(feat): first-class GriddedQuery tensor-product queries#184
Conversation
Validated 2D-linear separable path: per-axis (idx, alpha) anchors resolved once via _anchor_loc/_alpha_of, two passes reusing _linear_value_blend. Machine-eps vs point-wise; 0.80-1.04x vs hand-rolled separable baseline.
_LinearAnchor{T, Op, P}: public idx + Op-selected NamedTuple payload with
Val-dispatch virtual properties (house convention), matched _eval_anchor
kernel pair, _axis_anchor builder. Forwarder pin test asserts codegen
equality with a native-field twin (br/select/line counts).
test/Project.toml: add InteractiveUtils (needed by the forwarder pin
test's code_llvm introspection).
Batch builder resolves each axis once into an AoS anchor plan with an
identity flag (pass-elision hook). Output eltype pinned to point-wise
promotion incl. mixed target eltypes. Correctness matrix: {F64,F32} x
{Clamp,Extend} x {up,down,mixed,OOB,non-monotonic,M==1} + per-axis mixed
extraps.
Blend/gather pass kernels extracted as pool-agnostic pure functions; runtime argmin over the two order costs (measured seeds c_blend=0.2, c_gather=0.65 ns/elem) picks dim2-first (n1xN mid) vs dim1-first (Mxn2 mid). Forced-order test pins mathematical equivalence of both orders.
NoExtrap OOB throws DomainError naming the axis before any large buffer is touched; Wrap/Fill rejected with an explicit ArgumentError at dispatch (seam-aware anchors / OOB masks are roadmap items).
_gridded_linear_2d! writes into a user matrix (FI output-first convention); the pass intermediate comes from AdaptiveArrayPools so repeated calls have no O(MxN) allocation. DimensionMismatch on size, early return on empty axes, single-quantization mid eltype.
alpha in {0,1} blend columns degrade to copyto! (endpoint-exact kernel makes
this bit-equal); identity axis plans skip their pass entirely - one-axis
resizes run single-pass with no intermediate, double identity is a copy.
A/B verdict on the exact-node-heavy shape (512x512 data, 300x257 query,
255 exact-node hits on axis 2): WITH column-copy 64.2 us vs WITHOUT 69.4 us
(~7-8% faster, 3 repeats stable each side) - KEEP.
F64/F32 x {down,up,strong-up,mild-up,mixed} with forced-order A/B; verifies
the cost model picks the measured winner (or within 10% noise) on every
case. Calibrated c_blend/c_gather seeds recorded: F64 c_blend 0.15-0.25,
c_gather 0.44-0.47; F32 c_blend 0.08-0.13, c_gather 0.44-0.46 (existing
_GRIDDED_C_BLEND=0.2 / _GRIDDED_C_GATHER=0.65 already pick correctly in
every case; left unchanged).
… green Task 8 added BenchmarkTools to the ROOT Project.toml [deps]/[compat] so bench_gridded.jl's calibration section could use @belapsed. That makes BenchmarkTools a hard runtime dependency of the shipped package (every downstream Pkg.add installs it) even though src/ never imports it, and an unused [deps] entry fails test/test_aqua.jl's default stale-deps check. Revert both Project.toml additions and switch the calibration section to the file's own manual `best(f, reps)` timing helper (already used by the original 4-case table) instead of @belapsed. Timing mechanism only changes; MODEL OK/MISPICK logic, 10% agree-tolerance, and printed output are unchanged. Also fixes the stale top-of-file comment that no longer matched the (contradicted, now restored) BenchmarkTools-free design.
… branch+bit-exact tests - Split _gridded_linear_2d! into _gridded_plans (builds+validates axis anchors, owned Vectors, outside the pool scope) and _gridded_apply! (pool-scoped core that only consumes already-built plans). Both the allocating and in-place functors now build+validate plans BEFORE allocating/touching the O(M·N) output, so a NoExtrap OOB DomainError fires before any large buffer is acquired. - Replaced the byte-exact zero-alloc test bound (3515 B, one machine's measurement) with a proportional bound (4*(M+N)*16 + 1024) that tracks the true O(M+N) plan cost and stays far below the O(M·N) intermediate it must exclude. - Added a test for the p2.identity-only elision branch (axis-2 ≡ full grid, axis-1 resampled), the mirror of the existing p1.identity case. - Tightened the exact-node-column and axis-1-identity comparisons to strict == (both survive at full precision through the endpoint-exact kernel); added a light allocation assertion on the NoExtrap throwing path to pin the no-large-alloc structural guarantee.
- fused kernel: replace the same-cell run-scan with a branch-free plain loop — bit-identical, 26-61% faster at every ratio; fused now wins all pure downsampling and stays within 3x at strong upsampling - public GriddedQuery path evaluates via the fused strategy; two-pass full-buffer takes an explicit dim2_first fold order (auto cost models removed from src; benches carry their own choosers) - rename _AxisAnchorBatch -> _AxisPlan; length-gate the identity accumulation (plan-build overhead +27% -> +1.3% on range grids); _anchor_scalar_type accessor replaces positional A.parameters access - split experimental candidates (lazy Range plan, sliding-window cores, plan-agnostic fused fallback) into gridded_experimental.jl with measured verdicts in the header; drop dead non-pooled _gridded_plans
- drop _gridded_apply! (body was identical to _gridded_apply_fused_pooled!); the in-place functor calls the pooled entry directly - _gridded_eval stays as the one documented exception: plans (and NoExtrap validation) must be built before the O(M*N) output allocation, inside the same pool scope
Collapse the gridded path onto the one idea it adds — per-axis O(M+N)
anchors reused across O(M*N) outputs — consumed by exactly two strategies
picked per call: fused (pure downsampling) and two-pass full buffer
(otherwise, fold order from measured pass costs).
* plain _GriddedAnchor{T} (idx + alpha) replaces the op-payload anchor,
Val-dispatch forwarder, and _AxisPlan layering; extrap folding (Clamp
clamp / Extend raw / NoExtrap pre-alloc throw) is inlined in the builder
* identity elision and exact-node copy branches removed: the convex blend
is endpoint-exact, so node hits stay bit-identical to point-wise eval
(pinned by test); inert @simd annotation dropped (loop is ILP-bound)
* experimental candidates (lazy Range plan, sliding window) deleted with
their file; single src/gridded/gridded_query.jl remains
A/B vs 09147db (512x384 sweep): down2/down4/up2/mixed all within noise
(0.746/0.939/0.372/0.584 ns/out), warm allocs 0 on every path; only the
removed node-identity elision case regresses by design.
Replace the hand-written 2D fused loop with an N-generic generated kernel
consuming NTuple{N, Vector{_GriddedAnchor}}: loop d loads axis-d's anchor
at its own nesting level (the 2D hoisting, automated), and the corner
collapse emits the same blend tree the 2D code spelled out — axis-d blend
wrapping the two axis-d corners, (2^N - 1) blends per output. The
expression builder is a plain function defined before the generator; the
generator body only assembles expressions.
Same-process A/B vs the verbatim old 2D loop: bit-identical outputs and
1.000-1.010x runtime at every ratio (down2 0.726 vs 0.726 ns/out); public
down2 path unchanged at 0.744 ns/out, warm allocs 0. New 3D/4D kernel
tests pin correctness against point-wise eval (Clamp/Extend incl. OOB).
Swap the per-axis combine from the value blend to the shared 1D kernel _linear_kernel(op, yL, yR, inv_h, alpha) — the same collapse point-wise ND eval uses — so per-axis derivative ops fall out of the existing structure: * _GriddedAnchor gains inv_h (16 -> 24 B; geometry now computed with the same _get_inv_h/_alpha_of arithmetic as point-wise _locate_cell) * public entries take deriv (single op or per-axis tuple, resolved by _resolve_deriv_nd exactly like the scalar ND callable) * WrapExtrap: the anchor build folds coordinates via _anchor_loc's wrap path (shared _wrap_to_domain); closed-domain endpoint contract holds * FillExtrap: anchors clamp like Clamp for safe geometry, then a _gridded_fill_oob! post-pass overwrites OOB slabs mirroring the point-wise _try_fill_oob contract (first Fill axis's value for EvalValue, carrier zero for any deriv op); unsupported-extrap gate deleted — every extrap is now handled Pinned vs point-wise in new tests: deriv combinations on 2D public path (Clamp/Extend, vector grids) and 3D kernel, Wrap 2D/3D incl. mixed per-axis, Fill NaN/numeric/deriv/mixed. EvalValue perf unchanged (0.750/0.939/0.614/0.372/0.584 ns/out on the fixed sweep, allocs 0).
The whole pipeline is now generated/generic over N; the 2D-specialized code (blend_dim2/gather_dim1 passes, two-order fullbuffer, 2D apply/fill/ entries/functors) is deleted, subsumed by: * _gridded_pass! — one generated single-axis pass (axis D anchored, axis 1 innermost): Val(2) reproduces the old contiguous blend, Val(1) the old adjacent-pair gather, same arithmetic * N-pass fullbuffer with pass order from an adjacent-exchange cost comparator (gather ~0.65, contiguous ~0.2 ns/elem) that reduces EXACTLY to the old dim2-first model at N = 2; intermediates pooled, each pass a function barrier * strategy branch all(M_d < n_d) -> fused, else fullbuffer * fill post-pass via selectdim slabs; entries/functors take any N 2D behavior and perf unchanged (0.753/0.960/0.629/0.372/0.568 ns/out on the fixed sweep, within run-to-run noise; node bit-exactness tests still pass). 3D public path now live: 1.25/0.62/0.88 ns/out (down2/up1.5/ mixed), 5.3-10.5x over point-wise, warm allocs 0 everywhere.
g/t/ex -> grid/targets/extrap in the anchor builders, ts -> targets in entries and functors, nsz/msz/sz -> src_size/out_size/cur_size in the full-buffer core, fv/zref -> fill_value/data_sample in the fill pass. No behavior change; tests unchanged and green.
Add the extrap kwarg (single InBounds broadcast or per-axis tuple, resolved by _resolve_extrap_override_nd — same contract as point-wise ND eval, non-InBounds modes rejected). An InBounds axis skips the domain classification in the anchor build and takes the lean search directly; in-domain anchors are bit-identical to the default path's (pinned by test). The resolved extrap tuple threads through anchor build and the fill post-pass. Measured anchor-build saving: -42..-51% on range grids (1.5 -> 0.75 ns/anchor; classification was half the build), -8..-15% on Vector grids (binary search dominates). Build share of a full call is 0.02-2% at normal sizes, up to 32% for tiny outputs on Vector grids — the override's target regime.
On search-based (Vector) grids, when the mean consecutive-target stride is within 4 average cells, the anchor build chains a LinearBinarySearch hint (walk window 8) so each search starts at the previous anchor's interval. Range grids keep the O(1) direct arm; sparse targets keep plain binary. The searcher union splits at an @inline loop call — the inlining is load-bearing: it keeps the fresh RefHint inside the caller's compiled unit so escape analysis elides the Ref(1) heap allocation (same mechanism as _ensure_hint_nd on the scalar ND path); warm allocs stay 0 on every path. The threshold is measured, not guessed (n=2048 Vector grid, sorted targets): hinted/binary = 6.9x at 0.25-cell stride, 4.3x at 2, 1.2x at 6, and LOSES at 8 (walk exhaust + binary fallback) — 4 leaves margin for non-uniform cells. This targets exactly the M < n but CLUSTERED regime (zoom) that a size-ratio condition would miss: 256 targets in a 10-cell window build at 2.2 ns/anchor vs 11 binary (~5x); sparse full-span stays binary; range-grid sweep unchanged.
Each full-buffer pass now bounds its pass-through axes to the tap hull of the not-yet-resolved axes' anchors — slabs no downstream pass will read are never computed. A pure loop-bounds tightening of the existing strategy (identical arithmetic on computed entries, pinned bit-equal by test), unlike the retired sliding-window candidate which was a separate execution scheme. The hull comes from the query coordinates, not an anchor scan: separate minimum/maximum over the raw targets (SIMD, ~0.15 ns/elem; Base.extrema is ~34x slower and was end-to-end visible) mapped through two locates — valid because coordinate -> interval index is monotone for every non-wrapping extrap; Wrap keeps the full axis. Computed only on the full-buffer branch, so fused calls pay nothing. The pass call goes through a generated if-chain on the runtime axis id (_gridded_pass_dim!): N static call sites instead of one dynamic Val(d) dispatch, so the isbits ranges tuple crosses in registers (the dynamic form boxed it, 96 B/call) and the pass order tuple needs no pool buffer — the full-buffer core now touches the pool for intermediates only. Warm allocs stay 0 everywhere. Measured (512x384 -> 1024x768 fullbuffer): zoom 10x5 cells 2.65x, 50x40 2.28x, half domain 1.44x, full domain 1.00x (no regression); 3D zoom-up public path 0.620 -> 0.163 ns/out (3.8x), allocs 0.
Add a one-shot rectilinear-resize API that evaluates an N-D linear interpolant at every GriddedQuery axis combination without keeping a persistent interpolant alive. Decouple the gridded eval from LinearInterpolantND: _gridded_apply!, _gridded_fill_oob!, and _gridded_eval/! now take (grids, data) as a plain pair, and the persistent functor forwards itp.grids/itp.data (path and performance unchanged). The one-shot front resolves raw grids to the value-matched Tg in a pool-backed _CachedVector/_CachedRange (_cache_axis_pooled), released at call scope, so the gridded machinery sees exactly the axis type the persistent path feeds it and Vector grids pay no permanent cache allocation. Warm one-shot is zero-allocation apart from the output for every grid type (Range, Vector, OneTo) and bit-identical to the persistent functor. Scoped to NoExtrap/Clamp/Fill/Extend; WrapExtrap/periodic is reached by building the interpolant explicitly.
Display dimensionality, per-axis grid kind (Range/Vector), point count, and sampled [min, max] extent — mirroring the ND interpolant tree. min/max (not first/last) so an unsorted query axis reports its region honestly; empty axes render without throwing.
GriddedQuery threw an ad-hoc message without the domain bounds; route it through the shared _throw_domain_error so it reads like the scalar/ND path and reports the physical [lo, hi]. Add an optional axis index to _throw_domain_error (0 = axis-agnostic, 1D unchanged) and thread it through _validate_nd_domain via a thin _check_domain_axis wrapper, so ND scalar and batch NoExtrap OOB now name the offending axis too: query point on axis 2 outside interpolation domain [10.0, 20.0] The wrapper forwards the index only to the NoExtrap thrower; every other mode ignores it, so the 45 existing _check_domain call sites and the hot promotion path are untouched (no allocation change).
interp(grids, data, gq; method=LinearInterp()) previously fell through to the batch-query method and failed in _query_eltype (GriddedQuery is not a Real-vector container). Add GriddedQuery methods for the unified interp/ interp!: accept the query only when every axis method is LinearInterp (the gridded evaluator is linear-only today), forward per-axis bc from the methods, and reject any non-linear method with a clear ArgumentError. Placed in gridded_query.jl, not hetero/, because GriddedQuery is defined after hetero.jl is included whereas interp itself already exists there.
This reverts commit cd93fbd.
Make GriddedQuery flow through the existing generic interp path instead of a bespoke dispatch: it implements the ND query protocol — _query_length (product of axis lengths), _query_extract (column-major unravel to an N-tuple), _query_eltype (promoted axis eltype) — plus a new _query_size protocol hook (default (_query_length,), a flat vector) that a shaped container overrides. So interp(grids, data, gq; method=...) works for EVERY method (cubic, constant, mixed, ...) via pointwise batch and returns the N-D size(gq) array the GriddedQuery contract promises; ordinary SoA/ AoS batches keep their flat vector. interp! accepts any AbstractArray output and, for a shaped query, writes it through a flat alias view. A GriddedQuery on an all-LinearInterp method has a separable evaluator, so _try_gridded_separable! (dispatched, run before flattening) routes it to the gridded one-shot — per-axis anchors resolved once, 3-9x faster than pointwise — while every other query/method falls through; the hook writes the N-D output directly, so the fast path stays zero-allocation. Build the fullbuffer per-axis hull tuple with a @generated helper (_gridded_hulls) instead of an ntuple closure: the closure boxed a heterogeneous targets tuple (e.g. a UnitRange{Int} query axis beside a Vector), costing 3 allocs/call on that combination. Tests: protocol methods + column-major extract; interp routing for all method families; @which proof that GriddedQuery+linear resolves to the separable arm; fused and fullbuffer strategies both allocation-free including raw Int axes.
…adratic interpolation
There was a problem hiding this comment.
Pull request overview
Adds first-class GriddedQuery support for rectilinear (tensor-product) ND interpolation queries, enabling efficient resize-style evaluation (and shaped in-place output) by resolving per-axis anchors once and reusing them across the full output lattice.
Changes:
- Introduces the public
GriddedQuerytype plus query-protocol integration and REPL display support. - Adds separable gridded fast paths (linear/constant/local-Hermite/cubic/quadratic) and a unified persistent
itp(gq)/itp(out, gq)callable that returnssize(gq)-shaped output. - Expands test/benchmark coverage for correctness, allocation behavior, periodic seams, output type promotion, and error messaging.
Reviewed changes
Copilot reviewed 35 out of 35 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| test/test_show.jl | Adds show/REPL formatting tests for GriddedQuery. |
| test/test_nd_quadratic.jl | Adds regression test for N=1 quadratic ND collapse behavior across coeff strategies. |
| test/test_nd_output_type_promotion.jl | Pins output eltype behavior for ND constant vs linear batch results. |
| test/test_nd_noextrap_oob.jl | Adds tests for canonical axis-named NoExtrap OOB DomainError messages. |
| test/test_linear_blend_componentwise.jl | Adds inference/type-parity tests for colorant interpolation across scalar/ND/GQ paths. |
| test/test_gridded_query.jl | New comprehensive correctness/alloc/routing tests for GriddedQuery fast paths. |
| test/test_anchor_common.jl | Adds regression test for exclusive-periodic seam handling in anchor location. |
| test/Project.toml | Adds InteractiveUtils to support which usage in tests. |
| src/quadratic/nd/quadratic_nd_interpolant.jl | Adjusts N=1 quadratic ND entry to accept coeff strategy kw without breaking forwarding. |
| src/linear/linear_anchor.jl | Uses idxR from _anchor_loc to preserve exclusive-periodic seam pairs. |
| src/hetero/hetero_oneshot.jl | Refactors batch interp!/interp to support shaped outputs and a gridded fast-path hook. |
| src/hetero/hetero_nointerp.jl | Switches method-tuple normalization to _method_tuple. |
| src/hetero/hetero_interpolant.jl | Switches method-tuple normalization to _method_tuple. |
| src/gridded/gridded_query.jl | New core implementation of GriddedQuery + linear gridded evaluation and interp! hook for shaped output. |
| src/gridded/gridded_partials.jl | New fused-anchor gridded evaluation for cubic/quadratic partials-based methods. |
| src/gridded/gridded_hermite.jl | New separable fullbuffer gridded evaluation for local Hermite (PCHIP/Cardinal/Akima). |
| src/gridded/gridded_dispatch.jl | New unified persistent itp(gq) / itp(out,gq) entry for all ND interpolants. |
| src/gridded/gridded_constant.jl | New fused gather-only gridded evaluation for constant interpolation. |
| src/gridded/axis_anchor.jl | New shared _AxisAnchor backbone for per-axis anchor building across gridded families. |
| src/FastInterpolations.jl | Wires new gridded modules into the package and exports GriddedQuery. |
| src/cubic/cubic_oneshot_series.jl | Updates comments to reflect new seam-aware anchor behavior expectations. |
| src/cubic/cubic_anchor.jl | Uses idxR from _anchor_loc to preserve exclusive-periodic seam pairs. |
| src/core/utils.jl | Extends domain error helpers to optionally include axis index in messages. |
| src/core/show.jl | Adds show/show(::MIME"text/plain") support for GriddedQuery. |
| src/core/query_protocol.jl | Adds _query_size protocol + threads axis index into NoExtrap domain validation. |
| src/core/interpolant_protocol.jl | Factors shared point-wise batch core for reuse by GriddedQuery fallback. |
| src/core/interp_method_types.jl | Adds _method_tuple helper to normalize scalar vs per-axis method tuples. |
| src/core/anchor_common.jl | Extends _AnchorLoc to carry idxR for exclusive-periodic seam correctness. |
| src/constant/constant_anchor.jl | Uses idxR from _anchor_loc to preserve exclusive-periodic seam pairs. |
| ext/FastInterpolationsColorVectorSpaceExt.jl | Adds extrap-result promotion helpers to keep widened colorant types consistent on OOB shortcuts. |
| docs/src/nd/overview.md | Documents GriddedQuery as an ND query mode and clarifies scalar call forms. |
| docs/src/guides/performance_tips.md | Adds performance guidance for using GriddedQuery for tensor-product ND workloads. |
| docs/src/api/types.md | Documents GriddedQuery in the public API type listing. |
| benchmark/ci_benchmark.jl | Adds CI benchmark group 15_gridded_query for persistent/one-shot shaped evaluation. |
| bench_gridded.jl | Adds a standalone benchmarking script comparing gridded vs hand-rolled vs point-wise tensor evaluation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #184 +/- ##
==========================================
+ Coverage 96.43% 96.45% +0.02%
==========================================
Files 151 157 +6
Lines 12905 13502 +597
==========================================
+ Hits 12445 13024 +579
- Misses 460 478 +18
🚀 New features to boost your workflow:
|
FastInterpolations.jl Benchmarks
All benchmarks (56 total, click to expand)
|
| Benchmark | Current | Previous | Imm. Ratio | Grad. Ratio | Tier |
|---|---|---|---|---|---|
13_nd_oneshot_gridquery/bilinear_2d_sort_rand |
10118.1 ns |
9975.8 ns |
1.014 |
1.11 |
gradual |
Thresholds: immediate > 1.1x (vs latest master), gradual > 1.1x (vs sliding window)
Runner:
znver3|4c— AMD EPYC 7763 64-Core Processor, julia 1.12.6. Times are min-merged and compared only against this same machine's history.
This comment was automatically generated by Benchmark workflow.
…shot interpolation
… streamline interval search logic
…ounding near clamped boundaries
Summary
Adds
GriddedQuery, a rectilinear tensor-product query container for ND interpolation. Resize-style calls can now evaluateM1 x M2 x ... x MNquery axes directly into shaped output instead of expanding the query grid into a point-wise batch. This is the natural API for image resize, resampling a coarse field onto a denser grid, or sampling a surface/volume on a regular output lattice.The fast path covers the main separable families: linear, constant, local Hermite (
pchip/akima/cardinal), and cubic/quadratic partial-based interpolation. Persistent interpolants and one-shot calls both route through the GriddedQuery path when a supported method tuple is present; unsupported mixes still fall back to the generic point-wise batch path.API quick start
Mental model:
(xqs, yqs)is a pairwise ND batch:(xqs[i], yqs[i])->VectorGriddedQuery(xq, yq)is a tensor-product batch: every(x, y)pair ->MatrixPrinted in the REPL:
Allocating and in-place one-shot:
The in-place output is intentionally shaped: for
N > 1, pass anN-D array withsize(out) == size(gridded_query). A flat vector is not reshaped automatically.Changes
GriddedQueryAPI, query-protocol integration, display support, and shaped in-place output._AxisAnchor-based per-axis anchor pipeline with pooled storage, hint-chained search, fused downsampling, and full-buffer strategies.N-dimensional. Flat vectors no longer silently reshape forN > 1.N = 1coeff forwarding, generic SoA output-type promotion pins, Wrap/NoExtrap gridded coverage, and stale docstring cleanup.15_gridded_queryadds small regression sentinels for linear persistent/one-shot 2D and 3D, plus cubic persistent/one-shot 2D.Benchmark
Local BenchmarkTools min on Apple M1 Pro, persistent interpolants, allocating output in both cases. Source grid is
64 x 64; query isGriddedQuery(range(..., 128), range(..., 96)), i.e. 12,288 output points. The naive baseline is the equivalent scalar comprehension:GriddedQueryThe allocation is intentionally the same in this table: both forms allocate the returned
128 x 96matrix. The win is thatGriddedQueryresolves each query axis once and reuses those anchors across the tensor-product output instead of doing a full scalar locate/kernel call per pixel.