ENH: Out-of-core architecture rewrite and filter optimizations#1568
Draft
joeykleingers wants to merge 83 commits into
Draft
ENH: Out-of-core architecture rewrite and filter optimizations#1568joeykleingers wants to merge 83 commits into
joeykleingers wants to merge 83 commits into
Conversation
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
from
b4ef97f to
99b49ed
Compare
joeykleingers
requested review from
JDuffeyBQ,
imikejackson,
jmarquisbq,
mmarineBlueQuartz and
nyoungbq
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
2 times, most recently
from
b4ef97f to
bb09048
Compare
This was referenced Mar 24, 2026
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
4 times, most recently
from
102c436 to
b4c1358
Compare
joeykleingers
changed the title
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
6 times, most recently
from
2bd614a to
110c054
Compare
joeykleingers
changed the title
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
4 times, most recently
from
35aecd0 to
3a88bbf
Compare
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
from
bdfed87 to
6fbfc8d
Compare
…uresDirect for OOC dispatch Prepares the RemoveFlaggedFeatures algorithm for OOC dispatch by renaming the existing in-core implementation to RemoveFlaggedFeaturesDirect. No logic changes; the InputValues struct and enum are unchanged. A thin RemoveFlaggedFeatures dispatcher plus an OOC-optimized RemoveFlaggedFeaturesScanline variant will be added in a follow-up commit.
…anline dispatch Adds a thin RemoveFlaggedFeatures dispatcher plus a RemoveFlaggedFeaturesScanline algorithm that eliminates all per-voxel operator[]/copyTuple access on the (potentially out-of-core) FeatureIds array and its companion cell arrays: - The neighbor-vote pass (IdentifyNeighbors equivalent) reads FeatureIds through a 3-slice rolling window (copyIntoBuffer), matching ComputeBoundaryCellsScanline's approach, instead of random single-element reads whose +/-Z offset is a full Z-slice away. - The data-transfer pass (FindVoxelArrays equivalent) reuses the shared SliceBufferedTransfer utility to copy winning-neighbor data into every removed voxel for FeatureIds and all companion arrays, one Z-slice at a time, instead of per-voxel copyTuple calls. - FlagFeatures is rewritten as a chunked bulk copyIntoBuffer/copyFromBuffer scan instead of per-voxel operator[] read/write. The per-voxel neighbor-vote mapping remains a single O(n_cells) std::vector<int64> (documented in RemoveFlaggedFeaturesScanline.hpp): the vote pass must run to completion across the whole volume, strictly read-only, before any voxel is written, to preserve the Direct algorithm's exact convergence and tie-breaking behavior. This buffer is plain heap memory, not an OOC store, so it does not itself cause chunk thrashing. RemoveFlaggedFeaturesTest.cpp now runs its three execution-based TEST_CASEs through both the in-core and forced-OOC algorithm paths via ForceOocAlgorithmGuard + GENERATE(from_range(k_ForceOocTestValues)).
…eature accumulation
…two-pass streaming
…anline via rolling window
…ute Array Calculator
Track the observed FeatureIds range while streaming image and rectilinear-grid chunks, avoiding a separate full-array OOC scan while preserving validation errors. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
This change introduces storage-aware execution for data-intensive filters, preserving direct contiguous access for in-memory arrays while using bounded bulk transfers for disk-backed stores. It also converts histogram modal analysis and additional filter algorithms to chunked I/O and adds targeted correctness and large OOC benchmark coverage. 58 files changed, +8,494 / -2,103 lines.
================================================================================
1. Storage-aware direct and scanline algorithms
================================================================================
Files:
- SimplnxCore/CMakeLists.txt
- ComputeBoundingBoxStats{,Direct,Scanline}.*
- ComputeCoordinateThreshold{,Direct,Scanline}.*
- ComputeFeatureBounds{,Direct,Scanline}.*
- ComputeLargestCrossSections{,Direct,Scanline}.*
- CreateColorMap{,Direct,Scanline}.*
- PartitionGeometry{,Direct,Scanline}.*
Route each algorithm through storage-aware dispatch based on its participating arrays. Direct implementations retain or accelerate parallel contiguous-store processing for in-memory data, while scanline implementations use fixed-size row, plane, or chunk buffers for sequential out-of-core access.
Keep scratch storage independent of cell count for coordinate masks, feature bounds, cross sections, color generation, and geometry partitioning. Compute exact bounding-box frequency statistics with bounded reads and temporary external merge sorting instead of cell-sized in-memory collections.
================================================================================
2. Bounded histogram and modal processing
================================================================================
Files:
- ComputeArrayHistogram.cpp
- ComputeArrayHistogram.hpp
Stream masked and unmasked inputs through explicit range and binning passes using fixed-size buffers. Preserve direct modal calculation for in-memory stores and use temporary sorted runs plus bounded merge passes for out-of-core mode detection, including all-distinct inputs.
Write bin ranges and counts in bulk, retain overflow reporting, and add throttled progress and cancellation checks throughout the multi-pass work.
================================================================================
3. Chunked bulk-I/O conversions
================================================================================
Files:
- ReadChannel5Data.*
- CombineAttributeArrays.*
- ComputeDifferencesMap.*
- ComputeFeatureRect.cpp
- ExtractFeatureBoundaries2D.cpp
Replace per-element datastore traffic with bounded contiguous transfers. Chunk Channel 5 field and compatibility-array writes, combine and normalize attribute arrays through reusable buffers, stream difference-map and feature-rectangle inputs, and detect 2D feature boundaries with rolling row buffers.
Propagate bulk-transfer failures and add progress and cancellation checks at chunk boundaries without materializing full cell arrays.
================================================================================
4. Correctness and OOC benchmark coverage
================================================================================
Files:
- OrientationAnalysis/test/ReadChannel5DataTest.cpp
- SimplnxCore/test/*Test.cpp for the affected filters
Add hidden large-data OOC benchmarks that construct disk-backed inputs with bulk writes and validate analytical output values, ranges, colors, geometry, and feature IDs. Extend focused coverage for normalization, masked reads across chunk boundaries, all-distinct modal values, coordinate-threshold chunk boundaries, and both storage-dispatched paths.
================================================================================
Verification
================================================================================
- git diff --cached --check
- Build and tests not run; this is a history-only squash.
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
from
c6bb7fe to
ee6b370
Compare
Optimize six OrientationAnalysis and SimplnxCore filters for their backing datastore. Use bounded bulk I/O for out-of-core arrays and contiguous-memory execution where a direct path is beneficial. Add large-volume coverage for both forced algorithm paths. 18 files changed, +2458 / -365 lines. ================================================================================ 1. Bounded out-of-core algorithms ================================================================================ Files: * OrientationAnalysis: ComputeFZQuaternions, ComputeMisorientations * SimplnxCore: ComputeFeaturePhasesBinary, ComputeVectorColors, ConditionalSetValue, ConvertColorToGrayScale Process cell-level arrays through reusable fixed-size buffers and bulk datastore transfers instead of per-tuple out-of-core access. Preserve filter semantics for masks, phase lookups, feature assignment, value replacement, and grayscale conversion while propagating transfer errors. Add chunk-level cancellation and progress reporting, and cache small lookup data or feature-indexed output state where appropriate. ================================================================================ 2. Contiguous in-memory paths ================================================================================ Files: * ComputeFZQuaternions * ConditionalSetValue * ConvertColorToGrayScale Dispatch compatible in-memory stores to direct implementations that use contiguous pointers and parallel ranges. Retain abstract-store fallbacks so forced direct execution remains testable on non-contiguous stores, and precompute reusable phase operations outside tuple loops. ================================================================================ 3. Large-volume dispatch coverage ================================================================================ Files: * ComputeFZQuaternionsTest.cpp * ComputeMisorientationsTest.cpp * ComputeFeaturePhasesBinaryTest.cpp * ComputeVectorColorsTest.cpp * ConditionalSetValueTest.cpp * ConvertColorToGrayScaleTest.cpp Add hidden 200x200x200 OOC benchmarks that exercise both forced algorithm selections. Construct inputs and validate complete outputs through bounded slice buffers while checking tuple shapes, component counts, masks, and expected datastore types. ================================================================================ Verification ================================================================================ * git diff --cached --check * Build and tests not run; this is a history-only squash. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
…e type Adds a virtual getPlannedStoreType() to IDataStore that returns the store type an array will have once it is materialized. The default returns getStoreType(); EmptyDataStore overrides it to report InMemory or OutOfCore based on the intended data format recorded during preflight. This lets callers reflect the eventual backing of a preflight-only placeholder array without dispatching on the placeholder's element type. Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
Optimize eight OrientationAnalysis and SimplnxCore filters for their backing datastore. Dispatch contiguous stores to direct implementations and stream disk-backed stores through bounded bulk transfers. Add large-volume coverage for both forced algorithm paths. 25 files changed, +3479 / -341 lines. ================================================================================ 1. Storage-aware algorithm execution ================================================================================ Files: * OrientationAnalysis: ConvertQuaternion, WriteStatsGenOdfAngleFile * SimplnxCore: AddBadData, ChangeAngleRepresentation, ComputeCoordinatesImageGeom, SplitDataArrayByComponent, SplitDataArrayByTuple, WriteVtkStructuredPoints Use direct contiguous pointers and parallel ranges for compatible in-memory stores. Route out-of-core and forced scanline execution through fixed-size buffers, bulk datastore reads and writes, chunk-level cancellation, progress reporting, and transfer-error propagation. Retain bounded fallbacks when a forced direct path receives a non-contiguous store. ================================================================================ 2. Filter behavior and writer compatibility ================================================================================ Files: * AddBadData.cpp * SplitDataArrayByComponent.cpp * SplitDataArrayByTuple.cpp * WriteStatsGenOdfAngleFile.cpp * WriteVtkStructuredPoints.cpp Preserve seeded boundary and Poisson draw order while updating only selected tuples. Copy component and N-D tuple splits in row-major chunks without materializing full outputs. Stream phase-angle and VTK ASCII or binary payloads with bounded memory, endian conversion, explicit file errors, and compatibility helper APIs. ================================================================================ 3. Large-volume dispatch coverage ================================================================================ Files: * OrientationAnalysis ConvertQuaternion and StatsGen ODF tests * SimplnxCore algorithm tests and test registration Add 200x200x200 OOC benchmarks for all eight filters and exercise both forced algorithm selections. Populate and validate data through bounded slice buffers, check complete array results and file payloads, and register the new VTK structured-points test. ================================================================================ Verification ================================================================================ * git diff --cached --check * Build and tests not run; this is a history-only squash. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Replace element-wise datastore access in six filters with bounded bulk transfers and dispatch-backed direct paths where applicable. Cache feature-level data for microtexture grouping, add cancellation and progress reporting, and retain direct parallel execution for in-core stores. Add large OOC benchmarks that validate both forced algorithm paths. 18 files changed, +1769 / -279 lines. ================================================================================ 1. Bounded SimplnxCore filter transfers ================================================================================ Files: ComputeBoundaryElementFractions, CreateFeatureArrayFromElementArray, ExtractComponentAsArray, ComputeMomentInvariants2D, and ConvertData algorithms. Stream cell and value arrays through fixed-size copyIntoBuffer and copyFromBuffer transfers instead of per-element datastore access. Dispatch ConvertData, ExtractComponentAsArray, and ComputeMomentInvariants2D between direct in-memory implementations and scanline OOC implementations while preserving type conversion, extraction, reduction, and moment calculations. Propagate bulk-I/O errors and report cancellable progress at chunk boundaries. ================================================================================ 2. Cached microtexture grouping data ================================================================================ Files: GroupMicroTextureRegions.cpp and GroupMicroTextureRegions.hpp. Cache feature phases, parent IDs, crystal structures, quaternions, and volumes before grouping so random feature access remains in memory. Perform final cell parent remapping with bounded bulk transfers, keep randomized parent IDs in the cache until writeback, and check cancellation during grouping and remapping. ================================================================================ 3. Forced-path OOC benchmark coverage ================================================================================ Files: ComputeBoundaryElementFractionsTest.cpp, ComputeMomentInvariants2DTest.cpp, ConvertDataTest.cpp, CreateFeatureArrayFromElementArrayTest.cpp, ExtractComponentAsArrayTest.cpp, and GroupMicroTextureRegionsTest.cpp. Add large deterministic fixtures under the hidden OOC benchmark tag. Exercise both direct and forced-OOC algorithm dispatch, require the expected datastore type, and validate complete analytical outputs for fractions, moments, conversions, feature values, component transfers, and parent-ID remapping. Verification: Tests were not run as part of this history-only squash.
Convert large array imports, exports, and quaternion conjugation to bounded bulk I/O while retaining direct in-core paths for hot loops. Harden cancellation, error propagation, large-file offsets, VTK token parsing, CSV buffer flushing, and deferred legacy string-array creation. Add deterministic large OOC benchmarks covering direct and forced dispatch paths. 24 files changed, +2272 / -447 lines. ================================================================================ 1. Storage-aware computation and export ================================================================================ Files: ComputeQuaternionConjugate algorithms and CMake registration, WriteINLFile.cpp/.hpp, and WriteLosAlamosFFT.cpp/.hpp. Split quaternion conjugation into a parallel direct implementation and a fixed-capacity scanline implementation selected by datastore-aware dispatch. Stream INL and Los Alamos FFT inputs in bounded tuple chunks for disk-backed arrays while preserving contiguous direct access for RAM-backed arrays. Buffer formatted output where appropriate, retain coordinate and material metadata semantics, report progress, honor cancellation, and return file-write failures. ================================================================================ 2. Bounded import and parsing paths ================================================================================ Files: ReadBinaryCTNorthstar.cpp, ReadCSVFile.cpp, ReadVtkStructuredPoints.cpp/.hpp, FileUtilities.cpp/.hpp, and Dream3dIO.cpp. Bulk-initialize and populate Northstar density rows with checked large-file seeks. Buffer numeric CSV columns for contiguous datastore writes while preserving direct in-core and string assignment paths, flushing partial data on completion, cancellation, and parse errors. Replace VTK element-wise parsing with bounded ASCII and binary readers that preserve section boundaries, validate truncation and token limits, byte-swap binary chunks, and support vector arrays. Use empty legacy string stores during deferred metadata import. ================================================================================ 3. Forced-path OOC benchmark coverage ================================================================================ Files: ComputeQuaternionConjugateTest.cpp, WriteINLFileTest.cpp, ReadBinaryCTNorthstarTest.cpp, ReadCSVFileTest.cpp, ReadVtkStructuredPointsTest.cpp, and WriteLosAlamosFFTTest.cpp. Add large deterministic fixtures under the hidden OOC benchmark tag. Exercise both direct and forced-OOC algorithm paths, require expected datastore types, and validate complete quaternion, density, CSV, VTK, INL, and FFT outputs while checking input preservation and temporary-file cleanup. Verification: Builds and tests were not run as part of this history-only squash.
…Array Dream3dPreflightCache::RefreshStores isolated each cache handout's StringArray by rebuilding its store from StringArray::values(). The OOC architecture changed preflight string arrays to be backed by an EmptyStringStore placeholder whose element accessors throw, so values() (which iterates the store) threw "EmptyStringStore::operator[] called on placeholder store - data not loaded yet" on every fetch of a file containing a StringArray. This surfaced as a preflight failure when reading e.g. SmallIN100_Final.dream3d, whose ensemble MaterialName array is a StringArray; files with only numeric arrays were unaffected. Rebuild a placeholder EmptyStringStore for placeholder-backed arrays instead of reading their values; materialized stores are still rebuilt from values. Rework the now-invalid StringArray isolation test (which relied on the removed mutable StringStore) to prove isolation via resizeTuples, and add a regression test that fetching a file containing a StringArray succeeds. Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch does two things at once:
Reworks the core out-of-core (OOC) data-storage architecture in
simplnxso that disk-backed storage is supplied by a runtime-registered IO manager through clean interfaces, with the core library holding no reference to any concrete OOC format or symbol. Core gains a bulk-I/O data-store API (copyIntoBuffer/copyFromBuffer), an injectable store-format resolver, IO-manager lifecycle hooks, a tri-state storage-mode preference, a memory-budget manager, and memory-safety guards. The OOC implementation itself (chunked HDF5 stores, the chunk cache, deflate decompression) lives in a separate privateSimplnxOocsource set and is not part of this diff — it plugs into core at runtime through these interfaces.Adds chunk-aware (OOC-optimized) algorithm variants for ~50 filters across the SimplnxCore and OrientationAnalysis plugins, plus shared core utilities that benefit many more. Random-access algorithms that thrash the chunk cache on disk-backed arrays now either stream data with bulk I/O or dispatch to a sequential "scanline / CCL" variant, eliminating per-element virtual dispatch and HDF5 chunk churn. In-core behavior and performance are preserved via a runtime storage-type check, so optimized filters keep two code paths: one for in-memory data, one for disk-backed data.
Scope of this diff
develop…HEAD: 378 files, +42,853 / −13,112.src/simplnx(core library)Plugins/SimplnxCorePlugins/OrientationAnalysisHow to review
The diff is large but highly patterned, and the two layers are independent:
src/simplnx/DataStructure/IO/Generic/*,Core/Preferences.*,Utilities/{MemoryBudgetManager,AlgorithmDispatch,SegmentFeatures,UnionFind,SliceBufferedTransfer}.*, andFilter/IFilter.cpp(Part 1).Part 1 — Core out-of-core architecture (
src/simplnx)1.1 Bulk-I/O data-store API
copyIntoBuffer/copyFromBufferonAbstractDataStore<T>(a flat range form and a tuple-addressed form) are the single mechanism for bulk reads/writes, implemented byDataStore<T>andEmptyDataStore<T>; the OOC implementation supplies its own override.IDataStore::StoreType { InMemory, OutOfCore, Empty }, queried viagetStoreType(). The chunk-traversal API (loadChunk,getNumberOfChunks,getChunkLowerBounds,getChunkUpperBounds,getChunkShape) and the chunk-shape-based OOC detection are removed — bulk I/O plusgetStoreType()replace them entirely.EmptyStringStore,AbstractStringStore/StringStoreplaceholder support, andStringArray::isPlaceholder().1.2 Runtime store-format resolution and IO-manager hooks
OOC capability is supplied entirely at runtime; core never names the concrete format.
IDataStoreFormatResolver: a const, thread-safe policy interface deciding which registered format a soon-to-be-created array uses (""= in-memory).InMemoryFormatResolveris the default policy.ArrayCreationUtilities::ResolveStorageFormatis the single decision point for every array-creation call site, with a fixed precedence: the unstructured/poly-geometry gate (ParentGeometrySupportsOoc) forces in-core, then an explicit per-filter format override, then theDataStructure's resolver.DataStructurecarries a per-instance resolver plus a lazily-seeded process-wide default (neither serialized).IDataIOManagerlifecycle hooks (default no-ops):finalizesImport,onImportFinalize,onRecoveryWrite,onFinalizeStores,setBaseDirectory,shutdownManager.DataIOCollectionfans these out to every registered manager, so an OOC manager participates in import finalization, recovery writes, store read-only transition, and shutdown without core knowing the specifics.CoreDataIOManager: the always-present default manager that registers the in-memory data-store/list-store factories; an OOC manager registers on top at runtime.CreateDataStore/CreateListStoresingle entry points;CreateNeighborListAction/CreateNeighborsthread an explicitdataFormatoverride through to the store.1.3 Storage-mode preference (+ legacy migration)
DataStorageMode { Adaptive, ForceInCore, ForceOutOfCore }(persisted asdata_storage_mode) is the single source of truth for storage placement, replacing thelarge_data_format/force_ooc_datapreferences. The enum is deliberately OOC-vocabulary-free — core states user intent; the registered manager maps it to a concrete format.dataStorageMode()migrates older preference files from the retained legacy keys;useOocData()is a convenience view (true unlessForceInCore).Preferencesseeds the OOC base directory and default format on startup via the IO-collection hooks, and gains aremoveValuehelper.1.4 Memory budget + memory safety
MemoryBudgetManager: tracks the budget governing in-core vs OOC placement.defaultBudgetBytes()(50% of RAM, ≥1 GiB, clamped to the cap),maxBudgetBytes()(max(min(total−6 GiB, 0.95·total), 1 GiB)), andsetBudgetBytes()which clamps to the cap and reports whether it clamped. Total-RAM detection is centralized inMemory::GetTotalMemory(). The default is clamped to the cap so it never exceeds it on <12 GiB machines (e.g. CI runners).nxrunner --memory-budget <GB>: CLI flag wired through to the budget manager so the override takes effect in headless runs, with parsing hardened against NaN/inf/trailing garbage.-264total-RAM hard block):-271— non-blocking preflight warning when an in-core array would exceed currently-available RAM. OOC arrays are excluded (EmptyDataStore::memoryUsage()reports 0 for OOC placeholders; the format is resolved in preflight via the sharedResolveStorageFormathelper).-272— astd::bad_allocsafety net at the singleIFilter::execute → executeImplboundary, turning an out-of-memory condition into a clean pipeline error instead of a crash.1.5 HDF5 streaming write + compression
DatasetIOprovides the streaming OOC write path —createEmptyDataset+writeSpanHyperslabfor arrays too large to be resident — alongside the single-shotwriteSpan; reads usereadChunk/readChunkIntoSpan.createEmptyDatasetbuilds the chunked-deflate creation property list (BuildChunkedDeflateDcpl), so a large OOC array taking the two-step streaming write is compressed identically to the single-shot path. Contiguous storage is still used for compression level 0 and for arrays below the small-array threshold.1.6
.dream3dloader APIDream3dIOexposes aLoadDataStructurefamily:LoadDataStructure,LoadDataStructureMetadata(metadata-only for preflight),LoadDataStructureArrays(array subset), plus resolver-aware overloads that stamp a per-DataStructureresolver before import finalization (so a read-only visualization load can direct arrays to disk for fast first-show). Import is eager or deferred based onanyManagerFinalizesImport(); writes run under a recovery-write guard supplied by the registered manager.ImportH5ObjectPathsActionuses this API: metadata-only on preflight, full load on execute, with a selective shortest-path-first merge of only the requested paths.1.7 Core algorithm infrastructure
AlgorithmDispatch.hpp—DispatchAlgorithm<InCoreAlgo, OocAlgo>(arrays, args…), a free function template selecting between an in-core and an OOC algorithm class at runtime. Priority:ForceInCoreAlgorithm()> any array OOC >ForceOocAlgorithm()> in-core default. RAII test guardsForceOocAlgorithmGuard(bool)andForceInCoreAlgorithmGuardlet a single build exercise both paths.SegmentFeaturesOOC path —executeCCL(): Z-slice connected-component labeling with a 2-slice rolling buffer andUnionFindequivalence tracking (Face and FaceEdgeVertex connectivity, optional periodic BCs), replacing random-access BFS/DFS flood-fill on disk-backed data.UnionFind— vector-based disjoint set with union-by-rank and path-halving.SliceBufferedTransfer— type-dispatched Z-slice buffered tuple copy for morphological / neighbor-replacement transfer phases.Extent— region/range math helper (with unit tests).AlignSectionsOOC path — bulk slice read/write transfer for the align-sections family.1.8 Core utility bulk-I/O conversions
These live in core utilities and benefit every caller; each is guarded by a runtime storage-type check that preserves the original in-core code path:
DataArrayUtilities—ImportFromBinaryFile,AppendData,CopyData, and the mirrorswap_rangesops route through chunked bulk I/O when OOC. (PowersReadRawBinaryandAppendImageGeometry's mirror.)DataGroupUtilities::RemoveInactiveObjects— chunked featureIds renumbering viacopyIntoBuffer/copyFromBuffer.ClusteringUtilities::RandomizeFeatureIds— chunked bulk I/O (both overloads; benefits segmentation filters, SharedFeatureFace, MergeTwins).GeometryHelpers—FindElementsContainingVert/FindElementNeighborsuse 65K-element chunked passes with a current-chunk cache-hit check before falling back to per-element reads.ImageRotationUtilities— Z-slab source cache for nearest-neighbor and a ±2-slice trilinear margin, sliding-window slab updates (memmove + delta reads), and intra-slice parallelism. This is howApplyTransformationToGeometryandRotateSampleRefFrameget their OOC speedups (no plugin algorithm file changes for ApplyTransformation).TriangleUtilities— bulk-load triangles/labels for winding repair.H5DataStore— streaming row-batchFillOocDataStorereplacing full-dataset allocation on import.RectGridGeom/ImageGeomfindElementSizes— route through the resolver-awareCreateDataStore(voxel-sizes array can go OOC); the RectGrid inner loop refactored to per-axis precompute + Z-slicecopyFromBuffer.Part 2 — Filter optimizations
Optimization patterns
Every filter optimization is one of these four shapes:
DispatchAlgorithm<…Direct, …Scanline>: an in-coreDirectclass (unchanged or parallel) and an OOCScanlineclass that streams Z-slices / chunks. The originalFoo.cppbecomes a thin dispatcher.DispatchAlgorithm<…BFS, …CCL>(or an in-file branch): random-access flood-fill in-core, sequential Z-slice connected-component labeling for OOC.std::vectors, with a runtime storage-type check so in-core stays optimal.SliceBufferedTransfer, or an OOC-correctness guard (e.g. disabling threading for OOC stores) / progress-and-cancel additions.Algorithm structure: in-core + out-of-core variants
For dispatch-split filters the original algorithm file is renamed to the in-core variant and the OOC variant is added beside it; the original filename remains as a thin dispatcher:
FillBadDataBFSFillBadDataCCLIdentifySampleBFSIdentifySampleCCL…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Worklist…Scanline…Direct…ScanlineNew shared header
IdentifySampleCommon.hpp(SimplnxCore) provides theVectorUnionFindand per-slice functor shared by the BFS/CCL variants;TupleTransfer.hppgainsquickSurfaceTransferBatch/surfaceNetsTransferBatchbulk APIs used by the mesh Scanline variants.SimplnxCore inventory
DispatchAlgorithm<FillBadDataBFS, FillBadDataCCL>; CCL streams Z-slices with coreUnionFindDispatchAlgorithm<IdentifySampleBFS, IdentifySampleCCL>; scanline labeling viaVectorUnionFindSegmentFeatures::executeCCL()(Z-slice CCL)copyIntoBuffer(256K-tuple)canMergefindClusters; bounded per-cluster peak memorynodeIdsfor rolling 2-plane node buffers;quickSurfaceTransferBatchsurfaceNetsTransferBatchstd::vectoraccumulators (no DataStore); 64K-tuple chunked featureIdsassignBadVoxelsvoting; sparse changed-voxel trackingtriMaskbitset + sparsetriPrefixSumpopcount (~6.4× less memory); 65K-element streamed passesZSliceWorkerparallel Z-slice rasterize → mutex-guarded bulkcopyFromBufferSliceBufferedTransferper-Z commit + Z-slice neighbor readsSliceBufferedTransferper-Z commitSliceBufferedTransferper-Z best-neighbor commitfindShiftsOoc()with per-Z-slice mask readscopyIntoBufferfor coordinate/data outputOrientationAnalysis inventory
DispatchAlgorithm<…Direct, …Scanline>; Direct keeps parallel in-core, Scanline 65K-tuple chunks + cached crystal structures; color key forwarded to the OOC pathDispatchAlgorithm<…Worklist, …Scanline>; bool-mask reads routed through bulk I/OSegmentFeatures::executeCCL()slice-by-slice, replacing DFS flood-fillfindShiftsOoc(), 2-slice buffered quats/phases/mask + cached crystal structuresSliceBufferedTransfer; cached crystal structures[plane−kZ, plane+kZ]slab reads; cached crystal structurescopyFromBuffertriIncluded); feature-level caching; raw-pointer parallel selectorfmt::memory_buffercopyFromBufferfor all cell arrays; chunked Euler interleave; in-place phase validationcopyFromBufferin CopyData template, phase copy, Euler interleavecopyFromBufferfrom raw HDF5 reader bufferscopyFromBufferfor intensity/image outputsCancel + progress
In-core and OOC variants gained
m_ShouldCancelchecks at the top of major loops andThrottledMessenger-based progress reporting with per-phase messages and percent-complete.Part 3 — Performance results
All benchmarks use arm64 Release builds. OOC measurements force the disk-backed path through
DataStorageMode::ForceOutOfCoreorForceOocAlgorithmGuard.CRITICAL filter optimizations (existing unit-test data, filter.execute() only)
Small existing datasets do not expose every asymptotic win: mesh bucketing/pruning and bounded-memory streaming can add fixed overhead at this scale. All 17 filters passed their affected suites in both configurations; the table reports measured values rather than hiding those small-test regressions.
Large synthetic filter benchmarks (filter.execute() only)
HIGH and MEDIUM benchmarks use 8,000,000 cells (200³) except the 2D-only tests (2048²), the generated Channel5 reader dataset (8,000 × 1,000), and CSV's 8,000,000 rows. Figures are medians of five execute-only runs for the MEDIUM batches and repeated runs for HIGH where practical;
>values are conservative timeout bounds. Every listed filter passed its normal affected suite and hidden large benchmark in both configurations.Additional 200³ execute-only OOC benchmarks
Full-test wall-clock OOC benchmarks
Geometry / Mesh / Phase Filters
ImageRotationUtilitiesslab cachePart 4 — Test infrastructure
UnitTestCommon.hpp):CompareDataArrays,CompareDataArraysByComponent,CompareArrays,CompareFloatArraysWithNansstream both arrays in 40K-element chunks viacopyIntoBufferinstead of per-elementoperator[](per-element access on OOC arrays is pathologically slow).ExpectedStoreType()(derives the expectedStoreTypefrom the activeDataStorageMode+ whether an OOC manager is registered) andRequireExpectedStoreType().PreferencesSentineltakes aDataStorageModeand restores the original preferences on destruction.TestFileSentinelreference-counts archive extraction via per-process holder files, so parallel test runs don't delete shared decompressed data prematurely.LoadDataStructuretest helper callsDREAM3D::LoadDataStructure(path)directly.SegmentFeaturesTestUtils.hpp(~638 lines): shared builders/verifiers for the SegmentFeatures family.ForceOocAlgorithmGuard+GENERATE(from_range(k_ForceOocTestValues))runs each case in both in-core and forced-OOC modes — adopted by 23 plugin test files (16 SimplnxCore, 7 OrientationAnalysis).test/:DataStoreFormatResolverTest,DataStorageModeMigrationTest,Dream3dLoadingApiTest,EmptyStringStoreTest,ExtentTest,MemoryBudgetManagerTest,MemorySafetyTest,IParallelAlgorithmTest.RotateSampleRefFrame/RotateEulerRefFrametest paths use slab/chunked bulk I/O.Part 5 — Build system
cmake/SimplnxConfig.hpp.inis a generated PUBLIC, ODR-safe config header.SIMPLNX_TEST_ALGORITHM_PATHcache option (default0):0=Both,1=OOC-only,2=InCore-only; plumbed into every plugin test as a compile definition (cmake/Plugin.cmake). An in-core build can validate both algorithm paths (forcing the Scanline path against in-core data still runs correctly and fast).SIMPLNX_UNIT_TEST_TARGETSglobal property —create_simplnx_plugin_unit_testrecords each test target so consumer (add_subdirectory) builds can attach extra sources/settings.zlibas a direct vcpkg dependency — the OOC layer compiled into consumers uses it directly for parallel deflate-chunk decompression off the global HDF5 mutex (andUnitTestCommonuses it for the in-house tar.gz extractor)./bigobjfor the MSVC build ofDream3dIO.cpp(exceeds the COMDAT limit / C1128 in Debug).Extent,EmptyStringStore,IDataStoreFormatResolver,InMemoryFormatResolver,MemoryBudgetManager,AlgorithmDispatch,SliceBufferedTransfer,UnionFind,IdentifySampleCommon).Part 6 — Documentation
src/Plugins/*/docs/(27 SimplnxCore, 21 OrientationAnalysis; ~+1,177 lines), each adding an## Algorithmsection with### Performanceand paired In-Core / Out-of-Core subsections that explain the dual implementation, memory footprint, and chunk/slab streaming strategy. No docs deleted.Part 7 — Test data archives
Three
download_test_data()entries added:fill_bad_data_exemplars.tar.gz(SimplnxCore)identify_sample_exemplars.tar.gz(SimplnxCore)segment_features_exemplars.tar.gz(referenced by both SimplnxCore and OrientationAnalysis)No existing archive entries were removed.
Related PR
Test Plan
SIMPLNX_TEST_ALGORITHM_PATH=2) passesSIMPLNX_TEST_ALGORITHM_PATH=1) passesSIMPLNX_TEST_ALGORITHM_PATH=0) pass