Fix recursive deepJoin and flattenDeepArray traversal hangs#1041
Draft
He-Pin wants to merge 2 commits into
Draft
Fix recursive deepJoin and flattenDeepArray traversal hangs#1041He-Pin wants to merge 2 commits into
He-Pin wants to merge 2 commits into
Conversation
951e137 to
972f73f
Compare
22cd3c5 to
1779389
Compare
bca8902 to
705ad78
Compare
He-Pin
added a commit
to He-Pin/sjsonnet
that referenced
this pull request
Jun 28, 2026
Motivation: PR databricks#1041 adds active-path tracking for deepJoin and flattenDeepArray to stop recursive array hangs. Flat root arrays do not need that guard, but the rebased implementation still allocated traversal state for them. Modification: Scan root arrays directly until the first nested array. Only allocate TraversalState and DeepArrayActiveSet after nested traversal is actually needed. Also stop catching VM-level fatal errors in this traversal path. Result: Keeps recursive-array regression tests passing while reducing flat flatten/deepJoin traversal overhead in debug-stats and hyperfine checks.
705ad78 to
3a0faab
Compare
He-Pin
added a commit
to He-Pin/sjsonnet
that referenced
this pull request
Jun 28, 2026
Motivation: PR databricks#1041 added active-path tracking to make std.deepJoin terminate on recursive arrays. The nested acyclic workload std.makeArray(..., ["x"]) exposed a fixed cost from allocating traversal state and array frames for flat child arrays. Modification: Scan flat nested deepJoin arrays directly before entering traversal state. The root-level path now keeps scanning when child arrays contain only strings, and only allocates TraversalState after deeper array traversal is actually needed. Add a nested recursive-array regression test for that fast path. Result: Restores the nested deepJoin benchmark from a regression versus master to a small win while preserving recursive-array error behavior.
He-Pin
added a commit
to He-Pin/sjsonnet
that referenced
this pull request
Jun 28, 2026
Motivation: Recursive arrays in std.deepJoin and std.flattenDeepArray could hang, and the optimized nested deepJoin path left an unused private helper that failed Scala 2.12 compilation. Modification: Add cross-platform active-array tracking for deep array traversals, route flattenDeepArray/deepJoin through the shared traversal helper, preserve flat fast paths, and remove the stale unused nested deepJoin helper. Result: Recursive deep array traversal now fails fast, shared acyclic arrays still evaluate, and the PR CI matrix compiles/tests locally except Graal, which the user explicitly skipped. References: PR databricks#1041
d876e9f to
494b969
Compare
494b969 to
4fe29c5
Compare
4fe29c5 to
4032178
Compare
Motivation: The deep-array traversal PR accidentally removed the inline-object lookup size guard, allowing large imported objects with super lookups to fall back to repeated linear scans. Modification: Restore Obj.InlineScanMax and the guarded inline scan so large inline objects use the lazy lookup map. Add nested flattenDeepArray recursion coverage. Result: Keeps recursive deep array detection covered while avoiding the unrelated large-object lookup performance regression.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
std.deepJoinandstd.flattenDeepArraycan hang on recursive arrays because the old traversal recursed without tracking the active array path. They should terminate with a normal builtin error instead of looping indefinitely.Modification
Result
std.deepJoin(local a = [a]; a)max stack frames exceeded[std.deepJoin] max stack frames exceededstd.flattenDeepArray(local a = [a]; a)max stack frames exceeded[std.flattenDeepArray] max stack frames exceededFlat and acyclic nested arrays still produce the same values as before.
Risks
std.deepJoinandstd.flattenDeepArray.