Repository navigation
Conversation
Add a mapped hybrid reader, eager reference linking, field memoization and real Clava construction/codegen benchmarks. Preserve six-fixture fidelity checks, cache counters, memory probes, measurements and the recommendation to prefer eager integration with selective laziness.
…elds Generate typed Java bindings from the companion schema, map bounded blocks, and resolve all graph references before dropping construction indexes. Keep memoized scalar fields in existing Clava stores and retain immutable backing files. Add full-suite cache experiments and cross-format fidelity, mapping and lifetime checks.
Archive 27 baseline-equivalent runs, fidelity and memory checks, and excluded race/quota trials. Isolate experiment scratch files and stop matrices on unexpected failures. Document that the generated schema works but does not improve suite time or retained heap with blanket laziness.
Syntax validation now sends one native-only syntax-check option before compiler arguments. Add focused coverage for its argument placement.
Generate structural descriptors from official schema reflection and validate bounded offsets, vtables, vectors, strings, unions, scalar domains and required fields. Bound recursion and verification work, and reject references outside a single block. Validation: 15 isolated JUnit tests pass; templates, NAS LU and fidelity native streams verify across 191 blocks. Consumer generation and framing integrate this verifier in the following cutover change.
Exercise schema mismatch, header ordering, duplicate headers, missing and incorrect End records, empty blocks, trailing records and truncated framing. Verify mapped files can be deleted after both successful and failed reads. Validation: 25 structural and envelope tests pass in isolated JUnit runs on JDK 17 and JDK 26 against the current eager reader sources.
Reflect exact supplied Clava classes and their inherited public DataKeys into deterministic JSON with declaring constants and generic value types. Classify references, enums, optional values and lists without schema annotations or a naming alias map. Validation: Java 17 compilation succeeds; all 112 payload classes expose 2,186 keys and repeated inventory output is byte-identical.
Cover negative 64-bit IDs that would narrow to a valid 32-bit null sentinel, invalid zero and negative references, and scope isolation for translation-unit node IDs. Validation: 27 structural, envelope and ID tests pass against the current eager reader sources on JDK 17.
Verify schema-specific cache directories, reject invalid hashes and false CCACHE_DISABLE values, and check compressed cache invocation settings.
Cover exact release tag resolution, missing local manifests, canonical schema hashing, malformed archives and incompatible compiler versions.
Preserve experimental implementations on lazy-flatbuffers-experiment. Add fixed suite runners, structural corpus checks, batch consumer comparisons, cross-TU and round-trip probes, and source-identified repeated memory comparisons.
Keep nullable-node metadata beside the Java DataKey and reflect it into the consumer inventory, preserving VariableArrayType size expressions without producer annotations or binding aliases.
Resolve the exact selected schema and pinned compiler/runtime without sibling checkouts. Document schema evolution and add generated-source drift checks, including ARM compiler bootstrap and archive verification tests.
Find revision manifests stored directly in a frozen distribution and hash local native tools alongside the parser JAR.
Apply the same collection attempts and waiting period to every runtime, including historical controls that retain an AST. This avoids extra control-side GCs changing the peak measurement.
Run eager FlatBuffers, historical Text, and Protobuf runtimes sequentially against the same filtered Clava-JS suite. Record source and runtime identities, test counts, cache state, and the shared Vitest wall-time boundary. Remove ccache and its aliases from the runner PATH while setting CCACHE_DISABLE=true.
Use the shared filtered PATH for all transports and inspect live parser file descriptors alongside mappings, AST collection and temporary folders. Require every eager cycle to release those resources.
…anup Add a strict selected-corpus reparse mode with preserved compiler options, source hashes and stable generated bytes. Check mappings and open parser files across all previous parse iterations, and capture frozen runtime provenance from its own directory.
Give the first CUDA test thirty seconds to download and assemble its built-in headers. RC2 Linux CI passed the subsequent CUDA tests but timed out this initial setup at the default five-second limit.
Print capture initialization styles and pack placement from producer metadata, omit implicit captures from source lists, and reject misaligned capture vectors during eager reading. Validate source-stable roundtrips and rejection/file cleanup for each malformed vector.
Verify source/output hashes and exact compiler flags before checking each generated translation unit with LLVM 18. Record paired acceptance, baseline generator failures, commands, and diagnostics without waiving fidelity failures. The fixed RC2 corpus audit completed with no valid-Text-to-invalid-eager case.
Keep out-of-line template headers and specialization qualification, resolve lexical parameter names, and print bare constructor/destructor names. Preserve dependent braces and structured member-pointer precedence and suffixes. Validate grouped parameter metadata and stable roundtrips for partial templates, renamed parameters, member arrays, noexcept references, and dependent construction.
Expect bare constructor/destructor names, typed member-function pointer formatting, and one sizeof delimiter pair. Preserve source inputs and golden indentation; all five affected parser tests pass.
Consume the testing release with lambda, member pointer and template metadata. Preserve the user local dumper override while selecting RC3 in the committed release tag.
Keep CUDA and OpenMP outside the historical comparison workload while requiring the full integration suite to be checked separately.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| "compiler": command[0], | ||
| "compiler_version": compiler_versions[command[0]], | ||
| "command": command, | ||
| "command_shell_quoted": shlex.join(command), |
Comment on lines
+176
to
+177
| completed = subprocess.run(command, capture_output=True, text=True, | ||
| check=False, timeout=timeout_seconds) |
Comment on lines
+151
to
+152
| completed = subprocess.run([compiler, "--version"], capture_output=True, | ||
| text=True, check=False, timeout=10) |
|
|
||
| def read_json(path: Path) -> dict[str, Any]: | ||
| try: | ||
| value = json.loads(path.read_text()) |
| raise SystemExit(f"runtime {label} has no rows") | ||
|
|
||
| command = [str(part) for part in raw_command] | ||
| work_root = Path(command[-1]).resolve() |
Render class arguments from each enclosing template parameter group, including nested primary and partial-specialization scopes. Keep enclosing headers before the member template header. Verify renamed, packed, non-type and mixed nested parameters through stable parse-generate-reparse tests.
Pin full corpus and audit hashes, preserve exact input options and record every excluded case before strict round-trip validation. Select all eager syntax-pass cases without manual exclusions or waivers.
Unwrap parenthesized pointees for cv/ref/noexcept suffix discovery, preserve method ref qualifiers, and globally qualify nonlocal member-pointer classes. Cover namespace shadowing, local classes, and template specializations through stable source round trips.
Prevent anonymous-record normalization from applying a later const typedef to the mutable first alias. Exercise assignment and stable reparsing, and update the member-pointer golden for global class qualification.
Ignore empty declaration output when joining a translation unit, preserve terminal newlines while adding inline comments, and avoid doubling trailing record newlines inside heterogeneous declaration statements. Verify byte-stable source regeneration.
Document hash-verified automatic selection of every compiler-accepted output, record exclusions without waivers, and distinguish the historical comparison filter from full integration-suite coverage.
Clang reports unsupported dependent friend methods in the lexical class. Preserve their original declaration and enclosing headers instead of inventing an owner or placing friend before template. Document that AST edits within these declarations are not emitted, and reject raw strings or continued literals rather than change their contents. Verify direct and nested friend round trips and explicit rejection cases.
Print Clang's GNU malloc and nothrow attributes with valid spellings, and stabilize the order of GNU attributes without moving inline or other attribute forms. Drop only empty OpenMP captured placeholders anchored at the standalone directive, preserving structured and macro-expanded bodies. Add a focused parse-generate-reparse fixture covering attribute semantics, standalone target directives, and structured OpenMP bodies.
Avoid adding an unintended lvalue qualifier to factory-created C functions. Verify generated C parses and remains stable across regeneration.
Prevent a record pointee and globally qualified owner from being parsed as one nested name. Normalize parentheses introduced by reparsing while preserving const qualifiers and pointer/array declarators.
Capture exact release and consumer artifact provenance, full suite and corpus results, strict roundtrip evidence, repeated-parse cleanup, and matched transport measurements. Document measured costs and outstanding CI/security findings for review.
lm-sousa
changed the title
Record nine matched diagnostic observations with explicit GC disabled equally. Qualify the earlier 18.9% penalty and reopen the speed adoption gate; the residual reader cost remains unattributed. Preserve production sources and the local dumper selector.
Record source-verified GC and StoreDefinition cache omissions, plus the narrower generated-reader specialization candidate. Separate existing equivalents, compression-policy tradeoffs and benchmark settings from portable runtime changes.
Reuse StoreDefinition metadata per Clava class with ConcurrentHashMap.computeIfAbsent, matching the Protobuf reader fast path. Record the GC and concurrency ports and accepted XStream incompatibility. The full parser check, coverage and generated drift verification pass with 234 tests and four expected skips; both enabled-cache probes also pass.
Record 27 serial observations with matched App workloads, source provenance and per-call evidence. Document the production GC improvement and remaining App construction penalty without claiming an individual-port speedup.
Expose the recorded Java and JS command durations, distinguish common benchmark workloads from unfiltered suites, and audit the earlier cache-state ranking without attributing the remaining regression to an unprofiled phase.
Run four rounds of both historical suite workloads in bypass, cold and warm states for cumulative App and separate command timing. Preserve historical control values and original chart renderers, retain same-day results, and correct the JS heap description.
Replace historical comparison controls with 144 accepted measurements of current builds in rotated serial rounds. Preserve original App and separate wall-time methodology, record matching workloads and cache counters, and publish current boxplots with complete provenance.
Use LF line endings for embedded CSV and trim trailing report whitespace. Keep all measured values unchanged and refresh renderer provenance.
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Clava now imports completed eager FlatBuffers v2 ASTs through the production path. It verifies the exact schema and binaries selected by
clang-dumper-release.tag, generates bindings through strict Java naming/type checks, and rejects malformed or incompatible streams. Lazy work is preserved onlazy-flatbuffers-experiment; there is no legacy selector or Text fallback.The eager reader now caches store definitions per Clava class with
ConcurrentHashMap.computeIfAbsent, avoiding the shared synchronized metadata lookup on every node. specs-java-libs #37 also removes the duplicate GC request and uses the Protobuf immutable ordered metadata construction while retaining the stronger FlatBuffers concurrency regression.Compatibility warning: the immutable metadata map in specs-java-libs #37 breaks existing XStream XML persistence. XStream removal is being handled in a separate PR. Until it lands, 17 legacy XML tests fail in that dependency; its jOptions check is not green.
Latest validation at Clava
09542632awith specs-java-libsae7194a7: full parser check passed 234 tests with four expected skips, coverage and generated-output drift verification. Three release-resource tests skip under the preserved local dumper selector, and one cache-hit test skips underCCACHE_DISABLE=true; the separate enabled-cache run passed both GeneratedParseRootTest tests without skips. SpecsUtils check and coverage passed 6,654 tests with two skips. jOptions passed 1,725 tests and failed the 17 XML tests described above. The parser used an isolated temporary CUDA cache after the shared cache failed validation. The matched optimization benchmark completed 27 accepted serial observations at these exact source revisions. Production Java test-body medians were 38.638 s previous FlatBuffers, 34.010 s updated FlatBuffers and 31.926 s Protobuf, a 12.0% before/after improvement. App medians with diagnostics disabled were 27.280/27.122/25.284 s for Java and 9.933/9.847/9.507 s for JS in previous/updated/Protobuf order. App improvements were under 1%; updated FlatBuffers remained 7.3% and 3.6% slower than Protobuf. Before/after captured workloads match in every round, all 170 JS App calls match Protobuf, and the 207 matching Java calls retain a 7.5% penalty. These measurements do not isolate each port or remeasure memory.Earlier RC3 validation used the authorized v18.1.8_5-rc3 prerelease. The validation report and evidence identify consumer
f03eb45d8, producer158080298, dependency revisions and artifact hashes. Full parser checks passed 237 tests with one expected disabled-cache skip; the enabled-cache run passed both tests. The full Linux JS suite passed 162 tests with two pending. The published producer passed all 1,062 native corpus streams. All 778 syntax-valid eager corpus outputs passed strict regeneration, with no Text-valid/eager-invalid paired case. The raw 125 Text/eager differences remain recorded, including 26 outputs invalid in both printers; they are not waived.Final measurements found a cost as well as retained-heap savings. NAS median peak RSS was 562 MiB eager versus 515 MiB Text and 587 MiB Protobuf, while retained heap was 14.9/37.8/15.6 MiB. Small C++ eager peak/retained memory exceeded Text. All 120 eager cleanup cycles passed. The matched Java median was 38.016 s eager versus 35.635 s Text and 31.986 s Protobuf; matched JS snapshots were 44.319/42.813/39.253 s. This does not claim a speed or universal memory advantage. These frozen measurements predate the current optimization ports. The GC diagnostic reduced the Java eager-versus-Protobuf gap from 18.9% to 6.8% when explicit GC was disabled equally; it does not isolate the reader cost or establish a current speed advantage. The speed adoption gate remains unresolved.
GPT-6.1-Sol medium reviewed the integrated implementation and final fixes; its concrete findings were resolved. Field/node, regeneration, compatible-release and CI drift procedures are documented. Dependent friend member templates use bounded original-source printing, so edits inside those declarations are not emitted and raw/continued strings are rejected.
The follow-up CUDA test gate uses the manifest's host capability API. All three CUDA tests still pass on Linux and the TypeScript build passes; unsupported macOS hosts skip the CUDA group. Latest remote CI must be checked separately. SonarCloud findings remain open on this PR and native #56; overall CI is not claimed green.
Stacked on #258 and depends on specs-java-libs #37 and clang-dumper #56. Only testing RC publication is authorized; no master merge or stable publication is included.
Whole-command medians were also recorded for the common historical workloads, not unfiltered current-branch suites. With diagnostics disabled, Java was 33.819 s updated FlatBuffers versus 32.424 s Protobuf; JS was 34.827 versus 33.170 s. The remaining reversal survives the GC control. The earlier pilot did win App timing and Java command time; its JS whole-command ranking was narrow and Protobuf won the warm-cache median. Production additions have not been profiled individually.
The fresh comparison report now replaces historical Text and Protobuf controls with current builds and reruns FlatBuffers alongside them. It completed 144 accepted serial measurements on 7 October: four rotated rounds of three implementations, both common suite workloads and direct/cold/warm cache states, separately for cumulative App and uninstrumented whole-command timing. Protobuf uses the remotely verified branch heads Clava
ff5e58afa, specs-java-libs19c8e3e4and producerab9d0238. Text uses consumerbae332499, dependencyf49ca2c4and producerc549b4b2. All runtime JARs remained unchanged during measurement. Only Before-cache remains a clearly dated 2 October reference, excluded from paired statistics.FlatBuffers remained slower than both fresh controls in every suite/cache/metric cell. Median paired FlatBuffers-versus-Protobuf bypass differences were +4.01% JS and +7.83% Java for App creation, and +5.59% for each whole-command workload. Warm gaps were +10.74%/+16.44% for JS/Java App and +6.63%/+10.18% for JS/Java command time. All 170 JS App inputs match both controls; Java matches 207 of 216, with the matching-call diagnostic retaining a +7.91% bypass penalty versus Protobuf. Four observations are descriptive and do not establish significance of small differences. The committed report, evidence and SVGs retain exact revisions, all source/content/option differences, timer boundaries, original test-file orders, cache counters and renderer functions. Java uses 512 MiB; JS uses its original default heap observed at about 7.55 GiB. Diagnostics are disabled consistently in isolated benchmark builds. These compare branch implementations, not an isolated format switch; memory was not remeasured. The speed gate remains unresolved.