Use release-pinned eager FlatBuffers AST transport

.github/workflows/nightly.yml

@@ -94,6 +94,12 @@ jobs:
         run: |
           cd clava/ClangAstParser
           gradle test
+
+      - name: Verify generated wire bindings and release resolver
+        run: |
+          gradle -p clava/ClangAstParser verifyGeneratedWire
+          python3 -m unittest discover -s clava/ClangAstParser/scripts -p 'test_*.py'
+          python3 -m unittest discover -s clava/experiments/flatbuffers/validation -p 'test_*.py'
 
       - name: Test ClavaWeaver
         run: |

ClangAstDumper/src/ClangAstDumper/ClangAstDumperConstants.h

@@ -73,7 +73,6 @@ static const std::string TOP_VISIT_START = "<Top Visit Start>";
 
 static const std::string TOP_VISIT_END = "<Top Visit End>";
 
-static const std::string SKIPPED_NODES_MAP = "<Skipped Nodes Map>";
 
 static const std::string PRAGMA = "<Pragma>";
 

ClangAstDumper/src/plugin.cpp

@@ -30,4 +30,4 @@ class Plugin : public DumpAstAction, public PluginASTAction {
 
 // Register the plugin with Clang
 const static FrontendPluginRegistry::Add<Plugin>
-    DumpAst("DumpAst", "Dumps the AST information to feed ClangStreamParserV2");
+    DumpAst("DumpAst", "Dumps the AST information for Clava");

ClangAstParser/WIRE_FORMAT.md

@@ -0,0 +1,34 @@
+# Eager FlatBuffers AST transport
+
+Clava consumes completed, size-prefixed FlatBuffers v2 files. It creates the complete AST and resolves references before returning it. There is no lazy mode or Text fallback. The preserved `lazy-flatbuffers-experiment` branch contains the earlier experiments.
+
+## Release selection and toolchain
+
+`clang-dumper-release.tag` selects the producer release. The build downloads that release's `clang-dumper-release-manifest.json` and `clang-dumper-wire-schema-v2.zip`; it checks the archive SHA-256 and the canonical schema SHA-256 before generating anything. The runtime checks the native binary hash and rejects streams whose schema hash differs from the compiled bindings.
+
+An absolute local build directory is supported for development. It must contain the same manifest and schema bundle produced by the dumper build. A missing or incompatible manifest is an error. There is no implicit sibling schema checkout.
+
+The compiler and Java runtime are pinned to FlatBuffers 25.12.19, upstream commit `7e163021e59cca4f8e1e35a7c828b5c6b7915953`. The compiler archive and exact-commit runtime source archive have checked SHA-256 values in `scripts/resolve_wire_release.py`; the compiler version and runtime version marker are also checked. The Java runtime is compiled from that archive because this version has no Maven Central runtime artifact. Downloads are cached under Gradle's user cache, with hashes checked before reuse. Linux and Windows ARM64 build the compiler from the same verified source archive and require CMake and a C++ compiler.
+
+The producer owns the schema and native code generator. Clava owns the Java binding generator and verifier generator. Neither repository's generator imports the other repository's scripts.
+
+## Adding a field or node
+
+1. Add the producer schema field and native emitter in `clang-dumper`. Give node payload tables the exact name `JavaClassNameData`. Fields use snake case; the corresponding public Java `DataKey` name uses its camel case spelling. Rename existing mismatches rather than adding aliases.
+2. Add or update the Java node's `DataKey` in `ClavaAst`. `BindingInventory.java` inspects the compiled classes, including inherited keys and generic types. Generation fails when a class, key, enum or field type does not match. Node references use the actual key type to select required, optional or list reference resolution. Annotate a Clava node key with `@NullableNodeReference` when absence must become a typed Clava null node rather than an empty Optional.
+3. Keep protocol structure in the schema. Scalar presence, required offsets, discriminated unions, source locations and reference IDs are validated before applying records. Extend the consumer's compound-value decoder when introducing a new compound field type; do not attach Java class or key annotations to the producer schema.
+4. Build the dumper, select its local build directory for development, and run `gradle -p ClangAstParser generateCompleteWire`. This resolves the schema, runs the pinned compiler, inspects Java keys, and regenerates the bindings and structural verifier.
+5. Run the wire tests and both parser suites, then the cross-TU, source round-trip, malformed-input and memory gates described in `experiments/flatbuffers/README.md`.
+6. Run `gradle -p ClangAstParser updateGeneratedWireInventory`, review `generated-wire.sha256`, and commit it with the schema/binding changes. `verifyGeneratedWire` fails if regenerated Java sources differ from this inventory. `check` and CI invoke that verification.
+
+## Publishing a compatible release
+
+Publish the dumper binaries, generated enum bundle, schema bundle and manifest from one producer revision. The producer release workflow creates the schema bundle and manifest with its shared manifest script. The manifest records the FlatBuffers version and commit, schema version and canonical hash, schema archive hash, and each binary's hash.
+
+Select that published tag in Clava, regenerate the inventory, and validate the release through the ordinary build and parser runtime. A schema change requires regenerated consumer bindings; mismatches fail explicitly. Existing releases that lack the v2 schema manifest cannot serve this consumer.
+
+## Caching and cleanup
+
+AST ccache entries are isolated by FlatBuffers version, upstream commit, schema hash and executable hash. Every present `CCACHE_DISABLE` value disables Clava's ccache wrapper; `0`, `false` and `no` are rejected, so unset the variable to enable caching. Clava's explicit disable takes precedence over a conflicting `CCACHE_NODISABLE`. Bypass measurements must use the same policy for each control. Executable digest metadata is limited to 128 file identities; temporary resource paths cannot grow that cache indefinitely.
+
+Mapped windows are released when replaced and when the reader closes, including rejection paths. Parsing releases temporary native files after importing the eager graph. The repeated-parse probe checks collected ASTs, retained heap, mapped files and temporary folders; these checks are required in addition to a short-process peak memory measurement.

ClangAstParser/build.gradle

@@ -1,3 +1,6 @@
+import groovy.json.JsonSlurper
+import java.security.MessageDigest
+
 plugins {
     id 'distribution'
     id 'java'
@@ -24,6 +27,7 @@ dependencies {
 
     implementation 'com.google.code.gson:gson:2.12.1'
     implementation 'com.google.guava:guava:33.4.0-jre'
+    implementation 'com.github.luben:zstd-jni:1.5.7-16'
     implementation 'org.apache.commons:commons-compress:1.27.1'
     implementation 'org.tukaani:xz:1.9'
 
@@ -77,6 +81,16 @@ jacocoTestCoverageVerification {
     }
 }
 
+// Coverage applies to maintained parser code; generated bindings and the pinned
+// upstream FlatBuffers runtime are checked through regeneration and wire tests.
+tasks.withType(org.gradle.testing.jacoco.tasks.JacocoReportBase).configureEach {
+    classDirectories.setFrom(sourceSets.main.output.classesDirs.asFileTree.matching {
+        exclude 'astwire/**', 'com/google/flatbuffers/**'
+        exclude 'pt/up/fe/specs/clang/wire/GeneratedNodes*'
+        exclude 'pt/up/fe/specs/clang/wire/GeneratedWireVerifier*'
+    })
+}
+
 // Make sure jacoco report is generated after tests
 test {
     useJUnitPlatform()
@@ -86,3 +100,165 @@ test {
 
     finalizedBy jacocoTestReport
 }
+
+// Schema and compiler come from the exact selected release; no sibling schema checkout is read.
+def wireRoot = layout.buildDirectory.dir('wire').get().asFile
+def wireSchema = new File(wireRoot, 'schema')
+def wireTools = new File(wireRoot, 'tools')
+def wireFlatc = new File(wireTools, System.getProperty('os.name').startsWith('Windows') ? 'flatc.exe' : 'flatc')
+def wireReflection = new File(wireRoot, 'reflection')
+def wireGenerated = layout.buildDirectory.dir('generated/sources/wire').get().asFile
+def wireInventory = new File(wireRoot, 'binding-inventory.json')
+def wireResolved = new File(wireRoot, 'selected-release.json')
+def wireCache = new File(gradle.gradleUserHomeDir, 'caches/clava-wire')
+
+def runWire = { Closure configure ->
+    def process = providers.exec {
+        ignoreExitValue = true
+        configure.delegate = delegate
+        configure.resolveStrategy = Closure.DELEGATE_FIRST
+        configure.call()
+    }
+    def result = process.result.get()
+    def errors = process.standardError.asText.get()
+    if (errors) logger.lifecycle(errors.trim())
+    result.assertNormalExitValue()
+    process
+}
+
+tasks.register('resolveWireRelease', Exec) {
+    inputs.file('clang-dumper-release.tag')
+    inputs.file('scripts/resolve_wire_release.py')
+    // Local build manifests may change without changing the selected absolute path.
+    def selected = file('clang-dumper-release.tag').text.trim()
+    if (new File(selected).isAbsolute()) {
+        inputs.file(new File(selected, 'clang-dumper-release-manifest.json'))
+        inputs.file(new File(selected, 'clang-dumper-wire-schema-v2.zip'))
+    }
+    outputs.dir(wireSchema)
+    outputs.dir(wireTools)
+    outputs.file(wireResolved)
+    outputs.dir(new File(wireTools, 'java-runtime'))
+    commandLine 'python3', file('scripts/resolve_wire_release.py'),
+        '--tag-file', file('clang-dumper-release.tag'), '--cache-root', wireCache,
+        '--schema-out', wireSchema, '--flatc-out', wireFlatc,
+        '--reflection-fbs-out', new File(wireTools, 'reflection.fbs'), '--manifest-out', wireResolved,
+        '--java-runtime-out', new File(wireTools, 'java-runtime')
+}
+
+tasks.register('generateWireReflection') {
+    dependsOn('resolveWireRelease')
+    inputs.dir(wireSchema)
+    inputs.file(wireFlatc)
+    inputs.file(new File(wireTools, 'reflection.fbs'))
+    outputs.dir(wireReflection)
+    outputs.dir(new File(wireGenerated, 'astwire'))
+    doLast {
+        delete(wireReflection, new File(wireGenerated, 'astwire'))
+        wireReflection.mkdirs()
+        wireGenerated.mkdirs()
+        def schemaFiles = fileTree(new File(wireSchema, 'wire/v2')).matching { include '*.fbs' }.files.sort { it.name }
+        runWire {
+            commandLine([wireFlatc.absolutePath, '--java', '--gen-all', '--gen-object-api', '-I',
+                new File(wireSchema, 'wire/v2').absolutePath, '-o', wireGenerated.absolutePath] + schemaFiles*.absolutePath)
+        }
+        runWire {
+            commandLine wireFlatc.absolutePath, '--binary', '--schema', '--bfbs-comments', '--bfbs-builtins',
+                '-I', new File(wireSchema, 'wire/v2').absolutePath, '-o', wireReflection.absolutePath,
+                new File(wireSchema, 'wire/v2/complete.fbs').absolutePath
+        }
+        runWire {
+            commandLine wireFlatc.absolutePath, '--json', '--strict-json', '--defaults-json',
+                '-o', wireReflection.absolutePath, new File(wireTools, 'reflection.fbs').absolutePath,
+                '--', new File(wireReflection, 'complete.bfbs').absolutePath
+        }
+    }
+}
+
+tasks.register('generateWireBindingInventory') {
+    dependsOn('generateWireReflection', configurations.compileClasspath)
+    inputs.file('scripts/BindingInventory.java')
+    inputs.file(new File(wireReflection, 'complete.json'))
+    inputs.files(configurations.compileClasspath)
+    inputs.files(fileTree('../ClavaAst/src') { include '**/*.java' })
+    outputs.file(wireInventory)
+    doLast {
+        def schema = new JsonSlurper().parse(new File(wireReflection, 'complete.json'))
+        def union = schema.enums.find { it.name == 'astwire.v2.NodePayload' }
+        if (!union || !union.is_union) throw new GradleException('Schema lacks NodePayload union')
+        def sources = fileTree('../ClavaAst/src') { include '**/*.java' }.files
+        def classes = union.values.findAll { it.name != 'NONE' }.collect { value ->
+            def table = schema.objects[value.union_type.index].name.tokenize('.').last()
+            if (!table.endsWith('Data')) throw new GradleException("Node payload table must end in Data: ${table}")
+            def name = table.substring(0, table.length() - 4)
+            def matches = sources.findAll { it.name == name + '.java' }
+            if (matches.size() != 1) throw new GradleException("Expected one Java class for ${table}: ${matches}")
+            def matcher = matches[0].text =~ /(?m)^package\s+([\w.]+)\s*;/
+            if (!matcher.find()) throw new GradleException("Missing package in ${matches[0]}")
+            matcher.group(1) + '.' + name
+        }.unique().sort()
+        def helper = new File(wireRoot, 'inventory-helper')
+        helper.mkdirs()
+        def classpath = configurations.compileClasspath.asPath
+        runWire {
+            commandLine 'javac', '--release', '17', '-cp', classpath, '-d', helper.absolutePath,
+                file('scripts/BindingInventory.java').absolutePath
+        }
+        def inventoryProcess = runWire {
+            commandLine(['java', '-cp', helper.absolutePath + File.pathSeparator + classpath,
+                'BindingInventory'] + classes)
+        }
+        wireInventory.text = inventoryProcess.standardOutput.asText.get()
+        inventoryProcess.result.get().assertNormalExitValue()
+    }
+}
+
+tasks.register('generateCompleteWire') {
+    dependsOn('generateWireBindingInventory')
+    inputs.file('scripts/generate_clava_wire.py')
+    inputs.file('scripts/generate_wire_verifier.py')
+    inputs.file(wireInventory)
+    inputs.file(wireResolved)
+    inputs.file(new File(wireReflection, 'complete.json'))
+    outputs.dir(new File(wireGenerated, 'pt'))
+    doLast {
+        def selected = new JsonSlurper().parse(wireResolved)
+        runWire {
+            commandLine 'python3', file('scripts/generate_clava_wire.py'),
+                '--reflection', new File(wireReflection, 'complete.json'), '--inventory', wireInventory,
+                '--schema-hash', selected.schema_hash, '--out', wireGenerated
+        }
+        runWire {
+            commandLine 'python3', file('scripts/generate_wire_verifier.py'),
+                '--reflection', new File(wireReflection, 'complete.json'), '--out', wireGenerated
+        }
+    }
+}
+
+sourceSets.main.java.srcDir(wireGenerated)
+sourceSets.main.java.srcDir(new File(wireTools, 'java-runtime'))
+compileJava.dependsOn('generateCompleteWire')
+
+def generatedWireHashes = {
+    fileTree(wireGenerated).matching { include '**/*.java' }.files.sort {
+        wireGenerated.toPath().relativize(it.toPath()).toString().replace('\\', '/')
+    }.collect { source ->
+        def path = wireGenerated.toPath().relativize(source.toPath()).toString().replace('\\', '/')
+        MessageDigest.getInstance('SHA-256').digest(source.bytes).encodeHex().toString() + '  ' + path
+    }.join('\n') + '\n'
+}
+tasks.register('updateGeneratedWireInventory') {
+    dependsOn('generateCompleteWire')
+    doLast { file('generated-wire.sha256').text = generatedWireHashes() }
+}
+tasks.register('verifyGeneratedWire') {
+    dependsOn('generateCompleteWire')
+    inputs.file('generated-wire.sha256')
+    inputs.dir(wireGenerated)
+    doLast {
+        if (file('generated-wire.sha256').text != generatedWireHashes()) {
+            throw new GradleException('Generated wire bindings drifted. Review the schema and generator changes, then run updateGeneratedWireInventory.')
+        }
+    }
+}
+check.dependsOn('verifyGeneratedWire')

ClangAstParser/clang-dumper-release.tag

@@ -1 +1 @@
-v18.1.8_4
+v18.1.8_5-rc3