feat(metadata): extract the catalog crate, and give the schema a version - #27
Merged
Merged
Conversation
Two changes that belong together: the catalog types and loader become a crate of their own, and the catalog schema they read gains a version. src/metadata becomes crates/metadata, package sqd-metadata, so other projects can read a catalog without pulling in the engine. The engine re-exports it under the module's old path, so every crate::metadata:: consumer in the engine, tests, benches and examples is untouched. The catalogs stay at the repository root; the crate's tests reach them from there. The spec checker walked src/ and tests/ for coverage tags, which would have lost the loader's fourteen: it now walks crates/ too, and its self-tests stage that tree. The Makefile gates pass --workspace, since a root package makes plain `cargo test` skip the member. A catalog now opens with `version`, a string, and the loader refuses any value but the one it reads, `v2`, exported as SCHEMA_VERSION. Required rather than defaulted: a version that could be omitted tells a reader nothing, and a catalog written for a later schema should fail at load rather than be read as this one. The bundled catalogs and every inline test catalog carry it. A new version must stay backwards compatible with v2. A catalog is published once by its provider (the network scheduler, for one) and read by several consumers on their own release cycles, so a version a v2 reader cannot accept obliges the provider to publish one catalog per version in use. The rule is recorded beside the constant, in the crate's README and in the catalogs' README. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
define-null
marked this pull request as ready for review
September 8, 2026 15:16
A catalog is published once and read by consumers on their own release cycles, so a key added under v2 reaches readers that predate it. Every catalog type denied unknown fields, and the loader walked special_filters and virtual_fields for stray keys, so one new optional key made every older reader refuse the whole catalog. The types now skip a key they do not know, and the stray-key walk goes, with the per-kind key lists it read. A misspelled optional key is skipped the same way. A test loads a catalog carrying an unknown key in every shape a catalog has. The stale-key test and the misspelled alias key case asserted the opposite and are removed, as is the conformance note's citation of the former. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The loader skips a key it does not know, so a catalog that gained an optional key still loads in a release that predates it. A misspelled optional key is skipped the same way, and changes what the engine does: parent_hash_colum turns fork detection off, implicit_filter widens an alias to its whole table, knd: children makes a relation a plain join. A reader cannot tell the two apart; only the release a catalog is written for can. parse_dataset_description_strict and load_dataset_description_strict load as the others do, then refuse every key the parse skipped and name each by its path. They find one by comparing the document with the description written back out, which reaches into special_filters and virtual_fields too. A plain scalar key is compared as its text, and a key set to null is passed over, since it sets nothing. The bundled catalogs and the format doc's alias example load strictly, and both READMEs say a provider checks a catalog this way before publishing it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is this PR about?
Two changes that belong together: the catalog types and loader become a crate of their own, and the catalog schema they read gains a version.
src/metadata becomes crates/metadata, package sqd-metadata, so other projects can read a catalog without pulling in the engine. The engine re-exports it under the module's old path, so every crate::metadata:: consumer in the engine, tests, benches and examples is untouched. The catalogs stay at the repository root; the crate's tests reach them from there. The spec checker walked src/ and tests/ for coverage tags, which would have lost the loader's fourteen: it now walks crates/ too, and its self-tests stage that tree. The Makefile gates pass --workspace, since a root package makes plain
cargo testskip the member.A catalog now opens with
version, a string, and the loader refuses any value but the one it reads,v2, exported as SCHEMA_VERSION. Required rather than defaulted: a version that could be omitted tells a reader nothing, and a catalog written for a later schema should fail at load rather than be read as this one. The bundled catalogs and every inline test catalog carry it.A new version must stay backwards compatible with v2. A catalog is published once by its provider (the network scheduler, for one) and read by several consumers on their own release cycles, so a version a v2 reader cannot accept obliges the provider to publish one catalog per version in use. The rule is recorded beside the constant, in the crate's README and in the catalogs' README.