Add create_onboarding_experiments utility

[davidbaines]

Summary

Adds silnlp/common/create_onboarding_experiments.py, a utility that turns a production Serval onboarding request into ready-to-run NMT experiment folders.

Given a request folder name in MT/experiments/_OnboardingRequests/, it:

• Parses onboarding.log for the main project and every aligned reference project (both the request's onboarded references and the standard alignment set), collecting verse counts, parallel counts, alignment scores, and scripts.
• Looks up the target language's name and country from assets/languageFamilies.json and creates <Country>/<Language>/ folders (title case, underscores for spaces/hyphens).
• Selects references passing configurable thresholds (--min-parallel, default 2000; --min-alignment, default 0.2) and creates one experiment per passing reference plus two-source mixed_src combinations ordered by alignment score (interactive selection when more than 3 pairs qualify).
• Writes config.yml (NLLB lang codes, with <iso3>_<script> fallback for languages not in NLLB) and translate_config.yml (one translate entry per source project). corpus_books comes from --training-books, either a book selection or complete, which derives the books from verse_counts.csv using a 98% completeness rule.
• Prints the silnlp.nmt.experiment command for each experiment and offers to run them (--run skips the prompt). Experiments skipped because an identical config.yml already exists are still offered, since they may not have been run yet.
• Supports --dry-run to report everything without creating files (run commands are not printed — nothing will be run).

Distinct src/trg iso codes

The src and trg isos of a corpus pair must differ (silnlp would otherwise collapse both sides into one NLLB language token). When a passing reference shares the main project's iso — e.g. a closely related language whose translation team uses the major language's code — the tool synthesizes a target code that is neither a real iso nor in NLLB (first letter kept, last two letters mutated) and copies the target extract file in MT/scripture (plus its MT/terms renderings files) to the new stem, keeping the originals. The copy is deferred until an experiment folder is actually created, so a run that creates nothing leaves the shared store untouched, and the copied file on disk is the durable record: later runs reuse the same code even if the clash is no longer visible under different thresholds.

Translated books are excluded from training

Books in --translate-books are always excluded from corpus_books, so none of their text is used for training (a config that trains on the books it translates looks invalid even when the target books are empty). Explicit book lists are kept verbatim with subtraction selections appended (NT → NT;-MAT, GEN-DEU → GEN-DEU;-EXO, chapter selections subtract exactly the selected chapters); complete mode subtracts before NT/OT compaction. The user is warned with the list of excluded books.

Book-list arguments

--training-books and --translate-books are parsed and validated with machine.scripture.get_chapters: the full silnlp selection syntax is preserved verbatim (ranges like GEN-DEU, chapter selections like "MAT 1-4", subtractions like "NT;-REV"), comma- or space-separated lists are normalised (GEN,RUT works unquoted), input is case-insensitive, and invalid book IDs are rejected at the CLI instead of failing later at preprocess time.

Translation-source checks

Before the configs are written, each chosen source project (the src_project in translate_config.yml) is verified: its folder must exist in M/Paratext/projects and contain a book file for every book being translated, with file names derived from the project's Settings.xml naming convention via FileParatextProjectSettingsParser. A missing project or book produces a warning and an interactive offer to translate from a different project (checked the same way) or keep the original; the replacement only affects translate_config.yml. Dry runs warn without prompting.

Test-set options

--no-test creates train-only experiments (type: train, folder suffix _notest, --test dropped from the run command); --test100 uses a 100-verse test set (test_size: 100, folder suffix _test100). The default remains the standard random 250-verse test set with no suffix.

Corpus-stats input (preferred) with log fallback

corpus-stats.csv written by silnlp.common.analyze is the preferred input — it is the stable machine-readable artifact, whereas the log parser scrapes message wording. The tool probes <folder>/corpus-stats.csv then <folder>/alignments/corpus-stats.csv, falling back to onboarding.log only when neither exists (older request folders). The request argument therefore also accepts any folder relative to MT/experiments containing a stats file — e.g. an alignment folder like PNG/Taupota/Align. The main project is the stem appearing in every row on one side (trg for alignment runs, src for onboarding runs); when a log sits next to the CSV its main-project line orients the direction, so single-reference CSVs are never misread. verse_counts.csv is read from the same folder so --training-books complete works unchanged. A stats folder inside the experiments tree (but not under _OnboardingRequests) creates its experiments next to itself (PNG/Taupota/Align → PNG/Taupota), preserving the existing country/language folder naming.

--target override

--target names the target language by iso code or project name, overriding detection: it may match either corpus-stats.csv column (rows not involving it are ignored), iso codes match across their 2/3-letter forms, and matching two projects that share an iso — or both sides of a row — is an error suggesting the project name. With only an onboarding.log the value is validated against the log's main project. When the override flips the direction away from the analyze run's own target, a note is printed and the derived <Country>/<Language> location is used instead of the stats folder's parent.

Safeguards

The log-derived main-project hint is soft: if it does not fit the CSV the tool warns and falls back to auto-detection, then to parsing the log itself, so a stale or partial CSV never breaks a previously working folder. CSV rows with missing statistics are skipped with a warning. For a synthetic iso the target extract file (and its terms renderings) is copied to the new stem — the originals are kept, since they may be referenced by other experiments — after a y/N confirmation (default No), with a warning when --target overrode the direction and a caution when the target's iso is an NLLB code. If the original is later re-extracted, a warning flags the copy as possibly outdated. The create-next-to-stats location rule is disabled for request folders, folders directly under MT/experiments, and flipped targets.

Experiment selection

Every run displays one numbered list of possible experiments capped at the top 20 (--top adjusts the cap) — all single-source experiments first (ranked by alignment), then two-source pairs in best-source-first order — and asks which to create (1,3, all, or none; duplicates are deduplicated). Nothing is created automatically; --dry-run prints the list without prompting and reports all displayed experiments.

Usage

poetry run python -m silnlp.common.create_onboarding_experiments SFDH_2026_07_13 --translate-books MAT
poetry run python -m silnlp.common.create_onboarding_experiments SFDH_2026_07_13 --translate-books GEN,RUT --training-books "NT;-REV" --test100
poetry run python -m silnlp.common.create_onboarding_experiments PNG/Taupota/Align --translate-books MAT

Test plan

• 34 tests in tests/test_create_onboarding_experiments.py using a real-format fixture log: log parsing, folder naming, NLLB tag resolution, the 98% completeness rule and NT/OT compaction, book-list parsing/validation, translate-book exclusion (plain lists, ranges, chapter selections, complete mode), synthetic iso derivation, deferred extract/terms renames with dry-run and re-run/prior-rename behaviour, test-set variants, index handling and identical-config skipping, corpus-stats parsing (both directions, --target by iso/project/stem, mixed-direction rows, ambiguity detection), CSV-over-log precedence with the log hint, the stats-folder experiment location, the top-20 selection prompt, end-to-end folder/YAML creation, dry-run, and the run-offer behaviour.
• Validated with --dry-run against real onboarding request folders (A33_2026_07_02_Request, MKWBe_2026_07_16_Request) and used to create and launch real experiments.
• black / isort / flake8 / mypy clean.

🤖 Generated with Claude Code

---

This change is 

[TaperChipmunk32]

This will be a good foundation for me to add automatic model building to the automated onboarding requests sometime later this year.

@TaperChipmunk32 made 1 comment.
Reviewable status: 0 of 3 files reviewed, all discussions resolved.