[TTS] Harden MagpieTTS evalset config loading and the standalone evaluator by XuesongYang · Pull Request #16291 · NVIDIA-NeMo/Speech

XuesongYang · 2026-09-22T07:06:50Z

Move the per-dataset override resolution (asr_model, language) that examples/tts/magpietts_inference.py did by hand into resolve_evaluation_config_for_dataset() (dataclasses.replace on the CLI-level EvaluationConfig) and use it from both entry points, so every EvaluationConfig field is either overridden or inherited in one place.
validate_evalset_entry(): reject malformed language/asr_model values (including JSON null) with the dataset name in the message. Behavior change: an evalset entry with "language": null is now rejected instead of silently running with language=None; remove the key to inherit the CLI-level language. load_evalset_config() additionally warns about unrecognized evalset keys. ASR_MODEL_TYPES is shared by validate_evalset_entry(), --asr_model_type and the load_evaluation_models() error, which reported the model name instead of the unknown type.
Standalone evaluate_generated_audio.py could not complete: main() forwarded a hand-picked subset of EvaluationConfig to evaluate(), so with_fcd stayed True without a codec path and evaluate() raised before loading any model. It now builds the config with with_fcd=False and forwards it whole through evaluate_generated_audio_dir(). --evalset now takes --datasets_json_path/--datasets_base_path and honours the same per-dataset overrides as the example script. New CLI requirements: --generated_audio_dir is required; --manifest_path is required unless --evalset is given; --manifest_path/--audio_dir are rejected together with --evalset, and --datasets_json_path/--datasets_base_path are rejected without it; an unknown --evalset name is a parser error instead of an assert (argparse exit code 2).
Experiment CSV: header and append_metrics_to_csv() derive from one column tuple (EXPERIMENT_METRICS_CSV_COLUMNS); write_csv_header_if_needed() warns when an existing file was written with a different layout.
_get_record_texts(): "original_text": null counts as absent.
Docs: new "Evaluation set configuration" section; the inference example passes the required --datasets_json_path and --run_evaluation. EvaluationConfig docstring: duplicated asr_model_name line, missing asr_model_type/asr_batch_size/eou_batch_size, codec_model_path semantics.

Tests (CPU): resolver overrides and inheritance, malformed-value rejection in the resolver and the loader, unknown-key warning, base-path resolution, standalone CLI wiring with the real evaluate() running, argument validation, CSV header/row alignment and layout warning, and the example-script wiring through the resolver.

Important

The Update branch button must only be pressed in very rare occassions.
An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Collection: [Note which collection this PR will affect]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

Trusted PRs run automatically through copy-pr-bot. For an untrusted PR, a maintainer can trigger CI by commenting
/ok to test <head-sha>; repeat this after a new push if the PR remains untrusted.

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

copy-pr-bot · 2026-09-22T07:06:54Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

…uator - Move the per-dataset override resolution (asr_model, language) that examples/tts/magpietts_inference.py did by hand into resolve_evaluation_config_for_dataset() (dataclasses.replace on the CLI-level EvaluationConfig) and use it from both entry points, so every EvaluationConfig field is either overridden or inherited in one place. - validate_evalset_entry(): reject malformed language/asr_model values (including JSON null) with the dataset name in the message. Behavior change: an evalset entry with "language": null is now rejected instead of silently running with language=None; remove the key to inherit the CLI-level language. load_evalset_config() additionally warns about unrecognized evalset keys. ASR_MODEL_TYPES is shared by validate_evalset_entry(), --asr_model_type and the load_evaluation_models() error, which reported the model name instead of the unknown type. - Standalone evaluate_generated_audio.py could not complete: main() forwarded a hand-picked subset of EvaluationConfig to evaluate(), so with_fcd stayed True without a codec path and evaluate() raised before loading any model. It now builds the config with with_fcd=False and forwards it whole through evaluate_generated_audio_dir(). --evalset now takes --datasets_json_path/--datasets_base_path and honours the same per-dataset overrides as the example script. New CLI requirements: --generated_audio_dir is required; --manifest_path is required unless --evalset is given; --manifest_path/--audio_dir are rejected together with --evalset, and --datasets_json_path/--datasets_base_path are rejected without it; an unknown --evalset name is a parser error instead of an assert (argparse exit code 2). - Experiment CSV: header and append_metrics_to_csv() derive from one column tuple (EXPERIMENT_METRICS_CSV_COLUMNS); write_csv_header_if_needed() warns when an existing file was written with a different layout. - _get_record_texts(): "original_text": null counts as absent. - Docs: new "Evaluation set configuration" section; the inference example passes the required --datasets_json_path and --run_evaluation. EvaluationConfig docstring: duplicated asr_model_name line, missing asr_model_type/asr_batch_size/eou_batch_size, codec_model_path semantics. Tests (CPU): resolver overrides and inheritance, malformed-value rejection in the resolver and the loader, unknown-key warning, base-path resolution, standalone CLI wiring with the real evaluate() running, argument validation, CSV header/row alignment and layout warning, and the example-script wiring through the resolver. Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

XuesongYang · 2026-09-22T07:09:39Z

/ok to test 88072c0

Copilot

Copilot review overview

🟡 Changes recommended

Preserve or explicitly expose and document the standalone evaluator’s previous default ASR model.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

Preserve or expose the standalone evaluator's default ASR model · New

What changed in this PR

Hardens MagpieTTS evaluation configuration, standalone CLI behavior, CSV handling, tests, and documentation.

Changes:

Centralizes evalset overrides and validation.
Fixes standalone evaluator wiring and argument handling.
Aligns CSV metrics and expands tests/documentation.
Open issue: standalone evaluation changes the default ASR model without a CLI override.

File	Summary
`tests/collections/tts/modules/magpietts_inference/test_magpietts_inference.py`	Adds regression and CLI tests.
`nemo/collections/tts/modules/magpietts_inference/utils.py`	Shares ASR types and CSV layout handling.
`nemo/collections/tts/modules/magpietts_inference/evaluation.py`	Adds dataset-specific configuration resolution.
`nemo/collections/tts/modules/magpietts_inference/evaluate_generated_audio.py`	Updates evalset validation and standalone evaluator wiring.
`nemo/collections/tts/modules/magpietts_inference/__init__.py`	Exports the configuration resolver.
`examples/tts/magpietts_inference.py`	Integrates shared resolution and CSV handling.
`docs/source/tts/magpietts.rst`	Documents evaluation set configuration and usage.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Addresses the CodeQL alerts on #16291. py/cyclic-import: evaluate_generated_audio.main() imported evaluation.py inside the function while evaluation.py imports evaluate_generated_audio at module level. EvaluationConfig, validate_evalset_entry, resolve_evaluation_config_for_dataset, ASR_MODEL_TYPES and EVALSET_ENTRY_KEYS move to the new dependency-free module evaluation_config.py; evaluation.py re-exports EvaluationConfig and the resolver, so existing imports keep working. The EvaluationConfig -> evaluate() keyword mapping is now evaluate_generated_audio.evaluate_with_config(), used by evaluation.evaluate_generated_audio_dir() and by the standalone main(), which no longer imports evaluation.py. py/import-and-import-from: the test module imported three modules both as "import x as m" and "from x import ..."; it now uses from-imports only and patches by dotted path. No behavior change; the existing tests cover both call paths. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

Addresses the CodeQL alerts on #16291. py/cyclic-import: evaluate_generated_audio.main() imported evaluation.py inside the function while evaluation.py imports evaluate_generated_audio at module level. EvaluationConfig, validate_evalset_entry, resolve_evaluation_config_for_dataset, ASR_MODEL_TYPES and EVALSET_ENTRY_KEYS move to the new dependency-free module evaluation_config.py; evaluation.py re-exports EvaluationConfig and the resolver, so existing imports keep working. The EvaluationConfig -> evaluate() keyword mapping is now evaluate_generated_audio.evaluate_with_config(), used by evaluation.evaluate_generated_audio_dir() and by the standalone main(), which no longer imports evaluation.py. py/import-and-import-from: the test module imported three modules both as "import x as m" and "from x import ..."; it now uses from-imports only and patches by dotted path. No behavior change; the existing tests cover both call paths. Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

XuesongYang · 2026-09-22T08:09:08Z

/ok to test 77dfa5e

- test_load_evalset_config_resolves_relative_paths_against_base_path: the standalone --evalset test already asserts the manifest/audio paths resolved against --datasets_base_path, and every config-based test relies on that resolution through load_evalset_config's existence checks. - The eou_model_name forwarding pin is kept once, in the --evalset standalone test; both standalone tests go through the same evaluate_with_config call. Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

XuesongYang · 2026-09-22T08:17:28Z

/ok to test dadd90d

Copilot AI lite review requested due to automatic review settings September 22, 2026 07:06

github-actions Bot added the TTS label Sep 22, 2026

Copilot started reviewing on behalf of XuesongYang September 22, 2026 07:07 View session

XuesongYang force-pushed the xueyang/tts-eval-harden-evalset-and-standalone branch from d5d9d94 to fda3054 Compare September 22, 2026 07:07

XuesongYang force-pushed the xueyang/tts-eval-harden-evalset-and-standalone branch from fda3054 to 88072c0 Compare September 22, 2026 07:08

Copilot AI reviewed Sep 22, 2026

View reviewed changes

Comment thread nemo/collections/tts/modules/magpietts_inference/evaluate_generated_audio.py

copy-pr-bot Bot deployed to public September 22, 2026 07:10 Active

copy-pr-bot Bot deployed to test September 22, 2026 07:11 Active

github-advanced-security Bot found potential problems Sep 22, 2026

View reviewed changes

copy-pr-bot Bot deployed to public September 22, 2026 07:14 Active

copy-pr-bot Bot deployed to public September 22, 2026 07:15 Active

copy-pr-bot Bot deployed to public September 22, 2026 07:39 Active

XuesongYang force-pushed the xueyang/tts-eval-harden-evalset-and-standalone branch from 46ed727 to 77dfa5e Compare September 22, 2026 08:04

copy-pr-bot Bot deployed to public September 22, 2026 08:09 Active

copy-pr-bot Bot had a problem deploying to test September 22, 2026 08:12 Error

copy-pr-bot Bot deployed to public September 22, 2026 08:14 Active

copy-pr-bot Bot deployed to public September 22, 2026 08:15 Active

copy-pr-bot Bot deployed to public September 22, 2026 08:19 Active

copy-pr-bot Bot deployed to test September 22, 2026 08:19 Active

copy-pr-bot Bot deployed to public September 22, 2026 08:22 Active

copy-pr-bot Bot deployed to public September 22, 2026 08:23 Active

copy-pr-bot Bot deployed to public September 22, 2026 08:46 Active

Navigation Menu

[TTS] Harden MagpieTTS evalset config loading and the standalone evaluator - #16291

[TTS] Harden MagpieTTS evalset config loading and the standalone evaluator#16291
XuesongYang wants to merge 3 commits into
mainfrom
xueyang/tts-eval-harden-evalset-and-standalone

XuesongYang commented Sep 22, 2026

Uh oh!

copy-pr-bot Bot commented Sep 22, 2026

Uh oh!

XuesongYang commented Sep 22, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

XuesongYang commented Sep 22, 2026

Uh oh!

XuesongYang commented Sep 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

XuesongYang commented Sep 22, 2026

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Additional Information

Uh oh!

copy-pr-bot Bot commented Sep 22, 2026

Uh oh!

XuesongYang commented Sep 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

XuesongYang commented Sep 22, 2026

Uh oh!

XuesongYang commented Sep 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants