docs: Audit READMEs for v1.0 launch

[esthersweon]

Audits every README for GA: accuracy, consistent naming, and the current evaluator taxonomy.

• Rewrote the top-level and evals/ READMEs, and audited datasets/
• Added a README for each evaluator family and each of the 17 evaluators
• Added Mac/Linux and Windows setup guides, covering all four API keys the notebooks need
• Moved every doc link onto the current taxonomy and fixed a 404 in datasets/
• Aligned evaluator names with their contracts ("Strength Acknowledgement" → "Acknowledgment"; docs URL unchanged until Mintlify is renamed)
• TS SDK README: "Required key" → "Default provider", Batch section reduced to an example plus a link
• Batch README: added the missing feedback family, the full CLI flag list, and the programmatic API

Verified against source: every evaluator's name, grades, provider and doc link match its config.json; all README snippets typecheck against the published 1.1.0; no broken links across 53 markdown files.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[Copilot]

🟡 Changes recommended

Unresolved review findings include incomplete prompt inventories and inaccurate TypeScript SDK/batch guidance.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR refreshes GA-facing project, evaluator, SDK, setup, and dataset documentation.

Changes:

• Adds evaluator-family and evaluator-specific README coverage.
• Updates SDK, batch CLI/API, setup, taxonomy, and dataset guidance.
• Aligns documentation with evaluator contracts and current links.

File summaries

File	Reviewed changes and follow-up
sdks/typescript/src/batch/README.md	Documents normalization, CLI, and APIs; follow-ups remain for aliases, imports, precedence, and formatter/output semantics.
sdks/typescript/README.md	Updates provider guidance; qualify credentials as required for default models.
sdks/python/README.md	Points users to current Python SDK documentation.
README.md	Refreshes the evaluator catalog; add omitted prompt links for vocabulary, sentence structure, and math standards.
evals/student-facing-text/README.md	Adds the student-facing-text family index.
evals/student-facing-text/ela-reading/vocabulary-complexity/README.md	Documents the evaluator; add background-knowledge.txt.
evals/student-facing-text/ela-reading/sentence-structure/README.md	Documents the evaluator; add the three grade-specific rubric files.
evals/student-facing-text/ela-reading/reference-knowledge-demands/README.md	Adds evaluator documentation.
evals/student-facing-text/ela-reading/purpose-clarity/README.md	Adds evaluator documentation.
evals/student-facing-text/ela-reading/organizational-structure/README.md	Adds evaluator documentation.
evals/student-facing-text/ela-reading/meaning-directness/README.md	Adds evaluator documentation.
evals/student-facing-text/ela-reading/grade-level-appropriateness/README.md	Adds evaluator documentation.
evals/student-facing-text/ela-reading/background-knowledge-demands/README.md	Adds evaluator documentation.
evals/README.md	Reorganizes evaluator documentation; add omitted vocabulary, sentence-structure, and math prompt links.
evals/feedback/README.md	Adds the feedback family index.
evals/feedback/ela-writing/withholding-answers/README.md	Adds evaluator documentation.
evals/feedback/ela-writing/tone-appropriateness/README.md	Adds evaluator documentation.
evals/feedback/ela-writing/student-response-specificity/README.md	Adds evaluator documentation.
evals/feedback/ela-writing/strength-acknowledgment/README.md	Adds evaluator documentation.
evals/feedback/ela-writing/revision-manageability/README.md	Adds evaluator documentation.
evals/feedback/ela-writing/revision-actionability/README.md	Adds evaluator documentation.
evals/feedback/ela-writing/revision-accuracy/README.md	Adds evaluator documentation.
evals/durable-skills/README.md	Adds the durable-skills family index.
evals/durable-skills/ela-writing/critical-thinking/README.md	Adds Critical Thinking evaluator documentation.
evals/academic-standards-alignment/README.md	Adds the academic-standards family index.
evals/academic-standards-alignment/mathematics/math-standards-alignment/README.md	Documents math alignment; include the optional coarse-filter-user.txt prompt.
docs/setup-windows.md	Adds Windows setup and credential instructions.
docs/setup-mac-linux.md	Adds Mac/Linux setup and credential instructions.
datasets/README.md	Updates dataset documentation; correct the possessive apostrophe.

Review details

Suppressed comments (15)

README.md:25

• This row omits background-knowledge.txt, which is a prompt source used by the vocabulary evaluator's unconditional background_knowledge step. Since the column is presented as the evaluator's prompt links, add that file here as well.

|                                                                                                                                                                              | Vocabulary Complexity<br/>[Docs](https://docs.learningcommons.org/evaluators/student-facing-text-evaluators/vocabulary-complexity)<br/>[Directory](./evals/student-facing-text/ela-reading/vocabulary-complexity/)                 | [Input](./evals/student-facing-text/ela-reading/vocabulary-complexity/input_schema.json), [output](./evals/student-facing-text/ela-reading/vocabulary-complexity/output_schema.json)                         | [Notebook](./evals/student-facing-text/ela-reading/vocabulary-complexity/example_notebook.ipynb)             | [`grades-3-4-system.txt`](./evals/student-facing-text/ela-reading/vocabulary-complexity/grades-3-4-system.txt), [`grades-3-4-user.txt`](./evals/student-facing-text/ela-reading/vocabulary-complexity/grades-3-4-user.txt), [`other-grades-system.txt`](./evals/student-facing-text/ela-reading/vocabulary-complexity/other-grades-system.txt), [`other-grades-user.txt`](./evals/student-facing-text/ela-reading/vocabulary-complexity/other-grades-user.txt) |

README.md:26

• This row lists only the four LLM message prompts, but the sentence-structure config also injects the three grade-specific rubric-*.txt files. The prompt links are therefore incomplete for this evaluator.

|                                                                                                                                                                              | Sentence Structure<br/>[Docs](https://docs.learningcommons.org/evaluators/student-facing-text-evaluators/sentence-structure)<br/>[Directory](./evals/student-facing-text/ela-reading/sentence-structure/)                         | [Input](./evals/student-facing-text/ela-reading/sentence-structure/input_schema.json), [output](./evals/student-facing-text/ela-reading/sentence-structure/output_schema.json)                               | [Notebook](./evals/student-facing-text/ela-reading/sentence-structure/example_notebook.ipynb)                | [`analysis-system.txt`](./evals/student-facing-text/ela-reading/sentence-structure/analysis-system.txt), [`analysis-user.txt`](./evals/student-facing-text/ela-reading/sentence-structure/analysis-user.txt), [`complexity-system.txt`](./evals/student-facing-text/ela-reading/sentence-structure/complexity-system.txt), [`complexity-user.txt`](./evals/student-facing-text/ela-reading/sentence-structure/complexity-user.txt)                             |

README.md:38

• This row omits coarse-filter-user.txt, which is referenced by the math standards evaluator's optional coarse_filter LLM step. Add the missing prompt link so the table does not hide a contract input.

| Academic Standards<br/>[Docs](https://docs.learningcommons.org/evaluators/academic-standards-evaluators/introduction)<br/>[Directory](./evals/academic-standards-alignment/) | Math Standards Alignment<br/>[Docs](https://docs.learningcommons.org/evaluators/academic-standards-evaluators/math-standards-alignment)<br/>[Directory](./evals/academic-standards-alignment/mathematics/math-standards-alignment/) | [Input](./evals/academic-standards-alignment/mathematics/math-standards-alignment/input_schema.json), [output](./evals/academic-standards-alignment/mathematics/math-standards-alignment/output_schema.json) | [Notebook](./evals/academic-standards-alignment/mathematics/math-standards-alignment/example_notebook.ipynb) | [`system.txt`](./evals/academic-standards-alignment/mathematics/math-standards-alignment/system.txt), [`user.txt`](./evals/academic-standards-alignment/mathematics/math-standards-alignment/user.txt)                                                                                                                                                                                                                                                         |

datasets/README.md:18

• Use a right single quotation mark for the possessive here; ‘ is an opening quotation mark, so (SAP)‘s is punctuated incorrectly.

We expanded the dataset by scoring a subset of the rows for dimensions in Student Achievement Partners (SAP)‘s [Qualitative Text Complexity Rubric for Informational Text](https://learnwithsap.b-cdn.net/app/uploads/2026/04/Qualitative-Text-Complexity-Rubric-Informational.pdf) ↗.

evals/README.md:23

• The vocabulary row omits background-knowledge.txt, even though config.json uses it for the unconditional background-knowledge LLM step. Add the missing prompt link to keep this evaluator index complete.

|                                                                                                                                                                 | Vocabulary Complexity<br/>[Docs](https://docs.learningcommons.org/evaluators/student-facing-text-evaluators/vocabulary-complexity)<br/>[Directory](./student-facing-text/ela-reading/vocabulary-complexity/) | [Input](./student-facing-text/ela-reading/vocabulary-complexity/input_schema.json), [output](./student-facing-text/ela-reading/vocabulary-complexity/output_schema.json)                         | [Notebook](./student-facing-text/ela-reading/vocabulary-complexity/example_notebook.ipynb)             | [`grades-3-4-system.txt`](./student-facing-text/ela-reading/vocabulary-complexity/grades-3-4-system.txt), [`grades-3-4-user.txt`](./student-facing-text/ela-reading/vocabulary-complexity/grades-3-4-user.txt), [`other-grades-system.txt`](./student-facing-text/ela-reading/vocabulary-complexity/other-grades-system.txt), [`other-grades-user.txt`](./student-facing-text/ela-reading/vocabulary-complexity/other-grades-user.txt) |

evals/README.md:24

• The sentence-structure row lists only the four message prompts, while config.json also loads three grade-specific rubric files (rubric-grade-3.txt, rubric-grade-4.txt, and rubric-grades-5-12.txt). Add those links so this evaluator index exposes all prompt inputs.

|                                                                                                                                                                 | Sentence Structure<br/>[Docs](https://docs.learningcommons.org/evaluators/student-facing-text-evaluators/sentence-structure)<br/>[Directory](./student-facing-text/ela-reading/sentence-structure/) | [Input](./student-facing-text/ela-reading/sentence-structure/input_schema.json), [output](./student-facing-text/ela-reading/sentence-structure/output_schema.json)                               | [Notebook](./student-facing-text/ela-reading/sentence-structure/example_notebook.ipynb)                | [`analysis-system.txt`](./student-facing-text/ela-reading/sentence-structure/analysis-system.txt), [`analysis-user.txt`](./student-facing-text/ela-reading/sentence-structure/analysis-user.txt), [`complexity-system.txt`](./student-facing-text/ela-reading/sentence-structure/complexity-system.txt), [`complexity-user.txt`](./student-facing-text/ela-reading/sentence-structure/complexity-user.txt)                             |

evals/README.md:36

• The math standards row omits coarse-filter-user.txt, which is referenced by the optional coarse_filter LLM step in config.json. Add that prompt link so the evaluator index is complete.

| Academic Standards<br/>[Docs](https://docs.learningcommons.org/evaluators/academic-standards-evaluators/introduction)<br/>[Directory](./academic-standards-alignment/) | Math Standards Alignment<br/>[Docs](https://docs.learningcommons.org/evaluators/academic-standards-evaluators/math-standards-alignment)<br/>[Directory](./academic-standards-alignment/mathematics/math-standards-alignment/) | [Input](./academic-standards-alignment/mathematics/math-standards-alignment/input_schema.json), [output](./academic-standards-alignment/mathematics/math-standards-alignment/output_schema.json) | [Notebook](./academic-standards-alignment/mathematics/math-standards-alignment/example_notebook.ipynb) | [`system.txt`](./academic-standards-alignment/mathematics/math-standards-alignment/system.txt), [`user.txt`](./academic-standards-alignment/mathematics/math-standards-alignment/user.txt)                                                                                                                                                                                                                                             |

evals/academic-standards-alignment/mathematics/math-standards-alignment/README.md:7

• The contract has an optional coarse_filter LLM step sourced from coarse-filter-user.txt (config.json:60-70), but this README omits that prompt. Include it so the documented prompt assets cover both the single-item and filtered evaluation paths.

- Prompts: [system](./system.txt), [user](./user.txt)

evals/student-facing-text/ela-reading/sentence-structure/README.md:7

• The sentence-structure contract loads rubric-grade-3.txt, rubric-grade-4.txt, and rubric-grades-5-12.txt as preprocessing inputs, but this README lists only the four message prompts. Link the grade-specific rubric files too; otherwise the documented prompt assets are incomplete.

- Prompts: [analysis system](./analysis-system.txt), [analysis user](./analysis-user.txt), [complexity system](./complexity-system.txt), [complexity user](./complexity-user.txt)

evals/student-facing-text/ela-reading/vocabulary-complexity/README.md:7

• The evaluator config also runs the unconditional background_knowledge LLM step from background-knowledge.txt (config.json:64-72), but this prompt is omitted here. Add it so the evaluator README exposes every prompt used by the contract.

- Prompts: [grades 3–4 system](./grades-3-4-system.txt), [grades 3–4 user](./grades-3-4-user.txt), [other grades system](./other-grades-system.txt), [other grades user](./other-grades-user.txt)

sdks/typescript/README.md:171

• The unconditional credential guidance is false for the configuration modes documented below: modelOverride replaces the default provider set, and an injected llmProvider skips API-key validation. Qualify the requirement as applying to the default models so users do not infer that every listed provider key is always mandatory.

`metadata.model` names every model that ran, joined by `+` when an evaluator uses more than one — so a multi-step evaluator reports e.g. `openai:gpt-4o-…+openai:gpt-4.1-…`. Which models run can also depend on the input: Vocabulary Complexity takes a different branch for grades 3-4 than for 5-12. The **Default provider** column below is what each evaluator's contract declares, and you must supply a key for every provider listed — not a promise about which one serves a given call: construction validates the union of keys an evaluator could need across all its branches, so Vocabulary Complexity demands both keys even at a grade where only one provider runs. Model strings come from each evaluator's contract and change with it, so treat `metadata.model` as the record of what actually ran rather than something to assert on. When you need one comparable value per evaluation regardless of evaluator, use `readOutcome`:

sdks/typescript/src/batch/README.md:60

• This is not the full list of supported flags: parseArgs still accepts --model-override (the deprecated backward-compatible alias) and -h (the short help form), but neither is documented here. Add these aliases or narrow the claim so users can discover supported CLI inputs.

| `--version` / `--help` | — | Print version / full flag list |

sdks/typescript/src/batch/README.md:121

• renderOutputs does not always discard payload detail: the standards JSON and HTML preserve the per-component verdict, and the standards CSV embeds it as JSON. This contradicts the earlier standards-output description and may send users to the evaluators unnecessarily; distinguish the generic projections from the standards projections and mention output.results for raw payloads.

Those outputs are a flattened per-row summary — score, reasoning and status — not the full payloads, and their shape differs between the standards family and the others. For full payloads, call the evaluators directly.

sdks/typescript/src/batch/README.md:15

• normalizeRow only gives the canonical column priority when its value is non-empty; if the canonical header is present but empty, it falls back to a populated alias (covered by column-normalization.test.ts). As written, this overstates precedence and can lead users to think an empty canonical column will always win.

Column matching is case-insensitive and alias-aware; the canonical column wins when both it and an alias are present.

sdks/typescript/src/batch/README.md:121

• This description is inaccurate for the standards family: renderOutputs("math-standards-alignment", ...) emits family-specific JSON with full learningComponents, as the paragraph above says, while formatAsHTML is the text-complexity-only formatter. Calling formatAsHTML on standards or feedback output would render a misleading QTC report; clarify the formatter scope and the standards JSON exception.

`html` is absent for families without a report of their own. `formatAsCSV(output)`, `formatAsJSON(output, meta)` and `formatAsHTML(output, meta)` are the individual projections.

Those outputs are a flattened per-row summary — score, reasoning and status — not the full payloads, and their shape differs between the standards family and the others. For full payloads, call the evaluators directly.

• Files reviewed: 29/29 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[michael-berger-czi]

Should we be telling them to use it globally? Seems like we'd want them to do the npx approach first, and just do this if it's already installed globally, no?

[michael-berger-czi]

Comment from Claude: Review of the README GA audit — docs-only, nothing blocking on code correctness. Comments inline: five accuracy issues where a reader following the docs gets a wrong result, plus one low-priority note.

Verified clean along the way, for the record: all 54 external doc URLs return 200 (including the #sdk, #python-notebooks and #columns anchors), zero broken relative links across all 29 changed files, all markdown tables well-formed, and every batch-README fact checked against source (family ids/order, all 16 member ids, maxInputRows, column aliases, ReportMeta fields, CLI flag defaults). The strength-acknowledgement spelling is genuinely required — the -ment variant 404s, so that inline TODO is correct. Grade ranges and providers match each config.json.

Two things outside the inline comments that you may want to fold in: evals/student-facing-text/ela-reading/README.md is 0 bytes (a blank rendered README on the path every Student-Facing Text link routes through — its siblings have no README at all, which renders fine), and four internal <!-- TODO --> comments ship to npm and raw file views (sdks/typescript/README.md:214 and :295, both setup docs); the :295 one is already stale, since it lists three credential names but cli.ts:75 uses a fourth, LEARNING_COMMONS_API_KEY.

Reviewed at 57bef2a. Note the origin/esw/readmes-ga ref in my clone was stale (non-fast-forward fetch), so this is against refs/pull/213/head. Review produced by Claude.

[michael-berger-czi]

Comment from Claude: Omits background-knowledge.txt, the user message of the background_knowledge step. Its output feeds {student_background_knowledge}, which is a required placeholder in both grades-3-4-user.txt and other-grades-user.txt — so it's part of the prompt chain for every grade band, not an optional extra. Same class of gap as sentence-structure/README.md:7.

[michael-berger-czi]

Comment from Claude: evaluate_sync is Python-only, but this row covers both SDKs. git grep -IE 'evaluateSync|evaluate_sync' -- sdks/typescript/src returns 0 hits at this commit; Python has it at sdks/python/src/learning_commons_evaluators/evaluators/base.py:151. As written, a TypeScript reader goes looking for a sync entry point that doesn't exist.

Something like "call evaluate (Python also offers evaluate_sync)" would keep the row accurate for both.

[michael-berger-czi]

Comment from Claude: Low priority / non-blocking — flagging for awareness, since this wording is inherited rather than introduced here.

"five indicators" is conditionally wrong: synthesizing_sources (2.1) is omitted when source_count is 1, so single-source essays return four. That's specified in input_schema.json:27, in output_schema.json (indicators.required lists only the other four), and three times in system.txt (:33, :169, :176 — the last calls it "the only omission").

It also interacts with the headline score: system.txt:121-122 has a "with an even number of indicator ratings, the Overall median rounds up" rule, and an even count only occurs in the single-source case. Both fixtures cover the branch explicitly ("Multi-source prompt (2.1 rated)" / "Single-source prompt (2.1 omitted)"), and neither SDK models the field (0 hits for synthesizing_sources under sdks/), so the schema is the only contract a consumer has.

The same sentence already lives in config.json:7 and example_notebook.ipynb:9, both untouched by this PR — so a complete fix is a follow-up rather than something to expand scope for here. If you do touch this line, line 3 also implies the essay is the only input, while input_schema.json:6-11 requires assignment_text, sources and source_count as well.

[adnanrhussain]

ok as-is for now

[czi-fsisenda]

Looks good! A couple of comments.
Some can't be addressed in documentation alone. They'll need some implementation updates.
The documentation only comments are related to language could be broader that could save us frequent updates and questions around having some documentation live both in GH as well as Mintlify.

[czi-fsisenda]

nit: I think all the evals will load environment variables from a .env file in the directory. That may be a better way to go.

[czi-fsisenda]

nit: Same here. .env may be a better way to go.

[czi-fsisenda]

Should text-complexity be changed to student-facing-text?

[adnanrhussain]

yes, as part of the rename

[czi-fsisenda]

We should update this to be familyId

[esthersweon]

Here or in the SDK source code for a future release?

[czi-fsisenda]

I think this needs to be updated in the SDK first. Right, @adnanrhussain?
Both the name of the field and its value. It should be familyId: "student-facing-text"?

[czi-fsisenda]

I thought we were going for linking to Mintlify as much as we could to avoid having multiple sources of info that could drift apart?
We link to a Mintlify quickstart, but we have one in this page too.
We also have a lot of links that also exist elsewhere. Lots of opportunities for issues with links.

[adnanrhussain]

From my POV, and feedback from the field, we should have at-least an index of the evaluators + some top line info on what capabilities they cover.

Basic info + getting started (minimal) should be included in the repo, everything else should be in the documentation.

I am sure there is opportunity to streamline and reduce the bloat in the readmes

[czi-fsisenda]

Seems more specific than the other eval descriptions