feat(evals): add Critical Thinking onto the shared evaluator contract

[adnanrhussain]

Adds the Critical Thinking evaluator at evals/durable-skills/ela-writing/critical-thinking/, following the same shared contract as the student-facing-text and feedback families (#159, #161, #163, #173–#177, #179–#185). First evaluator under the durable-skills domain.

Contract-only: the TypeScript evaluator follows in its own PR, so the rubric, schemas and fixtures can be reviewed by the people who own them first.

Scope: grades 8-10 only. supported_grades is ["8", "9", "10"], and the rubric prompt states the same range in both places it appears. The validation evidence covers those grades and no others, so the evaluator is not offered for grades 6-7 or 11-12.

Evaluator

• evaluator.id: durable_skills.ela_writing.critical_thinking
• stable_id: 114fb24f-65d8-4736-bb23-300b0470fc4d
• Single LLM step (evaluate_critical_thinking) on claude-opus-5, provider: anthropic
• Inputs are assignment_text, sources, source_count, essay_text — no grade input, so the grade_level string standardisation doesn't apply
• supported_grades is 8–10 (see Scope above); the prompt's two grade references match
• Rates five indicators (synthesizing sources, evidence strength, counterarguments, facts over opinions, drawing conclusions) plus an Overall, on a five-level scale — not_evident, exploring, analyzing, integrating, extending. synthesizing_sources is omitted when source_count is 1

Contract

• system.txt / user.txt, hash-pinned in config.json
• output_schema.json defines indicators (each with evidence, rationale, rating) then overall. Property order is deliberate — evidence and rationale precede each rating, and all indicators precede the Overall, so the reasoning conditions the score
• indicators is an object keyed by indicator id, not an array. evidence_strength, counterarguments, facts_over_opinions and drawing_conclusions are required; synthesizing_sources is an optional fifth, present only on multi-source prompts. additionalProperties: false closes the rest. Nothing in the prompts required an array, and named keys say all of this in the schema itself rather than in allOf/contains — and make duplicate ids impossible, which no array constraint could
• Indicator.id is gone: the key carries the identity, so keeping both would let them disagree. The id-to-rubric mapping (2.1, 2.2, 3.1, 3.2, 4.1) moved to the indicators description
• fixtures.json carries two real cases covering both branches: one multi-source with all 5 indicators, one single-source with 4. Each case's description records the per-indicator expert consensus it was labelled from
• example_notebook.ipynb runs the config end-to-end, verifying both prompt hashes on load and deriving its generation settings from the config

generation.temperature is null

First use of the nullable temperature from #186. Verified against the live API on claude-opus-5:

sent	result
omitted	200
1	200
0	400 — "temperature is deprecated for this model"
0.5	400
literal null	400 — "Input should be a valid number"

Anthropic's guidance for Opus 4.7 and later is to omit the parameter entirely, which is exactly what null means in our contract.

null is also the only correct encoding here, not merely a convenient one: config.schema.json requires generation.temperature, and in the SDK an absent value falls back to the evaluator config's default while null explicitly omits. Removing the key would send a temperature rather than suppress one.

The model is claude-opus-5 with no dated suffix. Per Anthropic's versioning docs, dateless ids from the 4.6 generation onward are themselves pinned snapshots rather than evergreen pointers, so there is no dated form to pin instead.

Also fixed

• $schema was one level short for a three-deep path — eval-config caught it
• langchain-anthropic added to evals/requirements.txt — this is the first eval to use it, and the existing anthropic entry is the raw SDK, not the LangChain integration. eval-requirements caught it
• fixtures.tolerance added — the notebook already read tolerance.allow_adjacent_levels, but config never set it, so fixtures ran on a strict gate. Now matches the sibling evaluators
• Notebook switched to raw-bytes prompt hashing, matching the rest of the repo
• The notebook now derives temperature from the config rather than printing it and ignoring it. It previously built the model without a temperature, which was correct for this config by coincidence — null means omit and it omitted — but any config that set a number would have been silently disregarded
• registry-conformance.test.ts gains a UNIMPLEMENTED entry for this id. That test asserts every contract on disk is either implemented or explicitly named, so a contract-only PR has to declare itself; the entry says to drop it when the TypeScript evaluator lands. This is the only file touched outside the eval directory

Verification

scripts/check.py — all 6 pass. TypeScript unit tests, typecheck and lint pass, which this PR sees for the first time: it predates #223, so the old path filter (evals/prompts/**, evals/student-facing-text/**) never matched a new domain and the SDK jobs never ran.

The notebook was executed against the real API using the same command and timeout as eval-notebooks.yml. Exit 0, no cell errors, both fixtures exact:

fixture	predicted	expected	status
5076429_cleaned (multi-source, 5 indicators)	analyzing	analyzing	exact
AAAOPP13416000111182 (single-source, 4 indicators)	exploring	exploring	exact

Both matched exactly rather than relying on the adjacency tolerance. Re-run after each change: after the rating values moved to snake_case, after the level-token hint was removed from the prompt (confirming the schema enum alone constrains the model), and after indicators became an object. The live payload came back keyed by id with all five indicators and no id field.

A live run exercises one branch at a time, so the schema's accept/reject behaviour is checked directly as well:

case	result
single-source: four mandatory only	accepted
multi-source: all five	accepted
missing a mandatory indicator	rejected
unknown indicator id	rejected
old Title Case rating	rejected
redundant id field	rejected
evidence: [] at analyzing	accepted — the known gap below

Open items, not blocking

• evidence may be empty at any rating. The prompt requires at least one quote for anything above not_evident, but the schema permits an empty array, so the structured-output parser will not enforce it. Expressible as an if/then on rating; not done here.
• outcome is not declared, so readOutcome will return no score for this evaluator once it is implemented. Every scored evaluator except math declares one; worth adding when the SDK side lands.
• fixtures.tolerance.allow_adjacent_levels is read only by the notebook. Nothing in the SDK or the checks consumes it.
• max_tokens=16000 is hardcoded in the notebook. generation only permits temperature (additionalProperties: false), so there is nowhere in the contract to declare it. Measured output is ~4.5K tokens, so the ceiling is generous headroom rather than a tuned value.
• Expert sign-off on the output shape is still open — see the thread on output_schema.json.
• No evals/README.md row, consistent with the other in-flight evaluator PRs.

[Copilot]

Pull request overview

Adds a new “Critical Thinking” durable-skills evaluator (ELA writing) to the repo’s eval contract, including prompt assets, input/output schemas, fixtures, and an example notebook, plus the Python dependency needed to run the notebook’s Anthropic LangChain integration.

Changes:

• Introduces durable_skills.ela_writing.critical_thinking evaluator assets (config, prompts, schemas, fixtures).
• Adds an executable example notebook demonstrating structured-output invocation + fixture sniff-testing.
• Updates evals/requirements.txt to include langchain-anthropic.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
evals/requirements.txt	Adds langchain-anthropic dependency for the example notebook.
evals/durable-skills/ela-writing/critical-thinking/config.json	Registers the evaluator, model, generation, schemas, and fixtures configuration.
evals/durable-skills/ela-writing/critical-thinking/system.txt	Adds the rubric and decision rules as the system prompt.
evals/durable-skills/ela-writing/critical-thinking/user.txt	Adds the user prompt template with required placeholders.
evals/durable-skills/ela-writing/critical-thinking/input_schema.json	Defines the evaluator’s expected input shape.
evals/durable-skills/ela-writing/critical-thinking/output_schema.json	Defines the structured output contract for indicator + overall ratings.
evals/durable-skills/ela-writing/critical-thinking/fixtures.json	Adds two labeled fixtures for regression/sniff testing (overall rating).
evals/durable-skills/ela-writing/critical-thinking/example_notebook.ipynb	Demonstrates config/prompt hashing, invocation, and fixture comparison.

Suppressed comments (1)

evals/durable-skills/ela-writing/critical-thinking/output_schema.json:81

• The schema text says indicators rated above "Not Evident" must include at least one evidence quote, but Indicator.evidence currently allows an empty array for any rating. This makes the schema looser than the stated output contract and can let invalid outputs through the structured-output parser.

        "evidence": {
          "type": "array",
          "description": "Verbatim quote/comment pairs supporting the rating. At least one for any indicator rated above Not Evident; may be empty for Not Evident.",
          "items": {
            "$ref": "#/$defs/Evidence"

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[adnanrhussain]

This may need to be updated to snake_case for consistency. May require updates to the model prompt

[adnanrhussain]

@aychi1 - Did we align with our internal experts on the output shape of this Eval?

[aychi1]

Hi @adnanrhussain, I personally did not.

[adnanrhussain]

Reviewed with Michelle and made adjustments. Pls see the last commit b484496

[aychi1]

Overall LGTM. Several minor questions/ comments.

[aychi1]

Hi @adnanrhussain, I personally did not.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

[aychi1]

The fixtures should also include the scores for the indicators. Based on this spreadsheet, the indicator scores are all "Analyzing" as well.
https://docs.google.com/spreadsheets/d/1_sE_Fx2z_bx6l4iKO7PueVYD7jTJR8MR_6y9UgTcmpQ/edit?gid=2100920111#gid=2100920111

[adnanrhussain]

@aychi1 - I'll have to include these fields in a followup PR. Nested object verification will require updates to the fixture tests + notebook harness. I dont want to expand the current scope of this PR, and instead do that separately and add later. Is that ok?

[aychi1]

Yes, sounds good!

[aychi1]

Similar comment-- the fixture should include the indicators. Based on the consensus_labels_round3, the indicators are all "Exploring", aside from synthesizing_sources which does not have a score.

[Copilot]

🟡 Changes recommended

The output schema currently allows empty evidence arrays even when an indicator is rated above not_evident, which contradicts the prompt/contract requirements and can let invalid structured outputs validate.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 9/9 changed files
• Comments generated: 1
• Review effort level: Lite

[aychi1]

LGTM, thank you so much Adnan for your work on this