Skip to content

docs(evals): correct the temperature guidance in the config schema - #207

Merged
adnanrhussain merged 1 commit into
mainfrom
ahussain/temperature-schema-wording
Aug 27, 2026
Merged

adnanrhussain merged 1 commit into
mainfrom
ahussain/temperature-schema-wording

Conversation

@adnanrhussain

@adnanrhussain adnanrhussain commented Aug 27, 2026 •

Copy link
Copy Markdown
Collaborator

Salvaged from #187, which was closed because its Gemini-3-must-be-1 check contradicts a decision we've since made deliberately. The schema wording is independent of that check and still wrong on main.

1. "Google recommends removing the parameter" (Gemini 3) — false. Google's docs say the opposite: "For all Gemini 3 models, we strongly recommend keeping the temperature parameter at its default value of 1.0." Keeping it at a value is not removing it.

2. "GPT-5 (rejects any value but 1 with a 400)" — false. I verified gpt-5.4-2026-03-05 against the live API: it accepts a temperature and honours it.

The description also instructed authors to use null for Gemini 3, which is now backwards — the ela-reading evaluators pin an explicit 0, tested and preferred, except Organizational Structure at 1.

What replaces it

Sampling temperature to send, or null to omit the parameter entirely. Prefer an
explicit number: the value an evaluator was validated at should be recorded, not
inherited from a model default that can change under us. Use null only for models
that reject any non-default value -- Claude Opus 4.7 and later return a 400 --
where omitting is the documented approach. A model recommending its default is
not grounds for null; write the number out. Required rather than optional so
every step records a deliberate choice; an absent key reads as 'nobody
considered it'.

The per-model list is gone on purpose. It was a set of claims that rot — two had already rotted — and it invited null wherever a vendor merely recommended a default. The one hard constraint left is the only one that is a real API rejection rather than advice, and I verified it: Opus 5 returns 400 on a non-default temperature.

I did not reuse #187's phrasing verbatim. It argued for explicit numbers partly because our TypeScript provider coerced null to 0 via request.temperature ?? 0 — #199 fixed that, so the argument no longer holds and repeating it would be stating a bug we don't have.

One line changed. All 6 repo checks pass.

Copilot AI lite review requested due to automatic review settings August 27, 2026 06:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the author-facing JSON Schema guidance for generation.temperature to correct inaccurate vendor-specific claims and encourage recording the explicitly validated temperature per evaluator step.

Changes:

  • Rewrites the generation.temperature schema description to remove incorrect model-specific assertions (Gemini 3 “remove parameter”, GPT-5 “400 unless 1”).
  • Clarifies that null means “omit the parameter”, and should be used only when a model rejects non-default temperatures and omission is the documented approach.
  • Reinforces that temperature is required so each step records a deliberate choice.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@adnanrhussain
adnanrhussain merged commit c8b6282 into main Aug 27, 2026
9 checks passed
@adnanrhussain
adnanrhussain deleted the ahussain/temperature-schema-wording branch August 27, 2026 06:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants