Skip to content

fix(ts-sdk)!: bound text at 1-10,000 chars measured as the caller sent it - #202

Merged
adnanrhussain merged 1 commit into
mainfrom
ahussain/sdk-spec-text-limits
Aug 27, 2026
Merged

fix(ts-sdk)!: bound text at 1-10,000 chars measured as the caller sent it#202
adnanrhussain merged 1 commit into
mainfrom
ahussain/sdk-spec-text-limits

Conversation

@adnanrhussain

@adnanrhussain adnanrhussain commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Brings §4.1 bounds to their spec defaults and removes input normalization.

Limits

min 10 → 1, max 100,000 → 10,000. The SDK defaults carry no product opinion: the minimum excludes empty input and nothing else. Meaningful minimums belong to each evaluator's input schema — which math-question-alignment already does, reading maxLength straight from its input_schema.json.

One length, measured on the caller's text

The SDK previously validated the trimmed length while sending the original text to the model, so the bound enforced described a string the model never saw. Telemetry reported a third thing, agreeing with neither.

Before After
Validation bounds trimmed as sent
text_length_chars as sent as sent
Sent to model as sent as sent

trim survives only as a validity test — whitespace-only input is rejected outright, never repaired. Padding is the caller's to remove; absorbing it silently means never telling them their data is malformed, and billing them for characters we pretended weren't there.

Breaking

  • Text over 10,000 characters now raises InputValidationError, down from 100,000.
  • Text that only fits once trimmed is rejected, with the real length reported (received 10,004 characters). A trailing newline on an otherwise-maximal text is enough — deliberately, since that text is what reaches the model.
  • Short text is no longer rejected. min: 1 means anything non-empty passes; evaluators that need a real floor must declare it.

Nothing in the repo is affected: the largest fixture input is 2,910 characters.

Known dead branch

With min: 1 the minimum check is unreachabletext.length < 1 implies the empty string, which the whitespace-only check rejects first. Mutation testing confirms it: if (false) survives and the block reports as uncovered.

Left in place because it goes live the moment an evaluator declares a higher minimum, and because deleting it would mean re-deriving the bound later. Flagged here so the permanent coverage gap is not mistaken for a missing test — it should not be papered over with one.

Spec follow-up

§4.1 in #162 says "Trim … then validate", "Trimmed length < min", "Trimmed length > max", and "text_length_chars … is the trimmed length". This PR contradicts all four. §4.1 also never says whether the trimmed text is the input or only the measurement — the ambiguity that let the SDK validate one string and send another. Both need fixing before #162 merges.

Tests

12 in a new file, plus 6 existing suites updated where they asserted the old min: 10.

Verified load-bearing by mutation, not just by passing: reintroducing the trim in validateText fails 1 test, and in an evaluator's telemetry call fails 3 — so the validation and reporting paths are pinned independently. One test asserts the reported length equals the text the prompt actually carried, which is the invariant that was broken.

668/668 pass · tsc --noEmit clean · lint 0 errors.

Copilot AI lite review requested due to automatic review settings August 27, 2026 01:47

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR brings the TypeScript SDK’s text-length handling into §4.1 conformance by (1) capping validated text at the spec default of 10,000 characters and (2) ensuring telemetry/log-reported text_length_chars reflects the same trimmed text length that validation actually bounds.

Changes:

  • Reduced VALIDATION_LIMITS.MAX_TEXT_LENGTH from 100,000 to 10,000 and kept validation based on trim()ed text.
  • Introduced textLengthChars(text) and migrated evaluators to use it so logging/telemetry consistently report trimmed length.
  • Added unit tests covering trimmed-length reporting (success and error paths) and the 10,000-char cap behavior.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated no comments.

Show a summary per file
File Description
sdks/typescript/src/evaluators/base.ts Lowers max text length to 10,000, adds textLengthChars(), and uses it in validation logging; validation continues to use trimmed length.
sdks/typescript/src/evaluators/conventionality.ts Uses textLengthChars() for start/success/error telemetry/log length reporting.
sdks/typescript/src/evaluators/vocabulary.ts Uses textLengthChars() for consistent telemetry/log length reporting across paths.
sdks/typescript/src/evaluators/text-complexity.ts Uses textLengthChars() for consistent telemetry/log length reporting.
sdks/typescript/src/evaluators/smk.ts Uses textLengthChars() for consistent telemetry/log length reporting.
sdks/typescript/src/evaluators/sentence-structure.ts Uses textLengthChars() for consistent telemetry/log length reporting.
sdks/typescript/src/evaluators/purpose.ts Uses textLengthChars() for consistent telemetry/log length reporting.
sdks/typescript/src/evaluators/organizational-structure.ts Uses textLengthChars() for consistent telemetry/log length reporting.
sdks/typescript/src/evaluators/intertextuality.ts Uses textLengthChars() for consistent telemetry/log length reporting.
sdks/typescript/src/evaluators/grade-level-appropriateness.ts Uses textLengthChars() for consistent telemetry/log length reporting.
sdks/typescript/src/evaluators/math/standards-alignment.ts Uses textLengthChars() for consistent telemetry/log length reporting of question.
sdks/typescript/tests/unit/evaluators/text-length-reporting.test.ts Adds tests verifying trimmed-length telemetry reporting and enforcing the new 10,000-char max constraint.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@codecov

codecov Bot commented Aug 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@adnanrhussain
adnanrhussain force-pushed the ahussain/sdk-spec-text-limits branch from 66e86ab to d68f2d0 Compare August 27, 2026 02:23
@adnanrhussain adnanrhussain changed the title fix(ts-sdk)!: cap text at 10,000 chars and report trimmed length fix(ts-sdk)!: cap text at 10,000 chars and measure the caller's text as sent Aug 27, 2026
@adnanrhussain
adnanrhussain force-pushed the ahussain/sdk-spec-text-limits branch from d68f2d0 to 8e9cad5 Compare August 27, 2026 02:35
@adnanrhussain adnanrhussain changed the title fix(ts-sdk)!: cap text at 10,000 chars and measure the caller's text as sent fix(ts-sdk)!: bound text at 1-10,000 chars measured as the caller sent it Aug 27, 2026
@adnanrhussain
adnanrhussain merged commit d885e7b into main Aug 27, 2026
18 checks passed
@adnanrhussain
adnanrhussain deleted the ahussain/sdk-spec-text-limits branch August 27, 2026 02:45
@czi-github-helper czi-github-helper Bot mentioned this pull request Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants