AI Gateway Testkit

Know what your AI gateway actually supports before your users find out.

AI Gateway Testkit (agtk) is an open-source, black-box conformance and regression runner for Anthropic- and OpenAI-compatible gateways. It turns compatibility claims into repeatable test profiles, stable assertion IDs, CI-friendly verdicts, and canonical reports you can compare or safely share.

gateway target + compatibility profile
                    |
                    v
           versioned test catalog
                    |
                    v
       protocol / SDK / agent checks
                    |
                    v
      PASS / FAIL / INDETERMINATE
                    |
                    v
        comparable canonical report

Why agtk?

Claims of OpenAI or Anthropic compatibility can mean anything from basic authentication to full tool-calling and SDK interoperability. A successful request alone does not prove that a gateway is ready for production workloads.

agtk gives teams evidence they can act on:

• Precise failures: permanent scenario and assertion IDs such as OAI-TOOL-001/A04, instead of fragile log messages.
• Explicit compatibility claims: versioned profiles for core APIs, tool calling, official Go SDKs, behavioral diagnostics, Codex, and Claude Code.
• Honest verdicts: FAIL means observed incompatibility; missing or unavailable evidence becomes INDETERMINATE, never a false failure or pass.
• Regression-ready reports: compare two runs at assertion level and distinguish new failures from resolved or unchanged results.
• Security-conscious execution: environment-based credentials, HTTPS validation, bounded response bodies, redacted diagnostics, and sanitizable reports.
• CI-friendly behavior: deterministic ordering, bounded retries and costs, machine-readable JSON, and stable exit codes.

What it tests

Layer	Coverage	Profiles
OpenAI protocol	Authentication, model discovery, non-streaming Responses, forced function calls	oai-core, oai-tools
Anthropic protocol	Authentication, model discovery, non-streaming Messages, forced tool use	anthropic-core, anthropic-tools
Official Go SDKs	Request and response interoperability through the official clients	oai-sdk-go, anthropic-sdk-go
Model behavior	Exact instruction-following diagnostics, kept separate from protocol claims	behavioral-openai, behavioral-anthropic
Coding agents	Non-interactive Codex and Claude Code workflows in disposable Docker Sandboxes	codex-ready, claude-code-ready

Explore every executable contract with:

agtk profiles
agtk catalog list
agtk catalog show OAI-TOOL-001

The complete generated catalog is also available as human-readable case documentation and machine-readable JSON.

Quick start

1. Install

Go 1.25 or newer is required.

go install github.com/trungdlp/ai-gateway-testkit/cmd/agtk@latest

Or build the project locally:

git clone https://github.com/trungdlp/ai-gateway-testkit.git
cd ai-gateway-testkit
make build
./bin/agtk version

2. Configure a target

Copy examples/target.yaml and keep credentials in environment variables:

name: staging
openai:
  base_url: https://gateway.example.com/v1
  model: openai-compatible-model
  credential_env: OPENAI_GATEWAY_API_KEY
anthropic:
  base_url: https://gateway.example.com/v1
  model: anthropic-compatible-model
  credential_env: ANTHROPIC_GATEWAY_API_KEY
  api_version: "2023-06-01"

export OPENAI_GATEWAY_API_KEY='...'
export ANTHROPIC_GATEWAY_API_KEY='...'

Configure only the protocol your gateway exposes. Credential values are never accepted as CLI flags or written to reports.

3. Run a compatibility profile

agtk run --target target.yaml \
  --profile oai-core,oai-tools \
  --profile anthropic-core,anthropic-tools

Create a canonical JSON report for CI or later comparison:

agtk run --target target.yaml \
  --profile oai-tools,anthropic-tools \
  --format json --output report.json

For a quick single-endpoint OpenAI run, the environment-only interface is still available:

export AI_GATEWAY_BASE_URL='https://gateway.example.com/v1'
export AI_GATEWAY_MODEL='your-model-id'
export AI_GATEWAY_API_KEY='your-api-key'

agtk run

Understand the result

Each selected profile receives one readiness verdict:

Verdict	Meaning
PASS	Every required assertion passed.
FAIL	At least one required assertion produced evidence of incompatibility.
INDETERMINATE	Required evidence was missing, errored, blocked, skipped, or not applicable.

Reports expose coverage and success separately. This prevents unavailable evidence from masquerading as incompatibility, and prevents partial coverage from looking like a pass.

Exit codes are designed for automation:

Code	Meaning
0	Every selected profile passed.
1	At least one selected profile failed or was indeterminate.
2	Configuration, invocation, catalog, runner, or report I/O failed.

Diagnosing gateways without authentication

For a deliberately unauthenticated local gateway, use --skip-auth-checks to collect model, message, tool, and SDK evidence without first requiring missing and invalid credentials to be rejected:

ANTHROPIC_GATEWAY_API_KEY=diagnostic-placeholder agtk run \
  --target target.yaml --profile anthropic-core,anthropic-tools \
  --skip-auth-checks --format json --output diagnostic.json

Credentials are still resolved from the target's credential_env; use a synthetic placeholder when the gateway does not require one. This option skips the protocol authentication scenarios entirely: no missing/invalid-key probes are sent, and their assertions are reported as skipped with reason AUTH.CHECKS_SKIPPED. Only those authentication dependencies are waived; failures in other prerequisites still block their dependents. Without the flag, execution is unchanged.

Skipped authentication is neither a pass nor a failure. Profiles requiring it remain INDETERMINATE unless another required assertion fails, in which case that profile remains FAIL. Either verdict still exits with code 1. JSON records the option as runtime.skip_auth_checks; text and HTML reports also label authentication as skipped. Compare reports using the same setting: different authentication execution policies are marked not_comparable. Skipping authentication may permit downstream inference or explicitly enabled agent scenarios to run, with their usual request costs and runner requirements.

Report schema 1.2.0 adds the optional runtime.skip_auth_checks boolean; absent means false. This CLI still reads 1.0.0 and 1.1.0 reports. Older CLIs cannot read 1.2.0 reports and must be upgraded before rendering or comparing them.

Catch regressions in CI

Keep a known report as your baseline, then annotate a new run:

agtk run --target target.yaml \
  --profile oai-tools \
  --baseline baseline.json \
  --format json --output current.json

Or compare two existing canonical reports directly:

agtk compare baseline.json current.json

Assertions are classified as new_failure, resolved, unchanged_failure, unchanged_pass, or not_comparable. Catalog digests prevent results from different executable contracts from being compared as if they were identical.

Stable IDs, actionable evidence

Every scenario has a permanent ID, such as OAI-RESP-001, ANT-TOOL-001, or CDX-EXEC-001. Atomic assertions add a stable suffix, for example OAI-RESP-001/A04.

That identity stays useful across local debugging, CI, issue trackers, baselines, waivers, and release gates. Instead of saying that the response test broke, a team can identify the exact contract, profile version, scenario revision, and catalog digest that produced the result.

See docs/architecture.md for the scenario, assertion, profile, report, and versioning model.

Retries without hiding incompatibility

Transient transport failures and HTTP 408, 425, 429, 500, 502, 503, and 504 are retried twice by default with bounded exponential backoff. Retry-After is honored up to the configured maximum wait. Stable client, authentication, schema, and assertion failures are never retried.

agtk run --target target.yaml \
  --retries 3 \
  --retry-backoff 250ms \
  --retry-max-wait 5s

Recovered and exhausted retries are recorded in report provenance. Set --retries 0 when duplicate inference cost is unacceptable.

Agent readiness workflows

The experimental codex-ready and claude-code-ready profiles verify non-interactive coding workflows, including filesystem and shell effects. They require Docker Sandboxes (sbx) to be installed and authenticated:

agtk run --target target.yaml --profile codex-ready --agent-runner sbx
agtk run --target target.yaml --profile claude-code-ready --agent-runner sbx

Agent cases run against a disposable fixture, not this repository. They use a proxy-managed secret scoped to the sandbox and gateway host. Without --agent-runner sbx, agent assertions are skipped and the readiness verdict is INDETERMINATE.

Reports and security

The canonical report schema is schemas/report.schema.json. Reports include runtime provenance, a secret-free target fingerprint, catalog version and digest, scenario revisions, assertion statuses and reason codes, profile verdicts, retry evidence, and bounded diagnostics.

Before sharing a report, sanitize it:

agtk report sanitize report.json > shareable.json

Sanitization removes endpoint URLs, model IDs, evidence, expected and observed values, and diagnostic messages. It is defense in depth, not a data-loss-prevention boundary: always use synthetic prompts and test credentials.

Render a polished, self-contained HTML report for stakeholders:

agtk report render report.json --output report.html

The HTML report works offline, includes accessible assertion-distribution and profile-readiness charts, responsive profile and scenario views, search and result filters, dark and light themes, and print-to-PDF styling. Rendering is share-safe by default and removes the same sensitive details as report sanitize. Use --include-sensitive-details only when the report will remain in a trusted environment and diagnostic evidence is required.

Remote targets require HTTPS by default; loopback HTTP remains available for local fixtures. OpenAI requests set store: false. See SECURITY.md to report a vulnerability.

Contributing

Contributions that make gateway compatibility more measurable and reproducible are welcome. New protocol cases, provider reproductions, documentation, and focused bug fixes all help.

Start with CONTRIBUTING.md. Before adding a scenario or changing report semantics, read docs/architecture.md. Every scenario lives in one independently reviewable Go file and generates one JSON artifact plus one Markdown document.

The local quality gate is intentionally offline and free of API charges:

go generate ./...
make check

make check verifies formatting, generated catalog integrity, static analysis, race-enabled tests, and the CLI build.

License

Licensed under the Apache License 2.0.