[dtice25]

Summary

Adds a recurring catalog benchmark that fetches each AI gateway's model-list endpoint (/v1/models or equivalent) and writes a normalized JSON matrix of the models each gateway exposes, including routing/provider options where the response carries them. No per-model chat probes are issued.

What changed

• New benchmark: benchmarks/ai-gateway/ai-gateway-model-index.bench.ts
  • One benchsdk task per gateway from benchmarks/ai-gateway/model-index-providers.ts
  • onComplete writes results/ai-gateway-model-index/<YYYY-MM-DD>.json and latest.json for full runs; --provider or credential-filtered partial runs write suffixed latest-<provider>.json files so the shared snapshots are not overwritten.
• New task: benchmarks/ai-gateway/model-index-task.ts
  • Uses node:https to GET the models endpoint
  • Normalizes openai, openrouter, and anthropic response shapes to { id, name, displayName, ownedBy, providers, contextLength, maxOutputTokens, createdAt }
  • Extracts routing options from providers[].providerId (LLM Gateway) and keeps ownedBy derivation independent
  • Reads top_provider and per-provider max_output fields for output limits
• Provider roster covering the 10 gateways: openrouter, vercel-ai-gateway, cloudflare-ai-gateway, blazerail, llmgateway, pydantic-ai-gateway, concentrate-ai-gateway, ramp, neon, ngrok, llmapi
• package.json scripts for full and per-provider runs
• .github/workflows/ai-gateway-model-index.yml to run weekly on Monday 9am UTC and on workflow_dispatch, with results committed back to the repo

Running locally

pnpm bench:ai-gateway-model-index --no-ingest
pnpm bench:ai-gateway-model-index:vercel --no-ingest

The first CI run will commit the full results/ai-gateway-model-index/latest.json matrix.

Link to Devin session: https://app.devin.ai/sessions/1e7c8b6773ab42c788875c2a29df6804
Open in Devin Desktop: https://app.devin.ai/desktop/session/1e7c8b6773ab42c788875c2a29df6804?variant=devin
Requested by: @dtice25

---

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[open-cla]

Contributor License Agreement

All contributors are covered by a CLA.

[superagent-security]

Superagent found 4 security concern(s).

[superagent-security]

P1: Dependency installation and workspace builds execute lifecycle scripts with repository write credentials

pnpm install and recursive builds run lifecycle code while checkout credentials and contents write are available.

Disable install scripts, isolate builds with a read-only token, and grant write access only to the final push step.

AI prompt

Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name=".github/workflows/ai-gateway-model-index.yml">
<violation number="1" location=".github/workflows/ai-gateway-model-index.yml:66">
<priority>P1</priority>
<title>Dependency installation and workspace builds execute lifecycle scripts with repository write credentials</title>
<evidence>The new `pnpm install --frozen-lockfile` step runs package-manager lifecycle hooks, and the following recursive build runs package build scripts. The checkout action retains credentials by default and the workflow has contents: write; therefore a compromised dependency or build hook can use the GitHub token and runner environment before the intended benchmark step, including to alter or push repository contents.</evidence>
<recommendation>Install dependencies with scripts disabled (for example, `pnpm install --frozen-lockfile --ignore-scripts`) and invoke only audited build steps explicitly. Use a separate read-only token for install/build, disable checkout credential persistence for those steps, and perform the final results push in a narrowly isolated step with the minimum required permission.</recommendation>
</violation>
</file>

[superagent-security]

P2: Run the privileged model-index job only on an isolated trusted runner

A secret-bearing, contents-writing job uses a custom runner with wildcard Vault grants and OIDC access.

Use a dedicated ephemeral trusted runner and least-privilege Vault/token permissions; remove unused grants.

AI prompt

Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name=".github/workflows/ai-gateway-model-index.yml">
<violation number="1" location=".github/workflows/ai-gateway-model-index.yml:45">
<priority>P2</priority>
<title>Run the privileged model-index job only on an isolated trusted runner</title>
<evidence>The job selects the custom runner label `namespace-profile-default;permissions.additional_grant=vault/object:*:list;permissions.additional_grant=vault/object:*:describe` while the workflow has `contents: write` and `id-token: write` permissions and loads numerous API credentials. This creates a high-impact trust boundary: compromise or cross-job state on that runner could expose credentials or the write-capable token.</evidence>
<recommendation>Verify in the Namespace/GitHub runner configuration that this label resolves to an ephemeral, repository-scoped runner with no cross-job persistence and no untrusted-workload access. Prefer a GitHub-hosted or otherwise dedicated isolated runner, minimize the Vault grants to the exact paths needed, and remove any permission (including `id-token: write`) that the secret-loading implementation does not require.</recommendation>
</violation>
</file>

[superagent-security]

P3: All actions are pinned to mutable tags instead of commit SHAs in a secret-bearing workflow

All five actions (incl. third-party pnpm/action-setup and nscloud-setup@v0) are pinned to mutable tags, not commit SHAs.

Pin each uses: to a full commit SHA with a # vX.Y.Z comment so retagged action releases cannot silently alter this privileged job.

AI prompt

Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name=".github/workflows/ai-gateway-model-index.yml">
<violation number="1" location=".github/workflows/ai-gateway-model-index.yml:54">
<priority>P3</priority>
<title>All actions are pinned to mutable tags instead of commit SHAs in a secret-bearing workflow</title>
<evidence>Every `uses:` line is tag-pinned: actions/checkout@v4, pnpm/action-setup@v4, actions/setup-node@v4, namespacelabs/nscloud-setup@v0, actions/upload-artifact@v4. Tags are mutable and can be force-pushed if an action repository (including the third-party pnpm/action-setup and namespacelabs/nscloud-setup, the latter on a loose @v0 line) is compromised; this job runs with vault secrets, contents: write, and id-token: write, so a retagged action would execute in a privileged context.</evidence>
<recommendation>Pin each action to its full 40-character commit SHA with a version comment, e.g. `uses: actions/checkout@&lt;sha&gt; # v4.2.2`, prioritizing the third-party actions (pnpm/action-setup, namespacelabs/nscloud-setup), and keep pins updated via Dependabot/Renovate.</recommendation>
</violation>
</file>

[superagent-security]

P2: Dependency installation runs arbitrary lifecycle code in a privileged self-hosted job

pnpm install can run lifecycle hooks in a self-hosted job with repository-write and OIDC/Vault privileges.

Use --ignore-scripts where possible and isolate install/build from Vault access and contents:write permissions.

AI prompt

Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name=".github/workflows/ai-gateway-model-index.yml">
<violation number="1" location=".github/workflows/ai-gateway-model-index.yml:61">
<priority>P2</priority>
<title>Dependency installation runs arbitrary lifecycle code in a privileged self-hosted job</title>
<evidence>The workflow runs `pnpm install --frozen-lockfile` on a custom Namespace runner while the job has `contents: write`, `id-token: write`, and additional Vault permissions. pnpm installation can execute package-manager lifecycle scripts and install hooks; the same runner is subsequently used to load many API credentials and commit results. The workflow is not fork-PR-triggered, but this creates a large blast radius for a compromised dependency or persistent self-hosted runner.</evidence>
<recommendation>Verify that installation scripts are required, and prefer `pnpm install --frozen-lockfile --ignore-scripts` followed by an explicit allowlist of required build steps. Run dependency installation and builds on an ephemeral isolated runner with only the minimum token permissions; keep Vault access and repository write access in a separate narrowly scoped step or job.</recommendation>
</violation>
</file>

[devin-ai-integration]

Devin Review found 2 new potential issues.

7 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

[devin-ai-integration]

🟡 Image prices get token units

When a catalog exposes pricing.image, findPricingUnit misses it and assigns per_token. Current image entries are published with a unit contradicting their per-image prices.

Suggested change

	image_generation: 'per_image',
	image_generation: 'per_image',
	image: 'per_image',

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🟡 Mixed prices receive one wrong unit

When token prices accompany a duration surcharge, findPricingUnit selects the surcharge unit for the whole pricing object. Published token fields then appear to charge per second.

Prompt for agents

The singular pricingUnit is documented as applying to primary input/output pricing, but inferPricingUnit recursively scans every field and lets specialized fields such as transcription_duration_cost_per_second override those primary token prices. Some catalog entries contain both units. Update the representation or inference in benchmarks/ai-gateway/model-index-task.ts and benchmarks/ai-gateway/types.ts so each exposed price retains its own unit, or ensure pricingUnit describes only the primary input/output fields without being overridden by supplemental charges.

---

Was this helpful? React with 👍 or 👎 to provide feedback.