[Feat]: agent evaluation platform (prompt stats + AB test + LLM-as-judge + thumbs + branches)

[FireTable]

Feature: agent evaluation platform (prompt stats + AB test + LLM-as-judge + thumbs + branches)

A self-hosted evaluation layer for the chat agent. Captures every prompt execution, lets admins compare prompt versions head-to-head, runs LLM-as-judge scoring over a human-curated test set, AB-routes traffic between prompt variants, and surfaces user thumbs-up/down + assistant-ui message branching into the same dataset. The output is a single dashboard an admin uses to decide "should we ship the new system prompt?"

This is a sizable issue because it bundles 7 related surfaces that share infrastructure. They're listed in dependency order.

Goals

1. Capture every chat run with prompt version + tokens + latency + outcome. Backed by a new eval_run table, populated from the existing observability callback + the agent graph.
2. Compare prompt versions side-by-side in the admin dashboard: same input → different outputs, plus aggregate metrics (win rate, latency p50/p95, token cost).
3. LLM-as-judge scoring on a configurable rubric. Runs on a schedule over the test set; the judge model is separate from the agent model so it doesn't have the same blind spots.
4. Human-curated test sets, optionally referencing KB documents ([Feat]: knowledge-base agent (PDF/PNG/URL → markdown → RAG → graphRAG) #13). Each test case = { input, expected_output?, kb_document_ids?, rubric_weights? }.
5. AB routing between prompt variants — sticky per session, weighted by traffic %, with the assignment recorded so we can attribute outcomes.
6. Thumbs up/down on assistant messages. Lands in eval_feedback and joins against eval_run.
7. assistant-ui message branching — the existing <BranchPickerPrimitive> (verified in @assistant-ui/react@0.14.26/dist/index.d.ts) lets a user pick between alternative assistant responses. We track which branch the user kept and feed it back into the eval dataset.

Why a single feature, not seven

The seven sub-items share infrastructure: the eval_run row, the dashboard, the variant model. Splitting them into seven issues would mean each later issue re-derives the same schema. One issue, seven milestones.

Out of scope

• Hosted eval platforms (Braintrust, LangSmith, Langfuse). We build a lean self-hosted version; revisit if evaluation needs outgrow it.
• Multi-judge ensembles / inter-rater reliability stats — single-judge MVP.
• Online reinforcement learning (DPO / PPO loops). The data model is forward-compatible (we capture chosen vs rejected) but the RL training loop is a separate future issue.
• Public eval leaderboards / share links — single-tenant, operator-only dashboard.
• Per-tenant eval — single admin's view; multi-org is out of scope.

Architecture overview

                  ┌──────────────────────────────────────────────┐
                  │             agent graph (chat run)            │
                  │                                               │
                  │   START → quotaCheckNode → routerAgent → ...  │
                  │              ↓ (record prompt_variant_id)     │
                  │         recordEvalRunNode → END              │
                  └──────────────────────────────────────────────┘
                                       │
                                       ▼
                  ┌──────────────────────────────────────────────┐
                  │   eval_run table (one row per chat turn)      │
                  │   prompt_variant_id, tokens, latency,         │
                  │   branch_id, feedback_id (nullable)           │
                  └──────────────────────────────────────────────┘
                                       │
                ┌──────────────────────┼─────────────────────┐
                ▼                      ▼                     ▼
        thumbs up/down           branch picker       LLM-judge batch
        (per assistant msg)      (assistant-ui        (nightly / on-demand
                                  built-in)            over test set)
                │                      │                     │
                ▼                      ▼                     ▼
        eval_feedback          eval_branch            eval_judgment
                │                      │                     │
                └──────────────────────┴─────────────────────┘
                                       ▼
                          Admin dashboard @ /admin/eval
                          - runs list / detail
                          - compare 2 prompt versions
                          - test sets CRUD + run
                          - variants CRUD + AB weights
                          - thumbs distribution
                          - judge score distribution

Data model (Drizzle, lib/eval/schema.ts)

Prompt versions + variants

// One row per prompt template version. The agent reads "active variant" via
// a separate pointer table so we can change the active variant without
// rewriting history.
export const promptTemplate = pgTable("prompt_template", {
  id: text("id").primaryKey(),                          // nanoid
  agent: text("agent").notNull(),                       // "chat" | "weather" | "crypto" | "code" | "summarize" | ...
  parentId: text("parent_id"),                          // previous version (nullable for v1)
  content: text("content").notNull(),                   // the actual prompt body
  notes: text("notes"),                                 // admin's changelog
  createdByUserId: text("created_by_user_id").references(() => user.id),
  createdAt: timestamp("created_at").defaultNow().notNull(),
}, (t) => [index("prompt_template_agent_idx").on(t.agent)]);

// AB routing: a variant is a labeled branch with traffic weight.
// "active" variants sum to 100% (enforced server-side).
export const promptVariant = pgTable("prompt_variant", {
  id: text("id").primaryKey(),
  templateId: text("template_id").notNull().references(() => promptTemplate.id, { onDelete: "cascade" }),
  label: text("label").notNull(),                       // "control", "treatment-A", …
  trafficWeight: integer("traffic_weight").notNull(),   // 0..100
  enabled: boolean("enabled").notNull().default(true),
  createdAt: timestamp("created_at").defaultNow().notNull(),
}, (t) => [index("prompt_variant_template_idx").on(t.templateId)]);

// Sticky assignment per (user, template) so a user always sees the same
// variant during one experiment window. Re-assigned on variant disable.
export const promptVariantAssignment = pgTable("prompt_variant_assignment", {
  userId: text("user_id").notNull().references(() => user.id, { onDelete: "cascade" }),
  templateId: text("template_id").notNull().references(() => promptTemplate.id, { onDelete: "cascade" }),
  variantId: text("variant_id").notNull().references(() => promptVariant.id, { onDelete: "cascade" }),
  assignedAt: timestamp("assigned_at").defaultNow().notNull(),
}, (t) => [primaryKey({ columns: [t.userId, t.templateId] })]);

Runs + branches + feedback

export const evalRun = pgTable("eval_run", {
  id: text("id").primaryKey(),
  threadId: text("thread_id").notNull(),
  userId: text("user_id").notNull().references(() => user.id, { onDelete: "cascade" }),
  agent: text("agent").notNull(),                       // matches promptTemplate.agent
  templateId: text("template_id").notNull().references(() => promptTemplate.id),
  variantId: text("variant_id").notNull().references(() => promptVariant.id),
  branchId: text("branch_id"),                          // null if not branched
  inputTokens: integer("input_tokens"),
  outputTokens: integer("output_tokens"),
  totalMs: integer("total_ms").notNull(),
  status: text("status").notNull(),                     // 'success' | 'error' | 'rate_limited'
  errorMessage: text("error_message"),
  rubricId: text("rubric_id").references(() => evalRubric.id),
  kbDocumentIds: text("kb_document_ids").array(),       // empty if no @-mention
  createdAt: timestamp("created_at").defaultNow().notNull(),
}, (t) => [
  index("eval_run_user_idx").on(t.userId),
  index("eval_run_variant_idx").on(t.variantId),
  index("eval_run_thread_idx").on(t.threadId),
  index("eval_run_created_idx").on(t.createdAt),
]);

export const evalFeedback = pgTable("eval_feedback", {
  id: text("id").primaryKey(),
  runId: text("run_id").notNull().references(() => evalRun.id, { onDelete: "cascade" }),
  userId: text("user_id").notNull().references(() => user.id, { onDelete: "cascade" }),
  rating: text("rating").notNull(),                     // 'up' | 'down'
  reason: text("reason"),                               // optional free-text
  createdAt: timestamp("created_at").defaultNow().notNull(),
}, (t) => [
  index("eval_feedback_run_idx").on(t.runId),
  uniqueIndex("eval_feedback_user_run_idx").on(t.userId, t.runId),  // one rating per user per run
]);

Test sets + judge

export const evalTestSet = pgTable("eval_test_set", {
  id: text("id").primaryKey(),
  name: text("name").notNull(),
  description: text("description"),
  createdByUserId: text("created_by_user_id").references(() => user.id),
  createdAt: timestamp("created_at").defaultNow().notNull(),
});

export const evalTestCase = pgTable("eval_test_case", {
  id: text("id").primaryKey(),
  testSetId: text("test_set_id").notNull().references(() => evalTestSet.id, { onDelete: "cascade" }),
  ordinal: integer("ordinal").notNull(),
  input: text("input").notNull(),
  expectedOutput: text("expected_output"),              // optional golden answer
  kbDocumentIds: text("kb_document_ids").array(),       // optional @-mention references (#13)
  rubricWeights: jsonb("rubric_weights"),               // optional override of rubric default weights
});

export const evalRubric = pgTable("eval_rubric", {
  id: text("id").primaryKey(),
  name: text("name").notNull(),                         // "default", "code-correctness", …
  criteria: jsonb("criteria").notNull(),                // [{key, description, weight, scale}]
  judgeModel: text("judge_model").notNull(),            // "openai/gpt-4o-mini" etc.
  createdAt: timestamp("created_at").defaultNow().notNull(),
});

export const evalJudgment = pgTable("eval_judgment", {
  id: text("id").primaryKey(),
  runId: text("run_id").notNull().references(() => evalRun.id, { onDelete: "cascade" }),
  rubricId: text("rubric_id").notNull().references(() => evalRubric.id),
  scores: jsonb("scores").notNull(),                    // {relevance: 4, accuracy: 5, …}
  reasoning: text("reasoning"),                          // judge's explanation
  totalCostTokens: integer("total_cost_tokens"),
  createdAt: timestamp("created_at").defaultNow().notNull(),
}, (t) => [index("eval_judgment_run_idx").on(t.runId)]);

export const evalTestRun = pgTable("eval_test_run", {                    // batch run record
  id: text("id").primaryKey(),
  testSetId: text("test_set_id").notNull().references(() => evalTestSet.id),
  variantIds: text("variant_ids").array().notNull(),    // one or more variants tested in parallel
  rubricId: text("rubric_id").notNull().references(() => evalRubric.id),
  status: text("status").notNull(),                     // 'pending' | 'running' | 'completed' | 'failed'
  startedAt: timestamp("started_at").defaultNow().notNull(),
  completedAt: timestamp("completed_at"),
});

All new tables ON DELETE CASCADE from user (per the existing pattern in docs/DB.md). Update docs/DB.md in the same commit (rule #1 spirit).

Capture flow — agent integration

A new recordEvalRunNode runs at the END of the agent graph (after triggerBackgroundAgentNode), capturing the metadata of the just-finished turn:

async function recordEvalRunNode(state: AgentState): Promise<Partial<AgentState>> {
  const { userId, threadId, agent, templateId, variantId, branchId,
          inputTokens, outputTokens, totalMs, status, errorMessage,
          kbDocumentIds } = state;

  const id = nanoid(12);
  await db.insert(evalRun).values({
    id, userId, threadId, agent, templateId, variantId, branchId,
    inputTokens, outputTokens, totalMs, status, errorMessage, kbDocumentIds,
  });
  return { evalRunId: id };
}

templateId + variantId are populated by a new assignPromptVariantNode at the START of the agent graph (between quotaCheckNode and routerAgent). For non-admin users, it picks a variant per (userId, agent) — sticky via promptVariantAssignment. Admin users always get the "control" variant by default; admins can override per-thread.

AB routing algorithm

async function assignVariant(userId: string, agent: string): Promise<{ templateId: string; variantId: string }> {
  // 1. Check sticky assignment
  const existing = await db.select().from(promptVariantAssignment)
    .where(and(eq(userId, userId), eq(templateId, activeTemplateIdFor(agent))))
    .limit(1);
  if (existing.length) return existing[0];

  // 2. Otherwise pick weighted-random from enabled variants
  const variants = await db.select().from(promptVariant)
    .where(and(eq(templateId, activeTemplateIdFor(agent)), eq(enabled, true)));
  const pick = weightedPick(variants, v => v.trafficWeight);

  // 3. Persist the assignment
  await db.insert(promptVariantAssignment).values({ userId, templateId: pick.templateId, variantId: pick.variantId })
    .onConflictDoNothing();

  return { templateId: pick.templateId, variantId: pick.variantId };
}

weightedPick is a single-pass random — no bias, no PRNG state in DB. The DB-row sticky assignment is the consistency guarantee.

LLM-as-judge

// backend/eval/judge.ts
export async function judgeRun(runId: string, rubricId: string): Promise<EvalJudgment> {
  const run = await loadEvalRun(runId);
  const rubric = await loadRubric(rubricId);
  const prompt = buildJudgePrompt(run, rubric);        // see references below
  const out = await judgeModel.invoke(prompt);          // separate from agent model
  const { scores, reasoning } = parseJudgeOutput(out);
  await db.insert(evalJudgment).values({ id: nanoid(12), runId, rubricId, scores, reasoning, totalCostTokens: estimateTokens(out) });
}

Reference: the established LLM-as-judge pattern (Zheng et al., 2023 — "Judging LLM-as-a-Judge"). Keep the rubric dimensions explicit (relevance, accuracy, helpfulness, instruction-following); ask for a 1-5 per dimension plus a one-paragraph reasoning; require a JSON output envelope so parsing is robust. Hardcode a sensible default rubric (name: "default") seeded at startup.

The judge model is configured per-rubric (column judge_model). Default to a different model than the agent (cost + independence). Use the existing key pool (#14).

API surface (new routes, all withAuth + requireAdmin except feedback)

Route	Method	Auth	Notes
/api/eval/runs	GET	admin	Paginated runs list, filterable by variant / agent / date.
/api/eval/runs/[id]	GET	admin	Single run + linked feedback + judgment + branch.
/api/eval/runs/compare	POST	admin	Body: { variantIds, since, until } → aggregates.
/api/eval/templates	GET/POST/PATCH	admin	Prompt template CRUD.
/api/eval/variants	GET/POST/PATCH	admin	Variant CRUD + traffic weight updates.
/api/eval/test-sets	GET/POST	admin	Test set CRUD.
/api/eval/test-sets/[id]/cases	GET/POST/PATCH/DELETE	admin	Bulk case management; CSV import endpoint.
/api/eval/test-sets/[id]/run	POST	admin	Spawns an eval_test_run; background job.
/api/eval/test-runs/[id]	GET	admin	Batch run progress + per-case results.
/api/eval/rubrics	GET/POST	admin	Rubric CRUD.
/api/eval/feedback	POST	user	Body: { runId, rating, reason? }. ONE per (user, run).
/api/eval/feedback/[runId]	GET	user (own)	Returns the user's own rating.
/api/eval/branches/keep	POST	user	Body: { threadId, branchId }. Records which branch the user kept.

Rule #9: every route withAuth. Admin routes additionally gated by requireAdmin (#14). feedback is the only user-facing route — a user can rate their own runs only.

Update docs/APIS.md in the same commit (rule #1).

Frontend — admin dashboard

/admin/eval page (server component, requireAdmin guard, mirrors the Memory tab chrome — rule #11):

1. Overview cards — total runs (24h / 7d / 30d), thumbs-up rate, average judge score, p50/p95 latency, total tokens.
2. Runs table — paginated, filter by variant + agent + date range, click for run detail (input, output, branch tree, feedback, judgment scores).
3. Compare variants — pick 2-4 variants, side-by-side: latency box-plot, win rate from thumbs, judge score distribution, token cost. Render a simple bar chart via recharts (already in dep tree? verify; if not, hand-rolled SVG is fine per rule feat(backend): code sub-agent + lazy-register key-needing tools + shared UI primitives #5).
4. Templates — list prompt versions, diff view (we already have a JSON diff primitive in components/tool-ui/primitives/), create new version.
5. Variants — manage traffic weights (sliders), enable / disable, see current assignment distribution.
6. Test sets — list, edit, run, see past eval_test_runs with per-case results.
7. Rubrics — list / edit, set default.

Per-user scoping is automatic because all admin routes are requireAdmin and only return data for the admin viewing. No cross-tenant concerns.

Frontend — chat-side additions

Thumbs up/down

components/assistant-ui/thread.tsx already has message chrome. Add an action button row at the bottom of each assistant message:

<FeedbackButtons runId={run.evalRunId} initial={existingFeedback?.rating} />

Two icons (ThumbsUp, ThumbsDown per rule #8 — but rule #8 is for tool-UI; chat-thread icons are fine; verify). One click submits POST /api/eval/feedback. Subsequent clicks are debounced.

Branch picker

assistant-ui exposes <BranchPickerPrimitive> (verified in dist/index.d.ts). Wire it into the assistant message chrome:

<MessagePrimitive.If last>
  <BranchPickerPrimitive.Root>
    <BranchPickerPrimitive.Previous asChild><Button … /></BranchPickerPrimitive.Previous>
    <BranchPickerPrimitive.Count /> / <BranchPickerPrimitive.Total />
    <BranchPickerPrimitive.Next asChild><Button … /></BranchPickerPrimitive.Next>
  </BranchPickerPrimitive.Root>
</MessagePrimitive.If>

When the user picks a branch (or sends a new message after switching), POST /api/eval/branches/keep records it. The branchId flows into the next eval_run row so we can compute "this user prefers branch X of variant Y" stats.

Rule #8 says tool-UI buttons are text-only. Chat-thread action buttons can have icons because they're not tool cards — verify before shipping.

"Why these buttons?" — discoverability

A small inline hint the first time a user sees the feedback buttons: "👎 helps us improve — pick a reason". After the first interaction, hide.

Tool-call surface (rule #10)

// backend/tool/eval-tool.ts
export const getEvalSummaryTool: StructuredTool | null = process.env.EVAL_ENABLED
  ? tool(/* admin-only summary */, { name: "get_eval_summary", schema: … })
  : null;

Admin-only — guarded by role === 'admin' inside the tool's invoke. Lazy-registered (rule #10). Add a row to docs/TOOLS.md (rule #10).

Acceptance criteria

• Every chat run produces an eval_run row with template_id, variant_id, branch_id, tokens, latency, status.
• assignPromptVariantNode is the first node of the agent graph (after quotaCheckNode); sticky per (user, template).
• Traffic weights honored — admin can shift 100/0 → 50/50 and assignments change over time, never abruptly for the same user.
• Thumbs up/down button on each assistant message; one rating per (user, run); visible distribution in the dashboard.
• assistant-ui <BranchPickerPrimitive> wired; user's kept branch recorded via POST /api/eval/branches/keep.
• Test sets CRUD with CSV import.
• LLM-as-judge runs on a fixed rubric and produces eval_judgment rows; scores per-dimension + reasoning.
• Compare-variants view: side-by-side aggregates; admin can declare a winner ("promote variant X to 100%").
• Per-user isolation — users see only their own feedback; admin sees aggregate.
• pnpm lint, pnpm typecheck, pnpm test green.
• docs/APIS.md updated (rule feat(001): stage 1 — user auth (Better Auth + email verification + thread ownership) #1).
• docs/TOOLS.md updated (rule [Bug]: Memory tab Socials row missing email per linked provider #10).
• docs/DB.md updated with new tables.
• docs/AUTH.md env table updated (EVAL_ENABLED, JUDGE_MODEL).

TDD plan (rule #2)

1. tests/lib/eval/assign-variant.test.ts — seed 2 variants (70/30); assert that over 1000 simulated users, the assignment distribution is within ±5% of the weights.
2. tests/lib/eval/assign-variant.test.ts — sticky: call twice with the same user; assert the same variant comes back; disable one variant, assert re-assignment to the other.
3. tests/backend/agent/eval-record.test.ts — graph test: run a chat through agent; assert eval_run row exists with the right variant_id from the assignment table; assert branch_id matches the message's branch index.
4. tests/api/eval/feedback.test.ts — POST /api/eval/feedback twice with the same (user, runId); assert second is 409 (unique constraint), not a duplicate row.
5. tests/backend/eval/judge.test.ts — mocked judge model returns a JSON envelope; assert eval_judgment row has the parsed scores + reasoning; malformed JSON from judge → graceful error path + audit row, no crash.
6. tests/api/eval/compare.test.ts — seed 2 variants × 100 runs each; query /api/eval/runs/compare; assert win-rate + latency aggregates match the seeds.
7. tests/frontend/admin/eval-dashboard.test.tsx — render dashboard with mocked responses; assert per-admin data isolation (cross-admin doesn't render).
8. tests/observability/eval-coverage.test.ts — a thumbs-down on a chat run produces an observability span tagged eval_feedback_down so the existing per-turn panel can render it.

Notes for the implementer

• Capture is cheap; judge is expensive. Don't run the judge on every run — gate it on (a) explicit admin "judge this run" action, OR (b) nightly batch over recent runs, OR (c) all runs where rating === 'down'. Don't default to (a + b + c all on). Make it configurable.
• Prompt variants vs. prompt versions: a version is an immutable row; a variant is a label that points at a version. Don't conflate them. A/B experiment creates new variants pointing at the same or different versions.
• Don't run the judge on prompt X with the same model that produced prompt X's output. Use a different model (different family or different temperature) to break the bias loop. Make this a hard rule in the rubric config.
• Throttle LLM-as-judge at the system level — it's a new cost center. Default rate limit: 100 judgments per hour per admin. Configurable via env.
• Don't leak user PII into eval runs. The eval_run table stores user_id (FK) but the input/output payloads should be reviewed for PII before LLM-as-judge is invoked. For MVP, store the input/output as-is and trust the data classification of the underlying chat.
• CSV import for test cases — schema: input, expected_output (optional), kb_document_ids (semicolon-separated), rubric_weights (json). Validate on import; reject malformed rows with a per-row error report.
• The branch picker is built into assistant-ui — don't roll our own. Verified in @assistant-ui/react@0.14.26/dist/index.d.ts.
• Rule feat(backend): code sub-agent + lazy-register key-needing tools + shared UI primitives #5: comments only where the judge-prompt template / branch-picker quirks matter. Not on routine code.
• Rule [Feat]: Public landing page at / (replace auto-redirect to /chat) #11: dashboard chrome matches existing memory-view.tsx patterns; chart primitives match the stat-card pattern in observability.
• Self-host: no new services. Postgres-only. Operator can run an external judge model if they want.

Related

• [Feat]: chat attachments backed by Cloudflare R2 #12 — chat attachments (R2); kbDocumentIds on eval_run links to this.
• [Feat]: knowledge-base agent (PDF/PNG/URL → markdown → RAG → graphRAG) #13 — knowledge-base agent; @-mention of KB docs lands in eval_run.kbDocumentIds for attribution.
• [Feat]: admin-managed LLM API keys with sequential fallback #14 — admin role + LLM key pool; the requireAdmin gate + judge model keys come from here.
• [Feat]: per-user daily credit quota (free-tier rate limit) #15 — per-user daily credits; assignPromptVariantNode sits AFTER quotaCheckNode, so a quota-rejected run never consumes a variant slot.
• backend/agent.ts — where assignPromptVariantNode + recordEvalRunNode land.
• lib/observability/callback-collector.ts — span integration for thumbs / judge.
• @assistant-ui/react BranchPickerPrimitive — verified in dist/index.d.ts.
• docs/APIS.md — new routes (rule feat(001): stage 1 — user auth (Better Auth + email verification + thread ownership) #1).
• docs/DB.md — schema.
• docs/TOOLS.md — get_eval_summary (rule [Bug]: Memory tab Socials row missing email per linked provider #10).
• docs/AUTH.md — EVAL_ENABLED + JUDGE_MODEL env vars.