Add PR-level risk assessment score to the review pipeline

[maruiz93]

What happens

Fullsend has no composite risk score for pull requests. Several risk signals exist in isolation — protected-path detection in the review post-script (~18 prefixes), finding severity levels, triage risk classification, and RICE effort scoring — but they are evaluated independently by different agents with no unified model. A PR touching CODEOWNERS, CI workflows, and dependency files is treated the same as a docs-only PR in terms of review routing and effort.

What should happen

A PR-level risk assessment should produce a composite score (e.g., 1–5 scale from minimal to critical) before the review agent runs. The score should:

• Inform review effort: model selection (sonnet vs opus), sub-agent count, and whether human approval is required
• Be visible as a GitHub label (e.g., risk/low through risk/critical) and optionally as a PR comment with the score breakdown
• Gate auto-merge eligibility: only low-risk PRs qualify

The assessment should combine three tiers of signals: metadata-derivable signals (cheap, no LLM), git-history-derived signals (LLM-assisted), and linked-issue context signals (LLM-assisted).

Tier 1: Metadata signals (no LLM needed)

Dimension	Signal Source
Blast radius	Files changed count, lines changed
Path sensitivity	Protected path matches, security-sensitive files (.pem, .key, auth/, secrets/)
CI/workflow impact	Changes to .github/, Makefiles, Dockerfiles
Dependency risk	go.mod/go.sum, package.json, requirements.txt changes
Test coverage	Ratio of test files to source files in the diff
Author context	Bot vs human, first-time contributor

Tier 2: Git history signals (LLM-assisted)

These dimensions require parsing git history for the files touched by the PR. Raw git data (commit frequency, author counts, commit messages) is collected via CLI, then an LLM interprets the patterns.

Dimension	Signal Source
Churn hotspot	Files with high recent commit frequency — frequent changes correlate with higher defect rates. LLM contextualizes whether churn is healthy (active development) or concerning (repeated fixes)
Multi-author contention	Files recently touched by many different authors. LLM assesses whether the PR author is the primary maintainer or a newcomer to that code area
Recent regression history	git log --grep='fix' --grep='revert' on touched files. LLM distinguishes "fix typo" from "fix critical race condition" to weight severity
Change coupling	Files that historically change together but aren't all in the PR. LLM assesses whether missing co-changed files are intentional omissions or incomplete changes
Code age / stability	Files untouched for months/years carry higher risk when modified (Lindy effect). LLM assesses whether the change is a surgical fix or a broader refactor of stable code
Revert frequency	Prior reverts on touched files signal fragile areas
Commit message sentiment	History of "workaround", "hack", "temporary", "TODO" in commit messages signals technical debt zones where changes carry extra risk

Tier 3: Linked issue context (LLM-assisted)

When the PR links to an issue, the issue provides additional risk signals that complement the code-level analysis.

Dimension	Signal Source
Complexity vs scope mismatch	If the linked issue has a high RICE Effort score but the PR is small (or vice versa), the mismatch signals risk — oversimplification, incomplete fix, or scope creep
Issue label context	Labels like security, breaking-change, or priority/high on the linked issue directly inform risk beyond what file paths alone suggest
Acceptance criteria coverage	LLM compares the issue's requirements/acceptance criteria against what the PR actually changes, flagging potential gaps
Discussion history	Extensive debate, unresolved questions, or conflicting opinions on the issue indicate the change is contentious and more likely to need revision
Issue age and staleness	Issues open for months may have accumulated context drift — the codebase has evolved since the issue was filed, increasing risk of assumptions that no longer hold

Context

The prioritize agent already implements a RICE scoring model for issues (agents/agents/prioritize.md) with structured JSON output, GitHub Projects field updates, and issue comments. This proposal mirrors that pattern at the PR level.

The review agent's post-review script (agents/scripts/post-review.sh, lines 148–213) already implements protected-path blocking as a post-review gate — the most mature existing risk signal. The pre-review script (pre-review.sh) currently only validates inputs and PR state, making it the natural insertion point for a pre-review risk computation.

Architecture recommendation: Implement in phases:

1. Pre-review metadata step — Tier 1 signals computed in pre-review.sh with no LLM cost. Produces a baseline risk score and label.
2. LLM-assisted git history analysis — Tier 2 signals as a lightweight agent or review sub-agent that parses git log output for touched files and refines the risk score.
3. Linked issue context analysis — Tier 3 signals fetched via gh issue view on the linked issue (if any) and assessed by the same LLM step as Tier 2.

The combined score maps to risk levels:

• 1–2 (low): Sonnet-only review, auto-merge eligible
• 3 (medium): Standard review (current behavior)
• 4 (high): Opus-only review, human approval required
• 5 (critical): Opus-only review, senior reviewer required, blocks merge

Related issues:

• [feature-squad] Confidence-driven presentation: order agent output by risk and uncertainty #1340 — Confidence-driven presentation: order agent output by risk and uncertainty
• Prioritize review dispatch for bot-authored PRs that modify agent definitions #2988 — Prioritize review dispatch for bot-authored PRs that modify agent definitions
• Review agent post-script should derive 'Review effort' label from structured signals, not LLM prose format #4260 — Review agent post-script should derive 'Review effort' label from structured signals
• Review agent should flag large PRs and recommend splitting #2034 — Review agent should flag large PRs and recommend splitting
• Consider skipping or reducing agent effort on automated dependency update PRs #4177 — Consider skipping or reducing agent effort on automated dependency update PRs

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 8:47 AM UTC · Completed 8:51 AM UTC
Commit: 5cd495a · View workflow run →

[fullsend-ai-triage]

Triage Summary

Severity: Medium · Category: Feature

Problem

The review pipeline evaluates multiple risk signals independently — protected-path detection, finding severity, triage risk classification, RICE effort scoring — but has no unified model to produce a composite PR-level risk score. This means a high-blast-radius PR touching CI workflows, CODEOWNERS, and dependency files receives the same review treatment as a trivial docs change.

Recommended Approach

The phased architecture proposed in the issue is well-structured:

1. Phase 1 (Tier 1 — metadata signals): Compute file count, lines changed, protected-path matches, dependency file detection, test-to-source ratio, and author type in pre-review.sh. No LLM cost. Produces a baseline 1–5 risk score and risk/* label.
2. Phase 2 (Tier 2 — git history): LLM-assisted analysis of churn hotspots, multi-author contention, regression history, and code stability for touched files.
3. Phase 3 (Tier 3 — linked issue context): Cross-reference RICE effort, issue labels, acceptance criteria, and discussion history when a PR links to an issue.

The composite score should drive: model selection (sonnet vs opus), sub-agent count, human-approval gates, and auto-merge eligibility.

Related Issues

• Review agent post-script should derive 'Review effort' label from structured signals, not LLM prose format #4260 — Deriving the Review effort label from structured signals (subset; would be subsumed by this work)
• Consider skipping or reducing agent effort on automated dependency update PRs #4177 — Reducing agent effort on automated dependency PRs (downstream consumer of risk score)
• Prioritize review dispatch for bot-authored PRs that modify agent definitions #2988 — Prioritizing review dispatch for bot PRs modifying agent definitions (would use risk score)
• Review agent should flag large PRs and recommend splitting #2034 — Flagging large PRs and recommending splitting (blast radius is a Tier 1 signal)
• [feature-squad] Confidence-driven presentation: order agent output by risk and uncertainty #1340 — Confidence-driven presentation ordering (complementary)

Proposed Test Case

Tier 1 metadata risk scorer — synthetic PR metadata scenarios:

1. Docs-only PR (2 .md files, 10 lines, human author) → risk 1 (low)
2. Bot dependency bump (1 go.mod file, bot author) → risk 1–2
3. PR touching .github/workflows/, CODEOWNERS, and auth/ → risk 4–5 (high/critical)
4. Large PR (50 files, 2000 lines, 0 test files) → risk 3+ (blast radius + missing tests)
5. Verify score → label mapping and review config (model, sub-agents, auto-merge)

Edge cases:
- Test-only PR: low risk despite high file count
- First-time contributor on non-sensitive paths: moderate risk from author context
- Mixed sensitive/non-sensitive paths: risk reflects highest-sensitivity path

---

Labels: This is a feature request for the review agent pipeline, involving pre-review risk computation and dispatch/harness configuration. Medium priority — it improves review quality and efficiency but is not blocking current operations.

[maruiz93]

Architecture decision: review sub-agent with future pre-stage path

Current approach: Implement risk assessment as a new review sub-agent alongside the existing six (correctness, security, intent-coherence, style-conventions, docs-currency, cross-repo-contracts). The sub-agent computes all three tiers of signals (metadata, git history, and linked issue context) and produces structured risk findings. The post-review script reads the sub-agent's output, computes the composite risk score, and applies the risk/* label and score breakdown comment.

In this model, the risk score does not influence the current review's effort or model selection — it runs in parallel with the other sub-agents. The score is consumed by humans and downstream automation (merge gating, reviewer assignment).

Future approach: When the harness supports multi-agent stages (sequential phases within a review dispatch), the risk assessment sub-agent moves to a pre-review stage. The sequence becomes:

1. Risk assessment agent runs first (Tiers 1–3)
2. Review orchestrator reads the risk score and adjusts its strategy (model selection, sub-agent count, severity thresholds)
3. Post-review script applies labels from both

This is a non-breaking evolution — the same sub-agent definition works in both modes, only the dispatch ordering changes.