Skip to content

feat(agent-tasks): daily-ticket-status-drafts A/B harness (v0)#42

Open
alycda wants to merge 6 commits into
mainfrom
feat/ticket-drafts-ab-harness
Open

feat(agent-tasks): daily-ticket-status-drafts A/B harness (v0)#42
alycda wants to merge 6 commits into
mainfrom
feat/ticket-drafts-ab-harness

Conversation

@alycda

@alycda alycda commented Jul 9, 2026

Copy link
Copy Markdown
Owner

First Cowork scheduled task migrated to a runtime-agnostic harness, for an A/B quality comparison against the still-enabled Cowork incumbent (through 2026-09-01). Plan/requirements/ideation live in the private migration corpus (~/agentic-migration/docs/).

What

  • tools/agents/tasks/daily-ticket-status-drafts/ — task bundle: README, codex --output-schema contract (drafts XOR evidence-carrying skip, per-ticket dispositions, complete attestation), and the PreToolUse hook gate (hooks/never-post-linear.json: probe + deny-hook, hash-trusted install).
  • tools/agents/bin/ticket-drafts-run — drafting arm wrapper. Default-red: green requires pending drafts or an evidence-carrying skip. Distinct red taxonomy (auth | connector | rate-limit | timeout | malformed-output | incomplete | evidence-free-skip | placeholder-secret) so arm-unavailable days never count as quality data. Codex runs read-only sandbox; the wrapper owns all outbox writes; records named <date>-<ticket>-<runid>.json with supersede-by-state-transition (never overwrite). Sentinel refusal prevents drafting from the committed placeholder prompt.
  • tools/agents/bin/ticket-drafts-review — the only posting path (holds the Linear key). Per-draft approve / edit-then-approve / deny; approval freezes final text; posting is idempotent (attempt marker + frozen-body match on retry; editing a failed post revokes approval). Lazy calendar-date expiry with post_failed/approved exempt and resurfaced first. Cross-arm duplicate warning (24h window vs UTC-safe timestamps). Every decision → decisions.jsonl (the A/B dataset). Key passed to curl via fd, never argv.
  • home-manager/modules/tools/agent-tasks.nix — work-scoped, mkIf isDarwin-guarded (work.nix also backs the alyssa@work-dev Linux devcontainer — verified the guard: Linux config gains no secrets). Deploys scripts via writeShellApplication (shellcheck at build; jq/curl/rage/coreutils pinned) and the bundle to ~/.agents/tasks/.
  • justfile: ticket-drafts, ticket-drafts-one TICKET (injection-safe via quote()), ticket-drafts-review. Checkout scripts outrank stale PATH copies; everything works pre-activation via rage-direct secret fallback.

Verification

  • just ci clean; both script derivations build (shellcheck green).
  • Full darwin-rebuild switch on the dotfiles-ci tart VM: exit 0; scripts on PATH; bundle deployed; both secrets decrypt at 0400; expected-red --dry-run (auth, codex absent) writes its run record — default-red proven.
  • Offline scenario tests: sentinel refusal, invalid-ticket red, stale-pending spike, post_failed/approved midnight exemption + resurface, dry-run spikes excluded from decisions.jsonl, injection attempt reds at the wrapper.
  • Code review (8-angle find + verify) applied 11 fixes pre-merge, incl. a bash ${2:-{}} expansion bug that would have broken every Linear GraphQL call, and GNU-mktemp incompatibility on the nix-deployed path.

Before first real run (user actions)

  1. agenix -e secrets/personal/ticket-drafts-prompt.age — real prompt (adapted Cowork SKILL.md + comment-patterns.md).
  2. Mint a Linear personal API key → agenix -e secrets/personal/linear-api-key.age.
  3. Hook gate probe per hooks/never-post-linear.json, then echo true|false > ~/.agents/tasks/daily-ticket-status-drafts/hook-gate-state.

Known residuals

  • Linear GraphQL field shapes (issue(id:) accepting identifiers, user.isMe, commentCreate client-ID support) are unverifiable until the key exists — ticket-drafts-review isolates them in one documented integration point.
  • Hook-visibility of account-level connectors is undetermined until the probe runs; if invisible, the frozen-snapshot increment gets promoted per the plan.
  • Codex auth detection = codex login status exit code (probed on 0.143.0).

🤖 Generated with Claude Code

alycda and others added 6 commits July 9, 2026 13:43
…mktemp, stranded-approved retry, injection guard, provenance state file

Confirmed bugs fixed: bash ${2:-{}} expansion corrupted every Linear GraphQL
call; GNU mktemp -t dies under the nix-deployed coreutils PATH; records stuck
in state=approved after an interrupt/decline were invisible to review; invalid
ticket-id drafts were dropped while still reporting green with the full count;
success-path stderr chatter could fabricate rate-limit reds; unquoted {{TICKET}}
allowed shell injection via just; dry-run spikes polluted decisions.jsonl; UTC
createdAt vs local-date prefix missed evening duplicates (now a 24h window);
hook_gate_active provenance now reads a state file the probe flow writes;
--repo now outranks the deployed schema copy; checkout scripts outrank stale
PATH copies in the just recipes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant