Pipeline construction for Loom/Orbit: a `loom` cast target that lowers pipelines into Loom plans

[jmchilton]

Filed by Claude (AI assistant) on behalf of @jmchilton — drafted in a design session, not authored personally by them.

Summary

Foundry deliberately disowns the harness (ARCHITECTURE §15, GUIDING_PRINCIPLES "Harnesses own orchestration"): sophisticated, stateful, resumable orchestration "lives in its own repos and the Foundry produces the artifacts they load." Loom/Orbit is exactly such a harness — a Galaxy co-scientist runtime (Pi.dev brain + thin shells) that already fetches skills from github.com/galaxyproject/* via a skills_fetch tool and treats SKILL.md as authoritative.

This issue proposes the seam: a loom cast target that lowers a Foundry pipeline note into a "Loom plan" the Loom harness can drive, plus the lowering rules and a generated discovery fragment. It answers §15 ("where do harnesses live") with a concrete first consumer, and gives Loom a principled, validated, decomposed workflow-authoring capability it lacks today (it currently routes to hand-authored monolithic skills like galaxy-skills/nf-to-galaxy — precisely the class of skill the Foundry exists to supersede).

This is the graduation path for the stop-gap /assemble-pipeline linear harness (§15, #282): instead of a trivial in-repo linear runner, the Foundry emits a per-pipeline artifact that a real downstream harness executes.

Background: what a "Loom plan" actually is

A Loom plan is not a typed object. It is a thin regex contract plus LLM-honored markdown convention. The entire machine-checked surface (Loom's init-gate.ts / evals parser) is:

• Heading ^## Plan <X>: <title> [routing]$, routing ∈ {local, galaxy, hybrid, remote}.
• Steps: - [ ] / - [x] / - [!] checkboxes within the section (until the next ##).
• Optional step anchor {#plan-x-step-N} (referenced by Loom's loom-invocation YAML notebookAnchor).
• Optional ## Project context block with history_id: / galaxy_url: key-values.
• A heuristic: step description ≥ 8 chars after stripping title/number/anchor.

Everything else (### Steps, the ### Parameters table, Routing: / Verification: sub-bullets) is convention the model is told to honor, not parsed. /execute is a prompt nudge: "find the latest plan with an unchecked step, do the next one, verify, flip the checkbox."

Consequence: lowering a pipeline into a plan is a text-templating problem, not a data-structure mapping, because the target is barely a data structure. Freeing, but it means there is no schema on the Loom side catching a malformed assembly — the lowering rules must be exact.

Proposal

Unit: one loom-cast skill per pipeline

A new cast target loom, parallel to claude/web/generic, that casts a pipeline note (not a Mold) into a single pipeline entry-point skill — fetched by Loom via skills_fetch. Framed as "author a Galaxy workflow from {an interview / a paper / a Nextflow pipeline / CWL}"; its first deterministic act is to write a Loom plan into notebook.md, after which it dissolves into Loom's normal /execute + per-step skills_fetch loop.

• Source of truth stays the pipeline note (content/pipelines/<slug>/index.md). The skill is a generated, provenanced, drift-checked artifact — never hand-written. No two-sources-of-truth drift.
• Lowering rules live in the shared loom target adapter, not per-skill, so all ~6 pipelines are generated by one rule set and stay consistent.

Two-level progressive disclosure (don't inline)

This falls straight out of the existing principle "Pipelines disclose the journey; Molds disclose the action":

• pipeline skill = journey layer → writes the plan spine.
• each plan step → skills_fetches that step's mold cast = action layer, on demand.

Keep the indirection. Do not inline mold content into the pipeline skill — the pipeline skill stays small (spine + lowering), each action's detail loads only when its step runs.

Phase-kind lowering rules

Pipeline phase	Loom plan lowering
{mold: [[x]]}	one - [ ] N. **<x>** {#anchor} step; action = skills_fetch the cast for x; Verification: sourced from the mold cast's SKILL.md "validation hints" section
{mold, loop: true}	one self-perpetuating step ("Implement next draft step using "). On completion: if the looped mold's own oracle (gxwf draft-next-step) reports another drafty step, rewrite the completed line to name what it resolved and insert a fresh copy of the step above the downstream tail; else check off and fall through. Cardinality is never pre-enumerated — even after the template skeleton, N is unknown (discover-or-author can split a TODO, draft-validate --concrete can bounce a step).
{branch: ...}	one step whose description encodes the fallback chain (e.g. find-test-data → else ask user). Maps onto Loom's existing soft-fail-ask-user posture. Distinct from loop (one-shot, not self-perpetuating).
future {gate}	a plan-approval checkpoint: present the plan, await approval before /execute. Already how Loom skills behave ("skills follow planning/approval checkpoints internally").

Loop representation decision: self-perpetuating step with insert-above (always exactly one pending "implement next" step while looping; new copies inserted above the static tail; stop inserting when the oracle says done). Trusting the model to do positional inserts; revisit if that proves unreliable (append-and-defer is the fallback).

Discovery (router fragment)

Loom's skill router is rich only for hardcoded galaxy-skills; other repos get generic treatment. The pipeline skills need a router entry ("user wants to build a workflow from {interview/paper/nextflow/cwl} → fetch the matching pipeline skill"). Foundry already generates Index.md/dashboard from pipeline notes; it can generate a router fragment from the same phases/source/target frontmatter for Loom to surface.

Tensions / open questions to resolve in design

1. Routing-axis clash (biggest). Foundry target: galaxy = the workflow we're building runs on Galaxy. Loom [galaxy]/[remote] = this step's compute runs on Galaxy. A workflow-authoring plan is mostly [local] (summarize, design briefs, gxwf validate, file edits) with a single Galaxy leg at run-workflow-test. So the lowered plan is [local]/[hybrid], never [remote] — naïvely mapping target=galaxy→[remote] would hard-fail Loom's init-gate on Windows remote-only and mislead the agent. Authoring may even warrant a distinct plan notion on the Loom side; Loom's routing axis doesn't capture "we are building a workflow." (Also: Loom docs say routing ∈ 3 values but the code regex accepts a 4th, [galaxy] — pin the canonical set before emitting tags.)
2. gxwf availability in Loom's runtime. Authoring casts assume design-time gxwf; Loom today has zero gxwf references (it's Galaxy-MCP + runtime artifact checks). Blocker for any authoring plan. Who provides it — bundled, assumed-installed, Galaxy-side?
3. eval.md is never cast. Loom steps want Verification: text, but the per-mold oracle lives in eval.md, which casting deliberately never packages. The only castable verification source is the mold cast SKILL.md "validation hints" section — confirm it's rich enough to populate a Loom Verification: sub-bullet, or add per-phase verification metadata to the pipeline note.
4. Inter-mold artifacts. Molds already declare default_filename; artifacts-as-files-in-the-analysis-dir fits Loom's directory-as-project + markdown-as-state model. Interacts with Galaxy pipeline Molds write fixed filenames to cwd — namespace per-source run #282 (fixed filenames to cwd → namespace per run).
5. Provenance loop. When Loom assembles a plan ← pipeline X @ foundry-commit Y using casts @ hashes Z, record it. A Loom-side loom-pipeline-binding fenced block fits Loom's existing pattern (loom-invocation, loom-galaxy-page) and its stated direction (notebook-resident structured emitters).
6. Execution tier needs Planemo hosted in Loom's runtime. Foundry's run-workflow-test/debug-* molds (and the run half of implement-galaxy-workflow-test) call Planemo — the hardened, deterministic workflow test-runner/verifier. Planemo stays the test/verify authority; the harness calls it, it is not replaced. For these phases to run, Loom must host Planemo as a first-class runtime capability — a Loom runtime/hosting question (where Planemo runs, against which Galaxy, how it sits under the exec-guard as a long-running background job), not a Foundry-side re-cast. This is parallel to, not overlapping with, Loom's loom-invocation/Galaxy-MCP analysis-run model. Tracked separately.

Scope

In scope: the loom cast target, the four phase-kind lowering rules, the generated router fragment, a worked INTERVIEW → GALAXY reference lowering. Out of scope (separate issues): the Loom-side loom-pipeline-binding block, gxwf and Planemo runtime hosting in Loom (the binding constraint — both are hardened deterministic CLIs Loom must host as first-class capabilities; gxwf is static and lighter, Planemo needs a Galaxy and long-running background execution).

Relationship to existing work

• Graduates the stop-gap /assemble-pipeline linear harness (ARCHITECTURE §15) into a real downstream-consumed artifact.
• Related: Galaxy pipeline Molds write fixed filenames to cwd — namespace per-source run #282 (per-run filename namespacing — inter-mold artifacts), Thread an open-questions ledger artifact through the source→Galaxy pipeline #281 (open-questions ledger threaded through the pipeline — a natural plan companion).

Acceptance

• A loom target adapter that casts content/pipelines/<slug>/index.md → a pipeline entry-point skill.
• The four lowering rules implemented in the shared adapter; INTERVIEW → GALAXY cast end-to-end as the reference.
• A generated router fragment enumerating the pipeline skills by source/target.
• Provenance + drift-check parity with existing cast targets.

[mvdbeek]

Love it, when that's ready i'm happy to test!