[Contrib] Add SHAPER skill-harness evolution

[Control-derek]

Summary

• add SHAPER, a contrib algorithm for sequentially evolving a planner skill and an executable context harness around a frozen agent
• add runnable VLABench and ESI-Bench integrations with benchmark-specific prompts, observable-context contracts, environment checks, and train/evaluate entry points
• add setup and launch scripts for pinned benchmark revisions, the OpenPI VLABench actor, and an OpenAI-compatible multimodal planner

SHAPER uses rollout-level diagnostics, fixed validation data, branched artifact proposals, beam selection, and historical-best tracking. The skill is optimized first; the selected skill is then frozen while the context harness is optimized.

Paper: https://arxiv.org/abs/2608.11350

Benchmark integrations

VLABench

• runs the official simulator and reward implementation
• executes planner subgoals through a frozen OpenPI websocket policy
• includes the benchmark-specific judger, summarizer, skill optimizer, and harness optimizer prompts

ESI-Bench

• runs the pinned official active-exploration pipeline in isolated simulator processes
• routes primary and task-specific planner contexts through the selected harness
• limits harness inputs to official observable RGB/reference evidence, actions, reasoning, confidence, and sanitized action results

Usage

contrib/recipes/shaper/README.md documents environment setup and the training/evaluation commands. Runtime paths and endpoints are collected in run_vlabench.sh and run_esi_bench.sh.

The bundled vLLM launcher defaults to the official Qwen3.6-27B multimodal configuration: eight-way tensor parallelism, 262,144-token context, and the Qwen3 reasoning parser. Model and serving parameters remain configurable.

Validation

• Black and isort checks
• Pyright: 0 errors
• repository pre-commit hooks
• isolated core/contrib package installation and public import
• planner launcher argument expansion and VLABench/ESI-Bench planner configuration checks

Model weights, simulator assets, credentials, rollout outputs, and experiment results are not included.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a complete SHAPER (skill + context-harness evolution) implementation and runnable benchmark recipes for VLABench and ESI-Bench, including environment bootstrapping utilities and a separately packaged runtime extension.

Changes:

• Introduces VLABench + ESI-Bench SHAPER bundles (agents, datasets, role protocols, preflight checks, entry points).
• Adds reproducible setup/run scripts and pinned benchmark/actor identity contracts.
• Adds the agentlightning-contrib-shaper runtime extension package and typing stubs for repo-local static analysis.

Reviewed changes

Copilot reviewed 80 out of 80 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
contrib/recipes/shaper/vlabench/train.py	Adds VLABench SHAPER train entry point with environment preflight.
contrib/recipes/shaper/vlabench/roles.py	Implements VLABench-specific judger/summarizer/optimizer role protocol wiring.
contrib/recipes/shaper/vlabench/requirements-simulator.txt	Pins simulator-side dependencies for VLABench rollouts.
contrib/recipes/shaper/vlabench/prompts/skill_optimizer_user.txt	Adds VLABench skill optimizer user prompt template.
contrib/recipes/shaper/vlabench/prompts/skill_optimizer.txt	Adds VLABench skill optimizer system prompt.
contrib/recipes/shaper/vlabench/prompts/seed_skill.txt	Adds seed VLABench planner skill prompt.
contrib/recipes/shaper/vlabench/prompts/seed_harness.py	Adds seed VLABench context harness.
contrib/recipes/shaper/vlabench/prompts/round_judger_user.txt	Adds VLABench round judger user prompt template.
contrib/recipes/shaper/vlabench/prompts/round_judger.txt	Adds VLABench round judger system prompt.
contrib/recipes/shaper/vlabench/prompts/harness_optimizer_user.txt	Adds VLABench harness optimizer user prompt template.
contrib/recipes/shaper/vlabench/prompts/harness_optimizer.txt	Adds VLABench harness optimizer system prompt.
contrib/recipes/shaper/vlabench/prompts/episode_summarizer_user.txt	Adds VLABench episode summarizer user prompt template.
contrib/recipes/shaper/vlabench/prompts/episode_summarizer.txt	Adds VLABench episode summarizer system prompt.
contrib/recipes/shaper/vlabench/openpi_server.py	Adds a dependency-light OpenPI websocket policy server with pinned metadata.
contrib/recipes/shaper/vlabench/openpi_identity.py	Adds websocket metadata handshake + validation for the frozen actor.
contrib/recipes/shaper/vlabench/factory.py	Builds the VLABench SHAPER ReproductionBundle from environment variables.
contrib/recipes/shaper/vlabench/evaluate.py	Adds VLABench SHAPER evaluation entry point with preflight.
contrib/recipes/shaper/vlabench/dataset.py	Implements deterministic VLABench 15/24 episode split materialization.
contrib/recipes/shaper/vlabench/contracts.py	Adds VLABench artifact contracts, upstream pin checks, and harness validation setup.
contrib/recipes/shaper/vlabench/check_env.py	Adds non-invasive environment preflight checks (assets, actor, planner).
contrib/recipes/shaper/vlabench/actor_contract.py	Pins OpenPI repo/commit and checkpoint identity manifest hashing.
contrib/recipes/shaper/vlabench/init.py	Provides lazy imports to avoid heavy deps in non-simulator contexts.
contrib/recipes/shaper/typings/agentlightning/contrib/shaper/init.pyi	Adds repo-local typing bridge for the contrib runtime extension.
contrib/recipes/shaper/scripts/start_shaper_vlabench_actor.sh	Adds script to start the pinned OpenPI actor server.
contrib/recipes/shaper/scripts/start_shaper_planner_vllm.sh	Adds helper to run an OpenAI-compatible planner via vLLM.
contrib/recipes/shaper/scripts/run_vlabench.sh	Adds end-to-end runner script for VLABench check/train/eval.
contrib/recipes/shaper/scripts/run_esi_bench.sh	Adds end-to-end runner script for ESI-Bench check/train/eval.
contrib/recipes/shaper/scripts/download_shaper_vlabench_actor.py	Adds checkpoint download + identity verification helper.
contrib/recipes/shaper/scripts/checkout_shaper_benchmarks.sh	Adds pinned benchmark checkout script for VLABench/OpenPI/ESI/BEHAVIOR.
contrib/recipes/shaper/scripts/bootstrap_shaper_environment.sh	Adds bootstrap automation for common + benchmark-specific environments.
contrib/recipes/shaper/reproduce.py	Adds generic SHAPER reproduction runner for ReproductionBundle factories.
contrib/recipes/shaper/pyrightconfig.json	Adds strict pyright config for the recipe and extension code.
contrib/recipes/shaper/integration.py	Adds helpers to extract artifacts and load validated harness runtimes.
contrib/recipes/shaper/harness_bridge.py	Adds a private Unix-socket JSON bridge for harness execution from workers.
contrib/recipes/shaper/evaluate.py	Adds a generic evaluation runner for SHAPER artifacts on a bundle split.
contrib/recipes/shaper/esi_bench/train.py	Adds ESI-Bench SHAPER train entry point with environment preflight.
contrib/recipes/shaper/esi_bench/splits/reported_eval231.txt	Adds ESI-Bench reporting subset manifest.
contrib/recipes/shaper/esi_bench/splits/recipe_validation10.txt	Adds deterministic ESI-Bench validation recipe split.
contrib/recipes/shaper/esi_bench/splits/recipe_train10.txt	Adds deterministic ESI-Bench training recipe split.
contrib/recipes/shaper/esi_bench/splits/recipe_metadata.json	Records provenance for the contrib ESI-Bench recipe splits.
contrib/recipes/shaper/esi_bench/splits/README.md	Documents the purpose/constraints of the provided ESI-Bench recipe splits.
contrib/recipes/shaper/esi_bench/prompts/skill_optimizer_user.txt	Adds ESI-Bench skill optimizer user prompt template.
contrib/recipes/shaper/esi_bench/prompts/skill_optimizer.txt	Adds ESI-Bench skill optimizer system prompt.
contrib/recipes/shaper/esi_bench/prompts/seed_skill.txt	Adds ESI-Bench seed skill placeholder prompt.
contrib/recipes/shaper/esi_bench/prompts/seed_harness.py	Adds ESI-Bench seed context harness implementation.
contrib/recipes/shaper/esi_bench/prompts/round_judger_user.txt	Adds ESI-Bench judger user prompt template (includes official labels).
contrib/recipes/shaper/esi_bench/prompts/round_judger.txt	Adds ESI-Bench judger system prompt with taxonomy and scoring rules.
contrib/recipes/shaper/esi_bench/prompts/harness_optimizer_user.txt	Adds ESI-Bench harness optimizer user prompt template.
contrib/recipes/shaper/esi_bench/prompts/harness_optimizer.txt	Adds ESI-Bench harness optimizer system prompt and sandbox contract.
contrib/recipes/shaper/esi_bench/prompts/episode_summarizer_user.txt	Adds ESI-Bench batch summarizer user prompt template.
contrib/recipes/shaper/esi_bench/prompts/episode_summarizer.txt	Adds ESI-Bench batch summarizer system prompt.
contrib/recipes/shaper/esi_bench/patches/behavior_floor_maps.patch	Adds the required BEHAVIOR map-gen patch artifact for ESI-Bench.
contrib/recipes/shaper/esi_bench/factory.py	Builds the ESI-Bench SHAPER ReproductionBundle from environment variables.
contrib/recipes/shaper/esi_bench/evaluate.py	Adds ESI-Bench SHAPER evaluation entry point with preflight.
contrib/recipes/shaper/esi_bench/dataset.py	Adds explicit split loading while keeping labels out of planner inputs.
contrib/recipes/shaper/esi_bench/check_env.py	Adds static environment checks for ESI-Bench/OmniGibson without launching Isaac.
contrib/recipes/shaper/esi_bench/agent.py	Adds an Agent Lightning wrapper that runs the official ESI pipeline in a fresh process.
contrib/recipes/shaper/esi_bench/init.py	Provides lazy imports to avoid Agent Lightning in the isolated worker.
contrib/recipes/shaper/common.py	Adds shared helpers for prompt/LLM access, content normalization, and pin checks.
contrib/recipes/shaper/cli.py	Adds shared CLI helpers for preflight and endpoint parsing.
contrib/recipes/shaper/init.py	Adds recipe package marker.
contrib/recipes/shaper/README.md	Adds end-to-end documentation for installing, bootstrapping, and running both benchmarks.
contrib/agentlightning/contrib/shaper/types.py	Adds typed Pydantic records for traces, candidates, and optimization events.
contrib/agentlightning/contrib/shaper/trace.py	Adds trace emission helpers and a trace adapter to build EpisodeTrace.
contrib/agentlightning/contrib/shaper/roles.py	Adds role-protocol interfaces and request/response dataclasses.
contrib/agentlightning/contrib/shaper/pyproject.toml	Defines the separately packaged SHAPER runtime extension.
contrib/agentlightning/contrib/shaper/prompts/skill_optimizer.txt	Adds default extension skill optimizer role prompt.
contrib/agentlightning/contrib/shaper/prompts/round_judger.txt	Adds default extension round judger role prompt.
contrib/agentlightning/contrib/shaper/prompts/harness_optimizer.txt	Adds default extension harness optimizer role prompt.
contrib/agentlightning/contrib/shaper/prompts/episode_summarizer.txt	Adds default extension episode summarizer role prompt.
contrib/agentlightning/contrib/shaper/prompting.py	Adds prompt loading + strict JSON parsing helper.
contrib/agentlightning/contrib/shaper/init.py	Exposes the extension public API surface.
contrib/agentlightning/contrib/shaper/README.md	Documents the runtime extension and its public API + safety scope.
contrib/CODEOWNERS	Adds code ownership entries for the new shaper recipe and runtime extension.

Suppressed comments (3)

contrib/recipes/shaper/vlabench/prompts/harness_optimizer.txt:1

• The VLABench harness optimizer prompt describes an in-process runtime with encode_image(), attribute-based RoundRecords, and permission to call llm_client, but this recipe’s harness contract (and seed harness) uses a JSON-only sandbox with list-of-dicts history and forbids network calls. This internal contradiction is likely to degrade optimizer output quality; update this prompt to match the actual VLABench harness contract (or remove the conflicting sections) so the model isn’t trained on an incorrect interface.
  contrib/recipes/shaper/vlabench/prompts/harness_optimizer.txt:1
• The VLABench harness optimizer prompt describes an in-process runtime with encode_image(), attribute-based RoundRecords, and permission to call llm_client, but this recipe’s harness contract (and seed harness) uses a JSON-only sandbox with list-of-dicts history and forbids network calls. This internal contradiction is likely to degrade optimizer output quality; update this prompt to match the actual VLABench harness contract (or remove the conflicting sections) so the model isn’t trained on an incorrect interface.
  contrib/recipes/shaper/vlabench/requirements-simulator.txt:1
• Using an editable VCS dependency in a requirements file reduces reproducibility (install behavior can differ across pip versions and editable installs can affect isolation). If editability isn’t required, prefer a non-editable, PEP 508 style direct reference pinned to the commit so installs are more deterministic and closer to production usage.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Control-derek]

@microsoft-github-policy-service agree