Skip to content

refactor(cli): ship one Eval runtime in the npm CLI package - #3946

Open
liuxiaocs7 wants to merge 1 commit into
apache:mainfrom
liuxiaocs7:liuxiaocs7/refactor-cli-ship-one-eval-runtime-in-the-npm-cl
Open

refactor(cli): ship one Eval runtime in the npm CLI package#3946
liuxiaocs7 wants to merge 1 commit into
apache:mainfrom
liuxiaocs7:liuxiaocs7/refactor-cli-ship-one-eval-runtime-in-the-npm-cl

Conversation

@liuxiaocs7

@liuxiaocs7 liuxiaocs7 commented Aug 26, 2026

Copy link
Copy Markdown
Member

Summary

copyEvalMirror() in scripts/release-cli-package.mjs copied the entire staged node_modules/@maka/eval tree into packages/eval inside the published CLI package, so every install carried the Eval runtime twice and the two copies could diverge between releases.

The mirror is not pure redundancy. Eval containers bind-mount the CLI package root read-only at /opt/maka-agent (packages/cli/src/eval-bundle-path.ts sets MAKA_EVAL_MAKA_BUNDLE_PATH to the package root; the experiment mount resolves in packages/eval/src/harness-executor.ts), so /opt/maka-agent/packages/eval/... resolves to the mirror. Only a small container-facing subset is actually read from it; the host Eval runtime is read once from node_modules/@maka/eval.

stageEvalMirror() now ships only that subset (16 files, down from the whole runtime tree):

  • Container entry points (launched inside eval cells): dist/harbor-external-subject.js, dist/harbor-maka-subject.js.
  • Their transitive import closure (8 more dist/*.js).
  • Harbor data / egress assets the entry points resolve: harbor/deepseek-codex-models.json, the harbor/deepseek-harness-profile directory, harbor/docker-compose-egress-proxy.yaml, harbor/egress-proxy/network-policy.

Dropped from the mirror (all remain in node_modules/@maka/eval via releaseFiles): the host-only dist/*.js modules; the harbor Python launchers (read from node_modules/@maka/eval/harbor via BUNDLED_HARNESS_RELAY_ROOT, not through the mount); and the egress image-build inputs (CI-only). The relay_agent.py/run_trial.py release guards move from the mirror to node_modules/@maka/eval, matching where the runtime resolves them.

maka eval remains the only public Eval CLI. Behavior, standalone packaging, and every container path the experiments depend on are unchanged.

Fixes #3933

The reduction is proven structurally, not by text heuristics

  • Real parser for the closure. The import graph is walked with acorn, so imports split across comments are followed and a string that merely contains import( is not mistaken for a dependency. A dynamic import() with a computed specifier is refused (fail closed) because its closure cannot be resolved.
  • Structured config checking. Experiment configs are JSON.parsed and every string value is checked; each packages/eval reference is normalized (resolving ./..) and rejected if it escapes the mirror, so neither path traversal (.../deepseek-harness-profile/../secret.json) nor a JSON-escaped slash (packages\/eval\/...) can name an unshipped file undetected.
  • Single directory authority. deepseek-harness-profile is declared once, as a directory, in the eval package's releaseFiles; staging and the mirror both copy it whole, so a newly required profile file cannot reach the runtime yet be missed by the mirror.
  • Production-wiring assertion. validatePackedFiles recomputes the mirror inventory from the staged runtime independently of copyEvalMirror and asserts the packed packages/eval files match it exactly — so reverting to a whole-tree copy, or dropping a required file, fails the release. The installed smoke test additionally asserts no host-only file appears under the mirror.

Verification

  • node --test scripts/release-cli-file-policy.test.mjs — 24/24 pass, incl. comment-split imports, computed-import() rejection, path-traversal and JSON-escape rejection, the exact fixture inventory (a whole-tree copy fails it), and assertPackedEvalMirror rejecting a reintroduced host-only file. Affected suites together: 32/32.
  • Real development release build (node scripts/release-cli-package.mjs --development) succeeded end to end: the profile directory is copied whole into node_modules/@maka/eval, the mirror stages to exactly 16 files, and validatePackedFiles (with the new packed-mirror assertion) passed. Tarball: 13.7 MiB compressed, 6408 files.
  • biome check (lint + format) clean on all changed files.
  • Not run locally (needs Docker; runs in CI): release-cli-eval-package.mjs, which installs the tarball and runs real Harbor + Pier Docker cells. Please confirm the release:cli:eval job on CI.

Dependency

Adds acorn (already present transitively) as a direct devDependency so the release script can depend on it for parsing.

AI use

Select exactly one:

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: Claude Code (Opus) — mapped the runtime consumption of the mirror, implemented stageEvalMirror and its tests, and drafted this description. A human contributor reviewed and owns the change. The commit carries a Generated-by: Claude Opus trailer.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No

@liuxiaocs7
liuxiaocs7 force-pushed the liuxiaocs7/refactor-cli-ship-one-eval-runtime-in-the-npm-cl branch 2 times, most recently from 2927cd0 to 1f5e94d Compare August 26, 2026 18:46
@github-actions github-actions Bot added the effort/L Under 1000 readable lines label Aug 27, 2026
@liuxiaocs7
liuxiaocs7 force-pushed the liuxiaocs7/refactor-cli-ship-one-eval-runtime-in-the-npm-cl branch from 1f5e94d to 03c38e6 Compare August 27, 2026 04:17
copyEvalMirror() copied the whole staged node_modules/@maka/eval tree into
packages/eval, so every install carried the Eval runtime twice and the two
copies could diverge between releases.

The mirror is not pure redundancy: eval containers bind-mount the CLI package
root read-only at /opt/maka-agent (eval-bundle-path.ts sets
MAKA_EVAL_MAKA_BUNDLE_PATH to the package root; the experiment mount resolves in
harness-executor.ts), so /opt/maka-agent/packages/eval/... resolves to the
mirror. Only a small container-facing subset is actually read from it; the host
Eval runtime is read once from node_modules/@maka/eval.

stageEvalMirror() now ships only that subset (16 files): the two container entry
points launched inside eval cells — dist/harbor-external-subject.js and
dist/harbor-maka-subject.js — plus their transitive import closure and the
harbor assets those entry points resolve (deepseek-codex-models.json, the
deepseek-harness-profile directory, docker-compose-egress-proxy.yaml, and the
egress-proxy network-policy). The host-only dist modules, the harbor Python
launchers, and the egress image-build inputs are dropped from the mirror; they
remain in node_modules/@maka/eval (via releaseFiles), which is where the runtime
reads the launchers through BUNDLED_HARNESS_RELAY_ROOT and where the egress image
is built. The relay_agent.py and run_trial.py release guards move to
node_modules/@maka/eval to match. maka eval remains the only public Eval CLI.

The reduction is proven structurally, not by text heuristics:

- The import closure is computed with a real parser (acorn), so imports split
  across comments are followed and a string that merely contains "import(" is
  not mistaken for a dependency. A dynamic import with a computed specifier is
  refused (fail closed) because its closure cannot be resolved.
- Experiment configs are JSON-parsed and every string value is checked; each
  packages/eval reference is normalized (resolving . and ..) and rejected if it
  escapes the mirror, so neither path traversal nor a JSON-escaped slash can name
  an unshipped file undetected.
- The deepseek-harness-profile directory is declared once, as a directory, in the
  eval package's releaseFiles. Staging and the mirror both copy it whole, so a
  newly required profile file cannot reach the runtime yet be missed by the
  mirror.
- validatePackedFiles recomputes the mirror inventory from the staged runtime
  independently of copyEvalMirror and asserts the packed packages/eval files
  match it exactly, so reverting to a whole-tree copy — or dropping a required
  file — fails the release. The installed smoke test additionally asserts no
  host-only file appears under the mirror.

Generated-by: Claude Opus
@liuxiaocs7
liuxiaocs7 force-pushed the liuxiaocs7/refactor-cli-ship-one-eval-runtime-in-the-npm-cl branch from 03c38e6 to 09ab548 Compare August 27, 2026 04:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/L Under 1000 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

refactor(cli): ship one Eval runtime in the npm CLI package

1 participant