refactor(cli): ship one Eval runtime in the npm CLI package - #3946
Open
liuxiaocs7 wants to merge 1 commit into
Open
refactor(cli): ship one Eval runtime in the npm CLI package#3946liuxiaocs7 wants to merge 1 commit into
liuxiaocs7 wants to merge 1 commit into
Conversation
liuxiaocs7
force-pushed
the
liuxiaocs7/refactor-cli-ship-one-eval-runtime-in-the-npm-cl
branch
2 times, most recently
from
August 26, 2026 18:46
2927cd0 to
1f5e94d
Compare
liuxiaocs7
force-pushed
the
liuxiaocs7/refactor-cli-ship-one-eval-runtime-in-the-npm-cl
branch
from
August 27, 2026 04:17
1f5e94d to
03c38e6
Compare
copyEvalMirror() copied the whole staged node_modules/@maka/eval tree into
packages/eval, so every install carried the Eval runtime twice and the two
copies could diverge between releases.
The mirror is not pure redundancy: eval containers bind-mount the CLI package
root read-only at /opt/maka-agent (eval-bundle-path.ts sets
MAKA_EVAL_MAKA_BUNDLE_PATH to the package root; the experiment mount resolves in
harness-executor.ts), so /opt/maka-agent/packages/eval/... resolves to the
mirror. Only a small container-facing subset is actually read from it; the host
Eval runtime is read once from node_modules/@maka/eval.
stageEvalMirror() now ships only that subset (16 files): the two container entry
points launched inside eval cells — dist/harbor-external-subject.js and
dist/harbor-maka-subject.js — plus their transitive import closure and the
harbor assets those entry points resolve (deepseek-codex-models.json, the
deepseek-harness-profile directory, docker-compose-egress-proxy.yaml, and the
egress-proxy network-policy). The host-only dist modules, the harbor Python
launchers, and the egress image-build inputs are dropped from the mirror; they
remain in node_modules/@maka/eval (via releaseFiles), which is where the runtime
reads the launchers through BUNDLED_HARNESS_RELAY_ROOT and where the egress image
is built. The relay_agent.py and run_trial.py release guards move to
node_modules/@maka/eval to match. maka eval remains the only public Eval CLI.
The reduction is proven structurally, not by text heuristics:
- The import closure is computed with a real parser (acorn), so imports split
across comments are followed and a string that merely contains "import(" is
not mistaken for a dependency. A dynamic import with a computed specifier is
refused (fail closed) because its closure cannot be resolved.
- Experiment configs are JSON-parsed and every string value is checked; each
packages/eval reference is normalized (resolving . and ..) and rejected if it
escapes the mirror, so neither path traversal nor a JSON-escaped slash can name
an unshipped file undetected.
- The deepseek-harness-profile directory is declared once, as a directory, in the
eval package's releaseFiles. Staging and the mirror both copy it whole, so a
newly required profile file cannot reach the runtime yet be missed by the
mirror.
- validatePackedFiles recomputes the mirror inventory from the staged runtime
independently of copyEvalMirror and asserts the packed packages/eval files
match it exactly, so reverting to a whole-tree copy — or dropping a required
file — fails the release. The installed smoke test additionally asserts no
host-only file appears under the mirror.
Generated-by: Claude Opus
liuxiaocs7
force-pushed
the
liuxiaocs7/refactor-cli-ship-one-eval-runtime-in-the-npm-cl
branch
from
August 27, 2026 04:37
03c38e6 to
09ab548
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
copyEvalMirror()inscripts/release-cli-package.mjscopied the entire stagednode_modules/@maka/evaltree intopackages/evalinside the published CLI package, so every install carried the Eval runtime twice and the two copies could diverge between releases.The mirror is not pure redundancy. Eval containers bind-mount the CLI package root read-only at
/opt/maka-agent(packages/cli/src/eval-bundle-path.tssetsMAKA_EVAL_MAKA_BUNDLE_PATHto the package root; the experiment mount resolves inpackages/eval/src/harness-executor.ts), so/opt/maka-agent/packages/eval/...resolves to the mirror. Only a small container-facing subset is actually read from it; the host Eval runtime is read once fromnode_modules/@maka/eval.stageEvalMirror()now ships only that subset (16 files, down from the whole runtime tree):dist/harbor-external-subject.js,dist/harbor-maka-subject.js.dist/*.js).harbor/deepseek-codex-models.json, theharbor/deepseek-harness-profiledirectory,harbor/docker-compose-egress-proxy.yaml,harbor/egress-proxy/network-policy.Dropped from the mirror (all remain in
node_modules/@maka/evalviareleaseFiles): the host-onlydist/*.jsmodules; the harbor Python launchers (read fromnode_modules/@maka/eval/harborviaBUNDLED_HARNESS_RELAY_ROOT, not through the mount); and the egress image-build inputs (CI-only). Therelay_agent.py/run_trial.pyrelease guards move from the mirror tonode_modules/@maka/eval, matching where the runtime resolves them.maka evalremains the only public Eval CLI. Behavior, standalone packaging, and every container path the experiments depend on are unchanged.Fixes #3933
The reduction is proven structurally, not by text heuristics
acorn, so imports split across comments are followed and a string that merely containsimport(is not mistaken for a dependency. A dynamicimport()with a computed specifier is refused (fail closed) because its closure cannot be resolved.JSON.parsed and every string value is checked; eachpackages/evalreference is normalized (resolving./..) and rejected if it escapes the mirror, so neither path traversal (.../deepseek-harness-profile/../secret.json) nor a JSON-escaped slash (packages\/eval\/...) can name an unshipped file undetected.deepseek-harness-profileis declared once, as a directory, in the eval package'sreleaseFiles; staging and the mirror both copy it whole, so a newly required profile file cannot reach the runtime yet be missed by the mirror.validatePackedFilesrecomputes the mirror inventory from the staged runtime independently ofcopyEvalMirrorand asserts the packedpackages/evalfiles match it exactly — so reverting to a whole-tree copy, or dropping a required file, fails the release. The installed smoke test additionally asserts no host-only file appears under the mirror.Verification
node --test scripts/release-cli-file-policy.test.mjs— 24/24 pass, incl. comment-split imports, computed-import()rejection, path-traversal and JSON-escape rejection, the exact fixture inventory (a whole-tree copy fails it), andassertPackedEvalMirrorrejecting a reintroduced host-only file. Affected suites together: 32/32.node scripts/release-cli-package.mjs --development) succeeded end to end: the profile directory is copied whole intonode_modules/@maka/eval, the mirror stages to exactly 16 files, andvalidatePackedFiles(with the new packed-mirror assertion) passed. Tarball: 13.7 MiB compressed, 6408 files.biome check(lint + format) clean on all changed files.release-cli-eval-package.mjs, which installs the tarball and runs real Harbor + Pier Docker cells. Please confirm therelease:cli:evaljob on CI.Dependency
Adds
acorn(already present transitively) as a direct devDependency so the release script can depend on it for parsing.AI use
Select exactly one:
Tool(s) and scope: Claude Code (Opus) — mapped the runtime consumption of the mirror, implemented
stageEvalMirrorand its tests, and drafted this description. A human contributor reviewed and owns the change. The commit carries aGenerated-by: Claude Opustrailer.Checklist
Does this PR entail a change in behavior?