Summary
Two openhuman --lib tests depend on Cargo features they do not declare. Under --features "$(scripts/ci/product-features.sh)" — what CI runs — they pass. Under default features they fail, and they fail in a way that reads as a product defect rather than a missing feature.
This is a test-integrity problem, not tidiness: a test whose outcome depends on features it does not declare is a guard that cries wolf under the wrong profile.
The two tests
| test |
file |
every_prompt_names_at_least_one_tool_it_can_call |
crates/openhuman-core/src/agent/registry/agents/fleet_prompt_tests.rs:344 |
the_withheld_block_renders_for_a_renamed_session_with_a_filter |
crates/openhuman-core/src/agent/registry/agents/orchestrator/prompt_tests_session_routing_tests.rs:41 |
Neither carries a #[cfg(feature = …)] gate or a comment naming the profile it requires.
Mechanism
NodeExecTool is registered under #[cfg(feature = "runtime-node")] (crates/openhuman-core/src/tools/ops.rs:863-865). runtime-node is not in default:
default = ["media", "skills", "flows", "mcp", "channels", "medulla", "http-server", "scheduler-gate", "file-logging", "modules"]
but is in scripts/ci/product-features.txt.
So with default features node_exec / npm_exec do not exist. They are absent from tool_universe(), can_call (fleet_prompt_tests.rs:130-143) is therefore false for both, and the guard reports skill_creator as an agent whose prompt names none of the tools it can call — which is indistinguishable, from the failure output alone, from a prompt that genuinely names uncallable tools:
assertion `left == right` failed: agents that carry tools but whose prompt names none of them
left: ["tools_agent", "tool_maker", "skill_creator", "critic", "archivist", "skill_setup"]
right: ["tools_agent", "tool_maker", "critic", "archivist", "skill_setup"]
Nothing in that message mentions a feature.
The second test fails the same way for a documents-gated tool: it asserts the withheld block contains - skill `documents`: `make_presentation` , and documents is likewise product-only.
Demonstrated cost, not hypothetical risk
This is not a speculative concern. While triaging #6486 I ran the suite with default features, hit the first of these two, traced it through can_call → is_withheld_from, and filed a product defect against skill_creator's prompt — attributing it to a specific PR. It was wrong: the tools were simply not compiled in. The issue had to be retracted and closed as invalid, and the PR attribution withdrawn.
The failure output gave no signal that a feature was missing, and is_withheld_from is real code that produces exactly this symptom under different circumstances — so the wrong explanation was not implausible, it was correct about a situation that did not apply.
Suggested fix, cheapest first
Three options, and the third is the best because it converts a misleading failure into a self-explaining one:
- Gate the tests —
#[cfg(feature = "runtime-node")] / #[cfg(feature = "documents")]. Simple, but silently skips under default, which trades one invisibility for another.
- Assert the profile — fail early with "this test requires
runtime-node" if the tool is absent from tool_universe().
- Make the existing assertion self-explaining — when a named tool is missing from the universe entirely (as opposed to present-but-not-callable), say so in the message. The guard's job is to catch a belt narrowed out from under its prompt; "this tool is not compiled in this profile" is a different fact and should read differently.
Option 3 preserves the guard's coverage under every profile and removes the misreading at the same time.
Related
Summary
Two
openhuman --libtests depend on Cargo features they do not declare. Under--features "$(scripts/ci/product-features.sh)"— what CI runs — they pass. Under default features they fail, and they fail in a way that reads as a product defect rather than a missing feature.This is a test-integrity problem, not tidiness: a test whose outcome depends on features it does not declare is a guard that cries wolf under the wrong profile.
The two tests
every_prompt_names_at_least_one_tool_it_can_callcrates/openhuman-core/src/agent/registry/agents/fleet_prompt_tests.rs:344the_withheld_block_renders_for_a_renamed_session_with_a_filtercrates/openhuman-core/src/agent/registry/agents/orchestrator/prompt_tests_session_routing_tests.rs:41Neither carries a
#[cfg(feature = …)]gate or a comment naming the profile it requires.Mechanism
NodeExecToolis registered under#[cfg(feature = "runtime-node")](crates/openhuman-core/src/tools/ops.rs:863-865).runtime-nodeis not indefault:but is in
scripts/ci/product-features.txt.So with default features
node_exec/npm_execdo not exist. They are absent fromtool_universe(),can_call(fleet_prompt_tests.rs:130-143) is therefore false for both, and the guard reportsskill_creatoras an agent whose prompt names none of the tools it can call — which is indistinguishable, from the failure output alone, from a prompt that genuinely names uncallable tools:Nothing in that message mentions a feature.
The second test fails the same way for a
documents-gated tool: it asserts the withheld block contains- skill `documents`: `make_presentation`, anddocumentsis likewise product-only.Demonstrated cost, not hypothetical risk
This is not a speculative concern. While triaging #6486 I ran the suite with default features, hit the first of these two, traced it through
can_call→is_withheld_from, and filed a product defect againstskill_creator's prompt — attributing it to a specific PR. It was wrong: the tools were simply not compiled in. The issue had to be retracted and closed as invalid, and the PR attribution withdrawn.The failure output gave no signal that a feature was missing, and
is_withheld_fromis real code that produces exactly this symptom under different circumstances — so the wrong explanation was not implausible, it was correct about a situation that did not apply.Suggested fix, cheapest first
Three options, and the third is the best because it converts a misleading failure into a self-explaining one:
#[cfg(feature = "runtime-node")]/#[cfg(feature = "documents")]. Simple, but silently skips underdefault, which trades one invisibility for another.runtime-node" if the tool is absent fromtool_universe().Option 3 preserves the guard's coverage under every profile and removes the misreading at the same time.
Related