Repository navigation
check_answerable pre-flight and a with/without-OntologyEX agent eval - #80
Merged
Merged
Conversation
Port discover_repo.py to Rust with output that matches byte for byte: the same walk order and budgets, ID quoting, gap ordering, and Python's json.dumps formatting. The YAML loader is a port of PyYAML's reader, scanner, and parser plus the SafeLoader resolver, so it accepts and rejects exactly what PyYAML does (tabs, NEL line breaks, YAML 1.1 scalars such as 1:30). - discover_repo.py: refuse explicit YAML tags; bound YAML nesting (64) and JSON nesting (512) explicitly instead of at Python's recursion limit; a TOML float that overflows is an error, not infinity; parse failures report a stable category (YAMLError, JSONDecodeError, TOMLDecodeError, UnicodeError, DiscoveryError) - control_room --view-engine rust also discovers with the binary (native_view_bridge.native_discover, clean environment, trust checks) - engine_binary(): $ONTOLOGYEX_ENGINE_DIR, skill bin/, then the checkout's release and debug builds - tests/rust_discover_parity.py: 49 edge-case repositories plus seeded manifest mutations; 0 mismatches over 24,000 mutated manifests - Cargo.lock is now committed for reproducible release builds Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
…oofs A Rust drop-in for feeds.py: the same ONTOLOGYEX_FEEDS config, the same observations and status files. HTTP probes and Prometheus targets are polled in parallel; the OTLP/HTTP receiver (protobuf and JSON, gzip, bearer token) runs on a small worker pool. No proxy, no redirects, system trust store. On a 4-core container: ~1.74M spans/s vs ~98k for feeds.py (17.8x), 8.3 MiB vs 41.5 MiB resident. feeds.py is tightened where Rust could not match its quirks, refusing input that was malformed anyway: URLs must be printable RFC 3986 with a plain host and port; OTLP/JSON and Prometheus bodies are strict UTF-8 JSON with typed fields; truncated gzip is refused; the bearer token is compared as bytes (a non-ASCII header no longer raises); a non-object Prometheus response no longer crashes the runner; the status file's folder is created when no rows were written. Verification: - tests/rust_feeds_parity.py: config validation, OTLP decoding (seeded byte mutations), Prometheus parsing, span-window replays, store merges byte for byte, `once` end to end, and the live receiver - tests/feeds_conformance.py: classification, retention, and p95 rank tables compared across Python, Rust, and Lean - Lean (OntologyEX.Feeds): down_is_error, threshold_inclusive, classify_monotone, keep_at_most, keep_from_input, dropped_are_older, p95_index_in_range - CI runs both parity suites, the feeds conformance table for Rust and Lean, and cargo with --locked Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
- release-engine.yml: build the four binaries for Linux (static musl, x86_64 and arm64) and macOS (arm64, and x86_64 cross-built); run the discovery and feeds parity suites on each native build; on an engine-v* tag, publish the archives, SHA256SUMS, and a GitHub build provenance attestation. Pull requests build without publishing. - fetch_engine.py: download a release over HTTPS from github.com only, check the archive against SHA256SUMS, and install only the expected binaries into the skill's bin/ (no traversal, symlinks, or extra files; no silent overwrite). Prints the gh attestation verify command. Shipped with the installed skill. - docs/rust-engine.md: install, run, measured speedups, how parity is checked, what is proven, known differences, and how to release. - README, live feeds, operating-picture reference: point to the engine. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
Python 3.11.9's urlsplit accepts http://x[::1]/ and http://[::1]x/, newer versions refuse them; check the shape explicitly so feeds.py and ontologyex-feeds agree everywhere. The parity harness now prints only the rows that differ in the end-to-end case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
A manual run of "Engine binaries" with version=engine-vX.Y.Z builds, runs the parity suites, and publishes the release like a tag push, creating the tag on the commit it built. The version reaches the shell through an environment variable and must match engine-vX.Y.Z. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
ontologyex-feeds polls up to 16 targets at once. The fixture used socketserver's default listen queue of 5, so on macOS the extra connections were refused outright and the Rust run saw failed probes where the sequential Python run did not. Linux queues them, which is why only the macOS job failed (and only sometimes). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
…lled --discovery-engine auto (the default) uses the byte-identical ontologyex-discover when it is installed (fetch_engine.py or a cargo build) and Python otherwise; python and rust force a choice. A broken install still fails loudly instead of silently switching. The server prints which engine it uses. Graph views stay on Python unless --view-engine rust, since per-query process overhead outweighs the gain on small graphs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
feed_setup.py propose reads compose files (and a prometheus.yml a Prometheus service mounts) and lists candidate feeds, each with the file and line that declares it: HTTP probes from healthchecks on published ports, OTLP mappings from OTEL_SERVICE_NAME / service.name, and Prometheus scrape targets that name compose services. What no declaration backs is a gap, never a guess. feed_setup.py init writes only the accepted candidates and validates the config with feeds.load_config. operate.py up starts the feeds service (Rust when installed) and the control room on its observations together, and stops both together. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
Both feeds engines now decode trace, span, and parent span IDs. A CLIENT or PRODUCER span and the SERVER or CONSUMER span whose parent it is (same trace) count as one call from the first service to the second. Unmatched halves wait one window, bounded at 100,000 per side. Counts per service pair (at most 1,024) are written to *.calls.json beside the observations, and totals survive restarts; an unreadable calls file is refused, not overwritten. The parity suite covers ID decoding, pairing in seeded replays, reading mutated calls files, merging across emits, and the live receiver's calls file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
dependency_suggest.py reads the calls file the feeds write and proposes "A depends on B" for each call between mapped components that the repository does not already declare, with call and error counts and first and last seen. Calls the repository declares are reported as corroborated; calls from unmapped services are listed as gaps. Confirm or reject on the business page (opt-in, token-protected route /api/dependencies/decision) or with the CLI. A decision is appended atomically to a dependencies file with who made it and the evidence at that moment. The control room adds each confirmed dependency as a CALLS edge with its own evidence record and carries possible impact along it, like a declared DEPENDS_ON; the systems map draws it in violet. operate.py passes dependencies.json beside feeds.json and creates an empty observations file so a traces-only setup starts cleanly. The live demo now sends real cross-service calls (worker -> payments is suggested), and the Collector example keeps client spans so calls can be paired. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
decisions.py is the one place link and dependency decisions are written: confirm, reject, and undo, one or up to 100 at a time, checked whole and written in one atomic replace (an unknown ID or a file changed meanwhile refuses the whole batch; the row limit is checked before writing). Undo removes a decided row so its suggestion comes back; the decided row keeps the suggestion's ID. Every change is appended to <file>.history.jsonl with what, who, and when. - Business page: checkboxes, Select all / Select STRONG, Confirm/Reject N selected, Undo for the last action, and a Decided list with Undo per row, for both suggested links and observed dependencies. - Decision routes take "ids" (or one "id") and decision confirm, reject, or undo; the business payload lists decided rows (link_decided, dependency_decided). - CLIs: --id repeats, --all-strong, and a new undo command; suggest lists decided rows. A batch where every row got the same action reports CONFIRMED, REJECTED, or UNDONE as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
otel_setup.py propose lists every app in a repository (Node, Python, Java, Go, .NET, Ruby, PHP) with the file that declares it, the libraries it will trace, the name its traces carry (compose service, else package name) and the map component it stands for. otel_setup.py run --service NAME -- CMD starts a Node or Python service with OpenTelemetry's zero-code instrumentation and no code change: Node from a shipped lockfile (npm ci, 189 packages pinned with integrity hashes, install scripts off) loaded with NODE_OPTIONS; Python from the app's own interpreter (pip --target, the instrumentations opentelemetry-bootstrap picks) loaded by a sitecustomize that adds them after the app's packages. Installs go to a cache once. SIGTERM still stops both: Node's register hook swallowed it, Python's default skipped the last spans. feed_setup.py offers the name mappings. The feeds receivers (feeds.py and ontologyex-feeds) refused bodies without Content-Length, which is how OpenTelemetry JS sends them (chunked), so Node services could not report at all. Both now read chunked bodies with the same rules and the same 4 MB cap; the parity suite covers valid, extended, oversized and malformed chunked requests. CI runs a real end-to-end: a Node service calls a Flask service, both started with otel_setup.py run, and the receiver pairs the call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
… to source For "use OntologyEX to give me an overview of this repository and the business behind it", an agent now has one path, from AGENTS.md, the README and SKILL.md: scan, read, fill in a Business X-Ray blueprint, build one page. - overview.py scan reads a repository the way a new engineer would and writes scan.json and a blueprint draft: apps with languages and frameworks, compose services and datastores, outside services by area (payments, auth, email, AI, data, cloud, analytics), docs, and files to read first (entry points, routes, data model, money, users), each with its file and line. A rescan keeps an edited blueprint. - overview.py build refuses leftover placeholders and compiles the page. - The X-Ray compiler moved into the skill (scripts/business_xray.py; business/build_blueprints.py still runs it) and can cite a local checkout: every cited line must exist, symlinks out of the repository and paths that climb out are refused, and each citation records the quoted lines, their SHA-256, and the commit the checkout is on. - references/business-overview.md is the procedure; AGENTS.md is the first page an agent reads; otel_setup finds Rust apps too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
…n the map Four agents ran the "give this repo to your agent" flow on listmonk, documenso, saleor and nextjs-subscription-payments. Fixes from what they hit: - The scan crashed on compose files with YAML anchors; compose is now read with anchors and merges allowed, tags refused and expansion bounded. - A build could succeed while three page tabs were broken: risk severity and improvement priority are now checked, money-flow state and the rest defaulted, and the reference lists every field with one item of each kind. - The reading list is ranked by what files import and say, not only their names: money and access files, entry points from the Dockerfile or start script, docs including AGENTS.md and ADRs, configuration and background jobs. - Datastores are inferred from drivers, Prisma and Supabase when no compose runs them; Go modules, more integrations and frameworks; tooling-only packages marked. - The page gains a Systems tab and shows each citation's text; the build prints each citation's first line to check, and --github links a shared page to GitHub. code_surface.py reads routes, GraphQL/RPC operations, tables and jobs by syntax across common frameworks. overview.py lists them; the control room draws them on a local repository's map under the app that holds them (bounded, API first). Example page: docs/examples/nextjs-subscription-payments.html, linked from README. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
find_apps takes the discovery the scan already made, and both walks build relative paths from strings instead of pathlib. Same scan.json on saleor, documenso and listmonk; saleor 5s -> 3.2s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
Two more fresh agents ran the flow on maybe (Rails) and the FastAPI full-stack template. Fixes from what they hit: - Code: FastAPI/Flask route lines were two early (a leading \s* ate blank lines) and missed router prefixes; SQLModel and pydantic schemas were taken for Django tables. Rails routes follow namespaces, nesting, member and collection; ActiveJob and Sidekiq jobs and schedule.yml cron entries are found. - Scan: compose overrides and examples are read (compose.override.yml, compose.example.yml); a workspace-only pyproject is tooling; a compose build with dockerfile backend/Dockerfile maps to backend/; dev-only packages are not integrations; Ruby, banking (Plaid) and more catalog entries; entry points from the code that constructs the app, Procfile.*, config.ru; config.py and deploy workflows under configuration; .cursor/rules notes as docs; money ranks what the business charges above a finance app's own domain words; larger files first. - Output: scan prints a summary (scan.json has the rest); check_citations shows the first and last cited line. - Page: steps wrap instead of hiding, open questions on the Business tab, plain wording, no sideways scroll on phones. - Docs: drafted systems may be deleted; risk basis guidance; never copy secrets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
…ules Laravel route groups with prefix(...)->group(...) are followed and routes/api.php is served under /api; Schema::create migrations are tables; ShouldQueue classes and Schedule::command/job entries are jobs. The overview reads Laravel entry points (public/index.php, bootstrap/app.php, artisan), lists app/Models before one-table migrations, skips lang/ translation files and composer.json in topic ranking, names a root app after the repository, and knows common Composer packages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
…ecks Two more fresh agents ran the flow on Basecamp's handbook (documents, no code) and Monica (Laravel). Fixes from what they hit: - check_citations read the last line from a quote cut at 600 characters, so long prose ranges showed a middle line; it now reads the file. The page keeps up to 2,000 characters per citation, and the build lists unused citations. - Documents: money and access ranking reads Markdown too (salary, stipend, severance and benefits count as money); docs follow the README's own link order and business words rank root documents; the draft cites each document's first line; the scan says when a repository is documents, not code; a shallow clone is noted. The reference says what systems, stage and journeys mean for documents. - Laravel: entry points from the Dockerfile's shell scripts, config/<name>.php first under configuration, SQLite from .env.example, terms and privacy docs, license files out of topic lists. - Page: wording that fits any business (no "runtime", "blast radius", "business anatomy"), improvements name the risks they address, systems grouped by kind. - Written files take the user's umask instead of 0600. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
Spring routes take a class-level @RequestMapping prefix and path arrays, and annotations mentioned in comments are skipped; @entity classes are tables (with @table names when given); @scheduled methods are jobs. The overview reads Maven and Gradle dependencies (frameworks, integrations, database drivers), finds the @SpringBootApplication class as the entry point, and lists application*.properties and .yml under configuration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
…closed tests/bench_rust_engines.py measures discovery, entity context and the OTLP receiver the way OntologyEX runs them. Per engine: wall time (median, p10-p90 over measured runs after a warm-up), CPU seconds and cores used, peak memory (sampled from /proc on an untimed run; wait4's ru_maxrss counts the launcher's memory from before exec). Per workload: whether the answers were the same; a comparison is computed only when they were. The receiver check counts every acknowledged span and compares the emitted observation with the Python reference window. Conditions (machine, versions, build profile, binary hashes, commit, load) and caveats (debug, noisy, saturated, busy) are recorded. Missing binaries are NOT_RUN; differing answers exit 1. The docs quote a saved report through generated blocks (--write-docs), and a test fails when they disagree. CI runs a quick debug-build pass as a correctness gate with the report in the job summary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
docs/benchmarks/latest.{json,md} from a release build on a 4-CPU Linux
container: discovery 17.6x faster, the OTLP receiver 34.6x the spans per
second (19.1x per CPU-second), and the same answers on every workload.
Entity context as the control room asks for it is 40x slower on Rust (a
process per answer), which the docs now say instead of hiding.
The README and docs/rust-engine.md tables are generated from the report.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
… apart from proofs What relies on an entity used to stand as whichever chain the traversal reached first in name order, so renaming two entities could flip a result between INFERRED and CONFIRMED. The impact page already promised the strongest path. Python and Rust now keep, per hop count, each entity's strongest chain of exactly k hops and answer the strongest over 1..4 hops, fewest hops on ties; depth is that chain's hop count and path is the chain. Lean: OntologyEXModel/Impact.lean writes the computation; OntologyEXSpec/ Impact.lean states StrongestChain, NothingWithoutAChain, OrderFree and NameFree; OntologyEXProofs/Impact.lean proves them. The conformance table gains 300 generated graphs printed from the proven model and compared with Python (the old code differs on 12). tests/lean_statements.py pins the Model and Spec files by hash (changes need a recorded reason), forbids proofs in statements and sorry/axioms in proofs, and in CI holds each theorem to its exact statement and checks its axioms; leanchecker replays the new modules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
…ndone whole
Every decision is now a compare-and-set on one item: the version of the
decided row the person saw (sha256 prefix of its canonical entry) and what
to write. decisions.commit applies a batch only if every item is still at
that version; otherwise nothing is written (409 STALE_VIEW, naming the
items). A preview returns one digest per operation; applying with those
digests writes only the same batch. Decisions on other items never make an
approval stale. Each applied batch is recorded under a batch ID with every
item's entry before and after; undo_batch applies the inverse, puts back
exactly that batch's items, keeps later decisions on other items, and is
refused when a later decision changed one of its items. Undoing a re-confirm
now restores the previous row; a single-row undo is bound to the version
the page showed. GET /api/{kind}/batches lists batches.
Rust: decide.rs implements the same commit (ontologyex-view decide); the
control room uses it with --view-engine rust after checking at start that
the installed binary has it. tests/rust_decide_parity.py compares it with
Python (3,000 seeded batches in CI); engine/tests/decide.rs checks the laws
on the Rust code and the binary.
Lean: OntologyEXModel/Decisions.lean models commit; OntologyEXSpec/
Decisions.lean states ten laws (stale refused, twice refused, current
applies, whole batch, nothing else, undo restores, undo keeps later
decisions, undo refused after change, independent batches commute, others
never stale); OntologyEXProofs/Decisions.lean proves them. The conformance
table gains 300 generated batches compared with Python.
The page previews bulk confirm/reject and undo, lists recent batches with
Undo batch, and sends row versions with undo.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
… aggregates only operations.py reads payment-processor records dropped in REPO/.ontologyex/operations (Stripe dashboard exports, Stripe API objects saved as JSON, other processors through operations.json) and a read-only Stripe pull, and keeps weekly aggregates per currency: payments, refunds, disputes, payouts, subscriptions (active and monthly recurring revenue rebuilt from start, end, price and interval) and invoices. Only the columns a stream needs are read; customer names, emails, IDs, card details and free text never are. Amounts and breakdowns of weeks with fewer than three records are withheld. The pull aggregates each page as it arrives, writes only aggregates, and refuses secret keys. Signals compare the last four complete weeks with the four before (open disputes, failed payouts, rising refunds, falling revenue, high dispute rate, rising churn, failing payments, written-off invoices) and rank in the command center under Operations. The records are compared with the model and code: what happens that the sources do not show, and what is built that the records never show used. The control room serves /operations and /api/operations (cached on the folder's file stats), the pages link to it, agents get operations_status, and the demo carries 16 weeks of synthetic exports. Golden outputs record the demo's new command-center items and pin the ledger. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
…d costs spend.py reads business bank, card or ledger lines, vendor bills, and AWS, Google Cloud and Azure cost exports from the operations folder, as monthly aggregates per currency: money in and out, out by category, suppliers paid, deposits from payment processors, bills open and overdue, and cloud cost by provider and service. A payee is named only when it is a supplier in the catalog the supplier map uses (matched in payee and description, which are read for that and nothing else); people paid, shops and customers refunded count under their category only. operations.py pull aws runs the aws command line (ce:GetCostAndUsage, read-only) and writes aggregates only. operations.json now also maps bank files: currency, date format, and card exports that list charges as positive. Spend answers what code alone cannot: suppliers paid that the code does not use and suppliers the code uses that nobody is seen paying; whether processor payouts reached the bank; cloud cost against payments taken. Signals (payouts not in the bank, overdue bills, spending and cloud jumps, a new supplier paid) and the paid-vs-used findings rank in the command center under Spend; /operations shows a Spend section per currency. The demo gains six months of a bank account (payroll lines name fictional people, never kept), vendor bills, an AWS cost report, a missing payout deposit, and a web app declaring SendGrid and Sentry. Golden outputs record why the demo's answers changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
…ed with the payment records customers.py reads helpdesk exports (Zendesk, Freshdesk, Intercom, Help Scout or any with a created time and status) and CRM deal exports (HubSpot, Salesforce or any with a stage and created date) from the operations folder: tickets opened and solved, the backlog at each week's end, time to solve, topics, priority, channel and ratings; deals created, won and lost, amounts per currency, win rate, days to close, loss reasons and sources. No subject, requester, agent, deal name, company or owner is kept; a subject is read only to count which business item it names. Durations go into fixed buckets, never one number per ticket or deal. operations.py pull zendesk (incremental export, same host only) and pull hubspot (stage metadata decides won and lost) write aggregates only. Signals: tickets rising, backlog growing, slower solving, ratings dropping, a topic spiking, win rate down, one loss reason dominating. Joined with the payments: refunds and refund tickets rising together (two sources agree, ranked above either) and topics customers raise that the business model does not name. /operations gains a Customers section; the command center a Customers category; agents get it in operations_status. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
…rds show written.py reads policies, procedures, terms and contracts from the operations folder (Markdown, text, HTML, Word and simple PDFs) for commitments a business can keep or break, each cited to its line: refund windows, refund approval thresholds, ticket resolution times, dispute response times, payment terms, renewal notice periods and uptime. Each is held to the records where they can say: resolution time against the share of tickets solved within it, payment terms against overdue bills, an approval threshold against the business model, disputes against those open; the rest is marked as needing a person, with what to check. Two documents that state different values for the same commitment conflict, and refunds, disputes, payouts or support happening with no document describing them are not written down. Quotes mask emails, phone numbers and long numbers. Signals rank in the command center under Written; /operations gains a Written down section; agents get it in operations_status. The demo adds a refund policy, Word terms of service that disagree with it, a support promise the tickets break, a finance policy the overdue bill breaks, and a supplier order form. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
… the page Declare the pulls once in the operations folder (pulls.json: provider, interval, and the name of the environment variable holding the key, never the key), and the control room runs each on its interval when started with --pulls (operate.py up --pulls); `operations.py pull all --dir DIR` runs them once, for cron. Each runs the same read-only, aggregates-only pull as before. - pulls.json is checked: known providers, 15 minutes to a week, key_env an environment variable name, any other field (a key, say) refused, api_base only on this machine. An error that echoes a key has it masked. - pulls.status.json records each pull's last attempt and success, what it read (counts), the masked error and the next run. It and pulls.json are not records, and the status does not invalidate the cached ledger. - The ledger carries `live`, recomputed on every request; the command center ranks PULL_FAILING, PULL_STALE (two intervals without a success) and PULLS_MISCONFIGURED under Operations; /operations shows Live sources. - Pulled aggregate files are written whole (temporary file, then rename), so a read during a pull never sees half a file. aws takes --profile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
… needs repair drift.py finds each span the model cites (skill evidence quotes, confirmed rules' citations, confirmed links' bindings) in the repository as it is now: UNCHANGED, REFORMATTED (whitespace evened out; comments count as text), MOVED (the same text at other lines, with where), CHANGED (with the nearest look-alike lines) or FILE_GONE. - Business objects and actions read CURRENT, REPAIRED (evidence only moved) or STALE_REPAIR (evidence changed or gone); the command center ranks STALE_REPAIR under Decisions, the entity page shows "Cited code now", /api/drift serves the report (one entity, or Markdown), and agents get the read-only source_drift tool. In the control room it is a background scan, discarded when the repository changes under it. - Confirmed rules follow their lines: moved or reformatted stays confirmed and is read at the new lines; changed reopens. New confirmations also record a whitespace-evened hash; older ones are matched exactly. - Confirmed links record their binding's cited lines (span:PATH) instead of the whole file, so edits elsewhere in it no longer reopen them; links recorded by whole file keep that until re-confirmed. Change impact (Python and Rust) treats span:PATH as citing PATH, and lists it for re-confirmation only when recorded by whole file. - drift.py check (--fail-on-stale for CI) and an opt-in post-commit hook (hook install/remove; never blocks, never replaces a hook not its own). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
…hey run guardrail.py is a Claude Code PreToolUse hook. guardrails.json maps agent tools (names or globs) to a domain skill's task and says where each task input is in the tool's arguments; before a matching call runs, the task's modeled checks run on the call's own arguments (policy_check.evaluate): - a check fails: deny, naming the rule and what it says; - open questions (unknowns, conflicts, rules without a check), a missing or mistyped input, code under the rules that changed since (source drift), or a guard that cannot be read: ask a person, saying why (on_error can be deny); - every check passes: no decision; the normal permission flow applies. It never answers allow: passing the modeled checks is not an approval. init proposes a guard per task; check tries a call; install/remove add or take out only its own entry in .claude/settings.json (matching only guarded tools). Each guarded call is audited (decision and rules, input names, never values); the control room ranks a rule that stopped an agent in the last 7 days (AGENT_BLOCKED). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
…in the Rust engine
`ontologyex-discover surface` reads {"repo", "files"} on stdin and returns what
each Python, JavaScript/TypeScript and Go file declares, read from tree-sitter
syntax trees (grammars compiled in; parsing only). code_surface.scan(repo, engine)
asks it for those files and places the facts exactly as before (Next.js and Remix
file routes, table de-duplication, map budgets); other languages, and every file
when the engine is missing or predates the command, stay on the patterns, and the
result says which in `parser`.
From trees: comments, docstrings and strings are not code; calls and decorators
may span lines; each router keeps its own prefix (APIRouter/Blueprint per
router, include_router/register_blueprint, Express app.use, Hono route, gin/echo
Group, chi Route, gorilla PathPrefix().Subrouter()); Express route chains and
Hono chains, Fastify route({method, url}), gorilla .Methods(...), Go raw-string
paths and `export { handler as GET }` are routes.
On the patterns' own fixture both give the same answers (lines included), checked
in CI with the engine built (ONTOLOGYEX_REQUIRE_TREES), including on the musl and
macOS release builds. The control room uses it when discovery runs on Rust, and
overview.py scan when the engine is installed. The binary size budget for native
binaries is raised to 256 MiB for debug builds; release ontologyex-discover is ~5 MB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
Every command-center finding now names what it rests on: records of what was observed (health samples per source), read (file lines, skill citations), decided (confirmations, proposals) or aggregated (operations ledger, audits), each with its source, component, environment, when it was collected, its age, and whether it is COMPLETE, PARTIAL, STALE or MISSING; `role` separates the basis of the conclusion from context. `contradicted_by` says what would refute it. A conclusion whose basis is all expired or missing reads UNKNOWN, with why. The command center states a verdict on the systems: FAULT, DEGRADED, HEALTHY or INCONCLUSIVE, with the share of watched systems that have fresh evidence. Missing evidence is never healthy: with no fault seen and any watched system silent or expired, the verdict is INCONCLUSIVE and names them. Causes rank those with no failing system upstream first. replay.py captures what the control room knew at a moment (the discovered topology, the window of health observations, skills, links, events, rules and the operations ledger as aggregates) and replays it through the control room's own assembly (--bundle-only with --topology and --now) from a copy, checking the verdict, the cause within the top N, citation coverage, and that nothing was written. tests/replays holds seven synthetic situations (error, upstream error, degraded, healthy, one silent, expired, no evidence); all pass with 100% citation coverage. Also: --environment labels the evidence; Prometheus targets take a job_lag_seconds query (how long ago a job last did its work) with thresholds, in Python and Rust feeds; agent tools withhold evidence that names a denied file; /command shows the verdict and each finding's evidence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
…rson A guarded call that goes ahead is journaled as an intent (the tool and a fingerprint of its arguments, never their values), and the hook now also runs on PostToolUse, PostToolUseFailure and PermissionDenied to record how it ended: done, failed (the tool reported an error), not run, or unknown (interrupted, a timeout-like error, or no end recorded at all). While an identical call has an unknown outcome, the next one is not dispatched blindly: the hook asks a person to check the live state first. A later identical call that completes, or `guardrail.py reconcile --by NAME`, settles it; unknowns older than 72 hours stop holding calls. `guardrail.py outcomes` lists them, and the command center ranks unknown outcomes from the last seven days (OUTCOME_UNKNOWN), with the journal as evidence. The hook decides and records; it never retries, reverses or dispatches anything. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
- answerable.py: for each word of a question, the business items, rules, systems, records and written commitments that name it (with the lines behind them), the words nothing covers, open questions recorded there, and the kinds of source it needs and lacks. Deterministic, no model. ANSWERABLE only when every word is covered, every needed source is loaded and nothing open bears on it. - agent tool check_answerable (read-only; matches resting on withheld files are withheld whole) and /api/answerable in the control room. - evals/agent_ab.py: the same model and questions with and without OntologyEX's read-only tools, scored without a model judge (facts as whole words, abstention, citations that exist, grounding in gold spans), with the questions OntologyEX fixed and broke. Datasets: the demo shop (24) and fastapi/full-stack-fastapi-template at a pinned commit (13), each gold span held to an anchor phrase by `check`. No results published until a real run. - tests/test_agent_eval.py pins the pre-flight, the tool, the route and the harness (scripted agent, no key). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoxWvtYbajXo6hFeADTXYF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
1.
check_answerable(deterministic pre-flight, no model):ontology-extraction/scripts/answerable.pyANSWERABLEonly when every word is covered, every needed source is present and no open question bears on the question. Otherwise it isPARTLYorNOT_IN_SOURCES.check_answerable) and as/api/answerable?q=in the control room.2. Agent A/B eval:
evals/agent_ab.pydemo-shop(24 questions);fastapi/full-stack-fastapi-templateat a pinned commit (13 questions).agent_ab.py checkholds every gold span to an anchor phrase in the real file.anthropicSDK. It is an optional dependency for the eval only; the kit itself stays PyYAML-only.Tests
tests/test_agent_eval.pycovers the pre-flight, the tool and its policy, the route, and the harness via a scripted agent: scoring, abstention, invalid citations, whole-word facts, and the fixed/broken lists. It needs no key.Generated by Claude Code