Conversation
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Validation record for head Re-keyed from Measured configuration: a detached worktree of cle at the revision named, this workspace's pinned Python 3.12 environment,
The regression test loads
An earlier version of this record covered head The old Caveats:
Hosted CI at head Re-keyed 2026-09-05 to head The one conflict was the import block of The patch itself did not move. Two things measured again at this head rather than carried over:
Correcting a line above. This record says Hosted CI is re-running at this head; the run at Re-keyed 2026-10-03 to head This is a new candidate, not a formality rebase. The branch was rebased off What the rebase fixes beyond staleness. At The two malformed cases load committed fixtures.
The fixtures are Where executables actually declare
So on 64 bit the new code reads back exactly the constant it replaces for every unaffected executable, and on 32 bit the constant named the wrong address for 14 of the 15 images the corpus has. An earlier version of this record said "ld64 puts The message-less raise, re-counted at this baseline. The corpus measurement, and a figure withdrawn. An earlier version of the description said "The corpus holds 1,620 objects of this shape". That number is not reproducible and is withdrawn. Three complete populations, every sha256 verified against the extracted bytes:
The description quotes the first row, because those are the objects a sweep actually covered. 1,620 is none of them. No population holds an object whose Both sides, one environment each, cle master One thing to know about the environment the corpus run used. It ran against cle Why the description gives 695 as well as 699. Regression set: 487 other swept Mach-O objects, none affected, across 19 header strata. 0 worse. 400 load on both sides and 87 are refused on both sides with the same message; of the 400, 391 are identical in every loader fact and 9 differ, which with the 87 refusals makes the 478 unchanged rows. The 9 are four thin 32-bit x86 Function recovery was compared on the 26 objects that load on both sides and agree on the base address: of the 25 that produced a Local gate,
Caveats, extending the list above:
The two objects cited below are ones the corpus sweep recorded itself, and they are why this is cited rather than argued. Both are Go builds from a controlled generator corpus,
The first epoch ran The two epochs are not otherwise alike, so the isolation is worth stating as a measurement rather than as a claim about the ledger. Six of their seven components moved between them, cle's own base among them ( Re-measured directly at baseline Both are sighted on Hosted CI at this head. Run One mechanism accounts for all four non-green checks, and it is not this change.
So no single description satisfies both, and that is a property of the chain rather than a mistake in this candidate. Two sibling pull requests make it checkable without reading the action: What each job actually did, measured from its own log rather than from the check name:
|
|
Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_727 |
ca4a867 to
8b1c15a
Compare
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Independent corroboration from a survey of Mach-O binaries, in case it is useful They are interesting because the failure is only reachable after a different Before #729 these died in Measured over the same twenty-one objects, each loaded in an isolated
Entry points were checked against the No action requested; the branch already does the right thing. Recording the |
8b1c15a to
aba6625
Compare
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS A corpus measurement, in case it is useful for review. A decompilation sweep over 35,578 objects recorded 48 distinct Mach-O binaries Every one has the same shape: Method: the sweep's ledger stores full tracebacks, so each failure was Two details for anyone reading failure counts off a ledger like this one:
None of the 48 come from redistribution-restricted material. session: sharpen |
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Correction to my previous comment. The denominator I gave was wrong, and it I wrote "a sweep over 35,578 objects recorded 48 distinct Mach-O binaries that Measured over what has actually been swept:
So roughly 30% of the Mach-O binaries reached so far fail to load (49/163). Of those 49, 48 are the This agrees with the load-only A/B already recorded against this corpus, which session: sharpen |
Thirteen raise sites in the Mach-O backend raised CLECompatibilityError or
CLEInvalidBinaryError with no message. A caller that catches one, and any log
that records one, gets an empty string: every rejection is indistinguishable
from every other, and from the several unrelated conditions that raise the same
type. In a sweep over 35,578 objects, 48 binaries failed this way, and the only
thing separating them from a rejection that did carry a message ("Unsupported
Mach-O file type: 8...", classifiable on sight) was that someone had written
the message.
Most of these sites already computed the diagnosis and then threw it away into
a log.error immediately above the raise, where it is lost as soon as logging is
not configured at that level, and is not attached to the exception in any case.
Move that information into the exception, and add it where it did not exist:
each message now names what was found and what was expected, so the message
alone identifies the check that rejected the file.
_load_lc_unixthread's unknown-thread-flavor raise is deliberately left alone:
open PR #727 deletes that branch rather than messaging it, making an
unrecognized flavor non-fatal, and a message there would only conflict.
The regression test loads an existing ELF fixture through the Mach-O backend,
the one path of the thirteen reachable from a fixture that already exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two rebase messages name the segment they overran, and `segment`, `address`
and `reloc_type` are all None until a SET_* opcode assigns them, so reading them
for the message is only well typed once that is established. Reject a rebase
opcode that arrives before its state is set, which is a malformed blob and
previously died on an AttributeError several lines later, and hoist the segment
end out of the loop it does not vary in.
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Full load report for Before — all three loads are refused with a message-less cle at the merge base, 46a3733After — the stock fixture yields its entry point and its five segments, and both patched copies load with no entry point rather than no object: with this change, aba6625 |
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS A second corpus measurement, on a much broader and less Mach-O-heavy sample than Sample. 12,000 objects drawn uniformly at random, from a seeded permutation, Method. Each object is loaded with the catalogue's declared load recipe and Before. 350 of the 1,204 Mach-O objects fail at load — 29.1%, which agrees After. All 3 load, and all 3 complete a full The gap against the earlier 48-object figure is corpus composition, not a A 500-object control set that already reached CFG on master is unchanged — 0 of The corpus is not redistributable, so the objects are described by architecture, session: sharpen |
aba6625 to
6da0249
Compare
6da0249 to
8b59ffb
Compare
Thread state flavor numbers are only unique within a cputype, but _load_lc_unixthread dispatched on the flavor alone. Flavor 1 and 6 were read as ARM_THREAD_STATE and ARM_THREAD_STATE64 whatever the cputype was, and everything else aborted the load with an empty CLECompatibilityError. An x86_64 executable stores x86_THREAD_STATE64, flavor 4, so it never loaded at all. A 32-bit x86 executable stores x86_THREAD_STATE32, flavor 1, which is the same 16 words as ARM_THREAD_STATE but keeps __eip at index 10 rather than a trailing __pc, so it loaded with __gs as its entry point. Key the thread state layouts by (cputype, flavor) and cover both x86 states. Check the state against the length the command declares and against the end of the file before unpacking it; a binary truncated inside the thread state used to come back as a bare struct.error. An LC_UNIXTHREAD that cannot be read now leaves unixthread_pc unset and lets _resolve_entry report the missing entry point, because the entry point is the only thing the command contributes and the rest of the binary is still loadable.
The backend assumed every position independent MH_EXECUTE was linked at 0x100000000 on 64 bit and 0x4000 on 32 bit. Those are ld64's defaults, not properties of the format. Go's internal linker links darwin/amd64 executables at 0x1000000, and for one of those the mapped base ended up four gigabytes above every segment, so the load aborted in Loader._map_object on `assert obj.min_addr <= obj.max_addr` before any analysis could start. Read the vmaddr of __TEXT out of the load commands instead. That is the address the mach header itself lands at and what __mh_execute_header resolves to, so it is the linked base by definition. The old constants stay as the fallback for a binary that declares no usable __TEXT address. How far off the constants were differs sharply between the two widths. Measured over every MH_EXECUTE image a corpus sweep here has covered, fat slices included: all 5,616 of the 64 bit images that do not carry an LC_UNIXTHREAD declare __TEXT at exactly 0x100000000, so on 64 bit the new code reads back the constant it replaces. Of the 15 32 bit images, 1 is at 0x4000 and 14 are at 0x1000 -- nine x86 and five ppc -- so on 32 bit the constant names the wrong address far more often than the right one. This is also what makes the LC_UNIXTHREAD change observable on darwin/amd64. Go's linker still emits LC_UNIXTHREAD rather than LC_MAIN, and of the 699 x86_64 objects in that corpus which carry the command, 695 declare __TEXT at 0x1000000 and 4 at 0x100000000: for those 695 the old constant and the file disagree, and reading the entry point correctly is what exposes it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The test assembled its own Mach-O executables with struct.pack. Test inputs belong in angr/binaries, and a hand-built container is worse than a stray binary file in the wrong repository, because it is shaped to make the test pass: this one linked __TEXT at 0x100000000, where ld64 puts it, and where only 4 of the 699 objects in our corpus whose thread state this change rescues are actually linked -- the other 695 are at 0x1000000. The suite went green while every real binary that uses the command still failed to load. Load tests/x86_64/terramate.macho instead, the terramate executable out of the official Homebrew bottle for tenv 4.15.1. It is a Go-linked x86_64 macOS executable, so it takes its entry point from an x86_THREAD_STATE64 carried by LC_UNIXTHREAD, the flavor that used to abort the load with an empty CLECompatibilityError. The two malformed cases load committed fixtures too: tests/x86_64/terramate_float_thread_state.macho and tests/x86_64/terramate_short_thread_state.macho, each that same binary with one 32 bit field of its LC_UNIXTHREAD command header rewritten, so each differs from it in a single byte. angr/binaries carries them and tests_src/macho_unixthread_variants/build.sh, which rebuilds both from the tracked terramate.macho. Patching a copy at run time would be the same violation as assembling one: the input the test loads would not be a file anybody can look at, and nothing outside the test would ever see it. Two groups of cases went with the assembler: - The arm, arm64 and 32 bit x86 thread states. ld64 stopped emitting LC_UNIXTHREAD long ago and Go's linker only reaches for it on darwin/amd64. The corpus does hold objects carrying a flavor 1 state -- nine x86 and five ppc -- but all of them are samples we cannot publish a fixture from, so there is nothing to commit here for those layouts. They stay in the table and are simply not covered. - "Thread state running past the end of the file", which needs LC_UNIXTHREAD to be the last thing in the file. In a real binary it is not: in terramate it is the seventh of fourteen commands, so a file truncated inside its thread state has lost every segment too and the load fails on an empty backer well before the check matters. The check stays in the parser, where it keeps a short read from surfacing as a bare struct.error, but no real container reaches it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
8b59ffb to
3da1690
Compare
2 participants
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS
Problem
An x86_64 executable that carries its entry point in
LC_UNIXTHREADis refused outright, with an exception carrying no message.tests/x86_64/terramate.machois a Go binary, and Go's internal linker still emitsLC_UNIXTHREADrather thanLC_MAIN:The raise sits inside load-command parsing, so the binary's five segments, its sections and its symbol table are lost with the entry point.
terramate is not one fixture. Of 13,431 Mach-O objects in our corpus that a sweep has actually covered, 708 carry an
LC_UNIXTHREADand 699 carry a(cputype, flavor)pair cle master refuses, every one of themcputype=0x1000007withflavor=4. No object goes the other way.Root cause
MachO._load_lc_unixthreadchose the thread state layout from the flavor field alone:Mach-O flavor numbers are only unique within a cputype. Flavor 4 is
x86_THREAD_STATE64, which matches neither arm, so all 699 take theelse; and the same predicate reads flavor 1 on 32-bit x86 as an arm state.Fix
The layouts are keyed on
(cputype, flavor), with new enums inmacho_enums.py, and the state is bounded by the declared word count and the end of the file. An unknown layout, too small a count, or a truncated file logs a warning and leavesunixthread_pcunset rather than aborting;_resolve_entryalready reports a binary with no entry point.It also reads
linked_basefrom__TEXT's vmaddr for a position-independentMH_EXECUTErather than the ld64 default, because Go links darwin/amd64 at0x1000000and the load would otherwise abort inLoader._map_objectonce the entry point is read correctly. On 64 bit that reads back the constant it replaces, for all 5,616 sweptMH_EXECUTEimages that do not carry the command; on 32 bit it was already wrong for 9 of the 10 images on a cputype cle supports. A binary that declares no usable__TEXTaddress still gets the old constants.Measured on the corpus
One environment per side, cle master
997d4322against this branch, with angr, pyvex, archinfo and pypcode byte-identical across both:All 699 were measured. On master 699 of 699 are refused, every one at
macho.py:839in_load_lc_unixthreadwith an empty message; on this branch 699 of 699 load. The head's answers were checked against each file's own bytes, not against each other:linked_baseequals the object's own__TEXTvmaddr andunixthread_pcthe__ripword of its own thread state, for 699 of 699. 695 of the 699 carryMH_PIEin their own header. The other 4 reach that branch only throughMachO.pic, independently wrong atmacho.py:164; with that corrected they are refused on both sides, so 695 depends on nothing but this change.The two objects cited below are ones the sweep recorded itself: both
okin a 2026-09-21 epoch whose rollup carried this branch, andload-errorin the 2026-10-04 epoch, which does not. Both load here, at0x107bfc0and0x107c380, each the file's own__rip. Three more went through a CFG and one decompiled function each, recovering 2,284, 3,288 and 2,746 functions.$ nix/run.sh --feature pr-cle-727 -- python -P -c "import angr, logging; logging.disable(logging.ERROR); p=angr.Project('OBJECT', auto_load_libs=False); cfg=p.analyses.CFGFast(normalize=True); f=p.kb.functions['gogo']; print(p.analyses.Decompiler(f, cfg=cfg.model).codegen.text)"The samples are unpublished, so all five are identified by SHA-256:
eb9b8808c21a0b12139afc3eba6d503212daeecad7bf40604b9e05afa6665f20,eba86c6c5bb7fb64965bef2d8dcd824c6511a85b45755418ca76c942cf7a84b8,06f50408ac2c9d7c0d54c5f851102cb5dfb5624116a784811fe96c9f3ef3064b,6475878f31ba6fd2a17f0c543d2b6ccf52f664e3228b0b7358a2f1d28485a87fand93e46f3819df001496cba9f3f839114f13b1fa8abae7e38b834dabecb0155768.Regression set: 487 other swept Mach-O objects, none affected, across 19 header strata — arm64, x86_64, 32-bit x86 and arm thin images, fat and Universal2 containers, every file type the corpus has, PIE and not. None got worse. 400 load on both sides and 87 are refused on both with the same message; of the 400, 391 are identical in every loader fact and 9 differ. Those 9 are four thin 32-bit x86
MH_EXECUTEand five fat containers carrying such a slice; on each,__eipis read where master read__gsand got 0. None of the 9 is claimed here: each is a non-PIEMH_EXECUTEreaching this code only through thepicdefect above, and with that corrected both sides refuse all 9. Function recovery is identical on all 25 objects that load on both sides, agree on the base address and produced a count. Four more were excluded for disagreeing on it, and are four of those 9: each recovers 201 functions on master and 197 here, which nothing here settles.Testing
tests/test_macho_unixthread.pyloads three committed fixtures:terramate.machofor the entry point, and the two variants for the cases that must load without one. angr/binaries#245 adds the last two, so this cannot merge before it. Four checks are not green; the validation comment accounts for each. The durable reason: cle resolves a singleangr/binariespull request from this description, so with #245 named, master'stest_relocatable_object_no_symtabcannot reach its own fixture, unmerged in another. All three tests here fail on the merge base.Validation: #727 (comment)
session: sharpen