Repository navigation
Unbreak main: harden CI apt updates and disable broken branch preview - #281
Merged
Merged
Conversation
ikusteu
force-pushed
the
fix/ci-apt-third-party-sources
branch
2 times, most recently
from
September 9, 2026 21:30
56d7c57 to
00571c2
Compare
GitHub's runner images preinstall apt sources for vendors we never install from. When one serves a bad index, `apt-get update` exits 100 and takes the step with it — on jobs whose package list is entirely stock Ubuntu. On 2026-09-09 that turned all of main red: dl.google.com/linux/chrome-stable published a Release naming one Packages.gz and served another, and nine jobs across four workflows died on `Hash Sum mismatch`. The commit on main had nothing to do with it. The same failure arrived earlier from packages.microsoft.com, and the remedy then was an `rm` naming that host at three of the nine call sites — a deny-list that only buys time until the next vendor. So invert it. Every apt package this repo installs in CI comes from the Ubuntu archive, so ci-apt-update.sh drops the sources that are not the Ubuntu archive and updates against Ubuntu alone. All nine call sites now use it, replacing the three host-specific removals. The predicate keeps anything pointing at ubuntu.com rather than emptying the directory, because on Ubuntu 24.04 the main archive lives in that same directory as the deb822 file ubuntu.sources — wiping it would leave the runner unable to install anything. test-ci-apt-update.sh pins that case, along with the two vendors that have actually broken main, deb822 and one-line formats, lookalike hosts, and files apt does not read. It runs in the lint gate: temp-dir fixtures, no root, no apt, no network. Dropping a source list uninstalls nothing; the image's preinstalled Chrome, Node and gh are untouched. Verified end to end on ubuntu:24.04 with the runner's source layout: the vendor lists go, ubuntu.sources stays, and jq, xsltproc, libxml2-utils, shellcheck and ffmpeg all still install. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VAMMNv8AJv6qe2xXWNh3Ju
The UI-only preview rebuilds a per-branch viewer from main's published archive. All 39 meetings in that archive carry the exact-pcm audio integrity policy that 218c3f6 retired on 2026-09-02, so the export can read none of them. It fails for every branch, on every push, over data no branch can fix. A check that is guaranteed red teaches people to ignore red, which costs more than the preview is worth while it is broken. So the job is switched off until the archive is repaired. `false &&` in front of the existing condition rather than deleting the job: the real condition stays visible and correct, and re-enabling is deleting one line. Opt-in processing previews are untouched. They build their meetings from raw recordings, which are fine — the raw bucket is the source of truth here and was never the problem. Adds docs/runbooks/repair-published-archive.md: what broke, what the job is for, how to check whether every published meeting still has a raw recording behind it, how to reprocess, how to verify, and how to turn the job back on. The diagnosis step matters — anything missing from the raw bucket cannot be regenerated, and the upload is a copy rather than a sync, so a repair would leave such a meeting behind as an unreadable file. D-739 carries the decision that is not mine to make: reprocess the archive, or accept exact-pcm on read the way this PR already does for the schema identifier. It also blocks D-699 and D-645, whose cheap path reads these same files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VAMMNv8AJv6qe2xXWNh3Ju
ikusteu
force-pushed
the
fix/ci-apt-third-party-sources
branch
from
September 9, 2026 21:43
00571c2 to
46cae96
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
mainwent red on the merge of #280, and #280 is not the cause. Nine jobs across four workflows died the same way, in theapt-get updatestep:Google republished the Chrome repo's
Releaseat 17:16Z naming onePackages.gzwhile serving another of the same length. GitHub's runner image preinstalls that apt source, so every step that runsapt-get updateunderset -edied — on jobs whose package list is entirely stock Ubuntu and which have no use for Chrome at all.Failed:
Lint,CI,Build + publish ExApp image,Deploy Preview.Deploy Previewwas already red on the previous main commit for the same reason.This has happened before. Three call sites in
ci.ymlalready carry anrmnamingpackages.microsoft.com, with a comment describing exactly this failure. That is a deny-list of hosts we have already been burned by, and it only buys time until the next vendor. This is the next vendor.The fix
Every apt package this repo installs in CI —
jq,ffmpeg,shellcheck,xsltproc,libxml2-utils,ripgrep,python3-numpy— comes from the Ubuntu archive. Soscripts/ci-apt-update.shdrops the sources that are not the Ubuntu archive, then updates against Ubuntu alone. All nine call sites use it, replacing the three host-specific removals.The predicate keeps anything pointing at
ubuntu.comrather than emptying the directory. That distinction is load-bearing: on Ubuntu 24.04 the main archive lives in/etc/apt/sources.list.d/too, as the deb822 fileubuntu.sources. A pruner that wipes the directory leaves the runner unable to install anything, and I confirmed the layout before writing the rule.Dropping a source list uninstalls nothing. The image's preinstalled Chrome, Node and
ghstay exactly where they are; only which indexes apt fetches changes. A future job that genuinely wants a vendor package fails loudly on its own install, which is the right way round.Reproduced and verified
What I could not reproduce: Google's corruption itself. I checked, and the index is healthy again — a fresh
apt-get updateagainst the real repo now exits 0. Synthesising the exactHash Sum mismatchis not tractable, since apt only hard-fails it on a signed repo and only when the newReleasenames an index it has not cached. The four CI logs are the authoritative evidence for the failure.What I did verify, on
ubuntu:24.04with the runner's source layout (ubuntu.sourcesplus the Google and Microsoft vendor lists):scripts/test-ci-apt-update.shcovers the predicate offline — temp-dir fixtures, no root, no apt, no network — so it belongs in the fast lint gate, where it is now listed. Eight cases: the two vendors that have actually broken main, every Ubuntu mirror host the runners use, lookalike hosts (notubuntu.com,archive.ubuntu.com.vendor.net), a mixed file, comment-only files, both source formats, files apt does not read, and a missing directory.I mutation-tested it in both directions. A pruner that deletes everything fails 9 assertions including
ubuntu.sources; a pruner that keeps everything fails 6 including both vendors.Also run locally:
shellcheck -x --source-path=SCRIPTDIR --severity=styleover all 108 tracked scripts,actionlintover all 9 workflows, and the lint gate's script suite. The only local failure istest-prepare-synthetic-meeting.sh, which needs numpy that CI installs and I do not have.Note
Because the upstream index has recovered, a plain re-run of main would likely go green on its own. That is the argument for this change rather than against it: the third time should not need a human to notice.
🤖 Generated with Claude Code
https://claude.ai/code/session_01VAMMNv8AJv6qe2xXWNh3Ju