Skip to content

Unbreak main: harden CI apt updates and disable broken branch preview - #281

Merged
ikusteu merged 2 commits into
mainfrom
fix/ci-apt-third-party-sources
Sep 9, 2026
Merged

ikusteu merged 2 commits into
mainfrom
fix/ci-apt-third-party-sources

Conversation

@ikusteu

@ikusteu ikusteu commented Sep 9, 2026

Copy link
Copy Markdown
Member

What broke

main went red on the merge of #280, and #280 is not the cause. Nine jobs across four workflows died the same way, in the apt-get update step:

E: Failed to fetch https://dl.google.com/linux/chrome-stable/deb/dists/stable/main/binary-amd64/Packages.gz
   Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.
##[error]Process completed with exit code 100.

Google republished the Chrome repo's Release at 17:16Z naming one Packages.gz while serving another of the same length. GitHub's runner image preinstalls that apt source, so every step that runs apt-get update under set -e died — on jobs whose package list is entirely stock Ubuntu and which have no use for Chrome at all.

Failed: Lint, CI, Build + publish ExApp image, Deploy Preview. Deploy Preview was already red on the previous main commit for the same reason.

This has happened before. Three call sites in ci.yml already carry an rm naming packages.microsoft.com, with a comment describing exactly this failure. That is a deny-list of hosts we have already been burned by, and it only buys time until the next vendor. This is the next vendor.

The fix

Every apt package this repo installs in CI — jq, ffmpeg, shellcheck, xsltproc, libxml2-utils, ripgrep, python3-numpy — comes from the Ubuntu archive. So scripts/ci-apt-update.sh drops the sources that are not the Ubuntu archive, then updates against Ubuntu alone. All nine call sites use it, replacing the three host-specific removals.

The predicate keeps anything pointing at ubuntu.com rather than emptying the directory. That distinction is load-bearing: on Ubuntu 24.04 the main archive lives in /etc/apt/sources.list.d/ too, as the deb822 file ubuntu.sources. A pruner that wipes the directory leaves the runner unable to install anything, and I confirmed the layout before writing the rule.

Dropping a source list uninstalls nothing. The image's preinstalled Chrome, Node and gh stay exactly where they are; only which indexes apt fetches changes. A future job that genuinely wants a vendor package fails loudly on its own install, which is the right way round.

Reproduced and verified

What I could not reproduce: Google's corruption itself. I checked, and the index is healthy again — a fresh apt-get update against the real repo now exits 0. Synthesising the exact Hash Sum mismatch is not tractable, since apt only hard-fails it on a signed repo and only when the new Release names an index it has not cached. The four CI logs are the authoritative evidence for the failure.

What I did verify, on ubuntu:24.04 with the runner's source layout (ubuntu.sources plus the Google and Microsoft vendor lists):

[apt] dropping third-party source: google-chrome.list
[apt] dropping third-party source: microsoft-prod.list
[apt] kept 1 Ubuntu source(s), dropped 2 third-party source(s)
apt-get update exit=0

AFTER: ubuntu.sources

installed ok:
  jq /usr/bin/jq   xsltproc /usr/bin/xsltproc   xmllint /usr/bin/xmllint
  shellcheck /usr/bin/shellcheck   ffmpeg /usr/bin/ffmpeg   ffprobe /usr/bin/ffprobe

scripts/test-ci-apt-update.sh covers the predicate offline — temp-dir fixtures, no root, no apt, no network — so it belongs in the fast lint gate, where it is now listed. Eight cases: the two vendors that have actually broken main, every Ubuntu mirror host the runners use, lookalike hosts (notubuntu.com, archive.ubuntu.com.vendor.net), a mixed file, comment-only files, both source formats, files apt does not read, and a missing directory.

I mutation-tested it in both directions. A pruner that deletes everything fails 9 assertions including ubuntu.sources; a pruner that keeps everything fails 6 including both vendors.

Also run locally: shellcheck -x --source-path=SCRIPTDIR --severity=style over all 108 tracked scripts, actionlint over all 9 workflows, and the lint gate's script suite. The only local failure is test-prepare-synthetic-meeting.sh, which needs numpy that CI installs and I do not have.

Note

Because the upstream index has recovered, a plain re-run of main would likely go green on its own. That is the argument for this change rather than against it: the third time should not need a human to notice.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VAMMNv8AJv6qe2xXWNh3Ju

@ikusteu ikusteu added full-ci Run all CI tests, including the ones too heavy to run on each commit preview Build and push a preview docker image labels Sep 9, 2026
@ikusteu ikusteu changed the title Stop a third-party apt repo from turning main red Unbreak main: a vendor apt repo, and portable files rejected after a schema rename Sep 9, 2026
@ikusteu ikusteu removed the full-ci Run all CI tests, including the ones too heavy to run on each commit label Sep 9, 2026
@ikusteu
ikusteu force-pushed the fix/ci-apt-third-party-sources branch 2 times, most recently from 56d7c57 to 00571c2 Compare September 9, 2026 21:30
ikusteu and others added 2 commits September 9, 2026 23:43
GitHub's runner images preinstall apt sources for vendors we never install
from. When one serves a bad index, `apt-get update` exits 100 and takes the
step with it — on jobs whose package list is entirely stock Ubuntu.

On 2026-09-09 that turned all of main red: dl.google.com/linux/chrome-stable
published a Release naming one Packages.gz and served another, and nine jobs
across four workflows died on `Hash Sum mismatch`. The commit on main had
nothing to do with it. The same failure arrived earlier from
packages.microsoft.com, and the remedy then was an `rm` naming that host at
three of the nine call sites — a deny-list that only buys time until the next
vendor.

So invert it. Every apt package this repo installs in CI comes from the Ubuntu
archive, so ci-apt-update.sh drops the sources that are not the Ubuntu archive
and updates against Ubuntu alone. All nine call sites now use it, replacing the
three host-specific removals.

The predicate keeps anything pointing at ubuntu.com rather than emptying the
directory, because on Ubuntu 24.04 the main archive lives in that same
directory as the deb822 file ubuntu.sources — wiping it would leave the runner
unable to install anything. test-ci-apt-update.sh pins that case, along with
the two vendors that have actually broken main, deb822 and one-line formats,
lookalike hosts, and files apt does not read. It runs in the lint gate: temp-dir
fixtures, no root, no apt, no network.

Dropping a source list uninstalls nothing; the image's preinstalled Chrome,
Node and gh are untouched. Verified end to end on ubuntu:24.04 with the runner's
source layout: the vendor lists go, ubuntu.sources stays, and jq, xsltproc,
libxml2-utils, shellcheck and ffmpeg all still install.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VAMMNv8AJv6qe2xXWNh3Ju
The UI-only preview rebuilds a per-branch viewer from main's published
archive. All 39 meetings in that archive carry the exact-pcm audio integrity
policy that 218c3f6 retired on 2026-09-02, so the export can read none of
them. It fails for every branch, on every push, over data no branch can fix.

A check that is guaranteed red teaches people to ignore red, which costs more
than the preview is worth while it is broken. So the job is switched off until
the archive is repaired.

`false &&` in front of the existing condition rather than deleting the job: the
real condition stays visible and correct, and re-enabling is deleting one line.

Opt-in processing previews are untouched. They build their meetings from raw
recordings, which are fine — the raw bucket is the source of truth here and was
never the problem.

Adds docs/runbooks/repair-published-archive.md: what broke, what the job is
for, how to check whether every published meeting still has a raw recording
behind it, how to reprocess, how to verify, and how to turn the job back on.
The diagnosis step matters — anything missing from the raw bucket cannot be
regenerated, and the upload is a copy rather than a sync, so a repair would
leave such a meeting behind as an unreadable file.

D-739 carries the decision that is not mine to make: reprocess the archive, or
accept exact-pcm on read the way this PR already does for the schema
identifier. It also blocks D-699 and D-645, whose cheap path reads these same
files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VAMMNv8AJv6qe2xXWNh3Ju
@ikusteu
ikusteu force-pushed the fix/ci-apt-third-party-sources branch from 00571c2 to 46cae96 Compare September 9, 2026 21:43
@ikusteu ikusteu changed the title Unbreak main: a vendor apt repo, and portable files rejected after a schema rename Unbreak main: harden CI apt updates and disable broken branch preview Sep 9, 2026
@ikusteu
ikusteu merged commit 7002a52 into main Sep 9, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

preview Build and push a preview docker image

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant