Skip to content

Commit da076f0

Browse files
mikolalysenkoclaude
mikolalysenko
and
claude
authored
Re-download truncated e2e-bin archives once in CI (#1387)
* Re-download truncated e2e-bin archives once download-artifact v4 can finish "successfully" with a truncated file. Run 38059192473 attempt 1 (PR #1345) logged "Starting download" and nothing more, then the unpack step failed with zstd "Read error (39) : premature end" and the hatch e2e_vex_build leg went red. A re-run on the same SHA passed. In the merge queue the same blip evicts the entry and costs a full rebuild, and every merge group downloads e2e-bin once per e2e and cargo-vex leg. Add a composite download-artifact wrapper, alongside the upload one, that runs a caller-supplied check on the downloaded files. When the download or the check fails it clears the path and downloads once more; a second failure fails the job. The two e2e-bin consumers check the archive with `zstd -t`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KdaGJqi1VLtSWMnHY4hP6g * Require verify and gate retry checks precisely The second check now runs only after the second download succeeds, so a failed retry download reports its own error rather than a missing file. `verify` is required: a caller without a check would get no retry. The test pins every step's exact `if:` gate, which catches a broken gate that a substring match missed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KdaGJqi1VLtSWMnHY4hP6g --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent c174e67 commit da076f0

3 files changed

Lines changed: 123 additions & 2 deletions

File tree

‎.github/actions/download-artifact/action.yml‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
name: Download artifact with a verified retry
2+
description: Re-download once when the download fails or leaves files that fail a check
3+
inputs:
4+
pattern:
5+
description: Glob of artifact names to download
6+
required: true
7+
path:
8+
description: Directory to download into (cleared before the retry)
9+
required: true
10+
merge-multiple:
11+
description: Extract every matching artifact into the same directory
12+
default: 'false'
13+
verify:
14+
description: >-
15+
Bash script, run from the workspace, that exits non-zero when the
16+
downloaded files are incomplete or corrupt (e.g. `zstd -t <file>`)
17+
required: true
18+
runs:
19+
using: composite
20+
steps:
21+
# download-artifact v4 can finish "successfully" with a truncated file:
22+
# run 38059192473 attempt 1 logged "Starting download" and nothing more,
23+
# then `zstd -d` failed with "Read error (39) : premature end" in the
24+
# next step and the leg went red. A merge-queue leg that does that
25+
# evicts the entry, so check the files and fetch them once more.
26+
- name: Download (attempt 1)
27+
id: first
28+
if: ${{ !cancelled() }}
29+
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
30+
continue-on-error: true
31+
with:
32+
pattern: ${{ inputs.pattern }}
33+
merge-multiple: ${{ inputs.merge-multiple }}
34+
path: ${{ inputs.path }}
35+
36+
- name: Verify (attempt 1)
37+
id: first-check
38+
if: ${{ !cancelled() && steps.first.outcome == 'success' }}
39+
continue-on-error: true
40+
shell: bash
41+
env:
42+
VERIFY: ${{ inputs.verify }}
43+
run: bash --noprofile --norc -euo pipefail -c "$VERIFY"
44+
45+
- name: Clear the failed download
46+
if: ${{ !cancelled() && (steps.first.outcome == 'failure' || steps.first-check.outcome == 'failure') }}
47+
shell: bash
48+
env:
49+
DOWNLOAD_PATH: ${{ inputs.path }}
50+
run: |
51+
set -euo pipefail
52+
echo "::warning::Artifact download or check failed; downloading again"
53+
rm -rf -- "$DOWNLOAD_PATH"
54+
sleep 10
55+
56+
# Neither step below continues on error: a second bad download is not
57+
# a blip, and the job must fail rather than test corrupt files.
58+
- name: Download (attempt 2)
59+
id: second
60+
if: ${{ !cancelled() && (steps.first.outcome == 'failure' || steps.first-check.outcome == 'failure') }}
61+
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
62+
with:
63+
pattern: ${{ inputs.pattern }}
64+
merge-multiple: ${{ inputs.merge-multiple }}
65+
path: ${{ inputs.path }}
66+
67+
- name: Verify (attempt 2)
68+
if: ${{ !cancelled() && steps.second.outcome == 'success' }}
69+
shell: bash
70+
env:
71+
VERIFY: ${{ inputs.verify }}
72+
run: bash --noprofile --norc -euo pipefail -c "$VERIFY"

‎.github/workflows/ci.yml‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1288,11 +1288,13 @@ jobs:
12881288
persist-credentials: false
12891289

12901290
- name: Download the e2e binaries
1291-
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
1291+
# Re-downloads once if the archive arrives truncated.
1292+
uses: ./.github/actions/download-artifact
12921293
with:
12931294
pattern: e2e-bin-${{ matrix.os }}*
12941295
merge-multiple: true
12951296
path: target/e2e-archive
1297+
verify: zstd -q -t target/e2e-archive/e2e-bin.tar.zst
12961298

12971299
- name: Unpack the e2e binaries
12981300
shell: bash
@@ -2261,11 +2263,13 @@ jobs:
22612263
with:
22622264
persist-credentials: false
22632265
- name: Download the e2e binaries
2264-
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
2266+
# Re-downloads once if the archive arrives truncated.
2267+
uses: ./.github/actions/download-artifact
22652268
with:
22662269
pattern: e2e-bin-${{ matrix.os }}*
22672270
merge-multiple: true
22682271
path: target/e2e-archive
2272+
verify: zstd -q -t target/e2e-archive/e2e-bin.tar.zst
22692273
- name: Unpack the e2e binaries
22702274
shell: bash
22712275
# Decompress to a file, not into `| tar -xf -`: a tar that exits at

‎scripts/tests/test_ci_e2e_archive.py‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,51 @@ def test_corrupt_archive_fails_the_consumer_step(self):
119119
(archive_dir / "e2e-bin.tar.zst").write_bytes(b"not a zstd archive")
120120
self.assertNotEqual(run(consumer, work).returncode, 0)
121121

122+
def test_download_retries_when_the_archive_arrives_truncated(self):
123+
# download-artifact v4 once left a truncated e2e-bin.tar.zst and
124+
# still passed (run 38059192473): each consumer's verify check must
125+
# reject that file so the wrapper downloads it again.
126+
checks = []
127+
for job in JOBS.values():
128+
for step_name, step in rows.steps(job):
129+
if step_name == "Download the e2e binaries":
130+
self.assertIn("uses: ./.github/actions/download-artifact", step)
131+
verify = re.search(r"(?m)^ verify: (.+)$", step)
132+
self.assertIsNotNone(verify, "an e2e-bin download has no verify check")
133+
checks.append(verify[1])
134+
self.assertTrue(checks)
135+
for check in checks:
136+
with tempfile.TemporaryDirectory(prefix="e2e-truncated-") as directory:
137+
work = Path(directory)
138+
archive_dir = work / "target/e2e-archive"
139+
archive_dir.mkdir(parents=True)
140+
archive = archive_dir / "e2e-bin.tar.zst"
141+
payload = work / "payload"
142+
payload.write_bytes(os.urandom(1 << 20))
143+
subprocess.run(["zstd", "-q", str(payload), "-o", str(archive)], check=True)
144+
self.assertEqual(run(check, work).returncode, 0, check)
145+
archive.write_bytes(archive.read_bytes()[: archive.stat().st_size // 2])
146+
self.assertNotEqual(run(check, work).returncode, 0, check)
147+
archive.unlink()
148+
self.assertNotEqual(run(check, work).returncode, 0, "a missing archive passed " + check)
149+
150+
def test_download_wrapper_retries_a_failed_check_and_fails_on_the_second(self):
151+
action = (ROOT / ".github/actions/download-artifact/action.yml").read_text()
152+
retry = "${{ !cancelled() && (steps.first.outcome == 'failure' || steps.first-check.outcome == 'failure') }}"
153+
steps = action.split(" - name: ")[1:]
154+
gates = {step.split("\n", 1)[0]: re.search(r"(?m)^ if: (.+)$", step)[1] for step in steps}
155+
self.assertEqual(gates, {
156+
"Download (attempt 1)": "${{ !cancelled() }}",
157+
"Verify (attempt 1)": "${{ !cancelled() && steps.first.outcome == 'success' }}",
158+
"Clear the failed download": retry,
159+
"Download (attempt 2)": retry,
160+
"Verify (attempt 2)": "${{ !cancelled() && steps.second.outcome == 'success' }}",
161+
})
162+
# Only the first attempt may fail softly; the second must fail the job.
163+
for step in steps:
164+
name = step.split("\n", 1)[0]
165+
self.assertEqual("continue-on-error: true" in step, name.endswith("(attempt 1)"), name)
166+
122167

123168
if __name__ == "__main__":
124169
unittest.main()

0 commit comments

Comments
 (0)