Generic, longer, client-visible retry (and mirror fallback) for every external download; 15 s budget lost a ClearCore vendor fetch

[zackees]

What happened

FastLED's ci-full ClearCore job failed on a vendor download:

build error: package error: failed to download https://www.teknic.com/files/downloads/ClearCore-1.7.4.zip:
error sending request for url (https://www.teknic.com/files/downloads/ClearCore-1.7.4.zip)
❌ Compilation failed (fbuild) [167.8s]

Run: FastLED/FastLED actions run 36100730589 (job 107962450916), on PR FastLED/FastLED#4632. The change under test cannot affect board builds. The failure was an unreachable third-party host.

Why the existing retry did not save it

crates/fbuild-packages-fetch/src/downloader.rs already retries: MAX_ATTEMPTS = 5, RETRY_BACKOFFS = 1s, 2s, 4s, 8s, with is_transient covering connect, request, body and 5xx errors. Two limits:

1. The whole retry budget is about 15 s. A vendor site that is down for a minute, or a flaky CDN edge, outlasts it every time.
2. The retries are invisible to the caller. The download …: … on attempt N/5, retrying after … warnings go to the daemon log only. The client error says nothing about attempts, so CI output can't tell "retried 5×" from "never retried".

This is one instance of a general problem: every external fetch fbuild does (framework/platform archives, toolchains, esptool, vendor SDK zips like ClearCore's) depends on third-party hosts, and any of them can blip.

Ask: one generic, shared retry policy for every external fetch

• A single helper used by all fetchers (package archives, toolchains, tool binaries, metadata JSON). No per-call-site loops.
• A longer, bounded, jittered backoff, for example exponential with full jitter capped at 30 s per wait and about 3–5 min total, configurable by env (FBUILD_DOWNLOAD_MAX_ATTEMPTS, FBUILD_DOWNLOAD_MAX_WAIT). It should honour Retry-After on 429/503.
• Retry classification kept as today: connect, timeout, request, body, truncation and 5xx retry; 4xx, URL-parse and local disk errors fail fast.
• Resume on retry where the origin supports ranges. The Content-Range handling already exists.
• Mirror / fallback URLs per package. When the primary vendor host fails after its budget, try a mirror. An immutable mirror of pinned archives, e.g. release assets under a FastLED/fbuild org repo keyed by SHA-256, would make vendor outages a non-event for pinned versions.
• A content-addressed cache hit short-circuits the network, so a pinned archive already verified once is never re-fetched.
• Client-visible reporting: the final error names the URL, attempt count, total time waited, and each attempt's error class. Also log one line per retry at the client, not only in the daemon log.

FastLED-side follow-up (tracked here for visibility)

• FastLED/.github/workflows/build_clearcore.yml pins fbuild_version: "2.5.19", while the project pins fbuild==2.5.27. It should use the project pin, so fixes like this one actually reach the ClearCore job.

Acceptance

• A fault-injection test in which the first N attempts fail with connect errors and the (N+1)th succeeds. The download succeeds and the client output reports the retries.
• A test for a 4xx response, which fails immediately without retrying.
• A test for primary-down-then-mirror, which succeeds from the mirror and verifies the SHA-256.