Skip to content

fix(persistence,rest): pre-size bulk-submit byte progress via HEAD - #877

Merged
smunini merged 2 commits into
mainfrom
fix/874-presize-progress
Sep 2, 2026
Merged

fix(persistence,rest): pre-size bulk-submit byte progress via HEAD#877
smunini merged 2 commits into
mainfrom
fix/874-presize-progress

Conversation

@angela-helios

Copy link
Copy Markdown
Contributor

Closes #874

Stacked on #849 (which now carries #851's merge); merge that one first.

What

On a multi-file manifest the byte-progress denominator was learned lazily — each file's size arrived only when its stream opened — so the percentage kept recomputing against a partial total and every newly opened file yanked the bar backwards. Observed live on the #448 Postgres leg's 20-file run: 99% → 54% → 95% → 34% → ….

SubmitInputFetcher gains file_size() — a best-effort HEAD returning the advertised Content-Length, with a default implementation returning None so every existing mock keeps compiling — and the worker sums all output files up front, storing the complete denominator before ingestion starts. Fallbacks stay honest:

  • any file with an unknown size → lazy accumulation exactly as before;
  • encrypted (JWE) submissions skip pre-sizing entirely (wire length ≠ decrypted length);
  • HEAD requests deliberately skip Accept-Encoding: gzip so the size matches the decompressed bytes the ingest counts.

New worker test: a two-file manifest lands bytes_total == bytes_processed == sum of both files through the pre-sized path. Suites: worker units (6), rest bulk_submit (22) + bulk_submit_jwe (6).

On a multi-file manifest the byte denominator grew as each file's
stream opened, so the percentage repeatedly recomputed against a
partial total and every newly opened file yanked the bar backwards
(99% -> 54% -> 95% -> 34% on a 20-file run).

SubmitInputFetcher gains file_size — a best-effort HEAD returning the
advertised Content-Length, defaulting to None — and the worker sums
every output file's size before ingesting, storing the complete total
up front. Any unknown size (no HEAD support, sizeless response, or an
encrypted file whose wire length is not its decrypted length) falls
back to the previous lazy per-file accumulation.

Closes #874
Response::content_length() is the body size hint, and a HEAD response
has no payload — it reported 0 for every file, the presize summed to
zero, and the presized flag then suppressed the lazy per-file fallback
too: byte progress silently vanished on every backend. Caught live on
the #448 MongoDB leg (bytes_processed counted, bytes_total stuck at 0).

The size now comes from the Content-Length header itself, and a
zero-sum presize no longer claims the denominator.
@angela-helios

Copy link
Copy Markdown
Contributor Author

Follow-up pushed (0f716d4): live testing on the #448 MongoDB leg caught a bug in the first commit. reqwest's Response::content_length() is the body size hint — a HEAD response has no payload, so it reported 0 for every file regardless of the Content-Length header. The presize summed to zero and, worse, the presized flag then suppressed the lazy per-file fallback: byte progress silently vanished on every backend (bytes_processed counted fine, bytes_total stuck at 0). The size now comes from the Content-Length header itself, and a zero-sum presize no longer claims the denominator. Re-verified live: single-file manifest on SQLite lands bytes_total == bytes_processed == file size, and the multi-file MongoDB run reports a monotone percentage.

@smunini
smunini merged commit ef9c99e into main Sep 2, 2026
1 check passed
@smunini
smunini deleted the fix/874-presize-progress branch September 2, 2026 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ui/rest: byte-based bulk progress sawtooths on multi-file manifests — the denominator grows as each file opens

2 participants