fix(persistence,rest): pre-size bulk-submit byte progress via HEAD - #877
Merged
Conversation
On a multi-file manifest the byte denominator grew as each file's stream opened, so the percentage repeatedly recomputed against a partial total and every newly opened file yanked the bar backwards (99% -> 54% -> 95% -> 34% on a 20-file run). SubmitInputFetcher gains file_size — a best-effort HEAD returning the advertised Content-Length, defaulting to None — and the worker sums every output file's size before ingesting, storing the complete total up front. Any unknown size (no HEAD support, sizeless response, or an encrypted file whose wire length is not its decrypted length) falls back to the previous lazy per-file accumulation. Closes #874
Response::content_length() is the body size hint, and a HEAD response has no payload — it reported 0 for every file, the presize summed to zero, and the presized flag then suppressed the lazy per-file fallback too: byte progress silently vanished on every backend. Caught live on the #448 MongoDB leg (bytes_processed counted, bytes_total stuck at 0). The size now comes from the Content-Length header itself, and a zero-sum presize no longer claims the denominator.
Contributor
Author
|
Follow-up pushed (0f716d4): live testing on the #448 MongoDB leg caught a bug in the first commit. reqwest's Response::content_length() is the body size hint — a HEAD response has no payload, so it reported 0 for every file regardless of the Content-Length header. The presize summed to zero and, worse, the presized flag then suppressed the lazy per-file fallback: byte progress silently vanished on every backend (bytes_processed counted fine, bytes_total stuck at 0). The size now comes from the Content-Length header itself, and a zero-sum presize no longer claims the denominator. Re-verified live: single-file manifest on SQLite lands bytes_total == bytes_processed == file size, and the multi-file MongoDB run reports a monotone percentage. |
9 tasks
smunini
approved these changes
Sep 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #874
Stacked on #849 (which now carries #851's merge); merge that one first.
What
On a multi-file manifest the byte-progress denominator was learned lazily — each file's size arrived only when its stream opened — so the percentage kept recomputing against a partial total and every newly opened file yanked the bar backwards. Observed live on the #448 Postgres leg's 20-file run:
99% → 54% → 95% → 34% → ….SubmitInputFetchergainsfile_size()— a best-effort HEAD returning the advertisedContent-Length, with a default implementation returningNoneso every existing mock keeps compiling — and the worker sums all output files up front, storing the complete denominator before ingestion starts. Fallbacks stay honest:Accept-Encoding: gzipso the size matches the decompressed bytes the ingest counts.New worker test: a two-file manifest lands
bytes_total == bytes_processed == sumof both files through the pre-sized path. Suites: worker units (6), restbulk_submit(22) +bulk_submit_jwe(6).