Problem
On a one-shot bulk import (a single manifest, one big NDJSON file) the UI progress bar sweeps indeterminately for the entire run and the reported percentage stays at 0% until the very end. The percentage behind X-Progress is manifest-granular — completed manifests / total manifests — so a single-manifest submission is 0% for its whole life and jumps straight to done. The entry counter moves, but the bar itself never fills, which reads as "nunca termina de cargar".
Proposal: byte-based progress
The recipient knows how many bytes each output file advertises (Content-Length) and how many bytes the ingestion engine has consumed. Report the percentage as consumed/total bytes:
SubmitInputFetcher::open_file_stream returns the stream plus its content length (None for sizeless streams, e.g. gzip-decompressed bodies).
- The worker wraps each file stream in a counting reader and flushes
bytes_processed/bytes_total to the manifest row every few seconds from the lease keeper's task, so a status poll sees movement while a large file streams.
- The status endpoint computes the percentage from summed bytes (capped at 99 until terminal), falling back to the old manifest-count fraction when no totals are known.
- The UI renders 0% as a determinate empty bar that fills within seconds — the indeterminate sweep is reserved for recipients that report no percentage at all.
Needs a schema addition (bytes_processed/bytes_total on manifests) across the SQLite, PostgreSQL, MongoDB, and S3 backends.
Problem
On a one-shot bulk import (a single manifest, one big NDJSON file) the UI progress bar sweeps indeterminately for the entire run and the reported percentage stays at 0% until the very end. The percentage behind
X-Progressis manifest-granular — completed manifests / total manifests — so a single-manifest submission is 0% for its whole life and jumps straight to done. The entry counter moves, but the bar itself never fills, which reads as "nunca termina de cargar".Proposal: byte-based progress
The recipient knows how many bytes each output file advertises (
Content-Length) and how many bytes the ingestion engine has consumed. Report the percentage as consumed/total bytes:SubmitInputFetcher::open_file_streamreturns the stream plus its content length (Nonefor sizeless streams, e.g. gzip-decompressed bodies).bytes_processed/bytes_totalto the manifest row every few seconds from the lease keeper's task, so a status poll sees movement while a large file streams.Needs a schema addition (
bytes_processed/bytes_totalon manifests) across the SQLite, PostgreSQL, MongoDB, and S3 backends.