Skip to content

ui/rest: bulk import progress bar stays at 0% (indeterminate sweep) for one-shot submissions — report byte-based progress #848

Description

@angela-helios

Problem

On a one-shot bulk import (a single manifest, one big NDJSON file) the UI progress bar sweeps indeterminately for the entire run and the reported percentage stays at 0% until the very end. The percentage behind X-Progress is manifest-granular — completed manifests / total manifests — so a single-manifest submission is 0% for its whole life and jumps straight to done. The entry counter moves, but the bar itself never fills, which reads as "nunca termina de cargar".

Proposal: byte-based progress

The recipient knows how many bytes each output file advertises (Content-Length) and how many bytes the ingestion engine has consumed. Report the percentage as consumed/total bytes:

  • SubmitInputFetcher::open_file_stream returns the stream plus its content length (None for sizeless streams, e.g. gzip-decompressed bodies).
  • The worker wraps each file stream in a counting reader and flushes bytes_processed/bytes_total to the manifest row every few seconds from the lease keeper's task, so a status poll sees movement while a large file streams.
  • The status endpoint computes the percentage from summed bytes (capped at 99 until terminal), falling back to the old manifest-count fraction when no totals are known.
  • The UI renders 0% as a determinate empty bar that fills within seconds — the indeterminate sweep is reserved for recipients that report no percentage at all.

Needs a schema addition (bytes_processed/bytes_total on manifests) across the SQLite, PostgreSQL, MongoDB, and S3 backends.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions