Need to Rollback the data if the no of rejection exceeds 50  #1375

Activity

1. dimitri

   added

   WishList

   Sponsor needed

   on Jun 23, 2026
2. dimitri commented on Jun 23, 2026

   

   dimitri

   on Jun 23, 2026

   Owner

   More actions

   Today pgloader counts rejected rows and writes them to a .rej file, but it
   always finishes the load regardless of how many rows were rejected. There is
   no way to say "if more than N rows are bad, something is wrong with the source
   file — stop early."

   Design

   Two new WITH options that can be used independently or together:

   LOAD CSV FROM 'data.csv' ...
   WITH max errors 50,          -- abort after 50 rejected rows (absolute)
        max error ratio 0.01;   -- or after 1 % of rows are rejected

   The rejection counter is maintained in the batch writer alongside the existing
   stats. When either threshold is crossed the writer signals an abort: the
   in-progress COPY batch is rolled back and pgloader exits with a non-zero
   status and a clear message stating how many rows were rejected and which
   threshold was hit.

   Important caveat: pgloader loads data in batches of ~1 000 rows, each
   committed independently. There is no single wrapping transaction around the
   entire load. When the threshold is crossed, already-committed batches are
   not rolled back — pgloader aborts forward progress but leaves the target
   table in a partially-loaded state. You get to keep the pieces. If an
   all-or-nothing guarantee is required, wrap the load in a pre/post SQL step
   that truncates the table on failure, or use a staging table and swap on
   success.

   ---

   If you'd like to see this feature prioritised, sponsoring its development is
   possible through The Art of PostgreSQL OSS sponsorship
   program. Sponsored features get
   dedicated implementation time and are tracked to completion; the design above
   is already in place and ready to go.