Repository navigation
Commit ba7dbcb
feat: Single-pass WAL streaming for LOG_BASED replication (MeltanoLabs#772)
### problem
`PostgresLogBasedStream.get_records()` opens its own
`LogicalReplicationConnection` per selected stream. With N LOG_BASED
streams the tap runs N sequential WAL scans -- each rereads the same
segments, with add-tables discarding most records server-side.
End-to-end sync time scales ~linearly in N. For pipelines with multiple
LOG_BASED streams against a large backlog, this dominates run-time.
### changes
A new `SingleConnectionWALReader` opens one logical replication
connection with `add-tables` covering all selected LOG_BASED tables,
scans the WAL once, and dispatches each parsed wal2json message inline
to the owning stream's new `emit_record()` method for immediate Singer
RECORD emission. STATE flushes every 30s, and the slot is advanced to
the WAL tip on idle/max-run exit.
- new modules: `_wal_helpers.py` (FQN/escaping/parsing helpers) and
`wal_reader.py` (the reader and read loop).
- `client.py` gains `emit_record` method and a config-flag branch in
`get_records`
- `tap.py` adds the `log_based_single_connection` config setting
(default False!) and `_sync_log_based_streams_shared` orchestration
- new tests: `tests/test_wal_helpers.py`, `tests/test_wal_reader.py`,
`tests/test_consume.py`
Full disclosure, I had Claude Code implement those three test modules,
and then I iterated a bit. If it's still excessive / not testing
usefully -- something Claude is known to do, sigh -- just let me know,
and I will take a hatchet to it.
This is a very belated follow-up to PR MeltanoLabs#667 and Issue MeltanoLabs#587.
### constraints
- `Tap.sync_all` is `@typing.final`, so dispatch can't be restructured
at the SDK boundary -- this was a bummer. My next best option was
trigger at the first LOG_BASED stream's `get_records()` call, gated by a
`_shared_wal_run_completed` flag on the tap so siblings become no-ops.
- SCHEMA-before-RECORD across streams: `_sync_log_based_streams_shared`
pre-writes every stream's schema before the reader runs. Since the SDK's
`Stream.sync()` later calls `_write_schema_message()` again, the
override on `PostgresLogBasedStream` is idempotent. (Without that flag
every SCHEMA would be emitted twice, which is not great.)
- Per-stream LSN filter: Replication opens at `min(start_lsn)` across
all streams, so each stream's own bookmark is captured at construction
and used to drop messages that it's already past.
- I had to dip into private SDK calls -- `_write_record_message()` and
`_increment_stream_state()` -- for this to work. I consolidated them in
one place -- `emit_record()` -- so SDK renames hit one method. Not sure
how stable the API is here...
### questions
1. `_write_schema_message` idempotency: I made the smallest fix I could
for the duplicate-SCHEMA bug given the `@final` constraint on
`sync_all`. Is there another / cleaner approach?
2. `emit_record()` uses internal SDK calls. Is this going to be an
issue? Is there a safer / "public" equivalent?
3. `get_records()` as trigger: The "first stream's get_records fires the
shared reader" pattern is a not-great workaround for `sync_all` being
final. Is it okay as documented, or is there a cleaner SDK hook?
4. `replication_max_run_seconds` / `replication_idle_exit_seconds` now
bound the whole LOG_BASED batch instead of each stream. To me that feels
like an improvement, but I don't know the whole system / downstream use
caess. For example, does anything downstream assume per-stream bounds?
---------
Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
Co-authored-by: Edgar Ramírez Mondragón <16805946+edgarrmondragon@users.noreply.github.com>
Co-authored-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>1 parent 554ed05 commit ba7dbcb
8 files changed
Lines changed: 1558 additions & 125 deletions
File tree
- tap_postgres
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
261 | 261 | | |
262 | 262 | | |
263 | 263 | | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
264 | 277 | | |
265 | 278 | | |
266 | 279 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
0 commit comments