[2pc] SIGTERM may not clean up transactions saved in 2pc WAL

[levkk]

PgDog version
v0.1.48

Description
Upon receiving SIGTERM, some 2pc transactions were left in an unfinished state when pgdog rebooted. The WAL was on, so they should of been cleaned up, even though SIGTERM causes an immediate shutdown without running the WAL manager.

Configuration

2pc WAL was enabled.

[levkk]

Looking at our WAL implementation, this makes sense. We call fsync once a second which leaves ample time for a transaction to get lost. This is an incorrect implementation of a durable WAL.

We should change this to mimic what Postgres is doing:

1. All transactions state change require fsync to acknowledge
2. Transactions queue up for fsync
3. The first one to call fsync gets the LSN of the last transaction in the queue, calls fsync, and records that as the last flushed LSN

[willothy]

Hmm the 1s (configurable) is just the group-commit coalesce timeout iirc, so I don't think that's the issue. I think it's an issue related to GID prefixes when recovering from WAL. PR on the way.

[levkk]

Fixed in #1284.