Repository navigation
[ISSUE #141] fix: commit transactions on the node holding the context and close the remaining silent paths - #149
Merged
Conversation
…sage A proxy keeps the transaction context of a half message in its own process memory, keyed by producer group and transaction id, and expires it after a short TTL. END_TRANSACTION sent to any other node finds no context, and the server answers that miss with a success code while discarding the request, so the commit is lost with every health signal green. The committer used to resolve the target address from its own publish route table, which is populated independently of the writer's and may name a different node; with two committer subtasks each rolling its own dice, runs silently lost half or all of a checkpoint. Record the address the writer actually sent to on the SendCommittable right after the transactional send, and have the committer use it, falling back to route resolution only when a restored committable carries no address. The serializer moves to version 2 and still reads version 1 state. Verified on a commercial 5.5.1 cluster with clean-room TRANSACTION topics: parallelism 1 and 2 both land 101/101 with zero missing, unexpected or duplicate records, and the route-refresh fallback warning drops from 4 to 0.
…ad parameters A message whose deserialization throws used to fail the task, which restored from the checkpoint, re-read the same record and failed again, rewriting everything it had already emitted each round. Drop the record instead, log a WARN carrying message id, topic, queue and offset, and advance the split offset past it. Failures raised while emitting an already-deserialized record (the downstream sink runs inline inside collect) are still propagated, so a real write failure cannot hide behind the poison-message path. Also close the two remaining misleading sink parameters: send.pending.max was written into the configuration but never read, so the send buffer grew without limit; the writer now bounds in-flight async sends with a semaphore of that size. The SQL option retry.times mapped onto a key nobody read; it now acts as an alias for send.retry.times (new key wins, default 3 when neither is set), and the dead pass-through plus its unused field are gone. Finally, InnerConsumerImpl.commitOffset updated only the local offset store with increaseOnly=true and never persisted, which had no effect at all; it now publishes the checkpointed offset (rollbacks included) so the broker-side, display-only offset tracks the job instead of staying stale.
A synchronously throwing producer.send never reaches the whenComplete that returns the semaphore permit and decrements the pending count, which would wedge the writer at the send.pending.max limit after a single such failure.
…nded executor Committing ran each END_TRANSACTION on the producer's callback executor, which is bounded to four threads and a 2000-task queue with an abort policy. A checkpoint carrying more committables than that rejected the surplus with RejectedExecutionException; the committer treats such a failure as retryable, and a retry after the server-side transaction context expired is silently dropped, so a large checkpoint lost records with the job staying green. End the transaction on the thread that calls commit/rollback. The committer blocks until every request of the checkpoint is done anyway, so a burst now shows up as checkpoint duration instead of uncommitted half messages.
…times a second rocketmq.client.poll.timeout defaulted to 10ms, so a source with nothing to read re-entered fetch() a hundred times a second per subtask. The client poll waits on a blocking queue and returns as soon as records arrive, so raising the bound shortens nothing on the data path; it only cuts the idle spin, to one fetch a second. wakeUp() merely sets a flag and the consumer deliberately has no interruptible poll, which leaves this timeout as the sole bound on how long a fetch can outlive a close request. One second keeps a shutdown prompt. Adds a package-private constructor taking the InnerConsumer so the timeout that actually reaches poll() is covered by a test.
…ate TTL A proxy drops the transaction state of a half message after transactionDataExpireMillis, 30 seconds by default, and Flink only commits at the end of a checkpoint, so a long interval loses commits that the server still answers with a success code. Also list rocketmq.client.poll.timeout among the client options.
The acceptance round was formatted under JDK 21 where spotless silently skips; under the JDK 11 the project gates on, ten files violate the import order. Reformat them, no behaviour change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
An acceptance run of this connector against a commercial RocketMQ 5.5.1 cluster on Flink 1.20.3
(Application mode, K8s) found a family of defects with the same shape as #141: the job stays
RUNNING, checkpoints succeed, nothing is logged, and the data is wrong or absent. This PR carries
the fixes from that round that are still open, grouped into four topics. Each one was verified on
the cluster with clean-room topics before being committed; the per-defect root causes and code
evidence are in the acceptance report kept outside this repo.
What is in this PR
1. Transactional commits are lost when END_TRANSACTION lands on another node (the headline fix).
A proxy keeps the transaction context of a half message in its own process memory
(
TransactionDataManager, keyed by producer group + transaction id, expiring aftertransactionDataExpireMillis, 30s by default). The committer used to resolve the target addressfrom its own publish route table, which is populated independently of the writer's and may name a
different node; with two committer subtasks each rolling its own dice, runs silently lost half or
all of a checkpoint while every health signal stayed green. Seven controlled experiments (same
client version, one fresh TRANSACTION topic per run) pin the cause: same-address commits succeed
4/4, cross-pod commits are answered
code=0 SUCCESSbut land nothing 2/2, and a same-address commitdelayed 45s is discarded the same way (the 30s TTL). The fix records the address the writer actually
sent to on the
SendCommittableright after the send, and the committer uses it, falling back toroute resolution only for restored committables without an address.
SendCommittableSerializermoves to version 2 and still reads version 1 state.
2. Poison records and the remaining dead/misleading options. A record whose deserialization
throws used to fail the task, which restored from the checkpoint, re-read the same record and
rewrote everything it had already emitted each round. It is now dropped with a WARN carrying
message id, topic, queue and offset, while the split offset advances past it; failures raised while
emitting an already-deserialized record still fail the task, so a real write failure cannot hide
behind the poison path.
send.pending.maxwas declared with exactly this description but neverread; the writer now bounds in-flight async sends with a semaphore of that size (and releases the
permit when the send call throws synchronously). The SQL option
retry.timesmapped onto a keynobody read; it now acts as an alias for
send.retry.times.InnerConsumerImpl.commitOffsetupdated a local store with
increaseOnly=trueand never persisted, which had no effect at all; itnow publishes the checkpointed offset, rollbacks included, so the broker-side display offset tracks
the job.
3. Transactional commits were submitted on a bounded executor that rejects under backlog.
With 10k records queued,
Commit transaction messagewas logged 8055 times instead of 10000 and694
RejectedExecutionExceptions appeared; the sink stalled at 9650/10000 while the source hadalready read everything. Commit and rollback now run on the calling thread.
4. An idle source spun the fetch loop a hundred times a second.
rocketmq.client.poll.timeoutdefaulted to 10ms. The client poll waits on a blocking queue and returns as soon as records arrive,
so raising the default to 1000ms costs nothing on the data path; it only cuts the idle spin to one
fetch a second.
wakeUp()merely sets a flag and the consumer deliberately has no interruptiblepoll, so this timeout is the sole bound on how long a fetch can outlive a close request — one
second keeps that prompt. A test asserts both the default and a configured value reach
poll().The docs also record the deployment constraint that follows from topic 1: the checkpoint interval
must stay well below the server-side transaction state TTL, because Flink only commits at the end of
a checkpoint.
Server-side note (not fixable here)
When the transaction context is missing or expired,
AbstractRemotingActivitymapsTRANSACTION_DATA_NOT_FOUNDtoResponseCode.SUCCESSandTransactionActivitydiscards thereturned future, so the client sees success while nothing lands and nothing is logged. Topic 1
removes the wrong-node cause on the client side; the TTL cause remains a deployment constraint
(keep the checkpoint interval well below
transactionDataExpireMillis) until the server side isaddressed separately.
Verification
parallelism 1 and 2 (missing 0 / unexpected 0 / duplicate 0), and the route-refresh fallback
warning drops from 4 to 0.
compatibility, the writer recording the address, and the committer using it.
mvn spotless:checkunder JDK 11 and a fullmvn clean packageof all five modules pass.Happy to split this into per-topic PRs if maintainers prefer; the commits are already separated by
topic.