Skip to content

[ISSUE #141] fix: commit transactions on the node holding the context and close the remaining silent paths - #149

Merged
lizhimins merged 7 commits into
apache:mainfrom
lizhimins:fix-tx-commit-address
Oct 27, 2026
Merged

lizhimins merged 7 commits into
apache:mainfrom
lizhimins:fix-tx-commit-address

Conversation

@lizhimins

Copy link
Copy Markdown
Member

Context

An acceptance run of this connector against a commercial RocketMQ 5.5.1 cluster on Flink 1.20.3
(Application mode, K8s) found a family of defects with the same shape as #141: the job stays
RUNNING, checkpoints succeed, nothing is logged, and the data is wrong or absent. This PR carries
the fixes from that round that are still open, grouped into four topics. Each one was verified on
the cluster with clean-room topics before being committed; the per-defect root causes and code
evidence are in the acceptance report kept outside this repo.

What is in this PR

1. Transactional commits are lost when END_TRANSACTION lands on another node (the headline fix).
A proxy keeps the transaction context of a half message in its own process memory
(TransactionDataManager, keyed by producer group + transaction id, expiring after
transactionDataExpireMillis, 30s by default). The committer used to resolve the target address
from its own publish route table, which is populated independently of the writer's and may name a
different node; with two committer subtasks each rolling its own dice, runs silently lost half or
all of a checkpoint while every health signal stayed green. Seven controlled experiments (same
client version, one fresh TRANSACTION topic per run) pin the cause: same-address commits succeed
4/4, cross-pod commits are answered code=0 SUCCESS but land nothing 2/2, and a same-address commit
delayed 45s is discarded the same way (the 30s TTL). The fix records the address the writer actually
sent to on the SendCommittable right after the send, and the committer uses it, falling back to
route resolution only for restored committables without an address. SendCommittableSerializer
moves to version 2 and still reads version 1 state.

2. Poison records and the remaining dead/misleading options. A record whose deserialization
throws used to fail the task, which restored from the checkpoint, re-read the same record and
rewrote everything it had already emitted each round. It is now dropped with a WARN carrying
message id, topic, queue and offset, while the split offset advances past it; failures raised while
emitting an already-deserialized record still fail the task, so a real write failure cannot hide
behind the poison path. send.pending.max was declared with exactly this description but never
read; the writer now bounds in-flight async sends with a semaphore of that size (and releases the
permit when the send call throws synchronously). The SQL option retry.times mapped onto a key
nobody read; it now acts as an alias for send.retry.times. InnerConsumerImpl.commitOffset
updated a local store with increaseOnly=true and never persisted, which had no effect at all; it
now publishes the checkpointed offset, rollbacks included, so the broker-side display offset tracks
the job.

3. Transactional commits were submitted on a bounded executor that rejects under backlog.
With 10k records queued, Commit transaction message was logged 8055 times instead of 10000 and
694 RejectedExecutionExceptions appeared; the sink stalled at 9650/10000 while the source had
already read everything. Commit and rollback now run on the calling thread.

4. An idle source spun the fetch loop a hundred times a second. rocketmq.client.poll.timeout
defaulted to 10ms. The client poll waits on a blocking queue and returns as soon as records arrive,
so raising the default to 1000ms costs nothing on the data path; it only cuts the idle spin to one
fetch a second. wakeUp() merely sets a flag and the consumer deliberately has no interruptible
poll, so this timeout is the sole bound on how long a fetch can outlive a close request — one
second keeps that prompt. A test asserts both the default and a configured value reach poll().
The docs also record the deployment constraint that follows from topic 1: the checkpoint interval
must stay well below the server-side transaction state TTL, because Flink only commits at the end of
a checkpoint.

Server-side note (not fixable here)

When the transaction context is missing or expired, AbstractRemotingActivity maps
TRANSACTION_DATA_NOT_FOUND to ResponseCode.SUCCESS and TransactionActivity discards the
returned future, so the client sees success while nothing lands and nothing is logged. Topic 1
removes the wrong-node cause on the client side; the TTL cause remains a deployment constraint
(keep the checkpoint interval well below transactionDataExpireMillis) until the server side is
addressed separately.

Verification

  • Clean-room cluster runs on the commercial 5.5.1 deployment: transactional sink lands 101/101 at
    parallelism 1 and 2 (missing 0 / unexpected 0 / duplicate 0), and the route-refresh fallback
    warning drops from 4 to 0.
  • Module test suite: 138 tests green, including 4 new ones for the serializer round trip and v1
    compatibility, the writer recording the address, and the committer using it.
  • mvn spotless:check under JDK 11 and a full mvn clean package of all five modules pass.

Happy to split this into per-topic PRs if maintainers prefer; the commits are already separated by
topic.

…sage

A proxy keeps the transaction context of a half message in its own process
memory, keyed by producer group and transaction id, and expires it after a
short TTL. END_TRANSACTION sent to any other node finds no context, and the
server answers that miss with a success code while discarding the request,
so the commit is lost with every health signal green.

The committer used to resolve the target address from its own publish route
table, which is populated independently of the writer's and may name a
different node; with two committer subtasks each rolling its own dice, runs
silently lost half or all of a checkpoint.

Record the address the writer actually sent to on the SendCommittable right
after the transactional send, and have the committer use it, falling back to
route resolution only when a restored committable carries no address. The
serializer moves to version 2 and still reads version 1 state.

Verified on a commercial 5.5.1 cluster with clean-room TRANSACTION topics:
parallelism 1 and 2 both land 101/101 with zero missing, unexpected or
duplicate records, and the route-refresh fallback warning drops from 4 to 0.
…ad parameters

A message whose deserialization throws used to fail the task, which restored
from the checkpoint, re-read the same record and failed again, rewriting
everything it had already emitted each round. Drop the record instead, log a
WARN carrying message id, topic, queue and offset, and advance the split
offset past it. Failures raised while emitting an already-deserialized record
(the downstream sink runs inline inside collect) are still propagated, so a
real write failure cannot hide behind the poison-message path.

Also close the two remaining misleading sink parameters: send.pending.max was
written into the configuration but never read, so the send buffer grew without
limit; the writer now bounds in-flight async sends with a semaphore of that
size. The SQL option retry.times mapped onto a key nobody read; it now acts as
an alias for send.retry.times (new key wins, default 3 when neither is set),
and the dead pass-through plus its unused field are gone.

Finally, InnerConsumerImpl.commitOffset updated only the local offset store
with increaseOnly=true and never persisted, which had no effect at all; it now
publishes the checkpointed offset (rollbacks included) so the broker-side,
display-only offset tracks the job instead of staying stale.
A synchronously throwing producer.send never reaches the whenComplete that
returns the semaphore permit and decrements the pending count, which would
wedge the writer at the send.pending.max limit after a single such failure.
…nded executor

Committing ran each END_TRANSACTION on the producer's callback executor, which
is bounded to four threads and a 2000-task queue with an abort policy. A
checkpoint carrying more committables than that rejected the surplus with
RejectedExecutionException; the committer treats such a failure as retryable,
and a retry after the server-side transaction context expired is silently
dropped, so a large checkpoint lost records with the job staying green.

End the transaction on the thread that calls commit/rollback. The committer
blocks until every request of the checkpoint is done anyway, so a burst now
shows up as checkpoint duration instead of uncommitted half messages.
…times a second

rocketmq.client.poll.timeout defaulted to 10ms, so a source with nothing to read
re-entered fetch() a hundred times a second per subtask. The client poll waits on
a blocking queue and returns as soon as records arrive, so raising the bound
shortens nothing on the data path; it only cuts the idle spin, to one fetch a
second.

wakeUp() merely sets a flag and the consumer deliberately has no interruptible
poll, which leaves this timeout as the sole bound on how long a fetch can outlive
a close request. One second keeps a shutdown prompt.

Adds a package-private constructor taking the InnerConsumer so the timeout that
actually reaches poll() is covered by a test.
…ate TTL

A proxy drops the transaction state of a half message after
transactionDataExpireMillis, 30 seconds by default, and Flink only commits at the
end of a checkpoint, so a long interval loses commits that the server still
answers with a success code. Also list rocketmq.client.poll.timeout among the
client options.
The acceptance round was formatted under JDK 21 where spotless silently
skips; under the JDK 11 the project gates on, ten files violate the
import order. Reformat them, no behaviour change.
@lizhimins
lizhimins merged commit 7170770 into apache:main Oct 27, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant