fix: emit CONTROL config messages as a single atomic stdout write - #1090
Draft
devin-ai-integration[bot] wants to merge 1 commit into
Draft
fix: emit CONTROL config messages as a single atomic stdout write#1090devin-ai-integration[bot] wants to merge 1 commit into
devin-ai-integration[bot] wants to merge 1 commit into
Conversation
Co-Authored-By: bot_apk <apk@cognition.ai>
Contributor
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksTesting This CDK VersionYou can test this version of the CDK using the following: # Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1785392204-fix-control-message-line-atomic#egg=airbyte-python-cdk[dev]' --help
# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1785392204-fix-control-message-line-atomicPR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
|
PyTest Results (Full)4 144 tests 4 132 ✅ 12m 24s ⏱️ Results for commit f4b6b33. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
emit_configuration_as_airbyte_control_message()printed the CONTROL message with a bareprint(x). CPython'sprintwrites the payload and the trailing newline as two separatewrite()calls, so the emission is not line-atomic:Since #877,
SingleUseRefreshTokenOauth2Authenticator._emit_control_messagealways routes through this function so that rotated single-use refresh tokens are persisted immediately. In a concurrent declarative sync, token refreshes happen on the worker thread that made the HTTP request, while RECORD lines are printed from the main thread — which is line-atomic (entrypoint.launchdeliberately doesprint(f"{message}\n", end="")). A worker-thread CONTROL emission can therefore interleave between a main-thread RECORD write:The result is
<control><record>\n\n: the platform rejects the concatenated line asMalformed non-Airbyte recordand drops that RECORD, but the entrypoint'ssourceStats.recordCountalready counted it. Every subsequent state message then fails platform checksum validation with an off-by-one record count, so no state is committed at all and the sync ultimately fails.The fix makes the emission line-atomic by writing the payload and its newline in one call, mirroring the main read loop:
sys.stdout.writeis used rather thanprint(..., end="")because CPython'sprintstill issues a second (empty)write("")forend, which defeats a strict single-write guarantee.sys.stdoutis resolved at call time, so this is transparent to thePrintBufferredirection inentrypoint.launch.This preserves the immediate-persistence guarantee that PR 877 was added for — the CONTROL message is still written straight to stdout from the refreshing thread, just without the framing hazard.
ConfigObserver.update()benefits from the same fix.Resolves https://github.com/airbytehq/oncall/issues/13201:
Test Coverage
Both tests are in
unit_tests/test_config_observation.py:test_emit_configuration_as_airbyte_control_message_is_line_atomic— deterministic regression test. Monkeypatchessys.stdoutwith a stream that records everywrite()call and asserts exactly one write, ending in exactly one\n, parsing as aCONTROLmessage. This fails onmain(2 writes) and passes with the fix.test_emit_configuration_as_airbyte_control_message_concurrent_output_is_well_framed— end-to-end framing guard. Routes stdout through the realPrintBufferand runs one record-printing thread against four concurrent control-emitting threads (800 lines total), then asserts every emitted line is non-empty and valid JSON — i.e. no concatenated or split lines.ruff check,ruff format --check, andmypy --config-file mypy.ini airbyte_cdk(454 files) all pass.Declarative-First Evaluation
Not applicable in the connector sense — this is a fix in CDK core stdout emission, not a connector manifest change. The affected connector (manifest-only) needs no changes; no declarative component can influence how the CDK frames protocol messages on stdout.
Breaking Change Evaluation
Not breaking. No schema, spec, state, primary-key, cursor, or stream changes. The only observable difference is that the CONTROL message's newline is written together with its payload instead of in a follow-up write; the bytes on stdout are identical. No public interface changed.
Link to Devin session: https://app.devin.ai/sessions/ad48da8684fc45e29826c13cd2c85c8e