Repository navigation
Conversation
…and target Flink 2.2.1 Flink 2.x removed SourceFunction, SinkFunction, TwoPhaseCommittingSink, TableSchema and the ConfigOption-typed Configuration accessors. The connector still ships the legacy implementations next to the FLIP-27 / sink2 ones, so it cannot compile against 2.x and every fix lands twice. Following the maintenance model of apache/flink-connector-kafka (new API only, zero legacy, version differences isolated in branches): - Delete the legacy source/sink paths and their tests; the FLIP-27 Source and the sink2 Sink + SupportsCommitter become the only runtime paths. The SQL factory keeps accepting the existing DDL options but resolves them to the new implementations. - TableSchema / SchemaValidator -> ResolvedSchema; Configuration getXxx(ConfigOption) / setXxx -> get / set. - flink.version 1.20.3 -> 2.2.1, Java 17 bytecode, shade 3.6.0 (the ASM bundled by flink-connector-parent cannot read major-61 class files); the CI matrix drops JDK 11 accordingly. dependency:analyze keeps reporting but no longer fails the build: upstream CI only ever ran `package`, so the verify-phase gate's declaration list is stale, and 2.x additionally splits flink-core / flink-runtime / flink-annotations / flink-core-api into separate artifacts. - The async-send bounding and seek-retry behavior of apache#154 is preserved. Semantics are unchanged: at-least-once / exactly-once guarantees, offset management from Flink state, checkpoint-aligned transaction commits and the apache#154 permit accounting all stay as they are; this is an API migration, not a behavior change. Verified: mvn clean install green on JDK 17 (remoting 146 / grpc 80 unit tests, checkstyle and spotless clean), both SQL fat-jars produced. Follow-up proposed in apache#155: cut a 1.x maintenance branch from the pre-migration main and apply the same new-API-only migration against 1.20.3 (verified green with the same tests; the two lines differ by 5 files / ~124 lines), then propagate fixes main -> branch by cherry-pick, as Kafka does.
Member
Author
|
Restructuring per the branch-point discussion in #155: this combined commit would be squash-merged into one, leaving no branch point for the 1.x maintenance line. Replaced by a migration-only PR (new API, still 1.20.3) that serves as the branch point; the switch to Flink 2.2.1 follows as a separate single-commit PR once the maintenance branch is cut. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Migrate the connector to the new Source/Sink API only and target Flink 2.2.1 / Java 17 on
main, following the maintenance model of apache/flink-connector-kafka (new API only, zero legacy, version differences isolated in branches). Tracking issue: #155.Sourceand the sink2Sink+SupportsCommitterbecome the only runtime paths. The SQL factory keeps accepting the existing DDL options but resolves them to the new implementations.TableSchema/SchemaValidator->ResolvedSchema;Configuration.getXxx(ConfigOption)/setXxx->get/set.flink.version1.20.3 -> 2.2.1, Java 17 bytecode, shade 3.6.0 (the ASM bundled byflink-connector-parentcannot read major-61 class files); CI matrix drops JDK 11 accordingly.dependency:analyzekeeps reporting but no longer fails the build: CI only ever ranpackage, so the verify-phase gate's declaration list is stale, and 2.x additionally splitsflink-core/flink-runtime/flink-annotations/flink-core-apiinto separate artifacts.Why
Flink 2.x removed
SourceFunction,SinkFunction,TwoPhaseCommittingSink,TableSchemaand theConfigOption-typedConfigurationaccessors. With legacy and new implementations shipped side by side, the connector cannot compile against 2.x and every fix lands twice (e.g. #141 / #148). Kafka and Pulsar both keep every branch on the new API only and isolate the Flink version in branches; this PR bringsmainonto that basis.Semantics
Unchanged. At-least-once / exactly-once guarantees, offset management from Flink state, checkpoint-aligned transaction commits and the #154 async-send bounding / seek-retry behavior all stay as they are — this is an API migration, not a behavior change. The proxy-fronted EXACTLY_ONCE limitation documented in #152 still applies.
Verification
mvn clean installgreen on JDK 17: remoting 146 / grpc 80 unit tests, checkstyle / spotless zero violations, both remoting and grpc SQL fat-jars produced.Sink.InitContext/createSerializer(ExecutionConfig)bridges that 1.20 still declares abstract (the 1.20 runtime calls theWriterInitContext/SerializerConfigoverloads, so the main code path is identical), and one test package path.Follow-up (proposed in #155)
Cut a 1.x maintenance branch from the pre-migration
mainand land the 1.20.3 variant of this migration there, then propagate fixesmain-> branch by cherry-pick, as Kafka does.