Skip to content

[ISSUE #155] refactor: migrate to the new Source/Sink API only and target Flink 2.2.1 - #156

Closed
lizhimins wants to merge 1 commit into
apache:mainfrom
lizhimins:issue-155-flink-2x
Closed

lizhimins wants to merge 1 commit into
apache:mainfrom
lizhimins:issue-155-flink-2x

Conversation

@lizhimins

Copy link
Copy Markdown
Member

What

Migrate the connector to the new Source/Sink API only and target Flink 2.2.1 / Java 17 on main, following the maintenance model of apache/flink-connector-kafka (new API only, zero legacy, version differences isolated in branches). Tracking issue: #155.

  • Delete the legacy source/sink paths and their tests; the FLIP-27 Source and the sink2 Sink + SupportsCommitter become the only runtime paths. The SQL factory keeps accepting the existing DDL options but resolves them to the new implementations.
  • TableSchema / SchemaValidator -> ResolvedSchema; Configuration.getXxx(ConfigOption) / setXxx -> get / set.
  • flink.version 1.20.3 -> 2.2.1, Java 17 bytecode, shade 3.6.0 (the ASM bundled by flink-connector-parent cannot read major-61 class files); CI matrix drops JDK 11 accordingly.
  • dependency:analyze keeps reporting but no longer fails the build: CI only ever ran package, so the verify-phase gate's declaration list is stale, and 2.x additionally splits flink-core / flink-runtime / flink-annotations / flink-core-api into separate artifacts.

Why

Flink 2.x removed SourceFunction, SinkFunction, TwoPhaseCommittingSink, TableSchema and the ConfigOption-typed Configuration accessors. With legacy and new implementations shipped side by side, the connector cannot compile against 2.x and every fix lands twice (e.g. #141 / #148). Kafka and Pulsar both keep every branch on the new API only and isolate the Flink version in branches; this PR brings main onto that basis.

Semantics

Unchanged. At-least-once / exactly-once guarantees, offset management from Flink state, checkpoint-aligned transaction commits and the #154 async-send bounding / seek-retry behavior all stay as they are — this is an API migration, not a behavior change. The proxy-fronted EXACTLY_ONCE limitation documented in #152 still applies.

Verification

  • mvn clean install green on JDK 17: remoting 146 / grpc 80 unit tests, checkstyle / spotless zero violations, both remoting and grpc SQL fat-jars produced.
  • The same migration was verified against Flink 1.20.3 (Java 11 bytecode) with the same tests passing. The diff between the two lines is 5 files / ~124 lines: pom versions, the Sink.InitContext / createSerializer(ExecutionConfig) bridges that 1.20 still declares abstract (the 1.20 runtime calls the WriterInitContext / SerializerConfig overloads, so the main code path is identical), and one test package path.

Follow-up (proposed in #155)

Cut a 1.x maintenance branch from the pre-migration main and land the 1.20.3 variant of this migration there, then propagate fixes main -> branch by cherry-pick, as Kafka does.

…and target Flink 2.2.1

Flink 2.x removed SourceFunction, SinkFunction, TwoPhaseCommittingSink,
TableSchema and the ConfigOption-typed Configuration accessors. The
connector still ships the legacy implementations next to the FLIP-27 /
sink2 ones, so it cannot compile against 2.x and every fix lands twice.

Following the maintenance model of apache/flink-connector-kafka (new API
only, zero legacy, version differences isolated in branches):

- Delete the legacy source/sink paths and their tests; the FLIP-27 Source
  and the sink2 Sink + SupportsCommitter become the only runtime paths.
  The SQL factory keeps accepting the existing DDL options but resolves
  them to the new implementations.
- TableSchema / SchemaValidator -> ResolvedSchema; Configuration
  getXxx(ConfigOption) / setXxx -> get / set.
- flink.version 1.20.3 -> 2.2.1, Java 17 bytecode, shade 3.6.0 (the ASM
  bundled by flink-connector-parent cannot read major-61 class files);
  the CI matrix drops JDK 11 accordingly. dependency:analyze keeps
  reporting but no longer fails the build: upstream CI only ever ran
  `package`, so the verify-phase gate's declaration list is stale, and
  2.x additionally splits flink-core / flink-runtime / flink-annotations
  / flink-core-api into separate artifacts.
- The async-send bounding and seek-retry behavior of apache#154 is preserved.

Semantics are unchanged: at-least-once / exactly-once guarantees, offset
management from Flink state, checkpoint-aligned transaction commits and
the apache#154 permit accounting all stay as they are; this is an API
migration, not a behavior change.

Verified: mvn clean install green on JDK 17 (remoting 146 / grpc 80 unit
tests, checkstyle and spotless clean), both SQL fat-jars produced.

Follow-up proposed in apache#155: cut a 1.x maintenance branch from the
pre-migration main and apply the same new-API-only migration against
1.20.3 (verified green with the same tests; the two lines differ by 5
files / ~124 lines), then propagate fixes main -> branch by cherry-pick,
as Kafka does.
@lizhimins

Copy link
Copy Markdown
Member Author

Restructuring per the branch-point discussion in #155: this combined commit would be squash-merged into one, leaving no branch point for the 1.x maintenance line. Replaced by a migration-only PR (new API, still 1.20.3) that serves as the branch point; the switch to Flink 2.2.1 follows as a separate single-commit PR once the maintenance branch is cut.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant