[FLINK-37298] Added Pluggable Components for BatchStrategy & BufferWrapper in AsyncSinkWriter.

[Poorvankbhatia]

What is the purpose of the change

FLIP-509

• The current AsyncSinkWriter handles batching using fixed size constraints (maxBatchSize & maxBatchSizeInBytes). However, this approach lacks flexibility for more complex batching strategies like grouping records by partition key (important for sinks like Cassandra).
• This change introduces a pluggable batching mechanism where sink implementers can define custom batching logic instead of relying on fixed queue-based buffering.
• It also decouples the buffer implementation by replacing the direct use of Deque<RequestEntryWrapper> with a pluggable buffer wrapper, allowing for alternative data structures that optimize lookup, prioritization, or performance.

Brief change log

• Added BufferWrapper interface to allow pluggable buffering strategies instead of a fixed Deque.
• Implemented DequeBufferWrapper as the default buffer wrapper using ArrayDeque.
• Introduced BatchCreator interface to enable custom batching logic.
• Added SimpleBatchCreator as the default batch creator, replicating existing batch formation logic.
• Created BatchCreationResult class to encapsulate batch metadata (entries, size, record count).
• Modified AsyncSinkWriter to incorporate both the pluggable components.

Verifying this change

• Added tests for BufferWrapper and its default implementation (DequeBufferWrapper).
• Added tests for BatchCreator to validate the default batching behavior.
• Ensured batched requests are correctly formed and processed using the new interfaces.
• Verified that the default implementations (SimpleBatchCreator and DequeBufferWrapper) maintain existing behavior without requiring changes from users.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): yes
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? yes
• If yes, how is the feature documented? JavaDocs

[flinkbot]

CI report:

• f3c3386 Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[vahmed-hamdy]

Overall looks good, thanks for delivering this. Just a couple of questions and nits.

[vahmed-hamdy]

Looks Good, thanks for the huge efforts

[davidradl]

could we have the docs change to publicise this change and advice on supplying pluggable strategies and BufferWrapper.

[Poorvankbhatia]

could we have the docs change to publicise this change and advice on supplying pluggable strategies and BufferWrapper.

Thanks for the suggestion! The JavaDocs for BatchCreator and BufferWrapper already document their purpose and usage.

Regarding additional documentation changes since there's an ongoing vote on this FLIP, would it make sense to wait until finalized and then follow up with documentation updates in a separate PR?

Let me know what you think—happy to adjust based on the preferred approach! :)

[Poorvankbhatia]

Hey @dannycranmer can you help with this review?