You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the KafkaReadStream delivers item by batch of 10 (this should be configurable btw). When doing seek operations, the stream needs to be paused otherwise the handler will likely receive unwanted records that have been fetch and are delivered by the batch. This seems due to the fact that the current iterator is accessed from the worker thread only, unlike the paused flag.
One way to achieve it might be to pause() the consumer in seek operations. Alternatively the current iterator could be modified outside of the worker thread and checked when delivering items. Setting it to null would mean that a seek operation was required.
The text was updated successfully, but these errors were encountered:
@vietj Is pause a good way to handle slow consumers? Under a certain case, invoking pause will trigger operations on revoking/assigning paritions, right?
Currently the
KafkaReadStream
delivers item by batch of10
(this should be configurable btw). When doing seek operations, the stream needs to be paused otherwise the handler will likely receive unwanted records that have been fetch and are delivered by the batch. This seems due to the fact that thecurrent
iterator is accessed from the worker thread only, unlike thepaused
flag.One way to achieve it might be to pause() the consumer in seek operations. Alternatively the
current
iterator could be modified outside of the worker thread and checked when delivering items. Setting it to null would mean that a seek operation was required.The text was updated successfully, but these errors were encountered: