Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve KafkaReadStream control #75

Open
vietj opened this issue Oct 12, 2017 · 2 comments
Open

Improve KafkaReadStream control #75

vietj opened this issue Oct 12, 2017 · 2 comments

Comments

@vietj
Copy link
Contributor

vietj commented Oct 12, 2017

Currently the KafkaReadStream delivers item by batch of 10 (this should be configurable btw). When doing seek operations, the stream needs to be paused otherwise the handler will likely receive unwanted records that have been fetch and are delivered by the batch. This seems due to the fact that the current iterator is accessed from the worker thread only, unlike the paused flag.

One way to achieve it might be to pause() the consumer in seek operations. Alternatively the current iterator could be modified outside of the worker thread and checked when delivering items. Setting it to null would mean that a seek operation was required.

@vietj
Copy link
Contributor Author

vietj commented Oct 20, 2017

@ppatierno what's your opinion on this ?

@anhldbk
Copy link
Contributor

anhldbk commented Nov 19, 2018

@vietj Is pause a good way to handle slow consumers? Under a certain case, invoking pause will trigger operations on revoking/assigning paritions, right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

2 participants