Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BulkIngester should allow customizations of intermediate requests #703

Open
fabriziofortino opened this issue Nov 7, 2023 · 1 comment
Labels
Category: Enhancement New feature or request

Comments

@fabriziofortino
Copy link
Contributor

Description

Using the HLRC BulkProcessor, it became feasible to intercept the creation of requests by utilizing a BiConsumer that was invoked every time a bulk request was constructed (refer to https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/action/bulk/BulkProcessor.java#L213).

This capability proved particularly advantageous in various situations. For instance, if we consider a scenario where there are 1000 operations divided into 10 bulk requests, I could intercept the final request and modify the request refresh policy (for example, request.setRefreshPolicy(WriteRequest.RefreshPolicy.WAIT_UNTIL);).

Having a similar feature in the BulkIngester would be highly beneficial. One straightforward approach could involve adjusting the visibility of BulkIngester.newRequest() (refer to https://github.com/elastic/elasticsearch-java/blob/main/java-client/src/main/java/co/elastic/clients/elasticsearch/_helpers/bulk/BulkIngester.java#L243)

@amitaiporat
Copy link

@swallez do you know if this is something that will be addressed in a near term release?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Category: Enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants