Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Send documents by batch instead of one by one #186

Open
brunoocasali opened this issue Jul 23, 2024 · 1 comment
Open

Send documents by batch instead of one by one #186

brunoocasali opened this issue Jul 23, 2024 · 1 comment
Labels
enhancement New feature or request

Comments

@brunoocasali
Copy link
Member

Description
Basically, instead of pushing documents one by one, let's send them as a batch so Meilisearch can process them in batches, increasing the performance and the overall indexation time.

@brunoocasali brunoocasali added the enhancement New feature or request label Jul 23, 2024
@curquiza
Copy link
Member

⚠️ I want to add a warning here: we don't want to push too big batches to avoid memory crash issues by sending too big batches. I think 10 documents at the same time would be a good start. I would decrease by 10 the task queue without impacting too much the RAM, even if the documents are big.

A second iteration could be to add an option in this library to set up the size of the batch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants