Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CSV Exporter] Update prepareMeasureSearch to Support search_after #349

Open
Juiced66 opened this issue Jul 5, 2024 · 0 comments
Open
Assignees
Labels
enhancement New feature or request feature-request

Comments

@Juiced66
Copy link
Contributor

Juiced66 commented Jul 5, 2024

Description

Refactor the prepareMeasureSearch method to support pagination using the search_after parameter.
This intend to bypass Elastic Search 10K row limitations by using multiple requests

Tasks:

  • Ensure proper sorting by measuredAt and _id.
  • Return sorting criteria along with the search query.
  • Update sendExport method to use search_after for pagination.

Example Code:

const sort = [
  { measuredAt: "asc" },
  { _id: "asc" } // Ensuring unique sorting by adding _id
];

return { digitalTwin, searchQuery, sort };

Pros:

  • Efficiently handles large datasets by paginating results.
  • Avoids Elasticsearch limitations on result size.

Cons:

  • Requires proper handling of search_after logic.
  • Increases complexity of the search function.

Possible choices to made :

  • Use PIT for data consistency but larger impact on ES that have to maintain data state for the search duration
  • Do not use PIT as we export past data that should not be reordered between queries (with proper tie breaker on _ids)
@Juiced66 Juiced66 self-assigned this Jul 5, 2024
@Juiced66 Juiced66 changed the title [CSV Export] Update prepareMeasureSearch to Support search_after [CSV Exporter] Update prepareMeasureSearch to Support search_after Jul 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request feature-request
Projects
None yet
Development

No branches or pull requests

1 participant