Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate how Aries-VCR is performing paging when querying Solr #757

Open
WadeBarnes opened this issue Oct 6, 2023 · 3 comments
Open

Comments

@WadeBarnes
Copy link
Member

WadeBarnes commented Oct 6, 2023

As discussed some of the articles listed here certain workarounds for paging can be problematic for Solr. Particular search queries seem to have an effect on the search-engine container (Solr), causing it to use a significant amount of memory.

Example queries (note the high page numbers):

/api/v4/search/topic?format=json&ordering=-revoked_date&page=149713
/api/v4/search/topic?format=api&ordering=revoked_date&page=2721
/api/v4/search/credential/facets?format=api&page=318768
/api/v4/search/credential?ordering=effective_date&page=318772

This issue is related to the investigation happening here; bcgov/orgbook-configurations#137

@WadeBarnes
Copy link
Member Author

Try this first; #758

@WadeBarnes
Copy link
Member Author

WadeBarnes commented Oct 16, 2023

Disabling the Solr caches has been ineffective; #758 (comment), #758

I think it's time we get together to discuss next steps on how to proceed. A good start would be to assign someone to review, and then we can go over the tools we have for monitoring and what we've discovered about the issue so far.

@WadeBarnes
Copy link
Member Author

The issues we were seeing with Solr were caused by unbound queries (#762) and likely made worse by the paging implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant