-
|
I have been running version 0.3.4 with the dataset directly on the ephemeral SSD that comes with the node. It's running on a 8CPU, 16GB machine and the performance is excellent. It works well enough but the response times really tanks compared to my old installation. On the new version i'm seeing frequent slowdowns due to GC and memory swapping: Both are running with Are anybody having similar experiences, and have you found a way to improve this? |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 27 replies
-
|
It's hard to track regressions when you change two things at once. I'd test with opensearch without structured data first. |
Beta Was this translation helpful? Give feedback.
-
|
I will. I have already tested with normal 0.6.2 and the same dataset, and the results were equally bad which is why I opted to try the opensearch version, but I'll retry that test and come back with results. |
Beta Was this translation helpful? Give feedback.
-
|
If this all boils down to garbage collection problems, you could test with generational ZGC shipped with Java 21: |
Beta Was this translation helpful? Give feedback.
-
|
Very interesting! I'm actually playing around with ShenandoahGC but this might be even better. I'll add that in my tests 👍 |
Beta Was this translation helpful? Give feedback.
-
|
We don't use any |
Beta Was this translation helpful? Give feedback.
-
|
@lonvia the default maximum heap size is 25% of the RAM if not specified. On a beefy machine, that could be quite a large value. java -XX:+PrintFlagsFinal -version | grep MaxHeapSizeyields |
Beta Was this translation helpful? Give feedback.
-
|
I've made another test, comparing 0.3.4 and 0.6.2. Below are the details.
VERSION 0.3.4Rerun 50 requests: Total duration: 8 seconds VERSION 0.6.2Rerun 50 requests: Total duration: 133 seconds Conclusion: I'm happy to answer any questions or repeat any tests you would like to see. I can also provide the script and requests that I've used to test with if you're interested. |
Beta Was this translation helpful? Give feedback.
-
|
Yes! I work with all these languages: ar,bg,ca,cs,cy,da,de,el,en,es,et,fi,fo,fr,ga,hr,hu,is,it,ja,ko,lt,lv,nb,nl,pl,pt,ro,ru,sk,sv,th,tr,vi,zh I start it like this: Some typical requests could be: |
Beta Was this translation helpful? Give feedback.
-
|
Ah, known issue then, see also discussion at #876. There never was much request for highly multilingual instances, so it never floated up as a highly urgent issue. The OpenSearch variant should be doing better in theory, I haven't tested it out in practise. Really should be doing that before finalizing the index structure. |
Beta Was this translation helpful? Give feedback.
-
|
@lonvia I assume you meant to link maybe this? #615 (sorry, long thread) which sounds like what I'm facing. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, #615. See also #538 on the background why we changed that. The |
Beta Was this translation helpful? Give feedback.
-
|
@lonvia Hi and thank for all the great work on photon. I see that #615 is more than 3.5y old and the idea of changing the index structure hasn't been implemented yet. As you say it havn't been a highly urgent issue and therefore no huge gain for most "normal" photon installations. To us it would make a huge difference (cost) not to run 32 different language set ups with high availability in multiple regions, which for now could mean 256 EC2 instances instead of just 8 EC2 instances. If you're open to implementing a solution like the one in #615, we'd like to contribute that? |
Beta Was this translation helpful? Give feedback.
-
|
As said in #615, I don't think that this is a good solution because it does not address the route cause. Photon should be fast to respond, no matter how many languages are loaded. That's what needs to be fixed. If you think that it is important to search only in one specific language, then you'd need to tell us a bit more about your use case. Right now I don't see a useful application for that. |
Beta Was this translation helpful? Give feedback.
-
|
I also completely agree that fixing the root cause would be awesome, but for an outsider it's not a trivial task and if it's not really an issue for a projects core developers/users it only causes risk to the entire project with such a huge change. Our use-case is quick simple. We have an app which shows a list/map of stores near a specific location. You can change location by dragging the map, or searching for a specific place. The app runs in a specific language (usually the device language) and we expect results to be returned in that language or "default/original" name of that place. We see it as an unlikely use-case that somebody need to search through more languages than default/original index, app language index (and potential languages used within the user's country). We've been running with only default/original index+app language index for more than 6 years, and haven't received a single feature-request for better multi-lingual search. As for the complexity/confusion for adding another "language" parameter, I don't really see it:
Thank you for your time |
Beta Was this translation helpful? Give feedback.
-
|
In theory, you should be able to run multiple instances of Photon, all against the same database but with different In practice, ES currently complains about concurrent access when running in embedded mode. This is probably the correct behaviour as embedded mode was really meant to work for a stand-alone instance. However, it should be working when using an external ES instance. And if it doesn't, I'd take a PR that helps making it work. |
Beta Was this translation helpful? Give feedback.
Yes, #615. See also #538 on the background why we changed that.
The
langparameter only adds a preference for the given language and influences the language of the results.