-
|
Hi there, I've been trying to get an impression of the performance of using the shm provider and how can this be improved. But I couldn't find much information around this. I currently see the values: I would have expected better values for bandwidth and latency but maybe I'm missing something here. Are these correct? Any pointers/ideas much appreciated. Thanks in advance. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 11 replies
-
|
That is very slow. Did you perhaps compile Libfabric with |
Beta Was this translation helpful? Give feedback.
-
|
Good point, thanks. Indeed debug was enabled, my bad. Things improve slightly without it and more iterations (1000). In particular, the latency of small messages improves considerably. Yet, for large messages, I get similar results. Any further reasons for this? Thank you. |
Beta Was this translation helpful? Give feedback.
-
|
Another thing to check is CPU contention caused by the server and client processes trying to use the same CPU core |
Beta Was this translation helpful? Give feedback.
-
|
Thanks. It doesn't seem to be the case. I could not observe any contention between CPUs. Also when setting the CPU affinity of both processes to different CPUs I see similar results. It must be something else. |
Beta Was this translation helpful? Give feedback.
-
|
Interestingly I tried the same on another system (actually older). There the results are better although nothing different was done. Still not sure if that's the expected but nevertheless much better than before. I also tried to dig a bit into the benchmark code but could not find much yet except that quite some time is spent waiting for the completion ( |
Beta Was this translation helpful? Give feedback.
-
|
@rumach Something definitely looks off there. Here is what it should look like: As @sunkuamzn said you could try using taskset -c 1 and taskset -c 2 to make sure you bind the server and client to separate cores and see if that helps but I doubt that's the issue. Could you provide more details on the type of system, OS, CPU, etc that you're using? |
Beta Was this translation helpful? Give feedback.
-
|
Here the output of and of And thank you for mentioning that about By the way, is there a difference between Anyways, I guess something is wrong with this system. Would be nice to understand why but I suppose it's getting a bit out of scope (?!). |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the information. I don't see anything glaring in the output that would indicate an issue. Can you try running a different provider? Let's say tcp? If you see a similar performance issue on the system with other providers then I would say it's probably a system issue and not a provider issue. If shm is the only provider showing a problem, then still could be worth looking into. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the explanation on the different tests. Sometimes is a challenge with libfabric to have a clear view on what is what and what is best for which provider... too many knobs :). Keeping the focus, this is what I observe with tcp on "problematic" node: as opposed to (on the other "older" node): I suppose this confirms the system issue, right? Both systems have Ethernet (1 Gbit) and IB cards (CX-3 in old, CX-6 in problematic). I wonder which one gets used when the using the loopback address? Sorry, I know this is a bit off-topic. I would have other questions :). I guess the best (community) way is to have a different discussion thread for each one? |
Beta Was this translation helpful? Give feedback.
-
|
@rumach Thanks for running tcp and sending those numbers! I know, there are a lot of knobs... the never-ending battle between tunability and user-friendliness :) |
Beta Was this translation helpful? Give feedback.
-
|
@aingerson Many thanks for your help. Totally understand (tuning vs. user-friendliness). Good that we have forums like this :) Indeed, a system issue. But was helpful to understand some things. But as this is not the my "critical path", I'll mark this as answered. If something interesting or important arises related to this I'll surely post it here. As mentioned, I have other questions but will open new question(s)/discussion(s). Thanks again! |
Beta Was this translation helpful? Give feedback.
@rumach Something definitely looks off there. Here is what it should look like: