p_map() very slow compared to multiprocess.Pool.map()

[FlorinAndrei]

I'm trying to accelerate Pandas df.apply(), and also get a progress bar. The problem is, p_map is orders of magnitude slower than plain multiprocess.Pool.map() for a job where most of the processing is done by nltk.sentiment.vader.SentimentIntensityAnalyzer().

This notebook is self-explanatory:

https://github.com/FlorinAndrei/misc/blob/master/p_tqdm_bug_1.ipynb

p_map() is orders of magnitude slower.

However, the same function seems to work fine, fast enough, for another task - reading 25k files off the disk.

Windows 10, Python 3.8.8, Jupyter Notebook

[nuttyartist]

From my testing, it seems like tqdm is the culprit. If I use tqdm on regular multiprocessing.Pool() it slows it down significantly. Did someone else experience this?

[AeroTH310]

I have experienced this also. It appears that the pool is actually processing serially... I see many processes getting started in my system monitor according to the number of cores I set. Only one of these processes seem to be doing anything at any time though.

[BenjaminHoegh]

Also seems to very slow compared to joblib's parallel

[nphilou]

Same issue here so far but it seems that the issue is more on the pathos side.

Tests here: https://gist.github.com/nphilou/1296ebabc7b4de24f57de9452e25405e

Further testing on my side showed that pathos parallelization is very sensible to the number of operations.

If my workload consists of numerous short tasks, using pathos may lead to significantly slow performance. However, if I do a smaller number of computationally intensive tasks, the execution time with pathos might be comparable to using multiprocessing.Pool.

This is very high-level observations, I didn't dive further on the low-level aspect