-
Notifications
You must be signed in to change notification settings - Fork 20
Description
Dear Circle-Map team,
I'm using Circle-Map Realign with 100 CPUs, and it's getting slower and slower.
At the beginning, the logs shows something like:
0%| | 0/10000 [00:00<?, ?it/s]
^M0it [00:00, ?it/s]ESC[A^M 0%| | 1/10000 [00:02<7:12:41, 2.60s/it]
^M1it [00:02, 2.59s/it]ESC[A^M 0%| | 2/10000 [00:03<4:30:06, 1.62s/it]
^M2it [00:03, 1.62s/it]ESC[A^M 0%| | 3/10000 [00:03<2:34:33, 1.08it/s]
^M3it [00:03, 1.08it/s]ESC[A^M 0%| | 5/10000 [00:03<1:24:09, 1.98it/s]
^M5it [00:03, 1.98it/s]ESC[A^M 0%| | 7/10000 [00:04<53:09, 3.13it/s]
^M7it [00:04, 3.13it/s]ESC[A^M 0%| | 9/10000 [00:04<41:19, 4.03it/s]
^M9it [00:04, 4.03it/s]ESC[A^M 0%| | 10/10000 [00:04<40:13, 4.14it/s]
^M10it [00:04, 4.13it/s]ESC[A^M 0%| | 13/10000 [00:04<23:52, 6.97it/s]
,which looks normal. However, in the end, it takes hours to output a line:
M9991it [159:47:37, 31133.06s/it]ESC[A^M100%|█████████▉| 9992/10000 [161:26:49<52:23:50, 23578.76s/it]
^M9992it [161:26:49, 23578.76s/it]ESC[A^M100%|█████████▉| 9993/10000 [165:55:02<41:28:51, 21333.14s/it]
^M9993it [165:55:03, 21333.13s/it]ESC[A^M100%|█████████▉| 9994/10000 [187:21:24<63:28:46, 38087.69s/it]
^M9994it [187:21:24, 38087.71s/it]ESC[A^M100%|█████████▉| 9995/10000 [187:50:09<37:44:54, 27178.85s/it]
^M9995it [187:50:09, 27178.82s/it]ESC[A^M100%|█████████▉| 9996/10000 [188:35:33<22:02:50, 19842.54s/it]
^M9996it [188:35:34, 19842.54s/it]ESC[A^M100%|█████████▉| 9997/10000 [209:51:56<30:43:13, 36864.48s/it]
^M9997it [209:51:56, 36864.49s/it]ESC[
It have been taken several days from 9980 to 9997, and stayed at 9997 for more than one day.
I wonder why this happens and what can I do now. Can I just kill the process and use the results in the temp_files?
Thanks!