Madmax K34 Performance #996

bladeuserpi · 2022-02-20T12:38:22Z

bladeuserpi
Feb 20, 2022

Hello,

I would like to get feedback for expected K34 Performance:
Total plot creation time was 6749.71 sec (112.495 min)

I only checked ramdisk manually and max value I saw was about 1007GB.

Could the performance be improved further, it is only using 2568% CPU from 8000% available?

    1	Multi-threaded pipelined Chia k34 plotter - 2092041
     2	Network Port: 8444
     3	Final Directory: /plot/
     4	Number of Plots: 1
     5	Crafting plot 1 out of 1 (2022/02/19 23:27:23)
     6	Process ID: 8449
     7	Number of Threads: 80
     8	Number of Buckets P1:    2^8 (256)
     9	Number of Buckets P3+P4: 2^8 (256)
    10	Pool Puzzle Hash:  x
    11	Farmer Public Key: x
    12	Working Directory:   /ramdisk/
    13	Working Directory 2: /ramdisk/
    14	Plot Name: plot-k34-2022-02-19-23-27-x
    15	[P1] Table 1 took 21.5157 sec
    16	[P1] Table 2 took 273.704 sec, found 17179537808 matches
    17	[P1] Table 3 took 385.385 sec, found 17179175846 matches
    18	[P1] Table 4 took 417.257 sec, found 17178357339 matches
    19	[P1] Table 5 took 416.162 sec, found 17176916001 matches
    20	[P1] Table 6 took 374.045 sec, found 17173741705 matches
    21	[P1] Table 7 took 360.97 sec, found 17167515746 matches
    22	Phase 1 took 2249.05 sec
    23	[P2] max_table_size = 17179869184
    24	[P2] Table 7 scan took 25.5913 sec
    25	[P2] Table 7 rewrite took 147.017 sec, dropped 0 entries (0 %)
    26	[P2] Table 6 scan took 64.9572 sec
    27	[P2] Table 6 rewrite took 90.7885 sec, dropped 2325867920 entries (13.5432 %)
    28	[P2] Table 5 scan took 63.3569 sec
    29	[P2] Table 5 rewrite took 91.6393 sec, dropped 3048830625 entries (17.7496 %)
    30	[P2] Table 4 scan took 61.6832 sec
    31	[P2] Table 4 rewrite took 87.4706 sec, dropped 3316029493 entries (19.3035 %)
    32	[P2] Table 3 scan took 58.8058 sec
    33	[P2] Table 3 rewrite took 82.5131 sec, dropped 3420740003 entries (19.9121 %)
    34	[P2] Table 2 scan took 58.8942 sec
    35	[P2] Table 2 rewrite took 82.1456 sec, dropped 3462543688 entries (20.155 %)
    36	Phase 2 took 992.512 sec
    37	Wrote plot header with 252 bytes
    38	[P3-1] Table 2 took 569.302 sec, wrote 13716994120 right entries
    39	[P3-2] Table 2 took 110.386 sec, wrote 13716994120 left entries, 13716994120 final
    40	[P3-1] Table 3 took 384.764 sec, wrote 13758435843 right entries
    41	[P3-2] Table 3 took 115.912 sec, wrote 13758435843 left entries, 13758435843 final
    42	[P3-1] Table 4 took 381.511 sec, wrote 13862327846 right entries
    43	[P3-2] Table 4 took 116.546 sec, wrote 13862327846 left entries, 13862327846 final
    44	[P3-1] Table 5 took 393.774 sec, wrote 14128085376 right entries
    45	[P3-2] Table 5 took 119.581 sec, wrote 14128085376 left entries, 14128085376 final
    46	[P3-1] Table 6 took 394.467 sec, wrote 14847873785 right entries
    47	[P3-2] Table 6 took 122.81 sec, wrote 14847873785 left entries, 14847873785 final
    48	[P3-1] Table 7 took 449.14 sec, wrote 17167515746 right entries
    49	[P3-2] Table 7 took 141.725 sec, wrote 17167515746 left entries, 17167515746 final
    50	Phase 3 took 3319.81 sec, wrote 87481232716 entries to final plot
    51	[P4] Starting to write C1 and C3 tables
    52	[P4] Finished writing C1 and C3 tables
    53	[P4] Writing C2 table
    54	[P4] Finished writing C2 table
    55	Phase 4 took 188.307 sec, final plot size is 461382570726 bytes
    56	Total plot creation time was 6749.71 sec (112.495 min)
    57	Started copy to /plot/plot-k34-2022-02-19-23-27-x.plot
    58	Copy to /plot/plot-k34-2022-02-19-23-27-x.plot finished, took 369.715 sec, 1190.13 MB/s avg.
    59	170006.32user 12906.98system 1:58:40elapsed 2568%CPU (0avgtext+0avgdata 271484216maxresident)k

Machine info:
Architecture:        x86_64
CPU(s):              80
On-line CPU(s) list: 0-79
Thread(s) per core:  2
Core(s) per socket:  20
Socket(s):           2
NUMA node(s):        2
Model name:          Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz

For reference, other K-sizes on this machine:
K32 performance: Total plot creation time was 1292.62 sec (21.5437 min)
K33 performance: Total plot creation time was 4452.98 sec (74.2163 min)

brianwfreeman · 2022-02-27T04:36:14Z

brianwfreeman
Feb 27, 2022

You have at least 1TB of RAM to do this? Wow!

1 reply

bladeuserpi Mar 2, 2022
Author

Yes, as Xeon Gold 61xx has 6 memory channels, this has 1.5TB fully populated (24x 64GB LRDIMM, 2133MHz).
However it seems madmax performance is not improving by using full ramdisk (at least SSD endurance is saved),
see below for more performance analysis using top+"H".

brianwfreeman · 2022-02-27T04:36:55Z

brianwfreeman
Feb 27, 2022

I used 64 threads on a 64 core Threadripper today and the Intel Optane P5800X and got a 174 minute K=34 plot.

0 replies

brianwfreeman · 2022-02-27T20:22:31Z

brianwfreeman
Feb 27, 2022

Today I re-ran a K=34 plot using Linux Mint and the Intel Optane P5800X. Total Plot time in Linux was substantially faster -- total time 109 minutes.

0 replies

bladeuserpi · 2022-03-02T15:32:43Z

bladeuserpi
Mar 2, 2022
Author

There are large performance variations from run to run.
For my config the performance variations manifest mostly in P2 rewrite times
which can be much larger than below.

Another machine with Dual E5-2699v3 (2x18core) and 512GB RAM
completed in 108 minutes which is 4minutes faster than Xeon Gold machine
(and about the same performance as your Threadripper)

OS ist RHEL-8.5 for my tests.

Also note this used tmpdir on venerable P3600 SSD:
Node                  SN                   Model                                    Namespace Usage                      Format           FW Rev  
--------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme1n1          sn                   INTEL SSDPE2ME012T4                      1           1.20  TB /   1.20  TB      4 KiB +  0 B   8DV101H0

     1	Multi-threaded pipelined Chia k34 plotter - ecec17d
     2	(Sponsored by Flexpool.io - Check them out if you're looking for a secure and scalable Chia pool)
     3	
     4	Network Port: 8444 [chia]
     5	Final Directory: /plot06/
     6	Number of Plots: 1
     7	Crafting plot 1 out of 1 (2022/02/22 17:51:02)
     8	Process ID: 6674
     9	Number of Threads: 36
    10	Number of Buckets P1:    2^8 (256)
    11	Number of Buckets P3+P4: 2^8 (256)
    12	Pool Puzzle Hash:  p
    13	Farmer Public Key: p
    14	Working Directory:   /plot06/
    15	Working Directory 2: /ramdisk/
    16	Plot Name: plot-k34-2022-02-22-17-51-p
    17	[P1] Table 1 took 36.7124 sec
    18	[P1] Table 2 took 289.459 sec, found 17180029885 matches
    19	[P1] Table 3 took 444.283 sec, found 17180129970 matches
    20	[P1] Table 4 took 814.184 sec, found 17180415542 matches
    21	[P1] Table 5 took 508.546 sec, found 17181118681 matches
    22	[P1] Table 6 took 433.311 sec, found 17182387279 matches
    23	[P1] Table 7 took 415.19 sec, found 17185033637 matches
    24	Phase 1 took 2941.7 sec
    25	[P2] max_table_size = 17185033637
    26	[P2] Table 7 scan took 50.561 sec
    27	[P2] Table 7 rewrite took 196.34 sec, dropped 0 entries (0 %)
    28	[P2] Table 6 scan took 88.0299 sec
    29	[P2] Table 6 rewrite took 112.072 sec, dropped 2324681020 entries (13.5294 %)
    30	[P2] Table 5 scan took 82.5507 sec
    31	[P2] Table 5 rewrite took 124.643 sec, dropped 3047334499 entries (17.7365 %)
    32	[P2] Table 4 scan took 82.2928 sec
    33	[P2] Table 4 rewrite took 108.194 sec, dropped 3314911994 entries (19.2947 %)
    34	[P2] Table 3 scan took 82.0963 sec
    35	[P2] Table 3 rewrite took 115.239 sec, dropped 3419984976 entries (19.9066 %)
    36	[P2] Table 2 scan took 81.5556 sec
    37	[P2] Table 2 rewrite took 103.657 sec, dropped 3462054369 entries (20.1516 %)
    38	Phase 2 took 1310.63 sec
    39	Wrote plot header with 252 bytes
    40	[P3-1] Table 2 took 278.605 sec, wrote 13717975516 right entries
    41	[P3-2] Table 2 took 116.459 sec, wrote 13717975516 left entries, 13717975516 final
    42	[P3-1] Table 3 took 192.222 sec, wrote 13760144994 right entries
    43	[P3-2] Table 3 took 125.41 sec, wrote 13760144994 left entries, 13760144994 final
    44	[P3-1] Table 4 took 201.536 sec, wrote 13865503548 right entries
    45	[P3-2] Table 4 took 128.223 sec, wrote 13865503548 left entries, 13865503548 final
    46	[P3-1] Table 5 took 202.979 sec, wrote 14133784182 right entries
    47	[P3-2] Table 5 took 133.785 sec, wrote 14133784182 left entries, 14133784182 final
    48	[P3-1] Table 6 took 214.088 sec, wrote 14857706259 right entries
    49	[P3-2] Table 6 took 138.457 sec, wrote 14857706259 left entries, 14857706259 final
    50	[P3-1] Table 7 took 140.762 sec, wrote 17185033637 right entries
    51	[P3-2] Table 7 took 153.522 sec, wrote 17185033637 left entries, 17185033637 final
    52	Phase 3 took 2053.85 sec, wrote 87520148136 entries to final plot
    53	[P4] Starting to write C1 and C3 tables
    54	[P4] Finished writing C1 and C3 tables
    55	[P4] Writing C2 table
    56	[P4] Finished writing C2 table
    57	Phase 4 took 230.073 sec, final plot size is 461632649589 bytes
    58	Total plot creation time was 6536.32 sec (108.939 min)
    59	172134.98user 16600.73system 1:48:57elapsed 2887%CPU (0avgtext+0avgdata 124772548maxresident)k
    60	2291858200inputs+3182679016outputs (12859major+1069308525minor)pagefaults 0swaps

0 replies

bladeuserpi · 2022-03-02T15:48:10Z

bladeuserpi
Mar 2, 2022
Author

Also the scaling with such high core count is not as good
as expected as there are several bottlenecks (which are not
exposed when running with less cores).
As soon as a bottleneck is reached the performance maxes out,
i.e. even adding more cores does not improve perfomance.

One of these bottlenecks is "phase2/count".

I assume once "phase2/count" reaches 99% CPU the other threads are starving,
as they wait for more data to digest (see phase2/remap and phase2/add).

In Linux you can get insight what the threads are doing by running "top"
and then hitting "H":


top - 16:36:38 up 2 days,  4:03,  5 users,  load average: 15.84, 16.70, 17.47
Threads: 957 total,  13 running, 944 sleeping,   0 stopped,   0 zombie
%Cpu(s): 17.0 us,  0.5 sy,  0.0 ni, 79.8 id,  2.7 wa,  0.1 hi,  0.1 si,  0.0 st
MiB Mem : 515633.8 total,   2056.2 free,   8983.1 used, 504594.5 buff/cache
MiB Swap:  46100.0 total,  40149.3 free,   5950.7 used. 328341.5 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                     
  83416 ghia42    20   0 9509548   7.7g   1112 R  99.9   1.5   2:36.87 phase2/count                                                                                
  83426 ghia42    20   0 9509548   7.7g   1112 R  69.3   1.5   1:28.24 phase2/remap/9                                                                              
  83439 ghia42    20   0 9509548   7.7g   1112 S  69.0   1.5   1:11.52 phase2/remap/22                                                                             
  83424 ghia42    20   0 9509548   7.7g   1112 R  68.6   1.5   1:39.51 phase2/remap/7                                                                              
  83428 ghia42    20   0 9509548   7.7g   1112 R  68.6   1.5   1:48.42 phase2/remap/11                                                                             
  83430 ghia42    20   0 9509548   7.7g   1112 R  68.6   1.5   1:49.42 phase2/remap/13                                                                             
  83434 ghia42    20   0 9509548   7.7g   1112 R  66.7   1.5   1:48.67 phase2/remap/17                                                                             
  83437 ghia42    20   0 9509548   7.7g   1112 R  38.6   1.5   1:18.30 phase2/remap/20                                                                             
  83436 ghia42    20   0 9509548   7.7g   1112 R  38.3   1.5   0:59.57 phase2/remap/19                                                                             
  83418 ghia42    20   0 9509548   7.7g   1112 S  38.0   1.5   1:02.94 phase2/remap/1                                                                              
  83422 ghia42    20   0 9509548   7.7g   1112 S  37.3   1.5   1:03.24 phase2/remap/5                                                                              
  83423 ghia42    20   0 9509548   7.7g   1112 S  37.0   1.5   1:02.18 phase2/remap/6                                                                              
  83420 ghia42    20   0 9509548   7.7g   1112 S  36.6   1.5   1:01.60 phase2/remap/3                                                                              
  83417 ghia42    20   0 9509548   7.7g   1112 S  36.3   1.5   1:01.86 phase2/remap/0                                                                              
  83421 ghia42    20   0 9509548   7.7g   1112 S  36.3   1.5   1:02.05 phase2/remap/4                                                                              
  83425 ghia42    20   0 9509548   7.7g   1112 S  36.0   1.5   1:00.08 phase2/remap/8                                                                              
  83433 ghia42    20   0 9509548   7.7g   1112 R  36.0   1.5   0:56.68 phase2/remap/16                                                                             
  83419 ghia42    20   0 9509548   7.7g   1112 S  35.6   1.5   1:10.33 phase2/remap/2                                                                              
  83435 ghia42    20   0 9509548   7.7g   1112 R  35.6   1.5   1:13.98 phase2/remap/18                                                                             
  83440 ghia42    20   0 9509548   7.7g   1112 S  34.7   1.5   0:59.19 phase2/remap/23                                                                             
  83427 ghia42    20   0 9509548   7.7g   1112 S  33.7   1.5   1:02.62 phase2/remap/10                                                                             
  83438 ghia42    20   0 9509548   7.7g   1112 R  33.3   1.5   0:56.67 phase2/remap/21                                                                             
  83429 ghia42    20   0 9509548   7.7g   1112 S  32.7   1.5   1:01.49 phase2/remap/12                                                                             
  83431 ghia42    20   0 9509548   7.7g   1112 S  32.0   1.5   0:56.16 phase2/remap/14                                                                             
  83432 ghia42    20   0 9509548   7.7g   1112 R  31.7   1.5   0:58.25 phase2/remap/15                                                                             
  83405 ghia42    20   0 9509548   7.7g   1112 S   6.3   1.5   0:09.98 phase2/add/1                                                                                
  83408 ghia42    20   0 9509548   7.7g   1112 S   5.9   1.5   0:09.37 phase2/add/4                                                                                
  83406 ghia42    20   0 9509548   7.7g   1112 S   5.6   1.5   0:09.82 phase2/add/2                                                                                
  83404 ghia42    20   0 9509548   7.7g   1112 S   5.3   1.5   0:08.54 phase2/add/0                                                                                
  83409 ghia42    20   0 9509548   7.7g   1112 S   5.3   1.5   0:08.69 phase2/add/5                                                                                
  83413 ghia42    20   0 9509548   7.7g   1112 S   5.3   1.5   0:09.71 phase2/add/9                                                                                
  83415 ghia42    20   0 9509548   7.7g   1112 S   5.3   1.5   0:09.41 phase2/add/11                                                                               
  83446 ghia42    20   0 9509548   7.7g   1112 S   5.3   1.5   0:08.63 Table/read/5                                                                                
  83407 ghia42    20   0 9509548   7.7g   1112 S   5.0   1.5   0:08.54 phase2/add/3                                                                                
  83411 ghia42    20   0 9509548   7.7g   1112 S   5.0   1.5   0:08.28 phase2/add/7                                                                                
  83412 ghia42    20   0 9509548   7.7g   1112 S   5.0   1.5   0:08.25 phase2/add/8                                                                                
  83441 ghia42    20   0 9509548   7.7g   1112 S   5.0   1.5   0:08.34 Table/read/0                                                                                
  83442 ghia42    20   0 9509548   7.7g   1112 S   5.0   1.5   0:08.23 Table/read/1                                                                                
  83443 ghia42    20   0 9509548   7.7g   1112 S   5.0   1.5   0:08.40 Table/read/2                                                                                
  83444 ghia42    20   0 9509548   7.7g   1112 S   5.0   1.5   0:08.37 Table/read/3                                                                                
  83445 ghia42    20   0 9509548   7.7g   1112 S   5.0   1.5   0:08.53 Table/read/4                                                                                
  83410 ghia42    20   0 9509548   7.7g   1112 S   4.6   1.5   0:08.17 phase2/add/6                                                                                
  83414 ghia42    20   0 9509548   7.7g   1112 S   4.6   1.5   0:08.51 phase2/add/10

0 replies

bladeuserpi · 2022-03-02T15:58:00Z

bladeuserpi
Mar 2, 2022
Author

Another bottleneck with high core count might be in "Phase1/slice" and
also not enough Disk/read/n threads for feeding from ramdisk/ssd.

top - 16:52:52 up 2 days,  6:24,  5 users,  load average: 24.46, 27.62, 27.57
Threads: 714 total,  31 running, 683 sleeping,   0 stopped,   0 zombie
%Cpu(s): 51.9 us,  5.1 sy,  0.0 ni, 42.5 id,  0.0 wa,  0.4 hi,  0.1 si,  0.0 st
MiB Mem : 515654.1 total,   6170.3 free,  36586.3 used, 472897.5 buff/cache
MiB Swap:  30512.0 total,  28966.0 free,   1545.9 used.  24697.8 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                             
  79335 ghia42    20   0   40.9g  34.4g   3300 R  96.1   6.8   0:23.79 phase1/slice                                        
  79351 ghia42    20   0   40.9g  34.4g   3300 R  93.2   6.8   0:28.18 Disk/read/2                                         
  79352 ghia42    20   0   40.9g  34.4g   3300 R  90.3   6.8   0:21.78 Disk/read/3                                         
  79349 ghia42    20   0   40.9g  34.4g   3300 R  89.3   6.8   0:31.42 Disk/read/0                                         
  79350 ghia42    20   0   40.9g  34.4g   3300 R  89.3   6.8   0:29.00 Disk/read/1                                         
  79273 ghia42    20   0   40.9g  34.4g   3300 R  50.5   6.8   0:14.50 phase1/write/L                                      
  79319 ghia42    20   0   40.9g  34.4g   3300 R  43.7   6.8   0:08.97 phase1/match/8                                      
  79322 ghia42    20   0   40.9g  34.4g   3300 R  43.7   6.8   0:08.52 phase1/match/11                                     
  79318 ghia42    20   0   40.9g  34.4g   3300 S  40.8   6.8   0:09.18 phase1/match/7                                      
  79305 ghia42    20   0   40.9g  34.4g   3300 S  39.8   6.8   0:08.01 phase1/eval/18                                      
  79315 ghia42    20   0   40.9g  34.4g   3300 S  39.8   6.8   0:08.16 phase1/match/4                                      
  79314 ghia42    20   0   40.9g  34.4g   3300 S  38.8   6.8   0:08.64 phase1/match/3                                      
  79304 ghia42    20   0   40.9g  34.4g   3300 S  37.9   6.8   0:07.73 phase1/eval/17                                      
  79306 ghia42    20   0   40.9g  34.4g   3300 S  37.9   6.8   0:07.94 phase1/eval/19                                      
  79308 ghia42    20   0   40.9g  34.4g   3300 S  37.9   6.8   0:08.05 phase1/eval/21                                      
  79317 ghia42    20   0   40.9g  34.4g   3300 S  37.9   6.8   0:08.94 phase1/match/6                                      
  79325 ghia42    20   0   40.9g  34.4g   3300 R  37.9   6.8   0:09.34 phase1/match/14                                     
  79309 ghia42    20   0   40.9g  34.4g   3300 S  36.9   6.8   0:08.04 phase1/eval/22                                      
  79323 ghia42    20   0   40.9g  34.4g   3300 R  36.9   6.8   0:07.71 phase1/match/12                                     
  79287 ghia42    20   0   40.9g  34.4g   3300 S  35.9   6.8   0:07.65 phase1/eval/0                                       
  79310 ghia42    20   0   40.9g  34.4g   3300 S  35.9   6.8   0:07.54 phase1/eval/23                                      
  79324 ghia42    20   0   40.9g  34.4g   3300 R  35.9   6.8   0:09.02 phase1/match/13                                     
  79316 ghia42    20   0   40.9g  34.4g   3300 S  35.0   6.8   0:08.74 phase1/match/5                                      
  79320 ghia42    20   0   40.9g  34.4g   3300 S  35.0   6.8   0:09.12 phase1/match/9                                      
  79326 ghia42    20   0   40.9g  34.4g   3300 R  35.0   6.8   0:09.14 phase1/match/15                                     
  79332 ghia42    20   0   40.9g  34.4g   3300 S  35.0   6.8   0:09.57 phase1/match/21                                     
  79302 ghia42    20   0   40.9g  34.4g   3300 S  34.0   6.8   0:07.56 phase1/eval/15                                      
  79307 ghia42    20   0   40.9g  34.4g   3300 S  34.0   6.8   0:07.19 phase1/eval/20                                      
  79328 ghia42    20   0   40.9g  34.4g   3300 R  34.0   6.8   0:09.44 phase1/match/17                                     
  79333 ghia42    20   0   40.9g  34.4g   3300 S  34.0   6.8   0:09.48 phase1/match/22                                     
  79288 ghia42    20   0   40.9g  34.4g   3300 S  33.0   6.8   0:08.00 phase1/eval/1                                       
  79290 ghia42    20   0   40.9g  34.4g   3300 S  33.0   6.8   0:07.58 phase1/eval/3                                       
  79331 ghia42    20   0   40.9g  34.4g   3300 S  33.0   6.8   0:09.64 phase1/match/20                                     
  79292 ghia42    20   0   40.9g  34.4g   3300 R  32.0   6.8   0:07.56 phase1/eval/5                                       
  79301 ghia42    20   0   40.9g  34.4g   3300 S  32.0   6.8   0:07.77 phase1/eval/14                                      
  79303 ghia42    20   0   40.9g  34.4g   3300 S  32.0   6.8   0:07.77 phase1/eval/16                                      
  79321 ghia42    20   0   40.9g  34.4g   3300 R  32.0   6.8   0:08.96 phase1/match/10                                     
  79329 ghia42    20   0   40.9g  34.4g   3300 R  32.0   6.8   0:09.41 phase1/match/18                                     
  79289 ghia42    20   0   40.9g  34.4g   3300 S  31.1   6.8   0:07.58 phase1/eval/2                                       
  79294 ghia42    20   0   40.9g  34.4g   3300 R  31.1   6.8   0:07.89 phase1/eval/7                                       
  79297 ghia42    20   0   40.9g  34.4g   3300 S  31.1   6.8   0:07.78 phase1/eval/10                                      
  79311 ghia42    20   0   40.9g  34.4g   3300 S  31.1   6.8   0:08.98 phase1/match/0                                      
  79330 ghia42    20   0   40.9g  34.4g   3300 S  31.1   6.8   0:09.41 phase1/match/19                                     
  79293 ghia42    20   0   40.9g  34.4g   3300 R  30.1   6.8   0:07.56 phase1/eval/6                                       
  79298 ghia42    20   0   40.9g  34.4g   3300 S  30.1   6.8   0:07.64 phase1/eval/11                                      
  79300 ghia42    20   0   40.9g  34.4g   3300 S  30.1   6.8   0:07.55 phase1/eval/13                                      
  79312 ghia42    20   0   40.9g  34.4g   3300 S  30.1   6.8   0:08.73 phase1/match/1                                      
  79291 ghia42    20   0   40.9g  34.4g   3300 R  29.1   6.8   0:07.78 phase1/eval/4                                       
  79296 ghia42    20   0   40.9g  34.4g   3300 S  29.1   6.8   0:07.79 phase1/eval/9                                       
  79313 ghia42    20   0   40.9g  34.4g   3300 S  29.1   6.8   0:08.84 phase1/match/2                                      
  79327 ghia42    20   0   40.9g  34.4g   3300 R  29.1   6.8   0:08.78 phase1/match/16                                     
  79334 ghia42    20   0   40.9g  34.4g   3300 S  29.1   6.8   0:07.54 phase1/match/23                                     
  79339 ghia42    20   0   40.9g  34.4g   3300 S  29.1   6.8   0:08.30 Disk/sort/3                                         
  79295 ghia42    20   0   40.9g  34.4g   3300 S  28.2   6.8   0:07.60 phase1/eval/8                                       
  79340 ghia42    20   0   40.9g  34.4g   3300 S  28.2   6.8   0:08.22 Disk/sort/4                                         
  79341 ghia42    20   0   40.9g  34.4g   3300 R  28.2   6.8   0:08.50 Disk/sort/5                                         
  79342 ghia42    20   0   40.9g  34.4g   3300 R  28.2   6.8   0:08.35 Disk/sort/6

0 replies

bladeuserpi · 2022-03-02T16:11:16Z

bladeuserpi
Mar 2, 2022
Author

And in phase3 there are not enough read threads (existing threads using 99% CPU) for feeding data to the 
other threads buffer/merge/add/sort (which only run with 45% CPU or less):

op - 17:06:03 up 2 days,  4:32,  5 users,  load average: 23.82, 21.39, 19.43
Threads: 974 total,  27 running, 947 sleeping,   0 stopped,   0 zombie
%Cpu(s): 29.7 us,  2.6 sy,  0.0 ni, 65.9 id,  1.5 wa,  0.2 hi,  0.1 si,  0.0 st
MiB Mem : 515633.8 total,  21619.0 free,  36672.1 used, 457342.6 buff/cache
MiB Swap:  46100.0 total,  37324.1 free,   8775.9 used. 154275.1 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                     
  84193 ghia42    20   0   40.4g  34.6g   1752 R  99.9   6.9   1:07.58 Disk/read/1                                                                                 
  84194 ghia42    20   0   40.4g  34.6g   1752 R  99.9   6.9   1:08.63 Disk/read/2                                                                                 
  84195 ghia42    20   0   40.4g  34.6g   1752 R  99.9   6.9   1:07.04 Disk/read/3                                                                                 
  84196 ghia42    20   0   40.4g  34.6g   1752 R  99.9   6.9   1:07.96 Disk/read/4                                                                                 
  84203 ghia42    20   0   40.4g  34.6g   1752 R  99.9   6.9   1:04.74 Disk/read/11                                                                                
  84192 ghia42    20   0   40.4g  34.6g   1752 R  99.0   6.9   1:08.87 Disk/read/0                                                                                 
  84201 ghia42    20   0   40.4g  34.6g   1752 R  99.0   6.9   1:07.07 Disk/read/9                                                                                 
  84202 ghia42    20   0   40.4g  34.6g   1752 R  99.0   6.9   1:05.77 Disk/read/10                                                                                
  84134 ghia42    20   0   40.4g  34.6g   1752 R  45.2   6.9   0:42.00 phase3/buffer                                                                               
  84154 ghia42    20   0   40.4g  34.6g   1752 S  39.4   6.9   0:35.87 phase3/merge/0                                                                              
  84159 ghia42    20   0   40.4g  34.6g   1752 R  39.4   6.9   0:35.49 phase3/merge/5                                                                              
  84155 ghia42    20   0   40.4g  34.6g   1752 R  38.5   6.9   0:36.46 phase3/merge/1                                                                              
  84156 ghia42    20   0   40.4g  34.6g   1752 S  38.5   6.9   0:35.85 phase3/merge/2                                                                              
  84157 ghia42    20   0   40.4g  34.6g   1752 S  38.5   6.9   0:35.64 phase3/merge/3                                                                              
  84149 ghia42    20   0   40.4g  34.6g   1752 R  37.5   6.9   0:32.38 phase3/add/8                                                                                
  84158 ghia42    20   0   40.4g  34.6g   1752 R  37.5   6.9   0:36.31 phase3/merge/4                                                                              
  84145 ghia42    20   0   40.4g  34.6g   1752 S  35.6   6.9   0:34.03 phase3/add/4                                                                                
  84144 ghia42    20   0   40.4g  34.6g   1752 S  34.6   6.9   0:33.49 phase3/add/3                                                                                
  84141 ghia42    20   0   40.4g  34.6g   1752 S  32.7   6.9   0:32.78 phase3/add/0                                                                                
  84146 ghia42    20   0   40.4g  34.6g   1752 S  32.7   6.9   0:31.67 phase3/add/5                                                                                
  84152 ghia42    20   0   40.4g  34.6g   1752 S  32.7   6.9   0:31.50 phase3/add/10                                                                               
  84153 ghia42    20   0   40.4g  34.6g   1752 S  32.7   6.9   0:32.52 phase3/add/11                                                                               
  84147 ghia42    20   0   40.4g  34.6g   1752 R  31.7   6.9   0:31.94 phase3/add/6                                                                                
  84180 ghia42    20   0   40.4g  34.6g   1752 S  31.7   6.9   0:29.39 Disk/sort/13         
...

0 replies

bladeuserpi · 2022-04-14T18:16:40Z

bladeuserpi
Apr 14, 2022
Author

New Best Run: Total plot creation time was 6042.22 sec (100.704 min)
Machine is: 2x E5 2699v4, 512GB Ram + INTEL SSDPEDME016T4S

Crafting plot 6 out of 40 (2022/04/14 18:09:19)
Process ID: 114209
Number of Threads: 40
Number of Buckets P1:    2^8 (256)
Number of Buckets P3+P4: 2^8 (256)
Pool Puzzle Hash:  2576c1609d2f5171b994b5800f4afd742f3df34565d00266febfbee9ab3059b5
Farmer Public Key: b69d8206fa337bd693d094e980f5df1199538b53b1ed376338c634090d023859c9b097d3dd8ffaaea8af4b44979a2f9f
Working Directory:   /plot06/
Working Directory 2: /ramdisk/
Plot Name: plot-k34-2022-04-14-18-09-af2e9d546d4b4932a6b726f2341741e80c0171776f53748721dded4ce2e1c0ca
[P1] Table 1 took 35.2635 sec
[P1] Table 2 took 283.395 sec, found 17180102832 matches
[P1] Table 3 took 420.406 sec, found 17180320537 matches
[P1] Table 4 took 483.717 sec, found 17181029820 matches
[P1] Table 5 took 468.831 sec, found 17182252721 matches
[P1] Table 6 took 432.539 sec, found 17184512094 matches
[P1] Table 7 took 414.406 sec, found 17189238191 matches
Phase 1 took 2538.58 sec
[P2] max_table_size = 17189238191
[P2] Table 7 scan took 48.5164 sec
[P2] Table 7 rewrite took 192.429 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 79.4889 sec
[P2] Table 6 rewrite took 111.05 sec, dropped 2324366484 entries (13.5259 %)
[P2] Table 5 scan took 78.0263 sec
[P2] Table 5 rewrite took 117.393 sec, dropped 3047012353 entries (17.7335 %)
[P2] Table 4 scan took 78.2881 sec
[P2] Table 4 rewrite took 102.142 sec, dropped 3314700162 entries (19.2928 %)
[P2] Table 3 scan took 76.3594 sec
[P2] Table 3 rewrite took 102.758 sec, dropped 3419681559 entries (19.9046 %)
[P2] Table 2 scan took 75.3031 sec
[P2] Table 2 rewrite took 103.029 sec, dropped 3461878375 entries (20.1505 %)
Phase 2 took 1240.4 sec
Wrote plot header with 252 bytes
[P3-1] Table 2 took 326.636 sec, wrote 13718224457 right entries
[P3-2] Table 2 took 113.625 sec, wrote 13718224457 left entries, 13718224457 final
[P3-1] Table 3 took 184.868 sec, wrote 13760638978 right entries
[P3-2] Table 3 took 126.249 sec, wrote 13760638978 left entries, 13760638978 final
[P3-1] Table 4 took 191.141 sec, wrote 13866329658 right entries
[P3-2] Table 4 took 121.831 sec, wrote 13866329658 left entries, 13866329658 final
[P3-1] Table 5 took 189.31 sec, wrote 14135240368 right entries
[P3-2] Table 5 took 128.83 sec, wrote 14135240368 left entries, 14135240368 final
[P3-1] Table 6 took 200.624 sec, wrote 14860145610 right entries
[P3-2] Table 6 took 135.136 sec, wrote 14860145610 left entries, 14860145610 final
[P3-1] Table 7 took 145.565 sec, wrote 17189238191 right entries
[P3-2] Table 7 took 157.836 sec, wrote 17189238191 left entries, 17189238191 final
Phase 3 took 2049.24 sec, wrote 87529817262 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 213.782 sec, final plot size is 461694095744 bytes
Total plot creation time was 6042.22 sec (100.704 min)

0 replies

smilstea · 2022-05-13T20:17:13Z

smilstea
May 13, 2022

@bladeuserpi did you ever find the sweet spot of multi-socket / threads to use for madmax with K34?
I recently bought a Dell R720 with 2x E5-2690 V2 and 768GB of RAM. I am adding two PCIe v3.0 NVME drives in raid0 for tmp1.
It seems from what I have read that using a ramdisk for tmp1 doesn't really add much value (can actually hurt) compared to NVME.

0 replies

brianwfreeman · 2022-09-10T14:40:18Z

brianwfreeman
Sep 10, 2022

Number of Threads: 20
Number of Buckets P1: 2^8 (256)
Number of Buckets P3+P4: 2^8 (256)
Pool Puzzle Hash: f1102807deec714ae396fd282851a7865433585be34734fdb408a4cfc3a5ba5c
Farmer Public Key: a1e6bfe953ad89b5a61b1ad8f3e8bf83f2aa0b496abf3404198f66e3250aefa4339ff096c6e6d9b920a426b5ca92a2c8
Working Directory: /Volumes/Optane4x/
Working Directory 2: /Volumes/Optane4x/
Plot Name: plot-k34-2022-09-07-19-08-1e27a2eed785359ab5b830ac206752e65b628ecb4be9462508501728979d3065
[P1] Table 1 took 117.059 sec
[P1] Table 2 took 328.116 sec, found 17180071976 matches
[P1] Table 3 took 431.635 sec, found 17180256707 matches
[P1] Table 4 took 456.602 sec, found 17180521737 matches
[P1] Table 5 took 433.288 sec, found 17181343295 matches
[P1] Table 6 took 390.737 sec, found 17183030858 matches
[P1] Table 7 took 349.8 sec, found 17186318114 matches
Phase 1 took 2507.3 sec
[P2] max_table_size = 17186318114
[P2] Table 7 scan took 48.7415 sec
[P2] Table 7 rewrite took 151.467 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 51.0573 sec
[P2] Table 6 rewrite took 194.49 sec, dropped 2324597021 entries (13.5284 %)
[P2] Table 5 scan took 49.4438 sec
[P2] Table 5 rewrite took 188.675 sec, dropped 3047209834 entries (17.7356 %)
[P2] Table 4 scan took 47.3587 sec
[P2] Table 4 rewrite took 185.358 sec, dropped 3314918502 entries (19.2946 %)
[P2] Table 3 scan took 48.2893 sec
[P2] Table 3 rewrite took 183.55 sec, dropped 3420005777 entries (19.9066 %)
[P2] Table 2 scan took 48.3312 sec
[P2] Table 2 rewrite took 178.777 sec, dropped 3462131010 entries (20.152 %)
Phase 2 took 1383.42 sec
Wrote plot header with 252 bytes
[P3-1] Table 2 took 140.728 sec, wrote 13717940966 right entries
[P3-2] Table 2 took 105.12 sec, wrote 13717940966 left entries, 13717940966 final
[P3-1] Table 3 took 130.046 sec, wrote 13760250930 right entries
[P3-2] Table 3 took 107.052 sec, wrote 13760250930 left entries, 13760250930 final
[P3-1] Table 4 took 129.987 sec, wrote 13865603235 right entries
[P3-2] Table 4 took 107.627 sec, wrote 13865603235 left entries, 13865603235 final
[P3-1] Table 5 took 131.13 sec, wrote 14134133461 right entries
[P3-2] Table 5 took 109.022 sec, wrote 14134133461 left entries, 14134133461 final
[P3-1] Table 6 took 136.973 sec, wrote 14858433837 right entries
[P3-2] Table 6 took 113.706 sec, wrote 14858433837 left entries, 14858433837 final
[P3-1] Table 7 took 173.707 sec, wrote 17186318114 right entries
[P3-2] Table 7 took 131.978 sec, wrote 17186318114 left entries, 17186318114 final
Phase 3 took 1517.81 sec, wrote 87522680543 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 125.281 sec, final plot size is 461649575626 bytes
Total plot creation time was 5533.89 sec (92.2315 min)

Mac Studio Ultra with 4 Intel Optane 905ps in RAID0 (two thunderbolt ports)

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Madmax K34 Performance #996

{{title}}

Replies: 10 comments 1 reply

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Madmax K34 Performance #996

bladeuserpi Feb 20, 2022

Replies: 10 comments · 1 reply

brianwfreeman Feb 27, 2022

bladeuserpi Mar 2, 2022 Author

brianwfreeman Feb 27, 2022

brianwfreeman Feb 27, 2022

bladeuserpi Mar 2, 2022 Author

bladeuserpi Mar 2, 2022 Author

bladeuserpi Mar 2, 2022 Author

bladeuserpi Mar 2, 2022 Author

bladeuserpi Apr 14, 2022 Author

smilstea May 13, 2022

brianwfreeman Sep 10, 2022

bladeuserpi
Feb 20, 2022

Replies: 10 comments 1 reply

brianwfreeman
Feb 27, 2022

bladeuserpi Mar 2, 2022
Author

brianwfreeman
Feb 27, 2022

brianwfreeman
Feb 27, 2022

bladeuserpi
Mar 2, 2022
Author

bladeuserpi
Mar 2, 2022
Author

bladeuserpi
Mar 2, 2022
Author

bladeuserpi
Mar 2, 2022
Author

bladeuserpi
Apr 14, 2022
Author

smilstea
May 13, 2022

brianwfreeman
Sep 10, 2022