src: dynamic block sync size #9494

0xFFFC0000 · 2024-09-30T09:05:09Z

No description provided.

0xFFFC0000 · 2024-09-30T10:30:10Z

Same content. Only removed white-space changes.

sneurlax

LGTM! It was helpful for me to start reviewing out from here

nahuhh

Tested on stressnet and mainnet

src/cryptonote_protocol/cryptonote_protocol_handler.inl

src/cryptonote_protocol/cryptonote_protocol_handler.h

src/cryptonote_core/cryptonote_core.cpp

0xFFFC0000 · 2024-10-12T06:37:39Z

All comments have been addressed. Once discussion finished, I will update release branch too.

0xFFFC0000 · 2024-10-13T12:37:49Z

Updated to most latest master branch commit : 9866a0e9021e2422d8055731a586083eb5e2be67.

0xFFFC0000 · 2025-02-05T14:54:10Z

Rebased to updated master. Now CI should run without any issue.

nahuhh · 2025-02-21T00:51:27Z

src/cryptonote_core/cryptonote_core.cpp

+      MINFO("Last " << number_of_blocks
+             << " blocks median size is " << median_weight
+             << " bytes and the max average blocksize in the queue is " << average_blocksize_of_biggest_batch << " bytes");
+      uint64_t projected_blocksize = (average_blocksize_of_biggest_batch > median_weight) ? average_blocksize_of_biggest_batch : median_weight;


Suggested change

uint64_t projected_blocksize = (average_blocksize_of_biggest_batch > median_weight) ? average_blocksize_of_biggest_batch : median_weight;

uint64_t projected_blocksize = (average_blocksize_of_biggest_batch > median_weight) ? average_blocksize_of_biggest_batch : median_weight;

uint64_t blocks_huge_threshold = (batch_max_weight / 2);

nahuhh · 2025-02-21T00:54:30Z

src/cryptonote_core/cryptonote_core.cpp

+      else if (projected_blocksize >= batch_max_weight) {
+        res = 1;
+        MINFO("blocks are projected to surpass " << batch_max_weight << " bytes, syncing just a single block in next batch");
+      }
+      else if (projected_blocksize > BLOCKS_HUGE_THRESHOLD_SIZE) {
+        res = 1;
+        MINFO("blocks are huge, sync just a single block in next batch");


Suggested change

else if (projected_blocksize >= batch_max_weight) {

res = 1;

MINFO("blocks are projected to surpass " << batch_max_weight << " bytes, syncing just a single block in next batch");

}

else if (projected_blocksize > BLOCKS_HUGE_THRESHOLD_SIZE) {

res = 1;

MINFO("blocks are huge, sync just a single block in next batch");

else if (projected_blocksize >= blocks_huge_threshold) {

res = 1;

MINFO("blocks are projected to surpass 50% of " << batch_max_weight << " bytes, syncing just a single block in next batch");

}

condense

nahuhh · 2025-02-21T00:57:22Z

src/cryptonote_config.h

+#define BLOCKS_MEDIAN_WINDOW                            100      //by default, compute median weights of last 100 blocks
+#define BATCH_MAX_WEIGHT                                20       //by default, maximum size of batch in [mB]
+#define BATCH_MAX_ALLOWED_WEIGHT                        50       //maximum allowed size of batch in [mB]
+#define BLOCKS_HUGE_THRESHOLD_SIZE                      ((BATCH_MAX_WEIGHT * 1000000) / 2) //blocks that we consider huge [B]


Suggested change

#define BLOCKS_HUGE_THRESHOLD_SIZE ((BATCH_MAX_WEIGHT * 1000000) / 2) //blocks that we consider huge [B]

swapped here
#9494 (comment)

nahuhh · 2025-02-21T01:09:04Z

src/cryptonote_config.h

@@ -98,6 +98,10 @@
 #define BLOCKS_SYNCHRONIZING_DEFAULT_COUNT_PRE_V4       100    //by default, blocks count in blocks downloading
 #define BLOCKS_SYNCHRONIZING_DEFAULT_COUNT              20     //by default, blocks count in blocks downloading
 #define BLOCKS_SYNCHRONIZING_MAX_COUNT                  2048   //must be a power of 2, greater than 128, equal to SEEDHASH_EPOCH_BLOCKS
+#define BLOCKS_MEDIAN_WINDOW                            100      //by default, compute median weights of last 100 blocks


Suggested change

#define BLOCKS_MEDIAN_WINDOW 100 //by default, compute median weights of last 100 blocks

#define BLOCKS_MEDIAN_WINDOW CRYPTONOTE_REWARD_BLOCKS_WINDOW //by default, compute median weights of last 100 blocks (CRYPTONOTE_REWARD_BLOCKS_WINDOW)

src/cryptonote_core/cryptonote_core.cpp

nahuhh · 2025-02-21T05:46:18Z

src/cryptonote_protocol/cryptonote_protocol_handler.h

@@ -112,6 +113,15 @@ namespace cryptonote
    void log_connections();
    std::list<connection_info> get_connections();
    const block_queue &get_block_queue() const { return m_block_queue; }
+    const std::uint64_t max_average_of_blocksize_in_queue() {


Suggested change

const std::uint64_t max_average_of_blocksize_in_queue() {

std::uint64_t max_average_of_blocksize_in_queue() {

Warning in build log

0xFFFC0000 · 2025-03-29T05:19:06Z

Squashed to single commit. No change in content.

nahuhh

I think this is ready. I've tested numerous times

@vtnerd can you take another look

vtnerd · 2025-05-04T23:59:02Z

src/cryptonote_protocol/cryptonote_protocol_handler.h

@@ -192,6 +202,7 @@ namespace cryptonote
    uint64_t m_sync_download_chain_size, m_sync_download_objects_size;
    size_t m_block_download_max_size;
    bool m_sync_pruned_blocks;
+    std::atomic<size_t> m_bss;


I only see 1 write and no reads of this variable, can it be removed?

Yes, in this PR.

But in the dynamic span PR we need this variable. Long story short, it is fine to delete it here.

https://github.com/monero-project/monero/pull/9495/files#diff-d32229ecf55d414dcca43b6918ad711e1ffd313ef851492ecba1865323007f0fR225

vtnerd · 2025-05-04T23:59:38Z

src/cryptonote_protocol/cryptonote_protocol_handler.h

@@ -112,6 +113,15 @@ namespace cryptonote
    void log_connections();
    std::list<connection_info> get_connections();
    const block_queue &get_block_queue() const { return m_block_queue; }
+    std::uint64_t max_average_of_blocksize_in_queue() {


Small nitpick, but this bracing style isn't matching to this document. Same thing in other files.

Of course. Will be fixed in updated push.

vtnerd · 2025-05-05T00:04:40Z

src/cryptonote_core/cryptonote_core.cpp

+      MINFO("Max block size seen within the last " << number_of_blocks
+             << " blocks is " << max_weight
+             << " bytes and the max average blocksize in the queue is " << max_average_of_blocksize_in_queue << " bytes");
+      uint64_t projected_blocksize = (max_average_of_blocksize_in_queue > max_weight) ? max_average_of_blocksize_in_queue : max_weight;


looks like a std::max(max_average_of_blocksize_in_queue, max_weight)

Yes. Will be fixed in updated PR.

vtnerd · 2025-05-05T00:08:18Z

src/cryptonote_core/cryptonote_core.cpp

+             << " bytes and the max average blocksize in the queue is " << max_average_of_blocksize_in_queue << " bytes");
+      uint64_t projected_blocksize = (max_average_of_blocksize_in_queue > max_weight) ? max_average_of_blocksize_in_queue : max_weight;
+      uint64_t blocks_huge_threshold = (batch_max_weight / 2);
+      if ((projected_blocksize * BLOCKS_MAX_WINDOW) < batch_max_weight) {


Can this overflow? Does it make sense to flip the logic to:
if (projected_block_size < batch_max_weight / BLOCKS_MAX_WINDOW)?

It will not.

projected_block_size is uint64_t.
BLOCKS_MAX_WINDOW is 100.
batch_max_weight is uint64_t.

You do the math. Basically impossible with current ( and foreseeable ) network limitations to get close to overflow.

vtnerd · 2025-05-05T00:10:05Z

src/cryptonote_core/cryptonote_core.cpp

+        MINFO("blocks are tiny, " << projected_blocksize << " bytes, sync " << res << " blocks in next batch");
+      }
+      else if (projected_blocksize >= blocks_huge_threshold) {
+        res = 1;


If the block sizes exceed 5MB, then it drops down to one block? The max levin size is 100 MB? These values seem to too low.

We do have a serialization limit, too. So, because of the serialization limit, we have to limit it.

By default the estimations target 10mb batches

estimates are conservative, but can still be exceeded by ~100% under spike volume

current serialization limits break syncing if you exceed ~25mb in a batch

until serialization limits are extended / removed, 10mb is the largest we can safely default to. Any larger has potential to hit serialization limits

if a single block is expected to exceed 1/2 of the batch size target, then syncing more than 1 block would exceed the target, hence the "5mb"

the max this can be set to, is 50mb (1/2 of the levin limit to add safe-zone for underestimation due to spikes in volume). 50mb cant be used today though, because of serialization

Also to note

downloading of 10mb batches results in recalculating the estimation more often, and more reliable downloads (since monero likes to ban peers who dont complete the upload fast enough)

nahuhh · 2025-05-13T12:53:48Z

src/cryptonote_core/cryptonote_core.cpp

+      MWARNING("When --block-sync-size defined, the --batch-max-weight is not going to have any effect.");
+
+    batch_max_weight = command_line::get_arg(vm, arg_batch_max_weight);
+    if (batch_max_weight > BATCH_MAX_ALLOWED_WEIGHT) {


bracing in this section

nahuhh · 2025-05-13T12:55:57Z

src/cryptonote_core/cryptonote_core.cpp

-      res = BLOCKS_SYNCHRONIZING_DEFAULT_COUNT;
-    else
-      res = BLOCKS_SYNCHRONIZING_DEFAULT_COUNT_PRE_V4;
+    else {


Bracing in this section

nahuhh · 2025-05-13T12:56:48Z

src/cryptonote_protocol/cryptonote_protocol_handler.h

@@ -192,6 +203,7 @@ namespace cryptonote
    uint64_t m_sync_download_chain_size, m_sync_download_objects_size;
    size_t m_block_download_max_size;
    bool m_sync_pruned_blocks;
+    std::atomic<size_t> m_bss;


Remove? #9494 (comment)

Co-authored-by: nahuhh

0xFFFC0000 added pending review feature daemon labels Sep 30, 2024

0xFFFC0000 force-pushed the dev/0xfffc/dynamic-bss branch from 5b05971 to ce38fde Compare September 30, 2024 10:29

sneurlax approved these changes Sep 30, 2024

View reviewed changes

nahuhh approved these changes Oct 1, 2024

View reviewed changes

0xFFFC0000 mentioned this pull request Oct 1, 2024

src: dynamic block sync size [RELEASE] #9497

Open

0xFFFC0000 force-pushed the dev/0xfffc/dynamic-bss branch from ce38fde to 2ce35d9 Compare October 1, 2024 15:32

sneurlax approved these changes Oct 2, 2024

View reviewed changes

vtnerd suggested changes Oct 9, 2024

View reviewed changes

nahuhh reviewed Oct 11, 2024

View reviewed changes

src/cryptonote_core/cryptonote_core.cpp Outdated Show resolved Hide resolved

0xFFFC0000 force-pushed the dev/0xfffc/dynamic-bss branch from 2ce35d9 to 03d0dce Compare October 12, 2024 06:36

0xFFFC0000 force-pushed the dev/0xfffc/dynamic-bss branch from 03d0dce to 2195b6a Compare October 13, 2024 12:37

0xFFFC0000 force-pushed the dev/0xfffc/dynamic-bss branch from 2195b6a to 5459934 Compare February 5, 2025 14:53

nahuhh mentioned this pull request Feb 5, 2025

src: bump network speed 4x #9770

Merged

nahuhh suggested changes Feb 21, 2025

View reviewed changes

nahuhh reviewed Feb 21, 2025

View reviewed changes

0xFFFC0000 force-pushed the dev/0xfffc/dynamic-bss branch 4 times, most recently from 03bcd6f to fd06f00 Compare February 24, 2025 11:48

0xFFFC0000 force-pushed the dev/0xfffc/dynamic-bss branch 2 times, most recently from a4b2894 to dc80b29 Compare March 28, 2025 10:00

0xFFFC0000 force-pushed the dev/0xfffc/dynamic-bss branch 2 times, most recently from a71734d to ad988af Compare March 29, 2025 05:17

nahuhh mentioned this pull request Apr 7, 2025

Starting monerod with block-sync-size >~ 100 prevents daemon to sync #9897

Open

0xFFFC0000 force-pushed the dev/0xfffc/dynamic-bss branch 2 times, most recently from e405405 to 0a498c6 Compare May 1, 2025 11:01

nahuhh approved these changes May 3, 2025

View reviewed changes

vtnerd suggested changes May 5, 2025

View reviewed changes

Rucknium mentioned this pull request May 12, 2025

Monero Research Lab Meeting - Wed 07 May 2025, 17:00 UTC monero-project/meta#1200

Closed

0xFFFC0000 force-pushed the dev/0xfffc/dynamic-bss branch from 0a498c6 to 3096729 Compare May 13, 2025 06:43

nahuhh reviewed May 13, 2025

View reviewed changes

0xFFFC0000 force-pushed the dev/0xfffc/dynamic-bss branch 2 times, most recently from f54a254 to 3e36c30 Compare May 13, 2025 13:39

src: dynamic block sync size

2a1a489

Co-authored-by: nahuhh

0xFFFC0000 force-pushed the dev/0xfffc/dynamic-bss branch from 3e36c30 to 2a1a489 Compare May 13, 2025 13:52

spackle-xmr mentioned this pull request Jul 16, 2025

TODO list for launch seraphis-migration/monero#53

Open

62 tasks

	uint64_t projected_blocksize = (average_blocksize_of_biggest_batch > median_weight) ? average_blocksize_of_biggest_batch : median_weight;
	uint64_t projected_blocksize = (average_blocksize_of_biggest_batch > median_weight) ? average_blocksize_of_biggest_batch : median_weight;
	uint64_t blocks_huge_threshold = (batch_max_weight / 2);

	#define BLOCKS_MEDIAN_WINDOW 100 //by default, compute median weights of last 100 blocks
	#define BLOCKS_MEDIAN_WINDOW CRYPTONOTE_REWARD_BLOCKS_WINDOW //by default, compute median weights of last 100 blocks (CRYPTONOTE_REWARD_BLOCKS_WINDOW)

	const std::uint64_t max_average_of_blocksize_in_queue() {
	std::uint64_t max_average_of_blocksize_in_queue() {

Uh oh!

src: dynamic block sync size #9494

Are you sure you want to change the base?

src: dynamic block sync size #9494

Uh oh!

Conversation

0xFFFC0000 commented Sep 30, 2024

Uh oh!

0xFFFC0000 commented Sep 30, 2024

Uh oh!

sneurlax left a comment

Choose a reason for hiding this comment

Uh oh!

nahuhh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

0xFFFC0000 commented Oct 12, 2024

Uh oh!

0xFFFC0000 commented Oct 13, 2024

Uh oh!

0xFFFC0000 commented Feb 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0xFFFC0000 commented Mar 29, 2025

Uh oh!

nahuhh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nahuhh Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nahuhh Jul 10, 2025 •

edited

Loading