Asynchronous Block Download and Verification

[eval-exec]

What problem does this PR solve?

Working in progress.

Thinking and trying to make Synchronizer download block and ChainService verify block asynchronous.

What's Changed:

a sequence diagram to describe the IBD phase

The IBD phase on develop branch (before v0.111.0)

sequenceDiagram
  autonumber

  participant S as Synchronizer
  participant BP as BlockProcess
  participant C as ChainService


  box crate:ckb_sync
    participant S
    participant BP
  end


  box crate:ckb_chain
    participant C
  end
  
  Note left of S: synchronizer received <br>Block(122) from remote peer
  
  Note over S: try_process SyncMessageUnionReader::SendBlock


  S->>+BP: BlockProcess::execute(Block(122))
  BP->>+C: process_block(Block(122))
  Note over BP: waiting ChainService to return<br>the result of process_block(Block(123))
  Note over C: insert_block(Block(122))
  C->>-BP: return result of process_block(Block(122))
  BP->>-S: return result of BlockProcess::execute(Block(122))
  
  alt block is Valid
    Note over S: going on
  else block is Invalid
    Note over S: punish the malicious peer
  end

  Note left of S: synchronizer received <br>Block(123) from remote peer
  Note over S: try_process SyncMessageUnionReader::SendBlock
  S->>+BP: BlockProcess::execute(Block(123))
  BP->>+C: process_block(Block(123))
  Note over BP: waiting ChainService to return<br>the result of process_block(Block(123))
  Note over C: insert_block(Block(123))
  C->>-BP: return result of process_block(Block(123))
  BP->>-S: return result of BlockProcess::execute(Block(123))
  
  alt block is Valid
    Note over S: going on
  else block is Invalid
    Note over S: punish the malicious peer
  end

Loading

The IBD phase by this PR:

sequenceDiagram
  autonumber

  participant Sr as Synchronizer::received
  participant BP as BlockProcess

  participant Sp as Synchronizer::poll

  participant C as main thread
  participant CO as OrphanBlockPool thread
  participant CV as ConsumeUnverifiedBlocks thread

  box crate:ckb-sync
    participant  Sr
    participant  Sp
    participant  BP
  end

  box crate:ckb-chain
    participant C
    participant CO
    participant CV
  end



  Note left of Sr: synchronizer received <br>Block(122) from remote peer
  Note over Sr: try_process SyncMessageUnionReader::SendBlock
  Sr->>+BP: BlockProcess::execute(Block(122))
  BP->>+C: asynchronous_process_block(Block(122))
  Note over C: non_contextual_verify(Block(122))
  C->>+CO: send Block(122) to OrphanBlockPool via channel
  C->>-BP: return
  BP->>-Sr: return

  Note over CO: insert Block(122) to OrphanBlockPool

  Note left of Sr: synchronizer received <br>Block(123) from remote peer
  Note over Sr: try_process SyncMessageUnionReader::SendBlock
  Sr->>+BP: BlockProcess::execute(Block(123))
  BP->>+C: asynchronous_process_block(Block(123))
  Note over C: non_contextual_verify(Block(123))
  C->>+CO: send Block(123) to OrphanBlockPool via channel
  C->>-BP: return
  BP->>-Sr: return

  Note over CO: insert Block(123) to OrphanBlockPool

  loop Search Orphan Pool
    Note over CO: if a leader block have descendants
    Note over CO: load all descendants from OrphanBlockPool
    Note over CO: assume these descendants are valid, let BlockExt.verified = None
    Note over CO: insert them to RocksDB
    Note over CO: Increase Unverified TIP
    CO->>+CV: send the UnverifiedBlock to ConsumeUnverifiedBlocks via channel
  end

  loop Consume Unverified Blocks
    Note over CV: start verify UnverifiedBlock if the channel is not empty

    Note over CV: Verify Block in CKB VM


    alt Block is Valid
      Note over CV: remove Block block_status and HeaderMap
    else Block is Invalid
      Note over CV: Decrease Unverified TIP
      CV->>Sp: I received a Invalid Block, please punish the malicious peer
      Note over Sp: call nc.ban_peer() to punish the malicious peer
    end
    opt Execute Callback
      Note over CV: callback: Box<dyn FnOnce(Result<VerifiedBlockStatus, Error>) + Send + Sync>

    end
  end

Loading

Related changes

• When Synchronizer receive a block from remote peer, transfer it to ChainService.
• ChainService performs all validation tasks except for ScriptVerify, and optimistically assumes that the block is valid, storing blocks first in RocksDB, and using UnverifiedTip to represent the highest chain that has not been verified.
• do ScriptVerify task in a a asynchronous thread and update TipHeader
• move OrphanBlock to ckb_chain
• move BlockStatus and HeaderMap to ckb_shared, because both Synchronizer and ChainService need them
• Many things are still being considered/thought through.

Check List

Tests

Side effects

• Performance regression
• Breaking backward compatibility

Release note

Title Only: Include only the PR title in the release note.

[eval-exec]

On AWS c5.xlarge 4C 8G 200G machine, current branch's latest build got this:

[eval-exec]

I think we shouldn't derive[Clone] for Synchronizer and Relayer, in CKB process, there should be only one running Synchronizer instance:

ckb/sync/src/synchronizer/mod.rs

Lines 231 to 239 in 8dffa39

	/// Sync protocol handle
	#[derive(Clone)]
	pub struct Synchronizer {
	pub(crate) chain: ChainController,
	/// Sync shared state
	pub shared: Arc<SyncShared>,
	fetch_channel: Option<channel::Sender<FetchCMD>>,
	}

[eval-exec]

Since the ChainService is verifying blocks in an asynchronous manner, it is worth considering implementing CKBProtocolHandler::poll for the Synchronizer to obtain the verification result. If the verification fails, the peer can be banned.

[eval-exec]

Need to fix CI issues before release ckb-async-download-rc0.