Add Cell Dissemination via Partial Message Specification #4558

MarcoPolo · 2025-09-03T04:02:08Z

In draft until the libp2p/specs work is merged and we have multiple implementations of this. Also requires TODOs to be addressed.

This PR specifies how consensus clients use Gossipsub's Partial Messages to disseminate cells rather than only full columns.

More context in this ethresearch post.

fradamt · 2025-09-04T16:28:59Z

specs/fulu/das-core.md

+#### `PartialDataColumnSidecar`
+
+```python
+class PartialDataColumnSidecar(Container):
+    # Only provided if the index can not be inferred from the Gossipsub topic
+    index: ColumnIndex | None
+    # Encoded the same as an IHAVE bitmap
+    cells_present_bitmap: ByteVector[NUMBER_OF_COLUMNS/8] # ceiling if NUMBER_OF_COLUMNS is not divisible by 8
+    column: List[Cell, MAX_BLOB_COMMITMENTS_PER_BLOCK]
+    kzg_proofs: List[KZGProof, MAX_BLOB_COMMITMENTS_PER_BLOCK]
+     # The following are only provided on eager pushes.
+    kzg_commitments: None | List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK]
+    signed_block_header: None | SignedBeaconBlockHeader
+    kzg_commitments_inclusion_proof: None | Vector[Bytes32, KZG_COMMITMENTS_INCLUSION_PROOF_DEPTH]
+```


I think there's three possible ways to go with this:

Add an Optional SSZ type, which we are currently missing (see https://ethereum-magicians.org/t/eip-6475-ssz-optional/12891)

class PartialDataColumnSidecar(Container): index: ColumnIndex cells_bitmap: Bitlist[MAX_BLOB_COMMITMENTS_PER_BLOCK] cells: List[Cell, MAX_BLOB_COMMITMENTS_PER_BLOCK] kzg_proofs: List[KZGProof, MAX_BLOB_COMMITMENTS_PER_BLOCK] kzg_commitments: Optional[List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK]] signed_block_header: Optional[SignedBeaconBlockHeader] kzg_commitments_inclusion_proof: Optional[Vector[Bytes32, KZG_COMMITMENTS_INCLUSION_PROOF_DEPTH]

Separate the data part from the kzg commitments, header and proof part

class PartialDataColumnSidecar(Container): index: ColumnIndex cells_bitmap: Bitlist[MAX_BLOB_COMMITMENTS_PER_BLOCK] cells: List[Cell, MAX_BLOB_COMMITMENTS_PER_BLOCK] kzg_proofs: List[KZGProof, MAX_BLOB_COMMITMENTS_PER_BLOCK] class KZGCommitmentsSidecar(Container): kzg_commitments: List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK] signed_block_header: SignedBeaconBlockHeader kzg_commitments_inclusion_proof: Vector[Bytes32, KZG_COMMITMENTS_INCLUSION_PROOF_DEPTH]

Have two different types for a column with and without committments (hopefully with nicer names)

class PartialDataColumnSidecar(Container): index: ColumnIndex cells_bitmap: Bitlist[MAX_BLOB_COMMITMENTS_PER_BLOCK] cells: List[Cell, MAX_BLOB_COMMITMENTS_PER_BLOCK] kzg_proofs: List[KZGProof, MAX_BLOB_COMMITMENTS_PER_BLOCK] class PartialDataColumnSidecarWithCommitments(Container): index: ColumnIndex cells_bitmap: Bitlist[MAX_BLOB_COMMITMENTS_PER_BLOCK] cells: List[Cell, MAX_BLOB_COMMITMENTS_PER_BLOCK] kzg_proofs: List[KZGProof, MAX_BLOB_COMMITMENTS_PER_BLOCK] kzg_commitments: List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK] signed_block_header: SignedBeaconBlockHeader kzg_commitments_inclusion_proof: Vector[Bytes32, KZG_COMMITMENTS_INCLUSION_PROOF_DEPTH]

Also:

changed ByteVector to Bitlist (but could also be Bitvector)

NUMBER_OF_COLUMNS isn't the length/max-length of a column, should be MAX_BLOB_COMMITMENTS_PER_BLOCK

I'd just leave the column_index in all the time, 8 bytes doesn't seem worth worrying about imho

Preference for 1, as the having two containers makes things a bit more annoying as you need to then define when one or the other (or both) are transmitted.

I'd just leave the column_index in all the time, 8 bytes doesn't seem worth worrying about imho

The only reason we would need it is when the subnet != column index, right? 8 bytes is small, but a couple bytes here and a couple bytes there and soon we have a whole packet of redundant data. My preference might be to just remove this field and leave it as future work to define how to derive a column index when the subnet count != column count.

1 may seem better but the other options move the complexity from the type system (and the requisite push to add a new concept to the ssz spec) to other layers of the stack

that said, we have have Option as a non-canonical type for some time now, and it may be time to go ahead and add formal support for it

Including the commitments here is an optimization. We could rely on just receiving them in the block body.

Something worth paying attention to here is also how DataColumnSidedcar evolves in the gloas spec (though these changes will likely pre-date gloas). At the moment it seems like header + inclusion proof will be removed in favor of beacon_block_root only, requiring the beacon block to be seen in advance (though this still will not contain the kzg commitments), which is somewhat natural with epbs.

We could do the same thing here, requiring the block (or a full sidecar) to have been seen before receiving a partial column (and excluding the commitments, since at this point they're still in the beacon block). Similar to what you did below but with beacon_block_root instead of the whole signed_beacon_block_header

class PartialDataColumnSidecar(Container): index: ColumnIndex cells_present_bitmap: Bitlist[MAX_BLOB_COMMITMENTS_PER_BLOCK] partial_column: List[Cell, MAX_BLOB_COMMITMENTS_PER_BLOCK] kzg_proofs: List[KZGProof, MAX_BLOB_COMMITMENTS_PER_BLOCK] beacon_block_root: Root

This is what I am referring to re: gloas specs btw https://github.com/ethereum/consensus-specs/pull/4527/files

raulk

While this mechanistically introduces cell-level deltas, how they precisely integrate into existing PeerDAS processes isn't very clear. How about a section covering the end-to-end behaviour:

When nodes push DataColumnSidecars
When nodes send partial IWANTs
When nodes send partial IHAVEs
When nodes send PartialDataColumnSidecars
Dependency on getBlobsV3

specs/fulu/p2p-interface.md

raulk · 2025-09-05T12:51:07Z

specs/fulu/p2p-interface.md

+To address this tradeoff a Client may choose to eagerly push some (or all) of
+the cells it has. Clients SHOULD only do this when they are reasonably confident
+that a peer does not have the provided cells. For example, a proposer including
+private blobs SHOULD eagerly push the cells corresponding to the private blobs.


Unless we have a backpressure mechanism where a peer can ask an aggressively pushy peer to back off, and downscore if they don't follow through, this can be gamed to spam the peer under the pretence of "noble cause."

I don't see how this is worse than the current status quo. Currently we always eagerly push a whole column unless we receive an IDONTWANT.

This is the same except an IDONTWANT is replaced by an updated partial IHAVE.

There's a subtle difference, though. Before, our eager push was guaranteed to be useful if the peer was lacking any blob. Now, we can push data that's knowingly useless (e.g. an attacker always pushes the cells they know their peer already has, confidently learnt from EL tx announcements from the same IP). At 72 blobs, for a supernode that's only lacking 1 blob, this path exposes it to receiving 18MiB x D of knowingly redundant data, when all it needed was 256KiB (1 row x 128 columns x 2KiB cell size).

Thanks for explanation. You're saying that the status quo at least guarantees some useful information, even if it's a single cell, while partial messages can be used to only transmit redundant information.

I'm not yet convinced this is an interesting attack though. The cost to the attacker is 1:1 relative to the cost of the receiver. There are easier ways to transmit useless information to the peer at a 1:1 cost. Even in your example of 72 blobs with one missing blob, status quo would still receive (18MiB - 256KiB) x D redundant information.

cskiraly · 2025-09-05T15:46:45Z

specs/fulu/p2p-interface.md

+```
+
+For example, if the peer wanted cells at positions 0,3,9, the corresponding
+IWANT bitmap would be: `0000 1001 000 0010`.


What's n in the example?
000 -> 0000 , and also the byte order looks wrong, shouldn't we have MSB-first?

thanks for the 000 -> 0000 correction.

n can be anything from 9 to 16 in this case, but I should just say 12 to be clear in the example.

The byte order looks wrong, shouldn't we have MSB-first?

https://www.online-python.com/ietUIq50Ol

Yeah, still it might worth clarifying in what byte order it goes on wire.

actually isn't that defined by the SSZ encoding for Bitlist?

good point (but I don't know :-) )

github-project-automation bot added this to Lodestar Consensus Devnet Planning Sep 3, 2025

fradamt reviewed Sep 4, 2025

View reviewed changes

MarcoPolo force-pushed the cell-dissemination branch from 2e1e7f1 to 6113cf2 Compare September 5, 2025 02:29

raulk reviewed Sep 5, 2025

View reviewed changes

cskiraly reviewed Sep 5, 2025

View reviewed changes

MarcoPolo force-pushed the cell-dissemination branch 2 times, most recently from 8be8bff to 829184f Compare September 5, 2025 17:07

Welbert5753 approved these changes Sep 6, 2025

View reviewed changes

MarcoPolo force-pushed the cell-dissemination branch from 829184f to 8c70c5b Compare October 1, 2025 21:15

ppopth mentioned this pull request Oct 2, 2025

Add Marco Munizaga from EF P2P team protocolguild/documentation#425

Merged

Add Cell Dissemination via Partial Message Specification

c88bab2

MarcoPolo force-pushed the cell-dissemination branch from 8c70c5b to c88bab2 Compare October 14, 2025 23:57

Add Cell Dissemination via Partial Message Specification #4558

Are you sure you want to change the base?

Add Cell Dissemination via Partial Message Specification #4558

Conversation

MarcoPolo commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fradamt Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fradamt Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fradamt Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raulk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raulk Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

MarcoPolo commented Sep 3, 2025 •

edited

Loading

fradamt Sep 4, 2025 •

edited

Loading

fradamt Sep 5, 2025 •

edited

Loading

fradamt Sep 5, 2025 •

edited

Loading

raulk Sep 5, 2025 •

edited

Loading