Stabilize ChainConfig serde for Human-Readable & Binary Formats #2436

Wollac · 2025-05-13T10:55:48Z

Motivation

This PR refactors Genesis serde logic to fix roundtrip errors and ensure a clear distinction between human-readable (JSON) and binary formats.

Fixes #2435

Solution

This PR implements the following changes:

Updated deserialize_private_key, deserialize_storage_map, and introduced the ttd module. All now use is_human_readable() to ensure symmetric (de)serialization:
- TTD is now serialized as a raw JSON number (using u128) in human-readable contexts and via standard U256 serde for binary formats.
- The human-readable deserialisation was not changed.
deserialize_json_ttd_opt is kept for backward compatibility.
Other optional numeric fields in ChainConfig now also use is_human_readable() to ensure symmetric (de)serialization:
- The human-readable deserialisation still supports number or a "quantity" hex string.
- For non-human-readable formats the default is used for both serialization and deserialization.

With this PR it is now possible to (de)serialize Genesis using binary formats (such as serde_cbor). For JSON, the deserialization behavior remains completely unchanged to preserve backward compatibility, only for serialization it fixes the issue that the terminal_total_difficulty was not serialized as a number.

PR Checklist

Added Tests
Added Documentation
Breaking changes

mattsse · 2025-05-13T10:59:39Z

crates/genesis/src/lib.rs

removing hex support will break existing setups, so we cant remove this

Only for JSON files that contain hex, which neither alloy nor geth ever produce. Do you still think we should support hex deserialization in human-readable input?

Arguably, using alloy-serde::opt also for serialization, would be an even more breaking change.
So the only alternative would be to add a alloy-genesis specific deserializer, which uses default when non-human-readable and alloy_serde::quantity::opt::deserialize when human-readable.

Addressed in 823a16b

mattsse · 2025-05-13T11:00:43Z

crates/genesis/dumpgenesis/hoodi.json

this should remain unchanged to ensure this change is not breaking

I'm not sure I really understand why this would break, but I've reverted this change: d6521f1

This reverts commit 5485320.

mattsse · 2025-05-13T12:31:29Z

crates/serde/src/ttd.rs

+            Some(value) => {
+                // convert into an u128 when possible
+                let number = value.try_into().map_err(ser::Error::custom)?;
+                serializer.serialize_u128(number)


this does not work for json, the entire reason for this workaround is that this does not work for mainnet TTD because does not fit in json number without enabling arb precision, whcih we shoudl not do

It does work with serde_json, and I've also added a tests for that.
Serde has (limited) support for u128, even when the arbitrary precision feature is not enabled. However, there is no support for deserializing u128 inside an untagged enum: serde-rs/serde#2230
And as I understand the comments, the workaround has been implemented to support exactly untagged enums. (I even added a test for this particular case).

If we don't need to consider untagged enums (which are irrelevant for ChainConfig), we could even get rid of the workaround altogether and use u128.

Wollac · 2025-05-14T14:37:31Z

@mattsse I changed it so that the deserialization is now 100% identical to and compatible with the existing version.

mattsse

I'm a bit confused why we need to change this via the deserde impls

because this still keeps only deserialize_with = "deserialize_u64_opt"

which doesn't seem symmetrical to me

wouldn't this still deserialize hex and serialize u64?

mattsse · 2025-05-14T19:15:22Z

crates/genesis/src/lib.rs

+    if deserializer.is_human_readable() {
+        match Option::<String>::deserialize(deserializer)? {
+            Some(ref s) => {
+                if s == "0x" {
+                    return Ok(None);
+                }
+                B256::from_str(s).map(Some).map_err(D::Error::custom)
+            }
+            None => Ok(None),
        }
-        B256::from_str(s).map(Some).map_err(D::Error::custom)
    } else {
-        Ok(None)
+        Option::<B256>::deserialize(deserializer)
+    }


hmm, why we need to change this here, is this because the non human readable impl for b256 is different?

In binary, a B256 will not be serialized as a string, but as 32 bytes. Thus in a roundtrip, the Option::deserialize would fail. It only works for human-readable where it is serialized as a hex string.

mattsse · 2025-05-14T19:45:46Z

oh I think I get it now

Wollac · 2025-05-15T05:39:34Z

I think it would conceptually be cleaner (symmetry and Geth compatibility) to only support parsing JSON numbers and not also hex strings.
However, technically this would be a breaking change for config files in a strange format that used to work before. Thus, I've proposed to serialize to JSON numbers, and support the deserialization of both.

mattsse

okay, I think this makes sense, I only have one last question about removed test cases and one nit

crates/serde/src/ttd.rs

crates/genesis/src/lib.rs

mattsse · 2025-05-22T20:16:22Z

cool, I think this is good now
I'm delaying merging this a bit because we have fusaka genesis changes wip and it'd be easier to rebase this pr

mattsse · 2025-06-04T14:41:46Z

hey @Wollac sorry for the delay, we just merged #2542

mind rebasing this pr and update the new fields as well? then we can include this one

graphite-app · 2025-06-11T08:56:24Z

crates/genesis/src/lib.rs

+
+/// Custom deserialization function for `Option<u64>`.
+///
+/// This function allows it to be deserialized form a number or a "quantity" hex string.


There's a small typo in the docstring: form a number should be from a number.

Suggested change

/// This function allows it to be deserialized form a number or a "quantity" hex string.

/// This function allows it to be deserialized from a number or a "quantity" hex string.

Spotted by Diamond

Is this helpful? React 👍 or 👎 to let us know.

Wollac added 3 commits May 13, 2025 11:57

make serialization symmetric

5fc68e7

add hoodi dumpgenesis

5485320

keep deserialize_json_ttd_opt

fad683e

Wollac requested review from DaniPopes, mattsse, klkvr and yash-atreya as code owners May 13, 2025 10:55

github-project-automation bot added this to Alloy May 13, 2025

mattsse requested changes May 13, 2025

View reviewed changes

github-project-automation bot moved this to In Progress in Alloy May 13, 2025

Wollac added 5 commits May 13, 2025 13:01

fix clippy warning

a848124

support string and float when deserialiting TTD

a7e3847

Revert "add hoodi dumpgenesis"

d6521f1

This reverts commit 5485320.

fix clippy warning

95ed52f

improve documentation

7113751

mattsse requested changes May 13, 2025

View reviewed changes

Wollac added 4 commits May 13, 2025 14:53

improve test

7dd02a1

add missing #[serde(untagged)]

3c528a3

keep backwards compatibility

823a16b

Merge branch 'main' into serialize-genesis

6b7f2a9

Wollac requested a review from mattsse May 14, 2025 14:36

mattsse reviewed May 14, 2025

View reviewed changes

mattsse requested changes May 21, 2025

View reviewed changes

crates/serde/src/ttd.rs Outdated Show resolved Hide resolved

crates/genesis/src/lib.rs Outdated Show resolved Hide resolved

Wollac added 3 commits May 22, 2025 15:56

Merge branch 'main' into serialize-genesis

ada2e77

add accidentally removed tests

575d226

move private functions to end of file

08a6d86

Wollac requested a review from mattsse May 22, 2025 14:00

Wollac added 3 commits June 5, 2025 20:18

Merge branch 'main' into serialize-genesis

9ee6aa4

adapt new fields

fddda2f

Merge branch 'main' into serialize-genesis

39c007d

graphite-app bot reviewed Jun 11, 2025

View reviewed changes

	/// This function allows it to be deserialized form a number or a "quantity" hex string.
	/// This function allows it to be deserialized from a number or a "quantity" hex string.

Stabilize ChainConfig serde for Human-Readable & Binary Formats #2436

Are you sure you want to change the base?

Stabilize ChainConfig serde for Human-Readable & Binary Formats #2436

Uh oh!

Conversation

Wollac commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Solution

PR Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Wollac May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Wollac May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Wollac commented May 14, 2025

Uh oh!

mattsse left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattsse commented May 14, 2025

Uh oh!

Wollac commented May 15, 2025

Uh oh!

mattsse left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mattsse commented May 22, 2025

Uh oh!

mattsse commented Jun 4, 2025

Uh oh!

graphite-app bot Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Wollac commented May 13, 2025 •

edited

Loading

Wollac May 13, 2025 •

edited

Loading

Wollac May 13, 2025 •

edited

Loading