Error in block header serialization using msg pack

[scholtz]

Subject of the issue

I am trying to download a block using msgpack encoding, however decoding fails.

Your environment

4.1.2 algod from Urtho

Steps to reproduce

1. Download block 52243617, format msg pack, headers only

Expected behaviour

I expect json keys to be string not uints.

spt is map[bigint,sptobject], and should be encoded in the msgpack as map[bigint.ToString(),sptobject]

  Expected: "...8A0B9D14BE64AF887B4A373707481A13081A16ECE031D2D00A27463CEB..."
  But was:  "...8A0B9D14BE64AF887B4A3737074810081A16ECE031D2D00A27463CEBB6..."
  -------------------------------------------^

00 is raw 0, A130 is string "0"

I expect the algod follow the standards for encoding.

📚 References:

1. JSON Specification (RFC 8259)
   "An object is an unordered collection of name/value pairs... A name is a string."
   📌 Source: RFC 8259 – The JavaScript Object Notation (JSON) Data Interchange Format

2. MessagePack Specification
   "Map format family stores a key-value pair elements... A key and a value can be any types."
   📌 Source: MessagePack Spec – Map format

However, it also notes:

"If you want to convert MessagePack to JSON, keys must be strings."

3. MessagePack for C# (MessagePack-CSharp) Documentation
   "Map keys should be string for compatibility with JSON."
   📌 Source: MessagePack-CSharp GitHub Wiki

Actual behaviour

I am receiving

81A5626C6F636BDE0014A26269CE008FA891A46561726ECE000354B0A26663CD9858A466656573C420C7FCCDB258F0D4189C2BF8B6D68EE697508642B0AD001F31FCB918C354BA859AA466726163CF000000019A73D62AA367656EAC6D61696E6E65742D76312E30A26768C420C061C4D8FC1DBDDED2D7604BE4568E3F6D041987AC37BDE4B620B5AB39248ADFA27070CE008FF4BDA470726576C4208C32ECE62209FFE1D36921D7FBA4116F29955B18ED53BDD67F636697A0DE2EB4A570726F746FD95968747470733A2F2F6769746875622E636F6D2F616C676F72616E64666F756E646174696F6E2F73706563732F747265652F32333664636331386339633530376437393438313361623736386534363765613432643162346439A3707270C4207E7F342D4DB54C9982C5FE241C0CF3E724DF3609B9D1DEB8664DCD73983EE04FA3726E64CE031D2CA1A6727763616C72CE03211620A3727764C420FEFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFA473656564C4204EDE16B5C5C7E39F092F20659004E61C39462FD74AD488A0B9D14BE64AF887B4A3737074810081A16ECE031D2D00A27463CEBB651B7EA27473CE6888A6EFA374786EC420542FACDE15685204D8FF2FD140EFCCE0537E18C5E8909CDD224130E5D40EDCA2A674786E323536C420931F4E8F27F1AE024641CD3E4AA10E58B116B9A2815F9BA3C9AC499A247231C2

[jannotti]

Algorand uses msgpack. We won't change the protocol because another encoding, JSON, doesn't naturally express the msgpack types. In order to support json in these instances, a decision must be made how to deal with json's more limited types. When algod returns json, algod makes those decisions. For example, it often returns base64 encoded strings since JSON cannot represent arbitrary byte sequence conveniently. But here, I believe algod is only returning msgpack. It's up to clients that receive that msgpack to decode it. What library are you using to decode the msgpack? And more particularly, what library are using to re-encode to JSON? Perhaps it offers some knobs to indicate how you want these sort of maps to be encoded?

…

On Wed, Jul 30, 2025, 5:05 AM Ludovit Scholtz ***@***.***> wrote: *scholtz* created an issue (algorand/go-algorand#6395) <#6395> Subject of the issue I am trying to download a block using msgpack encoding, however decoding fails. Your environment 4.1.2 algod from Urtho Steps to reproduce 1. Download block 52243617, format msg pack, headers only Expected behaviour I expect json keys to be string not uints. spt is map[bigint,sptobject], and should be encoded in the msgpack as map[bigint.ToString(),sptobject] Expected: "...8A0B9D14BE64AF887B4A373707481A13081A16ECE031D2D00A27463CEB..." But was: "...8A0B9D14BE64AF887B4A3737074810081A16ECE031D2D00A27463CEBB6..." -------------------------------------------^ 00 is raw 0, A130 is string "0" I expect the algod follow the standards for encoding. 📚 References: 1. JSON Specification (RFC 8259) "An object is an unordered collection of name/value pairs... A name is a string." 📌 Source: RFC 8259 – The JavaScript Object Notation (JSON) Data Interchange Format <https://datatracker.ietf.org/doc/html/rfc8259#section-4> 2. MessagePack Specification "Map format family stores a key-value pair elements... A key and a value can be any types." 📌 Source: MessagePack Spec – Map format <https://github.com/msgpack/msgpack/blob/master/spec.md#map-format-family> However, it also notes: "If you want to convert MessagePack to JSON, keys must be strings." 3. MessagePack for C# (MessagePack-CSharp) Documentation "Map keys should be string for compatibility with JSON." 📌 Source: MessagePack-CSharp GitHub Wiki <https://github.com/neuecc/MessagePack-CSharp/wiki> Actual behaviour I am receiving 81A5626C6F636BDE0014A26269CE008FA891A46561726ECE000354B0A26663CD9858A466656573C420C7FCCDB258F0D4189C2BF8B6D68EE697508642B0AD001F31FCB918C354BA859AA466726163CF000000019A73D62AA367656EAC6D61696E6E65742D76312E30A26768C420C061C4D8FC1DBDDED2D7604BE4568E3F6D041987AC37BDE4B620B5AB39248ADFA27070CE008FF4BDA470726576C4208C32ECE62209FFE1D36921D7FBA4116F29955B18ED53BDD67F636697A0DE2EB4A570726F746FD95968747470733A2F2F6769746875622E636F6D2F616C676F72616E64666F756E646174696F6E2F73706563732F747265652F32333664636331386339633530376437393438313361623736386534363765613432643162346439A3707270C4207E7F342D4DB54C9982C5FE241C0CF3E724DF3609B9D1DEB8664DCD73983EE04FA3726E64CE031D2CA1A6727763616C72CE03211620A3727764C420FEFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFA473656564C4204EDE16B5C5C7E39F092F20659004E61C39462FD74AD488A0B9D14BE64AF887B4A3737074810081A16ECE031D2D00A27463CEBB651B7EA27473CE6888A6EFA374786EC420542FACDE15685204D8FF2FD140EFCCE0537E18C5E8909CDD224130E5D40EDCA2A674786E323536C420931F4E8F27F1AE024641CD3E4AA10E58B116B9A2815F9BA3C9AC499A247231C2 — Reply to this email directly, view it on GitHub <#6395>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADL7T2KXEMDNJNPNOZWC533LCDFRAVCNFSM6AAAAACCWIREOKVHI2DSMVQWIX3LMV43ASLTON2WKOZTGI3TMMRVGUYTSOI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[scholtz]

in the dotnet-algorand-sdk is used Newtonsoft lib which is using internally json, and does not allow parsing or creating json props without the proper json validation

[scholtz]

there are also issues with other properties like prev .. from msgpack its returned as byte array, but from json its returned as blk-base32 encoded value .. the decoding can be done normally depending on the type what comes, but further encoding is challanging as the converter does not usually knows if it should convert to msgpack or json

[scholtz]

is there some formal definition of what can be returned in which encoding from algod?

for example https://raw.githubusercontent.com/algorand/go-algorand/088f89dd02487a06497a79518c549aae8a3134ee/daemon/algod/api/algod.oas2.json does not have the model data for Block.. it just covers it as object and no type safety in other languages can be done there

    "BlockResponse": {
      "description": "Encoded block object.",
      "schema": {
        "type": "object",
        "required": [
          "block"
        ],
        "properties": {
          "block": {
            "description": "Block header data.",
            "type": "object",
            "x-algorand-format": "BlockHeader"
          },
          "cert": {
            "description": "Optional certificate object. This is only included when the format is set to message pack.",
            "type": "object",
            "x-algorand-format": "BlockCertificate"
          }
        }
      }
    },

[jannotti]

Is dotnet-algorand-sdk requesting the block as msgpack and trying to convert it to JSON? If so, that is a bug there, since msgpack cannot be (generally) converted to JSON.

Or is it requesting JSON from algod? I thought algod did not response to JSON requests for blocks. But looking at the code, it seems to do something. I doubt it does a great job (the clue is exactly what you found, the OpenAPI spec does not describe the fields). It probably allows Go to try to convert the Block to JSON. I do not know what comes out.

It would be a fairly big piece of work to return the block as JSON without losing information, including trying to decide how to encode these structures that JSON has difficulty with. I would recommend decoding the msgpack into a datastructure (probably a hashtable that doesn't impose any typing on its keys or values) directly and operating on that.

[scholtz]

dotnet algo sdk has option to allow get the block using msgpack which does not work because of the issue in the first message, or using the json.. default is the json, but i thought it would be better for the algod cpu just use the msgpack as i believe it is stored that way..

Have you considered changing the msgpack to something else? gRPC or Fast protocol would cut the network transfer and storage costs by at least 20% i believe.. it might also solve #6128
Clear definition of types what algod produces, so that it may be used in different languages is must have for the blockchain i believe..