feat: share protobuf definitions in sub-module with nwaku

[adklempner]

Description

Currently, we manually make updates to the protobuf definitions. This introduces potential incompatibility errors with nwaku (or any other project that uses the protocol).

User Story

As a developer, I want to be able to sync changes to the protobuf definitions without it affecting the repositories code-generation tools.

Proposed Solution / Feature Design

Have a shared repo for just protobuf files as a sub-module in js-waku and nwaku. Use a separate branch for each that can be manually synced using git. Both groups of maintainers can verify that the projects are using the same exact set of protobuf files by using the same commit hash (and vice versa, detect when the protobuf files are out of sync).

Optional: Diagram or Draft of Design

Notes

[fryorcraken]

Yes, this repo was created: https://github.com/waku-org/waku-proto

The issue is that nwaku does not use a protobuf library to generate types from definitions.

I believe such a library exists now, and is used by the new project (nim-sds?).

This specific user story is somethign we would need to properly plan in the roadmap.

1. Use protobuf code generator in nwaku
2. Revive waku-org/waku-proto and integrate/import it in js-waku and nwaku

cc @Ivansete-status @jm-clius @NagyZoltanPeter

[fryorcraken]

I think we could look into the Chat SDK demonstrating how this can be done with nim, and then later JS (aka having protobuf definitions in its own separate repo usable by several implementations) cc @jazzz @plopezlpz

[NagyZoltanPeter]

As I could check nim-sds is currently utilizing the same approach as nwaku - miniprotobuf. Which involves manual work for de/marshaling. I aware of two libraries: - https://github.com/PMunch/protobuf-nim This is in initial phase, but promises exactly what we would need, eats protobuf definition to generate Nim code. - https://github.com/status-im/nim-protobuf-serialization This one is more mature, but requires special Nim definition of structure, but after all is capable to encode/decode from/to protobuf byte sequence without manual work. Seems noon if them is battle tested yet.

…

On Fri, Jun 13, 2025, 03:04 fryorcraken ***@***.***> wrote: *fryorcraken* left a comment (waku-org/js-waku#2410) <#2410 (comment)> I think we could look into the Chat SDK demonstrating how this can be done with nim, and then later JS (aka having protobuf definitions in its own separate repo usable by several implementations) cc @jazzz <https://github.com/jazzz> @plopezlpz <https://github.com/plopezlpz> — Reply to this email directly, view it on GitHub <#2410 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3FU54NXEWCHV6ENEBITILD3DIPQNAVCNFSM6AAAAAB7GJ2TGKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSNRYGY3TIMZXGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[fryorcraken]

@NagyZoltanPeter great and thanks, very interesting.

I would suggest that we start to battle test one of those. Best maybe to do that in consultation with other nim developers in the org, so we can all focus towards the same goal. For example, does any Logos/Nimbus project use either?

While it may be best that we don't battle test with chat sdk per se, because we want ot move fast. Something like the segmentation library could be a great place to start as I expected the proto definition to be very simple. cc @plopezlpz @kaichaosun

[jm-clius]

Yes, repeating a lot of what has already been said:

• I know of no active Nim component that use any of the protobuf serialization libraries
• "battle testing" any of these typegen libraries should be relatively simple, by just comparing serialized bytes to those created by js-waku.
• the proto repo is a reasonable idea, but it would constantly become outdated and would diverge from the latest spec and nwaku implementation. It's best that we keep seeing the specs as the central source of truth and add as much automation as possible to keep the associated proto files up to date.

Happy to help review/revive the waku-proto repo.

[fryorcraken]

• the proto repo is a reasonable idea, but it would constantly become outdated and would diverge from the latest spec and nwaku implementation. It's best that we keep seeing the specs as the central source of truth and add as much automation as possible to keep the associated proto files up to date.

I think the end goal is that all (nim and JS) implementations of Waku protocol uses protobuf serialization libraries. Meaning waku-proto can be imported at development time in any implementations, which in terms means that whoever changes the protobuf definition in the spec, would change it in the code as part of the change they are doing.

What we don't want is to maintain 3 locations for protobuf:

1. the specs
2. the waku-proto repo
3. the actual protocol implementation

We can start simple by:

1. Starting using library generation:

• I believe js-waku already does that. can we import waku-proto in js-waku?
• @NagyZoltanPeter recommends using https://github.com/status-im/nim-protobuf-serialization can we try that for the next easy lib (aka segmentation cc @plopezlpz @kaichaosun)

2. Only update the protos in waku-proto that are used for generation - could even remove anything that is not used by generation (waku message, etc), and only have protos that are used for generation (eg segmentation
3. Slowly migrate existing codes to protobuf-serialization, for simple to complex (eg segmentation, then sds, chat sdk, and finally waku core protocols).

[kaichaosun]

The protobuf definition is mostly from specification, our research team has already put a lot of effort in the correctness of the specification, the protobuf definition better to be included in the spec's repo and we should adding tests to cover the changes of such definitions.

We probably needs to depends specific programming language if we add such tests, I propose to use Rust and slowly it would be great to fully replace our tech stack to be mainly Rust based, as Rust is the only option that logos accepted and has well community support.

Reuse the protobuf definition is good, but is not a "must" for me, because the protocol spec changes usually include broken changes, and manually include or duplicate such changes ensures the code owners in each implementation are aware of such changes' detail. It surely has some duplicated work, but looks like a rough consensus can be build with such duplication, which may bring extra value when Waku goes to wide public.

[NagyZoltanPeter]

The protobuf definition is mostly from specification, our research team has already put a lot of effort in the correctness of the specification, the protobuf definition better to be included in the spec's repo and we should adding tests to cover the changes of such definitions.

We probably needs to depends specific programming language if we add such tests, I propose to use Rust and slowly it would be great to fully replace our tech stack to be mainly Rust based, as Rust is the only option that logos accepted and has well community support.

Reuse the protobuf definition is good, but is not a "must" for me, because the protocol spec changes usually include broken changes, and manually include or duplicate such changes ensures the code owners in each implementation are aware of such changes' detail. It surely has some duplicated work, but looks like a rough consensus can be build with such duplication, which may bring extra value when Waku goes to wide public.

Hm, I tend to disagree here a bit.
Due protocol definitions, live as protobuf structure and tightly coupled with the spec/rfc text itself, I see it as a contract for each party regardless of platform or implementation.
So, having a single source of truth is a value for me.
I believe if spec and implementation could utilize the same - versioned - definitions can eliminate not just dup work but misunderstandings. Duplicated work for me is always a point of failure where misalignment starts to happen - that was the source of archiving the previous proto repository.
Question: Can the spec embed/reference the proto definition from an independent source?

[kaichaosun]

having a single source of truth is a value for me.

I fully agree, that's why I think the protobuf definition should be live in the spec repo and test covered.

Can the spec embed/reference the proto definition from an independent source?

It's unnecessary to referent an independent source since protobuf definition itself is the key part of one protocol.

Duplicated work for me is always a point of failure where misalignment starts to happen

I feel that it's hard to expect all implementations to follow the same approach, especially with third-party implementation involved. If we use the spec as the single source of truth, any unwanted duplicate work won’t be too harmful.

[Ivansete-status]

This repo was archived at the beginning of this year because no one used it: https://github.com/waku-org/waku-proto

@adklempner - isn't it enough with the interop tests to validate there's consistency between js-waku and nwaku?