What's the state of dcutr?

[MatthiasvB]

Hi everybody! I was wondering if you know the state of dcutr in rust-libp2p? The reason is that it doesn't work very well for me, but I'm not sure if that's because I'm using it wrong, have bad luck with my network environments, or it really just doesn't work well.

I'm aware that a few years back, there has been some effort to evaluate the performance of dcutr, for example by means of punchr. Talks I found on the Internet spoke of hole punching success rates between 90 and 60 %, depending on the network, VPN, etc.

My own success rates, tested by now on three different (semi)private networks, are more in between 1 and 10 % success rates. Is that just me, or do you get similar metrics?

I'm not actually doing much right now. More or less just walking the DHT, connecting to some random (10) relays, and having dcutr active, which does its thing pretty independently.

Any time I do get a dcutr Event, I update statistics of the form

Total successful: 2
Successful peers 2
Total failed: 200
Failed peers 160
Peers with successes and failures: 0

(these are particularly bad)

What I read from that is that the perfomance is, most interestingly, very consistent per peer. If it failed once, it'll fail again. If it's successful once, it'll be successful again (not seen in above stats but in others I got).

Never have I ever had a peer fail and then succeed, or the other way around.

This makes me wonder: Is there room for improvement on that protocol? Are there any plans to tackle that again? At the moment, assuming this is typical performance, it wouldn't be useful for any real world scenario at all.

There also the very wide open gaping hole between the dcutr spec and this popular article on NAT traversal that seems to explain in great detail why dcutr isn't, in fact, adequate.

Should I perhaps move away from dcutr and explore other options, I wonder. I haven't looked at the state of WebRTC in libp2p. Is that going to fix things, or is it a separate topic. Is direct connection between NATed peers high up on the prio list for future development at all?

Really looking forward to hearing your perspectives :)

[drHuangMHT]

Personally I don't really expect dcutr to work very well. I'm hoping for 10% success rate at best but I don't really have the data. I don't think WebRTC will be of any help because it still depends on the physical network, and we can't really do anything about that. Maybe with IPv6?

[elenaf9]

When the punchr project was first developed I ran a rust-client and had a good success rate, roughly matching what was reported in the campaign.
I don't think anything major was changed in the dcutr protocol since then, but I haven't tried it lately and currently don't have a good setup for testing. What type of router/ NATs does your networks have @MatthiasvB? Do you noticed a difference between using QUIC or TCP transport?

[dariusc93]

In my experience, it has been a bit of a hit or miss at times depending on the environment. In a good environment though, it was worked pretty well and had more of a success, although this was using TCP. With QUIC, I would say this can be the same but noticed some flakiness in the earlier days of quic and not sure about the success rate right now at this moment.

[MatthiasvB]

Actually, I hadn't had quic enabled, only TCP. Enabling quic raised the successrate significantly (this is from running multiple days):

Total successful: 6687
Successful peers 811
Total failed: 9980
Failed peers 1013
Peers with successes and failures: 36

So, at the moment, that's a 40% success rate per attempt and a 44% success rate per peer.

Again, a little sad that this "magic" improvement happens without any prior indication that missing quic may be part of the problem.

In the meantime, at least for my home network, I have figured out that it uses a "easy NAT", so no exceptional difficulties expected purely from the networking situation.

I'll probably focus on local network for now to get my application logic working and then see if I find the time to get further into the dcutr debugging

[jxs]

Hi Matthias!

As Elena I also run the rust-client for some time with the success rates matching the reported ones.
We also have the hole punch interop tests running on our CI and currently passing. You can take a look at their implementation here.

So as to make this discussion helpful for future reference

There also the very wide open gaping hole between the dcutr spec and this popular article on NAT traversal that seems to explain in great detail why dcutr isn't, in fact, adequate.

can you elaborate on this? DCUtR was thought with the decentralized nature of libp2p in mind, as you can read more about here. But the fundamental concepts are similar to what tailscale presents on their article.

@drHuangMHT can you elaborate on

Personally I don't really expect dcutr to work very well.

why do you not expect it to work very well?

And @dariusc93 can you elaborate also on

In my experience, it has been a bit of a hit or miss at times depending on the environment. In a good environment though, it was worked pretty well and had more of a success, although this was using TCP.

what in your usage experience is a good environment?

Thank you all!

[drHuangMHT]

@drHuangMHT can you elaborate on

Personally I don't really expect dcutr to work very well.

why do you not expect it to work very well?

That's based on my experience with networks in my country so I don't think it is a very useful piece of information.
But anyway, I've seen zero success when hole-punching with around 10 people, using dcutr and quic over IPv4 connections. That's probably because the ISPs are using symmetric NAT, which is the worst type of NAT for hole-punching.
Maybe it works better in other parts of the world, but not where I live, and it is terrible. That's also why I said it heavily depends on the actual setup, like how the ISPs deal with NAT.

[dariusc93]

And @dariusc93 can you elaborate also on

In my experience, it has been a bit of a hit or miss at times depending on the environment. In a good environment though, it was worked pretty well and had more of a success, although this was using TCP.

what in your usage experience is a good environment?

During the testing and use, the router or modem is one factor. If the NAT is strict, close or configured to block such use cases or the firewall is extremely strict, DCUtR will almost always fail, causing it to fall back to using the relay. Same for double nat, symmetric nat, etc. Another factor is the system firewall on the computer (which also does sometimes impact go-libp2p). Besides that, QUIC sometimes have higher success rate compared to TCP, but QUIC may not always be desirable for some use cases or may have existing issues that may be questionable (ie higher memory usage compared to tcp, especially when one is walking dht, though not sure how it is now these days so ymmv)

[jxs]

Thanks for elaborating folks. As you both referred, hole punching success may be influence by the type of NAT's prevalent on a certain geographic area.
Symmetric NAT's which are a subset of endpoind dependant NAT's are described both in the libp2p blogpost:

There are situations in which hole punching will not work, most notably when one of the nodes is behind a symmetric NAT. In such cases, nodes can instead explicitly add port mappings, either manually or by using UPnP (opens new window).

and the hole punching paper:

NATs using Endpoint-Independent Mapping lend themselves to reliable hole punching. Nodes can rely on their external address
to be stable, and advertise this address to other nodes. On the other hand, there is no reliable way to punch through NATs using Endpoint-Dependent Mappings; the difficulty lies in predicting the port the NAT will assign, which becomes exponentially harder with concurrent connection attempts.

but this is transversal to hole punching in general and not particular to DCUtR as the tailscale article posted by Matthias attests, when direct connection is impossible tailscale resorts to using relayed connections, which is also possible with circuit relay on libp2p world as the libp2p post also states

As a last resort, nodes can leverage external relay (opens new window)nodes.

This to challenge that

There also the very wide open gaping hole between the dcutr spec and this popular article on NAT traversal that seems to explain in great detail why dcutr isn't, in fact, adequate.

is not true, hole punching is adequate for endpoint independent mapping NATs and DCUtR protocol works successfully on that scenario, as the punchr experiment and results showcased.
On endpoint dependent mapping NATs relaying is the consensual solution and the combination of these both techniques should address all the connectivity cases.