Public node peering limits

[yakamok]

add a public node specific option to limit peering to prevent over saturation of that node.
The public node could when struggling have the chance to manually set a limit to prevent new peering to preserve stability of the network.

maybe this could be set as:
peer-limit = 40

if there is already more than the limit, the newest could be disconnected.

[Arceliar]

I see a few possible variations on this, depending on exactly what the use case is.

1. Limit on the maximum number of peers. If it grows too large, you close any new connection attempts.
2. Limit on the maximum number of incoming connections, but don't limit outgoing connections.
3. Limit on the maximum number of incoming manual connections (ignore connections from link-local discovery).

In all cases, we need to figure out exactly what we do to block a connection. In the case of 1, we can stop dialing out easily enough, but in all cases other nodes can still attempt to dial into us. We'd need to Close() the connection, but they can and will attempt to reconnect.

There's some weird edge cases too. If two public nodes are connected to eachother, get DCed, allow new peers to connect, and then try to reconnect to eachother again, that could block them from connecting. This could lead to netspits and other weirdness, so I'd like to think my way around that problem before attempting anything.

There's already a (admittedly indirect) way to deal with this. Public nodes could, in principle, use the AllowedEncryptionPublicKeys setting (configurable over the API for a running instance) to add a limited number of peers.

For example, in addition to copying/pasting a connection string into their Peers list, users may need to visit some webpage and submit their key. Then behind the scene, the page would add the allowed keys over the API and maybe save them to the persistent file (or add all keys from some external database whenever ygg starts up). While that's arguably a better solution in the long run (you could e.g. require users to register, and then use that to disconnect specific abusive nodes), that's also a lot more work to set up.

[neilalexander]

If we were able to use SO_REUSEPORT then it might be possible to compare an incoming TCP connection against the list of configured Peers, such that they would be exempt from any limits. That would be a good method of avoiding net split problems with statically configured peers in the event of a dropped connection. However, right now that isn't possible since the source ports on incoming connections are randomised.

That isn't to say that we couldn't add some kind of behaviour where anything listed in AllowedEncryptionPublicKeys are exempt from connection limits, but I guess that is further on in the negotiation phase and by that point is a switch port is already assigned?

I don't know if it makes any sense to limit anything from link-local discovery, as that is already easy to disable and those devices are very likely to be on a fast link anyway.

[Arceliar]

For the specific example of two public nodes that need to connect, we could only pay attention to the host part of the socket's RemoteAddr(), and allow it to bypass the connection limit if it's already in the Peers config, or something like that. I need to think about what other edge cases could exist, where we may accidentally prevent two nodes from connecting when they really should connect.

Also, even if public nodes don't allow new connections after some point, I would expect them to still have a disproportional amount of traffic get routed through them, just based on their tendency to cluster near the center of the network. So I'm not sure it will really reduce the load on those nodes, and it may force traffic to take a longer route than necessary (increasing load on public nodes). Need to figure out how likely that is to be a problem first.

[EternityForest]

As a related note, maybe preventing public peers from ever getting overloaded in the first place is worth investigating.

I like the idea I've seen discussed of letting people add themselves as public nodes in the DHT, so you can automatically find them.

But if one is already using the internet as opposed to long range 5GHZ links or something, there's not much reason to actually use the mesh.

An option to reveal your IP to the people you want to exchange traffic with, and temporarily peer with the node you're connecting to would take so much load off the mesh.

It would require a some extra code to have "non-routing peers" so you could talk directly without necessarily expecting them to route your traffic.

But I suspect that 20-95% of the load can be taked off a mesh by taking advantage of direct traditional internet connections, while still giving those users the benefits of encryption, key-derived addresses, and a mesh fallback if their internet goes out.

Especially if intermediate nodes had the ability to do this kind of direct connection, and you could cache the IP addresses of the people you were communicating with(Hopefully without breaking support for read only filesystems).

That would allow you to talk to people you've connected to recently without having to send packets to anyone at all except them, which is great for reliability.

[Arceliar]

@EternityForest I'd definitely rather spend my time reducing the need for and load on public nodes, rather than adding more configuration knobs that the pub operators need to tune themselves. Auto-peering over the internet would be one way to do that, but that particular fruit just doesn't hang low enough for us to reach it right now. It requires a bunch of little things to be in place first, some of which are useful on their own, but other's are not, before we could do a form of auto-peering that's good enough to be useful, without introducing some sort of perverse incentive to always have an internet connection (scalable routing doesn't matter if you encourage everyone to bypass, so I we need to field test that part on a larger network first, before I'm confident enough that it's OK to want a way to bypass it when it's inconvenient).

Also, supposedly, most of the cost of providing an internet connection is in the "last mile" infrastructure between the ISP and the point of service. By that logic, if most public nodes are either run by some organization with a particularly good connection (e.g. a university CS/IT department) or running by users with a VPS in a datacenter, then they the cost of routing traffic between them should be pretty low. I'd prefer to focus my attention on making sure the last mile meshed part works well in practice first, since that's the part that ultimately determines whether or not you have a connection at all, and optimize the over-the-internet part later.

[Revertron]

May be it will be easier to limit the speed, not connectoin counts?

[akhilman]

I have two nodes and two distinct physical routes between them. The first route always has the maximum available bandwidth, the second route always has the lower ping time, and suffers from oversaturation and reduced bandwidth. Yggdrasil always prefers the route with the lowest ping time, even if it cannot handle the load.

It would be nice if Yggdrasil could balance the load between multiple available physical routes with the lowest ping time, taking into account the transfer rate of those routes. Then nodes will be able to throttle transit traffic without causing bandwidth loss to endpoints. The endpoint would simply use another route if a transit node could not or would not handle the requested traffic rate.

[neilalexander]

@akhilman For now, if it's two peerings between the same two nodes, then you can append ?priority=N to the peering strings to prioritise one over the other. Lower numbers are preferred.

Likewise if it's MulticastInterfaces for more than one interface, then give each interface its own block and then set Priority: the same way for each.