Improve bandwidth

[aramperes]

I tested some sftp tunneling on some big files and found that each connection is capped at around 2MB/s. Naturally this depends on the latency and bandwidth with the WireGuard router, but with the official kernel implementation I get around 20MB/s for the same endpoint.

$ sftp [email protected]
# 20MB/s

$ sftp -P 2222 [email protected]
# onetun: 2.4MB/s

Parity with the kernel is not a goal since userspace will be a bit slower, but I would like to aim for something like ~50% instead of ~10%.

[zonyitoo]

Hello @aramperes , I may have encountered the same issue with smoltcp which have a lot smaller bandwidth than system network stack. Do you think the key problem is in smoltcp's design or there are any possible improvements could be done in application?

[aramperes]

@zonyitoo A few thoughts:

• According to smoltcp's benchmarks, it should be able to handle multiple Gbps, so I doubt the core libary is the main issue
• Tokio's design clashing with smoltcp might be however; the Device implementation needs to read from a local buffer, and so there is some locking in onetun as well:

onetun/src/virtual_device.rs

Lines 28 to 41 in 648154b

	tokio::spawn(async move {
	loop {
	match bus_endpoint.recv().await {
	Event::InboundInternetPacket(ip_proto, data) if ip_proto == protocol => {
	let mut queue = process_queue
	.lock()
	.expect("Failed to acquire process queue lock");
	queue.push_back(data);
	bus_endpoint.send(Event::VirtualDeviceFed(ip_proto));
	}
	_ => {}
	}
	}
	});

• The Bus design in onetun was to simplify the code and to make it more extendable (for example for pcap, reverse tunneling, etc.). Since every consumer has to read each message before going to the next one, I'm sure targeted channels between the components would be way more efficient. That's how I wrote onetun originally; however the amount of channels needed between the different parts was getting out of hand. I may tweak the Bus design to allow for direct connections with that in mind.

[jkcoxson]

Would this benefit from removing Tokio completely and switching to crossbeam? That should keep the architecture intact, but I don’t know how much that would improve performance.

Alternatively, crossfire should be an almost drop in replacement for Tokio’s broadcast while not having to remove the async runtime. It boasts higher speeds on its readme compared to even Tokio’s single consumer channels.

[aramperes]

Moving away from Tokio/async may indeed be an option, however I would need to see clear benefits/benchmarks before making that decision. Maybe it's something worth testing in a branch. Either way, I don't think this would solve the locking issue in the Device implementation, which I suspect is the main bottleneck.

crossfire doesn't seem maintained enough for me to make the switch, and hasn't been udpated past Tokio 0.2.