Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Bus performance #48

Open
aramperes opened this issue Jan 8, 2023 · 0 comments
Open

Improve Bus performance #48

aramperes opened this issue Jan 8, 2023 · 0 comments
Labels
enhancement New feature or request
Milestone

Comments

@aramperes
Copy link
Owner

aramperes commented Jan 8, 2023

Related to #46, #29

The Bus is the core of how onetun passes events around internally to different components, such as when data is transmitted, or a connection is closed.

There are some advantages to this design, mainly code readability and extendability. However, I realize that the implementation has important performance drawbacks.

The Bus is implemented using a tokio::sync::broadcast queue, which allows for multiple consumers and producers. However, some of the critical onetun events only need to reach 1-2 listeners, yet each Event is cloned for all receivers (even if dropped/ignored).

Here are the list of Bus endpoints in onetun:

  • tcp_tunnel: each local connection has an endpoint
  • udp_tunnel: each UDP port-forward server has an endpoint
  • tcp_virtual: there is only 0-1 TCP virtual interface with 1 endpoint
  • udp_virtual: there is only 0-1 UDP virtual interface with 1 endpoint
  • device: the virtual layer4 smoltcp Device implementation bridges data between tcp/udp_virtual and wg, it has 1 endpoint. There is 0-1 TCP device and 0-1 UDP device
  • wg: has 1 endpoint for the consumer (decapsulation) and 1 for the producer (encapsulation)
  • pcap: (optional) has 1 endpoint for recording inbound and outbound data

Here is the list of event channels in onetun (producer -> Event -> consumer):

  • tcp_tunnel -> ClientConnectionInitiated(PortForwardConfig, VirtualPort) -> tcp_virtual: When a client starts a TCP connection to the local port, this event creates a new client socket in the virtual interface (binding with the peer IP and the virtual port, i.e. this is the advertised port to remote peers). This a critical but quiet channel (only when new connections are created).
  • tcp_tunnel -> ClientConnectionDropped(VirtualPort) -> tcp_virtual: This event is dispatched when the tcp_tunnel has closed the connection with the local client (can be caused by an error, or if the local client terminated the connection, or if the remote peer has terminated the connection)
  • tcp_virtual -> ClientConnectionDropped(VirtualPort) -> tcp_tunnel: Similar to the one before, this is dispatched when communication with the remote peer has concluded to notify the tcp_tunnel to forcefully close the connection with the local client. Note: the tcp_tunnel will "reply" with the same event as the connection with the local client is closed (see previous).
  • tcp_tunnel -> LocalData(PortForwardConfig, VirtualPort, Bytes) -> tcp_virtual: Whenever data is polled from the local client, it is sent to the virtual interface. The virtual interface will enqueue this data and immediately send it off on the loop (poll delay is set to 0). This is a performance-critical channel and a major problem, since each packet is cloned for each endpoint.
  • udp_tunnel -> LocalData(PortForwardConfig, VirtualPort, Bytes) -> udp_virtual: Whenever data is polled from the local UDP socket, it is sent to the virtual interface. This is also a performance-critical channel.
  • tcp_virtual -> RemoteData(VirtualPort, Bytes) -> tcp_tunnel: Forwards data from the remote to the local client. This is also a performance-critical channel
  • udp_virtual -> RemoteData(VirtualPort, Bytes) -> udp_tunnel: See above. Also performance-critical
  • wg -> InboundInternetPacket(PortProtocol, Bytes) -> device: Forwards decrypted IP packets to the device, which is then propagated to the virtual interface. This is also a performance-critical channel
  • wg -> InboundInternetPacket(PortProtocol, Bytes) -> pcap: Used to record pcap files. This is a heavy and chatty channel but low-priority; the fact that each packet must wait for the pcap file to flush is bad though.
  • device -> OutboundInternetPacket(Bytes) -> wg: Forwards IP packets to boringtun (which encrypts and sends to wg). This is also a performance-critical channel
  • device -> OutboundInternetPacket(Bytes) -> pcap: Used to record pcap files. This is a heavy and chatty channel but low-priority; the fact that each packet must wait for the pcap file to flush is bad though.
  • device -> VirtualDeviceFed(PortProtocol) -> tcp_virtual: This is how the device informs the tcp_virtual interface that new data has arrived and is ready to be processed immediately (triggers a poll). This is performance-critical
  • device -> VirtualDeviceFed(PortProtocol) -> udp_virtual: This is how the device informs the tcp_virtual interface that new data has arrived and is ready to be processed immediately (triggers a poll). This is performance-critical

image

@aramperes aramperes added the enhancement New feature or request label Jan 8, 2023
@aramperes aramperes added this to the v1.0 milestone Jan 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant