Skip to content

Flow and Congestion Control

Jump to bottom
Martin Thompson edited this page Jul 20, 2020 · 27 revisions

Flow Control

Aeron uses flow control to keep publishers from sending too fast, ensuring that subscribers do not get overrun. To this end, if a publisher attempts to send data that is too far ahead of where subscribers are consuming data, that publisher's Publication.offer or Publication.tryClaim operation will not be allowed to succeed, thus providing back-pressure. This is a temporary condition and the publisher should attempt the offer again.

From a user's perspective, flow control is based on the subscriber's rate of consumption and the publisher's rate of publication. However, internally, flow control is more complex. In this section, we will try to discuss the intricacies of flow and congestion control/avoidance.

Flow and congestion control in Aeron are operational on a per Stream basis. The operation of one stream SHOULD NOT affect the operation of another.

Terminology

For background, please see Transport Protocol Specification for basic Aeron terminology. In addition, there is some terminology applied for flow and congestion control.

  • Consumption Point: The byte offset within a stream that has been delivered to a consuming entity, such as a driver publication or client application Subscription. Delivery to the application DOES NOT imply successful or complete processing by the application.
  • Status Message (SM): A special Aeron frame sent from subscriptions back to publication to inform publication of consumption state and reception window.
  • Receiver Window: The window of viable data buffer advertised by a Receiver in Status Messages.
  • Flow Control Strategy: The strategy that a driver publication employs in terms of how it treats SMs from multiple subscriptions.
  • Publication Term Window Length: The amount of window, or buffer, the client application publication(s) has/have to exceed the driver publication. In other words, how much additional buffering space the client application can be ahead of the driver when sending data.

Design Methodology

Low latency applications typically should not leave data in buffers for very long. Hence, the design of Aeron's flow control is to serve as a means to buffer bursts of activity in a safe manner for very high-throughput rates and to provide back-pressure when that buffering is at, or near, exhaustion. To this end, the buffering in use by Aeron is much larger than is typical for some transports that assume file transfer like semantics. However, this is configurable.

Unicast Operation

There are three points of flow control in an Aeron stream.

  1. From client application Publication to driver based Sender: An application is allowed to succeed a call to offer as long as it will not exceed the consumption point of the driver publication plus the Publication Term Window Length. The Publication Term Window Length is set by the media driver and can be controlled via the property, aeron.publication.term.window.length. When not set, a default value of 1/2 the term window length is used.
  2. From Sender to Receiver: A Sender is allowed to immediately send data frames if doing so will not exceed the Consumption Term Offset plus Receiver Window set by the Flow Control Strategy from SM information from the subscriptions.
  3. From driver based Receiver to client application Subscription: A Receiver is allowed to receive data and buffer it if the data lies within the last sent SM Consumption Term Offset plus the Receiver Window. The Receiver Window defaults to 128KB and may be controlled by the aeron.rcv.initial.window.length property, or by the congestion control strategy. No matter what it is set to, the Receiver Window is not allowed to be more than 1/2 the term window length.

From an end-to-end perspective, a publisher in a client application may be ahead of a subscriber by

Publication Term Window Length + Receiver Window

Multicast and Multi-Destination Operation

Multicast operation subsumes all of unicast operation. In addition, the handling of multiple subscriptions by driver publications is added. These are decoupled via a Flow Control Strategy.

Flow Control Strategies

Flow Control Strategy is concerned with how multiple driver receivers feedback, via SMs, is treated to consolidate a single driver publication view of the receivers. By default, Aeron uses a maximum (max) flow control strategy, i.e. the highest known consumption point from all receivers is used. A consequence of this default strategy is that if a slow receiver falls behind, it will eventually stop processing data, let the image go unavailable, then start again at the latest point in the stream.

Custom flow control strategies are supported for both unicast and multicast operation.

Custom Flow Control Strategies

The flow control strategy used by the driver Sender is set via the system properties, aeron.unicast.FlowControl.supplier and aeron.multicast.FlowControl.supplier for unicast and multicast/multi-destination channels respectively.

The suppliers are designed so that it is possible to assign different strategies based on channel and/or stream Id. However, convenient suppliers are provided for common strategies that can be applied to all instances.

  1. MinMulticastFlowControlSupplier: Track flow control to minimum (min) of all receivers. Slowest receiver sets the pace.
  2. MaxMulticastFlowControlSupplier: Track flow control to maximum (max) of all receivers. Fastest receiver sets the pace.
  3. TaggedMulticastFlowControlSupplier: Track flow control to minimum of tagged (tagged) receivers. Slowest tagged receiver sets the pace.

Beyond the suppliers, it is possible to customise a given FlowControl strategy as befits the application semantics desired by implementing the FlowControl interface. FlowControl strategy can be set via aeron.unicast.flow.control.strategy and aeron.multicast.flow.control.strategy properties respectively.

For more information, see the relevant classes and/or interfaces documentation.

Connectivity

Flow control strategies also have the ability to influence the connectivity of the Publication. The min and tagged strategies include support for this (max does not). It is possible to specify a minimum group size for the strategies and until that number of receivers are connected the Publication will indicate that it is not connected. This is orthogonal to the requirement of at least one Subscription to indicate connectivity, therefore the default value for the group minimum size is 0.

Note: If the ssc=true (Spies Simulate Connection) property is set for a publication the publication will be connected regardless of receiver group size if a spy subscription is connected.

Status Message Generation

SMs are generated by driver subscriptions and sent periodically back to driver publications. They include the following information:

  • Term ID: The term ID for the highest consumed byte in the stream.
  • Consumption Term Offset: The byte offset within the term of the highest consumed byte in the stream.
  • Receiver Window: The advertised window to the driver publication of how much data the driver subscription is willing to receive.

SMs are generated periodically via the following rules:

  • Generate an SM if the Term ID increases (i.e. rotation of Term buffers)
  • Generate an SM if the difference between the highest seen term offset and the last sent SM Completed Term Offset is greater than 25% of the Receiver Window.
  • Generate an SM if an SM not sent in more than a certain configurable timeout

A media driver that has multiple Subscriptions for the same driver subscription will take the minimum of the consumption points for the Subscriptions for use as the Consumption Term Offset and Term ID in the SM when sent.

Congestion Control

Flow control is concerned with avoiding a publisher from overrunning a subscriber. Congestion control tries to slow a publication down so that it does not overrun the network.

Congestion control is not mandatory for Aeron implementations and may or may not be used depending on use case.

Congestion control is enforced on a per stream basis and is totally controlled by the driver Receiver as it is the one sending the Receiver Window to the driver publication.

Congestion Control in Aeron is accomplished by manipulating the Receiver Window. To mimic TCP operation, the receiver window may be adjusted as a normal TCP sender would manipulate its own congestion window. Unlike TCP, however, Aeron flow control can deal with window lengths varying and changing. Specifically, when a window shrinks without ACK'ing data.

Aeron is designed to work with low-latency and high-throughput systems in controlled environments where lower latency is traded off against congestion control for burst situations. In such systems, it is unlikely that congestion control is desired or advisable. And if so, it is probably not TCP-friendly congestion control. As such, other manipulation techniques of the Receiver Window probably make more sense than strict TCP fairness.

Custom Congestion Control Strategies

The driver can be configured to provide various congestion control strategies via the property aeron.CongestionControl.supplier. A congestion control strategy is assigned per Image and can be highly customised for specific channel, stream Id, session Id, term length, etc.

The default congestion control supplier uses a static window, StaticWindowCongestionControl, which effectively disables congestion control.

The following are the provided congestion control suppliers and strategies.

  1. StaticWindowCongestionControl: A static window that effectively disables congestion control.
  2. CubicCongestionControl and CubicCongestionControlSupplier: A CUBIC implementation of receiver window modification with and without TCP Mode support.

Beyond the suppliers, it is possible to customise a given CongestionControl strategy as befits the application semantics by implementing the CongestionControl interface.

For more information, see the relevant classes and/or interfaces documentation.

Background Information

For more information on concerns with congestion control and TCP, please see the following.

IPC Operation

IPC operation does not use SMs. Instead the media driver periodically updates the limit that a publication can send based on the state of the Subscriptions. First the minimum of the Subscription consumption points is determined then a window is added and that limit updated for the Publications. This window is controlled by a single property, aeron.ipc.publication.term.window.length. This acts as the window that the Publication can directly use before it is back pressured. When not set, this value defaults to the length of a term buffer. The resulting window will be a minimum of any configured value and length of half a term buffer.

Subscription Tethering

If more than one subscription matches on the same driver then each can be considered a tether or not. If subscriptions are a tether then they tether the flow control locally, i.e. subscription position's factor in the computation of the flow control window. If a subscription is not a tether and it is in the bottom 1/8th of the window, when another consumer is at the top of the window, for the aeron.untethered.window.limit.timeout then subscription in the bottom 1/8th will be notified that the image is unavailable for them. The subscription then remains a tether on flow control for the same timeout to allow the client to notice the unavailable image, after which the window can move on without that subscription factoring in flow control. At this point the subscription will rest for the period defined by aeron.untethered.resting.timeout after which is can rejoin the stream when it gets notified the image is available once more. If the subscriber does not wish to rest then they can close the subscription and add a new one.

The default for if subscriptions tether or not is controlled by the aeron.tether.subscriptions boolean system property which can be overridden for each subscription with the tether URI channel param, e.g. aeron:ipc?tether=false.

Tethering on IPC or spy channels have the window as defined by aeron.ipc.publication.term.window.length. Tether is also available on UDP subscriptions where the window is the current window as advertised for Status Messages which begins at aeron.rcv.initial.window.length, and aeron.publication.term.window.length for spying on network publications.

Spy Subscriptions

A subscription which spies, e.g. aeron-spy:aeron:udp?endpoint=<multicast.address:port>, on an outgoing local UDP publication is not involved in receiver based flow control for min, max, or tagged. It participates in flow control in a similar fashion to IPC and can use tether URI param.

Clone this wiki locally