-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Fine Tuning PubSub Receive Performance
The PubSub client is a gRPC-based client with knobs for fine-tuning performance. We set reasonable defaults to these knobs, but we encourage users to adjust per their needs. This document describes some of these knobs and gives tips on setting them. This document is limited in scope to reception of messages and does not address publication.
A few implementation details must be described to fully understand the settings:
As the PubSub client receives messages from the PubSub server, it puts them into a local buffer. The client hands messages from the buffer to the user by way of the callback function passed in to Receive. The user must Ack or Nack a message in this function. Each invocation by the client of the passed-in callback occurs in a goroutine; that is, messages are processed concurrently.
The buffer holds a maximum of MaxOutstandingMessages messages or MaxOutstandingBytes bytes, and the client stops requesting more messages from the server whenever the buffer is full. Messages in the buffer have an ack deadline; that is, the server keeps a deadline for each outstanding messages. When that deadline expires, the server considers the message lost and re-sends the message. Each message in the buffer automatically has their deadline periodically extended. If a message is held beyond its deadline - for example, if your program hangs - the message will be redelivered.
This medium post describes tuning Pub/Sub performance in more detail.
This is the maximum amount of time that the client will extend a message's deadline. This value should be set as high as messages are expected to be processed, plus some buffer. It's fairly safe to set it quite high; the only downside is that it will take longer to recover from hanging programs. The higher the extension allowed, the longer it takes before the server considers messages lost and re-sends them to some other, healthy instance of your application.
This is the maximum amount of time to extend each message's deadline per ModifyAckDeadline RPC. Normally, the deadline is determined by the 99th percentile of previous message processing times. However, if normal processing times takes 10 minutes but an error occurs while processing a message within 1 minute, a message will be stuck and held by the client for the remaining 9 minutes. By setting the maximum amount of time to extend a message's deadline on a per-RPC basis, you can decrease the amount of time before message redelivery when errors occur. However the downside is that more ModifyAckDeadline RPCs will be sent.
This is the minimum amount of time to extend each message's deadline per ModifyAckDeadline RPC. This is the complement setting of MaxExtensionPeriod and represents the lower bound of modack deadlines to send. If processing time is very low, we might want to issue less frequent ModifyAckDeadline RPCs rather than every 10 seconds. Setting both MinExtensionPeriod and MaxExtensionPeriod to the same value effectively removes the automatic derivation of deadlines and fixes it to the value you wish to extend your messages deadlines by each time.
This is the maximum amount of messages that are to be processed by the callback function at a time. Once this limit is reached, the client waits for messages to be acked or nacked by the callback before requesting more messages from the server.
This value is set by default to a fairly conservatively low number. We strongly encourage setting this number as high as memory allows, since a low setting will artificially rate limit reception. Setting this value to -1 causes it to be unbounded.
This is the maximum amount of bytes (message size) that are to be processed by the callback function at a time. Once this limit is reached, the client waits for messages to be acked or nacked by the callback before requesting more messages from the server.
Note that there sometimes can be more bytes pulled and processing than MaxOutstandingBytes allows. This is due to the fact that the server doesn't consider byte sizes when provisioning server side flow control. For example, if the client sets MaxOutstandingBytes to 50 KiB, but receives a batch of messages totaling 100 KiB, there will be a temporary overflow of message byte size until messages are acked.
Similar to MaxOutstandingMessages, we recommend setting this higher to maximize processing throughput. Setting this value to -1 causes it to be unbounded.
This is the number of goroutines spawned to receive messages from the Pubsub server, where each goroutine opens a StreamingPull stream. This setting affects the rate of message intake from server to local buffer.
Setting this value to 1 is sufficient for many workloads. Each stream can handle about 10 MB/s of messages, so if you're throughput is under this, just set NumGoroutines=1. Reducing the number of streams can improve the performance by decreasing overhead.
Currently, there is an issue where setting NumGoroutines greater than 1 results in poor behavior interacting with flow control. Since each StreamingPull stream has its own flow control, the server side flow control will not match what is available locally. The default for NumGouroutines is 10 for legacy reasons, but we strongly recommend setting NumGoroutines to 1.
Going above 100 streams can lead to increasingly poor behavior, such as acks/modacks not succeeding in a reasonable amount of time, leading to message expiration. In these cases, we recommend horizontally scaling by increasing the number of subscriber client applications.
Note, if using the deprecated synchronous mode, NumGoroutines is capped at 1.
This configures the underlying message receiving mechanism. By default, we recommend turning off Synchronous mode, as this uses the streaming pull RPC. The streaming RPC has the benefit of reducing overhead compared to synchronous pull, which issues a Pull RPC every time new messages need to be pulled. In contrast, streaming pull periodically receives messages from the Pub/Sub server over the long-lived stream. The downside is that with streaming pull, the client does not have the fine grained control over how many messages are sent and it could exceed MaxOutstandingMessages.
Each application should use a single PubSub client instead of creating many. In addition, when publishing to a single topic, topic should be instantiated once and reused to take advantage of flow control and batching capabilities.