Make better use of available cpu on larger VMs #57
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
These changes allow the loader to better utilize all cpu available on a larger instance. **1. CPU-intensive parsing/transforming is now parallelized**. Parallelism is configured by a new config parameter `cpuParallelismFraction`. The actual parallelism is chosen dynamically based on the number of available CPU, so the default value should be appropriate for all sized VMs. **2. We now open a new Snowflake ingest client per channel**. Note the Snowflake SDK recommends to re-use a single Client per VM and open multiple Channels on the same Client. So here we are going against the recommendations. But, we justify it because it gives the loader better visiblity of when the client's Future completes, signifying a complete write to Snowflake. **3. Upload parallelism chosen dynamically**. Larger VMs benefit from higher upload parallelism, in order to keep up with the faster rate of batches produced by the cpu-intensive tasks. Parallelsim is configured by a new parameter `uploadParallelismFactor`, which gets multiplied by the number of available CPU. The default value should be appropriate for all sized VMs. These new settings have been tested on pods ranging from 0.6 to 8 available CPU.
istreeter
force-pushed
the
cpu-parallelism
branch
from
b4453db to
ed44537
Compare
benjben
approved these changes
Nov 18, 2024
config/config.azure.reference.hocon
Outdated
| # -- name to use for the snowflake channel. | ||
| # -- Prefix to use for the snowflake channels. | ||
| # -- The full name will be suffixed with a number, e.g. `snowplow-1` | ||
| # -- The prefix be unique per loader VM |
There was a problem hiding this comment.
Suggested change
| # -- The prefix be unique per loader VM | |
| # -- The prefix must be unique per loader VM |
maybe ?
config/config.azure.reference.hocon
Outdated
| @@ -75,10 +77,17 @@ | |||
| # - Events are emitted to Snowflake for a maximum of this duration, even if the `maxBytes` size has not been reached | |||
| "maxDelay": "1 second" | |||
|
|
|||
| # - How many batches can we send simultaneously over the network to Snowflake. | |||
| "uploadConcurrency": 1 | |||
| # - Controls ow many batches can we send simultaneously over the network to Snowflake. | |||
There was a problem hiding this comment.
Suggested change
| # - Controls ow many batches can we send simultaneously over the network to Snowflake. | |
| # - Controls how many batches can we send simultaneously over the network to Snowflake. |
modules/core/src/main/scala/com.snowplowanalytics.snowplow.snowflake/Environment.scala
Show resolved
Hide resolved
| @@ -152,8 +144,8 @@ object Processing { | |||
| } | |||
|
|
|||
| /** Parse raw bytes into Event using analytics sdk */ | |||
There was a problem hiding this comment.
Suggested change
| /** Parse raw bytes into Event using analytics sdk */ |
istreeter
added a commit
that referenced
this pull request
Nov 26, 2024
These changes allow the loader to better utilize all cpu available on a larger instance. **1. CPU-intensive parsing/transforming is now parallelized**. Parallelism is configured by a new config parameter `cpuParallelismFraction`. The actual parallelism is chosen dynamically based on the number of available CPU, so the default value should be appropriate for all sized VMs. **2. We now open a new Snowflake ingest client per channel**. Note the Snowflake SDK recommends to re-use a single Client per VM and open multiple Channels on the same Client. So here we are going against the recommendations. But, we justify it because it gives the loader better visiblity of when the client's Future completes, signifying a complete write to Snowflake. **3. Upload parallelism chosen dynamically**. Larger VMs benefit from higher upload parallelism, in order to keep up with the faster rate of batches produced by the cpu-intensive tasks. Parallelsim is configured by a new parameter `uploadParallelismFactor`, which gets multiplied by the number of available CPU. The default value should be appropriate for all sized VMs. These new settings have been tested on pods ranging from 0.6 to 8 available CPU.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These changes allow the loader to better utilize all cpu available on a larger instance.
1. CPU-intensive parsing/transforming is now parallelized. Parallelism is configured by a new config parameter
cpuParallelismFraction. The actual parallelism is chosen dynamically based on the number of available CPU, so the default value should be appropriate for all sized VMs.2. We now open a new Snowflake ingest client per channel. Note the Snowflake SDK recommends to re-use a single Client per VM and open multiple Channels on the same Client. So here we are going against the recommendations. But, we justify it because it gives the loader better visiblity of when the client's Future completes, signifying a complete write to Snowflake.
3. Upload parallelism chosen dynamically. Larger VMs benefit from higher upload parallelism, in order to keep up with the faster rate of batches produced by the cpu-intensive tasks. Parallelsim is configured by a new parameter
uploadParallelismFactor, which gets multiplied by the number of available CPU. The default value should be appropriate for all sized VMs.These new settings have been tested on pods ranging from 0.6 to 8 available CPU.