Skip to content

Turn off gzip for ElasticSearch in functional tests #9078

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
stephanos wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Turn off gzip for ElasticSearch in functional tests #9078

stephanos wants to merge 1 commit into from
+11 −2

Conversation

@stephanos
Copy link
Contributor

@stephanos stephanos commented Jan 19, 2026 •
edited
Loading

What changed?

Turn off gzip for ElasticSearch in functional tests.

Why?

Looking at a Go heap from a functional test that uses the ElasticSearch client (link), we can see that significant allocations (and likely CPU usage) are spent on compressing/decompressing.

Build ID: 5e6f217042f7638422b82855fafeec7bf59967d7
Type: alloc_space
Time: 2026-01-19 21:19:39 UTC
Showing nodes accounting for 28568.36MB, 69.16% of 41306.22MB total
Dropped 5182 nodes (cum <= 206.53MB)
      flat  flat%   sum%        cum   cum%
 3360.02MB  8.13%  8.13%  4099.66MB  9.93%  compress/flate.NewWriter (inline)
      ...
  712.32MB  1.72% 31.15%   712.32MB  1.72%  compress/flate.(*compressor).initDeflate (inline)

It turns out that this is only happening on setups with ElasticSearch. And since the tests run on a single machine, gzip has very little benefit. Turning it off reduces memory and CPU usage significantly.

The entries are completely removed from this PR's Go heap dump (example).

How did you test it?

  • built
  • run locally and tested manually
  • covered by existing tests
  • added new unit test(s)
  • added new functional test(s)

Potential risks

gzip is enabled by default in test envs, staging and production; therefore still being exercised sufficiently.

@stephanos stephanos changed the title Make elasticsearch gzip compression configurable for tests Turn off gzip for ElasticSearch in functional tests Jan 19, 2026
@stephanos stephanos force-pushed the elastic-disable-gzip-tests branch 4 times, most recently from ebaed3f to ff8e328 Compare January 19, 2026 22:54
@stephanos stephanos force-pushed the elastic-disable-gzip-tests branch from ff8e328 to ba2abcf Compare January 19, 2026 22:55
stephanos
stephanos commented Jan 19, 2026
View reviewed changes
common/persistence/visibility/store/elasticsearch/client/config.go
// httpClient is the awsHttpClient to be used for creating esClient.
httpClient *http.Client
// disableGzip disables gzip compression for Elasticsearch requests.
disableGzip bool
Copy link
Contributor Author

@stephanos stephanos Jan 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intentionally not exported as it's for testing purposes only.

Copy link
Contributor

@dnr dnr Jan 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tbh, I would even support making this public. Gzip is notoriously inefficient and is a terrible choice for protocol compression. (It's a terrible choice for any compression these days except for compatibility concerns.)

Copy link
Contributor Author

@stephanos stephanos Jan 20, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems we enabled it in production by default on a hunch:

It might help reduce/improve network traffic.

cc @rodrigozhou

@stephanos stephanos marked this pull request as ready for review January 19, 2026 23:14
@stephanos stephanos requested review from a team as code owners January 19, 2026 23:14
@stephanos stephanos requested a review from rodrigozhou January 19, 2026 23:14
dnr
dnr approved these changes Jan 19, 2026
View reviewed changes
common/persistence/visibility/store/elasticsearch/client/config.go
// httpClient is the awsHttpClient to be used for creating esClient.
httpClient *http.Client
// disableGzip disables gzip compression for Elasticsearch requests.
disableGzip bool
Copy link
Contributor

@dnr dnr Jan 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tbh, I would even support making this public. Gzip is notoriously inefficient and is a terrible choice for protocol compression. (It's a terrible choice for any compression these days except for compatibility concerns.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dnr dnr dnr approved these changes

@rodrigozhou rodrigozhou Awaiting requested review from rodrigozhou

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@stephanos @dnr