Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI is running parallel tests with undersized key buffers #2311

Closed
rzetelskik opened this issue Jan 9, 2025 · 0 comments · Fixed by #2312
Closed

CI is running parallel tests with undersized key buffers #2311

rzetelskik opened this issue Jan 9, 2025 · 0 comments · Fixed by #2312
Assignees
Labels
kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Milestone

Comments

@rzetelskik
Copy link
Member

We are running 32 test in parallel in which a single ScyllaCluster needs several keys to be generated, yet we configure the buffers to:

- --crypto-key-buffer-size-min=3
- --crypto-key-buffer-size-max=6

https://gcsweb.scylla-operator.scylladb.com/gcs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1955/pull-scylla-operator-master-e2e-openshift-aws-parallel/1868673877073203200/artifacts/deploy/0/operator/50_operator.deployment.yaml

This is barely enough to use the cache for one cluster and given how underprovisioned the operator is in the CI we can't afford to generate all keys on demand, so we should use the cache as much as possible to utilize the free cycles we have.

This is an example timing from a test that failed because cert were generated on demand for multiple test and too slow

Dec 16 16:10:05.479: INFO: basic-kb7wb-auth-token: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-local-client-ca: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-local-user-admin: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-local-serving-ca: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-alternator-local-serving-ca: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-alternator-local-serving-certs: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-local-cql-connection-configs-admin: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-local-serving-certs: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-alternator-local-serving-certs: SecretUpdated

This likely cause many e2e tests to flake.

/kind failing-test
/priority important-soon
/assign

Duplicate of #2274.

@scylla-operator-bot scylla-operator-bot bot added kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Jan 9, 2025
@rzetelskik rzetelskik added this to the v1.16.0 milestone Jan 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant