Skip to content

CI is running parallel tests with undersized key buffers #2311

Closed
@rzetelskik

Description

@rzetelskik

We are running 32 test in parallel in which a single ScyllaCluster needs several keys to be generated, yet we configure the buffers to:

- --crypto-key-buffer-size-min=3
- --crypto-key-buffer-size-max=6

https://gcsweb.scylla-operator.scylladb.com/gcs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1955/pull-scylla-operator-master-e2e-openshift-aws-parallel/1868673877073203200/artifacts/deploy/0/operator/50_operator.deployment.yaml

This is barely enough to use the cache for one cluster and given how underprovisioned the operator is in the CI we can't afford to generate all keys on demand, so we should use the cache as much as possible to utilize the free cycles we have.

This is an example timing from a test that failed because cert were generated on demand for multiple test and too slow

Dec 16 16:10:05.479: INFO: basic-kb7wb-auth-token: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-local-client-ca: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-local-user-admin: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-local-serving-ca: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-alternator-local-serving-ca: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-alternator-local-serving-certs: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-local-cql-connection-configs-admin: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-local-serving-certs: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-alternator-local-serving-certs: SecretUpdated

This likely cause many e2e tests to flake.

/kind failing-test
/priority important-soon
/assign

Duplicate of #2274.

Metadata

Metadata

Assignees

Labels

kind/failing-testCategorizes issue or PR as related to a consistently or frequently failing test.priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next release.

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions