Parallel Workflow Update Tests [experiment]

[stephanos]

What changed?

Migrated TestWorkflowUpdateSuite away from testify's Suite; enabling parallel test execution.

How it works

• a test invokes testcore.NewEnv(t) to obtain a new TestEnv
• TestEnv sets t.Parallel() (intentionally not giving a way to opt out!)
• TestEnv obtains a test cluster from clusterPool (or blocks if all are in-use right now)
• env var TEMPORAL_TEST_SHARED_CLUSTERS controls size of the pool
• if a test relies on APIs like InjectHook, a dedicated cluster is used to prevent overlap
• env var TEMPORAL_TEST_DEDICATED_CLUSTERS controls number of dedicated clusters

testify suites

Existing test suites are limited by the same dedicated cluster pool to prevent creating too many clusters.

Database connections

SQLite setup for TestEnv-based func tests (ie only TestWorkflowUpdateSuite so far) has been changed to a file-based approach since that supports much better concurrency due to its WAL that an in-memory SQLite database does not support.

Connection limits for other databases were also raised due to connection errors.

Planned follow-ups

• Migrating the other testify suites should be fairly straight-forward with the use of AI agents.
• Reduce need for dedicated clusters by leveraging isolated namespace-per-test more.
• Eliminate all time.Sleeps.
• Tweak test cluster pool behavior.

Why?

1. Local speedup: benchmarks show a ~50% speed increase (36.1s → 16.6s) for TestWorkflowUpdateSuite.

2. Namespace isolation: every test runs in its own namespace. This greatly reduces the risk of (accidental) collisions and also reduces the need to craft unique identifiers such as for task queues and workflow IDs.

3. Deprecate testify suites: Long-term strategy to remove use of testify suites in functional tests (one reason being their inability to run tests within a suite in parallel).

How did you test it?

• built
• run locally and tested manually
• covered by existing tests
• added new unit test(s)
• added new functional test(s)

Potential Issues

1. Logs become less useful since there is more interleaving of tests.
2. Higher resource consumption: it requires more concurrent connections to databases and shows higher memory consumption (see 3 and 4). This could cause some short-term instability on CI. Note that some other PRs were merged to add mechanics for monitoring memory usage much better; which will help here.
3. Until all functional tests are converted, there is an imbalance in test cluster creation: migrated tests use the shared pool while current tests create one cluster each. Especially given the fact that some tests don't allow for test cluster sharing as they use non-parallelizable actions such as InjectHook or dynamic config overrides. With some more effort the number of these can be reduced.
4. Setup of test clusters was designed around the idea of short-lived clusters, one per suite. But when re-using them for longer, some of the assumptions don't hold anymore and increase memory usage. There's a band aid in place to limit how often a test cluster can be used before it's torn down. A long-term solution requires some design changes to how test clusters are started/used/torn down.
5. If there are certain cross-namespace issues or bugs that affect multiple tests, it might be harder to identify the root cause now. However; the existing test re-runs should at least mitigate these short-term.

[stephanos]

Removed OTEL tracing for now as its design caused increased memory usage (details: the OTEL exporter is scoped to the lifetime of the cluster instead of a namespace which would be more efficient here).

[stephanos]

Recommended settings I found online.

[stephanos]

A little refactoring to make the persistence setup slightly more ergonomic.

[stephanos]

Superseded by testcore.Env.

[stephanos]

Changes here were kept to a minimum; helpers and tests were migrated to regular functions. The nested nature of testify suites was preserved, too.