Skip to content

[BLD]: fix Go test flakes #4760

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Jun 6, 2025
Merged

[BLD]: fix Go test flakes #4760

merged 13 commits into from
Jun 6, 2025

Conversation

codetheweb
Copy link
Contributor

@codetheweb codetheweb commented Jun 4, 2025

Description of changes

These changes seem to fix most causes of the Go tests flaking. Ran 70 times with no flake, whereas the flake rate seemed to be 5-20% prior to these changes.

Test plan

How are these changes tested?

  • Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Copy link

github-actions bot commented Jun 4, 2025

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

  • Can you think of any use case in which the code does not behave as intended? Have they been tested?
  • Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
  • If appropriate, are there adequate property based tests?
  • If appropriate, are there adequate unit tests?
  • Should any logging, debugging, tracing information be added or removed?
  • Are error messages user-friendly?
  • Have all documentation changes needed been made?
  • Have all non-obvious changes been commented?

System Compatibility

  • Are there any potential impacts on other parts of the system or backward compatibility?
  • Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

  • Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

@codetheweb codetheweb force-pushed the bld-debug-flaky-go-tests branch from 8f85ef7 to d4d1398 Compare June 5, 2025 17:28
@codetheweb codetheweb force-pushed the bld-debug-flaky-go-tests branch from be290d9 to 2ef29ad Compare June 5, 2025 19:54
@codetheweb codetheweb force-pushed the bld-debug-flaky-go-tests branch from 2ef29ad to 6f2e122 Compare June 5, 2025 20:42
@@ -235,6 +235,10 @@ func GetDBConfigForTesting() DBConfig {
WithStartupTimeout(5*time.Second)),
)

if err != nil {
panic(err)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is never hit, but realized any error here was swallowed during testing

@codetheweb codetheweb force-pushed the bld-debug-flaky-go-tests branch from f0ca173 to cc91374 Compare June 6, 2025 01:47
@codetheweb codetheweb marked this pull request as ready for review June 6, 2025 01:50
@codetheweb codetheweb requested a review from eculver June 6, 2025 01:50
Copy link
Contributor

propel-code-bot bot commented Jun 6, 2025

Improve Go Test Reliability by Reducing Flakiness in CI

This PR addresses Go test flakes by forcing sequential test execution in the Makefile, disabling the Ryuk container cleanup utility during CI runs, and adding stricter error handling for container startup in test database setup. The changes primarily target conditions causing race conflicts and resource contention with Docker-based database containers in CI environments, aiming to make Go tests more reliable.

Key Changes:
• Go tests are now run sequentially by adding '-p 1' to the test command in the Makefile, accompanied by explanatory comments on the cause of flakiness.
• CI workflow disables Testcontainers' Ryuk component, which was identified as a source of test flakiness.
• Error handling improved in Go test database setup: errors from starting a Postgres test container are now checked and will panic on failure.

Affected Areas:
• go/Makefile (test targets)
• go/pkg/sysdb/metastore/db/dbcore/core.go (test DB setup)
• .github/workflows/_go-tests.yml (GitHub Actions/CI for Go)

This summary was automatically generated by @propel-code-bot

if err != nil {
panic(err)
}

var ports nat.PortMap
ports, _ = container.Ports(context.Background())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BestPractice]

There's a missing error check after calling container.Ports(). Similar to how you've started properly handling the error from postgres2.RunContainer(), you should also check for errors when getting the ports.

@codetheweb codetheweb changed the title [BLD]: debug Go test flakes [BLD]: fix Go test flakes Jun 6, 2025
Copy link
Contributor

@eculver eculver left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool. I guess we might want to keep an eye on this a bit. The tests are already pretty fast and this doesn't seem to slow them down, presumably because there just aren't a ton of them? I guess it would still be nice to know why there are flakes when running in parallel, but since it doesn't "cost" much to run them sequentially, then who cares.

@codetheweb codetheweb merged commit 2a53fe3 into main Jun 6, 2025
71 checks passed
@codetheweb codetheweb deleted the bld-debug-flaky-go-tests branch June 6, 2025 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants