Skip to content

Conversation

@ikhoon
Copy link
Contributor

@ikhoon ikhoon commented Nov 12, 2025

Motivation:

Athenz Docker containers intermittently fail to start in our CI environment.

[Test worker] INFO tc.docker - Local Docker Compose is running command: compose up -d
[Thread-5] INFO tc.docker -  Network jedrmcc2cwgf_athenz  Creating
[Thread-5] INFO tc.docker -  Container jedrmcc2cwgf-zts-db-1  Creating
[Thread-5] INFO tc.docker -  Container jedrmcc2cwgf-zms-db-1  Creating
...
[Thread-5] INFO tc.docker -  Container jedrmcc2cwgf-zms-server-1  Starting
[Thread-5] INFO tc.docker -  Container jedrmcc2cwgf-zms-server-1  Started
[Thread-5] INFO tc.docker -  Container jedrmcc2cwgf-zts-db-1  Waiting
[Thread-5] INFO tc.docker -  Container jedrmcc2cwgf-zms-server-1  Waiting
[Thread-5] INFO tc.docker -  Container jedrmcc2cwgf-zts-db-1  Error
[Thread-5] INFO tc.docker -  Container jedrmcc2cwgf-zms-server-1  Healthy
[Thread-5] INFO tc.docker - dependency failed to start: container jedrmcc2cwgf-zts-db-1 is unhealthy
[Test worker] WARN com.linecorp.armeria.server.athenz.AthenzDocker - Failed to initialize Athenz Docker container
org.testcontainers.containers.ContainerLaunchException: Local Docker Compose exited abnormally with code 1 whilst running command: compose up -d
	at org.testcontainers.containers.LocalDockerCompose.invoke(LocalDockerCompose.java:119)
	at org.testcontainers.containers.ComposeDelegate.runWithCompose(ComposeDelegate.java:241)
	at org.testcontainers.containers.ComposeDelegate.createServices(ComposeDelegate.java:163)
	at org.testcontainers.containers.ComposeContainer.start(ComposeContainer.java:133)
	at com.linecorp.armeria.server.athenz.AthenzDocker.initialize(AthenzDocker.java:105)
	at example.armeria.athenz.AthenzTest$1.configure(AthenzTest.java:44)
	at com.linecorp.armeria.testing.junit5.server.ServerExtension$1.configure(ServerExtension.java:72)

Modifications:

  • Increase the health check timeout and retry count for Athenz databases.
  • Automatically retry up to 5 times when Docker startup fails.

Result:

Hopefully fix the flakiness

Motivation:

Athenz Docker containers intermittently fail to start in our CI environment.
```
[Test worker] INFO tc.docker - Local Docker Compose is running command: compose up -d
[Thread-5] INFO tc.docker -  Network jedrmcc2cwgf_athenz  Creating
[Thread-5] INFO tc.docker -  Container jedrmcc2cwgf-zts-db-1  Creating
[Thread-5] INFO tc.docker -  Container jedrmcc2cwgf-zms-db-1  Creating
...
[Thread-5] INFO tc.docker -  Container jedrmcc2cwgf-zms-server-1  Starting
[Thread-5] INFO tc.docker -  Container jedrmcc2cwgf-zms-server-1  Started
[Thread-5] INFO tc.docker -  Container jedrmcc2cwgf-zts-db-1  Waiting
[Thread-5] INFO tc.docker -  Container jedrmcc2cwgf-zms-server-1  Waiting
[Thread-5] INFO tc.docker -  Container jedrmcc2cwgf-zts-db-1  Error
[Thread-5] INFO tc.docker -  Container jedrmcc2cwgf-zms-server-1  Healthy
[Thread-5] INFO tc.docker - dependency failed to start: container jedrmcc2cwgf-zts-db-1 is unhealthy
[Test worker] WARN com.linecorp.armeria.server.athenz.AthenzDocker - Failed to initialize Athenz Docker container
org.testcontainers.containers.ContainerLaunchException: Local Docker Compose exited abnormally with code 1 whilst running command: compose up -d
	at org.testcontainers.containers.LocalDockerCompose.invoke(LocalDockerCompose.java:119)
	at org.testcontainers.containers.ComposeDelegate.runWithCompose(ComposeDelegate.java:241)
	at org.testcontainers.containers.ComposeDelegate.createServices(ComposeDelegate.java:163)
	at org.testcontainers.containers.ComposeContainer.start(ComposeContainer.java:133)
	at com.linecorp.armeria.server.athenz.AthenzDocker.initialize(AthenzDocker.java:105)
	at example.armeria.athenz.AthenzTest$1.configure(AthenzTest.java:44)
	at com.linecorp.armeria.testing.junit5.server.ServerExtension$1.configure(ServerExtension.java:72)
```

Modifications:

- Increase the health check timeout and retry count for Athenz databases.
- Automatically retry up to 5 times when Docker startup fails.

Result:

Hopefully fix the flakiness
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants