fix: server run race #7964

zirain · 2026-01-15T11:09:57Z

xref: envoyproxy/ai-gateway#1770 (comment) #7880

AIGW required Wait for the runners to close, see #5560.
There's a problem is that runnersDone is blocked until ctx.Done(), which cause goroutine leak when config reloading.

#7880 fix the leak, but there's a race when using WaitGroup with multi goroutinues.

This PR use semaphore with weight 2, to handle all of these cases. see Loader.Wait().

netlify · 2026-01-15T11:10:05Z

✅ Deploy Preview for cerulean-figolla-1f9435 canceled.

Name	Link
🔨 Latest commit	`976bc3f`
🔍 Latest deploy log	https://app.netlify.com/projects/cerulean-figolla-1f9435/deploys/696c3f8384c1e40008701acf

zirain · 2026-01-15T11:21:20Z

failed as excepted:

WARNING: DATA RACE
Write at 0x00c00201e958 by goroutine 129:
  runtime.racewrite()
      <autogenerated>:1 +0x1e
  github.com/envoyproxy/gateway/internal/cmd.server()
      /home/runner/work/gateway/gateway/internal/cmd/server.go:113 +0x491
  github.com/envoyproxy/gateway/internal/cmd.TestServerRun.func1()
      /home/runner/work/gateway/gateway/internal/cmd/server_test.go:43 +0x92

Previous read at 0x00c00201e958 by goroutine 131:
  runtime.raceread()
      <autogenerated>:1 +0x1e
  github.com/envoyproxy/gateway/internal/cmd.server.func1()
      /home/runner/work/gateway/gateway/internal/cmd/server.go:83 +0x84
  github.com/envoyproxy/gateway/internal/envoygateway/config/loader.(*Loader).runHook.func1()
      /home/runner/work/gateway/gateway/internal/envoygateway/config/loader/configloader.go:126 +0xc3
  github.com/envoyproxy/gateway/internal/envoygateway/config/loader.(*Loader).runHook.gowrap1()
      /home/runner/work/gateway/gateway/internal/envoygateway/config/loader/configloader.go:132 +0x4f

Goroutine 129 (running) created at:
  github.com/envoyproxy/gateway/internal/cmd.TestServerRun()
      /home/runner/work/gateway/gateway/internal/cmd/server_test.go:42 +0x147
  testing.tRunner()
      /opt/hostedtoolcache/go/1.25.5/x64/src/testing/testing.go:1934 +0x21c
  testing.(*T).Run.gowrap1()
      /opt/hostedtoolcache/go/1.25.5/x64/src/testing/testing.go:1997 +0x44

Goroutine 131 (running) created at:
  github.com/envoyproxy/gateway/internal/envoygateway/config/loader.(*Loader).runHook()
      /home/runner/work/gateway/gateway/internal/envoygateway/config/loader/configloader.go:124 +0x309
  github.com/envoyproxy/gateway/internal/envoygateway/config/loader.(*Loader).Start()
      /home/runner/work/gateway/gateway/internal/envoygateway/config/loader/configloader.go:46 +0xc4
  github.com/envoyproxy/gateway/internal/cmd.server()
      /home/runner/work/gateway/gateway/internal/cmd/server.go:94 +0x2d7
  github.com/envoyproxy/gateway/internal/cmd.TestServerRun.func1()
      /home/runner/work/gateway/gateway/internal/cmd/server_test.go:43 +0x92

codecov · 2026-01-15T13:16:15Z

Codecov Report

❌ Patch coverage is 61.76471% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.45%. Comparing base (30cabed) to head (976bc3f).

Files with missing lines	Patch %	Lines
internal/cmd/server.go	46.66%	7 Missing and 1 partial ⚠️
...nternal/envoygateway/config/loader/configloader.go	73.68%	3 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7964      +/-   ##
==========================================
+ Coverage   72.78%   73.45%   +0.67%     
==========================================
  Files         237      237              
  Lines       35475    35487      +12     
==========================================
+ Hits        25821    26068     +247     
+ Misses       7812     7562     -250     
- Partials     1842     1857      +15

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

mathetake · 2026-01-15T23:54:53Z

nice!

arkodg · 2026-01-16T03:30:42Z

can we enable the -race flag before releasing this

zhaohuabing · 2026-01-16T03:32:12Z

Hi @zirain is it easy to hit this?

zirain · 2026-01-16T03:36:48Z

Hi @zirain is it easy to hit this?

the testcase is reproducible.

zirain · 2026-01-16T03:37:51Z

can we enable the -race flag before releasing this

go.test.coverage: go.test.cel ## Run go unit and integration tests in GitHub Actions
	@$(LOG_TARGET)
	KUBEBUILDER_ASSETS="$$($(GO_TOOL) setup-envtest use $(ENVTEST_K8S_VERSION) -p path)" \
		go test ./... --tags=integration -race -coverprofile=coverage.xml -covermode=atomic -coverpkg=./...

it's enabled, the problem is lack of test case.

zhaohuabing · 2026-01-16T04:39:06Z

internal/envoygateway/config/loader/configloader.go

 				time.Sleep(3 * time.Second)

-				r.runHook(ctx)
+				r.runHook(ctx, wg)


It seems that there is still a race: wg.Add(1) is called from the watcher goroutine while shutdown calls wg.Wait() in server.

can you show a reproducible test case for better understanding?

If I understand this correctly, this PR moves the initial WaitGroup.Add into the same goroutine as WaitGrop.Wait, But during config reload, WaitGroup.Add in runHook and WaitGroup.Wait in Server are still called from different goroutines, which could introduce a race condition.

good catch, let me try to work out another solution.

nacx · 2026-01-16T17:11:10Z

We would love to have this merged, as there is an Envoy AI Gateway release waiting for this. @arkodg @jukie mind having another review?

jukie · 2026-01-16T23:01:06Z

internal/cmd/server.go

 				},
 			)
-			return server(cmd.Context(), cmd.OutOrStdout(), cmd.ErrOrStderr(), runnerErrors)
+			started := &atomic.Bool{}


It doesn't look like this is being read or used anywhere, can this be removed?

this's useful to allow us konw when the server is ready and when to trigger a config update.
we can update this to a callback.

Could we structure the tests differently if that's the only place it's being used?

Only a minor nit so non-blocking

arkodg · 2026-01-17T17:32:47Z

this is the 3rd fix in this area, can we take a step back and redesign what server/runner initialization, handling partial failures, and config restarts should look like ?

Signed-off-by: zirain <[email protected]>

zirain · 2026-01-18T01:49:25Z

this is the 3rd fix in this area, can we take a step back and redesign what server/runner initialization, handling partial failures, and config restarts should look like ?

the main reason of the regssion is lack of reason.

the first fix introduce a leak when config reload too much times.
chore: fix goroutine leak #7880 fix the leak but introduce a race condition.

I think the new test case cover them.

Signed-off-by: zirain <[email protected]>

arkodg · 2026-01-18T03:49:51Z

can you elaborate on what the logic is / should be for

dealing with partial runner termination
config reloads that impact runner restarts

zirain · 2026-01-18T04:36:12Z

can you elaborate on what the logic is / should be for

dealing with partial runner termination

config reloads that impact runner restarts

AIGW required Wait for the runners to close, see https://github.com/envoyproxy/gateway/pull/5560/changes.
There's a problem is that runnersDone is blocked until ctx.Done(), which cause goroutine leak when config reloading.

#7880 fix the leak, but there's a race when using WaitGroup with multi goroutinues.

This PR use semaphore with weight 2, to handle all of these cases. see Loader.Wait().

Update above to PR description.

nacx · 2026-01-19T23:51:14Z

this is the 3rd fix in this area, can we take a step back and redesign what server/runner initialization, handling partial failures, and config restarts should look like ?

I agree with the sentiment here, but I also think that given we're close to the desired date for 1.7 release, and the fact that this should be ideally cherry-picked to previous versions, it is convenient to merge the fix (it's safer to cherry-pick a small fix than a refactoring), unblock the projects that depend on this, and then plan and work for a proper redesign.

zirain requested a review from a team as a code owner January 15, 2026 11:09

This was referenced Jan 15, 2026

chore: fix goroutine leak #7880

Merged

fix: fail fast when unrecoverable discovery errors happens on checking optional CRDs #7872

Merged

zirain added cherrypick/release-v1.6.3 cherrypick/release-v1.5.8 labels Jan 15, 2026

jukie previously approved these changes Jan 16, 2026

View reviewed changes

zhaohuabing reviewed Jan 16, 2026

View reviewed changes

zirain dismissed jukie’s stale review via 47713f2 January 16, 2026 06:18

zirain force-pushed the fix/waitgroup-race branch 2 times, most recently from 9699599 to 6483c24 Compare January 16, 2026 07:56

zirain changed the title ~~fix: WaitGroup race~~ fix: server run race Jan 16, 2026

jukie reviewed Jan 16, 2026

View reviewed changes

zirain force-pushed the fix/waitgroup-race branch from a5fad1d to de06900 Compare January 17, 2026 10:36

zirain added 7 commits January 18, 2026 09:42

add test

39b77b4

Signed-off-by: zirain <[email protected]>

fix race

7d1a6d3

Signed-off-by: zirain <[email protected]>

fix lint

6a572ea

Signed-off-by: zirain <[email protected]>

fix

a14d603

Signed-off-by: zirain <[email protected]>

fix

d69b914

Signed-off-by: zirain <[email protected]>

fix lint

017afb8

Signed-off-by: zirain <[email protected]>

use Semaphore instead of WaitGroup

b48915d

Signed-off-by: zirain <[email protected]>

zirain added 5 commits January 18, 2026 09:43

comments

33baeb5

Signed-off-by: zirain <[email protected]>

lint

677c191

Signed-off-by: zirain <[email protected]>

fix

c12d9fb

Signed-off-by: zirain <[email protected]>

fix lint

7551dcb

Signed-off-by: zirain <[email protected]>

callback

2ec46b5

Signed-off-by: zirain <[email protected]>

zirain force-pushed the fix/waitgroup-race branch from de06900 to 2ec46b5 Compare January 18, 2026 01:47

fix lint

976bc3f

Signed-off-by: zirain <[email protected]>

fix: server run race #7964

Are you sure you want to change the base?

fix: server run race #7964

Uh oh!

Conversation

zirain commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for cerulean-figolla-1f9435 canceled.

Uh oh!

zirain commented Jan 15, 2026

Uh oh!

codecov bot commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mathetake commented Jan 15, 2026

Uh oh!

arkodg commented Jan 16, 2026

Uh oh!

zhaohuabing commented Jan 16, 2026

Uh oh!

zirain commented Jan 16, 2026

Uh oh!

zirain commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhaohuabing Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zirain Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

zhaohuabing Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

zirain Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

nacx commented Jan 16, 2026

Uh oh!

jukie Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zirain Jan 17, 2026

Choose a reason for hiding this comment

Uh oh!

jukie Jan 17, 2026

Choose a reason for hiding this comment

Uh oh!

jukie Jan 17, 2026

Choose a reason for hiding this comment

Uh oh!

arkodg commented Jan 17, 2026

Uh oh!

zirain commented Jan 18, 2026

Uh oh!

arkodg commented Jan 18, 2026

Uh oh!

zirain commented Jan 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nacx commented Jan 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

zirain commented Jan 15, 2026 •

edited

Loading

netlify bot commented Jan 15, 2026 •

edited

Loading

codecov bot commented Jan 15, 2026 •

edited

Loading

zirain commented Jan 16, 2026 •

edited

Loading

zhaohuabing Jan 16, 2026 •

edited

Loading

jukie Jan 16, 2026 •

edited

Loading

zirain commented Jan 18, 2026 •

edited

Loading