Add GrpcHealthCheckedEndpointGroupBuilder #6078

wvuong · 2025-01-22T18:54:54Z

Motivation:

Add GrpcHealthCheckedEndpointGroupBuilder which builds a health checked endpoint group whose health comes from a standard gRPC health check service result.

Modifications:

Adds GrpcHealthCheckedEndpointGroupBuilder which extends AbstractHealthCheckedEndpointGroupBuilder and creates a new health check function
Adds GrpcHealthChecker which is the health check function that creates and uses a gRPC HealthGrpc stub to check the gRPC health service on the endpoint. If the health check response is SERVING, it is healthy. It is unhealthy if the response is not SERVING or if there was a request failure.
Adds tests.

Result:

A user can create a health checked endpoint group that is backed by a gRPC health check service.
Closes Support for gRPC health check endpoint #5930

Motivation: Add `GrpcHealthCheckedEndpointGroupBuilder` which builds a health checked endpoint group whose health comes from a [standard gRPC health check service result](https://grpc.io/docs/guides/health-checking/). Modifications: * Adds `GrpcHealthCheckedEndpointGroupBuilder` which extends `AbstractHealthCheckedEndpointGroupBuilder` and creates a new health check function * Adds `GrpcHealthChecker` which is the health check function that creates and uses a gRPC `HealthGrpc` stub to check the gRPC health service on the endpoint. If the health check response is `SERVING`, it is healthy. It is unhealthy if the response is not `SERVING` or if there was a request failure. * Adds tests. Result: * A user can create a health checked endpoint group that is backed by a gRPC health check service. * Closes line#5930

CLAassistant · 2025-01-22T18:55:00Z

All committers have signed the CLA.

github-actions · 2025-01-22T19:34:15Z

🔍 Build Scan® (commit: `c077acc`)

Job name	Status	Build Scan®
build-ubicloud-standard-16-jdk-8	✅	https://ge.armeria.dev/s/xqo3l7j2j4n22
build-ubicloud-standard-16-jdk-21-snapshot-blockhound	✅	https://ge.armeria.dev/s/yohl7y7lu2py2
build-ubicloud-standard-16-jdk-17-min-java-17-coverage	✅	https://ge.armeria.dev/s/h7jbmyx5ajf6m
build-ubicloud-standard-16-jdk-17-min-java-11	✅	https://ge.armeria.dev/s/xrkt5m75dwobc
build-ubicloud-standard-16-jdk-17-leak	✅	https://ge.armeria.dev/s/yi7na53yqlxx6
build-ubicloud-standard-16-jdk-11	✅	https://ge.armeria.dev/s/kil472rqcm4z6
build-macos-latest-jdk-21	✅	https://ge.armeria.dev/s/5auqluiydnclm

jrhee17 · 2025-02-10T09:08:55Z

grpc/src/main/java/com/linecorp/armeria/internal/client/grpc/GrpcHealthChecker.java

+                        if (healthCheckResponse.getStatus() == HealthCheckResponse.ServingStatus.SERVING) {
+                            ctx.updateHealth(HEALTHY, reqCtx, null, null);
+                        } else {
+                            // not sure about the response headers but it needs to be non-null


I think it's fine to leave the headers as null for now.

Alternatively, we can also extract the headers from the ctx log

ResponseHeaders responseHeaders = null; if (reqCtx.log().isAvailable(RequestLogProperty.RESPONSE_HEADERS)) { responseHeaders = reqCtx.log().partial().responseHeaders(); }

jrhee17 · 2025-02-10T09:09:35Z

grpc/src/main/java/com/linecorp/armeria/internal/client/grpc/GrpcHealthChecker.java

+        lock();
+        try {
+            final HealthCheckRequest.Builder builder = HealthCheckRequest.newBuilder();
+            if (this.service != null) {


Suggested change

if (this.service != null) {

if (service != null) {

jrhee17 · 2025-02-10T09:10:55Z

grpc/src/main/java/com/linecorp/armeria/internal/client/grpc/GrpcHealthChecker.java

+                    public void onError(Throwable throwable) {
+                        final ClientRequestContext reqCtx = reqCtxCaptor.get();
+                        // same here
+                        ctx.updateHealth(UNHEALTHY, reqCtx, UNHEALTHY_RESPONSE_HEADERS, throwable);
+                    }
+
+                    @Override
+                    public void onCompleted() {
+                    }


Question) If the connection is dropped, is there no need to retry the check (with a backoff)?

jrhee17 · 2025-02-10T09:11:29Z

grpc/src/main/java/com/linecorp/armeria/internal/client/grpc/GrpcHealthChecker.java

+    }
+
+    public void start() {
+        check();


Question) Is there no need to also implement watch for completeness? Also, what do you think of allowing users to configure which method to use?

I think that's a good idea.

https://grpc.io/docs/guides/health-checking/

The health check service on a gRPC server supports two modes of operation:

Unary calls to the Check rpc endpoint

Useful for centralized monitoring or load balancing solutions, but does not scale to support a fleet of gRPC client constantly making health checks

Streaming health updates by using the Watch rpc endpoint

Used by the client side health check feature in gRPC clients

Watch seems to be the preferred mode for standard gRPC clients.

ikhoon

I would be satisfied as long as check runs periodically and watch is implemented.

ikhoon · 2025-02-19T04:50:11Z

grpc/src/main/java/com/linecorp/armeria/internal/client/grpc/GrpcHealthChecker.java

+            }
+
+            try (ClientRequestContextCaptor reqCtxCaptor = Clients.newContextCaptor()) {
+                stub.check(builder.build(), new StreamObserver<HealthCheckResponse>() {


Check is a unary call so it can't continuously update the upstream's status.
For Check method, we need a scheduler to periodically send Check requests.

ikhoon · 2025-02-19T04:53:24Z

grpc/src/main/java/com/linecorp/armeria/internal/client/grpc/GrpcHealthChecker.java

+    }
+
+    public void start() {
+        check();


https://grpc.io/docs/guides/health-checking/

The health check service on a gRPC server supports two modes of operation:

Unary calls to the Check rpc endpoint

Useful for centralized monitoring or load balancing solutions, but does not scale to support a fleet of gRPC client constantly making health checks

Streaming health updates by using the Watch rpc endpoint

Used by the client side health check feature in gRPC clients

Watch seems to be the preferred mode for standard gRPC clients.

wvuong · 2025-02-19T23:01:25Z

Hi, I have started working through these changes...

* Add GrpcHealthCheckWatcher which uses watch method * Refactor commoh class AbstractGrpcHealthChecker * Add support for configuring GrpcHealthCheckedEndpointGroupBuilder to select health check method with GrpcHealthCheckMethod * Add and update tests, redo HealthGrpcServerExtension

wvuong · 2025-02-25T01:09:24Z

Hi, I've implemented the requested changes:

Added the ability to configure GrpcHealthCheckedEndpointGroupBuilder to use either Check or Watch method to health check
GrpcHealthCheckWatcher which does the health checking with the Watch method
Updated both methods to use the executor to schedule the next check as necessary
Other minor cleanup, etc.

Unit tests and checkstyle pass. For the tests, an exception gets logged during the tear down when the health check tries to fire during tear down. I'm not really too sure what to do with that.

And the build-windows-latest-jdk-21 fails but all others pass. 🤷‍♂️

wvuong requested review from ikhoon, jrhee17, minwoox and trustin as code owners January 22, 2025 18:54

Fix tests

05f93c7

jrhee17 reviewed Feb 10, 2025

View reviewed changes

minwoox added this to the 1.33.0 milestone Feb 12, 2025

ikhoon added the new feature label Feb 19, 2025

ikhoon reviewed Feb 19, 2025

View reviewed changes

github-actions bot added the Stale label Mar 27, 2025

ikhoon added the priority: P2 label Apr 28, 2025

github-actions bot removed the Stale label Apr 29, 2025

github-actions bot added the Stale label May 29, 2025

ikhoon modified the milestones: 1.33.0, 1.34.0 Aug 1, 2025

github-actions bot removed the Stale label Aug 2, 2025

github-actions bot added the Stale label Sep 2, 2025

jrhee17 modified the milestones: 1.34.0, 1.35.0 Nov 24, 2025

github-actions bot removed the Stale label Nov 25, 2025

github-actions bot added the Stale label Dec 28, 2025

minwoox modified the milestones: 1.35.0, 1.36.0 Dec 30, 2025

github-actions bot removed the Stale label Jan 2, 2026

Add GrpcHealthCheckedEndpointGroupBuilder #6078

Are you sure you want to change the base?

Add GrpcHealthCheckedEndpointGroupBuilder #6078

Conversation

wvuong commented Jan 22, 2025

Uh oh!

CLAassistant commented Jan 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Build Scan® (commit: c077acc)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ikhoon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wvuong commented Feb 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wvuong commented Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

CLAassistant commented Jan 22, 2025 •

edited

Loading

github-actions bot commented Jan 22, 2025 •

edited

Loading

🔍 Build Scan® (commit: `c077acc`)

wvuong commented Feb 19, 2025 •

edited

Loading

wvuong commented Feb 25, 2025 •

edited

Loading