Is there any sample code for using circuit breaker in grpc client?

[hitoshi-isawa]

https://armeria.dev/docs/client-circuit-breaker/
I was able to implement circuit-breaker for the HTTP client according to this page.

Next, I would like to implement it for the grpc client, but I am wondering how to set the rules for the CircuitBreakerRule.
I'd appreciate any sample code or information on how to implement it.

ref) Rules implemented in HTTP client

    CircuitBreakerRule.of(
        // as failure
        CircuitBreakerRule.builder()
            .onServerErrorStatus()
            .onException()
            .thenFailure(),

        // as success
        CircuitBreakerRule.builder()
            .onStatusClass(HttpStatusClass.SUCCESS)
            .thenSuccess(),

        // ignore
        CircuitBreakerRule.builder().thenIgnore(),
    )

[minwoox]

Hi! The grpc-status is included in the trailer, so you need to use CircuitBreakerRuleWithContent.
You can check out this test case for reference:

armeria/grpc/src/test/java/com/linecorp/armeria/client/grpc/GrpcRetryWithCircuitBreakerTest.java

Lines 69 to 77 in 8618f8e

	final CircuitBreakerRuleWithContent<HttpResponse> cbRuleWithContent =
	CircuitBreakerRuleWithContent.<HttpResponse>builder()
	.onStatusClass(HttpStatusClass.SERVER_ERROR)
	.onResponseTrailers((ctx, trailers) -> {
	final String status =
	trailers.get(GrpcHeaderNames.GRPC_STATUS);
	return !"0".equals(status);
	})
	.thenFailure();

[hitoshi-isawa]

@minwoox
Thank you so much.
I implemented it based on the information you gave me, and tried to make it work in a local development environment by setting the URI of the destination to a dummy value of localhost, which causes an UnprocessedRequestException.
However, it seems that CircuitBreakerRuleWithContent does not seem to work, and replacing it with CircuitBreakerRule seems to work.
Do you know what the cause is?

ref) https://javadoc.io/doc/com.linecorp.armeria/armeria-javadoc/latest/com/linecorp/armeria/client/circuitbreaker/CircuitBreakerRuleWithContent.html

        GrpcClients.builder(properties.uri)
            .decorator(CircuitBreakerClient.newDecorator(circuitBreaker, createCircuitBreakerRule()))
            .serializationFormat(GrpcSerializationFormats.PROTO)
            .build(clientType)

private fun createCircuitBreakerRule() = CircuitBreakerRuleWithContent.of(
        // as failure
        CircuitBreakerRuleWithContent.builder<HttpResponse>()
            .onException()
            .onResponseTrailers { ctx: ClientRequestContext?, trailers: HttpHeaders ->
                setOf(
                    Status.Code.UNKNOWN.value().toString(),
                    Status.Code.DEADLINE_EXCEEDED.value().toString(),
                    Status.Code.UNIMPLEMENTED.value().toString(),
                    Status.Code.INTERNAL.value().toString(),
                    Status.Code.UNAVAILABLE.value().toString(),
                    Status.Code.DATA_LOSS.value().toString(),
                ).contains(trailers[HttpHeaderNames.of("grpc-status")])
            }
            .thenFailure(),

        // as success
        CircuitBreakerRuleWithContent.builder<HttpResponse>()
            .onResponseTrailers { ctx: ClientRequestContext?, trailers: HttpHeaders ->
                setOf(
                    Status.Code.OK.value().toString()
                ).contains(trailers[HttpHeaderNames.of("grpc-status")])
            }
            .thenSuccess(),

        // ignore
        CircuitBreakerRuleWithContent.builder<HttpResponse>().thenIgnore(),
    )

[minwoox]

I have no idea at the moment. Would you mind providing a reproducer?

[hitoshi-isawa]

It is difficult to submit a well hidden reproducer due to the implementation at work (although the company is related to LINE).
However, I have tried several implementation patterns, some successful and some unsuccessful, and I would like to share them.

Is this consideration likely to be correct?

Consideration (imo)

• I think that with CircuitBreakerRuleWithContent, only the decision by the object set in the Response is effective.
  • If it is a CircuitBreakerRule, can it handle other settings as well as judging by Response?
• If we want the circuit breaker to make a decision based on the grpc status code value, don't we need to use onGrpcTrailers?
  • perhaps onResponseTrailers is for HTTP client?.

---

What I tried

• Test Case
  • Test case 1: Prepared a dummy grpc server and implemented it to return Status.Code.INTERNAL.
    • Expect a decision by grpc status code with a trailer.
  • Test case 2: Stops dummy server and raises UnprocessedRequestException.
    • Expect onException behavior.

Successful implementation (using CircuitBreakerRule, onGrpcTrailers)

• Test case 1: OK
• Test case 2: OK

code

    private fun createCircuitBreakerRule() = CircuitBreakerRule.of(
        // as failure
        CircuitBreakerRule.builder()
            .onException()
            .onGrpcTrailers { ctx: ClientRequestContext?, trailers: HttpHeaders ->
                setOf(
                    Status.Code.UNKNOWN.value().toString(),
                    Status.Code.DEADLINE_EXCEEDED.value().toString(),
                    Status.Code.UNIMPLEMENTED.value().toString(),
                    Status.Code.INTERNAL.value().toString(),
                    Status.Code.UNAVAILABLE.value().toString(),
                    Status.Code.DATA_LOSS.value().toString(),
                ).contains(trailers[HttpHeaderNames.of("grpc-status")])
            }
            .thenFailure(),

        // as success
        CircuitBreakerRule.builder()
            .onGrpcTrailers { ctx: ClientRequestContext?, trailers: HttpHeaders ->
                setOf(
                    Status.Code.OK.value().toString()
                ).contains(trailers[HttpHeaderNames.of("grpc-status")])
            }
            .thenSuccess(),

        // ignore
        CircuitBreakerRule.builder().thenIgnore(),
    )

Failed implementations 1 (using CircuitBreakerRule, onResponseTrailers )

• Test case 1: NG
• Test case 2: OK

code

    private fun createCircuitBreakerRule() = CircuitBreakerRule.of(
        // as failure
        CircuitBreakerRule.builder()
            .onException()
            .onResponseTrailers { ctx: ClientRequestContext?, trailers: HttpHeaders ->
                setOf(
                    Status.Code.UNKNOWN.value().toString(),
                    Status.Code.DEADLINE_EXCEEDED.value().toString(),
                    Status.Code.UNIMPLEMENTED.value().toString(),
                    Status.Code.INTERNAL.value().toString(),
                    Status.Code.UNAVAILABLE.value().toString(),
                    Status.Code.DATA_LOSS.value().toString(),
                ).contains(trailers[HttpHeaderNames.of("grpc-status")])
            }
            .thenFailure(),

        // as success
        CircuitBreakerRule.builder()
            .onResponseTrailers { ctx: ClientRequestContext?, trailers: HttpHeaders ->
                setOf(
                    Status.Code.OK.value().toString()
                ).contains(trailers[HttpHeaderNames.of("grpc-status")])
            }
            .thenSuccess(),

        // ignore
        CircuitBreakerRule.builder().thenIgnore(),
    )

Failed implementations 2 (using CircuitBreakerRuleWithContent, onGrpcTrailers)

• Test case 1: OK
• Test case 2: NG

code

    private fun createCircuitBreakerRule() = CircuitBreakerRuleWithContent.of(
        // as failure
        CircuitBreakerRuleWithContent.builder<HttpResponse>()
            .onException()
            .onGrpcTrailers { ctx: ClientRequestContext?, trailers: HttpHeaders ->
                setOf(
                    Status.Code.UNKNOWN.value().toString(),
                    Status.Code.DEADLINE_EXCEEDED.value().toString(),
                    Status.Code.UNIMPLEMENTED.value().toString(),
                    Status.Code.INTERNAL.value().toString(),
                    Status.Code.UNAVAILABLE.value().toString(),
                    Status.Code.DATA_LOSS.value().toString(),
                ).contains(trailers[HttpHeaderNames.of("grpc-status")])
            }
            .thenFailure(),

        // as success
        CircuitBreakerRuleWithContent.builder<HttpResponse>()
            .onGrpcTrailers { ctx: ClientRequestContext?, trailers: HttpHeaders ->
                setOf(
                    Status.Code.OK.value().toString()
                ).contains(trailers[HttpHeaderNames.of("grpc-status")])
            }
            .thenSuccess(),

        // ignore
        CircuitBreakerRuleWithContent.builder<HttpResponse>().thenIgnore(),
    )

Failed implementations 3 (using CircuitBreakerRuleWithContent, onResponseTrailers )

• Test case 1: NG
• Test case 2: NG

code

private fun createCircuitBreakerRule() = CircuitBreakerRuleWithContent.of(
        // as failure
        CircuitBreakerRuleWithContent.builder<HttpResponse>()
            .onException()
            .onResponseTrailers { ctx: ClientRequestContext?, trailers: HttpHeaders ->
                setOf(
                    Status.Code.UNKNOWN.value().toString(),
                    Status.Code.DEADLINE_EXCEEDED.value().toString(),
                    Status.Code.UNIMPLEMENTED.value().toString(),
                    Status.Code.INTERNAL.value().toString(),
                    Status.Code.UNAVAILABLE.value().toString(),
                    Status.Code.DATA_LOSS.value().toString(),
                ).contains(trailers[HttpHeaderNames.of("grpc-status")])
            }
            .thenFailure(),

        // as success
        CircuitBreakerRuleWithContent.builder<HttpResponse>()
            .onResponseTrailers { ctx: ClientRequestContext?, trailers: HttpHeaders ->
                setOf(
                    Status.Code.OK.value().toString()
                ).contains(trailers[HttpHeaderNames.of("grpc-status")])
            }
            .thenSuccess(),

        // ignore
        CircuitBreakerRuleWithContent.builder<HttpResponse>().thenIgnore(),
    )

[minwoox]

@hitoshi-isawa Thanks for sharing, and I'm really sorry about the misguidance. 😓
Yeah, we need to use onGrpcTrailers with the CircuitBreakerRule. The grpc-status header can be in response headers or response trailers so we need to use onGrpcTrailers to check both. 😓
However, I still don't understand why the third case didn't work. Let me investigate. Thanks!

[minwoox]

Hi! I've tried the third case which was using CircuitBreakerRuleWithContent and onGrpcTrailers and they all worked as expected.
If you could provide the detailed information about how you've tested it, it would help me a lot in identifying the problem. Thanks!

[hitoshi-isawa]

@minwoox
By the way, can you still succeed using CircuitBreakerRule and onGrpcTrailers?
If so, I would like to know if it is necessary to use CircuitBreakerRuleWithContent.

[minwoox]

Yeah, it perfectly works with CircuitBreakerRule and onGrpcTrailers.
I was just wondering why CircuitBreakerRuleWithContent with onGrpcTrailers didn't work. 😓
Since your issue is resolved, let me just close this discussion.
Sorry about the misguidance from the first. 🙇

[hitoshi-isawa]

I was just wondering why CircuitBreakerRuleWithContent with onGrpcTrailers didn't work

Actually, me too... 😓

Anyway, I would like to close this discussion as I believe this matter has been resolved. Thank you for your cooperation. 🙌