Liveness Health Check getting executed on eventloop threads even after adding @Blocking annotation

[snavlani]

We implemented a custom health check as mentioned below

import io.smallrye.common.annotation.Blocking;
import lombok.extern.slf4j.Slf4j;
import org.eclipse.microprofile.health.HealthCheck;
import org.eclipse.microprofile.health.HealthCheckResponse;
import org.eclipse.microprofile.health.Liveness;

import jakarta.enterprise.context.ApplicationScoped;

@Slf4j
@Blocking
@Liveness
@ApplicationScoped
public class ServiceHealthCheck implements HealthCheck {

    @Override
    public HealthCheckResponse call() {
        log.info("Custom health check is being executed");
        return HealthCheckResponse.up("Service is up and running");
    }
}

We enabled quarkus HTTP debug logs and got to know that some of the liveness check requests were being executed using eventloop threads (Most of the requests were executed by executor threads).
Whenever eventloop threads execute the requests, it takes around 30 seconds to respond due to which kubernetes liveness check fails (As we have configured it to wait for maximum 30 seconds).

Even after adding @Blocking annotation why the requests are executed by eventloop threads ?

[mitkama]

Can you try similar code logic

`import io.smallrye.health.api.AsyncHealthCheck;
import io.smallrye.mutiny.Uni;
import org.eclipse.microprofile.health.HealthCheck;
import org.eclipse.microprofile.health.HealthCheckResponse;
import org.eclipse.microprofile.health.HealthCheckResponseBuilder;
import org.eclipse.microprofile.health.Readiness;
import javax.enterprise.context.ApplicationScoped;
import javax.ws.rs.Produces;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionStage;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

@readiness
@ApplicationScoped
public class CustomHealthCheck implements AsyncHealthCheck {

private ExecutorService executorService = Executors.newSingleThreadExecutor();  

@Override  
public Uni<HealthCheckResponse> call() {  
    return Uni.createFrom()  
            .completionStage(() -> CompletableFuture.supplyAsync(this::checkHealth, executorService));  
}  

private HealthCheckResponse checkHealth() {  
    HealthCheckResponseBuilder responseBuilder = HealthCheckResponse.named("Custom readiness check");  
    // Perform actual health check here  
    if (isHealthy()) {  
        return responseBuilder.up().build();  
    } else {  
        return responseBuilder.down().build();  
    }  
}  

private boolean isHealthy() {  
    // Add your health check logic here  
    return true;  
}  

}
`

[xstefank]

Hi @snavlani, this is not the correct place for this issue since the executing thread is specific to Quarkus. But I'll respond here.

In SR health (Quarkus integration) HealthCheck is always executed on the worker thread, and AsyncHealthCheck is always run on the eventloop thread. What you might see in the debug logs is that the collecting of the health check responses (and collective outcome) is computed on the eventloop thread. But if you'll do:

@Liveness
public class BlockingHealthCheck implements HealthCheck {
    @Override
    public HealthCheckResponse call() {
        System.out.println(getClass().getName() + ": " + Thread.currentThread().getName());
        return HealthCheckResponse.up("blocking");
    }
}

and

@Liveness
public class NonblockingHealthCheck implements AsyncHealthCheck {
    @Override
    public Uni<HealthCheckResponse> call() {
        System.out.println(getClass().getName() + ": " + Thread.currentThread().getName());
        return Uni.createFrom().item(HealthCheckResponse.up("async"));
    }
}

you'll see that your code is always run on the correct thread.

It shouldn't happen that HealthCheck is executed on the eventloop thread because - https://github.com/quarkusio/quarkus/blob/main/extensions/smallrye-health/runtime/src/main/java/io/quarkus/smallrye/health/runtime/QuarkusAsyncHealthCheckFactory.java#L27-L31

So if it does, please file the issue in the quarkus repository. Basically if it might happen that HealthCheck sometimes run on the eventloop this is a major problem.

Here is the reproducer https://github.com/xstefank/quarkus-reproducers/tree/main/health-thread.