Optimize Armeria server performance — event loop, thread pool, and JVM tuning to reduce tail latency

[evolved37]

🐞 Armeria Server Performance — Questions for Tuning Event Loop, Thread Pool & JVM

📌 Context

During load testing with about 300 parallel requests per second, we’re seeing the tail latency grow to around 30 seconds for the last request.
The test machine has 8 CPU cores and 30 GB RAM, so the issue is likely related to server configuration and JVM tuning, not hardware limits.

The service runs on a REST API built with Armeria.

---

❓ Questions for the Team

A. Event Loop & Thread Pool

• What is the best practice to configure the Armeria event loop and worker threads?
• How can we know if any of our request handlers contain blocking operations (e.g. DB queries, I/O, heavy CPU)?
• How should we handle those — e.g. should we use a dedicated BlockingTaskExecutor?
• How many threads should our blocking executor have relative to the number of CPU cores?
• Should we be using HTTP/2 or keep-alive to minimize connection overhead?

B. JVM & Build Configuration

• What JVM tuning parameters should we currently use for our services (heap size, GC type, pause time targets, etc.)?
• Are there preferred GC configurations (e.g. G1GC vs ZGC) for heavy workload?
• How should we size the heap for our 30 GB RAM server?
• Do we need to consider any specific optimizations for the JAR build (e.g. fat JAR, GraalVM native image, or JDK version upgrades)?

C. Profiling & Load Testing

• What metrics or tools should we be using to measure latency and throughput (e.g. P99 latency)?
• Do you have any tips how to load testing the service?
• is there any profiling data (e.g. from Async Profiler, jvisualvm) showing where the bottlenecks are?
• How should we want to approach scaling (thread pool adjustments, async code, horizontal scaling, etc.)?

---

🧾 Optional Discussion Points

• What tail latency (P99) do we consider acceptable at 300 RPS?
• How much headroom do we want to maintain before scaling horizontally?
• What monitoring or alerting thresholds should we define?

[ikhoon]

A. Event Loop & Thread Pool

• What is the best practice to configure the Armeria event loop and worker threads?

Armeria tries to provide a sensible default value for the number of event loops. However, it may be adjustable depending on the type of the workload.

armeria/core/src/main/java/com/linecorp/armeria/common/Flags.java

Lines 705 to 710 in 38d8d25

	* <p>The default value of this flag is {@code 2 * <numCpuCores>} for {@link TransportType#NIO},
	* {@link TransportType#EPOLL} and {@link TransportType#KQUEUE} and {@code <numCpuCores>} for
	* {@link TransportType#IO_URING}. Specify the {@code -Dcom.linecorp.armeria.numCommonWorkers=<integer>}
	* JVM option to override the default value.
	*/
	public static int numCommonWorkers() {

• How can we know if any of our request handlers contain blocking operations (e.g. DB queries, I/O, heavy CPU)?

We recommend using https://github.com/reactor/BlockHound to detect blocking IO.
Armeria provides BlockHound integration to avoid false positive reports.

armeria/core/src/main/java/com/linecorp/armeria/common/CoreBlockHoundIntegration.java

Line 30 in b841f0b

public final class CoreBlockHoundIntegration implements BlockHoundIntegration {

• How should we handle those — e.g. should we use a dedicated BlockingTaskExecutor?

To handle blocking tasks, you could enable useBlockingTaskExecutor() option, use @Blocking annotation, or manually execute tasks in BlockingTaskExecutor.

• How many threads should our blocking executor have relative to the number of CPU cores?

As blocking tasks are not CPU-intensive, it may depend on your system memory.

• Should we be using HTTP/2 or keep-alive to minimize connection overhead?

I think so. HTTP/2 would make your server more performant.

B. JVM & Build Configuration
...

Those are good questions. However, there are no recommended JVM tuning parameters of Armeria. They can vary depending on the type of your workload.

What metrics or tools should we be using to measure latency and throughput (e.g. P99 latency)?

Armeria exports request metrics via Micrometer. https://armeria.dev/docs/advanced-metrics
You can use PrometheusExpositionService to export them in the Prometheus format.

armeria/prometheus1/src/main/java/com/linecorp/armeria/server/prometheus/PrometheusExpositionService.java

Line 47 in 00d421f

public final class PrometheusExpositionService extends AbstractHttpService

---

There were many questions, but I skipped some of them because they were too general or lacked enough context to answer.
If you’d like to continue the discussion on those I didn’t address, feel free to leave additional comments.

[evolved37]

This is my startup class

@Slf4j
public class ServerSingleStartup {
    /* -- Registry meter for prometheus -- */
    public static PrometheusMeterRegistry meterRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);;

    /* -- BlobUtil init -- */
    public final BlobUtil blobUtil;
    public static Server runnableServer;
    public static BlobMemoryOptimizer blobMemoryOptimizer;
    public static RollingLogPolicy rollingLogPolicy;

    /* -- WssOptimizer Init --*/
    public static WssMemoryOptimizer wssMemoryOptimizer;

    /* -- Mdc Verify Key Map -- */
    public static ConcurrentHashMap<String, Verifier> jwtMdcVerifier;

    /* -- AI Capabilities optimizers Map = NomeModulo, Lista chiavi di traduzioni delle descrizioni delle funzioni -- */
    public static ConcurrentHashMap<String, List<String>> aiCapabilitiesFunctionMap;

    /* -- AI FunctionExecutor optimizer Map = ListaNomeModulo, Lista funzioni associate al modulo --*/
    public static ConcurrentHashMap<ModuleName,List<FunctionDefCustomWrapper>> mapModulesListFunctionDefList;

    /* -- AI report status Map --*/
    public static HashMap<String, Double> refIdReportStatus = new HashMap<>();

    private static final Tracing tracing = Tracing.newBuilder().build();

    public ServerSingleStartup(BlobUtil blobUtil,ConcurrentHashMap<String, Verifier> jwtMdcVerifier) {
        this.blobUtil = blobUtil;
        ServerSingleStartup.jwtMdcVerifier = jwtMdcVerifier;
    }

    public static void main(String[] args) throws Exception {
        /* -- LOAD LOGBACK CONFIG ON STARTUP -- */
        LoggerUtil.setSession("main");

        /* -- GET PROPERTY SERVER FILE -- */
        final ServerProperty conf = ServerResource.getConfiguration();

        /* -- ENABLE LOG ON STARTUP -- */
        final LoggerContext context = (LoggerContext) LoggerFactory.getILoggerFactory();
        final JoranConfigurator configurator = new JoranConfigurator();
        configurator.setContext(context);

        /* -- INIT SERVER STARTUP  -- */
        runnableServer = new ServerSingleStartup(new BlobUtil(),new ConcurrentHashMap<>()).run(conf);
        runnableServer.closeOnJvmShutdown();

        /* -- RUN SERVER -- */
        runnableServer.start().join();

        /* -- CREATE STATIC INSTANCE OF MONGO -- */
        MongoDb.instance(conf);

        /* -- LOAD POOLCONNECTOR MANAGER AREE -- */
        new PoolConnector().loadHikariPoolConnector();
        log.info("\n {} Server has been started http://localhost:{} {}", "\uD83E\uDD84 \uD83E\uDD84",runnableServer.activeLocalPort(), "\uD83E\uDD84 \uD83E\uDD84");
    }
    private Server run(final ServerProperty conf) throws IOException, InvocationTargetException, NoSuchMethodException, InstantiationException, IllegalAccessException {
        /* -- LOAD PROMETHEUS SERVICE -- */
        new JvmThreadMetrics().bindTo(meterRegistry);
        final ServerBuilder sb = Server.builder();
        sb.meterRegistry(meterRegistry);
        sb.http(conf.getServerPort());
        /* -- LOAD ALL ENDPONT CONFIGURED -- */
        this.loadHandler(sb);
        /* -- LOAD ALL ADDITION SERVICE -- */
        this.loadService(sb,meterRegistry,conf);
        /* -- LOAD ALL ADDITION DECORATOR -- */
        this.loadDecorator(sb);
        return sb.build();
    }

    private void loadHandler(final ServerBuilder sb) {
        .... More endpoint ....     
            sb.annotatedService(GlobalInfo.__API_VO__,new InfoHandler())
             .annotatedService(GlobalInfo.__API_VO__.concat("/reservationSmartHandler"),new ReservationSmartLegacyHandler())
        .... More endpoint ....
        ;

        /*-- LOAD WEBSOCKET SERVICE --*/
        final WebSocketService wsService = WebSocketService.builder(AiSocket.getWebSocketServiceHandler()).maxFramePayloadLength(200000).allowedOrigins("*").build();
        sb.service("/chat", wsService.decorate(GhrSocketUserSessionValidation::new));

        /*-- GET SERVER PROCESSOR AND SET AVAILABLE THREADS ON EVENT LOOP I/O OPERATION --*/
        final int cores = Runtime.getRuntime().availableProcessors();
        log.info("--- 🧮 DETECTED {} CPU CORES ---", cores);
        int numberOfThreads = cores * 2;
        log.info("--- AVAILABLE THREADS {} ---", numberOfThreads);
        sb.workerGroup(EventLoopGroups.newEventLoopGroup(numberOfThreads), true);
    }

    private void loadDecorator(final ServerBuilder sb){

        sb.decorator(ContentPreviewingService.newDecorator(1000));
        /* -- LOAD TRACING REQUEST AND LOG HTTP TRACE -- */
            sb.decorator((delegate, ctx, req) -> {
                final Tracer tracer = tracing.tracer();
                final Span span = tracer.newTrace().name(ctx.path()).start();
                /* -- LOGGIN CURRENT REQUEST -- */
                ctx.log().whenAvailable(RequestLogProperty.REQUEST_CONTENT_PREVIEW).thenAccept(log -> {
                    assert log.isAvailable(RequestLogProperty.REQUEST_CONTENT_PREVIEW);
                    LoggerServerHttp.logRequest(span.context().traceIdString(),req.method(),log.requestHeaders(),req.path(),req.contentType(),log.requestContentPreview());
                });
                /* -- LOGGIN CURRENT RESPONSE -- */
                ctx.log().whenComplete().thenAccept(log -> {
                    assert log.isComplete();
                    LoggerServerHttp.logResponse(span.context().traceIdString(),req.method(),log.responseHeaders(),req.path(),req.contentType(),log.responseContentPreview());
                });
                try (Tracer.SpanInScope ws = tracer.withSpanInScope(span)) {
                    ContextManager.setContextValue(ctx, ContextManager.ContextKey.TRACE_ID, span.context().traceIdString());
                    LoggerUtil.setTraceId(span.context().traceIdString());
                    return delegate.serve(ctx, req);
                } finally {
                    span.finish();
                }
            });

        /* -- ENABLE CORS FOR ALL SERVICE -- */
        sb.decorator(CorsService.builder("....MY DOMAIN....")
                .allowCredentials()
                .allowRequestMethods(HttpMethod.GET, HttpMethod.POST, HttpMethod.PUT, HttpMethod.DELETE, HttpMethod.OPTIONS)
                .allowRequestHeaders("content-type","x-scopes","authorization","x-lang","x-apikey","x-source")
                .newDecorator());
        /* -- LOAD METRICS -- */
        sb.decorator(MetricCollectingService.newDecorator(new CustomMeterIdPrefixFunction()));
        sb.decorator(MetricCollectingService.newDecorator(GrpcMeterIdPrefixFunction.of("grpc.service")));
        sb.decorator((delegate, ctx, req) -> {
            /* -- LOAD CURRRENT LANGUAGE FROM REQUEST X-Lang Header -- */
            final String language = req.headers().get("X-Lang");
            ContextManager.setContextValue(ctx, ContextManager.ContextKey.LANGUAGE, StringUtil.isEmpty(language) ? "en" : language);
            return delegate.serve(ctx, req);
        });
    }

    private  void loadService(final ServerBuilder sb,final PrometheusMeterRegistry meterRegistry,final ServerProperty conf ) throws IOException, InvocationTargetException, NoSuchMethodException, InstantiationException, IllegalAccessException {
        final DocService docService = DocService.builder().build();
        sb.serviceUnder("/docs", docService);
        sb.serviceUnder("/metrics", PrometheusExpositionService.of(meterRegistry.getPrometheusRegistry()));
        /* -- AVVIO IL BLOB MEMORY OPTIMIZER --*/
        blobMemoryOptimizer = new BlobMemoryOptimizer(conf.getCleanerSecondsInterval(),conf.getIncompleteSecondsAlive());
        /* -- AVVIO ROLLING POLICY LOG --*/
        rollingLogPolicy = new RollingLogPolicy(conf.getLogCleanerIntervalDays(),conf.getLogRoot());
        Runtime.getRuntime().addShutdownHook(new Thread(this::stop));
        /* -- AVVIO IL WSS MEMORY OPTIMIZER --*/
        wssMemoryOptimizer = new WssMemoryOptimizer(conf.getWssChatCleanerSecondsInterval(), conf.getWssChatSecondsAlive());
        /* -- AVVIO AI FUNCTIONS OPTIMIZER --*/
        log.info("Initializing AI Memory optimizer");
        final List<Class<?>> classList = OpenAiComponent.getClassesInPackage(OpenAiComponent.BASE_PACKAGE);
        aiCapabilitiesFunctionMap = OpenAiComponent.populateAiCapabilitiesMap(classList);
        mapModulesListFunctionDefList = OpenAiComponent.populateEnabledFunctionMap(classList);
    }

    public void stop() {
        log.info("Shutting down... blob optimizer memory");
        blobMemoryOptimizer.stop();
        log.info("Shutting down... rolling log policy");
        rollingLogPolicy.stop();
        log.info("Shutting down... wss memory optimizer");
        wssMemoryOptimizer.stop();

        log.info("Waiting for log flush...");
        final LoggerContext loggerContext = (LoggerContext) LoggerFactory.getILoggerFactory();
        loggerContext.stop(); 

        log.info("Shutting down... armeria server");
        runnableServer.stop();
    }
}

This is my handler

@ProducesJson
    @GhrUserSession
    @HttpLogger(type = WoHttpLogType.RESERVATION_SMART_SYNC, section = WoHttpLogSection.RESERVATION, logOnlyInternalRequest = true)
    @Post("/finalize")
    public ReservationFinalizationResponse finalizeReservation(final FinalizationReservationRequest finalizationReservationRequest, final ServiceRequestContext ctx) throws Exception {
        try (Connection conn = PoolConnector.getConnection(ctx)) {
            return new ReservationSmartCreateComponent(conn,ctx).finalizationReservation(finalizationReservationRequest);
        }
    }

I have this REST service, and inside it there are some external HTTP calls (using Armeria’s HTTP client): one call to Outlook for calendar creation and two calls to an external service for data registration.
The responses from these calls contain essential information for the procedure, so they cannot be executed asynchronously with respect to the request.
The server is very large and hosts many REST services.
For the connection pool, we use HikariCP and jOOQ with MySQL 8.
I have enabled metrics for both the handlers and the connection pool, as well as aggregated metrics with jOOQ.

How can I optimize or change the Armeria configuration to reduce the impact on my server’s performance?

Sorry if these are more management-related questions, but we are an Italian team and your European division has shut down, so we’re struggling to find documentation and use cases.

[evolved37]

no i do not use @Blocking annotation or serving in a blocking task executor.
should I do this ?

@ProducesJson
@GhrUserSession
@Post("/user/reservations")
public CompletableFuture<ReservationRangeResponse> getDataReservation(final ReservationDataRequest reservationDataRequest, final ServiceRequestContext ctx) throws Exception {
    /* ACTUAL
    try(Connection conn = PoolConnector.getConnection(ctx)){
        return ReservationAppComponent.getDataReservation(reservationDataRequest, conn);
    }
    */
    return CompletableFuture.supplyAsync(() -> {
        try (Connection conn = PoolConnector.getConnection(ctx)) {
            return ReservationAppComponent.getDataReservation(reservationDataRequest, conn);
        } catch (Exception e) {
            throw new RuntimeException("Error", e);
        }
    }, ctx.blockingTaskExecutor());
}

[ikhoon]

You can either specify ctx.blockingTaskExecutor() or annotate @Blocking to the method to serve requests in a blocking task executor.

@Blocking
@ProducesJson
@GhrUserSession
@HttpLogger(type = WoHttpLogType.RESERVATION_SMART_SYNC, section = WoHttpLogSection.RESERVATION, logOnlyInternalRequest = true)
@Post("/finalize")
public ReservationFinalizationResponse finalizeReservation(final FinalizationReservationRequest finalizationReservationRequest, final ServiceRequestContext ctx) throws Exception {
    try (Connection conn = PoolConnector.getConnection(ctx)) {
        return new ReservationSmartCreateComponent(conn,ctx).finalizationReservation(finalizationReservationRequest);
    }
}

[evolved37]

Do you recommend using the blocking task on all calls that access the database, or only on those that access the database and make HTTP calls to external services?

[ikhoon]

It depends on how HTTP calls are implemented. The general rule is to use the @Blocking annotation when the logic blocks a thread. If you find it difficult to distinguish, apply the @Blocking annotation to every class.

[evolved37]

Ok thanks,
you did a fantastic job with Armeria — it’s really an excellent framework