Skip to content

Enable health check to use mtls to connect to backends #807

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
felicity3786 wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Enable health check to use mtls to connect to backends #807

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8bf5a96
Allow health check monitor to use mtls on backend connection
felicity3786 Dec 11, 2025
5d266ed
small fix
felicity3786 Dec 12, 2025
00524ae
add docs
felicity3786 Dec 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions docs/installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -558,3 +558,32 @@ a username and password using `backendState` as with the `JDBC` option.
#### NOOP

This option disables health checks.

### Enable mTLS for monitor HttpClient (JMX and METRICS)

For JMX and Metrics monitors, you can enable mutual TLS (mTLS) for the monitor HttpClient.
When enabled, the monitors authenticate to backends with a client certificate and ignore any
configured `backendState.username` and `backendState.password`.

Enable it with a toggle in `backendState` and provide TLS material for the named `monitor` HttpClient
under `serverConfig`:

```yaml
backendState:
monitorMtlsEnabled: true

serverConfig:
monitor.http-client.key-store-path: /path/to/keystore
monitor.http-client.key-store-password: keystore_password
monitor.http-client.trust-store-path: /path/to/truststore
monitor.http-client.trust-store-password: changeit
```

Notes:

- The `monitor` client is bound via Airlift's `httpClientBinder`, so properties must use the
`monitor.http-client.*` prefix.
- With `monitorMtlsEnabled: true`:
- JMX and Metrics monitors do not send `Authorization` or `X-Trino-User` headers and do not apply Basic Auth.
- Authentication relies solely on the configured client certificate.
- The gateway validates these TLS properties on startup when mTLS is enabled and fails fast if any are missing.
10 changes: 7 additions & 3 deletions gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/ClusterStatsJmxMonitor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,13 +47,15 @@ public class ClusterStatsJmxMonitor
private final String username;
private final String password;
private final boolean xForwardedProtoHeader;
private final boolean monitorMtlsEnabled;

public ClusterStatsJmxMonitor(HttpClient client, BackendStateConfiguration backendStateConfiguration)
{
this.client = requireNonNull(client, "client is null");
this.username = backendStateConfiguration.getUsername();
this.password = backendStateConfiguration.getPassword();
this.xForwardedProtoHeader = backendStateConfiguration.getXForwardedProtoHeader();
this.monitorMtlsEnabled = backendStateConfiguration.isMonitorMtlsEnabled();
}

private static void updateClusterStatsFromDiscoveryNodeManagerResponse(JmxResponse response, ClusterStats.Builder clusterStats)
Expand Down Expand Up @@ -132,16 +134,18 @@ private Optional<JmxResponse> queryJmx(ProxyBackendConfiguration backend, String
.setUri(uriBuilderFrom(URI.create(jmxUrl))
.appendPath(JMX_PATH)
.appendPath(mbeanName)
.build())
.addHeader("X-Trino-User", username);
.build());
if (!monitorMtlsEnabled) {
requestBuilder.addHeader("X-Trino-User", username);
}
if (xForwardedProtoHeader) {
requestBuilder.addHeader(X_FORWARDED_PROTO, "https");
}
Request preparedRequest = requestBuilder.build();

boolean isHttps = preparedRequest.getUri().getScheme().equalsIgnoreCase("https");

if (isHttps) {
if (isHttps && !monitorMtlsEnabled) {
HttpRequestFilter filter = new BasicAuthRequestFilter(username, password);
preparedRequest = filter.filterRequest(preparedRequest);
}
Expand Down
19 changes: 14 additions & 5 deletions gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/ClusterStatsMetricsMonitor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ public class ClusterStatsMetricsMonitor
private final int retries;
private final MetricsResponseHandler metricsResponseHandler;
private final Header identityHeader;
private final boolean monitorMtlsEnabled;
private final String metricsEndpoint;
private final String runningQueriesMetricName;
private final String queuedQueriesMetricName;
Expand All @@ -65,12 +66,18 @@ public ClusterStatsMetricsMonitor(HttpClient httpClient, BackendStateConfigurati
{
this.httpClient = requireNonNull(httpClient, "httpClient is null");
retries = monitorConfiguration.getRetries();
if (!isNullOrEmpty(backendStateConfiguration.getPassword())) {
identityHeader = new Header("Authorization",
new BasicCredentials(backendStateConfiguration.getUsername(), backendStateConfiguration.getPassword()).getBasicAuthHeader());
monitorMtlsEnabled = backendStateConfiguration.isMonitorMtlsEnabled();
if (!monitorMtlsEnabled) {
if (!isNullOrEmpty(backendStateConfiguration.getPassword())) {
identityHeader = new Header("Authorization",
new BasicCredentials(backendStateConfiguration.getUsername(), backendStateConfiguration.getPassword()).getBasicAuthHeader());
}
else {
identityHeader = new Header("X-Trino-User", backendStateConfiguration.getUsername());
}
}
else {
identityHeader = new Header("X-Trino-User", backendStateConfiguration.getUsername());
identityHeader = null;
Copy link
Member

@oneonestar oneonestar Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use Optional instead of null.

}
metricsEndpoint = monitorConfiguration.getMetricsEndpoint();
runningQueriesMetricName = monitorConfiguration.getRunningQueriesMetricName();
Expand Down Expand Up @@ -139,8 +146,10 @@ private Map<String, String> getMetrics(String baseUrl, int retriesRemaining)

Request.Builder requestBuilder = prepareGet()
.setUri(uri.build())
.addHeader(identityHeader.name, identityHeader.value)
.addHeader("Content-Type", "application/openmetrics-text; version=1.0.0; charset=utf-8");
if (identityHeader != null) {
requestBuilder.addHeader(identityHeader.name, identityHeader.value);
}
if (xForwardedProtoHeader) {
requestBuilder.addHeader(X_FORWARDED_PROTO, "https");
}
Expand Down
11 changes: 11 additions & 0 deletions gateway-ha/src/main/java/io/trino/gateway/ha/config/BackendStateConfiguration.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ public class BackendStateConfiguration
private String password = "";
private Boolean ssl = false;
private boolean xForwardedProtoHeader;
private boolean monitorMtlsEnabled;

public BackendStateConfiguration() {}

Expand Down Expand Up @@ -61,4 +62,14 @@ public void setXForwardedProtoHeader(boolean xForwardedProtoHeader)
{
this.xForwardedProtoHeader = xForwardedProtoHeader;
}

public boolean isMonitorMtlsEnabled()
{
return monitorMtlsEnabled;
}

public void setMonitorMtlsEnabled(boolean monitorMtlsEnabled)
{
this.monitorMtlsEnabled = monitorMtlsEnabled;
}
}
32 changes: 32 additions & 0 deletions gateway-ha/src/main/java/io/trino/gateway/ha/module/HaGatewayProviderModule.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@
import jakarta.ws.rs.container.ContainerRequestFilter;
import org.jdbi.v3.core.Jdbi;

import java.util.ArrayList;
import java.util.List;
import java.util.Map;

Expand All @@ -81,6 +82,11 @@
public class HaGatewayProviderModule
extends AbstractModule
{
private static final String MONITOR_HTTP_CLIENT_KEY_STORE_PATH = "monitor.http-client.key-store-path";
private static final String MONITOR_HTTP_CLIENT_KEY_STORE_PASSWORD = "monitor.http-client.key-store-password";
private static final String MONITOR_HTTP_CLIENT_TRUST_STORE_PATH = "monitor.http-client.trust-store-path";
private static final String MONITOR_HTTP_CLIENT_TRUST_STORE_PASSWORD = "monitor.http-client.trust-store-password";

Comment on lines +85 to +89
Copy link
Member

@Peiyingy Peiyingy Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I was thinking of something like this, WDYT?

Suggested change
private static final String MONITOR_HTTP_CLIENT_KEY_STORE_PATH = "monitor.http-client.key-store-path";
private static final String MONITOR_HTTP_CLIENT_KEY_STORE_PASSWORD = "monitor.http-client.key-store-password";
private static final String MONITOR_HTTP_CLIENT_TRUST_STORE_PATH = "monitor.http-client.trust-store-path";
private static final String MONITOR_HTTP_CLIENT_TRUST_STORE_PASSWORD = "monitor.http-client.trust-store-password";
private static final List<String> REQUIRED_MONITOR_MTLS_KEYS = List.of(
"monitor.http-client.key-store-path",
"monitor.http-client.key-store-password",
"monitor.http-client.trust-store-path",
"monitor.http-client.trust-store-password");

private final LbOAuthManager oauthManager;
private final LbFormAuthManager formAuthManager;
private final AuthorizationManager authorizationManager;
Expand All @@ -106,6 +112,10 @@ public HaGatewayProviderModule(HaGatewayConfiguration configuration)
{
this.configuration = requireNonNull(configuration, "configuration is null");
pathFilter = new PathFilter(configuration.getStatementPaths(), configuration.getExtraWhitelistPaths());
// Enforce required TLS properties for the named HttpClient "monitor" when mTLS is enabled for monitors
if (configuration.getBackendState() != null && configuration.getBackendState().isMonitorMtlsEnabled()) {
validateMonitorMtlsConfig(configuration.getServerConfig());
}
Map<String, UserConfiguration> presetUsers = configuration.getPresetUsers();

oauthManager = getOAuthManager(configuration);
Expand All @@ -127,6 +137,28 @@ public HaGatewayProviderModule(HaGatewayConfiguration configuration)
queryHistoryManager = new HaQueryHistoryManager(jdbi, configuration.getDataStore().getJdbcUrl().startsWith("jdbc:oracle"));
}

private static void validateMonitorMtlsConfig(Map<String, String> serverConfig)
{
String[] requiredKeys = new String[] {
MONITOR_HTTP_CLIENT_KEY_STORE_PATH,
MONITOR_HTTP_CLIENT_KEY_STORE_PASSWORD,
MONITOR_HTTP_CLIENT_TRUST_STORE_PATH,
MONITOR_HTTP_CLIENT_TRUST_STORE_PASSWORD
Comment on lines +145 to +146
Copy link
Member

@oneonestar oneonestar Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A user could skip setting truststore. Airlift would fallback to some system default truststore.

};
List<String> missing = new ArrayList<>();
for (String key : requiredKeys) {
String value = serverConfig.get(key);
if (value == null || value.isBlank()) {
missing.add(key);
}
}
if (!missing.isEmpty()) {
throw new IllegalArgumentException(
"backendState.monitorMtlsEnabled=true requires monitor HttpClient TLS configuration. Missing: "
+ String.join(", ", missing));
}
}

private LbOAuthManager getOAuthManager(HaGatewayConfiguration configuration)
{
AuthenticationConfiguration authenticationConfiguration = configuration.getAuthentication();
Expand Down