Can not create JWT auth method with TLS enabled on Nomad

[conamu]

Feature Description

Hello,
I have a small cluster running, setup with TLS. In order to use the Workload Identities in Nomad and do proper ACL things I need to setup this JWT Auth Method between consul and nomad. I use self-signed certificates.

When I supply this configuration:

{
  "JWKSUrl": "https://10.0.1.3:4646/.well-known/jwks.json",
  "JWKSCACert": "REDACTED",
  "JWTSupportedAlgs": ["RS256"],
  "BoundAudiences": ["consul.io"],
  "ClaimMappings": {
    "nomad_namespace": "nomad_namespace",
    "nomad_job_id": "nomad_job_id",
    "nomad_task": "nomad_task",
    "nomad_service": "nomad_service"
  }
}

I get this error:
Failed to create new auth method: Unexpected response code: 500 (Invalid Auth Method: error checking JWKSURL: fetching keys oidc: get keys failed Get "https://10.0.1.3:4646/.well-known/jwks.json": remote error: tls: certificate required)

I searched and tried a lot, and there seems to not be a way to supply a client certificate and key to this auth method to make it able to access the needed JWKS.

My suggestion:

Add 2 more fields to supply a client certificate and a client key, or even a .p12 file.

(I am not entirely sure if we would need the key here as well though)

...
"JWKSClientCert": "...",
"JWKSClientKey": "...",
...

Or maybe its just not at all documented since I don't see people having issues with this apparently. One workaround of course is to proxy it but that shouldn't be necessary...

Use Case(s)

Its basically impossible to run Jobs with proper security measures. Consul ACLs cant be used with services running on nomad in a connect service mesh when Consul ACLs are setup without a general key in the Agent and with a default deny rule.

[tgross]

Hi @conamu! I'm from the Nomad team but happened to be looking for a different Consul issue and saw this. The Nomad docs on securing via TLS include the following recommendation (ref https://developer.hashicorp.com/nomad/docs/secure/traffic/tls#configuring-nomad):

verify_https_client requires HTTP API clients to present a certificate signed by the same CA as Nomad's certificate. It may be disabled to allow HTTP API clients (e.g. Nomad CLI, Consul, or curl) to communicate with the HTTPS API without presenting a client-side certificate. If verify_https_client is enabled only HTTP API clients presenting a certificate signed by the same CA as Nomad's certificate are allowed to access Nomad.

If you click thru to the tls.verify_https_client docs you'll see:

By default, verify_https_client is set to false, which is safe so long as ACLs are enabled.

Although mTLS is absolutely required for Nomad agent-to-agent RPC to be secured, it's totally fine if you don't have the HTTP clients present client certs so long as you've got non-mutual TLS + ACLs enabled. This is accounted for in our published security model requirements.

That being said, this would be a nice feature for Consul too. 😁

[conamu]

Hey @tgross!
In my understanding browsers accessing the UI wouldnt get the prompt for certificates as well then, correct? I would really like everything to require mTLS since we are not using VPNs. For now we halted the integration of the service mesh since it doesn't seem to work correctly for permanent tcp connections between same instances of services for now and we will need this later along the line.
Another good option would be to allow a second non tls port to be specified which should only be exposed on localhost and to cluster servers.

[tgross]

In my understanding browsers accessing the UI wouldnt get the prompt for certificates as well then, correct?

Correct. Most users don't want that because of how painful distributing client certs is to browsers. But if you want that and want mTLS on the HTTP API, then you're going to need this Consul feature as well for sure.