TLS: Support separate certificate for HTTPS like Consul does

[olljanat]

Proposal

Consul supports configuration where tls.defaults is used to configure internal CA for mTLS in gRPC and RPC protocols.
Documentation also mentions that for security reason mTLS CA should be dedicated for this purpose.

Same time another CA (private or public) can be used for HTTPS traffic.

Full configuration example:

tls {
    defaults{
      verify_incoming = true
      verify_outgoing = true
      verify_server_hostname = true
      ca_file = "/opt/consul/agent-certs/ca.crt"
      cert_file = "/opt/consul/agent-certs/agent.crt"
      key_file = "/opt/consul/agent-certs/agent.key"
    }
}

tls {
    https {
        verify_incoming = false
        verify_outgoing = true
        ca_file = "/opt/node-cert/ca.crt"
        cert_file = "/opt/node-cert/node.crt"
        key_file = "/opt/node-cert/node.key"
    }
}

Currently Nomad supports only one CA which is used for all purposes and it would be nice to have this same functionality available in it.

Use-cases

Target is use mTLS for everything and avoid certificate errors when admins accessing to Consul UI with server name/IP address without going through load balancer/reverse proxy.

Attempted Solutions

Option 1: Enable TLS only to RPC
It is possible to enable TLS to RPC only and use non-encrypted HTTP for web traffic like this:

tls {
    http = false
    rpc = true
    ca_file = "/etc/nomad.d/agent-certs/ca.crt"
    cert_file = "/etc/nomad.d/agent-certs/agent.crt"
    key_file = "/etc/nomad.d/agent-certs/agent.key"
    rpc_upgrade_mode = false
}

However nowadays web browsers warns about that situation.

Option 2: Use same TLS for everything
It is possible to use certificate from external CA for mTLS too and it is possible to increase security by enabling verify_server_hostname = true setting because then these certificates must also contain name server.<region>.nomad

tls {
    http = true
    rpc = true
    ca_file = "/opt/node-cert/ca.crt"
    cert_file = "/opt/node-cert/node.crt"
    key_file = "/opt/node-cert/node.key"
    rpc_upgrade_mode = false
    verify_https_client = false
    verify_server_hostname = true
}

However it is bit hacky solution and that why not preferred.

Option 3: Offline Root CA + dedicated CA in Vault
I haven't fully tested this configuration but it seems that it would be possible to use Vault with external root to issue these certificates and then do configuration in way that:

1. External clients trust to offline root CA
2. ca_file value in Nomad configuration uses intermediate CA instead of actual root CA.

However, I'm not sure if that causes some other issues which I haven't think about yet.

[jrasell]

Hi @olljanat and thanks for raising this issue. This is an interesting idea and something that makes sense; I'll add it to our backlog, but note we would likely need some discussion on it as the changes would be broad and will need to be backwards compatible.

[olljanat]

@jrasell Sure backward compatibility is important.

It seems that in Consul it was handled by leaving old syntax intact so it is also possible configure example ca_file in root level of config file instead of tls.defaults.ca_file and it still can be overwritten with tls.https.ca_file

So in Nomad configuration like this would be backward compatible:

tls {
    http = true
    rpc = true
    ca_file = "/path/to/default-ca.crt"
    https {
        ca_file = "/path/to/https-ca.crt"
    }
}

[arodd]

@olljanat can you describe the use case that is driving you to want to define separate certificates/CA's for each protocol? If we allowed customization of the RPC server/client verification domains(currently locked to "server.region.nomad") would this help allow you to continue using one common cert/ca for both protocols?

Or is it more of an operational concern where outward facing certs(HTTPS) are generated by a centralized CA that doesn't allow as much automation flexibility, and the RPC certs can be maintained by a separate less widely trusted CA that you have more control over at the platform level when spinning up new clients?

[olljanat]

can you describe the use case that is driving you to want to define separate certificates/CA's for each protocol?

@arodd I don't necessarily need separate certificate for each protocol but more like separating client and server certificates.

Perhaps this guidance (which I wrote my colleagues as part of reviewing their Consul+Nomad+Vault setup) helps understanding what I mean:

---

When dealing with certificates it is important to understand difference between TLS server certificates and TLS client certificates.

Server certificates are used for:

1. Encrypt connection between client and server.
2. Verify server identity by comparing certificate's Subject and Subject Alternative Name values to fully qualified domain name (FQDN) which client used when it connected to server.

When we are talking about connections between out internal servers, the encryption of connection is more important feature than identity verification. That why it is recommended to use certificates issued by our internal certificate authority (which is already trusted by all computers in our environment) for this purpose and use long validity period of those certificates (1 year or more).

Client certificates are used for authenticate to target service. Valid certificate provides access to service with identity defined in certificate and it can be used by anyone who has access to this certificate. That why it is recommended to use certificates issued by certificate authority which dedicated (which is not trusted by any computer outside of Nomad environment) for this purpose and use short validity period of those certificates (max 7 days).

---

Or is it more of an operational concern where outward facing certs(HTTPS) are generated by a centralized CA that doesn't allow as much automation flexibility, and the RPC certs can be maintained by a separate less widely trusted CA that you have more control over at the platform level when spinning up new clients?

There is both operational and security concerns because:

• CA outside of Nomad environment has limitations in both technology and process which why it cannot be used to create and rotate client certificates.
• Option 3 in my initial message cannot be used because it would allow Vault admins to create certificates for any services even if those are outside of Nomad environment.