Skip to content
This repository has been archived by the owner on Jun 6, 2024. It is now read-only.

CAP 1.5.2 Notes (tag 2.20.x)

Jump to bottom
Jan Dubois edited this page Jan 31, 2020 · 6 revisions

CAP-1.5.2 Notes

Most information in this file is specific to the CAP-1.5.2 release.

So far it includes information about SCF, UAA, and minibroker, but is missing information about Stratos and the metrics provider.

Namespaces

UAA can be installed separately in it's own namespace, or be included in the SCF namespace via the enable.uaa=true option, in which case it shares the mysql/proxy/secret generation services with SCF.

Cloud Foundry applications run inside diego-cell pods inside the SCF namespace when using the Diego scheduler. Under Eirini the applications will run as individual pods inside a separate eirini namespace.

Minibroker is installed into a separate namespace. All services installed via minibroker will be placed into the minibroker namespace as well. It is possible, but not recommended, to install minibroker into the SCF namespace as well.

PodSecurityPolicies (PSPs)

SCF and UAA

Unless the users explicitly specifies a default PSP via kube.psp.default the SCF (and UAA) helm charts will generate a default PSP themselves:

spec:
  allowPrivilegeEscalation: true
  allowedCapabilities:
  - "*"
  defaultAllowPrivilegeEscalation: true
  fsGroup:
    rule: "RunAsAny"
  hostPorts:
  - max: 65535
    min: 0
  privileged: true
  runAsUser:
    rule: "RunAsAny"
  seLinux:
    rule: "RunAsAny"
  supplementalGroups:
    rule: "RunAsAny"
  volumes:
  - "configMap"
  - "secret"
  - "emptyDir"
  - "downwardAPI"
  - "projected"
  - "persistentVolumeClaim"
  - "nfs"

Any user provided PSP must be a superset of the capabilities granted by the default PSP.

Many of the SCF pods run containers within containers, either via the "Bosh process manager" to isolate Cloud Foundry jobs from each other, or via "Garden Runc" to run user applications inside diego-cell pods. This requires the outer container to be privileged.

Typically the container starts running as root, but after initial setup drops some privileges and runs under the regular vcap user.

minibroker

The minibroker chart has no provision to provide or specify a PSP; if a particular PSP is required to deploy a specific service, then a role authorizing the use of this PSP for the minibroker namespace must be bound to the minibroker service account.

Service Accounts and Role Bindings

SCF (and UAA) namespace

The SCF (and UAA when using a separate UAA) namespace contains multiple service accounts. Each one is authorized to use the default PSP described above (or overridden via kube.psp.default).

default

No additional role bindings; used when no special resource access is required.

configgin

The configgin service account can create/update/delete secrets. It can patch stateful sets and services. It is used to configure SCF pods during startup, importing properties from other roles (aka "Bosh links") and generally expanding property values with information only available at runtime.

The only pod running directly with the configgin service account is the configgin-helper pod. It periodically stores the token for this service account in a secret, in case the cluster is using expiring service account tokens.

This token is then imported by all pods using Configgin via an environment variable. Configgin will remove this token once initialization is complete, so the actual jobs running in the containers don't have access to it.

active-passive

The active-passive service account can patch pods and services. It is used by service that use multiple instances in HA mode, only one of which will be active at any moment in time. Their readiness probe is going to select which of the pods is the active/leader one, and will add a label to the pods or services, which can be used as a selector by kube.

secret-generator

The secret-generator service account is used by the secret generator and provides full access to configmaps and secrets. It will generate passwords, SSH keys and SSL certificates for all internal secrets, unless they are overridden by the user during helm install.

When SCF is installed with Eirini enabled, then the secret-generator service account will also get a cluster role binding to create, get, and delete certificate signing requests to create a trusted certificate for the Eirini registry.

When env.KUBE_CSR_AUTO_APPROVAL is true then an additional cluster role binding to allow approval of signing requests is added.

garden-runc

The garden-runc service account is only used by diego-cell (and therefore only when Diego is used as the scheduler). It allows querying its own pod definition to find out which node the pod is running on. It also has a cluster role binding to query the node, which allows it to determine its availability zone. This is required to let the Diego scheduler spread multiple instances of the same app accross multiple zones.

eirini

The eirini service account is only used by the Eirini Scheduler. It has full access to the eirini namespace, where it will create pods for application instances.

In addition the eirini service account can create and delete mutating web hook admission controller configurations. This is needed by various Eirinix extensions to provide access to persisten volumes or allow ssh access.

Certificates

Subject Names

All the internal certificates are generated by the secret-generator by default. Starting with 1.5.2 the role names and subject names for each certificate are documented in the values.yaml file of the Helm chart.

The names may include these template variables:

  • DOMAIN

    The base domain of the SCF installation; speicied during helm install.

  • KUBERNETES_CLUSTER_DOMAIN

    The kubernetes cluster domain, often cluster.local by default, but can be modified by cluster setup, or via helm install.

  • KUBERNETES_NAMESPACE

    The value provided via the helm install --namespace option.

The role name is expanded to a list of 9 subject names:

  • {{.ROLENAME}}
  • *.{{.ROLENAME}}
  • {{.ROLENAME}}.{{.KUBERNETES_NAMESPACE}}.svc
  • *.{{.ROLENAME}}.{{.KUBERNETES_NAMESPACE}}.svc
  • {{.ROLENAME}}.{{.KUBERNETES_NAMESPACE}}.svc.{{.KUBERNETES_CLUSTER_DOMAIN}}
  • *.{{.ROLENAME}}.{{.KUBERNETES_NAMESPACE}}.svc.{{.KUBERNETES_CLUSTER_DOMAIN}}
  • *.{{.ROLENAME}}-set
  • *.{{.ROLENAME}}-set.{{.KUBERNETES_NAMESPACE}}.svc
  • *.{{.ROLENAME}}-set.{{.KUBERNETES_NAMESPACE}}.svc.{{.KUBERNETES_CLUSTER_DOMAIN}}

Any additional names are appended to this list.

In values.yaml of the Helm chart the UAA_SERVER certificate is documented as:

# This certificate uses the role name "uaa" and the additional names
# "uaa.{{.DOMAIN}}" and "*.uaa.{{.DOMAIN}}".
UAA_SERVER_CERT: ~

This expands to:

$ openssl s_client -showcerts -connect cf.uaa.cf-dev.io:2793 </dev/null 2>/dev/null \
| openssl x509 -noout -text \
| perl -nE 'if (/DNS:/) { s/ //g && say for split /,/ }'
DNS:uaa
DNS:*.uaa
DNS:uaa.cf.svc
DNS:*.uaa.cf.svc
DNS:uaa.cf.svc.cluster.local
DNS:*.uaa.cf.svc.cluster.local
DNS:*.uaa-set
DNS:*.uaa-set.cf.svc
DNS:*.uaa-set.cf.svc.cluster.local
DNS:uaa.cf-dev.io
DNS:*.uaa.cf-dev.io

127.0.0.1 certificates

Some services need to connect to other services running inside the same container. Since hairpin-NAT isn't supported in all configurations, the services may include 127.0.0.1 on their certificate to allow local connection as well.

The Eirini registry running in the bits pod uses a nodePort service. It uses a certificate for 127.0.0.1 that is not signed by the secret-generator itself, but via a certificate signing request to the kube certificate signer (see secret-generator service account above). This is necessary because the certificate needs to be trusted by the container runtime running on each node of the cluster.

UAA_CA_CERT

Starting with CAP-1.5.2 you no longer need to set UAA_CA_CERT when using an external UAA with a cert signed by a well known CA. It is only needed when you use an external UAA with either a secret-generator generated cert, or a self-signed cert.

External Database

SCF can be configured to use an external database. This must be configured during the initial installation and cannot be changed afterwards.

database-seeder

The database-seeder runs during installation and created databases inside the service for the various clients (cloud controller, diego, uaa, etc). It uses these configuration variables:

env.DB_EXTERNAL_HOST: Hostname for an external database server to use for the CF-internal databases. If not set, the internal database is used and the remaining DB_EXTERNAL_* variables are ignored.

env.DB_EXTERNAL_USER_HOST_SUFFIX: A suffix that has to be appended to every user name for the external database; usually @host. Must include the @ sign. Empty by default.

env.DB_EXTERNAL_PORT: Port for an external database server to use for the CF-internal databases. Default: 3306.

env.DB_EXTERNAL_SSL_MODE: SSL configuration for the external database server. Valid values: false, skip-verify, preferred, and true. Default: true.

env.DB_EXTERNAL_USER: Administrator user name for an external database server; this is required to create the necessary databases. DB_EXTERNAL_USER_HOST_SUFFIX will be appended to this user name, so this variable should include just the user name without the host suffix.

secrets.DB_EXTERNAL_PASSWORD: Administrator password for an external database server; this is required to create the necessary databases.

The user and password are only used by the seeder to create the databases. All clients will then use database specific usernames and passwords.

credhub

secrets.CREDHUB_DB_CA_CERT: CA trusted for making TLS connections to targeted database server.

env.CREDHUB_DB_HOST_VALIDATION: Enables hostname verification for TLS connections to targeted database server. Default: true.

env.CREDHUB_DB_REQUIRE_TLS: Requires only TLS connections to targeted database server. Default: true.

uaa

env.UAADB_TLS: Use TLS connection for UAA database. Valid options are: enabled (use TLS with full certificate validation), enabled_skip_hostname_validation (use TLS but skip validation of common and alt names in the host certificate), enabled_skip_all_validation (use TLS but do not validate anything about the host certificate), and disabled (do not use TLS). Default: enabled.

Clone this wiki locally