Procedure for ARC on GHES

[casanova-21]

Hi, all. 👋 I was able to get ARC working with ephemeral, auto-scaling runners triggered via webhook on our internal release candidate GHES 3.3.0.rc1 using a self-signed certificate. The process is much simpler for getting ARC working on GHEC due to the fact that github.com is signed by DigiCert and is trusted by both the controller and the runners. In attempting to get ARC connected to GHES, there were problems with both the controller's and the runners' SSL connections back to the GHES appliance. With the help of our internal Actions team, we worked through how to get the cert from the GHES appliance to be made available on both the runner and the controller.

We'd like to use this discussion to figure out how to make the process more efficient and what changes may need to be made to the Helm chart(s). I haven't been able to test this yet with a GHES appliance enabled with Let's Encrypt as my GHES instances are deployed into AWS and it won't me allow to sign them with the AWS domain.

Here's the procedure:

Configuring the actions-runner-controller(ARC) for GHES 3.3.0.rc1

This procedure assumes you're using a self-signed certificate for your GHES appliance, and the commands are being executed from a laptop/workstation unless otherwise specified. The ARC will be deployed at the org level for an organization called harkonnens. I used EKS in AWS with an EKS node group for the Kubernetes cluster.

1. Create EKS cluster with at least one node group.
2. aws eks --region us-east-1 update-kubeconfig --name <EKS cluster name>
3. kubectl config set-context <context name>
4. helm repo add jetstack https://charts.jetstack.io
5. git clone [email protected]:actions-runner-controller/actions-runner-controller.git
6. helm repo update
7. helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.6.0 --set installCRDs=true
8. kubectl create ns actions-runner-system
9. On your GHES instance, create a PAT with the appropriate scope for ARC.
10. kubectl create secret generic controller-manager -n actions-runner-system --from-literal=github_token=<PAT value>
11. The self-signed certificate is located on your appliance at /etc/haproxy/ssl.crt. Copy the contents of this file into a local file on your workstation. For this example we will name the file ghe.crt.
12. kubectl create secret generic ghecert -n actions-runner-system --from-file=ghe.crt (Secret for the GHES cert to be used by the actions-runner-controller)
13. kubectl create secret generic ghecert --from-file=ghe.crt (Secret for the GHES cert to be used by the runners which will be deployed into the default namespace)
14. Create a file called values.yaml with the following contents:

githubWebhookServer:
  enabled: true
  secret:
    create: true
    name: "github-webhook-server"
    github_webhook_secret_token: "<some token string>"
  service:
    type: LoadBalancer
    ports:
      - port: 80
        targetPort: http
        protocol: TCP
        name: http
        nodePort: 32080

15. Edit your local copy of the Helm chart(cloned in step 5) actions-runner-controller/templates/deployment.yaml to expand these volumeMounts and volumes blocks to include mounting the ghecert secret that contains our GHES certificate as the file /etc/ssl/certs/ghe.crt on the container:

        volumeMounts:
        - mountPath: "/etc/actions-runner-controller"
          name: secret
          readOnly: true
        - mountPath: /tmp
          name: tmp
        - mountPath: /tmp/k8s-webhook-server/serving-certs
          name: cert
          readOnly: true
        - mountPath: /etc/ssl/certs
          name: ghe-cert
          readOnly: true

and

      volumes:
      - name: secret
        secret:
          secretName: {{ include "actions-runner-controller.secretName" . }}
      - name: cert
        secret:
          defaultMode: 420
          secretName: {{ include "actions-runner-controller.servingCertName" . }}
      - name: ghe-cert
        secret:
          secretName: ghecert
      - name: tmp
        emptyDir: {}

16. helm upgrade --install -f values.yaml -n actions-runner-system actions-runner-controller ./actions-runner-controller

Point the ARC at your GHES appliance.

17. kubectl set env deploy -n actions-runner-system actions-runner-controller -c manager GITHUB_ENTERPRISE_URL=https://<hostname of your GHES appliance or load balancer> --namespace actions-runner-system

Tell the controller where to look for the GHES certificate

18. kubectl set env deploy -n actions-runner-system actions-runner-controller -c manager SSL_CERT_DIR=/etc/ssl/certs --namespace actions-runner-system

19. Create a runnerdeployment.yaml with the following contents:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: k8s-runner-deployment
spec:
  template:
    spec:
      organization: harkonnens
      env: []
      labels:
      - k8s-arc-runner
      volumes:
        - name: ghe-cert
          secret:
            secretName: ghecert
      volumeMounts:
        - mountPath: /etc/ssl/certs
          name: ghe-cert
          readOnly: true

20. kubectl apply -f runnerdeployment.yaml
    If you look at the logs for the RunnerDeployment pod, you should now see it connected to GitHub and listening for jobs.

21. Create an hra.yaml with the following contents(tweak the settings as desired):

apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: hra-csa
spec:
 scaleDownDelaySecondsAfterScaleOut: 60
 scaleTargetRef:
   name: k8s-runner-deployment
 minReplicas: 1
 maxReplicas: 5
 scaleUpTriggers:
 - githubEvent: {}
   duration: "30s"

22. kubectl apply -f hra.yaml
23. At this point, you should now be able to see a single runner in an Idle state in the Settings/Actions/Runners for your org on the GHES appliance.
24. You can now create a webhook at the org or repo level with the following settings:

Paylord URL: http://<DNS name of the load balancer in front of the controller deployed by step 16>
Content type: application/json
Secret: <Use the token value from your values.yaml created in Step 14 e.g. 1234567890>
Events: Workflow jobs

25. At this point you can create a helloworld workflow_dispatch job to test out the webhook. If you launch multiple workflows at once, you should see the pods scale up to our maximum of 5 runner pods as the jobs queue up and trigger the autoscaling.

One question I have is regarding the webhook server. The load balancer services that we spin up is listening on port 80. Will this support 443? I believe I've tried changing the values.yml to use 443 for the service, and the instance behind the load balancer never went into service.

[casanova-21]

I've also confirmed that if you're using Let's Encrypt, there's no need to pass the certificate to the controller and runners. It's already trusted.

[toast-gear]

With #952 merged (chart released soon) you no longer need to hand edit your deployment.yaml in step 15 if you are using Helmt to install actions-runner-controller

[casanova-21]

@toast-gear Thanks for getting this update done. I'm ready to test it out but am not sure how to properly pull the changes down having not worked with helm very much prior to this. If I do a helm update and helm pull, I still see the older deployment.yaml. I can pull the deployment.yaml from this repo and place it in my local helm copy but would prefer to understand the right way to do this.

NAME                                              	CHART VERSION	APP VERSION	DESCRIPTION
actions-runner-controller/actions-runner-contro...	0.15.0       	0.20.3     	A Kubernetes controller that operates self-host...
jetstack/cert-manager                             	v1.6.1       	v1.6.1     	A Helm chart for cert-manager
jetstack/cert-manager-approver-policy             	v0.2.0       	v0.2.0     	A Helm chart for cert-manager-approver-policy
jetstack/cert-manager-csi-driver                  	v0.2.0       	v0.2.0     	A Helm chart for cert-manager-csi-driver
jetstack/cert-manager-csi-driver-spiffe           	v0.1.0       	v0.1.0     	A Helm chart for cert-manager-csi-driver-spiffe
jetstack/cert-manager-istio-csr                   	v0.3.1       	v0.3.0     	A Helm chart for istio-csr
jetstack/cert-manager-trust                       	v0.1.1       	v0.1.0     	A Helm chart for cert-manager-trust```

[toast-gear]

We need to release a new version, it's just in master atm so you'd need to clone the repo and install reference the chart locally. I was hoping to bundle it with a few other changes before doing another chart release but perhaps I'll just release it as is.

[casanova-21]

@brunocous Hi! 👋 Thanks for doing the work in this Issue. Can you provide an example for the formatting to be used in the values.yaml? I'm trying a few different ways, and they're all failing. I'm sure I'm not formatting it properly.

## specify additional volumes to mount in the manager container, this can be used
## to specify additional storage of material or to inject files from ConfigMaps
## into the running container
additionalVolumes:
  name: ghe-cert
  secret:
    secretName: ghecert

## specify where the additional volumes are mounted in the manager container
additionalVolumeMounts: []
  mountPath: /etc/ssl/certs
  name: ghe-cert
  readOnly: true

This is producing this error:
> helm upgrade --install -f values.yaml -n actions-runner-system actions-runner-controller ./actions-runner-controller Error: UPGRADE FAILED: failed to create resource: Deployment.apps "actions-runner-controller" is invalid: [spec.template.spec.volumes[2].secret: Forbidden: may not specify more than 1 volume type, spec.template.spec.containers[0].volumeMounts[1].name: Not found: "tmp"]

cc @toast-gear

[gscho]

@casanova-21 I opened a PR to update the nindent settings and there is an example in the PR description for the formatting.

[casanova-21]

@gscho Thank you for getting that fixed and also for figuring out the settings to use in values.yml for the SSL dir and the GHES URL. I was able to get the ARC deployed successfully with the following settings in my values.yml:

githubEnterpriseServerURL: "https://<hostname of the GHES instance>"

env:
  SSL_CERT_DIR: "/etc/ssl/certs"

additionalVolumeMounts:
- mountPath: /etc/ssl/certs
  name: ghe-cert
  readOnly: true
additionalVolumes:
- name: ghe-cert
  secret:
    secretName: ghecert

githubWebhookServer:
  enabled: true
  secret:
    create: true
    name: "github-webhook-server"
    github_webhook_secret_token: "1234567890"
  service:
    type: LoadBalancer
    ports:
      - port: 80
        targetPort: http
        protocol: TCP
        name: http
        nodePort: 32080

@toast-gear Once you get a new helm release done, let me know and I'll update the documentation above.

I have a couple of additional questions. The first is that we still can't seem to get this working using https for the webhook server load balancer. @gscho was able to see in the code that it binds to port 8000 so we have tried updating the values.yml service definition to look like this:

  service:
    type: LoadBalancer
    ports:
      - port: 443
        targetPort: 8000
        protocol: TCP
        name: https

The instance successfully comes into the service in the load balancer, but when configuring the webhook in the repo to use https instead of http, the delivery response that comes back says "We couldn't deliver this payload: server gave HTTP response to HTTPS client" in the webhook deliveries section.

The other question I have is regarding the mounting to the ssl certificate at /etc/ssl/certs for both the runner and controller. I believe there are already a bunch of certs in that directory before we're mounting our GHES certificate in that location. Are we impacting anything negatively by doing this such that it would make sense to mount it in a different location?