Support kubernetes secret for providing user-data to cloud-init

[umangachapagain]

Context

When using Hegel, currently, if we want to provide user-data to cloud-init, we need to pass it via Hardware spec.
For example:

apiVersion: tinkerbell.org/v1alpha1
kind: Hardware
metadata:
  name: hw1
  namespace: tink-system
spec:
  userData: |
    #cloud-config
    ---
    user: <USERNAME>
    password: <PLAINTEXT_PASSWORD>
    chpasswd: {expire: False}
    ssh_pwauth: True

Hegel then serves the user-data on HEGEL_IP:HEGEL_PORT/2009-04-04/user-data and
meta-data on HEGEL_IP:HEGEL_PORT/2009-04-04/meta-data/

cloud-init can read these user-data and meta-data when datasource is configured correctly.

This behavior works ok as long as user-data does not contain any sensitive information. However, it could still cause formatting issues with user-data.

Proposal

If user-data contains sensitive data like passwords, license keys etc it might not be desirable to put these in Hardware spec in plaintext format which can be read by anyone with read access to Hardware CR.

To help with this, we could move the user-data to a kubernetes secret object and reference that object in Hardware spec.
This secret object reference can be used by Hegel to pull user-data.
New spec example:

apiVersion: tinkerbell.org/v1alpha1
kind: Hardware
metadata:
  name: hw1
  namespace: tink-system
spec:
  userDataRef:
     name: <SECRET_NAME>
     namespace: <SECRET_NAMESPACE>

This approach has a few benefits,

• We can avoid sensitive user-data information in Hardware spec.
• Access to this Secret can be restricted to only required users i.e. cluster-admin and hegel serviceaccount.
• Secret stores data in base64 encoded form. This helps preserve formatting of user-data by creating a secret directly from the user-data file.

[displague]

This brings crossplane-contrib/provider-cloudinit#8 to mind.

I don't know that Hardware should be concerned with CloudInit directly, but a CloudInit controller could piece together secrets, configmaps, and plain text blobs into a secret that userdata could consume.

This is just food for thought.

In the "New spec example" snippet, userDataRef should be userDataSecretRef.
The namespace would be omitted as it should need to match the Hardware resource.