Does not seem to detect provider-helm release is ready

[james-nofrixion]

What happened?

I have created the following composition pipeline step:

  - step: sequence-resources
    functionRef:
      name: function-sequencer
    input:
      apiVersion: sequencer.fn.crossplane.io/v1beta1
      kind: Input
      rules:
        - sequence:
          - aws-load-balancer-controller-release
          - metrics-server-release
          - external-secrets-release
          - workload-identity-webhook-release
          - cert-manager-release
          - rabbitmq-cluster-operator-release
          - crossplane-release

I would expect that when the metrics-server-release is ready that the external-secrets-release is deployed. This does not happen.

From kubectl get releases:

devops-2406-aws-load-balancer-controller-release   aws-load-balancer-controller   1.8.1     True     True    deployed   1          Install complete   36m
devops-2406-ingress-nginx-release                  ingress-nginx                  4.10.1    True     True    deployed   1          Install complete   36m
devops-2406-kubernetes-replicator-release          kubernetes-replicator          2.9.2     True     True    deployed   1          Install complete   36m
devops-2406-metrics-server-release                 metrics-server                 3.12.1    True     True    deployed   1          Install complete   14m

The following error is in the events:

12m         Normal    ComposeResources                  xcluster/devops-2406-nswbh                                             Pipeline step "sequence-resources": Delaying creation of resource "external-secrets-release" because "metrics-server-release" is not ready or does not exist yet

Note the release names without the devops-2406 prefix is the composition resource name. The function does seem to wait for the load balancer release before installing the metrics server chart but after that nothing gets installed.

How can we reproduce it?

Composition steps to render charts and sequence function:

  - step: render-helm-releases
    functionRef:
      name: function-go-templating
    input:
      apiVersion: gotemplating.fn.crossplane.io/v1beta1
      kind: GoTemplate
      source: Inline  # Or Remote, if you want to pull templates from git.
      inline:
        template: |
          {{- $clusterName := .observed.composite.resource.spec.parameters.id }}
          {{ with .observed.composite.resource.spec.parameters }}
          {{- range $index, $chart := .helmCharts }}
          ---
          apiVersion: helm.crossplane.io/v1beta1
          kind: Release
          metadata:
            name: {{ $clusterName }}-{{ $chart.name }}-release
            annotations:
              {{ setResourceNameAnnotation (print $chart.name "-release") }}
          spec:
            forProvider:
              chart:
                name: {{ $chart.name }}
                repository: {{ $chart.repository }}
                version: {{ $chart.version }}
              namespace: {{ $chart.namespace }}
              {{- if $chart.set}}
              set:
                {{ toYaml $chart.set | nindent 6 }}
              {{- end }}
              {{- if $chart.values }}
              values:
                {{ toYaml $chart.values | nindent 6 }}
              {{- end }}
              {{- if $chart.wait }}
              wait: {{ $chart.wait }}
              {{- end }}
            providerConfigRef:
              name: {{ $clusterName }}-helm-provider
            rollbackLimit: 3
            {{- end }} # range
          {{ end }}    # with

  - step: patch-and-transform-helm-charts
    functionRef:
      name: crossplane-contrib-function-patch-and-transform
    input:
      apiVersion: pt.fn.crossplane.io/v1beta1
      kind: Resources
      resources:
      - name: aws-load-balancer-controller-release
        patches:
        - type: FromCompositeFieldPath
          fromFieldPath: status.eks.awsLoadBalancerControllerRoleArn
          toFieldPath: spec.forProvider.values.serviceAccount.annotations[eks.amazonaws.com/role-arn]
        - type: FromCompositeFieldPath
          fromFieldPath: spec.parameters.id
          toFieldPath: spec.forProvider.values.clusterName
        - type: FromCompositeFieldPath
          fromFieldPath: spec.parameters.region
          toFieldPath: spec.forProvider.values.region
        - type: FromCompositeFieldPath
          fromFieldPath: status.eks.vpcId
          toFieldPath: spec.forProvider.values.vpcId
        - type: ToCompositeFieldPath
          fromFieldPath: status.eks.aws-load-balancer-state
          toFieldPath: spec.atProvider.state

      - name: ingress-nginx-release
        patches:
        - type: FromCompositeFieldPath
          fromFieldPath: spec.parameters.id
          toFieldPath: spec.forProvider.values.controller.service.annotations[service.beta.kubernetes.io/aws-load-balancer-name]
          transforms:
            - type: string
              string:
                fmt: '%s-nlb'
                type: Format

      - name: cert-manager-release
        patches:
        - type: FromCompositeFieldPath
          fromFieldPath: status.eks.certManagerRoleArn
          toFieldPath: spec.forProvider.values.serviceAccount.annotations[eks.amazonaws.com/role-arn]
  - step: sequence-resources
    functionRef:
      name: function-sequencer
    input:
      apiVersion: sequencer.fn.crossplane.io/v1beta1
      kind: Input
      rules:
        - sequence:
          - aws-load-balancer-controller-release
          - metrics-server-release
          - external-secrets-release
          - workload-identity-webhook-release
          - cert-manager-release
          - rabbitmq-cluster-operator-release
          - crossplane-release

XR manifest extract for rendering the helm charts:

    helmCharts:
      # Only install crossplane in the 'it-opos-xxxx' cluster
      - name: crossplane
        repository: https://charts.crossplane.io/stable
        version: "1.16.0"
        namespace: crossplane-system
        set:
          - name: args
            value: '{"--enable-usages"}'
      - name: metrics-server
        repository: https://kubernetes-sigs.github.io/metrics-server/
        version: 3.12.1
        namespace: kube-system
        values:
          replicas: 2
          podDisruptionBudget:
            enabled: true
            maxUnavailable: 1
      - name: aws-load-balancer-controller
        repository: https://aws.github.io/eks-charts
        version: 1.8.1
        namespace: kube-system
        values: 
          # enableServiceMutatorWebhook: false
          serviceAccount:
            annotations:
              eks.amazonaws.com/role-arn: "" # leave blank for template to fill in
            create: true
            name: aws-load-balancer-controller
          serviceAnnotations:
            service.beta.kubernetes.io/aws-load-balancer-attributes: load_balancing.cross_zone.enabled=true
        wait: true
      - name: ingress-nginx
        repository: https://kubernetes.github.io/ingress-nginx
        version: "4.10.1"
        namespace:  ingress-nginx
        values:
          controller:
            replicaCount: 2
            enableAnnotationValidations: true
            service:
              type: LoadBalancer
              annotations:
                service.beta.kubernetes.io/aws-load-balancer-name: "" # leave blank for template to fill in
                service.beta.kubernetes.io/aws-load-balancer-type: external
                service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
                service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
                service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: tcp
                service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: /healthz
                service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true
                service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
      - name: kubernetes-replicator
        repository: https://helm.mittwald.de
        version: "2.9.2"
        namespace: kubernetes-replicator
      - name: external-secrets
        repository: https://charts.external-secrets.io
        version: "0.9.18"
        namespace: external-secrets
      - name: cert-manager
        repository: https://charts.jetstack.io/
        version: "v1.15.0"
        namespace: cert-manager
        values:
          installCRDs: true
          serviceAccount:
            name: cert-manager
            annotations:
               eks.amazonaws.com/role-arn: "" # this get's patched in the pipeline
          securityContext:
            fsGroup: 1001
      - name: workload-identity-webhook
        repository: https://azure.github.io/azure-workload-identity/charts
        version: "1.2.2"
        namespace: azure-workload-identity-system
        values:
          azureTenantID: xxxxxxxxxxxxxxx    
      - name: rabbitmq-cluster-operator
        namespace: rabbitmq-system
        repository: https://charts.bitnami.com/bitnami
        version: 4.3.6
        values:
          useCertManager: true
          extraDeploy:
          - apiVersion: v1
            kind: Namespace
            metadata:
              name: rabbitmq
              labels:
                app: rabbitmq
          - apiVersion: rabbitmq.com/v1beta1
            kind: RabbitmqCluster
            metadata:
              name: rabbitmq-ops
              namespace: rabbitmq
            spec:
              persistence:
                storageClassName: gp2
                storage: 1Gi
              rabbitmq:
                additionalConfig: |
                  disk_free_limit.absolute = 50MB
                additionalPlugins:
                - rabbitmq_consistent_hash_exchange
                - rabbitmq_delayed_message_exchange
              replicas: 3
              resources:
                limits:
                  cpu: 500m
                  memory: 1Gi
                requests:
                  cpu: 200m
                  memory: 512Mi
              service:
                type: ClusterIP
          - apiVersion: networking.k8s.io/v1
            kind: Ingress
            metadata:
              annotations:
              name: rabbitmq-ops-ingress
              namespace: rabbitmq
            spec:
              ingressClassName: nginx
              rules:
              - host: rabbitmq-ops.example.com
                http:
                  paths:
                  - backend:
                      service:
                        name: rabbitmq-ops
                        port:
                          number: 15672
                    path: /
                    pathType: Prefix
              tls:
              - hosts:
                - rabbitmq-ops.example.com
                secretName: wildcard-tls-secret
    ```

### What environment did it happen in?

Function versions: 
- function-sequencer:v0.1.2
- function-patch-and-transform:v0.6.0
- function-go-templating:v0.4.1

Crossplane version: 1.16
AWS provider family: 1.17.0

All running in AWS EKS 1.30