TLS support for elastic/open search visibility store

[infurio]

I've been trying to setup a temporal cluster with Cassandra as the default store and OpenSearch as the visibility store.

After going through the CRDs and source it looks like all the pieces are in place to support TLS access to elastic/open search for the visibility store.

For example:

apiVersion: temporal.io/v1beta1
kind: TemporalCluster
metadata:
  name: prod-test
  namespace: temporal
spec:
  version: 1.25.2
  numHistoryShards: 1
  admintools:
    enabled: true
  ui:
    enabled: true
  persistence:
    defaultStore:
      passwordSecretRef:
        name: cassandra-password
        key: PASSWORD
      cassandra:
        hosts:
          - test-cluster-dc1-service.k8ssandra-operator.svc.test-us-east4
        keyspace: temporal
        port: 9042
        user: test-cluster-superuser
    visibilityStore:
      tls:
        enableHostVerification: false
        caFileRef:
          name: opensearch-cluster-http-cert
          key: ca.crt
        certFileRef:
          name: opensearch-cluster-http-cert
          key: tls.crt
        keyFileRef:
          name: opensearch-cluster-http-cert
          key: tls.key
        enabled: true
        serverName: opensearch-cluster.opensearch.svc.test-us-east4
      elasticsearch:
        enableHealthcheck: true
        enableSniff: true
        closeIdleConnectionsInterval: 10s
        indices:
          visibility: temporal_visibility
        url: https://opensearch-cluster.opensearch.svc.test-us-east4:9200
        username: admin
        version: v8
      passwordSecretRef:
        name: opensearch-password
        key: PASSWORD

However the schema-script-runner running setup-visibility-schema.sh fails with TLS errors from the curl requests:

curl: (60) SSL certificate problem: unable to get local issuer certificate

The curl requests in

temporal-operator/internal/resource/persistence/template.go

Line 85 in c4be5cd

curl --fail --user "{{ .Username }}":"${{ .PasswordEnvVar }}" -X PUT "{{ .URL }}/_cluster/settings" -H "Content-Type: application/json" --data-binary @/etc/temporal/schema/elasticsearch/visibility/cluster_settings_{{ .Version }}.json --write-out "\n"

don't seem to support the addition of TLS certificates.

It looks like the tls section in the visibility store manifest section is used for the other database options just not for open/elastic search, is that being added?

I don't see a work around at the moment apart from switching to the helm based installation of temporal, or modifying the operator to support this feature.

Temporal added support for elasticsearch TLS with this PR: temporalio/temporal#5134