ECK APM no output defined, please define one under the output section

[kaykhan]

I'm settng up APM using the eck-apm-server chart https://artifacthub.io/packages/helm/elastic/eck-apm-server but i get the following error

: Error: no output defined, please define one under the output section

    count: 1
    version: 8.15.2
    fullnameOverride: ${local.apm_server_name}
    spec:
      elasticsearchRef:
        name: ${data.terraform_remote_state.eck_elasticsearch.outputs.elasticsearch_name}
      kibanaRef: 
        name: ${data.terraform_remote_state.eck_kibana.outputs.kibana_name}
      http:
        service:
          spec:
            type: NodePort

I came across an example with config.output in the docs - https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-apm-advanced-configuration.html#k8s-apm-existing-es

Just wondering if i should be setting this given that im also providing an elasticsearchRef?

In the examples - https://github.com/elastic/cloud-on-k8s/blob/main/deploy/eck-stack/examples/apm-server/basic.yaml#L33-L52 - there is no output set only the ref

kubectl logs -f apm-server-prod-apm-server-7b684854f7-5ct82 -n elastic
-system

-system
{"log.level":"info","@timestamp":"2024-10-28T08:22:31.781Z","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.(*Beat).init","file.name":"beatcmd/beat.go","file.line":143},"message":"Home path: [/usr/share/apm-server] Config path: [/usr/share/apm-server] Data path: [/usr/share/apm-server/data] Logs path: [/usr/share/apm-server/logs]","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-28T08:22:31.783Z","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.(*Beat).init","file.name":"beatcmd/beat.go","file.line":150},"message":"Beat ID: 371aae38-b8e7-4c30-b3e4-e6c3ab1880dc","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-28T08:22:31.785Z","log.logger":"api","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start","file.name":"api/server.go","file.line":69},"message":"Starting stats endpoint","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-28T08:22:31.785Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo","file.name":"beatcmd/beat.go","file.line":574},"message":"Beat info","service.name":"apm-server","system_info":{"beat":{"path":{"config":"/usr/share/apm-server","data":"/usr/share/apm-server/data","home":"/usr/share/apm-server","logs":"/usr/share/apm-server/logs"},"type":"apm-server","uuid":"371aae38-b8e7-4c30-b3e4-e6c3ab1880dc"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-10-28T08:22:31.785Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo","file.name":"beatcmd/beat.go","file.line":582},"message":"Build info","service.name":"apm-server","system_info":{"build":{"commit":"1f0cd7772b3718cc1e5e57e6fec76abd53bc4ce9","time":"2024-09-19T08:15:14.000Z","version":"8.15.2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-10-28T08:22:31.785Z","log.logger":"api","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start.func1","file.name":"api/server.go","file.line":71},"message":"Metrics endpoint listening on: 127.0.0.1:5068 (configured: localhost)","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-28T08:22:31.785Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo","file.name":"beatcmd/beat.go","file.line":585},"message":"Go runtime info","service.name":"apm-server","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":4,"version":"go1.22.6"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-10-28T08:22:31.786Z","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.adjustMaxProcs.diffInfof.func2","file.name":"beatcmd/maxprocs.go","file.line":68},"message":"maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-28T08:22:31.786Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo","file.name":"beatcmd/beat.go","file.line":589},"message":"Host info","service.name":"apm-server","system_info":{"host":{"architecture":"x86_64","native_architecture":"x86_64","boot_time":"2024-09-30T09:44:01Z","containerized":false,"name":"apm-server-prod-apm-server-7b684854f7-5ct82","ip":["127.0.0.1/8","::1/128","10.1.15.117/32","fe80::ec25:90ff:fe5e:963c/64"],"kernel_version":"6.1.109-118.189.amzn2023.x86_64","mac":["ee:25:90:5e:96:3c"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"22.04.5 LTS (Jammy Jellyfish)","major":22,"minor":4,"patch":5,"codename":"jammy"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-10-28T08:22:31.787Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo","file.name":"beatcmd/beat.go","file.line":618},"message":"Process info","service.name":"apm-server","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/apm-server","exe":"/usr/share/apm-server/apm-server","name":"apm-server","pid":1,"ppid":0,"seccomp":{"mode":"disabled","no_new_privs":false},"start_time":"2024-10-28T08:22:31.650Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-10-28T08:22:31.787Z","log.logger":"api","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start.func1","file.name":"api/server.go","file.line":73},"message":"Stats endpoint (127.0.0.1:5068) finished: accept tcp 127.0.0.1:5068: use of closed network connection","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-28T08:22:31.788Z","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.(*Beat).Run","file.name":"beatcmd/beat.go","file.line":380},"message":"apm-server stopped.","service.name":"apm-server","ecs.version":"1.6.0"}
Error: no output defined, please define one under the output section
Usage:
  apm-server run [flags]

Flags:
  -N, --N                   Disable actual publishing for testing
      --cpuprofile string   Write cpu profile to file
  -h, --help                help for run
      --httpprof string     Start pprof http server
      --memprofile string   Write memory profile to this file

Global Flags:
  -E, --E setting=value      Configuration overwrite
  -c, --c string             Configuration file, relative to path.config (default "apm-server.yml")
  -d, --d stringArray        Enable certain debug selectors
  -e, --e                    Log to stderr and disable syslog/file output
      --environment string   Set the environment in which the process is running (default "default")
      --path.config string   Configuration path
      --path.data string     Data path
      --path.home string     Home path
      --path.logs string     Logs path
      --strict.perms         Strict permission checking on config files (default true)
  -v, --v                    Log at INFO level

After explicit setting config.output - run into 2 errors.

     output.elasticsearch:
        hosts: ["${data.terraform_remote_state.eck_elasticsearch.outputs.elasticsearch_name}-es-http.elastic-system.svc:9200"]
        username: "elastic"
        password: ${data.terraform_remote_state.eck_elasticsearch.outputs.elasticsearch_elastic_user_password}
        protocol: "https"

{"log.level":"error","@timestamp":"2024-10-28T09:04:44.199Z","log.logger":"agentcfg","log.origin":{"function":"github.com/elastic/apm-server/internal/agentcfg.(*ElasticsearchFetcher).Run.func1","file.name":"agentcfg/elasticsearch.go","file.line":150},"message":"refresh cache error: context deadline exceeded","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-10-28T09:04:50.509Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.waitReady","file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}

Again i'm really not sure if i should be manually setting output.elasticsearch here.