Bug: `slack_api_url_file` not respected by mimir alertmanager so secrets are required in the values file

[jessebot]

What is the bug?

Hoi Grafana team!

Grafana Mimir's pod in Kubernetes throws the following in the logs:

ts=2025-07-21T13:24:33.9987957Z caller=multitenant.go:725 level=warn component=MultiTenantAlertmanager msg="error applying config" err="unable to load fallback configuration for anonymous: no global Slack API URL set" user=anonymous

Even though I have the global.slack_api_url_file parameter set in my fallback config. Slack alerts are not working because of this, even though the individual receivers do have their slack api url files also set.

Please let me know if you need any further info and if this isn't intended to be supported, what I should do to hide secret data instead?

Thank you for any help you can provide 🙏

How to reproduce it?

Via the latest helm chart values, I use this fallback config:

alertmanager:
  enabled: true

  # secrets with slackURLs
  extraVolumes:
    - name: alertmanager-api-urls
      secret:
        secretName: alertmanager-api-urls

  extraVolumeMounts:
    - name: alertmanager-api-urls
      mountPath: /etc/secrets/alertmanager
      readOnly: true

  fallbackConfig: |
    global:
      slack_api_url_file: '/etc/secrets/alertmanager/slack_url'

# let me know if you need my whole fallbackconfig but I've added the most crucial part

the secret that has the url looks like this (but I've redacted the actual secret):

apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-api-urls
  namespace: monitoring
type: Opaque
data:
  slack_url: redacted

What did you think would happen?

It should respect the global.slack_api_url_file parameter as this is respected in Prometheus alertmanager (which I have running in the same cluster working fine). The reason it matters is that this parameter contains sensitive information. I expect this based on the docs that say:

Each tenant has an Alertmanager configuration that defines notifications receivers and alerting routes. The Mimir Alertmanager uses the same configuration file that the Prometheus Alertmanager uses.

What was your environment?

kubernetes: v1.30.13-eks-5d4a308
Helm chart version: 5.8.0-weekly.347

Let me know if you need anything else!

Any additional context to share?

No response

[jessebot]

If there's a way to provide the entire config as a file, I could pass it in as a Secret, but I don't see a place in the helm chart to do that right now. Is there a file that the alertmanager always looks at for the config in the docker container? If that's the case, I can just pull in the secret data in an ExternalSecret and I'll have a work around. Right now, I cannot afford to check in secret data such as passwords and token to git though, which is where I keep my helm values files.

Edit: Ah! I found a work around as we can specify the fallbackconfig file here:

mimir/operations/helm/charts/mimir-distributed/values.yaml

Line 165 in dd00fb2

fallback_config_file: /configs/alertmanager_fallback_config.yaml

Edit again: No, that changing that doesn't work :(

[jessebot]

Ok, so an ask for helping with a work around would be one of the following:

• document where to put the default config so that mimir alertmanager finds it (can't actually pop a shell into the pod because it's not allowed, so I can't go in and take a look in the docker container to figure it out myself)
• accept the fallback_config_file as an existing Kubernetes Secret (right now we HAVE to pass it in as a string to the helm chart, and there's no other options)