Bug: install failed

[daddyauden]

What is the bug?

I just follow the steps in https://grafana.com/docs/helm-charts/mimir-distributed/latest/get-started-helm-charts/

steps:

1. kubectl create namespace monitoring
2. helm repo add grafana https://grafana.github.io/helm-charts
3. helm repo update
4. helm -n monitoring install mimir grafana/mimir-distributed
5. kubectl -n monitoring get pods

NAME                                        READY   STATUS             RESTARTS        AGE
mimir-alertmanager-0                        0/1     CrashLoopBackOff   4 (35s ago)    5m57s
mimir-compactor-0                           0/1     Running            3 (3m3s ago)   5m57s
mimir-distributor-7664846b46-5txn7          0/1     CrashLoopBackOff   6 (9s ago)     5m58s
mimir-ingester-zone-a-0                     0/1     Error              5              5m58s
mimir-ingester-zone-b-0                     0/1     CrashLoopBackOff   4 (28s ago)    5m57s
mimir-ingester-zone-c-0                     0/1     CrashLoopBackOff   4 (26s ago)    5m58s
mimir-make-minio-buckets-5.4.0-xfdfl        1/1     Running            4 (55s ago)    5m58s
mimir-minio-5477c4c7b4-z2hzc                1/1     Running            0              5m59s
mimir-minio-post-job-5f28c                  0/2     CrashLoopBackOff   4 (13s ago)    5m57s
mimir-nginx-7c65ff9dfd-k7ct2                1/1     Running            0              5m59s
mimir-overrides-exporter-84fcc88767-9nbhj   1/1     Running            0              5m59s
mimir-querier-56c7f8864b-8dnpx              0/1     CrashLoopBackOff   5 (65s ago)    5m58s
mimir-querier-56c7f8864b-8pcdb              0/1     Running            4 (60s ago)    5m58s
mimir-query-frontend-5995d7b875-wfzdl       0/1     CrashLoopBackOff   6 (7s ago)     5m57s
mimir-query-scheduler-5fdfc8d578-6gztr      1/1     Running            0              5m58s
mimir-query-scheduler-5fdfc8d578-98dhr      1/1     Running            0              5m58s
mimir-rollout-operator-5bc66dbb95-6wvw7     1/1     Running            0              5m59s
mimir-ruler-67596574dc-2svbg                0/1     Running            0              5m59s
mimir-store-gateway-zone-a-0                0/1     Error              4 (66s ago)    5m58s
mimir-store-gateway-zone-b-0                0/1     Running            4 (59s ago)    5m58s
mimir-store-gateway-zone-c-0                0/1     CrashLoopBackOff   4 (37s ago)    5m58s

logs:

1. kubectl -n monitoring logs mimir-alertmanager-0

ts=2025-07-21T03:13:19.762905996Z caller=main.go:225 level=info msg="Starting application" version="(version=2.16.0, branch=HEAD, revision=b4f36da)"
ts=2025-07-21T03:13:19.766667772Z caller=server.go:368 level=info msg="server listening on addresses" http=[::]:8080 grpc=[::]:9095
ts=2025-07-21T03:13:19.783621373Z caller=modules.go:1000 level=info msg="Starting Alertmanager in classic mode"
ts=2025-07-21T03:13:19.783642123Z caller=featurecontrol.go:97 level=warn msg="Classic mode enabled"
ts=2025-07-21T03:13:19.784511799Z caller=memberlist_client.go:463 level=info msg="Using memberlist cluster label and node name" cluster_label= node=mimir-alertmanager-0-193eec75
ts=2025-07-21T03:13:19.78575232Z caller=module_service.go:82 level=info msg=starting module=activity-tracker
ts=2025-07-21T03:13:19.785792886Z caller=module_service.go:82 level=info msg=starting module=sanity-check
ts=2025-07-21T03:13:19.785860859Z caller=sanity_check.go:32 level=info msg="Checking directories read/write access"
ts=2025-07-21T03:13:19.786067798Z caller=sanity_check.go:37 level=info msg="Directories read/write access successfully checked"
ts=2025-07-21T03:13:19.786082278Z caller=sanity_check.go:39 level=info msg="Checking object storage config"
ts=2025-07-21T03:13:29.786788045Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="alertmanager storage: unable to successfully send a request to object storage: Get \"http://mimir-minio.monitoring.svc:9000/mimir-ruler/?location=\": context deadline exceeded"
ts=2025-07-21T03:13:41.246381743Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="alertmanager storage: unable to successfully send a request to object storage: Get \"http://mimir-minio.monitoring.svc:9000/mimir-ruler/?location=\": context deadline exceeded"
ts=2025-07-21T03:13:49.79284971Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="alertmanager storage: unable to successfully send a request to object storage: Get \"http://mimir-minio.monitoring.svc:9000/mimir-ruler/?location=\": dial tcp: lookup mimir-minio.monitoring.svc on 30.30.0.10:53: no such host"
ts=2025-07-21T03:14:04.695803963Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="alertmanager storage: unable to successfully send a request to object storage: Get \"http://mimir-minio.monitoring.svc:9000/mimir-ruler/?location=\": context deadline exceeded"
ts=2025-07-21T03:14:19.330218856Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="alertmanager storage: unable to successfully send a request to object storage: Get \"http://mimir-minio.monitoring.svc:9000/mimir-ruler/?location=\": context deadline exceeded"
ts=2025-07-21T03:14:34.055491168Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="alertmanager storage: unable to successfully send a request to object storage: Get \"http://mimir-minio.monitoring.svc:9000/mimir-ruler/?location=\": context deadline exceeded"
ts=2025-07-21T03:14:39.061363228Z caller=sanity_check.go:44 level=info msg="Object storage config successfully checked"
ts=2025-07-21T03:14:39.061467763Z caller=module_service.go:82 level=info msg=starting module=server
ts=2025-07-21T03:14:39.061591111Z caller=module_service.go:82 level=info msg=starting module=runtime-config
ts=2025-07-21T03:14:39.061610662Z caller=module_service.go:82 level=info msg=starting module=memberlist-kv
ts=2025-07-21T03:14:39.061786526Z caller=mimir.go:958 level=error msg="module failed" module=memberlist-kv err="starting module memberlist-kv: invalid service state: Stopping, expected: Running"
ts=2025-07-21T03:14:39.061896168Z caller=mimir.go:958 level=error msg="module failed" module=runtime-config err="starting module runtime-config: context canceled"
ts=2025-07-21T03:14:39.06206575Z caller=mimir.go:958 level=error msg="module failed" module=alertmanager err="failed to start alertmanager, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: starting module memberlist-kv: invalid service state: Stopping, expected: Running"
ts=2025-07-21T03:14:39.063594844Z caller=server_service.go:55 level=info msg="server stopped"
ts=2025-07-21T03:14:39.063616416Z caller=module_service.go:120 level=info msg="module stopped" module=server
ts=2025-07-21T03:14:39.063653666Z caller=module_service.go:120 level=info msg="module stopped" module=sanity-check
ts=2025-07-21T03:14:39.066895458Z caller=module_service.go:120 level=info msg="module stopped" module=activity-tracker
ts=2025-07-21T03:14:39.066923725Z caller=mimir.go:945 level=info msg="Application stopped"
ts=2025-07-21T03:14:39.067003213Z caller=log.go:134 level=error msg="error running application" err="failed services\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:1001\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:227\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"

2.kubectl -n monitoring logs mimir-minio-5477c4c7b4-z2hzc

INFO: Formatting 1st pool, 1 set(s), 1 drives per set.
INFO: WARNING: Host local has more than 0 drives of set. A host failure will result in data becoming unavailable.
MinIO Object Storage Server
Copyright: 2015-2025 MinIO, Inc.
License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
Version: RELEASE.2024-12-18T13-15-44Z (go1.23.4 linux/amd64)

API: http://20.20.225.30:9000  http://127.0.0.1:9000 
WebUI: http://20.20.225.30:9001 http://127.0.0.1:9001   

Docs: https://docs.min.io

3. kubectl -n monitoring logs mimir-make-minio-buckets-5.4.0-xfdfl

Connecting to MinIO server: http://mimir-minio:9000
mc: <ERROR> Unable to initialize new alias from the provided credentials. Get "http://mimir-minio:9000/probe-bsign-3m0atuog0wywsmf5ej3wuv41bggas1/?location=": dial tcp: lookup mimir-minio: i/o timeout.
"Failed attempts: 1"
Added `myminio` successfully.
Suspending versioning for 'mimir-tsdb'
mc: <ERROR> Unable to suspend versioning. Get "http://mimir-minio:9000/mimir-tsdb/?location=": dial tcp: lookup mimir-minio: i/o timeout.

4. kubectl -n monitoring logs mimir-querier-56c7f8864b-8dnpx

ts=2025-07-21T03:16:53.026595612Z caller=main.go:225 level=info msg="Starting application" version="(version=2.16.0, branch=HEAD, revision=b4f36da)"
ts=2025-07-21T03:16:53.050443092Z caller=server.go:368 level=info msg="server listening on addresses" http=[::]:8080 grpc=[::]:9095
ts=2025-07-21T03:16:53.070106087Z caller=memberlist_client.go:463 level=info msg="Using memberlist cluster label and node name" cluster_label= node=mimir-querier-56c7f8864b-8dnpx-02d390ec
ts=2025-07-21T03:16:53.078929902Z caller=worker.go:144 level=info msg="Starting querier worker connected to query-scheduler" scheduler=mimir-query-scheduler-headless.monitoring.svc:9095
ts=2025-07-21T03:16:53.080013274Z caller=module_service.go:82 level=info msg=starting module=sanity-check
ts=2025-07-21T03:16:53.080036421Z caller=module_service.go:82 level=info msg=starting module=usage-stats
ts=2025-07-21T03:16:53.080093472Z caller=sanity_check.go:32 level=info msg="Checking directories read/write access"
ts=2025-07-21T03:16:53.080123897Z caller=sanity_check.go:37 level=info msg="Directories read/write access successfully checked"
ts=2025-07-21T03:16:53.08013223Z caller=sanity_check.go:39 level=info msg="Checking object storage config"
ts=2025-07-21T03:16:53.080093738Z caller=module_service.go:82 level=info msg=starting module=activity-tracker
ts=2025-07-21T03:17:03.081199509Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Get \"http://mimir-minio.monitoring.svc:9000/mimir-tsdb/?location=\": context deadline exceeded"
ts=2025-07-21T03:17:13.089521997Z caller=sanity_check.go:44 level=info msg="Object storage config successfully checked"
ts=2025-07-21T03:17:13.089578056Z caller=module_service.go:82 level=info msg=starting module=server
ts=2025-07-21T03:17:13.089628975Z caller=module_service.go:82 level=info msg=starting module=runtime-config
ts=2025-07-21T03:17:13.089658129Z caller=module_service.go:82 level=info msg=starting module=memberlist-kv
ts=2025-07-21T03:17:13.089846552Z caller=mimir.go:958 level=error msg="module failed" module=memberlist-kv err="starting module memberlist-kv: invalid service state: Stopping, expected: Running"
ts=2025-07-21T03:17:13.089893213Z caller=mimir.go:958 level=error msg="module failed" module=ingester-ring err="failed to start ingester-ring, because it depends on module runtime-config, which has failed: context canceled"
ts=2025-07-21T03:17:13.089929974Z caller=mimir.go:958 level=error msg="module failed" module=store-queryable err="failed to start store-queryable, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: starting module memberlist-kv: invalid service state: Stopping, expected: Running"
ts=2025-07-21T03:17:13.089973558Z caller=mimir.go:958 level=error msg="module failed" module=querier err="failed to start querier, because it depends on module distributor-service, which has failed: context canceled"
ts=2025-07-21T03:17:13.089992391Z caller=mimir.go:958 level=error msg="module failed" module=distributor-service err="failed to start distributor-service, because it depends on module ingester-ring, which has failed: context canceled"
ts=2025-07-21T03:17:13.090010802Z caller=mimir.go:958 level=error msg="module failed" module=runtime-config err="starting module runtime-config: context canceled"
ts=2025-07-21T03:17:13.090201408Z caller=server_service.go:55 level=info msg="server stopped"
ts=2025-07-21T03:17:13.090224761Z caller=module_service.go:120 level=info msg="module stopped" module=server
ts=2025-07-21T03:17:13.09027625Z caller=module_service.go:120 level=info msg="module stopped" module=sanity-check
ts=2025-07-21T03:17:13.090494417Z caller=seed.go:127 level=warn msg="failed to read cluster seed file from object storage" err="Get \"http://mimir-minio.monitoring.svc:9000/mimir-tsdb/__mimir_cluster/mimir_cluster_seed.json\": context canceled"
ts=2025-07-21T03:17:13.090546476Z caller=module_service.go:120 level=info msg="module stopped" module=usage-stats
ts=2025-07-21T03:17:13.094391915Z caller=module_service.go:120 level=info msg="module stopped" module=activity-tracker
ts=2025-07-21T03:17:13.094422928Z caller=mimir.go:945 level=info msg="Application stopped"
ts=2025-07-21T03:17:13.094517696Z caller=log.go:134 level=error msg="error running application" err="failed services\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:1001\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:227\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"

I can not provide all the issue, in general I can not deploy.

How to reproduce it?

just do it with steps in https://grafana.com/docs/helm-charts/mimir-distributed/latest/get-started-helm-charts/

What did you think would happen?

deploy succeed.

What was your environment?

mimir-helm(5.7.0)
helm(3.18.4)
k8s(1.33.2)

Any additional context to share?

No response

[mkhong9999]

hi me also same issue about mimir-distributed-5.7.0, mimir-distributed-5.1.2
mimir-alertmanager pod , mimir-ingester-zone pods, mimir-ruler pod, mimir-store-gateway-zone pods

before when i installed mimir-distributed-5.1.2 not issues but version update to 5.7.0 have a issue
also rollback has same issue

ts=2025-07-24T07:08:29.732778718Z caller=main.go:225 level=info msg="Starting application" version="(version=2.16.0, branch=HEAD, revision=b4f36da)" ts=2025-07-24T07:08:29.740984476Z caller=server.go:368 level=info msg="server listening on addresses" http=[::]:8080 grpc=[::]:9095 ts=2025-07-24T07:08:29.759219082Z caller=memberlist_client.go:463 level=info msg="Using memberlist cluster label and node name" cluster_label= node=mimir-ingester-zone-a-0-6fa7df7d ts=2025-07-24T07:08:29.761243495Z caller=ingester.go:445 level=info msg="TSDB idle compaction timeout set" timeout=1h11m45.6493889s ts=2025-07-24T07:08:29.762442716Z caller=module_service.go:82 level=info msg=starting module=usage-stats ts=2025-07-24T07:08:29.763062236Z caller=module_service.go:82 level=info msg=starting module=sanity-check ts=2025-07-24T07:08:29.763161948Z caller=sanity_check.go:32 level=info msg="Checking directories read/write access" ts=2025-07-24T07:08:29.76328458Z caller=sanity_check.go:34 level=error msg="Unable to access directory" err="ingester: failed to access directory ./tsdb/: open .check: read-only file system" ts=2025-07-24T07:08:29.763355301Z caller=mimir.go:958 level=error msg="module failed" module=sanity-check err="starting module sanity-check: invalid service state: Failed, expected: Running, failure: ingester: failed to access directory ./tsdb/: open .check: read-only file system" ts=2025-07-24T07:08:29.763378781Z caller=mimir.go:958 level=error msg="module failed" module=ingester-service err="failed to start ingester-service, because it depends on module activity-tracker, which has failed: context canceled" ts=2025-07-24T07:08:29.763386081Z caller=module_service.go:82 level=info msg=starting module=active-groups-cleanup-service ts=2025-07-24T07:08:29.763397402Z caller=mimir.go:958 level=error msg="module failed" module=server err="failed to start server, because it depends on module activity-tracker, which has failed: context canceled" ts=2025-07-24T07:08:29.763426872Z caller=module_service.go:82 level=info msg=starting module=activity-tracker ts=2025-07-24T07:08:29.763448192Z caller=mimir.go:958 level=error msg="module failed" module=memberlist-kv err="failed to start memberlist-kv, because it depends on module server, which has failed: context canceled" ts=2025-07-24T07:08:29.763459763Z caller=mimir.go:958 level=error msg="module failed" module=active-groups-cleanup-service err="starting module active-groups-cleanup-service: context canceled" ts=2025-07-24T07:08:29.763476453Z caller=mimir.go:958 level=error msg="module failed" module=ingester-ring err="failed to start ingester-ring, because it depends on module activity-tracker, which has failed: context canceled" ts=2025-07-24T07:08:29.766481423Z caller=mimir.go:958 level=error msg="module failed" module=activity-tracker err="starting module activity-tracker: context canceled" ts=2025-07-24T07:08:29.823810494Z caller=module_service.go:120 level=info msg="module stopped" module=usage-stats ts=2025-07-24T07:08:29.823851245Z caller=mimir.go:945 level=info msg="Application stopped" ts=2025-07-24T07:08:29.823964347Z caller=log.go:134 level=error msg="error running application" err="failed services\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:1001\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:227\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"

[mkhong9999]

https://community.grafana.com/t/mimir-upgrade-to-helm-chart-4-4-1-now-getting-bucket-errors-in-ingester-and-store-gateway/89272

see this guide about delete pvc
i tried but not solved...