Open
Description
What did you do to encounter the bug?
Steps to reproduce the behavior:
- Follow the docs in https://github.com/mongodb/mongodb-kubernetes-operator/blob/master/docs/deploy-configure.md
- Run kubectl -n mongodb get mongodbcommunity
- Run kubectl -n mongodb get pod
- Run kubectl -n mongodb describe mongodb-0
The only differences from https://github.com/mongodb/mongodb-kubernetes-operator/blob/master/config/samples/mongodb.com_v1_mongodbcommunity_cr.yaml
- metadata.name: mongodb
- spec.version: 7.0.15 (although i've tried version 6.0.5 and had the same behaviour)
- spec.members: 1 (after testing with 2 - same behaviour independent of amount)
- statefulSet.spec.volumeClaimTempates: added definitions for data-volume and logs-volume to give them some meaningfull sizes, had the same behaviour without the PVC templates.
What did you expect?
kubectl -n mongodb get mongodbcommunity
NAME PHASE VERSION
mongodb Running 7.0.15
kubectl -n mongodb get pod
NAME READY STATUS RESTARTS AGE
mongodb-0 2/2 Running 0 9m44s
mongodb-kubernetes-operator-7c967f54d4-vrhk4 1/1 Running 0 2d19h
What happened instead?
kubectl -n mongodb get mongodbcommunity
NAME PHASE VERSION
mongodb Pending
kubectl -n mongodb get pod
NAME READY STATUS RESTARTS AGE
mongodb-0 1/2 Running 0 9m44s
mongodb-kubernetes-operator-7c967f54d4-vrhk4 1/1 Running 0 2d19h
kubectl -n mongodb describe pod mongodb-0
(some output ommited)
Name: mongodb-0
Namespace: mongodb
Priority: 0
Service Account: mongodb-database
Status: Running
Containers:
mongod:
Image: docker.io/mongodb/mongodb-community-server:7.0.15-ubi8
Image ID: docker.io/mongodb/mongodb-community-server@sha256:bd2e8e00a36d89eeb67eb7886630eaeb68c445c8474fc8ed95286ee82456d44f
State: Running
Ready: True
Mounts:
/data from data-volume (rw)
/healthstatus from healthstatus (rw)
/hooks from hooks (rw)
/tmp from tmp (rw)
/var/lib/mongodb-mms-automation/authentication from mongodb-keyfile (rw)
/var/log/mongodb-mms-automation from logs-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-p89bx (ro)
mongodb-agent:
Image: quay.io/mongodb/mongodb-agent-ubi:108.0.2.8729-1
Image ID: quay.io/mongodb/mongodb-agent-ubi@sha256:dda6762d4b53da3230c8acc925aeaaa45fc2b3e4c38e180a83053ced1528306d
State: Running
Ready: False
Mounts:
/data from data-volume (rw)
/opt/scripts from agent-scripts (rw)
/tmp from tmp (rw)
/var/lib/automation/config from automation-config (ro)
/var/lib/mongodb-mms-automation/authentication from mongodb-keyfile (rw)
/var/log/mongodb-mms-automation from logs-volume (rw)
/var/log/mongodb-mms-automation/healthstatus from healthstatus (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-p89bx (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data-volume:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-volume-mongodb-0
ReadOnly: false
logs-volume:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: logs-volume-mongodb-0
ReadOnly: false
agent-scripts:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
automation-config:
Type: Secret (a volume populated by a Secret)
SecretName: mongodb-config
Optional: false
healthstatus:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
hooks:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
mongodb-keyfile:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-api-access-p89bx:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10m default-scheduler Successfully assigned mongodb/mongodb-0 to k3s-master-1-pi4
Normal SuccessfulAttachVolume 10m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-f5a8a0e6-61c7-439d-a6d7-6cfd693e012c"
Normal SuccessfulAttachVolume 10m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-73984433-b0c0-4f5b-b2ec-e568e2352e11"
Normal Pulling 10m kubelet Pulling image "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.9"
Normal Pulled 10m kubelet Successfully pulled image "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.9" in 834ms (834ms including waiting). Image size: 55380047 bytes.
Normal Created 10m kubelet Created container mongod-posthook
Normal Started 10m kubelet Started container mongod-posthook
Normal Pulling 10m kubelet Pulling image "quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.22"
Normal Pulled 10m kubelet Successfully pulled image "quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.22" in 613ms (613ms including waiting). Image size: 56850989 bytes.
Normal Created 10m kubelet Created container mongodb-agent-readinessprobe
Normal Started 10m kubelet Started container mongodb-agent-readinessprobe
Normal Pulling 10m kubelet Pulling image "docker.io/mongodb/mongodb-community-server:7.0.15-ubi8"
Normal Pulled 8m2s kubelet Successfully pulled image "docker.io/mongodb/mongodb-community-server:7.0.15-ubi8" in 2m10.069s (2m10.069s including waiting). Image size: 382255288 bytes.
Normal Created 8m2s kubelet Created container mongod
Normal Started 8m2s kubelet Started container mongod
Normal Pulling 8m2s kubelet Pulling image "quay.io/mongodb/mongodb-agent-ubi:108.0.2.8729-1"
Normal Pulled 8m1s kubelet Successfully pulled image "quay.io/mongodb/mongodb-agent-ubi:108.0.2.8729-1" in 776ms (776ms including waiting). Image size: 259631097 bytes.
Normal Created 8m1s kubelet Created container mongodb-agent
Normal Started 8m1s kubelet Started container mongodb-agent
Warning Unhealthy 7m49s kubelet Readiness probe failed: {"level":"info","ts":"2025-03-23T17:31:17.529Z","msg":"logging configuration: &{Filename:/var/log/mongodb-mms-automation/readiness.log MaxSize:5 MaxAge:0 MaxBackups:5 LocalTime:false Compress:false size:0 file:<nil> mu:{state:0 sema:0} millCh:<nil> startMill:{done:{_:{} v:0} m:{state:0 sema:0}}}"}
{"level":"info","ts":"2025-03-23T17:31:17.632Z","msg":"Mongod is not ready"}
{"level":"info","ts":"2025-03-23T17:31:17.632Z","msg":"Reached the end of the check. Returning not ready."}
2025-03-23 17:31:17.52957058 +0000 UTC m=+0.576564662 write error: can't open new logfile: open /var/log/mongodb-mms-automation/readiness.log: permission denied
2025-03-23 17:31:17.632262316 +0000 UTC m=+0.679256139 write error: can't open new logfile: open /var/log/mongodb-mms-automation/readiness.log: permission denied
2025-03-23 17:31:17.63252335 +0000 UTC m=+0.679517173 write error: can't open new logfile: open /var/log/mongodb-mms-automation/readiness.log: permission denied
Warning Unhealthy 7m49s kubelet Readiness probe failed: {"level":"info","ts":"2025-03-23T17:31:17.726Z","msg":"logging configuration: &{Filename:/var/log/mongodb-mms-automation/readiness.log MaxSize:5 MaxAge:0 MaxBackups:5 LocalTime:false Compress:false size:0 file:<nil> mu:{state:0 sema:0} millCh:<nil> startMill:{done:{_:{} v:0} m:{state:0 sema:0}}}"}
{"level":"info","ts":"2025-03-23T17:31:17.782Z","msg":"Mongod is not ready"}
{"level":"info","ts":"2025-03-23T17:31:17.783Z","msg":"Reached the end of the check. Returning not ready."}
Operator Information
- Operator Version 0.12.0
- MongoDB Image used 7.0.15, 6.0.5 (same behaviour on both)
Kubernetes Cluster Information
- Distribution: k3s on RPi (arm64, two nodes, RPi4 master and RPi3 worker)
- Version v1.31.2+k3s1
Additional context
Possibly same problem as: #1384 #1143 #949
The volumes are RWO, correctly provisioned and bound.
kubectl -n mongodb get mdbc -oyaml
apiVersion: v1
items:
- apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"mongodbcommunity.mongodb.com/v1","kind":"MongoDBCommunity","metadata":{"annotations":{},"name":"mongodb","namespace":"mongodb"},"spec":{"additionalMongodConfig":{"storage.wiredTiger.engineConfig.journalCompressor":"zlib"},"members":1,"security":{"authentication":{"modes":["SCRAM"]}},"statefulSet":{"spec":{"volumeClaimTemplates":[{"metadata":{"name":"data-volume"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"10Gi"}}}},{"metadata":{"name":"logs-volume"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}}}}]}},"type":"ReplicaSet","users":[{"db":"admin","name":"admin","passwordSecretRef":{"name":"admin-password"},"roles":[{"db":"admin","name":"clusterAdmin"},{"db":"admin","name":"userAdminAnyDatabase"}],"scramCredentialsSecretName":"my-scram"}],"version":"7.0.15"}}
creationTimestamp: "2025-03-23T17:28:25Z"
generation: 1
name: mongodb
namespace: mongodb
resourceVersion: "5233111"
uid: dfe7acc8-7de8-4cf2-a567-26433c2d5ac8
spec:
additionalMongodConfig:
storage.wiredTiger.engineConfig.journalCompressor: zlib
members: 1
security:
authentication:
ignoreUnknownUsers: true
modes:
- SCRAM
statefulSet:
spec:
volumeClaimTemplates:
- metadata:
name: data-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
- metadata:
name: logs-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
type: ReplicaSet
users:
- db: admin
name: admin
passwordSecretRef:
name: admin-password
roles:
- db: admin
name: clusterAdmin
- db: admin
name: userAdminAnyDatabase
scramCredentialsSecretName: my-scram
version: 7.0.15
status:
currentMongoDBMembers: 0
currentStatefulSetReplicas: 0
message: ReplicaSet is not yet ready, retrying in 10 seconds
mongoUri: ""
phase: Pending
kind: List
metadata:
resourceVersion: ""kubectl -n mongodb get sts -oyaml
apiVersion: v1
items:
- apiVersion: apps/v1
kind: StatefulSet
metadata:
creationTimestamp: "2025-03-23T17:28:25Z"
generation: 1
name: mongodb
namespace: mongodb
ownerReferences:
- apiVersion: mongodbcommunity.mongodb.com/v1
blockOwnerDeletion: true
controller: true
kind: MongoDBCommunity
name: mongodb
uid: dfe7acc8-7de8-4cf2-a567-26433c2d5ac8
resourceVersion: "5233118"
uid: b8243dc6-fd22-4a65-8c0f-ea658425980f
spec:
persistentVolumeClaimRetentionPolicy:
whenDeleted: Retain
whenScaled: Retain
podManagementPolicy: OrderedReady
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: mongodb-svc
serviceName: mongodb-svc
template:
metadata:
creationTimestamp: null
labels:
app: mongodb-svc
spec:
containers:
- args:
- ""
command:
- /bin/sh
- -c
- "\nif [ -e "/hooks/version-upgrade" ]; then\n\t#run post-start hook
to handle version changes (if exists)\n /hooks/version-upgrade\nfi\n\n#
wait for config and keyfile to be created by the agent\nwhile ! [ -f /data/automation-mongod.conf
-a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep
3 ; done ; sleep 2 ;\n\n# start mongod with this configuration\nexec mongod
-f /data/automation-mongod.conf;\n\n"
env:
- name: AGENT_STATUS_FILEPATH
value: /healthstatus/agent-health-status.json
image: docker.io/mongodb/mongodb-community-server:7.0.15-ubi8
imagePullPolicy: IfNotPresent
name: mongod
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /data
name: data-volume
- mountPath: /healthstatus
name: healthstatus
- mountPath: /hooks
name: hooks
- mountPath: /var/log/mongodb-mms-automation
name: logs-volume
- mountPath: /var/lib/mongodb-mms-automation/authentication
name: mongodb-keyfile
- mountPath: /tmp
name: tmp
- command:
- /bin/bash
- -c
- |-
current_uid=$(id -u)
declare -r current_uid
if ! grep -q "${current_uid}" /etc/passwd ; then
sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
export NSS_WRAPPER_PASSWD=/tmp/passwd
export LD_PRELOAD=libnss_wrapper.so
export NSS_WRAPPER_GROUP=/etc/group
fi
agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile /var/log/mongodb-mms-automation/automation-agent.log -logLevel INFO -maxLogFileDurationHrs 24
env:
- name: AGENT_STATUS_FILEPATH
value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
- name: AUTOMATION_CONFIG_MAP
value: mongodb-config
- name: HEADLESS_AGENT
value: "true"
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: quay.io/mongodb/mongodb-agent-ubi:108.0.2.8729-1
imagePullPolicy: Always
name: mongodb-agent
readinessProbe:
exec:
command:
- /opt/scripts/readinessprobe
failureThreshold: 40
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/scripts
name: agent-scripts
- mountPath: /var/lib/automation/config
name: automation-config
readOnly: true
- mountPath: /data
name: data-volume
- mountPath: /var/log/mongodb-mms-automation/healthstatus
name: healthstatus
- mountPath: /var/log/mongodb-mms-automation
name: logs-volume
- mountPath: /var/lib/mongodb-mms-automation/authentication
name: mongodb-keyfile
- mountPath: /tmp
name: tmp
dnsPolicy: ClusterFirst
initContainers:
- command:
- cp
- version-upgrade-hook
- /hooks/version-upgrade
image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.9
imagePullPolicy: Always
name: mongod-posthook
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /hooks
name: hooks
- command:
- cp
- /probes/readinessprobe
- /opt/scripts/readinessprobe
image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.22
imagePullPolicy: Always
name: mongodb-agent-readinessprobe
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/scripts
name: agent-scripts
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 2000
serviceAccount: mongodb-database
serviceAccountName: mongodb-database
terminationGracePeriodSeconds: 30
volumes:
- emptyDir: {}
name: agent-scripts
- name: automation-config
secret:
defaultMode: 416
secretName: mongodb-config
- emptyDir: {}
name: healthstatus
- emptyDir: {}
name: hooks
- emptyDir: {}
name: mongodb-keyfile
- emptyDir: {}
name: tmp
updateStrategy:
type: RollingUpdate
volumeClaimTemplates:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
name: data-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
volumeMode: Filesystem
status:
phase: Pending
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
name: logs-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
volumeMode: Filesystem
status:
phase: Pending
status:
availableReplicas: 0
collisionCount: 0
currentReplicas: 1
currentRevision: mongodb-d79cb44df
observedGeneration: 1
replicas: 1
updateRevision: mongodb-d79cb44df
updatedReplicas: 1
- apiVersion: apps/v1
kind: StatefulSet
metadata:
creationTimestamp: "2025-03-23T17:28:25Z"
generation: 1
name: mongodb-arb
namespace: mongodb
ownerReferences:
- apiVersion: mongodbcommunity.mongodb.com/v1
blockOwnerDeletion: true
controller: true
kind: MongoDBCommunity
name: mongodb
uid: dfe7acc8-7de8-4cf2-a567-26433c2d5ac8
resourceVersion: "5233113"
uid: 15468191-c29a-45f9-864d-0e34bc565081
spec:
persistentVolumeClaimRetentionPolicy:
whenDeleted: Retain
whenScaled: Retain
podManagementPolicy: OrderedReady
replicas: 0
revisionHistoryLimit: 10
selector:
matchLabels:
app: mongodb-svc
serviceName: mongodb-svc
template:
metadata:
creationTimestamp: null
labels:
app: mongodb-svc
spec:
containers:
- args:
- ""
command:
- /bin/sh
- -c
- "\nif [ -e "/hooks/version-upgrade" ]; then\n\t#run post-start hook
to handle version changes (if exists)\n /hooks/version-upgrade\nfi\n\n#
wait for config and keyfile to be created by the agent\nwhile ! [ -f /data/automation-mongod.conf
-a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep
3 ; done ; sleep 2 ;\n\n# start mongod with this configuration\nexec mongod
-f /data/automation-mongod.conf;\n\n"
env:
- name: AGENT_STATUS_FILEPATH
value: /healthstatus/agent-health-status.json
image: docker.io/mongodb/mongodb-community-server:7.0.15-ubi8
imagePullPolicy: IfNotPresent
name: mongod
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /data
name: data-volume
- mountPath: /healthstatus
name: healthstatus
- mountPath: /hooks
name: hooks
- mountPath: /var/log/mongodb-mms-automation
name: logs-volume
- mountPath: /var/lib/mongodb-mms-automation/authentication
name: mongodb-keyfile
- mountPath: /tmp
name: tmp
- command:
- /bin/bash
- -c
- |-
current_uid=$(id -u)
declare -r current_uid
if ! grep -q "${current_uid}" /etc/passwd ; then
sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
export NSS_WRAPPER_PASSWD=/tmp/passwd
export LD_PRELOAD=libnss_wrapper.so
export NSS_WRAPPER_GROUP=/etc/group
fi
agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile /var/log/mongodb-mms-automation/automation-agent.log -logLevel INFO -maxLogFileDurationHrs 24
env:
- name: AGENT_STATUS_FILEPATH
value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
- name: AUTOMATION_CONFIG_MAP
value: mongodb-config
- name: HEADLESS_AGENT
value: "true"
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: quay.io/mongodb/mongodb-agent-ubi:108.0.2.8729-1
imagePullPolicy: Always
name: mongodb-agent
readinessProbe:
exec:
command:
- /opt/scripts/readinessprobe
failureThreshold: 40
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/scripts
name: agent-scripts
- mountPath: /var/lib/automation/config
name: automation-config
readOnly: true
- mountPath: /data
name: data-volume
- mountPath: /var/log/mongodb-mms-automation/healthstatus
name: healthstatus
- mountPath: /var/log/mongodb-mms-automation
name: logs-volume
- mountPath: /var/lib/mongodb-mms-automation/authentication
name: mongodb-keyfile
- mountPath: /tmp
name: tmp
dnsPolicy: ClusterFirst
initContainers:
- command:
- cp
- version-upgrade-hook
- /hooks/version-upgrade
image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.9
imagePullPolicy: Always
name: mongod-posthook
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /hooks
name: hooks
- command:
- cp
- /probes/readinessprobe
- /opt/scripts/readinessprobe
image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.22
imagePullPolicy: Always
name: mongodb-agent-readinessprobe
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/scripts
name: agent-scripts
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 2000
serviceAccount: mongodb-database
serviceAccountName: mongodb-database
terminationGracePeriodSeconds: 30
volumes:
- emptyDir: {}
name: agent-scripts
- name: automation-config
secret:
defaultMode: 416
secretName: mongodb-config
- emptyDir: {}
name: healthstatus
- emptyDir: {}
name: hooks
- emptyDir: {}
name: mongodb-keyfile
- emptyDir: {}
name: tmp
updateStrategy:
type: RollingUpdate
volumeClaimTemplates:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
name: data-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
volumeMode: Filesystem
status:
phase: Pending
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
name: logs-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
volumeMode: Filesystem
status:
phase: Pending
status:
availableReplicas: 0
collisionCount: 0
currentRevision: mongodb-arb-d79cb44df
observedGeneration: 1
replicas: 0
updateRevision: mongodb-arb-d79cb44df
kind: List
metadata:
resourceVersion: ""kubectl -n mongodb get pods -oyaml
apiVersion: v1
items:
- apiVersion: v1
kind: Pod
metadata:
annotations:
agent.mongodb.com/version: "-1"
creationTimestamp: "2025-03-23T17:28:25Z"
generateName: mongodb-
labels:
app: mongodb-svc
apps.kubernetes.io/pod-index: "0"
controller-revision-hash: mongodb-d79cb44df
statefulset.kubernetes.io/pod-name: mongodb-0
name: mongodb-0
namespace: mongodb
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: StatefulSet
name: mongodb
uid: b8243dc6-fd22-4a65-8c0f-ea658425980f
resourceVersion: "5233280"
uid: 1c9884b4-8548-424b-9696-6efc8df4b6af
spec:
containers:
- args:
- ""
command:
- /bin/sh
- -c
- "\nif [ -e "/hooks/version-upgrade" ]; then\n\t#run post-start hook to handle
version changes (if exists)\n /hooks/version-upgrade\nfi\n\n# wait for
config and keyfile to be created by the agent\nwhile ! [ -f /data/automation-mongod.conf
-a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3
; done ; sleep 2 ;\n\n# start mongod with this configuration\nexec mongod
-f /data/automation-mongod.conf;\n\n"
env:
- name: AGENT_STATUS_FILEPATH
value: /healthstatus/agent-health-status.json
image: docker.io/mongodb/mongodb-community-server:7.0.15-ubi8
imagePullPolicy: IfNotPresent
name: mongod
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /data
name: data-volume
- mountPath: /healthstatus
name: healthstatus
- mountPath: /hooks
name: hooks
- mountPath: /var/log/mongodb-mms-automation
name: logs-volume
- mountPath: /var/lib/mongodb-mms-automation/authentication
name: mongodb-keyfile
- mountPath: /tmp
name: tmp
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-p89bx
readOnly: true
- command:
- /bin/bash
- -c
- |-
current_uid=$(id -u)
declare -r current_uid
if ! grep -q "${current_uid}" /etc/passwd ; then
sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
export NSS_WRAPPER_PASSWD=/tmp/passwd
export LD_PRELOAD=libnss_wrapper.so
export NSS_WRAPPER_GROUP=/etc/group
fi
agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile /var/log/mongodb-mms-automation/automation-agent.log -logLevel INFO -maxLogFileDurationHrs 24
env:
- name: AGENT_STATUS_FILEPATH
value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
- name: AUTOMATION_CONFIG_MAP
value: mongodb-config
- name: HEADLESS_AGENT
value: "true"
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: quay.io/mongodb/mongodb-agent-ubi:108.0.2.8729-1
imagePullPolicy: Always
name: mongodb-agent
readinessProbe:
exec:
command:
- /opt/scripts/readinessprobe
failureThreshold: 40
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/scripts
name: agent-scripts
- mountPath: /var/lib/automation/config
name: automation-config
readOnly: true
- mountPath: /data
name: data-volume
- mountPath: /var/log/mongodb-mms-automation/healthstatus
name: healthstatus
- mountPath: /var/log/mongodb-mms-automation
name: logs-volume
- mountPath: /var/lib/mongodb-mms-automation/authentication
name: mongodb-keyfile
- mountPath: /tmp
name: tmp
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-p89bx
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
hostname: mongodb-0
initContainers:
- command:
- cp
- version-upgrade-hook
- /hooks/version-upgrade
image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.9
imagePullPolicy: Always
name: mongod-posthook
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /hooks
name: hooks
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-p89bx
readOnly: true
- command:
- cp
- /probes/readinessprobe
- /opt/scripts/readinessprobe
image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.22
imagePullPolicy: Always
name: mongodb-agent-readinessprobe
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/scripts
name: agent-scripts
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-p89bx
readOnly: true
nodeName: k3s-master-1-pi4
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 2000
serviceAccount: mongodb-database
serviceAccountName: mongodb-database
subdomain: mongodb-svc
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: data-volume
persistentVolumeClaim:
claimName: data-volume-mongodb-0
- name: logs-volume
persistentVolumeClaim:
claimName: logs-volume-mongodb-0
- emptyDir: {}
name: agent-scripts
- name: automation-config
secret:
defaultMode: 416
secretName: mongodb-config
- emptyDir: {}
name: healthstatus
- emptyDir: {}
name: hooks
- emptyDir: {}
name: mongodb-keyfile
- emptyDir: {}
name: tmp
- name: kube-api-access-p89bx
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2025-03-23T17:28:48Z"
status: "True"
type: PodReadyToStartContainers
- lastProbeTime: null
lastTransitionTime: "2025-03-23T17:28:54Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2025-03-23T17:28:25Z"
message: 'containers with unready status: [mongodb-agent]'
reason: ContainersNotReady
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2025-03-23T17:28:25Z"
message: 'containers with unready status: [mongodb-agent]'
reason: ContainersNotReady
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2025-03-23T17:28:25Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: containerd://3722ad0bcb691b4c467bd94ca4f876f773c88fd48307d8cdbb4755e841a7b4c6
image: docker.io/mongodb/mongodb-community-server:7.0.15-ubi8
imageID: docker.io/mongodb/mongodb-community-server@sha256:bd2e8e00a36d89eeb67eb7886630eaeb68c445c8474fc8ed95286ee82456d44f
lastState: {}
name: mongod
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2025-03-23T17:31:04Z"
volumeMounts:
- mountPath: /data
name: data-volume
- mountPath: /healthstatus
name: healthstatus
- mountPath: /hooks
name: hooks
- mountPath: /var/log/mongodb-mms-automation
name: logs-volume
- mountPath: /var/lib/mongodb-mms-automation/authentication
name: mongodb-keyfile
- mountPath: /tmp
name: tmp
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-p89bx
readOnly: true
recursiveReadOnly: Disabled
- containerID: containerd://495a47b2ad004a3ccd33921fd04239d1936ae39409eefbf4456879d3f33cf9b9
image: quay.io/mongodb/mongodb-agent-ubi:108.0.2.8729-1
imageID: quay.io/mongodb/mongodb-agent-ubi@sha256:dda6762d4b53da3230c8acc925aeaaa45fc2b3e4c38e180a83053ced1528306d
lastState: {}
name: mongodb-agent
ready: false
restartCount: 0
started: true
state:
running:
startedAt: "2025-03-23T17:31:05Z"
volumeMounts:
- mountPath: /opt/scripts
name: agent-scripts
- mountPath: /var/lib/automation/config
name: automation-config
readOnly: true
recursiveReadOnly: Disabled
- mountPath: /data
name: data-volume
- mountPath: /var/log/mongodb-mms-automation/healthstatus
name: healthstatus
- mountPath: /var/log/mongodb-mms-automation
name: logs-volume
- mountPath: /var/lib/mongodb-mms-automation/authentication
name: mongodb-keyfile
- mountPath: /tmp
name: tmp
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-p89bx
readOnly: true
recursiveReadOnly: Disabled
hostIP: 192.168.1.241
hostIPs:
- ip: 192.168.1.241
initContainerStatuses:
- containerID: containerd://cfd9da564e938a4e07736d6580d6be86f526ed059b7c55b2f128687f7ecd5858
image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.9
imageID: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:a72e929e623ed8450991f444e21d50db1f791c28452c193a3a33b7d3c159928e
lastState: {}
name: mongod-posthook
ready: true
restartCount: 0
started: false
state:
terminated:
containerID: containerd://cfd9da564e938a4e07736d6580d6be86f526ed059b7c55b2f128687f7ecd5858
exitCode: 0
finishedAt: "2025-03-23T17:28:48Z"
reason: Completed
startedAt: "2025-03-23T17:28:48Z"
volumeMounts:
- mountPath: /hooks
name: hooks
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-p89bx
readOnly: true
recursiveReadOnly: Disabled
- containerID: containerd://8acb60714fe19f1b1d4d8dddefae3df0873849fadb58448ee5041d6fd2ac4dbb
image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.22
imageID: quay.io/mongodb/mongodb-kubernetes-readinessprobe@sha256:a6701b892f7989b46bddfae1d9e3ba8d279e15d3f035dc27dae52fd8838b981d
lastState: {}
name: mongodb-agent-readinessprobe
ready: true
restartCount: 0
started: false
state:
terminated:
containerID: containerd://8acb60714fe19f1b1d4d8dddefae3df0873849fadb58448ee5041d6fd2ac4dbb
exitCode: 0
finishedAt: "2025-03-23T17:28:51Z"
reason: Completed
startedAt: "2025-03-23T17:28:51Z"
volumeMounts:
- mountPath: /opt/scripts
name: agent-scripts
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-p89bx
readOnly: true
recursiveReadOnly: Disabled
phase: Running
podIP: 10.42.0.218
podIPs:
- ip: 10.42.0.218
qosClass: Burstable
startTime: "2025-03-23T17:28:25Z"
- apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2025-03-20T22:20:44Z"
generateName: mongodb-kubernetes-operator-7c967f54d4-
labels:
name: mongodb-kubernetes-operator
pod-template-hash: 7c967f54d4
name: mongodb-kubernetes-operator-7c967f54d4-vrhk4
namespace: mongodb
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: mongodb-kubernetes-operator-7c967f54d4
uid: 3e7cc32d-e63b-4442-977e-646df7dd2d46
resourceVersion: "5053687"
uid: 6d4d49f5-a3a6-4589-a1a9-5386ec215169
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: name
operator: In
values:
- mongodb-kubernetes-operator
topologyKey: kubernetes.io/hostname
containers:
- command:
- /usr/local/bin/entrypoint
env:
- name: WATCH_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: OPERATOR_NAME
value: mongodb-kubernetes-operator
- name: AGENT_IMAGE
value: quay.io/mongodb/mongodb-agent-ubi:108.0.2.8729-1
- name: VERSION_UPGRADE_HOOK_IMAGE
value: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.9
- name: READINESS_PROBE_IMAGE
value: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.22
- name: MONGODB_IMAGE
value: mongodb-community-server
- name: MONGODB_REPO_URL
value: docker.io/mongodb
- name: MDB_IMAGE_TYPE
value: ubi8
image: quay.io/mongodb/mongodb-kubernetes-operator:0.12.0
imagePullPolicy: Always
name: mongodb-kubernetes-operator
resources:
limits:
cpu: 1100m
memory: 1Gi
requests:
cpu: 500m
memory: 200Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-v7bxb
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: k3s-worker-1-pi3
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
runAsNonRoot: true
runAsUser: 2000
serviceAccount: mongodb-kubernetes-operator
serviceAccountName: mongodb-kubernetes-operator
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: kube-api-access-v7bxb
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2025-03-20T22:21:49Z"
status: "True"
type: PodReadyToStartContainers
- lastProbeTime: null
lastTransitionTime: "2025-03-20T22:20:45Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2025-03-20T22:21:49Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2025-03-20T22:21:49Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2025-03-20T22:20:45Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: containerd://1afaf481f5cec5be83c1387961b6997a058ee0244b8261173a4898af678a49ec
image: quay.io/mongodb/mongodb-kubernetes-operator:0.12.0
imageID: quay.io/mongodb/mongodb-kubernetes-operator@sha256:d0bdb84582ce3645aa57aac28fe007848ebb88722e863ea6c9eb57f124451a7f
lastState: {}
name: mongodb-kubernetes-operator
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2025-03-20T22:21:49Z"
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-v7bxb
readOnly: true
recursiveReadOnly: Disabled
hostIP: 192.168.1.102
hostIPs:
- ip: 192.168.1.102
phase: Running
podIP: 10.42.1.49
podIPs:
- ip: 10.42.1.49
qosClass: Burstable
startTime: "2025-03-20T22:20:45Z"
kind: List
metadata:
resourceVersion: ""kubectl -n mongodb logs mongodb-0
Defaulted container "mongod" out of: mongod, mongodb-agent, mongod-posthook (init), mongodb-agent-readinessprobe (init)
2025-03-23T17:31:05.666Z INFO versionhook/main.go:33 Running version change post-start hook
2025-03-23T17:31:05.670Z INFO versionhook/main.go:40 Waiting for agent health status...
2025-03-23T17:31:06.671Z INFO versionhook/main.go:46 Agent health status file not found, mongod will start
kubectl -n mongodb exec -it mongodb-0 -c mongodb-agent -- cat /var/lib/automation/config/cluster-config.json
{
"version":1,
"processes":[
{
"name":"mongodb-0",
"disabled":false,
"hostname":"mongodb-0.mongodb-svc.mongodb.svc.cluster.local",
"args2_6":{
"net":{
"port":27017
},
"replication":{
"replSetName":"mongodb"
},
"storage":{
"dbPath":"/data",
"wiredTiger":{
"engineConfig":{
"journalCompressor":"zlib"
}
}
}
},
"featureCompatibilityVersion":"7.0",
"processType":"mongod",
"version":"7.0.15",
"authSchemaVersion":5
}
],
"replicaSets":[
{
"_id":"mongodb",
"members":[
{
"_id":0,
"host":"mongodb-0",
"arbiterOnly":false,
"votes":1,
"priority":1
}
],
"protocolVersion":"1",
"numberArbiters":0
}
],
"auth":{
"usersWanted":[
{
"mechanisms":[],
"roles":[
{
"role":"clusterAdmin",
"db":"admin"
},
{
"role":"userAdminAnyDatabase",
"db":"admin"
}
],
"user":"admin",
"db":"admin",
"authenticationRestrictions":[],
"scramSha256Creds":{
"iterationCount":15000,
"salt":".......",
"serverKey":".....",
"storedKey":"....."
},
"scramSha1Creds":{
"iterationCount":10000,
"salt":"......",
"serverKey":".....",
"storedKey":"....."
}
}
],
"disabled":false,
"authoritativeSet":false,
"autoAuthMechanisms":["SCRAM-SHA-256"],
"autoAuthMechanism":"SCRAM-SHA-256",
"deploymentAuthMechanisms":["SCRAM-SHA-256"],
"autoUser":"mms-automation",
"key":".......",
"keyfile":"/var/lib/mongodb-mms-automation/authentication/keyfile",
"keyfileWindows":"%SystemDrive%\\MMSAutomation\\versions\\keyfile",
"autoPwd":"....."
},
"tls":{
"CAFilePath":"",
"clientCertificateMode":"OPTIONAL"
},
"mongoDbVersions":[
{
"name":"7.0.15",
"builds":[
{
"platform":"linux",
"url":"",
"gitVersion":"",
"architecture":"amd64",
"flavor":"rhel",
"minOsVersion":"",
"maxOsVersion":"",
"modules":[]
},{
"platform":"linux",
"url":"",
"gitVersion":"",
"architecture":"amd64",
"flavor":"ubuntu",
"minOsVersion":"",
"maxOsVersion":"",
"modules":[]
},{
"platform":"linux",
"url":"",
"gitVersion":"",
"architecture":"aarch64",
"flavor":"ubuntu",
"minOsVersion":"",
"maxOsVersion":"",
"modules":[]
},{
"platform":"linux",
"url":"",
"gitVersion":"",
"architecture":"aarch64",
"flavor":"rhel",
"minOsVersion":"",
"maxOsVersion":"",
"modules":[]
}
]
}
],
"backupVersions":[],
"monitoringVersions":[],
"options":{
"downloadBase":"/var/lib/mongodb-mms-automation"
}
}kubectl -n mongodb exec -it mongodb-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
{
"statuses":{
"mongodb-0":{
"IsInGoalState":false,
"LastMongoUpTime":0,
"ExpectedToBeUp":true,
"ReplicationStatus":-1
}
},
"mmsStatus":{
"mongodb-0":{
"name":"mongodb-0",
"lastGoalVersionAchieved":-1,
"plans":[
{
"automationConfigVersion":1,
"started":"2025-03-23T17:31:06.349704425Z",
"completed":null,
"moves":[
{
"move":"Start",
"moveDoc":"Start the process",
"steps":[
{
"step":"StartFresh",
"stepDoc":"Start a mongo instance (start fresh)",
"isWaitStep":false,
"started":"2025-03-23T17:31:06.349778998Z",
"completed":null,
"result":"error"
}
]
},{
"move":"WaitAllRsMembersUp",
"moveDoc":"Wait until all members of this process' repl set are up",
"steps":[
{
"step":"WaitAllRsMembersUp",
"stepDoc":"Wait until all members of this process' repl set are up",
"isWaitStep":true,
"started":null,
"completed":null,
"result":""
}
]
},{
"move":"RsInit",
"moveDoc":"Initialize a replica set including the current MongoDB process",
"steps":[
{
"step":"RsInit",
"stepDoc":"Initialize a replica set",
"isWaitStep":false,
"started":null,
"completed":null,
"result":""
}
]
},{
"move":"WaitFeatureCompatibilityVersionCorrect",
"moveDoc":"Wait for featureCompatibilityVersion to be right",
"steps":[
{
"step":"WaitFeatureCompatibilityVersionCorrect",
"stepDoc":"Wait for featureCompatibilityVersion to be right",
"isWaitStep":true,
"started":null,
"completed":null,
"result":""
}
]
}
]
}
],
"errorCode":0,
"errorString":"
\u003cmongodb-0\u003e [18:04:55.151] Plan execution failed on step StartFresh as part of move Start :
\u003cmongodb-0\u003e [18:04:55.151] Failed to apply action. Result = \u003cnil\u003e :
\u003cmongodb-0\u003e [18:04:55.151] Error starting mongod :
\u003cmongodb-0\u003e [18:04:55.151] Error getting start process cmd for executable=mongod, stip=[args=
{
"net":{
"bindIp":"0.0.0.0",
"port":27017
},
"replication":{
"replSetName":"mongodb"
},
"security":{
"authorization":"enabled",
"keyFile":"/var/lib/mongodb-mms-automation/authentication/keyfile"
},
"setParameter":{
"authenticationMechanisms":"SCRAM-SHA-256"
},
"storage":{
"dbPath":"/data",
"wiredTiger":{
"engineConfig":{
"journalCompressor":"zlib"
}
}
}
}[],
confPath=/data/automation-mongod.conf,version=7.0.15-(),
isKmipRotateMasterKey=false,useOldConfFile=false] :
\u003cmongodb-0\u003e [18:04:55.150] Failed to create conf file :
\u003cmongodb-0\u003e [18:04:55.150] Failed to create file /data/automation-mongod.conf :
\u003cmongodb-0\u003e [18:04:55.150] Error creating /data/automation-mongod.conf : open /data/automation-mongod.conf: permission denied",
"waitDetails":{
"RunSetParameter":"process not up",
"UpdateFeatureCompatibilityVersion":"process isn't up",
"WaitAllRsMembersUp":"[]",
"WaitCannotBecomePrimary":"Wait until the process is reconfigured with priority=0 by a different process",
"WaitClusterReadyForFCVUpdate":"process isn't up",
"WaitDefaultRWConcernCorrect":"waiting for the primary to update defaultRWConcern",
"WaitForResyncPrimaryManualInterventionStep":"A resync was requested on a primary. This requires manual intervention",
"WaitHealthyMajority":"[]",
"WaitMultipleHealthyNonArbiters":"[]",
"WaitNecessaryRsMembersUpForReconfig":"[]",
"WaitPrimary":"This process is expected to be the primary member. Check that the replica set state allows a primary to be elected",
"WaitProcessUp":"The process is running, but not yet responding to agent calls",
"WaitResetPlacementHistory":"config servers haven't seen the marker"
}
}
}
}kubectl -n mongodb exec -it mongodb-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/automation-agent-verbose.log
cat: /var/log/mongodb-mms-automation/automation-agent-verbose.log: No such file or directory
command terminated with exit code 1
kubectl -n mongodb exec -it mongodb-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/automation-agent.log
cat: /var/log/mongodb-mms-automation/automation-agent.log: No such file or directory
command terminated with exit code 1
kubectl -n mongodb exec -it mongodb-0 -c mongodb-agent -- ls -al /var/log/mongodb-mms-automation/
total 28
drwxr-xr-x 4 root root 4096 Mar 23 16:59 .
drwxr-xr-x 1 root root 4096 Mar 23 04:26 ..
drwxrwsrwx 2 root 2000 4096 Mar 23 18:15 healthstatus
drwx------ 2 root root 16384 Mar 23 16:59 lost+found
kubectl -n mongodb exec -it mongodb-0 -c mongodb-agent -- ls -al /var/log/
total 312
drwxr-xr-x 1 root root 4096 Mar 23 04:26 .
drwxr-xr-x 1 root root 4096 Mar 13 07:22 ..
lrwxrwxrwx 1 root root 39 Mar 23 04:25 README -> ../../usr/share/doc/systemd/README.logs
-rw-rw---- 1 root utmp 0 Mar 23 04:25 btmp
-rw-rw-r-- 1 root utmp 295704 Mar 23 04:25 lastlog
drwxr-xr-x 4 root root 4096 Mar 23 16:59 mongodb-mms-automation
drwx------ 2 root root 4096 Mar 23 04:25 private
-rw------- 1 root root 0 Mar 23 04:25 tallylog
-rw-rw-r-- 1 root utmp 0 Mar 23 04:25 wtmp
kubectl -n mongodb exec -it mongodb-0 -c mongodb-agent -- ls -al /var/log/mongodb-mms-automation/healthstatus
total 12
drwxrwsrwx 2 root 2000 4096 Mar 23 18:16 .
drwxr-xr-x 4 root root 4096 Mar 23 16:59 ..
-rw------- 1 2000 2000 3353 Mar 23 18:16 agent-health-status.json
kubectl -n mongodb exec -it mongodb-0 -c mongod -- ls -al /data
total 32
drwxr-xr-x 5 root root 4096 Mar 23 16:59 .
drwxr-xr-x 1 root root 4096 Mar 23 17:31 ..
drwxr-xr-x 2 mongod mongod 4096 Mar 23 06:20 configdb
drwxr-xr-x 2 mongod mongod 4096 Mar 23 06:20 db
drwx------ 2 root root 16384 Mar 23 16:59 lost+found
I'd expect the /data and /var/log/mongodb-mms-automation to be owneb by uid=2000,gid=2000 or at least writiable by the group in both containers.
Right now i see permission denied errors both from the mongod and mongodb-agent containers
mongod:Failed to create file /data/automation-mongod.confmongodb-agent:open /var/log/mongodb-mms-automation/readiness.log: permission denied