mongodb-agent: can't open new logfile: open /var/log/mongodb-mms-automation/readiness.log: permission denied #1700
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
{{ message }}
What did you do to encounter the bug?
Steps to reproduce the behavior:
The only differences from https://github.com/mongodb/mongodb-kubernetes-operator/blob/master/config/samples/mongodb.com_v1_mongodbcommunity_cr.yaml
What did you expect?
kubectl -n mongodb get mongodbcommunitykubectl -n mongodb get podWhat happened instead?
kubectl -n mongodb get mongodbcommunitykubectl -n mongodb get podkubectl -n mongodb describe pod mongodb-0(some output ommited)
Operator Information
Kubernetes Cluster Information
Additional context
Possibly same problem as: #1384 #1143 #949
The volumes are RWO, correctly provisioned and bound.
kubectl -n mongodb get mdbc -oyamlkubectl -n mongodb get sts -oyamlkubectl -n mongodb get pods -oyamlkubectl -n mongodb logs mongodb-0kubectl -n mongodb exec -it mongodb-0 -c mongodb-agent -- cat /var/lib/automation/config/cluster-config.json{ "version":1, "processes":[ { "name":"mongodb-0", "disabled":false, "hostname":"mongodb-0.mongodb-svc.mongodb.svc.cluster.local", "args2_6":{ "net":{ "port":27017 }, "replication":{ "replSetName":"mongodb" }, "storage":{ "dbPath":"/data", "wiredTiger":{ "engineConfig":{ "journalCompressor":"zlib" } } } }, "featureCompatibilityVersion":"7.0", "processType":"mongod", "version":"7.0.15", "authSchemaVersion":5 } ], "replicaSets":[ { "_id":"mongodb", "members":[ { "_id":0, "host":"mongodb-0", "arbiterOnly":false, "votes":1, "priority":1 } ], "protocolVersion":"1", "numberArbiters":0 } ], "auth":{ "usersWanted":[ { "mechanisms":[], "roles":[ { "role":"clusterAdmin", "db":"admin" }, { "role":"userAdminAnyDatabase", "db":"admin" } ], "user":"admin", "db":"admin", "authenticationRestrictions":[], "scramSha256Creds":{ "iterationCount":15000, "salt":".......", "serverKey":".....", "storedKey":"....." }, "scramSha1Creds":{ "iterationCount":10000, "salt":"......", "serverKey":".....", "storedKey":"....." } } ], "disabled":false, "authoritativeSet":false, "autoAuthMechanisms":["SCRAM-SHA-256"], "autoAuthMechanism":"SCRAM-SHA-256", "deploymentAuthMechanisms":["SCRAM-SHA-256"], "autoUser":"mms-automation", "key":".......", "keyfile":"/var/lib/mongodb-mms-automation/authentication/keyfile", "keyfileWindows":"%SystemDrive%\\MMSAutomation\\versions\\keyfile", "autoPwd":"....." }, "tls":{ "CAFilePath":"", "clientCertificateMode":"OPTIONAL" }, "mongoDbVersions":[ { "name":"7.0.15", "builds":[ { "platform":"linux", "url":"", "gitVersion":"", "architecture":"amd64", "flavor":"rhel", "minOsVersion":"", "maxOsVersion":"", "modules":[] },{ "platform":"linux", "url":"", "gitVersion":"", "architecture":"amd64", "flavor":"ubuntu", "minOsVersion":"", "maxOsVersion":"", "modules":[] },{ "platform":"linux", "url":"", "gitVersion":"", "architecture":"aarch64", "flavor":"ubuntu", "minOsVersion":"", "maxOsVersion":"", "modules":[] },{ "platform":"linux", "url":"", "gitVersion":"", "architecture":"aarch64", "flavor":"rhel", "minOsVersion":"", "maxOsVersion":"", "modules":[] } ] } ], "backupVersions":[], "monitoringVersions":[], "options":{ "downloadBase":"/var/lib/mongodb-mms-automation" } }kubectl -n mongodb exec -it mongodb-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json{ "statuses":{ "mongodb-0":{ "IsInGoalState":false, "LastMongoUpTime":0, "ExpectedToBeUp":true, "ReplicationStatus":-1 } }, "mmsStatus":{ "mongodb-0":{ "name":"mongodb-0", "lastGoalVersionAchieved":-1, "plans":[ { "automationConfigVersion":1, "started":"2025-03-23T17:31:06.349704425Z", "completed":null, "moves":[ { "move":"Start", "moveDoc":"Start the process", "steps":[ { "step":"StartFresh", "stepDoc":"Start a mongo instance (start fresh)", "isWaitStep":false, "started":"2025-03-23T17:31:06.349778998Z", "completed":null, "result":"error" } ] },{ "move":"WaitAllRsMembersUp", "moveDoc":"Wait until all members of this process' repl set are up", "steps":[ { "step":"WaitAllRsMembersUp", "stepDoc":"Wait until all members of this process' repl set are up", "isWaitStep":true, "started":null, "completed":null, "result":"" } ] },{ "move":"RsInit", "moveDoc":"Initialize a replica set including the current MongoDB process", "steps":[ { "step":"RsInit", "stepDoc":"Initialize a replica set", "isWaitStep":false, "started":null, "completed":null, "result":"" } ] },{ "move":"WaitFeatureCompatibilityVersionCorrect", "moveDoc":"Wait for featureCompatibilityVersion to be right", "steps":[ { "step":"WaitFeatureCompatibilityVersionCorrect", "stepDoc":"Wait for featureCompatibilityVersion to be right", "isWaitStep":true, "started":null, "completed":null, "result":"" } ] } ] } ], "errorCode":0, "errorString":" \u003cmongodb-0\u003e [18:04:55.151] Plan execution failed on step StartFresh as part of move Start : \u003cmongodb-0\u003e [18:04:55.151] Failed to apply action. Result = \u003cnil\u003e : \u003cmongodb-0\u003e [18:04:55.151] Error starting mongod : \u003cmongodb-0\u003e [18:04:55.151] Error getting start process cmd for executable=mongod, stip=[args= { "net":{ "bindIp":"0.0.0.0", "port":27017 }, "replication":{ "replSetName":"mongodb" }, "security":{ "authorization":"enabled", "keyFile":"/var/lib/mongodb-mms-automation/authentication/keyfile" }, "setParameter":{ "authenticationMechanisms":"SCRAM-SHA-256" }, "storage":{ "dbPath":"/data", "wiredTiger":{ "engineConfig":{ "journalCompressor":"zlib" } } } }[], confPath=/data/automation-mongod.conf,version=7.0.15-(), isKmipRotateMasterKey=false,useOldConfFile=false] : \u003cmongodb-0\u003e [18:04:55.150] Failed to create conf file : \u003cmongodb-0\u003e [18:04:55.150] Failed to create file /data/automation-mongod.conf : \u003cmongodb-0\u003e [18:04:55.150] Error creating /data/automation-mongod.conf : open /data/automation-mongod.conf: permission denied", "waitDetails":{ "RunSetParameter":"process not up", "UpdateFeatureCompatibilityVersion":"process isn't up", "WaitAllRsMembersUp":"[]", "WaitCannotBecomePrimary":"Wait until the process is reconfigured with priority=0 by a different process", "WaitClusterReadyForFCVUpdate":"process isn't up", "WaitDefaultRWConcernCorrect":"waiting for the primary to update defaultRWConcern", "WaitForResyncPrimaryManualInterventionStep":"A resync was requested on a primary. This requires manual intervention", "WaitHealthyMajority":"[]", "WaitMultipleHealthyNonArbiters":"[]", "WaitNecessaryRsMembersUpForReconfig":"[]", "WaitPrimary":"This process is expected to be the primary member. Check that the replica set state allows a primary to be elected", "WaitProcessUp":"The process is running, but not yet responding to agent calls", "WaitResetPlacementHistory":"config servers haven't seen the marker" } } } }kubectl -n mongodb exec -it mongodb-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/automation-agent-verbose.logkubectl -n mongodb exec -it mongodb-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/automation-agent.logkubectl -n mongodb exec -it mongodb-0 -c mongodb-agent -- ls -al /var/log/mongodb-mms-automation/kubectl -n mongodb exec -it mongodb-0 -c mongodb-agent -- ls -al /var/log/kubectl -n mongodb exec -it mongodb-0 -c mongodb-agent -- ls -al /var/log/mongodb-mms-automation/healthstatuskubectl -n mongodb exec -it mongodb-0 -c mongod -- ls -al /dataI'd expect the
/dataand/var/log/mongodb-mms-automationto be owneb by uid=2000,gid=2000 or at least writiable by the group in both containers.Right now i see permission denied errors both from the
mongodandmongodb-agentcontainersmongod:Failed to create file /data/automation-mongod.confmongodb-agent:open /var/log/mongodb-mms-automation/readiness.log: permission deniedThe text was updated successfully, but these errors were encountered: