Shard Removal Fails Due to Direct mongod Connection Check

[Fluder-Paradyne]

Report

When attempting to remove a shard from a sharded Percona Server for MongoDB cluster, the operator's reconcile loop fails. The failure is caused by a pre-delete safety check (checkIfUserDataExistInRS) that attempts to connect directly to the mongod instance of the shard being removed. In a sharded topology, this direct connection is improper and is rejected by MongoDB, causing the operator to error out and preventing the scale-down from completing.

More about the problem

Error log:

2025-09-21T15:13:11.721Z	INFO	Deleting STS component from replst	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"people-psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "people-psmdb-db", "reconcileID": "0e825148-6106-4fc9-bc25-38fc1ebe8cdd", "sts": "people-psmdb-db-rs2", "rs": "rs2", "port": 27017}
2025-09-21T15:13:11.748Z	ERROR	failed to list databases	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"people-psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "people-psmdb-db", "reconcileID": "0e825148-6106-4fc9-bc25-38fc1ebe8cdd", "rs": "rs2", "error": "listDatabases: (Unauthorized) You are connecting to a sharded cluster improperly by connecting directly to a shard. Please connect to the cluster via a router (mongos).", "errorVerbose": "(Unauthorized) You are connecting to a sharded cluster improperly by connecting directly to a shard. Please connect to the cluster via a router (mongos).\nlistDatabases\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/mongo.(*mongoClient).ListDBs\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/mongo/mongo.go:490\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).checkIfUserDataExistInRS\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:859\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:361\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:334\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1223"}
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).checkIfUserDataExistInRS
	/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:861
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile
	/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:361
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:334
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255

Steps to reproduce

1. Deploy a sharded PSMDB cluster with two or more replica sets (e.g., rs0, rs1, rs2).
2. Follow the official MongoDB procedure to drain a shard by connecting to a mongos instance and successfully running b.adminCommand({ removeShard: "rs2" })..
3. Modify the PerconaServerMongoDB custom resource to remove the rs2 replica set from the spec.replsets array.
4. Apply the updated configuration.

Versions

1. Kubernetes -- 1.32
2. Operator -- 1.20.0
3. Database -- 8.0.4-1-multi

Anything else?

[direct: mongos] config> db.adminCommand({ removeShard: "rs2" });
{
  msg: 'removeshard completed successfully',
  state: 'completed',
  shard: 'rs2',
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1758467128, i: 17 }),
    signature: {
      hash: Binary.createFromBase64('dv0FyFVESLFebpbU2Is3YB6bJlI=', 0),
      keyId: Long('7541359738756268055')
    }
  },
  operationTime: Timestamp({ t: 1758467128, i: 17 })
}
``

[Fluder-Paradyne]

I think for sharded cluster we can run listShards on admin and check if the shard still exsist in the output

[eleo007]

Hi!

I have tried to reproduce the described issue on both 1.20.0 and main and could not. Could you provide more details about your setup? Could you provide a cr example that you have used?

Thanks.

[eleo007]

Hi @Fluder-Paradyne ,

An additional note: there is no need to manually remove the shard using the db.adminCommand({ removeShard: "rs2" }) as operators runs it from its side after removing shard from cr. The manual actions are required though to move/drop DBs/collections from shard in case there are any to move.

[Fluder-Paradyne]

This was the initial config, the issue might have been caused because of me running db.adminCommand({ removeShard: "rs2" }) before removing it from cr, but rs2 was primary of a db, I then moved the primary to rs0, then removeShard went through.

resource "helm_release" "mongodb_operator" {
  name       = "mongodb-operator"
  namespace  = kubernetes_namespace.mongodb.metadata[0].name
  repository = "https://percona.github.io/percona-helm-charts"
  chart      = "psmdb-operator"
  version    = "1.20.0"
}

resource "helm_release" "people_mongodb" {
  chart      = "psmdb-db"
  name       = "people"
  namespace  = kubernetes_namespace.mongodb.metadata[0].name
  repository = "https://percona.github.io/percona-helm-charts"
  version    = "1.16.3"

  values = [
    templatefile("${path.module}/values/people-data-mongodb.yaml", {
      users_secret       = kubernetes_secret.people_mongodb.metadata[0].name
      bucket             = data.aws_s3_bucket.people_mongo_data_extract_backups.id
      credentials_secret = kubernetes_secret.people_mongo_backup_credentials_new.metadata[0].name
      region             = data.aws_region.current.name
      backup_prefix      = "people"
    })
  ]

  timeout = 300
}

pause: false
image:
  tag: 8.0.4-1-multi
systemLog:
  verbosity: 0
tls:
  mode: disabled
backup:
  enabled: true
  image:
    tag: 2.10.0
  storages:
    us-east-1:
      main: true
      type: s3
      s3:
        bucket: ${bucket}
        credentialsSecret: ${credentials_secret}
        region: ${region}
        prefix: ${backup_prefix}
  tasks:
    - name: daily-backup
      enabled: true
      schedule: "45 16 * * *"
      keep: 7
      compressionType: "gzip"
      storage: "us-east-1"
      storageName: "us-east-1"
clusterServiceDNSMode: internal
unsafeFlags:
  backupIfUnhealthy: true
  tls: true
secrets:
  users: ${users_secret}
  systemUsers: ${users_secret}
replsets:
  rs0:
    name: rs0
    size: 3
    nodeSelector:
      karpenter.sh/capacity-type: on-demand
    configuration: |
      systemLog:
        quiet: true
    expose:
      enabled: true
      exposeType: ClusterIP
    resources:
      requests:
        memory: "8Gi"
        cpu: "4"
      limits:
        memory: "8Gi"
        cpu: "4"
    volumeSpec:
      pvc:
        storageClassName: gp3
        resources:
          requests:
            storage: 2048Gi
  rs1:
    name: rs1
    size: 3
    nodeSelector:
      karpenter.sh/capacity-type: on-demand
    configuration: |
      systemLog:
        quiet: true
    expose:
      enabled: true
      exposeType: ClusterIP
    resources:
      requests:
        memory: "8Gi"
        cpu: "4"
      limits:
        memory: "8Gi"
        cpu: "4"
    volumeSpec:
      pvc:
        storageClassName: gp3
        resources:
          requests:
            storage: 2048Gi
  rs2:
    name: rs2
    size: 3
    nodeSelector:
      karpenter.sh/capacity-type: on-demand
    configuration: |
      systemLog:
        quiet: true
    expose:
      enabled: true
      exposeType: ClusterIP
    resources:
      requests:
        memory: "8Gi"
        cpu: "4"
      limits:
        memory: "8Gi"
        cpu: "4"
    volumeSpec:
      pvc:
        storageClassName: gp3
        resources:
          requests:
            storage: 2048Gi
sharding:
  enabled: true
  mongos:
    size: 5
    nodeSelector:
      karpenter.sh/capacity-type: on-demand
    resources:
      requests:
        memory: "1Gi"
        cpu: "1"
      limits:
        memory: "4Gi"
        cpu: "2"
  configrs:
    size: 3
    expose:
      enabled: true
      exposeType: ClusterIP
    nodeSelector:
      karpenter.sh/capacity-type: on-demand

[Fluder-Paradyne]

could it be because of me running db.adminCommand({ removeShard: "rs2" }) before removing it from the config ?