Volume failed to attach : "Input/output error (os error 5)"

[veenadong]

Mayastor 2.7.1, after 1 node is disconnected, the volume failed to attach with error:

Events:
  Type     Reason       Age                 From     Message
  ----     ------       ----                ----     -------
  Warning  FailedMount  90s (x18 over 29m)  kubelet  MountVolume.MountDevice failed for volume "pvc-209c15bd-a46d-41d6-b9f5-6b63f5f90f39" : rpc error: code = Internal desc = Failed to stage volume 209c15bd-a46d-41d6-b9f5-6b63f5f90f39: attach failed: IO error: Input/output error (os error 5), args: hostnqn=nqn.2019-05.io.openebs:node-name:sc-os-160-node1.glcpdev.cloud.hpe.com,hostid=42168bd1-20a2-2591-5779-353d1054684f,nqn=nqn.2019-05.io.openebs:209c15bd-a46d-41d6-b9f5-6b63f5f90f39,transport=tcp,traddr=10.245.244.161,trsvcid=8420,reconnect_delay=10,ctrl_loss_tmo=1980,nr_io_queues=2

Logs:
mayastor-2024-10-15--20-08-44-UTC_failedto_attach.tar.gz

[mattlqx]

Same here after cluster restart, some pods are stuck.

Events:
  Type     Reason       Age                   From               Message
  ----     ------       ----                  ----               -------
  Normal   Scheduled    9m                    default-scheduler  Successfully assigned monitoring/prometheus-kube-prom-kube-prometheus-prometheus-0 to node1.example.com
  Warning  FailedMount  42s (x12 over 8m58s)  kubelet            MountVolume.MountDevice failed for volume "pvc-3de351ab-9647-4b6c-831c-c95cb5eb8d33" : rpc error: code = Internal desc = Failed to stage volume 3de351ab-9647-4b6c-831c-c95cb5eb8d33: attach failed: IO error: Input/output error (os error 5), args: hostnqn=nqn.2019-05.io.openebs:node-name:node1.example.com,hostid=03000200-0400-0500-0006-000700080009,nqn=nqn.2019-05.io.openebs:3de351ab-9647-4b6c-831c-c95cb5eb8d33,transport=tcp,traddr=192.168.2.27,trsvcid=8420,reconnect_delay=10,ctrl_loss_tmo=1980,nr_io_queues=2

The volume is currently rebuilding...

 3de351ab-9647-4b6c-831c-c95cb5eb8d33  2c06cd21-dcfc-4898-a5d9-1ba003f597cf  node1.example.com  node1-1    Online   450GiB    28.3GiB    0 B        Degraded      <none>  6%    
 └─                                    616926d1-91c2-4597-a418-6964bf3a4a88  node2.example.com  node2-2    Online   450GiB    347.8GiB   0 B        Online        <none>  <none>

[dsharma-dc]

This can be closed it seems.
@tiagolobocastro