CEPH 19.2.2 Squid NVMeOF install / CEPH 18.2.7

[H3yRepo]

Hello,

I am trying to setup an NVMeOF setup for test in Ceph 19.2.2 with 2 nodes, but i have some issues....

Process:

I have created the nvmeof service on the services page for ceph-nvme1 node (runs ubuntu22.04 and podman), the daemon runs, the container gets deployed to the node. I configured the subsystem, listener, namespace and initiator. When i do a discovery on the esxi 8.0.2 host it sees the controller. When i connect to it, it sees the namespace, but the lun stays standby.

Log on the esxi side:

2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NVMEDEV:6278 Discover namespace on controller 452
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu47:2097610)NvmeDiscover: 2192: Failed to get valid NS for controller nqn.2001-07.com.ceph:1753261226568#vmhba65#10.10.11.8:4420 with with status Invalid or missing namespace
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NVMEDEV:5417 Controller 452, construct namespace 1
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NVMEDEV:5359 Namespace identifier type 0x2, length 0x10.
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NVMEDEV:5359 Namespace identifier type 0x3, length 0x10.
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NVMEDEV:5301 Set name of ns 1 to eui.610465c484794101931f35ab92209987
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NvmeDiscover: 6804: Scan operation 3 received on adapter vmhba65
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NvmeDiscover: 6241: Add namespace on controller = nqn.2001-07.com.ceph:1753261226568#vmhba65#10.10.11.8:4420
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NvmeDiscover: 6274: Failed to get valid NS handle from controller nqn.2001-07.com.ceph:1753261226568#vmhba65#10.10.11.8:4420 with status Invalid or missing namespace
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NVMEPSA:1631 adpater: vmhba65, action: 0
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NvmeDiscover: 3167: Created path for NS (under nqn.2001-07.com.ceph:1753261226568#vmhba65#10.10.11.8:4420) 0x4306f6a6b858 = 0x431e6b807800 for ns = 0x1 for path vmhba65:C0:T0:L0
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NvmeDiscover: 3221: ANA enabled = 1 on ns=0x1
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NvmeDiscover: 2955: NGUID based 16-byte identifier: 0x61 0x4 0x65 0xc4 0x84 0x79 0x41 0x1 0x93 0x1f 0x35 0xab 0x92 0x20 0x99 0x87
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NvmeDiscover: 3045: Path 'vmhba65:C0:T0:L0' : Current UID is eui.610465c484794101931f35ab92209987
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)ns = 0x1
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NvmeDiscover: 6828: Scan operation 3 completed with status Success on adapter vmhba65
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NVMEDEV:6419 Discover namespaces on controller 452 is complete
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NVMEDEV:8533 Controller 452 setting event[2,3]: aec=2048
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NVMEDEV:8617 Configuring aec: 0x900 on controller 452.
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NVMEDEV:8645 Async event configuration value is 0x900 on controller 452.
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NVMEDEV:9252 Succeeded to create recovery world for controller 452
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NVMFDEV:1245 Controller 452 (nqn.2001-07.com.ceph:1753261226568#vmhba65#10.10.11.8:4420) registered successfully.
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NVMFVSI:1342 Connected to controller successfully
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NVMFVSI:693 Masking out trType 1
2025-07-23T09:05:01.234Z In(182) vmkernel: cpu30:2098764 opID=772545d0)NVMFVSI:693 Masking out trType 2
2025-07-23T09:05:02.236Z In(182) vmkernel: cpu37:2097607)NvmeDiscover: 6200: NS discover waited 1002 msec for controller nqn.2001-07.com.ceph:1753261226568#vmhba65#10.10.11.8:4420 , now: 417980701 nsAddTimestamp: 417979699
2025-07-23T09:05:02.236Z In(182) vmkernel: cpu13:2097592)HPP: HppCreateDevice:2807: Created logical device 'eui.610465c484794101931f35ab92209987'.
2025-07-23T09:05:02.236Z In(182) vmkernel: cpu13:2097592)HPP: HppClaimPath:3582: ALUA/ANA target (vmhba65:C0:T0:L0)
2025-07-23T09:05:02.236Z In(182) vmkernel: cpu13:2097592)HPP: HppClaimPath:3647: Added path 'vmhba65:C0:T0:L0' to logical device 'eui.610465c484794101931f35ab92209987'. Total Path Count 1
2025-07-23T09:05:02.236Z In(182) vmkernel: cpu36:2098204)HPP: HppRegisterDevice:3109: Registering logical device with uid 'eui.610465c484794101931f35ab92209987'.
2025-07-23T09:05:02.236Z In(182) vmkernel: cpu36:2098204)StorageApdHandler: 965: APD Handle Created with lock[StorageApd-0x4306f6a5eaa0]
2025-07-23T09:05:02.236Z In(182) vmkernel: cpu36:2098204)StorageDevice: 1142: Alloc'd device 0x4306f6a7a980
2025-07-23T09:05:02.237Z In(182) vmkernel: cpu36:2098204)HPP: HppPathGroupMovePath:688: Path "vmhba65:C0:T0:L0" state changed from "dead" to "standby"
2025-07-23T09:05:02.237Z Wa(180) vmkwarning: cpu36:2098204)WARNING: HPP: HppRegisterDeviceEvents:3005: Could not register reservation events on device "Unregistered Device", Status: Failure. Event Registration will be retried on next path eval.
2025-07-23T09:05:02.237Z Wa(180) vmkwarning: cpu36:2098204)WARNING: HPP: HppRegisterDevice:3131: Not registering device eui.610465c484794101931f35ab92209987 - no active paths
2025-07-23T09:05:02.237Z In(182) vmkernel: cpu36:2098204)StorageApdHandler: 1051: Freeing APD handle 0x4306f6a5eaa0 []
2025-07-23T09:05:02.237Z In(182) vmkernel: cpu36:2098204)StorageApdHandler: 1135: APD Handle freed!
2025-07-23T09:05:21.139Z In(182) vmkernel: cpu36:2098204)HPP: HppRegisterDevice:3109: Registering logical device with uid 'eui.610465c484794101931f35ab92209987'.
2025-07-23T09:05:21.139Z In(182) vmkernel: cpu36:2098204)StorageApdHandler: 965: APD Handle Created with lock[StorageApd-0x4306f6a5eaa0]
2025-07-23T09:05:21.139Z In(182) vmkernel: cpu36:2098204)StorageDevice: 1142: Alloc'd device 0x4306f6a774c0
2025-07-23T09:05:21.140Z Wa(180) vmkwarning: cpu36:2098204)WARNING: HPP: HppRegisterDeviceEvents:3005: Could not register reservation events on device "Unregistered Device", Status: Failure. Event Registration will be retried on next path eval.
2025-07-23T09:05:21.140Z Wa(180) vmkwarning: cpu36:2098204)WARNING: HPP: HppRegisterDevice:3131: Not registering device eui.610465c484794101931f35ab92209987 - no active paths
2025-07-23T09:05:21.140Z In(182) vmkernel: cpu36:2098204)StorageApdHandler: 1051: Freeing APD handle 0x4306f6a5eaa0 []
2025-07-23T09:05:21.140Z In(182) vmkernel: cpu36:2098204)StorageApdHandler: 1135: APD Handle freed!
2025-07-23T09:05:21.140Z In(182) vmkernel: cpu36:2098204)HPP: HppRegisterDevice:3109: Registering logical device with uid 'eui.610465c484794101931f35ab92209987'.
2025-07-23T09:05:21.140Z In(182) vmkernel: cpu36:2098204)StorageApdHandler: 965: APD Handle Created with lock[StorageApd-0x4306f6a5eaa0]
2025-07-23T09:05:21.140Z In(182) vmkernel: cpu36:2098204)StorageDevice: 1142: Alloc'd device 0x4306f6a7a980
2025-07-23T09:05:21.140Z Wa(180) vmkwarning: cpu36:2098204)WARNING: HPP: HppRegisterDeviceEvents:3005: Could not register reservation events on device "Unregistered Device", Status: Failure. Event Registration will be retried on next path eval.
2025-07-23T09:05:21.140Z Wa(180) vmkwarning: cpu36:2098204)WARNING: HPP: HppRegisterDevice:3131: Not registering device eui.610465c484794101931f35ab92209987 - no active paths
2025-07-23T09:05:21.140Z In(182) vmkernel: cpu36:2098204)StorageApdHandler: 1051: Freeing APD handle 0x4306f6a5eaa0 []
2025-07-23T09:05:21.140Z In(182) vmkernel: cpu36:2098204)StorageApdHandler: 1135: APD Handle freed!

I checked the configuration multiple times and it looks fine by me.
I tried to make it work with the base image of 1.2.5 and tried a couple more. 1.3 and 1.4 images says ana group cannot be 0, and the 1.5 says server shutting down when i try to create the namespace.

That one explained the same issue imo:
#683
But even if i try to do it with 1 nvme node, not with 2, the same issue happens on the ceph 19.2.2

Interesing part is i tried to setup a ceph 18.2.7 version of that and it does work with the base image until i reboot the ceph nvme node. After i do reboot it, the container doesnt appear again on the node. Thats what i thought.... I have spammed podman ps, and seen it, the container gets created, and delete, this happens 3-4 times and after it stops. If i remove the daemon, and reconfigure everything on the nvmeof part, it works again if i am not rebooting it.

Is there any way to make them work? Or am i doing smth wrong? I checked the github issues on the opened and closed part, and i didnt really see anything what can fix it.

[caroav]

The nvmeof GW is not supported in squid. Please use tentacle or main branches.

[H3yRepo]

Hello caroav, thanks for the answer. Noted.

I have tried to setup the main branch with the details on the webpage, but the ninja -j3 build fails when it arrives to cephfs, if i have the log, i am going to attach it, i am currently trying to build it again.

Is there any recommended OS to build it? I tried with ubuntu 22.04.
I never really done that before.

This is the guide what i followed:
https://github.com/ceph/ceph/blob/main/README.md

[caroav]

ceph main is built daily, you can find the builds here https://shaman.ceph.com/builds/ceph/ - look for 'main' , and you will find the built targets. Then you can use the build sha to pull the ceph image.

[H3yRepo]

Hello, i have installed one of the main builds, but i have some problems.... The ceph not able to create the container on the nvme node, if i get it correctly, i am attaching every step what i have made, and the logs what i see for the nvme daemon:

Steps:

Installing ceph on ubuntu 22.04

wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo tee /etc/apt/trusted.gpg.d/ceph.asc

echo "deb [trusted=yes] https://1.chacra.ceph.com/r/ceph/main/8f62e8f08f8b4f52fec667e6daa3325e599b11a6/ubuntu/jammy/flavors/default jammy main" | sudo tee /etc/apt/sources.list.d/ceph.list

(https://shaman.ceph.com/builds/ceph/main/8f62e8f08f8b4f52fec667e6daa3325e599b11a6/default/444359)

apt update && apt install ceph -y && ceph --version

ceph version 20.3.0-1809-g8f62e8f0 (8f62e8f08f8b4f52fec667e6daa3325e599b11a6) tentacle (dev)

Creating the cluster:

sudo cephadm bootstrap --mon-ip 10.72.0.31

This is a development version of cephadm.
For information regarding the latest stable release:
https://docs.ceph.com/docs/tentacle/cephadm/install
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit systemd-timesyncd.service is enabled and running
Repeating the final host check...
podman (/usr/bin/podman) version 3.4.4 is present
systemctl is present
lvcreate is present
Unit systemd-timesyncd.service is enabled and running
Host looks OK
Cluster fsid: acfbd948-6934-11f0-b1f7-e545c5687884
Verifying IP 10.72.0.31 port 3300 ...
Verifying IP 10.72.0.31 port 6789 ...
Mon IP 10.72.0.31 is in CIDR network 10.72.0.0/24
Mon IP 10.72.0.31 is in CIDR network 10.72.0.0/24
Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network
Pulling container image quay.ceph.io/ceph-ci/ceph:main...
Ceph version: ceph version 20.3.0-1865-g92b1eca7 (92b1eca70229ca571f98bd4d86f62efcf43fbd2a) tentacle (dev)
Extracting ceph user uid/gid from container image...
Creating initial keys...
Creating initial monmap...
Creating mon...
Waiting for mon to start...
Waiting for mon...
mon is available
Assimilating anything we can from ceph.conf...
Generating new minimal ceph.conf...
Restarting the monitor...
Setting public_network to 10.72.0.0/24 in global config section
Wrote config to /etc/ceph/ceph.conf
Wrote keyring to /etc/ceph/ceph.client.admin.keyring
Creating mgr...
Verifying port 0.0.0.0:9283 ...
Verifying port 0.0.0.0:8765 ...
Verifying port 0.0.0.0:8443 ...
Waiting for mgr to start...
Waiting for mgr...
mgr not available, waiting (1/15)...
mgr not available, waiting (2/15)...
mgr not available, waiting (3/15)...
mgr not available, waiting (4/15)...
mgr is available
Enabling cephadm module...
Waiting for the mgr to restart...
Waiting for mgr epoch 5...
mgr epoch 5 is available
Verifying orchestrator module is enabled...
Setting orchestrator backend to cephadm...
Generating ssh key...
Wrote public SSH key to /etc/ceph/ceph.pub
Adding key to root@localhost authorized_keys...
Adding host ceph-testnode05...
Deploying mon service with default placement...
Deploying mgr service with default placement...
Deploying crash service with default placement...
Deploying ceph-exporter service with default placement...
Deploying prometheus service with default placement...
Deploying grafana service with default placement...
Deploying node-exporter service with default placement...
Deploying alertmanager service with default placement...
Enabling the dashboard module...
Waiting for the mgr to restart...
Waiting for mgr epoch 9...
mgr epoch 9 is available
Using certmgr to generate dashboard self-signed certificate...
Creating initial admin user...
Fetching dashboard port number...
Ceph Dashboard is now available at:

         URL: https://ceph-testnode05.h3yrepo.lan:8443/
        User: admin
    Password: xxxxxxxx

Enabling client.admin keyring and conf on hosts with "admin" label
Saving cluster configuration to /var/lib/ceph/acfbd948-6934-11f0-b1f7-e545c5687884/config directory
You can access the Ceph CLI as following in case of multi-cluster or non-default config:

    sudo /usr/sbin/cephadm shell --fsid acfbd948-6934-11f0-b1f7-e545c5687884 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring

Or, if you are only running a single cluster on this host:

    sudo /usr/sbin/cephadm shell

Please consider enabling telemetry to help improve Ceph:

    ceph telemetry on

For more information see:

    https://docs.ceph.com/en/latest/mgr/telemetry/

Bootstrap complete.
Enabling the logrotate.timer service to perform daily log rotation.

ceph orch daemon add osd ceph-testnode05:/dev/node05_vg-storage/node05_lvm-storage -> i have added an osd to the cluster, i can see it at the Cluster->OSDs as an in and up status

i've made a pool with rbd for the nvme -> name: nvme-pool

At the services tab i have added the nvmeof service to the node.

I am getting an error when the ceph wants to deploy the nvmeof daemon.

log:

nvmeof.nvme-pool.default.ceph-testnode05.zvomyg |   | error | 3 minutes ago | - | - |

An hour ago - daemon:nvmeof.nvme-pool.default.ceph-testnode05.zvomyg Failed to deploy: ['Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-acfbd948-6934-11f0-b1f7-e545c5687884-nvmeof-nvme-pool-default-ceph-testnode05-zvomyg\n/usr/bin/podman: stderr Error: error inspecting object: no such container ceph-acfbd948-6934-11f0-b1f7-e545c5687884-nvmeof-nvme-pool-default-ceph-testnode05-zvomyg\nNon-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-acfbd948-6934-11f0-b1f7-e545c5687884-nvmeof.nvme-pool.default.ceph-testnode05.zvomyg\n/usr/bin/podman: stderr Error: error inspecting object: no such container ceph-acfbd948-6934-11f0-b1f7-e545c5687884-nvmeof.nvme-pool.default.ceph-testnode05.zvomyg\nDeploy daemon nvmeof.nvme-pool.default.ceph-testnode05.zvomyg ...\nVerifying port 0.0.0.0:5500 ...\nVerifying port 0.0.0.0:4420 ...\nVerifying port 0.0.0.0:8009 ...\nVerifying port 0.0.0.0:10008 ...\nCreating ceph-nvmeof config...\nWrite file: /var/lib/ceph/acfbd948-6934-11f0-b1f7-e545c5687884/nvmeof.nvme-pool.default.ceph-testnode05.zvomyg/ceph-nvmeof.conf\nWrite file: /var/lib/ceph/acfbd948-6934-11f0-b1f7-e545c5687884/nvmeof.nvme-pool.default.ceph-testnode05.zvomyg/enable_dsa_acceleration\nFound a sysctl config file for a cluster with a different FSID (/etc/sysctl.d/90-ceph-acfbd948-6934-11f0-b1f7-e545c5687884-osd.conf).\nThis might obstruct the setting of new values. Consider deleting the file.\nNon-zero exit code 1 from systemctl start ceph-acfbd948-6934-11f0-b1f7-e545c5687884@nvmeof.nvme-pool.default.ceph-testnode05.zvomyg\nsystemctl: stderr Job for ceph-acfbd948-6934-11f0-b1f7-e545c5687884@nvmeof.nvme-pool.default.ceph-testnode05.zvomyg.service failed because the control process exited with error code.\nsystemctl: stderr See "systemctl status ceph-acfbd948-6934-11f0-b1f7-e545c5687884@nvmeof.nvme-pool.default.ceph-testnode05.zvomyg.service" and "journalctl -xeu ceph-acfbd948-6934-11f0-b1f7-e545c5687884@nvmeof.nvme-pool.default.ceph-testnode05.zvomyg.service" for details.\nsystemctl start failed for ceph-acfbd948-6934-11f0-b1f7-e545c5687884@nvmeof.nvme-pool.default.ceph-testnode05.zvomyg: Failed command: systemctl start ceph-acfbd948-6934-11f0-b1f7-e545c5687884@nvmeof.nvme-pool.default.ceph-testnode05.zvomyg: Job for ceph-acfbd948-6934-11f0-b1f7-e545c5687884@nvmeof.nvme-pool.default.ceph-testnode05.zvomyg.service failed because the control process exited with error code.\nSee "systemctl status ceph-acfbd948-6934-11f0-b1f7-e545c5687884@nvmeof.nvme-pool.default.ceph-testnode05.zvomyg.service" and "journalctl -xeu ceph-acfbd948-6934-11f0-b1f7-e545c5687884@nvmeof.nvme-pool.default.ceph-testnode05.zvomyg.service" for details.']

I have checked the node with podman ps, and the container is not here.

I have checked the fsid error part, but everywhere the cluster id is correct.

I have tried the latest 5 builds of the main (ubuntu 22.04 jammy), with the base nvmeof image, and the 1.5.1.

I tried to redeploy it, the same error happens.

Is there any version of the main branch where i can test the nvmeof for future plans? Do you have any recommendation?

Thanks for the help.

[caroav]

Please provide the output of :
"ceph config get mgr mgr/cephadm/container_image_nvmeof"

[H3yRepo]

I have tried again with the latest build of the main. (https://shaman.ceph.com/repos/ceph/main/1fc5167a56074cca34204da6b8f44052f0fc1718/default/354035)

error:

daemon:nvmeof.nvmeof-pool.default.ceph-testnode01.vmhoqw
Failed to deploy: ['Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-51ca5ca8-6b86-11f0-ad23-2bdf15d3371b-nvmeof-nvmeof-pool-default-ceph-testnode01-vmhoqw\n/usr/bin/podman: stderr Error: error inspecting object: no such container ceph-51ca5ca8-6b86-11f0-ad23-2bdf15d3371b-nvmeof-nvmeof-pool-default-ceph-testnode01-vmhoqw\nNon-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-51ca5ca8-6b86-11f0-ad23-2bdf15d3371b-nvmeof.nvmeof-pool.default.ceph-testnode01.vmhoqw\n/usr/bin/podman: stderr Error: error inspecting object: no such container ceph-51ca5ca8-6b86-11f0-ad23-2bdf15d3371b-nvmeof.nvmeof-pool.default.ceph-testnode01.vmhoqw\nDeploy daemon nvmeof.nvmeof-pool.default.ceph-testnode01.vmhoqw ...\nVerifying port 0.0.0.0:5500 ...\nVerifying port 0.0.0.0:4420 ...\nVerifying port 0.0.0.0:8009 ...\nVerifying port 0.0.0.0:10008 ...\nCreating ceph-nvmeof config...\nWrite file: /var/lib/ceph/51ca5ca8-6b86-11f0-ad23-2bdf15d3371b/nvmeof.nvmeof-pool.default.ceph-testnode01.vmhoqw/ceph-nvmeof.conf\nWrite file: /var/lib/ceph/51ca5ca8-6b86-11f0-ad23-2bdf15d3371b/nvmeof.nvmeof-pool.default.ceph-testnode01.vmhoqw/enable_dsa_acceleration\nFound a sysctl config file for a cluster with a different FSID (/etc/sysctl.d/90-ceph-51ca5ca8-6b86-11f0-ad23-2bdf15d3371b-osd.conf).\nThis might obstruct the setting of new values. Consider deleting the file.\nNon-zero exit code 1 from systemctl start ceph-51ca5ca8-6b86-11f0-ad23-2bdf15d3371b@nvmeof.nvmeof-pool.default.ceph-testnode01.vmhoqw\nsystemctl: stderr Job for ceph-51ca5ca8-6b86-11f0-ad23-2bdf15d3371b@nvmeof.nvmeof-pool.default.ceph-testnode01.vmhoqw.service failed because the control process exited with error code.\nsystemctl: stderr See "systemctl status ceph-51ca5ca8-6b86-11f0-ad23-2bdf15d3371b@nvmeof.nvmeof-pool.default.ceph-testnode01.vmhoqw.service" and "journalctl -xeu ceph-51ca5ca8-6b86-11f0-ad23-2bdf15d3371b@nvmeof.nvmeof-pool.default.ceph-testnode01.vmhoqw.service" for details.\nsystemctl start failed for ceph-51ca5ca8-6b86-11f0-ad23-2bdf15d3371b@nvmeof.nvmeof-pool.default.ceph-testnode01.vmhoqw: Failed command: systemctl start ceph-51ca5ca8-6b86-11f0-ad23-2bdf15d3371b@nvmeof.nvmeof-pool.default.ceph-testnode01.vmhoqw: Job for ceph-51ca5ca8-6b86-11f0-ad23-2bdf15d3371b@nvmeof.nvmeof-pool.default.ceph-testnode01.vmhoqw.service failed because the control process exited with error code.\nSee "systemctl status ceph-51ca5ca8-6b86-11f0-ad23-2bdf15d3371b@nvmeof.nvmeof-pool.default.ceph-testnode01.vmhoqw.service" and "journalctl -xeu ceph-51ca5ca8-6b86-11f0-ad23-2bdf15d3371b@nvmeof.nvmeof-pool.default.ceph-testnode01.vmhoqw.service" for details.']

root@ceph-testnode01:/home/admin# ceph config get mgr mgr/cephadm/container_image_nvmeof

quay.io/ceph/nvmeof:1.5

[caroav]

Please provide output of:

1. ceph orch ls and ceph orch ps
2. journactl of the gw:
   run systemctl -l | grep nvmeofpool //to find the nvmeof gw service name
   journalctl -u //to see the journal logs

[H3yRepo]

root@ceph-node01:/home/admin# ceph orch ls
NAME PORTS RUNNING REFRESHED AGE PLA CEMENT
alertmanager ?:9093,9094 1/1 10m ago 4d cou nt:1
ceph-exporter ?:9926 3/3 10m ago 4d *
crash 3/3 10m ago 4d *
grafana ?:3000 1/1 10m ago 4d cou nt:1
mgr 2/2 10m ago 4d cou nt:2
mon 3/5 10m ago 4d cou nt:5
node-exporter ?:9100 3/3 10m ago 4d *
nvmeof.nvmeof-pool.default ?:4420,5500,8009,10008 0/1 3m ago 4d cep h-node03
osd.default 3 10m ago 4d cep h-node03
prometheus ?:9095 1/1 10m ago 4d cou nt:1

root@ceph-node01:/home/admin# ceph orch ps
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
alertmanager.ceph-node01 ceph-node01 *:9093,9094 running (4d) 18s ago 4d 30.9M - 0.27.0 11f11916f8cd 6c0e707ae843
ceph-exporter.ceph-node01 ceph-node01 *:9926 running (4d) 18s ago 4d 23.1M - 20.3.0-1924-g1fc5167a 8a13b4874518 1ea3ccf7bed6
ceph-exporter.ceph-node02 ceph-node02 *:9926 running (4d) 8m ago 4d 27.0M - 20.3.0-1924-g1fc5167a 8a13b4874518 440b1f806bd1
ceph-exporter.ceph-node03 ceph-node03 *:9926 running (4d) 4m ago 4d 31.4M - 20.3.0-1924-g1fc5167a 8a13b4874518 69e306eb2aab
crash.ceph-node01 ceph-node01 running (4d) 18s ago 4d 18.4M - 20.3.0-1924-g1fc5167a 8a13b4874518 218c1c6b64e2
crash.ceph-node02 ceph-node02 running (4d) 8m ago 4d 7633k - 20.3.0-1924-g1fc5167a 8a13b4874518 e9cc7a339779
crash.ceph-node03 ceph-node03 running (4d) 4m ago 4d 4744k - 20.3.0-1924-g1fc5167a 8a13b4874518 d7db31894c7b
grafana.ceph-node01 ceph-node01 *:3000 running (4d) 18s ago 4d 163M - 11.6.0 5c42a1c2e40b e222b9ff25d9
mgr.ceph-node01.lhxrnf ceph-node01 *:9283,8765,8443 running (4d) 18s ago 4d 591M - 20.3.0-1924-g1fc5167a 8a13b4874518 e8d1461c113b
mgr.ceph-node02.hbntgi ceph-node02 *:8443,9283,8765 running (4d) 8m ago 4d 506M - 20.3.0-1924-g1fc5167a 8a13b4874518 ba4f6dedfcd5
mon.ceph-node01 ceph-node01 running (4d) 18s ago 4d 444M 2048M 20.3.0-1924-g1fc5167a 8a13b4874518 191b2155649d
mon.ceph-node02 ceph-node02 running (4d) 8m ago 4d 445M 2048M 20.3.0-1924-g1fc5167a 8a13b4874518 ff3846fb5af0
mon.ceph-node03 ceph-node03 running (4d) 4m ago 4d 418M 2048M 20.3.0-1924-g1fc5167a 8a13b4874518 0f298c9d39db
node-exporter.ceph-node01 ceph-node01 *:9100 running (4d) 18s ago 4d 12.9M - 1.7.0 72c9c2088986 2779c2abaf69
node-exporter.ceph-node02 ceph-node02 *:9100 running (4d) 8m ago 4d 10.8M - 1.7.0 72c9c2088986 c1c13fa2d659
node-exporter.ceph-node03 ceph-node03 *:9100 running (4d) 4m ago 4d 14.7M - 1.7.0 72c9c2088986 1b08fa29aee8
nvmeof.nvmeof-pool.default.ceph-node03.omydty ceph-node03 *:5500,4420,8009,10008 error 4m ago - - - 3c937764e6f5 e574cdede2e6
osd.0 ceph-node01 running (4d) 18s ago 4d 68.3M 4096M 20.3.0-1924-g1fc5167a 8a13b4874518 e712326c2b03
osd.1 ceph-node02 running (4d) 8m ago 4d 81.0M 4096M 20.3.0-1924-g1fc5167a 8a13b4874518 c971d46bf823
osd.2 ceph-node03 running (4d) 4m ago 4d 59.5M 4096M 20.3.0-1924-g1fc5167a 8a13b4874518 792366b51e6a
prometheus.ceph-node01 ceph-node01 *:9095 running (4d) 18s ago 4d 202M - 2.51.0 1d3b7f56885b ef6f3efed1b5

I didnt get any data for the systemctl one. I did run it where the container should be deployed. This is the /etc/systemd/system directory with grepping ceph:

root@ceph-node03:/home/admin# ls -l /etc/systemd/system |grep ceph
-rw-r--r-- 1 root root 1181 Jul 28 09:36 ceph-c6112f7c-6b92-11f0-bc8e-1d5fc47cb660@.service
-rw-r--r-- 1 root root 157 Jul 28 09:36 ceph-c6112f7c-6b92-11f0-bc8e-1d5fc47cb660.target
drwxr-xr-x 2 root root 4096 Jul 28 09:36 ceph-c6112f7c-6b92-11f0-bc8e-1d5fc47cb660.target.wants
-rw-r--r-- 1 root root 88 Jul 28 09:36 ceph.target
drwxr-xr-x 2 root root 4096 Jul 28 09:19 ceph.target.wants

These services running on the ceph-node03, where the container should be deployed:

root@ceph-node03:/home/admin# systemctl status
● ceph-node03
State: degraded
Jobs: 0 queued
Failed: 2 units
Since: Mon 2025-07-28 09:00:38 UTC; 4 days ago
CGroup: /
├─user.slice
│ ├─user-0.slice
│ │ ├─session-10.scope
│ │ │ └─8115 sshd: root@notty
│ │ ├─session-9.scope
│ │ │ └─8067 sshd: root@notty
│ │ ├─session-14.scope
│ │ │ └─8307 sshd: root@notty
│ │ ├─session-6.scope
│ │ │ └─7912 sshd: root
│ │ ├─session-11.scope
│ │ │ └─8163 sshd: root@notty
│ │ ├─session-15.scope
│ │ │ └─8355 sshd: root@notty
│ │ ├─session-12.scope
│ │ │ └─8211 sshd: root@notty
│ │ ├─session-16.scope
│ │ │ └─8403 sshd: root@notty
│ │ ├─session-8.scope
│ │ │ └─7930 sshd: root@notty
│ │ ├─user@0.service …
│ │ │ └─init.scope
│ │ │ ├─7915 /lib/systemd/systemd --user
│ │ │ └─7916 (sd-pam)
│ │ └─session-13.scope
│ │ └─8259 sshd: root@notty
│ └─user-1000.slice
│ ├─user@1000.service …
│ │ └─init.scope
│ │ ├─1192 /lib/systemd/systemd --user
│ │ └─1193 (sd-pam)
│ ├─session-123.scope
│ │ ├─1193274 sshd: kardosl_admin [priv]
│ │ ├─1193435 sshd: kardosl_admin@pts/1
│ │ ├─1193436 -bash
│ │ ├─1193450 sudo su
│ │ ├─1193451 sudo su
│ │ ├─1193452 su
│ │ ├─1193453 bash
│ │ ├─1193589 systemctl status
│ │ └─1193590 less
│ └─session-1.scope
│ ├─ 903 /bin/login -p --
│ ├─1200 -bash
│ ├─1217 sudo su
│ ├─1219 sudo su
│ ├─1220 su
│ └─1221 bash
├─init.scope
│ └─1 /sbin/init
└─system.slice
├─irqbalance.service
│ └─874 /usr/sbin/irqbalance --foreground
├─open-vm-tools.service
│ └─760 /usr/bin/vmtoolsd
├─packagekit.service
│ └─1589 /usr/libexec/packagekitd
├─systemd-networkd.service
│ └─835 /lib/systemd/systemd-networkd
├─systemd-udevd.service
│ └─539 /lib/systemd/systemd-udevd
├─cron.service
│ └─866 /usr/sbin/cron -f -P
├─polkit.service
│ └─876 /usr/libexec/polkitd --no-debug
├─networkd-dispatcher.service
│ └─875 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
├─system-ceph\x2dc6112f7c\x2d6b92\x2d11f0\x2dbc8e\x2d1d5fc47cb660.slice
│ ├─ceph-c6112f7c-6b92-11f0-bc8e-1d5fc47cb660@crash.ceph-node03.service …
│ │ ├─supervisor
│ │ │ └─12771 /usr/bin/conmon --api-version 1 -c d7db31894c7bbdac1872cb70a71feffc1c22ed999e9518f544ef4a578d673585 -u d7db31894c>
│ │ └─libpod-payload-d7db31894c7bbdac1872cb70a71feffc1c22ed999e9518f544ef4a578d673585
│ │ ├─12775 /dev/init -- /usr/bin/ceph-crash -n client.crash.ceph-node03
│ │ └─12778 /usr/bin/python3 -s /usr/bin/ceph-crash -n client.crash.ceph-node03
│ ├─ceph-c6112f7c-6b92-11f0-bc8e-1d5fc47cb660@osd.2.service …
│ │ ├─libpod-payload-792366b51e6ac66df42f5c9d19cd2df5bee0f110f22c510d066d030ddb81e368
│ │ │ ├─15560 /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-l>
│ │ │ └─15564 /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journal>
│ │ └─supervisor
│ │ └─15556 /usr/bin/conmon --api-version 1 -c 792366b51e6ac66df42f5c9d19cd2df5bee0f110f22c510d066d030ddb81e368 -u 792366b51e>
│ ├─ceph-c6112f7c-6b92-11f0-bc8e-1d5fc47cb660@mon.ceph-node03.service …
│ │ ├─libpod-payload-0f298c9d39db2672f46141944b520e28ccc6e917c11d92bef6f5fe35440d1909
│ │ │ ├─16990 /dev/init -- /usr/bin/ceph-mon -n mon.ceph-node03 -f --setuser ceph --setgroup ceph --default-log-to-file=false ->
│ │ │ └─16994 /usr/bin/ceph-mon -n mon.ceph-node03 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log->
│ │ └─supervisor
│ │ └─16986 /usr/bin/conmon --api-version 1 -c 0f298c9d39db2672f46141944b520e28ccc6e917c11d92bef6f5fe35440d1909 -u 0f298c9d39>
│ ├─ceph-c6112f7c-6b92-11f0-bc8e-1d5fc47cb660@ceph-exporter.ceph-node03.service …
│ │ ├─libpod-payload-69e306eb2aab71bc61a037c6565c782c2ae3c2b32d41946283d8c28540dadbec
│ │ │ ├─12368 /dev/init -- /usr/bin/ceph-exporter -n client.ceph-exporter.ceph-node03 -f --sock-dir=/var/run/ceph/ --addrs=:: ->
│ │ │ └─12371 /usr/bin/ceph-exporter -n client.ceph-exporter.ceph-node03 -f --sock-dir=/var/run/ceph/ --addrs=:: --port=9926 -->
│ │ └─supervisor
│ │ └─12365 /usr/bin/conmon --api-version 1 -c 69e306eb2aab71bc61a037c6565c782c2ae3c2b32d41946283d8c28540dadbec -u 69e306eb2a>
│ └─ceph-c6112f7c-6b92-11f0-bc8e-1d5fc47cb660@node-exporter.ceph-node03.service …
│ ├─libpod-payload-1b08fa29aee8668030f5c01b54d7f2e6578ee4e5b25ecdcb89d8c767114e2d7f
│ │ ├─13129 /dev/init -- /bin/node_exporter --no-collector.timex --path.procfs=/host/proc --path.sysfs=/host/sys --path.rootf>
│ │ └─13132 /bin/node_exporter --no-collector.timex --path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs
│ └─supervisor
│ └─13126 /usr/bin/conmon --api-version 1 -c 1b08fa29aee8668030f5c01b54d7f2e6578ee4e5b25ecdcb89d8c767114e2d7f -u 1b08fa29ae>
├─multipathd.service
│ └─544 /sbin/multipathd -d -s
├─ModemManager.service
│ └─934 /usr/sbin/ModemManager
├─systemd-journald.service
│ └─505 /lib/systemd/systemd-journald
├─unattended-upgrades.service
│ └─933 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
├─ssh.service
│ └─1848 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
├─ceph-crash.service
│ └─6413 /usr/bin/python3 /usr/bin/ceph-crash
├─vgauth.service
│ └─759 /usr/bin/VGAuthService
├─snapd.service
│ └─7046 /usr/lib/snapd/snapd
├─rsyslog.service
│ └─865434 /usr/sbin/rsyslogd -n -iNONE
├─upower.service
│ └─135988 /usr/libexec/upowerd
├─systemd-resolved.service
│ └─837 /lib/systemd/systemd-resolved
├─udisks2.service
│ └─888 /usr/libexec/udisks2/udisksd
├─dbus.service
│ └─867 @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
├─systemd-timesyncd.service
│ └─757 /lib/systemd/systemd-timesyncd
└─systemd-logind.service
└─886 /lib/systemd/systemd-logind
lines 92-134/134 (END)
│ │ └─16986 /usr/bin/conmon --api-version 1 -c 0f298c9d39db2672f46141944b520e28ccc6e917c11d92bef6f5fe35440d1909 -u 0f298c9d39db2672f46141944b520e28ccc6e917c11d92bef6f5fe35440d1909 -r /usr/bin/crun -b /var/lib/containers/s>
│ ├─ceph-c6112f7c-6b92-11f0-bc8e-1d5fc47cb660@ceph-exporter.ceph-node03.service …
│ │ ├─libpod-payload-69e306eb2aab71bc61a037c6565c782c2ae3c2b32d41946283d8c28540dadbec
│ │ │ ├─12368 /dev/init -- /usr/bin/ceph-exporter -n client.ceph-exporter.ceph-node03 -f --sock-dir=/var/run/ceph/ --addrs=:: --port=9926 --prio-limit=5 --stats-period=5
│ │ │ └─12371 /usr/bin/ceph-exporter -n client.ceph-exporter.ceph-node03 -f --sock-dir=/var/run/ceph/ --addrs=:: --port=9926 --prio-limit=5 --stats-period=5
│ │ └─supervisor
│ │ └─12365 /usr/bin/conmon --api-version 1 -c 69e306eb2aab71bc61a037c6565c782c2ae3c2b32d41946283d8c28540dadbec -u 69e306eb2aab71bc61a037c6565c782c2ae3c2b32d41946283d8c28540dadbec -r /usr/bin/crun -b /var/lib/containers/s>
│ └─ceph-c6112f7c-6b92-11f0-bc8e-1d5fc47cb660@node-exporter.ceph-node03.service …
│ ├─libpod-payload-1b08fa29aee8668030f5c01b54d7f2e6578ee4e5b25ecdcb89d8c767114e2d7f
│ │ ├─13129 /dev/init -- /bin/node_exporter --no-collector.timex --path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs
│ │ └─13132 /bin/node_exporter --no-collector.timex --path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs
│ └─supervisor
│ └─13126 /usr/bin/conmon --api-version 1 -c 1b08fa29aee8668030f5c01b54d7f2e6578ee4e5b25ecdcb89d8c767114e2d7f -u 1b08fa29aee8668030f5c01b54d7f2e6578ee4e5b25ecdcb89d8c767114e2d7f -r /usr/bin/crun -b /var/lib/containers/s>
├─multipathd.service
│ └─544 /sbin/multipathd -d -s
├─ModemManager.service
│ └─934 /usr/sbin/ModemManager
├─systemd-journald.service
│ └─505 /lib/systemd/systemd-journald
├─unattended-upgrades.service
│ └─933 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
├─ssh.service
│ └─1848 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
├─ceph-crash.service
│ └─6413 /usr/bin/python3 /usr/bin/ceph-crash
├─vgauth.service
│ └─759 /usr/bin/VGAuthService
├─snapd.service
│ └─7046 /usr/lib/snapd/snapd
├─rsyslog.service
│ └─865434 /usr/sbin/rsyslogd -n -iNONE
├─upower.service
│ └─135988 /usr/libexec/upowerd
├─systemd-resolved.service
│ └─837 /lib/systemd/systemd-resolved
├─udisks2.service
│ └─888 /usr/libexec/udisks2/udisksd
├─dbus.service
│ └─867 @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
├─systemd-timesyncd.service
│ └─757 /lib/systemd/systemd-timesyncd
└─systemd-logind.service
└─886 /lib/systemd/systemd-logind

Is it possible the ceph orch not able to create the systemd on the nvmeof target server, and thats why everything is failing?

[caroav]

we must see the log of the nvmeof daemons. What do you get when you run:
"systemctl -l | grep nvmeofpool" on the node where you try to run the gw?
can you send me a mail to aviv.caro@ibm.com?