Create namespace failed when I use all the capacity tnvmcap with an NVMe disk which has nn=1

[yizhanglinux]

# nvme list -v
Subsystem        Subsystem-NQN                                                                                    Controllers
---------------- ------------------------------------------------------------------------------------------------ ----------------
nvme-subsys0     nqn.1994-11.com.samsung:nvme:PM1733:HHHL:S55JNC0T900416                                          nvme0
nvme-subsys1     nqn.2014.08.org.nvmexpress:80868086PHLJ108400MJ2P0BGN  INTEL SSDPE2KX020T8                       nvme1
nvme-subsys2     nqn.1994-11.com.samsung:nvme:PM1733:HHHL:S55JNC0T900429                                          nvme2
nvme-subsys3     nqn.2014.08.org.nvmexpress:80868086PHLJ912000ZQ1P0FGN  INTEL SSDPE2KX010T8                       nvme3

Device           Cntlid SN                   MN                                       FR       TxPort Address        Slot   Subsystem    Namespaces
---------------- ------ -------------------- ---------------------------------------- -------- ------ -------------- ------ ------------ ----------------
nvme0    65     S55JNC0T900416       SAMSUNG MZPLJ1T6HBJR-00007               EPK9CB5Q pcie   0000:3b:00.0          nvme-subsys0 nvme0n1
nvme1    0      PHLJ108400MJ2P0BGN   INTEL SSDPE2KX020T8                      VDV10131 pcie   0000:88:00.0   168    nvme-subsys1 nvme1n1
nvme2    65     S55JNC0T900429       SAMSUNG MZPLJ1T6HBJR-00007               EPK9CB5Q pcie   0000:5e:00.0          nvme-subsys2 nvme2n1
nvme3    0      PHLJ912000ZQ1P0FGN   INTEL SSDPE2KX010T8                      VDV10131 pcie   0000:89:00.0   169    nvme-subsys3

Device            Generic           NSID       Usage                      Format           Controllers
----------------- ----------------- ---------- -------------------------- ---------------- ----------------
/dev/nvme0n1 /dev/ng0n1   0x1          1.60  TB /   1.60  TB    512   B +  0 B   nvme0
/dev/nvme1n1 /dev/ng1n1   0x1          2.00  TB /   1.95  TB    512   B +  0 B   nvme1
/dev/nvme2n1 /dev/ng2n1   0x1          1.60  TB /   1.60  TB    512   B +  0 B   nvme2

# nvme id-ctrl /dev/nvme1 -v | grep -E "^nn|cntlid|nvm|NS Man"
cntlid    : 0
nvmsr     : 0
  [3:3] : 0x1	NS Management and Attachment Supported
tnvmcap   : 2000398934016
unvmcap   : 50483781632
nn        : 1
# nvme delete-ns /dev/nvme1n1
delete-ns: success
nsid: 1
# nsze=$((2000398934016/512))
# echo $nsze
3907029168
# ctrl_dev=nvme1
# nvme create-ns /dev/"$ctrl_dev" --nsze="$nsze" --ncap="$nsze" -b 512
NVMe status: Namespace Insufficient Capacity: Creating the namespace requires more free space than is currently available(0x4115)
NS management and attachment not supported
# echo $nsze
3907029168
# nsze=3900029168
# nvme create-ns /dev/"$ctrl_dev" --nsze="$nsze" --ncap="$nsze" -b 512
create-ns: success
nsid: 1
# nvme delete-ns /dev/nvme1n1
delete-ns: success
nsid: 1
# nsze=3906029168
# nvme create-ns /dev/"$ctrl_dev" --nsze="$nsze" --ncap="$nsze" -b 512
create-ns: success
nsid: 1
# nvme delete-ns /dev/nvme1n1
delete-ns: success
nsid: 1
# nsze=3907029168
# nvme create-ns /dev/"$ctrl_dev" --nsze="$nsze" --ncap="$nsze" -b 512
NVMe status: Namespace Insufficient Capacity: Creating the namespace requires more free space than is currently available(0x4115)
NS management and attachment not supported
# nsze=3907029100
# nvme create-ns /dev/"$ctrl_dev" --nsze="$nsze" --ncap="$nsze" -b 512
NVMe status: Namespace Insufficient Capacity: Creating the namespace requires more free space than is currently available(0x4115)
NS management and attachment not supported
# nsze=3907028168
# nvme create-ns /dev/"$ctrl_dev" --nsze="$nsze" --ncap="$nsze" -b 512
NVMe status: Namespace Insufficient Capacity: Creating the namespace requires more free space than is currently available(0x4115)
NS management and attachment not supported
# nsze=3906028168
# nvme create-ns /dev/"$ctrl_dev" --nsze="$nsze" --ncap="$nsze" -b 512
create-ns: success
nsid: 1
# cntlid=0
#  nvme attach-ns /dev/"$ctrl_dev" --namespace-id="$nid" --controllers="$cntlid"
attach-ns: success
nsid: 1
# nvme list
Node                  Generic               SN                   Model                                    Namespace  Usage                      Format           FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
/dev/nvme0n1          /dev/ng0n1            S55JNC0T900416       SAMSUNG MZPLJ1T6HBJR-00007               0x1          1.60  TB /   1.60  TB    512   B +  0 B   EPK9CB5Q
/dev/nvme1n1          /dev/ng1n1            PHLJ108400MJ2P0BGN   INTEL SSDPE2KX020T8                      0x1          2.00  TB /   1.95  TB    512   B +  0 B   VDV10131
/dev/nvme2n1          /dev/ng2n1            S55JNC0T900429       SAMSUNG MZPLJ1T6HBJR-00007               0x1          1.60  TB /   1.60  TB    512   B +  0 B   EPK9CB5Q
# uname -r
6.15.0-0.rc6.49.eln148.x86_64

[igaw]

Not sure what's happening here. There is some logic for calculating the LBAs. create-ns is using parse_lba_num_si to for this. Maybe there is something wrong. Can you run your test and use -vv, this will show the commands issued. If the numbers are not always identically it points towards parse_lba_num_si I think.

[yizhanglinux]

Here is the output with -vv

# nvme id-ctrl /dev/nvme0 -v | grep -E "^nn|cntlid|NS M|nvm"
cntlid    : 0
nvmsr     : 0
  [3:3] : 0x1	NS Management and Attachment Supported
tnvmcap   : 2000398934016
unvmcap   : 2000398934016
nn        : 1
# nsze=$((2000398934016/512))
# nvme create-ns /dev/"$ctrl_dev" --nsze="$nsze" --ncap="$nsze" -b 512  -vv
opcode       : 06
flags        : 00
rsvd1        : 0000
nsid         : ffffffff
cdw2         : 00000000
cdw3         : 00000000
data_len     : 00001000
metadata_len : 00000000
addr         : 55e9232a3000
metadata     : 0
cdw10        : 00000000
cdw11        : 00000000
cdw12        : 00000000
cdw13        : 00000000
cdw14        : 00000000
cdw15        : 00000000
timeout_ms   : 00000000
result       : 00000000
err          : 0
latency      : 68 us
opcode       : 06
flags        : 00
rsvd1        : 0000
nsid         : 00000000
cdw2         : 00000000
cdw3         : 00000000
data_len     : 00001000
metadata_len : 00000000
addr         : 55e9232a5000
metadata     : 0
cdw10        : 00000001
cdw11        : 00000000
cdw12        : 00000000
cdw13        : 00000000
cdw14        : 00000000
cdw15        : 00000000
timeout_ms   : 00000000
result       : 00000000
err          : 0
latency      : 46 us
opcode       : 0d
flags        : 00
rsvd1        : 0000
nsid         : 00000000
cdw2         : 00000000
cdw3         : 00000000
data_len     : 00001000
metadata_len : 00000000
addr         : 55e9232a7000
metadata     : 0
cdw10        : 00000000
cdw11        : 00000000
cdw12        : 00000000
cdw13        : 00000000
cdw14        : 00000000
cdw15        : 00000000
timeout_ms   : 0001d4c0
result       : 00000000
err          : 16661
latency      : 22 us
NVMe status: Namespace Insufficient Capacity: Creating the namespace requires more free space than is currently available(0x4115)
opcode       : 06
flags        : 00
rsvd1        : 0000
nsid         : 00000000
cdw2         : 00000000
cdw3         : 00000000
data_len     : 00001000
metadata_len : 00000000
addr         : 55e9232a9000
metadata     : 0
cdw10        : 00000001
cdw11        : 00000000
cdw12        : 00000000
cdw13        : 00000000
cdw14        : 00000000
cdw15        : 00000000
timeout_ms   : 00000000
result       : 00000000
err          : 0
latency      : 42 us

[igaw]

Darn, the inteesting part is not in the command dwords. Could you check if tnvmcap does not change after the ns delete? Just to rule this out.

[yizhanglinux]

The tnvmcap not changed from the log:

# nvme id-ctrl -v /dev/nvme0 | grep -E "^nn|cntlid|tnvm|unvm|NS M"
cntlid    : 0
  [3:3] : 0x1	NS Management and Attachment Supported
tnvmcap   : 2000398934016
unvmcap   : 17915904
nn        : 1
# nvme delete-ns /dev/nvme0n1
delete-ns: Success, deleted nsid:1
# nvme id-ctrl -v /dev/nvme0 | grep -E "^nn|cntlid|tnvm|unvm|NS M"
cntlid    : 0
  [3:3] : 0x1	NS Management and Attachment Supported
tnvmcap   : 2000398934016
unvmcap   : 2000398934016
nn        : 1

[igaw]

Are you running these steps without a delay? One thing is that you need to wait for the firmware and kernel to finish the operation before doing the next step: https://github.com/linux-nvme/nvme-cli/blob/master/tests/nvme_create_max_ns_test.py

[ikegami-t]

Looks the NS capacity not unallocated completely soon in the drive after the NS deletion as the NS creation succeeded after repeated since the time elapsed to complete the NS deletion.

[yizhanglinux]

Are you running these steps without a delay? One thing is that you need to wait for the firmware and kernel to finish the operation before doing the next step: https://github.com/linux-nvme/nvme-cli/blob/master/tests/nvme_create_max_ns_test.py

I also tried to delay and reboot the server, but the create-ns operation still failed.

[yizhanglinux]

Just tried the test nvme_create_max_ns_test.py, and got the same error.

#  nose2 --verbose --start-dir tests nvme_create_max_ns_test
test_attach_detach_ns (nvme_create_max_ns_test.TestNVMeCreateMaxNS)
Testcase main ...
Using nvme binary 'nvme'
NVMe status: Namespace Insufficient Capacity: Creating the namespace requires more free space than is currently available(0x4115)
FAIL



======================================================================
FAIL: test_attach_detach_ns (nvme_create_max_ns_test.TestNVMeCreateMaxNS)
Testcase main
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/nvme-cli/tests/nvme_create_max_ns_test.py", line 50, in setUp
    super().setUp()
  File "/root/nvme-cli/tests/nvme_test.py", line 80, in setUp
    self.create_and_attach_default_ns()
  File "/root/nvme-cli/tests/nvme_test.py", line 114, in create_and_attach_default_ns
    self.assertEqual(err, 0)
AssertionError: 1 != 0

----------------------------------------------------------------------
Ran 1 test in 5.723s

FAILED (failures=1)

[igaw]

The nightly build runs nvme_create_max_ns_test against a disk using latest kernel. We haven't seen something like you report here. I don't think this is a nvme-cli problem.

Also it survives a reboot, I am pretty sure it origin from the disk itself. I suggest your report this to your disk vendor.

[ikegami-t]

You can test with the admin-passthru command instead of the create-ns command to check if the error caused by (and depended) the command or not to make sure.

[yizhanglinux]

Yeah, to reproduce it, we need to find one NVMe disk with "nn: 1." and I cannot reproduce it when I use one NVMe disk with "nn:32".

[HaroPanosyan]

We were able to reproduce this issue at Solidigm (formerly Intel SSD division).
Without documentation to confirm fully, this might be related to over-provisioning for better endurance.

Here is a question on chrome:
what is over provisioning size on intel cliffdale 2tb drive

And here is partial answer from AI:
Over-provisioning on an Intel Cliffdale 2TB drive is generally managed by the SSD controller itself. The amount of over-provisioning can vary, but SSD manufacturers typically reserve 7% of physical storage space for background activities like garbage collection and wear-leveling

As you can see we might have 7% reserved for over-provisioning and the maximum NS size creation in this post I could find is 3906029168. This number is ~90% of the total nvme capacity of 3907029168, so 10% being more than 7% the controller could successfully create the namespace.

If I find actual specification related to what this drives reserves for over-provisioning, I will post it here, but looks like AI knows it already :)

[ikegami-t]

Is this really related to the overprovisioning?

1. Actually the same size: 3907029168 NS creation reported as both failed and succeeded actually as below.

# echo $nsze
3907029168
# ctrl_dev=nvme1
# nvme create-ns /dev/"$ctrl_dev" --nsze="$nsze" --ncap="$nsze" -b 512
NVMe status: Namespace Insufficient Capacity: Creating the namespace requires more free space than is currently available(0x4115)
NS management and attachment not supported
# echo $nsze
3907029168
# nsze=3900029168
# nvme create-ns /dev/"$ctrl_dev" --nsze="$nsze" --ncap="$nsze" -b 512
create-ns: success
nsid: 1

2. Seems the TNVMCAP not included the overprovisioning capacity.
   
3. Overprovision is hidden storage so seems as not should be included by the TNVMCAP.
   

By the way about the nvme list NS usage output below the namespace utilization (NUSE) bytes: 2.00 TB exceeded the namespace utilization (NUSE) bytes: 1.95 TB this looks not correct behavior. But not sure if related to the error behavior.

# nvme list -v
...
Device            Generic           NSID       Usage                      Format           Controllers
----------------- ----------------- ---------- -------------------------- ---------------- ----------------
...
/dev/nvme1n1 /dev/ng1n1   0x1          2.00  TB /   1.95  TB    512   B +  0 B   nvme1