CI nighly build: creating namespaces fails sometimes

[igaw]

The nightyli CI build fails because the nvme create-ns sometimes fails.

last good: https://github.com/linux-nvme/nvme-cli/actions/runs/23993741402
first broken: https://github.com/linux-nvme/nvme-cli/actions/runs/24018026373
with extensive logging: https://github.com/linux-nvme/nvme-cli/actions/runs/24180791444

Note this against the some nvme-cli version, so we can rule out changes in nvme-cli which are causing this. And as far I can there wasn't many changes in the kernel. Anyway I don't know what is causing it.

The failure looks like this:

=================================== 62/81 ====================================
test:         nvme-cli - nvme_error_log_test
start time:   09:20:37
duration:     3.74s
result:       exit status 0
command:      MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 PATH=/nvme-cli/.build-ci:/usr/bin:/usr/sbin MALLOC_PERTURB_=99 MESON_TEST_ITERATION=1 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 /usr/bin/python3 /nvme-cli/tests/tap_runner.py --start-dir /nvme-cli/tests nvme_error_log_test
----------------------------------- stdout -----------------------------------
TAP version 13
1..1
# Using nvme binary '/nvme-cli/.build-ci/nvme'
...
# /nvme-cli/.build-ci/nvme create-ns /dev/nvme1 --nsze=1875385008 --ncap=1875385008 --flbas=0 --dps=0 --verbose --output-format=json
not ok 1 - test_get_error_log (TestNVMeErrorLogCmd)
----------------------------------- stderr -----------------------------------
  ---
  traceback: |
    Traceback (most recent call last):
      File "/usr/lib/python3.13/unittest/case.py", line 58, in testPartExecutor
        yield
      File "/usr/lib/python3.13/unittest/case.py", line 647, in run
        self._callSetUp()
        ~~~~~~~~~~~~~~~^^
      File "/usr/lib/python3.13/unittest/case.py", line 603, in _callSetUp
        self.setUp()
        ~~~~~~~~~~^^
      File "/nvme-cli/tests/nvme_error_log_test.py", line 42, in setUp
        super().setUp()
        ~~~~~~~~~~~~~^^
      File "/nvme-cli/tests/nvme_test.py", line 90, in setUp
        self.create_and_attach_default_ns()
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
      File "/nvme-cli/tests/nvme_test.py", line 125, in create_and_attach_default_ns
        self.assertEqual(err, 0)
        ~~~~~~~~~~~~~~~~^^^^^^^^
    AssertionError: 1 != 0
  ...
==============================================================================

but sometimes during the same run:

=================================== 80/81 ====================================
test:         nvme-cli - nvme_create_max_ns_test
start time:   09:20:36
duration:     14.09s
result:       exit status 0
command:      MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MALLOC_PERTURB_=104 PATH=/nvme-cli/.build-ci:/usr/bin:/usr/sbin MESON_TEST_ITERATION=1 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 /usr/bin/python3 /nvme-cli/tests/tap_runner.py --start-dir /nvme-cli/tests nvme_create_max_ns_test
----------------------------------- stdout -----------------------------------
TAP version 13
1..1
# Using nvme binary '/nvme-cli/.build-ci/nvme'
...
# /nvme-cli/.build-ci/nvme create-ns /dev/nvme1 --nsze=1875385008 --ncap=1875385008 --flbas=0 --dps=0 --verbose --output-format=json
# {
#   "create-ns":"success",
#   "nsid":"1"
# }
# 
# /nvme-cli/.build-ci/nvme id-ns /dev/nvme1 --namespace-id=1
# ##### Attaching 1
# /nvme-cli/.build-ci/nvme attach-ns /dev/nvme1 --namespace-id=1 --controllers=0 --verbose
# ##### Running IOs in 1
# /nvme-cli/.build-ci/nvme id-ns /dev/nvme1n1 --output-format=json
# {
#   "nsze":7325728,
#   "ncap":7325728, 
...
# ##### Creating 2
# /nvme-cli/.build-ci/nvme create-ns /dev/nvme1 --nsze=7325722 --ncap=7325722 --flbas=0 --dps=0 --verbose --output-format=json
# {
#   "create-ns":"success",
#   "nsid":"2"
# }
# 
# /nvme-cli/.build-ci/nvme id-ns /dev/nvme1 --namespace-id=2
# ##### Attaching 2
# /nvme-cli/.build-ci/nvme attach-ns /dev/nvme1 --namespace-id=2 --controllers=0 --verbose
not ok 1 - test_attach_detach_ns (TestNVMeCreateMaxNS)
# /nvme-cli/.build-ci/nvme create-ns /dev/nvme1 --nsze=7325722 --ncap=7325722 --flbas=0 --dps=0 --verbose --output-format=json
not ok 2 - test_attach_detach_ns (TestNVMeCreateMaxNS)
----------------------------------- stderr -----------------------------------
  ---
  traceback: |
    Traceback (most recent call last):
      File "/usr/lib/python3.13/unittest/case.py", line 58, in testPartExecutor
        yield
      File "/usr/lib/python3.13/unittest/case.py", line 651, in run
        self._callTestMethod(testMethod)
        ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
      File "/usr/lib/python3.13/unittest/case.py", line 606, in _callTestMethod
        if method() is not None:
           ~~~~~~^^
      File "/nvme-cli/tests/nvme_create_max_ns_test.py", line 98, in test_attach_detach_ns
        self.assertEqual(self.attach_ns(self.ctrl_id, nsid), 0)
        ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    AssertionError: 1 != 0
  ...
  ---
  traceback: |
    Traceback (most recent call last):
      File "/usr/lib/python3.13/unittest/case.py", line 58, in testPartExecutor
        yield
      File "/usr/lib/python3.13/unittest/case.py", line 654, in run
        self._callTearDown()
        ~~~~~~~~~~~~~~~~~~^^
      File "/usr/lib/python3.13/unittest/case.py", line 611, in _callTearDown
        self.tearDown()
        ~~~~~~~~~~~~~^^
      File "/nvme-cli/tests/nvme_create_max_ns_test.py", line 78, in tearDown
        self.assertEqual(self.create_and_validate_ns(self.default_nsid,
        ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                                     self.nsze,
                                                     ^^^^^^^^^^
                                                     self.ncap,
                                                     ^^^^^^^^^^
                                                     self.flbas,
                                                     ^^^^^^^^^^^
                                                     self.dps), 0)
                                                     ^^^^^^^^^^^^^
    AssertionError: 1 != 0
  ...
==============================================================================

[igaw]

Ohhh wait a minute! This could be caused by parallel execution of the tests....

[igaw]

After marking the tests as non parallel workload, only one test failed:

=================================== 64/81 ====================================
test:         nvme-cli - nvme_create_max_ns_test
start time:   11:50:22
duration:     12.97s
result:       exit status 0
command:      MESON_TEST_ITERATION=1 MALLOC_PERTURB_=251 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 PATH=/nvme-cli/.build-ci:/usr/bin:/usr/sbin UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 /usr/bin/python3 /nvme-cli/tests/tap_runner.py --start-dir /nvme-cli/tests nvme_create_max_ns_test
----------------------------------- stdout -----------------------------------
TAP version 13
1..1
...
# ##### Testing max_ns: 128
# ##### Creating 1
# /nvme-cli/.build-ci/nvme create-ns /dev/nvme0 --nsze=7325722 --ncap=7325722 --flbas=0 --dps=0 --verbose --output-format=json
# {
#   "create-ns":"success",
#   "nsid":"1"
# }
# 
# /nvme-cli/.build-ci/nvme id-ns /dev/nvme0 --namespace-id=1
# ##### Attaching 1
# /nvme-cli/.build-ci/nvme attach-ns /dev/nvme0 --namespace-id=1 --controllers=0 --verbose
# ##### Running IOs in 1
# /nvme-cli/.build-ci/nvme id-ns /dev/nvme0n1 --output-format=json
# {
#   "nsze":7325728,
#   "ncap":7325728,
#   "nuse":0,
#   "nsfeat":2,
#   "nlbaf":4,
#   "flbas":0,
#   "mc":3,
#   "dpc":23,
#   "dps":0,
#   "nmic":0,
#   "rescap":255,
#   "fpi":128,
#   "dlfeat":9,
#   "nawun":255,
#   "nawupf":255,
#   "nacwu":0,
#   "nabsn":0,
#   "nabo":0,
#   "nabspf":0,
#   "noiob":0,
#   "nvmcap":4294967296,
#   "mssrl":0,
#   "mcl":0,
#   "msrc":0,
#   "kpios":0,
#   "nulbaf":0,
#   "kpiodaag":0,
#   "anagrpid":0,
#   "nsattr":0,
#   "nvmsetid":0,
#   "endgid":0,
#   "nguid":"b2310000000000000014ee83023aad80",
#   "eui64":"0014ee83023aad80",
#   "lbafs":[
#     {
#       "lbaf":0,
#       "ms":0,
#       "ds":9,
#       "rp":0,
#       "in_use":1
#     },
#     {
#       "lbaf":1,
#       "ms":8,
#       "ds":9,
#       "rp":0,
#       "in_use":0
#     },
#     {
#       "lbaf":2,
#       "ms":0,
#       "ds":12,
#       "rp":0,
#       "in_use":0
#     },
#     {
#       "lbaf":3,
#       "ms":8,
#       "ds":12,
#       "rp":0,
#       "in_use":0
#     },
#     {
#       "lbaf":4,
#       "ms":64,
#       "ds":12,
#       "rp":0,
#       "in_use":0
#     }
#   ],
#   "vs":[]
# }
# 
# dd if=/dev/nvme0n1 of=/dev/null bs=512 count=1 > /dev/null 2>&1
# dd if=/dev/zero of=/dev/nvme0n1 bs=512 count=1 > /dev/null 2>&1
# ##### Creating 2
# /nvme-cli/.build-ci/nvme create-ns /dev/nvme0 --nsze=7325722 --ncap=7325722 --flbas=0 --dps=0 --verbose --output-format=json
# {
#   "create-ns":"success",
#   "nsid":"2"
# }
# 
# /nvme-cli/.build-ci/nvme id-ns /dev/nvme0 --namespace-id=2
# ##### Attaching 2
# /nvme-cli/.build-ci/nvme attach-ns /dev/nvme0 --namespace-id=2 --controllers=0 --verbose
not ok 1 - test_attach_detach_ns (TestNVMeCreateMaxNS)
# /nvme-cli/.build-ci/nvme create-ns /dev/nvme0 --nsze=7325722 --ncap=7325722 --flbas=0 --dps=0 --verbose --output-format=json
# {
#   "create-ns":"success",
#   "nsid":"3"
# }
# 
not ok 2 - test_attach_detach_ns (TestNVMeCreateMaxNS)
----------------------------------- stderr -----------------------------------
  ---
  traceback: |
    Traceback (most recent call last):
      File "/usr/lib/python3.13/unittest/case.py", line 58, in testPartExecutor
        yield
      File "/usr/lib/python3.13/unittest/case.py", line 651, in run
        self._callTestMethod(testMethod)
        ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
      File "/usr/lib/python3.13/unittest/case.py", line 606, in _callTestMethod
        if method() is not None:
           ~~~~~~^^
      File "/nvme-cli/tests/nvme_create_max_ns_test.py", line 98, in test_attach_detach_ns
        self.assertEqual(self.attach_ns(self.ctrl_id, nsid), 0)
        ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    AssertionError: 1 != 0
  ...
  ---
  traceback: |
    Traceback (most recent call last):
      File "/usr/lib/python3.13/unittest/case.py", line 58, in testPartExecutor
        yield
      File "/usr/lib/python3.13/unittest/case.py", line 654, in run
        self._callTearDown()
        ~~~~~~~~~~~~~~~~~~^^
      File "/usr/lib/python3.13/unittest/case.py", line 611, in _callTearDown
        self.tearDown()
        ~~~~~~~~~~~~~^^
      File "/nvme-cli/tests/nvme_create_max_ns_test.py", line 78, in tearDown
        self.assertEqual(self.create_and_validate_ns(self.default_nsid,
                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
                                                     self.nsze,
                                                     ^^^^^^^^^^
                                                     self.ncap,
                                                     ^^^^^^^^^^
                                                     self.flbas,
                                                     ^^^^^^^^^^^
                                                     self.dps), 0)
                                                     ^^^^^^^^^
      File "/nvme-cli/tests/nvme_test.py", line 449, in create_and_validate_ns
        self.assertEqual(int(json_output['nsid']), nsid,
        ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                         "ERROR : create namespace failed")
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    AssertionError: 3 != 1 : ERROR : create namespace failed
  ...
==============================================================================

[igaw]

Today, a manual run did pass:

https://github.com/linux-nvme/nvme-cli/actions/runs/24238055035

The interesting thing is that this run took around 54 minutes, about the same amount the last good run took. All failing ones are around the 50 minutes mark.

Also the nvme_create_max_ns_test only created 1 namespace, while in the other runs it tried to create more.

The good case:

# Running: /nvme-cli/.build-ci/nvme id-ctrl /dev/nvme1 --output-format=json
# {
#   "vid":7062,
#   "ssvid":7062,
#   "sn":"A053A059            ",
#   "mn":"WUS4BB096D7P3E3                         ",
#   "fr":"R1110200",

The bad case:

# /nvme-cli/.build-ci/nvme id-ctrl /dev/nvme0 --output-format=json
# {
#   "vid":7062,
#   "ssvid":7062,
#   "sn":"205033900221        ",
#   "mn":"WUS4B7696DSP303                         ",
#   "fr":"R611000F",

What has changed since yesterday, is the 'modernized subprocess' patch is in, which makes the input and output handling more consistent. Might have changed the nvme_create_max_ns_test test.

[ikegami-t]

Just confirmed the failure test log below.
https://github.com/linux-nvme/nvme-cli/actions/runs/24298377793/job/70947267237#logs
Artifact download URL: https://github.com/linux-nvme/nvme-cli/actions/runs/24298377793/artifacts/6390326562

artifact-20260412-045933.zip: artifact-20260412-045933/nvmetests/meson-logs/meson-log.txt
meson-log.txt

# ##### Attaching 2
# Running: /nvme-cli/.build-ci/nvme attach-ns /dev/nvme1 --namespace-id=2 --controllers=0 --verbose
# attach-ns: success
# nsid: 2
# 
not ok 1 - test_attach_detach_ns (TestNVMeCreateMaxNS)
# Running: /nvme-cli/.build-ci/nvme create-ns /dev/nvme1 --nsze=14651362 --ncap=14651362 --flbas=0 --dps=0 --verbose --output-format=json
# {
#   "create-ns":"success",
#   "nsid":"3"
# }
# 
not ok 2 - test_attach_detach_ns (TestNVMeCreateMaxNS)
----------------------------------- stderr -----------------------------------
  ---
  traceback: |
    Traceback (most recent call last):
      File "/usr/lib/python3.13/unittest/case.py", line 58, in testPartExecutor
        yield
      File "/usr/lib/python3.13/unittest/case.py", line 651, in run
        self._callTestMethod(testMethod)
        ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
      File "/usr/lib/python3.13/unittest/case.py", line 606, in _callTestMethod
        if method() is not None:
           ~~~~~~^^
      File "/nvme-cli/tests/nvme_create_max_ns_test.py", line 98, in test_attach_detach_ns
        self.assertEqual(self.attach_ns(self.ctrl_id, nsid), 0)
        ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    AssertionError: 1 != 0
  ...
  ---
  traceback: |
    Traceback (most recent call last):
      File "/usr/lib/python3.13/unittest/case.py", line 58, in testPartExecutor
        yield
      File "/usr/lib/python3.13/unittest/case.py", line 654, in run
        self._callTearDown()
        ~~~~~~~~~~~~~~~~~~^^
      File "/usr/lib/python3.13/unittest/case.py", line 611, in _callTearDown
        self.tearDown()
        ~~~~~~~~~~~~~^^
      File "/nvme-cli/tests/nvme_create_max_ns_test.py", line 78, in tearDown
        self.assertEqual(self.create_and_validate_ns(self.default_nsid,
                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
                                                     self.nsze,
                                                     ^^^^^^^^^^
                                                     self.ncap,
                                                     ^^^^^^^^^^
                                                     self.flbas,
                                                     ^^^^^^^^^^^
                                                     self.dps), 0)
                                                     ^^^^^^^^^
      File "/nvme-cli/tests/nvme_test.py", line 410, in create_and_validate_ns
        self.assertEqual(int(json_output['nsid']), nsid,
        ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                         "ERROR : create namespace failed")
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    AssertionError: 3 != 1 : ERROR : create namespace failed

---

There are two errors as not ok for nvme_create_max_ns_test as below.

1. Error 1: NS 2 attach
   The attach itself not faield failed as below.

# attach-ns: success
# nsid: 2

So looks failed to find the block device for 5 seconds after the NS 2 attach. We may avoid the timeout error by trying to test increase the timeout value: 5 seconds to more longer.

    def attach_ns(self, ctrl_id, nsid):
        """ Wrapper for attaching the namespace.
            - Args:
                - ctrl_id : controller id to which namespace to be attached.
                - nsid : new namespace id.
            - Returns:
                - 0 on success, error code on failure.
        """
        attach_ns_cmd = f"{self.nvme_bin} attach-ns {self.ctrl} " + \
            f"--namespace-id={str(nsid)} --controllers={ctrl_id} --verbose"
        err = self.run_cmd(attach_ns_cmd).returncode
        if err != 0:
            return err

        # Try to find block device for 5 seconds
        device_path = f"{self.ctrl}n{str(nsid)}"
        stop_time = time.time() + 5
        while time.time() < stop_time:
            if os.path.exists(device_path) and stat.S_ISBLK(os.stat(device_path).st_mode):
                return 0
            time.sleep(0.1)

        return 1

2. Error 2: NS 3 create
   This seems caused by the NS 2 attach error above since NS 1 and 2 not deleted after the error then the default NS is not crated by the tear down but NS 3 created instead and the error caused.

    def tearDown(self):
        """
        Post Section for TestNVMeAttachDetachNSCmd

            - Create primary namespace.
            - Attach it to controller.
            - Call super class's destructor.
        """
        self.assertEqual(self.create_and_validate_ns(self.default_nsid,
                                                     self.nsze,
                                                     self.ncap,
                                                     self.flbas,
                                                     self.dps), 0)
        self.attach_ns(self.ctrl_id, self.default_nsid)
        super().tearDown()

---

Just FYI: Just searched the closed issues before by attach_ns as below.
https://github.com/linux-nvme/nvme-cli/issues?q=is%3Aissue%20attach_ns

[igaw]

We can try with a bigger timeout, but this whole thing is still a bit fragile, IMO. Maybe we should look into a proper block layer monitoring, e.g. look at udevs event instead and add some proper logging with time stamps. I really like to avoid busy loop polling. That is usually a 'code smell'.

[ikegami-t]

Right and agreed. Will do consider the refactoring later I also. Thank you.

[ikegami-t]

Just created the PR: #3286 to just change the NS attaching timeout value to 10 seconds from 5 sedonds. Still needed more time to fix this by using the udev event so can we fix the issue by the changes at first to make sure the issue cause? Sorry for late to do this.

[igaw]

I don't think it's going to be addressed by extending the busy loop to 10s. The device is idle, so if the namespace is not ready in 5 seconds it wont be ready in 10 seconds.

FWIW, I've reviewed the attach-ns command and nothing obvious broken. Unfortunately, I don't have hardware to test locally.

4173 │ # ##### Testing max_ns: 128
4174 │ # ##### Creating 1
4175 │ # Running: /nvme-cli/.build-ci/nvme create-ns /dev/nvme1 --nsze=14651362 --ncap=14651362 --flbas=0 --dps=0 --verbose --output-format=json
4176 │ # {
4177 │ #   "create-ns":"success",
4178 │ #   "nsid":"1"
4179 │ # }
4180 │ #
4181 │ # Running: /nvme-cli/.build-ci/nvme id-ns /dev/nvme1 --namespace-id=1
4182 │ # NVME Identify Namespace 1:
4183 │ # nsze    : 0
4184 │ # ncap    : 0
4185 │ # nuse    : 0
4186 │ # nsfeat  : 0
4187 │ # nlbaf   : 0
[...]
4218 │ # ##### Attaching 1
4219 │ # Running: /nvme-cli/.build-ci/nvme attach-ns /dev/nvme1 --namespace-id=1 --controllers=0 --verbose
4220 │ # attach-ns: success
4221 │ # nsid: 1
4222 │ #
4223 │ # ##### Running IOs in 1
4224 │ # Running: /nvme-cli/.build-ci/nvme id-ns /dev/nvme1n1 --output-format=json
4225 │ # {
4226 │ #   "nsze":14651368,
4227 │ #   "ncap":14651368,
4228 │ #   "nuse":0,
4229 │ #   "nsfeat":2,
[...]
4299 │ # Running: dd if=/dev/nvme1n1 of=/dev/null bs=512 count=1
4300 │ # 1+0 records in
4301 │ # 1+0 records out
4302 │ # 512 bytes copied, 0.000962724 s, 532 kB/s
4303 │ #
4304 │ # Running: dd if=/dev/zero of=/dev/nvme1n1 bs=512 count=1
4305 │ # 1+0 records in
4306 │ # 1+0 records out
4307 │ # 512 bytes copied, 0.000394388 s, 1.3 MB/s
4308 │ #
4309 │ # ##### Creating 2
4310 │ # Running: /nvme-cli/.build-ci/nvme create-ns /dev/nvme1 --nsze=14651362 --ncap=14651362 --flbas=0 --dps=0 --verbose --output-format=json
4311 │ # {
4312 │ #   "create-ns":"success",
4313 │ #   "nsid":"2"
4314 │ # }
4315 │ #
4316 │ # Running: /nvme-cli/.build-ci/nvme id-ns /dev/nvme1 --namespace-id=2
4317 │ # NVME Identify Namespace 2:
4318 │ # nsze    : 0
4319 │ # ncap    : 0
4320 │ # nuse    : 0
4321 │ # nsfeat  : 0
4322 │ # nlbaf   : 0
[...]
4352 │ #
4353 │ # ##### Attaching 2
4354 │ # Running: /nvme-cli/.build-ci/nvme attach-ns /dev/nvme1 --namespace-id=2 --controllers=0 --verbose
4355 │ # attach-ns: success
4356 │ # nsid: 2
4357 │ #
4358 │ not ok 1 - test_attach_detach_ns (TestNVMeCreateMaxNS)

One problem with the test framework is, it assumes that the new block device uses the namespace id which is used for the create and attach operation. But the kernel does not use this id for creating the block device name. Thus what we should do is install a udev handler wait for new block devices and then do a 'nvme id-ns' on them and compare the nsid.

[igaw]

Another thing we should attach the kernel messages to report.

[ikegami-t]

Just changed the PR: #3286 to only print the elapsed time for attach_ns. Then we can make sure how long time needed to attach the NS.

I don't think it's going to be addressed by extending the busy loop to 10s. The device is idle, so if the namespace is not ready in 5 seconds it wont be ready in 10 seconds.

If so then I think this issue is not caused by attach_ns itself but seems the kernel or drive issue.

[ikegami-t]

One problem with the test framework is, it assumes that the new block device uses the namespace id which is used for the create and attach operation. But the kernel does not use this id for creating the block device name. Thus what we should do is install a udev handler wait for new block devices and then do a 'nvme id-ns' on them and compare the nsid.

Just confirmed the nvme driver implementation then as far as confirmed seems the driver NS info (the block device) depended on the scan_work so looks possible to be delayed. And seems similar issue caused before on the issue: #2668 also.

[ikegami-t]

Another thing we should attach the kernel messages to report.

Seems we can check the block device nsid sysfs attribute value to make sure. Also to do scan_work immidiately seems we can use the sysfs rescan_controller attribute.

[MaisenbacherD]

The nvme_create_max_ns_test test case is failing to create more than one namespace due to the lack of access permissions in the /dev tree within the test container.

PR address this.

[igaw]

yes, this is working again. thanks everyone!