libnvme segfault during blktests nvme/003 fc transport

[kawasaki]

When I tested the latest libnvme and nvme-cli, I observed blktests test case nvme/003 failed for fc transport.

nvme/003 (tr=fc) (test if we're sending keep-alives to a discovery controller) [failed]
    runtime  10.435s  ...  10.385s
    --- tests/nvme/003.out      2024-05-09 16:48:17.745065811 +0900
    +++ /home/shin/Blktests/blktests/results/nodev_tr_fc/nvme/003.out.bad       2026-02-20 16:13:48.546035453 +0900
    @@ -1,3 +1,3 @@
     Running nvme/003
    -disconnected 1 controller(s)
    +disconnected 2 controller(s)
     Test complete

dmesg

[89463.214967][T218430] run blktests nvme/003 at 2026-02-20 16:18:39
[89463.326168][T218477] loop0: detected capacity change from 0 to 2097152
[89463.342311][T218480] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
[89463.366460][T218484] nvmet_tcp: enabling port 0 (127.0.0.1:4420)
[89463.409843][T211598] nvmet: Created discovery controller 1 for subsystem nqn.2014-08.org.nvmexpress.discovery for NQN nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349.
[89463.420179][T218488] nvme nvme5: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 127.0.0.1:4420, hostnqn: nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
[89473.578561][T218505] nvme nvme5: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[89489.442516][T218622] run blktests nvme/003 at 2026-02-20 16:19:05
[89489.535263][T218670] loop0: detected capacity change from 0 to 2097152
[89489.554982][T218673] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
[89489.625862][T190232] nvme nvme5: NVME-FC{0}: create association : host wwpn 0x20001100aa000001  rport wwpn 0x20001100ab000001: NQN "nqn.2014-08.org.nvmexpress.discovery"
[89489.629893][T213747] nvmet_fc: {0:0}: Association created
[89489.631665][T195922] nvmet: Created discovery controller 1 for subsystem nqn.2014-08.org.nvmexpress.discovery for NQN nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349.
[89489.636739][T190232] nvme nvme5: NVME-FC{0}: controller connect complete
[89489.638590][T218694] nvme nvme5: NVME-FC{0}: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", hostnqn: nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
[89489.696322][T192710] nvme nvme6: NVME-FC{1}: create association : host wwpn 0x20001100aa000001  rport wwpn 0x20001100ab000001: NQN "nqn.2014-08.org.nvmexpress.discovery"
[89489.702879][T213747] nvmet_fc: {0:1}: Association created
[89489.705234][T190232] nvmet: Created discovery controller 2 for subsystem nqn.2014-08.org.nvmexpress.discovery for NQN nqn.2014-08.org.nvmexpress:uuid:81edead3-7664-4eda-801e-a1b6966e8e9d.
[89489.710923][T192710] nvme nvme6: NVME-FC{1}: controller connect complete
[89489.713167][T218696] nvme nvme6: NVME-FC{1}: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", hostnqn: nqn.2014-08.org.nvmexpress:uuid:81edead3-7664-4eda-801e-a1b6966e8e9d
[89489.717846][T218696] nvme[218696]: segfault at 30 ip 00007f8e3e506729 sp 00007fff9ee2fb90 error 4 in libnvme.so.1.16.0[13729,7f8e3e4f3000+1a000] likely on CPU 3 (core 3, socket 0)
[89489.721180][T218696] Code: f0 00 00 00 48 c7 83 e0 00 00 00 00 00 00 00 e8 2d d5 fe ff 48 c7 83 f0 00 00 00 00 00 00 00 5b c3 55 48 89 fd 53 48 83 ec 18 <48> 8b 5f 30 48 85 db 74 0d 48 8b 5b 30 48 85 db 74 04 48 8b 5b 20
[89499.742717][T218729] nvme nvme5: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[89499.765065][T218729] nvme nvme6: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[89499.768158][T190232] nvmet_fc: {0:0}: Association deleted
[89499.777287][T190232] nvmet_fc: {0:0}: Association freed
[89499.780138][T192710] nvmet_fc: {0:1}: Association deleted
[89499.780160][T211103] (NULL device *): Disconnect LS failed: No Association
[89499.789536][T192710] nvmet_fc: {0:1}: Association freed
[89499.790864][T212241] (NULL device *): Disconnect LS failed: No Association
[89499.871937][T218742] nvme_fc: nvme_fc_create_ctrl: nn-0x10001100ab000001:pn-0x20001100ab000001 - nn-0x10001100aa000001:pn-0x20001100aa000001 combination not found

The failure was not observed with older libnvme/nvme-cli (1.15/2.15). I tried to bisect pinning libnvme version to the git hash d65b44cd. The failure was observed with nvme-cli git hash dcf6799, and was not observed with git hash 149aeb0. There are 6 commits between these two commits, but it looks like nvme-cli build fails for these commits.

I guess and hope that libnvme/nvme-cli experts' knowledge to resolve this failure quicker than my effort.

[ikegami-t]

The changes between the commits dcf6799 and 149aeb0 seem not related to the failure behavior.

58bfcb60d ioctl: rework nvme_set_features commands

dcf6799ee nvme: nvme_get_log use nvme_passthru_cmd directly /* "the failure observed" */
78c44c5bc ioctl: add nvme_init_log_lpo helper
dff48ff95 ioctl: introduce get log helper functions
950dc09db ioctl: rework nvme_get_log commands
c1e7a4cc6 ioctl: introduce nvme_identify helper functions
e2cc8be73 src: rework nvme_identify commands
4561bb87c ioctl: rework nvme_identify commands

149aeb0fc src: rename struct command *cmd to acmd /* "not observed" */
6ea76b6ef test: fix ioctl/logs.c -Wint-in-bool-context warnings
9b34209f1 ioctl: remove struct args backwards compatibility
a3adf2a96 fabrics: fix nvme_get_adrfam function description parameter ctrl as c
a27a08bdf mi: remove nvme_mi_admin_identify{_partial}
6c0c8c229 ioctl: add nvme_identify_partial
4fb93ad2d nvme-cli: update error handling
f4c6eee29 rrc: return error codes directly
b6d4f1557 nvme-cli: update to libnvme API changes
ac68c9124 mi: replace dedicated MI API with generic one
870e4cd79 ioctl: add alignment and length check to nvme_fw_download
3acfeaa78 ioctl: add support for MI passthru
047784862 linux: add nvme_transport_handle_is_{direct|mi}
836cc566e linux: add MI support to nvme_{open|close}
23424dd80 mi: replace nvme_mi_ctrl_t with struct nvme_transport_handle
5917db735 linux: add nvme_transport_handle_is_{blk|char}dev
5efc019ea linux: add nvme_transport_handle_get_name
0bd541250 linux: add nvme_transport_handle_get_fd
ec7b4626f api-types: remove unused fd members
4f1574faf ioctl: fix get_log_partial to use uring for link type direct
7d0cc6ebc src: introduce transport handle abstraction
074d70cc8 api-types: replace nvme_root_t with struct nvme_global_ctx
e62dacaef src: merge libnvme-mi into libnvme
d52131860 ioctl: remove deprecated functions
77b6d0c72 map: merge all symbols into 2.0 section
dba98408c build: disable force-fallback-for libnvme
65037520d build: add native compiler to cross compiler config
9ed7ba789 libnvme: hookup the library into the build
89646d01b lib: import libnvme sources
faf7326a2 (tag: v2.16, origin/nvme-cli-2.x) Release v2.16

By the way the segfault caused in libnvme.so.1.16.0 but seems libnvme.so.1.16.0 not used by the commit: dcf6799 so seems also as a strange behavior.

[89489.717846][T218696] nvme[218696]: segfault at 30 ip 00007f8e3e506729 sp 00007fff9ee2fb90 error 4 in libnvme.so.1.16.0[13729,7f8e3e4f3000+1a000] likely on CPU 3 (core 3, socket 0)

[igaw]

The ctrl pointer is NULL.

0x00007ffff7fa4714 in nvme_disconnect_ctrl (c=c@entry=0x0) at ../libnvme/src/nvme/tree.c:1375
1375            struct nvme_global_ctx *ctx = c->s && c->s->h ? c->s->h->ctx : NULL;
(gdb) bt
#0  0x00007ffff7fa4714 in nvme_disconnect_ctrl (c=c@entry=0x0) at ../libnvme/src/nvme/tree.c:1375
#1  0x00007ffff7f96917 in _nvmf_discovery (ctx=ctx@entry=0x5555556ddb40, fctx=fctx@entry=0x5555556df410, connect=connect@entry=true, c=0x5555556df4e0)
    at ../libnvme/src/nvme/fabrics.c:2260
#2  0x00007ffff7f9918c in nvmf_discovery (ctx=ctx@entry=0x5555556ddb40, fctx=fctx@entry=0x5555556df410, connect=connect@entry=true, force=<optimized out>)
    at ../libnvme/src/nvme/fabrics.c:3370
#3  0x0000555555571a82 in fabrics_discovery (desc=<optimized out>, argc=<optimized out>, argv=<optimized out>, connect=<optimized out>) at ../fabrics.c:574
#4  0x00005555555b5290 in handle_plugin (argc=argc@entry=7, argv=argv@entry=0x7fffffffea60, plugin=0x5555556caea0 <builtin>) at ../plugin.c:190
#5  0x000055555556f426 in main (argc=8, argv=0x7fffffffea58) at ../nvme.c:11292

[igaw]

With the fix from #3108

nvme/003 (tr=fc) (test if we're sending keep-alives to a discovery controller) [passed]
    runtime  11.241s  ...  11.178s

[  269.931625] run blktests nvme/003 at 2026-02-23 08:44:11
[  270.083166] loop0: detected capacity change from 0 to 2097152
[  270.153758] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
[  270.482179] nvme nvme1: NVME-FC{0}: create association : host wwpn 0x20001100aa000001  rport wwpn 0x20001100ab000001: NQN "nqn.2014-08.org.nvmexpress.discovery"
[  270.487112] nvmet_fc: {0:0}: Association created
[  270.488386] nvmet: Created discovery controller 1 for subsystem nqn.2014-08.org.nvmexpress.discovery for NQN nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349.
[  270.494073] nvme nvme1: NVME-FC{0}: controller connect complete
[  270.494397] nvme nvme1: NVME-FC{0}: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", hostnqn: nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
[  280.867748] nvme nvme1: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[  280.878834] nvmet_fc: {0:0}: Association deleted
[  280.885556] nvmet_fc: {0:0}: Association freed
[  280.885930] (NULL device *): Disconnect LS failed: No Association

[kawasaki]

Thank you for the quick fix!

[igaw]

Thanks for the report! Really appreciated. It's good to know that the heavy refactoring not all is broken. I am sure there a few more bugs from the refactoring but at least blktests is working.