nvme discover over RDMA parses Discovery Log Entry incorrectly on nvme-cli 2.13 / libnvme 1.13, while direct nvme connect works

[iops-hunter]

Environment

Working node:

• Proxmox VE 8.4
• Debian 12 (bookworm)
• nvme-cli 2.4+really2.3-3
• libnvme 1.3-1+deb12u1

Broken node:

• Proxmox VE 9.1
• Debian 13 (trixie)
• nvme-cli 2.13-2
• libnvme 1.13-2

Target:

• Huawei NVMe-oF target
• transport: RDMA / RoCE v2

NIC:

• Mellanox ConnectX-6 Lx

Summary

On the old node (nvme-cli 2.3 / libnvme 1.3), nvme discover returns a correct Discovery Log Entry.

On the new node (nvme-cli 2.13 / libnvme 1.13), the discovery controller connection succeeds, Get Log Page commands succeed, and direct nvme connect to the same subsystem works correctly, but nvme discover prints a bogus Discovery Log Entry:

trtype:  fc
adrfam:
subtype: unrecognized
treq:    not specified
portid:  0
trsvcid:
subnqn:
traddr:
eflags:  none

This looks like the discovery log is successfully retrieved but decoded incorrectly in userspace.

Expected behavior

nvme discover -t rdma -a <target> -s 4420 should return something like:

Discovery Log Number of Records 1, Generation counter 2
=====Discovery Log Entry 0======
trtype:  rdma
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified
portid:  0
trsvcid: 4420
subnqn:  nqn.2020-02.huawei.nvme:nvm-subsystem-sn-[XXX]
traddr:  192.168.10.10
eflags:  none
rdma_prtype: roce-v2
rdma_qptype: connected
rdma_cms:    rdma-cm
rdma_pkey: 0x0000

Actual behavior

Discovery Log Number of Records 1, Generation counter 2
=====Discovery Log Entry 0======
trtype:  fc
adrfam:
subtype: unrecognized
treq:    not specified
portid:  0
trsvcid:
subnqn:
traddr:
eflags:  none

Important observation

Direct connect to the same subsystem works correctly on the same host:

nvme connect -t rdma -n nqn.2020-02.huawei.nvme:nvm-subsystem-sn-[XXX] -a 192.168.10.10 -s 4420

and then nvme list-subsys shows:

nvme-subsys1 - NQN=nqn.2020-02.huawei.nvme:nvm-subsystem-sn-[XXX]
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:[YYY]
\
 +- nvme1 rdma traddr=192.168.10.10,trsvcid=4420 live

So the RDMA transport and controller connection are functional.

Verbose discover output

nvme discover -t rdma -a 192.168.10.10 -s 4420 -vv

connect ctrl, 'nqn=nqn.2014-08.org.nvmexpress.discovery,transport=rdma,traddr=192.168.10.10,trsvcid=4420,
hostnqn=nqn.2014-08.org.nvmexpress:uuid:[YYY],hostid=[YYY],ctrl_loss_tmo=600'
connect ctrl, response 'instance=1,cntlid=0'
nvme1: nqn.2014-08.org.nvmexpress.discovery connected

opcode       : 06
...
data_len     : 00001000
cdw10        : 00000001
...
err          : 0

nvme1: get header (try 0/10)

opcode       : 02
...
data_len     : 00000014
cdw10        : 00040070
...
err          : 0

nvme1: get 1 records (genctr 2)

opcode       : 02
...
data_len     : 00000400
cdw10        : 00ff0070
cdw12        : 00000400
...
err          : 0

nvme1: get header again

opcode       : 02
...
data_len     : 00000014
cdw10        : 00040070
...
err          : 0

Discovery Log Number of Records 1, Generation counter 2
=====Discovery Log Entry 0======
trtype:  fc
adrfam:
subtype: unrecognized
treq:    not specified
portid:  0
trsvcid:
subnqn:
traddr:
eflags:  none

nvme1: nqn.2014-08.org.nvmexpress.discovery disconnected

What was already ruled out

• RDMA stack mismatch was fixed
• nvme_rdma, nvme_fabrics, rdma_cm, ib_core are all now loaded from the same DKMS stack
• direct nvme connect works
• host NQN and host ID are consistent
• the same target works fine on the older node with nvme-cli 2.3 / libnvme 1.3

Version info

Broken node:

nvme version
nvme version 2.13 (git 2.13)
libnvme versio n 1.13 (git 1.13)

dpkg -l | egrep 'nvme-cli|libnvme'
ii  libnvme1t64  1.13-2  amd64  NVMe management library (library)
ii  nvme-cli     2.13-2  amd64  NVMe management tool

Working node:

nvme version
nvme version 2.3 (git 2.3)
libnvme version 1.3 (git 1.3)

dpkg -l | egrep 'nvme-cli|libnvme'
ii  libnvme1   1.3-1+deb12u1     amd64
ii  nvme-cli   2.4+really2.3-3   amd64

Related issues (what I've found so far)

nvme-cli #1555
nvme-cli #2555
libnvme #807

Question

Given the related issues, could this be a regression in discovery log parsing/decoding in newer nvme-cli / libnvme, possibly triggered by a Huawei discovery log format or vendor-specific quirk?
Or something else? What do you think?

Respectfully tagging some of the related people: @igaw @davidrohr @hreinecke

Thanks all for contribution.

[igaw]

This seems to be very similar to #807.

Could you try forcing the discovery log page to be read in 4k chunks to see if that fixes the problem?

LIBNVME_FORCE_4K=1 nvme discover ...

[iops-hunter]

Hi @igaw, thanks for the response.

Unfortunately, I see no change in response:

# LIBNVME_FORCE_4K=1 nvme discover -t rdma -a 192.168.10.10 -s 4420

Discovery Log Number of Records 1, Generation counter 2
=====Discovery Log Entry 0======
trtype:  fc
adrfam:
subtype: unrecognized
treq:    not specified
portid:  0
trsvcid:
subnqn:
traddr:
eflags:  none

That seems to suggest the issue is either a different decoding/parsing problem, or some target-specific discovery log format quirk that newer nvme-cli/libnvme does not currently handle correctly.

Is there, eventually, some another way how to dump discovery response for further analysis?

[iops-hunter]

Hi @igaw,

I made the test even more explicit: I copied the older (aka working) nvme userspace stack from the working node to the broken node, and it works there without any problems:

mkdir -p /root/nvme-test/bin /root/nvme-test/lib # on broken node

cp -a /usr/sbin/nvme /root/nvme-test/bin/ # from working to broken
cp -a /lib/x86_64-linux-gnu/libnvme.so* /root/nvme-test/lib/ # from working to broken
cp -a /lib/x86_64-linux-gnu/libnvme-mi.so* /root/nvme-test/lib/ # from working to broken

continue on broken node only:

chmod +x /root/nvme-test/bin/nvme

LD_LIBRARY_PATH=/root/nvme-test/lib /root/nvme-test/bin/nvme version

nvme version 2.3 (git 2.3)
libnvme version 1.3 (git 1.3)

LD_LIBRARY_PATH=/root/nvme-test/lib /root/nvme-test/bin/nvme discover -t rdma -a 192.168.10.10 -s 4420

Discovery Log Number of Records 1, Generation counter 2
=====Discovery Log Entry 0======
trtype:  rdma
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified
portid:  0
trsvcid: 4420
subnqn:  nqn.2020-02.huawei.nvme:nvm-subsystem-sn-[XXX]
traddr:  192.168.10.10
eflags:  none
rdma_prtype: roce-v2
rdma_qptype: connected
rdma_cms:    rdma-cm
rdma_pkey: 0x0000

So this strongly suggests a regression in newer nvme-cli / libnvme, specifically in discovery log parsing/decoding.

I'll also notify Huawei R&D to be aware of this issue.

[iops-hunter]

I did also a targeted (and GPT-assisted) review of libnvme discovery-related history between the version that works for me (libnvme 1.3) and the version that reproduces the issue (libnvme 1.13). The full range is large (v1.3...v1.13 shows 614 commits), so I filtered specifically for discovery / fabrics.c / discovery-log parsing related changes. 

Based on that review, these are the most suspicious changes, ordered from most likely to least likely.

1. PR fabrics: restructrure nvmf_get_discovery_log #378 / commit ed79826

Title: fabrics: restructrure nvmf_get_discovery_log
Why it is suspicious: this changes the core control flow of nvmf_get_discovery_log(). The PR text explicitly says it moved the initial discovery-log header read into the loop so that genctr and numrec stay in sync across both Get Log Page invocations, and it explicitly references nvme-cli#1555. My failing -vv output shows exactly that path: connect to discovery controller succeeds, header is read, records are read, header is read again, and then the resulting entry is decoded incorrectly. This is the single most suspicious change in the range. 

2. PR python: Use nvmf_get_discovery_wargs() #486 / commit af3241e

Title: python: Use nvmf_get_discovery_wargs()
Why it is suspicious: despite the title, this PR changes discovery behavior by switching to nvmf_get_discovery_wargs(), specifically to support setting the LSP field for TP8010 / PLEO handling, and adds supported_log_pages() checks related to that support. Any change that alters how discovery log pages are requested, especially LSP-related behavior, is a plausible source of interoperability regressions against vendor targets that previously worked with older libnvme. 

3. PR fabrics: Add No CDC Connectivity to EFLAGS #420

Title: fabrics: Add No CDC Connectivity to EFLAGS
Why it is suspicious: this changes interpretation of discovery log entry flags (EFLAGS) and updates the corresponding formatting / reporting path. My symptom is not a transport failure: the discovery controller connects and the log page reads complete successfully, but the entry is decoded into nonsense (trtype: fc, empty traddr, empty subnqn). Any change in the decode / formatting path for discovery entries is therefore relevant. I rank this below #378 and #486 because it looks narrower, but it is still worth reviewing. 

4. Issue Incorrect check for discovery controllers #454

Title: Incorrect check for discovery controllers
Why it is suspicious: this issue points out that after TP8013, discovery controllers can no longer be identified just by matching the well-known discovery NQN, and that there were still multiple places in libnvme doing exactly that, including in src/nvme/fabrics.c. This looks more like controller identification / classification logic than raw discovery entry parsing, so I rank it lower, but it is still in the same codepath and spec area. 

5. Issue "could not alloc memory for discovery log page" and mount failure with Corsair 8TB drive over RDMA nvmet #807

Title: "could not alloc memory for discovery log page" and mount failure with Corsair 8TB drive over RDMA nvmet
Why it is relevant: this is not the same symptom, but it is another case where nvme discover misbehaves while direct nvme connect can still work, and the report explicitly says the problem seems related to the numrec value coming out of discovery-log handling. That makes it useful corroborating evidence that the discovery path in libnvme has had target-specific / RDMA-specific parsing problems. 

Additional context

There is already historical precedent for discovery regressions in this area. nvme-cli#1555 reports nvme discover failures with newer userspace, and PR #378 was merged specifically to address that class of problem. That makes #378 and follow-on discovery-path refactors the most likely suspect range for my regression. 

Why I think this is a userspace parsing regression:

On my setup:
• old userspace (nvme-cli 2.3 / libnvme 1.3) prints the Huawei discovery record correctly,
• new userspace (nvme-cli 2.13 / libnvme 1.13) connects to the discovery controller and successfully issues Get Log Page, but prints the entry as:
• trtype: fc
• empty adrfam
• empty trsvcid
• empty subnqn
• empty traddr
• direct nvme connect -t rdma -n -a -s 4420 works correctly on the same host.

So the transport path is working; the failure appears to be in discovery-log handling / decoding in newer userspace. The most suspicious changes are therefore the discovery-path refactors above, especially PR #378 and PR #486. 

[davidrohr]

For reference, the issue I observed and reported in #807 does not seem a regression in a recent version, it was present in all versions which I tested.
I did not manage to fix it on the NVME side, but it went away when I disabled the iommu on the server side, which is for now a workaround for me.
I would like to mention that for me also, the transport part was working, but the discovery handling failed. However, I had some partitions which I could not mount, and I did not check deeper why that failed, but in that case it could actually have been also a problem on the transport side. I could, however, read the partition with dd, and got the correct content of the disk.

[gridd20-pixel]

I confirm this behaviour on Dorado 5000V6 NVME and Mellanox CX-5.
any versions nvme newer than 2.3 return nvme discover with this odd output

[igaw]

@iops-hunter @gridd20-pixel is your HBA officially supported? I ask because when @davidrohr and I tried to figure out what's happening on his system, the HBA vendor just said, but luck not supported. There is something fishy going with RDMA and nvme-cli just able to trigger this bug.

[iops-hunter]

@igaw Yes, it is supported, Mellanox ConnectX-6 Lx is a mainstream card for RoCE V2 with fully supported lifecycle.

As I already presented in A/B test above, the only real difference is version of NVMe libs.

Based on this, I'm quite sure it's not a problem of HW (and firmware, as I also flashed to the latest image).

[igaw]

Okay, that might come handy. One important information from @davidrohr is disabled the iommu on the server side which seems to help.

@keithbusch I wonder if we run into some IOMMU issue here. IIRC, when allocate memory in userspace and it is properly aligned etc., the nvme subsystem is able to directly use the userland allocated buffer (no double buffering necessary). And there is actually a change how the buffers are allocated between v1.3 and v1.13:

libnvme v1.3:

static struct nvmf_discovery_log *nvme_discovery_log(nvme_ctrl_t c,
						     struct nvme_get_log_args *args,
						     int max_retries)
{
[...]
		size = sizeof(struct nvmf_discovery_log);

		free(log);
		log = calloc(1, size);
[...]

libnvme v1.13:

void *__nvme_alloc(size_t len)
{
	size_t _len = round_up(len, 0x1000);
	void *p;

	if (posix_memalign((void *)&p, getpagesize(), _len))
		return NULL;

	memset(p, 0, _len);
	return p;
}

static struct nvmf_discovery_log *nvme_discovery_log(
	const struct nvme_get_discovery_args *args)
{
[...]
	log = __nvme_alloc(sizeof(*log));
[...]
}

[iops-hunter]

For anyone interested, this is how I solve this bogus discovery in production.

Initial notes:

• nvme connect-all does discovery first, only nvme connect is a direct way to connect known target
• nvme connect-all is automatically called by /etc/systemd/system/default.target.wants/nvmf-autoconnect.service:

[Unit]
Description=Connect NVMe-oF subsystems automatically during boot
ConditionPathExists=|/etc/nvme/config.json
ConditionPathExists=|/etc/nvme/discovery.conf
Wants=modprobe@nvme_fabrics.service
After=modprobe@nvme_fabrics.service
After=network-online.target
Before=remote-fs-pre.target

[Service]
ProtectSystem=full
ProtectHome=true
ProtectHostname=true
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectControlGroups=true
ProtectProc=invisible
RestrictRealtime=true
LockPersonality=yes
MemoryDenyWriteExecute=yes
RemoveIPC=yes
RestrictAddressFamilies=AF_INET AF_INET6
Type=oneshot
ExecStart=/usr/sbin/nvme connect-all --context=autoconnect

[Install]
WantedBy=default.target

• /etc/systemd/system/default.target.wants/nvmf-autoconnect.service symlink is created automatically by apt install nvme-cli

Original content of /etc/nvme/discovery.conf:

# Used for extracting default parameters for discovery
#
# Example:
# --transport=<trtype> --traddr=<traddr> --trsvcid=<trsvcid> --host-traddr=<host-traddr> --host-iface=<host-iface>

--transport rdma --traddr ip-target-A --trsvcid 4420 --ctrl-loss-tmo 3600
--transport rdma --traddr ip-target-B --trsvcid 4420 --ctrl-loss-tmo 3600
--transport rdma --traddr ip-target-C --trsvcid 4420 --ctrl-loss-tmo 3600
--transport rdma --traddr ip-target-D --trsvcid 4420 --ctrl-loss-tmo 3600
--transport rdma --traddr ip-target-E --trsvcid 4420 --ctrl-loss-tmo 3600
--transport rdma --traddr ip-target-F --trsvcid 4420 --ctrl-loss-tmo 3600
--transport rdma --traddr ip-target-G --trsvcid 4420 --ctrl-loss-tmo 3600
--transport rdma --traddr ip-target-H --trsvcid 4420 --ctrl-loss-tmo 3600

As discovery process is broken, I don't use discovery at all (on new proxmox nodes, aka with newer nvme libs).

This means:

• renamed /etc/nvme/discovery.conf to discovery.conf.bak
• created /etc/nvme/config.json with this content:

[
  {
    "hostnqn": "nqn.2014-08.org.nvmexpress:uuid:[my-hostnqn]",
    "hostid": "[my-hostid],
    "subsystems": [
      {
        "nqn": "nqn.2020-02.huawei.nvme:nvm-subsystem-sn-[XXX]",
        "ports": [
          { "transport": "rdma", "traddr": "ip-target-A", "trsvcid": "4420", "ctrl_loss_tmo": 3600 },
          { "transport": "rdma", "traddr": "ip-target-B", "trsvcid": "4420", "ctrl_loss_tmo": 3600 },
          { "transport": "rdma", "traddr": "ip-target-C", "trsvcid": "4420", "ctrl_loss_tmo": 3600 },
          { "transport": "rdma", "traddr": "ip-target-D", "trsvcid": "4420", "ctrl_loss_tmo": 3600 },
          { "transport": "rdma", "traddr": "ip-target-E", "trsvcid": "4420", "ctrl_loss_tmo": 3600 },
          { "transport": "rdma", "traddr": "ip-target-F", "trsvcid": "4420", "ctrl_loss_tmo": 3600 },
          { "transport": "rdma", "traddr": "ip-target-G", "trsvcid": "4420", "ctrl_loss_tmo": 3600 },
          { "transport": "rdma", "traddr": "ip-target-H", "trsvcid": "4420", "ctrl_loss_tmo": 3600 }
        ]
      }
    ]
  }
]

Next, I use modified nvmf-autoconnect.service (but in a clear way), so:

• systemctl edit nvmf-autoconnect.service
• add this content there:

[Service]
ExecStart= # blank to reset previous nvme connect-all
ExecStart=/usr/sbin/nvme connect -J /etc/nvme/config.json -v # direct way to connect without bogus discovery
ExecStartPre=/bin/sh -c "echo -n round-robin > /sys/module/nvme_core/parameters/iopolicy" # not relevant for this issue

After that:

• systemctl daemon-reload
• systemctl restart nvmf-autoconnect.service
• systemctl status nvmf-autoconnect.service

Final check:
systemctl cat nvmf-autoconnect.service:

# /usr/lib/systemd/system/nvmf-autoconnect.service
[Unit]
Description=Connect NVMe-oF subsystems automatically during boot
ConditionPathExists=|/etc/nvme/config.json
ConditionPathExists=|/etc/nvme/discovery.conf
Wants=modprobe@nvme_fabrics.service
After=modprobe@nvme_fabrics.service
After=network-online.target
Before=remote-fs-pre.target

[Service]
ProtectSystem=full
ProtectHome=true
ProtectHostname=true
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectControlGroups=true
ProtectProc=invisible
RestrictRealtime=true
LockPersonality=yes
MemoryDenyWriteExecute=yes
RemoveIPC=yes
RestrictAddressFamilies=AF_INET AF_INET6
Type=oneshot
ExecStart=/usr/sbin/nvme connect-all --context=autoconnect

[Install]
WantedBy=default.target

# /etc/systemd/system/nvmf-autoconnect.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/sbin/nvme connect -J /etc/nvme/config.json
ExecStartPre=/bin/sh -c "echo -n round-robin > /sys/module/nvme_core/parameters/iopolicy"

• as you can see, /etc/systemd/system/nvmf-autoconnect.service.d/override.conf is active and valid

And that's all. From this point all is working as intended, i.e. RoCE LUNs are mounted and no issues so far.

Only problem is - as discovery is broken - you need to know target nqn (here provided in /etc/nvme/config.json).

---

For @igaw @davidrohr @keithbusch: as I'm on AMD EPYC, I just use iommu=pt in /etc/default/grub, so I'm using IOMMU Passthrough, as confirmed by dmesg | grep -i IOMMU:

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.17.13-2-pve root=/dev/mapper/pve-root ro quiet iommu=pt
[    0.526618] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.17.13-2-pve root=/dev/mapper/pve-root ro quiet iommu=pt
[    1.628656] iommu: Default domain type: Passthrough (set via kernel command line)

• so IOMMU shouldn't be active/effective here (and full BIOS IOMMU deactivation is not preferred on my side)

Additional note to system libs and driver versions: I've installed full doca-roce stack with mlnx-nvme-dkms, so I've unified nvme/rdma/mlx5 kernel stack, compiled/deployed from one common source. (DOCA is a new way to install NVIDIA/Mellanox stack, as OFED is deprecated).

For reference: CX-6 Lx firmware 26.48.1000 with DOCA 3.3.0 (doca-host_3.3.0-088000-26.01-debian13_amd64.deb)

[hreinecke]

Okay, that might come handy. One important information from @davidrohr is disabled the iommu on the server side which seems to help.

@keithbusch I wonder if we run into some IOMMU issue here. IIRC, when allocate memory in userspace and it is properly aligned etc., the nvme subsystem is able to directly use the userland allocated buffer (no double buffering necessary). And there is actually a change how the buffers are allocated between v1.3 and v1.13:

libnvme v1.3:

static struct nvmf_discovery_log *nvme_discovery_log(nvme_ctrl_t c,
struct nvme_get_log_args *args,
int max_retries)
{
[...]
size = sizeof(struct nvmf_discovery_log);

  free(log);
  log = calloc(1, size);

[...]

libnvme v1.13:

void *__nvme_alloc(size_t len)
{
size_t _len = round_up(len, 0x1000);
void *p;

if (posix_memalign((void *)&p, getpagesize(), _len))
return NULL;

memset(p, 0, _len);
return p;
}

static struct nvmf_discovery_log *nvme_discovery_log(
const struct nvme_get_discovery_args *args)
{
[...]
log = __nvme_alloc(sizeof(*log));
[...]
}

Ah, that might be it. There is quite a bit of magic involved when doing RDMA to userland memory (page pinning etc), so I wouldn't be surprised if we fall into one of these issues here. But it should be relatively easy to port the memalign trick over to the current version. Plus: I think we need to limit discovery to single pages; I don't think we can force 'malloc' to allocate two consecutive physical pages.

[igaw]

The current version uses memalign which avoids double buffering as far I can tell. My understanding is that with calloc the kernel uses bio_copy_user_iov path. And with posix_memalign the kernel is using bio_map_user_iov:

int blk_rq_map_user_iov(struct request_queue *q, struct request *rq,
			struct rq_map_data *map_data,
			const struct iov_iter *iter, gfp_t gfp_mask)
{
	bool copy = false, map_bvec = false;
	unsigned long align = blk_lim_dma_alignment_and_pad(&q->limits);
	struct bio *bio = NULL;
	struct iov_iter i;
	int ret = -EINVAL;

	if (map_data)
		copy = true;
	else if (iov_iter_alignment(iter) & align)
		copy = true;
	else if (iov_iter_is_bvec(iter))
		map_bvec = true;
	else if (!user_backed_iter(iter))
		copy = true;
	else if (queue_virt_boundary(q))
		copy = queue_virt_boundary(q) & iov_iter_gap_alignment(iter);

	if (map_bvec) {
		ret = blk_rq_map_user_bvec(rq, iter);
		if (!ret)
			return 0;
		if (ret != -EREMOTEIO)
			goto fail;
		/* fall back to copying the data on limits mismatches */
		copy = true;
	}

	i = *iter;
	do {
		if (copy)
			ret = bio_copy_user_iov(rq, map_data, &i, gfp_mask);
		else
			ret = bio_map_user_iov(rq, &i, gfp_mask);
		if (ret) {
			if (ret == -EREMOTEIO)
				ret = -EINVAL;
			goto unmap_rq;
		}
		if (!bio)
			bio = rq->bio;
	} while (iov_iter_count(&i));

	return 0;

unmap_rq:
	blk_rq_unmap_user(bio);
fail:
	rq->bio = NULL;
	return ret;
}

[igaw]

@iops-hunter could you do an experiment? That means are you able to hack on the code base? I'd like to see if it would work again, if we use calloc instead posix_memalign.

diff --git a/libnvme/src/nvme/fabrics.c b/libnvme/src/nvme/fabrics.c
index 129aa134d33a..e833bc689c13 100644
--- a/libnvme/src/nvme/fabrics.c
+++ b/libnvme/src/nvme/fabrics.c
@@ -1242,7 +1242,7 @@ static int nvme_discovery_log(const struct nvme_get_discovery_args *args,
        struct nvme_transport_handle *hdl = nvme_ctrl_get_transport_handle(args->c);
        struct nvme_passthru_cmd cmd;

-       log = __nvme_alloc(sizeof(*log));
+       log = calloc(1, sizeof(*log));
        if (!log) {
                nvme_msg(ctx, LOG_ERR,
                         "could not allocate memory for discovery log header\n");

[iops-hunter]

@igaw I'm not sure if I did something meaningful, but here are the steps:

• test 1: compiled current master of libnvme and exported libnvme.so and libnvme-mi.so (with symlinks, 6 files total) to dedicated directory
• test 2: compiled patched master of libnvme and exported libnvme.so and libnvme-mi.so (with symlinks, 6 files total) to dedicated directory, only libnvme.so was changed

Patched file was fabrics.c, only changed was one line (1153) from log = __nvme_alloc(sizeof(*log)); to log = calloc(1, sizeof(*log));

Source: https://github.com/linux-nvme/libnvme/blob/master/src/nvme/fabrics.c

Next tried to use compiled/modified libs:

~# LD_LIBRARY_PATH=/root/debug/libnvme/build-1-head/ ldd /usr/sbin/nvme | grep libnvme
        libnvme.so.1 => /root/debug/libnvme/build-1-head/libnvme.so.1 (0x0000732f487a1000)
        libnvme-mi.so.1 => /root/debug/libnvme/build-1-head/libnvme-mi.so.1 (0x0000732f48796000)

~# LD_LIBRARY_PATH=/root/debug/libnvme/build-1-head/ /usr/sbin/nvme discover -t rdma -a 192.168.10.10 -s 4420 -v
nvme1: nqn.2014-08.org.nvmexpress.discovery connected

Discovery Log Number of Records 1, Generation counter 2
=====Discovery Log Entry 0======
trtype:  fc
adrfam:
subtype: unrecognized
treq:    not specified
portid:  0
trsvcid:
subnqn:
traddr:
eflags:  none
nvme1: nqn.2014-08.org.nvmexpress.discovery disconnected

~# LD_LIBRARY_PATH=/root/debug/libnvme/build-2-patch/ ldd /usr/sbin/nvme | grep libnvme
        libnvme.so.1 => /root/debug/libnvme/build-2-patch/libnvme.so.1 (0x00007e771910e000)
        libnvme-mi.so.1 => /root/debug/libnvme/build-2-patch/libnvme-mi.so.1 (0x00007e7719103000)

~# LD_LIBRARY_PATH=/root/debug/libnvme/build-2-patch/ /usr/sbin/nvme discover -t rdma -a 192.168.10.10 -s 4420 -v
nvme1: nqn.2014-08.org.nvmexpress.discovery connected

Discovery Log Number of Records 1, Generation counter 2
=====Discovery Log Entry 0======
trtype:  fc
adrfam:
subtype: unrecognized
treq:    not specified
portid:  0
trsvcid:
subnqn:
traddr:
eflags:  none
nvme1: nqn.2014-08.org.nvmexpress.discovery disconnected

So still the same bogus output.