"could not alloc memory for discovery log page" and mount failure with Corsair 8TB drive over RDMA nvmet

[davidrohr]

I am posting here as libnvme issue, but I am not really sure if the problem is in libnvme, nvme-cli, the kernel or somewhere else.

I have 2 computers, pc1 and pc2, connected via ConnectX3 Infiniband adapters, with pc1 mounting 3 remote nvme SSDs from pc2: a samsung 990 Pro 2TB and 2 Corsair MP600 Pro 8TB.

I am having 2 problems:

• In the nvme discover nvme discover -t rdma -a [IP] -s 4420, I get the error:

could not alloc memory for discovery log page
failed to get discovery log: Cannot allocate memory

• I can still connect, via nvme connect -n [SUBSYS] -t rdma -a [IP] -s 4420, and then the Samsung 2TB SDD seem to work normally, but I cannot mount a file system from the Corsair SSDs (even though I see them normally in /proc/partitions`).

I tried the following things already:

• When I switch everything to TCP via IPoIB, and I just use tcp type instead of rdma in all nvme commands, everything works.
• I have another pc pc3 with 2 Samsung NVME drives and the same ConnectX3 InfiniBand card. Mounting these 2 SSDs from pc1 via RDMA works without a problem, and I also do not get the error when doing nvme discover.
• I tried with linux kernels 6.6, 6.7, 6.8, and nvme-cli and libnvme 1.7, 1.8, and the latest master version - all behave the same.
• For the nvme discovery error: I debugged what happens in libnvme when I get the could not alloc memory for discovery log page, and it seems the numrec value here

  libnvme/src/nvme/fabrics.c

  Line 1089 in 691f809

  numrec = le64_to_cpu(log->numrec);

  is bogus, thus libnvme tries to allocate a very large amount of memory and fails.
• For the mount failure: In the kernel log, when trying to mount the EFI FAT partition on the disk, I am getting FAT-fs (nvme1n1p1): bogus number of reserved sectors and from the mount command: wrong fs type, bad option, bad superblock on ..., missing codepage or helper program, or other error. Mounting the very same file system when connecting via tcp instead of rdma works correctly. I wanted to check if I get corrupted data from this disk via rdma, so I dumped the whole filesystem to an external disk with dd, and compared what I read when connecting with rdma and with tcp, and it is fully identical. So the data I am getting from the disk is correct, mounting fails for a different reason. Sector size and disk size I get when opening the disk with gdisk are also identical.

So in summary, everything works when connecting via tcp, but when connecting via rdma the nvme discovery fails since the numrec entry is bogus, and with rdma I can mount filesystems from the 2TB samsung SSD correctly, but filesystems from the 8TB corsair SSDs fail to mount, despite the data I am reading from the disk with dd is fully correct.

I would be thankful for any advise, or recommendation where to ask. I you'd need me to conduct any tests, please let me know.

[martin-belanger]

Not sure this will make any difference for RDMA. I'm more familiar with TCP.

For TCP, the Service ID used for Discovery is typically 8009. And the Service ID for I/O is 4420. You could try nvme discover -t rdma -a [IP] -s 8009 to see if that makes a difference.

[igaw]

This looks like the RMDA transport is having an issue. libnvme/nvme-cli doesn't differentiate between the transports. Also the nvme subsystem core doesn't differentiate between the transport.

What you could do is to dump the discovery log page for TCP and RDMA (nvme get-log). This would give us a clue.
Also try to add '-vvv' when doing the nvme discover command. This also sheds some light on nvme_discovery_log

FWIW, Linux's nvmet implementation (aka soft target) doesn't implement the discovery controller on port 8009.

[davidrohr]

For the service ID: I am explicity setting this when I set up the nvmet on the server. I have changed it to 8009, but it doesn't make any difference. Also, I am quite sure it connects to the right service ID, since when the ID differs between client and server, nvme discover / connect complain that they cannot connect.

I ran nvme discover -vvv [...]. The logs are attached, but don't tell anything to me.
logs.tar.gz

Could you be more specific which log page to obtain with nvme get-log, i.e. which log-id and how many bytes. I could not find any information which one is the discovery log.

[igaw]

The logs show that the initial get header already is corrupted for RDMA. So it's not that the loop is going wrong.

TCP:

nvme3: get header (try 0/10)
nvme3: get 4 records (genctr 9)
nvme3: get header again

RDMA:

nvme3: get header (try 0/10)
could not alloc memory for discovery log page
failed to get discovery log: Cannot allocate memory

I just realized we need a block device in order to run the get-log command. So you need first to connected to the discovery controller and then do a nvme get-log:

# nvme connect -t rdma -a XXX -s 4420 -n nqn.2014-08.org.nvmexpress.discovery
[ get the nvme decvice from dmesg, it wont show up in nvme list]
# nvme get-log /dev/nvme$X -i 0x70 -l 4096

[davidrohr]

The logs show that the initial get header already is corrupted for RDMA. So it's not that the loop is going wrong.

Yes, as I wrote in the original post, the numrec field in the header has a bogus value. Thus it computes a very large size to allocate for the log page, and fails the allocation.

# nvme get-log /dev/nvme$X -i 0x70 -l 4096

thx, the 0x70 was what I was missing. Will get that tonight connecting to the server both via TCP and via RDMA, and will post the logs here.

Thx for the quick support!

[igaw]

I was just making sure the loop works. We had some bugs in this loop thus I am a bit paranoid.

FWIW, in the base spec the output is specified in 5.16.1.23 Discovery Log Page (Log Page Identifier 70h)

[davidrohr]

I cannot get a log for id 70h / 112:

~ # nvme get-log /dev/nvme0n1 --log-id=112 --log-len 4096
NVMe status: Invalid Field in Command: A reserved coded value or an unsupported value in a defined field(0x6002)
~ # nvme get-log /dev/nvme0n1 --log-id=70h --log-len 4096
NVMe status: Invalid Field in Command: A reserved coded value or an unsupported value in a defined field(0x6002)

While e.g. obtaining the log for id 2 (smart) works, so the nvme get-log is working in general, just not for id 70h.

[igaw]

Are you sure you connected to the discovery controller. A discovery controller doesn't have name spaces. So it's important to use the correct NQN: -n nqn.2014-08.org.nvmexpress.discovery

[hreinecke]

The header is not necessarily corrupted. We never evaluate the CQE status when calling nvme_get_log_page(); it surely will return an errno when the ioctl fails, but unfortunately an ioctl success just means that we got an CQE correctly, not that the CQE does not have an error status ...

[hreinecke]

Check if #810 helps here.

[davidrohr]

-n nqn.2014-08.org.nvmexpress.discovery

I am not fully sure what to do :(:

~ # nvme list
Node                  Generic               SN                   Model                                    Namespace  Usage                      Format           FW Rev  
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
/dev/nvme0n1          /dev/ng0n1            6722de001c9e5e3af9e8 Linux                                    0x1          8.00  TB /   8.00  TB    512   B +  0 B   6.8.2-ge
~ # nvme get-log /dev/nvme0n1 --log-id=70h --log-len=512 -n 1
NVMe status: Invalid Field in Command: A reserved coded value or an unsupported value in a defined field(0x6002)
~ # nvme get-log /dev/nvme0n1 --log-id=70h --log-len=512 -n nqn.2014-08.org.nvmexpress.discovery
Expected word argument for 'namespace-id' but got 'nqn.2014-08.org.nvmexpress.discovery'!
~ # nvme get-log /dev/nvme0n1 --log-id=70h --log-len=512
NVMe status: Invalid Field in Command: A reserved coded value or an unsupported value in a defined field(0x6002)

I have done nvme connect, and the device is there. I can read it with dd, but mounting works only when I connect via tcp, not rdma. Where shall I use -n nqn.2014-08.org.nvmexpress.discovery? As you see above, nvme get-log doesn't accept that option, it expects a namespace. If I put in the namespace 1 of the device, it still doesn't work.

[hreinecke]

'discovery' works without a controller connected.
You could try with

nvme discover -t rdma -a [IP] -s 4420

[davidrohr]

'discovery' works without a controller connected. You could try with

nvme discover -t rdma -a [IP] -s 4420

But this is what I did before, see the logs here: #807 (comment)
It works if I connect over tcp, but over rdma it fails with

could not alloc memory for discovery log page
failed to get discovery log: Cannot allocate memory

[davidrohr]

Check if #810 helps here.

I tried with this PR on top of master, but I am still getting:

~ # nvme discover -t rdma -a 192.168.110.9 -s 4421
could not alloc memory for discovery log page
failed to get discovery log: Cannot allocate memory

I deployed the new version only on the client so far, do I also need to update it on the server?

[hreinecke]

Can you try with this patch:

diff --git a/src/nvme/fabrics.c b/src/nvme/fabrics.c
index 96ba9d4b..44a8ac62 100644
--- a/src/nvme/fabrics.c
+++ b/src/nvme/fabrics.c
@@ -1106,7 +1106,7 @@ retry_header:
                log = __nvme_alloc(sizeof(*log) + entries_size);
                if (!log) {
                        nvme_msg(r, LOG_ERR,
-                                "could not alloc memory for discovery log page\n");
+                                "could not alloc %" PRIu64 " discovery log records\n", numrec);
                        errno = ENOMEM;
                        return NULL;
                }