[Debian 13] Mounted NVME over TCP Share Locks System on Reboot or Shutdown: Error -101

[johntdavis84]

Hello,

I'm running Debian 13 inside a Proxmox 9 VM. I'm using nvme-cli to mount an NVME-over-TCP share from my NAS for persistent storage. The mounted NVME-over-TCP share appears rock solid (it even survives my NAS rebooting with no issues, at least so far).

However, I've hit a snag. The VM locks up when I need to shut down or reboot the VM, with a -101 error.

**Untested Suspicion:** I believe that Debian 13 is shutting down the network interfaces before `nvme-cli` can gracefully unmount the NVME-over-TCP share, and `nvme-cli`'s tools don't understand what happened, so the software gets stuck in an infinite loop waiting for the NVME-over-TCP share to answer back so it can unmount it.

I've had to manually stop the VM when this happens. The NVME is formated as ext4 and lives inside a zVol on TrueNAS, so these hard reboots don't seem to have damaged it or cost me any data, since the system is already in the middle of shutting down when they happen.

Diagnostic Info: Please let me know what else would be useful.

johntdavis@flynn:~$ uname -a
Linux flynn 6.12.74+deb13+1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.74-2 (2026-03-08) x86_64 GNU/Linux
johntdavis@flynn:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 13 (trixie)
Release:	13
Codename:	trixie
johntdavis@flynn:~$ apt info nvme-cli
Package: nvme-cli
Version: 2.13-2

Diagnostic Info: VM Network Configuration: The NVME-over-TCP traffic is going over my storage VLAN.

root@flynn:/etc/network# batcat interfaces
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: interfaces
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ # This file describes the network interfaces available on your system
   2   │ # and how to activate them. For more information, see interfaces(5).
   3   │ 
   4   │ source /etc/network/interfaces.d/*
   5   │ 
   6   │ # The loopback network interface
   7   │ auto lo
   8   │ iface lo inet loopback
   9   │ 
  10   │ # The primary network interface
  11   │ # Changed from "allow-hotplug"; we want to require this at boot to make sure SSH, etc. don't get grumpy.
  12   │ auto ens18
  13   │ iface ens18 inet dhcp
  14   │ 
  15   │ # VLAN 200 - Storage Network
  16   │ auto ens19
  17   │ iface ens19 inet dhcp
───────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────

ETA: discovery.conf

# Used for extracting default parameters for discovery
#
# Example:
# --transport=<trtype> --traddr=<traddr> --trsvcid=<trsvcid> --host-traddr=<host-traddr> --host-iface=<host-iface>
--transport=tcp --traddr=$ADDR --trsvcid=$PORT --nqn=$NQN.nvmexpress.discovery

[johntdavis84]

Looks like this is related to at least:

• This Pull Request: Allow service to also disconnect at shutdown #3066
• This issue re: auto-disconnecting non-boot NVME-oF devices: Prevent NVMe-oF boot devices from being removed by disconnect-all #2913

Just from skimming those, this looks like a symptom of some other things that might need serious refactoring, so I'm not expecting this to be a quickly patched thing.

In the meantime, if it is going to take some time to address this behavior, is there a recommended way to force Debian (or any other SystemD-based distro, I suppose), to unmount the NVME share before the network goes down? Not being able to reboot or restart a VM set to autoboot when the hypervisor comes online without manual intervention is a bit of a problem.

[igaw]

nvme-cli's tools don't understand what happened

As you mentioned, it really comes down to how the system is configured and how it orchestrates the shutdown sequence. In this case, the proper order during shutdown should be to unmount the disk first, then run nvme disconnect, and only after that shut down the network.

This isn’t something nvme-cli can address on its own. It’s a low-level administrative tool that simply carries out the commands it’s given.

Did you tried something like:

[Unit]
After=network-online.target
Before=network.target shutdown.target
Wants=network-online.target

[johntdavis84]

nvme-cli's tools don't understand what happened

As you mentioned, it really comes down to how the system is configured and how it orchestrates the shutdown sequence. In this case, the proper order during shutdown should be to unmount the disk first, then run nvme disconnect, and only after that shut down the network.

This isn’t something nvme-cli can address on its own. It’s a low-level administrative tool that simply carries out the commands it’s given.

Did you tried something like:

[Unit]
After=network-online.target
Before=network.target shutdown.target
Wants=network-online.target

Thanks. :) I'm really not familiar enough with how SystemD works to have tried writing my own shutdown commands. I'll have to look into that.

This seems to be a problem specific to Debian 13. I might try opening a bug report with Debian, as well. This seems like something their package maintainer should be aware of, as it's fairly easily reproducible and I've seen multiple people discussing it.

[igaw]

The core issue is that system integration for the “mounting a disk via NVMe-oF” use case isn’t very smooth right now. In short, the shutdown sequence doesn’t work properly out of the box.

If all dependencies were modeled correctly in systemd, this should work automatically. At the moment, though, admins often need to handle parts of it manually, since there are still some technical challenges to solve before this can be made generic.