NVMe-oF Setup Guide for Proxmox VE

Discovery

NVMe-oF TCP targets can now be discovered as follows:

nvme discover --transport=tcp --traddr=<IP-STORAGE>

Depending on the implementation of the NVMe-oF protocol on the SAN, this command returns all available controllers and IP addresses on the SAN. In some cases, however, you will only receive a log entry from the IP address on which the request was made. In most cases, though, additional IP addresses (e.g., in high-availability multi-controller systems) are available for connection.

An output listing all endpoints looks like this, for example:

root@pve-01:~# nvme discover --transport=tcp --traddr=192.168.50.10 --trsvcid=4420

Discovery Log Number of Records 2, Generation counter 7
=====Discovery Log Entry 0======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified, sq flow control disable supported
portid:  1
trsvcid: 4420
subnqn:  [FAKE]
traddr:  192.168.50.10
eflags:  none
sectype: none
=====Discovery Log Entry 1======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified, sq flow control disable supported
portid:  2
trsvcid: 4420
subnqn:  [FAKE]
traddr:  192.168.51.10
eflags:  none
sectype: none
=====Discovery Log Entry 2======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified, sq flow control disable supported
portid:  2
trsvcid: 4420
subnqn:  [FAKE]
traddr:  192.168.50.11
eflags:  none
sectype: none
=====Discovery Log Entry 3======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified, sq flow control disable supported
portid:  2
trsvcid: 4420
subnqn:  [FAKE]
traddr:  192.168.51.11
eflags:  none
sectype: none

Checking Targets

The targets/paths can be listed using nvme list-subsys -v, along with the I/O policy.

However, the ANA states (comparable to ALUA) are not displayed.

These can be viewed using nvme list-subsys -o json /dev/nvme0n1 (e.g., optimized / non-optimized).

Automatic Setup at Startup

Connecting Targets

Connections established using nvme connect or nvme connect-all do not survive a reboot. To automatically establish connections at boot time, the discovery parameters are specified in the file /etc/nvme/discovery.conf, with one entry per discovery portal:

--transport=tcp --traddr=192.168.50.10 --trsvcid=4420 [--ctrl-loss-tmo=-1 --keep-alive-tmo=10] 
--transport=tcp --traddr=192.168.51.10 --trsvcid=4420 [--ctrl-loss-tmo=-1 --keep-alive-tmo=10] 
--transport=tcp --traddr=192.168.50.11 --trsvcid=4420 [--ctrl-loss-tmo=-1 --keep-alive-tmo=10] 
--transport=tcp --traddr=192.168.51.11 --trsvcid=4420 [--ctrl-loss-tmo=-1 --keep-alive-tmo=10]

If necessary, please verify that the configuration in [] matches the optimized multipathing settings.

At system startup, the systemd service nvmf-autoconnect.service effectively executes an nvme connect-all command against all entries stored there, thereby automatically re-establishing all paths.

The service must be enabled:

systemctl enable nvmf-autoconnect.service

You can check the status with:

systemctl is-enabled nvmf-autoconnect.service

Change the I/O Policy

By default, the NVMe-oF targets are set to the numa I/O policy. To automatically change this to, for example, Round-Robin at startup, a udev rule is required.

The following I/O policies are available:

At runtime (not persistent across reboots):

echo “round-robin” > /sys/class/nvme-subsystem/nvme-subsys0/iopolicy #Adjust subsys0 accordingly

To ensure the policy is set automatically at startup, the udev rule /etc/udev/rules.d/71-nvme-io-policy.rules is required:

ACTION==“add|change”, SUBSYSTEM==“nvme-subsystem”, ATTR{iopolicy}=“round-robin”

Enable and verify at runtime:

udevadm control --reload-rules
udevadm trigger --subsystem-match=nvme-subsystem
cat /sys/class/nvme-subsystem/nvme-subsys*/iopolicy

Missing Targets After Boot

Persistent configuration is missing

Please take “Automatic Setup at Startup” (see above) into account

Network connection became available too late

Although the nvmf-autoconnect.service does not start until after network-online.target, this target is considered reached as soon as ifupdown2 has configured the interfaces—not when the network actually begins transmitting packets.

LACP negotiation on the bond or STP listening/learning on the switch port (up to 30 seconds without Edge Port/PortFast configuration) may still be in progress at this point. The one-time nvme connect-all command then runs during this time window: TCP connects to portals that are not yet reachable fail and are not retried.

Typical symptom: After boot, individual paths are missing, but a manual connect subsequently works without any problems. --ctrl-loss-tmo=-1 does not help here, since the parameter only protects existing connections, and a controller that never established a connection cannot reconnect.

Diagnosis

Check the timing relationship between the network startup and connect errors:

journalctl -b -u networking.service -u nvmf-autoconnect.service -o short-precise

If the connect failed messages occur only milliseconds to a few seconds after the completion of networking.service, the race condition is confirmed.

Possible Solution (Static): Start Delay via systemd Override

Instead of editing the unit file directly (which would be overwritten during package updates), create a drop-in file:

systemctl edit nvmf-autoconnect.service

Content:

[Service]
ExecStartPre=/bin/sleep 15

This causes the service to wait 15 seconds before running nvme connect-all, which is sufficient for LACP connections; for STP without an edge port, increase this to 30–45 seconds if necessary. (A more permanent solution is to resolve the issue on the switch side by configuring the host ports as RSTP edge ports (PortFast), which completely eliminates the forwarding delay window.) The drop-in file is then located at /etc/systemd/system/nvmf-autoconnect.service.d/override.conf.

Verification

Check the override and perform a reboot test:

systemd-delta --type=extended | grep nvmf
systemctl cat nvmf-autoconnect.service

After the restart, all paths must be present:

nvme list-subsys

Also check the log to verify that the service was executed only after the wait time had elapsed and that no Connect errors are occurring anymore:

journalctl -b -u nvmf-autoconnect.service