NVMe/TCP and NVMe/FC

SCSI was designed for spinning disks in the 1980s. Flash is so fast now that the old protocol itself has become a cost, and that’s why new ONTAP builds for databases and VMware often use NVMe over Fabrics instead of iSCSI or FC.

The good news: if you understood LUNs and igroups, NVMe is the same idea with new names. This lesson maps those names, builds an NVMe/TCP subsystem on ONTAP, and connects a Linux server to it, using the practice shell for both the cluster and the Ubuntu host.

Where this lesson fits

Lesson 28 of 56 in NetApp ONTAP Storage Administration · Phase 2: Protocols and day-to-day administration · Before this: Lesson 27, FC target ports and zoning · Next: Lesson 29, NFS versions and name services

TimeYou needBy the end you can
70 minutesThe practice shell (version 1.2 or later) in two terminalsExplain subsystem, namespace and host NQN, create NVMe/TCP LIFs, and connect and troubleshoot an NVMe/TCP host

What it is, in one line

NVMe over Fabrics carries the NVMe protocol over TCP or FC; on ONTAP a namespace plays the LUN, a subsystem plays the igroup, and the host is identified by its NQN instead of an IQN or WWPN.

In Simple Words

Express counter at the post office

The old post office counter makes you fill a long form for every parcel, even a tiny one. The new express counter has a short form and several clerks working side by side, so the queue moves much faster.

It’s the same post office and the same parcels. NVMe is the express counter: same storage, much less paperwork per I/O and many queues at once.

Engineer's View

How an engineer describes it

NVMe uses up to 64K queues with up to 64K commands each, against SCSI’s single queue per device, so it cuts CPU and latency on flash. ONTAP supports NVMe/FC on FC adapters (since 9.4) and NVMe/TCP on ordinary Ethernet (since 9.10.1).

Objects map one to one: namespace (a LUN, lives in a volume), subsystem (an igroup plus the mapping, with an ostype), host NQN (the initiator name, in /etc/nvme/hostnqn on Linux). NVMe/TCP LIFs use the default-data-nvme-tcp service policy and listen on port 4420 (discovery on 8009).

Instead of ALUA, NVMe uses ANA (Asymmetric Namespace Access): paths are “optimized” or “non-optimized” the same way. Linux uses native NVMe multipath (nvme_core multipath=Y), and nvme list-subsys shows each path’s ANA state.

How it works, step by step

1Turn on NVMe in the SVM and add LIFs

vserver nvme create, then one NVMe/TCP LIF per node (two per node on separate networks in production).

ONTAP NVMe names compared with SCSI SAN names

2Create the namespace and subsystem

The namespace goes in a volume, like a LUN. The subsystem gets the host’s NQN and the namespace is mapped to it.

3The host discovers and connects

nvme discover asks a LIF what it offers, nvme connect (or connect-all) opens the controllers, and the namespace appears as /dev/nvmeXnY.

ONTAP NVMe/TCP steps from LIFs to a Linux disk

4Make it permanent and check paths

Save the discovery targets and enable the autoconnect service so the host reconnects at boot, then check ANA states with nvme list-subsys.

SCSI SAN (iSCSI / FC)NVMe-oF (TCP / FC)
LUNNamespace
igroup + mappingSubsystem (host NQNs + mapped namespaces)
IQN or WWPNHost NQN
ALUA (AO / ANO)ANA (optimized / non-optimized)
dm-multipathNative NVMe multipath
TCP 3260 (iSCSI)TCP 4420, discovery 8009 (NVMe/TCP)

In System Manager: Storage › NVMe namespaces › + Add creates the namespace and subsystem together, and Hosts › NVMe subsystems lists the hosts.

System Manager NVMe namespaces page listing /vol/vol_nvme1/ns1 in subsystem sub_linux1
System Manager: Storage > NVMe namespaces; + Add creates the namespace and subsystem together.

Hands-on lab: an NVMe/TCP subsystem and a Linux host

Before you run any command, double-check it

Read every command twice before you press Enter. Check the target: the server name, the user, the disk or device name, the path and the sizes. These commands are for learning and were tested in our lab, not on your system. At work, run the read-only commands first, follow your change process, make sure a backup exists, and ask a second person to check anything that changes or deletes data. You run them at your own risk.

This lab uses the SaatVedha ONTAP practice shell, our own learning tool. It is not NetApp software: real ONTAP output looks like this but is not identical.

Everything here runs in the practice shell: the cluster in one terminal (python ontapsim.py) and the Ubuntu host linux1 in a second (python ontapsim.py --host linux1). The ONTAP commands are the real ONTAP 9.18.1 ones, and the host side uses nvme-cli exactly as on a real Ubuntu server.

1Start from a clean lab

Save these three lines as setup-28.txt, then reset and run them from your normal terminal:

storage aggregate create -aggregate aggr1_01 -node saat-cluster-01 -diskcount 8 -disktype SAS
storage aggregate create -aggregate aggr1_02 -node saat-cluster-02 -diskcount 8 -disktype SAS
vserver create -vserver svm1 -aggregate aggr1_01 -rootvolume-security-style unix
python ontapsim.py -c "simctl reset"
python ontapsim.py --run setup-28.txt

✅ Checkpoint: Two aggregates and svm1 are created with no errors.

2Turn on NVMe, add LIFs, a namespace and a subsystem

Log in to the cluster shell and run:

vserver show -vserver svm1 -fields allowed-protocols
network interface create -vserver svm1 -lif svm1_nvme1 -service-policy default-data-nvme-tcp -home-node saat-cluster-01 -home-port e0d -address 192.168.56.141 -netmask 255.255.255.0
network interface create -vserver svm1 -lif svm1_nvme2 -service-policy default-data-nvme-tcp -home-node saat-cluster-02 -home-port e0d -address 192.168.56.142 -netmask 255.255.255.0
vserver nvme create -vserver svm1
vserver nvme show-interface -vserver svm1
volume create -vserver svm1 -volume vol_nvme1 -aggregate aggr1_01 -size 24g -space-guarantee none -snapshot-policy none
vserver nvme namespace create -vserver svm1 -path /vol/vol_nvme1/ns1 -size 20g -ostype linux
vserver nvme subsystem create -vserver svm1 -subsystem sub_linux1 -ostype linux

You should see this:

saat-cluster::> vserver show -vserver svm1 -fields allowed-protocols
vserver allowed-protocols
------- -------------------------------------
svm1    nfs, cifs, fcp, iscsi, ndmp, s3, nvme
saat-cluster::> network interface create -vserver svm1 -lif svm1_nvme1 -service-policy default-data-nvme-tcp -home-node saat-cluster-01 -home-port e0d -address 192.168.56.141 -netmask 255.255.255.0
saat-cluster::> network interface create -vserver svm1 -lif svm1_nvme2 -service-policy default-data-nvme-tcp -home-node saat-cluster-02 -home-port e0d -address 192.168.56.142 -netmask 255.255.255.0
saat-cluster::> vserver nvme create -vserver svm1
saat-cluster::> vserver nvme show-interface -vserver svm1
        Logical                               Transport
Vserver Interface       Home Node:Port        Protocols  Transport Address
------- --------------- --------------------- ---------- --------------------
svm1    svm1_nvme1      saat-cluster-01:e0d   nvme-tcp   192.168.56.141
svm1    svm1_nvme2      saat-cluster-02:e0d   nvme-tcp   192.168.56.142
2 entries were displayed.
saat-cluster::> volume create -vserver svm1 -volume vol_nvme1 -aggregate aggr1_01 -size 24g -space-guarantee none -snapshot-policy none
[Job 104] Job succeeded: Successful
saat-cluster::> vserver nvme namespace create -vserver svm1 -path /vol/vol_nvme1/ns1 -size 20g -ostype linux
saat-cluster::> vserver nvme subsystem create -vserver svm1 -subsystem sub_linux1 -ostype linux

One NVMe/TCP LIF per node, so the host gets a path through each node. The namespace is the disk; the subsystem is the list of hosts allowed to see it, still empty for now. Your job number may differ.

✅ Checkpoint: show-interface lists svm1_nvme1 and svm1_nvme2 with protocol nvme-tcp.

3Get linux1’s host NQN

Open the second terminal with python ontapsim.py --host linux1 and run:

nvme list
apt install -y nvme-cli
cat /etc/nvme/hostnqn

You should see this:

root@linux1:~# nvme list
Command 'nvme' not found, but can be installed with:

apt install nvme-cli
root@linux1:~# apt install -y nvme-cli
(apt output trimmed)
Setting up nvme-cli (2.8-1build2) ...
root@linux1:~# cat /etc/nvme/hostnqn
nqn.2014-08.org.nvmexpress:uuid:a1ce8f6a-977d-0b92-5b5d-02921e0195a9

Installing nvme-cli writes a host NQN to /etc/nvme/hostnqn. That string is to NVMe what the IQN is to iSCSI: the subsystem must list it, character for character. Copy yours.

✅ Checkpoint: You have linux1’s host NQN copied.

4Allow the host and map the namespace

Back in the cluster shell (put your own NQN in the first line):

vserver nvme subsystem host add -vserver svm1 -subsystem sub_linux1 -host-nqn nqn.2014-08.org.nvmexpress:uuid:a1ce8f6a-977d-0b92-5b5d-02921e0195a9
vserver nvme subsystem map add -vserver svm1 -subsystem sub_linux1 -path /vol/vol_nvme1/ns1
vserver nvme subsystem show -vserver svm1
vserver nvme namespace show -vserver svm1

You should see this:

saat-cluster::> vserver nvme subsystem host add -vserver svm1 -subsystem sub_linux1 -host-nqn nqn.2014-08.org.nvmexpress:uuid:a1ce8f6a-977d-0b92-5b5d-02921e0195a9
saat-cluster::> vserver nvme subsystem map add -vserver svm1 -subsystem sub_linux1 -path /vol/vol_nvme1/ns1
saat-cluster::> vserver nvme subsystem show -vserver svm1
Vserver Subsystem       Target NQN
------- --------------- --------------------------------------------------------
svm1    sub_linux1      nqn.1992-08.com.netapp:sn.c84b2d40ac3f32ca940fcb7e13e6d933:subsystem.sub_linux1
saat-cluster::> vserver nvme namespace show -vserver svm1
Vserver   Path                             State      Size Subsystem  NSID
--------- -------------------------------- ------- ------- ---------- ---------
svm1      /vol/vol_nvme1/ns1               online     20GB sub_linux1 00000001h

The subsystem has its own target NQN, and the namespace now shows Subsystem sub_linux1 with NSID 1. In System Manager the same job is Storage › NVMe namespaces › + Add, which creates the namespace and subsystem together.

System Manager NVMe subsystem sub_linux1 details with linux1 host NQN and namespace ID 1
System Manager: Hosts > NVMe subsystems shows the subsystem, its host NQN and the mapped namespace.

✅ Checkpoint: The namespace is online and mapped to sub_linux1.

5Discover and connect from linux1

modprobe nvme-tcp
nvme discover -t tcp -a 192.168.56.141 -s 8009
nvme connect-all -t tcp -a 192.168.56.141 -s 8009
nvme list
nvme list-subsys

You should see this:

root@linux1:~# modprobe nvme-tcp
root@linux1:~# nvme discover -t tcp -a 192.168.56.141 -s 8009

Discovery Log Number of Records 4, Generation counter 4
=====Discovery Log Entry 0======
trtype:  tcp
adrfam:  ipv4
subtype: current discovery subsystem
treq:    not specified
portid:  7
trsvcid: 8009
subnqn:  nqn.1992-08.com.netapp:sn.c84b2d40ac3f32ca940fcb7e13e6d933:discovery
traddr:  192.168.56.141
eflags:  explicit discovery connections, duplicate discovery information
sectype: none
=====Discovery Log Entry 1======
trtype:  tcp
adrfam:  ipv4
subtype: current discovery subsystem
treq:    not specified
portid:  8
trsvcid: 8009
subnqn:  nqn.1992-08.com.netapp:sn.c84b2d40ac3f32ca940fcb7e13e6d933:discovery
traddr:  192.168.56.142
eflags:  explicit discovery connections, duplicate discovery information
sectype: none
=====Discovery Log Entry 2======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified
portid:  7
trsvcid: 4420
subnqn:  nqn.1992-08.com.netapp:sn.c84b2d40ac3f32ca940fcb7e13e6d933:subsystem.sub_linux1
traddr:  192.168.56.141
eflags:  none
sectype: none
=====Discovery Log Entry 3======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified
portid:  8
trsvcid: 4420
subnqn:  nqn.1992-08.com.netapp:sn.c84b2d40ac3f32ca940fcb7e13e6d933:subsystem.sub_linux1
traddr:  192.168.56.142
eflags:  none
sectype: none
root@linux1:~# nvme connect-all -t tcp -a 192.168.56.141 -s 8009
root@linux1:~# nvme list
Node                  Generic               SN                   Model                                    Namespace Usage                      Format           FW Rev  
--------------------- --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1          /dev/ng0n1            0iAuZq8QZ5UL         NetApp ONTAP Controller                  1           0.00  GB /  21.47  GB      4 KiB +  0 B   FFFFFFFF
root@linux1:~# nvme list-subsys
nvme-subsys0 - NQN=nqn.1992-08.com.netapp:sn.c84b2d40ac3f32ca940fcb7e13e6d933:subsystem.sub_linux1
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:a1ce8f6a-977d-0b92-5b5d-02921e0195a9
               iopolicy=round-robin
\
 +- nvme0 tcp traddr=192.168.56.141,trsvcid=4420,src_addr=192.168.56.21 live optimized
 +- nvme1 tcp traddr=192.168.56.142,trsvcid=4420,src_addr=192.168.56.21 live non-optimized

Discovery on port 8009 returned the two discovery controllers and the subsystem through both LIFs; connect-all connected to both on port 4420. nvme list shows one namespace, and nvme list-subsys shows its two paths with their ANA states: optimized through the namespace’s own node, non-optimized through the partner. Both paths under one subsystem means Linux’s native NVMe multipath is doing its job.

✅ Checkpoint: nvme list-subsys shows one live optimized and one live non-optimized path.

6Match the device to ONTAP, then use it

With NetApp’s nvme-cli plugin you can tie the Linux device back to its ONTAP path:

nvme netapp ontapdevices -o column
root@linux1:~# nvme netapp ontapdevices -o column
Device           Vserver   Namespace Path                                   NSID UUID                                   Size
---------------- --------- ------------------------------------------------ ---- -------------------------------------- ---------
/dev/nvme0n1     svm1      /vol/vol_nvme1/ns1                               1    d887fe41-d6c4-34e9-3f00-9162a3b9146c   21.47GB

Then make a file system, mount it and turn on reconnect at boot:

mkfs.xfs /dev/nvme0n1
mkdir /mnt/nvme
mount /dev/nvme0n1 /mnt/nvme
df -h /mnt/nvme
echo "-t tcp -a 192.168.56.141 -s 8009" >> /etc/nvme/discovery.conf
systemctl enable nvmf-autoconnect.service

You should see this:

root@linux1:~# mkfs.xfs /dev/nvme0n1
Discarding blocks...Done.
meta-data=/dev/nvme0n1           isize=512    agcount=4, agsize=1310720 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=1
         =                       reflink=1    bigtime=1 inobtcount=1 nrext64=1
data     =                       bsize=4096   blocks=5242880, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=16384, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
root@linux1:~# mkdir /mnt/nvme
root@linux1:~# mount /dev/nvme0n1 /mnt/nvme
root@linux1:~# df -h /mnt/nvme
Filesystem   Size  Used Avail Use% Mounted on
/dev/nvme0n1   20G  142M   20G   1% /mnt/nvme
root@linux1:~# echo "-t tcp -a 192.168.56.141 -s 8009" >> /etc/nvme/discovery.conf
root@linux1:~# systemctl enable nvmf-autoconnect.service
Created symlink /etc/systemd/system/multi-user.target.wants/nvmf-autoconnect.service → /usr/lib/systemd/system/nvmf-autoconnect.service.

The discovery.conf line plus nvmf-autoconnect make linux1 reconnect to the subsystem after a reboot. For a mount that survives the reboot too, add it to /etc/fstab with _netdev,nofail, as you did for iSCSI in Lesson 25.

✅ Checkpoint: df -h /mnt/nvme shows a 20G XFS file system.

ONTAP NVMe lab: linux1 connected to an NVMe/TCP subsystem with one optimized and one non-optimized path
linux1 in the practice host: one namespace, two paths with their ANA states, and the file system mounted.

7See the host from ONTAP

In the cluster shell:

vserver nvme subsystem controller show -vserver svm1
saat-cluster::> vserver nvme subsystem controller show -vserver svm1
                     Controller
Vserver Subsystem    ID         LIF           Node            Host NQN
------- ------------ ---------- ------------- --------------- ----------------------------------------------------------
svm1    sub_linux1   0x0021     svm1_nvme1    saat-cluster-01 nqn.2014-08.org.nvmexpress:uuid:a1ce8f6a-977d-0b92-5b5d-02921e0195a9
svm1    sub_linux1   0x0022     svm1_nvme2    saat-cluster-02 nqn.2014-08.org.nvmexpress:uuid:a1ce8f6a-977d-0b92-5b5d-02921e0195a9
2 entries were displayed.

One controller per path, each on its LIF and node, with the host NQN. This is the NVMe version of vserver iscsi session show.

✅ Checkpoint: Two controllers, one on each node.

Try it yourself: Check the SVM’s LIFs and service policies in the Command Builder.

Break It, Then Fix It

NVMe’s version of “wrong IQN in the igroup”: the host NQN changes. It happens when a server is rebuilt or a VM is cloned, because a fresh nvme-cli install writes a brand new host NQN.

1Pretend linux1 was rebuilt

On linux1, unmount, disconnect and generate a new NQN, the way a rebuilt server would get one:

umount /mnt/nvme
nvme disconnect-all
nvme gen-hostnqn
root@linux1:~# umount /mnt/nvme
root@linux1:~# nvme disconnect-all
root@linux1:~# nvme gen-hostnqn
nqn.2014-08.org.nvmexpress:uuid:2023ab1f-0868-afb9-aa5d-1793201d9f16

Now connect using that new NQN, as the rebuilt server would. Use the NQN your own gen-hostnqn printed:

nvme discover -t tcp -a 192.168.56.141 -s 8009 -q nqn.2014-08.org.nvmexpress:uuid:2023ab1f-0868-afb9-aa5d-1793201d9f16
nvme connect-all -t tcp -a 192.168.56.141 -s 8009 -q nqn.2014-08.org.nvmexpress:uuid:2023ab1f-0868-afb9-aa5d-1793201d9f16
nvme list

You’ll get this:

root@linux1:~# nvme discover -t tcp -a 192.168.56.141 -s 8009 -q nqn.2014-08.org.nvmexpress:uuid:2023ab1f-0868-afb9-aa5d-1793201d9f16

Discovery Log Number of Records 2, Generation counter 3
=====Discovery Log Entry 0======
trtype:  tcp
adrfam:  ipv4
subtype: current discovery subsystem
treq:    not specified
portid:  7
trsvcid: 8009
subnqn:  nqn.1992-08.com.netapp:sn.c84b2d40ac3f32ca940fcb7e13e6d933:discovery
traddr:  192.168.56.141
eflags:  explicit discovery connections, duplicate discovery information
sectype: none
=====Discovery Log Entry 1======
trtype:  tcp
adrfam:  ipv4
subtype: current discovery subsystem
treq:    not specified
portid:  8
trsvcid: 8009
subnqn:  nqn.1992-08.com.netapp:sn.c84b2d40ac3f32ca940fcb7e13e6d933:discovery
traddr:  192.168.56.142
eflags:  explicit discovery connections, duplicate discovery information
sectype: none
root@linux1:~# nvme connect-all -t tcp -a 192.168.56.141 -s 8009 -q nqn.2014-08.org.nvmexpress:uuid:2023ab1f-0868-afb9-aa5d-1793201d9f16
root@linux1:~# nvme list
Node                  Generic               SN                   Model                                    Namespace Usage                      Format           FW Rev  
--------------------- --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------

Discovery still answers, but only with the two discovery controllers: there’s no “nvme subsystem” record, so connect-all quietly connects to nothing and nvme list is empty. ONTAP only advertises a subsystem to hosts it knows.

✅ Checkpoint: Discovery shows 2 records instead of 4 and nvme list is empty.

2Read it on the storage side

In the cluster shell:

vserver nvme subsystem host show -vserver svm1
saat-cluster::> vserver nvme subsystem host show -vserver svm1
Vserver Subsystem       Host NQN
------- --------------- ----------------------------------------------------------
svm1    sub_linux1      nqn.2014-08.org.nvmexpress:uuid:a1ce8f6a-977d-0b92-5b5d-02921e0195a9

The subsystem only knows the old NQN. Compare it with the NQN the host is now using and the problem is obvious.

✅ Checkpoint: You can see the subsystem lists a different NQN from the one the host used.

ONTAP NVMe break: a new host NQN gets only discovery records and no namespace
A rebuilt host with a new NQN: discovery answers, but no subsystem and no namespace.

3Fix it

Add the host’s new NQN to the subsystem (and remove the old one later, once nothing uses it), then connect again from linux1:

vserver nvme subsystem host add -vserver svm1 -subsystem sub_linux1 -host-nqn nqn.2014-08.org.nvmexpress:uuid:2023ab1f-0868-afb9-aa5d-1793201d9f16
vserver nvme subsystem host show -vserver svm1
saat-cluster::> vserver nvme subsystem host add -vserver svm1 -subsystem sub_linux1 -host-nqn nqn.2014-08.org.nvmexpress:uuid:2023ab1f-0868-afb9-aa5d-1793201d9f16
saat-cluster::> vserver nvme subsystem host show -vserver svm1
Vserver Subsystem       Host NQN
------- --------------- ----------------------------------------------------------
svm1    sub_linux1      nqn.2014-08.org.nvmexpress:uuid:a1ce8f6a-977d-0b92-5b5d-02921e0195a9
svm1    sub_linux1      nqn.2014-08.org.nvmexpress:uuid:2023ab1f-0868-afb9-aa5d-1793201d9f16
2 entries were displayed.

Back on linux1:

nvme connect-all -t tcp -a 192.168.56.141 -s 8009 -q nqn.2014-08.org.nvmexpress:uuid:2023ab1f-0868-afb9-aa5d-1793201d9f16
nvme list
mount /dev/nvme0n1 /mnt/nvme
df -h /mnt/nvme
root@linux1:~# nvme connect-all -t tcp -a 192.168.56.141 -s 8009 -q nqn.2014-08.org.nvmexpress:uuid:2023ab1f-0868-afb9-aa5d-1793201d9f16
root@linux1:~# nvme list
Node                  Generic               SN                   Model                                    Namespace Usage                      Format           FW Rev  
--------------------- --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1          /dev/ng0n1            0iAuZq8QZ5UL         NetApp ONTAP Controller                  1           0.15  GB /  21.47  GB      4 KiB +  0 B   FFFFFFFF
root@linux1:~# mount /dev/nvme0n1 /mnt/nvme
root@linux1:~# df -h /mnt/nvme
Filesystem   Size  Used Avail Use% Mounted on
/dev/nvme0n1   20G  142M   20G   1% /mnt/nvme

The namespace is back with its data. The better long-term fix is on the host: keep /etc/nvme/hostnqn when you rebuild a server, so the storage side never has to change.

✅ Checkpoint: nvme list shows the namespace again and /mnt/nvme is mounted.

This happened at work

The cluster that forgot its names

A university research group in Patna moved its genome analysis scratch space to NVMe/TCP namespaces on an all-flash ONTAP system. The speed-up was real, and for two months nobody thought about storage.

Then the HPC team reimaged all twelve compute nodes with a new OS build. Every node came back without its NVMe disk, and nvme connect failed with “failed to write to nvme-fabrics device”. The storage admin was on leave, and the HPC team spent most of a day blaming the new kernel.

The cause was simple: installing nvme-cli on a fresh OS writes a brand new /etc/nvme/hostnqn, so none of the nodes matched the subsystem’s host list any more. They added the new NQNs with vserver nvme subsystem host add, and changed the image build to restore each node’s saved hostnqn file, the same way they already kept SSH host keys.

Common mistakes and troubleshooting

MistakeWhat happensDo this instead
Reimaging hosts without keeping /etc/nvme/hostnqnHosts can’t connect to their subsystemSave and restore hostnqn, or update the subsystem host list
Native NVMe multipath turned offTwo separate nvme devices for one namespaceCheck /sys/module/nvme_core/parameters/multipath is Y
One NVMe/TCP LIF onlyNo path through the partner during failoverAt least one LIF per node, ideally two networks
Treating NVMe as “just another LUN” in backup toolsSome older tools don’t support namespacesCheck backup and host software support before moving
No autoconnect at bootNamespaces missing after rebootdiscovery.conf plus nvmf-autoconnect, or your distro’s equivalent
Wrong subsystem ostypeHost-side quirks and support issuesMatch the host OS (linux, windows, vmware)

Interview questions

Click a question to open it. Each one is written the way an interviewer would ask it. Under it you will see why they ask it and one way you could answer, in your own words.

Fresher level

1. How would you explain NVMe/TCP to someone who knows iSCSI?

Why they ask this: They want to see you can map new ideas onto old ones.

How you might answer: It does the same job as iSCSI, block storage over normal Ethernet, but it carries the NVMe protocol instead of SCSI. NVMe has many parallel queues, so it uses less CPU and gives lower latency on flash. The namespace is the LUN, the subsystem is the igroup and the host NQN is the IQN.

2. Where do you find a Linux host’s NVMe host NQN?

Why they ask this: They’re checking you know the value the subsystem needs.

How you might answer: In /etc/nvme/hostnqn. nvme-cli creates it when it’s installed, so it changes if the host is rebuilt, which is worth remembering.

2–5 years experience

3. A Linux host can’t connect to its ONTAP NVMe/TCP subsystem after a rebuild. What do you check?

Why they ask this: They want a methodical answer that starts with the likely cause.

How you might answer: First I’d compare cat /etc/nvme/hostnqn on the host with vserver nvme subsystem host show on ONTAP, because a rebuild usually creates a new NQN. If they match, I’d check the LIFs are up, port 4420 is reachable, and the nvme-tcp module is loaded.

4. What is ANA and how does it compare with ALUA?

Why they ask this: They’re testing whether you understand path optimization in NVMe.

How you might answer: ANA, Asymmetric Namespace Access, is NVMe’s version of ALUA. Paths through the node that owns the namespace are optimized, paths through the partner are non-optimized. Native NVMe multipath reads those states and sends I/O to the optimized paths; nvme list-subsys shows them.

Senior level

5. A customer wants to move their Oracle LUNs from FC to NVMe/TCP. What would you look at before saying yes?

Why they ask this: They want to see you weigh benefits against support, network and operational risk.

How you might answer: I’d check host OS and kernel support in the NetApp interoperability tool, the backup and snapshot tools they use, and whether their Ethernet network has the bandwidth and redundancy FC gave them.

I’d test one non-production database first, compare latency and CPU, and plan the move as a copy at the host level, since a LUN doesn’t convert in place to a namespace.

6. How would you design NVMe/TCP networking for a two-node AFF cluster?

Why they ask this: They’re checking redundancy and separation of storage traffic.

How you might answer: I’d use two separate VLANs or switches, with one NVMe/TCP LIF per node on each, so every host has four paths, two optimized. Storage traffic stays on its own VLANs, and I’d keep the MTU the same end to end, using 9000 only if every hop supports it.

Quick quiz

Q1. In NVMe-oF on ONTAP, what plays the role of a LUN?

  1. Subsystem
  2. Namespace
  3. Host NQN
  4. Portset
Show answer

B — a namespace is the block device, stored in a volume.

Q2. What identifies the host in NVMe-oF?

  1. IQN
  2. WWNN
  3. Host NQN
  4. MAC address
Show answer

C — the host NQN, in /etc/nvme/hostnqn on Linux.

Q3. Which service policy do NVMe/TCP LIFs use?

  1. default-data-files
  2. default-data-iscsi
  3. default-data-nvme-tcp
  4. default-intercluster
Show answer

C — default-data-nvme-tcp.

Q4. What is NVMe’s equivalent of ALUA?

  1. ANA
  2. SLM
  3. MPIO
  4. NPIV
Show answer

A — Asymmetric Namespace Access.

Q5. Why might every NVMe host lose access after being reimaged?

  1. The namespace was deleted
  2. A new host NQN was generated
  3. The LIFs moved
  4. NVMe needs a licence each boot
Show answer

B — a fresh nvme-cli install writes a new hostnqn.

Quick recap

  • NVMe-oF is faster, leaner block storage over TCP or FC.
  • Namespace = LUN, subsystem = igroup + mapping, host NQN = initiator name.
  • NVMe/TCP LIFs use default-data-nvme-tcp; one per node at least.
  • ANA gives optimized and non-optimized paths; Linux uses native NVMe multipath.
  • Keep /etc/nvme/hostnqn across rebuilds.

Next lesson

Lesson 29: NFS versions and name services → You’ll compare NFSv3 and NFSv4.1, see why user IDs matter, and mount a volume both ways.

Further reading: ONTAP NVMe

Want to go deeper into ONTAP NVMe? Read NetApp’s ONTAP concepts guide for where NVMe fits in ONTAP SAN, and the nvme-cli documentation for host commands. You can also see the full plan for this course on the NetApp ONTAP Storage Administration course page.

Previous Lesson
Next Lesson
Scroll to Top

This website uses cookies

We use a few cookies to keep this site working. If you allow it, we also use Google Analytics to count visits so we can improve our lessons. You can accept all, deny the optional ones, or choose what to allow.

Cookie consent by
saatvedha