Final project: build it, break it, fix it
This is where you put the whole course together. You’ll build a small but real setup from an empty cluster, prove it works from two clients, then let three faults loose on it and fix them using nothing but the troubleshooting order from Phase 4.
There’s no new theory here. The point is to practise the habits that matter on the job: build in a clean order, check from the client side, save a baseline, and when things break, find the cause before you touch anything.
Give yourself about two hours. Try each fault on your own before you read the answers at the end of the lab.
Where this lesson fits
Lesson 55 of 56 in NetApp ONTAP Storage Administration · Phase 5: Final project and career · Before this: Lesson 54, SnapMirror and takeover failures · Next: Lesson 56, NetApp interview questions
| Time | You need | By the end you can |
|---|---|---|
| 2 hours | The practice shell (Phase 4 setup not needed: you start from a reset) | Build NFS and iSCSI storage end to end, then find and fix three faults on your own |
What it is, in one line
The final project is a complete ONTAP job in one sitting: build an SVM that serves an NFS share to a Linux web server and a LUN to a database server, then troubleshoot three faults the way you would on a real night shift.
In Simple Words
A driving test, not another lesson
After weeks of driving lessons, the test doesn’t teach you anything new. The examiner watches you do everything in order: mirrors, signal, move off, park, and handle the surprise when a car pulls out in front of you.
This lesson is your driving test. The surprises are planted on purpose, and you pass when you can explain what went wrong, not when you get lucky with a fix.
Engineer's View
What you’ll build
One SVM, svm_fp, with a NAS LIF on node 01 and an iSCSI LIF on each node. NFS v3 and v4.1 are on. The SVM root lets the lab subnet read (to walk the namespace), and a separate export policy web_hosts gives read-write only to linux1 (192.168.56.21).
A 50 GB volume vol_web at /web serves the web files. A thin 60 GB volume vol_db on the other node holds a 50 GB LUN mapped to an igroup for rhel1, so the database server should see two paths: optimized through node 02, which owns the LUN, and non-optimized through node 01.
That covers aggregates, SVMs, LIFs, NFS exports, iSCSI, LUN masking and ALUA, the core of what a storage admin does every week.
How it works, step by step
1Build in layers
Aggregates first, then the SVM, then LIFs, then protocols, then volumes and LUNs, and access rules last. Building in this order means every command finds the thing it depends on.
2Prove it from the clients
ONTAP saying “Job succeeded” isn’t proof. A mount that can write a file and a multipath map with two paths are.
3Save a baseline
Before anything breaks, save the health list and the key show commands. Every fix later is a comparison against this.
4Troubleshoot in order
For each complaint: write the symptom, find the scope, check health, read events, then walk the layers (Lesson 47). Fix one thing at a time and check from the client after each fix.
| Complaint | Start with | Lessons |
|---|---|---|
| NFS Permission denied | Export rules, check-access | 30, 49 |
| LUN missing | Session, igroup, mapping, LUN state | 48 |
| Only one path, or slow path | LIFs on both nodes, ALUA | 24, 53 |
Hands-on lab: the full build and three faults
Before you run any command, double-check it
Read every command twice before you press Enter. Check the target: the server name, the user, the disk or device name, the path and the sizes. These commands are for learning and were tested in our lab, not on your system. At work, run the read-only commands first, follow your change process, make sure a backup exists, and ask a second person to check anything that changes or deletes data. You run them at your own risk.
This lab uses the SaatVedha ONTAP practice shell, our own learning tool. It is not NetApp software: real ONTAP output looks like this but is not identical.
1Reset and build
Reset the lab, create fp-build.txt with these lines, and run it:
storage aggregate create -aggregate aggr1_01 -node saat-cluster-01 -diskcount 8 -disktype SAS
storage aggregate create -aggregate aggr1_02 -node saat-cluster-02 -diskcount 8 -disktype SAS
vserver create -vserver svm_fp -aggregate aggr1_01 -rootvolume-security-style unix
network interface create -vserver svm_fp -lif fp_nas1 -service-policy default-data-files -home-node saat-cluster-01 -home-port e0c -address 192.168.56.150 -netmask 255.255.255.0 -auto-revert true
network interface create -vserver svm_fp -lif fp_iscsi1 -service-policy default-data-iscsi -home-node saat-cluster-01 -home-port e0d -address 192.168.56.151 -netmask 255.255.255.0
network interface create -vserver svm_fp -lif fp_iscsi2 -service-policy default-data-iscsi -home-node saat-cluster-02 -home-port e0d -address 192.168.56.152 -netmask 255.255.255.0
vserver nfs create -vserver svm_fp -v3 enabled -v4.1 enabled
vserver export-policy rule create -vserver svm_fp -policyname default -clientmatch 192.168.56.0/24 -rorule sys -rwrule never -superuser none
vserver export-policy create -vserver svm_fp -policyname web_hosts
vserver export-policy rule create -vserver svm_fp -policyname web_hosts -clientmatch 192.168.56.21 -rorule sys -rwrule sys -superuser sys -protocol nfs
volume create -vserver svm_fp -volume vol_web -aggregate aggr1_01 -size 50g -junction-path /web -policy web_hosts -security-style unix
vserver iscsi create -vserver svm_fp
volume create -vserver svm_fp -volume vol_db -aggregate aggr1_02 -size 60g -space-guarantee none -snapshot-policy none
lun create -vserver svm_fp -path /vol/vol_db/lun_db -size 50g -ostype linux -space-reserve disabled
lun igroup create -vserver svm_fp -igroup ig_rhel1 -protocol iscsi -ostype linux -initiator iqn.1994-05.com.redhat:8f3d2c1b9a7e
lun mapping create -vserver svm_fp -path /vol/vol_db/lun_db -igroup ig_rhel1python ontapsim.py -c "simctl reset"
python ontapsim.py --run fp-build.txtThe only warnings should be that each node has no spare SAS disks left, which is expected in this small lab.
✅ Checkpoint: No line starts with Error.
2Prove both clients work
Log in and run:
simctl host linux1 mount 192.168.56.150:/web /mnt/web vers=4.1
simctl host linux1 touch /mnt/web/index.html
simctl host linux1 ls /mnt/web
simctl host rhel1 iscsi-discover 192.168.56.151
simctl host rhel1 iscsi-login
simctl host rhel1 rescan
simctl host rhel1 multipathYou should see the share and two paths:
(mounted 192.168.56.150:/web on /mnt/web type nfs4)
...
index.html
...
Scanning SCSI subsystem for new devices... 2 new or changed device(s) found.
saat-cluster::> simctl host rhel1 multipath
3600a0980337a4868374d525946676d4 dm-2 NETAPP,LUN C-Mode
size=50G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 30:0:0:0 sdj 8:16 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
`- 32:0:0:0 sdl 8:48 active ready running✅ Checkpoint: linux1 lists index.html and rhel1 shows a 50G LUN with two paths.
3Save your baseline
cluster show
storage failover show
network interface show -vserver svm_fp
vserver export-policy rule show -vserver svm_fp
lun igroup show -vserver svm_fp -instanceCopy the output into a text file. You’ll compare against it in a few minutes.
✅ Checkpoint: You have the baseline saved.
4Let the faults loose
Run these lines without thinking about them too much, then pretend three different teams have just called you:
simctl break export-drop svm_fp web_hosts
simctl break iqn-typo svm_fp ig_rhel1
simctl break lif-down svm_fp fp_iscsi2
simctl tick 1The web team says their server can’t read its files. The database team says their disk is gone after a reboot.
A third person says “one of the storage paths alerted last night”. Now find and fix everything, using only ONTAP and the host views. Stop here and try it yourself before reading on.
✅ Checkpoint: You have written down three symptoms.
5Answers: what the clients see
simctl host linux1 ls /mnt/web
simctl host rhel1 rescanYou’ll see:
ls: cannot open directory '/mnt/web': Permission denied
saat-cluster::> simctl host rhel1 rescan
Scanning SCSI subsystem for new devices... 0 new or changed device(s) found.
(No LUN paths. Check: sessions or FC logins, igroup initiators, LUN mapping, LUN/volume online, reporting nodes.)✅ Checkpoint: You see Permission denied and no LUN paths.
6Answers: health first, then the layers
cluster show
event log show -severity <=ERROR
network interface show -vserver svm_fp
vserver export-policy rule show -vserver svm_fp -policyname web_hosts
vserver export-policy check-access -vserver svm_fp -volume vol_web -client-ip 192.168.56.21 -authentication-method sys -protocol nfs4 -access-type read-write
lun igroup show -vserver svm_fp -instance
vserver iscsi initiator show -vserver svm_fpThe important lines:
saat-cluster::> event log show -severity <=ERROR
There are no entries matching your query.
...
fp_iscsi2 down/down 192.168.56.152/24 saat-cluster-02 e0d true
saat-cluster::> vserver export-policy rule show -vserver svm_fp -policyname web_hosts
There are no entries matching your query.
saat-cluster::> vserver export-policy check-access -vserver svm_fp -volume vol_web -client-ip 192.168.56.21 -authentication-method sys -protocol nfs4 -access-type read-write
Policy Policy Rule
Path Policy Owner Owner Type Index Access
---------------------- ------------ ------------ ---------- ------ ----------
/ default svm_fp_root volume 1 read
/web web_hosts vol_web volume 0 denied
2 entries were displayed.
...
Initiators: iqn.1994-05.com.redhat:8f3d2c1b9a70 (not logged in)
saat-cluster::> vserver iscsi initiator show -vserver svm_fp
Tpgroup Initiator Igroup
Vserver Name Name Name
---------- ------------- ---------------------------------------------- ------------
svm_fp fp_iscsi1 iqn.1994-05.com.redhat:8f3d2c1b9a7e -
svm_fp fp_iscsi2 iqn.1994-05.com.redhat:8f3d2c1b9a7e -
2 entries were displayed.The cluster is healthy and the event log is empty, so these are configuration changes, not hardware.
Fault 1: web_hosts has no rules, so /web is denied. Fault 2: the igroup holds an IQN ending in 9a70 while rhel1 really is 9a7e. Fault 3: fp_iscsi2 is administratively down.
✅ Checkpoint: You found all three faults, or know which one you missed.
7Answers: fix one at a time and check
lun igroup remove -vserver svm_fp -igroup ig_rhel1 -initiator iqn.1994-05.com.redhat:8f3d2c1b9a70
lun igroup add -vserver svm_fp -igroup ig_rhel1 -initiator iqn.1994-05.com.redhat:8f3d2c1b9a7e
simctl host rhel1 rescan
simctl host rhel1 multipathAfter the igroup fix you get one path only:
Scanning SCSI subsystem for new devices... 1 new or changed device(s) found.
saat-cluster::> simctl host rhel1 multipath
3600a0980337a4868374d525946676d4 dm-2 NETAPP,LUN C-Mode
size=50G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=10 status=enabled
`- 30:0:0:0 sdj 8:16 active ready runningLook closely: the only path has prio=10, the non-optimized one through node 01. The LUN lives on node 02, and node 02’s iSCSI LIF is the one that’s down. The database works, but every I/O takes the long way.
network interface modify -vserver svm_fp -lif fp_iscsi2 -status-admin up
simctl host rhel1 rescan
simctl host rhel1 multipath
vserver export-policy rule create -vserver svm_fp -policyname web_hosts -clientmatch 192.168.56.21 -rorule sys -rwrule sys -superuser sys -protocol nfs
simctl host linux1 ls /mnt/webYou should see both paths back and the web files:
3600a0980337a4868374d525946676d4 dm-2 NETAPP,LUN C-Mode
size=50G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 30:0:0:0 sdj 8:16 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
`- 32:0:0:0 sdl 8:48 active ready running
saat-cluster::> vserver export-policy rule create -vserver svm_fp -policyname web_hosts -clientmatch 192.168.56.21 -rorule sys -rwrule sys -superuser sys -protocol nfs
saat-cluster::> simctl host linux1 ls /mnt/web
index.html✅ Checkpoint: rhel1 has two paths again and linux1 lists index.html.
8Close it out
Run the baseline commands again and compare. Then write three short lines, one per fault: what the user saw, the cause, the fix. That’s your incident note, and it’s the habit interviewers ask about.
✅ Checkpoint: Your new output matches the baseline.
Try it yourself: Open the cluster health check in the Command Builder.
Break It, Then Fix It
A classic mistake made while fixing: you re-create the export rule in a hurry and copy the safe-looking “superuser none” from the default policy.
1Re-create the rule the wrong way
From the fixed state, delete rule 1 and add it back with -superuser none:
vserver export-policy rule delete -vserver svm_fp -policyname web_hosts -ruleindex 1
vserver export-policy rule create -vserver svm_fp -policyname web_hosts -clientmatch 192.168.56.21 -rorule sys -rwrule sys -superuser none -protocol nfs
simctl host linux1 touch /mnt/web/new.htmlYou’ll get this:
touch: cannot touch '/mnt/web/new.html': Permission denied
(you are root on the client, but superuser is "none" in export rule 1, so root is mapped to the anonymous user 65534, which cannot write here)Reading works, but root on linux1 can’t write. With -superuser none, ONTAP maps the client’s root user to the anonymous user (65534 here), which has no write permission on the volume. The practice shell explains this in brackets; a real client just says Permission denied.
✅ Checkpoint: Root on linux1 gets Permission denied when writing.
2Fix it the right way
Decide what the web server needs. If its application runs as root and must write, set superuser to sys for that one host. Better still, run the application as a normal user who owns the files, and keep root squashed.
vserver export-policy rule modify -vserver svm_fp -policyname web_hosts -ruleindex 1 -superuser sys
simctl host linux1 touch /mnt/web/new.html
simctl host linux1 ls /mnt/webYou should see:
index.html new.html✅ Checkpoint: linux1 can write again.
Try it yourself: Build an export rule in the Command Builder.
This happened at work
Three alerts, one night shift
A junior storage engineer at an IT services company in Warangal was on her third night shift when, just after 1 a.m., three tickets arrived within ten minutes of each other, right after a planned power test in the data centre. Her first instinct was to call her lead.
Instead she opened the team runbook and went in order. Cluster health was fine and there were no hardware events, so she treated each ticket as a configuration problem. An automation job had reapplied an old export policy, a rebuilt host had a new IQN, and an iSCSI LIF had been left down after switch work earlier that evening.
She fixed all three in about forty minutes, checked each one from the client side, and wrote a short note per ticket. Her lead’s only change the next morning was to add “check for automation jobs that ran” to the runbook.
Common mistakes and troubleshooting
| Mistake | What happens | Do this instead |
|---|---|---|
| Fixing before finding all faults | You stop at the first fix and miss the rest | List every symptom, then fix one at a time |
| Trusting “Job succeeded” | A share that can’t be written still looks fine | Prove it from the client: write a file, check multipath |
| Ignoring a single path | The LUN works but slowly, and has no spare path | Expect an optimized and a non-optimized path |
| Copying options without reading them | superuser none breaks root writes | Know what each export option does |
| No incident note | The same fault comes back next month | Write symptom, cause and fix for every ticket |
Interview questions
Click a question to open it. Each one is written the way an interviewer would ask it. Under it you will see why they ask it and one way you could answer, in your own words.
Fresher level
1. Walk me through how you’d give a Linux server an NFS share on ONTAP, from scratch.
Why they ask this: They want to see you know the build order and don’t skip access control.
How you might answer: I’d make sure there’s an aggregate and an SVM with NFS turned on and a data LIF. Then I’d create an export policy with a rule for the server’s IP, create the volume with a junction path and that policy, and check it with vserver export-policy check-access before the server mounts it and writes a test file.
2. After you give a LUN to a host, how do you know the job is really done?
Why they ask this: They’re checking that you verify from the host side.
How you might answer: I ask the host team to rescan and show me multipath -ll or the Windows MPIO view. I expect the right size and two or more paths, one optimized through the owning node, and I check lun igroup show -instance says logged in.
2–5 years experience
3. Three tickets arrive at once after a maintenance window. How do you handle them?
Why they ask this: They want calm prioritisation and a method.
How you might answer: First I check cluster health and the event log to see if there’s one shared cause, like a node or port. If the cluster is healthy, I treat them as separate configuration problems, work through them in business-impact order, and check each fix from the client before moving on.
4. A database host has only one path to its LUN and it’s the non-optimized one. What’s likely wrong?
Why they ask this: They’re testing ALUA understanding in a real symptom.
How you might answer: The path through the node that owns the LUN is missing. I’d check the iSCSI or FC LIF on that node, whether it’s down or not zoned, and the reporting nodes in lun mapping show. Until it’s fixed every I/O crosses the cluster network and there’s no spare path.
Senior level
5. How would you turn a set of one-off fixes like these into something your team doesn’t repeat?
Why they ask this: They want to see you improve the system, not just the incident.
How you might answer: I’d write a short note for each incident, then look for patterns: automation jobs that change config, hosts rebuilt without updating igroups, LIFs left down after network work. Each pattern becomes a check in the runbook or a guardrail, like RBAC limits or a post-change verification script.
6. If you could only monitor five things on an ONTAP cluster, what would they be?
Why they ask this: They want judgement about what matters most.
How you might answer: Node health and Takeover Possible, aggregate and volume space, SnapMirror health and lag, LIFs that are down or away from home, and latency per important volume. With AutoSupport working on top, those catch most problems before users do.
Quick quiz
Q1. In what order should you build NFS storage?
- Export rule, volume, SVM, aggregate
- Aggregate, SVM, LIF, NFS, export policy, volume
- Volume, LIF, aggregate, SVM
- It doesn’t matter
Show answer
B — each step depends on the one before it.
Q2. The event log is empty and the cluster is healthy, but three things broke. What does that suggest?
- Hardware failure
- Configuration changes
- A full aggregate
- A failed node
Show answer
B — no hardware events points to settings that changed.
Q3. A host shows only a prio=10 path to its LUN. What does that mean?
- Everything is perfect
- Only the non-optimized path through the partner node is working
- The LUN is offline
- The igroup is empty
Show answer
B — the optimized path through the owning node is missing.
Q4. With -superuser none, what happens to root from the client?
- It keeps full access
- It is mapped to the anonymous user
- It is blocked from mounting
- It becomes vsadmin
Show answer
B — root squashing maps root to the anonymous user.
Q5. What proves an NFS share works for a web server?
- The volume create job succeeded
- The server can mount it and write a test file
- The aggregate is online
- The LIF exists
Show answer
B — proof comes from the client side.
Quick recap
- Build in layers: aggregate, SVM, LIFs, protocols, volumes and LUNs, access rules.
- Prove every service from the client and save a baseline.
- Health and events first; empty events point to configuration changes.
- Fix one fault at a time and check from the client after each.
- Write an incident note: symptom, cause, fix.
Next lesson
Lesson 56: NetApp interview questions → You’ll practise the questions interviewers ask at fresher, mid and senior level, and how to answer them.
Further reading: ONTAP final project
Want to go deeper into ONTAP final project? Keep NetApp’s ONTAP concepts guide open while you build; it explains every layer you touch in this project. You can also see the full plan for this course on the NetApp ONTAP Storage Administration course page.