Inside the frame: the main hardware parts

DS8000 hardware components fit into one 42U frame: two Power9+ servers, pairs of I/O enclosures and flash enclosures, two management consoles and the power units feeding them. Once you know what each box does, alerts and service calls stop being scary, because you can read a hardware message and tell straight away whether hosts are at risk.

You’ll walk the frame from bottom to top, the way an IBM service engineer (SSR) does on installation day, and pick up the rule that explains almost every part: everything comes in twos. Then you list each part in the practice shell, fail a flash drive on purpose and watch the array rebuild onto a spare.

You need the practice shell from Lesson 1. The models from Lesson 2 all use the same parts, only in different numbers.

TimeYou needBy the end you can
60 to 75 minutesThe practice shell from Lesson 1 and Python 3.8 or newerName every part in a DS8A00 frame, say what it does and what backs it up, and list the parts with lshmc, lspnode, lsioenclosure, lsdacard, lsstgencl and lsdrive

What it is, in one line

A DS8000 frame holds two CPC servers, I/O enclosures with host and device adapters, HPFE Gen3 flash enclosures, two HMCs and two power domains, and every one of them comes in pairs so the system keeps running when a part fails.

In Simple Words

A railway station built twice over

Think of a busy railway station with two ticket offices, two platforms for every line, two staircases and two power lines from two different power stations.

If one ticket office closes for lunch, the queue moves to the other, and if a staircase is being cleaned people use the second one, so the trains keep running. The station master sits in a small office upstairs watching screens and calling the repair team, but no passenger ever walks through his office to get on a train.

A DS8000 frame is that station: the ticket offices are the two servers, the platforms and staircases are the I/O enclosures and cables, the tracks and storerooms are the flash enclosures, and the station master is the HMC, who manages everything without data ever passing through him. The two power lines are the two power domains, and they only help if they come from two different power stations.

Engineer's View

The parts list of a DS8A00 base frame

This is what IBM puts into a DS8A00 base frame, from the bottom up. The numbers are for the base frame, and the DS8A50 and DS8A80 can add one E05 expansion frame.

PartHow manyWhat it does
HMC (7063-CR2)2Management consoles for the DS GUI, DS CLI, Call Home and code loads. Not in the data path.
Keyboard and display tray1 (1U)Local access to both HMCs at the rack.
Rack power control hubs2 Ethernet switchesPrivate black and gray networks between HMCs, CPCs and iPDUs. No customer equipment allowed.
CPC (Power9+ server)2CPC 0 and CPC 1. Each runs an AIX partition with the storage code (server0, server1).
I/O enclosurePairs, up to 4 per frame3U box with slots for up to 4 host adapters, 2 device adapters and 2 zHyperLink adapters.
HPFE Gen3 enclosurePairs, up to 4 pairs per frame2U flash drawers. Each pair holds 16, 32 or 48 NVMe drives.
iPDUPairs, 1 or 2 pairs per framePower distribution with 12 C13 outlets each. Left side and right side form two power domains.

A few details matter at work. Each I/O enclosure connects to both CPCs with x8 PCIe Gen4 cables, and the CPCs also talk to each other through the I/O enclosures. Each CPC has a flexible service processor (FSP) that looks after its power, cooling and error recovery.

The device adapters (DAs) are the flash RAID adapters. One DA pair serves one HPFE Gen3 pair, an I/O enclosure pair supports up to two DA pairs, and it’s the DA pair that runs the RAID 6 or RAID 10 arrays so the CPCs don’t have to.

Inside each HPFE pair, drives come in sets of 16, split 8 and 8 across the two enclosures, and each set makes two array sites of 8 drives. Every HPFE pair keeps two global spares taken from the first set, so the first two RAID 6 arrays are 5+P+Q+S and the later ones are 6+P+Q.

On power IBM is clear that each power domain should be fed from a separate building power source. The iPDUs on the left are one domain (green) and those on the right the other (yellow), and every CPC, I/O enclosure, HPFE and HMC has power supplies on both sides.

On DS8900F the layout is the same with Power9 CPCs, PCIe Gen3 cables and HPFE Gen2 drawers, and the pairs-of-everything rule doesn’t change.

How it works, step by step

The best way to learn the frame is to walk it as the SSR does on installation day, starting at the bottom and ending at the top, and asking two questions at every part: what does it do, and what happens if it dies?

1Bottom: HMCs, keyboard tray and network hubs

At the bottom sit the two HMCs, small 1U Power9 servers, with a fold-out keyboard and display above them that reaches both, and two Ethernet switches (the rack power control hubs) that carry the private black and gray networks. If one HMC dies you log in through the other, and if both die the hosts keep reading and writing while you lose the GUI, the DS CLI and Call Home until one is back, which is why the HMC counts as a management part and not a data part.

2Middle: the two CPCs

Next come CPC 0 and CPC 1, the two Power9+ servers holding the processor cores and the memory used as cache. Part of that memory sits on NVDIMMs, special memory that saves itself to onboard flash if power fails, kept alive by small backup power modules (BPMs). If one CPC fails, the other already has a copy of its unwritten data in NVS and takes over all the work, and you’ll walk through that failover step by step in Lesson 18.

DS8000 hardware components in a base frame: HPFE Gen3, I/O enclosures, CPCs, HMCs and iPDUs

3Above them: the I/O enclosure pairs

I/O enclosures are 3U boxes that come in pairs, each wired to both CPCs over PCIe Gen4. Inside are up to 4 host adapters (HAs) where the SAN cables plug in, up to 2 device adapters (DAs) that talk to the flash and up to 2 zHyperLink adapters for IBM Z. If one I/O enclosure fails, hosts zoned to adapters in the other enclosure keep working, but only if you spread every host across both, and Lesson 6 covers the rules for that.

4Top: the HPFE Gen3 flash enclosure pairs

The 2U flash drawers come in pairs, and each pair is connected to one DA pair. The drives are NVMe, either IBM FlashCore Modules with built-in compression or industry-standard flash, and one pair holds only one type. When a drive fails, the DA rebuilds its data onto a global spare while RAID 6 keeps the data safe, since RAID 6 can lose two drives in one array without losing anything.

5Along the sides: power

iPDUs run down both sides at the back of the rack, and every part has two power supplies, one plugged into each side: one side is the green power domain and the other the yellow. If one side loses power, the other carries the whole frame. If both sides are out for more than 20 ms, the system shuts down in order and the NVDIMMs save the write cache to their own flash. That’s the short version, and Lesson 8 covers power properly.

How DS8000 parts connect: hosts, I/O enclosures, CPCs, HPFE Gen3 flash and HMCs

Put the five steps together and you get the most useful sentence in this course: every part of a DS8000 has a partner. Two HMCs, two CPCs, I/O enclosures, DAs, HPFEs and iPDUs in pairs, and two power supplies in every box. That’s also how you read a hardware alert, because one failed part almost never means an outage, but it does mean the system is running without a backup for that part, and at work you want it replaced before its partner fails too.

One part is different, though. The pairs only protect you if your own setup respects them: hosts zoned to both I/O enclosures, both power domains fed from different sources, both HMCs on the network. IBM builds the pairs, and it’s up to your team to use them.

Hands-on lab: walk the frame in the practice shell

Before you run any command, double-check it

Read every command twice before you press Enter. Check the target: the system or account, the IDs, the names and the sizes. These commands are for learning and were tested in our lab, not on your system. At work, run the read-only list commands first, follow your change process, make sure a backup exists, and ask a second person to check anything that changes or deletes data. You run them at your own risk.

You’ll do the same frame walk in the practice shell, bottom to top with one command per part, and end up with a parts count for the lab DS8A50, the kind of list you keep for every box you look after.

Practice shell note: the practice shell is our teaching tool, not IBM software. Real DS CLI output is similar but not identical, and the part IDs, location codes and serial numbers here are made up for the lab; on a real DS8000 these commands show the real frame.

1Start clean

Start the shell with python ds8sim.py, log in and reset the lab system.

Run this:

simctl reset

You should see this:

SIM0100I simctl: factory state restored (users and passwords kept; use simctl reset -all to reset them too).

✅ Checkpoint: You are at the dscli> prompt.

2Bottom of the frame: the two HMCs

Run this:

lshmc

You should see this:

Date/Time: 24 October 2026 10:52:54 UTC IBM DSCLI Version: 7.10.10.348 DS: IBM.2107-78SV001
Name          State  Role         Release                  Management-IP Location-Code    Machine-Type Machine-Model Memory Disk
================================================================================================================================
DS8A50-1A.LAB Online Primary(1)   R10.1 bundle 10.10.106.0 10.0.0.1      U7063CR2.78SV1DD 7063         CR2           33GB   1TB
DS8A50-1B.LAB Online Secondary(2) R10.1 bundle 10.10.106.0 10.0.0.2      U7063CR2.78SV2DD 7063         CR2           33GB   1TB

Machine type 7063, model CR2, is the Power9 HMC from the parts table. One is primary and one secondary, each with its own IP address on your network.

✅ Checkpoint: Two HMCs, both Online.

3The two CPCs

Run this:

lspnode

You should see this:

Date/Time: 24 October 2026 10:52:54 UTC IBM DSCLI Version: 7.10.10.348 DS: IBM.2107-78SV001
ID State  Release Processor       Memory   Location
==============================================================
00 Online R10.1   Power9+ 20-core 512.0 GB U9009.42A.78SV00-P1
01 Online R10.1   Power9+ 20-core 512.0 GB U9009.42A.78SV01-P1

Node 00 is CPC 0 and node 01 is CPC 1, and both run the same release. On a real system, a node that isn’t Online means its partner is carrying all the work.

✅ Checkpoint: Two nodes, same release, both Online.

4The I/O enclosures

Run this:

lsioenclosure

You should see this:

Date/Time: 24 October 2026 10:52:54 UTC IBM DSCLI Version: 7.10.10.348 DS: IBM.2107-78SV001
ID                 Frame State  Host adapters Device adapters Location code   S/N
========================================================================================
IBM.1400-1B1-78SV0 1     Online 1             2               U1400.1B1.78SV0 YE10SV0001
IBM.1400-1B2-78SV0 1     Online 1             2               U1400.1B2.78SV0 YE10SV0002
IBM.1400-1B3-78SV0 1     Online 2             0               U1400.1B3.78SV0 YE10SV0003
IBM.1400-1B4-78SV0 1     Online 0             0               U1400.1B4.78SV0 YE10SV0004

Four enclosures make two pairs. The first pair (1B1, 1B2) carries the device adapters for the flash and the second pair (1B3, 1B4) holds only host adapters, with none in 1B4 yet, which is a design gap you’ll understand in Lesson 6.

✅ Checkpoint: You counted 4 I/O enclosures, 4 host adapters and 4 device adapters.

5The device adapters

Run this:

lsdacard

You should see this:

Date/Time: 24 October 2026 10:52:54 UTC IBM DSCLI Version: 7.10.10.348 DS: IBM.2107-78SV001
ID                           I/O enclosure      Frame State  DA pair Location code         S/N
=====================================================================================================
IBM.1400-1B1-78SV0/R-1-P1-C6 IBM.1400-1B1-78SV0 1     Online 0       U1400.1B1.78SV0-P1-C6 YA30SV0010
IBM.1400-1B2-78SV0/R-1-P1-C6 IBM.1400-1B2-78SV0 1     Online 0       U1400.1B2.78SV0-P1-C6 YA30SV0020
IBM.1400-1B1-78SV0/R-1-P1-C8 IBM.1400-1B1-78SV0 1     Online 2       U1400.1B1.78SV0-P1-C8 YA30SV0012
IBM.1400-1B2-78SV0/R-1-P1-C8 IBM.1400-1B2-78SV0 1     Online 2       U1400.1B2.78SV0-P1-C8 YA30SV0022

Look at the pairs: DA pair 0 has one card in 1B1 and one in 1B2, and DA pair 2 is laid out the same way. Because each pair is split across two enclosures, losing one enclosure never cuts off the flash.

✅ Checkpoint: Each DA pair has one card in each enclosure of the pair.

6Top of the frame: flash enclosures and drives

Run this:

lsstgencl IBM.2107-78SV001

You should see this:

Date/Time: 24 October 2026 10:52:54 UTC IBM DSCLI Version: 7.10.10.348 DS: IBM.2107-78SV001
ID                    Frame Enclnum Loc   FC Interfaces State  DA pair
======================================================================
IBM.2107-D04-8SV01/R1 1     1       R1-D1 HPFE Gen3     Normal 0
IBM.2107-D04-8SV01/R2 1     2       R1-D2 HPFE Gen3     Normal 0
IBM.2107-D04-8SV02/R1 1     3       R1-D3 HPFE Gen3     Normal 2
IBM.2107-D04-8SV02/R2 1     4       R1-D4 HPFE Gen3     Normal 2

Two HPFE Gen3 pairs, 8SV01 on DA pair 0 and 8SV02 on DA pair 2, which fits the rule “one DA pair per HPFE pair”.

Practice shell lsdacard and lsstgencl output: two DA pairs and two HPFE Gen3 pairs
Device adapter pairs and the HPFE Gen3 pairs they serve, in the practice shell.

Now count the drives. lsdrive prints one line per drive, and -s prints only the IDs.

Run this:

lsdrive -s

You should see this (output shortened):

Date/Time: 24 October 2026 11:05:30 UTC IBM DSCLI Version: 7.10.10.348 DS: IBM.2107-78SV001
IBM.2107-D04-8SV01/R1-P1-D1
IBM.2107-D04-8SV01/R1-P1-D2
...
IBM.2107-D04-8SV02/R1-P1-D31
IBM.2107-D04-8SV02/R1-P1-D32

Look at the IDs: each HPFE pair (8SV01, 8SV02) numbers its drives D1 to D32, so that’s 32 drives per pair in two sets of 16, and 64 drives in total. The 8-drive groups inside each pair are the array sites you saw with lsarraysite in Lesson 1.

✅ Checkpoint: You found 2 HPFE pairs and 64 drives, 32 in each pair.

7Work out one HPFE pair on paper, with Python

The shell shows you the parts, and a small script shows the rules behind them: sets of 16, two array sites per set and two global spares per HPFE pair. Save this as drive_sets.py in your lab folder, next to ds8sim.py.

It’s a planning tool that runs on your laptop, and the plan part needs no DS8000 at all. Step 8 uses its --check part on the lab system.

# drive_sets.py - SaatVedha HPFE Gen3 drive set checker
# A planning tool: it runs on your laptop or jump server, not on the DS8000.
# Plan mode needs no DS8000. --check reads the installed drives through the DS CLI
# (auto.profile: banner off, header on, format delim) and tests them against the same rules.
# Rules from the IBM Redbook SG24-8559-01 (DS8A00, Release 10.1).
# Usage: python drive_sets.py <drives in the HPFE pair> <drive size in TB> <fcm|std> [second type]
#        python drive_sets.py --check
import csv, subprocess, sys
DSCLI = ["python", "ds8sim.py", "-cfg", "auto.profile"]   # at work: ["dscli", "-cfg", "auto.profile"]

def dscli(*args):
    """Run one DS CLI command and return its rows as dictionaries."""
    out = subprocess.run(DSCLI + list(args), capture_output=True, text=True).stdout
    lines = [l for l in out.splitlines() if l and not l.startswith("SaatVedha") and not l.startswith("CMUC")]
    return list(csv.DictReader(lines))

SIZES = {"fcm": [4.8, 9.6, 19.2], "std": [1.6, 1.92, 3.84, 7.68]}

def plan(drives, size, kinds):
    """Print the layout of one HPFE pair, or the rule it breaks. Returns True if it is valid."""
    if len(set(kinds)) > 1:
        print("ERROR: FCM and industry-standard drives can't be mixed in one HPFE pair (Release 10.1).")
        return False
    kind = kinds[0]
    if drives not in (16, 32, 48):
        print(f"ERROR: an HPFE Gen3 pair holds 16, 32 or 48 drives (sets of 16), not {drives}.")
        return False
    if size not in SIZES.get(kind, []):
        print(f"ERROR: {size} TB is not a {kind} size. Choose from {SIZES.get(kind)}.")
        return False
    sets = drives // 16
    sites = drives // 8
    data = 0
    for i in range(sites):
        data += 5 if i < 2 else 6          # first two RAID 6 arrays 5+P+Q+S, the rest 6+P+Q
    print(f"Drive sets      : {sets} x 16 drives, {drives // 2} in each enclosure of the pair")
    print(f"Array sites     : {sites} (8 drives each)")
    print(f"Global spares   : 2")
    print(f"RAID 6 arrays   : 2 x 5+P+Q+S" + (f", {sites - 2} x 6+P+Q" if sites > 2 else ""))
    print(f"Raw capacity    : {drives} x {size} TB = {round(drives * size, 2)} TB")
    print(f"Data drives     : {data} -> {round(data * size, 2)} TB before formatting")
    return True

if len(sys.argv) > 1 and sys.argv[1] == "--check":
    # After installation: group the drives that lsdrive reports by HPFE pair and test each pair.
    pairs = {}
    for d in dscli("lsdrive"):
        pairs.setdefault(d["Storage enclosure"], []).append(d)
    if not pairs:
        print("No drives returned. Check auto.profile and the password file.")
        sys.exit(2)
    bad = 0
    for pair, ds in sorted(pairs.items()):
        classes = sorted({d["Class"] for d in ds})
        kinds = ["fcm" if d["Class"] == "FlashCoreModule" else "std" for d in ds]
        sizes = sorted({round(float(d["Capacity"]) / 1000, 2) for d in ds})
        print(f"HPFE pair {pair}: {len(ds)} drives, {' + '.join(classes)}, {' / '.join(str(s) for s in sizes)} TB")
        if len(sizes) > 1 and len(set(kinds)) == 1:
            print("ERROR: more than one drive size in this pair. Check the order and the drive sets.")
            ok = False
        else:
            ok = plan(len(ds), sizes[0], kinds)
        bad += not ok
    print(f"Result          : {len(pairs) - bad} of {len(pairs)} HPFE pair(s) follow the rules.")
    sys.exit(1 if bad else 0)

if len(sys.argv) < 4:
    print("Usage: python drive_sets.py <drives in the HPFE pair> <drive size in TB> <fcm|std> [second type]")
    sys.exit(1)

drives, size, kind = int(sys.argv[1]), float(sys.argv[2]), sys.argv[3].lower()
kinds = [kind] + ([sys.argv[4].lower()] if len(sys.argv) > 4 else [])
sys.exit(0 if plan(drives, size, kinds) else 2)

Run this (PowerShell):

python drive_sets.py 32 4.8 fcm

You should see this:

Drive sets      : 2 x 16 drives, 16 in each enclosure of the pair
Array sites     : 4 (8 drives each)
Global spares   : 2
RAID 6 arrays   : 2 x 5+P+Q+S, 2 x 6+P+Q
Raw capacity    : 32 x 4.8 TB = 153.6 TB
Data drives     : 22 -> 105.6 TB before formatting

This is one HPFE pair of the lab system. Of the 32 drives, 2 are spares and 8 hold RAID 6 parity, which leaves 22 for data, and FlashCore Modules then compress that data (more on that in Lesson 7).

Try the biggest pair you can buy:

Run this:

python drive_sets.py 48 19.2 fcm

You should see this:

Drive sets      : 3 x 16 drives, 24 in each enclosure of the pair
Array sites     : 6 (8 drives each)
Global spares   : 2
RAID 6 arrays   : 2 x 5+P+Q+S, 4 x 6+P+Q
Raw capacity    : 48 x 19.2 TB = 921.6 TB
Data drives     : 34 -> 652.8 TB before formatting

Eight of those pairs give you the 7,372.8 TB raw maximum of a DS8A50 or DS8A80 from Lesson 2. Type quit when you’ve finished in the shell.

✅ Checkpoint: You can explain why 32 drives give 22 data drives in RAID 6.

8Check the installed drives against the rules

That’s the plan on paper, and after installation you check the real box against the same rules. --check asks the DS CLI for lsdrive, groups the drives by HPFE pair and runs each pair through those rules.

A script can’t answer password prompts, so save your password in a password file first. If security.dat is already in your lab folder from an earlier lesson, skip this command (the shell would tell you the record already exists).

Run this (PowerShell):

python ds8sim.py -hmc1 10.0.0.1 -user admin -passwd YOURPASSWORD managepwfile -action add -name admin -pw YOURPASSWORD -pwfile security.dat

You should see this:

CMUC00206I managepwfile: Record 10.0.0.1/admin successfully added to password file security.dat.

Then save these eight lines as auto.profile in the same folder. banner: off, header: on and format: delim make every list come back as one header row plus comma-separated rows, which Python reads easily. Implementation Lesson 2 explains profiles in full.

auto.profile:

hmc1: 10.0.0.1
username: admin
pwfile: security.dat
devid: IBM.2107-78SV001
banner: off
header: on
format: delim
delim: ,

Run this:

python drive_sets.py --check

You should see this:

HPFE pair IBM.2107-D04-8SV01: 32 drives, FlashCoreModule, 4.8 TB
Drive sets      : 2 x 16 drives, 16 in each enclosure of the pair
Array sites     : 4 (8 drives each)
Global spares   : 2
RAID 6 arrays   : 2 x 5+P+Q+S, 2 x 6+P+Q
Raw capacity    : 32 x 4.8 TB = 153.6 TB
Data drives     : 22 -> 105.6 TB before formatting
HPFE pair IBM.2107-D04-8SV02: 32 drives, FlashCoreModule + FlashTier2, 3.84 / 4.8 TB
ERROR: FCM and industry-standard drives can't be mixed in one HPFE pair (Release 10.1).
Result          : 1 of 2 HPFE pair(s) follow the rules.

Pair 8SV01 passes with 32 FlashCore Modules of one size, the same as the plan from step 7, and pair 8SV02 fails, which is the check doing its job. The practice shell simplifies that pair by putting FCM and tier 2 drives in it together, something a real Release 10.1 system doesn’t allow, and Lesson 7 comes back to it.

At work you run --check after every drive upgrade, before anyone builds arrays on the new drives.

✅ Checkpoint: The check passes pair 8SV01 and flags the mixed drive types in pair 8SV02.

Break It, Then Fix It

Two things go wrong in real frames all the time: a flash drive fails, or someone plans hardware the rules don’t allow. You’ll do both on purpose.

1Break it: fail a flash drive

You need an array to fail a drive in, so save this one-line setup file as setup-03.txt and run it in script mode (arrays get built properly in Lesson 10).

setup-03.txt:

mkarray -raidtype 6 -arsite S1

Run this (PowerShell):

python ds8sim.py -hmc1 10.0.0.1 -user admin -passwd YOURPASSWORD -script setup-03.txt

You should see this:

CMUC00004I mkarray: Array A0 successfully created.

Now start the shell, fail a drive in array site S1 and see what it did.

Run this:

simctl drivefail S1
lsarray -l
lsdrive -l

You’ll see this (lsdrive output shortened):

dscli> simctl drivefail S1
SIM0110I simctl: drive IBM.2107-D04-8SV01/R1-P1-D1 failed. Array A0 is rebuilding onto a spare (about 10 simulated minutes; use simctl tick).
dscli> lsarray -l
Date/Time: 24 October 2026 10:52:54 UTC IBM DSCLI Version: 7.10.10.348 DS: IBM.2107-78SV001
Array State      Data       RAIDtype    arsite Rank DA Pair DDMcap (10^9B) diskclass       encrypt
====================================================================================================
A0    Unassigned Rebuilding 6 (5+P+Q+S) S1     -    0       4800.0         FlashCoreModule supported
dscli> lsdrive -l
ID                           Storage enclosure  Frame State  Class           Capacity Array Location  Interface Firmware S/N
===============================================================================================================================
IBM.2107-D04-8SV01/R1-P1-D1  IBM.2107-D04-8SV01 1     Failed FlashCoreModule 4800.0   A0    R1-P1-D1  NVMe      4.1.2    SVS101

Why: one drive in array A0 died, and the device adapter pair started rebuilding its data onto a global spare straight away. Data Rebuilding means the array still serves I/O but has used up one of its two RAID 6 safety margins, while the enclosure stays Normal because only one slot failed.

Fix: on a real DS8000 the HMC raises a serviceable event and calls IBM, and an SSR replaces the drive, so your part is to check the call went out and watch the rebuild. In the shell, move the clock forward.

Run this:

simctl tick
lsarray -l

You should see this:

SIM0111I simctl: array A0 rebuild complete; the failed drive was replaced and is now the spare.
SIM0112I simctl: clock advanced 10 minutes.
Date/Time: 24 October 2026 11:02:54 UTC IBM DSCLI Version: 7.10.10.348 DS: IBM.2107-78SV001
Array State      Data   RAIDtype    arsite Rank DA Pair DDMcap (10^9B) diskclass       encrypt
================================================================================================
A0    Unassigned Normal 6 (5+P+Q+S) S1     -    0       4800.0         FlashCoreModule supported
Practice shell drive failure: array A0 rebuilding onto a spare, then back to Normal
A failed drive, the rebuild and the array back to Normal in the practice shell.

✅ Checkpoint: Array A0 shows Normal again.

2Break it: plan an HPFE pair the rules don’t allow

A buyer wants to save money: “40 drives of 3.84 TB, and let’s put a few FlashCore Modules in the same drawer.” Check both ideas.

Run this (PowerShell):

python drive_sets.py 40 3.84 std
python drive_sets.py 32 4.8 fcm std

You’ll get this:

ERROR: an HPFE Gen3 pair holds 16, 32 or 48 drives (sets of 16), not 40.
ERROR: FCM and industry-standard drives can't be mixed in one HPFE pair (Release 10.1).

Why: flash comes in drive sets of 16, split 8 and 8 across the two enclosures, so 40 isn’t possible. And with Release 10.1, one HPFE pair (and one storage pool) holds either FlashCore Modules or industry-standard drives, never a mix.

Fix: order 32 or 48 drives, and put each drive type in its own HPFE pair.

Run this:

python drive_sets.py 48 3.84 std

You should see this:

Drive sets      : 3 x 16 drives, 24 in each enclosure of the pair
Array sites     : 6 (8 drives each)
Global spares   : 2
RAID 6 arrays   : 2 x 5+P+Q+S, 4 x 6+P+Q
Raw capacity    : 48 x 3.84 TB = 184.32 TB
Data drives     : 34 -> 130.56 TB before formatting

✅ Checkpoint: You can explain why “40 drives” and “mixed drawers” both fail.

This happened at work

Two power cords, one fuse board

A regional hospital ran its patient records and lab results on a DS8000 with two iPDU pairs, green on the left and yellow on the right, as IBM ships it, so on paper it could lose a whole side and carry on. Then during a building upgrade an electrician moved both rack feeds to a new distribution board to tidy the cabling. Both cords now came from the same board through two breakers side by side, and nobody on the storage team was asked.

Six weeks later that board tripped during a generator test. Both power domains went dark at the same moment. The DS8000 did what it was built to do: it shut down in order, and the NVDIMMs saved the write cache to flash.

No data was lost.

But the hospital lost its records system for 70 minutes while power came back, the system restarted and the databases recovered. Doctors went back to paper charts in the emergency ward.

When the team dug in, the two power domains turned out to be only as independent as the building wiring behind them, and nobody had looked at that wiring since installation. They moved one feed back to a separate board on a different UPS, labelled both feeds at the board and at the rack, and added “power path check” to every facilities change form. The frame gives you two of everything, but your data centre has to give it two different sources.

Common mistakes and troubleshooting

SymptomLikely causeWhat to check / fix
A storage “outage” turns out to be a management outageBoth HMCs unreachable, hosts were fineHMCs aren’t in the data path, so check host I/O first, then the HMC network and power.
Both power domains fail togetherBoth iPDU feeds come from one building sourceTrace both feeds to separate boards or UPSs and label them at both ends.
Hosts lose paths when one I/O enclosure is servicedAll of a host’s ports sit in one enclosureZone each host to host adapters in both enclosures of a pair (Lesson 6).
An array shows RebuildingA drive failed and a spare is taking overCheck lsdrive -l for the failed drive, confirm the Call Home went out, watch until Normal.
Someone plugs a laptop into the rack switchMistaking the rack power control hubs for a normal switchThey carry only the private black and gray networks, and nothing else may be plugged in.
A capacity order is rejectedDrive count not a multiple of 16, or FCM mixed with standard drivesOrder in sets of 16; one drive type per HPFE pair.
Old scripts still call lsddm and lsdaPre-G10 command namesThey still run on G10 but have no help page, so move to lsdrive and lsdacard.

Keep one habit from this lesson: when any part fails, ask what its partner is and whether the partner is healthy. A failed part with a healthy partner is a service ticket, but a failed part with a sick partner is an emergency.

Interview questions

Click a question to open it. Each is worded the way an interviewer would ask it, and underneath you’ll find why they ask it and one way you could answer in your own words.

Fresher level

1. Walk me through the main hardware parts of a DS8000 frame.

Why they ask this: They want to know you can picture the box rather than recite a list, and a clear bottom-to-top walk shows you understand it.

How you might answer: From the bottom up: two HMCs for management, then the two Power9+ CPCs that run the storage code and hold the cache, and above them I/O enclosures in pairs with host adapters, device adapters and zHyperLink adapters. HPFE Gen3 flash enclosure pairs sit at the top, iPDUs along both sides form two power domains, and everything has two power supplies.

2. What does an HMC do, and what happens to hosts if both HMCs are down?

Why they ask this: It’s the classic check of whether you know the management path from the data path.

How you might answer: The HMC runs the GUI and DS CLI, Call Home and code loads, and talks to the iPDUs and CPCs over the private networks. If both are down, hosts keep doing I/O because data never goes through the HMC; I lose management and alerts until one comes back, so it’s urgent but it isn’t an outage.

2–5 years experience

3. A drive failed overnight. What do you check in the morning?

Why they ask this: They want to see a calm routine and that you know about spares and RAID 6, which tells them you won’t panic or ignore it.

How you might answer: I’d run lsdrive -l to find the failed drive and lsarray -l to see if the array is still rebuilding or back to Normal, then check that the HMC opened a Call Home and IBM has scheduled a replacement. RAID 6 can lose two drives in one array, but until the rebuild finishes there’s less margin, so I keep watching it.

4. What is the relationship between device adapters and HPFE Gen3 enclosures?

Why they ask this: It tests hardware knowledge beyond the marketing slide, and anyone who has done a capacity upgrade knows this.

How you might answer: Each HPFE Gen3 pair is connected to one device adapter pair, and the two DAs of that pair sit in the two enclosures of an I/O enclosure pair. Since one I/O enclosure pair supports up to two DA pairs, a third HPFE pair needs a second I/O enclosure pair.

Senior level

5. How would you check that a DS8000 really has no single point of failure in your data centre?

Why they ask this: Seniors have to look past the box to the facility and the SAN, so the interviewer is looking for a checklist mindset.

How you might answer: Inside the box IBM gives you pairs, so I’d look at everything outside it. I’d trace both iPDU feeds to different power sources, check both HMCs are on the network with working Call Home, and confirm every host is zoned to two I/O enclosures on two fabrics. Then I’d test it in a planned window, say by pulling one feed, and watch for path or power alerts.

6. A facilities team wants to do power work on one side of the rack. What do you tell them?

Why they ask this: They want to know you follow the safe procedure and protect the system while facilities work goes on.

How you might answer: First I’d confirm both power domains are healthy. Once power is back on the side they worked on, every power supply input LED on that side has to be solid green before anyone touches the other side. I’d also make sure the two feeds come from different sources, and keep HMC Call Home active during the work.

Quick quiz

Q1. How many HMCs does every DS8A00 base frame have?

  1. One
  2. Two
  3. Four
  4. One per CPC plus a spare
Show answer

B — Every DS8A00 base frame has two HMCs (Power9, 7063-CR2) for redundant management.

Q2. What sits inside an I/O enclosure?

  1. Flash drives
  2. Host adapters, device adapters and zHyperLink adapters
  3. The CPC memory
  4. The iPDUs
Show answer

B — An I/O enclosure has slots for up to 4 host adapters, 2 device adapters and 2 zHyperLink adapters.

Q3. How many flash drives can one HPFE Gen3 pair hold?

  1. 8, 16 or 24
  2. 16, 32 or 48
  3. 24 only
  4. Any number up to 48
Show answer

B — Drives come in sets of 16, so a pair holds 16, 32 or 48.

Q4. An array shows Data “Rebuilding”. What does it mean?

  1. The array is offline
  2. A drive failed and its data is being rebuilt onto a spare
  3. A new array is being created
  4. The CPC is restarting
Show answer

B — The array keeps serving I/O while the device adapters rebuild the failed drive onto a global spare.

Q5. Why should the two power domains be fed from separate building sources?

  1. To save energy
  2. So one building failure can’t cut both sides at once
  3. Because IBM charges per source
  4. To balance the CPC load
Show answer

B — The pairs only protect you if each side has its own source. Two cords from one board fail together.

Quick recap

  • A DS8A00 frame holds two HMCs, two CPCs, I/O enclosure pairs, HPFE Gen3 flash pairs and iPDU pairs.
  • Every part has a partner, and every box has two power supplies, one per power domain.
  • I/O enclosures hold host adapters, device adapters and zHyperLink adapters, and connect to both CPCs over PCIe Gen4.
  • One DA pair serves one HPFE Gen3 pair, and drives come in sets of 16 with two global spares per HPFE pair.
  • lshmc, lspnode, lsioenclosure, lsdacard, lsstgencl and lsdrive list each part on G10.
  • A failed drive starts a rebuild onto a spare, the HMC calls IBM, and you watch until the array is Normal.

Next lesson

Lesson 4: The two Power9+ servers (CPCs) → You open up CPC 0 and CPC 1 and see how they share the work.

Further reading: DS8000 hardware components

To go deeper into DS8000 hardware components, IBM Storage DS8A00 Architecture and Implementation Guide (IBM Redbooks) describes every part in chapter 2. To compare the three models again, go back to Lesson 2: DS8000 models.

You can also see the full plan for this course on the IBM DS8000 Architecture course page.

Scroll to Top

This website uses cookies

We use a few cookies to keep this site working. If you allow it, we also use Google Analytics to count visits so we can improve our lessons. You can accept all, deny the optional ones, or choose what to allow.