The two Power9+ servers (CPCs)

You’ll build one even volume and one odd volume, check which server owns each, then fail CPC 0 and bring it back. Don’t worry about understanding every setup command yet, because arrays, ranks and pools get their own lessons from Lesson 10 on.

Practice shell note: the practice shell is our own teaching tool, not IBM software, and real DS CLI output is similar but not identical. On a real DS8000 there’s no command to “fail” a CPC. simctl serverfail only exists in the practice shell, as do the lines starting with SIM and the Note line after lsserver.

1Reset and load the setup script

Start the shell, run simctl reset, then quit. Save these lines as setup-04.txt in your lab folder.

setup-04.txt:

mkarray -raidtype 6 -arsite S1
mkarray -raidtype 6 -arsite S2
mkrank -array A0 -stgtype fb -extsize 16MiB -keygrp 1
mkrank -array A1 -stgtype fb -extsize 16MiB -keygrp 1
mkextpool -rankgrp 0 -stgtype fb -keygrp 1 pool_even
mkextpool -rankgrp 1 -stgtype fb -keygrp 1 pool_odd
chrank -extpool P0 R0
chrank -extpool P1 R1
mkfbvol -extpool P0 -cap 100 -sam ese -name db_even 2000
mkfbvol -extpool P1 -cap 100 -sam ese -name db_odd 2100

Run this (PowerShell):

python ds8sim.py -hmc1 10.0.0.1 -user admin -passwd YOURPASSWORD -script setup-04.txt

You should see this:

CMUC00004I mkarray: Array A0 successfully created.
CMUC00004I mkarray: Array A1 successfully created.
CMUC00007I mkrank: Rank R0 successfully created.
CMUC00007I mkrank: Rank R1 successfully created.
CMUC00000I mkextpool: Extent pool P0 successfully created.
CMUC00000I mkextpool: Extent pool P1 successfully created.
CMUC00008I chrank: Rank R0 successfully modified.
CMUC00008I chrank: Rank R1 successfully modified.
CMUC00025I mkfbvol: FB volume 2000 successfully created.
CMUC00025I mkfbvol: FB volume 2100 successfully created.

The lab flash is FlashCore Modules, and that’s why the volumes use -sam ese (thin provisioned). Lesson 15 explains the reason.

✅ Checkpoint: Ten success messages, ending with volumes 2000 and 2100.

2See who owns what

Start the shell again and run:

Run this:

lslss
lsextpool

You should see this:

Date/Time: 24 October 2026 10:59:15 UTC IBM DSCLI Version: 7.10.10.348 DS: IBM.2107-78SV001
ID Group addrgrp stgtype confgvols subsys
=========================================
20 0     2       fb      1         0xFF20
21 1     2       fb      1         0xFF21
Date/Time: 24 October 2026 10:59:15 UTC IBM DSCLI Version: 7.10.10.348 DS: IBM.2107-78SV001
Name      ID stgtype rankgrp status availstor (2^30B) %allocated available reserved numvols
===========================================================================================
pool_even P0 fb      0       below  99576             0          6372864   64       1
pool_odd  P1 fb      1       below  99576             0          6372864   64       1

The Group column of lslss tells you the owner: LSS 20 is in group 0 (server0) and LSS 21 in group 1 (server1). Pools follow the same rule through rankgrp, so volume 2000, which lives in LSS 20 and pool P0, sits entirely on server0.

Volume 2100, though, is on server1.

✅ Checkpoint: You can say which server owns volume 2000 and which owns 2100.

3Fail CPC 0

Run this:

simctl serverfail 0
lsserver
lspnode

You should see this:

SIM0107I simctl: server0 is offline. Failover: server1 now serves rank groups 0 and 1. Host I/O continues; write cache runs in a single copy until failback.
Date/Time: 24 October 2026 10:59:15 UTC IBM DSCLI Version: 7.10.10.348 DS: IBM.2107-78SV001
ID      Image ID         Image Name   Power Control SFI State
=============================================================
server0 IBM.2107-78SV001 ds8k-g10-lab Offline       Offline
server1 IBM.2107-78SV001 ds8k-g10-lab Online        Online
Note (practice shell): server0 is offline. server1 now runs rank groups 0 and 1 (failover).
Date/Time: 24 October 2026 10:59:15 UTC IBM DSCLI Version: 7.10.10.348 DS: IBM.2107-78SV001
ID State   Release Processor       Memory   Location
===============================================================
00 Offline R10.1   Power9+ 20-core 512.0 GB U9009.42A.78SV00-P1
01 Online  R10.1   Power9+ 20-core 512.0 GB U9009.42A.78SV01-P1

✅ Checkpoint: server0 and node 00 show Offline; server1 is Online.

4Check that the volumes are still online

Run this:

lsfbvol

You should see this:

Date/Time: 24 October 2026 10:59:15 UTC IBM DSCLI Version: 7.10.10.348 DS: IBM.2107-78SV001
Name    ID   accstate datastate configstate deviceMTM datatype extpool cap (2^30B) cap (10^9B) cap (blocks)
===========================================================================================================
db_even 2000 Online   Normal    Normal      2107-900  FB 512   P0      100.0       -           209715200
db_odd  2100 Online   Normal    Normal      2107-900  FB 512   P1      100.0       -           209715200

Volume 2000 belongs to the server that just died, yet it still shows Online because server1 is serving it now. A real host would notice no more than a short pause, and that’s what having two CPCs buys you.

✅ Checkpoint: Both volumes show accstate Online while server0 is offline.

5Fail back

Run this:

simctl serverfix 0
lsserver

You should see this:

SIM0108I simctl: server0 is online. Failback: rank group 0 moved back to server0.
Date/Time: 24 October 2026 10:59:15 UTC IBM DSCLI Version: 7.10.10.348 DS: IBM.2107-78SV001
ID      Image ID         Image Name   Power Control SFI State
=============================================================
server0 IBM.2107-78SV001 ds8k-g10-lab Online        Online
server1 IBM.2107-78SV001 ds8k-g10-lab Online        Online

✅ Checkpoint: Both servers are Online again.

6Check failover headroom with numbers

The shell can show you a failover, but it can’t tell you how busy the CPCs are. At work you’d read CPC utilisation from the performance tools (Lesson 22 of the Implementation course), so here you use a small rule-of-thumb script instead: add both loads together and see whether one CPC could carry the total.

Save it as cpc_headroom.py on your laptop, in the lab folder next to ds8sim.py. It’s a planning tool that runs there and not on the DS8000, and its --check part (you’ll use it in step 7) adds what the DS CLI can tell you.

# cpc_headroom.py - SaatVedha failover headroom check (a planning rule of thumb, not an IBM tool)
# A planning tool: it runs on your laptop or jump server, not on the DS8000.
# The busy % comes from your performance tools. --check adds what the DS CLI can tell you
# (auto.profile: banner off, header on, format delim): server state and volumes per server.
# During a failover or a code load, ONE CPC must carry the work of BOTH.
# Usage: python cpc_headroom.py <busy % of CPC 0> <busy % of CPC 1> [safe limit %] [--check]
import csv, subprocess, sys
DSCLI = ["python", "ds8sim.py", "-cfg", "auto.profile"]   # at work: ["dscli", "-cfg", "auto.profile"]

def dscli(*args):
    """Run one DS CLI command and return its rows as dictionaries."""
    out = subprocess.run(DSCLI + list(args), capture_output=True, text=True).stdout
    lines = [l for l in out.splitlines() if l and not l.startswith("SaatVedha") and not l.startswith("CMUC")]
    return list(csv.DictReader(lines))

check = "--check" in sys.argv
args = [a for a in sys.argv[1:] if a != "--check"]
if len(args) < 2:
    print("Usage: python cpc_headroom.py <busy % of CPC 0> <busy % of CPC 1> [safe limit %] [--check]")
    sys.exit(1)

cpc0, cpc1 = float(args[0]), float(args[1])
limit = float(args[2]) if len(args) > 2 else 80.0
alone = cpc0 + cpc1

print(f"CPC 0 busy        : {cpc0:.0f} %")
print(f"CPC 1 busy        : {cpc1:.0f} %")
print(f"One CPC alone     : {alone:.0f} % (both workloads on one server)")
if alone <= limit:
    print(f"Result            : OK, below your {limit:.0f} % limit. A failover or code load is safe.")
elif alone <= 100:
    print(f"Result            : TIGHT, above {limit:.0f} %. Expect higher latency during a failover.")
else:
    print(f"Result            : OVERLOAD. One CPC can't carry {alone:.0f} %. Plan the code load off-peak,")
    print("                    move workload, or size up (more cores and memory).")
skew = abs(cpc0 - cpc1)
if skew > 20:
    print(f"Balance           : CPC loads differ by {skew:.0f} points. Check even/odd LSS and pool balance.")

if check:
    # Both servers must be Online before a code load: if one is down, you already run on one CPC.
    servers = dscli("lsserver")
    print("Servers (lsserver): " + ", ".join(f"{s['ID']} {s['SFI State']}" for s in servers))
    for s in servers:
        if s["SFI State"] != "Online":
            print(f"STOP              : {s['ID']} is not Online. No code load until it is repaired and failed back.")
    # lslss Group 0 = even LSSs = server0, Group 1 = odd LSSs = server1.
    vols = {"0": 0, "1": 0}
    for lss in dscli("lslss"):
        vols[lss["Group"]] += int(lss["confgvols"])
    total = vols["0"] + vols["1"]
    print(f"Volumes (lslss)   : server0 {vols['0']}, server1 {vols['1']}")
    if total and abs(vols["0"] - vols["1"]) * 100 / total > 20:
        side = "server0" if vols["0"] > vols["1"] else "server1"
        print(f"Balance           : most volumes sit on {side}. New volumes should go to the other side.")

The 80% default limit is a habit many teams use when planning, not an IBM number, so change it with the third argument if your team works with another value.

Run this (PowerShell):

python cpc_headroom.py 30 25

You should see this:

CPC 0 busy        : 30 %
CPC 1 busy        : 25 %
One CPC alone     : 55 % (both workloads on one server)
Result            : OK, below your 80 % limit. A failover or code load is safe.

✅ Checkpoint: You get OK for 30 % and 25 %.

Type quit in the shell when you are done.

7Add what the DS CLI knows

The busy numbers come from your performance tools, but two facts come straight from the box: whether both servers are Online, and how the volumes are split between them. --check gets both by asking the DS CLI for lsserver and lslss.

A script can’t answer password prompts, so save your password in a password file first. If security.dat is already in your lab folder from an earlier lesson, skip this command, or the shell will tell you the record already exists.

Run this (PowerShell):

python ds8sim.py -hmc1 10.0.0.1 -user admin -passwd YOURPASSWORD managepwfile -action add -name admin -pw YOURPASSWORD -pwfile security.dat

You should see this:

CMUC00206I managepwfile: Record 10.0.0.1/admin successfully added to password file security.dat.

Then save these eight lines as auto.profile in the same folder. banner: off, header: on and format: delim make every list come back as one header row plus comma-separated rows, which Python reads easily. Implementation Lesson 2 explains profiles in full.

auto.profile:

hmc1: 10.0.0.1
username: admin
pwfile: security.dat
devid: IBM.2107-78SV001
banner: off
header: on
format: delim
delim: ,

Run this (PowerShell):

python cpc_headroom.py 30 25 --check

You should see this:

CPC 0 busy        : 30 %
CPC 1 busy        : 25 %
One CPC alone     : 55 % (both workloads on one server)
Result            : OK, below your 80 % limit. A failover or code load is safe.
Servers (lsserver): server0 Online, server1 Online
Volumes (lslss)   : server0 1, server1 1

Both servers are Online and each owns one volume (in lslss, Group 0 belongs to server0 and Group 1 to server1). If a server were offline the script would print a STOP line, because you’d already be running on one CPC.

At work the change team runs this on the morning of every code load with the latest busy numbers and attaches the report to the change record.

✅ Checkpoint: --check shows both servers Online and one volume on each server.