Extend LVM online: hot-add a disk and grow volumes live

You’ll switch between two terminals: one on your laptop for VBoxManage, and an SSH session to rocky01 as student. Keep rocky01 running the whole time; the point of the lab is that nothing on it stops. Commands for the laptop are shown for Windows; on macOS or Linux use ~/Lab/Linux/rocky01/rocky01-data3.vdi as the path and grep instead of findstr.

1See where you are starting from

On rocky01, run this:

df -h /srv/app /srv/logs
sudo vgs vg_data
sudo lvs vg_data

You should see something like this:

Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/vg_data-lv_app   2.0G   71M  1.9G   4% /srv/app
/dev/mapper/vg_data-lv_logs  974M   24K  907M   1% /srv/logs
  VG      #PV #LV #SN Attr   VSize  VFree
  vg_data   1   2   0 wz--n- <5.00g <2.00g
  LV      VG      Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv_app  vg_data -wi-ao---- 2.00g
  lv_logs vg_data -wi-ao---- 1.00g

This is the state Lesson 44 left: lv_app is 2 GiB, lv_logs is 1 GiB, and vg_data has a little under 2 GiB free. You could grow lv_app by 2 GiB from that free space alone, but here you’ll add a disk first, which is the usual situation at work when a VG has run out.

✅ Checkpoint: df shows /srv/app at 2.0G and vgs shows VFree <2.00g.

2Start a writer that keeps the app busy

Create the script (paste this whole block):

mkdir -p ~/lab45
cat > ~/lab45/writer.sh <<'EOF'
#!/bin/bash
# writer.sh: appends a timestamp to /srv/app/heartbeat.log every second, like a busy app
while true; do
  date +%s.%N >> /srv/app/heartbeat.log
  sleep 1
done
EOF
bash -n ~/lab45/writer.sh && echo syntax ok

Start it in the background as root, then look at the log:

sudo -v
sudo bash ~/lab45/writer.sh &
sleep 3; tail -n 3 /srv/app/heartbeat.log

You should see something like this:

[1] 2318
1793081525.402118734
1793081526.406731902
1793081527.411022157

Each line is a Unix timestamp with nanoseconds, written once a second to a file on lv_app. sudo -v asks for your password first, so the background sudo doesn’t sit waiting for it. Leave this running until step 8.

✅ Checkpoint: wc -l /srv/app/heartbeat.log gives a bigger number every time you run it.

3Hot-add a 5 GB disk to the running VM

On the laptop, run this:

VBoxManage createmedium disk --filename="C:\Lab\Linux\rocky01\rocky01-data3.vdi" --size=5120 --format=VDI
VBoxManage storageattach rocky01 --storagectl=SATA --port=3 --device=0 --type=hdd --medium="C:\Lab\Linux\rocky01\rocky01-data3.vdi"
VBoxManage showvminfo rocky01 --machinereadable | findstr SATA-3-0

You should see something like this:

0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Medium created. UUID: 9e4b7c1d-2f6a-4d83-b5e0-7a1c3f8d6e42
"SATA-3-0"="C:\Lab\Linux\rocky01\rocky01-data3.vdi"

createmedium made a 5120 MB dynamically allocated VDI, and storageattach plugged it into SATA port 3 while rocky01 kept running, with no output when it works. Don’t add --hotpluggable on here: VirtualBox marks a disk attached at run time as hot-pluggable by itself, and asking to change that flag on a running VM fails with “The machine is not mutable (state is Running)”, after the disk is already attached, which is confusing.

If storageattach says the controller doesn’t support hot-plugging, your disks are on an IDE or SCSI controller instead of SATA. Lesson 2 created a SATA controller with four ports; check the name with VBoxManage showvminfo rocky01.

✅ Checkpoint: showvminfo lists the new VDI on SATA-3-0 and rocky01 is still running.

4Find the new disk on rocky01

Back on rocky01, rescan every SCSI host and list the disks:

ls /sys/class/scsi_host/
for h in /sys/class/scsi_host/host*; do echo "- - -" | sudo tee "$h/scan" >/dev/null; done
lsblk -S -o NAME,HCTL,TYPE,MODEL,SIZE,TRAN

You should see something like this:

host0  host1  host2  host3  host4  host5

NAME HCTL    TYPE MODEL          SIZE TRAN
sda  2:0:0:0 disk VBOX HARDDISK   20G sata
sdb  3:0:0:0 disk VBOX HARDDISK    5G sata
sdc  4:0:0:0 disk VBOX HARDDISK    5G sata
sdd  5:0:0:0 disk VBOX HARDDISK    5G sata
sr0  1:0:0:0 rom  VBOX CD-ROM   1024M ata

Six host adapters: two for the IDE controller that holds the DVD drive and four for the SATA ports. sdd is the new disk at H:C:T:L 5:0:0:0, which means host 5, channel 0, target 0, LUN 0. Your host numbers can differ, which is why the loop scans all of them instead of guessing. Now see what the kernel logged:

sudo dmesg | tail -n 8

You should see something like this:

[ 3712.418236] ata6: irq_stat 0x00400040, connection status changed
[ 3712.418245] ata6: hard resetting link
[ 3713.150612] ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 3713.151048] ata6.00: ATA-6: VBOX HARDDISK, 1.0, max UDMA/133
[ 3713.151052] ata6.00: 10485760 sectors, multi 128: LBA48 NCQ (depth 32)
[ 3713.151790] scsi 5:0:0:0: Direct-Access     ATA      VBOX HARDDISK    1.0  PQ: 0 ANSI: 5
[ 3713.152279] sd 5:0:0:0: [sdd] 10485760 512-byte logical blocks: (5.37 GB/5.00 GiB)
[ 3713.153190] sd 5:0:0:0: [sdd] Attached SCSI disk

The AHCI driver noticed the new drive on its own (“connection status changed”) before you even ran the rescan, which is normal for VirtualBox SATA. On VMware, Hyper-V or a SAN you often get no such message until you rescan, so the rescan stays in your routine.

Try it yourself: build the rescan command for new or resized disks in the Command Builder.

✅ Checkpoint: lsblk shows sdd, 5G, with no partitions and no file system.

5Add sdd to vg_data

Run this:

sudo pvcreate /dev/sdd
sudo vgextend vg_data /dev/sdd
sudo pvs
sudo vgs vg_data

You should see something like this:

  Physical volume "/dev/sdd" successfully created.
  Volume group "vg_data" successfully extended
  PV         VG      Fmt  Attr PSize  PFree
  /dev/sda3  rl      lvm2 a--  18.41g     0
  /dev/sdb   vg_data lvm2 a--  <5.00g <2.00g
  /dev/sdd   vg_data lvm2 a--  <5.00g <5.00g
  VG      #PV #LV #SN Attr   VSize VFree
  vg_data   2   2   0 wz--n- 9.99g 6.99g

vg_data now has two PVs and its size doubled to 9.99g, with 6.99g free: the 2 GiB that was left on sdb plus the whole of sdd. Nothing about lv_app or lv_logs has changed yet, and df still shows the old sizes.

✅ Checkpoint: vgs shows vg_data with #PV 2, VSize 9.99g and VFree 6.99g.

6Grow lv_app and its XFS in one command

Run this:

sudo lvextend -r -L +2G /dev/vg_data/lv_app
df -h /srv/app

You should see something like this (xfs_growfs output shortened):

  File system xfs found on vg_data/lv_app mounted at /srv/app.
  Size of logical volume vg_data/lv_app changed from 2.00 GiB (512 extents) to 4.00 GiB (1024 extents).
  Extending file system xfs to 4.00 GiB (4294967296 bytes) on vg_data/lv_app...
xfs_growfs /dev/vg_data/lv_app
meta-data=/dev/mapper/vg_data-lv_app isize=512    agcount=4, agsize=131072 blks
         =                       sectsz=512   attr=2, projid32bit=1
...
data blocks changed from 524288 to 1048576
xfs_growfs done
  Extended file system xfs on vg_data/lv_app.
  Logical volume vg_data/lv_app successfully resized.
Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/vg_data-lv_app  4.0G  110M  3.9G   3% /srv/app

Read it top to bottom: LVM found XFS mounted on /srv/app, added 512 extents, then ran xfs_growfs, which reported the data section going from 524288 to 1048576 blocks of 4 KiB. df shows 4.0G straight away, and the writer kept going the whole time. -L +2G means “2 GiB more”; -L 4G would mean “make it 4 GiB in total”, and -l +100%FREE would take everything left.

Try it yourself: build the grow-an-LV command (new disk, grown disk, or free space) in the Command Builder.

✅ Checkpoint: df -h /srv/app shows 4.0G and tail -n 1 /srv/app/heartbeat.log shows a timestamp from the last second or two.

7Grow lv_logs and ext4 in two steps

First grow only the LV:

sudo lvextend -L +512M /dev/vg_data/lv_logs
df -h /srv/logs

You should see something like this:

  Size of logical volume vg_data/lv_logs changed from 1.00 GiB (256 extents) to 1.50 GiB (384 extents).
  Logical volume vg_data/lv_logs successfully resized.
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/vg_data-lv_logs  974M   24K  907M   1% /srv/logs

The LV is now 1.5 GiB, but without -r LVM didn’t touch the file system, so df still shows 974M. Grow the file system yourself:

sudo resize2fs /dev/vg_data/lv_logs
df -h /srv/logs

You should see something like this:

resize2fs 1.47.1 (20-May-2024)
Filesystem at /dev/vg_data/lv_logs is mounted on /srv/logs; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/vg_data/lv_logs is now 393216 (4k) blocks long.

Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/vg_data-lv_logs  1.5G   24K  1.4G   1% /srv/logs

resize2fs grew ext4 while it was mounted (“on-line resizing”) to 393216 blocks of 4 KiB, which is 1.5 GiB. For XFS the second command would be sudo xfs_growfs /srv/app. Note the difference: resize2fs wants the device, while xfs_growfs is documented with the mount point.

Look at the result in LVM:

sudo lvs vg_data
sudo vgs vg_data
sudo lvs -o lv_name,lv_size,seg_le_ranges vg_data

You should see something like this:

  LV      VG      Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv_app  vg_data -wi-ao---- 4.00g
  lv_logs vg_data -wi-ao---- 1.50g
  VG      #PV #LV #SN Attr   VSize VFree
  vg_data   2   2   0 wz--n- 9.99g 4.49g
  LV      LSize LE Ranges
  lv_app  4.00g /dev/sdb:0-511
  lv_app  4.00g /dev/sdb:768-1278
  lv_app  4.00g /dev/sdd:0-0
  lv_logs 1.50g /dev/sdb:512-767
  lv_logs 1.50g /dev/sdd:1-128   

lv_app used the rest of sdb and its first extent on sdd, and lv_logs grew onto sdd. Both LVs now span two disks, which is how LVM is meant to work, and 4.49g is still free.

✅ Checkpoint: df shows /srv/logs at 1.5G and vgs shows VFree 4.49g.

8Prove the app never stopped

Stop the writer and check the biggest gap between two lines:

sudo pkill -f 'lab45/[w]riter.sh'
awk '{if (NR > 1 && $1 - p > g) g = $1 - p; p = $1} END {printf "%d lines, biggest gap %.2f s\n", NR, g}' /srv/app/heartbeat.log

You should see something like this:

[1]+  Terminated              sudo bash ~/lab45/writer.sh
362 lines, biggest gap 1.01 s

The square brackets in 'lab45/[w]riter.sh' are a small trick: the pattern still matches the writer, but not pkill’s own command line. The awk program compares each timestamp with the one before it and keeps the largest difference. About 1.0 seconds is the loop itself; if the grow had frozen I/O you would see a gap of several seconds, and there isn’t one.

✅ Checkpoint: awk reports a biggest gap close to 1 second, over a few hundred lines.

9Practise the grown-disk path on sdb

At work the disk under a PV is often made bigger instead. sdb hasn’t changed, so this is a harmless rehearsal:

readlink -f /sys/class/block/sdb/device
echo 1 | sudo tee /sys/class/block/sdb/device/rescan
sudo pvresize /dev/sdb
sudo pvs /dev/sdb

You should see something like this:

/sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0
1
  Physical volume "/dev/sdb" changed
  1 physical volume(s) resized or updated / 0 physical volume(s) not resized
  PV         VG      Fmt  Attr PSize  PFree
  /dev/sdb   vg_data lvm2 a--  <5.00g    0 

The readlink shows the H:C:T:L address (3:0:0:0) behind sdb, so you know which SCSI device you’re poking. The rescan made the kernel read sdb’s size again (tee echoes the 1), and pvresize re-read the PV and found nothing new, so PFree stays 0. If the disk had grown, pvs would now show the extra space as PFree and you’d finish with lvextend -r as in step 6.

✅ Checkpoint: pvresize reports 1 physical volume resized or updated, and nothing else about vg_data changed.