Azure VM performance: credits and disk caps

Both breaks need time for credits to drain. Start each one, then wait as shown.

1Break it: the batch job that slows down after half an hour

Start a CPU-heavy job for one hour, wait 45 minutes, then look again:

az vm run-command invoke --resource-group rg-saat-l72 --name vm-perf --command-id RunShellScript --scripts 'systemd-run --unit burn stress-ng --cpu 1 --timeout 3600s; echo started' --query "value[0].message" -o tsv
Start-Sleep -Seconds 2700
az monitor metrics list --resource $vmId --metric "CPU Credits Remaining" "Percentage CPU" --interval PT5M --aggregation Average --query "value[].{metric:name.value, latest:timeseries[0].data[-2].average}" -o table
az vm run-command invoke --resource-group rg-saat-l72 --name vm-perf --command-id RunShellScript --scripts 'vmstat 5 2 | tail -1' --query "value[0].message" -o tsv

You should see something like this:

Metric                 Latest
---------------------  ------
CPU Credits Remaining  0.0
Percentage CPU         10.2
[stdout]
 1  0      0 498112  21500 250012    0    0     0     4  210  160 10  0  0  0 90

The job still wants 100 percent, but the VM runs at about 10 percent with steal time near 90: the hypervisor is holding it at the B1s baseline because the credits are gone. In the first 20 to 30 minutes the same job ran at full speed, which is how it passed testing.

Fix it: a steady CPU load needs a size whose baseline is 100 percent. Stop the job and resize to a non-burstable size (the VM restarts):

az vm run-command invoke --resource-group rg-saat-l72 --name vm-perf --command-id RunShellScript --scripts 'systemctl stop burn; echo stopped' --query "value[0].message" -o tsv
az vm resize --resource-group rg-saat-l72 --name vm-perf --size Standard_D2as_v5 -o none
az vm show --resource-group rg-saat-l72 --name vm-perf --query hardwareProfile.vmSize -o tsv

You should see Standard_D2as_v5. B-series is still a good choice for VMs that are mostly idle, such as small web servers and jump boxes.

2Break it: the disk that was fast on day one

Run a steady random-read test against the P4 disk for 45 minutes, reporting every 5 minutes, then read the start and the end:

az vm run-command invoke --resource-group rg-saat-l72 --name vm-perf --command-id RunShellScript --scripts 'systemd-run --unit io fio --name=steady --filename=/dev/disk/azure/scsi1/lun0 --rw=randread --bs=4k --iodepth=32 --ioengine=libaio --direct=1 --runtime=2700 --time_based --status-interval=300 --output=/root/fio.log; echo started' --query "value[0].message" -o tsv
Start-Sleep -Seconds 2760
az vm run-command invoke --resource-group rg-saat-l72 --name vm-perf --command-id RunShellScript --scripts 'grep -E read:.IOPS= /root/fio.log | head -2; grep -E read:.IOPS= /root/fio.log | tail -2' --query "value[0].message" -o tsv
az monitor metrics list --resource $vmId --metric "Data Disk Used Burst IO Credits Percentage" --interval PT5M --aggregation Average --query "value[].{metric:name.value, latest:timeseries[0].data[-2].average}" -o table

You should see something like this:

[stdout]
  read: IOPS=3496, BW=13.7MiB/s (14.3MB/s)(4097MiB/300001msec)
  read: IOPS=3412, BW=13.3MiB/s (13.9MB/s)(3998MiB/300002msec)
  read: IOPS=121, BW=484KiB/s (496kB/s)(142MiB/300003msec)
  read: IOPS=120, BW=482KiB/s (493kB/s)(141MiB/300001msec)
Metric                                     Latest
-----------------------------------------  ------
Data Disk Used Burst IO Credits Percentage  100.0

For about half an hour the disk burst to around 3,500 IOPS. When its burst credits were used up, it dropped to the P4 baseline of 120 IOPS. Anyone who benchmarked it for five minutes would have expected 3,500 forever.

Fix it: raise the disk’s performance tier to match the sustained need, without resizing or detaching it, and test again for 5 minutes:

az disk update --resource-group rg-saat-l72 --name data-p4 --performance-tier P30 -o none
az disk show --resource-group rg-saat-l72 --name data-p4 --query "{tier:tier, iops:diskIOPSReadWrite, mbps:diskMBpsReadWrite}" -o table
az vm run-command invoke --resource-group rg-saat-l72 --name vm-perf --command-id RunShellScript --scripts 'fio --name=check --filename=/dev/disk/azure/scsi1/lun0 --rw=randread --bs=4k --iodepth=32 --ioengine=libaio --direct=1 --runtime=300 --time_based --group_reporting | grep IOPS=' --query "value[0].message" -o tsv

You should see the tier P30 with 5,000 IOPS, and fio showing a steady result in the thousands. On D2as_v5 the VM’s own uncached limit, about 3,750 IOPS, may now be the ceiling, which you can confirm with VM Uncached IOPS Consumed Percentage.

3Clean up

Delete the lab resource group:

az group delete --name rg-saat-l72 --yes

✅ Checkpoint: You proved a CPU credit throttle and a disk burst throttle with metrics, and fixed each by sizing for sustained load.