EC2 and EBS Performance: Find the Ceiling

Sign in and launch perf61 with a separate gp3 data volume:

cd C:\aws-lab
aws login --profile saat-admin
$env:AWS_PROFILE = 'saat-admin'
$v = Get-Content C:\aws-lab\saat-vpc.json | ConvertFrom-Json
'[{"DeviceName":"/dev/sdf","Ebs":{"VolumeSize":20,"VolumeType":"gp3","Encrypted":true,"DeleteOnTermination":true}}]' | Set-Content bdm61.json
$ami = aws ssm get-parameter --name /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 --query Parameter.Value --output text
$id = aws ec2 run-instances --image-id $ami --instance-type t3.micro --credit-specification CpuCredits=unlimited --subnet-id $v.PublicA --security-group-ids $v.WebSg --iam-instance-profile Name=saat-ec2-ssm-profile --metadata-options HttpTokens=required --block-device-mappings file://bdm61.json --user-data "#!/bin/bash
dnf install -y fio" --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=perf61},{Key=course,Value=saatvedha-aws},{Key=lesson,Value=61}]' 'ResourceType=volume,Tags=[{Key=course,Value=saatvedha-aws},{Key=lesson,Value=61}]' --query "Instances[0].InstanceId" --output text
do { Start-Sleep 20; $p = aws ssm describe-instance-information --filters "Key=InstanceIds,Values=$id" --query "InstanceInformationList[0].PingStatus" --output text } until ($p -eq 'Online')
$vol = aws ec2 describe-instances --instance-ids $id --query "Reservations[0].Instances[0].BlockDeviceMappings[?DeviceName=='/dev/sdf'].Ebs.VolumeId | [0]" --output text
aws ec2 describe-volumes --volume-ids $vol --query "Volumes[0].[VolumeType,Size,Iops,Throughput]" --output text

You should see something like this:

gp3     20      3000    125

perf61 sits in a public subnet so it reaches Systems Manager without endpoints; it has no inbound rules. The data volume has gp3’s default 3,000 IOPS and 125 MiB/s.

1Measure CPU

Paste this whole block. It adds helpers, then runs a single-core benchmark and reads CPU steal:

function Invoke-Linux([string]$Id, [string[]]$Lines) {
  @{ commands = $Lines; executionTimeout = @('900') } | ConvertTo-Json | Set-Content lx-cmd.json
  $cmd = aws ssm send-command --document-name AWS-RunShellScript --instance-ids $Id --parameters file://lx-cmd.json --query Command.CommandId --output text
  do { Start-Sleep 5; $st = aws ssm get-command-invocation --command-id $cmd --instance-id $Id --query Status --output text 2>$null } until ($st -in 'Success', 'Failed', 'TimedOut')
  aws ssm get-command-invocation --command-id $cmd --instance-id $Id --query StandardOutputContent --output text
}
$bench = 'python3 -c "import time; t = time.time(); s = sum(i * i for i in range(30000000)); print(round(time.time() - t, 1), \"seconds\")"; top -bn2 -d1 | grep "%Cpu" | tail -1'
$parse = ' --output-format=json | python3 -c ''import json, sys; j = json.load(sys.stdin)["jobs"][0]["write"]; print(round(j["bw"] / 1024), "MiB/s", round(j["iops"]), "IOPS")'''
$seq = 'fio --name=seq --filename=/dev/nvme1n1 --rw=write --bs=1M --direct=1 --ioengine=libaio --iodepth=32 --runtime=30 --time_based' + $parse
$rand = 'fio --name=rand --filename=/dev/nvme1n1 --rw=randwrite --bs=4k --direct=1 --ioengine=libaio --iodepth=64 --runtime=30 --time_based' + $parse
function Get-Metric([string]$Ns, [string]$Name, [string]$DimName, [string]$DimValue, [string]$Stat = 'Maximum') {
  $e = (Get-Date).ToUniversalTime(); $s = $e.AddMinutes(-10)
  aws cloudwatch get-metric-statistics --namespace $Ns --metric-name $Name --dimensions "Name=$DimName,Value=$DimValue" --start-time $s.ToString('o') --end-time $e.ToString('o') --period 60 --statistics $Stat --query "max_by(Datapoints, &Timestamp).$Stat" --output text
}
Invoke-Linux $id @($bench)

You should see something like this:

2.9 seconds
%Cpu(s): 50.2 us,  0.3 sy,  0.0 ni, 49.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

About 3 seconds, one of two vCPUs busy, and 0.0 st (steal). This is your baseline. In unlimited mode, a t3 can run at 100% as long as it likes; you just pay for the surplus.

2Measure the data volume

Paste this whole block. fio writes to the empty data volume for 30 seconds each, in big and small blocks:

Invoke-Linux $id @('lsblk -o NAME,SIZE,MOUNTPOINT /dev/nvme1n1', $seq, $rand)
"VolumeThroughputExceededCheck = $(Get-Metric AWS/EBS VolumeThroughputExceededCheck VolumeId $vol)"

You should see something like this:

NAME    SIZE MOUNTPOINT
nvme1n1  20G
126 MiB/s 126 IOPS
12 MiB/s 3001 IOPS
VolumeThroughputExceededCheck = 1.0

Big blocks stop at gp3’s 125 MiB/s; small blocks stop at 3,000 IOPS. Two different limits on the same volume, and CloudWatch confirms the app tried to go faster. fio on a raw device is only safe on an empty volume like this one.

3Read the CloudWatch side

Paste this whole block:

"CPUUtilization max   = $(Get-Metric AWS/EC2 CPUUtilization InstanceId $id)"
"CPUCreditBalance     = $(Get-Metric AWS/EC2 CPUCreditBalance InstanceId $id Average)"
"VolumeAvgWriteLatency = $(Get-Metric AWS/EBS VolumeAvgWriteLatency VolumeId $vol) ms"
"InstanceEBSThroughputExceededCheck = $(Get-Metric AWS/EC2 InstanceEBSThroughputExceededCheck InstanceId $id)"

You should see something like this:

CPUUtilization max   = 50.8
CPUCreditBalance     = 4.2
VolumeAvgWriteLatency = 10.4 ms
InstanceEBSThroughputExceededCheck = 0.0

The volume, not the instance, was the ceiling during fio. Credit metrics arrive every 5 minutes, so the balance may show None at first; run it again later.

✅ Checkpoint: The benchmark takes a few seconds with 0 steal, fio stops at about 125 MiB/s and 3,000 IOPS, and VolumeThroughputExceededCheck shows 1.