CloudWatch metrics, alarms and dashboards

Start as usual, then set your email address:

cd C:\aws-lab
aws login --profile saat-admin
$env:AWS_PROFILE = 'saat-admin'
$env:Path += ';C:\Program Files\Amazon\SessionManagerPlugin\bin'
$v = Get-Content C:\aws-lab\saat-vpc.json | ConvertFrom-Json
$email = 'you@example.com'
.\Start-SsmEndpoints.ps1

1Create the alert topic and confirm your email

Paste this whole block, then open your inbox:

$topic = aws sns create-topic --name saat-alerts --tags Key=course,Value=saatvedha-aws --query TopicArn --output text
aws sns subscribe --topic-arn $topic --protocol email --notification-endpoint $email --query SubscriptionArn --output text

You should see this:

pending confirmation

Click Confirm subscription in the email from AWS Notifications. Until you do, every alarm will “work” and you will receive nothing. Check it:

aws sns list-subscriptions-by-topic --topic-arn $topic --query "Subscriptions[0].SubscriptionArn" --output text

After confirming, you should see something like this:

arn:aws:sns:ap-south-1:111122223333:saat-alerts:6f1c2d3e-4a5b-6c7d-8e9f-0a1b2c3d4e5f

Keep saat-alerts. Budgets, GuardDuty and backups will send to it in later lessons.

2Prepare the CloudWatch agent

Paste this whole block. It lets the role publish metrics, stores the agent config and adds a monitoring endpoint:

aws iam attach-role-policy --role-name saat-ec2-ssm-role --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
@'
{
  "agent": { "metrics_collection_interval": 60 },
  "metrics": {
    "namespace": "CWAgent",
    "append_dimensions": { "InstanceId": "${aws:InstanceId}", "InstanceType": "${aws:InstanceType}" },
    "metrics_collected": {
      "mem": { "measurement": [ "mem_used_percent" ] }
    }
  }
}
'@ | Set-Content cwagent.json
aws ssm put-parameter --name AmazonCloudWatch-saat-linux --type String --value file://cwagent.json --overwrite | Out-Null
$mon = aws ec2 create-vpc-endpoint --vpc-id $v.VpcId --vpc-endpoint-type Interface --service-name com.amazonaws.ap-south-1.monitoring --subnet-ids $v.PrivateA --security-group-ids $v.EndpointSg --private-dns-enabled --tag-specifications 'ResourceType=vpc-endpoint,Tags=[{Key=Name,Value=saat-monitoring-ep},{Key=course,Value=saatvedha-aws},{Key=lesson,Value=23}]' --query VpcEndpoint.VpcEndpointId --output text
do { Start-Sleep 15; $s = aws ec2 describe-vpc-endpoints --vpc-endpoint-ids $mon --query "VpcEndpoints[0].State" --output text } until ($s -eq 'available')
$s

You should see this:

available

Private servers have no internet, so the agent reaches CloudWatch through this endpoint. The policy and the parameter stay for Lesson 24; the endpoint is deleted at the end.

3Launch mon01 and build the CPU alarm

Paste this whole block:

[IO.File]::WriteAllText('C:\aws-lab\mon-userdata.sh', "#!/bin/bash`ndnf install -y amazon-cloudwatch-agent`n/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c ssm:AmazonCloudWatch-saat-linux`n")
$base = aws ssm get-parameter --name /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 --query Parameter.Value --output text
$i = aws ec2 run-instances --image-id $base --instance-type t3.micro --subnet-id $v.PrivateA --security-group-ids $v.AppSg --iam-instance-profile Name=saat-ec2-ssm-profile --metadata-options HttpTokens=required --monitoring Enabled=true --user-data file://mon-userdata.sh --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=mon01},{Key=course,Value=saatvedha-aws},{Key=lesson,Value=23}]' --query "Instances[0].InstanceId" --output text
aws cloudwatch put-metric-alarm --alarm-name saat-mon01-cpu-high --alarm-description "mon01 CPU above 40% for 2 of 3 minutes" --namespace AWS/EC2 --metric-name CPUUtilization --dimensions "Name=InstanceId,Value=$i" --statistic Average --period 60 --evaluation-periods 3 --datapoints-to-alarm 2 --threshold 40 --comparison-operator GreaterThanThreshold --treat-missing-data missing --alarm-actions $topic --ok-actions $topic
aws cloudwatch set-alarm-state --alarm-name saat-mon01-cpu-high --state-value ALARM --state-reason "Testing the alert path"
aws cloudwatch describe-alarms --alarm-names saat-mon01-cpu-high --query "MetricAlarms[0].[StateValue,StateReason]" --output text

You should see this, and an ALARM email within a minute:

ALARM   Testing the alert path

set-alarm-state is how teams test that alerts actually reach people, without breaking anything. A minute later the alarm re-evaluates real data and sends an OK email.

4Cause real load, then build the dashboard

Paste this whole block. The load command keeps the prompt busy for five minutes:

do { Start-Sleep 20; $p = aws ssm describe-instance-information --filters "Key=InstanceIds,Values=$i" --query "InstanceInformationList[0].PingStatus" --output text } until ($p -eq 'Online')
aws ssm start-session --target $i --document-name AWS-StartInteractiveCommand --parameters command="timeout 300 yes > /dev/null"
Start-Sleep 90
aws cloudwatch describe-alarm-history --alarm-name saat-mon01-cpu-high --history-item-type StateUpdate --query "AlarmHistoryItems[].HistorySummary" --output text
$alarmArn = aws cloudwatch describe-alarms --alarm-names saat-mon01-cpu-high --query "MetricAlarms[0].AlarmArn" --output text
@'
{
  "widgets": [
    { "type": "metric", "x": 0, "y": 0, "width": 12, "height": 6,
      "properties": { "title": "mon01 CPU %", "region": "ap-south-1", "stat": "Average", "period": 60,
        "metrics": [ [ "AWS/EC2", "CPUUtilization", "InstanceId", "INSTANCE_ID" ] ] } },
    { "type": "metric", "x": 12, "y": 0, "width": 12, "height": 6,
      "properties": { "title": "mon01 memory %", "region": "ap-south-1", "stat": "Average", "period": 60,
        "metrics": [ [ "CWAgent", "mem_used_percent", "InstanceId", "INSTANCE_ID", "InstanceType", "t3.micro" ] ] } },
    { "type": "alarm", "x": 0, "y": 6, "width": 24, "height": 3,
      "properties": { "title": "Alarms", "alarms": [ "ALARM_ARN" ] } }
  ]
}
'@ -replace 'INSTANCE_ID', $i -replace 'ALARM_ARN', $alarmArn | Set-Content dash.json
aws cloudwatch put-dashboard --dashboard-name saat-ops --dashboard-body file://dash.json

You should see something like this (newest first):

Alarm updated from ALARM to OK   Alarm updated from OK to ALARM   Alarm updated from ALARM to OK   Alarm updated from OK to ALARM   Alarm updated from INSUFFICIENT_DATA to OK
{
    "DashboardValidationMessages": []
}

yes keeps one of the two vCPUs busy, so CPU sat near 50 percent and the alarm fired for real; you got a second ALARM email and then OK. Open CloudWatch, then Dashboards, then saat-ops to see CPU, memory and the alarm on one screen.

✅ Checkpoint: saat-alerts has a confirmed email, the CPU alarm fired from set-alarm-state and from real load and returned to OK, and saat-ops shows CPU, memory and the alarm. Keep everything for the break lab.