CloudWatch Logs and Logs Insights

Start as usual. This lab adds two endpoints of its own, for CloudWatch Logs and metrics:

cd C:\aws-lab
aws login --profile saat-admin
$env:AWS_PROFILE = 'saat-admin'
$env:Path += ';C:\Program Files\Amazon\SessionManagerPlugin\bin'
$v = Get-Content C:\aws-lab\saat-vpc.json | ConvertFrom-Json
.\Start-SsmEndpoints.ps1
$extra = foreach ($svc in 'logs', 'monitoring') {
  aws ec2 create-vpc-endpoint --vpc-id $v.VpcId --vpc-endpoint-type Interface --service-name com.amazonaws.ap-south-1.$svc --subnet-ids $v.PrivateA --security-group-ids $v.EndpointSg --private-dns-enabled --tag-specifications "ResourceType=vpc-endpoint,Tags=[{Key=Name,Value=saat-$svc-ep},{Key=course,Value=saatvedha-aws},{Key=lesson,Value=24}]" --query VpcEndpoint.VpcEndpointId --output text
}
do { Start-Sleep 15; $s = aws ec2 describe-vpc-endpoints --vpc-endpoint-ids $extra --query "VpcEndpoints[].State" --output text } until ($s -notmatch 'pending')
$s

You should see this:

available       available

1Add a logs section to the agent config

Paste this whole block:

@'
{
  "agent": { "metrics_collection_interval": 60 },
  "metrics": {
    "namespace": "CWAgent",
    "append_dimensions": { "InstanceId": "${aws:InstanceId}", "InstanceType": "${aws:InstanceType}" },
    "metrics_collected": { "mem": { "measurement": [ "mem_used_percent" ] } }
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          { "file_path": "/var/log/saat-app.log", "log_group_name": "/saat/app", "log_stream_name": "{instance_id}", "retention_in_days": 7 }
        ]
      }
    }
  }
}
'@ | Set-Content cwagent.json
aws ssm put-parameter --name AmazonCloudWatch-saat-linux --type String --value file://cwagent.json --overwrite --query Version --output text

You should see something like this:

2

Version 2 of the parameter now tells the agent to tail the app log, name the stream after the instance and keep logs for 7 days.

2Launch log01 with the demo app

Paste this whole block:

$ud = @'
#!/bin/bash
dnf install -y amazon-cloudwatch-agent
cat > /usr/local/bin/saat-loggen.py <<'PY'
import json, random, time, datetime
paths = ['/', '/login', '/cart', '/checkout', '/api/orders']
codes = [200, 200, 200, 200, 200, 301, 404, 500, 503]
while True:
    s = random.choice(codes)
    rec = {"ts": datetime.datetime.now(datetime.timezone.utc).isoformat(), "level": "ERROR" if s >= 500 else "INFO",
           "path": random.choice(paths), "status": s, "ms": random.randint(5, 900), "user": "u" + str(random.randint(1, 40))}
    print(json.dumps(rec), flush=True)
    time.sleep(0.5)
PY
cat > /etc/systemd/system/saat-loggen.service <<'UNIT'
[Unit]
Description=saat demo log generator
[Service]
ExecStart=/usr/bin/python3 /usr/local/bin/saat-loggen.py
StandardOutput=append:/var/log/saat-app.log
Restart=always
[Install]
WantedBy=multi-user.target
UNIT
systemctl daemon-reload
systemctl enable --now saat-loggen
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c ssm:AmazonCloudWatch-saat-linux
'@
[IO.File]::WriteAllText('C:\aws-lab\log-userdata.sh', ($ud -replace "`r`n", "`n"))
$base = aws ssm get-parameter --name /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 --query Parameter.Value --output text
$i = aws ec2 run-instances --image-id $base --instance-type t3.micro --subnet-id $v.PrivateA --security-group-ids $v.AppSg --iam-instance-profile Name=saat-ec2-ssm-profile --metadata-options HttpTokens=required --user-data file://log-userdata.sh --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=log01},{Key=course,Value=saatvedha-aws},{Key=lesson,Value=24}]' --query "Instances[0].InstanceId" --output text
do { Start-Sleep 20; $streams = aws logs describe-log-streams --log-group-name /saat/app --query "logStreams[].logStreamName" --output text 2>$null } until ($streams)
$streams
aws logs describe-log-groups --log-group-name-prefix /saat --query "logGroups[].[logGroupName,retentionInDays]" --output text

You should see something like this (it takes three or four minutes):

i-0e1f2a3b4c5d60031
/saat/app       7

The agent created the log group, set 7-day retention and opened a stream named after the instance. The demo app writes two lines a second, about one in five of them errors.

3Query the logs

Paste the helper function first, then the queries. Wait two or three minutes after step 2 so there is data:

function Invoke-LogsQuery([string]$Group, [string]$Query, [int]$Minutes = 15) {
  $end = [DateTimeOffset]::UtcNow.ToUnixTimeSeconds()
  $qid = aws logs start-query --log-group-name $Group --start-time ($end - $Minutes * 60) --end-time $end --query-string $Query --query queryId --output text
  do { Start-Sleep 2; $r = aws logs get-query-results --query-id $qid --output json | ConvertFrom-Json } until ($r.status -in 'Complete', 'Failed', 'Cancelled')
  if (-not $r.results) { '(no rows)'; return }
  foreach ($row in $r.results) { ($row | Where-Object { $_.field -ne '@ptr' } | ForEach-Object { "$($_.field)=$($_.value)" }) -join '  ' }
}
Invoke-LogsQuery /saat/app 'fields @timestamp, path, status, ms | sort @timestamp desc | limit 5'
Invoke-LogsQuery /saat/app 'filter status >= 500 | stats count(*) as errors by path | sort errors desc'
Invoke-LogsQuery /saat/app 'stats count(*) as requests, pct(ms, 95) as p95_ms by bin(5m)'

You should see something like this (your numbers will differ):

@timestamp=2026-10-20 18:14:51.513  path=/cart  status=200  ms=118
@timestamp=2026-10-20 18:14:51.012  path=/checkout  status=503  ms=712
@timestamp=2026-10-20 18:14:50.511  path=/login  status=200  ms=64
@timestamp=2026-10-20 18:14:50.010  path=/api/orders  status=404  ms=331
@timestamp=2026-10-20 18:14:49.509  path=/  status=200  ms=27
path=/checkout  errors=31
path=/api/orders  errors=29
path=/login  errors=27
path=/cart  errors=26
path=/  errors=24
bin(5m)=2026-10-20 18:10:00.000  requests=600  p95_ms=856

You never told CloudWatch about status, path or ms; it found them in the JSON. The helper strips the internal @ptr field and prints each row on one line. The same queries work in the console under Logs Insights, with charts.

4Turn errors into a metric

Paste this whole block, then wait three minutes:

aws logs put-metric-filter --log-group-name /saat/app --filter-name app-5xx --filter-pattern '{ $.status >= 500 }' --metric-transformations metricName=App5xx,metricNamespace=Saat/App,metricValue=1,defaultValue=0
Start-Sleep 180
aws cloudwatch get-metric-statistics --namespace Saat/App --metric-name App5xx --start-time (Get-Date).ToUniversalTime().AddMinutes(-10).ToString('o') --end-time (Get-Date).ToUniversalTime().ToString('o') --period 60 --statistics Sum --query "sort_by(Datapoints, &Timestamp)[].Sum" --output text

You should see something like this:

0.0     24.0    25.0    23.0

The first minute is 0 because the filter only counts events that arrive after it’s created. App5xx is now an ordinary CloudWatch metric, so an alarm such as “more than 50 errors in 5 minutes, notify saat-alerts” is one put-metric-alarm away.

✅ Checkpoint: /saat/app receives logs with 7-day retention, the three queries return rows, and App5xx shows error counts per minute. Keep everything for the break lab.