The AWS Troubleshooting Workflow: What Changed?

Sign in and set names:

cd C:\aws-lab
aws login --profile saat-admin
$env:AWS_PROFILE = 'saat-admin'
$v = Get-Content C:\aws-lab\saat-vpc.json | ConvertFrom-Json
$acct = aws sts get-caller-identity --query Account --output text

1Build the app and a teammate’s role

Paste this whole block. It creates app01 with a security group rule and a config parameter, plus a limited role that a “teammate” can use:

$sg = aws ec2 create-security-group --group-name saat-app01-sg --description "app01 HTTPS from web tier" --vpc-id $v.VpcId --query GroupId --output text
aws ec2 authorize-security-group-ingress --group-id $sg --ip-permissions "IpProtocol=tcp,FromPort=443,ToPort=443,UserIdGroupPairs=[{GroupId=$($v.WebSg),Description=443-from-web}]" | Out-Null
$ami = aws ssm get-parameter --name /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 --query Parameter.Value --output text
$app = aws ec2 run-instances --image-id $ami --instance-type t3.micro --subnet-id $v.PrivateA --security-group-ids $sg --metadata-options HttpTokens=required --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=app01},{Key=course,Value=saatvedha-aws},{Key=lesson,Value=56}]' --query "Instances[0].InstanceId" --output text
aws ssm put-parameter --name /saat/app01/db-endpoint --type String --value 'saat-sql.internal:1433' --overwrite | Out-Null
@{ Version = '2012-10-17'; Statement = @(@{ Effect = 'Allow'; Principal = @{ AWS = "arn:aws:iam::$($acct):root" }; Action = 'sts:AssumeRole' }) } | ConvertTo-Json -Depth 5 | Set-Content intern-trust.json
@{ Version = '2012-10-17'; Statement = @(
  @{ Effect = 'Allow'; Action = @('ec2:Describe*', 'ec2:StopInstances', 'ec2:RevokeSecurityGroupIngress'); Resource = '*' },
  @{ Effect = 'Allow'; Action = 'ssm:PutParameter'; Resource = "arn:aws:ssm:ap-south-1:$($acct):parameter/saat/app01/*" }) } | ConvertTo-Json -Depth 5 | Set-Content intern-policy.json
$roleArn = aws iam create-role --role-name saat-lab-intern --assume-role-policy-document file://intern-trust.json --query Role.Arn --output text
aws iam put-role-policy --role-name saat-lab-intern --policy-name intern-ops --policy-document file://intern-policy.json
aws configure set profile.saat-intern.role_arn $roleArn
aws configure set profile.saat-intern.source_profile saat-admin
aws configure set profile.saat-intern.role_session_name priya-intern
aws configure set profile.saat-intern.region ap-south-1
aws ec2 wait instance-running --instance-ids $app
"$app $sg"

You should see something like this:

i-0b2c3d4e5f6a70819 sg-0c1d2e3f4a5b6c7d8

The saat-intern profile assumes saat-lab-intern with the session name priya-intern, which is what CloudTrail will record. Using named sessions like this is why real teams can tell people apart in CloudTrail.

2Stage the incident

Run these as the teammate. Then pretend you know nothing about them:

Start-Sleep 20
aws ec2 revoke-security-group-ingress --group-id $sg --ip-permissions "IpProtocol=tcp,FromPort=443,ToPort=443,UserIdGroupPairs=[{GroupId=$($v.WebSg)}]" --profile saat-intern | Out-Null
aws ssm put-parameter --name /saat/app01/db-endpoint --value 'old-sql.internal:1433' --overwrite --profile saat-intern | Out-Null
aws ec2 stop-instances --instance-ids $app --profile saat-intern | Out-Null
"Incident staged at $((Get-Date).ToUniversalTime().ToString('HH:mm')) UTC"

You should see something like this:

Incident staged at 05:01 UTC

The ticket you’d get says only: “app01 is down since about 10:30”. Now work it like the on-call engineer.

3Scope it, check AWS, and list what changed

Paste this whole block. It saves Get-RecentChanges.ps1 (you’ll reuse it all through Phase 4), then runs the first three questions:

$grc = @'
param([int]$Minutes = 60, [string]$Region = 'ap-south-1', [string]$Who = '')
$start = (Get-Date).ToUniversalTime().AddMinutes(-$Minutes).ToString('o')
$raw = aws cloudtrail lookup-events --region $Region --lookup-attributes AttributeKey=ReadOnly,AttributeValue=false --start-time $start --max-results 200 --query "Events[].CloudTrailEvent" --output json | ConvertFrom-Json
$rows = foreach ($r in $raw) {
  $e = $r | ConvertFrom-Json
  if ($e.userIdentity.type -eq 'AWSService') { continue }
  [pscustomobject]@{
    Time   = ([datetime]$e.eventTime).ToLocalTime().ToString('HH:mm:ss')
    Who    = ($e.userIdentity.arn -split '/')[-1]
    Action = ($e.eventSource -replace '\.amazonaws\.com$', '') + '/' + $e.eventName
    Error  = $e.errorCode
  }
}
if ($Who) { $rows = $rows | Where-Object { $_.Who -like "*$Who*" } }
if ($rows) { $rows | Format-Table -AutoSize } else { "No write events in the last $Minutes minutes in $Region." }
'@
$grc | Set-Content C:\aws-lab\Get-RecentChanges.ps1
aws ec2 describe-instances --instance-ids $app --query "Reservations[0].Instances[0].[State.Name,StateTransitionReason]" --output text
Start-Sleep 300
.\Get-RecentChanges.ps1 -Minutes 30 -Who intern

You should see something like this:

stopped User initiated (2026-10-20 05:01:09 GMT)

Time     Who          Action                         Error
----     ---          ------                         -----
10:31:09 priya-intern ec2/StopInstances
10:31:04 priya-intern ssm/PutParameter
10:31:01 priya-intern ec2/RevokeSecurityGroupIngress

“User initiated” tells you it wasn’t a hardware failure. CloudTrail tells you who, what and when, in local time thanks to the script. For question 2, open AWS Health Dashboard > Your account health: no open issues means AWS isn’t the cause.

4Fix it and see the old values

Paste this whole block. Parameter history shows what the value used to be:

aws ssm get-parameter-history --name /saat/app01/db-endpoint --query "Parameters[].[Version,Value,LastModifiedUser]" --output text
aws ssm put-parameter --name /saat/app01/db-endpoint --value 'saat-sql.internal:1433' --overwrite | Out-Null
aws ec2 authorize-security-group-ingress --group-id $sg --ip-permissions "IpProtocol=tcp,FromPort=443,ToPort=443,UserIdGroupPairs=[{GroupId=$($v.WebSg),Description=443-from-web}]" | Out-Null
aws ec2 start-instances --instance-ids $app | Out-Null
aws ec2 wait instance-running --instance-ids $app
aws ec2 describe-instances --instance-ids $app --query "Reservations[0].Instances[0].State.Name" --output text

You should see something like this:

1       saat-sql.internal:1433  arn:aws:sts::111122223333:assumed-role/AWSReservedSSO_.../saat-admin
2       old-sql.internal:1433   arn:aws:sts::111122223333:assumed-role/saat-lab-intern/priya-intern
running

All three changes are reverted. The “prevent” part is a conversation, not a command: why could this role stop production instances, and should changes like these go through a pipeline?

✅ Checkpoint: StateTransitionReason says User initiated, Get-RecentChanges.ps1 lists the three priya-intern events, parameter history shows version 2 by priya-intern, and app01 is running with its rule and value restored.