DevOps Rescue Story: Recovering an EC2 Instance Without a PEM Key

"Lost PEM? No SSH? SSM dead? Don't panic — AWS always leaves a backdoor for those who know where to look."

Yesterday I ran into one of those heart-sinking moments: an EC2 instance was completely locked out. - PEM key gone → SSH impossible - SSM agent broken → root volume full, wouldn't start even after EBS expansion - EC2 Instance Connect failing

Basically… the instance was bricked. Or so it seemed.

The Recovery Playbook I Followed

• Spun up a helper EC2 instance with a fresh key pair.
• Detached the root volume from the locked instance → attached it to the helper.
• Mounted the volume → discovered the partition still capped at 100GB even though the EBS size was already 150GB.
• Ran growpart + resize2fs → filesystem finally stretched to the full 150GB. (49GB free instantly.)
• Cleared old logs and temp files for breathing room.
• Added a new SSH public key into ~/.ssh/authorized_keys.
• Detached the fixed root volume → reattached it back to the original instance.
• Rebooted → boom! SSH worked with the new PEM, and the SSM Agent sprang back to life.