The SSH habit that breaks production audit trails

What this guide covers

Replace ad-hoc SSH into production with AWS Systems Manager Session Manager so your debug sessions are IAM-scoped, auditable in CloudTrail, and don’t require inbound SSH ports (reduces shared keys and blind spots). After this you can start an auditable shell on an EC2 instance and remove an SSH security group rule without losing incident debugging capability. (AWS Systems Manager Session Manager documentation)

When to use it

• You or on-call still SSH into an EC2 instance to debug incidents and want the same interactive access but with audit logs. (Session Manager supports interactive shell sessions.)
• You need to remove port 22 from production security groups but must retain the ability to run live diagnostics.
• You want session-level access control tied to IAM (no shared SSH keys) and recorded API events for postmortems. (CloudTrail records Session Manager API calls.)

The move, step by step

1. Ensure the SSM Agent runs on the instance. Amazon Linux 2 and some AMIs include it by default; otherwise install the agent per the docs (SSM Agent requirement — AWS Systems Manager Session Manager documentation).

2. Create an EC2 role with the managed policy AmazonSSMManagedInstanceCore and attach it to the instance:

# create trust policy file
cat > trust-policy.json <<'JSON'
{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Effect":"Allow",
      "Principal":{"Service":"ec2.amazonaws.com"},
      "Action":"sts:AssumeRole"
    }
  ]
}
JSON

aws iam create-role --role-name SSMProdRole --assume-role-policy-document file://trust-policy.json
aws iam attach-role-policy --role-name SSMProdRole --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

# attach the role to the instance (replace INSTANCE_ID with your EC2 id)
aws ec2 associate-iam-instance-profile --instance-id i-0123456789abcdef0 --iam-instance-profile Name=SSMProdRole

3. Start an interactive session from your workstation using the AWS CLI (this opens a shell to the instance via Session Manager; no inbound port required):

aws ssm start-session --target i-0123456789abcdef0

(Session Manager uses IAM for session authorization and negotiates a connection without opening port 22 — AWS Systems Manager Session Manager documentation.)

4. Verify auditability: search CloudTrail for StartSession or ResumeSession events after you connect:

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=StartSession --max-results 5

(CloudTrail records Session Manager API calls — see Session Manager docs.)

5. Once sessions work, remove SSH access from the instance security group (example revokes a wide 0.0.0.0/0 rule on port 22; tailor to your policy):

aws ec2 revoke-security-group-ingress --group-id sg-0abc1234def56789a --protocol tcp --port 22 --cidr 0.0.0.0/0

6. Tighten IAM: replace broad permissions with least-privilege by scoping StartSession to specific instance ARNs or using condition keys (see Session Manager IAM examples in the docs).

Example

Input: an EC2 instance i-0123456789abcdef0 with SSM agent installed.

Commands → expected output:

• Attach role and start session:

aws iam attach-role-policy --role-name SSMProdRole --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
aws ec2 associate-iam-instance-profile --instance-id i-0123456789abcdef0 --iam-instance-profile Name=SSMProdRole
aws ssm start-session --target i-0123456789abcdef0

Expected: your terminal switches into a remote shell prompt on the instance (e.g. sh-4.2$ or ec2-user@ip-…$). In CloudTrail you’ll later see an event with EventName = “StartSession” for that instance. (AWS Systems Manager Session Manager documentation)

Common mistakes

• Mistake: Assuming SSM agent is installed → Fix: verify agent on the AMI or install it per the docs before removing SSH.
• Mistake: Giving everyone broad IAM StartSession permission → Fix: scope the role/policy to the minimal principals and instance ARNs.
• Mistake: Removing SSH before testing Session Manager → Fix: confirm start-session works from your workstation first.
• Mistake: Thinking Session Manager logs full terminal output by default → Fix: enable Session Manager logging (CloudWatch/ S3) if you need full session transcripts (see docs).
• Mistake: Not checking CloudTrail → Fix: validate that StartSession events appear in CloudTrail for your account.

Next step

In the next 10 minutes: pick one non-production EC2 instance, attach an IAM role with AmazonSSMManagedInstanceCore, run aws ssm start-session —target , confirm you get a shell, then revoke SSH on that instance’s security group. (AWS Systems Manager Session Manager documentation) Then come back and try the next move from the video.