One of the principles of running large numbers of instances is that consistency is key. Config deviations cause oddities that’ll drain your time with investigations and nothing causes entropy on your hosts like an admin investigating an issue. In this post we’ll configure our instances to mark themselves as contaminated when someone logs in. We can then use other tooling to query, collate and probably reap, machines corrupted by the keystrokes of humans.

While the example here is step-by-step and interactive, you’d normally bake this in to your AMI or deploy it very early in your config stage, possibly using cloud-init . For our test we’ll spin up an instance and grant it an ec2 instance profile so it can alter its own tags.

In terms of moving parts we’ll install the awscli package, add a short script that’ll tag the instance when run and configure PAM to invoke the script when an ssh session opens to the machine.

# install required dependency sudo apt-get install awscli

cat > /usr/local/bin/add-dirty-tag <<EOF #!/bin/bash [ "$PAM_TYPE" = "open_session" ] || exit 0 INSTANCE_ID=$(ec2metadata --instance-id) REGION=$(ec2metadata --availability-zone) REGION=${REGION%?} # remove the last letter aws ec2 --region $REGION create-tags --resources $INSTANCE_ID --tags Key=contaminated,Value=true EOF

sudo chmod a+rx /usr/local/bin/add-dirty-tag

Now we have a script to add the ‘contaminated’ tag to our instance we’ll configure PAM to run it when a new ssh session starts. On a Ubuntu system the config should be placed in /etc/pam.d/sshd .

# tag the machine as contaminated if anyone sshs in. session optional pam_exec.so /usr/local/bin/add-dirty-tag

It’s worth opening another ssh session and logging in to confirm this works. That will leave you with an established connection in case you misconfigure PAM in some way. Once you’ve successfully logged in and caused the new tag to be added to the instance you can run a cli filter from outside the instance to show all hosts that have been interactively connected to:

aws --region eu-west-1 ec2 describe-instances \ --filters "Name=tag:contaminated,Values=true" \ --query 'Reservations[].Instances[].{id: InstanceId}' [ { "id": "i-x134x34x" } ]