Synchronizing SSH keys for Elastic Beanstalk

Photo by Fernanda kovacs on Unsplash

One of the benefits to session manager is no longer being required to manage SSH keys. That being said, SSH tunnels do require keys and we’d rather not use the shared key provided by Elastic Beanstalk. How do we deal with this? There was a fantastic StackOverflow solution posted by Sam Maleyek. I cleaned it up and converted it into a full Elastic Beanstalk extension. Let’s look at the extension and what it does (this would saved to $project_root/.ebextensions/sync_ssh_keys.config or something similar)

files:

"/home/ec2-user/get-ssh-keys.sh":

mode: "000755"

content: |

#!/bin/bash # Only users in this group will have their keys sync'ed to instances

iam_group_allowed_users=beanstalk-ssh-users auth_file=/home/ec2-user/.ssh/authorized_keys

tempfile=/home/ec2-user/.ssh/authorized_keys.tmp.$$ logMsg()

{

INITPID=$$

PROG="get-ssh-keys"

logger -t ${PROG}[$INITPID] $1

echo $1

} # Delay the start so we don't get a storm of instances crushing IAM

sleep_duration=$(( (RANDOM % 10) + 1 ))

logMsg "Delaying startup by ${sleep_duration} seconds"

sleep ${sleep_duration}s # Save off the "root" beanstalk SSH key if we have not already done so

if [ ! -e "/home/ec2-user/.ssh/root_key" ]; then

logMsg "Saving beanstalk root key to /home/ec2-user/.ssh/root_key"

head -1 "/home/ec2-user/.ssh/authorized_keys" > "/home/ec2-user/.ssh/root_key"

fi # Set the perms now or SSH will fail

touch "${tempfile}"

chmod 600 "${tempfile}"

chown ec2-user:ec2-user "${tempfile}" # Always start off with the "root" key

logMsg "Initializing ${tempfile} with beanstalk root SSH key"

cat "/home/ec2-user/.ssh/root_key" > "${tempfile}" allowed_users=$(aws iam get-group \

--group-name "${iam_group_allowed_users}" \

--query 'Users[].UserName' \

--output text) for iam_user in ${allowed_users}

do

logMsg "Retrieving SSH keys for user ${iam_user}" ssh_key_ids=$(aws iam list-ssh-public-keys \

--user-name "${iam_user}" \

--query 'SSHPublicKeys[?Status==`Active`].SSHPublicKeyId' \

--output text) if [ -n "${ssh_key_ids}" ]; then

for ssh_key_id in ${ssh_key_ids}

do

logMsg "Retrieving public key for ${iam_user} with ID ${ssh_key_id}" ssh_key=$(aws iam get-ssh-public-key \

--encoding SSH --user-name "${iam_user}" \

--ssh-public-key-id "${ssh_key_id}" \

--query 'SSHPublicKey.SSHPublicKeyBody' \

--output text) echo "${ssh_key}" >> "${tempfile}"

done

else

logMsg "No active SSH keys found for user ${iam_user}"

fi

done # Drop in the new authorized_keys file

logMsg "Replacing ${auth_file} with ${tempfile}"

mv "${tempfile}" "${auth_file}" "/tmp/get-ssh-keys":

mode: "000644"

owner: root

group: root

content: |

*/10 * * * * root /home/ec2-user/get-ssh-keys.sh > /dev/null 2>&1 container_commands:

01update_cron:

command: "mv -f /tmp/get-ssh-keys /etc/cron.d"

02postinit_hook:

command: "cp -f /home/ec2-user/get-ssh-keys.sh /opt/elasticbeanstalk/hooks/postinit/12_get-ssh-keys.sh"

Let’s have a look at the interesting parts…

iam_group_allowed_users=beanstalk-ssh-users

We will only synchronize SSH keys for users who are a member of this IAM group.

head -1 "/home/ec2-user/.ssh/authorized_keys" > "/home/ec2-user/.ssh/root_key"

Always save the “root” Elastic Beanstalk SSH key to fallback on should we somehow lock out all users.

ssh_key_ids=$(aws iam list-ssh-public-keys \

--user-name "${iam_user}" \

--query 'SSHPublicKeys[?Status==`Active`].SSHPublicKeyId' \

--output text)

This is the crux of the solution. Here, we get all active public SSH key IDs for a specific user.

IAM allows uploading SSH public keys to an IAM user for use with CodeCommit. But nothing says we can only use them with CodeCommit! These keys can all be managed by the users themselves and rotated like IAM keys. Perfect!

ssh_key=$(aws iam get-ssh-public-key \

--encoding SSH --user-name "${iam_user}" \

--ssh-public-key-id "${ssh_key_id}" \

--query 'SSHPublicKey.SSHPublicKeyBody' \

--output text)

Next, we can get the actual public key and write it to the authorized_keys file on the instance

container_commands:

01update_cron:

command: "mv -f /tmp/get-ssh-keys /etc/cron.d"

02postinit_hook:

command: "cp -f /home/ec2-user/get-ssh-keys.sh /opt/elasticbeanstalk/hooks/postinit/12_get-ssh-keys.sh"

Finally, we will synchronize the keys every 10 minutes in a cron on the instance. We also need to add to a Beanstalk postinit hook in order for the keys to be synchronized right away at boot time, rather than potentially waiting for the first run of the cron.

Wrapping up

Having worked in tech for 25 years, two principles have really stuck with me in regards to security: