In the Cloud world, security is paramount. Many cloud providers are still coming up with new features to make the infra secure. In this post, Here we are going to setup a bastion server to secure the SSH access on any Cloud infra like AWS, GCP, and Azure.

Is Bastion same as Jump host?

Actually not. Jump hosts are living in the same network as you want to connect. But the Bastion host is located somewhere else and need additional security options to access other networks.

At Searce, we manage cloud infrastructure for multiple customers across AWS, GCP and Azure on 24*7 basis. When have to provide access to our CloudOps teams — it’s not efficient making writing multiple firewall rules / security groups, hop on to customer’s bastion host etc. So we have deployed our own bastion Server to access all the customer's infra.

What’s new here?

Its just bastion host setup, why we need to blog about it?

There are so many blogs and documentation about setting up a bastion server. But here we have added few ingredients to the recipe.

SSH session recording 2Factor authentication enabled. Transfer.sh for sharing the files between any servers. Automated user management. Easy to restore the users on the new bastion server.

Enable SSH Session Recording:

This option will track every command that an user executed along the command results. With the help of this, we can easily find out if someone did something wrong. This is already published in AWS Blog, but we have done this on GCP.

Create the directory for session recording log files.

mkdir /var/log/bastion

Here out admin user is sqladmin. So make this user to own this bastion log directory.

chown sqladmin:sqladmin /var/log/bastion

chmod -R 770 /var/log/bastion

setfacl -Rdm other:0 /var/log/bastion

Make OpenSSH execute a custom script on logins.

echo -e "

ForceCommand /usr/bin/bastion/shell" >> /etc/ssh/sshd_config

Remove some features from SSH and make it more restricted.

awk '!/AllowTcpForwarding/' /etc/ssh/sshd_config > temp && mv temp /etc/ssh/sshd_config

awk '!/X11Forwarding/' /etc/ssh/sshd_config > temp && mv temp /etc/ssh/sshd_config

echo "AllowTcpForwarding no" >> /etc/ssh/sshd_config

echo "X11Forwarding no" >> /etc/ssh/sshd_config

Create a shell script to capture the user's session.

mkdir /usr/bin/bastion

cat > /usr/bin/bastion/shell << 'EOF' if [[ -z $SSH_ORIGINAL_COMMAND ]]; then LOG_FILE="`date --date="today" "+%Y-%m-%d_%H-%M-%S"`_`whoami`"

LOG_DIR="/var/log/bastion/" echo ""

echo "NOTE: This SSH session will be recorded"

echo "AUDIT KEY: $LOG_FILE"

echo "" SUFFIX=`mktemp -u _XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX` script -qf --timing=$LOG_DIR$LOG_FILE$SUFFIX.time $LOG_DIR$LOG_FILE$SUFFIX.data --command=/bin/bash else

echo "This bastion supports interactive sessions only. Do not supply a command"

exit 1 fi

EOF chmod a+x /usr/bin/bastion/shell

chown root:sqladmin /usr/bin/script

chmod g+s /usr/bin/script

Prevent bastion host users from viewing processes owned by other users, because the log file name is one of the scipt execution parameters.

mount -o remount,rw,hidepid=2 /proc

awk '!/proc/' /etc/fstab > temp && mv temp /etc/fstab

echo "proc /proc proc defaults,hidepid=2 0 0" >> /etc/fstab

Restart the SSH service to apply /etc/ssh/sshd_config modifications.

service sshd restart

You can sync the logs files which are in /var/log/bastion/* to S3, GCS or Azure Blob.

To verify this session recording, we can login to the bastion server and we’ll get the below welcome message.

Also we can find the log files.

ls /var/log/bastion/ [root@XXXXXXX bhuvaneshwaran_r]# ls /var/log/bastion/

2019-02-26_06-38-27_bhuvaneshwaran_r_eZpI0sOqp3VGJi8z73sHV9nikMIZXY1Z.data

2019-02-26_06-38-27_bhuvaneshwaran_r_eZpI0sOqp3VGJi8z73sHV9nikMIZXY1Z.time

2019-02-26_17-55-02_bhuvaneshwaran_r_MFUQ71xnZ6tDrNfHSzIuP10vlXBmpgRT.time

2019-02-26_17-59-07_bhuvaneshwaran_r_yrOZFBA6XOzkNiMXYoNfRFz45jIWyD3N.data

2019-02-26_17-59-07_bhuvaneshwaran_r_yrOZFBA6XOzkNiMXYoNfRFz45jIWyD3N.time

2019-02-26_18-01-56_sqladmin_z1KsXV4J9cwWSJnPlbFi2AqfgbBkolh1.data

2019-02-26_18-01-56_sqladmin_z1KsXV4J9cwWSJnPlbFi2AqfgbBkolh1.time

2019-02-26_18-04-00_sqladmin_5DRUr5fh0OwVKMkh98sumeBFhrMsrjIn.data

Configure SMTP with Postfix and integrate AWS SES:

Before installing google authenticator and other things, we are going to install postfix mail agent and integrate with AWS SES. (if you have any own mail servers then you can ignore this part)

yum remove sendmail

yum install postfix

yum install cyrus-sasl-plain

alternatives --set mta /usr/sbin/postfix #Replace email-smtp.us-west-2.amazonaws.com with your SES region or SMTP server.

vi /etc/postfix/main.cf

smtp_tls_note_starttls_offer = yes

smtp_tls_security_level = encrypt

smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd

smtp_sasl_security_options = noanonymous

relayhost = [email-smtp.us-west-2.amazonaws.com]:587

smtp_sasl_auth_enable = yes

smtp_use_tls = yes

smtp_tls_CAfile = /etc/ssl/certs/ca-bundle.crt vi /etc/postfix/sasl_passwd

[email-smtp.us-west-2.amazonaws.com]:587 ACCESSKEY:SECRETKEY sudo postmap hash:/etc/postfix/sasl_passwd

sudo chown root:root /etc/postfix/sasl_passwd /etc/postfix/sasl_passwd.db

sudo chmod 0600 /etc/postfix/sasl_passwd /etc/postfix/sasl_passwd.db sudo postconf -e 'smtp_tls_CAfile = /etc/ssl/certs/ca-bundle.crt'

sudo postfix start; sudo postfix reload

Send the test message:

echo "test mail from postfix" | mailx -r " sender@domain. com (Bastion Admin)" -s "Bastion host test" receiver.r@searce.com

Enable 2FA with Google Authenticator:

Install Google authenticator:



sudo yum install google-authenticator sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm sudo yum install google-authenticator

Setup Google Authentication in SSH Config

vi /etc/pam.d/sshd #remove this line or comment it.

auth substack password-auth #add this line in the end

auth required pam_google_authenticator.so vi /etc/ssh/sshd_config

#Please take a copy of the sshd_config file #Use custom SSH Port

Port 46285 #Disable root login

PermitRootLogin no #Enable Key based authendication and disable password authendication

PubkeyAuthentication yes

PasswordAuthentication no #Enale PAM

UsePAM yes #My admin user is sqladmin, so I don't want to use MFA for this user, so if Google Authendicator is not working, I can login with this admin user and fix it. So exclude sqladmin user from MFA. Match User "sqladmin"

AuthenticationMethods "publickey"

Match User "*,!sqladmin"

AuthenticationMethods "keyboard-interactive,publickey"

Save the config file and restart it.

service sshd restart

Automate User Creation:

This shell script will help to create the user with google authenticator along with the Publickey authentication. Then automatically then send the private key and Google authenticator’s QR code in email.

Also it’ll sync the user’s public/private keys and google authenticator files to GCS bucket. (you can modify this script to sync this information to S3 or Azure blob.

Input values:

Username: Name of the user Email: email address to send the private key and google QR code. Super User: Does this user need root access? If yes 'Y' else 'N'

Create an user and test the MFA:

./bastio-user-create.sh

Email Address:

Do you want Super User access [Y or N]: Y User Name: bhuvaneshEmail Address: bhuvaneshwaran.r@searce.com Do you want Super User access [Y or N]: Y

Configure Transfer.sh on bastion:

Now we are going to setting up transfer.sh via docker.

yum install docker

service docker start docker pull dutchcoders/transfer.sh:latest

I have a 100G volume and attached it to /transfer and I want all the uploaded files should go to this mount point.

tmp dir: /transfer/tmp

upload files location: /transfer/uploads

docker run -d --restart always --privileged=true -i -v "/transfer/uploads:/uploads" -v "/transfer/tmp:/tmp" --publish 80:8080 dutchcoders/transfer.sh:latest --provider local --basedir /uploads --temp-path=/tmp

Open your browser and hit the IP of the bastion server.