Before I get into this post, I do want to highlight that you probably will not do this in any production type environment. The reason why I implemented this, and how this post came about, is because I was helping out with our new edition of the VSAN 6.2. Hands-On-Lab (which should be available imminently by the way). Part of the lab involved demonstrating checksum functionality. Since VSAN has a distributed architecture, there was a requirement to run commands on different hosts. Rather than having lab participants input the password each and every time to run a command on the remote hosts, which becomes tedious very quickly, we decided to try to implement a public/private key pair, creating a trust between the ESXi 6.0U2 hosts and avoid having to input the password to run commands remotely. This proves a little problematic on ESXi hosts as not every file on ESXi is persisted on reboot. The following are the steps we implemented to allow us to do this in the lab.

Step 1. Create folder(s) on persistent storage

In order to save the keys and have them available on next reboot, you will need some persistent storage, such as VMFS or NFS, presented to the ESXi host. In my case, I had an NFS datastore. I then proceed to create a folder on the persistent storage in /vmfs/volumes/<NFS>/<hostname>/keys. Why we need to do this will become clearer shortly.

[root@esxi-hp-06:~] mkdir /vmfs/volumes/NFSA/esxi-hp-06/ [root@esxi-hp-06:~] mkdir /vmfs/volumes/NFSA/esxi-hp-06/keys/

Step 2. Create an .ssh directory for root



As I am doing this as a root user, I created /.ssh.

Step 3. Generate the keys using ssh-keygen



First, change directory to /.ssh. I then used the following command to create RSA keys. I used the -b option to use 4096 bits in the key:

[root@esxi-hp-06:~] cd /.ssh [root@esxi-hp-06:/.ssh] /usr/lib/vmware/openssh/bin/ssh-keygen -t rsa -b 4096 Generating public/private rsa key pair. Enter file in which to save the key (//.ssh/id_rsa): Enter passphrase (empty for no passphrase): << leave empty>> Enter same passphrase again: << leave empty>> Your identification has been saved in //.ssh/id_rsa. Your public key has been saved in //.ssh/id_rsa.pub. The key fingerprint is: SHA256:v6dVmqvVYZc147BubdgbU1rMQUC9eic+87ovm/5Ftuk root@esxi-hp-06 The key's randomart image is: +---[RSA 4096]----+ | .oo. | | .. | | . +o| | ++*| | S .=oO| | . .B=O=| | . ==+Xo| | +o.B.=| | ++..*E+| +----[SHA256]-----+ [root@esxi-hp-06:/.ssh] ls id_rsa id_rsa.pub known_hosts

Step 4. Copy public key to all hosts

You now need to copy the public key (id_rsa.pub) that you just created to the other hosts that you wish to login to without providing a password. In ESXi 6.0U2, the file that the key must be appended to on the remote host is /etc/ssh/keys-root/authorized_keys. I used this very useful command to copy the public key to the other hosts:

[root@esxi-hp-06:/.ssh] cat id_rsa.pub | ssh root@esxi-hp-07 \ 'cat >> /etc/ssh/keys-root/authorized_keys' The authenticity of host 'esxi-hp-07 (10.27.51.7)' can't be established. RSA key fingerprint is SHA256:QMkmf/vrqPU2GnN8dzRz01JBQRqD6hyZBA+nB8SLTcA. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'esxi-hp-07,10.27.51.7' (RSA) to the list of known hosts. Password:

Obviously, at this point, I still need to provide passwords. Repeat the above step for all hosts that you wish to login to without providing a root password.

Step 5. Make sure authorized_keys is sticky/persists

One other thing to check – make sure that the contents of authorized_keys is sticky/persistent. This ensures that it contents gets saved and persists through reboots. To do this, ensure that there is a “T” bit in the permission as follows:

[root@esxi-hp-06:~] cd /etc/ssh/keys-root/ [root@esxi-hp-06:/etc/ssh/keys-root] ls -l total 4 -rw------T 1 root root 380 Dec 27 2014 authorized_keys [root@esxi-hp-06:/etc/ssh/keys-root]

If it is not sticky for whatever reason, run the following command to make it so:

[root@esxi-hp-06:~] chmod +t /etc/ssh/keys-root/authorized_keys

Step 6. Copy the public key to the persistent storage

We are now going to store the contents of /.ssh to the persistent storage. In my setup, This is what I did.

[root@esxi-hp-06:/.ssh] cp /.ssh/* /vmfs/volumes/NFSA/esxi-hp-06/keys/*

*** Repeat steps 1 through 6 for all hosts ***

Step 7. Modify sshd_config

The next step is to have keys used for authentication rather than passwords. This necessitates a change to the /etc/ssh/sshd_config file on each host that you will to login to. Ensure that the following entries appear in the sshd_config file.

PermitRootLogin yes UsePAM yes # only use PAM challenge-response (keyboard-interactive) PasswordAuthentication no ChallengeResponseAuthentication no

In my /etc/ssh/sshd_config I simply needed to add the ChallengeResponseAuthentication entry, but you should verify that the other settings are in place too.

Step 8. Restart SSH

After modifying the ssh_config file, we now need to restart SSH to pull in the new settings:

[root@esxi-hp-07:/etc/ssh/keys-root] /etc/init.d/SSH restart SSH login disabled SSH login enabled

Step 9. Do a test login

Now we can see if we can login to the remote ESXi host without a password. I would however recommend keeping the original session open and creating a new session to verify this, as this will allow you to rectify any mistakes should you get it wrong.

[root@esxi-hp-06:/.ssh] ssh root@esxi-hp-07 The time and date of this login have been sent to the system logs. VMware offers supported, powerful system administration tools. Please see www.vmware.com/go/sysadmintools for details. The ESXi Shell can be disabled by an administrative user. See the vSphere Security documentation for more information. [root@esxi-hp-07:~]

Success – everything is working as expected and we can login without a password.

*** Repeat steps 7 through 9 for all hosts ***

However there is one issue. The authorized_keys and sshd_config files are persisted through reboots, but the /.ssh folder that we created earlier (and where we store our keys) is not persisted during reboots. So how to we make it persistent?

Step 10. Recreate the .ssh directory and put the keys back in place on reboot



On reboot, we will need to create the /.ssh directory, and copy the keys back from persistent storage to this directory. So how do we do that? Well, we are fortunate in that we have a file on the ESXi host that can run commands on boot up. This file is /etc/rc.local.d/local.sh.

However, note that editing this file is unsupported, and you need to be extremely careful editing this file as you can end up with an unusable system if you make a mistake.

Let’s look at the contents of the file first – please read it carefully:

[root@esxi-hp-06:/etc/rc.local.d] cat local.sh #!/bin/sh # local configuration options # Note: modify at your own risk! If you do/use anything in this # script that is not part of a stable API (relying on files to be in # specific places, specific tools, specific output, etc) there is a # possibility you will end up with a broken system after patching or # upgrading. Changes are not supported unless under direction of # VMware support. exit 0

There are two commands that we wish to add:

Make the /.ssh directory Copy the files from the persistent storage to the /.ssh directory

Once we are finished add those commands, the script should now look something like this:

[root@esxi-hp-06:/etc/rc.local.d] cat local.sh #!/bin/sh # local configuration options # Note: modify at your own risk! If you do/use anything in this # script that is not part of a stable API (relying on files to be in # specific places, specific tools, specific output, etc) there is a # possibility you will end up with a broken system after patching or # upgrading. Changes are not supported unless under direction of # VMware support. mkdir /.ssh cp /vmfs/volumes/17602f20-1df66551/esx-hp-06/keys/* /.ssh exit 0

This now means that when this ESXi host boots up, a new .ssh directory is created, and the .ssh files stored on persistent storage are copied in to the .ssh directory.

Step 11. Make sure the changes you make are saved

Now, a normal reboot of the ESXi host should save off this file to the bootbank, and it should have the changes on subsequent boot up. However, to be doubly sure, you can run the script /sbin/auto-backup.sh to save the changes immediately:

[root@esxi-hp-06:~] /sbin/auto-backup.sh Files /etc/vmware/dvsdata.db and /tmp/auto-backup.3869706//etc/vmware/dvsdata.db differ Saving current state in /bootbank Clock updated. Time: 08:14:48 Date: 04/13/2016 UTC [root@esxi-hp-06:~]

Step 12. Reboot the host, and verify you can still login

Now you can try rebooting each of the ESXi hosts in turn and verify that you can login. If everything is working, this now means that we can SSH to remote hosts without the need for passwords, and that this behaviour will be maintained through reboots.

*** Repeat steps 10 through 12 for all hosts ***

References and Disclaimer

There are some useful references out there if you want to learn more about this process. I referenced the following articles while researching a method on how to do this:

Disclaimer: Just to finish, this procedure is shown here purely for convenience. We set this up in the confines of a hands-on-lab, where we were running multiple commands/scripts remotely between hosts. As already stated, I would not recommend using this in production. Note that you are also acknowledging the risk involved by editing the local.sh file. (Hopefully this disclaimer means Mike Foley won’t come after me 😀 )

Disclaimer2: This is what worked for us. There may be other ways to do this procedure, but this is the way it worked for me. If you know of an alternate method, let me know.

Finally, this was implemented for the new Hands-On-Lab based on VSAN 6.2. These labs should come online soon as part of the Spring 2016 lab refresh. I’d urge you to check it out to see features like RAID5/6, Dedupe/Compression, Checksum and stretched cluster/ROBO implementations.