Well Defined and Parallized Environment Control

The latest Massh RPM is : massh-2.0-58.noarch.rpm

The latest Massh OSX Package is : massh-2.0-58.zip

The latest Massh Tar/Gzipped Source is : massh-2.0-57.tgz

Massh makes it possible to perform the following on hundreds, or even thousands of hosts in a parallelized fashion:

Run Remote Commands

Push and Execute Pre-Written Scripts

Push Files

Pull Files

Massh uses Ambit to enumerate a list of targets for which it will execute one of the aforementioned operations. Hosts can be logically grouped and dynamically managed allowing for consistent operational changes and overall management of identically purposed hosts.

Massh can display the full output resulting from its various operations or it can be set to display an operation's success or failure for individual target hosts. The full ouput produced by Massh is completely organized and labeled by host, regardless of what order output is returned from target hosts.

Starting with version 2.x Massh combines flexible option control management and a simplified command + subcommand sytax. This reduces the complexity of typical Massh commands and makes it much easier to learn how to perform specific operations with specific options with Massh.

Contents

Massh 2.x introduces a new simplified CLI sytax that is best broken down in the following way:

COMMAND => OPTION + SUBCOMMAND => SUBOPTION ===================================================================== massh allhosts verbose gpasswd -a mm wheel massh db[1..30] push my.cnf massh hosts.txt worked sudo service restart httpd

The Massh executable itself.

Currently the values that are valid for OPTION are those that can be understood, expanded and enumerated by Ambit or some other hostlist generator:

A command line argument in the form of an Ambit expandable string.

$ massh [2,3].a.tt verbose whoami 3.a.tt : mm 2.a.tt : mm $ massh [1..3].a.tt verbose sudo whereami 2.a.tt : Fremont, CA, US 1.a.tt : Atlanta, GA, US 3.a.tt : Dallas, TX, US

If a file in the user's current working directory ($PWD) contains hostnames (one per line) or Ambit expandable strings it can be passed to OPTION by its name (full path not needed).

1 $ cat somehosts.txt 1.a.tt 2.a.tt 3.a.tt $ massh somehosts.txt verbose sudo whereami 1.a.tt : Atlanta, GA, US 3.a.tt : Dallas, TX, US 2.a.tt : Fremont, CA, US

If a file located somewhere on the current host contains hostnames (one per line) or Ambit expandable strings it can be passed to OPTION by listing the files full path.

1 $ cat /usr/tmp/otherhosts.txt db[1..3].a.tt 1 $ ambit /usr/tmp/otherhosts.txt db1.a.tt db2.a.tt db3.a.tt $ massh /usr/tmp/otherhosts.txt verbose 'sudo monit summary | grep bind' db1.a.tt : Process 'bind' running db3.a.tt : Process 'bind' running db2.a.tt : Process 'bind' running

Users can create personal or User HostGroups (via Ambit) that can be passed to OPTION by name.

1 $ ambit create hostgroup HostGroup Name : myradservers HostGroup Summary : My Rad Servers Enter Hostnames Below [Ctrl-d to End]: db[1..3].a.tt 1 $ ambit list hostgroups User HostGroups ---------------- myradservers: My Rad Servers 1 $ massh myradservers push /etc/hosts db1.a.tt : Push Succeeded db3.a.tt : Push Succeeded db2.a.tt : Push Succeeded

Users with root access can create System HostGroups (via Ambit) that all users on the system can pass the OPTION by name.

Network accessible HostGroups can be established by creating a DNS TXT record (alas they have a use ;^) containing an Ambit expandable string. Ambit will check for such records if the 'Domain' Ambit option is set. The shortname for the aforementioned DNS TXT records can be passed to OPTION. Example: The domain a.tt has the following TXT record to establish a Network HostGroup for the Yum package repositories:

$ host -t TXT yum yum.a.tt descriptive text "[1..3].a.tt"

This mean the HostGroup's name (yum) can be passed to OPTION.

The SUBCOMMAND is a one word descriptive indicator that best describes the overall operation that Massh will be performing. Unfortunately the initial goal of establishing SUBCOMMAND's that were gramatically consistent (i.e. all verbs or even adverbs) was sacrificed in in favor of memorable SUBCOMMAND names.

A SUBOPTION will either be a command (to be run on target hosts) or a filename (i.e. file to be pulled, pushed or containing a script to be executed). The following are possible SUBCOMMANDS:

SSH to all targets enumerated from OPTION, execute the command stipulated by SUBOPTION and display only the success or failure of the executed command.

$ massh east10 terse uptime easthost5 : Command Succeeded easthost7 : Command Succeeded easthost1 : Command Succeeded easthost3 : Command Succeeded easthost8 : Command Succeeded easthost2 : Command Succeeded easthost9 : Command Succeeded easthost6 : Command Succeeded easthost10 : Command Succeeded easthost4 : Command Succeeded

SSH to all targets enumerated from OPTION, execute the command stipulated by SUBOPTION and display the full output resulting from the command.

$ massh east10 verbose uname -vr easthost10 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 easthost3 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 easthost5 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 easthost6 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 easthost8 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 easthost1 : 2.6.9-67.ELsmp #1 SMP Wed Nov 7 13:58:04 EST 2007 easthost9 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 easthost2 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 easthost4 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 easthost7 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006

SSH to all targets enumerated from OPTION, execute the command stipulated by SUBOPTION and display only the hostnames where the command executed successfully.

$ ambit west10 | wc -l 10 $ massh west10 worked uname -vr westhost1 westhost2 westhost4 westhost3 westhost7 westhost10 westhost6 westhost9 westhost8 $ massh west10 worked uname -vr | wc -l 9

The example above shows that the HostGroup 'west10' is made up of ten hosts. However the Massh run only returned 9 hostnames.

SSH to all targets enumerated from OPTION, execute the command stipulated by SUBOPTION and display only the hostnames where the command failed.

$ massh west10 bombed uname -vr westhost5

This example demonstrates how the option 'bombed' only returns hostnames that are not able to fulfill the Massh run. The host 'westhost5' was the only host that is not listed in the results from the previous example. These options are very useful for large scale operations where success is expected across 100% of hosts (production code releases and production package updates come to mine). Capturing all hosts that fail a code release provides the immediate opportunity to fix the issue or the immediate opportunity to make sure the failed hosts are effectively taken out of service.

SCP the file stipulated by SUBOPTION, to all targets enumerated from OPTION.

$ massh east10 push /etc/passwd easthost7 : Push Succeeded easthost2 : Push Succeeded easthost6 : Push Succeeded easthost5 : Push Succeeded easthost10 : Push Succeeded easthost3 : Push Succeeded easthost4 : Push Succeeded easthost8 : Push Succeeded easthost1 : Push Succeeded easthost9 : Push Succeeded

SCP the file stipulated by SUBOPTION, from all targets enumerated from OPTION.

$ massh east10 pull /etc/resolv.conf easthost9 : Pull Succeeded easthost2 : Pull Succeeded easthost1 : Pull Succeeded easthost7 : Pull Succeeded easthost4 : Pull Succeeded easthost8 : Pull Succeeded easthost10 : Pull Succeeded easthost3 : Pull Succeeded easthost5 : Pull Succeeded easthost6 : Pull Succeeded

The pull subcommand saves each file separately in directories named after the hosts:path they were pulled from:

$ find .massh/pull/ -name resolv.conf .massh/pull/easthost10/etc/resolv.conf .massh/pull/easthost1/etc/resolv.conf .massh/pull/easthost7/etc/resolv.conf .massh/pull/easthost2/etc/resolv.conf .massh/pull/easthost8/etc/resolv.conf .massh/pull/easthost9/etc/resolv.conf .massh/pull/easthost6/etc/resolv.conf .massh/pull/easthost5/etc/resolv.conf .massh/pull/easthost3/etc/resolv.conf .massh/pull/easthost4/etc/resolv.conf

SSH to all targets enumerated from OPTION, piping the script stipulated by SUBOPTION to all targets enumerated from OPTION, execute the script and display the full output of the executed command.

$ cat dateup.sh #!/bin/bash -e MyDate=$(date '+%a %B %d') MyTime=$(sed 's/^.*up\ \(.*days\).*$/\1/' The example above demonstrates that things work quite well - with one slight annoyance. Massh is excellent at ensuring that multi-line output is always returned and displayed just like it would be displayed directly on each host. It also ensures that multi-line output is accurately separated by host regardless of the order that output is returned Massh. These capabilities alone separate Massh from most other parallelized shells, but they don't equate to perfect or even optimal output. The addition of the option 'Separator' goes a long way towards achieving well formatted, multi-line parallel output: $ massh ed opt Separator Updating Option : Separator Current Value : no Enter New Value : yes $ massh east10 verbose df -h easthost2 : Filesystem Size Used Avail Use% Mounted on easthost2 : /dev/sda3 72G 9.8G 58G 15% / easthost2 : /dev/sda1 99M 17M 78M 18% /boot easthost2 : none 2.0G 0 2.0G 0% /dev/shm - easthost6 : Filesystem Size Used Avail Use% Mounted on easthost6 : /dev/sda3 72G 13G 56G 19% / easthost6 : /dev/sda1 99M 17M 78M 18% /boot easthost6 : none 2.0G 0 2.0G 0% /dev/shm - easthost4 : Filesystem Size Used Avail Use% Mounted on easthost4 : /dev/sda3 72G 13G 55G 19% / easthost4 : /dev/sda1 99M 17M 78M 18% /boot easthost4 : none 2.0G 0 2.0G 0% /dev/shm - easthost7 : Filesystem Size Used Avail Use% Mounted on easthost7 : /dev/sda3 72G 13G 56G 19% / easthost7 : /dev/sda1 99M 17M 78M 18% /boot easthost7 : none 2.0G 0 2.0G 0% /dev/shm - easthost3 : Filesystem Size Used Avail Use% Mounted on easthost3 : /dev/sda3 72G 13G 55G 19% / easthost3 : /dev/sda1 99M 17M 78M 18% /boot easthost3 : none 2.0G 0 2.0G 0% /dev/shm - easthost5 : Filesystem Size Used Avail Use% Mounted on easthost5 : /dev/sda3 72G 13G 56G 19% / easthost5 : /dev/sda1 99M 17M 78M 18% /boot easthost5 : none 2.0G 0 2.0G 0% /dev/shm - easthost1 : Filesystem Size Used Avail Use% Mounted on easthost1 : /dev/sda1 3.7G 1.4G 2.2G 38% / easthost1 : /dev/sda3 63G 2.6G 57G 5% /home easthost1 : none 2.0G 0 2.0G 0% /dev/shm - easthost8 : Filesystem Size Used Avail Use% Mounted on easthost8 : /dev/sda3 72G 13G 55G 19% / easthost8 : /dev/sda1 99M 17M 78M 18% /boot easthost8 : none 2.0G 0 2.0G 0% /dev/shm - easthost9 : Filesystem Size Used Avail Use% Mounted on easthost9 : /dev/sda3 72G 6.0G 62G 9% / easthost9 : /dev/sda1 99M 17M 78M 18% /boot easthost9 : none 1014M 0 1014M 0% /dev/shm - easthost10 : Filesystem Size Used Avail Use% Mounted on easthost10 : /dev/sda3 72G 6.0G 62G 9% / easthost10 : /dev/sda1 99M 17M 78M 18% /boot easthost10 : none 1014M 0 1014M 0% /dev/shm - CONFIGURABLE OPTIONS Starting with version 2.x Massh establishes a system wide option baseline that can only be changed with root privileges, but can be overridden by users on a per user basis. Massh provides the ability to view and edit options via simple interactive dialogues. Option Controls list options List both System and User Tunable Options. $ massh list options System Default Options ---------------------- Debug="no" ForcePseudoTTY="no" SyslogMassh="yes" SSHLogLevel="Error" TimeOut="30" Parallel="30" Random="$RANDOM" ControlPath="$HOME/.ssh/masshter-%r@%h:%p" ControlMaster="auto" ControlPersist="300" HostsFrom="ambit" Separator="no" SubCommand="" User Specified Options ---------------------- SubCommand="" ControlMaster="no" SSHLogLevel="error" TimeOut="5" Separator="yes" edit option Edit or Set a User Option. User options override System Options. Let's say I want to set the subcommand to always show verbose output (making for an even shorter command): $ massh edit option SubCommand Updating Option : SubCommand Current Value : Enter New Value : verbose SubCommand="verbose" $ massh list options System Default Options ---------------------- Debug="no" ForcePseudoTTY="no" SyslogMassh="yes" SSHLogLevel="Error" TimeOut="30" Parallel="30" Random="$RANDOM" ControlPath="$HOME/.ssh/masshter-%r@%h:%p" ControlMaster="auto" ControlPersist="300" HostsFrom="ambit" Separator="no" SubCommand="" User Specified Options ---------------------- ControlMaster="no" SSHLogLevel="error" TimeOut="5" Separator="yes" SubCommand="verbose" $ massh east10 uname -vr easthost1 : 2.6.9-67.ELsmp #1 SMP Wed Nov 7 13:58:04 EST 2007 easthost5 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 easthost10 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 easthost2 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 easthost9 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 easthost3 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 easthost6 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 easthost8 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 easthost4 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 easthost7 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 SHORT COMMANDS Massh supports abbreviate command sytax for its Option Controls:

massh list options

massh ls opts

massh show opts

massh edit option [OptionName]

massh ed opt [OptionName]

There's a special User HostGroup called 'results' ($HOME/.massh/hosts/results) that will contain the results of the SUBCOMMAND's 'worked' and 'bombed', along with a time stamp of the time the command was run and the command itself. This is mostly to address anything that 'bombed' during Massh runs where 100% success was either expected or required. Ambit will not show this group by default when running: ambit list hostgroups. Nor will Ambit know it exists since it is not in any of Ambit's data paths. If you would like this group to be listed simply hard link it to ($HOME/.ambit/hosts/results):

$ ln $HOME/.massh/hosts/results $HOME/.ambit/hosts/results $ ambit list hostgroups | grep results results: 2011-07-11 09:26:10 grep massh grep Marschall /etc/passwd