I’ve decided to take on a greenfield deployment of saltstack. I’ve used puppet in the past and I don’t really enjoy learning a DSL that’s not applicable in any other area. I’ve also worked with Ansible which is a great tool for quick and dirty configuration management, but it lacks real depth.

SaltStack is an interesting beast. I would consider it on the lesser side of a steep learning curve. Working with YAML is very easy. Understanding Jinja takes a little bit of work but it’s not the hardest thing in the world and can be applied to anywhere you use python and text rendering. The hardest time I had was really determining the structure of how you want your configuration management and modularity to occur. I want this to be the focus on this article for two reasons, 1) so I can get feedback, 2) so you can understand how pillars, states, and jinja2 all work together.

I’m going to use a particular task: Installing erlang from source (because erlang gets no love). In my example my salt-master is ‘salt-mater’ and my minion is ‘client01p’

So first and for most let me show you my file structure using /srv/ as the root

. ├── pillar │ ├── core │ │ ├── open_ports.sls │ │ └── packages.sls │ ├── erlang │ │ ├── common.sls │ │ └── packages.sls │ ├── haraka │ │ ├── open_ports.sls │ │ └── packages.sls │ ├── nodejs │ │ └── common.sls │ ├── redis │ │ ├── open_ports.sls │ │ └── packages.sls │ └── top.sls └── salt ├── core │ ├── map.jinja │ ├── open_ports.sls │ ├── packages.sls │ ├── python34.sls │ ├── selinux.sls │ ├── ssh.sls │ ├── stash_keys.sls │ └── sysctl.sls ├── erlang │ ├── install.sls │ └── testinstall.sls ├── files │ ├── core │ │ ├── authorized_keys │ │ ├── issue │ │ ├── sshd_config │ │ └── sysctl.conf │ ├── haraka │ │ └── haraka.service │ └── stash │ ├── id_rsa_stash_automated │ └── id_rsa_stash_automated.pub ├── haraka │ ├── install.sls │ └── map.jinja ├── _modules │ └── customuser.py ├── nodejs │ └── install.sls └── top.sls

Let’s explain these folders quickly. We are working out of a /srv/ root directory. This is where all configuration around salt’s pillars and states live. We see there’s two folders under this root, /pillar/ and /salt/. The pillar directory holds pillar information, and the salt directory holds state information. I then further divide my pillars and states to logical functions. In this example we are going to only worry about the the sub-folders “core” and “erlang”. Before we dig into these sub-folders let me quickly explain what pillars and state files are.

Pillars

Pillars hold YAML data structures which you can “tag” onto your minion and are assigned to the minion via the top.sls file. Pillars do not actually do ANY work. Pillars are simple a way for us to pair information, and a minion, together. I like to think of it as defining attributes on an object in a programming language. Typically in object oriented program we define a class, instantiate that object, and then use it’s parameters to perform a task. I.e.

>>> class Adder: ... def __init__(self): ... self.number = 4 >>> >>> >>> Add = Adder() >>> def addit(Adder): ... print(Add.number + 4) ... >>> addit(Add) 8

In the above example I want you to think of the number 4 as a pillar and the function ‘addit’ as a state. We have an object with a number ‘4’(pillar) which our function(state) ‘looks-up’ and applies a ‘state’ too.

States

I use states as my actual unit of work. My state should be ‘fed’ what ever it needs to complete it’s tasks. You can place logic and variables in your states, however best practices would guide you to place this logic else where and keep your state as simple as possible.

With the above example let’s see how we can install a specific set of packages required to build erlang on a minion, based on a pillar and a state file.

Installing and removing packages with pillars and states

First I want to ‘attach’ some kind of data structure to my minion which will facilitate the install and removal of packages on our minion. We are interested in creating an ‘erlang’ minion who’s only function is to host the erlang language. I then create the following pillar file

#/srv/pillar/erlang/packages.sls packages: install: {% if grains['virtual'] != none %} - open-vm-tools {% endif %} - lsof - tcpdump - mtr - traceroute - telnet - bind-utils - curl - wget - ftp - tftp - samba - samba-client - ntp - git - gcc - gcc-c++ - m4 - ncurses-devel - autoconf - java-1.8.0-openjdk-devel - openssl-devel - make {% if grains['os_family'] == 'RedHat' %} - yum-utils {% endif %} remove: - postfix - NetworkManager

This is a list of base packages I want on all my machines along with packages that are required to build erlang.

I then target this pillar to my minion client01p in our pillar top.sls file

#/srv/pillar/top.sls base: 'client01p': - erlang.packages - erlang.common

As you see above, we target our pillar by the specifying the sub-folder the pillar belongs in under the pillar/ root, a ‘.’, and then the .sls file name leaving the .sls out. You notice I targed ‘client01p’ to erlang.common also – more on that in part 2.

So now if we were to do a lookup on the pillar items that ‘client01p’ posses we will see our list of packages. Only pay attention to our ‘Packages’ object.

[root@salt-master srv]# salt 'client01p' pillar.items client01p: ---------- erlang_version: 17.5 open_ports: ---------- 22: tcp 161: udp packages: ---------- install: - open-vm-tools - lsof - tcpdump - mtr - traceroute - telnet - bind-utils - curl - wget - ftp - tftp - samba - samba-client - ntp - git - gcc - gcc-c++ - m4 - ncurses-devel - autoconf - java-1.8.0-openjdk-devel - openssl-devel - make - yum-utils remove: - postfix - NetworkManage

So this is great, we have a minion object client01p, we ‘tagged’ parameters on this object indicating which packages it should install and remove by use of a pillar.sls an top.sls file. So now how do we use this?

We turn to our core.packages state file now. I placed all my really common tasks in a ‘core’ folder. This is how I decided to do it, and by no means indicates that this is how you should do it. Let’s take a look at the core.packages state file.

#/srv/salt/core/packages.sls install_packages: pkg.installed: - pkgs: {% for pkg in salt['pillar.get']('packages:install') %} - {{ pkg }} {% endfor %} remove_packages: pkg.purged: - pkgs: {% for pkg in salt['pillar.get']('packages:remove') %} - {{ pkg }} {% endfor %}

So first thing you’re going to ask, what is that crazy syntax in our state file? Welcome to Jinja2. Jinja2 is a rendering engine, which means it does nothing more then determine what the text in a file looks like after being rendered.

Let’s walk through this state file step by step:

First we have an ID for the state. This can be any arbitrary name as long as it’s unique within the state file. Correction: the ID must be unique across all states that are running not just within the state file.

Next we have the state module we want to use. In our case we want to use the state module named pkg.installed. For a full list of state modules available to you reference here:

https://docs.saltstack.com/en/develop/ref/states/all/index.html

The next line is a state module directive. If you reference the documentation at the link above we see the usage definition for this directive:

pkgs (list) -- A list of packages to install from a software repository. All packages listed under pkgs will be installed via a single command

Now our first piece of Jinja2 code. In Jinja when we want to add any logic, flow control, or variables we begin these statements with {% and end with %}. We are initiating a for loop in this example. We are going to iterate over a function within the salt dictionary. This function takes the arguments ‘packages:install’ which will return each item in our Yaml data structure following install: Let me clarify this,

salt['pillar.get']('packages:remove') ^ ^ ^ | | | | | The arguments to this module | | | The execution module you're interested in running | Dictionary containing all built in execution modules

What we are doing here is running a remote execution model inside our state file to pull the results into our state and use it there. To get an idea of the output this produces you can do the following

[root@salt-master srv]# salt 'client01p' pillar.get packages:install client01p: - open-vm-tools - lsof - tcpdump - mtr - traceroute - telnet - bind-utils - curl - wget - ftp - tftp - samba - samba-client - ntp - git - gcc - gcc-c++ - m4 - ncurses-devel - autoconf - java-1.8.0-openjdk-devel - openssl-devel - make - yum-utils

So we now have this list of packages to feed into our for loop. For ever item in this list we are telling Jinja2 to render the line

– {{ pkg }}

The resulting FILE we arrive at would be this (even though this is invisible to you)

#/srv/salt/core/packages.sls install_packages: pkg.installed: - pkgs: - open-vm-tools - lsof - tcpdump - mtr - traceroute - telnet - bind-utils - curl - wget - ftp - tftp - samba - samba-client - ntp - git - gcc - gcc-c++ - m4 - ncurses-devel - autoconf - java-1.8.0-openjdk-devel - openssl-devel - make - yum-utils remove_packages: pkg.purged: - pkgs: {% for pkg in salt['pillar.get']('packages:remove') %} - {{ pkg }} {% endfor %}

So to stress this point, all Jinja2 is doing here is rendering a new TEXT file which salt will then use to run our state. These are two distinct and mutually exclusive processes. First Jinja2 is called to render our state file, then salt evaluates the syntax and performs our state. The same exact thing happens for remove_packages and all the above can just be implied.

Ok so if this all makes sense, all we do now is target this state to our minion in the STATE top.sls file

base: 'client01p': - core.packages

We can now run highstate or do a one off execution of our state on the minion

salt 'client01p' state.highstate

or

salt 'client01p' state.sls core.packages

Let’s summarize:

We created a pillar file with a data structure which allowed us to easily query which packages we want to install and remove for a particular minion

We targeted this pillar to the minion via the /srv/pillar/top.sls file

We then created a state file in /srv/salt/core/packages.sls. This state file ‘looksup’ the installed and removed packages from the minion’s pillar and runs a for loop rendering these packages in the correct YAML syntax. After this is rendered salt runs the state and performs the code which actually installs the packages.

Now as you can tell from my overall folder structure, we are creating a solid framework for modularity. I can easily create a new sub-folder for a machine type, i.e. mysql. We can then copy the /srv/pillar/core/packages.sls pillar file into /srv/pillar/mysql/ pillar directory and edit the packages.sls file, adding the new packages we want. Then in our /srv/pillar/top.sls pillar file we can even target ‘*mysql* to apply this pillar to all minions with mysql in the name.

One thing I’d love to see is simple list merging. As things are today, each salt/pillar/*/package.sls pillar file MUST contain ALL packages you want to install on the images. If we tried to do something like the following

#/srv/salt/top.sls base: '*': - core.packages 'client01p': - erlang.packages

One would assume that the wild card will install our base packages, and we would only need to place the erlang specific packages in our /srv/pillar/erlang/packages.sls file. This is not the case. they will overwrite each other, leaving the minion only with the packages listed in /srv/pillar/erlang/packages.sls

I hope this helps clear some stuff up. In part two I’ll be moving forward and showing you guys how to use pillars, states, and map.jinja files to download erlang from source and compile.

Stay tuned.