What's new in Salt 3000 Neon 33 minute read Updated: Jun 9, 2020

This is an unofficial summary of what’s new in the Salt Neon release. As with Salt Fluorine, it also started as a series of tweets and mostly mentions new features. If you want to read about other changes and deprecations (for example, RAET is gone), then go read the official release notes and the new handcrafted changelog.

UPDATE (2020.05.20): the 3000.3 bugfix release is available and is strongly recommended instead of 3000.

What's new in Salt 3000 Neon: Chroot, Saltcheck, Unless/Onlyif, Slots, Multiple instances, Cloud, Mine, Slack Webhooks, Nifty tricks

New release strategy

Starting with the Neon release, Salt adopted a new, single branch release strategy and a non-date based version schema beginning at 3000. The version format is MAJOR.PATCH. For a planned release containing features and bug fixes, the MAJOR version will be incremented. The expected release cadence is 3 - 4 months. Since all future development is happening in the master branch (except for critical bugfixes), this means that most of the PRs from the old neon and develop branches are not included in this release and need to be backported. Read the SEP 14, FAQ, and watch the Salt Office Hours for more details.

Security note

For historical reasons, Salt requires PyCrypto as a “lowest common denominator”. However, PyCrypto is unmaintained and best practice is to manually upgrade to use a more maintained library such as PyCryptodome.

There is a picking order as to which package is used:

PyCrypto - basic level

PyCryptodome - preferred over PyCrypto if installed

M2Crypto - preferred over PyCryptodome and PyCrypto if installed

Due to issues found during package testing for the several supported Linux distributions, Salt temporarily switched back the crypto depencency to PyCrypto.

See the relevant issues: #52674, #56039, #56095

Chroot support

The first execution module (inspired by dockermod ) can execute Salt modules and states in a chroot environment. It provides the following functions:

chroot.exist (e.g., salt minion1 chroot.exist /chroot , although exists would be a better name)

(e.g., , although would be a better name) chroot.create ( salt minion1 chroot.create /chroot )

( ) chroot.call ( salt minion1 chroot.call /chroot test.ping )

( ) chroot.apply ( salt minion1 chroot.apply /chroot state_name )

( ) chroot.sls ( salt minion1 chroot.sls /chroot state_name pillar='{"foo": "bar"}' )

( ) chroot.highstate ( salt minion1 chroot.highstate /chroot )

There are two requirements for this to work:

chroot.create creates an empty chroot directory (with dev and proc dirs) and doesn’t install any binaries there (you have to do so yourself) chroot.call , chroot.apply , chroot.sls and chroot.highstate unpack a Salt thin archive and run it inside a chroot, so having Python installed is also necessary

The second execution module can freeze and restore package sets. It provides the following functions:

freezer.list

freezer.status

freezer.freeze

freezer.restore

The produced freeze files (lists of repos and packages) are in yaml format and stored in the /var/cache/salt/minion/freezer folder.

The chroot feature also brings a few related changes:

The cmd.run_chroot function got the binds option to export directories from the parent system. Also, now it mounts the /sys directory.

function got the option to export directories from the parent system. Also, now it mounts the directory. The cmd.wait and cmd.run states gained the root parameter to run commands inside a chroot

and states gained the parameter to run commands inside a chroot Many functions in the systemd_service module also got the optional root parameter to support chroot environments

module also got the optional parameter to support chroot environments The same thing happened with groupadd , shadow (if the link doesn’t work, ask for this PR to get merged), and useradd modules. Unfortunately, this is not supported in the corresponding state modules, so you can’t write a state to manage users inside a chroot environment (unless you resort to module.run ).

Details

Saltcheck was introduced by William Cannon in Salt 2013.3.0 and is probably the easiest (built-in) way to test your Salt states automatically, and also validate any existing infrastructure that is not managed with Salt. Saltcheck uses Salt execution modules and a bit of YAML/Jinja (or any other renderer syntax) to make assertions against the desired system state. It may look like someone who saves himself from being drowned by pulling on his own hair, but actually, it is quite effective. Here is an example:

{ # baron/munchausen.sls #} /tmp/swamp.txt : file.managed : - contents : | Münchhausen

{ # baron/saltcheck-tests/muchausen.tst #} ensure_the_swamp_contains_munchausen : module_and_function : file.search args : - /tmp/swamp.txt - Münchhausen assertion : assertTrue

% sudo salt minion1 state.apply baron.munchausen minion1: ---------- ID: /tmp/swamp.txt Function: file.managed Result: True Comment: File /tmp/swamp.txt updated Started: 11:03:04.452589 Duration: 10.000000009313226 ms Changes: ---------- diff: New file Summary for minion1 ------------ Succeeded: 1 (changed=1) Failed: 0 ------------ Total states run: 1 Total run time: 10.000 ms

% sudo salt minion1 saltcheck.run_state_tests baron.munchausen minion1: |_ ---------- baron.munchausen: ---------- ensure_the_swamp_contains_munchausen: ---------- duration: 3.9526 status: Pass |_ ---------- TEST RESULTS: ---------- Execution Time: 3.9526 Failed: 0 Missing Tests: 0 Passed: 1 Skipped: 0

In Salt Neon it gained several improvements:

support for saltenv environments

ability to associate tests with states by naming convention (or use check_all=True to run all tests in the state’s saltcheck-tests directory)

to run all tests in the state’s directory) ability to run tests over salt-ssh

assertEmpty and assertNotEmpty checks

and checks skip keyword to skip a test

keyword to skip a test print_result keyword to show assertion results

keyword to show assertion results assertion_section and assertion_section_delimiter keywords (useful to inspect nested dictionaries or lists returned by the module_and_function )

and keywords (useful to inspect nested dictionaries or lists returned by the ) saltcheck.state_apply function for test setup or teardown

function for test setup or teardown saltcheck.report_highstate_tests function to report on tests for states assigned to the minion through highstate

function to report on tests for states assigned to the minion through highstate ability to inject pillars via pillar-data keyword (useful for setup/teardown states)

keyword (useful for setup/teardown states) saltcheck_test_location minion configuration setting

minion configuration setting test duration display in the output

If you haven’t written any tests for your Salt states yet, give it a try. You’ll have an automated safety net and less need to pull out your hair 🙂

Details

Also, Christian wrote an intro titled Validation with SaltStack’s Saltcheck that contains examples of how to:

write a test for Salt formula

reuse a map.jinja in a Saltcheck test

in a Saltcheck test access pillar data

validate an existing Hadoop cluster deployment

validate the Windows registry

Cloud modules

Tencent Cloud

A module for Tencent Cloud (the 2nd largest cloud provider in China). It requires the tencentcloud-sdk-python package and provides the following functions:

avail_locations

avail_images

avail_sizes

list_securitygroups

list_custom_images

list_availability_zones

list_nodes

list_nodes_full

list_nodes_select

list_nodes_min

create

start

stop

reboot

destroy

script

show_image

show_instance

show_disk

PR #54526 by likexian (docs: salt.cloud.tencentcloud)

Azure ARM DNS modules

Provide CRUD operations on zones and record sets:

record_set_create_or_update

record_set_delete

record_set_get

record_sets_list_by_type

record_sets_list_by_dns_zone

zone_create_or_update

zone_delete

zone_get

zones_list_by_resource_group

zones_list

And a couple of states:

zone_present

zone_absent

record_set_present

record_set_absent

PR #55424 by Nicholas Hughes

AWS SSM

New boto_ssm module to manage AWS Systems Manager parameters:

get_parameter

put_parameter

delete_parameter

PR #54982 by Christian McHugh

AWS Elasticsearch

Functions:

add_tags

cancel_elasticsearch_service_software_update

create_elasticsearch_domain

delete_elasticsearch_domain

delete_elasticsearch_service_role

describe_elasticsearch_domain

describe_elasticsearch_domain_config

describe_elasticsearch_domains

describe_elasticsearch_instance_type_limits

describe_reserved_elasticsearch_instance_offerings

describe_reserved_elasticsearch_instances

get_compatible_elasticsearch_versions

get_upgrade_history

get_upgrade_status

list_domain_names

list_elasticsearch_instance_types

list_elasticsearch_versions

list_tags

purchase_reserved_elasticsearch_instance_offering

remove_tags

start_elasticsearch_service_software_update

update_elasticsearch_domain_config

upgrade_elasticsearch_domain

exists

wait_for_upgrade

check_upgrade_eligibility

States:

present

absent

upgraded

latest

tagged

PR #55768 by Herbert (docs: salt.modules.boto3_elasticsearch, salt.states.boto3_elasticsearch)

vSphere tagging

Add tagging ability to vCenter proxymodule through the following functions:

list_tag_categories

list_tags

attach_tag

list_attached_tags

create_tag_category

delete_tag_category

create_tag

delete_tag

PR #54058 by @xeacott (docs: salt.modules.vsphere)

Update to venafi pillar and runner to restore interoperability with Venafi Cloud (automated certificate issuance service). Also adds support for Venafi Trust Protection Platform. Requires the vcert-python library.

PR #55858 by Ryan Treat and Aleksander Rykalin

Other cloud features

Modify the Proxmox VM when using clone in the salt-cloud profile if settings are present. PR #55060 by Brian Sidebotham and Akmod

and Add the internal flag to openvswitch . PR #55666 by Akmod

flag to . PR #55666 by Fix nova module cooperation with python-novaclient>6.0.1 . PR #49977 by @slivik

Module calls in unless/onlyif

This is probably THE coolest feature in Salt Neon. It allows using any execution module in onlyif/unless requisites (at runtime, unlike Jinja):

{%- from 'postgresql/map.jinja' import pg with context -%} Enable idle_in_transaction_session_timeout for Postgres 9.6 and later : file.managed : - name : {{ pg.conf_d }}/session_timeout.conf - contents : 'idle_in_transaction_session_timeout = 60000' - watch_in : service : postgresql - onlyif : - fun : postgres.psql_query query : "SELECT setting FROM pg_catalog.pg_settings WHERE name = 'server_version_num' AND setting::int >= 90600" runas : postgres postgresql : service.running : - enable : True - reload : True

CAVEAT: Unfortunately, this feature doesn’t work everywhere. The following state modules have their own mod_run_check implementations that do not support module calls:

states.cmd

states.docker_container

states.git

This is going to be fixed in Salt Sodium, see the open PR #55974 by Christian McHugh

PRs #51846 and #52969 by Daniel Wallace and Christian McHugh (docs: Unless and onlyif Enhancements)

Slots

Slots were introduced in Salt 2018.3.0 with the following disclaimer: This functionality is under development and could be changed in future releases . Since then, several issues were filed but haven’t received enough attention from SaltStack employees in the previous major release (2019.2.0, which is almost a year). Another year later, in Salt Neon, some of the issues were fixed with help from the community:

Extended slot syntax to support dictionary lookups

Ability to append strings to slot results

Support slot parsing inside dicts and lists

Ability to use slots as unless/onlyif function arguments

{% set user = 'joeblade' %} zshrc for {{ user }} : file.managed : # Dictionary lookup and appended text - name : __slot__ : salt : user.info({{ user }}).home ~ /.zshrc - user : {{ user }} - group : {{ user }} - template : jinja - contents : | # ALMIGHTY SYSADMIN, PLEASE DO [NOT] MANAGE THIS FILE FOR ME autoload -Uz compinit compinit alias l= 'ls -a' alias ll= 'ls -lF' alias la= 'ls -laF' alias ..= 'cd ..' alias ...= 'cd ../..' alias ....= 'cd ../../..' alias .....= 'cd ../../../..' alias cd= ' builtin cd' - context : # Context is not used in this example and is here only # to demonstrate that slot parsing works inside dicts user_name : __slot__ : salt : user.info({{ user }}).fullname - unless : - fun : file.search args : # Slot as unless argument - __slot__ : salt : user.info({{ user }}).home ~ /.zshrc - "ALMIGHTY SYSADMIN, PLEASE DO NOT MANAGE THIS FILE FOR ME" ignore_if_missing : True

Multiple instances

Engines

Existing Salt engines are singletons, i.e. can interact with only one thing. In Salt Neon, it is possible to run multiple instances of any particular engine. Just specify a different engine alias and use the same engine_module parameter:

engines : - production_logstash : engine_module : logstash host : production_log.my_network.com port : 5959 proto : tcp - develop_logstash : engine_module : logstash host : develop_log.my_network.com port : 5959 proto : tcp

The PR is titled Initial work to allow running multiple instances of a Salt engine . The word “initial” could mean that some bits are not implemented yet. During my tests, it worked fine; however, I haven’t tried running anything complex like the reactor engine . Some engines like stalekey are inherently singular and can’t work in parallel. Also, I think it would be helpful if engine name was appended to the process name (when the setproctitle module is installed), because right now it is not apparent what engine a MinionProcessManager runs:

% ps ax | grep '[s]alt' 15988 ? Ss 0:00 /usr/bin/python3 /usr/bin/salt-minion 16014 ? Sl 0:03 /usr/bin/python3 /usr/bin/salt-minion KeepAlive MultiMinionProcessManager MinionProcessManager 16017 ? S 0:00 /usr/bin/python3 /usr/bin/salt-minion KeepAlive MultiprocessingLoggingQueue 16044 ? S 0:00 /usr/bin/python3 /usr/bin/salt-minion KeepAlive MultiMinionProcessManager MinionProcessManager 16045 ? S 0:00 /usr/bin/python3 /usr/bin/salt-minion KeepAlive MultiMinionProcessManager MinionProcessManager

PR #50059 by Gareth J. Greenaway (docs: Engines)

Beacons

Unlike engines, some Salt beacons can monitor multiple things. For example, inotify beacon can watch multiple files. Other beacons (e.g., log ) can monitor only one thing. In Salt Neon, it is possible to run multiple instances of any Salt beacon. Just specify a different beacon alias and use the same beacon_module parameter:

beacons : watch_importand_file : - beacon_module : inotify - files : /etc/important_file : {} watch_another_file : - beacon_module : inotify - files : /etc/another_file : {}

Again, the word “initial” in the commit message could potentially mean that some bits are not implemented yet.

PR #55794 by Gareth J. Greenaway (docs: Configuring beacons)

Metaproxy

The description is quite vague: This PR creates a new loadable module type called a metaproxy and abstracts the existing proxy minion to become a type of metaproxy. This enables us to build different types of proxy minions that can still load existing proxymodules . Below is my personal guess on what this feature could actually mean.

A few facts about proxy minions:

Proxy minions enable controlling devices that cannot run a standard salt-minion (e.g., a network gear that has an API, devices with limited CPU or memory, devices that have SSH but no Python, or devices that should not run a minion for security reasons)

A separate proxy process is required for each device: https://docs.saltstack.com/en/latest/topics/proxyminion/index.html#getting-started

Each proxy process consumes up to 40-70MB of memory. To manage 100 devices, you need to start 100 different proxy processes (who wants to do that for countless console servers?): https://github.com/saltstack/salt/issues/34132

If you have lots of devices, it could be necessary to distribute proxies between different servers or Docker containers: https://mirceaulinic.net/2018-09-27-network-automation-at-scale/

A few details about the metaproxy PR:

It touches loadable matchers that were introduced in Salt 2019.2.0

Matchers are used for minion/device targeting

PR #48809 that introduced loadable matchers, states that currently new matchers cannot be created because the required plumbing for the CLI does not exist yet

In Salt Neon, a CLI interface for custom matchers is not yet materialized (this means it was not a priority, and custom matchers are fine as is)

Both PRs were submitted by C. R. Oldham , the original engineer and inventor of the Salt proxy system

Given all of the above, I think the metaproxy system is designed to control multiple devices from a single proxy process (probably, one per device vendor). It could save a lot of memory/CPU resources and avoid the need to manage countless proxy minion processes.

And since the PR contains no alternative proxy implementations (and no metaproxy-capable matcher modules), I think the actual solution will be offered only in Salt Enterprise.

After this feature was submitted, a few things happened:

In June 2019, an open-source salt-sproxy (Salt Super-Proxy) package was released by Mircea Ulinic. It uses agentless architecture (similar to Salt SSH or Ansible), and I think it pretty much solves the same problem as the Enterprise Metaproxy - manage more devices with fewer resources. As of Apr 30, 2020 it was downloaded more than 20k times. In July 2019, metaproxy was backported to 2019.2.1. It was done against the official development policy, which says that new features should go into the develop branch. As a result, it completely broke the salt-proxy process. Finally, in November 2019, it was announced that the missing Enterprise Metaproxy piece is called Delta Proxy. SaltStack did consider open-sourcing it , but I was informed that the decision hadn’t been made, and it is better not to expect it in the foreseeable future.

<RANT>

I honestly believe that this is a tough position to be in (i.e., decide which feature should be open and which one shouldn’t). It creates an inherent tension between the open-source community and the company, plus wastes a lot of efforts.

For example, there are multiple attempts to build an open-source GUI for Salt: Alcali, Silica, Molten, SaltGUI, Foreman Salt, Salt Dash, SaltPad, Obdi, Saltshaker, and also Uyuni, the upstream project for SUSE Manager. Instead, those efforts could be spent on improving the official one if it was open.

While it is hard to make money on Open Source projects, I believe that a better way is to have a single open codebase (without artificial Elasticsearch-like license restrictions) and sell high-order (or orthogonal) products and services.

</RANT>

A type of metaproxy is controlled by the following proxy config option (currently, custom metaproxy modules can’t be loaded/synchronized via the fileserver’s _metaproxy folder):

metaproxy : proxy

Since the Delta Proxy isn’t open-source (hopefully yet), the Metaproxy feature has no practical use. If you can provide more insights about Metaproxy or Delta Proxy, I’ll be happy to update this section of the post.

PRs #50183 and #53616 by C. R. Oldham.

Salt Mine

Previously, all mine functions (and their data) were retrievable by all minions. In Salt Neon, it is possible to define a minion-side ACL. When a minion requests a function from the Salt Mine that is not allowed to be requested by that minion, it will get no data, just as if the requested function is not present in the Salt Mine.

mine_functions : network.ip_addrs : - interface : eth0 - cidr : 10.0 . 0.0 / 8 - allow_tgt : 'I@role:master' - allow_tgt_type : 'compound'

When Salt master receives mine data from a minion, it also receives the defined ACL and stores it together with the data. When another minion requests the mine data, Salt master checks the minion ID against the stored ACL. Obviously, the ACL will be secure only if the targeting criteria can’t be spoofed by a rogue minion (e.g., grains are insecure).

In addition to that, the format to define mine_functions has been extended to allow lists (like in module.run ), to support both positional and keyword arguments:

# Old format mine_functions : test.arg : foo : foo bar : bar # New format mine_functions : test.arg : - foo - bar - kwarg1 : foo - kwarg2 : bar

The format to define mine aliases also has been extended:

# Old format mine_functions : test_alias : - mine_function : test.arg - foo - bar # New format mine_functions : test_alias : - mine_function : test.arg - foo - bar - kwarg1 : foo - kwarg2 : bar

Both formats are supported simultaneously without any config options or deprecations!

CAVEAT: Unfortunately, it looks like this change broke the mine data format if you use an older master with the new minion: #56118.

PR #55760 by Herbert (docs: Mine functions)

Event relay engines

Fluent engine

The fluent engine reads messages from the Salt event bus and pushes them onto a fluentd endpoint.

Engine configuration (needs the fluent-logger-python module installed):

engines : - fluent : host : localhost port : 24224 app : engine

Example fluentd configuration:

<source> @type forward port 24224 </source> <match saltstack.** > @type file path /var/log/td-agent/saltstack </match>

PR #55711 by Christian McHugh (docs: salt.engines.fluent)

Script engine

Sometimes it is necessary to consume a structured event log (or an external data source) and inject the result into Salt’s event bus. The script engine creates events based on a command’s output and can run on a Salt master or minion. It reads the output line by line and deserializes each line into a data structure using any serializer. If the resulting data structure contains the tag key, then an event will be fired on the event bus (with an optional data payload).

#/etc/salt/minion.d/engines.conf engines : - script : cmd : /etc/salt/script.sh output : json

#!/bin/bash # /etc/salt/script.sh while true ; do echo -e '{"tag": "thermal_zone0", "data": {"temp": "' $( cat /sys/class/thermal/thermal_zone0/temp ) '"}}' sleep 5 done

% sudo salt-run state.event pretty=True thermal_zone 0 { "_stamp" : "2019-09-04T15:08:46.220902" , "cmd" : "_minion_event" , "data" : { "id" : "minion1" , "temp" : "50000" }, "id" : "minion1" , "pretag" : null , "tag" : "thermal_zone0" } thermal_zone 0 { "_stamp" : "2019-09-04T15:08:51.224414" , "cmd" : "_minion_event" , "data" : { "id" : "minion1" , "temp" : "50000" }, "id" : "minion1" , "pretag" : null , "tag" : "thermal_zone0" }

If a script doesn’t run indefinitely, you can run it periodically by adding interval: N to the engine configuration.

Two caveats:

json deserializer doesn’t like empty lines

deserializer doesn’t like empty lines The data['id'] attribute could be easily spoofed by a rogue minion. Use the event id key instead.

PR #50005 by austin (docs: salt.engines.script)

Certificate expiration beacon

SSL/TLS certificates tend to expire at inappropriate times. If you do not have any certificate renewal automation (e.g., Certbot), it is worth setting up expiry reminders. You can use the new cert_info beacon to do so:

# This can go either to a minion config or to a pillar beacons : cert_info : - files : - /tmp/puppet.pem - notify_days : 45 - interval : 86400

When a certificate is about to expire, you will get a Salt event that looks like this:

% sudo salt-run state.event 'salt/beacon/*/cert_info/' pretty=True

salt/beacon/minion 1 /cert_info/ { "_stamp" : "2020-01-09T09:21:43.184517" , "certificates" : [ { "cert_path" : "/tmp/puppet.pem" , "extensions" : [ { "ext_data" : "DNS:puppet.com, DNS:docs.puppet.com, DNS:www.puppet.com" , "ext_name" : "subjectAltName" } ], "has_expired" : true , "issuer" : "C=\"FR\",ST=\"Paris\",L=\"Paris\",O=\"Gandi\",CN=\"Gandi Standard SSL CA 2\"" , "issuer_dict" : { "C" : "FR" , "CN" : "Gandi Standard SSL CA 2" , "L" : "Paris" , "O" : "Gandi" , "ST" : "Paris" }, "notAfter" : "2019-11-05 23:59:59Z" , "notAfter_raw" : "20191105235959Z" , "notBefore" : "2018-11-05 00:00:00Z" , "notBefore_raw" : "20181105000000Z" , "serial_number" : "333090271171444178457395344333836335102" , "signature_algorithm" : "sha256WithRSAEncryption" , "subject" : "OU=\"Domain Control Validated\",OU=\"PositiveSSL Multi-Domain\",CN=\"puppet.com\"" , "subject_dict" : { "CN" : "puppet.com" , "OU" : "PositiveSSL Multi-Domain" }, "version" : 2 } ], "id" : "minion1" }

Now let’s do something more useful - forward these events to a Slack channel using the new webhook-based state module.

1. Place the following snippet into /etc/salt/master.d/reactor.conf

reactor : - salt/beacon/ */cert_info/: - /srv/salt/reactor/cert_info.sls slack : # Create an incoming webhook here: https://api.slack.com/messaging/webhooks hook : https : //hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX

2. Define the reactor state using the new json_query Jinja filter

# /srv/salt/reactor/cert_info.sls Send expired certs to Slack : runner.state.orchestrate : - args : - mods : slack - pillar : expired_cert_minion : {{ data [ 'id' ] }} expired_cert_message : | {{ data['certificates'] | json_query("[].[cert_path, extensions[?ext_name=='subjectAltName'].ext_data | [0], notAfter] | map(&[] | join(' :: ', @), @) | join('

', @)") }}

Do not forget to run sudo apt-get install jmespath on the Salt master to enable the filter.

3. Define the Slack notification state

# /srv/salt/slack.sls slack-message : slack.post_message : - message : | The following certificate(s) are about to expire on "{{ pillar["expired_cert_minion"] }}": {{ pillar [ "expired_cert_message" ] }} - webhook : {{ salt [ 'config.get' ] ( 'slack:hook' ) }}

And then restart the Salt master. You should see something like this when a certificate is about to expire:

PR #54902 by Nicholas Hughes

Support for Slack webhooks

Slack state

The slack.post_message state has been updated to support webhooks (in addition to API keys). If a webhook argument is defined, the state will use it to post the message:

slack-message : slack.post_message : - message : Hello there! - username : SaltStack - icon_emoji : ":glitch_crab:" - channel : "#random" - webhook : https : //hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX

For security reasons, it is better to store the webhook URL in a master or minion config:

slack : hook : https : //hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX

And then use - webhook: {{ salt['config.get']('slack:hook') }} in your state instead of the URL.

The old way of using API keys is still supported:

slack-message : slack.post_message : - message : Hello there! - from_name : SaltStack - icon : https : //cdn.mirantis.com/wp-content/uploads/ 2017 / 02 /image01.png - channel : "#random" - api_key : AAABBBCCC

PR #52715 by Gareth J. Greenaway (docs: salt.states.slack.post_message)

Slack webhook returner

The slack_webhook is a new webhook returner based on the existing slack returner that uses API keys.

Place the following snippet into /etc/salt/minion.d/slack.conf and restart the minion:

slack_webhook : webhook : T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX show_tasks : true

Then you’ll be able to use the returner as follows:

% sudo salt minion1 state.sls neon.warning --return slack_webhook

CAVEAT: It only works with state.apply , state.sls , and state.highstate functions. This limitation will be fixed in #55968.

PR #55342 by Carlos D. Álvaro

Misc modules

XML module and state

This is a simple XML execution module to get and set XML attributes and values. The corresponding state module has just one function: xml.value_present .

# rss.sls HackerNews : file.managed : - name : /tmp/hn.rss # Also try https://hnrss.org/newest - source : https : //hnrss.org/frontpage ? count= 30 - skip_verify : true {% set idx = salt [ 'random.rand_int' ] (start= 1 , end= 30 ) %} Random HN item : module.run : - name : test.arg - kwargs : title : __slot__ : salt : xml.get_value(/tmp/hn.rss , channel/item [ {{ idx }} ] /title) url : __slot__ : salt : xml.get_value(/tmp/hn.rss , channel/item [ {{ idx }} ] /link)

% sudo salt minion 1 state.apply rss --out json | \ jq '. [][] | select(.__id__ | contains( "Random HN item" )) | .changes.ret.kwargs' { "title" : "The Terrors and Joys of Terraform" , "url" : "https://medium.com/driven-by-code/the-terrors-and-joys-of-terraform-88bbd1aa4359" }

PR: #54977 by Christian McHugh (docs: XML module)

ssh_auth.manage state

This state makes unnecessary having both ssh_auth.present and ssh_auth.absent to control authorized ssh keys. Now you can specify a single list of authorized keys to ensure that there are no stale ones:

all_ssh_keys : ssh_auth.manage : - user : joeblade - enc : ssh-dss - options : - option1= "value1" - ssh_keys : - AAAABBBBCCCCDDD - ssh-rsa DDDDEEEEFFFFGGGG== colin@swinbourne.local - GGGGHHHHIIIIJJJJ== interceptor@micros.local

PR #54981 by Christian McHugh (docs: ssh_auth.manage )

Keystore module

New keystore execution and state modules to manage Java Keystore files are shipped in Salt Neon. To enable them, install the pyjks Python library.

The following execution functions are provided:

keystore.list ( salt minion1 keystore.list /tmp/test.store changeit )

( ) keystore.add ( salt minion1 keystore.add aliasname /tmp/test.store changeit /tmp/testcert.crt )

( ) keystore.remove ( salt minion1 keystore.remove aliasname /tmp/test.store changeit )

# salt-call keystore.list /path/to/keystore.jks changeit local: | _ ---------- alias: hostname1 expired: True sha1: CB:5E:DE:50:57:99:51:87:8E:2E:67:13:C5:3B:E9:38:EB:23:7E:40 type: TrustedCertEntry valid_start: August 22 2012 valid_until: August 21 2017

The state module contains the keystore.managed state:

define_keystore: keystore.managed: - name: /tmp/statestore.jks - passphrase: changeit - force_remove: True - entries: - alias: hostname1 certificate: /tmp/testcert.crt - alias: remotehost certificate: /tmp/512.cert private_key: /tmp/512.key - alias: stringhost certificate: | -----BEGIN CERTIFICATE----- MIICEjCCAX Hn+GmxZA -----END CERTIFICATE-----

PR #54991 by Christian McHugh (docs: salt.modules.keystore, salt.states.keystore)

IIS webconfiguration settings

A module and state to manage IIS WebConfiguration settings:

win_iis.get_webconfiguration_settings function

function win_iis.set_webconfiguration_settings function

function win_iis.webconfiguration_settings state

PR #54879 by Thomas Lemarchand (docs: salt.modules.win_iis, salt.states.win_iis)

Elasticsearch functions

elasticsearch.cluster_get_settings

elasticsearch.cluster_put_settings

elasticsearch.flush_synced

elasticsearch.index_get_settings

elasticsearch.index_put_settings

PRs #54917 and #55721 by Proskurin Kirill by Proskurin Kirill (docs: salt.modules.elasticsearch)

RabbitMQ upstreams management

rabbitmq.upstream_exists function

function rabbitmq.list_upstreams function

function rabbitmq.set_upstream function

function rabbitmq.delete_upstream function

function rabbitmq_upstream.present state

state rabbitmq_upstream.absent state

PR #55767 by Herbert (docs: salt.modules.rabbitmq, salt.states.rabbitmq_upstream)

Django migrations

A little helper function to apply Django migrations:

% sudo salt minion1 django.migrate <settings_module>

Or via state:

From South to North : module.run : - name : django.migrate - settings_module : my_django_app.settings

PR #54632 by @jrbeilke (docs: django.migrate)

Custom module helpers

saltutil state

Previously, it was only possible to run saltutil.sync_* functions via module.run , and it always reported changes. Now there is a corresponding state module that provides the following states:

saltutil.sync_all

saltutil.sync_beacons

saltutil.sync_clouds

saltutil.sync_engines

saltutil.sync_executors

saltutil.sync_grains

saltutil.sync_log_handlers

saltutil.sync_matchers

saltutil.sync_modules

saltutil.sync_output , saltutil.sync_outputters

, saltutil.sync_pillar

saltutil.sync_proxymodules

saltutil.sync_renderers

saltutil.sync_returners

saltutil.sync_sdb

saltutil.sync_serializers

saltutil.sync_states

saltutil.sync_thorium

saltutil.sync_utils

The usage is straightforward:

sync_everything : saltutil.sync_all : - refresh : True

Test mode ( test=True ) is supported, but it doesn’t actually test for possible changes and just prints saltutil.sync_* would have been run .

PRs #50197 by Christian McHugh and #51900 by Max Arnold (docs: salt.states.saltutil)

Ability to sync custom executor modules

Allow syncing custom executor modules from salt://_executors directory via the saltutil.sync_executors or saltutil.sync_all functions. Executor modules (do not confuse them with execution modules) wrap state execution calls globally, for example, to use sudo or docker.call .

PR #55190 by Matt Phillips and Max Arnold

__utils__ in grains

Since custom utils (synced via saltutil.sync_utils ) modules aren’t importable; it was impossible to use them in custom grain modules. Now you can call custom utility functions from grain functions via the loader:

# _utils/my_util.py def my_func (): return 'abc'

# _grains/my_grain.py def my_grain (): return { 'my_grain' : __utils__ [ 'my_util.my_func' ]()}

% sudo salt minion1 saltutil.sync_utils,saltutil.sync_grains , minion1: ---------- saltutil.sync_grains: - grains.my_grain saltutil.sync_utils: - utils.my_util

% sudo salt minion1 grains.get my_grain minion1: Hello there!

This is useful if you have a couple of custom grains that use common utility functions.

PR #49128 by @mirceaulinic

This is a new utility function to emit date-based deprecation warnings in Salt modules:

import salt.utils.versions salt . utils . versions . warn_until_date ( '20201201' , 'Please stop using X before {date} and instead use Y' )

After the specified date, the warning will turn into a RuntimeError .

Previously it was only possible to emit warnings based on release code names:

import salt.utils.versions salt . utils . versions . warn_until ( 'Neon' , 'RAET is deprecated and will be removed in Salt {version}' )

PR #55047 by Pedro Algarvio

Version-aware dependency decorator

The @salt.utils.decorators.depends decorator gained an optional version keyword. This decorator will check the module when it is loaded, check that the dependencies passed in are in the globals of the module and that the version requirements are met. If not, it will cause the function to be unloaded (or replaced if fallback_function is specified).

import salt.utils.decorators @depends ( 'botocore' , version = '1.12.21' ) def a_function_that_depends_on_botocore (): pass

PR #55590 by Herbert

filter_falsey

The salt.utils.data.filter_falsey function filters values from an iterable that evaluate to false (a typical use-case for most Boto calls). Removes None , {} and [] , 0 , '' , but does not remove False .

In [ 1 ]: import salt.utils.data In [ 2 ]: salt . utils . data . filter_falsey ({ 'foo' : None , 'bar' : True }) Out [ 2 ]: { 'bar' : True } In [ 3 ]: salt . utils . data . filter_falsey ([ True , False , None , {}, [{}, 'blah' , '' ]]) Out [ 3 ]: [ True , False , [{}, 'blah' , '' ]] In [ 3 ]: salt . utils . data . filter_falsey ( ... : [ True , False , None , {}, [{}, 'blah' , '' ]], ... : ignore_types = [ dict ], ... : recurse_depth = True ) Out [ 3 ]: [ True , False , {}, [{}, 'blah' ]]

PR #52499 by Herbert

recursive_diff

The salt.utils.data.recursive_diff function is able to produce a dict with old and new keys for any combination of nested maps or iterable types:

In [ 1 ]: import salt.utils.data In [ 2 ]: salt . utils . data . recursive_diff ( ... : { 'foo' : [ 1 , 2 , 3 ]}, ... : { 'foo' : [ 3 , 2 , 1 ]}) Out [ 2 ]: { 'new' : { 'foo' : [ 3 , 1 ]}, 'old' : { 'foo' : [ 1 , 3 ]}}

PR #55759 by Herbert

Jinja helpers

json_query filter

This is a port of Ansible json_query Jinja filter to make complex queries against JSON data structures. The query language parser depends on jmespath python library.

It could be used to filter pillar data, event data, yaml maps, and in combination with http_query . Can replace lots of ugly Jinja loops and simplify data parsing (especially if you use the TOFS pattern). In some cases, it can help to avoid writing trivial custom modules.

# json_query_example.sls {% set services = ' {"services": [ {"name": "http", "host": "1.2.3.4", "port": 80}, {"name": "smtp", "host": "1.2.3.5", "port": 25}, {"name": "ssh", "host": "1.2.3.6", "port": 22} ]}' | load_json %} {% set ports = services | json_query ( "services[].port" ) %} {% do salt.log.warning ( ports ) %} % sudo salt-call state.apply json_query_example -l warning [WARNING ] [80, 25, 22]

PR #50428 by Max Arnold (docs: json_query)

A helper to debug Jinja map files

Troubleshooting TOFS-based formulas could be tricky, especially when multiple yaml maps are merged and filtered inside a map.jinja file. The jinja execution module can help you debug these data structures:

% sudo salt minion1 jinja.import_yaml template-formula/defaults.yaml minion1: ---------- template: ---------- added_in_defaults: defaults_value config: /etc/template pkg: ---------- name: template rootgroup: root service: ---------- name: template winner: defaults

% sudo salt minion1 jinja.load_map template-formula/map.jinja defaults minion1: ---------- config: /etc/template.d/custom-ubuntu-18.04.conf pkg: ---------- name: template-ubuntu

PR #51047 by Erik Johnson (docs: salt.modules.jinja)

This feature adds a few functions to modify nested dicts, and exposes several Jinja filters (with support of configurable key delimiters and optional use of ordered dicts):

salt.utils.dictupdate.ensure_dict_key - a function to ensure that a dictionary contains the series of recursive keys

- a function to ensure that a dictionary contains the series of recursive keys salt.utils.dictupdate.set_dict_key_value - a function (and Jinja filter) to set a value in a nested dictionary without having to worry if all the nested keys actually exist

- a function (and Jinja filter) to set a value in a nested dictionary without having to worry if all the nested keys actually exist salt.utils.dictupdate.update_dict_key_value - a function (and Jinja filter) to update a dictionary nested (deep) in another dictionary without having to worry if all the nested keys actually exist

- a function (and Jinja filter) to update a dictionary nested (deep) in another dictionary without having to worry if all the nested keys actually exist salt.utils.dictupdate.append_dict_key_value - a function (and Jinja filter) to append to a list nested (deep) in a dictionary without having to worry if all the nested keys (or the list itself) actually exist

- a function (and Jinja filter) to append to a list nested (deep) in a dictionary without having to worry if all the nested keys (or the list itself) actually exist salt.utils.dictupdate.extend_dict_key_value - a function (and Jinja filter) to extend a list nested (deep) in a dictionary without having to worry if all the nested keys (or the list itself) actually exist

PR #52455 by Herbert

hmac_compute filter

The new hashutil.hmac_compute function and @hmac_compute Jinja filter to calculate an HMAC digest using SHA-256:

% sudo salt-call hashutil.hmac_compute "Hello there!" "A secret" local: 0c9d76aa8bcb2c4138af07e1f9d6a7393352948f42f3b18f3d8043c9242a260c % sudo salt-call slsutil.renderer default_renderer = jinja \ string = '{{ "Hello there!" | hmac_compute("A secret") }}' local: 0c9d76aa8bcb2c4138af07e1f9d6a7393352948f42f3b18f3d8043c9242a260c

PR #55506 by @Ajnbro

Camels and snakes

The following two functions could be used either in Jinja templates or directly in Python modules. The primary use-case seems to be converting Boto parameter names. The functions named camel_to_snake_case and snake_to_camel_case are located in the salt.utils.stringutils module. They are also exposed as Jinja filters to_snake_case and to_camelcase (yes, the naming is inconsistent). The usage is simple:

% sudo salt-call slsutil.renderer default_renderer = jinja \ string = "{{ 'snake_case_for_the_win' | to_camelcase }}" local: snakeCaseForTheWin

% sudo salt-call slsutil.renderer default_renderer = jinja \ string = "{{ 'camelsWillLoveThis' | to_snake_case }}" local: camels_will_love_this

PR #52458 by Herbert

macOS

PyObjC bindings

The PyObjC library is now shipped with the macOS package. It allows calling native Objective-C APIs from Python without shelling out. AFAIK, there are no builtin modules in Salt that use it (yet). However, you can check out the mosen/salt-osx repo to see what is possible.

PR #49657 by Wesley Whetstone (also backported to 2019.2.0)

Homebrew improvements

List the right namespace for cask packages from a tap different from the default one. The caskroom/cask/ namespace for brew-cask packages is deprecated in favor of homebrew/cask/ and will be removed in Sodium release.

PR #54216 by Carlos D. Álvaro

Virt improvements

Enhancements in package modules

Grains

HTTP improvements

Multimaster tweaks

Use UTC time (instead of local time) to generate job IDs. This is useful in SaltStack Enterprise deployments where masters are in different timezones so that jobs across the infrastructure can easily be sorted chronologically. PR #55557 by Michael Steed

Properly fire events to syndics when using list targeting and master is configured with order_masters . PR #55607 by Lukas Raska

Performance tweaks

Add performance tracing/logging to gitfs file_lists cache rebuild. PR #55420 by Duane Waddle

cache rebuild. PR #55420 by Don’t refresh modules twice per saltutil sync. PR #46713 by Dmitry Kuzmenko

sync. PR #46713 by Use ss filter to match TCP connections on Linux. PR #55501 by Andrea Agosti

filter to match TCP connections on Linux. PR #55501 by Add event_listen_queue_max_seconds option to cause the queue to be flushed regardless of how many events are in the queue. This is useful to prevent stale events when event_listen_queue on the master is set to large values. PR #53412 by C. R. Oldham

option to cause the queue to be flushed regardless of how many events are in the queue. This is useful to prevent stale events when on the master is set to large values. PR #53412 by Add git_pillar_update_interval option so the pillar data can be collected less frequently than the main loop_interval timer allows. PR #53621 by Mathieu Parent

Filesystems and partitions

Files and archives

New file.hardlink state. PR #55000 by Ali Rizvi-Santiago

state. PR #55000 by Add clean_parent argument to the archive.extracted state. It deletes the directory into which the archive is going to be extracted. PR #55418 by Proskurin Kirill

argument to the state. It deletes the directory into which the archive is going to be extracted. PR #55418 by Add skip_files_list_verify argument to the archive.extracted state. It prevents the archive from being extracted if its hash hasn’t changed. PR #55700 by Proskurin Kirill

Other notable features

Nifty tricks

loop.until_no_eval state

This is a generic state that blocks the execution process until a specific Salt function returns an expected result. Unlike the loop.until state it doesn’t use potentially unsafe Python eval() function. Instead, it can use a function from Python’s operator module, __salt__ , or __utils__ . Below is an example of how to wait until an Amazon ELB instance is healthy:

Wait for service to be healthy : loop.until_no_eval : - name : boto_elb.get_instance_health - expected : '0:state:InService' - compare_operator : data.subdict_match - period : 5 - timeout : 20 - args : - {{ elb }} - kwargs : keyid : {{ access_key }} key : {{ secret_key }} instances : "{{ instance }}"

PR #55639 by Herbert (docs: salt.states.loop)

salt_version execution module

Did you ever have to invent a workaround in a state file to support older Salt versions? Or maybe wrote formulas that worked across Salt versions? You might find this new module quite useful:

% sudo salt minion1 salt_version.equal Neon minion1: True

# And also in Jinja: {% if salt [ 'salt_version.equal' ]( 'Sodium' ) or salt [ 'salt_version.greater_than' ]( 'Sodium' ) %} superseded_syntax_for_module_run: module.run: - test.echo: - text: 'Salt Sodium or newer' {% else %} legacy_syntax_for_module_run: module.run: - name: test.echo - text: 'Older than Salt Sodium' {% endif %}

PR #55195 by Nicole Thomas and Max Arnold

Custom state warnings

You may want to set up a highstate process warning to notify yourself about something important. Below is a little helper that allows you to propagate custom warnings back to salt CLI output (unlike the salt.log.warning jinja function that is only visible in a minion log):

# warning.sls A warning : test.configurable_test_state : - result : True - warnings : - Hello - there

% sudo salt minion1 state.apply warning minion1: ---------- ID: A warning Function: test.configurable_test_state Result: True Comment: Started: 06:25:55.749509 Duration: 2.846 ms Changes: ---------- testing: ---------- new: Something pretended to change old: Unchanged Warnings: Hello there Summary for minion1 ------------ Succeeded: 1 (changed=1) Failed: 0 Warnings: 1 ------------

PR #53959 by Max Arnold

Enhanced config.option

The config.option function has been enhanced:

Lookup order is on par with config.get : minion config, grains, pillar, master options, sane defaults, explicit defaults

: minion config, grains, pillar, master options, sane defaults, explicit defaults Wildcards are supported when wildcard=True

New omit_all option to return only default option(s)

To view the sane defaults, you can use the following master and minion commands respectively:

% sudo salt-run salt.cmd config.option '*' omit_all = True wildcard = True % sudo salt minion1 config.option '*' omit_all = True wildcard = True

Additionally, configuration for Docker registries is no longer restricted only to pillar data and is now loaded using config.option (see #51531).

PR #55637 by Erik Johnson (docs: salt.modules.config.option)

Attaching grains to minion start events

Device onboarding often requires getting initial system attributes and storing them in a CMDB. Previously, the common way to do that was through the Salt reactor, which required an additional round trip. The new feature reduces the master load by automatically attaching any grains to the salt/minion/*/start event. To enable it, add the list of grains to a minion config file:

start_event_grains : - machine_id - uuid

After you restart the minion you’ll see the following events on the master bus:

minion_start { "_stamp" : "2020-01-15T07:40:13.655558" , "cmd" : "_minion_event" , "data" : "Minion minion1 started at Tue Jan 14 23:40:13 2020" , "grains" : { "machine_id" : "599118d5b9619443ac3166fb0e59349e" , "uuid" : "599118d5-b961-9443-ac31-66fb0e59349e" }, "id" : "minion1" , "pretag" : null , "tag" : "minion_start" } salt/minion/minion 1 /start { "_stamp" : "2020-01-15T07:40:13.662584" , "cmd" : "_minion_event" , "data" : "Minion minion1 started at Tue Jan 14 23:40:13 2020" , "grains" : { "machine_id" : "599118d5b9619443ac3166fb0e59349e" , "uuid" : "599118d5-b961-9443-ac31-66fb0e59349e" }, "id" : "minion1" , "pretag" : null , "tag" : "salt/minion/minion1/start" }

The event output above is a reminder to set the enable_legacy_startup_events: False in a minion config to get rid of the legacy event.

PRs #54948 and #55885 by Abid Mehmood (docs: start_event_grains)

minion_id domain removal

When there is no explicit minion_id defined in a minion config, Salt uses the following algorithm:

If an id_function name is defined in the minion config (with optional kwargs), it will be called to generate an id Otherwise, the salt.utils.network.generate_minion_id() will be invoked to generate an id based on host’s FQDN Then, if the minion_id_lowercase option is true , the resulting id will be lowercased Then the new minion_id_remove_domain option is considered And finally, the optional append_domain value is appended

The minion_id_remove_domain option can take the following values:

false (default) - do nothing A specific domain name (e.g., example.com ) - the domain name will be removed from a minion id ( minion.example.com -> minion , minion.foo.bar -> minion.foo.bar ) true - anything after the first dot will be removed from a minion id (e.g., minion.example.com -> minion , minion.foo.bar -> minion )

PR #54622 by @markuskramerIgitt (docs: minion_id_remove_domain)

Saltenv support in slsutil.renderer

The slsutil.renderer function is a useful way to debug Jinja templates. Now it can handle saltenvs:

% sudo salt minion1 slsutil.renderer salt://state.sls default_renderer = jinja saltenv = base

PR #52293 by Alexander Fischer

DSON outputter

This outputter deserves to be highlighted. It uses DSON format (Doge Serialized Object Notation) to represent the output of any Salt command. So needed! Just run pip install dogeon and then:

% sudo salt minion1 test . echo 'Such salty!' -- out dson such "minion1" is "Such salty!" wow

PR #49338 by Erik Johnson

Also, you can write your Salt states using DSON. The corresponding renderer was added in Salt 2016.11 after SaltConf 16. Much readable!