Often when writing ansible playbooks you might find yourself needing to perform some text transformation. When doing simple things like direct data type conversion or running an algorithm against the string this can usually be easily performed by the built in Jinja2 filters in ansible.

# For human readable output, you can use: as_json: "{{ some_variable | to_nice_json }}" as_yaml: "{{ some_variable | to_nice_yaml }}" # Get the min and max of lists min_num: "{{ [3, 4, 2] | min }}" max_num: "{{ [3, 4, 2] | max }}" # Perform complex operations on strings sha1_hash: "{{ 'test1' | hash('sha1') }}" checksum: "{{ 'test1' | checksum }}"

However, when we being to have complex text transformation needs using just the built-in ansible filters you will probably end up in the same spot each time, an unreadable string of chained filters. I'll go over how you can make these complex transformations easier by using custom filters in your ansible playbooks so that they are readable and reusable.

Creating a custom filter

Take this example for a good candidate for a custom ansible filter. We have an S3 bucket with each top-level folder representing a package. We want to create a playbook to audit the bucket and find each top-level folder present in the bucket.

However, if we were to perform a list on the s3 bucket we would get the following output.

- folder1/file1.js - folder1/sub-folder/file1.js - folder2/ - folder3/file1.js - folder3/file2.js - folder3/file3.js

This list shows the key of each object in the bucket as the concept of folders is emulated by using paths in keys. Now to get each top-level folder in the bucket we can simply get the first folder in the path and filter these folders to there unique items. And hopefully get something like below:

folder1 folder2 folder3

To create a new filter module just create a .py file in a directory named filter_plugins (You can name this file anything, all .py files are automatically picked up from the filter_plugins folder) that is adjacent to our playbook in the file tree. We will create a python module that defines a FilterModule class. This class must have a method called filters that returns a map of names and callable implementations for the filter.

To create a custom filter to gather the unique root folders of our bucket we could write something like:

#!/usr/bin/python class FilterModule(object): def filters(self): return {'unique_folders': self.unique_base_paths} def unique_base_paths(self, keys): # A Set allows for no duplication of keys base_paths = set() for key in keys: # A path contains a directory if it has a "/" if "/" in key: # Get the first folder in each path base_folder = key.split("/")[0] base_paths.add(base_folder) # Sets aren't handled by ansible so convert to list return list(base_paths)

Our custom ansible functions can now be called in our playbooks by calling our filter named unique_folders just like any other filter.

- hosts: localhost vars: deployment_bucket: name: "some-bucket-with-packages" region: ap-southeast-2 tasks: - name: List keys aws_s3: bucket: "{{ deployment_bucket.name }}" region: "{{ deployment_bucket.region }}" mode: list register: bucket_content - name: All bucket paths debug: var: bucket_content.s3_keys - name: Get all top level folders in bucket set_fact: deployed_packages: "{{ bucket_content | unique_folders }}" - name: Unique top level folders debug: var: deployed_packages

And once we run our playbook the final debug output should show just the top level folders in the bucket:

TASK [Unique top level folders] ********************* ok: [localhost] => { "deployed_packages": [ "folder1", "folder2", "folder3" ] }

Final thoughts

Custom filters are a powerful tool to move some of your complex text transformations to some more readable python code rather than the alternative of long chaining of the built-in ansible filters. These filters as they are just python code are reusable across projects allowing you to build useful filters for validating your custom data types and structures easily in your ansible playbooks.

Further reading

Cover photo by Anders Jildén on Unsplash