If you watched the ChefConf keynote,

attended last years’ community summits,

or follow our open source mailing lists, you’ve probably heard about

Policyfiles.

If you haven’t, here’s the deal: Policies are a new feature of Chef that

combine the very best parts of Roles, Environments, and client-side

dependency resolvers like Berkshelf into a single easy to use workflow.

Policies are built by defining a Policyfile, which looks similar to a

Chef Role combined with a Berksfile. When a Policy is ready for upload,

a workstation command included with the ChefDK compiles the Policyfile

into a Policyfile.lock file. This locked Policy, along with all of the

cookbooks it references, are treated as a single unit by the Chef

tooling. The bundle of Policyfile.lock and cookbooks are uploaded to the

server simultaneously. They are also promoted simultaneously through the

deployment lifecycle, from dev to QA to production.

Policies make your chef-client runs completely repeatable, because

cookbooks referenced in a Policy are identified by a unique hash based

on their contents. This means that once the lock file + cookbook bundle

has been generated, the code underlying it will never change.

For more information, see the

Policyfile README in the ChefDK repo.

Previously we’ve recommended that you only use Policyfiles in

specialized testing environments because using Policyfiles in

compatibility mode along side existing infrastructure could cause

unexpected behavior. With Chef Server 12.1 and ChefDK 0.7, Policyfile

data is now stored via specialized APIs, making it safe (and a lot

easier) to use Policyfiles in your existing Chef Server setup.

To help you get familiar with the workflows Policyfiles make possible,

we’ll walk through deploying a simple demo application using Policyfiles

to manage our dependencies across the application’s lifecycle.

The code here is based on

a demo my colleague created for the London Chef meetup.

It deploys the “Awesome Appliance Repair” Python application.

Getting Started With Local Development and Testing

To follow along with this example, you’ll need:

Initialize Shell

If you have previously installed ChefDK or Test Kitchen on your machine

using rubygems, you might have older versions of these tools in your

PATH. If that’s the case, you can use chef shell-init to setup your

environment variables, like so (I use zsh , be sure to update the

command for your shell):

eval "$(chef shell-init zsh)" which kitchen # => /opt/chefdk/bin/kitchen

Generate:

For this example, we’ll structure our code as if we were developing our

infrastructure code alongside the application, in the same source repo.

Note that Policyfiles don’t require you to manage your code this way,

you can use individual git repos per cookbook or a monolithic repo if

you prefer. Just follow along for now :)

We use chef generate to create the required files and directories for

us:

mkdir aar cd aar chef generate app . # Policyfiles will be the default someday, 'till then: chef generate policyfile

Commit

We’ll commit our work now so we can roll back to a fairly blank slate if

we make a mistake later:

git add . git commit -m 'initial policyfile demo commit'

Edit Policyfile:

We describe how we want Chef to compose our cookbooks to configure a

machine to run our application by editing the Policyfile.rb . This file

defines a few things:

name : This describes the kind of machine we are creating. We name

this “aar”, which is our abbreviation for “Awesome Appliance Repair.”

: This describes the kind of machine we are creating. We name this “aar”, which is our abbreviation for “Awesome Appliance Repair.” default_source : The place where we get shared cookbooks. The default

source is :community , which is the Chef Supermarket site. You can

also use an internal supermarket instance or a monolithic Chef Repo.

: The place where we get shared cookbooks. The default source is , which is the Chef Supermarket site. You can also use an internal supermarket instance or a monolithic Chef Repo. run_list : The list of recipes, in order, that you want Chef to

evaluate to configure your system. When using Policyfiles, you set the

run_list in the Policyfile instead of on the node.

: The list of recipes, in order, that you want Chef to evaluate to configure your system. When using Policyfiles, you set the in the Policyfile instead of on the node. cookbook : A Policyfile.rb can have multiple cookbook statements;

these can configure specific cookbooks to be loaded from alternative

sources or set additional version constraints on them.

: A can have multiple statements; these can configure specific cookbooks to be loaded from alternative sources or set additional version constraints on them. Default and override attributes: these define attributes at the ‘role’

precedence level. We’ll look at these later.

Edit the Policyfile.rb as follows:

# Policyfile.rb - Describe how you want Chef to build your system. # # For more information on the Policyfile feature, visit # https://github.com/opscode/chef-dk/blob/master/POLICYFILE_README.md # A name that describes what the system you're building with Chef does. name "aar" # Where to find external cookbooks: default_source :community # run_list: chef-client will run these recipes in the order specified. run_list "aar::default" # Specify a custom source for a single cookbook: cookbook "aar", path: "cookbooks/aar"

Chef Install

With our basic Policyfile.rb , we run chef install to fetch

dependencies and generate a Policyfile.lock.json . We haven’t specified

any dependencies yet, so we don’t need to fetch anything, but we will

need the lockfile to be generated before we can proceed to the next

step.

chef install

Let’s take a peek at the Policyfile.lock.json we just created. We’ll

go over each part individually:

Revision ID

"revision_id": "5f750bf464100b487cd7c276c5d532341b79fbeb5e8accd29538ae972896992b",

Each time we create or update the lock, chef will automatically

generate a revision_id based on the content. These values are used to

automatically version your policies, so that you can apply different

revsions of a policy to different set of servers. We’ll see this in

action a little later.

Name and Run List

"name": "aar", "run_list": [ "recipe[aar::default]" ],

The lock includes the name and run list we specified previously. The run

list is normalized to the least ambiguous form.

Cookbook Locks

"cookbook_locks": { "aar": { "version": "0.1.0", "identifier": "cff2d37260c04b21053ad30b68aa20e674e52e6c", "dotted_decimal_identifier": "58532310150070347.9294424438433962.36174175743596", "source": "cookbooks/aar", "cache_key": null, "scm_info": { "scm": "git", "remote": null, "revision": "3455fb415d56f9a7cabbb76f2063942a6547b2eb", "working_tree_clean": true, "published": false, "synchronized_remote_branches": [ ] }, "source_options": { "path": "cookbooks/aar" } } },

For each cookbook we use, there is a corresponding entry in the

cookbook_locks section. The exact data collected about each cookbook

is dependent on the cookbook’s source. In this case, we have a cookbook

sourced from the local disk which happens to be in a git repo. In the

event we need to debug this cookbook later, ChefDK has collected

information about the cookbook’s git revision. If we’d setup a remote,

git would tell us the cookbook’s git URL and whether we’d pushed this

commit to a branch on the remote.

Attributes

"default_attributes": { }, "override_attributes": { },

Policyfiles have attributes that replace role attributes. We’ll see

these a little later.

The Rest

"solution_dependencies": {

You can ignore the solution_dependencies section. It’s used to keep

track of dependencies in your cookbooks so ChefDK can check whether

changes to your cookbooks are compatible with their dependencies without

having to download the full cookbook list from supermarket every time.

Commit the lockfile

We’ll want to compare it to an updated version

later, to see what changed.

git add Policyfile.lock.json git commit -a -m 'updated Policyfile and created lock'

Edit .kitchen.yml

ChefDK ships with a policyfile_zero provisioner for Test Kitchen that

allows us to test our policies in local (or cloud) VMs. Note that

currently Chef Zero doesn’t fully support “native” Policyfile APIs, so

instead it runs in compatibility mode. This isn’t a problem on an

isolated server like Chef’s local mode, but it’s something you might

notice when debugging.

--- driver: name: vagrant network: - ["forwarded_port", {guest: 80, host: 8080}] provisioner: name: policyfile_zero require_chef_omnibus: 12.3.0 platforms: - name: ubuntu-14.04 suites: - name: default attributes:

Run TK with Empty Cookbook

To verify our test rig works, we’ll run kitchen with our empty

cookbook:

kitchen converge

If you get an error like Message: Could not load the 'policyfile_zero' provisioner from the load path

then you didn’t run the chef shell-init step above. Run that and try

again.

If that worked without a problem, you can throw that VM away:

kitchen destroy

Develop your Cookbook

This is where we’d normally run a TDD testing loop, but for the purpose

of this walkthrough, we’ll just import the aar cookbook fully formed:

cd cookbooks curl -LO https://github.com/danielsdeleo/aar/releases/download/draft-1/aar-cookbook.tgz tar zxvf aar-cookbook.tgz rm aar-cookbook.tgz cd ..

If you inspect cookbooks/aar/metadata.rb , you’ll notice that our

cookbook now has some dependencies, but we haven’t yet downloaded them.

We’ll do that next.

Chef Update and Commit

Now that we’ve added some dependencies to our cookbook, we need to run

chef update to fetch them.

chef update

This will recompute our dependencies and cache all the cookbooks we

need for our Policy.

To see what’s changed since our last commit, we could just run git

diff , but chef diff gives us itemized output, listing added, removed,

and changed cookbooks. Let’s give it a go:

chef diff --head

Now that we’re satisfied with our changes, we’ll commit again.

git add . git commit -m 'Update aar cookbook and deps'

Run Kitchen Again

We can run test kitchen again to see the result of our changes:

kitchen converge

Visit the Site

With the port forwarding, we can visit

http://localhost:8080 and see the site. For

more info on using the application, see the awesome appliance repair README on github.

Deploy

We can use the chef provision feature to create a “staging” and then a

“production” node. We’ll use these to see how we can deploy different

revisions of a policy to different machines. Since chef provision is

new and somewhat experimental, it’s not yet integrated with chef

generate , however, we can generate a cookbook like this (make sure it’s

named “provision”, that name is special):

chef generate cookbook provision

Then overwrite the generated provision/recipes/default.rb with the

following. NOTE: It’s vital that you set the convergence_options

as shown here. Nodes currently don’t have any attributes to set

Policyfile options; instead you must set policy_group and

policy_name in the config file. The provisioning convergence options

will take care of that for you automatically.

If you’d like to extend this example to use something other than

Vagrant, you can learn more about Chef Provisioning on docs.chef.io.

context = ChefDK::ProvisioningData.context # Set the port dynamically via the command line: target_port = context.opts.port with_driver 'vagrant:~/.vagrant.d/boxes' do options = { vagrant_options: { 'vm.box' => 'opscode-ubuntu-14.04', 'vm.network' => ":forwarded_port, guest: 80, host: #{target_port}" }, convergence_options: context.convergence_options } machine context.node_name do machine_options(options) # This forces a chef run every time, which is sensible for `chef provision` # use cases. converge(true) action(context.action) end end

Notice that we are making the forwarded port configurable via the

command line. This lets us run multiple VMs on the same host without the

forwarded ports colliding with each other.

We can sync the policy to the server and create our “staging” node with

a single command:

chef provision staging --sync -n aar-staging-01 -o port=8000

That will sync our local policy lock to a policy group called ‘staging’

(creating that policy group in the process), then run an embedded Chef

Client which creates a VM (via Chef Provisioning), configures it and

converges it.

We can see the site running by visiting http://localhost:8000

(notice that’s port 8000 this time).

Since it works in staging, we’ll create a “production” node with the

same policy:

chef provision production --sync -n aar-production-01 -o port=8888

Update the Attributes via Policyfile

Policyfiles allow us to set attributes. Since Policyfiles don’t support

roles, these attributes replace role attributes in the precedence

hierarchy. In our Policyfile.rb, we set attributes using the same syntax

we use in cookbooks. In this example, we’ll change the version number

that appears on the home page. Add the following line to your

Policyfile.rb

default['aar']['version'] = "19.7.4"

To apply the changes to the Policyfile.lock.json, use chef update :

chef update --attributes

We can see the effect of our changes with chef diff :

chef diff --head

And we can see that our local policy differs from what we’ve deployed to

our staging group:

chef diff staging

Now that we’re satisfied with our changes, commit to git again:

git commit -a -m 'update aar version'

Deploy it to Staging:

In a normal TDD workflow, we’d run kitchen again to see our changes, but

this time we’ll just deploy it to staging by running chef provision

again:

chef provision staging --sync -n aar-staging-01 -o port=8000

If we visit the site at http://localhost:8000

we see “Awesome Appliance v.19.7.4” right under the login dialog.

Let’s suppose we’re not ready to apply the change in production, but we

want to run chef-client on our production machine. We can do this by

using the --policy-name option instead of the --sync option:

chef provision production --policy-name aar -n aar-production-01 -o port=8888

Note that since we did not update the policy, nothing is updated. If we

visit the web page at http://localhost:8888

we’ll see that nothing has changed. Though we only changed the

attributes, the same is true if we updated the cookbooks, since they’re

locked down by content in our policy.

Oh No! A Bug in Our Cookbook

To demonstrate how cookbook code is automatically versioned and

sandboxed by policyfiles, let’s introduce a “bug” into our cookbook. Add

this line to the top of cookbooks/aar/recipes/default.rb :

raise "OH NO THIS IS A BUG"

Since this cookbook is local, chef will automatically pull in updates

when we upload (no need to run chef update ). Lets upload to staging

and run chef-client again. This time, we’ll use chef push to upload

our changes without invoking provisioning (which is probably what you’d

want to do in your normal workflow).

chef push staging

We can invoke provisioning without syncing the policy like so:

chef provision staging -p aar -n aar-staging-01 -o port=8000

This will cause an error like this in the chef run on the VM (which will

cause another error in chef provision on your workstation):

================================================================================ Recipe Compile Error in /var/chef/cache/cookbooks/aar/recipes/default.rb ================================================================================ RuntimeError ------------ OH NO THIS IS A BUG Cookbook Trace: --------------- /var/chef/cache/cookbooks/aar/recipes/default.rb:1:in `from_file'

But if we run Chef on our production node, everything is roses and

sunshine:

chef provision production -p aar -n aar-production-01 -o port=8888 # => Chef Client finished, 0/41 resources updated in 10.634577518 seconds

We can also confirm that the policies applied to each group are

different with the show-policy subcommand:

chef show-policy aar

The output should be similar to:

aar === * production: 8312cd89c9 * staging: eb0fedf311

Cleaning Up

To shut down the Vagrant VMs, you can use the -d option to chef provision

to set the default action to destroy:

chef provision production --policy-name aar -n aar-production-01 -o port=8888 -d chef provision staging -p aar -n aar-staging-01 -o port=8000 -d

Fin

Policyfiles give us a consistent and repeatable description of how we

want Chef to configure our machines, with minimal hassle. Because

versioning is built-in and automatic at both the cookbook and policy

level, we can make changes to our infrastructure code safely and

explicitly. While we still have work to do to fill out the feature set

around Policyfiles, enough of it exists for you to get started today.

Currently, the most complete documentation is in the

Policyfile README in the ChefDK repo,

but we’ll be moving documentation to docs.chef.io

as we complete work on Policyfiles.

If you give it a try and find that any missing feature is a deal breaker

for you, let us know and we’ll do our best to make you successful.

Happy Cheffing!