Puppet is often described as "model-based" configuration management, which enables some pretty amazing feats of systems administration: simulating the effects of a potential change across your infrastructure, detecting dependency loops, letting you audit and trace every event on a machine, and so forth. But as more people find out about Puppet and start using it to solve their problems, we're continually looking for ways to smooth out the learning curve and help new users get value from it as quickly as possible. One of the first stumbling blocks people run into as they start writing Puppet modules is the order in which the agent evaluates resources.

"Why isn't it doing what I want?"

In the early days of Puppet, before 2.6.0, the only guarantee the agent made about the order in which a catalog's resources would be evaluated was that any metaparameters like 'before' and 'require' would be honored. Even the same catalog evaluated twice in a row on the same system might evince different ordering. This principle led to conventional wisdom (and in truth was the result of a philosophical approach) that long-time Puppeteers hold to be self-evident:

STATE YOUR DEPENDENCIES.

In 2.6.0, the change described in Redmine #6911 became the standard. This behavior—though its underpinnings are deliberately opaque—is what most Puppet users have come to expect through observation:

Without explicit ordering/dependency information, you can't predict where in a puppet run a given resource will be evaluated, but ...

... it will be evaluated in a similar place relative to other resources in the same catalog, even if you add/remove/reorder things in your modules and manifests.

So, in order to ensure that things happen in the order you expect, you're obligated to (you guessed it).... STATE YOUR DEPENDENCIES.

This has generally worked pretty well, with the notable exception of people who are new to Puppet but generally experienced with scripting and programming, for whom the deterministic-but-opaque ordering is incredibly confusing.

"I wrote it in the order it needs to run, why isn't it doing what I want?" they ask.

Then the resident Puppet expert launches into an hour-long whiteboard discussion of directed acyclic graphs, and their audience of one—that well-intentioned, intelligent person—forms a distinct impression that Puppet is very hard to use.

Introducing MOAR

So, in the spirit of experimenting more rapidly and more radically in releases of open source Puppet, then delivering the best of the best in Puppet Enterprise, we implemented configurable resource ordering algorithms in Puppet 3.3.0 and began to solicit feedback on how "manifest order analysis of resources" (MOAR) would work in the real world. Andy Parker and Patrick Carlisle extracted the resource evaluation routines so they could be examined in isolation, and implemented the following setting to pick between them:

# How unrelated resources should be ordered when applying a catalog. # Allowed values are `title-hash`, `manifest`, and `random`. This # setting affects puppet agent and puppet apply, but not puppet master. # # * `title-hash` (the default) will order resources randomly, but will use # the same order across runs and across nodes. # * `manifest` will use the order in which the resources were declared in # their manifest files. # * `random` will order resources randomly and change their order with each # run. This can work like a fuzzer for shaking out undeclared dependencies. # # Regardless of this setting's value, Puppet will always obey explicit # dependencies set with the before/require/notify/subscribe metaparameters # and the `->`/`~>` chaining arrows; this setting only affects the relative # ordering of _unrelated_ resources. # The default value is 'title-hash'. # ordering = title-hash

Some amusing alternate names for the pluggable algorithms arose during a spirited conversation on Twitter, sparked off by Dean Wilson's excellent blog post exploring the behavior. The random order could also be called --ordering=0.24 and MOAR waggishly referred to as --ordering=chef-mode .

But the question persisted, internally at Puppet Labs as people explored the feature and externally as the community thought about the ramifications: Would the contravention of such a deeply-held tenet of Puppet's operation cause failures on production code? A breakdown of the social order? CATS AND DOGS, LIVING TOGETHER? We decided to find out.

Doing Science on MOAR

Led by our fearless Puppet Test Pilot research team, we formed the following hypotheses:

New users will find manifest ordering easier to understand and use as they're starting out with Puppet, thus avoiding a critical "I GIVE UP!" threshold and keeping them engaged long enough to learn about when dependencies are actually necessary. Experienced users have gotten used to creating explicit ordering in their code, but they (a) can switch to MOAR without causing problems for existing modules, and (b) are willing to switch in order to improve code clarity and help junior admins work with Puppet.

The first hypothesis was proven by both contrapositive evidence (a quick search for 'puppet ordering' reveals dozens of new users independently baffled by what they observe) and very simple user tests at PuppetConf 2012 with people brand-new to Puppet.

The second hypothesis is more interesting, and cuts to the heart of the criticisms surrounding MOAR. The Test Pilots agreed to try out manifest ordering on their existing codebases, on real (if not customer-facing) systems. We worked with five users, running sites ranging in size from the low-hundreds into the multiple-thousands of nodes, spanning industries from education to scientific research to cloud service provider, and the results were pretty fascinating.

First, nobody experienced problems running their existing code with manifest ordering. Even when the catalog consisted completely of in-house modules (rather than Forge modules, which tend to have a greater proportion of explicit dependencies) on brand-new machines, things ran without a hitch. In a way, this makes sense: Since the hash-title ordering algorithm was totally opaque, manifest ordering on existing code should behave exactly like a different, still opaque, algorithm to the nodes. But it was still very encouraging to witness live via Google Hangout screen-sharing.

Second, even in complicated codebases where the Test Pilots removed explicit before/require ordering metaparameters (but left notify/subscribe metaparameters to propagate events between dependent resources), the result was increased code clarity and compactness with no unexpected failures. This was the real win (and honestly, the real surprise). In many cases, the manifests were already written in the logical order top-to-bottom, changed only to comment out the before/require statements, and things just worked.

"In working on a small team and using manifests written by my predecessor, I often find myself re-ordering a manifest so that it is ordered logically," Jesse Martinich, senior systems administrator at Southern Oregon University, said. "In this way, I have also found instances of resources not being ordered properly. I think it would end up motivating people to write better code, by making them organize it on the page the way they intend it to happen."

Making MOAR Official

Based on the evidence, we're super excited to make manifest order evaluation the default for future releases of Puppet and Puppet Enterprise. A couple of caveats are probably in order, though:

If you try out MOAR at your site and run into evidence that contradicts these findings, please let us know on the puppet-users group! This is science, after all, so we welcome evidence contrary to theory, because it's the only way to make the theory more accurate. The pluggability of the ordering algorithm in Puppet can actually be a huge win for sites with sophisticated continuous delivery pipelines. Andy wrote the '0.24' -- err, 'random' -- fuzzer algorithm to help smoke out unexpressed dependencies in modules that ought to be explicit. Anything published on the Forge, for example, shouldn't make any assumptions about the ordering algorithm in use by its users and should probably run through a random-order test on its way out the door. If you really love title-hash ordering, even after MOAR becomes the default, a puppetlabs/inifile resource like the following will let you keep using it ad infinitum:

ini_setting { "title-hash ordering": ensure => present, path => '/etc/puppetlabs/puppet/puppet.conf', section => 'agent', setting => 'ordering', value => 'title-hash', }

I'd like to extend a huge 'thank you' to both the community members who participated in the vetting of this feature and the Puppet Test Pilots who took it for a spin on their actual infrastructure.