This blog is moo-ving Just a quick note, I’ve been busy with a new job but not too busy to move my blog across to http://nemski.github.io. All the content here has been replicated there and I this will no longer be updated.

On a side note I really like Jekyll

Performance bug in RedHat’s virt-who A few months ago I was troubleshooting RedHat’s Subscription Asset Manager (SAM). This is used as an internal licensing server and RPM repository, as opposed to the RedHat Network. The issue I was chasing was when we deployed a new Virtual Machine, it was taking the SAM server up to 30 minutes to identify which ESXi host the VM was running on and hence correctly register it and allow it to download packages. Our deployment process for a new server was already 30+ minutes, so adding another 30 minutes to wait for SAM to catch up was a little over the top. The process which discovers VMs is called virt-who, written in Python. It was scheduled to run every 15 minutes but was taking 15 minutes to complete. So having barely touched Python I dived in deep with plenty of time and print statements (i’m sure there’s a better library to do this for me). This eventually led me to a section of code, the core of the entire program.

# Get list of host uuids, names and virtual machines At first glance it appeared to be well optimised: object_contents = self.RetrieveProperties(‘HostSystem’, ['vm’, 'hardware’], hostObjs) This line got a list of Hosts, it’s VM and Hardware properties and then proceeded to iterate through them. for host in object_contents:

vmObjs = [] # List of objs for 'VirtualMachine’ query

if not hasattr(host, 'propSet’):

continue

for propSet in host.propSet:

if propSet.name == “hardware”:

self.hosts[host.obj.value].uuid = propSet.val.systemInfo.uuid

elif propSet.name == “vm”:





Then it iterates through each property and if it’s a VM we obviously want to learn more about it. Now we get to the crust of the issue for vm in propSet.val.ManagedObjectReference:

vmObjs.append(vm)

v = VM()

self.vms[vm.value] = v

self.hosts[host.obj.value].vms.append(v)

# Get list of virtual machine uuids

vm_object_contents = self.RetrieveProperties('VirtualMachine’, ['config’], vmObjs)

for obj in vm_object_contents:

# Get each VMs uuid and put into a dict

The program keeps a list of VMs it finds in the vmObjs array, then gets the config property for each VM with a single API call to vSphere. I was quite impressed, that’s a very efficient way of doing it that I wasn’t aware of. But… I’ll give you one last chance to read that code again, remembering to take into account Python’s whitespace sensitive for loops. Got it? Ok so Python doesn’t start and close blocks in the way traditional languages do, usually with curly brackets ({}). Instead they start with a colon (:) and any code indented by some white space (i’m not sure on the amount, usually 4 or a tab) is within that block. There is no real end to the block, code after the block is just not indented. So why then are we getting a list of properties for VMs in vmObjs more than once? Why is that for loop embedded within the loop that gets a list of VMs? The reason is, it’s not meant to be. Un-indenting that second for loop resulted, in my relatively mediocre vSphere environment of 326 Virtual Machines, in a 90% performance increase. 90 seconds as opposed to 15 minutes. Someone else I spoke with who had over 1000 VMs reported a 98% increase in performance. I submitted a pull request to RedHat via Github, but I’m not sure if the fix has been released yet. I think it’s being tracked via this Bugzilla request

Dashboards with dashing I was reading an article by Dataloop.io about using dashboards to change behaviour. It was a good coincidence because I’d just stood up a Continuous Integration tool to let the team know when our puppet code fails tests, but had this niggling thought in the back of my mind “who, other than me, will look at this?” The problem is we have so many dispirit systems that it’s easy to forget about one for a period of time, or the person who usually looks after it is on leave. So how do we compile all that information in one place? Mentioned in that article (which I recommend reading btw) was something called Dashing, a Ruby framework for building dashboards. As someone who’s only done web development under duress, I dived into it and was surprised at what I managed to achieve in two days. I was able to churn out integration with ServiceNow, Opsview and EMC Networker to present the three most valuable pieces of information to our team on a day to day basis. And, most importantly, I barely had to get my hands dirty with the web dev side! Most of my time was spent creating dashing jobs to collect the data.

Continuous Integration of Puppet code We have a project with almost 10,000 lines of puppet code, which until recently had 0 tests. I’ve been working for the past few months to change this by implementing unit tests using rspec-puppet and acceptance tests using beaker (which heavily leverages server-spec under the hood).



What kicked off my desire to start this was a Puppet Camp I attended in Melbourne last year where a couple of the speakers talked about using Bamboo and Cucumber to execute them. Being the curious admin I am, I decided to trial Atlassian Bamboo along side JetBrains TeamCity to see which I liked better (I eliminited Jenkins without really looking at it due to it’s perceived lack of useability). Bamboo, TeamCity and Jenkins are a range of tools used to test, compile/build code and deploy it into test/QA/prod. Really I only need it for testing, there is no compilation or build and there isn’t enough effort involved to justify automating deployment of code yet.

Bamboo vs. TeamCity



Both tools offer what I needed, integration with Rake to run our tests (Bamboo required a plugin to be installed) and the ability to view test results in a well formatted way (Bamboo required some massaging see my previous post). But TeamCity provided a much easier interface to configure my tests and the ability to use build configuration templates with parameters set in individual build configs that modified the default behaviour. This was very useful, as we have two seperate VCS roots; a local repository of code and a vendor import. The location of the vendor import depends on which code stream is being used (branch vs. trunk). Both are great tools though and I recommend trialing both if you’re looking for a Continuous Integration (CI) tool.



Enter RubyMine



I’d also been trialling RubyMine, another JetBrains product, as an IDE for Ruby development. I’d been working on a number of Gems and found it’s project templates as well as integration with rubygems, rake and bundler an eye-opener into the value of IDEs (having been a text-editor coder up until now). So much so that I forked out AU$133 of my own cold hard cash for it. What I discovered today was it’s native support for Puppet and integration with TeamCity. I can now modify my Puppet code in RubyMine (which brings with it support for automatically renaming variables and auto indenting) and then kick off TeamCity builds before I check my code in! This is a game changer, I will never develop puppet code the same way again.

Entering the BIOS devopsreactions: by @just_hank_moody, andreibkn and necessaryaegis



Multiple format outputs for Rspec I’ve been playing with Atlassian Bamboo to automatically execute and report on our rspec tests. We have unit and acceptance tests, which are executed via rake tasks. I found a fantastic gem rspec_junit_formatter which allows you configure Bamboo (and other CI tools) to parse your rspec test results. My problem was I wanted to keep the original formatting (documentation) for when the command is run interactively, and I needed to separate the unit test results from the acceptance test results. The project readme suggests adding options to .rspec file to append command line options, but that means writing outputs from unit and acceptance tests to the same file, which would make it difficult for bamboo to pick both results up. So here’s how you do it, from within your spec helper! RSpec.configure do |c|

…

c.add_formatter “RspecJunitFormatter”, “spec.xml”

c.add_formatter :documentation

end

Don't panic: learning to be on call Some articles to encourage the developer who is new to being on call (becoming more common):http://www.unwiredcouch.com/2015/01/06/learning-on-call.html TL:DR; moral of the tale: don’t panic. An Etsy…

Introducing a Ruby Gem for ServiceNow integration rs_service_now is a Ruby Gem that acts as an interface to ServiceNow. At the moment it only supports two operations, request and export (which do the same thing, but via different mechanisms), but I plan on expanding this to support more methods in the future. Export is an experimental approach at the moment, where as the request method is tried and true. This was my first foray into writing a Gem, being motivated by a previous experience re-writing a backup script that was procedural with a more Object Oriented approach. I also used an IDE called RubyMine, which I recommend you check out if you do a lot of Ruby coding. It highlights code in real time to let you know of variables that are not used, not declared or code that is not syntactically correct and allows you to build and push Gems from within the IDE. Anyway, rs_service_now will act as a wrapper for much of the work I do with custom code exporting data from ServiceNow, by allowing you to get the information with a single method call. The ServiceNow SOAP interface is infamous for restricting such exports to a limited number of rows and makes it difficult to get a large dump. The process to then get the export is difficult and the resulting data type differs if you are returned one vs. more than one record (hash vs. array of hashes).

Driving automation for better and quicker software releases As the year has come to a close I’ve been looking back on what I’ve achieved and where I was 12 months ago. Last year we were in the same place we had been for several years, in a constant patch and release cycle for code provided by our developers. We would continually have to fix software ourselves, sometimes bad by design sometimes just day to day bugs that we find. We also had to add new functionality ourselves as new requirements came through from the business. Often this doesn’t get merged upstream. Puppet came along early in the year and this enabled us to centralise and standardise the configuration for much of our infrastructure, so we could more quickly and easily roll out changes. This was a huge improvement for us, but it didn’t really decrease the number and severity of software and configuration bugs. The fact was our Quality Assurance processes were duplicated across the dev team and our team. The Dev team would do their own manual set of testing and we would run our own. As the year went on we started to included much more rigorous testing, as we had to test for bugs we had previously picked up but that had not been fixed upstream as well as functionality we’d implemented. But this was all still manual, time consuming and error prone. So how did we streamline this approach and improve quality in the end product at the same time? Automation was key. We had automated, somewhat, the release mechanism, so I looked at going backwards and automating the deployment mechanism. We already had some degree of automation with deploying new Virtual Machines, but it was clunky and had far out grown the original scope. This was achieved by implementing vCenter Orchestrator, as I’ve discussed in my previously blogs. So next on my list was automating tests. I came away from a Puppet Camp in Melbourne in November charged and ready to tackle this problem, as I’d heard two case studies where clients were using cucumber to automate testing. For testing puppet code the default testing framework is rspec, but cucumber seems to have gained quite a following in recent years for its simplistic language of test steps (implemented in a language called Gherkin that uses Given, When and Then keywords to develop test cases). I had a lot of trouble initially wrapping my head around how I would complete all the testing, as we needed to do a mix of white-box and black-box testing. The puppet code we had needed to be white-box tested to ensure the internal logic was sound. But we also needed to test the proprietary products we use, for which we cannot inspect the internal state. This requires a different mechanism of testing called black-box testing, where we would need to simply throw information into a lab environment and inspect the result to ensure it matched our requirements. Eventually, after a discussion with a friend who is a software developer who told me to just jump right in, I started to build my tests for puppet first. So I implemented rspec testing for our puppet modules and after a few days of bashing my head against a brick wall I was able to confidently eliminate a small number of our manual tests. What frustrated me about rspec though was it’s insistence on unit tests, testing just the functionality of a single module and nothing else. This seemed pointless to me as a System Administrator as I wasn’t concerned with if a single module had an issue, but if the end to end system functioned. I hacked away and implemented some integration testing that compiled a puppet catalog of almost an entire server and tested all the pieces existed, and all the variables and logic worked end to end. At this point I decided to give cucumber a try, to see what all the hype was about, although I was initially hesitant to implement my “testing code” and “test steps” in two different files. Cucumber-puppet is sporadically maintained but still gives the same functionality as rspec. Another few days later and I started to fall in love with the Gherkin syntax that allowed me to define my test code once and write almost human readable tests that eventually consisted of 127 individual tests. Not only that it insisted on a more top down approach, testing all my modules together rather than individual functionality. But the end test steps still are not readable by management, they contain various phrases like “should have resource File[server.xml]” as well as “file should contain content matching <host>db01<\/host>” which really isn’t meaningful. Then I stumbled upon beaker and the solution to how to do black-box testing became apparent. Beaker is a wrapper of various shell tools and hypervisor APIs that allows you to build/configure a VM, kick off a puppet run with code from your branch and run some rspec tests. It’s exactly what I needed, end-to-end testing (called acceptance testing). There’s still some work to do, such as integrating with vCenter Orchestrator which I’ve already invested heavily in to leverage automated provisioning (checkout my branch here), but I’m almost there. Also I need to fit it in with our existing release workflow, but our designated “testing engineer” should be happy. Once completed I believe I might well have reached Nirvāṇa. I know this isn’t really a new concept within the Software Development community, but too many businesses still fail at the basics. Especially the many companies that spent many many years just “integrating” products that various companies provided without doing real software development. In the Software Defined World, every company is a software company.