At the Tokyo Summit, we provided an operational comparison of the two major OpenStack distros — Mirantis OpenStack 7.0 and Red Hat Enterprise Linux OpenStack Platform 7 (OSP7). It was so well received that we decided to write a follow-on blog. We are expanding the original comparison of just the software and installation to include aspects of the overall ownership experience. There are eight areas that we compare here, including the two from the Tokyo summit session:



We compare these distros against each other and score each review on their available merits. The basis for this experiment is as objective as possible. Some people at Mirantis may not be pleased when they see how we rated ourselves, but so be it. Ditto for Red Hat.

We used only public documentation and available software. No scripts, professional services, or corporate support have been used. For the software comparison we are looking at the OpenStack distributions and not the host OS, hypervisor, or guest OS. Finally, Fuel is now an official Big Tent OpenStack project, but it was not the case when these distros were released. For this reason, we are not treating Fuel as an OpenStack project in this blog to ensure that Mirantis doesn’t get an unfair advantage.

Review #1: Lifecycle management (Day 1)

Level set – Infrastructure

For the purposes of this comparison, we are focused on splitting it into Day 1 and Day 2, where Day 1 involves the ease of installation and deployment, and Day 2 involves ongoing operations like scaling the environment up and down, patches, and upgrades. Using the available documentation, we designed equivalent physical environments and used the exact same hardware for each distribution deployment.

Installation Procedures

This review consists of the overall process for getting both distributions from bare metal and an ISO to an up-and-running OpenStack environment capable of launching instances and workloads.

Scoring Metrics:

Guided vs manual installation steps

Time required for installation and deployment

Frequency and types of tripping points/issues

Troubleshooting required

Mirantis

We start the review with Mirantis. As shown in the Installation and Deployment Steps diagram, from start to finish Mirantis OpenStack took 12 steps, of which more than 80% were guided procedures, either by a wizard or a GUI. The only point that caused any significant amount of troubleshooting was caused by user error, where I failed to “Apply” my network changes on each page of the opening installation GUI of the Fuel Master. Rather, I made all the changes and then clicked “Apply,” losing the changes I made on previous pages. But, as a testament to Fuel’s network verification check, I was able to quickly diagnose and fix the issue.

Red Hat

As for getting Red Hat Enterprise Linux OpenStack Platform 7 off the ground, that was significantly more challenging. This is both the first version of their TripleO deployment model and the first version of their Director installer; both have several tripping points. From start to finish, the preponderance of steps are pure CLI commands with a heavy dependence on the documentation. It is also the first version release of the documentation for TripleO and Director, and it suffers from several syntax typographical errors, and—in some areas—missing steps altogether. At the time of this article’s writing, there were 67 outstanding errata filed against the installation documentation. Our experience with these errors resulted in many showstoppers, such as failed installations and deployments with cryptic logs. It took hours of research to turn up the correct context and syntax for many CLI commands. The Red Hat OpenStack distribution lacks any active pre-deployment checks, so several issues were discovered only after attempting deployment.

Adding the comparison of errors found in Red Hat’s documentation, at 67 known errata, to Mirantis’ 6 outstanding documentation errata and the number of issues that came up during the install and deployment, Mirantis clearly has a better installation method.

Review #1 Scores:

Mirantis OpenStack 7.0 with Fuel

The user experience to install from scratch and get to a working environment was easy considering the vast majority of the steps were wizard driven, requiring only that I enter prompted information; the rest was basically automatic. There were very few issues along the path of installation.

Total Steps: 12 mostly guided steps (includes some defaults, sub-steps, and 2 optional health verifications)

Total Time: 1 hour 20 minutes.

Red Hat OpenStack Platform 7 with Director

The experience here was significantly difficult. The majority of steps were numerous CLI commands to get things off the ground, and with them came several syntactical errors, both in the documentation and in user-error. The higher number of user-error issues are to be expected when dealing with a large set of manual CLI, context, and syntax related commands. However, the many errors in the documentation were unexpected. Coupling the two together resulted in an extended and frustrating experience to get a minimally viable environment. In addition, the extra steps required after deploying OpenStack to get to the point of being able to launch instances was another letdown.

Total Steps: 48 mostly manual CLI (includes some default values, sub-steps, and one optional health verification)

Total Time: 3 hours 5 minutes (not including troubleshooting time)

Review #2: Lifecycle management (Day 2)

Continuing operations for OpenStack is a different challenge to meet than the initial configuration and deployment, so we have broken it out separately. In this category, we look at the abilities for each distribution to scale up and down, patch an existing environment, and eventually upgrade it to the next major revision.

Scoring Metrics:

Ease and automation of scaling the physical infrastructure up/down

Patching methods and documentation

Method and documentation for updating

Managing multiple environments

Scaling the physical infrastructure

Mirantis – Using Fuel, existing environments are easily scaled both up and down, with automated discovery of PXE bootstrapped nodes and GUI or CLI based scaling of nodes.

Red Hat – Red Hat’s scaling, while functional, is lackluster. While it performs as expected, there aren’t any shining features that benefit administrators in making their jobs easier or less involved. One aspect that is appreciated is the use of Ironic for bare metal node registration. While Ironic is still young and without numerous features, and requires some manual data collection and entry (for example, Node MAC addresses), it has the potential to grow into an essential component. But with the added steps of data collection and the added complexity of the IPMI-to-Management networking requirement countering the benefit of Ironic registration, it’s a wash for scoring. Additionally, Red Hat documentation is limited to adding and removing nodes through manual CLI operations rather than through their Director GUI.

Patching

Patching for bugs and security are a commonplace event in operating systems and software. Both distributions have documented processes to apply patches, and both use similar methods for distributing patches, including customer notifications and normal system level commands such as yum and apt-get.

Upgrades

Upgrading OpenStack is a long-standing pain point in operations. The community and distribution vendors continue to make improvements to the process, but it still remains a manual, though practiced, process. The procedures exist in both of these distributions to allow for minimal downtime, albeit a relatively long and time-consuming process, or if more downtime is available within a change window, a shorter bulk style process can be undertaken.

Mirantis has well documented manual processes for each procedure type, and focused upgrade scripts in their Octane tooling for Fuel to help automate where possible.

Red Hat does have documentation on the processes required for upgrading their environment. Currently, there are hundreds of individual CLI commands required to achieve an upgrade; they have not yet provided any automation in scripting.

Management of multiple environments

The deployment engine built into Mirantis OpenStack, Fuel, can maintain management of multiple OpenStack environments, each at a different version. This flexibility enables organizations to concurrently run test and development environments that have a more dynamic versioning structure, alongside production environments with more rigid versioning cadences. This has the further benefit of allowing for gradual, granular staging of upgrades.

Red Hat’s OpenStack distribution only supports a single overcloud environment per undercloud and Director seed node. (This is a current limitation of the TripleO deployment model.)

Review #2 Scores

Mirantis

Scaling – Mirantis has the edge here, especially in the scaling department. Fuel management of scaling in multiple environments and a completely automated discovery mechanism is clearly a strong benefit for administrators.

Patching – The patching aspect is on par with Red Hat.

Upgrade – Full upgrades continue to be challenging, but Mirantis does have a slight advantage with the inclusion of their Octane tooling for Fuel.

Multiple management – Mirantis gets points here allowing for numerous environments at different versions with centralized management.

Red Hat

Scaling – Using Director and Ironic for node management and registration/discovery is more involved. Ironic registration of nodes requires some manual data collection that, especially at scale, can be troublesome.

Patching – The patching is on par with Mirantis in process and method.

Upgrade – Upgrades are documented but slightly more involved than Mirantis due to the lack of tooling similar to Fuel Octane. The requirement of a significantly greater number of CLI commands and lack of scripting offered are a disadvantage.

Multiple management – Red Hat loses points here with the inability to manage multiple environments with a single installation. The difficult installation process has to be repeated for any additional environments.

Review #3: Downstream Testing & Stability Improvements

In this review we compare the stability enhancements of the two distributions. We could try to use metrics such as failures at real customers or number of critical bugs filed by customers, but generally this data is hard to come by or proprietary (or both). So instead, we will score reliability at scale based on these four factors:

Scoring Metrics:

Validation testing

Packaging

Reference architectures

Validation Testing

Both Mirantis & Red Hat perform testing over and above what the community does to ensure their respective distros are resilient at scale.

The respective statistics are:

Test Suites Bugs fixed in OpenStack Kilo Additional bugs fixed in Distro Mirantis OpenStack 7.0 8 test suites over & above community testing: Functional System Scale (Rally) Package validation Destructive Negative Longevity Compatibility 922 328 Red Hat Enterprise Linux OSP7 Unknown number of test suites. The only one we were able to confirm is Scale testing. 794 Unknown

While we believe this to be a relevant aspect of overall product distribution, given that we do not have visibility into Red Hat’s validation testing, this comparison is clearly inconclusive, so we are omitting this section from the scoring until we can gather more concrete data. That said, we chose not to completely remove this section from the blog because we think it’s an important aspect that should be considered by potential customers. When coupled with the installation and usage experience in reviews #1 and #2, questions are raised regarding the validation and stability (as they should be, with all vendors). (And if Red Hat wants to provide the information, we’re happy to include this metric in the scoring.)

Packaging

There are three levels at which we look at packaging:

The differences look like this — first, which projects are included. Next, what middleware is packaged. Finally, which patches are included from trunk.

Both distributions include core projects, i.e. Nova, Swift, Neutron, Keystone, Cinder, and Glance. They also include Horizon, Ceilometer, Heat, and Sahara. Where the two distros deviate is that Mirantis OpenStack 7.0 includes Fuel for deployment and ongoing management and as well as Murano application catalog; RHEL OSP7 includes Director for deployment and ongoing management and Ironic for bare-metal provisioning. OSP7 also has a few projects in technology preview e.g., Trove, Designate, and Manila. However, because these are in the state of technology preview and not supported for production environments, we can’t weight them heavily.

As for middleware, both Mirantis and RH use MariaDB, Galera MySQL, MongoDB, Pacemaker and Corosync in an HA configuration.

Finally, the specific patches utilized or rejected from trunk is proprietary information on both sides, so no scoring differences are made on this point, but as mentioned before, it can still be an important aspect of the customer’s decision and brought up with each vendor.

With basically equivalent pros and cons for both distros, the packaging comparison results in a draw.

Reference Architectures

Reference architectures are a critical piece of a distro because these documents state what configurations and combinations are recommended and assure that they have been tested together. A distro can have its own reference architecture document or one created in conjunction with a third-party technology.

Both distributions offer documentation explaining the lower-level architectures involving the communication paths of services and the best practices for implementing them in suggested design scenarios.

This section of comparison results in a draw.

Review #3 Score

Mirantis

Mirantis scores reasonably well due to the completeness around a variety of factors related to resilience at scale. More OpenStack projects could be included.

Red Hat

Red Hat also scores reasonably well here. Given that the validation testing section is held from the score, in terms of other factors, neither distro beats the other one on packaging.

Review #4: Developer productivity



OpenStack, or any private cloud, is not an end in itself. Most companies build it to enable workloads like developer productivity tools, big data, NFV, etc. We see unlocking developer productivity as the most important use case, and therefore we made it a comparison review.

Scoring Metrics:

PaaS support

CMP support

Container framework support

App Catalog support

In general, Mirantis’ philosophy is

the developer space is very fast-moving, and partnering with best-in-class vendors gives users maximum choice and flexibility; and less opinionated is better.

Red Hat’s philosophy seems to be in contrast that

OpenShift based on Kubernetes is the right choice for PaaS; and a prescriptive approach is better.

Specifically on the application catalog piece, Mirantis is putting its stock behind an OpenStack project called Murano (which also powers the OpenStack community app catalog). The Red Hat approach is to use their CloudForms project as an app catalog. The plus with Red Hat’s approach is that CloudForms supports hybrid clouds out of the box — Murano requires some extra work for that — but the drawback is that, though it is open source, it is not a meritocracy-based community project. Here is a comparison along the various scoring metrics:

Mirantis Red Hat PaaS Support Approach: partnerships

Basic Cloud Foundry

Pivotal Cloud Foundry Approach: in-house

OpenShift CMP Support Approach: partnerships

Scalr

Apcera

Cloudify

Avni Approach: in-house

CloudForms Container framework Support Approach: partnerships

Basic Kubernetes

Tectonic Kubernetes

CoreOS Approach: in-house

OpenShift Kubernetes

Atomic App Catalog Approach: community

Murano Approach: in-house

CloudForms



We are going to declare this review as a draw. Even though the Mirantis view is that a pure-play, user-makes-the-choice, and fluid approach in a fast-moving space make more sense, a user may disagree and prefer a fully integrated stack from one vendor. So we are going to set our views aside. We are giving both distros a 7 since there is much work to be done on the developer front.

Review #4 Score

Review #5: Open Source & Community

Users get creative when trying to figure out which distro to buy. One way they figure this out is by studying the contributions of the vendor and to see how well they play in the community. The arguments for open source are well known; the ability to ‘look under the hood’, so to speak, and freely change and customize the code to fit your requirements. But not all players in the open source community are created equal. We look at the following aspects as to how Mirantis and Red Hat play in the OpenStack community:

Scoring Metrics:

Open-source approach

Contributions by commits

Contributions by bug fixes

Thought leadership

Open-Source Approach

Mirantis uses open source for all their projects. All Mirantis projects are developed 100% through a meritocracy-based community approach. Mirantis also develops all their code, bug fixes, etc. upstream-first.

Red Hat also uses open source for all their projects. However, not all projects are developed in a community. For example, aspects of Director are not being developed in the community.

As a testament to Mirantis’ efforts to be upstream, Mirantis pioneered Fuel as a project upstream that has become part of the OpenStack Big Tent and is included in the distribution. In addition, Mirantis has but one version of the distribution (Mirantis OpenStack) that can be downloaded and used for free by the community. It is the same distribution for commercial purposes with the addition of support services. All fixes to Mirantis OpenStack distributions are upstream; there’s no conflict in versioning. Red Hat, on the other hand, has RDO (community edition) and RHEL-OSP 7 (commercial version), each with their respective repositories, packages, and patches.

Contributions by Commits (Kilo)

We are using the Kilo release of OpenStack as a comparison point because both distros are based on that release. Both Red Hat and Mirantis are top 3 contributors to OpenStack, where both have give and take in different projects. For example, Red Hat has an advantage in the number of overall commits performed with over 5,200 to Mirantis’ over 3,400.

Contributions by Bugs Resolved (Kilo)

While Mirantis has an advantage in overall bug fixes with 922 to Red Hat’s 794.

Thought Leadership

The Mirantis-initiated Murano orchestrator/app catalog and the Fuel deployer are both official OpenStack projects. Outside of core projects, but still included in the distro, Mirantis leads on Murano, Sahara, Heat, Fuel, and Rally.

Outside of core projects, but still included in the distro, Red Hat leads on Ironic and have several projects included in their technology preview, like Designate and Trove.

Review #5 Score

Mirantis

Mirantis scores reasonably well by leading on the open-source approach, contributions by bugs resolved, and earns points in thought leadership.

Red Hat

WIth Red Hat’s scores higher in overall contributions by commits and their long standing, strong community presence, they gain the advantage in this review.

Review #6: Extensibility

This review is about the ability for users to customize and extend their OpenStack solution. This involves the ability to choose from lists of third-party plugins for their specific use case and the capability of developing their own unique plugin for deploying and managing their environments.

Scoring Metrics:

Catalog of plugins

Availability of a documented framework for DIY plugins

Mirantis has a strong advantage in this review. There is a constantly growing ecosystem with a wide array of certified third-party plugins to provide the desired functionality for customers’ custom deployments. In addition, by including Fuel, Mirantis OpenStack gains the ability to automate the deployment of these plugins, with the added benefit of being able to granularly deploy plugin-specific roles to dedicated nodes, all through the GUI. In addition, Mirantis has a well-documented open-source framework for creating your own plugins for a truly pure-play, open source, customized experience to meet each use cases’ requirements.

Red Hat also has a list of plugins available for their distribution; they do get good points for that. However, the deployment of the plugins is completely manual; CLI commands are required on each node that the plugin affects, which can be an administrative nightmare in large-scale deployments. Also, we could not find a publicly available open-source framework for creating your own plugins for their distribution. (Again, we’re happy to revisit this review if such a framework is indeed available.)

Review #6 Score

Review #7: Interoperability

OpenStack requires numerous integrations “above” the stack in terms of PaaS, containers, CMP, and other middleware, and “below” the stack in terms of hypervisors, storage, networking/SDN, and so on. This is one section that we couldn’t get hard evidence supporting the numbers of partners for Red Hat’s distribution across the technology silos, so we are focusing on the transparency of the validation process.

Review #7 Score

Mirantis has a wide array of published partners across the technology silos offering pure-play options for end users in every aspect of their solution. Mirantis’ ‘Unlocked’ program provides transparent procedures, templates, scripts and tools for validating hardware; OpenStack drivers and infrastructure-level solutions; and application-level and/or OpenStack API-integrated solutions. Mirantis offers additional validation guidance for Fuel plugins (which automates deployment of drivers and complex products with Mirantis OpenStack) and Murano app catalog packages (which orchestrate application deployments on a running OpenStack cluster). Mirantis insists that partners revalidate their products with each major release (six-month cadence). They also incorporate many community-approved open-source technology options (e.g., Ceph) in their regular distributions, in robust reference configurations, configured and deployed automatically with Fuel.

Because Mirantis OpenStack remains extremely close to OpenStack trunk, moreover, it is likely that non-validated OpenStack drivers compatible with a given OpenStack release (e.g., Kilo) will work with the corresponding Mirantis OpenStack release (e.g., MOS 7.0). Mirantis led the effort to create DriverLog, a community-maintained index of available OpenStack drivers, compatible versions, automated test results, and driver-builder contact information.

Red Hat develops their own goods in the PaaS, CMP, and Containers. We didn’t have direct access to their partner counts in many of the categories, and as a disclaimer, they are grandfathering in the hardware servers from their RHEL operating system partner program. While this would make the number of partners looks high for Red Hat, they are not actively revalidating the partner solutions specifically with their distro. These may or may not be functional with each specific OSP release.

We score the two distros highly as both do offer additional functionality through other offerings. However, since our data is incomplete with regards to Red Hat specifics, and considering their presence across the technology fields, we score them equally here. Again, if Red Hat wants to provide more information we’ll be happy to review scoring for this review.

Review #8: Completeness of offering

In comparing the completeness of the distribution offering we look beyond the software itself at aspects surrounding the use of the product: available training, professional services, and support.

Scoring metrics:

Training

Professional services and support offerings

Mirantis’ roots are in professional services and training. Training, certification, and professional services are available worldwide. Mirantis offers training and certification in both pure-play (vendor agnostic) open-source OpenStack and the Mirantis distribution. In more than 4 years, Mirantis has trained over 5000 people with a 95% recommendation rate–significantly higher than the industry averages.

Mirantis offers a broad catalog of nearly 20 productized services specific to OpenStack, from design assessments to fully managed environments. Mirantis has received a long standing customer support satisfaction score of 9.2 out of 10 with their multiple tiered offerings.

Red Hat also offers OpenStack training, although it is specific to their distribution only. We don’t have the quantities of people trained and certified, but given Red Hat has a strong training corp around their operating system, we’ll give them the benefit of the doubt for their OpenStack training and score them equally.

Red Hat also has a professional services organization offering several consulting and delivery services. We don’t have their overall customer satisfaction numbers for their services and support, so giving them the benefit of the doubt again, we’ll score them on par.

Review #8 Scores

Mirantis gains a slight advantage here for their training offerings in both the pure-play community and distribution specific OpenStack, whereas Red Hat only offers their distribution specific training. Their support and pro-serve offerings are on par.

Summary

Mirantis has the overall advantage with a score of 62 to 53. There is a heavy advantage in the complete solution of Mirantis’ OpenStack distribution. From direct user experience benefits to community direction and participation, Mirantis offers more than the other vendor, Red Hat.

For more information and to download Mirantis OpenStack 7.0 to try it for yourself, visit www.mirantis.com.