How Sitecore CMS Architecture Decisions Interact With Future Multivariate Testing Options

What is CMS Architecture?

A key property of content management systems (CMS) is that they provide structure to the content management experience—largely through content modeling*, which structures the input of data; the way that templates are logically and modularly constructed; through the design of metadata and tree relationships among content; and through workflows and permissions. The many decisions that happen to achieve usability behind-the scenes we call CMS Architecture, as distinct from Information Architecture, which is more commonly called UX these days.

The exact same user experience on the front-end can be created by many different back-end CMS architectures, which are a reflection of both general back-end usability practices, and considerations specific to the business context. Many developers do not understand how to do this well, or cut corners to make their jobs go faster—and it has all sorts of implications for the usability of CMS implementations and the flexibility of their front-ends.

To elaborate further—in one site, a particular type of layout is comprised of swappable “modules” each one made up of a series of carefully connected content items to create maximum granularity. In another site, that looks and functions the same to the user, the exact same page is a single “rich text” area, with very little that is supporting its structure on the back-end. The first may be completely testable; the second may have very limited testing capabilities.

Multivariate Testing in Sitecore 8 & Sitecore 7

One of the great features of Sitecore 8 and previous versions is fully integrated multivariate testing, allowing users to easily run permutations of different text, image and layout elements in an automated way against site traffic. Like natural selection in evolution, multivariate testing goes beyond what any Web specialist might think to provide objective (hopefully statistically significant) data about what really works.

This can be easily set up in the Sitecore interface by marketers with a minimum of training—and have great benefits over 3rd party testing tools since the content and HTML of the tested variations are all managed through the same system and interfaces. Sitecore keeps track of all the different possible combinations of tested elements behind the scenes and does the hard work for you of scoring the different permutations and allowing you to find easily the most objectively successful combinations.

How Multivariate Testing & CMS Architectural Decisions Interact

That rosy circumstance described above is not the end of the story, however. In order for page elements to participate in multivariate tests in Sitecore, they have to be discrete items in the Sitecore back-end, not just as fields (which cannot be tested as such), but as either of these two types of entities:

Separate managed content items, or

Separate swappable layout modules within a template (called “sublayouts”)

If these conditions are not met, it is still possible to create A/B testing (gaming complete different versions against one another, page to page), or to otherwise test performance, but true “multivariate” testing, with its crucial matrix and score of permutational conversion performance across elements, is not possible. You might be able to test, for example, a title and an image as a block but not varied titles against images. It is quite possible to get to a point where you have a beautifully functioning site launched, decide you want to start optimizing with multivariate testing and—BLAMMMO!—you just cannot. At least not without recoding and reconfiguring the pages through a significant development effort. You might be able to test some aspects, but not what you want to, like testing the slogan and an image as independent multivariate variables if they are part of the same Sitecore component rendering.

In other words, how one decides to build the site on the back-end in Sitecore directly affects what tests are possible. If the pieces to be multivariate tested are separated as different content items and different related sublayouts that are swappable in the page, it then becomes possible to test them. The items that are testable in a multivariate test are either different content items or different sublayouts, or both—NOT just fields within a content item. If they are grouped together, (for instance, as fields in complex content types or without any back-end structure at all, as in the case of rich text areas), they won’t be truly multivariate testable.

One might leap to the conclusion that if one cares about testing, one should always implement sites in the maximally engineered way—with all page elements broken up as separate sublayouts or at least as separate content items on the back-end. However, this has the potential to create a number of secondary problems, to wit:

If the page and content management is over-engineered, the back-end may become less usable for other regular tasks. The structures that preserve testability on the front-end may lead to confusing data entry and editing in relation to day-to-day tasks. One finds oneself having to make edits in three content items just to change a small piece of a page—to preserve full testability. Development time slows down since there are many more tasks per element that have to happen and be tested to complete programming. Although the other extreme is also bad back-end usability, (for instance, the “whole page as rich-HTML-area” disaster), something in the middle is more reasonable for most builds. There is also a danger in creating unsustainable implementations by over-engineering. Too much back-end complexity can make the next development task in the given area onerous to figure out to developers who may not be familiar with the original development work.

When is "Unstructured" Actually Better?

As discussed above, the ability to perform true multivariate tests is directly related to how elements and sublayouts are implemented in Sitecore. This generally pushes in the direction of more engineered approaches to content and presentation.

However, it is worth mentioning that there are also some scenarios where a less-structured, richer text approach can be advantageous. Although combining elements into a rich text field generally precludes true multivariate testing, there are times when testing chunks of HTML is the straightest path of testing at all—because what is being tested are a large number of very different experiences that are by their complexity and number far easier to mock up as self-contained HTML.

Trying to model fully each in the CMS back-end if they are potentially disposable may not be practical. Engineering something complex in the CMS back-end takes time, which adds overhead that may limit the range of different ideas that can be tested or at the very least provide poor ROI if there is a lot of ongoing testing of varied, complex experiences. In these cases, true multivariate testing may not be practical, but the same data may be achievable by mocking up the multivariate permutations—assuming there are not too many.

To give an example, imagine if you had a financial grid on a page and you want to test two versions, one with labels on rows and the other with labels on columns (or permutations that are more complex). Rather than do the work to create two automated sublayouts that draw from Sitecore managed data, it might make more sense to mock up the options in rich HTML with hard-coded data for the moment until testing is complete.

Then, once the test has provided the appropriate data to make a decision, one can put time into executing the final, fully content managed implementation of the winning component.

An Art, Not a Science

All of this further emphasizes that CMS architecture is an art more than a formal science. It's all about the trade-offs that give contextually appropriate usability and once one understands the contents of this blog post, there is another dimension to consider. Good CMS architects think about a variety of elements when they design implementations: how will this content (and these relationships) be edited? Who will edit what content? How will these page templates be reused, and how do they need to be flexible? What is the right level of granularity, and separation of elements into separate content items, to drive future content reusage?

Add to this one more consideration: how will these page elements be tested? Of course, it is not always easy to imagine all future testing scenarios, but we know this definitively: thinking about testing will influence back-end choices in many cases, in which testing and data-driven design is at a premium.

*called “data templates” in Sitecore