Measuring Success

“Measurement is the first step that leads to control and eventually to improvement. If you can’t measure something, you can’t understand it. If you can’t understand it, you can’t control it. If you can’t control it, you can’t improve it.” – H. James Harrington, Author & CEO (1)

Before we get directly to the topic of increasing the success rate of BI Initiatives, we must first touch upon a few common approaches to measuring the success of BI within an Organization.

How do you measure the success of a BI Initiative? If you’re like many of the organizations I’ve come across during my fifteen plus years in BI, you don’t. Organizations that do attempt to understand the success of BI Initiatives likely use one or more of the following approaches:

Usage Tracking – Track the number of users who have a “BI” license, the number of times per week/month a user logs into BI Systems, the amount of time users are actively engaged with the System, etc.

ROI – Measure the ROI of BI Initiatives. A typical approach to this strategy might be to work with Business Partners to define the opportunity cost (i.e. what does it cost the organization every day it does NOT move forward with a given BI Initiative) and then compare the estimated opportunity cost with actual results post implementation.

Word of Mouth – Look for empirical evidence, such as the number of senior leaders who mention “partnering with BI” on strategic initiatives, as an indicator of BI success.

Each of the above approaches has its merits, but with failure rates of BI Projects somewhere in the 30-75% range, more work is needed.

“It’s ironic that we are the group that measures everything in the organization, but we don’t have a good way to measure ourselves and our effectiveness.” – Eric Colson, Chief Data Officer (2)

In my opinion, the best indicator of BI success within an Organization is user adoption. I use the term “User Adoption”, rather than Usage Tracking, because adoption is more than having a license and logging into a BI System once a week. Dictionary.com defines adopt as “to choose or take as one’s own”.

User Adoption

User Adoption of BI Systems is primarily driven by three characteristics: the system provides tangible business value, the system is easy to use and the system is flexible enough to meet end users’ needs over time. Here are a few questions to consider when thinking about or attempting to measure each of these characteristics:

Tangible Business Value: What is the business need or opportunity driving this implementation? How will business value be measured? How secure, accurate and reliable is the System’s underlying data? How will users share information and insights gleaned from the System?



Easy to Use: How will users access the System? Will users require mobile access? Is the User Experience (UX) intuitive? How much training will be required for users to fully leverage the System’s capabilities? How will training and documentation be made available to users? What is the simplest design possible to meet the intended business objective(s)?



Flexibility: How easily can the System adapt to typical changes that occur within the Organization, Industry, etc.? How quickly can users investigate potential issues or answer ad-hoc questions directly within the System? What “call(s) to action” does the System provide? How does the System enable users to take immediate action in the event of a valuable insight and/or issue?



If User Adoption is the key indicator of the success of a BI Initiative, how do you improve it and thus increase the overall success rate of BI within your organization?

Four Steps to Increase BI User Adoption via Business Engagement

The first step in increasing BI User Adoption and therefore the overall success rate of BI within your organization is to recognize BI Initiatives cannot be thought of in the same way as traditional IT projects. The primary purpose of BI is to transform data into meaningful information for business analysis and decision making purposes. Therefore, success is dependent upon the support of business leaders as well as alignment with business objectives in a way typical IT infrastructure and support initiatives are not.

The second step in increasing BI User Adoption is to require an Executive Business Sponsor for every BI Initiative. There are myriad reasons Executive Sponsorship is critical to the success of a BI Initiative, but for purposes of this discussion I will highlight two:

Executive Sponsors provide the high-level visibility, support and resources (i.e. the buy-in and commitment) needed

Executive Sponsors are typically responsible for the types of “business outcomes” required to define strong BI Business Cases

The third step in increasing BI User Adoption is to develop a strong BI Business Case. There is plenty of information available regarding how to write a business case. For my purposes, I will simply highlight the key components of a strong BI Business Case:

Executive Summary: The key to ensuring Senior Leaders within the Organization can understand the initiative without difficulty

Business Problem Statement: A clear, easy-to-understand overview of the business problem and/or opportunity

Cost Benefit Analysis: A summary of the pros and cons, including associated financials, of moving forward with the Business Case

Recommendation: Highlighting key aspects of the Cost Benefit Analysis, the Recommendation summarizes what the organization should do and why

It doesn’t take a Rocket Scientist (perhaps I should use Data Scientist given the audience) to see the connection and overlap between the objectives of an Executive Sponsor and essential components of a strong BI Business Case.

The fourth and final step in increasing BI User Adoption is the use of an Agile Delivery Methodology. For those familiar with the Agile Manifesto and its associated principles it probably comes as no surprise we ended up here given my focus on User Adoption & Business Engagement. Agile simply provides the best approach available to continuously deliver tangible Business Value to the Organization via Business & IT collaboration and therefore offers the greatest chance for strong End User Adoption.

In this whitepaper, Pentaho describes Agile BI as “the ability to create BI solutions quickly and easily, in a nimble and well-coordinated way” and states “using an agile approach improves the success of BI projects, and enables you to start more projects. It does this by changing the economics, the technical solution, and the execution of the projects.”

Following the four steps recommended here will not guarantee the success of your next BI Initiative of course, but it will certainly increase your odds of victory.

Follow us on Twitter, Facebook & Google+

1. http://tinyurl.com/osze93m

2. http://tinyurl.com/ojj7jdh