The Myth of Self-Service Business Intelligence

Liberating IT?

Remember D-Day? That was the day in June 1944 when Allied troops stormed the beaches of Normandy to liberate Europe from the Nazi occupation. But there is another D-Day, one that is less well known but yet significant in its own right. That was the day in 1990 that data warehousing came to life. This is the day that information technology (IT) managers celebrate with gratitude and fondness. (Or at least they should!)

At the time, data warehouses promised to liberate IT from the drudgery of creating custom reports. Previously, the IT department was swamped. It couldn’t keep pace with the demand for custom development and was slowly drowning in a backlog of requests, a good portion of which were for custom versions of corporate reports. (Alas, things haven’t changed much for some organizations where IT groups are still bogged down with requests for custom reports.)

Admittedly, IT was cautious at first about creating redundant data stores—a data warehousing prerequisite and a long-time IT taboo emanating from the days of expensive disk storage. Once IT realized it could offload a huge portion of its work to end users, however, it began to evangelize the benefits of data warehousing to the business. Soon enough, data warehousing and its successor, business intelligence, became a clarion call for IT and a booming career opportunity as well.

The heart and soul of data warehousing, at least to an IT professional, is the notion of self-service reporting or self-service BI. Here, business users create their own custom reports using end user–oriented query and reporting tools running against a data warehouse. IT steps aside as an intermediary between users and the data and gives users what they’ve demanded for years: complete and unfettered access to data without IT interference. All that IT needs to do is set up the data warehouse and provide query and reporting tools. It can then focus on more value-added activities, such as developing new applications.

Data warehousing created a win-win situation in which business users gained direct access to data and IT managers eliminated the need to create custom reports—or at least that was how it was supposed to work.

Downsides of Self-Service BI

Unfortunately, as many organizations have discovered the hard way, self-service BI is a myth and doesn’t translate well to reality. Although the concept is valid, implementation is misguided. The result is reporting chaos.

One cause of the chaos is that organizations, keen to empower users with data warehouses and BI tools, go overboard. They give users too much responsibility for generating the information and reports they need to do their jobs. In reality, most users don’t want this responsibility, and it’s not part of their job descriptions. It takes too much time, and users often make mistakes and get frustrated. If they do get training, users usually forget how to use the tool by the time they need to create a report. Consequently, they either stop using the tool or call IT to create the report for them. The organization then finds itself exactly where it was before it spent hundreds of thousands, if not millions, of dollars on its DW/BI program.

For example, the human resources department in one large organization I worked with discovered that it had 26,000 different reports serving 450 active users out of a potential of 3,500. Most of the reports were variations on a couple of themes, and most hadn’t been used in months or years, but they were still sucking up disk space and cluttering report folders. The majority of the 3,500 employees who could benefit from the BI infrastructure found the tools difficult to use or couldn’t find the right report in the directory.

There is another, more deleterious, downside to reporting chaos than lack of usage and BI tool shelfware. It occurs when the small percentage of users who do employ the new tools don’t define key metrics, accounts, and terms in a consistent fashion. This makes it impossible for executives to get a harmonious view of business activity, and they may unwittingly make critical decisions based on inaccurate information.

Another problem is that overambitious business users may submit costly, long-running queries, which bog down query performance for all other users. When result sets take a painfully long time to return, business users accustomed to split-second response times for Google queries stop using the BI tools altogether. What’s ironic is that most of these network-clogging, runaway queries are usually unnecessary; users typically need only a fraction of the data that they include in their queries.