Survey Analysis: BI Platform Challenges and Limitations, 2014

This research will help alert business intelligence leaders to potential issues with their business intelligence platform or vendor by highlighting specific problems and concerns reported by organizations surveyed as part of Gartner's research for the Magic Quadrant for BI and Analytics Platforms.

Key Findings

Negative sentiment about the future of the megavendors is increasing while users in other platform categories are indicating an increase in positive opinion about their future, further validating the transition from traditional BI platforms to new delivery models.

Approximately 20% of references cited license cost as a limitation to wider deployment, which highlights the fact that software cost is the main consideration and focal point for organizations attempting to deploy BI on a larger scale, both internally and externally.

References using data discovery platforms indicate the highest percentage of "other" problems and limitations indicating there are new and unique challenges in data discovery platforms that should be considered.

Recommendations

Business intelligence (BI) leaders evaluating options for BI platform purchases should:

Use this research in conjunction with the BI and Analytics Magic Quadrant and related survey analysis notes as well as BI RFP templates and critical capabilities research to ensure that a comprehensive view of each option is used to inform purchase decisions.

Consider running a proof of concept or pilot program with data and use cases specific to your organization to determine the viability of each BI platform prior to purchasing.

Realize that license cost is a relatively small contributor to BI platform ownership cost (BIPOC) on a per-user basis over three years, and it should not be the only criteria used when comparing total cost of ownership metrics across platforms.

Survey Objective

The purpose of this research is to utilize the extensive "Magic Quadrant for Business Intelligence and Analytics Platforms" customer survey data that Gartner collects annually. It helps provide additional insight into the platform problems and limitations that references have revealed, as well as allowing us to assess the future state of each BI vendor. Table 1 provides a detailed definition of the vendor category segments used throughout this analysis.

Table 1. Vendor List by Category

Vendor Category Vendors Cloud BI Birst

GoodData Data Discovery Leaders Tableau

Tibco Spotfire

Qlik Large Independents Information Builders

MicroStrategy

SAS Institute Megavendors IBM

Microsoft

Oracle

SAP Open Source Actuate (BIRT)

Jaspersoft (acquired by Tibco Spotfire)

Pentaho Small Independents Alteryx

arcplan

Bitam

Board International

Infor

Logi Analytics

Panorama Software

Prognoz

Pyramid Analytics

Salient Management

Targit

Yellowfin Other Vendors Adaptive Insights

Advizor Solutions

Chartio

Dimensional Insight

Dundas Data Visualization

eQ Technologic

InetSoft

Jedox

Jinfonet Software (JReport)

Kofax (Altosoft)

Lavastorm Analytics

Phocas Software

SiSense

Software AG (JackBe)

SpagoBI

Strategy Companion

BI = business intelligence

Source: Gartner (August 2014)

The "Magic Quadrant for Business Intelligence and Analytics Platforms" customer survey results used in this analysis included a total of 1,589 responses that are used in all vendor and vendor category level views.

Total survey responses received from:

Vendor-provided references — 1,470 (93%)

BI users from Gartner BI Summits — 33 (2%)

Respondents from last year's survey — 86 (5%)

Figure 1 shows a breakdown of survey responses by vendor category and vendor, for reference purposes.

Figure 1. Number of References Included in Survey Analysis Vendor Category and Vendor

BI = business intelligence

Source: Gartner (August 2014)

In 2014, we included OEM references in the actual survey, but those responses were not factored into the Magic Quadrant scoring process and consequently these will be excluded from all survey analysis notes this year, with the exception of "Survey Analysis: BI and Analytics Spending Intentions, 2014" that details market spending patterns, which is not included in the Magic Quadrant scoring methodology.

We include vendors in this research that may not have met the Magic Quadrant inclusion criteria for an actual position within the Magic Quadrant, and as such, are reported in aggregate as "Other Vendors" in the figures depicted throughout this analysis. Please see Note 1 detailing the inclusion criteria for a position on the "Magic Quadrant for Business Intelligence and Analytics Platforms" and Note 2 for specific vendor exclusions.

For the product level views, the total number of responses included in the analysis is 1,551 due to the fact that some products did not have a significant number of responses to enable an accurate assessment, so were consequently excluded. However, these responses are included in the vendor and vendor category aggregated views, so were also factored into the Magic Quadrant scoring and positioning. Figure 2 shows the number of responses received for each product included for consideration in this analysis when evaluating the survey results reported for each of the product level views.

It is important to note that because the product level views include fewer responses than vendor or vendor category views, overall survey average results reported in this analysis for the same metric will be different in product views in comparison with those reported for vendor or vendor category.

Figure 2. Number of References Included in Survey Analysis by Product

BI = business intelligence

Source: Gartner (August 2014)

Data Insights

BI leaders are challenged with the task of providing business intelligence and analytic tools that meet the current and future demands of the user community, while also satisfying the requirements of IT departments (those responsible for supporting the platform). This research is intended to assist BI leaders in making informed technology-related purchase decisions by revealing the feedback received from survey references regarding problems faced with the BI platforms they are using, and the limitations that they have discovered, which have resulted in restricting wider deployment and adoption. In addition to questions regarding specific problems and limitations, we also asked references to provide feedback on sentiment toward their vendor as well as providing their perspective on the prospects of the vendor in the future, and ultimately, whether they are considering discontinuing use of their current BI vendor.

Future State Assessment of BI Platform Vendor

View of Vendor Future

As part of this year's survey, we asked reference organizations to assess the future prospects of their current BI platform vendor, by asking whether they felt positive or negative about their vendor's future, and also whether their view had undergone any change. (However, those references indicating that they have been using the platform for less than one year have not been represented in the following figures.)

As shown in Figure 3, 4.8% of references indicated that their view has changed to negative with the megavendor category customer references (since last year), and are now indicating the highest overall percentage of new negative sentiment toward their current vendor at 11.3%. More encouragingly, since last year, 26.7% of references indicated that they have changed their view of their BI platform vendor to positive within the categories cloud BI, data discovery and open-source vendor showing the highest degree of positive change compared with last year's figures.

Figure 3. Percentage of References by Vendor Category Indicating Changed View of Vendor Future Compared With Prior Year

BI = business intelligence

n = 1,589 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

The vendor category results shown in Figure 3 are further broken down by individual vendors in Figure 4. This illustrates the results from references that indicated that their view of the future of their vendor has changed since last year. Organizations using MicroStrategy, SAP, Infor and Oracle as their BI platform vendor reported the highest levels of negative sentiment (which has changed from the prior year). It is clear that those references that use BI platforms in the categories of cloud BI, data discovery and open source have a more positive view of their vendor's future compared with last year's results — with almost 50% of references indicating a change in sentiment with positive feedback for vendors such as Birst, Tableau, Jaspersoft and Pentaho.

Figure 4. Percentage of References by Vendor Indicating Changed View of Vendor Future Compared to Prior Year

n = 1,589 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

Figure 5 depicts the results for references indicating that their view of their vendor's future has not altered since last year. Overall, 6% of reference organizations indicated that their view of their BI platform vendor remains negative with the megavendor; and the category of other vendors references reporting the highest level of continued negative sentiment. At the other end of the spectrum, many organizations remain positive about their vendor's future, with 56.9% of references indicating positive sentiment — led by the large independents and other vendor categories.

Figure 5. Percentage of References by Vendor Category With No Changed View of Vendor Future Compared With Prior Year

BI = business intelligence

n = 1,589 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

Figure 6 expands on the vendor category results depicted in Figure 4, based on references indicating that their view of their vendor's future has not changed since last year. The highest levels of persistent negative sentiment are found in the category of other vendors, with several vendors well above the overall survey average — in terms of the percentage of references indicating a negative view. Within the Magic Quadrant vendor group, over 10% of references for Infor, MicroStrategy and Microsoft report that they maintain a negative view of the future for their BI platform vendor.

Figure 6. Percentage of References by Vendor With No Changed View of Vendor Future Compared With Prior Year

n = 1,589 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

Plans to Discontinue Use

As shown in Figure 7, an overall total of 3.2% (combined plans within one- and three-year categories) of survey references have plans to discontinue use of their current BI platform over the next one to three years; with an additional 6% indicating that they are considering discontinuing use but as yet, do not have actual plans to do so. At the individual product level — within the Magic Quadrant vendor category — users of SAP BusinessObjects BI Platform 4.1, Board International, Actuate (BIRT) and Bitam report the highest overall percentage of references that have actual plans to discontinue use (as indicated by the red and orange bars on the left side of the chart in Figure 7). Also, within the Magic Quadrant vendor category, references using SAS, arcplan, IBM Cognos 10 and Infor appear to be considering alternate platform options, given the high percentage of references who indicated that they are considering discontinuing use of their current BI platform (as indicated by the circles on the right side of the chart in Figure 7). In the other vendors category, there appears to be more consideration given to discontinuing use by reference organizations, with several products in the group ranking well above the vendor average in terms of the percentage of organizations indicating that they are planning or considering discontinuing use of their BI product.

Figure 7. Percentage of References Planning or Considering Discontinuing Product Use

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

BI Platform General Problems

Survey references were asked to report on any general product problems they have experienced across the areas specified (in Table 2). Results from the survey are indicated in the following figures in this section and are reported based on the percentage of references using each product that indicated they had experienced a problem across the areas covered in the survey. Table 2 summarizes the platform problems featured in the survey with results ordered by significance.

Table 2. Summary of Product Issues (in Order of Significance)

Problems Identified % Average (Significance) Figure No. Other Problems With Platform 14.9% 9 Poor Performance 9.4% 10 Absent or Weak Functionality 9.3% 11 Unreliable, Unstable, and/or Buggy 9.1% 12 Difficult to Implement 8.0% 13 Difficult to Use 6.1% 14 Inability to Handle the Required Data Volumes 5.9% 15 Inability to Support Large Numbers of Users 2.1% 16

Source: Gartner (August 2014)

The heat map in Figure 8 provides an overall summary (by vendor category) across the general platforms of problems that will be explored in greater detail in the following sections, which are all reported at the product level.

Figure 8. Summary of Product Issues by Vendor Category

BI = business intelligence

Source: Gartner (August 2014)

Other Problems With Platform

Survey references were also given the opportunity to stipulate whether they had experienced any other platform problems that were not actually specified in the list of options in the survey. As depicted in Figure 9, approximately 15% of references indicated that they experienced problems that were not related to issues with performance, scalability, complexity and quality.

MicroStrategy, SAP BusinessObjects Enterprise XI 3.1 or earlier, SAP BusinessObjects BI Platform 4.1, Pyramid Analytics and GoodData were also rated poorly by their reference organization with at least 20% reporting other issues with the platform.

Figure 9. Percentage of References Citing Other Platform Issues Not Specified in the Survey

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

Of particular interest is the fact that the data discovery products, Tibco Spotfire, Tableau and Qlik's QlikView, were cited by more than 25% of their users as having experienced other platform problems. Actual reference responses collected from the survey, highlighting the types of problems experienced specific to data discovery are included in Table 3 through to Table 5.

Table 3. Verbatim Responses of Other Platform Problems for Data Discovery Vendor Category — Tibco Spotfire

Tibco Spotfire Verbatim Response Availability of IT consultants for Spotfire Compatibility with Excel could be much better Lack of consulting services follow-through Conversion of users from legacy platforms Difficult to find trained professionals in my region (LATAM) Difficult to find developers Does not look good outside-of-the-box, therefore we've had to write a top layer ourselves Effort required to upgrade In application load balancing, "smart" file loading across multiple VMs iPad version needs a multiselect for their list box Lack of support for older versions of the product Limited formatting options Migration from DecisionSite Not a problem, per se, but it is hard to extend the tool via custom code (e.g., adding buttons with customized programs) Not all functionalities work out-of-the-box, some require installation of additional programs Not enough time spent on admin capabilities, one feature was even dropped from the product (Configuration Console) Oracle Server Outdated graphics Subvendor knowledge UI not very flexible Version control for visualizations, deployment automation Viewer cannot customize analysis We have had multiple issues with new versions Working with SAP Hana and the overall IT SAP strategy

LATAM = Latin America region; UI = user interface; VM = virtual machine

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

Table 4. Verbatim Responses of Other Platform Problems for Data Discovery Vendor Category — Tableau

Tableau Verbatim Response Connection with SAS database Data cleanup issues caused performance issues, but were resolved Enterprise metadata management needs to be easier to do Finding resources to create visualizations Importing metadata Issues associated with not having a true enterprise-grade product Lack of templating abilities Limited administrative features MDX Cube Queries perform slowly Minor problems, needs additional development Minor technical issues, nothing material Needs integration with SAP — Business Objects Universe Not available on Linux OS X Requires too much in-depth knowledge of creating complex formulas Server hanging issue with Version 8 Some integration issues Symantec layer is weak We are "young" X32 memory limitations

OS = operating system

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

Table 5. Verbatim Responses of Other Platform Problems for Data Discovery Vendor Category — QlikView

QlikView Verbatim Response Absence of semantic layer Amount of developing work needed to build new ground-up views and dashboards Challenges engaging operational stakeholders Data loading process Difficult to implement QlikView Expressor Enterprise management capabilities weak Full self-service for non-IT minded business developers Integration of security across applications iPad App not prime time for iPad Lack of data modelling and cleansing features Licensing unclear Sector knowledge within the organization Support is not very responsive and does not live up to expectations Weak standard reporting capabilities

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

Poor Performance

As shown in Figure 10, 9.4% of survey references cited performance issues with their BI platform; specifically with IBM Cognos 10, references reported poor performance in 33.3% of reference organizations. However, IBM Cognos 10 is used to query very large data volumes — and is second only to MicroStrategy in the Magic Quadrant vendor product group, in terms of average query size as indicated by the blue line chart plotted along the top axis (in Figure 10). Notably, this statistic represents the size of the dataset being returned in the query result set, not the data repository itself. The size of the data repository can also contribute to performance issues but are often related to the underlying database architecture — and not necessarily the BI platform being used to access the database. Despite being used for some of the smallest queries, out of all the products evaluated, Infor was cited by 18.4% of its users for poor performance. Not surprisingly, references using in-memory platforms such as QlikView cited performance as an issue in only 2% of reference organizations.

Figure 10. Percentage of References Citing Poor Performance With the Platform

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

Absent or Weak Functionality

As shown in Figure 11, 9.3% of survey references indicated that their BI platform is either guilty of providing weak functionality or is completely lacking in some expected capabilities entirely. There is significant diversity in the vendor platforms that performed weakest in this area. For example, over 20% of references using SAP BusinessObjects Enterprise XI Versions 3.1 or earlier, SAP BusinessObjects BI Platform 4.0 and SAP BusinessObjects BI Platform 4.1, as well as users of Microsoft SQL Server 2012 in the megavendor category, reported that there are weak or absent functionalities with the platform — despite the fact that both contain products within the stack covering a broad range of capabilities. It is somewhat less surprising to see smaller vendors such as Targit and GoodData here, because they may indeed be missing functionality or providing capabilities that are weaker than other platforms, given the fact that they are niche vendors, which tend to provide less of the spectrum of BI and analytic capabilities. However, there are other similar vendor platforms that were not considered by references to be absent or lacking in functionality. One possible explanation for this is that the products that scored well such as SAS, Prognoz and arcplan are being deployed against very specific use cases, whereas organizations using Targit and GoodData are attempting to deploy the platform widely — and are therefore identifying functionality gaps along the way, as shown in Figure 12.

Figure 11. Percentage of References Citing Absent or Weak Functionality With the Platform

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

Unreliable, Unstable, and/or Buggy

Overall, 9.1% of survey references reported that their platform is unreliable, unstable and/or buggy, but most vendors scored better than the survey average, since most of these problems are concentrated in a small subset of platforms as shown in Figure 12. Oracle, SAP, MicroStrategy, Yellowfin and GoodData were all cited by 15% to 25% of their respective references as having quality-related product issues. Of particular concern is the fact that all three SAP BusinessObjects products included in the survey received poor ratings by reference organizations for product quality. Fortunately, there is some improvement as SAP BusinessObjects BI Platform 4.1 was rated somewhat more favorably than either version 3.1 or 4.0, still well above the overall average for the products included in the survey.

Figure 12. Percentage of References Indicating the Platform Is Unreliable, Unstable or Buggy

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

Difficult to Implement

Overall, 8% of survey references reported difficulty implementing their vendor's platform, as depicted in Figure 13. Implementation may include activities such as:

Installation and configuration of software

Semantic layer development

Security administration

Software distribution to users

Content creation

The vendor platforms that were rated most difficult to implement by references such as SAP BusinessObjects BI Platform 4.0 and SAP BusinessObjects Enterprise XI 3.1, Actuate (BIRT), MicroStrategy, Infor and SAS, do typically involve complex and/or large deployments that can complicate implementation tasks. It is somewhat surprising that the cloud vendors, Birst and GoodData, did not perform as well as expected — since the perception is that cloud deployments generally equate to an easier implementation. It is also quite surprising to see Microsoft (SQL Server 2008 and 2012) had no references reporting that the platform is difficult to implement — given that the general perception is that the various components within the Microsoft BI stack require expertise to install and configure correctly. This may indicate that most organizations are relying on outside third-party implementation partners to deploy Microsoft BI or that they are implementing only select components of the Microsoft BI stack, which would simplify its implementation.

Figure 13. Percentage of References Reporting the Platform as Difficult to Implement

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

Difficult to Use

Overall, the megavendor and large independent vendor platforms are cited by references as being the most difficult to use. As shown in Figure 14, references using products in those categories such as SAS, IBM Cognos 10, MicroStrategy, SAP BusinessObjects BI Platform 4.0 and SAP BusinessObjects Enterprise XI 3.1 and Oracle BI EE 11g.x rated their platforms difficult to use at a rate of over twice the vendor average. SAS topped the list with one out of four references citing it is as difficult to use — which is consistent with its reputation for being a very powerful tool for statistical modeling and predictive analysis, which essentially means it also comes with a steep learning curve. It should be noted, however, that these responses do not reflect experience with SAS Visual Analytics — which is likely to be perceived as an easier-to-use platform for SAS customers — given its visual interface and data discovery-based capabilities. As expected, data discovery, cloud and several niche vendor platforms were not rated as difficult to use by survey references, with most receiving a difficult-to-use rating well below the vendor average.

Figure 14. Percentage of References Reporting the Platform as Difficult to Use

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

Inability to Handle the Required Data Volumes

As organizations attempt to introduce additional data sources and larger volumes of data into their BI and analytic applications, some survey references are discovering limitations in their BI platform's ability to handle larger datasets. As illustrated in Figure 15, the overall survey average is 5.9%, but GoodData, Tableau and Actuate (BIRT) references cite data volume related problems at more than twice the overall survey average. As shown by the blue line chart, plotted against the top axis in Figure 15, Tableau users access data repositories making it the third largest among all products included in the survey. However, GoodData and Actuate (BIRT) are used against much smaller data repositories, but are cited by a significant percentage of references as unable to handle those required data volumes. The platforms that were not cited by references as unable to handle required data volumes, such as Birst, Prognoz and Targit, are generally deployed in environments with relatively small data volumes — and with great success — as indicated by the fact that no references reported data volume related issues with either platforms for these vendors.

Figure 15. Percentage of References Citing Inability of the Platform to Handle Required Data Volumes

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

Inability to Support Large Numbers of Users

Only 2.1% of survey references indicated that their platform is unable to support large numbers of users, as illustrated in Figure 16. Platform problems related to large deployments appear to be somewhat rare across all of the vendors included in the survey, with single-digit percentages reported for each. Users of the Actuate (BIRT) platform report very large deployments with an average of greater than 3,500 users as indicated by the blue line plotted along the top axis in the chart in Figure 16; but 7.3% of references have cited some difficulty in supporting their large end-user deployments. There were 12 products overall with zero references, citing that the platform was unable to support the number of users in their particular deployment — further indicating that user volume has not been a widely reported issue in most organizations. This may change and has the potential to become problematic, moving forward, as BI adoption continues to grow and usage expands to both internal and external customers, with content delivered via mobile devices and being embedded in websites and applications.

Figure 16. Percentage of References Citing Inability of the Platform to Support Large Numbers of Users

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

No Problems Encountered

In addition to the individual platform problems covered in the previous sections, survey references were also offered the option to indicate if they had not experienced any issues with the platform in their organization. As depicted in Figure 17, approximately 57% of references, overall, indicated that they had not experienced any issues with their product. Topping the list was Prognoz, Salient Management, arcplan and Bitam — with more than 75% of references indicating a problem-free deployment. On the other hand, users of SAP BusinessObjects (versions 3.1 or earlier, 4.0 and 4.1) reported problem-free deployments at a rate of only approximately 33% (on average) across all versions of the product. GoodData ties with IBM Cognos 10, Actuate (BIRT) and Oracle BI EE 11g.x at the bottom, with approximately two-thirds of their references citing some trouble with their deployments, which is somewhat surprising — given the overall favorable ratings of the cloud and data discovery vendors in the survey.

Figure 17. Percentage of References Citing No Platform Problems

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

Product Limitations to Broader Deployment

Survey references were asked to report on any product-specific limitations they consider as barriers to broader deployment across the areas, specified in Table 6. Results from the survey are indicated in the following figures in this section, and are reported based on the percentage of references using each product that indicated whether they considered any of the areas covered in the survey to be a limitation to wider deployment. Table 6 summarizes those problems featured in the survey with results ordered by significance.

Table 6. Summary of Product Limitations to Broader Deployment (in Order of Significance)

Problems Identified % Average (Significance) Figure No. Cost of Software Limiting Wider Adoption 18.9% 19 Lack of Ease of Use for Business Users Limiting Wider Adoption 11.2% 20 Other Product Issues Limiting Wider Adoption 10.0% 21 Lack of Ease of Use for Developers Limiting Wider Adoption 9.5% 22 Software Quality Limiting Wider Adoption 5.5% 23 Support Quality Limiting Wider Adoption 5.2% 24 Poor Performance Limiting Wider Adoption 4.8% 25

Source: Gartner (August 2014)

The heat map in Figure 18 provides an overall summary by vendor category, across the platforms of problems which limits broader deployment and these will be explored in greater detail in the following sections and reported at the product level.

Figure 18. Summary of Product Limitations to Broader Deployment by Vendor Category

BI = business intelligence

Source: Gartner (August 2014)

Cost of Software Limiting Wider Adoption

Overall, almost 20% of survey references cited the cost of software as a limitation to broader deployment — as illustrated in Figure 19 — making this the most commonly reported barrier to broader deployment, out of all the limitations included in the survey. Alteryx, SAP BusinessObjects Enterprise XI 3.1 or earlier, MicroStrategy, SAP BusinessObjects BI Platform 4.0 and Actuate (BIRT) top the list with over 40% of references for each product, indicating that license cost is a significant barrier to wider deployment. As vendors across all categories attempt to simplify their product offerings and offer more streamlined and transparent pricing structures that align with the simplified product offerings, this may yet improve. However, as organizations seek to expand the use and adoption of BI and analytics both internally and externally, by embedding content to reach wider audiences, license cost will likely continue to be the primary focus for organizations as they evaluate their platform options.

Figure 19. Percentage of References Citing Cost of Software Responsible for Limiting Broader Deployment

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

Lack of Ease of Use for Business Users Limiting Wider Adoption

An important consideration for organizations seeking to expand their deployment of BI is ease of use for business users who consume, and in a growing number of organizations, create content. As shown in Figure 20, 11.2% of survey references indicated that they have concerns about ease of use for nontechnical end users. Organizations using SAS and SAP BusinessObjects BI Platform 4.0 reported the highest level of concerns on this topic in the Magic Quadrant vendor product group by a wide margin — both with over 33% of references indicating business user ease-of-use as a limitation to wider deployment. It should be noted that SAP BusinessObjects BI 4.1 received much better scores on this question, indicating that many of the issues experienced in version 4.0 from a usability perspective has been addressed in the 4.1 release. It is no surprise that the products that were rated most positively and with very few concerns regarding business user ease-of-use are products such as QlikView, Logi Analytics and Birst, known for their ease of consumption characteristics.

Figure 20. Percentage of References Citing Lack of Ease of Use for Business Users Limiting Broader Deployment

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

Other Product Issues Limiting Wider Adoption

As illustrated in Figure 21, 10% of survey references reported "other" issues that may limit broader deployment. In the Magic Quadrant vendor product group, Microsoft SQL Server 2008, QlikView, Pyramid Analytics, Tibco Spotfire, SAP BusinessObjects Enterprise XI 3.1 and Tableau were all cited by over 17% of references as having other limitations (not specified in the survey) that may limit broader deployment. An analysis of the verbatim responses (such as those given in Table 3, Table 4 and Table 5) highlights some of the same issues mentioned in the other platform problem category, and for data discovery vendors in particular. Limitations, such as security integration and administrative burden, limited source connectivity options, lack of user interest and adoption and weak or missing metadata layers, were mentioned uniformly across the products, but topping the list with the highest percentage of references were reports of "other" concerns.

Figure 21. Percentage of References Citing Other Issues Not Specified in Survey Limiting Broader Deployment

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

Lack of Ease of Use for Developers Limiting Wider Adoption

The other aspect regarding ease of use (that is often a consideration for organizations considering deployment to a wider audience) is ease-of-use from a development perspective. As depicted in Figure 22, 9.5% of reference organizations indicated concerns about developer ease of use. Users of MicroStrategy, GoodData and Jaspersoft cited the highest levels of concern about developer ease of use across all of the Magic Quadrant vendor products. In contrast to the business user ease-of-use topic, SAP BusinessObjects Enterprise XI 3.1 or earlier was not cited by any reference organization as having concerns about developer ease of use that would limit its broader deployment. Users of Tableau, Microsoft SQL Server 2008 and Bitam also had no concerns regarding developer ease of use.

Figure 22. Percentage of References Citing Lack of Ease of Use for Developers Limiting Broader Deployment

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

Software Quality Limiting Wider Adoption

As depicted in Figure 23, 5.5% of survey references reported concerns with software quality and consider it a limitation to broader deployment to wider audiences. Oracle BI EE 11g.x, Yellowfin, MicroStrategy, SAP BusinessObjects (version 4.0 and earlier) were cited by the highest percentage of reference organizations with concerns about broader deployment due to software quality issues. Similar to other topics covered in the survey, SAP BusinessObjects BI Platform 4.1 was rated much more favorably than its previous versions, and actually did not receive a response from any reference indicating that its software quality was a potential barrier to limitation — thereby reflecting product improvements in the current version of the software.

Figure 23. Percentage of References Citing Software Quality Limiting Broader Deployment

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

Support Quality Limiting Wider Adoption

Survey references were also asked about support quality — as it pertains to broader deployment considerations — and, as shown in Figure 24, approximately 5% of references indicated concern regarding support. The results depicted in the chart in Figure 24 are consistent with the overall support ratings covered in the survey where references are asked about response time, time to resolution, level of expertise (with the exception of Jaspersoft). On the topic of support, SAP, MicroStrategy and IBM scored the lowest overall, so it is not surprising that references have concerns about support and consider it a potential barrier to wider deployment for SAP BusinessObjects Enterprise XI 3.1, MicroStrategy, IBM Cognos 10 and SAP BusinessObjects BI Platform 4.0. However, Jaspersoft was rated favorably by references across the three subcategories of overall support, but 15% of references cited support-related concerns when considering broader deployment.

Figure 24. Percentage of References Citing Support Quality Limiting Broader Deployment

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

Source: Gartner (August 2014)

Poor Performance Limiting Wider Adoption

As shown in Figure 25, 4.8% of references consider poor performance to be a limitation to broader deployment. Unlike the Poor Performance section in the platform problems segment, references citing concerns related to broader deployment limitations are reporting on whether they feel that they will encounter scalability issues as usage expands — not necessarily on actual performance issues experienced. There are many variables that factor into the issue of scalability related to performance (such as underlying database design, performance optimization approach in database layer, workload, among others), which are not directly related to the BI product being used for deployment. In some cases, it is likely that organizations are developing strategies to address the underlying issues that they have experienced with their BI delivery to date, which lessens the concern about scalability. This appears to the be the case with IBM Cognos 10, where 33% of references indicated that poor performance was a problem, yet only 10% consider poor performance a limitation to wider deployment. The same is observed for products such as Infor, SAP BusinessObjects BI Platform 4.0, Pentaho and Board International, where there is inconsistency in the performance-related questions covered in the survey.

Figure 25. Percentage of References Citing Poor Performance Limiting Broader Deployment

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

ource: Gartner (August 2014)

No Product Limitations to Wider Adoption

Slightly less than half of survey references reported that there aren't any limitations to broader deployment with their current BI platform, as depicted in Figure 26. Logi Analytics, Birst, Prognoz and Pentaho were rated highest overall in the Magic Quadrant vendor product group, with ratings of "limitation-free" by over two out of three of the reference organizations. At the other end of the spectrum, SAP BusinessObjects (versions 4.0 and earlier) along with Alteryx were not rated well by references, with fewer than one out of five viewing their platform as not having any limitations to broader deployment. This is a topic that BI vendors will have to address rapidly in order to be considered for wider deployment to larger audiences — both internal and external to the organization.

Figure 26. Percentage of References Citing No Limitations to Broader Deployment

n = 1,551 (See Note 2)

Chart represents customer perception and not Gartner's opinion.

ource: Gartner (August 2014)

Methodology

Each year, Gartner evaluates the business intelligence (BI) and analytics platforms market with the ultimate objective of publishing Magic Quadrant research, scrutinizing those results in depth. Part of this process is a large user survey of vendor-supplied references and other organizations. This includes IT, business, or hybrid IT business leaders disclosing their experiences with their vendor's BI and analytics products, as well as how those products contributed to overall business success.

The online survey was developed and hosted by Gartner to support the Magic Quadrant BI platforms analysis. More than 3,763 unique companies were invited to participate. Vendor-provided references (direct customers and OEMs), participants in Gartner's BI summit conferences (globally) and respondents from last year's survey also provided data.

To ensure the integrity of the survey data, each survey response was verified by company respondent email. For survey responses from nonidentified email accounts such as Gmail or Yahoo accounts, the respondent was contacted and had to provide Gartner with a company email address, a company role and other contact information (this amounted to fewer than five responses), all of which were further vetted and ultimately included. Only completed surveys are included in the results.

Acronym Key and Glossary Terms

BI business intelligence BIPOC business intelligence platform ownership cost GB gigabyte LATAM Latin America region OEM original equipment manufacturer RFP request for proposal UI user interface VM virtual machine

Evidence

The survey was conducted over a four-week period in 4Q13, hosted and executed by Gartner. Survey results were used as input to the Gartner "Magic Quadrant for Business Intelligence and Analytics Platforms" research. This research provides details on how survey respondents view their vendor's future prospects, if there are any plans to discontinue use, or any general platform problems and platform-specific limitations to wider deployment.

Note 1

BI Platforms Magic Quadrant 2014 Inclusion Criteria

To be included in the Magic Quadrant graphic, software vendors had to meet all of the following criteria:

Each had to generate at least $15 million in total BI-related software license revenue annually. (Gartner defines total BI-related software license revenue as revenue that is generated from appliances, new licenses, updates, subscriptions and hosting, technical support and maintenance. Professional services revenue and hardware revenue are not included.)

In the case of vendors that also supply transactional applications, each had to show that its BI platform is used routinely by organizations that do not use its transactional applications.

Each had to deliver at least 12 out of the 17 capabilities in the BI and Analytics Platform Capabilities Definition (for further details, see "Magic Quadrant for Business Intelligence and Analytics Platforms"). OEM components from other vendors were not included.

Each had to receive a minimum of 30 survey responses from customers that use its platform in a production environment.

Note 2

Vendor/Product Exclusion

In order to be included in the product level views, a minimum of 12 survey responses is required. This restriction primarily impacted products in the "other vendor" category as many did not receive enough responses to be shown. The 16 vendors/products in the "other vendor" category received a minimum of 12 references and have been included in the figures. One exception is that references responding on behalf of Domo have been excluded from this survey analysis report, despite the fact that 14 survey responses were received.

There were also some products within the Magic Quadrant vendor category that did not receive enough responses to be reported separately, but were included in the vendor and vendor category views, which is the reason for the difference of 38 responses between the vendor and vendor category views (1,589 responses) and the product level views (1,551 responses) as shown in the Survey Objective section of this research.

Source: Gartner Research, G00262570, Josh Parenteau, Rita L. Sallam, 22 August 2014