Last week the Gartner blog became the platform for some data industry click bait.

Beats me, but for some reason organizations think that they can build A SECURITY DATA LAKE and/or their own CUSTOM BIG DATA SECURITY ANALYTICS tools. Let me tell you what will happen — it will FAIL.

[ … ]

OK, let me tone this down a bit — it will be successful (however this is defined) for 0.1% of those who try [the percentages are approximate and are meant to increase the dramatic impact of this post, not to share data]

The opening stanzas of the diatribe (quoted above) set a hyperbolic tone that rambled on before not getting particularly anywhere. If you filter out the sarcasm you can see that the author has indeed done the leg work talking to security organizations and presents some of the very real problems they face. Filtered here:

Dirty data

Trouble collecting data

Trouble accessing data

No value beyond collection

No value for threat detection

Failure to conceptualize security analytics use cases

Good security analytics use case design is hard

No value beyond keyword search

Level of required technical talent out of reach

A rewrite of the opening statement can give us some very valuable insight into the current state of the industry.

“Facing the current landscape of threats fortune 500 organizations believe that building a security data lake and their own custom big data security analytics and tools will improve their security posture.”

Everyone’s next question should be “Why do they feel that way?”. The implication, from the Gartner blog, is that collectively; security professionals are easy marks who have been sucked into the big data hype. The real answer is significantly more complex.

Many security professionals find them selves incredibly disappointed by the current state of the security platform market. The tools within reach of their budgets are often flashy but don’t get to the basic premise of their work (defending the company from threats).

Let’s imagine a theoretical large organization trying to re-posture. The gateway drug for a big data implementation is a pre-existing SIEM. The organization may even already have a foot in the Big Data world. A number of large fortune size organizations tail their SIEM’s into Hadoop or systems like it to hold history for compliance reasons. Meeting with security operators from large orgs in and out of the tech industry surfaces patterns of frustration with their SIEM initiatives. Here is a sampling of some of the top frustrations:

Dirty data

Trouble collecting data

Trouble accessing data

No value beyond collection

No value for threat detection

Failure to conceptualize security analytics use cases

Good security analytics use case design is hard

Analysis doesn’t scale outside recent history

This list should look familiar, its practically the same as the “reasons your big data security project will fail”. The issue of scale is solved by moving to platforms like Hadoop but the core challenges, which are challenges for all big security organizations still remain.

Internet explorer dominated the “number one browser in the world” spot until the first time you used Firefox and realized what was missing.

The goal of Apache Spot is to solve these specific challenges. The community recognizes the complexity of the operational challenge in security and strongly believes that industry wide collaboration is needed to solve them.

Spot Core is working to solve:

No value beyond collection

No value for threat detection

Good security analytics use case design is hard

Spot Operational Analytics is working to solve:

Trouble accessing data

Failure to conceptualize security analytics use cases

Spot Information Accessibility Initiative is working to solve:

Dirty data

Trouble collecting data

We hope Anton keeps his eye on the project and that we can help him lose his skepticism. But realistically we will inevitably fail unless the skeptics become contributors and help us solve the real challenges.