The Gartner Group believes that Big Data will be the new normal in the near future and defines it as:

“Big data” is high-volume, -velocity and -variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.

The 3 ‘V’’s of big data, Volume, Velocity and Variety. Here at Logentries we identified this pattern back in 2007 and since then have been focused on delivering a class leading solution to this problem.

Volume

The amount of data generated these days can be mind numbing. Even something as simple as a customer purchasing a book online could create thousands of lines of logs when you examine the full stack. We handle billions of lines of log data every day from some of the biggest companies on the net. But with volume comes cost. We know this and have always had value as a priority and are the best value provider in the market today. This is to help ensure that you get the most bang/data for your dollar.

What good is logging data if you have to restrict the quantity of data due to the costs of your provider? You could be missing the most important information! You should be logging as much data as you feel is necessary to aid in quickly and effectively identifying issues as well as helping the business make good decisions. The quicker you can identify, the faster you can resolve and hence provide a better level of service.

Velocity

Logs are a real time concern for companies and are a vital link in their issue resolution chain. Some log analysis providers do not see this as an issue and can only provide you access to your logs 10’s of minutes after the event happened. This does not address one of the primary uses of log files (resolving an issue ASAP) and leaves companies that are dependent on these services vulnerable.

We have always understood the importance of speed to a business and the quicker you get notified and access to your information the better the quality of service you can provide.

Our system is as close to real time as it gets. Try using the live tail feature on your logs, where you can see the events in near real time, i.e. arriving in Logentries within a second or two after the event happened. Our alerts are just as fast, so as soon as a log entry comes into our system and contains a pattern that you wanted to be notified on, a notification is generated and on it’s way to you be it through email, phone app or web hook.

Variety

I doubt there are very many companies out there that do not have their own unique stack in one way or another. There are so many different tools, OS’s, versions, etc that the permutations grow exponentially. So your log management company should not limit themselves to handling certain logs or applications.

We have always ensured that we are as open and flexible as possible when it comes to the log files that we can accept. So to ensure this we have built a system that will accept any type of log file that is in a text based format. We have also delivered tools, which convert some binary type logs to text based logs on the server.

By having this approach of unrestricted file types that we accept, (once they’re text based) means that we can evolve and service your requirements as they evolve over time.

Note we also provide a range of approaches for collecting your log data including libraries for popular languages and frameworks, an open source agent that runs out of the box on Windows and Linux systems as well as providing syslog support. We like to think there isn’t a system we can’t collect your data from!

So when it comes down to Big Data, Logentries have always been focused on the heart of the matter: taking large amounts data being rapidly created from multiple sources and amalgamating them into a place/tool where you can extract the relevant information to help your business grow and be successful.