Monitoring with Big Data ~ Splunk

Traditional monitoring is nice and usually does a very good job at their tasks, such as monitoring a sensor, temperature or pinging a server and checking levels, but it is often something which you set up and have to constantly maintain while your system is changing, and this is quite hard to do when the environment is many hundreds of servers where different departments are changing stuff and once something is missed, the data is lost, as it is not collected.

This is where big data comes in and saves you from your ordeals.

With big data you're able to save yourself many times, but it is not something which would replace your traditional monitoring solution, it is there to help you out from another angle. where traditional monitoring just is not enough.

Big Data will help you become a forensics expert in your own environment where you can catch many bugs and issues before they become issues, with a simple reason, you are able to corelate data from many different sources and see trends and find dependencies which you would otherwise miss out on.

All you need for big data to work is already in your local environment, simply your own log files.

If you have your internal system values logged in some logfile on your servers, same log files you have your sytem statuses/error messages in or if you are going to add new ones, done... The data is there and you only need to make a dashboard in your big data system to display that data in a visibly and interactive way.

Splunk is such a tool which does just that.

As a simple example from a security point of view, you have accesslogs on your systems, but in order to view them, you need to login to the servers and check them manually, you locate accounts which have logged on and then try to open a different log file to see what has happened on the system, a lot of manual work which is very time consuming.

but at the same time if big data was collecting that same logfiles from all of your servers, you could easily spot every account which has accessed any system, what the accounts were doing and at the same time make the system alert you in a very easy manner that an account tried to access or change something which was not allowed or wrote the wrong password 2-3 times and could be compromised.


Check out some nice commands to use in splunk



Posted on: Thursday 11 June 2015