The term “big data” has been around for a while now. Today, the big data analytics is mainly focused on for example, analyze customer transactions information to determine spending patterns and trends to enable business to better understanding of consumers, resulting greater customer loyalty, increase sales and brand value.  It’s more driven by finding business opportunities, what the consumer needs and what don’t.

So by collecting vast data sets from various resources and applying the analytics to the information, customized products can be offered to consumer based on their needs and likes or dislikes. This approach can also be very much used in understanding and managing enterprise information security requirement, and setup the relationship between the security events & incidents occurring in the enterprise to detect the malicious activity, be better prepared to stop security breaches with predictable & actionable information.

This article is focused on leveraging the big data approaches to manage the enterprise information security, improve response time and achieve greater efficiency in supporting day-to-day business operation activities.

Current state of big data analytics in managing information security

Almost every organizations uses security information and event management (SIEM) tool to monitor and analyze security events & incidents in the enterprises. Traditional SIEM solutions are kind of point solutions that offer the capability for the enterprises to collect, aggregate, store and generate alerts based on certain business rules and logics, and produce reports. SIEM solutions itself utilizes big data analytics with limited scope or capabilities, with keeping specific objectives in the mind like getting logs and events from predefined application or systems with some predefined formats while providing excellent results for the enterprises to an extent.

Now that the security threat landscape is increased in last 2-3 years by the emerging & evolving technologies like cloud, mobile and social networks etc, managing information security is becoming a challenging task for the enterprises, and it increases the risk to the enterprise security posture, due to the fact that when SIEM was developed, today’s complex digital environment was not envisioned at that time.

What are the challenges, why big data is needed to be adopted in managing enterprise security?

The SIEM technology provides quite good information for managing security. But if an endpoint system gets infected with a malware and it tries to talk to the server outside the network then there are many challenges in dealing with it and tracking down the endpoint. Further, at times making sure that the malware doesn’t remain undetected within the network even after taking down infected endpoint.

Take a typical scenario; a SOC team gets an alert indicting an infected endpoint in the network. SOC team start investigating it immediately, during the investigation, the key lies how soon the infected endpoint can be tracked down. In most of the cases, it’s a challenging task and not easy to get required information within a time frame say first 5-10 min of incident.

It’s because of the fact that there is no system or mechanism for SOC team to enter a system IP or system name and pull a detailed report showing connections to/from it in the network both internally or externally. In general, SOC team starts with digging into the logs at different places like web filtering gateway, firewall & IPS, endpoint protection solution or any other places to get the details or pinpoint the endpoint to get basic information like city, building, floor etc the endpoint is located.

This exercise take quite a long time, because all security systems are running in silos mode with no single point integrations, logs are not being forwarded to the SIEM tools or the SIEM tool doesn’t provide sufficient capacities to do that. Completing the investigation and reporting timeframe could take up to 12 to 24 hrs. The long investigation and response time can adversely impact the enterprise should a sophisticated malware attacks it. 85 percent of initial compromises occurred within minutes or less according to the Verizon 2013 Data Breach Investigation Report that stated “2012 reminded us that breaches are a multi-faced problem and any one dimensional attempt to describe them fails to adequately capture their complexity”

Big data analytics in improving response time & efficiency in managing information security

In every enterprises there has been massive increase in volume of data being generated from both web and from within the corporate operational systems. This requires the technologies to capture, store, manage and retrieve the data with exploding in volume, variety, velocity. In addition, lot of information is unstructured or semi-structured data which is more difficult to query then structured data (in case of traditional SIEM solutions).

To better manage the enterprise security, enterprises need to adopt the big data technology approaches that will help access, aggregate, analyze and visualize large volume of data regardless of its sources or formats. In addition, to quickly respond to a security incident, enterprises must connect and integrate information security events across the enterprise for quick accessibility and analysis with timely fashion. By leveraging the big data technologies, enterprises can leverage security event and incident information to better manage the enterprise security and find predictive pattern with deep insights.

The insights information from the analysis can help enterprises provide better attack detection, identification and intelligence, resulting improved response time with greater protection to potential targets whether the threat is inside or external.

Big data analytics in managing enterprise security further can help enterprises to identify threat trends and discover evolving attack patterns for example; Symantec produces its Internet Threat Report to present trend and evolutions in attacks, potential targets and malicious traffic.