Posted March 9, 2017
In our last blog, Chief Technology Officer Allan Thomson and Principal Data Scientist Dr. Jamison Day discussed Threat Intelligence: Evidence-based knowledge about an existing hazard designed to help organizations make inform decisions regarding their response to the threat. and security telemetry for assessing targeting attacks. Today, they delve into different threat correlation techniques security professionals can use to assess targeted attacks.
Threat correlation identifies new cyber threat insights by associating events from multiple data sources. There are many approaches to automating threat correlation between threat intelligence indicators and network flow data. Each has its benefits and drawbacks.
Manual Threat Correlation
Given the vast amount of known threat indicators and level of network activity today, automation has become a necessity. It’s often difficult and time consuming for human analysts to efficiently manage large amounts of granular data and a wide range of cognitive biases. Therefore, manual threat correlation is often too slow to keep up with the amount of data generated, results include a high number of false negatives and positives, and outputs are not always reproducible.
However, performing manual threat correlation processes will remain crucial. The human brain’s ability to leverage well-formed biases and perform higher-order reasoning is essential for assessing the validity and value being provided by whatever solutions your organization uses as well as building your cyber threat management team’s knowledge base. Thus, even when automated methods are employed, the final tier of analysis typically uses these human abilities for sense-making before any actions are taken.
The simplest form of automated threat correlation uses field comparison, in which a field entry from network activity data is matched against an identical field entry from known threat data and matches provide hits for likely suspicious activity. For example, when a threat feed identifies the IP address of a newly-discovered watering hole, previous and ongoing network flow data can be correlated via field comparison to see if any computers within the organization’s network have accessed that address. Offending systems can then be checked for any signs of infection.
Field comparison is a simple, fast, and scalable method for automating threat correlation. However, it is a fairly naïve approach that relies on identifying a single element from each data source that provides sufficient evidence for raising suspicion. Sophisticated attackers will likely avoid detection via this method.
Rules-based threat correlation allows cyber threat managers to define a set of criteria that, when seen together, will raise a red flag for further review. The combinations of criteria may include something as simple as an IP address range, port range, and UDP transport protocol used by a threat actor, or it may rely on a complex regular expressions and Boolean logic to allow for alternative attack vectors that are used in several variants of a Malware: Software that is intended to damage or disable computers and computer systems. family.
Identifying more complex attacks becomes possible with rules-based correlation, and it is quite possible to scale up checking a large number of rules within a distributed architecture. Certainly, creating and managing an expanding list of complex rule sets can become cumbersome, especially as many threats today tend to be slightly modified on each use.
Fuzzy matching is based on approximate similarities between elements from both data sets. Where rule-based approaches produce matches with true-or-false certainty, fuzzy matching generates probabilities of a match when elements from both data sets show only slight deviation from desired patterns.
A big strength of this approach is that it can help identify new attack tactics that are slightly modified from those that are already known. Unfortunately, fuzzy matching algorithms tend to either be computationally expensive or require explicit definition about the type of approximate results that are examined.
Machine learning algorithms such as neural nets, evolutionary algorithms, support vector machines, and Bayesian networks allow computers to learn without being explicitly programmed. Once trained with sample data, these algorithms provide the potential for identifying many known attacks as well as some previously unknown ones. Depending on how they are used, they can also continue to learn more from the real-world data they encounter.
Unfortunately, machine learning algorithms usually do not provide your cyber threat management team with any insight about why something has been deemed suspicious. Instead, analysts must scrutinize machine learning results for potential threats and figure out why they may have been singled out by the algorithm. This process can help enhance your analysts’ understanding, grow a more valuable threat knowledge base, and suggest refinements to the machine learning approaches used so they become even more valuable over time.
How Threat Actors Can Evade Detection via Threat Correlation
No cyber security approach is completely safe. However, investing in a more advanced threat correlation approach will increase the level of effort required for threat actors to evade detection.
Assessing Targeted Attacks
Regardless of the method used, correlation of threat intelligence and network activity data can help identify active attacks on your organization. Knowing which threats are active can help your cyber threat management team focus its efforts on shoring up the right defenses to improve your organization’s unique security context.
Information provided by threat intel feeds often includes some severity rating for each threat. It is advisable that any additions made to your organization’s threat knowledge base include similar ratings. As threat correlation matches any network flows to known threats, it can associate that information with the attack.
Internal network flow data also helps assess the value of various computing assets within your organization. Observing the amount and type of data flowing between computers in your organization can be used to assess the potential impact of an attack that targets that asset.
As attacks are identified, the criticality of each attack as well as the value of the targeted asset can be used to automate prioritization of threat incidents to guide your cyber threat management team’s response and mitigation efforts to generate the greatest value.
Next Step Recommendations for Your Organization
There are several recommended steps for harnessing threat correlation to automate targeted attack assessment:
- Determine which threat intelligence feeds are best for your organization. These feeds should be machine-readable so that as new data becomes available, it automatically flows into your organization’s threat knowledge base. Ensure that the feeds you select contain high quality data, provide a wide view of your cyber threat landscape, and have minimal overlap. You may consider subscribing to a curated composite feed that takes the hassle out of this effort for you.
- Integrate threat intelligence into your automated threat management. A single source of truth for all your organization’s known threats is desirable. However, realize that large organizations will have many departments that may require different views of the information it contains, including the ability to manage different severities for the same threats.
- Capture and analyze your network activity. Even without automated threat correlation, network activity data is extremely valuable for informing cyber threat management activities. Both packet and network flow data provide different insights about the threats impacting your organization. Critical decisions will include what data to capture, methods for analysis, and how long to store it (to allow for historical analysis).
- Automate correlation of network activity with threat intelligence. If your threat correlation needs are simple, you may be able to find an off-the-shelf solution that integrates well into your existing cyber threat management capabilities. Organizations seeking more tailored solutions may require extensive customization to successfully create an automated solution.
- Maximize impact with feedback loops that continuously improve your organization’s abilities. Whether you choose simple field comparison capabilities, rules for strict or fuzzy matching, or machine learning techniques, understand the effort required to effectively manage the approach so that new knowledge is fed back into your inputs and algorithms. This includes understanding what threats you are likely to miss so that your cyber threat management team can augment the automation with additional directed effort that covers the gaps.
It is always important to remember that, while threat correlation can enhance the efficiency of your cyber security efforts, no automated solution can replace your need for the right people working your cyber threat management.