- The Role of Machine Learning in Intelligent Threat Response
- Bulletproof Cybersecurity, Starts with the Best Threat Intelligence
- Challenging the Economics of Cybersecurity with Threat Intelligence Sharing Programs: Part 1
- Challenging the Economics of Cybersecurity with Threat Intelligence Sharing Programs: Part 2
Posted September 13, 2018
We often use anomaly detection to identify potential malicious activity within our organizations, with the data of choice being network telemetry obtained from firewalls and intrusion detection systems, as well as system and application logs.
However, anomaly detection has much greater uses, such as identifying how the broader threat environment is changing. This activity provides threat analysts with insights about emerging threats in specific industries, intensively targeted phishing activity, and malware behaviors including their associated tactics, techniques, and procedures (TTPs). This enhanced situational awareness allows security teams to better-prepare for the threats they are most likely to encounter next.
Finding anomalies is not easy. The patterns that can be considered “normal” are different in every domain, most change over time, and inherent variations give rise to noise that can often obscure actual anomalies. Sometimes anomalies aren’t just single points of data, but they arise from several points of data interacting with each other. LookingGlass uses automated anomaly detection that simultaneously examines millions of interdependent contexts, identifying single points as well as groups of points that exhibit abnormal activity with statistical significance.
It is important to bear in mind, however, that all of the data we examine contains information about malicious activity. There are many types of threats that continue to be present over long periods of time or that pop up and subside with very limited impact. These are part of the “normal” threat landscape; however, our interest in this effort is to gain insight about what is novel within our large set of threat intelligence data. Therefore, the anomalies that we seek are those data points (and groups of data points) that do not fall into the category of “normal” malicious activity.
Identified anomalies have included:
- Trends of increasing (or decreasing) phishing activity, sometimes from a specific source
- Sudden spikes (or drop-offs) in infections from a certain type of malware
- General shifts in source or target geolocations
Enabling Success with Anomalies
Automating the identification of anomalies is only effective when it informs the cyber threat intelligence analysts that are impacting sIT security operational decisions. The features of an anomaly detection capability must ensure the results remain relevant for analysts.
We recommend that any anomaly detection capability not only score each anomaly based on its statistical significance, but also allow analysts to custom-tune which types of anomalous data are most important to each of them, prioritizing which anomalies float to the top of each of their lists for review. That way, for example, some analysts can focus on internet infrastructure anomalies while others focus on phishing activity or malware propagation.
Anomaly detection algorithms should define what is “normal,” adapting with threat intelligence data in two different ways. First, they should evolve slowly to incorporate the emerging trends so that analysts don’t keep seeing those same trends as anomalous over and over again. Eventually, a sustained trend is no longer an anomalous emerging trend. Second, they should detect when revolutionary changes have occurred and re-train entirely new models as new data beyond that detected revolution arrive. This way, analysts are alerted to the major shift, but the new normal is determined as soon as possible.
Finally, anomaly detection capabilities should provide analysts with the option to pivot through the anomaly results to explore root causes or broader extent of the anomalous activity. For example, when examining an anomalous spike in phishing volume at a customer, an analyst might want to pivot down into exactly which sources of phishing were of greatest impact during that spike or pivot up into whether other customers in that industry also experienced a spike in activity.
LookingGlass Threat Intelligence
Anomaly detection is only successful when paired with high-quality threat intelligence. As we’ve discussed before, not all threat intelligence is created equal. Data is not intelligence, and there are many different types of threat intelligence. For example, the threat intelligence LookingGlass uses contains significant historical intelligence context covering many years and many forms of malicious activities across diverse collections of networks. This includes:
- Threat Reports: Articles, strategic intelligence reports, threat summaries
- Threat Definitions: Names, definitions, technical indicators of compromise
- Threat Associations: Specific sightings of C2 servers, infrastructure, phishing domains, malware infections
- Internet Structure: ASN CIDR announcements, domain name records
- Internet Information: Ownership, geolocation, languages
While these feed sources are excellent for helping analysts develop a deep understanding of various threat domains, they are seldom applicable to automated anomaly detection due to the unstructured nature of the information they contain. Luckily, the rest of the threat intelligence feeds LookingGlass uses contain structured data that lends well towards automated anomaly detection.
If you would like to learn more about our anomaly detection work at LookingGlass, contact us.