Posted May 5, 2016
In this blog series by CTO Allan Thomson, we are exploring the three fundamental challenges that users often have with leveraging a Threat Intelligence Platform (TIP). Last week we went over collection, aggregation, and distribution. Today’s blog will discuss the critical aspects of analysis and enrichment.
scoutPRIME® Distributed Architecture
ScoutVision showed us that to truly scale from smaller customer environments to large scale Internet intelligence data (hundreds of billions of observations), the product has to be capable of growing as both our customers and their data grow over time. The system also needs to be able to handle increasing historical context that will be used for search, analytics, and reporting. We solved this challenge by implementing changes at the core of scoutPRIME.
scoutPRIMEis a homogenous, distributed software architecture (database, message bus, as well as back end and front end software nodes) that runs within a cluster of server nodes. The number of nodes is dependent on the scale of data, performance, and functions the system needs to handle. Each node runs an identical software stack making horizontal growth of the system easier. As customer data requirements change, we simply add more nodes to the cluster and the system automatically re-provisions itself to load balance all data and processing across the additional nodes.
The key challenge we solved was how to distribute the data ingestion, threat analysis, threat scoring, and user operations such as search, updates, etc. across N nodes without requiring a duplicate data load on each node. We solved many of these technical challenges based on our experience building ScoutVision. With the potential to load several hundred billion threat data observations per day, the ability to distribute that across scoutPRIME was vital to ensure we could ingest the data within an acceptable time. Secondly, scoring on threat data that is fundamentally a connected graph requires sophisticated algorithms to divide the Internet into relevant chunks, and a back end communication system to allow distributed processors to communicate and share data results between themselves in real-time.
Threat Data Immutability
With ScoutVision, significant time was spent updating threat data in a database over the lifetime of the system. Historical tables had to be maintained so that as new data was ingested, the previous context was shifted to the historical tables. This often meant data tables would be locked for database writes by multiple parallel processes attempting writes.
Threat context over time is extremely relevant to analysis and scoring, so ensuring that you have an efficient mechanism to analyze both current and historical threat data and their scores requires a system that keeps all data separate. We solved the challenge through immutable (unchangeable) data storage, and built a highly scalable distributed system that writes without blocking for reads while providing all historical context necessary for analysis and user reports.
Our immutable data storage mechanisms also improved scoutPRIME’s search function – a primary operation performed on any TIP. Most searches are combinational, where an analyst may be searching across disparate types based on time. scoutPRIME’s highly scalable data write and read requirements leveraging immutable data storage drastically improved the search functionality. This is key because threat teams rely on search to find relevant data quickly.
Threat Analytics & Scoring
At its core, scoutPRIMEhas a sophisticated threat scoring processing engine, which provides customers with relevant, actionable threat intelligence. The engine itself is made up of multiple sub-engines distributed across the scoutPRIME server cluster. Each sub-engine is automatically provisioned for a section of the global Internet and any customer relevant networks.
The scoring system was introduced in our blog, “How You Stay Confident in a World Full of Cyber Threats” for background.
The scoring engine is similar to a brain that continuously calculates threat scores based on any new ingested data. The analogy to the human stops there. A human analyst can at best consume and consider several thousand threat Observables a day. However, scoutPRIME consumes literally hundreds of billions of observations every 24 hours, as well as any human entered threat data. The scoring engine not only keeps up with this significant amount of data, but also allows scoring to drive relevant threat intel for human analysts. It calculates scores in real-time and keeps historical scoring as part of the analysis.
Let’s discuss the basic scoring mechanism.
scoutPRIME’s most basic unit of threat analysis is an Observable. An Observable is a broad definition of any threat data intel that is associated with another element in the system. An Observable’s score is based on its source, criticality (severity), and one or more classifications.
All data sources in scoutPRIME are associated with the provider of the data. Some may be open-source, some may be commercial, and some may be LookingGlass-provided.
We learned from our customers and scoutPRIME that not all sources are considered equally reliable. This means that a source’s score should be associated with all data from that source. Sometimes, a single source may have multiple data feeds that have better or worse value and assessment, so we created a mechanism to score individual data feeds and their rating for use in the scoring engine.
An Observable’s criticality is the component of an observable’s score based on the severity of the specific Observable. Severity is partly subjective, based on the source’s assessment, but can be tuned. For example, we can change the severity of a threat based on its relevance to the organization.
For example, a DSHIELD Suspicious Domain observable may not be that significant to an organization that is focused on actors.
Whereas information Flashpoint regarding a State Actor may be considered more serious.
Classifications are a way for our threat analysis teams and our customers to group observables together that have a similar assessment. All Observables have at least one coarse grained-classification and zero or more fine-grained classifications. Classifications are a way for our threat analysis teams and our customers to group observables together that have a similar assessment.
A simple example is all command & control (C&C) observables are rated based on a C&C classification. This means that all observables – from the 100s of data feeds – that have C&C will have a similar score based on the grouping within the C&C classification.
The Observable Score is a combination of all three aspects (source, classification, criticality). Organizations can easily determine how source, criticality, and classifications influence the Observable score by visual inspection as individual aspects are changed. As organizations change each aspect they influence more or less Observables that are within that category. For example, if they change the source score, then all observables from that source will be influenced somewhat by that score change. scoutPRIME automatically recalculates the score for that source’s score change, updating all Observables and all elements associated with that Observable. A single change allows organizations to update scoring and relevancy across hundreds of millions of elements and Observables.
An element is any networked asset. It could be an application (malware or not), a device (IP or Domain) or a user.
All elements have scores. If there are no Observables associated with a particular element, then that element receives a default assessment score. As Observables are associated with elements, those elements’ scores are calculated based on the combination of Observables with which they are directly associated.
Networked elements do not exist in isolation from other elements. Therefore, the scoring engine takes into account the combination of elements and their scores and provides scores on relationships and containers. For example, a CIDR may have multiple IP elements that have a score. That CIDR would assume a score based on the composite elements. A domain name that has multiple IPs associated with it would receive a score based on the multiple IPs, as well as any directly associated Observables.
scoutPRIME introduced a user-definable object that allows customers (and systems) to define and score a collection of elements. One example might be where a customer has a set of disparate network domain names that represent their company, but there’s no direct relationship between those domain names. The customer could create a collection for all domain names within their organization and be able to see a score associated across their entire organization.
Time Impact on Scoring
The timeliness of threat intelligence largely influences how relevant that threat is to an organization.
For example, observing a Command & Control (C&C) server actively engaging with your network within the last hour is clearly more relevant than similar behavior observed a week or month ago. Many bad actors rely on new or infected domain names to quickly infect a target. Once a target is impacted, the bad actor may not use that malicious site again, rendering any threat intelligence associated with this stale site somewhat irrelevant.
Similarly, a domain name that has acted as a C&C server multiple times in the past but is not currently exhibiting that behavior due to cleanup, may still influence that server’s score. This will influence your analysis if internal assets are communicating with that external site.
scoutPRIME considers the time that threat intelligence is observed and distributed as critical factors in scoring relevancy of threats. The scoring engine automatically scores Observables and their associated network elements based on when the observations occurred and updates those scores as time decay occurs without any new observations to refresh the score.
Not all score decay occurs the same, and customers can fine tune the decay depending on source, classification, and Observable type.
Threat Scoring Visibility & Customization
Because scoutPRIME is customizable based on your organization, we’ve provided a view that shows the basis for scoutPRIME scores. You can visualize the composite scores that make up each Observable and element. So, if a customer has overridden the scoring engine, they can see the override clearly. Customers can also change the composite score and immediately see the impact of those changes in their view.
For each observable that supports the element score, the customer can visualize the composite scores that make up each observable score.
In our next post, we will describe in more detail how scoutPRIME allows our customers to change scoring without necessarily impacting other users of the same system.