Posted April 28, 2016
This blog series from CTO Allan Thomson will explore the three fundamental challenges LookingGlass sees organizations facing when it comes to leveraging a Threat Intelligence Platform (TIP). Today’s blog reviews the first issue: Collection, Aggregation, and Distribution.
As a threat centric company, LookingGlass understands that having a comprehensive Threat Intelligence Platform (TIP) is a critical component in managing threat for organizations. Our experience with developing and implementing these platforms, such as with our proven ScoutVision solution, has taught us that not all TIPs are created equal. For many of our discerning customers, ScoutVision has shown them how critical it is for their TIP to support large scale data ingest, correlation, analysis, data visualization, and data sharing.
As more people use the Internet for more things, threat intelligence data will continue to grow in coverage, volume, complexity, and through its impact inside the enterprise. LookingGlass considers it vital that TIPs continue to support security organizations with the increasing scale, context, and performance that future threat intelligence demands.
With that firmly in mind, we developed ScoutPrime, our next-generation threat intelligence management platform.
ScoutPrime was created to address the three fundamental challenges that we see most security organizations facing when it comes to leveraging a TIP in their environments:
- Collection, Aggregation, and Distribution of Network Threat Data
- This is the basis for any downstream mechanisms that allow customers to collect relevant data, casting the net wide enough to start the process of turning data into intelligence
- Machine-Readable Threat Intelligence (MRTI) products
- Analysis and Enrichment
- This is where data can be turned into threat intelligence, allowing security teams (threat analysis, security operations, compliance, etc.) to collaborate and share perspectives on actionable threat intelligence
- ScoutPrime details
- Threat Defense Operations
- This is where actionable intelligence drives the network security infrastructure to make real-time decisions that protect customer environments or enable further inspection of ongoing threats
- Dynamic Threat Defense solutions
Collection, Aggregation, and Distribution
A typical cycle for organizations considering or leveraging Threat Intelligence (TI) data to improve their security posture can be broken into three phases.
- Determine a business security need that potentially can be enhanced by integrating TI in that process or offering
- Evaluate one or more TI feeds including open source and commercial providers
- Determine which data feed(s) provides relevant and valuable data to the organization
- Determine what compute, storage, and network infrastructure must exist to collect chosen data feed(s); how long data must be stored for; what normalization of the data must take place to allow ingestion with other feeds; what access is necessary; and what enrichment and classification of raw data can take place as part of the collection process
- Build an automated collection process that gathers those feeds continuously, such that the feed is fed into the other security infrastructure
- Deploy the process in a security operations environment
- Monitor the collection process such that data is continuously gathered, monitored for data format changes on vendor feeds, and provides reports on data feed collection
- Repeat from Evaluation as new feeds or existing feeds change
For many organizations, the investment in both time and resources to support such a cycle is not a priority. LookingGlass ScoutPrime provides customers with a solution to significantly reduce, simplify, and eliminate the three phases, thus providing these organizations with a solution to their business needs.
LookingGlass has evaluated, and continues to evaluate, a large number of both open source and commercial threat intelligence feeds. ScoutPrime integrates over 140 data feeds including many from our own LookingGlass Cyveillance Machine-Readable Threat Intelligence (MRTI). The LookingGlass Threat Intelligence Analysis team evaluates TI feeds for coverage, efficacy, and uniqueness. This analysis has helped LookingGlass provide one of the most comprehensive data sets in the industry, and available only to our customers via their LookingGlass Threat Intelligence Platform.
A common part of the system architecture that supports ScoutPrime is the backend data collection system, known as the Core Intelligence Processor (CIP).
The CIP plays a central role in the system architecture, as it is the data collection, aggregation, and initial enrichment system supporting all LookingGlass customer Threat Intelligence Platforms.
High Level Threat Data Flow
For all TI feeds supported in the platform, LookingGlass’ development teams have developed a complete Extract-Transform-Load (ETL) process to gather, normalize, and aggregate those data feeds with all other network intelligence and threat intelligence data.
The CIP’s primary function is to gather and distribute all publicly accessible threat data feeds, as well as LookingGlass’ MRTI portfolio.
A key value-add for our customers is the enrichment and classification of all TI feeds the CIP system gathers. Our Threat Intelligence Analysis team analyzes and scores all incoming raw data incorporating threat specific information, and provides a common set of classification and scoring information.
Our CIP system is continuously gathering TI feeds at different intervals dependent on the source provider, 24 hours a day, 7 days a week. Our operations team monitors the CIP system for uptime performance, ensuring our ScoutPrime customers have a reliable threat intelligence data set.
ScoutPrime can be deployed both in a hosted data center and on customer premises, enabling organizations to choose which option matches their requirements for both flexibility and security.
The CIP runs 24×7, gathering data feeds, classifying threat data, and providing initial threat scoring across all data feeds supporting all customer deployed systems. In addition, certain data feeds may be cross-correlated across other LookingGlass sensor networks to corroborate and gather further metadata associated with the initial findings from the other feeds.
In the next blog, we will dig deeper into the ScoutPrime platform itself.