From Chaos to Clarity: Leveraging Cyber Threat Intelligence from Heterogeneous Sources

Cyber threat intelligence (CTI) enhances cybersecurity by giving organizations the means to proactively identify and counter potential threats. In this dissertation, we develop efficient methods to leverage the CTI data in various scenarios, including effective detection of Advanced and Persistent Threats (APT), providing a comprehensive insight into the APT landscape, and attributing threats to respective APTs. We began by curating a comprehensive collection of CTI from a wide array of sources that present threat intelligence in various formats and structures. By developing several innovative methodologies, we refine this highly complex and heterogeneous data to extract threat intelligence that often can be mapped into low-level systems and network logs and can be utilized in various real-world scenarios. We use this data to underpin our analytics. This includes indicators of compromise (IOC) alongside more intricate threat intelligence representations that describe the threat, context, and behavioral patterns. In this dissertation, we introduce Extractor which distills system-level attack behaviors from threat reports. Extractor employs novel techniques to navigate the complexities of threat reports, effectively capturing and illustrating the causality, information flow, and chronological sequence of attack steps as provenance attack behavior graphs. These graphs are then leveraged by Extractor for threat detection in various real-world contexts. Furthermore, we introduce TIPCE, a framework aimed at providing a comprehensive analysis of threat intelligence sources. To support this, we have developed a unique dataset of Indicators of Compromise (IOCs) gleaned from threat reports, which we use to assess the various Threat Intelligence Sharing Platforms (TISP) in terms of coverage, overlap, and timeliness, among other metrics. Lastly, we introduce APTOracel which automatically attributes threats to respective APTs. APTOracl emulates a security expert’s intuitive approach to APT attribution by utilizing historical evidence that outlines the defining traits and characteristics of APTs. By applying dynamic field weighting methods, APTOracl associates threats with their corresponding low-level system behaviors and synthesizes this information across different levels of detail. The efficacy of the proposed methods is evaluated against real-world APT incidents recorded in threat reports, as well as through dynamic malware analyzers and sandboxes, and against APT scenarios designed for adversarial engagements. Our evaluations indicate that these approaches can effectively convert unstructured threat intelligence into a structured, machine-readable format, enhancing various threat detection operations with high accuracy and low rates of false positives. In conclusion, this dissertation shows that publicly available CTI, typically offered in an unstructured form, can be effectively employed in threat detection, APT attribution, and providing a detailed view of the APT landscape.