Threat Detection using Information Flow Analysis on Kernel Audit Logs

Kernel audit logs are a rich source of information containing the history of causal dependencies and information flows among system entities in a host system. The mainstream use of kernel audit logs is for forensic tasks to investigate cyberattacks retrospectively. In this dissertation, we develop efficient methods that make use of kernel audit logs for complex real-time security tasks, such as Advanced and Persistent Threat (APT) detection, attack scenario reconstruction, and cyber threat-hunting. To this end, we first process kernel audit logs into a platform-neutral provenance graph stored in the main memory and use it as a foundation to run various analytics. For APT detection, we develop techniques to produce a detection signal indicating the presence of a coordinated set of suspicious activities. For real-time attack scenario reconstruction, we develop an approach that utilizes information flow policies to identify entities and events that are involved in cyberattacks. For cyber threat-hunting, we develop an inexact graph pattern matching approach to align a query graph extracted from cyber threat intelligence to a provenance graph constructed out of kernel audit logs. The efficacy of the proposed methods is evaluated against real-world APT scenarios designed for adversarial engagements. These experiments contain millions of records and collectively involve months of audit log collection activity from a variety of hosts that run OS platforms such as Linux, FreeBSD, and Windows. The results indicate that the proposed methods are capable of efficiently searching these audit logs and pinpoint threats in real-time with high precision and low false alarm rate. Besides, these methods effectively produce summaries of attack campaigns that assist investigators in cyber response operations. In summary, this dissertation demonstrates that the low-level causal information inferred from kernel audit logs could be utilized to achieve robust and reliable threat detection methods that efficiently pinpoint threats and reveal the high-level picture of attacks by producing compact visual graphs of attack steps.