The Cost Challenge of Modern Security Operations
Cost management with Microsoft Sentinel has been the biggest limitation preventing SOC teams from adopting Big Data and AI event detection in the enterprise. Many organisations have resorted to supplementing Sentinel with problematic solutions like the ELK stack to retain the Big Data capabilities that Sentinel was never designed to support.
The Cost Comparison
The graphic above shows the current representation of Pay-as-you-go Sentinel pricing for Analytics Logs (in Australian dollars) compared to the same data storage and ingest capabilities with Azure Data Explorer (ADX). It's still not an apples-with-apples comparison, as the ADX pricing includes:
- A Premium Event Hub Namespace
- Two ADX clusters in separate geographies to provide Active/Active resilience
- DR capabilities or an externalised cache of Security data in a dedicated Security tenant
Architectural Best Practices
For some years, the Azure Architecture Center has recommended using Azure Data Explorer as the long-term data store for Security data. Most of us are familiar with this design pattern:
Azure Data Explorer primarily ingests data from a series of Event Hubs, each typically dedicated to one type of log. Microsoft's recommendation was to forward Defender logs directly to Event Hubs and archive additional Sentinel data in real time to Event Hubs and ADX using the Log Analytics Export feature. This design pattern allows any other type of raw logs to be forwarded to a dedicated Event Hub for ingestion into Azure Data Explorer as well.
The Economics of Big Data Security
In a previous post, I discussed how (as a Rough Order of Magnitude) Azure Data Explorer can ingest and store 1000 GB a day for 30 months at a monthly cost of about $9,600. View the detailed analysis. This makes Big Data SIEM achievable for all enterprise SOCs while allowing ELK Stack and Splunk environments to be retired completely.
Sentinel Alerting on ADX Data
Azure Data Explorer is designed to be a Big Data Analytics tool, and it's phenomenal. It delivers exceptional query performance across large data sets, and Function Apps can continuously run KQL alert queries. When notable data is discovered, these events can be forwarded to Sentinel using Data Collection Rules.
The Step Change in SOC Capability
Azure Data Explorer allows SOC teams to collect all Security Events for long-term, online hunting and modelling. With total signal visibility from an enterprise environment, the primary role of Defenders evolves into a data engineering role with opportunities for reliable AI analysis of events.
Getting Started
Engineers keen to experiment with Azure Data Explorer can find an example project with documentation referenced at: Azure Data Explorer - Security Data Warehouse: A Reference Implementation View the implementation guide
- Log in to post comments