Laurie Rhodes' Info

A frustration in dealing with table schemas for Log Analytics and Azure Data Explorer is inaccuracies with Microsoft's published documentation for the schema.  The schema returned by using Microsoft's recommended method  (https://learn.microsoft.com/en-us/rest/api/loganalytics/schema/get?view=rest-loganalytics-2023-09-01&tabs=HTTP) can mangle data types.  This can lead to unexpected difficulties when working with data in a large scale. 

 The following script provides an example of using PowerShell to directly query Azure Data Explorer Records using REST.

Prerequisites

You will also need to ensure that the Application you use is granted the Database Viewer role permission.   This role can be added through the portal on the database Overview -> Permissions -> Add.   Note that ADX does not support table level viewer permissions.  

The following script provides an example of directly writing to an Azure Data Explorer table using Powershell.  For resiliency, the preference for ADX data ingest remains a data connection against an Event Hub (Kafka) but there can be situations where ad-hoc writing is needed.

Prerequisites

Streaming ingestion must be enabled on the cluster level of ADX.  This may be done through the portal under Settings -> Configurations.

This example of using REST to populate Syslog is based off a Microsoft guide for using Logstash:
https://learn.microsoft.com/en-us/azure/sentinel/connect-logstash-data-connection-rules#create-a-sample-file

The following is an rsyslog template for parsing Common Event Format messages and writing output to a dedicated log file.  The created log file can be transformed with the Azure AMA agent (JSON format) for ingestion into Log Analytics.

For Microsoft Sentinel a 'syslog forwarder' that acts as a centralisation point for linux system and the Azure Monitor Agent (AMA) forwards messages received to a designated Log Analytics Workspace.  AMA provides the ability to filter logs using KQL queries at source, protentially reducing cost for the eingestion of a large amount of noise.

AMA does have a catch that's in the fine-print of its billing:

https://azure.microsoft.com/en-au/pricing/details/monitor/

Microsoft's strategy for allowing integration with security entities and incidents is through the use of Playbooks (Logic Apps).  Any engineers who have been involved in complex automation will prefer to script instead of using workflows.  The only form of automation avalable for use within the console of Sentinel are Playbooks.

There can be a number of reasons for wanting to backup Azure (or Office 365) to GitHub.  As an increasing number of SaaS services (like Microsoft Sentinel) are designed for being configured and deploying Azure services through the console, traditional CI/CD code promotion doesn't work.

For some years I've been backing up my Azure subscription to Github using automated workflows.  It ensures that I can compare changes in my subscription over time and by using MarkDown I can look through backups to reference previous versions of KQL queries.

At times it's necessary to be able to get tokens for different Microsoft cloud platforms without having access to Developer kits or .dll modules.  This function shows how different tokens can be gained using passwords, interactive OICD login, passwords etc.

It's one of my best examples of how agile code development needs refactoring after years of ongoing updates.  It will produce a hashtable that I can use in REST calls to the various Microsoft portals like Azure or Office 365.

This example brings together various examples I've used on this site to show how Incidents from Microsoft Sentinel can be queried using PowerShell.  References to pages that demonstrate the utilised code are at the bottom of this article.

A core example of the code is below