Have you ever wondered...
-
...how you could get data from a web service like the Office 365 Reporting Web Service or the Microsoft Graph reports Api using PowerShell?
-
...how you could store that data to investigate it later or simple archiving purposes?
-
...how you could automate the entire process, so that you don't have to get the data manually every time new data is generated?
-
...how you could investigate the collected data?
If yes, CAML is here to help you!
This repository demonstrates an end-to-end solution for gathering log data from an web service or Api and storing it in a Log Analytics workspace using PowerShell only. In this tutorial, you will request message trace data and message trace detail data from the Office 365 Reporting web service, as well as Microsoft Teams device usage reports and Microsoft Teams user activity reports and store it in a Azure Log Analytics workspace.
This is just an example of how an web service or API like the Office 365 Reporting web service or Microsoft Graph reportings api can be triggered and the received data redirected. You can customize the functions in this repository to receive data from any other API with ease. If you want to learn more about each file or you want to further customize them, please have a look at the descriptions, readme.md files and help-blocks of each folder or script-file. If you encounter any problems, want to add a new feature or want to improve the documentation, please open an issue in this repository.
To learn more about the entire project, check out the .md files in the docs folder, as well as the .md files in each subfolder of this project (if created). The .md files outside the docs folder will explain the content of it's own folder in detail.
IMPORTANT NOTE before we get started: Please try the solution in a test/demo environment before using it in production.
If you want to use this to collect ALL inbound and outbound emails of your environment, please consider the limitations of the Office 365 reporting web service and also read the uncontrollable events.
The CustomAzureMonitorLog powershell module must be used on a Windows machine (MacOS and Linux support coming soon!). You can use the module on your local machine interactively or in an Azure Function automatically. If you want to use it in an Azure Function, you have to deploy the solution to your Azure tenant.
In both cases you need at least the following:
- PowerShell 7.3 or later
- PowerShell Module Az.Accounts
- PowerShell Module Azure.Function.Tools
Further pre-requisites are described in the related deployment description.
There are several Exchange Online reports, which can be requested via the Office 365 Reporting web services. Learn more about the available Exchange reports of the Office 365 Reporting web service at Exchange reports available in Office 365 Reporting web service
Additionally, there are lots of Microsoft 365 usage reports, which can be requested via the Microsoft Graph reports API. Learn more about the available Microsoft 365 usage reports of the Microsoft Graph reports API at Working with Microsoft 365 usage reports in Microsoft Graph
As described in the introduction, you can easily add new functionalities to get data from any other web service or API. You could, for example, utilize Microsoft 365 Defender APIs or any custom API to collect data and write it to the Log Analytics workspace. In those cases, remember to add the necesarry permissions to your Azure App registration, if needed.
To customize the solution, check out the customizeFunctions documentation.
To learn what you can do with your collected data inside the Log Analytics Workspace, check out the investigateData documentation.
You can find more information about the manual removal process in the removeSolution documentation.
See knownIssues for known issues and workarounds. If you encounter any other issues, please open an issue in this repository. Thanks!
- Friedrich Weinmann - thanks for supporting me with your knowledge and your PowerShell modules (e.g. Azure.Function.Tools and PSModuleDevelopment)!
- Vini Costa - thanks for contributing to this project and the documentation!
- Thomas Meyer - thanks for contributing to this project and the documentation!
Modern authentication (OAuth) support for the Reporting Web Service in Office 365
Exchange reports available in Office 365 Reporting web service
MessageTrace report documentation
Introduction to Azure Functions
Timer trigger for Azure Functions
Azure Monitor REST API reference
Tutorial: Send data to Azure Monitor Logs using REST API (Azure portal)
Log Analytics REST API Reference