Figure out how to route logs from Azure Native ISV Service

[zmoog]

Context

The Azure Native ISV Service forwards logs from Azure to Elastic Cloud.

Currently, the log forwarder in Azure applies the following routing rules:

Log category	Data stream
Activity logs (for the subscription only)	logs-azure.activitylogs-default
Sign-in logs	logs-azure.signinlogs-default
Audit logs	logs-azure.auditlogs-default
Spring cloud logs	logs-azure.platformlogs-default
Resource logs	logs-azure.platformlogs-default
(None of the above)	logs-azure.platformlogs-default

Since Azure Logs and other Azure-focused integrations now support more log categories, we can leverage the reroute processor to send these log categories to the most appropriate data stream.

Goals

Add a custom pipeline to route incoming logs to the target data stream. Fallback to the logs-azure.platformlogs-default data stream.

[zmoog]

Setup

I created a diagnostic setting to route blob storage logs from Azure to an Elastic partner solution.

Logs coming from Azure Native ISV Service include an additional field named azure_log_forwarder.category we can use for routing.

[zmoog]

Routing

Using the Dev Tools, let's check if we already have a custom ingest pipeline for the platform logs:

GET _ingest/pipeline/logs-azure.platformlogs@custom
{}

If you get anything other than {} it means you already have a custom pipeline and you have to add the following change instead of replacing it.

Since our logs-azure.platformlogs@custom pipeline is empty, we can replace it:

PUT _ingest/pipeline/logs-azure.platformlogs@custom
{
  "processors": [
    {
      "reroute": {
        "dataset": [
          "azurex.storage.blob"
        ],
        "if": "ctx.azure_log_forwarder?.category == \"StorageWrite\" || ctx.azure_log_forwarder?.category == \"StorageRead\"",
        "description": "storage-blob-logs"
      }
    }
  ]
}

Here's the result before and after applying the custom pipeline change.

This mechanism allows users to route log categories to the new data streams from Elastic beyond those currently supported by the log forwarder in Azure.

In this example, I used log categories that the Azure Logs integration currently does not support, like StorageWrite and StorageRead to show that users can build their own index template to fit their needs and route log categories to them.

Note on the dataset: I recommend users avoid using datasets like azure.<service> (e.g., azure.blobstorage) to avoid conflicts if Elastic adds support for this service in the future.