Azure Monitor planned Format Change

Hello World,

Using external log analytics or SIEM tool with Azure Monitor? Get ready to November 1^st 2018

On this page Microsoft announced that on Nov 1 it will change its diagnostic logs format in the storage accounts.

The change will take place across all subscriptions at once except Azure China, Azure Germany, and Azure Government clouds.

What will change:

JSON records will now be in JSON Lines format with new line separations and no commas between JSON records.

What it will impact:

• This change impacts the following data types:
  • Azure resource diagnostic logs(see list of resources here)
  • Azure resource metrics being exported by diagnostic settings
  • Azure Activity log data being exported by log profiles
• This change does not impact:
  • Network flow logs
  • Azure service logs not made available through Azure Monitor yet (for example, Azure App Service diagnostic logs, storage analytics logs)
  • Routing of Azure diagnostic logs and activity logs to other destinations (Event Hubs, Log Analytics)

Who Is impacted and need to make changes:

If you have custom tooling that ingests these log files for further processing like external log analytics or SIEM tool then you need to make them work with both formats

The old and the new:

{“time”: “2016-01-05T01:32:01.2691226Z”,”resourceId”: “/SUBSCRIPTIONS/361DA5D4-A47A-4C79-AFDD-XXXXXXXXXXXX/RESOURCEGROUPS/CONTOSOGROUP/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/CONTOSOKEYVAULT”,”operationName”: “VaultGet”,”operationVersion”: “2015-06-01″,”category”: “AuditEvent”,”resultType”: “Success”,”resultSignature”: “OK”,”resultDescription”: “”,”durationMs”: “78”,”callerIpAddress”: “104.40.82.76”,”correlationId”: “”,”identity”: {“claim”: {“http://schemas.microsoft.com/identity/claims/objectidentifier”: “d9da5048-2737-4770-bd64-XXXXXXXXXXXX”,”http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn”: “live.com#username@outlook.com”,”appid”: “1950a258-227b-4e31-a9cf-XXXXXXXXXXXX”}},”properties”: {“clientInfo”: “azure-resource-manager/2.0″,”requestUri”: “https://control-prod-wus.vaultcore.azure.net/subscriptions/361da5d4-a47a-4c79-afdd-XXXXXXXXXXXX/resourcegroups/contosoresourcegroup/providers/Microsoft.KeyVault/vaults/contosokeyvault?api-version=2015-06-01″,”id”: “https://contosokeyvault.vault.azure.net/”,”httpStatusCode”: 200}}

Good luck

I’m on my lab already working on the change for one client but as Microsoft recommends using Azure Event Hub, in that case any future change will make 0 change for me.