AZ-104 · Skill Domain

Monitoring & Security

Log Analytics workspaces, KQL queries, metric and log-based alert rules, VM backup with Recovery Services vault, Azure Site Recovery concepts, and zero-credential Key Vault architecture with full audit logging — observability and security across the stack.

3Projects
0Stored credentials
3Restore paths
P9–P14Phase 3 + 4
01

Projects

09
Log Analytics · KQL · Alerts · AMA + DCR
Azure Monitor & Log Analytics
LAW · Azure Monitor Agent · Data Collection Rules · metric alerts · log-based alerts · KQL queries

Built the observability layer for the lab environment: a Log Analytics workspace receiving performance and event data from both VMs via Azure Monitor Agent and Data Collection Rules, metric alert rules firing on CPU threshold breaches, a log-based heartbeat alert detecting VM availability, and an Action Group delivering email notifications. Verified end-to-end by stress-testing CPU on the Linux VM and confirming the alert fired and the email arrived.

The legacy Log Analytics agent (MMA) and Azure Diagnostics extension (LAD) were deprecated in March 2026. The current pattern is Azure Monitor Agent with Data Collection Rules. DCRs decouple the definition of what data to collect from the agent — the same DCR can be associated with multiple VMs, and data collection can be changed without touching the agent.

KQL — VM heartbeat availability
Heartbeat
| summarize LastSeen=max(TimeGenerated) by Computer
| extend MinutesSince=datetime_diff('minute',now(),LastSeen)
| order by MinutesSince asc
KQL — Average CPU over time
Perf
| where CounterName == "% Processor Time"
| where ObjectName == "Processor"
| where InstanceName == "_Total"
| summarize AvgCPU=avg(CounterValue)
    by Computer, bin(TimeGenerated, 5m)
| render timechart
View architecture diagram — project9-architecture.svg
  • AMA over legacy agents — MMA and LAD are deprecated. Azure Monitor Agent is the current correct deployment pattern. Remove legacy extensions before installing AMA to avoid conflicts — both agents writing to the same workspace produce duplicate records.
  • DCR association has no -g flagaz monitor data-collection rule association create does not accept -g. The resource group is implicit in the --resource ARM ID. Use the full resource ID.
  • Metric alert vs log alert — Metric alerts evaluate a numeric measurement against a threshold in near-real-time. Log alerts run a KQL query on a schedule. Use metric alerts for threshold breaches on known counters; use log alerts for absence detection (heartbeat missing) or complex multi-resource patterns.
  • Action Group is reusable — Defined once, referenced by multiple alert rules. Adding a new notification target means updating the Action Group, not each individual alert rule.
⚠ Gotcha — diagnostic logs have 5–15 min lag

After creating a diagnostic setting, logs take 5–15 minutes to appear in Log Analytics. Trigger a read operation first, wait, then query. Running the KQL query immediately returns zero results and looks like a misconfiguration — it isn't.

Log Analytics Azure Monitor Agent Data Collection Rules Metric Alerts Log Alerts Action Groups KQL
Alert fired
CPU alert fired
Alert detail
Alert — additional detail
KQL heartbeat query
KQL — VM heartbeat query
14
Key Vault · Managed Identity · Zero-Credential · Audit Logs
Key Vault & Managed Identity
Zero-credential architecture · RBAC model · KV references · diagnostic logs · KQL audit trail

Deployed a zero-credential architecture connecting App Service to Key Vault using system-assigned managed identity. No passwords, no API keys, no service principal secrets anywhere in code, configuration files, deployment scripts, or git history. The App Service proves its identity to Microsoft Entra ID, receives a scoped token, and uses that token to resolve Key Vault references at runtime — surfacing secrets as ordinary environment variables to the application without any Key Vault dependency in application code. Every secret access is logged to Log Analytics as an AuditEvent and queryable via KQL.

The traditional approach stores a credential somewhere the app can read it. Every step is a failure point: committed to git, stored in plaintext, forgotten after rotation. Managed identity eliminates the credential entirely. Azure manages the identity lifecycle — when the App Service is deleted, the identity is deleted with it. Nothing to rotate, nothing to leak, nothing to accidentally expose.

KQL — Key Vault secret access audit
AzureDiagnostics
| where ResourceType == "VAULTS"
| where OperationName == "SecretGet"
| project TimeGenerated, CallerIPAddress,
          OperationName, ResultType
| order by TimeGenerated desc
View architecture diagram — project14-architecture.svg
  • RBAC model, never Access Policies — Access Policies grant access at vault level — all or nothing. RBAC grants access at any scope, integrates with standard Azure IAM tooling, and produces an audit trail consistent with every other RBAC operation in the subscription. Access Policies are deprecated for new vaults.
  • Key Vault Secrets User — least privilege — The App Service identity has read-only access to secret values only. It cannot list secrets, create, modify, or delete. The application gets exactly what it needs and no more.
  • KV references over SDK retrieval — Configuring secrets as App Service references (@Microsoft.KeyVault(SecretUri=...)) means the application has no Key Vault dependency in its code. The runtime resolves the reference before surfacing it as an environment variable. Infrastructure manages secrets; application consumes environment variables.
  • RBAC scope — Key Vault resource, not resource group — Scoping to the resource group grants the same role against every resource in the group. Scope every RBAC assignment as narrowly as possible.
⚠ Gotcha — KV reference strings corrupt in local Bash

The @ symbol and parentheses in @Microsoft.KeyVault(...) are mishandled by Git Bash and WSL even inside quotes. All az webapp config appsettings set commands containing KV references must run from Azure Cloud Shell.

App Service masks KV reference values by design

After setting KV references via CLI, output shows "value": null. This is expected — App Service hides the reference string for security. Verify in the portal under Settings → Environment variables: correctly resolved references show a green checkmark and "Key vault" in the Source column.

Key Vault Managed Identity RBAC KV References Diagnostic Settings Log Analytics KQL Entra ID
KV references in App Service
KV references — App Service
KV audit logs KQL
KQL — KV audit logs
02

Lessons Learned

Managed identity eliminates the rotation problem
Every stored credential has a rotation schedule, and every rotation is a change event that can break dependent applications. Managed identity has no credential to rotate. The identity lifecycle is tied to the resource — deleted with it, never orphaned in Entra ID. This removes an entire class of security risk, not just a maintenance burden.
Legacy agents are gone — AMA + DCR is the pattern
The Log Analytics agent (MMA) and Azure Diagnostics extension (LAD) were deprecated in March 2026. Any guide referencing these agents describes a deprecated workflow. Azure Monitor Agent with Data Collection Rules is the current correct approach. DCRs decouple data collection definition from the agent, making it possible to change what's collected without touching the VM.
RPO and RTO are negotiated, not discovered
RPO and RTO are business requirements that determine infrastructure choices. Daily backup gives 24-hour RPO. Hot standby with ASR gives near-zero RTO. The cost difference is significant. Every backup architecture is a deliberate tradeoff between data loss tolerance, recovery time tolerance, and budget — not a default setting to accept.
Diagnostic log categories are resource-type specific
AuditEvent is the Key Vault diagnostic category. App Service exposes AppServiceHTTPLogs, AppServiceConsoleLogs, and others. Never guess category names — query them: az monitor diagnostic-settings categories list --resource $RESOURCE_ID returns exactly what is valid for that specific resource type.
Soft delete is the last line of defense
Ransomware attacks frequently target backups before encrypting the primary workload. Soft delete on Recovery Services vault retains deleted backup data for 14 days regardless of who deleted it. Purge protection on Key Vault prevents vault deletion during the retention period. Both should be enabled on any production resource from day one.
KQL is the common language across Azure monitoring
The same KQL syntax queries VM performance data, Key Vault audit logs, App Service HTTP logs, and diagnostic data from every Azure service that integrates with Log Analytics. The heartbeat query, the CPU timechart, and the Key Vault SecretGet audit query all follow identical patterns. Proficiency in KQL compounds across every new service you monitor.