Azure Monitor alerts for AKS: the 03:47 page with no runbook, the SLO dashboards we built, and the routing that wakes the right engineer within 60 seconds

At 03:47 on a Wednesday, the on-call engineer got paged for a memory alert on a node pool called npool-batch-v1. The node pool had been decommissioned six weeks earlier as part of a migration to spot instances. The alert rule had not been deleted. The node pool was gone, the metric had no data, and Azure Monitor fired the alert anyway because "no data" evaluates as a breach on a threshold alert unless you explicitly configure it otherwise.

That page was the fourth false positive that month. The postmortem produced an audit: how many alert rules do we actually have, how many are attached to things that still exist, and how many have a runbook link and a named owner?

The answers were 94 rules, 37 attached to decommissioned or renamed resources, 61% with no runbook link, and 100% routing to a single shared ops email that nobody read proactively. The audit took four hours and the results were worse than expected for infrastructure we thought was well-monitored.

This is the rebuild: 94 rules reduced to 31, SLO-based alert conditions replacing raw threshold alerts for the core signals, per-team routing through Action Groups, and a weekly Azure DevOps pipeline that checks alert rule health and flags orphans before they page anyone.

Why raw threshold alerts fail at scale

The 03:47 page was not an anomaly. Raw threshold alerts have three failure modes that compound as a cluster grows:

First, they alert on symptoms rather than user impact. A node at 85% memory is not an incident. A deployment that cannot schedule because every node is above 85% memory is an incident. The threshold alert fires in both cases.

Second, they do not age gracefully. When a node pool is decommissioned, the alert rule stays. When a deployment is renamed, the alert rule targets the old name. The rule appears healthy in the portal because it has no error state; it simply fires on nothing, or fires on data that no longer represents what you think it does.

Third, routing is usually an afterthought. One Action Group, one email, one Slack channel. Every alert goes to the same place regardless of which team owns the affected workload or what severity the condition represents.

SLO-based alerting fixes the first problem. Alert hygiene fixes the second. Per-team Action Groups fix the third.

The SLO model

Rather than alerting on individual metrics, we define three SLOs per service and alert when the error budget burn rate exceeds a threshold over a rolling window.

The three SLOs for an AKS-hosted service:

Availability: percentage of requests returning 2xx or 3xx over a 28-day window. Target: 99.5%.

Latency: percentage of requests completing under 500ms over a 28-day window. Target: 95%.

Error rate: percentage of requests returning 5xx over a 28-day window. Target below 0.5%.

Burn rate alerting uses two windows to distinguish a fast burn (severe, page now) from a slow burn (concerning, ticket):

// Fast burn: p95 request duration SLO — 1-hour window
let slo_threshold_ms = 500.0;
let window = 1h;
let burn_rate_threshold = 14.4; // 1h window, 99.5% target
requests
| where timestamp > ago(window)
| summarize
    total = count(),
    slow = countif(duration > slo_threshold_ms)
| extend error_rate = toreal(slow) / toreal(total)
| extend burn_rate = error_rate / (1 - 0.995)
| where burn_rate > burn_rate_threshold

// Slow burn: 6-hour window, lower threshold
let slo_threshold_ms = 500.0;
let window = 6h;
let burn_rate_threshold = 6.0;
requests
| where timestamp > ago(window)
| summarize
    total = count(),
    slow = countif(duration > slo_threshold_ms)
| extend error_rate = toreal(slow) / toreal(total)
| extend burn_rate = error_rate / (1 - 0.995)
| where burn_rate > burn_rate_threshold

Fast burn pages immediately. Slow burn creates a ticket and sends to Slack without waking anyone. The distinction matters: a fast burn means you are spending error budget at 14x the sustainable rate and will exhaust it in under two hours. A slow burn means you have time to investigate in the morning.

The alert rule structure

Each alert rule carries four required tags enforced by an Azure Policy:

{
  "owner-team": "payments",
  "service": "checkout-api",
  "runbook-url": "https://wiki.internal/runbooks/checkout-api-latency",
  "severity": "sev1"
}

The policy runs in audit mode and posts violations to a Log Analytics table. Any rule missing a required tag appears in the weekly hygiene report without blocking deployment.

Creating a rule with the required tags via Bicep:

resource sloLatencyAlert 'Microsoft.Insights/scheduledQueryRules@2023-03-15-preview' = {
  name: 'slo-latency-fast-burn-checkout-api'
  location: resourceGroup().location
  tags: {
    'owner-team': 'payments'
    'service': 'checkout-api'
    'runbook-url': 'https://wiki.internal/runbooks/checkout-api-latency'
    'severity': 'sev1'
  }
  properties: {
    displayName: 'Checkout API latency SLO fast burn'
    description: 'P95 latency SLO burn rate exceeding 14.4x over 1 hour'
    severity: 1
    enabled: true
    evaluationFrequency: 'PT5M'
    windowSize: 'PT1H'
    scopes: [appInsightsResourceId]
    criteria: {
      allOf: [
        {
          query: sloLatencyFastBurnQuery
          timeAggregation: 'Count'
          operator: 'GreaterThan'
          threshold: 0
          failingPeriods: {
            numberOfEvaluationPeriods: 1
            minFailingPeriodsToAlert: 1
          }
        }
      ]
    }
    actions: {
      actionGroups: [paymentsTeamActionGroupId]
    }
  }
}

Per-team Action Groups

Each team has its own Action Group with three notification channels: PagerDuty webhook for sev1, Slack webhook for sev2 and sev3, and email for the weekly hygiene digest.

resource paymentsActionGroup 'Microsoft.Insights/actionGroups@2023-01-01' = {
  name: 'ag-payments-team'
  location: 'global'
  properties: {
    groupShortName: 'payments'
    enabled: true
    webhookReceivers: [
      {
        name: 'pagerduty-sev1'
        serviceUri: paymentsPagerdDutyWebhookUrl
        useCommonAlertSchema: true
      }
    ]
    webhookReceivers: [
      {
        name: 'slack-sev2-sev3'
        serviceUri: paymentsSlackWebhookUrl
        useCommonAlertSchema: true
      }
    ]
    emailReceivers: [
      {
        name: 'hygiene-digest'
        emailAddress: 'payments-oncall@company.com'
        useCommonAlertSchema: true
      }
    ]
  }
}

Routing logic in the alert rule maps severity to the correct receiver: sev1 rules reference the Action Group with PagerDuty wired; sev2 and sev3 rules reference the same Action Group but the webhook filters by severity before escalating to PagerDuty.

The weekly hygiene pipeline

A scheduled Azure DevOps pipeline runs every Monday at 07:00 and produces a hygiene report for each team:

schedules:
  - cron: "0 7 * * 1"
    displayName: Weekly alert hygiene check
    branches:
      include:
        - main
    always: true

jobs:
  - job: alert_hygiene
    steps:
      - task: AzureCLI@2
        inputs:
          azureSubscription: $(SERVICE_CONNECTION)
          scriptType: bash
          scriptLocation: inlineScript
          inlineScript: |
            echo "Fetching all alert rules..."
            RULES=$(az monitor scheduled-query list               --resource-group rg-monitoring-prod               --query "[].{name:name,tags:tags,enabled:properties.enabled}"               -o json)

            echo "$RULES" | jq -r '
              .[] |
              select(
                (.tags["owner-team"] == null) or
                (.tags["runbook-url"] == null) or
                (.enabled == false)
              ) |
              "HYGIENE: (.name) missing=(
                [
                  if .tags["owner-team"] == null then "owner-team" else empty end,
                  if .tags["runbook-url"] == null then "runbook-url" else empty end,
                  if .enabled == false then "disabled" else empty end
                ] | join(",")
              )"
            '

            ORPHAN_COUNT=$(echo "$RULES" | jq '[.[] | select(.tags["owner-team"] == null)] | length')
            echo "##vso[task.setvariable variable=orphanCount]$ORPHAN_COUNT"

      - script: |
          if [ "$(orphanCount)" -gt "5" ]; then
            echo "##vso[task.complete result=SucceededWithIssues]$( echo $(orphanCount) ) orphaned alert rules found"
          fi
        displayName: Flag if orphan count exceeds threshold

The pipeline posts a summary to the platform Slack channel and marks the run as SucceededWithIssues if the orphan count crosses five. It does not fail the pipeline — hygiene debt is a warning, not a blocker — but it is visible in the weekly standup report.

Where we ended up

94 alert rules reduced to 31. Every active rule has an owner tag, a runbook link, and routes to a team-specific Action Group. Fast-burn pages go to PagerDuty and wake someone. Slow-burn alerts go to Slack and create a ticket. The weekly hygiene job has caught three orphaned rules in the two months since it shipped, all from resources decommissioned without cleaning up their alert rules.

The 03:47 page in October was the last false positive. The engineer who got paged that night wrote the hygiene pipeline. The rule that woke them is gone. The decommissioned node pool it was watching is gone. The shared ops email that used to receive every alert now receives only the weekly digest.

The thing that changed most was not the alert conditions but the ownership model. When every alert rule has a named team attached to it, "who owns this alert" stops being a question you ask at 03:47 and becomes something the system already knows.