AKS cluster upgrade from 1.28 to 1.30: the PDB that blocked eviction for 47 minutes and the automated rollout that replaced it

The cluster upgrade started at 21:00 on a Saturday to minimise customer impact. The plan was to upgrade the node pool in rolling fashion, two nodes at a time, with the drain completing on each pair before the next pair started. The first pair drained cleanly in eleven minutes. The second pair stalled at 21:23 and did not move.

The kubectl drain output told the story clearly enough: error when evicting pods/"analytics-exporter-7d9f6b" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. The analytics-exporter deployment had a PodDisruptionBudget with maxUnavailable: 0. The deployment was running one replica. The drain could not evict the pod without violating the PDB, and the upgrade controller was retrying every five seconds, waiting for the situation to resolve itself, which it would not.

At 21:47 we patched the PDB manually from a laptop to allow one unavailable pod, the drain completed in 40 seconds, and the upgrade finished at 23:14 with no further issues. The actual downtime for the analytics-exporter was zero because the pod came back up on a new node within the upgrade window, but the 47-minute stall delayed the entire node pool rotation and burned the maintenance window buffer we had budgeted.

The post-incident action items: automate the upgrade pipeline, add pre-flight checks that catch PDB misconfigurations before the drain starts, and never run a cluster upgrade from a laptop again.

Why PDBs block upgrades and when they should

A PodDisruptionBudget is a good thing. It prevents Kubernetes from evicting too many pods from a deployment simultaneously, which protects service availability during node maintenance. The misconfiguration is not the existence of the PDB but the combination of maxUnavailable: 0 and a single-replica deployment.

With one replica and maxUnavailable: 0, the only safe way to evict the pod is if a second replica is already running on another node. The scheduler will not start the second replica until the first is evicted. The drain cannot evict the first pod until the PDB is satisfied. The system deadlocks.

The correct PDB for a single-replica deployment that can tolerate a brief outage during maintenance is maxUnavailable: 1. The correct PDB for a single-replica deployment that cannot tolerate any outage is to run more than one replica. maxUnavailable: 0 on a single-replica deployment is almost always a mistake.

The pre-flight check

The pre-flight runs before any node cordon and outputs a list of deployments that would block a drain.

#!/bin/bash
# pre-flight-pdb-check.sh
# Exits non-zero if any PDB would deadlock a drain

NAMESPACE=${1:-"--all-namespaces"}
ERRORS=0

while IFS= read -r line; do
  NAME=$(echo "$line" | awk '{print $1}')
  NS=$(echo "$line" | awk '{print $2}')
  MIN_AVAILABLE=$(echo "$line" | awk '{print $3}')
  MAX_UNAVAILABLE=$(echo "$line" | awk '{print $4}')
  CURRENT_HEALTHY=$(echo "$line" | awk '{print $5}')
  DESIRED_HEALTHY=$(echo "$line" | awk '{print $6}')

  # Flag: maxUnavailable is 0 (or min-available equals current healthy)
  if [ "$MAX_UNAVAILABLE" = "0" ] && [ "$CURRENT_HEALTHY" -le 1 ]; then
    echo "BLOCK: PDB $NAME in $NS has maxUnavailable=0 with only $CURRENT_HEALTHY healthy pod(s)"
    ERRORS=$((ERRORS + 1))
  fi

  # Flag: disruptions allowed is 0 meaning no pod can be evicted
  ALLOWED=$(kubectl get pdb "$NAME" -n "$NS" -o jsonpath='{.status.disruptionsAllowed}' 2>/dev/null)
  if [ "$ALLOWED" = "0" ]; then
    echo "BLOCK: PDB $NAME in $NS currently allows 0 disruptions"
    ERRORS=$((ERRORS + 1))
  fi

done < <(kubectl get pdb $( [ "$NAMESPACE" = "--all-namespaces" ] && echo "-A" || echo "-n $NAMESPACE")   -o custom-columns="NAME:.metadata.name,NS:.metadata.namespace,MINAVAIL:.spec.minAvailable,MAXUNAVAIL:.spec.maxUnavailable,CURRENTHEALTHY:.status.currentHealthy,DESIREDHEALTHY:.status.desiredHealthy"   --no-headers 2>/dev/null)

if [ "$ERRORS" -gt 0 ]; then
  echo ""
  echo "$ERRORS PDB issue(s) found. Resolve before proceeding with upgrade."
  exit 1
fi

echo "Pre-flight PDB check passed."
exit 0

Running this before the Saturday upgrade would have caught the analytics-exporter PDB in 30 seconds.

The automated upgrade pipeline

The upgrade pipeline runs in Azure Pipelines and is parameterised by cluster name, target Kubernetes version, and node pool name. It runs the pre-flight, performs the control plane upgrade, upgrades the system node pool, then upgrades the user node pools in sequence.

parameters:
  - name: clusterName
    type: string
  - name: resourceGroup
    type: string
  - name: targetVersion
    type: string
  - name: nodePools
    type: object
    default: []

stages:
  - stage: PreFlight
    displayName: Pre-flight checks
    jobs:
      - job: pdb_check
        displayName: PDB eviction check
        steps:
          - task: AzureCLI@2
            inputs:
              azureSubscription: $(SERVICE_CONNECTION)
              scriptType: bash
              scriptLocation: inlineScript
              inlineScript: |
                az aks get-credentials                   --name ${{ parameters.clusterName }}                   --resource-group ${{ parameters.resourceGroup }}                   --overwrite-existing
                bash scripts/pre-flight-pdb-check.sh --all-namespaces

      - job: version_check
        displayName: Version availability check
        steps:
          - task: AzureCLI@2
            inputs:
              azureSubscription: $(SERVICE_CONNECTION)
              scriptType: bash
              scriptLocation: inlineScript
              inlineScript: |
                AVAILABLE=$(az aks get-upgrades                   --name ${{ parameters.clusterName }}                   --resource-group ${{ parameters.resourceGroup }}                   --query "controlPlaneProfile.upgrades[].kubernetesVersion"                   -o tsv)
                if ! echo "$AVAILABLE" | grep -q "${{ parameters.targetVersion }}"; then
                  echo "Version ${{ parameters.targetVersion }} not available for upgrade"
                  echo "Available: $AVAILABLE"
                  exit 1
                fi
                echo "Version ${{ parameters.targetVersion }} available."

  - stage: UpgradeControlPlane
    displayName: Upgrade control plane
    dependsOn: PreFlight
    jobs:
      - deployment: upgrade_cp
        environment: aks-upgrade-prod
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureCLI@2
                  inputs:
                    azureSubscription: $(SERVICE_CONNECTION)
                    scriptType: bash
                    scriptLocation: inlineScript
                    inlineScript: |
                      az aks upgrade                         --name ${{ parameters.clusterName }}                         --resource-group ${{ parameters.resourceGroup }}                         --kubernetes-version ${{ parameters.targetVersion }}                         --control-plane-only                         --yes
                      echo "Control plane upgrade complete."

  - stage: UpgradeNodePools
    displayName: Upgrade node pools
    dependsOn: UpgradeControlPlane
    jobs:
      - ${{ each pool in parameters.nodePools }}:
        - deployment: upgrade_pool_${{ pool }}
          displayName: Upgrade node pool ${{ pool }}
          environment: aks-upgrade-prod
          strategy:
            runOnce:
              deploy:
                steps:
                  - task: AzureCLI@2
                    inputs:
                      azureSubscription: $(SERVICE_CONNECTION)
                      scriptType: bash
                      scriptLocation: inlineScript
                      inlineScript: |
                        az aks nodepool upgrade                           --cluster-name ${{ parameters.clusterName }}                           --resource-group ${{ parameters.resourceGroup }}                           --name ${{ pool }}                           --kubernetes-version ${{ parameters.targetVersion }}                           --no-wait

                        echo "Waiting for node pool ${{ pool }} upgrade to complete..."
                        az aks nodepool wait                           --cluster-name ${{ parameters.clusterName }}                           --resource-group ${{ parameters.resourceGroup }}                           --name ${{ pool }}                           --updated                           --interval 60                           --timeout 3600
                        echo "Node pool ${{ pool }} upgrade complete."

The upgrade pipeline is triggered manually from the Azure DevOps UI with a required approval on the aks-upgrade-prod environment before any node pool rotation begins. The pre-flight stage runs automatically; if it fails, the pipeline stops before any cluster state is touched.

The cordon-and-drain wrapper

For cases where the pre-flight identifies a PDB issue and the team needs to understand the scope before deciding how to handle it, we have a wrapper script that cordons a node, lists what would be evicted, and reports which pods have PDB conflicts without actually draining.

#!/bin/bash
# drain-dry-run.sh NODE_NAME

NODE=$1
if [ -z "$NODE" ]; then
  echo "Usage: $0 <node-name>"
  exit 1
fi

echo "Pods on $NODE that would be evicted:"
kubectl get pods --all-namespaces --field-selector spec.nodeName="$NODE"   -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind"   --no-headers

echo ""
echo "PDB conflicts:"
kubectl drain "$NODE"   --ignore-daemonsets   --delete-emptydir-data   --dry-run=client 2>&1 | grep -i "disruption budget"

This runs in under 10 seconds and gives the on-call engineer a clear view of what would block the drain before they commit to running it.

Handling the PDB conflict in automation

When the pre-flight catches a PDB conflict, the pipeline does not auto-patch the PDB. That would be a destructive action performed without human review. Instead, the pipeline posts the list of conflicting PDBs as a comment on the pipeline run and fails the pre-flight stage with a clear remediation message:

Pre-flight failed: 1 PDB issue found

BLOCK: PDB analytics-exporter-pdb in monitoring has maxUnavailable=0 with 1 healthy pod

Remediation options:
1. Scale the deployment to 2+ replicas before upgrading
2. Patch the PDB to allow maxUnavailable: 1 if brief unavailability is acceptable
3. Delete the PDB if the workload does not require disruption protection

Re-run the pipeline after resolving the above.

The remediation decision stays with the team that owns the workload, not with the upgrade pipeline.

Where we ended up

Three clusters upgraded from 1.28 to 1.30 over two weekends using the automated pipeline. The pre-flight caught two PDB issues across the three clusters: the analytics-exporter we already knew about and a second single-replica deployment in a staging namespace that had copied a misconfigured PDB from an older manifest without anyone noticing.

Both were resolved before the drain started. Neither upgrade stalled. Total elapsed time per cluster was between 90 minutes and 140 minutes depending on node count. The manual Saturday incident took 2 hours and 14 minutes for a single cluster, with 47 of those minutes spent stalled on a PDB that a 30-second script would have caught.

The other thing the automation changed was the upgrade cadence. Before the pipeline, upgrades were quarterly events with two-week planning cycles because the manual process was error-prone enough to warrant significant preparation. With the pre-flight and the parameterised pipeline, upgrades are now scheduled monthly, each one taking less time than the planning cycle used to.

Kubernetes versions go end-of-support on a known schedule. Running current is not optional; the question is whether you do it deliberately with good tooling or reactively under pressure. The 47-minute stall was the pressure that built the tooling.