Azure Container Registry geo-replication across three regions: pull failures at 08:14, the 400ms cold-start that was actually a cache miss, and the token scope that was the real problem

At 08:14 on a Monday, the latency alert fired for our West Europe workload cluster. Pod startup times had jumped from a normal 18 seconds to 4 minutes 12 seconds. The on-call engineer opened the pod events and saw what looked like slow image pulls. Eight pods were in ContainerCreating and the event stream showed image pull requests succeeding, but slowly.

By 08:31 we had the root cause. The regional Azure Container Registry replica in West Europe had entered a degraded state overnight following a storage backend issue Microsoft had already mitigated, but ACR geo-replication has a client-side failover: when the regional replica is degraded, the ACR client falls back to the home registry, which for us was East US. Pods in West Europe were pulling 1.2 GB images from East US. The pulls succeeded but slowly, and the 400ms cold-start we had blamed on container initialisation for weeks was, in fact, the latency penalty from the occasional cross-region pull that was happening before the full failover.

This is the full setup: geo-replication across three regions done correctly, ACR Tasks for automated base image patching, and the service connection token scope that was routing pulls to the wrong endpoint.

Why geo-replication and what it actually does

Azure Container Registry geo-replication places read replicas of your registry in additional Azure regions. When a pod in West Europe pulls an image, the ACR client resolves the registry hostname to the nearest healthy replica and pulls from there. The round-trip stays within the region, which matters significantly for large images.

The important behaviour to understand: geo-replication is a read-replication model. Pushes always go to the home registry and are replicated asynchronously to the regional replicas. A push in your Azure Pipelines build stage lands in East US first; West Europe and Southeast Asia get the image via replication, which is usually sub-minute but is not instantaneous.

The failover behaviour is what caught us. When a regional replica is degraded, the client falls back to the home registry silently. There is no error, no log entry at the pod level, just slower pulls. The only signal is pull duration, which we were not alerting on.

The geo-replication setup

Three regions: East US (home), West Europe, Southeast Asia. Replication is configured at the registry level.

REGISTRY="acrprodshared"
az acr replication create   --registry "$REGISTRY"   --location westeurope   --region-endpoint-enabled true

az acr replication create   --registry "$REGISTRY"   --location southeastasia   --region-endpoint-enabled true

The --region-endpoint-enabled true flag creates a region-specific endpoint (acrprodshared.westeurope.data.azurecr.io) in addition to the global endpoint. AKS pulls against the global endpoint (acrprodshared.azurecr.io) by default; the geo-replication client resolves this to the nearest healthy regional endpoint at DNS lookup time.

Verify replication status:

az acr replication list --registry "$REGISTRY" --output table

You want provisioningState: Succeeded and regionEndpointEnabled: True for each replica before you rely on regional pull performance.

The token scope that was the real problem

When we investigated the 08:14 incident further, we found a secondary issue that had been contributing to the occasional 400ms cold-start for weeks before the full failover event. Our Azure Pipelines service connection was using a registry-scoped token with repository-level permissions, but the token had been created against the home registry endpoint rather than the global endpoint.

az acr token create   --name pipeline-pull-token   --registry "$REGISTRY"   --repository myapp/api content/read   --repository myapp/worker content/read

This token works, but when used with docker pull acrprodshared.azurecr.io/myapp/api:latest, the token exchange goes through the global endpoint and the geo-replication client resolves the pull to the nearest replica. However, when the Azure Pipelines Docker task uses the service connection, it was explicitly setting the registry server to acrprodshared.azurecr.io without the geo-routing headers, and the ACR geo-replication client was not activating. Every pull was going to East US regardless of where the agent was running.

The fix is to use the AKS-attached registry integration for in-cluster pulls and to set the registry server correctly for pipeline-side pulls.

For AKS, attach the registry to the cluster rather than using a pull secret:

az aks update   --name aks-prod-weu   --resource-group rg-prod-weu   --attach-acr "$REGISTRY"

This grants the kubelet managed identity AcrPull on the registry and means image pulls use the identity-based auth flow which does correctly geo-route.

For pipeline-side pulls (base image scanning, security checks), use the regional endpoint explicitly rather than the global one:

variables:
  ACR_LOGIN_SERVER: acrprodshared.azurecr.io

steps:
  - task: Docker@2
    inputs:
      containerRegistry: acr-service-connection
      command: pull
      arguments: $(ACR_LOGIN_SERVER)/myapp/api:$(Build.BuildId)

The service connection itself should be created with the global endpoint. The geo-routing happens at the ACR network layer, not at the client endpoint string.

ACR Tasks for automated base image patching

The second thing the 08:14 incident exposed was that our West Europe replica was serving a patched base image while East US had the older version, because our ACR Task that rebuilds images on base image update had only been configured to push to East US and relied on geo-replication to distribute the result.

During the degraded replica window, pods in West Europe that were pulled to East US got the newer patched image, while pods that had been pulled from the cached West Europe replica before the degradation had the older base. This created a two-version split across the same deployment that was invisible at the application level but would have been material for a CVE-bearing patch.

ACR Tasks automate this correctly when configured to push to the global endpoint:

az acr task create   --name base-image-patch-myapp-api   --registry "$REGISTRY"   --image myapp/api:{{.Run.ID}}   --image myapp/api:latest   --context https://github.com/org/myapp.git   --file Dockerfile.api   --commit-trigger-enabled false   --base-image-trigger-enabled true   --base-image-trigger-type Runtime   --auth-mode Default

The --base-image-trigger-type Runtime flag means the task fires whenever the base image used in the Dockerfile is updated in the registry. Because the push target is the global endpoint, geo-replication distributes the rebuilt image to all replicas within the normal replication window.

The replica health alert we added

The pull-duration signal turned out to be the most reliable early indicator of a degraded replica. We added a Log Analytics query that alerts if the p95 pull duration for any cluster exceeds a threshold for two consecutive five-minute windows:

ContainerLog
| where LogEntry has "pulling image"
| extend DurationMs = todouble(extract("duration=([0-9.]+)s", 1, LogEntry)) * 1000
| where isnotnull(DurationMs)
| summarize p95_pull_ms = percentile(DurationMs, 95) by bin(TimeGenerated, 5m), ClusterName
| where p95_pull_ms > 30000

This alert has fired three times since we set it up. Each time it was either a genuine replica degradation (once) or a large image being pulled for the first time after a deployment with no warm cache (twice). The false-positive rate is acceptable because the alert does not page; it goes to a Slack channel that the platform team checks in the morning.

Where we ended up

Three-region geo-replication running correctly, AKS clusters attached to the registry via managed identity, ACR Tasks rebuilding on base image update and pushing to the global endpoint, and a pull-duration alert that catches degraded replicas before they become a 4-minute pod startup incident.

The Monday morning incident was caused by a combination of a Microsoft-side storage issue and our misconfigured token scope that had been silently routing some pulls to the wrong region for weeks. The token scope issue would have been invisible without the deeper investigation the incident forced. The 400ms cold-start we had dismissed as container initialisation overhead was, in fact, a consistent signal we had not read correctly.

Geo-replication is not a passive feature. It requires the client-side integration to be correct, the task automation to push to the right endpoint, and alerting that catches degraded replicas at the symptom level rather than waiting for a user-visible outage.