YAML pipeline templates across 58 repositories: versioning, the breaking change that shipped to 23 pipelines at once, and the extend-only governance model that stopped it happening again

At 09:14 on a Tuesday, the build queue went orange across 23 repositories simultaneously. The failure message was identical in every case: Error: 'publishTestResults' is not a valid task name. A task had been renamed in the shared steps template overnight. The engineer who made the change had tested it against one pipeline, confirmed it worked, and merged. The 22 other pipelines that referenced the same template were collateral damage.

The incident lasted 47 minutes. Every affected repository was blocked from merging until the template was reverted, and two releases that were scheduled for 10:00 slipped. The postmortem produced three action items: template versioning, a consumer validation gate on the template repository, and a governance model that made breaking changes structurally harder to ship by accident.

This is that rebuild, applied across 58 repositories and running for six months without a repeat incident.

The problem with unversioned templates

The root cause was simple. Every repository that used the shared template referenced it like this:

resources:
  repositories:
    - repository: templates
      type: git
      name: platform/pipeline-templates
      ref: refs/heads/main

steps:
  - template: steps/build.yml@templates

ref: refs/heads/main means "use whatever is at the tip of main right now." That is convenient during development and catastrophic when a breaking change lands. The 23 affected pipelines were all pinned to main, and they all picked up the renamed task the moment the merge happened.

Template versioning with tags

The fix is to pin to a tag rather than a branch. The platform team owns the template repository and publishes versioned releases by creating a git tag for each breaking-change-free increment.

Consumer repositories reference a specific tag:

resources:
  repositories:
    - repository: templates
      type: git
      name: platform/pipeline-templates
      ref: refs/tags/v2.4.1

A tag reference does not move. A merge to main in the template repository does not affect any consumer pinned to v2.4.1. The consumer upgrades on their own schedule by bumping the tag in their YAML.

The version scheme is straightforward: major version for breaking changes, minor version for new optional parameters, patch for bug fixes and documentation. Breaking changes require a major version bump and a migration guide in the tag release notes.

The migration from ref: refs/heads/main to a tag was scripted across all 58 repositories:

LATEST_TAG=$(git -C /tmp/templates describe --tags --abbrev=0)

for REPO_DIR in /tmp/repos/*/; do
  YAML_FILES=$(find "$REPO_DIR" -name "*.yml" -path "*/.azure/*")
  for YAML in $YAML_FILES; do
    if grep -q "ref: refs/heads/main" "$YAML"; then
      sed -i "s|ref: refs/heads/main|ref: refs/tags/$LATEST_TAG|g" "$YAML"
      echo "Updated: $YAML"
    fi
  done
done

Fifty-eight repositories updated, one PR per repository, merged over two days.

The consumer validation gate

Tagging solves the "breaking change hits everyone at once" problem but it does not prevent a broken template from being tagged. If v2.5.0 has a syntax error or a behaviour regression, every consumer that upgrades will hit it.

The gate is a pipeline in the template repository that runs on every PR. It checks out a representative set of consumer repositories and runs their pipelines against the candidate template version before the merge is allowed.

trigger: none
pr:
  branches:
    include:
      - main

pool:
  vmImage: ubuntu-latest

jobs:
  - job: validate_consumers
    displayName: Validate downstream consumers
    steps:
      - checkout: self

      - script: |
          echo "Testing consumer: service-api"
          az pipelines run             --org "https://dev.azure.com/$(ADO_ORG)"             --project "$(ADO_PROJECT)"             --name "service-api"             --branch "main"             --variables "TEMPLATE_REF=refs/pull/$(System.PullRequest.PullRequestNumber)/merge"             --open
        displayName: Trigger service-api validation run
        env:
          AZURE_DEVOPS_EXT_PAT: $(ADO_PAT)

      - script: |
          echo "Testing consumer: payments-worker"
          az pipelines run             --org "https://dev.azure.com/$(ADO_ORG)"             --project "$(ADO_PROJECT)"             --name "payments-worker"             --branch "main"             --variables "TEMPLATE_REF=refs/pull/$(System.PullRequest.PullRequestNumber)/merge"             --open
        displayName: Trigger payments-worker validation run
        env:
          AZURE_DEVOPS_EXT_PAT: $(ADO_PAT)

The consumer pipelines accept a TEMPLATE_REF variable that overrides the tag reference at runtime. This lets the validation gate test a PR branch of the template without merging it first.

resources:
  repositories:
    - repository: templates
      type: git
      name: platform/pipeline-templates
      ref: ${{ variables.TEMPLATE_REF }}

If TEMPLATE_REF is not set, the pipeline uses the pinned tag as before. If it is set by the validation gate, it uses the PR branch. The gate waits for both consumer runs to complete and fails the template PR if either consumer run fails.

We run five representative consumers in the gate: two services with complex multi-stage pipelines, one monorepo with path filters, one repository that uses the container job template, and one that uses the security scanning step. These five cover about 80% of the template surface area.

The extends governance model

Versioning and the consumer gate solve the stability problem. The governance problem is different: with 58 repositories referencing shared templates, how do you enforce that all pipelines go through the platform-approved template rather than writing ad-hoc steps that bypass security scanning, test gates, or artifact signing?

The answer is the extends resource. Instead of exposing individual step templates that consumers assemble freely, the platform exposes a single pipeline template that consumers must extend, with a fixed structure that enforces required stages.

The extends template looks like this:

parameters:
  - name: buildSteps
    type: stepList
    default: []
  - name: testSteps
    type: stepList
    default: []
  - name: deployEnvironment
    type: string

stages:
  - stage: Build
    jobs:
      - job: build
        steps:
          - template: steps/checkout-and-cache.yml
          - ${{ each step in parameters.buildSteps }}:
            - ${{ step }}
          - template: steps/security-scan.yml
          - template: steps/sign-artifact.yml

  - stage: Test
    dependsOn: Build
    jobs:
      - job: test
        steps:
          - ${{ each step in parameters.testSteps }}:
            - ${{ step }}
          - template: steps/publish-results.yml

  - stage: Deploy
    dependsOn: Test
    jobs:
      - deployment: deploy
        environment: ${{ parameters.deployEnvironment }}
        strategy:
          runOnce:
            deploy:
              steps:
                - template: steps/deploy.yml
                  parameters:
                    environment: ${{ parameters.deployEnvironment }}

A consumer pipeline extends this template and supplies its build and test steps as parameters:

resources:
  repositories:
    - repository: templates
      type: git
      name: platform/pipeline-templates
      ref: refs/tags/v2.4.1

extends:
  template: pipelines/service.yml@templates
  parameters:
    deployEnvironment: my-service-prod
    buildSteps:
      - script: dotnet build src/MyService.csproj --configuration Release
        displayName: Build
    testSteps:
      - script: dotnet test src/MyService.Tests.csproj --configuration Release
        displayName: Unit tests

The consumer cannot skip the security scan or the artifact signing because they are hardcoded in the template. The consumer can only supply the build and test steps that are specific to their service. If they try to add a stage or modify the deploy step, the pipeline rejects it at parse time because the extends model does not allow consumers to add top-level stages.

Azure DevOps enforces this via the required templates setting on the pipeline. In the pipeline settings for each repository, you mark the template repository as required and specify the allowed template file. Any pipeline that does not use extends pointing at the approved template is blocked from running.

Where we ended up

Fifty-eight repositories, all pinned to tagged template versions, all running through the extends governance model. The consumer validation gate runs on every template PR and has caught three breaking changes in the six months since we shipped it, each of which was caught before it reached a tag and therefore before any consumer was affected.

The Tuesday incident was not just a bad merge. It was the natural outcome of a setup that treated shared templates as mutable global state. A reference to main is an implicit dependency on whatever future state main happens to be in. That dependency is invisible in normal operation and only becomes visible when it breaks something. Pinning to tags makes the dependency explicit and voluntary. The consumer upgrades when they are ready, not when the platform team merges.

The extends model solved a problem we did not articulate clearly until we were building it. The problem was not "how do we share pipeline logic" but "how do we guarantee that certain steps always run regardless of what the consumer does." Security scanning and artifact signing are not optional. An extends template with those steps hardcoded makes them structurally unavoidable rather than reliant on the consumer remembering to include them.

The 23-pipeline outage lasted 47 minutes. The consumer validation gate adds about 12 minutes to a template PR. That is the trade that is worth making.