Teaching the Pipeline to Explain Its Own Failures

Most pipeline failure triage isn't debugging. It's reading. And reading is exactly what large language models are good at.

The 4,000-line log problem

A pipeline fails. The notification tells you that it failed. It never tells you why.

So you open the run, expand the failing stage, and start scrolling. The answer is on line 3,847 — and half the time it isn't even the real error. The real error is the package feed that returned a 401 two stages earlier, or the vendor CLI that printed a stack trace and then exited 0, letting the pipeline march on until something downstream hit the missing artifact.

If you own the pipelines, you become the person who reads these logs. Most developers don't read a failing log at all — they see the red X and send the ping: "hey, why did my pipeline fail?" Every failure looks like a pipeline problem until proven otherwise, so it lands on you. I was spending real time every week answering that ping, and the answer was almost always sitting in a log the developer never opened. Someone just had to read it. That's a reading comprehension task. So I stopped doing it by hand.

What I built

The pattern is simple: an on-failure stage at the end of the pipeline. When anything before it fails, it collects the tails of the failing tasks' logs, scrubs them, and sends them to an LLM with a narrow prompt — what failed, what's the likely cause, and is this the code, the environment, or the pipeline? The summary lands in the one place the developer is guaranteed to look: the failed run page itself. A one-line verdict sits in the failure section right next to the native error, and the full write-up — cause, category, suggested fix — is on the run's summary tab. (Routing it to Slack is an easy next step, but starting in the run means nobody can miss it.)

The developer still sees the red X. But next to it, there's now a paragraph in plain text: "The deploy failed because the artifact download returned 404. The build stage published the artifact under a different name than the deploy stage requested. This is a pipeline configuration issue, not a code issue." That's the ping, pre-answered. And every summary ends with a "Does this need SRE?" section that says "No — dev-actionable" unless the log shows genuine platform failure — build agent, network, permissions, quota. When it says no, the ping never happens; when it says yes, I get a head start instead of a cold report.

Here's the part that made it cheap to ship: in my last post I wrote about centralizing our pipeline logic into extends templates, and I claimed the payoff was that new capabilities arrive by default instead of by migration. This was the proof. The triage step went into the shared templates once. Every service riding the platform picked it up on their next run. No repo-level rollout, no migration, no asking teams to adopt anything.

The interesting part: what to send

The model turned out to be the easy part. The engineering is deciding what goes into the context window.

Sending the whole log doesn't work. A verbose build log is hundreds of thousands of tokens of dependency-resolution noise, and the signal drowns. Sending only the last 50 lines of the failing task doesn't work either — the last lines are usually the symptom ("artifact not found"), while the cause is upstream.

The answer that worked: don't pick a log — walk the run. The step pulls the run's timeline from the REST API and collects every failed task in the run, whatever stage it's in. For each one it sends the task's full path (stage / job / task), the error messages the platform recorded against it, and the last 150 lines of its log — plus run metadata like the pipeline name, branch, and the commit message that triggered it.

The first row is the one that matters. The task a developer stares at is often downstream of the one that actually broke — the deploy fails because the build quietly did. Collecting every failed task across the whole run is what lets the model say "the failure you're looking at started two stages ago," which is precisely the diagnosis humans are slowest at.

Everything is capped: five failed tasks, 150 lines each, ~40k characters total. When the bundle has to shrink, it keeps the end of each log, because that's where errors live.

Security and data handling

Scrubbing happens in two layers before anything leaves the step. The platform already masks registered secrets; a regex pass then catches what masking can't know about — JWTs, bearer tokens, key=value credential shapes, long base64 blobs a task echoed by accident. And the model call goes to an organization-approved Azure OpenAI deployment, so build logs stay inside our governed Azure environment instead of being posted to a public consumer endpoint. Honestly, that decision did more to settle the "should build logs go to an AI?" question than any scrubbing regex.

If you build this, treat that as a requirement, not a preference. Build logs leak things no scrubber reliably catches — internal hostnames, infrastructure layout, the occasional secret in a format your regexes have never seen. Send them to an endpoint your organization controls: an in-tenant Azure OpenAI deployment, AWS Bedrock, a gateway in front of a model your security team has already blessed. The version of this pattern that posts log tails to a public API with a personal key is not a smaller version of the same thing — it's a different thing, and at most enterprises it's an incident report waiting to be written.

The shape of it

This is a from-scratch sketch of the pattern, not production code — but it's honestly not far off, because there isn't much to it. One structural point matters: it's a dedicated stage at the end of the pipeline, not a step inside the deploy job. It depends on every prior stage and runs when any of them failed — so a build failure that stops deploy from ever starting still gets triaged, because the step reads the run's timeline rather than any one job's state:

# The last stage of every pipeline, added by the shared template.
- stage: Triage
  dependsOn: [Build, Deploy]       # every prior stage, listed directly
  condition: failed()
  jobs:
    - job: ExplainFailure
      steps:
        - checkout: none           # works entirely off the REST API
        - task: Bash@3
          displayName: 'Explain failure'
          continueOnError: true    # must never add a second failure
          env:
            SYSTEM_ACCESSTOKEN: $(System.AccessToken)
            LLM_API_KEY: $(TriageApiKey)
          inputs:
            targetType: inline
            script: |
              # 1. Walk the run's timeline via the REST API;
              #    collect every failed task, whatever stage it's in
              # 2. For each: error messages + last 150 log lines
              # 3. Scrub: JWTs, bearer tokens, credentials, base64 blobs
              # 4. POST the bundle to the model with a fixed triage prompt
              # 5. Publish: full write-up on the summary tab, one-line
              #    verdict in the failure section next to the native error
              ./triage.sh

The prompt is deliberately rigid. It forces a fixed structure — a one-sentence verdict, what failed, probable root cause with the key log lines quoted, a category (application code, test failure, pipeline configuration, infrastructure, or transient/external), a suggested fix, and the SRE-needed call — all capped under 300 words. It also carries one instruction that earns its keep: if the evidence is insufficient to be sure, say so plainly instead of inventing a cause. Constraining the output is what keeps the summaries short enough that people actually read them.

Advisory only, on purpose

One rule shaped every design decision: the triage step never blocks, never retries the build, and never mutates anything. It writes an explanation. That's all it's allowed to do.

The rule extends to its own failures. If the API key is missing, the model call times out, or the log fetch fails, the step logs a warning and exits clean — it never adds a second failure to a run that's already broken. A triage step that can itself break the pipeline would be worse than no triage step at all.

This isn't timidity — it's the math of being wrong. A wrong summary costs a developer thirty seconds of skepticism before they read the log themselves, which is what they would have done anyway. A wrong action — an automatic retry that masks a real failure, a rollback triggered by a misread log — costs an incident. Advisory-only means the failure mode of the AI is "mildly unhelpful paragraph," and that's a failure mode I can live with while trust builds.

One small decision I'd defend hard: the output is disclosed, but not branded. The heading is just Reason for failure, with a one-line note underneath identifying it as automated analysis of the failed tasks' logs and pointing back to the raw logs as the source of truth. Leading with "AI-powered" invites both reflexive over-trust and reflexive dismissal, and neither helps. The point isn't the technology — it's that the question gets answered.

The cost side is just as boring, in a good way. Failures are rare relative to runs, the input is a capped, scrubbed bundle rather than a whole log, and a small model handles reading comprehension fine. At those caps the per-failure model cost is negligible — the expensive resource was never the API, it was the engineer reading line 3,847.

Where it should earn its keep — and where it won't

It's early. This has been running for weeks, not quarters, and I don't have a full catalog of wins and flops yet. What I do have is a clear picture of which failure classes it's built for, and which failure modes I'm watching for — because both were design inputs, not surprises I'm waiting to discover.

The target: failures a given developer sees once a year but that follow patterns repeated across thousands of builds. Expired feed credentials. Version conflicts buried in dependency-resolution noise. Tasks that fail with a misleading exit code. And the upstream-cause case — symptom in the deploy stage, cause in the build stage — which is exactly what the timeline walk exists to catch.

The flop modes I'm watching for: summaries that confidently paraphrase the stack trace, adding nothing the developer couldn't see in the last ten lines. And failures whose true cause never made it into the log at all — agent networking issues, quota exhaustion, an outage on the far end of a curl. The model can only read what's written down, and some failures aren't. The prompt tells it to admit when the evidence is thin; whether it reliably does is one of the things the next few months will answer.

That second flop mode is worth being honest about, because it's also the argument for keeping it advisory. The model can't be relied on to notice when it's outside its evidence. The design has to assume it won't.

What I'd tell past me

Spend your effort on context selection, not prompt wording. The change that mattered most wasn't in the prompt at all — it was walking the run's timeline and collecting every failed task instead of grabbing the last log.

Make it advisory from day one. Trust is the actual product here, and it builds fastest when the cost of being wrong is thirty seconds.

And ship it through the platform layer. The triage step is one file in one repository; every improvement to the scrubber or the prompt reaches every pipeline on its next run. That's the argument I made for centralizing pipelines in the first place — this was the first capability to prove it.