We’ve all seen it happen. The CI pipeline is green, every check passes, and somehow, a customer still loses money.

We’ve gotten so caught up in Jenkins pipelines and the promise of AI writing our tests that we forgot a basic truth: test automation was never supposed to replace thinking.

The Illusion of Coverage

Think about what a regression suite actually is. It’s just institutional memory. It reminds us of the bugs we’ve already seen and fixed. It doesn’t imagine the bugs we haven't found yet. But lately, as codebases balloon because everyone is prompt-engineering their way through sprints, we just keep adding tests. The suite grows taller, but it never grows wider.

Let's be honest about what automation is: it's just running a checklist. And who wrote that checklist? Probably someone who was stressed out and rushing, working on a version of the app that looks nothing like what we have today.

I’ve seen teams proudly claim "100% coverage" while the actual integration paths are rotting from the inside out. Payments pass in the test environment because the mock returns a nice little 200 OK. But in production, the app is hitting a deprecated endpoint because nobody noticed the vendor migrated. Pipeline stays green. Revenue goes red. The checklist passed, but the customer still got screwed.

The Volume Problem

AI hasn't just changed how we write code; it’s radically changed how much code we push per sprint. The surface area is massive now, but the number of QA hours? Exactly the same, if not less.

Automated tests scale execution cheaply, but they completely fail at scaling judgment. You need a human looking at this stuff asking, "Wait, should we even be testing this?" or "What are we ignoring just because it's a pain to mock?" When you let models generate tests without human review, it gets worse. AI loves to just spam assert response.status_code == 200. It rarely checks actual business logic. So when you see "Test coverage up 40%," that usually just means you have 40% more noise in your suite.

Gatekeepers vs. Green Lights

Somewhere along the line, we confused gatekeepers with green lights. A green light just tells you something ran. A gatekeeper actually stops bad code from going out. We replaced the latter with the former and called it "modernization."

Real gatekeeping looks like:

  • A QA engineer refusing to sign off because they don't fully understand the new auth flow.
  • A senior dev rejecting a bloated 600-line AI-generated PR and asking for an 80-line manual rewrite.
  • A release manager saying "no" to a Friday deploy because nobody did any exploratory testing.

It’s unpopular. It’s slow. But it actually works.

A Boring Ritual That Works

There’s a boring maintenance ritual I think every team needs to adopt. Once a month:

  1. Pick five automated tests at random.
  2. Try to trace each one back to a user story. Does that story even exist anymore? Kill the orphan tests.
  3. Add one new test based on a real support ticket from last month.
  4. Spend two hours just doing exploratory testing. Have humans click around, do the weird things users do, and file the bugs that automation could never dream up.

And if you're trusting AI to write your tests, you need to actually read them. If the person who generated it can't explain the assertion, that test isn't protecting you. It's just decoration for a dashboard.

The Real Quality System

We need to stop treating velocity like a virtue. Velocity is just a tempo. Customers don't care about your sprint commit count; they remember the outage. Quality isn't just nostalgia for the "good old days" of software. It’s the difference between code that technically runs and software you can actually defend when things go wrong.

Right now, we’re shipping massive amounts of code with fewer people validating it—or even understanding it. Then we act surprised when user trust erodes.

The checklist was never the quality system. The real quality system was the people who knew exactly when to ignore the checklist.