Optimizing the Test Process: What to Measure Before Shrinking the Regression Suite

Shrinking a regression suite is usually proposed in a meeting and executed with a text editor. Someone says “these tests take too long,” a list of candidates gets assembled from memory, and two hundred tests disappear. Four weeks later a bug reaches production, nobody can prove whether a deleted test would have caught it, and the whole set is restored. The suite is now exactly as slow as before, plus the team has lost the appetite to try again.

The problem is not the cutting. It is that the cut was made without a baseline, so there was nothing to compare against afterwards. Everything below is about the measurements you take before you delete anything, and the decision rule those measurements feed.

Define the Decision First: What Are You Optimizing?

“The regression suite is too big” is three different complaints wearing one coat.

PR feedback time. Developers wait for a pipeline before merging. The metric is wall-clock duration of the PR job, and the tests that matter are the ones on the critical path — not the slowest tests overall, the slowest branch.

CI bill. Finance is looking at the invoice. The metric is total compute minutes per day across every trigger, including nightly runs nobody reads. Here a fast test that runs 400 times a day costs more than a slow test that runs twice.

Maintenance load. Engineers spend hours per week fixing tests instead of writing code. The metric is edits-per-test over the last quarter and the share of failures that turned out to be the test’s fault. Deleting slow tests does nothing for this; deleting brittle ones does.

Pick one and write it as a number with a date: “PR pipeline from 42 minutes to 12 minutes by the end of the quarter.” Without a target, deletion becomes taste, and taste loses the first argument it has after an escaped defect.

The Test Inventory: Six Fields You Need per Test

Every decision below runs off one table. Six columns per test:

1. Mean and p95 duration. The mean tells you cost; p95 tells you which tests occasionally block the pipeline.
2. Failure count over the last 30–90 runs. Raw count, no interpretation yet.
3. How many of those failures were real defects. Requires a human pass over failure records. There is no shortcut.
4. Date of the last real defect caught.
5. Business rule or risk covered. One short tag: payment.refund, auth.session-expiry.
6. Level in the pyramid. unit / integration / API / E2E.

Columns 1 and 2 come from the report files CI already produces. Column 3 is the expensive one and the reason most teams skip it — and skipping it is exactly what turns measurement into guessing. Columns 5 and 6 come from test metadata, which means tagging tests is a prerequisite, not a nice-to-have.

The practical obstacle is almost always artifact retention. CI providers expire build artifacts on a default window, and sharded runs write one report per shard, so a single pipeline produces a dozen files that have to be merged before anything can be counted. Fix both before you start collecting: push report files to object storage on every run, and name them with run ID and shard index.

Failure Yield: Real Defects per Test

A test’s value is the number of real regressions it has caught, not the number of times it has executed. Divide column 3 by the number of runs and you have failure yield. Most suites, once measured, split into a small group of tests that catch nearly everything and a long tail that has never caught anything.

Do not convert that observation into “never failed, therefore delete.” Yield is frequency, and some risks are rare by nature. A test covering refund calculation on a currency used by 2% of customers may catch one defect every three years — and that defect would be a financial correction, not a cosmetic bug. The same applies to authorization boundaries and data-integrity constraints: low frequency, high blast radius. Insurance is not worthless because the house hasn’t burned down.

So read yield together with a risk weight on the business-rule tag. Low yield plus low risk is a candidate. Low yield plus high risk is a keeper, possibly at reduced frequency.

Flake Rate: The Noise You Must Clear Before Shrinking

Flaky tests corrupt every number in the inventory. A test that fails one run in six because of a race condition shows up with a high failure count, which looks like “this test catches a lot.” It catches nothing; it just fails. Meanwhile the retries it triggers inflate pipeline duration, so the same test also pollutes your duration data.

Compute flake rate per test first: runs where the test failed and then passed on retry with no code change, divided by total runs. Set a threshold — many teams use 1% — and before any shrink decision, either fix the tests above it or quarantine them into a separate job. Only then are the failure counts worth reading. If you need a method for the fixing part, I wrote one up in diagnosing flaky tests in Playwright, and a large share of instability traces back to shared fixtures, which is a test data ownership problem rather than a timing one.

Duration Distribution and the Critical Path

Average test duration is the wrong statistic. Pipeline wall-clock time is set by the longest parallel branch, so a single 9-minute test in an otherwise 4-minute shard defines your feedback loop no matter what the other 800 tests do.

Sort tests by duration descending. Typically a small number of tests produce a large share of total runtime, and a meaningful fraction of that time is setup, not verification: logging in through the UI, seeding data through forms, waiting for an app shell that the test does not actually assert on. Those tests can get faster without being deleted — replace UI login with a token injected into storage state, create fixture data through the API, and rebalance shards by measured duration instead of file count. Exhaust this list before you open the deletion discussion; see performance optimization in test automation for the specific techniques.

Coverage Overlap: How Many Tests Verify the Same Risk

The safest cuts come from overlap, not from line coverage. When the same business rule is verified by a unit test, an API test, and an E2E test, the E2E layer is usually paying for a confirmation you already have — at ten to a hundred times the runtime, with the full flakiness surface of a browser attached.

Extract overlap by grouping the inventory on the business-rule tag and listing the pyramid level of each test in the group. Groups with three levels and one rule are where the reduction lives. A code coverage tool will not tell you this, because it cannot distinguish intent: two tests that execute identical lines may be checking a calculation and a navigation flow respectively, and coverage reports them the same way.

The reverse pattern matters too. A rule covered only at E2E level is not a deletion candidate; it is a signal that the test should be pushed down, which is the practical shape of scaling an automation suite.

Defect Escape Rate: The Only Honest Scorecard

Before cutting, count the defects that reached production in the last quarter and were catchable by the regression suite — meaning a deterministic check at some level could have found them. That number is your baseline. Everything after the cut is compared to it.

Tag each escaped defect with the level that should have caught it: unit, API, E2E, or none-of-these (config, data migration, third-party behavior). The distribution tells you where your suite is actually thin, and it frequently contradicts the intuition that drove the shrink proposal. Without this baseline, a post-cut incident has no denominator, and the argument collapses into opinion — which is how reductions get reverted wholesale.

Converting CI Cost into Money

Managers fund changes that have a number attached:

minute price × pipeline minutes × runs per day × 21 working days

That is the compute side. Add the human side: how many times per day does a developer wait on a pipeline, and what does the lost context cost at a loaded hourly rate. The waiting cost is usually larger than the invoice, which is why “PR feedback time” and “CI bill” lead to different cuts.

Then put the risk of deletion in the same table, also in money: expected defect escapes × average incident cost (engineering hours for the hotfix, plus whatever the business already knows an outage-hour costs). If you only quantify the cost side, the cost side wins by default and quality enters the meeting as an adjective.

A Small Script That Collects the Measurements

This reads a folder of CI reports — JUnit XML or Playwright JSON — and writes one CSV row per test with run count, failures, flaky runs, mean and p95 duration. No dependencies, Node 18+.

#!/usr/bin/env node
// collect-test-inventory.js
// Usage: node collect-test-inventory.js ./reports > inventory.csv

const fs = require('fs');
const path = require('path');

function walk(dir, acc = []) {
  for (const entry of fs.readdirSync(dir, { withFileTypes: true })) {
    const full = path.join(dir, entry.name);
    if (entry.isDirectory()) walk(full, acc);
    else if (/\.(xml|json)$/i.test(entry.name)) acc.push(full);
  }
  return acc;
}

function attrs(raw) {
  const out = {};
  for (const m of raw.matchAll(/([\w:-]+)="([^"]*)"/g)) out[m[1]] = m[2];
  return out;
}

function parseJUnit(xml) {
  const cases = [];
  const re = /<testcase\b([^>]*?)(\/>|>([\s\S]*?)<\/testcase>)/g;
  for (const m of xml.matchAll(re)) {
    const a = attrs(m[1]);
    const body = m[3] || '';
    if (/<skipped\b/.test(body)) continue;
    cases.push({
      name: `${a.classname || ''} > ${a.name || 'unnamed'}`.trim(),
      durationMs: Math.round(parseFloat(a.time || '0') * 1000),
      failed: /<failure\b|<error\b/.test(body),
      flaky: /<flakyFailure\b/.test(body),
    });
  }
  return cases;
}

function parsePlaywright(report) {
  const cases = [];
  const visit = (suite, trail) => {
    const title = [...trail, suite.title].filter(Boolean);
    for (const spec of suite.specs || []) {
      for (const t of spec.tests || []) {
        const results = t.results || [];
        const durationMs = results.reduce((s, r) => s + (r.duration || 0), 0);
        cases.push({
          name: [...title, spec.title].join(' > '),
          durationMs,
          failed: t.status === 'unexpected',
          flaky: t.status === 'flaky',
        });
      }
    }
    for (const child of suite.suites || []) visit(child, title);
  };
  for (const suite of report.suites || []) visit(suite, []);
  return cases;
}

function percentile(sorted, p) {
  if (sorted.length === 0) return 0;
  const idx = Math.min(sorted.length - 1, Math.ceil((p / 100) * sorted.length) - 1);
  return sorted[Math.max(0, idx)];
}

function csvCell(value) {
  return `"${String(value).replace(/"/g, '""')}"`;
}

const root = process.argv[2];
if (!root) {
  console.error('Usage: node collect-test-inventory.js <reports-dir>');
  process.exit(1);
}

const stats = new Map();

for (const file of walk(root)) {
  const raw = fs.readFileSync(file, 'utf8');
  let cases = [];
  try {
    cases = file.endsWith('.json') ? parsePlaywright(JSON.parse(raw)) : parseJUnit(raw);
  } catch (err) {
    console.error(`skipped ${file}: ${err.message}`);
    continue;
  }
  for (const c of cases) {
    if (!stats.has(c.name)) {
      stats.set(c.name, { runs: 0, failures: 0, flaky: 0, durations: [] });
    }
    const s = stats.get(c.name);
    s.runs += 1;
    if (c.failed) s.failures += 1;
    if (c.flaky) s.flaky += 1;
    s.durations.push(c.durationMs);
  }
}

const rows = [...stats.entries()].map(([name, s]) => {
  const sorted = [...s.durations].sort((a, b) => a - b);
  const mean = Math.round(sorted.reduce((a, b) => a + b, 0) / sorted.length);
  return {
    name,
    runs: s.runs,
    failures: s.failures,
    flaky: s.flaky,
    flakeRatePct: ((s.flaky / s.runs) * 100).toFixed(2),
    meanMs: mean,
    p95Ms: percentile(sorted, 95),
    totalMs: sorted.reduce((a, b) => a + b, 0),
  };
}).sort((a, b) => b.totalMs - a.totalMs);

console.log('test,runs,failures,flaky_runs,flake_rate_pct,mean_ms,p95_ms,total_ms');
for (const r of rows) {
  console.log([
    csvCell(r.name), r.runs, r.failures, r.flaky,
    r.flakeRatePct, r.meanMs, r.p95Ms, r.totalMs,
  ].join(','));
}

The output is a CSV, not a dashboard. The point is to walk into the decision meeting with real numbers. Set it up in an afternoon, let it accumulate two weeks of runs, then add the three human columns — real-defect count, last catch date, business-rule tag — by hand.

The Decision Matrix: Delete, Move Down, Run Less Often, Keep

Every test gets exactly one of four outcomes, and the justification has to come from the inventory row, not from the meeting.

Delete is legitimate only when all three hold: proven overlap at a lower level, zero real defects caught, low risk weight. Two out of three is not enough.

Move down is for valuable but expensive tests: the assertion is worth keeping, the browser is not. An E2E checkout test that really verifies tax calculation becomes an API test plus one thin UI smoke path.

Run less often is for low-frequency, high-impact risks — the insurance cases from the yield section. They move to nightly or release pipelines, not to the trash.

Keep covers the critical flows: login, payment, permission boundaries, anything regulatory. These stay in the PR set regardless of their failure yield, because the cost of catching them late is categorically different.

One warning that comes up repeatedly in review: a test that looks worthless on metrics may be worthless only because something upstream is broken. A test that has been silently skipped by a misconfigured tag, or that asserts on a feature flag which has been off for a year, will show zero failures and zero catches — and the metric says delete while the correct action is to fix the configuration. Before deleting on a zero-yield row, confirm the test actually ran and actually asserted.

Implementing Reduced Frequency with Tags

Splitting by tag instead of deleting makes the decision reversible. Tag at the test level, select at the pipeline level.

// tests/checkout.spec.ts
import { test, expect } from '@playwright/test';

test('rejects an expired card @pr @payments', async ({ page }) => {
  await page.goto('/checkout');
  await page.getByLabel('Card number').fill('4000000000000069');
  await page.getByRole('button', { name: 'Pay' }).click();
  await expect(page.getByRole('alert')).toContainText('expired');
});

test('refunds in a secondary currency @nightly @payments', async ({ page }) => {
  await page.goto('/orders/8841');
  await page.getByRole('button', { name: 'Refund' }).click();
  await expect(page.getByTestId('refund-total')).toHaveText('CHF 240.00');
});

test('full cross-browser purchase journey @release @payments', async ({ page }) => {
  await page.goto('/catalog');
  await page.getByRole('link', { name: 'Noise cancelling headphones' }).click();
  await page.getByRole('button', { name: 'Add to cart' }).click();
  await page.getByRole('link', { name: 'Checkout' }).click();
  await expect(page).toHaveURL(/\/checkout/);
});

# .github/workflows/tests.yml
name: tests

on:
  pull_request:
  schedule:
    - cron: '0 2 * * *'
  workflow_dispatch:
    inputs:
      scope:
        description: 'Tag scope to run'
        required: true
        default: 'release'

jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx playwright install --with-deps chromium

      - name: Select scope
        id: scope
        run: |
          if [ "${{ github.event_name }}" = "pull_request" ]; then
            echo "grep=@pr" >> "$GITHUB_OUTPUT"
          elif [ "${{ github.event_name }}" = "schedule" ]; then
            echo "grep=@pr|@nightly" >> "$GITHUB_OUTPUT"
          else
            echo "grep=@${{ github.event.inputs.scope }}" >> "$GITHUB_OUTPUT"
          fi

      - name: Run tests
        run: npx playwright test --grep "${{ steps.scope.outputs.grep }}"
        env:
          PLAYWRIGHT_JSON_OUTPUT_NAME: report.json

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: report-${{ github.run_id }}
          path: report.json
          retention-days: 90

Moving a test back into the PR set is a one-word edit. That reversibility is what makes the cut defensible in the first place.

The Verification Window After the Cut

Watch three numbers for at least four weeks: escaped defect rate against your pre-cut baseline, hotfix count, and pipeline duration against the target you wrote down in the first section.

If escapes exceed the baseline, do not restore the whole set. Take each escaped defect, find the business-rule tag it belongs to, look up which removed tests carried that tag, and move those specific tests to nightly. That is a two-line change with a documented reason, and it keeps the duration win for everything unrelated.

A reduction without a written rollback rule is not an experiment, it is a bet. Write the rule before the cut: “if more than N catchable defects escape in four weeks, tests tagged X return to the PR set.”

What to Do This Week

1. Turn on long artifact retention and push JUnit/Playwright reports to storage with run ID and shard index in the filename. Start accumulating today; you need two weeks.
2. Run the collection script against whatever reports you already have and set a flake threshold. Quarantine or fix everything above it before trusting any other number.
3. Pull the last quarter’s production defects and tag each with the test level that should have caught it. That is your escape baseline.
4. Pick one module — not the whole suite — and apply the four-outcome matrix to its tests. Prefer tags over deletion for everything that isn’t a clear overlap case.
5. Track escapes, hotfixes and pipeline duration for four weeks, then decide whether the method earned the right to be applied to the next module.

One team, one module, one measured window. Suite-wide cuts made on the first attempt are the ones that get reverted.