Diagnosing Flaky Tests: Finding the Root Cause of Instability in Playwright

What a Flaky Test Is — and What It Isn’t

A flaky test is a test that produces different results on the same commit, in the same environment, without anything in the application changing. Run it ten times: seven green, three red. That’s flakiness.

A test that catches a genuine bug is not flaky. Neither is a test that fails because the feature it covers is broken under concurrency, or because the backend returns a 500 one request in fifty. Those are real defects with intermittent symptoms — and they are exactly the defects teams destroy when they slap the “flaky” label on every red build. Once a team learns that red means “run it again,” it has stopped reading its own test results. The suite still runs, but it no longer carries information.

Retries make this worse, not better. retries: 2 in your Playwright config is damage control for a CI pipeline that has to keep moving; it is not a diagnosis and it is not a fix. A retried test that passes on attempt two is a test you now know nothing about, except that it is unstable. If you run retries, treat every retried-and-passed result as an open defect record, not as a green tick. Playwright already marks these as flaky in its report — that status is your input queue, not your success metric.

You Cannot Diagnose Instability Without Measuring It

“I think that test is flaky” is a hypothesis. “That test failed 9 times out of 214 runs last week, always on the CI runner, never locally” is evidence you can act on. The gap between those two sentences is one script.

Playwright’s JSON reporter gives you everything you need. Configure it to write a file per run, keep the files as CI artifacts, and aggregate them.

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  reporter: [
    ['list'],
    ['json', { outputFile: `reports/run-${process.env.GITHUB_RUN_ID ?? Date.now()}.json` }],
  ],
});

Then aggregate every report file into a per-test stability table:

// scripts/flakiness.mjs
// Usage: node scripts/flakiness.mjs reports
import { readdirSync, readFileSync } from 'node:fs';
import { join } from 'node:path';

const dir = process.argv[2] ?? 'reports';
const stats = new Map(); // key -> { runs, failures, flaky }

function record(key, result) {
  const entry = stats.get(key) ?? { runs: 0, failures: 0, flaky: 0 };
  entry.runs += 1;
  if (result.status === 'failed' || result.status === 'timedOut') entry.failures += 1;
  stats.set(key, entry);
}

function walk(suite, titlePath = []) {
  const path = suite.title ? [...titlePath, suite.title] : titlePath;
  for (const spec of suite.specs ?? []) {
    const key = [...path, spec.title].join(' > ');
    for (const test of spec.tests ?? []) {
      for (const result of test.results ?? []) record(key, result);
      if (test.status === 'flaky') {
        const entry = stats.get(key);
        if (entry) entry.flaky += 1;
      }
    }
  }
  for (const child of suite.suites ?? []) walk(child, path);
}

for (const file of readdirSync(dir).filter((f) => f.endsWith('.json'))) {
  const report = JSON.parse(readFileSync(join(dir, file), 'utf8'));
  for (const suite of report.suites ?? []) walk(suite);
}

const rows = [...stats.entries()]
  .map(([key, s]) => ({ test: key, ...s, rate: s.failures / s.runs }))
  .filter((r) => r.failures > 0)
  .sort((a, b) => b.rate - a.rate);

console.table(
  rows.slice(0, 20).map((r) => ({
    test: r.test.length > 70 ? '…' + r.test.slice(-69) : r.test,
    runs: r.runs,
    failures: r.failures,
    'fail %': (r.rate * 100).toFixed(1),
  })),
);

That gives you historical data. For a suspect test you can produce data on demand instead of waiting a week:

npx playwright test tests/checkout.spec.ts --grep "applies discount code" \
  --repeat-each=30 --workers=4 --retries=0 --reporter=line

Thirty runs with retries off. If three fail, you have a ~10% instability rate and a reproduction. If thirty pass, the instability lives in something you haven’t replicated locally — worker count, machine speed, or data left behind by other tests. Both outcomes are progress.

Track two numbers per suite over time: instability rate (failed runs ÷ total runs, excluding real regressions) and the number of tests contributing to it. A suite where one test causes all the noise is a different problem than a suite where forty tests each fail occasionally.

The Five Root Causes of Instability

Almost every flaky test traces back to one of five causes: timing, shared state, network dependencies, environment differences, and brittle selectors. The classification matters because each one has a different fix and, more usefully, a different diagnostic signature. Before touching code, ask:

• Does it fail at the same action every time, or at random points?
• Does it pass in isolation and fail inside the suite?
• Does it fail more at certain times of day, or after certain deploys?
• Does it pass locally and fail only in CI?
• Did it start failing right after a UI change?

The answers usually narrow it to one or two candidates.

Timing and Race Conditions

The most common cause: the test acts before or after the application is ready. page.waitForTimeout(2000) is the signature move — it works on your machine and fails on a loaded CI runner, or it makes every run three seconds slower for no reason.

Playwright’s auto-waiting covers actionability: before clicking, it waits for the element to be attached, visible, stable (not animating), enabled, and able to receive events. It does not know whether your framework has hydrated the component, whether a React event handler is bound yet, or whether the store finished loading. A server-rendered button is visible and enabled long before it does anything on click. That gap is where the race lives.

The fix is to assert on a state that can only exist after the application is ready, using web-first assertions that retry:

import { test, expect } from '@playwright/test';

test('adds item to cart', async ({ page }) => {
  await page.goto('/products/42');

  const addToCart = page.getByRole('button', { name: 'Add to cart' });

  // Wrong: the button exists immediately, the handler may not be attached.
  // await page.waitForTimeout(1000);
  // await addToCart.click();

  // Right: wait for a state that only exists after hydration/data load.
  await expect(addToCart).toBeEnabled();
  await expect(page.getByTestId('stock-status')).toHaveText(/In stock/);

  await Promise.all([
    page.waitForResponse(
      (res) => res.url().includes('/api/cart') && res.request().method() === 'POST',
    ),
    addToCart.click(),
  ]);

  await expect(page.getByTestId('cart-count')).toHaveText('1');
});

Note what changed: no fixed sleeps, and every wait is tied to an observable application state. If the assertion is wrong, the test fails with a message that tells you what it was waiting for.

Test Data and Shared State

If two tests log in as te**@*****le.com and both modify that user’s profile, they will collide the moment you raise --workers. Same for a shared order record, a shared feature flag, or a fixed database row.

Diagnostic signature: the test passes with --workers=1 and fails with --workers=4, or the result changes when the run order changes (--shuffle or simply running a single file). If that’s what you see, stop looking at selectors and look at data ownership.

The fix is per-test data, created by a fixture:

// tests/fixtures.ts
import { test as base, expect } from '@playwright/test';
import { randomUUID } from 'node:crypto';

type User = { email: string; password: string; id: string };

export const test = base.extend<{ user: User }>({
  user: async ({ request }, use) => {
    const email = `e2e-${randomUUID()}@example.test`;
    const password = 'Pw!' + randomUUID().slice(0, 12);

    const created = await request.post('/api/test/users', {
      data: { email, password },
    });
    expect(created.ok()).toBeTruthy();
    const { id } = await created.json();

    await use({ email, password, id });

    await request.delete(`/api/test/users/${id}`);
  },
});

export { expect };

// tests/profile.spec.ts
import { test, expect } from './fixtures';

test('user can change display name', async ({ page, user }) => {
  await page.goto('/login');
  await page.getByLabel('Email').fill(user.email);
  await page.getByLabel('Password').fill(user.password);
  await page.getByRole('button', { name: 'Sign in' }).click();

  await page.goto('/settings/profile');
  await page.getByLabel('Display name').fill('Ada Lovelace');
  await page.getByRole('button', { name: 'Save' }).click();

  await expect(page.getByRole('status')).toHaveText('Profile updated');
});

Every test now owns its data and cleans it up. Parallelism stops being a source of randomness. This is also what makes a suite scalable rather than merely large — a point worth reading alongside scalability practices for large automation projects.

Network and External Dependencies

Analytics scripts, chat widgets, CDN fonts, payment provider iframes, a staging backend that gets slow when someone runs a migration — any of these can turn a deterministic test into a coin flip.

Diagnostic signature: failures correlate with time rather than with code. Red builds cluster during office hours, during deploy windows, or whenever a third party has an incident. If your failure timestamps look like a work schedule, the problem is outside your application.

Decide, per dependency, whether it is under test. Your checkout flow needs your API to be real. It does not need Google Analytics, Intercom, or a font from a CDN. Block or stub everything that isn’t the subject of the test:

test.beforeEach(async ({ context }) => {
  await context.route(/(googletagmanager|google-analytics|intercom|hotjar)\./, (route) =>
    route.abort(),
  );
  await context.route('**/api/exchange-rates', (route) =>
    route.fulfill({
      status: 200,
      contentType: 'application/json',
      body: JSON.stringify({ base: 'EUR', rates: { USD: 1.1 } }),
    }),
  );
});

Where to draw the line between real and stubbed is a design decision, not a convenience one; mocking and stubbing in test automation covers the trade-off. The rule: stub what you don’t own, keep real what you’re validating.

Environment Differences: Green Locally, Red in CI

Your laptop has fast cores, a warm cache, a headed browser, your locale and timezone, and no other test process competing for CPU. The CI runner has none of that. Under CPU pressure, animations take longer, hydration takes longer, and a test that passed with 40 ms of slack now misses by 10 ms.

Don’t debug this by guessing. Reproduce CI conditions locally:

# Same image CI uses, CPU-constrained, same worker count, headless.
docker run --rm -it --cpus="1.0" --ipc=host \
  -v "$PWD":/work -w /work \
  mcr.microsoft.com/playwright:v1.49.0-noble \
  npx playwright test --workers=4 --repeat-each=20 --retries=0 \
    --grep "applies discount code"

Pin the browser image to the version in your CI config. Pin timezoneId, locale, and viewport in playwright.config.ts so they are identical everywhere — a test that renders a date differently in Europe/Istanbul and UTC is not flaky, it is under-specified. If the test only fails under --cpus="1.0", you have found a timing assumption, and you now have a fast way to verify the fix.

Brittle Selectors

Selectors tied to CSS classes, nth-child positions, or generated IDs (#mui-4821) break on every UI change. Usually that produces a consistently broken test, not a flaky one — which makes it easy to fix and easy to notice.

They become flaky when the DOM order isn’t deterministic: a list sorted by a timestamp, results returned in whatever order the backend felt like, a notification banner that appears only sometimes and shifts every index by one. Then nth-child(3) is right most of the time and wrong the rest.

// Brittle: depends on order and on generated class names.
await page.locator('.MuiTableRow-root:nth-child(3) .actions button').click();

// Stable: identifies the row by its content, the control by its role.
const row = page.getByRole('row').filter({ hasText: 'INV-2024-0042' });
await row.getByRole('button', { name: 'Download' }).click();

// When there is no accessible name to anchor on, use an explicit test id.
await page.getByTestId('invoice-row-INV-2024-0042').getByTestId('download').click();

Anchor on identity (invoice number, user email, order ID), never on position. If the DOM offers no identity, add a data-testid that encodes one — that’s a five-minute change in the application and it removes an entire class of failures.

Reading the Evidence Playwright Gives You

Playwright collects more diagnostic data than most teams use. Configure it to capture artifacts only where they’re worth the storage:

// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  timeout: 30_000,
  expect: { timeout: 5_000 },
  retries: process.env.CI ? 1 : 0,
  workers: process.env.CI ? 4 : undefined,
  use: {
    baseURL: process.env.BASE_URL ?? 'http://localhost:3000',
    timezoneId: 'UTC',
    locale: 'en-US',
    viewport: { width: 1280, height: 720 },
    trace: 'on-first-retry',      // trace only when something already failed
    video: 'retain-on-failure',
    screenshot: 'only-on-failure',
  },
  projects: [{ name: 'chromium', use: { ...devices['Desktop Chrome'] } }],
});

trace: 'on-first-retry' is the right default for CI: zero overhead on green runs, a full trace for exactly the runs you need to investigate. Open it with npx playwright show-trace trace.zip.

In the trace, look for four specific things:

1. Which action timed out. The failing action is highlighted with its full wait log — “waiting for element to be visible, enabled and stable” tells you which actionability check never passed.
2. The DOM snapshot at that moment. Before/after snapshots are live DOM, not screenshots. Was the element present but covered by an overlay? Present but disabled? Absent entirely?
3. Pending network requests. The network tab shows what was still in flight when the action gave up. A request stuck at pending points at a dependency, not at your selector.
4. The delta between the action’s before and after snapshots. If nothing changed, the click landed on a node with no handler attached — a hydration race.

The Reproduction Loop

Evidence gives you a hypothesis. The loop confirms it. The rule is simple: do not ship a fix for a failure you cannot reproduce on demand. Otherwise you’re changing code and waiting to see whether the noise goes away, which takes days and teaches you nothing.

# 1. Isolate. Does it fail alone, with no parallelism?
npx playwright test --grep "applies discount code" \
  --repeat-each=30 --workers=1 --retries=0

# 2. Add parallelism only. If it now fails -> shared state.
npx playwright test --grep "applies discount code" \
  --repeat-each=30 --workers=6 --retries=0

# 3. Add the whole suite back. If it fails only here -> order dependency.
npx playwright test --repeat-each=3 --workers=6 --retries=0 --shuffle

# 4. Add CPU pressure only. If it now fails -> timing assumption.
docker run --rm -v "$PWD":/work -w /work --cpus="0.5" --ipc=host \
  mcr.microsoft.com/playwright:v1.49.0-noble \
  npx playwright test --grep "applies discount code" \
    --repeat-each=30 --workers=1 --retries=0

One variable per step. Record the failure rate at each step — 0/30, 4/30, 11/30 — because that number is how you’ll know whether your fix worked. After the change, run the same command with the same repeat count. A fix that takes 11/30 to 0/30 is proven. A fix that takes it to 1/30 didn’t address the root cause; it moved the timing window. Long repeat loops are cheap if your suite is fast, which is one more reason to care about execution performance.

Quarantine: Damage Control, Not a Fix

Sometimes you can’t fix it today and the red build is blocking twelve people. Quarantine is legitimate — but only in a specific form.

Deleting the test destroys coverage silently. Writing test.skip() with no note is the same thing with extra steps: within a month nobody remembers why it’s skipped, and the feature it covered ships untested.

A quarantine that works has three properties:

1. An owner. A name, in the annotation, in the code.
2. A deadline. A date after which the quarantine expires and someone must decide: fix, rewrite, or delete the test on purpose.
3. Continued execution. The test keeps running in a separate, non-blocking CI job, so your flakiness data keeps accumulating. A quarantined test that stops running stops being diagnosable.

test('applies discount code at checkout', { tag: '@quarantine' }, async ({ page }) => {
  test.info().annotations.push({
    type: 'quarantine',
    description: 'owner: payments-team, since: 2025-01-14, review by: 2025-02-14',
  });
  // test body unchanged
});

Main pipeline runs --grep-invert @quarantine and blocks merges. A scheduled job runs --grep @quarantine --repeat-each=10 and reports without blocking. Add a check that fails the build when a quarantine annotation passes its review date — otherwise the list only grows, and a quarantine list that grows unbounded ends with a team that has quietly stopped testing.

Rules That Prevent Instability in the First Place

Everything you learn from one diagnosis should become a rule that stops the next one. Rules enforced in code review get forgotten; rules enforced in CI don’t.

Ban fixed sleeps with lint, not with good intentions:

// eslint.config.mjs
export default [
  {
    files: ['tests/**/*.ts'],
    rules: {
      'no-restricted-syntax': [
        'error',
        {
          selector: "CallExpression[callee.property.name='waitForTimeout']",
          message: 'No fixed sleeps. Use a web-first assertion or waitForResponse.',
        },
        {
          selector: "CallExpression[callee.property.name='$$']",
          message: 'Use locators (getByRole/getByTestId), not element handles.',
        },
      ],
      'playwright/no-conditional-in-test': 'error',
      'playwright/expect-expect': 'error',
    },
  },
];

Then make every new or modified test prove its stability before it can merge:

# .github/workflows/e2e.yml
name: e2e
on: pull_request

jobs:
  changed-tests-stability:
    runs-on: ubuntu-latest
    container: mcr.microsoft.com/playwright:v1.49.0-noble
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - run: npm ci
      - name: Collect changed spec files
        id: changed
        run: |
          FILES=$(git diff --name-only origin/${{ github.base_ref }}...HEAD \
            | grep -E '^tests/.*\.spec\.ts$' | tr '\n' ' ')
          echo "files=$FILES" >> "$GITHUB_OUTPUT"
      - name: Run changed tests 20 times
        if: steps.changed.outputs.files != ''
        run: npx playwright test ${{ steps.changed.outputs.files }} \
          --repeat-each=20 --workers=4 --retries=0

A new test that cannot pass twenty consecutive runs under parallelism does not enter the suite. This single gate prevents more flakiness than any amount of after-the-fact debugging, because it forces the author to confront timing and data assumptions while the code is still in their head.

The rest is discipline you already know: every test creates its own data through a fixture, no test depends on execution order, no test depends on a shared account, third-party requests are blocked by default. Your choice of tooling affects how easily you can enforce all of this — a comparison of popular automation tools is worth reading if you’re still deciding.

What to Do Next

Don’t try to stabilize the whole suite. Do this instead, this week:

1. Collect the JSON reports from the last week of CI runs and run the aggregation script above. You’ll get a ranked list of failure rates.
2. Take the top five. For each, open the trace from one failed run and classify it into one of the five root causes using the diagnostic questions.
3. Pick the one with the highest failure rate — not the one that looks easiest — and put it through the reproduction loop until you can trigger the failure on demand with a known rate.
4. Fix one variable. Re-run the same command with the same repeat count. Confirm the rate is zero.

One test, root cause understood, fix proven by measurement. Then the next one. A suite stabilizes test by test, and each diagnosis you finish makes the next one faster, because the same five causes keep coming back.