Running Tests in Parallel on CI: Shards, Workers and Merging Reports

Why CI Duration Is the First Wall You Hit

A browser test suite grows the way application features grow: one spec file at a time. Nothing feels slow on the day you add it. Then a pull request sits for 40 minutes waiting on a green check, people start merging on faith, and someone proposes “deleting the slow tests.”

The problem is almost never that the tests are slow. A login flow that takes 9 seconds is a 9-second flow; no amount of selector tuning turns it into 2. The problem is that 300 such tests run sequentially on one machine, so total wall-clock time is the sum of every test. Parallelism attacks the sum. It has two independent axes, and confusing them is where most CI configurations go wrong:

• Inside one machine: Playwright spawns multiple worker processes, each with its own browser instance.
• Across machines: the suite is split into shards, and each shard runs as a separate CI job on its own runner.

You need both, in that order, and for different reasons.

Workers and Shards Are Not the Same Thing

A worker is a Node process on a single machine. Four workers means four Chromium instances competing for the same CPU, the same RAM, the same network interface and the same page cache. Scaling workers is cheap — no extra runner, no extra checkout, no extra npm ci — but it saturates fast. Past the core count you stop gaining time and start producing timeouts, because every worker’s expect is waiting on a machine that is busy context-switching.

A shard is a slice of the test list handed to a separate CI job. --shard=2/4 means “run the second quarter of the suite.” Each shard pays a full setup tax: runner provisioning, dependency install, browser download. In exchange it gets its own CPU, its own memory and total process isolation from the other shards.

So workers=4 and shard=4 produce the same nominal parallelism with completely different cost and isolation profiles. Pick by constraint:

• CPU cores and RAM decide the worker ceiling. A headless Chromium instance under load wants roughly a core and a few hundred MB; two cores do not host six workers.
• Setup cost decides whether another shard is worth it. If install plus browser download is 3 minutes, a 4-minute suite should not be cut into ten pieces.
• Your application’s load tolerance decides the absolute ceiling for both. Forty concurrent browsers against a single staging database, a rate-limited auth provider or a sandbox payment API will produce failures that look like test bugs and are not.

Choosing the Right Worker Count on a Single Machine

The standard GitHub Actions Linux runner gives you 2 vCPUs. Your laptop gives you 8 or 10 performance cores. If playwright.config.ts leaves workers undefined, Playwright defaults to half the core count — which means a different number locally and on CI, and on CI that number is derived from hardware you don’t control.

Leaving CI workers high is worse than leaving them at one. Over-subscribed workers make every timeout non-deterministic: the same test passes on a quiet runner and fails on a loaded one. That is manufactured flakiness, and it will cost you more hours than the parallelism saved. Be explicit.

// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';

const isCI = !!process.env.CI;

export default defineConfig({
  testDir: './tests',

  // Run tests inside a single file in parallel, not just across files.
  fullyParallel: true,

  // Two vCPUs on a standard runner: two workers, no more.
  workers: isCI ? 2 : undefined,

  retries: isCI ? 1 : 0,

  // Fail the build if someone leaves test.only in a spec.
  forbidOnly: isCI,

  timeout: 30_000,
  expect: { timeout: 7_000 },

  reporter: isCI
    ? [['blob'], ['github']]
    : [['html', { open: 'never' }], ['list']],

  use: {
    baseURL: process.env.BASE_URL ?? 'http://localhost:3000',
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
  },

  projects: [
    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
  ],
});

fullyParallel: true is the setting people forget. Without it, Playwright parallelizes files but runs the tests inside one file serially. A spec with 20 tests then becomes a single-threaded block that no amount of sharding can break up. With it, every test is schedulable independently — which also means every test must be independently runnable. That constraint is the subject of a later section, and it is a feature.

Wiring Shards Into the CI Matrix

--shard=i/n splits the suite deterministically. The GitHub Actions matrix strategy turns those slices into jobs. Each shard writes a blob report — an intermediate, machine-readable format designed exactly for later merging — and uploads it as an artifact.

# .github/workflows/e2e.yml
name: E2E Tests

on:
  pull_request:
  push:
    branches: [main]

jobs:
  test:
    name: Shard ${{ matrix.shardIndex }} of ${{ matrix.shardTotal }}
    runs-on: ubuntu-latest
    timeout-minutes: 30
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm

      - name: Install dependencies
        run: npm ci

      - name: Cache Playwright browsers
        uses: actions/cache@v4
        id: playwright-cache
        with:
          path: ~/.cache/ms-playwright
          key: playwright-${{ runner.os }}-${{ hashFiles('package-lock.json') }}

      - name: Install Playwright browsers
        if: steps.playwright-cache.outputs.cache-hit != 'true'
        run: npx playwright install --with-deps chromium

      - name: Install system deps (cache hit path)
        if: steps.playwright-cache.outputs.cache-hit == 'true'
        run: npx playwright install-deps chromium

      - name: Run tests
        run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
        env:
          BASE_URL: ${{ secrets.BASE_URL }}

      - name: Upload blob report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: blob-report-${{ matrix.shardIndex }}
          path: blob-report
          retention-days: 3

Three details that are not decorative. fail-fast: false keeps the other shards running when one fails — otherwise you see one failure and lose the information about the rest. The browser cache is keyed on the lockfile hash, so it invalidates when the Playwright version changes; a stale browser binary against a new client is a confusing class of failure. And if: always() on the upload step is what makes failed runs debuggable, since that is precisely when you need the report.

Shard Imbalance and How to Fix It

Playwright distributes shards by test file, not by duration. It has no way to know that checkout.spec.ts takes 11 minutes and footer-links.spec.ts takes 20 seconds. The typical result is a matrix where one job finishes in 3 minutes and another takes 12 — and since the pipeline is only green when the last job finishes, your suite’s duration equals your slowest shard. The three fast shards bought you nothing except four invoices.

The fix is boring and it works:

1. Measure before you tune. Every shard’s job duration is already in the CI run summary, and the merged HTML report gives per-file timings. Imbalance is visible in ten seconds of looking; it is invisible if you never look.
2. Split the heavy spec files. A 40-test checkout.spec.ts is one indivisible unit to the shard algorithm. Four files of 10 tests are four schedulable units. This is also better suite design — large spec files tend to accumulate shared setup that quietly couples tests together.
3. Redistribute based on the previous run’s timings. Keep slow files from clustering. If you don’t want to maintain this by hand, Playwright’s --shard can be combined with a custom test ordering, but splitting files is the lower-maintenance option for most suites.
4. Stop adding shards once the slowest file dominates. If one file takes 6 minutes, no shard count brings the suite under 6 minutes. At that point the work is in the test, not the pipeline.

More on keeping a growing suite tractable: test otomasyonunda ölçeklenebilirlik.

Merging Reports: Blob Reporter and merge-reports

If each shard produces its own HTML report, you end up with four partial reports and no view of the suite. Nobody opens four artifacts. People start judging the run by the red/green of the job list, which throws away every trace, screenshot and timing you paid to collect.

The blob reporter exists for this. Each shard writes an intermediate archive; a separate job downloads all of them and merges them into one real report.

  merge-reports:
    name: Merge reports
    if: always()
    needs: [test]
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm

      - name: Install dependencies
        run: npm ci

      - name: Download blob reports
        uses: actions/download-artifact@v4
        with:
          path: all-blob-reports
          pattern: blob-report-*
          merge-multiple: true

      - name: Merge into HTML and JUnit
        run: |
          npx playwright merge-reports \
            --reporter html,junit \
            ./all-blob-reports

      - name: Upload merged HTML report
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report
          path: playwright-report
          retention-days: 14

      - name: Upload JUnit XML
        uses: actions/upload-artifact@v4
        with:
          name: junit-results
          path: results.xml
          retention-days: 14

The merged HTML report shows the suite the way it would look if it had run on one machine: a single pass/fail/flaky summary, per-test traces, and the real distribution of durations. That duration view is the input for the imbalance work in the previous section, so set this up before you start tuning shard counts. The JUnit output is the format most CI dashboards and test-management tools ingest, which is why it is worth emitting alongside the HTML.

Hidden Dependencies That Parallelism Exposes

The first parallel run after a sequential one usually fails in places nobody touched. The instinct is “parallelism broke the tests.” It didn’t. Parallelism removed the accidental ordering that was hiding the coupling.

The usual suspects: one shared test account whose session is invalidated when a second worker logs in, a fixed record ID that two tests mutate, a “create then assert then delete” trio spread over three tests, a counter in the database that one test asserts an exact value on.

Every test must create its own data and carry its own authenticated state. A fixture is the right place for both:

// tests/fixtures/auth.ts
import { test as base, expect } from '@playwright/test';
import { randomUUID } from 'node:crypto';

type TestUser = {
  email: string;
  password: string;
  id: string;
};

type Fixtures = {
  user: TestUser;
};

async function createUser(): Promise<TestUser> {
  const email = `e2e-${randomUUID()}@example.test`;
  const password = 'Test-' + randomUUID().slice(0, 12);

  const response = await fetch(`${process.env.API_URL}/test/users`, {
    method: 'POST',
    headers: {
      'content-type': 'application/json',
      authorization: `Bearer ${process.env.API_TOKEN}`,
    },
    body: JSON.stringify({ email, password }),
  });

  if (!response.ok) {
    throw new Error(`User creation failed: ${response.status}`);
  }

  const { id } = await response.json();
  return { email, password, id };
}

export const test = base.extend<Fixtures>({
  user: async ({ page }, use) => {
    const user = await createUser();

    await page.goto('/login');
    await page.getByLabel('Email').fill(user.email);
    await page.getByLabel('Password').fill(user.password);
    await page.getByRole('button', { name: 'Sign in' }).click();
    await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible();

    await use(user);

    await fetch(`${process.env.API_URL}/test/users/${user.id}`, {
      method: 'DELETE',
      headers: { authorization: `Bearer ${process.env.API_TOKEN}` },
    });
  },
});

export { expect };

Creating the user over the API rather than through the signup UI keeps the cost per test at one HTTP round trip instead of a full browser flow. The deeper discussion of why each test owning its data is non-negotiable: test verisi yönetimi. For failures that survive this cleanup, the diagnosis path is a separate skill: flaky testleri teşhis etmek.

Retries, Flaky Reporting and a False Sense of Safety

retries: 2 on CI turns red builds green. That is its appeal and its danger. A test that fails once and passes twice is reported as flaky, not failed, and flaky does not block a merge. Set the number high enough and your pipeline stops telling you anything.

Treat retries as a debt instrument, not a fix. One retry absorbs genuine infrastructure noise — a runner hiccup, a cold start — without masking much. Two or more starts hiding real race conditions in the application, which is exactly the category of bug your E2E suite exists to catch.

So instrument the debt:

• Read the flaky count from the merged report as its own metric, separate from pass and fail. The merged summary reports it explicitly; a per-shard report cannot, because a retry may land in a different worker.
• Review the flaky list weekly and attribute each entry to a cause: test coupling, application race, or infrastructure. Unattributed flakiness compounds.
• Set a threshold and enforce it. If more than a small, agreed number of tests pass only on retry, fail the build. A quality gate nobody can breach is not a gate.

trace: 'on-first-retry' in the config above exists for this loop: the retry that produced the green result also produced the trace file that explains why the first attempt failed.

Cost: How Many Shards Actually Make Sense

Every shard job pays a fixed tax before a single test runs: runner provisioning, checkout, npm ci, browser install. Call that tax S and your sequential test time T. Splitting into n shards gives you roughly:

wall clock ≈ S + (T / n) + merge job time, and billed minutes ≈ n × (S + T/n) + merge.

The arithmetic is unforgiving at small T. If setup is 3 minutes and the suite is 4 minutes, one shard costs 7 minutes of wall clock and 7 billed minutes. Ten shards cost about 3.4 minutes of wall clock plus a merge job, and roughly 34 billed minutes. You paid nearly five times the money to save three minutes, and the merge job gave part of that back.

The usable rule: sharding pays when T is several times larger than S, and you stop adding shards when T/n approaches S. Below that point you are mostly paying for runners to boot. Caching the browser binaries and the npm cache lowers S directly, which raises the shard count that makes sense — tune setup before you tune the matrix. More ways to buy time back inside the suite itself: performans optimizasyonu.

What to Do Next

Do it in this order, because each step changes the measurement the next step depends on.

1. Measure the current distribution. Record total wall-clock time and, if you already shard, per-shard duration. Without this number every later change is a guess.
2. Fix the worker count to the CI hardware. Two vCPUs, two workers. Set it explicitly in playwright.config.ts rather than inheriting a default derived from the runner.
3. Add sharding with a modest matrix — four is a sane starting point — and wire the blob reporter from day one.
4. Add the merge job before you tune anything else, so you have one report with real per-test timings.
5. Rebalance using those timings, splitting heavy spec files until the slowest shard stops dominating.
6. Track flaky counts weekly and enforce a threshold.

Inverting this order is the common mistake: people add ten shards first, then discover their tests share a user account, then add retries to cover it, and end up with an expensive pipeline nobody trusts. Isolation and measurement come before parallelism. Parallelism only multiplies what is already there.