Adding AI to a Playwright + TypeScript Setup: Where It Helps and Where It Doesn’t

Every automation team is being asked the same question this year: can AI write our end-to-end tests? The honest answer is that it can write some of them, badly, and it can do a different job — one nobody asks about — very well. The gap between those two outcomes is not about model quality. It is about which part of the work you handed over.

This post draws the line in a concrete Playwright + TypeScript setup: what to automate with a model, what to keep in human hands, and which guardrails to put in CI before AI-generated code reaches main.

The problem AI actually solves in test automation

Writing a test is two jobs stapled together. The first is mechanical production: scaffolding a spec file, declaring the fixture types, building a factory for a twelve-field form, repeating the same four assertions across six variants. The second is decision-making: what deserves an end-to-end test at all, whether this failure is a real regression or an environment artifact, whether two tests can safely run in parallel against the same database.

A language model is good at the first job because the output is constrained, verifiable and immediately checked by the TypeScript compiler. It is unreliable at the second because those decisions depend on your risk model, your data model and your infrastructure — none of which fit in a prompt, and most of which are not written down anywhere. Treat those as one task (“let AI write the tests”) and you get volume without judgement: a suite that grows faster than it earns its runtime.

Where AI plugs into a Playwright project

There are three distinct integration points, and lumping them together is what makes AI adoption feel risky without anyone being able to say why. The real variable is how much write authority each layer has over code that ends up in the repository.

In-editor completion: the lowest-risk layer

Copilot-style completion is the most controlled layer, for a boring reason: its output is read by a human the moment it appears, and it cannot reach main without passing review. The model proposes, the human commits.

TypeScript raises the hit rate here more than any prompt trick. When @playwright/test types are in scope, hallucinated API calls — page.waitForElement, expect(locator).toBeDisplayed() — fail at compile time instead of surviving into a spec file that nobody runs until next week. This is a good argument for strict tsconfig settings in a test repo even if the product code is looser: the type system is doing free review work on machine-generated code.

Browser-driving agents and MCP

An agent connected to a live browser — Playwright’s MCP server is the common path — can read the page’s accessibility tree, click through a flow and emit a scenario it just performed. For exploration and first drafts this is genuinely useful, especially on an application you do not know yet: the agent discovers the actual role and name of each control instead of guessing from a DOM dump.

The failure mode is treating the agent session as the test. An agent run is not deterministic, not reviewable and not reproducible in CI. If you do not convert the output into a committed spec file with explicit locators and assertions, you are re-inventing the test on every run, and a test you re-invent cannot detect a regression — it just describes whatever the app does today.

The post-CI analysis layer

Feeding traces, error stacks and screenshots to a model after the run is the highest-value use with the lowest risk, because this layer has no write access to anything. It reads artifacts only and produces suggestions; the decision stays with a human.

It works: repetitive scaffolding and data factories

Writing a typed factory for a twelve-field checkout form and a fixture that creates and deletes the record takes real minutes, and requires zero decisions once you have chosen the pattern. This is exactly the work to delegate: the shape is obvious, the output is typed, and anything the model invents breaks the build.

// tests/factories/customer.ts
import { randomUUID } from 'node:crypto';

export type CustomerDraft = {
  email: string;
  firstName: string;
  lastName: string;
  company: string;
  vatId: string;
  street: string;
  houseNumber: string;
  postalCode: string;
  city: string;
  country: string;
  phone: string;
  newsletterOptIn: boolean;
};

export function makeCustomerDraft(
  overrides: Partial<CustomerDraft> = {},
): CustomerDraft {
  const unique = randomUUID().slice(0, 8);

  return {
    email: `qa.${unique}@example.test`,
    firstName: 'Ada',
    lastName: `Lovelace-${unique}`,
    company: `Acme ${unique}`,
    vatId: `DE${unique.replace(/\D/g, '0').padEnd(9, '0')}`,
    street: 'Hauptstrasse',
    houseNumber: '42',
    postalCode: '10115',
    city: 'Berlin',
    country: 'DE',
    phone: '+49 30 1234567',
    newsletterOptIn: false,
    ...overrides,
  };
}

// tests/fixtures/customer.ts
import { test as base } from '@playwright/test';
import { makeCustomerDraft, type CustomerDraft } from '../factories/customer';

type CustomerFixtures = {
  customer: CustomerDraft & { id: string };
};

export const test = base.extend<CustomerFixtures>({
  customer: async ({ request }, use) => {
    const draft = makeCustomerDraft();

    const response = await request.post('/api/customers', { data: draft });
    if (!response.ok()) {
      throw new Error(
        `Customer seed failed: ${response.status()} ${await response.text()}`,
      );
    }
    const { id } = (await response.json()) as { id: string };

    await use({ ...draft, id });

    await request.delete(`/api/customers/${id}`);
  },
});

export { expect } from '@playwright/test';

// tests/checkout/address.spec.ts
import { test, expect } from '../fixtures/customer';

test('saved billing address is prefilled in checkout @new', async ({
  page,
  customer,
}) => {
  await page.goto(`/checkout?customerId=${customer.id}`);

  await expect(page.getByRole('textbox', { name: 'Email' })).toHaveValue(
    customer.email,
  );
  await expect(page.getByRole('textbox', { name: 'Postal code' })).toHaveValue(
    customer.postalCode,
  );
  await expect(
    page.getByRole('checkbox', { name: 'Subscribe to newsletter' }),
  ).not.toBeChecked();
});

The pattern is yours; the twelve fields are the model’s. That is the right split.

It works: triaging a failed run

When a 200-test run comes back with 7 failures, the first question is not “why did test 43 fail” but “how many distinct causes am I looking at”. Grouping similar error messages, stack frames and failing steps is pattern work, and models are good at it. Paste the seven error messages plus a trace summary and ask how many root causes they represent; you will usually open two traces instead of seven.

What you get back is a hypothesis, not evidence. “All five failures happen on the same getByRole('button', { name: 'Continue' }) after a deploy that renamed the button” is a lead you confirm by opening one trace. Treat it as a conclusion and you will close a real bug as “flaky infrastructure” at some point — which is the most expensive mistake in this entire article.

Partially works: locator suggestions

Given a DOM snapshot, a model will usually propose a getByRole locator, and that is a reasonable starting point. The problem appears on interfaces with poor accessibility semantics: with no roles and no accessible names to work with, the model falls back to brittle CSS or XPath and presents it with exactly the same confidence as a good suggestion.

// Rejected: structural coupling, breaks on any layout refactor
await page.locator('div.cart > div:nth-child(3) > button.btn-primary').click();
await page.locator('//*[@id="root"]/div[2]/form/div[4]/input').fill('10115');

// Accepted: role-based contract, readable by a human reviewer
await page.getByRole('button', { name: 'Proceed to payment' }).click();
await page.getByRole('textbox', { name: 'Postal code' }).fill('10115');

// Accepted fallback when the UI has no usable semantics:
// add the attribute to the product, then lock the test to it
await page.getByTestId('checkout-submit').click();

The rule is simple: never accept a machine-suggested locator as-is. Convert it into a contract first — a role plus accessible name, or a data-testid you add to the product code. If the UI cannot support either, you have found an accessibility bug, and the correct output of that moment is a ticket, not an XPath.

Doesn’t work: the root cause of a flaky test

Flakiness does not live inside a single run. It lives in the difference between runs: a race between a network response and an assertion, shared state left behind by a sibling test, a timer, a retry at the proxy layer. A model handed one trace sees one timeline, and the statistically most common fix in its training data is “add a wait”. That suppresses the symptom and hides the cause — the next person sees a green test with a three-second sleep and no idea why it exists.

Real diagnosis requires repeated runs, running the test in isolation versus in a parallel shard, and removing one variable at a time until the failure rate changes. None of that is prompt work; it is experimental procedure. The method is covered in diagnosing flaky tests in Playwright, and nothing in it gets shorter because a model is in the room.

Doesn’t work: designing test isolation

Deciding that every test creates and destroys its own data is an architectural call with several sub-decisions: which entities are seeded through the API because the UI path is slow and irrelevant, which must go through the UI because the UI is what you are testing, which records collide when ten workers run at once, and what cleanup looks like when a run is killed mid-flight.

A model does not know your data model, your environment constraints or which account is shared with the staging demo. Generated tests therefore lean on fixed data that already exists — a login that “should” be there, an order ID someone hard-coded last quarter. They pass once, locally, and collapse under --workers=4. The reasoning behind the alternative is in test data management: why every test must own its data; the fixture above is what enforcing it looks like in code.

Doesn’t work: deciding what to test

Scope is a risk judgement. Which flow loses money when it breaks? Which bug is visible to a customer within minutes? Which scenario is already covered by a unit test and only adds runtime at the E2E level? Those answers come from knowing the product, the incident history and the cost of a slow pipeline.

A model will happily derive scenarios from acceptance criteria — including the three that belong in unit tests and the two that duplicate an existing spec. That is why AI-assisted suites bloat so quickly: generation is cheap, deletion requires someone to argue for it, and the regression suite quietly grows from eight minutes to twenty-five. Scope decisions stay with the person who owns the pipeline’s runtime budget.

Why the self-healing locator promise is dangerous

Self-healing locators re-bind to an element when the original selector stops matching, keeping the test green after a button moves or gets renamed. The pitch is reduced maintenance. The flaw is categorical: sometimes that button moving is the bug. A checkout button that silently relocated into a collapsed menu is a conversion incident, and a self-healing suite reports it as a pass.

A test’s job is not to adapt to change. It is to report change and let a human decide whether the change was intended. If you use self-healing at all, cap it at advisory level: let it raise a warning and open a suggested diff, never let it silently repair and move on. A healing event should cost someone thirty seconds of attention — that cost is the entire value of the test. The broader framing of what AI can and cannot own in a test process is in AI-powered test automation.

Guardrails before merging AI output

Manual review is the first thing to break when generation volume goes up, so do not rely on discipline. Encode the rules as lint errors and CI steps.

// eslint.config.js
import tseslint from 'typescript-eslint';

export default tseslint.config({
  files: ['tests/**/*.ts'],
  rules: {
    'no-restricted-syntax': [
      'error',
      {
        selector: "CallExpression[callee.property.name='waitForTimeout']",
        message: 'No fixed waits. Use web-first assertions or waitForResponse.',
      },
      {
        selector:
          "CallExpression[callee.property.name='locator'] > Literal:not([value=/^\\[data-testid=/])",
        message:
          'No raw CSS/XPath. Use getByRole/getByLabel/getByTestId, or [data-testid=...].',
      },
    ],
    'no-restricted-imports': [
      'error',
      {
        paths: [
          {
            name: '@playwright/test',
            importNames: ['test'],
            message:
              'Import test from tests/fixtures so every test owns its data.',
          },
        ],
      },
    ],
  },
});

# .github/workflows/e2e.yml
name: e2e
on: pull_request

jobs:
  suite:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: Lint test rules
        run: npx eslint tests --max-warnings=0
      - name: Full suite
        run: npx playwright test --workers=4
      - name: New tests must pass three times in a row
        run: npx playwright test --grep "@new" --repeat-each=3 --workers=4

Four rules, all machine-enforced: no fixed waits, no raw CSS or XPath locators, every spec imports test from a fixture that owns its data, and anything tagged @new must survive three consecutive runs before it is allowed to join the suite. A model cannot talk its way past a failing CI step.

What to do next

Measure before you adopt. Record three numbers for two weeks: median time to write a new spec, flaky rate (tests that changed verdict without a code change), and total suite runtime. Without a baseline you cannot tell a tooling gain from a quieter sprint.

Then introduce AI at exactly two points — scaffolding and triage — and leave locator design, isolation and scope selection in human hands. Put the lint rules and the --repeat-each=3 gate in place first, because they are what keeps generated volume from turning into maintenance debt. Measure the same three numbers again after two weeks. If nothing moved, the tool is not paying for itself, no matter how convincing the demo was. If you are still choosing a framework underneath all this, the trade-offs are laid out in comparing Playwright, Cypress and Selenium on real criteria.