What Breaks Your Regression Suite Is Data, Not Tests
When a nightly regression run comes back with fourteen red tests and the code hasn’t changed, the cause is almost never the tests. It’s the state of the database they ran against. A test that passed yesterday now fails because another test consumed the coupon code it expected, because a colleague manually cancelled the order that test looks up by ID, or because a previous run left a user in a state nobody designed for.
This is the specific problem: shared, uncontrolled state creates dependencies between tests that don’t exist in the source code. Test A works because test B ran first and created something. Nothing in the diff shows that. Code review cannot catch it, because there is no code to review — the coupling lives in the database. And the moment you try to run those tests in parallel or in a different order, the hidden contract collapses. Test data management is the practice of making every dependency explicit and owned by the test that needs it.
Three Sources of Test Data and What Each Actually Costs
There are exactly three ways data gets into your test environment, and they have very different bills.
Static fixtures are files checked into the repository — SQL seeds, JSON dumps, CSV rows loaded before the suite starts. Setup cost is low: write it once, load it once. Fragility is high the moment tests mutate it, because a mutation in test 3 becomes an input to test 40. Privacy risk is zero if you authored the data yourself. Maintenance cost is proportional to schema churn: every column with a NOT NULL constraint added upstream breaks every fixture file at once.
Factories build data at runtime, usually through the application’s own API or ORM. Setup cost is the highest of the three — you have to write and maintain code. Fragility is the lowest, because each test owns what it created and nothing is shared. Privacy risk is zero. Maintenance cost is real but centralized: a new required field is one default in one factory function, not a search-and-replace across forty fixture files.
Production clones copy a real dataset into a lower environment. Setup cost looks low (“just restore the dump”) and is in fact the highest once you count the anonymization pipeline. Fragility is high because the data changes underneath you every refresh. Privacy risk is the whole GDPR surface. Maintenance cost never stops.
The allocation that works: fixtures for reference data at every level; factories for transactional data in integration, API and end-to-end tests; production clones only for performance and migration testing, and never for functional assertions.
When a Static Fixture Is the Right Call
A fixture is correct when the data is not the subject of the test but part of the environment. Country lists, currency codes, VAT rates, product categories, role and permission definitions, feature flag defaults. No test creates a country. No test deletes one. This data is read-only by nature, it has to exist before the application boots, and every test in the suite can share it without any risk of interference.
Two rules make this work. First, reference fixtures must be loaded by the same mechanism the application uses in production — the real migration or seed command, not a test-only script. Otherwise your tests validate a schema that doesn’t ship. Second, no test may write to reference data. If a test needs a VAT rate of 27% and your fixture has 19%, it does not update the fixture row; it creates its own rate record, or the test belongs at unit level with the rate injected.
Using fixtures for transactional data — orders, invoices, subscriptions, users with balances — guarantees collisions. The fixture user has one password, one email, one state. Two tests that both log in as fi**********@*****le.com and change their profile will interfere, and the interference will be intermittent because it depends on timing.
The Factory Pattern: Every Test Creates Its Own Data
A factory fills in every field the system requires with a valid default, and lets the test override only the fields it actually cares about. That’s the whole idea, and its main benefit is readability: when a test reads data.customer({ plan: 'enterprise' }), you know the plan matters and nothing else does. A twenty-line JSON payload inline in the test tells you nothing about intent.
Uniqueness needs a strategy, not luck. Random UUIDs in every field work but make failures unreadable. The pattern that holds up is a run identifier plus a monotonic counter: one ID per suite execution, incremented per object. You get global uniqueness across parallel runs and a value you can grep for in application logs when something fails.
Prefer building data through the API over direct database inserts. The API applies validation, defaults, derived fields and side effects (audit rows, search indexing, outbox events) that your INSERT statement will silently skip. Data created by raw SQL is frequently in a state the application can never produce, and you end up debugging tests that fail against perfectly correct code. Direct DB access is the fallback for states the API genuinely cannot create — an expired trial, a soft-deleted record, a legacy row shape.
// tests/support/run-context.ts
import { randomBytes } from 'node:crypto';
// One ID per suite execution. Set it in CI so all workers share the prefix.
const RUN_ID = process.env.TEST_RUN_ID ?? randomBytes(4).toString('hex');
let counter = 0;
export function runId(): string {
return RUN_ID;
}
// RUN_ID scopes the run, pid scopes the worker, counter scopes the object.
export function uniqueSuffix(): string {
counter += 1;
return `${RUN_ID}-${process.pid}-${counter}`;
}
// tests/support/factories/customer.ts
import type { APIRequestContext } from '@playwright/test';
import { uniqueSuffix } from '../run-context';
export type CustomerInput = {
email: string;
name: string;
country: string;
plan: 'free' | 'pro' | 'enterprise';
creditLimit: number;
};
export type Customer = CustomerInput & { id: string };
export function customerInput(
overrides: Partial<CustomerInput> = {},
): CustomerInput {
const suffix = uniqueSuffix();
return {
email: `customer+${suffix}@test.example.com`,
name: `Customer ${suffix}`,
country: 'DE',
plan: 'pro',
creditLimit: 1000,
...overrides,
};
}
export async function createCustomer(
api: APIRequestContext,
overrides: Partial<CustomerInput> = {},
): Promise<Customer> {
const payload = customerInput(overrides);
const response = await api.post('/api/customers', { data: payload });
if (!response.ok()) {
throw new Error(
`createCustomer failed: ${response.status()} ${await response.text()}`,
);
}
const body = await response.json();
return { ...payload, id: body.id };
}
Build Setup Through the API, Not the UI
An end-to-end test that clicks through registration, email confirmation, a wizard and three forms just to reach the screen it wants to assert on is not testing those steps — it’s paying for them. It runs slow, and worse, it fails for reasons unrelated to its purpose. A change to the signup form breaks forty tests that were never about signup.
Preconditions belong in the API layer. Only the behavior under verification goes through the interface. The pattern below wires factories into Playwright fixtures and tracks everything created so it can be removed in reverse order.
// tests/support/fixtures.ts
import { test as base, request, type APIRequestContext } from '@playwright/test';
import { createCustomer, type Customer, type CustomerInput } from './factories/customer';
import { createInvoice, type Invoice, type InvoiceInput } from './factories/invoice';
type Created = { path: string; id: string };
type DataFactory = {
customer(overrides?: Partial<CustomerInput>): Promise<Customer>;
invoice(overrides: Partial<InvoiceInput> & { customerId: string }): Promise<Invoice>;
};
export const test = base.extend<{ api: APIRequestContext; data: DataFactory }>({
api: async ({ playwright }, use) => {
const api = await request.newContext({
baseURL: process.env.API_BASE_URL,
extraHTTPHeaders: {
Authorization: `Bearer ${process.env.API_TEST_TOKEN}`,
},
});
await use(api);
await api.dispose();
},
data: async ({ api }, use) => {
const created: Created[] = [];
const factory: DataFactory = {
async customer(overrides = {}) {
const customer = await createCustomer(api, overrides);
created.push({ path: '/api/customers', id: customer.id });
return customer;
},
async invoice(overrides) {
const invoice = await createInvoice(api, overrides);
created.push({ path: '/api/invoices', id: invoice.id });
return invoice;
},
};
await use(factory);
// Reverse order: children before parents, so foreign keys stay valid.
for (const item of created.reverse()) {
await api.delete(`${item.path}/${item.id}`).catch(() => undefined);
}
},
});
export { expect } from '@playwright/test';
// tests/invoices/overdue-badge.spec.ts
import { test, expect } from '../support/fixtures';
test('an open invoice past its due date is shown as overdue', async ({ page, data }) => {
const customer = await data.customer({ plan: 'enterprise' });
const invoice = await data.invoice({
customerId: customer.id,
dueDate: '2020-01-01',
status: 'open',
});
await page.goto(`/customers/${customer.id}/invoices`);
const row = page.getByRole('row', { name: invoice.number });
await expect(row).toContainText('Overdue');
});
The test is six lines of intent. If the invoice list page breaks, this test fails. If the signup flow breaks, it doesn’t.
Cleanup Strategies: Deletion, Rollback and Isolated Schemas
Reverse-order deletion, as in the fixture above, is the most portable option. It works against any environment you can reach over HTTP and needs no database privileges. Its weaknesses: a crashed worker leaves orphans, and cascade rules you don’t control can delete more than you intended. Parallel-safe, because each worker only removes what it created.
Transaction rollback — open a transaction in setup, roll it back in teardown — is the fastest and cleanest option, and it only works when the test and the code under test share one database connection. That means in-process integration tests. It is unavailable for end-to-end tests against a deployed application, and it hides anything your code does outside the transaction, such as publishing events or writing to another store.
A separate schema or tenant per run gives you real isolation at the cost of infrastructure. You need migrations that run fast enough to provision on demand, and an application that can be pointed at a schema or tenant at request level. When it’s available, it is the best answer: teardown is a single DROP SCHEMA, and worker count is limited only by your database.
No cleanup at all is a legitimate strategy, not laziness — if and only if every object carries a unique identifier and no test queries by anything but its own IDs. You accumulate garbage, so you need a periodic reset of the environment and enough storage headroom. It is the most parallel-friendly option, because nothing is ever deleted and no two workers can race. It breaks down the moment a test asserts on a list count or a “latest record” query.
The Bill for Cloning Production Data
A production copy buys realism: real cardinalities, real edge cases, real data shapes nobody would have invented. Then it sends four invoices.
Personal data liability. Under GDPR, a copy in your test environment is processing. You need a lawful basis, the same access controls as production, deletion within retention limits, and a place in your records of processing. A restored dump on a developer laptop is a reportable breach waiting for a lost device.
The anonymization pipeline is software. It has a schema dependency, transformation rules per column, referential integrity constraints and no test suite of its own. Somebody owns it, and that ownership never ends. It typically decays quietly: a new column ships, nobody adds a rule, and real data flows into the test environment unnoticed until an audit finds it.
Refresh frequency fights test stability. Refresh often and your assertions break, because the data they depend on changed. Refresh rarely and the “realism” argument disappears — you’re testing against a dataset that no longer resembles production, while still carrying the privacy risk.
Schema changes break the copy. Every migration has to be applied to the clone and to the anonymization rules. Two artifacts to keep in sync instead of one.
Production data earns its cost in exactly two places: performance testing, where volume and distribution are the subject, and migration testing, where you need to know whether your ALTER TABLE survives twelve years of accumulated weirdness. Neither of those needs stable, named records — which is precisely why they’re a fit.
Masking Is Not Enough — You Need Synthetic Generation
Masking replaces sensitive values in place: jo**@**me.com becomes us******@*****le.com, a name becomes a random name. It has two failure modes.
The first is re-identification. Masking single fields leaves combinations intact. A postcode, a birth date and a transaction timestamp identify a person even when the name is gone. Masking one column at a time does not address inference across columns.
The second is semantic destruction. Replace an IBAN with random digits and the checksum validation rejects it. Randomize names independently of the country column and every locale assertion becomes nonsense. Break the link between a customer row and its invoices and your test data no longer represents any state the application can produce. You end up with data that is neither private nor useful.
The alternative is synthetic generation: profile production statistically — row counts per table, cardinality per column, value distributions, null ratios, the shape of the join fan-out — then generate a dataset that matches those properties without containing any real record. Nothing to re-identify, because nothing came from a person.
The hard part is referential integrity. Generating one valid customer is trivial. Generating a hundred thousand customers whose invoice counts follow the real long-tail distribution, whose payment records point to existing invoices, whose totals match line item sums, and whose date ordering respects business rules, is a graph problem. You have to generate in dependency order and carry constraints forward. This is why synthetic generation is worth building for the tables at the center of your domain and not for the whole database at once.
Moving From a Shared Database to Parallel Execution
You cannot parallelize without data isolation. Run four workers against a shared dataset and you get failures that reproduce only under load — the hardest class of flaky test to diagnose, because the root cause is another worker.
Three levels of isolation are available, and which one you get depends on the application’s architecture.
Tenant-level isolation is the strongest and requires a genuinely multi-tenant application where every query is scoped by tenant ID. One tenant per worker, created in global setup, dropped at the end. Worker count equals the number of tenants you can provision.
User-level isolation is the cheapest to adopt and the most common. Each worker authenticates as its own account and only ever touches records it owns. It works when all data is user-scoped; it fails for anything global — admin settings, shared catalogs, system-wide counters. Those tests go in a serial group.
Schema-level isolation gives each worker its own schema or database. It works regardless of application design as long as the connection string is configurable, and the ceiling is your database server, not your code. The prerequisite is migrations fast enough to run per worker — if provisioning takes three minutes, parallelism costs more than it saves.
Whichever you pick, set worker count from the isolation unit, not from CPU count. Eight CPUs and three tenants means three workers. Scaling beyond the isolation boundary reintroduces exactly the coupling you removed. The same reasoning applies to how you scale a suite overall.
Locking Data Ownership Into the Process
Test data decays without an owner. Typically it decays in a predictable way: the person who wrote the factories moves teams, a required column ships, half the factories break, and the fix is a hardcoded value in one test file that nobody removes. Six months later the factories are unused and every test constructs its own payload inline.
Assign ownership explicitly. Factories belong to the same team that owns the API endpoint they call — if backend engineers change the contract, they update the factory, because they’re the ones who know what the new field means. Reference fixtures belong to whoever owns the migration that introduced the reference table. QA owns the isolation strategy and the cleanup mechanism, because that’s suite infrastructure.
Then put it in the pull request checklist:
- If a request payload gained a required field, the corresponding factory default was updated.
- If a migration changed a reference table, the fixture or seed was updated in the same PR.
- New tests create their own data; no test reads a record it did not create.
- No test depends on execution order or on data left behind by another test.
- Test data is identified by the run-scoped unique suffix, never by a hardcoded ID or email.
- Preconditions are built through the API; only the behavior under verification goes through the UI.
For someone writing their first test in the repository, the rule set is four lines: use the factory, override only what the test asserts on, never hardcode an identifier, never read another test’s data. That’s enough to keep the suite parallelizable. Building setup through the API also means your API tests and your UI tests share one data layer instead of two.
What to Do Next
Start with evidence, not with a rewrite.
- Label the last 30 days of failures. Pull every failed run from CI and tag each failure as data-related or not. Data-related means: missing or unexpected record, uniqueness violation, stale state, order dependency. You need the ratio before you can argue for the work.
- Write factories for your three most shared data objects. They’re the ones appearing in the most test files — usually user, and the two entities at the center of your domain. Convert the tests that use them; leave the rest.
- Document your production copy, if you have one. Write down the refresh interval, which tables contain personal data, who owns the anonymization rules, and when they were last reviewed against the current schema. If you can’t answer all four, that’s the finding.
- Isolate one test file for parallel execution. Pick the file with the least shared state, give it its own data ownership and cleanup, and run it with two workers. The failures you see there are the full list of coupling you’ll have to remove everywhere else — and they’re much cheaper to read in one file than in forty.
One question worth asking your team while you do this: when a regression run last collapsed because of shared database state, what was the actual root cause, and which approach closed it — deletion, a separate tenant, or synthetic generation? The answer usually tells you more about the environment than any tooling decision will.