Flaky Tests and the Five Failure Types Behind Them

TL;DR: A flaky test passes and fails without a relevant code change, but the cause is rarely random. Most browser-test flake belongs to one of five groups: timing, leaked state, unstable data, variable network behavior, or locator drift. Classify the first mismatch before adding retries. A passing rerun only proves the conditions changed. It does not prove the product or test is reliable.

A flaky test is a test that sometimes passes and sometimes fails even though the application and test code appear unchanged. Flake damages trust because teams stop knowing whether a red build means a real regression or a noisy test.

The fastest way to fix flaky tests is not to increase the retry count. It is to identify the first condition that differs between a passing run and a failing run.

For browser tests, use this order:

1. timing and asynchronous work;

2. leaked browser or application state;

3. unstable test data;

4. variable network or service behavior;

5. locator drift or ambiguous targeting.

That order is practical rather than absolute. It starts with the failures most often hidden by a rerun and ends with the targeting problems that are easiest to inspect directly.

What makes a test flaky

A test is flaky when the same intended scenario produces inconsistent results under conditions the team believed were equivalent. The important phrase is believed were equivalent. A browser can carry state through cookies, storage, cached objects, open tabs, service workers, feature flags, permissions, account roles, locale, and page history. Two runs that look identical in the test file may not begin from the same state.

Flaky tests are different from consistently failing tests:

A retry can be useful for measuring frequency or collecting another trace. It should not be mistaken for a fix.

Failure type 1 is timing

Timing flake happens when the test acts before the application reaches the state the test assumes.

Common causes include:

• a request that has not completed;

• an animation or overlay that still blocks input;

• a component that rendered before its data arrived;

• a save that acknowledged the click before the write persisted;

• a background job whose completion time varies;

• an arbitrary sleep that is too short on a slower worker.

The repair is usually to wait for a meaningful condition, not a longer clock delay. Wait for the button to become enabled, the expected response to arrive, the persisted value to appear, or the next state to become visible.

A good timing assertion names the contract:

await page.getByRole('button', { name: 'Save' }).click();
await expect(page.getByText('Saved')).toBeVisible();
await page.reload();
await expect(page.getByLabel('Plan')).toHaveValue('enterprise');

The reload matters when the product promise is persistence. A success message can appear before a failed or discarded write.

Failure type 2 is leaked state

State flake appears when an earlier action changes what a later test sees. The state can live in the browser, the page, the account, shared test infrastructure, or a long-lived application object.

Typical examples include:

• a previous test leaves an admin role active;

• local storage survives into a later scenario;

• a workspace setting changes while an open page keeps an old parser or permission model;

• one tab updates shared state while another tab continues with a stale view;

• a service worker or cache serves an older response;

• tests reuse the same account and mutate its data in a different order.

Playwright creates a fresh browser context for each test by default. That is a strong isolation control, but it can also hide a real product defect when the user changes state inside an existing session. Keep both tests when the transition matters: a fresh-page control and an in-session transition path.

Our earlier locale experiment demonstrated this pattern. A fresh UK page interpreted a date correctly, while an already-open US page changed its visible locale but retained a US parser. The final error looked intermittent. The real variable was page history.

Failure type 3 is unstable data

Data flake occurs when the scenario depends on records that are shared, reused, expired, reordered, or created without a deterministic identity.

Watch for:

• a username or order number reused across parallel runs;

• a record that another test deletes;

• date-sensitive fixtures that cross midnight or a billing boundary;

• a list whose order changes when new data arrives;

• eventually consistent reads immediately after a write;

• random fixture generation without a retained seed.

The fix is to make ownership and cleanup explicit. Give each run unique records, freeze clocks where appropriate, retain random seeds, and assert the data preconditions before testing the UI behavior.

Do not use production-like data as an excuse for uncontrolled data. Realism helps only when the test can still explain what changed.

Failure type 4 is network or service variability

A browser test may depend on APIs, third-party scripts, queues, authentication providers, or rate-limited services. A test can look flaky when the browser path is stable but the service contract is not.

Separate these cases:

• the request never started;

• the request started with the wrong method or payload;

• the response was delayed;

• the response returned an error;

• the response succeeded but the page ignored it;

• the page showed success but the server did not retain the result.

Network mocking can make a test deterministic, but broad mocks can also hide real contract failures. Mock only the boundary you intend to control. Keep at least one test against the real integration path when that contract is important.

Record the request, response, visible state, and durable readback as different facts. A green UI assertion does not prove the backend state persisted.

Failure type 5 is locator drift

Locator flake happens when the test does not have a stable, unambiguous way to identify the intended control.

Common signals include:

• a CSS class changes during a refactor;

• text appears twice and the test selects whichever renders first;

• a hidden copy of the component remains in the DOM;

• a generated selector depends on brittle ancestry;

• a button label changes with locale or experiment assignment;

• a broad locator matches a different element after a layout change.

Prefer user-facing roles, labels, and explicit test contracts. Use CSS when the DOM relationship is itself the contract. Require strict matches where ambiguity would otherwise pass silently.

A stable locator is necessary, but it is not sufficient. The test must still verify the result after interacting with the correct element.

A controlled stale-state experiment

We built a fake approval dashboard to show why a passing fresh-page rerun can hide the original failure. The fixture uses synthetic data and performs no real approval.

The scenario has two paths:

The 409 response is correct. The browser problem is that the stale page still presents an action the current server state no longer permits.

A tester who opens a fresh page sees no button and cannot reproduce the report. A screenshot of the final 409 shows the rejection but not why the action was available. The useful evidence is the ordered transition:

1. open the record while approval is still allowed;

2. change the record to locked elsewhere;

3. return to the stale page;

4. click the still-visible approval action;

5. observe the 409 response;

6. compare with a fresh locked page where the action is hidden.

That comparison classifies the failure as stale browser state, not timing noise, bad test data, or an unstable endpoint.

How to triage a flaky test

Use a passing run as a control, not as a reason to close the failure.

Preserve the first failing run

Keep the trace, console output, requests, screenshots, test data identity, browser configuration, and relevant application state before rerunning. The first failure often contains the only copy of the condition that matters.

Re-run the smallest scope

Run the failing test alone. Then run it with its immediate predecessor or the smallest suite segment that reproduces the issue. If the result changes, suspect state, order, data, or shared resources.

Compare the first mismatch

Do not start with the final assertion. Compare the earliest point where the passing and failing runs disagree:

• Was the same account and role active?

• Did the same request leave the page?

• Did the same element receive the action?

• Did the same data exist?

• Did the page and server agree on the current state?

Change one variable

Use a fresh context, fixed dataset, controlled response, explicit wait, or stricter locator one at a time. Changing several things together can make the test pass without identifying the cause.

Verify the fix across clean runs

Remove diagnostic retries. Repeat the repaired test from clean conditions, then run the surrounding suite and CI environment. Record the denominator. “It passed again” is weaker than “it passed 20 clean runs across local and CI conditions.”

What to send the teammate fixing it

A useful flaky-test report should answer:

• What was the intended starting state?

• Which action first differed?

• What did the browser show?

• What did the request and response show?

• What state existed after reload or a fresh session?

• Which control path passed?

• Which of the five failure types best fits the evidence?

When the failure depends on browser history, deliberately start recording before the state-changing action. Samelogic is a platform for recording browser bug steps, playing them back, inspecting the failing moment, and handing the useful path to engineering. It is deliberately started and is not passive background capture. It does not replace deterministic tests, traces, or server logs.

For a browser-specific triage sequence, use the flaky Playwright tests workflow. For a worked stale-locale example, see flaky tests caused by locale changes and stale browser state. For action-by-action diagnosis, use the Playwright Trace Viewer workflow.

A practical flaky-test checklist

Before closing a flaky test, confirm that you have:

• preserved the first failing run;

• named the failure type;

• compared a passing control;

• checked browser and application state;

• verified data ownership and time assumptions;

• separated UI success from durable state;

• inspected the exact matched element;

• removed diagnostic retries;

• repeated the fix under clean conditions;

• given the fixing teammate the first mismatch and next diagnostic step.

Flaky tests become manageable when the team stops treating them as mysterious red builds. The failure frequency may vary. The evidence still belongs to a specific contract: time, state, data, network, or target. Find that contract, then make it explicit.

Sources

• Playwright test isolation

• Playwright auto-waiting

• pytest guidance on flaky tests

• Practitioner discussion on managing flaky tests