How we triage flaky tests without freezing the merge queue
A practical sequence for quarantining noise, measuring flake rate, and restoring trust in green builds.
When flake rate climbs, teams often respond by re-running jobs until green. That hides the cost: every re-run burns runner minutes and trains engineers to ignore red signals.
We start by sampling failures across a two-week window, grouping by suite and assertion fingerprint, then quarantine only the suites that exceed an agreed flake threshold. Quarantine is temporary and visible on a shared board.
Next we add a nightly unquarantine attempt with stricter timing. Suites that stay green for five consecutive nights return to the merge path. Suites that fail again keep their quarantine label and get an owner.
The goal is not a perfect suite on day one. It is a merge queue that engineers trust again, with a shrinking quarantine list instead of an endless re-run habit.