One of my accessibility tests failed on CI about one run in six, and passed every single time I ran it locally. That combination is close to the worst kind of bug to have, because “works on my machine” is not a debugging strategy, it’s a confession.
The actual cause
The test ran an axe-core scan on a card component mid-fade — CI machines are slower and less predictably scheduled than my laptop, so the scan sometimes landed while a card’s opacity was still animating between 0 and 1. Axe doesn’t care about opacity, but it does flag aria-hidden mismatches, and the fade transition briefly toggled a hidden state on elements that hadn’t finished appearing yet.
await page.waitForSelector('[data-reveal="done"]');
const results = await new AxeBuilder({ page }).analyze();The fix was one attribute: adding a data-reveal="done" flag once the fade transition’s IntersectionObserver callback finished, and waiting for it before scanning. Scan the settled state, not the mid-animation one.
Why it’s worth writing down
A flaky test on CI is easy to treat as noise — rerun it, move on, and quietly stop trusting the suite a little more each time it happens. The actual cause here had nothing to do with the test framework and everything to do with the thing under test racing against the thing checking it. Once I stopped assuming the test was wrong and started asking what was different about CI’s timing, it took about ten minutes to find.