Abstract

There are a lot of articles, blog posts, podcasts, conference talks about what are “Flaky tests” and how to avoid them.

Some of the ideas proposed are:
Automatically rerun failed tests a couple of times, and hope they pass
Automatically retry certain operations in the test (ex: retry click / checking for visibility of elements, etc.) and hope the test can proceed

Unfortunately, I do not agree with the above ideas, and I would term these as anti-patterns for fixing flaky / intermittent tests.

We need to understand the reasons for flaky / intermittent tests. Some of these reasons could be because of issues like:
– timing issues (i.e. page loading taking time)
– network issues
– browser-timing issues (different for different browsers / devices)
– data related (dynamic, changing, validity, etc.)
– poor locator strategy (ex: weird & hard-wired xpaths / locators)
– environment issue
– actual issue in the product-under-test

In this talk we will explore the anti-patterns and its impact. We will also look at ideas how you can actually reduce / eliminate the flakiness of your test execution.

Top Takeaways:

• Stop with the retry / rerun of failing tests
• Fix the root-cause, not the symptoms to make your tests stable, robust and scalable!