When to stop testing? This question gets asked more often than not. However, as simple and straightforward it sounds, the answer to this question spans multiple aspects and variables which should be considered when we really want to stop. We just wish the answer could be “When all defects are found!”. Like humans, the software is also mere mortal and never bug-free.
This blog aims to discuss multiple factors which should be pondered before we make a decision to stop testing.
The Release Deadline Has Been Met
We can stop when the release deadlines have been met. We live in a world full of constraints, so time like any other resource is constrained too. Every testing cycle will have a schedule, and we need to stop when we reach the end of the schedule.
The key point here is to plan the testing well so that all the requirements have been tested and coverage has been done before the scheduled end date. After all, you won’t like to ship a partly tested product just because the deadline had arrived.
Too Buggy To Go Deep
One of the interesting answers to the question would be when it is too buggy that you can’t go beyond the first few screens. There will be times when the software will have a lot of bugs on the surface, which will not allow the testing team(s) to go deep into the software or perform more sophisticated tests. In such cases, even though schedule permits, we might have to stop (read pause) testing, only till the time blockers or superficial bugs are solved, and it allows us to go deep into the product.
Management Decisions Leading to Stop Software Testing
While as testers we only have visibility of the product we test, but the company’s management has the visibility of all the products under its umbrella. The company’s management may take decisions to maximize the interests of the
company, which could lead to the stopping of testing of the project you are assigned.
Some of the common reasons in such cases are – a) Budget or/and resources were allocated to some other high priority project b) The project was canceled (from the client’s side or the company’s side) c) The allocated budget is over and no more budget is being allocated.
Exit Criteria Has Been Met
By far one of the most important criteria to decide is when exit criteria are met. However, the key point is here it not just meeting the criteria, the key point here is designing them. Designing exit criteria for any product or software testing cycle is easier said than done. It is imperative that we consider multiple factors while designing exit criteria, some of which are mentioned below:
1. Strong buy-in from the software development team. In the end, your job is to discover defects and theirs is to fix.
2. If there are clients who will further test the product, that should also be considered as to what will be an acceptable input quality.
Know When to Stop testing with These Examples of Exit Criteria:
1. When all the planned tests have been run (it could include multiple cycles of some or all tests as well)
2. When the number of open defects reaches a certain milestone (this could also be enhanced based on the allowed open defects across various priorities/severity)
3. When the planned regression tests have been done.
4. If your product involves stability tests like MTBF, what is the acceptable value of the MTBF Test?
Why Testing Is A Never-Ending Process?
Criteria once designed are not meant to be carved in stone. It should be reviewed constantly in light of the adherence and the post-release issues which come after we meet a set criterion.
How to Track Testing Progress to Meet Exit Criteria?
While designing and concluding exit criteria is indeed major work done. The next step is to regularly monitor the progress that the exit criteria are met. As QA or software testers, we should understand that if the product is not shipped on time (after meeting exit criteria), then we might have a severe business impact in terms of lost sales, and competition gaining significant advantages to name a few. It is extremely important to monitor the progress in the testing process viz-a-viz exit criteria regularly. Some of the suggested methods of doing it are as follows:
1. Set up a Dashboard that can powerfully summarize all the variables against
exit criteria.
2. Set up regular sync-ups with various stakeholders to unblock blocked
people discuss risks and mitigation plans.
Conclusion
In conclusion, determining when to stop testing is a complex and subjective matter, lacking a definitive answer. However, throughout this blog, we have provided you with valuable insights and considerations to help guide your decision-making process. By exploring factors such as meeting release deadlines, the severity of bugs, management decisions, and establishing effective exit criteria, we have equipped you with the necessary thinking points to make informed choices. While the decision may still require careful consideration and evaluation of specific project circumstances, we are confident that the information shared in this blog will empower you to determine appropriate exit criteria and identify the right time to conclude testing. Remember, success lies in finding the right balance between thorough testing and project constraints. So go forth, armed with newfound knowledge, and make impactful decisions to deliver high-quality software products.
