When to stop testing? This question gets asked more often than not. However, as simple and straightforward it sounds, the answer to this question spans multiple aspects and variables which should be considered when we really want to stop testing. We just wish the answer could be “When all defects are found!”. Like humans, the software is also mere mortal and never bug-free.
This blog aims to discuss multiple factors which should be pondered before we make a decision to stop testing.
The Release Deadline Has Been Met
We can stop testing when the release deadlines have been met. We live in a world full of constraints, so time like any other resource is constrained too. Every testing cycle will have a schedule, and we need to stop testing when we reach the end of the schedule.
The key point here is to plan the testing well so that all the requirements have been tested and coverage has been done before the scheduled end date. After all, you won’t like to ship a partly tested product just because the deadline had arrived.
Too Buggy To Go Deep
One of the interesting answers to the question, when to stop testing is when it is too buggy that you can’t go beyond the first few screens.There will be times when the software will have a lot of bugs on the surface which will not allow the testing team(s) to go deep into the software or perform more sophisticated tests. In such cases, even though schedule permits, we might have to stop (read pause) testing, only till the time blockers or superficial bugs are solved and it allows us to go deep into the product.
Management Decisions
While as testers we only have visibility of the product we test, but the company’s management has the visibility of all the products under its umbrella. The company’s management may take decisions to maximize the interests of the
company which could lead to the stopping of testing of the project you are assigned.
Some of the common reasons in such cases are – a) Budget or/and resources were allocated to some other high priority project b) The project was canceled (from the client’s side or the company’s side) c) The allocated budget is over and no more budget is being allocated.
Exit Criteria Has Been Met
By far one of the most important criteria to decide testing stop is when exit criteria are met. However, the key point is here it not just meeting the exit criteria, the key point here is designing them. Designing exit criteria for any product or
software testing cycle is easier said than done. It is imperative that we consider multiple factors while designing exit criteria, some of which are mentioned below:
1. Strong buy-in from the development team. In the end, your job is to discover defects and theirs is to fix.
2. If there are clients who will further test the product, that should also be considered as to what will be an acceptable input quality.
Here Are Some Examples of Exit Criteria:
1. When all the planned tests have been run (it could include multiple cycles of some or all tests as well)
2. When the number of open defects reaches a certain milestone (this could also be enhanced based on the allowed open defects across various priorities/severity)
3. When the planned regression tests have been done.
4. If your product involves stability tests like MTBF, what is the acceptable value of the MTBF Test.
Exit criteria once designed are not meant to be carved in stone. It should be reviewed constantly in light of the adherence and the post-release issues which come after we meet a set exit criterion.
How to Track Testing Progress to Meet Exit Criteria?
While designing and concluding exit criteria is indeed major work done. The next step is to regularly monitor the progress that the exit criteria are met. As testing teams, we should understand that if the product is not shipped on time (after meeting exit criteria), then we might have a severe business impact in terms of lost sales, competition gaining significant advantages to name a few. It is extremely important to monitor the testing progress viz-a-viz exit criteria regularly. Some of the suggested methods of doing it are as follows:
1. Set up a Dashboard that can powerfully summarize all the variables against
exit criteria.
2. Set up regular sync-ups with various stakeholders to unblock blocked
people, discuss risks and mitigation plans.
As mentioned earlier, while there is no straightforward answer to when to stop testing, we hope this blog gives you enough thinking points to decide your exit criteria and stop testing points.
To read more insightful blogs related to software testing, visit our page – https://thetesttribe.com/blog/
