Products are getting complex with time and release times are getting equally shorter. The tolerance for bugs and poor quality has decreased significantly, largely due to the abundance of software products. You name a category, and there are at least 50 companies working towards it, however niche you might think of it being. The competition forces everyone to optimize their time, effort, and resources even more than before. People don’t want to rework; they want the best quality from the start. So, before we dive deep into testing a product, a comprehensive sanity testing strategy acts as a much needed checkpoint in the release cycle.
In this blog:
- What is a Sanity Test?
- Objective of A Sanity Test
- Sanity Testing Checklist
- Building a Sanity and Regression Test Suite Together
- Common Mistakes to Avoid in Your Sanity Testing Strategy
- Conclusion
What is a Sanity Test?
It is a subset of the exhaustive list of tests that you could perform on your application. This is the set usually executed first most of the time. Based on the results, the exhaustive list of tests could be picked up for execution or the build be rejected. Does it sound similar to a smoke test? In some cases, yes. Every company and team has its own definition. So, work with what your context demands and justifies a test to be a sanity test.
Objective of a Sanity Test
A strong sanity test suite helps everyone understand the health of a product, and based on the results, we can then decide whether to spend more time testing it or reject it. The objective of a sanity test should be to confirm that the following types of functions are working fine:
- The main use case of the product
- Customer facing features
- Most critical features
- Most used features
- Features promised in the specific release
- No blockers in overall application (tricky ask)
We have seen multiple testers spend a lot of time testing multiple features and reporting many bugs; they forget to test for sanity. That is a risky approach. You never know if those bugs are a consequence of the few failing smoke tests.
Sanity Testing Checklist
Let us look at the checklist for creating a comprehensive sanity testing strategy:
1. Systematic Product Modeling
As soon as you receive a product to test, spend a few hours playing with the product. Go through every feature systematically to understand how the feature behaves. Figure out how the feature is interacting with other features. Pay attention to the defaults and the error information. Any information about the product now is worth its weight in gold. This helps create a mental model of the entire product under test. If you are wondering how this is related to a sanity testing strategy, let me explain. The more we know about the product, the stronger our tests will be. If our knowledge of the product is shallow or limited, we would be scratching the surface. We would be getting a false hope that we have tested the sanity but we would have left many features unexplored. There cannot be a bigger risk than that.
Following figure illustrates an example of partial understanding of the product. The functionality marked red might not be discovered till we dive deep into the product. If the sanity test suite is not updated frequently, it might be on the regression suite but never get on the sanity suite. The overlapping circles denote interacting features. So, by performing a systematic product modeling, we understand the scope of the product first. With this understanding, we will now be able to find the right tests for the sanity suite. Check out this course to learn more about systematic product modeling: Link
2. Understanding The Users
While every user is unique, you can group them by behavior or personas. Once we understand the types of users and their typical usage patterns, we can design our testing strategy accordingly. If your users hardly use the website and are mostly on the app, it makes more sense to test the app more than the web. Taking help of the analytics, we can also focus on the most critical flows in our sanity suite. Some teams dive too deep into certain features and as the sanity suite is too large now, they ignore the other features. If you want to have comprehensive coverage, pay attention to all the features equally. Spending some time with the support team also helps in understanding the calls made by the users and which flows are trending.
3. Take Help of Heuristics
A heuristic is a fallible way of solving a problem. When we don’t know how to solve a problem, heuristics can help us, though they are not guaranteed to solve the problem. There are multiple heuristics like SFDIPOT, RCRCRC and FEW HICCUPS to assist in testing software. These can help in creating a robust sanity suite.
For example, RCRCRC stands for
R: Recent
C: Core
R: Risk
C: Configuration
R: Repaired
C: Chronic
Using this heuristic, one can design both the sanity and regression suites.
4. Keep It Up-to-Date
A bigger mistake than not having a sanity suite is to not update it often. When the features get updated, bug-fixes change the previous implementation, some features get deprioritized, it is time to update the tests too. Some of the features mature with time and there are not many changes to those features as well. With time, the sanity test suite might bloat and try to trim the suite as well. The purpose of every test in the suite is to find a bug. If the test doesn’t give you the confidence of finding a bug, remove or update it with a more powerful test.
5. Lean Documentation to the Rescue
In one of the projects, the sanity suite itself took 2 hours and the overall regression suite took 2 days. Teams need quick information on whether to reject or accept the build. In some contexts, 2 hours is a lot of time. Keep it lean. See if the sanity testing checklists, guidewords, mind maps work well compared to the test cases, guide words and scenarios. Having lean documentation helps in easy review, update and execution.
6. Complement with the Regression Strategy
Having a case in the sanity suite for a feature but no more cases in the regression suite doesn’t help. While there is rarely a need to repeat a case in both the sanity and regression suite, it is important to have both the suites complementary. Based on the results of the sanity suite results, the order of execution in the regression suite can be modified.
Building a Sanity and Regression Test Suite Together
Let us take an example to describe both the sanity and regression test suite for the ‘Increase font size’ feature in Google Docs.
Sanity Test:
This could be answered by the question – What is that one thing this feature is expected to do without fail in most of the cases?
The answer would be: “Increase font size” when you select some text and click on this option or use the shortcut to enable it. And that becomes the sanity test!
Regression Test:
Every test that one can think of as important in the context of the product’s release. A few examples of test ideas are as follows:
- Does the font increase for
- A single letter
- A symbol
- A word
- Multiple words
- Multiple sentences
- Multiple paragraphs
- The whole multi-page document
- Can we have different font sizes for the different sections of the document
- Will undo, redo work with the font size changes
- What is the max limit till which the font size can be increased
- What is the increment in which increase/decrease font size work
- Is the increment consistent for all kinds of text
- Will the setting be preserved during printing
and so on.
Common Mistakes to Avoid in Your Sanity Testing Strategy
Also, beware of the following mistakes in creating a sanity suite
- Not covering all the features
- Either being too shallow or too deep, both not helping the purpose of a sanity suite
- Not using automation if there is a possibility to automate the suite
- Ignoring the analytics and keeping the suite static
- Not taking the sanity results seriously
Summing Up:
Once we have a comprehensive sanity suite, we can expect key gains in productivity as teams don’t spend a lot of time testing the buggy build. The sanity suite acts as a strong gatekeeper. It is easy to implement and needs discipline to maintain it as a strong suite.
This post is a guest blog by Ajay Balamurugadas. We’d like to thank him for contributing yet another informative piece for The Test Tribe. You can connect with Ajay at :
