The world is continuously amazed by ChatGPT and its applications. Every day multiple people post the applications of ChatGPT in their respective industries ranging from marketing, copywriting, sales, operations, and more. While we thought about ChatGPT for software testing, we have a mixed bag of opinions.
We do not doubt its benefits to the software testing community and at the same time, we are aware of the dangers of blindly accepting every output by the software. At best, it is a good heuristic to use and by nature – heuristics are fallible and not rules. As the output of ChatGPT is heavily reliant on the prompts, it is largely dependent on the skill of the software tester providing the prompts.
If you also want to learn how to use ChatGPT in test automation, listen to Siddharth Shukla now!
Making Use of ChatGPT in Software Testing:
Here are seven different ways one can use ChatGPT in software testing:
1. Test Idea Generator
Wouldn’t it be wonderful if you can have someone cross-check your list of test ideas and also suggest an initial set of test ideas based on your prompt? Let us try this out on a flight booking website for a specific type of booking – Round Trip
Few quick ideas before we ask ChatGPT:
- Valid cities in both From and To
- Autocomplete / Autosuggest
- Validation of the airport code and spelling
- Validation of the search results (count, accuracy, speed, sorted display, pagination…)
- Multiple combinations allowed (Same date – Different date, Number of travelers and class, fare type)
- Match with the API and consistency across platforms
- Error messages (I added this after typing the next sentence)
Now let’s ask ChatGPT for the test ideas with as detailed prompts as possible
Based on the output, it appears to have focused more on the idea – Multiple combinations allowed (Same date – Different date, Number of travelers and class, fare type)
Let’s use Pairwise Online Tool to generate combinations along with exclusions
The two exclusions are:
- 1 Adult cannot travel with 2 children
- 1 Adult cannot travel with 2 infants
The other exclusion I wanted to highlight but couldn’t do because of the limitation of the tool was:
- 1 Adult cannot travel with 1 child and 1 infant
The output from the Pairwise tool was a total of 21 cases. We will exclude Case #4 for the above exclusion rule.
Now, who did better? ChatGPT’s output was a good starting point for one of the test ideas which could be improved using another tool.
We could also dig deep into each of the test ideas with pointed prompts to get a better response.
For example, when we asked ChatGPT to generate ideas for error messages, the output was not disappointing.
Some of these cases are outside the scope of the current feature variables under focus (ex: insufficient funds, invalid card details…)
You get the drift, it is a good starting point provided you know the drill. You have to critically analyze the output, pick the useful points and leave the rest.
2. Analyze key statements
Imagine a sentence from a requirement document and what if ChatGPT can analyze the sentence and provide us with multiple interpretations of the phrases?
Let’s try it out. We will use the example from Google Meet
For users with personal Google accounts: You can have a maximum of 100 participants in a meeting.
I feel that 3,4,5 are convey the same meaning but in different words. I would use ChatGPT to give me alternate interpretations to cross-check against my understanding and interpretations.
3. Learning Guide
One could use it to kickstart their learning about a topic and build on the outputs. For example, if we want to understand how to test multimedia, we can ask ChatGPT for initial pointers.
I would then refine more to ask more deep questions.
- Kind of audio and video issues to expect
- Parameters affecting audio/video
- Tools to test audio/video
- Terms associated with audio/video testing
Now, is this the perfect answer? I don’t know. But it gives me enough guidewords to start my journey. I can now read more about codecs and their relation to multimedia.
4. Snippets of code
Can it generate snippets of code that can be built upon? I have seen multiple examples posted on social media on how ChatGPT can generate code in multiple languages.
Example: Write code in Python to read the contents of a file and list out the number of duplicate elements and also sort them in descending order based on the number of characters in each element
Did ChatGPT achieve the objective?
5. Summarize a document or video
As of date, ChatGPT is not able to create images or watch videos. Providing the transcript of a 15 min video could generate a summary for the video – “Become the CEO of your testing” by B. Ajay | testing scope | TestFlix2022 | Testing tips and tricks.
You could also use extensions like YouTube Summary with ChatGPT to get the transcript.
6. Topic Outlines
I thought for a while about a blog topic and then asked ChatGPT to create an outline for this article itself
The result is a well-structured outline and saves a lot of time.
Will ChatGPT replace jobs? Not so soon. It is a good assistant and needs skilled folks to leverage it to its fullest potential.
7. Brainstorming partner
Ideas can strike anytime and if you want to brainstorm on different software testing topics, get a different perspective, compare and contrast topics or even have an assistant to help you with your research, ChatGPT can be a good starting point.
Conclusion
For the different use cases highlighted so far, analyzing the ChatGPT responses, we can arrive at the following conclusion:
- ChatGPT is a good starting point and needs skilled testers to build on top of the responses
- It is a language model and with the appropriate learning techniques, the quality of the output will improve
- Treat the responses as a heuristic and not as a rule.
- Know the power of ChatGPT and make use of it.
- Continue experiments with ChatGPT and pick the right responses to implement.
This post is a guest blog by Ajay Balamurugadas. We’d like to thank him for contributing yet another informative piece for The Test Tribe.
