ChatGPT and Software Testing: Embrace or Tread with Caution?

chatgpt in testing

The world is continuously amazed by ChatGPT and its applications. Every day multiple people post the applications of ChatGPT in their respective industries ranging from marketing, copywriting, sales, operations, and more. While we thought about ChatGPT for software testing, we have a mixed bag of opinions.

We do not doubt its benefits to the software testing community and at the same time, we are aware of the dangers of blindly accepting every output by the software. At best, it is a good heuristic to use and by nature – heuristics are fallible and not rules. As the output of ChatGPT is heavily reliant on the prompts, it is largely dependent on the skill of the software tester providing the prompts.

If you also want to learn how to use ChatGPT in test automation, listen to Siddharth Shukla now!

Making Use of ChatGPT in Software Testing:

Here are seven different ways one can use ChatGPT in software testing:

1. Test Idea Generator

Wouldn’t it be wonderful if you can have someone cross-check your list of test ideas and also suggest an initial set of test ideas based on your prompt? Let us try this out on a flight booking website for a specific type of booking – Round Trip

chatgpt in software testing

Few quick ideas before we ask ChatGPT:

  • Valid cities in both From and To
  • Autocomplete / Autosuggest
  • Validation of the airport code and spelling
  • Validation of the search results (count, accuracy, speed, sorted display, pagination…)
  • Multiple combinations allowed (Same date – Different date, Number of travelers and class, fare type)
  • Match with the API and consistency across platforms
  • Error messages (I added this after typing the next sentence)

Now let’s ask ChatGPT for the test ideas with as detailed prompts as possible

chatgpt for software testing

Based on the output, it appears to have focused more on the idea – Multiple combinations allowed (Same date – Different date, Number of travelers and class, fare type)

Let’s use Pairwise Online Tool to generate combinations along with exclusions

The two exclusions are:

  • 1 Adult cannot travel with 2 children
  • 1 Adult cannot travel with 2 infants

The other exclusion I wanted to highlight but couldn’t do because of the limitation of the tool was:

  • 1 Adult cannot travel with 1 child and 1 infant

The output from the Pairwise tool was a total of 21 cases. We will exclude Case #4 for the above exclusion rule.

GAl ysRD36d4XmoYphJzGnsfwGPO8swlQ5kyfbVre

Now, who did better? ChatGPT’s output was a good starting point for one of the test ideas which could be improved using another tool. 

We could also dig deep into each of the test ideas with pointed prompts to get a better response.

For example, when we asked ChatGPT to generate ideas for error messages, the output was not disappointing.

chatgpt use cases

Some of these cases are outside the scope of the current feature variables under focus (ex: insufficient funds, invalid card details…)

You get the drift, it is a good starting point provided you know the drill. You have to critically analyze the output, pick the useful points and leave the rest.

2. Analyze key statements

Imagine a sentence from a requirement document and what if ChatGPT can analyze the sentence and provide us with multiple interpretations of the phrases?

Let’s try it out. We will use the example from Google Meet 

For users with personal Google accounts: You can have a maximum of 100 participants in a meeting. 

9jcgOUhTkBYkT4mM1VQ8mftfhopxN5Rvy6ab amTa5qk6XF8Ntw5qoCt7XfVmAOm1n2BmotZo97n16zGobT9Euk IEPFZlcZ7hmisUUoNqmdW15C8IGYLF 7UEKZM0W J7oXYuJXvMs2osmREFZh U

I feel that 3,4,5 are convey the same meaning but in different words. I would use ChatGPT to give me alternate interpretations to cross-check against my understanding and interpretations.

3. Learning Guide

One could use it to kickstart their learning about a topic and build on the outputs. For example, if we want to understand how to test multimedia, we can ask ChatGPT for initial pointers.

SvbMaD4iz9ko9tePXGiMMPzogEnzkAubBbVYO07VOv6TpMpGxKWRJL9W yuSG5EBSSDB3jO L 5bm XBs1CSExCIO1aHvxG z jiRb1lz7v8BkN 62OQ69BOfUuZ31zLta2HBbCbk2yaAaq0b34way8

I would then refine more to ask more deep questions.

  • Kind of audio and video issues to expect
  • Parameters affecting audio/video
  • Tools to test audio/video
  • Terms associated with audio/video testing

Now, is this the perfect answer? I don’t know. But it gives me enough guidewords to start my journey. I can now read more about codecs and their relation to multimedia.

4. Snippets of code

Can it generate snippets of code that can be built upon? I have seen multiple examples posted on social media on how ChatGPT can generate code in multiple languages. 


Example: Write code in Python to read the contents of a file and list out the number of duplicate elements and also sort them in descending order based on the number of characters in each element

Xq8L1vqVLyTpC9RsF7H6hlubzQrxVGMxhRxz1aqRaKxB14VThK6RmbAjTCcElETs07SK6jEiGkDsRd0 pZQ3zNW0yQg9HGgjwr1kOyWzAstvzWNAnvAsbwLTaVjLXoIv3Kt9dwOD wZmcy3ZfUFdsDs

Did ChatGPT achieve the objective?

5. Summarize a document or video

As of date, ChatGPT is not able to create images or watch videos. Providing the transcript of a 15 min video could generate a summary for the video – “Become the CEO of your testing” by B. Ajay | testing scope | TestFlix2022 | Testing tips and tricks

wvblkajdtEgt99qmHxInYIsftznf85pfsqSjr2f2tOJU4Qj4S6KCDwznjsSjRxn8I3ymvx cFjVXVD8uqOTtvBC0i4JR0GHdCWzpfl9QtwUrfUNQiMjloxaviXb8Qb1KxS E

You could also use extensions like YouTube Summary with ChatGPT to get the transcript. 

6. Topic Outlines

I thought for a while about a blog topic and then asked ChatGPT to create an outline for this article itself

The result is a well-structured outline and saves a lot of time.

Akb0WFE7J2847oxQtP0TD0ZJV2kIo4MCFnYEOjCvsTPSlTDMGmfMsxcKih26uJ 0AAgHopD7uLpLCp F61L84enpZmYnzzYB

Will ChatGPT replace jobs? Not so soon. It is a good assistant and needs skilled folks to leverage it to its fullest potential.

7. Brainstorming partner

Ideas can strike anytime and if you want to brainstorm on different software testing topics, get a different perspective, compare and contrast topics or even have an assistant to help you with your research, ChatGPT can be a good starting point.

Conclusion

For the different use cases highlighted so far, analyzing the ChatGPT responses, we can arrive at the following conclusion:

  • ChatGPT is a good starting point and needs skilled testers to build on top of the responses
  • It is a language model and with the appropriate learning techniques, the quality of the output will improve
  • Treat the responses as a heuristic and not as a rule. 
  • Know the power of ChatGPT and make use of it.
  • Continue experiments with ChatGPT and pick the right responses to implement.

This post is a guest blog by Ajay Balamurugadas. We’d like to thank him for contributing yet another informative piece for The Test Tribe.

Written by

Ajay Balamurugadas, goes by the handle ‘ajay184f’ in the testing community and is continuously re-inventing his testing methodology. He co-founded Weekend Testing – a worldwide movement for skilled testing, authored multiple books available. His friends associate the terms – ‘Change Agent, Idea Man, Motivational’ to him. He tweets under @ajay184f and loves to have long conversations on software testing and life in general. He is currently working at GSPANN Technologies, Inc. as SeniorDirector – QE. When not testing, he spends time with his wife and two children.

Leave a Reply

Your email address will not be published.

Related Posts

Testing Courses at Thrive EdSchool

Advertisement (Know More)

Get Top Community News

    Top Event and other The Test Tribe updates to your Inbox.

     

    Categories

    Tags