Learn about automating test design.

Software testing is intractable.  Even the simplest feature has more possible tests than can be done in a lifetime.  Automated test design helps you select a reasonable set of test conditions to meaningfully explore important risks.  Automated test design guides selection of meaningful combinations, permutations, samples, and distributions of factors which influence, or are influenced by, the behaviour of software under test.

There exist a plethora of platforms and system configurations which testing cannot ignore.  Different versions, configurations and locales of hardware, software, operating systems and even third party coresident software are an enormous risk, especially for mobile and web-based technologies. Automated test design can establish a reasonable set of platforms to exercise thoroughly and efficiently.  

Automated test design can help identify interesting pathways, through any complex series of operations, can be identified automatically deriving a reasonably small number of tests to exercise complex patterns and expose important bugs and weakness and point out areas of instability. 

Automated test design can extract great tests from absurd collections of complex business rules.

Learn about designing test automation.

Today many practitioners learn test automation by studying how to use specific tools to manipulate software simulating transactions in a myriad of ways.  Almost all test automation is taught as part of a regression testing strategy to repeat manual checking repeatedly.  But not here. In this course we clearly demonstrate that all automation is not regression and automation can a helpful tool to solve a wide variety of testing conundrums.  Test automation does not replace testers, it enables them!

In this course we will look at how test automation can be an important utility in solving all manners of testing problems.  Real testing problems can be solved leveraging automation the right way in the right time. 

In this course test automation examples are explored and discussed in depth.  Fundamentally, automation provides a means of controlling and observing software under test.  What we need to see and observe depends on our goals.

Each case studies review will expose automation strategies which delegates will explore through a series of homework assignments.

We will look at many different automation models and examples of the real-world problems they have solved.

Case studies include:

• Embedded in device unit test system.
• Safety critical software exercised with keyword driven approaches.
• Qonfidence building Agile story testing with gherkin based transactional regression.
• Failure modes in business-critical software tested with mocking strategies.
• Live system monitoring with run time dynamic analysis automation.
• Live system failure mode testing with chaos monkeys.
• Security risk assessment with build time static analysis.
• In sprint (Scrum) data sensitive performance testing with J Meter.
• Applications developed with example driven approaches. (ATDD and BDD).
• Code developed with test driven approaches (TDD).
• Several examples of Continuous Integration Continuous Delivery pipelines.

About Automating Test Design

Testers can save a lot of time—and focus on what really matters—by using automated test design and implementation tools. Traditional test automation helps testers execute tests, report progress, and offer insights into test coverage and completeness, but little emphasis has been made on automating effective test design. 

Rob Sabourin will share automated test design techniques using a selection of free and open-source tools. 

Rob has implemented automated test design in conjunction with automated unit tests and automated story tests on many software projects, and he will share examples demonstrating automatic test design on many technology stacks using popular unit and story test frameworks. 

Rob will illustrate how complex transactional business tests are defined with automated decision table analyzers using state-of-the-art logic reduction algorithms, as well as construct test cases navigating complex integration paths, microservice APIs, and functionality chains using automatic path analysis tools. 

You will learn to automate test generation on many technology stacks for several architectures, including REST APIs, microservice architectures, cloud computing solutions, multi-tiered architectures, and mobile and embedded systems, to efficiently discover critical bugs.

About Designing Test Automation

Many of Rob’s customers, students and professional colleagues are concerned about building skills in test Automation. In this course Rob shares a problem-solving approach which can help you leveraging test automation tools and technologies to help solve some complex testing problems.  Rather than mastering any specific tool or technology stack Rob shares many ways that automation can support testing.  Automation can be an important part of your testing solution space.  Rob has succeeded in many projects by tooling and instrumenting projects leveraging a multitude of automation strategies over the past 40 years.

Automated Test Design and Designing Test Automation

This course addresses the topic of design.

Instead of prescribing how test automation should be used in software testing, this course looks at the testing from a problem-solving approach.

Given a clear test objective, tied to project-specific business goals, Rob Sabourin will share how a wide variety of tools and techniques can be used to efficiently deliver valuable meaningful results.

The issue of test design will be reviewed looking at how tools and technologies can be used to support the design of excellent tests. Test design is about focusing, and different test design techniques can be applied to help fulfill the test objective.  Automated test design tools and techniques can be used in the identification of a meaningful relevant and minimal set of variables, conditions, and factors, selecting appropriate trials to exercise the application, and learn about the risks that really matter.

Specifically starting with variable identification Robs will investigate collaboration tools, mind maps, spreadsheets, and technology-oriented tools such as Postman and Static Analysis tools such as SonarQube and J-Test to help identify the variables of interest.

Multiple variable test design techniques focused on business rules can be supported with tools such as decision table editors and APIs such as Open Rules.

Test designs for transactional, data-oriented, or stateful systems can be implemented with path analysis tools used to help identify a reasonable subset of the vast number of possible pathways through a network.

Combinations test design techniques, such as Pareto analysis can be performed with spreadsheets and with many publicly available libraries and tools.

Rob will show how simple Python scripts can be assembled to generate excellent combinations, permutation, selection, and sample-oriented tests.

About Teaching Approach

See it. A Bit of Show and Tell.

Rob provides a series of illustrative examples of each test design and test automation approach.  The context, testing problem and proposed solutions are reviewed.

Try it out yourself. 

Delegates are given a chance to implement some of the test design approaches as in-class and homework exercises.  Although some tools may be required all are free or open-sourced and have been used by Rob on both Windows and Mac Platforms.

Do it.

Homework problems give delegates a chance to attempt more complex problems, combine techniques and even apply techniques to their own problem space.

Make it real. Case Studies and Experience Reports.

Rob presents case studies from recent project contexts which blend many test automation tools and techniques to help solve complex problems.  All the case studies come from professional task analysis and include relevant business, technical and organizational factors.

Dynamic Energetic and Relevant.

Rob provides a fun, educational, and high-energy learning experience.  Rob encourages delegates to bring up their real-world project concerns so that Rob can selection of a wide set of relevant recent examples and case studies.  Rob is not prescriptive; he shows what he and his many customers have done to solve similar problems in various contexts.

Topical Outline

• Automating Test Design
  
  • Variable Identification.
  • Mind Mapping.
  • Equivalence Analysis.
  • Boundary Testing.
  • Decision Modeling.
  • Story Testing.
  • Scenario Based Testing.
  • Control Flow Testing.
  • Path Analysis.
  • Pareto Analysis.
  • Pairwise Combinations Testing.
  • Permutations.
  • Combinations.
  • Selections.
  • Sampling.
  • Randomizing.
• Automation Fundamentals
  
  • Observing
    
    • What do we want to learn about?
    • How does this relate to purpose?
    • Direct information.
    • Indirect information.
    • Shadows on a wall.
      
      • The parable of the cave.
    • Variance.
    • Time.
    • Frequency.
    • Pattern.
    • States.
    • Data.
    • Noise.
  • Controlling
    
    • Hooks.
    • Application Program Interfaces.
    • Data driven.
    • Simulating Operation.
    • Triggers.
    • Events.
    • Messages.
    • What is good enough.
  • Scripting
    
    • Data Driven.
    • Keyword Driven.
    • Probes.
    • How to learn a new scripting language.
      
      • Fieldstone approach.
    • Dabbling with Scripts
      
      • Python Combinations and Permutations Test Design.
      • Python Keyword Driven Example.
      • Python Rest API Testing.
      • Ruby Keyword Driven Example.
      • Ruby Rest API Testing.
• Assessing correctness.
  
  • Test oracles.
  • Knowing the truth.
  • Domain expert validation.
  • Comparators.
  • Fuzzy Matches.
  • Heuristics.
  • Algorithms.
• Data integrity.
  
  • Check summing.
  • Self Verifying Data.
  • Self Correcting Data.
  • Stained Data.
  • Corruption.
  • Inexplicable Data Variation.
  • Fuzzing.
• Notion of Regression
  
  • Regression Testing
    
    • Automation is not Regression.
    • All automation is not regression.
    • The automatability of regression.
    • All regression is not automation.
    • The regress-ability of automation.
  • Regression confidence.
    
    • Warm fuzzy feelings
    • Dashboards
    • Reality
    • Risks
    • Rewards
  • Regression strategies.
    
    • Smoke Testing Build Readiness
    • Smoke Testing Environment Readiness
    • Functional Acceptance Simple Tests (FAST)
    • Railroading
    • Shot Gunning