Learn about automating test design.
Software testing is intractable. Even the simplest feature has more possible tests than can be done in a lifetime. Automated test design helps you select a reasonable set of test conditions to meaningfully explore important risks. Automated test design guides selection of meaningful combinations, permutations, samples, and distributions of factors which influence, or are influenced by, the behaviour of software under test.
There exist a plethora of platforms and system configurations which testing cannot ignore. Different versions, configurations and locales of hardware, software, operating systems and even third party coresident software are an enormous risk, especially for mobile and web-based technologies. Automated test design can establish a reasonable set of platforms to exercise thoroughly and efficiently.
Automated test design can help identify interesting pathways, through any complex series of operations, can be identified automatically deriving a reasonably small number of tests to exercise complex patterns and expose important bugs and weakness and point out areas of instability.
Automated test design can extract great tests from absurd collections of complex business rules.
Learn about designing test automation.
Today many practitioners learn test automation by studying how to use specific tools to manipulate software simulating transactions in a myriad of ways. Almost all test automation is taught as part of a regression testing strategy to repeat manual checking repeatedly. But not here. In this course we clearly demonstrate that all automation is not regression and automation can a helpful tool to solve a wide variety of testing conundrums. Test automation does not replace testers, it enables them!
In this course we will look at how test automation can be an important utility in solving all manners of testing problems. Real testing problems can be solved leveraging automation the right way in the right time.
In this course test automation examples are explored and discussed in depth. Fundamentally, automation provides a means of controlling and observing software under test. What we need to see and observe depends on our goals.
Each case studies review will expose automation strategies which delegates will explore through a series of homework assignments.
We will look at many different automation models and examples of the real-world problems they have solved.
Case studies include:
- Embedded in device unit test system.
- Safety critical software exercised with keyword driven approaches.
- Qonfidence building Agile story testing with gherkin based transactional regression.
- Failure modes in business-critical software tested with mocking strategies.
- Live system monitoring with run time dynamic analysis automation.
- Live system failure mode testing with chaos monkeys.
- Security risk assessment with build time static analysis.
- In sprint (Scrum) data sensitive performance testing with J Meter.
- Applications developed with example driven approaches. (ATDD and BDD).
- Code developed with test driven approaches (TDD).
- Several examples of Continuous Integration Continuous Delivery pipelines.
About Automating Test Design
Testers can save a lot of time—and focus on what really matters—by using automated test design and implementation tools. Traditional test automation helps testers execute tests, report progress, and offer insights into test coverage and completeness, but little emphasis has been made on automating effective test design.
Rob Sabourin will share automated test design techniques using a selection of free and open-source tools.
Rob has implemented automated test design in conjunction with automated unit tests and automated story tests on many software projects, and he will share examples demonstrating automatic test design on many technology stacks using popular unit and story test frameworks.
Rob will illustrate how complex transactional business tests are defined with automated decision table analyzers using state-of-the-art logic reduction algorithms, as well as construct test cases navigating complex integration paths, microservice APIs, and functionality chains using automatic path analysis tools.
You will learn to automate test generation on many technology stacks for several architectures, including REST APIs, microservice architectures, cloud computing solutions, multi-tiered architectures, and mobile and embedded systems, to efficiently discover critical bugs.
About Designing Test Automation
Many of Rob’s customers, students and professional colleagues are concerned about building skills in test Automation. In this course Rob shares a problem-solving approach which can help you leveraging test automation tools and technologies to help solve some complex testing problems. Rather than mastering any specific tool or technology stack Rob shares many ways that automation can support testing. Automation can be an important part of your testing solution space. Rob has succeeded in many projects by tooling and instrumenting projects leveraging a multitude of automation strategies over the past 40 years.
Automated Test Design and Designing Test Automation
This course addresses the topic of design.
Instead of prescribing how test automation should be used in software testing, this course looks at the testing from a problem-solving approach.
Given a clear test objective, tied to project-specific business goals, Rob Sabourin will share how a wide variety of tools and techniques can be used to efficiently deliver valuable meaningful results.
The issue of test design will be reviewed looking at how tools and technologies can be used to support the design of excellent tests. Test design is about focusing, and different test design techniques can be applied to help fulfill the test objective. Automated test design tools and techniques can be used in the identification of a meaningful relevant and minimal set of variables, conditions, and factors, selecting appropriate trials to exercise the application, and learn about the risks that really matter.
Specifically starting with variable identification Robs will investigate collaboration tools, mind maps, spreadsheets, and technology-oriented tools such as Postman and Static Analysis tools such as SonarQube and J-Test to help identify the variables of interest.
Multiple variable test design techniques focused on business rules can be supported with tools such as decision table editors and APIs such as Open Rules.
Test designs for transactional, data-oriented, or stateful systems can be implemented with path analysis tools used to help identify a reasonable subset of the vast number of possible pathways through a network.
Combinations test design techniques, such as Pareto analysis can be performed with spreadsheets and with many publicly available libraries and tools.
Rob will show how simple Python scripts can be assembled to generate excellent combinations, permutation, selection, and sample-oriented tests.
About Teaching Approach
See it. A Bit of Show and Tell.
Rob provides a series of illustrative examples of each test design and test automation approach. The context, testing problem and proposed solutions are reviewed.
Try it out yourself.
Delegates are given a chance to implement some of the test design approaches as in-class and homework exercises. Although some tools may be required all are free or open-sourced and have been used by Rob on both Windows and Mac Platforms.
Do it.
Homework problems give delegates a chance to attempt more complex problems, combine techniques and even apply techniques to their own problem space.
Make it real. Case Studies and Experience Reports.
Rob presents case studies from recent project contexts which blend many test automation tools and techniques to help solve complex problems. All the case studies come from professional task analysis and include relevant business, technical and organizational factors.
Dynamic Energetic and Relevant.
Rob provides a fun, educational, and high-energy learning experience. Rob encourages delegates to bring up their real-world project concerns so that Rob can selection of a wide set of relevant recent examples and case studies. Rob is not prescriptive; he shows what he and his many customers have done to solve similar problems in various contexts.
Topical Outline
- Automating Test Design
- Variable Identification.
- Mind Mapping.
- Equivalence Analysis.
- Boundary Testing.
- Decision Modeling.
- Story Testing.
- Scenario Based Testing.
- Control Flow Testing.
- Path Analysis.
- Pareto Analysis.
- Pairwise Combinations Testing.
- Permutations.
- Combinations.
- Selections.
- Sampling.
- Randomizing.
- Automation Fundamentals
- Observing
- What do we want to learn about?
- How does this relate to purpose?
- Direct information.
- Indirect information.
- Shadows on a wall.
- Variance.
- Time.
- Frequency.
- Pattern.
- States.
- Data.
- Noise.
- Controlling
- Hooks.
- Application Program Interfaces.
- Data driven.
- Simulating Operation.
- Triggers.
- Events.
- Messages.
- What is good enough.
- Scripting
- Data Driven.
- Keyword Driven.
- Probes.
- How to learn a new scripting language.
- Dabbling with Scripts
- Python Combinations and Permutations Test Design.
- Python Keyword Driven Example.
- Python Rest API Testing.
- Ruby Keyword Driven Example.
- Ruby Rest API Testing.
- Assessing correctness.
- Test oracles.
- Knowing the truth.
- Domain expert validation.
- Comparators.
- Fuzzy Matches.
- Heuristics.
- Algorithms.
- Data integrity.
- Check summing.
- Self Verifying Data.
- Self Correcting Data.
- Stained Data.
- Corruption.
- Inexplicable Data Variation.
- Fuzzing.
- Notion of Regression
- Regression Testing
- Automation is not Regression.
- All automation is not regression.
- The automatability of regression.
- All regression is not automation.
- The regress-ability of automation.
- Regression confidence.
- Warm fuzzy feelings
- Dashboards
- Reality
- Risks
- Rewards
- Regression strategies.
- Smoke Testing Build Readiness
- Smoke Testing Environment Readiness
- Functional Acceptance Simple Tests (FAST)
- Railroading
- Shot Gunning
- Hybrid Models
- Tagged Models
- Relationship to
- Having Fun with Spreadsheets
- Spreadsheets in test design
- Pareto Analysis
- Monte Carlo Approaches
- Randomizing
- Spreadsheets in test automation
- Data sources
- Sampling
- Log file analysis
- Learning how to use testing tools & technologies.
- Computing models.
- Books about programming.
- Books about tools.
- References.
- Your milage may vary.
- Driven to confusion.
- Test Driven Development.
- Acceptance Test Driven Development.
- Behaviour Driven Development.
- Example Driven Development.
- Ticket Driven Software Development.
- Static Analysis.
- Code Metrics.
- Code Standards Compliance.
- Application Program Interface.
- Potential Security Risks.
- Maintainability.
- Unreachable Code.
- Risky Patterns.
- Automated unit test generation. (Tools like J-Test)
- Reusable Code.
- Defect Detection.
- Redundant Code.
- Dynamic Analysis.
- CPU Usage.
- Memory Usage.
- Memory Leaks.
- Memory Access Violations.
- Bounds Checking.
- Process Monitoring.
- Performance Monitoring.
- Storage Monitoring.
- Network Traffic Monitoring.
- Counters.
- Flip Tests.
- Coverage Monitoring.
- Transaction Log Monitoring.
- Notions of Coverage.
- The Myths.
- The Methods.
- The Risks.
- The Rewards.
- The Benefits.
- The Consequences.
- Automation in Multilingual Computing.
- Localization
- Internationalization
- Platform compatibility
- Automation in Non-Functional Testing
- Topic of Non-Functional Requirements.
- Designing experiments.
- Automation in Non-Functional experiments.
- Load Testing.
- Stress Testing.
- Usability.
- Security.
- Automating in Live Testing
- Chaos Monkeys
- Dynamic Analysis
- Case Studies
- Data
- Test Design
- Real.
- Sampled.
- Generated.
- Hybrid.
- Simulated.
- Obfuscated.
- Issues of size.
- Issues of dynamics.
- Data integrity.
- Data age.
- Software Development Life Cycle Dependencies.
- Waterfall
- Spiral
- Structured
- Agile
- Hybrid
- Service Oriented
- Mocking.
- Approach.
- Test Doubles.
- Unit Testing.
- Story Testing.
- Sandbox Testing.
- Code.
- Process.
- Tools.
- Mechanics.
- Failure mode simulation.
- Technology Stack Dependencies.
- Plenty of examples.
- Technical Collaboration.
- Developers.
- Contractors.
- Technology Experts.
- So Called Test Automator.
- Risks.
- Hired Gun.
- Ownership.
- Maintenance.
- Support.
- Evolution.
- Porting.
- The crazy world of frameworks.
- Framework is a vague term.
- Myth.
- Reality.
- Waste.
- Benefit.
- Continuous Integration.
- Frequent Builds.
- Automated Builds.
- Self-Tested Builds.
- Fast Builds.
- Accessible Builds.
- Automating Story Tests
- FitNesse
- Structure
- Test Definitions
- Gherkin
- Structure
- Test Definitions
- Instrumentation
- Testing Hooks
- Policy
- Practise
- Technical Work Breakdown
- Automation Library Models.
- Controller Models.
- Step Based Models.
- Domain Specific Models.
- Technology Specific Models.
- Basic Atomic Operations.
- Test Procedure
- Test Logic
- Implementation Free.
- Same Test Should Work on Dramatically Different Application Environments.
- Some Automation Programming Practises
- Think like a stack.
- Named objects.
- Design Patterns.
- Some More Automation Paradigms.
- Contract Testing.
- Cloud robot device testing farms.
- Model Driven Test Automation.
- Automation Return on Investment Myths.
- Skills matter
- Testing
- Programming
- Automation
- Domain
- Modeling
- Metaphors
- Dash Boards
- VCR
- Capture
- Playback
