Pairwise Testing

I have learned about pairwise testing (or sometimes called all-pairs testing) in the past as being a good test specification technique to come up with a meaningful and manageable set of test cases. Until now I have experimented with it but never used it in anger.

All-pairs in practice

At the moment I am working on a test for a complex feature of the system pictured below:

In a workshop session this morning I gathered subject matter experts, and defined a set of parameters relevant for the feature under test. And there are a lot of them, with each a number of possible values.

Getting a full coverage with all possible permutations would have resulted in 29952 test cases. For reference, with the way the system works (keep complex in mind) it takes 1/2 an hour to fully run and analyse 1 test case. This would take almost 2 years(!) back-to-back testing to get through.

With using the test specification technique of pairwise testing, I managed to reduce the set of test cases to 62. That is a big reduction but it doesn’t compromise on coverage. With execution these tests (still 31 hours but it only will be run over the weekend) there is confidence this feature will be adequately covered.

Tools

To make life easier applying pairwise testing it is highly recommended to use a tool. There are various ones out there; a list of them are mentioned in this article.

The one tool I particularly like is the Pairwise Independent Combinatorial Tool (PICT) from Microsoft. (turns out they do make useful products 😉). The mean reason I like is, that it is one executable which completely runs from the command line. It has got a lot of flexibility and executes quick.

A command line tool might not be your cup of tea; have a look at the other tools 😃.

Happy pairwising!

This post is part of a series of articles about “What I Have Learned Today”; acknowledging every day the thing I have learned. Feel free to read the other posts. all opinions in this article are my own and not necessarily represent the views of my employer.