Test coverage assessment and pairwise test generator

I am a kind of tester who is concerned about coverage as much as I am concerned about bugs. Most people use Requirements Based Testing to show a sense of coverage to the customers and that's how the customers get fooled. Neither are the requirements complete nor are any coverage gimmicks using test cases good enough.

There are several kinds of coverage I am concerned about. This is what I wrote in Test Republic recently on coverage, "I see RBT as the most and bare minimum approach that is widely practiced in the places that I have been in and heard of. Achieving a fair RBT itself is a challenge due to ambiguity and other factors.

Code coverage is one kind of modeling. Error message coverage is another. So are, time, data, operations, platform etc. Our actual coverage ( that can't be measured by a number ) depicts our ability to model and re model a system under test.

After experimenting, I have started to think about cross coverage. When I test for a timing coverage, I might have done some part of RBT, some part of data coverage and others although I can quantify only the timing coverage at that moment."

It struck to me that one of the ways I would want to test for data coverage is by help of tools. One of them is definitely TestersDesk Pairwise Test Data Generator. Testing is sampling and that says you can't execute all possible tests. If I'd use Pairwise tool @ Testersdesk, it helps me with set of samples that is of value to my testing.

For instance, if I'd be in a situation of just one day to test a product ( which is what usually I am in ) and want to provide my client with coverage of my tests and I'd want data coverage and justify why this set of data I chose was meaningful, I'd be using Pairwise Test Data Generator for my tests.

Oh, you challenge me. Assuming I am testing an auction system ( that I did test a couple of years ago ), I'd have the following challenges for data coverage:

Auction status: Closed, Open, Yet to be opened
Goods Category: F&B, Electronics, Computers, Electrical, Peripherals, Machines, Jewelery, Garments
Time Zones: GMT, PST, EST, IST
Availability: Stocked, Sold out, Last few pending
Bidder_Category: Registered, New, Unregistered

Which leads to about 800+ combinations to test. So, I wouldn't want to run all of them ( unless my client has so much of money and time to give it to me ) and hence the Pairwise Test Data Generator offers me the following 33 combinations

Auction status,Goods Category,Time Zones,Availability,Bidder_Category
Closed, F&B, GMT, Stocked, Registered
Open, F&B, PST, Sold out, New
Yet to be opened , F&B, EST, Last few pending , Unregistered
Closed, F&B, IST , Sold out, Unregistered
Open, Electronics, GMT, Last few pending , Registered
Yet to be opened , Electronics, PST, Stocked, New
Closed, Electronics, EST, Sold out, Registered
Open, Electronics, IST , Stocked, Unregistered
Yet to be opened , Computers, GMT, Sold out, New
Closed, Computers, PST, Last few pending , Registered
Open, Computers, EST, Stocked, New
Yet to be opened , Computers, IST , Last few pending , Registered
Closed, Electrical, GMT, Stocked, Unregistered
Open, Electrical, PST, Sold out, New
Yet to be opened , Electrical, EST, Last few pending , Registered
Closed, Electrical, IST , Last few pending , New
Closed, Peripherals, GMT, Stocked, Unregistered
Open, Peripherals, PST, Sold out, Unregistered
Yet to be opened , Peripherals, EST, Last few pending , Registered
any_value_of_Auction status, Peripherals, IST ,any_value_of_Availability, New
Closed, Machines, GMT, Stocked, Unregistered
Open, Machines, PST, Sold out, Registered
Yet to be opened , Machines, EST, Last few pending , New
any_value_of_Auction status, Machines, IST ,any_value_of_Availability,any_value_of_Bidder_Category
Closed, Jewelery, GMT, Stocked, Unregistered
Open, Jewelery, PST, Sold out, Registered
Yet to be opened , Jewelery, EST, Last few pending , New
any_value_of_Auction status, Jewelery, IST ,any_value_of_Availability,any_value_of_Bidder_Category
Closed, Garments, GMT, Stocked, Unregistered
Open, Garments, PST, Sold out, Registered
Yet to be opened , Garments, EST, Last few pending , New
any_value_of_Auction status, Garments, IST ,any_value_of_Availability,any_value_of_Bidder_Category
any_value_of_Auction status, Computers,any_value_of_Time Zones,any_value_of_Availability, Unregistered

So, what's my strategy to test?

I'd use the above test data as a sample of my test data coverage and investigate for problems. The moment I find a problem. So let's assume, I found a problem with IST Time zone causing a crash ( which I did when I was testing the auction system a couple of years back ) I'd want to do more combinations of IST with other parameters. That gives me a need to use N-Way-Random Combination Generator to sequence all tests I need for my investigation that surrounds IST Time Zone.

Wow! That's exciting.