Test coverage assessment and pairwise test generator

I am a kind of tester who is concerned about coverage as much as I am concerned about bugs. Most people use Requirements Based Testing to show a sense of coverage to the customers and that's how the customers get fooled. Neither are the requirements complete nor are any coverage gimmicks using test cases good enough.

There are several kinds of coverage I am concerned about. This is what I wrote in Test Republic recently on coverage, "I see RBT as the most and bare minimum approach that is widely practiced in the places that I have been in and heard of. Achieving a fair RBT itself is a challenge due to ambiguity and other factors.

Code coverage is one kind of modeling. Error message coverage is another. So are, time, data, operations, platform etc. Our actual coverage ( that can't be measured by a number ) depicts our ability to model and re model a system under test.

After experimenting, I have started to think about cross coverage. When I test for a timing coverage, I might have done some part of RBT, some part of data coverage and others although I can quantify only the timing coverage at that moment."

It struck to me that one of the ways I would want to test for data coverage is by help of tools. One of them is definitely TestersDesk Pairwise Test Data Generator. Testing is sampling and that says you can't execute all possible tests. If I'd use Pairwise tool @ Testersdesk, it helps me with set of samples that is of value to my testing.

For instance, if I'd be in a situation of just one day to test a product ( which is what usually I am in ) and want to provide my client with coverage of my tests and I'd want data coverage and justify why this set of data I chose was meaningful, I'd be using Pairwise Test Data Generator for my tests.

Oh, you challenge me. Assuming I am testing an auction system ( that I did test a couple of years ago ), I'd have the following challenges for data coverage:

Auction status: Closed, Open, Yet to be opened
Goods Category: F&B, Electronics, Computers, Electrical, Peripherals, Machines, Jewelery, Garments
Time Zones: GMT, PST, EST, IST
Availability: Stocked, Sold out, Last few pending
Bidder_Category: Registered, New, Unregistered

Which leads to about 800+ combinations to test. So, I wouldn't want to run all of them ( unless my client has so much of money and time to give it to me ) and hence the Pairwise Test Data Generator offers me the following 33 combinations

Auction status,Goods Category,Time Zones,Availability,Bidder_Category
Closed, F&B, GMT, Stocked, Registered
Open, F&B, PST, Sold out, New
Yet to be opened , F&B, EST, Last few pending , Unregistered
Closed, F&B, IST , Sold out, Unregistered
Open, Electronics, GMT, Last few pending , Registered
Yet to be opened , Electronics, PST, Stocked, New
Closed, Electronics, EST, Sold out, Registered
Open, Electronics, IST , Stocked, Unregistered
Yet to be opened , Computers, GMT, Sold out, New
Closed, Computers, PST, Last few pending , Registered
Open, Computers, EST, Stocked, New
Yet to be opened , Computers, IST , Last few pending , Registered
Closed, Electrical, GMT, Stocked, Unregistered
Open, Electrical, PST, Sold out, New
Yet to be opened , Electrical, EST, Last few pending , Registered
Closed, Electrical, IST , Last few pending , New
Closed, Peripherals, GMT, Stocked, Unregistered
Open, Peripherals, PST, Sold out, Unregistered
Yet to be opened , Peripherals, EST, Last few pending , Registered
any_value_of_Auction status, Peripherals, IST ,any_value_of_Availability, New
Closed, Machines, GMT, Stocked, Unregistered
Open, Machines, PST, Sold out, Registered
Yet to be opened , Machines, EST, Last few pending , New
any_value_of_Auction status, Machines, IST ,any_value_of_Availability,any_value_of_Bidder_Category
Closed, Jewelery, GMT, Stocked, Unregistered
Open, Jewelery, PST, Sold out, Registered
Yet to be opened , Jewelery, EST, Last few pending , New
any_value_of_Auction status, Jewelery, IST ,any_value_of_Availability,any_value_of_Bidder_Category
Closed, Garments, GMT, Stocked, Unregistered
Open, Garments, PST, Sold out, Registered
Yet to be opened , Garments, EST, Last few pending , New
any_value_of_Auction status, Garments, IST ,any_value_of_Availability,any_value_of_Bidder_Category
any_value_of_Auction status, Computers,any_value_of_Time Zones,any_value_of_Availability, Unregistered

So, what's my strategy to test?

I'd use the above test data as a sample of my test data coverage and investigate for problems. The moment I find a problem. So let's assume, I found a problem with IST Time zone causing a crash ( which I did when I was testing the auction system a couple of years back ) I'd want to do more combinations of IST with other parameters. That gives me a need to use N-Way-Random Combination Generator to sequence all tests I need for my investigation that surrounds IST Time Zone.

Wow! That's exciting.

End of July 2009- What's going on?

Testing community will benefit more from version 2.0 of TestersDesk.com that will be released end of August. It brings more usability based on feedback received; increased limits on inputs and outputs, a new set of tools like an Image Generator for a given set of test requirements and more... stay tuned.

We are also working on some new innovative tools that are too early to reveal at the moment.

We are pleased to see the daily increase in the registered users as well as returning users. When testers use our products to increase their productivity, it gives us a long lasting happiness and the motivation to do more. Each minute is an exciting moment as testers get help from our tools in their daily activities of finding bugs in the software through our new/innovative testing tool.

Testers who perform manual test execution anyway have been using our tools regularly and recently we have seen increased growth even in the usage from test engineers working automation tools (for example, our N-way Test Case Generator is being used to automatically data-drive the test script with different possibilities through the generated CSV output of test combinations, essentially working like a tool-centric decision table).

Every day, we receive mails from users thanking us for the free online toolkit, passionate testers who have explored TestersDesk.com started blogging on us (for example, Ajay Balamurugadas in his blog wrote his experiences with TestersDesk.com) and so on.

Do write to us with any use cases on how you benefited from TestersDesk.com and any new requests/suggestions/whatever you may want to communicate in growing the online toolkit.

How humble looking 'Template Based File Generator' exploded bomb at my work palce !

This post is about “Template-based File Generator” provided by TestersDesk, oh sorry, wait a minute, let me correct- this post is about my encounter with that tool. For technical help about the feature, of course there is a manual within the tool, but this is an off tangent from my experience. Should you be interested to know of a use case, do read on…

During my school days, in a debate competition, we were supposed to speak on behalf of science and other group against it. Their argument was that science developed lot of bombs and subsequently causing loss of human life, after a heated discussion one of our team members said ‘it’s the place where bomb is exploded’. Wow! That was an explosion in all our minds, a mind blowing answer. Bombs are utilized in numerous ways apart from shelling them on people.

That’s fine but what does bombs have got to do with testing? I see similar thing going on in my career. We developed a beautiful test suite for 13 modules of an ERP product, where each module approximately has not less than 40 test scripts for regression. Total test script assets would be around 520 (data files). Once the test suite is ready, for every regression we have to change test data in almost all the test data files.

I don’t know whether you believe if I say that it is taking on an average approximately 2 complete days for single resource to update all 520 test data files. Few of my concerns while preparing test data for every regression were…

1. Some test data becomes obsolete, we have to change it every time we run regression, or else tests would stop. For example
a. New test data for invoices, purchase orders, receipts, journal names to name a few.
b. Amount to be entered in all of the above type of bills.
c. Passwords for few users need to be changed every time we do regression.

2. Add new column to test data files
a. Sometimes few screens in application has additional text fields (this is when they add descriptive flex field), which forces us to open each test data file and include a new column.
b. When additional details are to be entered into application we have to add few columns in test data files.
c. When a new custom screen is developed ( although it is rare, it does happens)

Features @Testersdesk look very simple but more I understand and keep on using them, more powerful they are turning out.

Now look at ‘Template based file generator’ feature, it looks so simple and humble with just 2 files for input. Once you feed the input files it throws a set of files at you. Long back around a year ago when I looked at this feature, I could not understand much of its strength.

Let me tell you how this bomb exploded in right place. In my case I just kept all test data I ever need to run my gigantic test suite in single master test data file. Once all the new data is updated, I just take a Template file along with and throw these two into ‘Template based file generator’. In almost no time I am getting beautiful set of all test data files, which I directly use to drive my test suite.

You see, now it is taking one of my resources just 3 – 4 hours to fill the correct test data in master test data file, and almost no time to get all those 520 test data files filled with new data.

Here is the beauty with this tool, if I am supposed to add new column to all test data files, effort from my side would be to just add another column in my master test data file.

What a great reduction in mundane manual tasks, when substituted with TestersDesk to my team.

There is a plethora of areas where the humble looking ‘Template based file generator’ can explode a bomb at your work place.

I never imagined life would be so easy with Testersdesk. I understood true meaning of these pictures-

Happy TestersDesking !

Cheers,
Virtualkey(Suresh)

Exploring Test Design Capabilities of TestersDesk

What is a test tool without it undergoing testing? The kind of testing you would witness in the following video is a kind of acceptance test that I did to help myself believe, if TestersDesk could be of help to me for my testing work.

I am a kind of tester who is focused on cost versus value and I'd not want to end up wasting a lot of time on something that is of little value and that was one of the key things I plan to test it out. I wanted to take a complex product and attack an area that looks like simple but might not be so. I found Microsoft Word as a very good product that matches my requirement.



Does TestersDesk clear my tests? Watch the video to find it out!






After you are done watching the video (of course you may use any ideas that you witnessed) check if you do any kind of acceptance tests before you bring in your tool to your armory.

I'd be interested to know what kind of acceptance tests you would do for a tool and what results did you get on TestersDesk. Contribute ideas and that will be fun to learn.

My first encounter with TestersDesk and my first post as its product blogger

Hi TestDesking Enthusiasts !

I was wondering on how many different areas does a tester might leverage features of testersdesk.com while testing any typical web application and I started with an email service provider (gmail). (Note: Usage of TestersDesk doesn't confine to just web applications, infact any application with its functionality exposed through screens of fields can benefit from TD).

Coming back to the point. Let us closely examin all features of a typical E-Mail service provider (gmail) right from Registration to requesting for deactivation of our email account and identify...

1. All different areas where Test Design and Test Data tools of TestersDesk can be leveraged.
   
2. Areas where TestersDesk can help us prepare better test data which yield better bug removal efficiency.

Scenario: Possible test cases for Registration Screen

Let's examin possible types of test data required to test a registration page for gmail account. If you look at top right corner there is something called 'Change Language' option (list box), and if i am not wrong there are 52 values in that list box. Which means you have to prepare following (i.e we need to generate test data in 52 different languages and test it).

Do you think like "What the heck ! who the hell on earth does all that stuff ! ". Ok ! what do you think google releases without testing them all ? I don't think so ! This implies guys at google must have undergone all the pain or used some or other technique to generate test data for all 52 languages.

Lets examine test data required for one language (English-US), only for registration page.There are 9 text fields in total, out of which one field is 'Terms of service' field. Here is a list of features of TestersDesk which can be used (classified according to Category of Tools).

Field Name	Category	TD Feature
First Name	Test Data	Person Name Generator
Last Name	Test Data	Person Name Generator
Desired Login Name	Test Data	Person Name Generator
Desired Login Name	Test Data	Email Address Generator
Change Password	Test Data	Password Generator
Re enter Password	Test Data	Password Generator
Answer	Test Data	Quantified String Generator
Secondary E-Mail	Test Data	Email Address Generator
Word Verification	Test Data	Quantified String Generator
All above fields individually	Test Data	Quantified String Generator
All above fields together	Test Design	Pairwise TestCase Generator

 

Lets roughly estimate the test data required to be generated for registration page alone.

Test Data = (All types of test input values for each field ) * (8 fields) * (52 Languages)

Lets imagine the time it would have taken for guys at google or where ever to prepare this amount of test data just for one page.

Using TestersDesk any novice could generate this test data on the fly, now do you think we need to discuss more on how much time is saved... stuff like that , that should be quite clear by now, isn't it !

I think we need to discuss more on every area where we can leverage TD.

All that one need is little understanding of areas where tools at Testersdesk can be leveraged.

How about gaining little insight into using TD for Login Screen in next post?

There are lot of other features yet to be explored, keep visiting.

Until then, Happy Testdesking !

Virtualkey

Current sources of help to use TestersDesk.com

Dear reader,

This is to list the sources we have today (27th April, 2009) that could help in using www.TestersDesk.com.

The basic purpose of TestersDesk.com can be read in a minute or two, from this link.

You could read the Frequently Asked Questions here. If you have questions that are not covered in it, do write to me at my email (ap at testersdesk dot com).

As many of you know, the key features of TestersDesk.com is its Pairwise TestCase Generator that is rich in using the test coverage techniques to produce a minimal set of test combinations. Please do read about it here.

Must read - There are at least 20 different ways how the Online Toolkit can benefit Software Test Design and Test Data Generation activities (whether you are doing manual testing or using test automation tools) - They are documented here.

There are a few videos at this page. Some done by me and some by Pradeep Soundararajan (an independent test consultant and also a trainer who is known to many through his blog) here. Do watch them in a sequence when time permits.

And please stay tuned, there is more to come, help-wise or tool-wise.

Of course it is good that many users are straight on to the regular usage of the tools, and the above links are for the rest of us.

Cheers
Ashwin

Bug reporting skills, Good testers and "I entered something and it crashed" syndrome

I have roamed around India speaking to testers from different cities. I notice common patterns and region specific patterns about how testers think and or communicate.

I know there would be a never ending argument of what are the attributes of good testers because everyone seem to have a different opinion. However, I think there exists a set of attributes that any school of thought would agree upon.

Let me discuss one such and you might have guessed it from the title. I have never seen a good tester say, "I entered something and something and it crashed". But why do they not say?

I have heard the above sentence from a lot of testers and most of them have been clueless about software testing. I shall explain to you my reasoning of the same.

Assume you are the cockroach catcher and that's what you do for a living. I hire you to catch a cockroach that I saw in my home and you are going to ask me, "Where did you see the cockroach?" and if I dont have an answer for that how helpful am I being to myself in getting the cockroach out of my home?

If I am staying in a 2 acre bungalow, your job is all the more complicated. Instead, I say, "I last spotted a couple in the kitchen and the bathroom", I am helping myself and you.

Look into your organization's bug tracking system and you'd find testers reporting problem like, "I entered some input and application crashed". Now, that's like reporting, there is a cockroach in this country find it.

Unless you are specific as a tester, how do you suppose a developer to fix the problem. Testers are finding a bug to get it resolved or help the management take a better decision. Reporting it in such a way that it is of no help to anyone is an extreme bad job.

Why would a developer ever want to respect such a tester? Why would the developer community want to respect such community of testers?

So, if you have been doing it and want to be better, here is a way:

You may want to look into what the Quantified String Generator can do to solve this problem of yours or any other toolkit at Common Test Data Generator in Testers Desk

Here is an output:

2^4^6^8^11^14^17^20^23^26^29^32^35^38^41^44^47^50^53^56^59^62^65^68^71^74^77^80^83^86^89^92^95^98^102^106^110^114^118^122^126^130^134^138^142^146^150^154^158^162^166^170^174^178^182^186^190^194^198^202^206^210^214^218^222^226^230^234^238^242^246^250^254^258^262^266^270^274^278^282^286^290^294^298^302^306^310^314^318^322^326^330^334^338^342^346^350^354^358^362^366^370^374^378^382^386^390^394^398^402^406^410^414^418^422^426^430^434^438^442^446^450^454^458^462^466^470^474^478^482^486^490^494^498^502^506^510^514^518^522^526^530^534^538^542^546^550^554^558^562^566^570^574^578^582^586^590^594^598^602^606^610^614^618^622^626^630^634^638^642^646^650^654^658^662^666^670^674^678^682^686^690^694^698^702^706^710^714^718^722^726^730^734^738^742^746^750^754^758^762^766^770^774^778^782^786^790^794^798^802^806^810^814^818^822^826^830^834^838^842^846^850^854^858^862^866^870^874^878^882^886^890^894^898^902^906^910^914^918^922^926^930^934^938^942^946^950^954^958^962^966^970^974^978^982^9

Now, whether the developer or specification talks about Boundary or not, you might be able to find it out using this Quantified String Generator. For instance, let me assume you copy paste the above on to your input field and it accepts 983 characters and it then appears to crash as you hit the submit button.

Now, investigating this bug is an interesting challenge: Typical bad testers would directly report the problem however here is a pattern that I observe in most of the good testers I have met - They would want to know from which number does this crash occur.

Is it 983 characters or is it a number below that?

So, we go for a smaller sample than 983 characters: And here is a sample output from Quantified String Generator:

2^4^6^8^11^14^17^20^23^26^29^32^35^38^41^44^47^50^53^56^59^62^65^68^71^74^77^80^83^86^89^92^95^98^102^106^110^114^118^122^126^130^134^138^142^146^150^154^158^162^166^170^174^178^182^186^190^194^198^202^206^210^214^218^222^226^230^234^238^242^246^250^254^258^262^266^270^274^278^282^286^290^294^298^30

Tip: Did you notice the beauty that you dont need to count the characters and it counts itself?

That's 300 characters.

Now, in case we see that it does not crash for 300 characters, we have learned that it is somewhere between 300 and 983. And as we zero down, we might find that great number from which the application starts crashing.

Imagine you reporting a problem like, "I used the Quantified String Generator and discovered that 432 characters ( and thereafter ) appears to crash the application" and ask a striking question to the developers, "Why would it crash at 432?", "What is so magical about 432?"

Note: If you'd want to see that in demonstration, you may consider watching this video.

That's when the developer would start respecting a tester for his skills of bug reporting, bug investigation and communication.

I leave the choice to you. You need to do lots of things and change the way you do things to become a good testers and you just need to do whatever you are doing right now to be a bad tester.