I am a kind of tester who is concerned about coverage as much as I am concerned about bugs. Most people use Requirements Based Testing to show a sense of coverage to the customers and that's how the customers get fooled. Neither are the requirements complete nor are any coverage gimmicks using test cases good enough.
There are several kinds of coverage I am concerned about. This is what I wrote in Test Republic recently on coverage, "I see RBT as the most and bare minimum approach that is widely practiced in the places that I have been in and heard of. Achieving a fair RBT itself is a challenge due to ambiguity and other factors.
Code coverage is one kind of modeling. Error message coverage is another. So are, time, data, operations, platform etc. Our actual coverage ( that can't be measured by a number ) depicts our ability to model and re model a system under test.
After experimenting, I have started to think about cross coverage. When I test for a timing coverage, I might have done some part of RBT, some part of data coverage and others although I can quantify only the timing coverage at that moment."
It struck to me that one of the ways I would want to test for data coverage is by help of tools. One of them is definitely TestersDesk Pairwise Test Data Generator. Testing is sampling and that says you can't execute all possible tests. If I'd use Pairwise tool @ Testersdesk, it helps me with set of samples that is of value to my testing.
For instance, if I'd be in a situation of just one day to test a product ( which is what usually I am in ) and want to provide my client with coverage of my tests and I'd want data coverage and justify why this set of data I chose was meaningful, I'd be using Pairwise Test Data Generator for my tests.
Oh, you challenge me. Assuming I am testing an auction system ( that I did test a couple of years ago ), I'd have the following challenges for data coverage:
Auction status: Closed, Open, Yet to be opened
Goods Category: F&B, Electronics, Computers, Electrical, Peripherals, Machines, Jewelery, Garments
Time Zones: GMT, PST, EST, IST
Availability: Stocked, Sold out, Last few pending
Bidder_Category: Registered, New, Unregistered
Which leads to about 800+ combinations to test. So, I wouldn't want to run all of them ( unless my client has so much of money and time to give it to me ) and hence the Pairwise Test Data Generator offers me the following 33 combinations
Auction status,Goods Category,Time Zones,Availability,Bidder_Category
Closed, F&B, GMT, Stocked, Registered
Open, F&B, PST, Sold out, New
Yet to be opened , F&B, EST, Last few pending , Unregistered
Closed, F&B, IST , Sold out, Unregistered
Open, Electronics, GMT, Last few pending , Registered
Yet to be opened , Electronics, PST, Stocked, New
Closed, Electronics, EST, Sold out, Registered
Open, Electronics, IST , Stocked, Unregistered
Yet to be opened , Computers, GMT, Sold out, New
Closed, Computers, PST, Last few pending , Registered
Open, Computers, EST, Stocked, New
Yet to be opened , Computers, IST , Last few pending , Registered
Closed, Electrical, GMT, Stocked, Unregistered
Open, Electrical, PST, Sold out, New
Yet to be opened , Electrical, EST, Last few pending , Registered
Closed, Electrical, IST , Last few pending , New
Closed, Peripherals, GMT, Stocked, Unregistered
Open, Peripherals, PST, Sold out, Unregistered
Yet to be opened , Peripherals, EST, Last few pending , Registered
any_value_of_Auction status, Peripherals, IST ,any_value_of_Availability, New
Closed, Machines, GMT, Stocked, Unregistered
Open, Machines, PST, Sold out, Registered
Yet to be opened , Machines, EST, Last few pending , New
any_value_of_Auction status, Machines, IST ,any_value_of_Availability,any_value_of_Bidder_Category
Closed, Jewelery, GMT, Stocked, Unregistered
Open, Jewelery, PST, Sold out, Registered
Yet to be opened , Jewelery, EST, Last few pending , New
any_value_of_Auction status, Jewelery, IST ,any_value_of_Availability,any_value_of_Bidder_Category
Closed, Garments, GMT, Stocked, Unregistered
Open, Garments, PST, Sold out, Registered
Yet to be opened , Garments, EST, Last few pending , New
any_value_of_Auction status, Garments, IST ,any_value_of_Availability,any_value_of_Bidder_Category
any_value_of_Auction status, Computers,any_value_of_Time Zones,any_value_of_Availability, Unregistered
So, what's my strategy to test?
I'd use the above test data as a sample of my test data coverage and investigate for problems. The moment I find a problem. So let's assume, I found a problem with IST Time zone causing a crash ( which I did when I was testing the auction system a couple of years back ) I'd want to do more combinations of IST with other parameters. That gives me a need to use N-Way-Random Combination Generator to sequence all tests I need for my investigation that surrounds IST Time Zone.
Wow! That's exciting.
Tuesday, July 28, 2009
End of July 2009- What's going on?
Testing community will benefit more from version 2.0 of TestersDesk.com that will be released end of August. It brings more usability based on feedback received; increased limits on inputs and outputs, a new set of tools like an Image Generator for a given set of test requirements and more... stay tuned.
We are also working on some new innovative tools that are too early to reveal at the moment.
We are pleased to see the daily increase in the registered users as well as returning users. When testers use our products to increase their productivity, it gives us a long lasting happiness and the motivation to do more. Each minute is an exciting moment as testers get help from our tools in their daily activities of finding bugs in the software through our new/innovative testing tool.
Testers who perform manual test execution anyway have been using our tools regularly and recently we have seen increased growth even in the usage from test engineers working automation tools (for example, our N-way Test Case Generator is being used to automatically data-drive the test script with different possibilities through the generated CSV output of test combinations, essentially working like a tool-centric decision table).
Every day, we receive mails from users thanking us for the free online toolkit, passionate testers who have explored TestersDesk.com started blogging on us (for example, Ajay Balamurugadas in his blog wrote his experiences with TestersDesk.com) and so on.
Do write to us with any use cases on how you benefited from TestersDesk.com and any new requests/suggestions/whatever you may want to communicate in growing the online toolkit.
We are also working on some new innovative tools that are too early to reveal at the moment.
We are pleased to see the daily increase in the registered users as well as returning users. When testers use our products to increase their productivity, it gives us a long lasting happiness and the motivation to do more. Each minute is an exciting moment as testers get help from our tools in their daily activities of finding bugs in the software through our new/innovative testing tool.
Testers who perform manual test execution anyway have been using our tools regularly and recently we have seen increased growth even in the usage from test engineers working automation tools (for example, our N-way Test Case Generator is being used to automatically data-drive the test script with different possibilities through the generated CSV output of test combinations, essentially working like a tool-centric decision table).
Every day, we receive mails from users thanking us for the free online toolkit, passionate testers who have explored TestersDesk.com started blogging on us (for example, Ajay Balamurugadas in his blog wrote his experiences with TestersDesk.com) and so on.
Do write to us with any use cases on how you benefited from TestersDesk.com and any new requests/suggestions/whatever you may want to communicate in growing the online toolkit.
Sunday, July 5, 2009
How humble looking 'Template Based File Generator' exploded bomb at my work palce !
This post is about “Template-based File Generator” provided by TestersDesk, oh sorry, wait a minute, let me correct- this post is about my encounter with that tool. For technical help about the feature, of course there is a manual within the tool, but this is an off tangent from my experience. Should you be interested to know of a use case, do read on…
During my school days, in a debate competition, we were supposed to speak on behalf of science and other group against it. Their argument was that science developed lot of bombs and subsequently causing loss of human life, after a heated discussion one of our team members said ‘it’s the place where bomb is exploded’. Wow! That was an explosion in all our minds, a mind blowing answer. Bombs are utilized in numerous ways apart from shelling them on people.
That’s fine but what does bombs have got to do with testing? I see similar thing going on in my career. We developed a beautiful test suite for 13 modules of an ERP product, where each module approximately has not less than 40 test scripts for regression. Total test script assets would be around 520 (data files). Once the test suite is ready, for every regression we have to change test data in almost all the test data files.
I don’t know whether you believe if I say that it is taking on an average approximately 2 complete days for single resource to update all 520 test data files. Few of my concerns while preparing test data for every regression were…
1. Some test data becomes obsolete, we have to change it every time we run regression, or else tests would stop. For example
a. New test data for invoices, purchase orders, receipts, journal names to name a few.
b. Amount to be entered in all of the above type of bills.
c. Passwords for few users need to be changed every time we do regression.
2. Add new column to test data files
a. Sometimes few screens in application has additional text fields (this is when they add descriptive flex field), which forces us to open each test data file and include a new column.
b. When additional details are to be entered into application we have to add few columns in test data files.
c. When a new custom screen is developed ( although it is rare, it does happens)
Now look at ‘Template based file generator’ feature, it looks so simple and humble with just 2 files for input. Once you feed the input files it throws a set of files at you. Long back around a year ago when I looked at this feature, I could not understand much of its strength.
Let me tell you how this bomb exploded in right place. In my case I just kept all test data I ever need to run my gigantic test suite in single master test data file. Once all the new data is updated, I just take a Template file along with and throw these two into ‘Template based file generator’. In almost no time I am getting beautiful set of all test data files, which I directly use to drive my test suite.
You see, now it is taking one of my resources just 3 – 4 hours to fill the correct test data in master test data file, and almost no time to get all those 520 test data files filled with new data.
Here is the beauty with this tool, if I am supposed to add new column to all test data files, effort from my side would be to just add another column in my master test data file.
There is a plethora of areas where the humble looking ‘Template based file generator’ can explode a bomb at your work place.
I never imagined life would be so easy with Testersdesk. I understood true meaning of these pictures-
Happy TestersDesking !
Cheers,
Virtualkey(Suresh)
During my school days, in a debate competition, we were supposed to speak on behalf of science and other group against it. Their argument was that science developed lot of bombs and subsequently causing loss of human life, after a heated discussion one of our team members said ‘it’s the place where bomb is exploded’. Wow! That was an explosion in all our minds, a mind blowing answer. Bombs are utilized in numerous ways apart from shelling them on people.
That’s fine but what does bombs have got to do with testing? I see similar thing going on in my career. We developed a beautiful test suite for 13 modules of an ERP product, where each module approximately has not less than 40 test scripts for regression. Total test script assets would be around 520 (data files). Once the test suite is ready, for every regression we have to change test data in almost all the test data files.
I don’t know whether you believe if I say that it is taking on an average approximately 2 complete days for single resource to update all 520 test data files. Few of my concerns while preparing test data for every regression were…
1. Some test data becomes obsolete, we have to change it every time we run regression, or else tests would stop. For example
a. New test data for invoices, purchase orders, receipts, journal names to name a few.
b. Amount to be entered in all of the above type of bills.
c. Passwords for few users need to be changed every time we do regression.
2. Add new column to test data files
a. Sometimes few screens in application has additional text fields (this is when they add descriptive flex field), which forces us to open each test data file and include a new column.
b. When additional details are to be entered into application we have to add few columns in test data files.
c. When a new custom screen is developed ( although it is rare, it does happens)
Features @Testersdesk look very simple but more I understand and keep on using them, more powerful they are turning out.
Now look at ‘Template based file generator’ feature, it looks so simple and humble with just 2 files for input. Once you feed the input files it throws a set of files at you. Long back around a year ago when I looked at this feature, I could not understand much of its strength.
Let me tell you how this bomb exploded in right place. In my case I just kept all test data I ever need to run my gigantic test suite in single master test data file. Once all the new data is updated, I just take a Template file along with and throw these two into ‘Template based file generator’. In almost no time I am getting beautiful set of all test data files, which I directly use to drive my test suite.
You see, now it is taking one of my resources just 3 – 4 hours to fill the correct test data in master test data file, and almost no time to get all those 520 test data files filled with new data.
Here is the beauty with this tool, if I am supposed to add new column to all test data files, effort from my side would be to just add another column in my master test data file.
What a great reduction in mundane manual tasks, when substituted with TestersDesk to my team.
There is a plethora of areas where the humble looking ‘Template based file generator’ can explode a bomb at your work place.
I never imagined life would be so easy with Testersdesk. I understood true meaning of these pictures-
 |
 |
Happy TestersDesking !
Cheers,
Virtualkey(Suresh)
Subscribe to:
Posts (Atom)
Posts
