Going Through the Motions

Michael Bolton posted a list of heuristics to use when deciding if it is time to stop testing.  As usual he provides a pretty comprehensive list, but for a change I feel like I can add something to it.

The seventh heuristic is “The Pause that Refreshes Heuristic” which means that testing isn’t stopped, but it is suspended until the testers feel inspired to start testing again.  I don’t think that this is a heuristic trigger to stop testing though.  I think it is a heuristic action that takes place when a test team uses some other rule of thumb.  Michael identified two reasons that the testers might stop, and I would suggest that this one heuristic should be broken out into two separate ones.

Change in Priorities - Michael mentioned that testing may pause if there is a higher priority at the moment.  I think this one is largely self-explanatory, but a heuristic that I often don’t see applied is the next one.

Lights are Off - This heuristic says that if you don’t know why you are testing then pause and refresh.  I’ve seen very large test teams continue to run the same test cases over and over, because they feel like they need to be doing something even if they don’t know why.  Let’s explore some of the reasons that this might occur:

1. “We’ve always done it this way.” – The testers may just have faith in the fact that if the team has been doing it this way then it must be good.
2. The testers are rewarded by numbers – If the testers reduce the amount of test cases that are “executed” then it could look like less work is being done.
3. There is time left – The test team might be given more time than is actually necessary to test a feature so they fill it up with old test cases that don’t have much meaning.

I would suggest that testing has technically stopped at this point, because the testers are no longer thinking about their actions.  That is precisely the reason that the guise of testing should stop.  It’s time to rethink the mission, analyze the risks in the application or perhaps the “testing” that is performed is actually important, but the testers don’t understand why.  In the latter case it would be a better use of time to take a step back and help the testers understand why specific test cases were chosen for execution rather than telling them to “just finish your test cases”.

So, Michael’s list of heuristics are helpful, but I’d recommend changes.  In the end it doesn’t matter if you use his list, my list or any list at all.  These heuristics are rules of thumb to help thinking testers decide when testing should stop.  The most important thing is that you are thinking about it.

Read Full Post »

Tests and Test Cases

As a test manager, I like to interact with new testers on my team. So, I had a question for our newest tester, Venkat, after he completed his first test case.  The test case instructs the tester to enter an invalid value into a field to ensure that the application rejects it.

“So, how many tests have you run so far, Venkat?”

He looked puzzled.  “Well, I’m still on my first test.”

“Well, maybe we should talk about what I think a test is. I actually think that you have ‘executed’ countless tests at this point.”

I explained that I define a test as the action that someone takes to answer a question about the application.  These actions are abstract.  It could be a thought, a visual check, or a feeling.  Each action is testing the application’s abilities to meet our expectations, and they are difficult to separate sometimes.  I then asked Venkat to list all the things that were going on in his head while he executed that one ‘test’.  Here is the list he provided:

• It was difficult to enter the invalid value.
• The application invalidated his input rather quickly.
• The other labels and edit boxes on the screen seemed correct.
• There was a UI bug on a screen that he needed to use prior to the current one.
• He liked the invalidation message, but he was confused by the location of some of the edit boxes.
• He learned some business rules and how to navigate the system.

I asked Venkat to stop there.  Venkat ran numerous (maybe countless) tests against the application while following the steps in the document.  This document is a “test case”.

I define a test case as a documented model of a predicted test.  The documented tests are  attempts by humans to turn the abstract into the concrete, and I think it can be a valuable tool.  Testers should consider documenting tests if the goal is to:

• remember them for the future (personally or for a test team).
• submit to others for review (peer review or auditing purposes)
• assign to others for execution (more than likely these are checks)
• submit to others for some kind of scripting support

These are rather ambiguous categories, but the method of documentation will differ depending on what the end goal is. It also isn’t very helpful to count the tests or test cases, as I’ve demonstrated that a skilled tester (even a junior one as Venkat demonstrated) will perform all kinds of tests that in the end is counted as one.  The benefit of counting is more in showing progress towards a total.

After Venkat completed 10 of the 20 introductory test cases, I asked him another question. “When will you be done with your training test cases?”

Venkat looked quickly at the clock, and then did some quick mental math. “It took 95 minutes to complete the first 10 tests, so I would expect it to take 95 more minutes.”

“Aw, you let me trick you again!  Tell me about your testing so far.”

Venkat found four bugs that took 40 of the 95 minutes to write.  He also couldn’t understand a step in one of the test cases, so he had to ask his neighbor about that.  Consider the possibilities with the remaining test cases:

• The remaining tests are somewhat redundant, and Venkat has already technically performed the work that the test cases suggest.
• There are more or fewer bugs to be found with the remaining tests.
• The remaining test cases could be more or less difficult to understand.
• The remaining test cases require more or fewer steps to complete.

So, even reporting a percentage of the total test cases can be misleading and unhelpful. I recommend that testers be able to tell a story about the testing that they have completed so far, and the testing that remains.  If numbers are required, then at least a qualitative story is available.

A week after having this discussion with Venkat I decided to stop by and see if it shaped the way he talked about his testing.

“How many tests have you run today, Venkat?”, I said in a half-joking tone.

He smiled, and said “Can you please help me understand why the answer would be helpful to you?  The number of tests aren’t the same as number of test cases, and I’m not sure why either number would be useful.”

I just gave him a thumbs up and kept walking.

Read Full Post »