Michael Bolton posted a list of heuristics to use when deciding if it is time to stop testing. As usual he provides a pretty comprehensive list, but for a change I feel like I can add something to it.
The seventh heuristic is “The Pause that Refreshes Heuristic” which means that testing isn’t stopped, but it is suspended until the testers feel inspired to start testing again. I don’t think that this is a heuristic trigger to stop testing though. I think it is a heuristic action that takes place when a test team uses some other rule of thumb. Michael identified two reasons that the testers might stop, and I would suggest that this one heuristic should be broken out into two separate ones.
Change in Priorities - Michael mentioned that testing may pause if there is a higher priority at the moment. I think this one is largely self-explanatory, but a heuristic that I often don’t see applied is the next one.
Lights are Off - This heuristic says that if you don’t know why you are testing then pause and refresh. I’ve seen very large test teams continue to run the same test cases over and over, because they feel like they need to be doing something even if they don’t know why. Let’s explore some of the reasons that this might occur:
- “We’ve always done it this way.” – The testers may just have faith in the fact that if the team has been doing it this way then it must be good.
- The testers are rewarded by numbers – If the testers reduce the amount of test cases that are “executed” then it could look like less work is being done.
- There is time left – The test team might be given more time than is actually necessary to test a feature so they fill it up with old test cases that don’t have much meaning.
I would suggest that testing has technically stopped at this point, because the testers are no longer thinking about their actions. That is precisely the reason that the guise of testing should stop. It’s time to rethink the mission, analyze the risks in the application or perhaps the “testing” that is performed is actually important, but the testers don’t understand why. In the latter case it would be a better use of time to take a step back and help the testers understand why specific test cases were chosen for execution rather than telling them to “just finish your test cases”.
So, Michael’s list of heuristics are helpful, but I’d recommend changes. In the end it doesn’t matter if you use his list, my list or any list at all. These heuristics are rules of thumb to help thinking testers decide when testing should stop. The most important thing is that you are thinking about it.