Ben Simo tweeted the following the other day: “When bugs swarm, writing bug reports for each one becomes difficult. And reporting together looks like big ball of mud.”
First of all I like the term “swarming” for bugs. I’ve tested several releases where one or more features had bugs around every corner and it severely slowed down testing. Swarming is a good name for that.
It also reminded me of my frustrations working in a waterfallish environment where every bug must be documented the same way, and people held “best practices” above the heads of others. I responded to Ben’s tweet with my thoughts on what testers should do when faced with a swarm of bugs: don’t document all of those bugs, the feature/product needs more dev time and rigorous bug documentation may be wasteful. I didn’t show much context-driven thinking by replying in that manner. Michael Bolton, one of my favorite testing thinkers, called me on it:
“Easy: you’re the waiter, not the client. If the client wants four salads, offer different things, but remember who’s who.”
“For an exercise, list at least five reasons to keep going.”
It appears that I’m coming across as trying to control the development process instead of supporting it. To be clear, I agree with Michael. Testers are here to provide quality related information, not force a process on the project. I was, however, trying to provide a heuristic to testers and others on software development projects. Sometimes it’s cheaper to do more programming without the aid of testing than it is with the bug reports. In other words, once you have the information you need (“this feature needs more programming”) then testing is no longer needed.
I will respond to Michael’s exercise though. After all, he provided more than five reasons to stop testing, so I should be able to think of five reasons to keep testing:
Ultimately we should keep testing a product/feature if we want more information. So, when bugs swarm here are some reasons that we may still want more information:
- We want to find a pattern behind the swarm – We wouldn’t be very good testers if all we did is report to the project that “feature x” is broken. Perhaps we should do additional testing to help them figure out if there is a general pattern. For example, on an insurance application that I was testing we couldn’t issue any auto policies due to several bugs on various screens. It didn’t seem worth it to even worry about the auto line of business. However, after some additional testing we discovered that the majority of the issues were related to IE6 only. It was beneficial for us to continue testing to discover what that pattern was.
- We want to find the worst possible failure - So, there are so many bugs it’s hard to see through the dense cloud. Maybe it’s not too helpful to report all of them right now, but it would be useful to know if there are any particularly nasty bugs.
- We really do want to know about all the failures we can find - The programmers may not be able to just redevelop a feature without understanding what all of the bugs are. Perhaps the bugs are business rule problems. In this case it will be necessary to find all of the business rule problems in the application and report them to the programmers. I recommend (note: I don’t enforce) that the testers report this in a lightweight way. A spreadsheet should suffice, instead of documenting every last problem in a bug tracking system.
- We want to demonstrate the failures - The programmers or management may want to see these bugs first hand. In that case we will have to work our way through the swarm again.
- We are compelled by politics - Let’s face it. We don’t work in perfect environments. Sometimes management will expect the testers to “keep their heads down” and “plow through test cases”. The appearance of hard work is often more important in some environments than actually doing good work.
In summary, we should respond in a context-driven way when bugs swarm and that means we have a lot of options. We need to keep our eyes and brains open to the possible solutions. So, what do you think? Why might you still test when faced with a swarm of bugs?