As a test manager, I like to interact with new testers on my team. So, I had a question for our newest tester, Venkat, after he completed his first test case. The test case instructs the tester to enter an invalid value into a field to ensure that the application rejects it.
“So, how many tests have you run so far, Venkat?”
He looked puzzled. “Well, I’m still on my first test.”
“Well, maybe we should talk about what I think a test is. I actually think that you have ‘executed’ countless tests at this point.”
I explained that I define a test as the action that someone takes to answer a question about the application. These actions are abstract. It could be a thought, a visual check, or a feeling. Each action is testing the application’s abilities to meet our expectations, and they are difficult to separate sometimes. I then asked Venkat to list all the things that were going on in his head while he executed that one ‘test’. Here is the list he provided:
- It was difficult to enter the invalid value.
- The application invalidated his input rather quickly.
- The other labels and edit boxes on the screen seemed correct.
- There was a UI bug on a screen that he needed to use prior to the current one.
- He liked the invalidation message, but he was confused by the location of some of the edit boxes.
- He learned some business rules and how to navigate the system.
I asked Venkat to stop there. Venkat ran numerous (maybe countless) tests against the application while following the steps in the document. This document is a “test case”.
I define a test case as a documented model of a predicted test. The documented tests are attempts by humans to turn the abstract into the concrete, and I think it can be a valuable tool. Testers should consider documenting tests if the goal is to:
- remember them for the future (personally or for a test team).
- submit to others for review (peer review or auditing purposes)
- assign to others for execution (more than likely these are checks)
- submit to others for some kind of scripting support
These are rather ambiguous categories, but the method of documentation will differ depending on what the end goal is. It also isn’t very helpful to count the tests or test cases, as I’ve demonstrated that a skilled tester (even a junior one as Venkat demonstrated) will perform all kinds of tests that in the end is counted as one. The benefit of counting is more in showing progress towards a total.
After Venkat completed 10 of the 20 introductory test cases, I asked him another question. “When will you be done with your training test cases?”
Venkat looked quickly at the clock, and then did some quick mental math. “It took 95 minutes to complete the first 10 tests, so I would expect it to take 95 more minutes.”
“Aw, you let me trick you again! Tell me about your testing so far.”
Venkat found four bugs that took 40 of the 95 minutes to write. He also couldn’t understand a step in one of the test cases, so he had to ask his neighbor about that. Consider the possibilities with the remaining test cases:
- The remaining tests are somewhat redundant, and Venkat has already technically performed the work that the test cases suggest.
- There are more or fewer bugs to be found with the remaining tests.
- The remaining test cases could be more or less difficult to understand.
- The remaining test cases require more or fewer steps to complete.
So, even reporting a percentage of the total test cases can be misleading and unhelpful. I recommend that testers be able to tell a story about the testing that they have completed so far, and the testing that remains. If numbers are required, then at least a qualitative story is available.
A week after having this discussion with Venkat I decided to stop by and see if it shaped the way he talked about his testing.
“How many tests have you run today, Venkat?”, I said in a half-joking tone.
He smiled, and said “Can you please help me understand why the answer would be helpful to you? The number of tests aren’t the same as number of test cases, and I’m not sure why either number would be useful.”
I just gave him a thumbs up and kept walking.