As testers, one of our most important tasks is to find problems with the product or service that we are testing. The things which reduce value for any stakeholder that matters. To be able to distinguish between what is a problem and what is not a problem we need to compare our understanding of the product with our understanding of what threatens value for any stakeholders who matters. I will in this post focus on our understanding, our model, of the product. And especially how to find flaws in our product model at later stages, when we believe that we have a pretty firm grasp of the product.
Understanding the product
When we first start testing a new product we have a very basic understanding of it. Our mental model might be as crude as that we know that it is an application and the name of it. As we start exploring the product we learn and our model gets fuller and more detailed. Our understanding of the product increases as our mental models more and more reliable and accurately can predict different aspects of the product. Or to be a bit more precise, the understanding of the product in a small system comprised of three main components; the product, the product environment and ourselves.
Gaining understanding about the product
Initially any observation of the product gives us information that requires us to change our mental model to be able to explain the product. Any new observed element will be added to the model. Within just a few moments “random” observations will not be a very efficient source of information to build, refine and refactor our model. At this stage a more systematic approach to observing is necessary. One popular such approach is touring the software.
As we observe the product two different scenarios can occur. Observation can be explained with our current model, which builds confidence that our model is accurate. Although this might feel good, we do not learn anything new about the product. The second outcome, that the observation cannot be explained by our current model is much more interesting and will be the one that I elaborate on.
Adapting our model
When observations cannot be explained with our current model of the product we don’t throw away the whole model. We fix whatever flaw we see in it. We can remove parts, add parts or replace parts of it, so that the new model does a better job at explaining the product. In order to explain a complex product with a simpler model, a model that is useful for testers who has well-developed but still limited cognitive capacity, we need limit the amount of information that our model contains. This can be done by choosing, more or less consciously, which aspects that is important to us. Of many aspects, the one of model granularity is often substantially reduced. Sometimes we make generalizations, we assume based on inductive reasoning that similar things are identical. We assume that very similar buttons are in fact identical. That observed behavior to the stimuli received in this test will be the same of the observed behavior of a later test with same stimuli. When we decide which aspects that are important we do this from another model, where or product model is one part, which also can be amended as new information is revealed. Our product model is not the sum of our observations. Many observations is not included in the model and some parts of the model are based on inductive reasoning.
“Our product model is not the sum of our observations.”
That the model is not an exact representation of the product is important, since it will be an integral part when we test the product for value or lack of such. Decisions about which aspects are important to test, which risks there are and how design powerful tests will depend on the product model. As we test for value we will continue to improve on our product model. But some of our assumptions that are false might be very hard to refute. Especially as our focus is more on testing for value than for understanding of the product. Assumptions, even false ones, which are hard to falsify tend to grow stronger over time. Since they survived for such long time it is tempting to regard them as absolute true, or even cling to them despite of contradicting evidence.
Detecting false assumptions
To increase our chances of detecting false assumptions we first of all need to know what our assumptions are. But here we need to restrict ourselves to certain kinds of assumptions. For every piece of our model is either built from something that we assumed to have observed correctly or inferred from such assumptions. The kind of assumptions we are after is those that have no empiric evidence as support. The assumptions that are made wholly from our own reasoning, reasoning that makes sense in the light of our own faulty model. The things we “know” aren’t so.
As a first step we need to identify these assumptions of what is not, what never can happen. It is probably easier to detect these when one are learning a new aspect of a product. Example: While testing an embedded system I notice that the load on an internal bus is about 10%, I do some testing trying to provoke the system so that the bus load goes up. My best effort stresses the system to a bus load of 15%. Not much. Here I make an assumption, bus load will be low. I would be very surprised to see bus loads higher than 20%. By thorough critical thinking of you product model you can discover these assumptions long after making them.
Given that we have some way of observing the aspect that we make an assumption about, we can add checks for those. Performance checks for things like bus load, memory usage or CPU usage. And also logical restrictions like this and this element will always be shown or never both be shown at the same time, or that this variable never will exceed 10. Then, with the right tool support, we can design continuous checks, restrictions, for these aspects. And be notified by the tool if these restrictions where ever violated. These background checks lie dormant while you carry on doing testing only to be notified if an restriction is violated. These checks are more often used in automated regression check suites. There they also help us detect violations but aren’t as effective since they run the same tests over and over whereas testing continuously make use of new tests.
So what if these restrictions are violated? Do we have a problem? The answer is both yes and no. It isn’t necessarily a problem that busload jumped up to 21%, that could still be an acceptable bus load which doesn’t result in any significant risk. But the fact that the bus load is 21% exposes an error in our model of the product. A serendipitous finding that gives us an excellent opportunity to learn more of the product in terms of what influence bus load. This adaptation of our mental product model reveal new potential risks, give us new test ideas and thereby affect another model of ours, the coverage model.
Enablers of Continuous Model Checks
To be able to add the power of these Continuous Model Checks or Restrictions if you may, there are things that are needed. (1) Known assumptions, (2) Observability of the aspect that we are making an assumption about, so that (3) a tool can help us monitor this aspect of the product.
That a tool is detecting a violation of a restriction does us little good if we are to focused on the aspects of the product that we are currently testing to notice it. To make sure these serendipitous events catches our immediate attention the alert needs to be forceful. Good alerts use both audio and graphics. And the graphical notification should be big, signaling colors with high contrast from background and if possible move around (jumping or shaking). All these properties of an alert increases our chances of notifying is in a variety of situations like headphones lying on desk or looking away from the screen of your monitoring tool.
As a closing remark I want to add that we need to be careful when adding monitoring to our system. Our monitoring test system is affecting the product that we are testing to some extent. It is our job to figure out how to balance the value of the information extracted and the risk associated with introducing monitoring.
This became a slightly longer post than I originally planned for. Maybe two years of non-blogging does that to you. Please share your thoughts and own experiences!