Context: Have you ever had a conversation about test automation where the manager wanted you to automate “everything”? Or the manager asked you to show him/her all the scenarios? And how can we explain it in a simple way? Below is a small/simple dialog trying to do this.
Manager: Is there a place I can see the testing outputs for all applications? What scenarios are tested and which are not?
Tester: We cannot simply show “all the scenarios”. Testing is not a finite set of scenarios but an exploration of the product to discover new information.
Manager: I do not understand. Everywhere you look they do this.
Tester: Indeed. Look. Maybe there is more to say here. I understand your preoccupation. May I try to offer a perspective? Maybe it will help us. That’s what people say they do. The reality is more complex.
Manager: <<the manager makes a gesture that is ready to listen, but the tester perceived some doubts>>
Tester: I like to use an example that is connected with a problem posed by international terrorism. Yes, it sounds strange and curious at the same time, I think. It was a guy, Max Boisot(1), that tried to explain via a simple analogy why they were not able to catch lots of attacks. Imagine that we have 10 points. These points can represent people, requirements, code packages/branches and other things regarding product development.
Look at this image, please do not stress with math formulas, take it as it is:
Manager: So those points can represent different things in a project?
Tester: yes. Actually just imagine the realities of a project like a woven fabric of different things(events, actions, interactions, feedback, dangers, determinations) that are inseparably associated.
Manager: Ok <<said with a long Ooo>>
Tester: Do you see in the drawing that multiple geometric figures can be created via 2 or 3 or 4 dots?
Manager: Yes, different kind of triangles, polygons
Tester: Yes, Max Boisot named them patterns. The total number of these is 3.5 trillion. In real life some of them matter, some of them not. But in testing we must look for those that matter through all the existing ones. Somehow I froze at your request because it is impossible for me to give you what you said. I would do it but it is hard if not impossible.
Manager: Hmmm. And what about automation?
Tester: We can automate a thing which is explicit and deterministic in an algorithmic way. We cannot automate sensemaking, critical thinking, studying, modeling, interpretation. All these , among other things, are part of the process of testing.
Manager: But every company I look at does this. It seems that every company follows the same thread/trend. Why do they consider it that it can be done, that you can automate everything?
Tester: First , I think that there is a lot of marketing out there. Then how do we expect people who were hired or built a career or had a certification to admit they were not so right? Is not easy.
Manager: You did not finish the idea with automation and the geometric figures
Tester: Yes. There are three important things:
– even with the best programers we can’t cover and prove those 3.5 trillion patterns
– I say this because there are different types of coverages(2), I know of at least 100 of them. If you’ll ask a programmer she/he will tell you, maybe, about code coverage.
– As people, we are pattern detectors, etc. If you encourage our agency, we can see patterns in the data/product a tool can’t see.
Manager: But I need a way to show that we are on top of the situation and that we can detect the problems.
Tester: There are ways. For example, to manage testing we can use Session Based Test Management(3). To discover problems we have a long range of heuristics and techniques to do it.
Manager: Ok. But we have to discuss these also. Let’s fix another meeting for this.
Tester: Of course
(1) Connecting the dots, Max Boisot, https://web.archive.org/web/20091123034027/http://www.hlswatch.com/2009/11/12/nidal-hasan-and-the-problem-of-connecting-the-dots/
Note: I searched for the original link of the article “Connecting the dots” but is no longer available.
(2) Software Negligence and Testing Coverage, Cem Kaner, http://www.badsoftware.com/coverage.htm
(3) Session-Based Test Management, James Bach, Jon Bach , https://www.satisfice.com/download/session-based-test-management