Automation in testing – part 1

Do you think it is normal to say test automation? Really everything can be automated in testing? Are we focusing automation goodness in a good direction?

For me, also as a programmer, testing is, among other things, about finding those relevant patterns that matter within a complex system. But in order to understand complex systems a person must interact with it. But how do we do that?

If you look carefully in the image of the post, you will see a forest. Forest represents testing. Trees represent parts which form the forest. But let’s not confuse the forest with a tree. Each tree has its role in that environment.

Small note about words: Some time ago I worked at a company in a management position. Weekly we had a meeting between all the middle managers. What I noticed in these meetings, which were held in my mother language, Romanian, is that we used a lot of English words. I am speaking about words which have a good translation in Romanian. In one of those meetings I realized how superficial I was. Why can’t I speak about those concepts in my language? So I began to be more careful about this aspect. When I tried to translate and adapt those words in my mother language, I began to understand even more the topics I wanted to speak about. It is then when I really understood the power of words. How they can be used to stimulate thoughts, awareness and insights.

As a programmer I wrote and I write a lot of code, and also code to verify the code written. And now when reading the previous sentence, I realize that I do not write code to verify the verification code. Interesting insight I had by just changing the formulation of an idea.

But I noticed something also. The verification code I was writing was on things which were clear, deterministic which could be approached in an algorithmic way. How does this relate to testing? For me this is a technique to find problems in code. 

But I can also use other things when testing like modeling, questioning, inferring.(see the blog post image). 

So those automated verifications are part of testing like the others. Personally, I like to call these automated verifications also automated checks.

Therefore this checking stuff is included in the testing, it is an integral and fundamental part of the testing. But we can’t say that only this aspect defines testing. We can say that it is a component of testing. And this part can be automated because it is an algorithmic verification. 

If I got involved to understand what testing is , it is because I had the occasion to work with wonderful testers. And I wanted to understand their craft. These testers actually did not have coding skills, but still they were able to find interesting and important bugs. I saw how they used their social knowledge to make a lot of judgments about what to test, how to test, when to stop. There is no algorithm to make these reflections, judgments, assessments. What actually I noticed was how important is the human aspect in testing. Like a conductor who conducts an orchestra and the musicians who use the instruments(tools) to generate the music. Those instruments can be mechanical, but also digital, whatever is needed to transpose the music.

I asked at the beginning of this post the following question: Do you think it is normal to say test automation? I did this as an invitation to see what we might miss if we frame it this way. For me, each time when I witness this framing I see an overemphasis on one dimension of testing and all the others dimensions ignored. That’s why I like to say automation in testing because it serves as a way to increase awareness on how automation relates to testing and that other things might be considered also.

This is a three-part series:

Automation in testing – Part 1 – Do you think it is normal to say test automation? Really everything can be automated in testing? Are we focusing automation goodness in a good direction? 

Automation in testing – Part 2 – Do we really need to concentrate our automation efforts in simulating users via Selenium or other tools? Isn’t this a sign of other problems that cannot or will not be solved?

Automation in testing – Part 3 – What bugs you aren’t finding while you automate checks?

What is testing for me

Some days ago I had the occasion to make a presentation regarding testing. But I had one dilemma before. How could I have been able to do this presentation without me being able to say what testing is for me?

There is a concept, praxis, that started my quest to understand how theory can inform and make sense in the practices, in my day to day work. And when I say practices I think about practices in testing, management and programming.  There are lots of them. I began to cover some of them, please look here for examples. The concept I am speaking about now is the complexity theory. 

Is not easy to explain this complexity thing. I like how Edgar Morin explains the complexity. For him complexity is like a tissue or fabric of inseparable associated diverse elements. But at a second look, those elements represent: events, actions, interactions, feedbacks, hazards. 

I like the header image because it is trying  visually to show this complexity. You can see in the image some dots, connection between the dots, but it also shows patterns that can be formed (different lines, triangles, pentagons, polygons and so many others). In our day to day life those dots might represent things like:

  • Technical details in our product(components and objects relating to each other, the fractality of the code, mixing up unrelated abstraction and so on)
  • Humans with their roles working on a product
  • Risks
  • Requirements 

And then there is another important detail and that is the fact that humans think in patterns.

So, what is testing for me? Testing is about finding those relevant patterns that matters within a complex system.

Those patterns might mean whatever is relevant in spotting problems within the product or process. These patterns are not static, they continuously change.

Risks, feelings, management

What if everyone –our managers, auditors, testers — got it wrong with the topic of risks? 

What if what we witness regarding risk management  is just a charade ?

Long ago, I read Jerry Weinberg say, “The definition of quality is always political and emotional.”  That resonated with me then and still does.

I had to get out of a project. At that time I occupied a management position. Usually in taking over a project there are some intermediate steps that need to be done. There was no pressure. It was time for my replacement, let’s call him John, to take it over.  

John was also in a management position. We both worked on the same big suite of products. My project was an integration project, making the communication between products from the suite work. Part of the problem in my project, before taking over, was that it contained hacks. By hacks I mean things which did not belong there at all but on the other products,at the source.Unfortunately, managers from the other products did not want to apply the needed fixes on their parts, to remove those hacks. So all the garbage needed to be fixed on their part was thrown on this integration project, before I came. 

My team fixed these problems in both the integration product and the other products from the suite. 

As I reflect back on my conversations with John,I remember several things that appeared “off.” He seemed  very detached about topics we needed to discuss in order for him to take up the project .  My perception was that John had exaggerated detachment. When he spoke with me everything was ok. But with the others(management from client , our management, teams) his conversation was all about risks. 

I did not mind at all that John was concerned about risks. What I did not understand in those moments was his lack of convergence. John spoke with me about risks but in a calm way. When he spoke with others he was doing it in an alarming and exaggerating way. Also I have noticed certain resistance from the teams he had. Anyway that exaggerated perceived detachment in combination with how he handled the risks puzzled me. 

Feelings, as much as data, play an important role in decision making. 

Then, I found some scientific research about risks, the risks as feelings hypothesis. This hypothesis says that response to risky situations and how we evaluate them are influenced directly by our emotions/affect. The traditional models of evaluating risks tell us about probabilities and consequences but nothing else. But these, actually, are seriously influenced by feelings.(1)

But there is more. You may wonder why us , in IT, should we care about these studies? Because, is about validated scientific knowledge. Things which other disciplines already studied and proved it. We only have to cross the corridor to these disciplines and learn from them. I like  the word praxis to describe this. 

So, I spoke about the psychological part. But there are also sociological and anthropological studies. These studies show that risk perception has roots also in sociological and cultural elements. For example, the sociological component is telling us how we are influenced in dealing with risks by colleagues, friends, family. The anthropological component tells us that people within groups minimize certain risks and exaggerate others as a way to maintain or control a group.(2)

I realized John had feelings, and those feelings might lead him to assess risks differently than I did. Maybe he felt fear because of the history of the project. 

As said above there is also a sociological part which might help in making sense of this situation. I tried to understand from whom he was taking certain technical information. Who was advising him on this part. And I realized it was about some of  his team members who did not want to be involved in the take over. 

The anthropological component helped me to understand the exaggerated escalation to management. It was a way to maneuver the management groups by playing with risks.

Till that moment I only witnessed a shallow view of how risks were used by management. It was an eye opener to me regarding risks and how they should be used in management, testing, audits. 

I mentioned audits. I would like to see an auditor knowing these aspects of risks. Maybe this will help in order to be able to see the spirit of the law and not just the letter of the law. Risks combined with the tacit knowledge might help understand the why’s of why certain risks are articulated, handled and made public in a very specific way. Audits is not about simple receipts ignoring the context and professionalism of a discipline.

Emotions play an important aspect in, risks, judgment/evaluation/assessment. How right Jerry Weinberg was.


(1) Risks as Feelings, Loewenstein, George F. and Weber, Elke U. and Hsee, Christopher K., Psychological Bulletin, Vol. 127, 2001, Available at SSRN: https://ssrn.com/abstract=929947 

(2) Perception of Risk, Paul Slovic, Science 17 Apr 1987:Vol. 236, Issue 4799, pp. 280-285, https://science.sciencemag.org/content/236/4799/280