December 2020

I would like to share with you an example that totally changed my perception of how I look at automated checking, whether it is unit or integration.

I am programming a lot in .NET. The language I use is C#. The code written in the end is packaged in a .dll or .exe file. If you will look at the content of this file you will not see C# code. You will see MSIL code which is like an assembly language. This MSIL at runtime is translated into processor code.

Here is a C# code, it is taken from the book CLR via C#:

This code is a little bit weird. I admit that it is an extreme example. But it helps me communicate what I want. Let’s suppose that we covered, so to speak, this code with unit checks done ok. Lets also suppose that the code review was quite rigorous and carefully done.

I have some questions:

Is it just a line of code?
How many memory allocation problems can occur?

Let’s change the perspective. What you see bellow is the MSIL code generated from the C# code you see above:

Side Note: By the way, this MSIL code generated from a .Net Core code. Do you think it’s very different from .NET 4.8 code? The answer is no. I ask this because there are people whose CVs will not be accepted because they have not worked with .NET Core, but have experience with .NET. For me is an indication that we focus too much on shallow things.

All those things you see marked with red indicate a place where memory allocation problems might occur.

From my experience not a lot of .NET programmer’s look at the MSIL code. But let’s think about unit checks. You see, what you do know isn’t canceled, it’s okay. But now don’t you think about code coverage in a different way? It’s something else, isn’t it?

Note: For Java we have Java bytecode

This perspective helped me, in this context, to understand that I can think, for example, of memory corruption coverage. Which is just another type of coverage. But I wrote this post to try to answer another question which is: What bugs you aren’t finding while you automate checks? For me, as a programmer, it is very clear that I focus on a certain direction when developing an automated check and nothing else. But focusing on a certain direction means ignoring other dimensions. In the example above I would have ignored memory problems or what MSIL code is being generated and the kind of instructions which will actually be run.

This is a three-part series:

Automation in testing – Part 1 – Do you think it is normal to say test automation? Really everything can be automated in testing? Are we focusing automation goodness in a good direction?

Automation in testing – Part 2 – Do we really need to concentrate our automation efforts in simulating users via Selenium or other tools? Isn’t this a sign of other problems that cannot or will not be solved?

Automation in testing – Part 3 – What bugs you aren’t finding while you automate checks?

Thank you Johanna Rothman

Do we really need to concentrate our automation efforts in simulating users via Selenium or other tools? Isn’t this a sign of other problems that cannot or will not be solved?

I really understood how I should look at the role of automation in testing when I participated in an audit. The role of that audit was to help a company to obtain information, also technical information, in order to know if the acquisition of that company would be made. My responsibility was on the technical part to look at the code and the technologies being used. When investigating the code I noticed some patterns. There was no tool on the market to help me, somehow expose what I observed in that context. So I wrote it. In this way I was able to show and articulate the more complicated parts from the less complicated ones in that code which were specific for that project.

That’s the moment when I realized why we should use the “tools” word instead of the “automation” word. Also is then when I realized how limited we are when using the idea of automation/tool. We can use/create a tool to, among other things:

create test data, for example by using combinatorics, Montecarlo simulations, de bruijn sequences;
to analyze and display the logs in a certain way;
to see where to focus our attention at a certain time, depending on what is in the source control;
exercise the system in a specific way so that the tester can observe tangential and oblique problems.

In other words, we do not use this tool idea to its value and power. We limit ourselves through the UI to press some buttons. We use this automation thing, as I observed, only to try to simulate a user. How strange, we use a machine to simulate a thing only a human can really do, but do not use the tool/automation for things which are supposed to be used.

So, back to the questions from the beginning of this post: Do we really need to concentrate our automation efforts in simulating users via Selenium or other tools? Isn’t this a sign of other problems that cannot or will not be solved?

As a programmer, so from a technical perspective, I have observed this obsession to simulate users in an automated way when:

the product code is very bad, so is hard to include any automation checks or testability points at a fine granular level and because of that nothing will be done about it;
when there is no trust in the automated checks done at a fine granular level;
when programmers do not know how or do not want to do automation cheks;

As a manager, I have observed this obsession to simulate users in an automated way when:

Testers see that what is payed in the industry is this user simulation automation, so they find any excuse to do it;
When testers do not know what testing is and how to make a test strategy. Because choosing a tool as selenium and implement automation checks is not a test strategy – but this is for another post;
When managers have no clue what testing is. But they are delighted when they see some encoded test cases in a wiki page and even happier when those (preferably all) are automated whatever that would mean. I wonder how their reaction would be if we would insist that their whole work be automated, probably not too well.

So, I have a new heuristic to spot problems in testing regarding automation which I named it “Automated user simulation heuristic”.

I think that using automations/tools in testing is really cool and useful and there is no doubt about it, not for me at least. Too bad that we focus on the wrong direction in using it.

This is a three-part series:

Automation in testing – Part 1 – Do you think it is normal to say test automation? Really everything can be automated in testing? Are we focusing automation goodness in a good direction?

Automation in testing – Part 3 – What bugs you aren’t finding while you automate checks?

Month: December 2020

Automation in testing – part 3

Automation in testing – part 2