Do we really need to concentrate our automation efforts in simulating users via Selenium or other tools? Isn’t this a sign of other problems that cannot or will not be solved?
I really understood how I should look at the role of automation in testing when I participated in an audit. The role of that audit was to help a company to obtain information, also technical information, in order to know if the acquisition of that company would be made. My responsibility was on the technical part to look at the code and the technologies being used. When investigating the code I noticed some patterns. There was no tool on the market to help me, somehow expose what I observed in that context. So I wrote it. In this way I was able to show and articulate the more complicated parts from the less complicated ones in that code which were specific for that project.
That’s the moment when I realized why we should use the “tools” word instead of the “automation” word. Also is then when I realized how limited we are when using the idea of automation/tool. We can use/create a tool to, among other things:
- create test data, for example by using combinatorics, Montecarlo simulations, de bruijn sequences;
- to analyze and display the logs in a certain way;
- to see where to focus our attention at a certain time, depending on what is in the source control;
- exercise the system in a specific way so that the tester can observe tangential and oblique problems.
In other words, we do not use this tool idea to its value and power. We limit ourselves through the UI to press some buttons. We use this automation thing, as I observed, only to try to simulate a user. How strange, we use a machine to simulate a thing only a human can really do, but do not use the tool/automation for things which are supposed to be used.
So, back to the questions from the beginning of this post: Do we really need to concentrate our automation efforts in simulating users via Selenium or other tools? Isn’t this a sign of other problems that cannot or will not be solved?
As a programmer, so from a technical perspective, I have observed this obsession to simulate users in an automated way when:
- the product code is very bad, so is hard to include any automation checks or testability points at a fine granular level and because of that nothing will be done about it;
- when there is no trust in the automated checks done at a fine granular level;
- when programmers do not know how or do not want to do automation cheks;
As a manager, I have observed this obsession to simulate users in an automated way when:
- Testers see that what is payed in the industry is this user simulation automation, so they find any excuse to do it;
- When testers do not know what testing is and how to make a test strategy. Because choosing a tool as selenium and implement automation checks is not a test strategy – but this is for another post;
- When managers have no clue what testing is. But they are delighted when they see some encoded test cases in a wiki page and even happier when those (preferably all) are automated whatever that would mean. I wonder how their reaction would be if we would insist that their whole work be automated, probably not too well.
So, I have a new heuristic to spot problems in testing regarding automation which I named it “Automated user simulation heuristic”.
I think that using automations/tools in testing is really cool and useful and there is no doubt about it, not for me at least. Too bad that we focus on the wrong direction in using it.
This is a three-part series:
Automation in testing – Part 2 – Do we really need to concentrate our automation efforts in simulating users via Selenium or other tools? Isn’t this a sign of other problems that cannot or will not be solved?
Automation in testing – Part 3 – What bugs you aren’t finding while you automate checks?