Rapid Software Testing

Unit tests, programmers, and testers

These days I am participating, as a Peer Advisor, at the Online Rapid Software Testing Explored class with James Bach.

I had an interesting experience that made me think and articulate more carefully about a certain subject. I say this because although we might know we have the answer to some questions we usually see just a part of a situation and sometimes we might forget (not consider) more clearly the context(s) for which the answer might apply. But there is more. Is about the context(s) of the one who poses the question, the context(s) of the ones who might participate in the conversation and the context(s) that influenced my answer.

The example described below helped me realize this and adapt my message.

At a certain moment, the topic of testers and unit checks[tests] and whether testers should be involved or do them, popped up. I gave a response. Afterwards somebody in the class thought that I was saying that testers should never touch the product on a unit level. Then another student in the class asked if a tester could do pair programming with a programmer, when the programmer was doing the unit checks[tests].

I am glad that those two students stepped in and made me aware of their contexts.

At that moment I realized that my message was not accompanied with all the premises I had in my head and the realities that affected my response.

I say premises because of a blog post, written by Michael Bolton, where he says that: “Most arguments seem to be about conclusions, when they’re really about premises” (1)

So here are premises that I had in my mind:

unit checking[testing] with mutation checking[testing] AND TDD are techniques that the programmer uses to take care of the semantic stability of the code. (2)

it is helpful for testers to have interactional expertise, as Harry Collins intends it, when it is about code. Allways a session of pair programming or a code review done with a tester is something that is special. Special in the sense that the programmer feels very well that is something else. Is like the tester without wanting it, it triggers some tacit aspects in programmers mind regarding code, that usually with another programmer does not necessarily happen.

me witnessing and discussing with testers, often, that they are being forced to do unit tests[checks] and selenium via UI. Testers which have no programming background and for which the message is that if they don’t do these, they are out.

for me testing is this one that James Bach is teaching, the RST (more broadly I am thinking about Context Driven Testing where we have the RST, BBST, JIT, Buccaneer Testing)

I have been a programmer for 19 years already. It was hard, but at this point I can say that I no longer make any confusion between testing and checking (3). And also, I am very aware of how management might misuse, abuse, play these disciplines, not in a naturalistic way of how things should fit, if may I say so.

I deeply appreciate and encourage when both programmers and testers try to find out about each other’s nature of work. And I say this because of several reasons: empathy, helping in sensing and navigating the complexity of the product needed to be done, helping with test ideas the tester.

context is everything, I have no doubt. And when I say this, I am fully aware about the cognitive patterns activations which are triggered by the situational context. (4)

knowing the origin of a theory, principle, … is good because when making deviations from it, we will know why it might work or not work. For example, we have been doing at industry level OOP wrong, I believe, for the past 40 years now (5). We have just taken some principles/techniques out and popularized them without understanding the whole thing as it should be, and we wonder why it does not work.

the importance of identity/role and how in the IT industry we try to smash these and use in reckless ways just some techniques.

testing is an intellectual discipline because in RST we say that testing is applied epistemology.

a tester is or should be closer to the people using, in the end, the application.

So, should a tester do TDD or unit checks[tests]? And I realized that it depends also on what it means by ‘do’. Does ‘do’ mean:

being forced by a polluted context where the real testing has no chance of being done and the tester otherwise might get fired.

because the tester wants to learn what it means for a programmer to do it.

because the tester wants to learn programming in this way.

because the tester thinks it’s his/her responsibility.

because the tester is also a programmer and switching context might be needed (Side note: Very often I find myself in this situation. But I am fully aware about the programming and testing disciplines and their particularities, intentions, goals…).

because the tester wants to help, she/he loves so much the context that will do whatever it takes to help.

I have answered initially in the class by considering just one interpretation of ‘do’. Mea maxima culpa. But I did it because I am deeply affected by the realities I am witnessing. Realities which try to reduce testing from a discipline to just some techniques which can be fully automated, if I may say so. And as a programmer I know that I need a complement that will not confirm and follow my biases but will really complement my actions/work/ideas.

(1) Testing, Checking, and Changing the Language. James Bach, Michael Bolton. http://www.developsense.com/blog/2009/09/testing-checking-and-changing-language/

(2) Robert C. Martin. https://cleancoders.com/

(3) Testing and Checking Refined. James Bach, Michael Bolton. https://www.satisfice.com/blog/archives/856

(4) The Context–Object–Manipulation Triad: Cross Talk during Action Perception Revealed by fMRI. Journal of Cognitive Neuroscience. Wurm, M., Cramon, D., & Schubotz, R. (2012), 24, 1548-1559. https://doi.org/10.1162/jocn_a_00232

Implicitly activated memories are associated to general context cues. Memory & Cognition. Nelson, D., Goodmon, L., & Akirmak, U. (2007), 35, 1878-1891. https://doi.org/10.3758/BF03192922

(5) When I think about OOP, I have in my mind the work of Trygve Reenskaug and James Coplien regarding OOP: https://fulloo.info/

Automation in testing – part 3

I would like to share with you an example that totally changed my perception of how I look at automated checking, whether it is unit or integration.

I am programming a lot in .NET. The language I use is C#. The code written in the end is packaged in a .dll or .exe file. If you will look at the content of this file you will not see C# code. You will see MSIL code which is like an assembly language. This MSIL at runtime is translated into processor code.

Here is a C# code, it is taken from the book CLR via C#:

This code is a little bit weird. I admit that it is an extreme example. But it helps me communicate what I want. Let’s suppose that we covered, so to speak, this code with unit checks done ok. Lets also suppose that the code review was quite rigorous and carefully done.

I have some questions:

Is it just a line of code?
How many memory allocation problems can occur?

Let’s change the perspective. What you see bellow is the MSIL code generated from the C# code you see above:

Side Note: By the way, this MSIL code generated from a .Net Core code. Do you think it’s very different from .NET 4.8 code? The answer is no. I ask this because there are people whose CVs will not be accepted because they have not worked with .NET Core, but have experience with .NET. For me is an indication that we focus too much on shallow things.

All those things you see marked with red indicate a place where memory allocation problems might occur.

From my experience not a lot of .NET programmer’s look at the MSIL code. But let’s think about unit checks. You see, what you do know isn’t canceled, it’s okay. But now don’t you think about code coverage in a different way? It’s something else, isn’t it?

Note: For Java we have Java bytecode

This perspective helped me, in this context, to understand that I can think, for example, of memory corruption coverage. Which is just another type of coverage. But I wrote this post to try to answer another question which is: What bugs you aren’t finding while you automate checks? For me, as a programmer, it is very clear that I focus on a certain direction when developing an automated check and nothing else. But focusing on a certain direction means ignoring other dimensions. In the example above I would have ignored memory problems or what MSIL code is being generated and the kind of instructions which will actually be run.

This is a three-part series:

Automation in testing – Part 1 – Do you think it is normal to say test automation? Really everything can be automated in testing? Are we focusing automation goodness in a good direction?

Automation in testing – Part 2 – Do we really need to concentrate our automation efforts in simulating users via Selenium or other tools? Isn’t this a sign of other problems that cannot or will not be solved?

Automation in testing – Part 3 – What bugs you aren’t finding while you automate checks?

Thank you Johanna Rothman

Automation in testing – part 2

Do we really need to concentrate our automation efforts in simulating users via Selenium or other tools? Isn’t this a sign of other problems that cannot or will not be solved?

I really understood how I should look at the role of automation in testing when I participated in an audit. The role of that audit was to help a company to obtain information, also technical information, in order to know if the acquisition of that company would be made. My responsibility was on the technical part to look at the code and the technologies being used. When investigating the code I noticed some patterns. There was no tool on the market to help me, somehow expose what I observed in that context. So I wrote it. In this way I was able to show and articulate the more complicated parts from the less complicated ones in that code which were specific for that project.

That’s the moment when I realized why we should use the “tools” word instead of the “automation” word. Also is then when I realized how limited we are when using the idea of automation/tool. We can use/create a tool to, among other things:

create test data, for example by using combinatorics, Montecarlo simulations, de bruijn sequences;
to analyze and display the logs in a certain way;
to see where to focus our attention at a certain time, depending on what is in the source control;
exercise the system in a specific way so that the tester can observe tangential and oblique problems.

In other words, we do not use this tool idea to its value and power. We limit ourselves through the UI to press some buttons. We use this automation thing, as I observed, only to try to simulate a user. How strange, we use a machine to simulate a thing only a human can really do, but do not use the tool/automation for things which are supposed to be used.

So, back to the questions from the beginning of this post: Do we really need to concentrate our automation efforts in simulating users via Selenium or other tools? Isn’t this a sign of other problems that cannot or will not be solved?

As a programmer, so from a technical perspective, I have observed this obsession to simulate users in an automated way when:

the product code is very bad, so is hard to include any automation checks or testability points at a fine granular level and because of that nothing will be done about it;
when there is no trust in the automated checks done at a fine granular level;
when programmers do not know how or do not want to do automation cheks;

As a manager, I have observed this obsession to simulate users in an automated way when:

Testers see that what is payed in the industry is this user simulation automation, so they find any excuse to do it;
When testers do not know what testing is and how to make a test strategy. Because choosing a tool as selenium and implement automation checks is not a test strategy – but this is for another post;
When managers have no clue what testing is. But they are delighted when they see some encoded test cases in a wiki page and even happier when those (preferably all) are automated whatever that would mean. I wonder how their reaction would be if we would insist that their whole work be automated, probably not too well.

So, I have a new heuristic to spot problems in testing regarding automation which I named it “Automated user simulation heuristic”.

I think that using automations/tools in testing is really cool and useful and there is no doubt about it, not for me at least. Too bad that we focus on the wrong direction in using it.

This is a three-part series:

Automation in testing – Part 1 – Do you think it is normal to say test automation? Really everything can be automated in testing? Are we focusing automation goodness in a good direction?

Automation in testing – Part 3 – What bugs you aren’t finding while you automate checks?

Automation in testing – part 1

Do you think it is normal to say test automation? Really everything can be automated in testing? Are we focusing automation goodness in a good direction?

For me, also as a programmer, testing is, among other things, about finding those relevant patterns that matter within a complex system. But in order to understand complex systems a person must interact with it. But how do we do that?

If you look carefully in the image of the post, you will see a forest. Forest represents testing. Trees represent parts which form the forest. But let’s not confuse the forest with a tree. Each tree has its role in that environment.

Small note about words: Some time ago I worked at a company in a management position. Weekly we had a meeting between all the middle managers. What I noticed in these meetings, which were held in my mother language, Romanian, is that we used a lot of English words. I am speaking about words which have a good translation in Romanian. In one of those meetings I realized how superficial I was. Why can’t I speak about those concepts in my language? So I began to be more careful about this aspect. When I tried to translate and adapt those words in my mother language, I began to understand even more the topics I wanted to speak about. It is then when I really understood the power of words. How they can be used to stimulate thoughts, awareness and insights.

As a programmer I wrote and I write a lot of code, and also code to verify the code written. And now when reading the previous sentence, I realize that I do not write code to verify the verification code. Interesting insight I had by just changing the formulation of an idea.

But I noticed something also. The verification code I was writing was on things which were clear, deterministic which could be approached in an algorithmic way. How does this relate to testing? For me this is a technique to find problems in code.

But I can also use other things when testing like modeling, questioning, inferring.(see the blog post image).

So those automated verifications are part of testing like the others. Personally, I like to call these automated verifications also automated checks.

Therefore this checking stuff is included in the testing, it is an integral and fundamental part of the testing. But we can’t say that only this aspect defines testing. We can say that it is a component of testing. And this part can be automated because it is an algorithmic verification.

If I got involved to understand what testing is , it is because I had the occasion to work with wonderful testers. And I wanted to understand their craft. These testers actually did not have coding skills, but still they were able to find interesting and important bugs. I saw how they used their social knowledge to make a lot of judgments about what to test, how to test, when to stop. There is no algorithm to make these reflections, judgments, assessments. What actually I noticed was how important is the human aspect in testing. Like a conductor who conducts an orchestra and the musicians who use the instruments(tools) to generate the music. Those instruments can be mechanical, but also digital, whatever is needed to transpose the music.

I asked at the beginning of this post the following question: Do you think it is normal to say test automation? I did this as an invitation to see what we might miss if we frame it this way. For me, each time when I witness this framing I see an overemphasis on one dimension of testing and all the others dimensions ignored. That’s why I like to say automation in testing because it serves as a way to increase awareness on how automation relates to testing and that other things might be considered also.

This is a three-part series:

Automation in testing – Part 1 – Do you think it is normal to say test automation? Really everything can be automated in testing? Are we focusing automation goodness in a good direction?

Automation in testing – Part 3 – What bugs you aren’t finding while you automate checks?

What is testing for me

Some days ago I had the occasion to make a presentation regarding testing. But I had one dilemma before. How could I have been able to do this presentation without me being able to say what testing is for me?

There is a concept, praxis, that started my quest to understand how theory can inform and make sense in the practices, in my day to day work. And when I say practices I think about practices in testing, management and programming. There are lots of them. I began to cover some of them, please look here for examples. The concept I am speaking about now is the complexity theory.

Is not easy to explain this complexity thing. I like how Edgar Morin explains the complexity. For him complexity is like a tissue or fabric of inseparable associated diverse elements. But at a second look, those elements represent: events, actions, interactions, feedbacks, hazards.

I like the header image because it is trying visually to show this complexity. You can see in the image some dots, connection between the dots, but it also shows patterns that can be formed (different lines, triangles, pentagons, polygons and so many others). In our day to day life those dots might represent things like:

Technical details in our product(components and objects relating to each other, the fractality of the code, mixing up unrelated abstraction and so on)
Humans with their roles working on a product
Risks
Requirements

And then there is another important detail and that is the fact that humans think in patterns.

So, what is testing for me? Testing is about finding those relevant patterns that matters within a complex system.

Those patterns might mean whatever is relevant in spotting problems within the product or process. These patterns are not static, they continuously change.

Risks, feelings, management

What if everyone –our managers, auditors, testers — got it wrong with the topic of risks?

What if what we witness regarding risk management is just a charade ?

Long ago, I read Jerry Weinberg say, “The definition of quality is always political and emotional.” That resonated with me then and still does.

I had to get out of a project. At that time I occupied a management position. Usually in taking over a project there are some intermediate steps that need to be done. There was no pressure. It was time for my replacement, let’s call him John, to take it over.

John was also in a management position. We both worked on the same big suite of products. My project was an integration project, making the communication between products from the suite work. Part of the problem in my project, before taking over, was that it contained hacks. By hacks I mean things which did not belong there at all but on the other products,at the source.Unfortunately, managers from the other products did not want to apply the needed fixes on their parts, to remove those hacks. So all the garbage needed to be fixed on their part was thrown on this integration project, before I came.

My team fixed these problems in both the integration product and the other products from the suite.

As I reflect back on my conversations with John,I remember several things that appeared “off.” He seemed very detached about topics we needed to discuss in order for him to take up the project . My perception was that John had exaggerated detachment. When he spoke with me everything was ok. But with the others(management from client , our management, teams) his conversation was all about risks.

I did not mind at all that John was concerned about risks. What I did not understand in those moments was his lack of convergence. John spoke with me about risks but in a calm way. When he spoke with others he was doing it in an alarming and exaggerating way. Also I have noticed certain resistance from the teams he had. Anyway that exaggerated perceived detachment in combination with how he handled the risks puzzled me.

Feelings, as much as data, play an important role in decision making.

Then, I found some scientific research about risks, the risks as feelings hypothesis. This hypothesis says that response to risky situations and how we evaluate them are influenced directly by our emotions/affect. The traditional models of evaluating risks tell us about probabilities and consequences but nothing else. But these, actually, are seriously influenced by feelings.(1)

But there is more. You may wonder why us , in IT, should we care about these studies? Because, is about validated scientific knowledge. Things which other disciplines already studied and proved it. We only have to cross the corridor to these disciplines and learn from them. I like the word praxis to describe this.

So, I spoke about the psychological part. But there are also sociological and anthropological studies. These studies show that risk perception has roots also in sociological and cultural elements. For example, the sociological component is telling us how we are influenced in dealing with risks by colleagues, friends, family. The anthropological component tells us that people within groups minimize certain risks and exaggerate others as a way to maintain or control a group.(2)

I realized John had feelings, and those feelings might lead him to assess risks differently than I did. Maybe he felt fear because of the history of the project.

As said above there is also a sociological part which might help in making sense of this situation. I tried to understand from whom he was taking certain technical information. Who was advising him on this part. And I realized it was about some of his team members who did not want to be involved in the take over.

The anthropological component helped me to understand the exaggerated escalation to management. It was a way to maneuver the management groups by playing with risks.

Till that moment I only witnessed a shallow view of how risks were used by management. It was an eye opener to me regarding risks and how they should be used in management, testing, audits.

I mentioned audits. I would like to see an auditor knowing these aspects of risks. Maybe this will help in order to be able to see the spirit of the law and not just the letter of the law. Risks combined with the tacit knowledge might help understand the why’s of why certain risks are articulated, handled and made public in a very specific way. Audits is not about simple receipts ignoring the context and professionalism of a discipline.

Emotions play an important aspect in, risks, judgment/evaluation/assessment. How right Jerry Weinberg was.

(1) Risks as Feelings, Loewenstein, George F. and Weber, Elke U. and Hsee, Christopher K., Psychological Bulletin, Vol. 127, 2001, Available at SSRN: https://ssrn.com/abstract=929947

(2) Perception of Risk, Paul Slovic, Science 17 Apr 1987:Vol. 236, Issue 4799, pp. 280-285, https://science.sciencemag.org/content/236/4799/280

Tacit knowledge, testing, requirements, management

Have you ever wondered why it does not make sense to ask or expect for complete written requirements?

Do you know why trying to write all test cases to capture the testing needed to be done will bring you to a dead end?

Sometime ago I worked on a big project. There were about 50 people involved. These people were divided into various teams. And these teams handled various products from the entire suite.

One of those products was hard to install. Imagine that simple settings that could be saved in a config file, like xml or json, were saved in windows registries. The installation of the suite in production was a ‘special’ event on its own. I use the word ‘special’ not in a good sense. Each time an install was being done; people were awake in the middle of the night to stay to test it. But this was one problem in a sea of problems.

One day, the man responsible for the product installation announced his resignation. His resignation worried me. I asked my colleagues in the office. They weren’t worried. They said, “it’s a different situation, but look, the man who is leaving also writes in a wiki. So, it will be ok”. They had the impression that all what is needed will be made explicit.

For me this did not make any sense. Not all knowledge can be made explicit. We have a lot of implicit knowledge. Harry Collins in his wonderful book named Tacit and Explicit Knowledge explains why. Imagine that knowledge can be represented like an iceberg. What is above the water is the explicit part and beneath the water the tacit part. So:

weak or relational tacit knowledge – Is tacit because of the relationships between people that arise in a social life context. For example:

* concealed knowledge – knowledge which is kept secret intentionally;

* ostensive knowledge – knowledge which is hard to be described/comprehended. Because of that we point to a word, object or a practice to describe it;

* logistically demanding knowledge -imagine a person who knows where everything is, but would not be able to list it if asked),

* mismatched silences – knowledge kept secret without intention;

* unrecognized knowledge – a person is doing certain things in certain ways. This person would not tell another person because it might not know if these are worth to be told for the second person.

medium or somatic tacit knowledge – is tacit because it is incorporated in the human body. How we type, how we juggle, how we balance on a bicycle. What is also important here is about the contrast between conscious and unconscious processing, just think of the process of learning/riding a bike.

strong or collective tacit knowledge – is about the social aspects/interactions/relations people have and the knowledge that derives from this. These will influence also the social judgment of why certain things were done. The most important thing is that these things cannot be described and learned by explanation; it must be practiced in a social environment.

For me this was the decipher key of this type of situation. It was more than clear to me that even with all the goodwill, that man could not have written everything in a wiki and that things would have escaped to him. And this is ok, normal. I felt calm. I don’t know how to describe this calm; maybe it was a calm because it helped me focus on what I knew would come.

Do you wonder how the story ended? That moment I anticipated arrived, and it was a difficult moment for my colleagues in that project. So difficult that replacement also suffered enormous stress and left.

Connection with test cases:

In the example above I said about that man who had to write about the steps (and nifty details) related to an installation. But this also applies to requirements, test cases. We have the impression that if we will concentrate all our efforts on writing and managing by test cases is the way to go. Actually, it is a waste, a not so good direction because on testing we do much much more. There are so many aspects which can be covered that it is really hard to articulate it.

With this overemphasis on test cases we ignore the way we are and how we gain knowledge. Testing encompases many things like learning, critical thinking, experiencing and many others. A lot of this is not explicit but tacit.

For example, I said about critical thinking, this is for me a form of ostensive knowledge. I used 2 words to point to an entire discipline not so easy to describe and comprehend.

Learning about the product involves also understanding why certain decisions were made about that product. Being aware that certain knowledge will be kept secret without intention, mismatched silences or not being aware that an information is so important that needs to be shared-unrecognized knowledge.

Experiencing a certain product might not be easy at all. Information might popup only when discussing with persons involved in the process of working/developing with that product(collective tacit knowledge).

Connection with requirements:

Is helpful to have things written down especially for remembering them. In Scrum, for example, we write to remember but we do not write to communicate the Product Backlog Items(PBI’s) – the PO tells them.

Also this helps in setting more realistic expectations about written requirements.

Connection with management:

The story I gave was the moment when I understood the importance of theory and practice (praxis) , in this case regarding some aspects of tacit and explicit knowledge. As a manager it helped me to size up an unexpected situation and also be aware of things that might happen.

Rapid Software Testing course with James Bach

In November 2017 I attended the Rapid Software Testing course held by James Bach. Now, this course has been renamed as Rapid Software Testing Explored.

When I was on the train, going to the training, I imagined what I would have written about the course in those days. But I did not do that. After the training, I decided not to write about it and wait for a good period of time, to see how it will influence me in real life. I think this is the most important thing in the end.

I think that the time to write about this course has arrived. I hope you will find it useful.

Premises

There were some things preceding the training which, I believe, are very important and worth mentioning:

– I knew about the work of James Bach and Michael Bolton. Before going to the training I have fully read their entire blogs, and also the book “Lessons Learned in Software Testing: A Context-Driven Approach” written by James Bach;

– I knew about the work of Jerry Weinberg and have read a dozen of his books, especially the book “Perfect Software: And Other Illusions about Testing” ;

– For 2 years, before going to this training, I tried to understand the testing envisioned by Jerry Weinberg, James Bach, Michael Bolton, Cem Kaner;

– I was a software developer and team lead;

– Going to this training involved also personal money/cost contribution. I mention this because I am strongly influenced here by the “skin in the game” concept of Nicholas Taleb;

– I have read the rst.pdf file, which was the course support material..twice;

– I have searched the internet for testimonials regarding this course with enough details, but I did not found a lot of them.

Feelings

I remember the strange feeling I had when I entered the class room. James Bach was there at his desk, but I did not dare to say anything. I can say that I was overwhelmed. It was unreal for me to be there.

After the first 2 hours of course, I realized that I knew so little about testing…

Details about course

I will try to write about this course via 3 dimensions:

-structure: the flow of the course in those 3 days;

-key words: it’s interesting to describe a training via some keywords used in the course, which can be enumerated coherently in a minute or so. I think it can have a good impact;

-pearls of wisdom: these are insights from the training, words used by James Bach. You will see that each of the texts is within quotes and preceded and followed by “….”. This is because the text has a context(story, experience, people, time, history, feelings, exercise, place). Some of those phrases took even an hour or 30 minutes to be described by James Bach for us. So there is a risk, for the reader, not to decipher the real meaning/lesson/message, but I put them there in the hope that it will convince/entice you to go at his, and Michael’s, training and find out more;

Structure

The course was centered around exercises, experiments and interesting surprises. These exercises were not easy at all, it was hard stuff(even if at the beginning, sometimes, for us it felt easy, we found out that it needs to be analysed first and we needed to ask for more information). I saw damn good testers being tricked. But James warned us.

Since these exercises and challenges were hard, he spoke to us about mistakes, about his mistakes. He told us that we should not be scared of them and that we should learn from them.

These exercises were inspired from real life difficult situations that James Bach experienced.

Each exercise was followed by a lesson/message. This was the beauty. If someone would have told me in advance the same lesson, I would have not understood it as I should – I am sure of that 100%. But you know why? Because we struggled with it, because it had put us in a delicate situation…try to imagine social pressure or a hostile environment, it is not enough to read about it, you have to live it.

The explanations were full of metaphors, so the message can be easily understood.

The interaction with James was via the Socratic method.

James intention was to make us learn more rapidly the things he learned in a much slower way.

Although I have read in advance the pdf which accompanied the course, I realized that I have missed A LOT of insights…

Key words

Initially, I was overwhelmed by all the details from the course and I did not knew how to describe it in a short and coherent way. But, shortly after, I realized that I can use some words used in the training. So here they are:

context based risk analysis, compatibility regression testing, sanity check, test strategy, business risk, normalization analysis(from therapy), leader and its attitude toward tester, thinking outside the box, parenting, biases, process of thinking, pattern(s) recognition, efficiency and relative efficiency, sense making, common sense questioning, interview for hiring, diminishing returns principle, fallacies, design tests, cope with complexity and uncertainty, scepticism, risk mitigation, quality, agile, think negatively, tacit test procedures, normalized test sessions, cognitive science, social dynamics, psychology, social skills, qualitative methods, social pressure, functional blindness, hostile environment, data types bugs, boundary testing, heuristics, perimeter assumptions, defocusing technique, factoring, randomness, heuristic test strategy model, professionalism in software testing, hazop, inattentional blindness, audit testing, penetration testing, anchoring bias, mental testing models, adversarial images(from AI) , statistical correlation, product coverage outline, oracles, lifeboat argument, explainability/confusion oracle, person whose opinion matters oracle, defocusing, diffusion limited aggregation, deep testing, exploratory and scripted testing.

Pearl’s of wisdom

As I have already said, it is important to mention that these phrases do not have context around them. This is intentional. It is an invitation, if you like or find it problematic or… to dig more, even to go to the course. So:

-“… Sanity checking means shallow test coverage not deep test coverage. For the purposes of identifying insanities. You are looking for, not just something that is a little bit wrong, you are looking for something that is totally broken. If it’s totally broken I want to know about it …”

-“… All testing problems have a context, and if you know that context you will be able to make a sensible decision about what to do next. And if you don’t know the critical context variables then you will suggest a ridiculous answer …”

-“…. Rapid Software Testing isn’t rapid because you are typing faster than other people. It’s about thinking: Do I really need to do this? Why do I need to do this? What things can I set aside? And what things do I really need to do? …”

-“… What a good leader should say, knowing how delicate and emotional testing is: I want to know if there is a problem. Don’t worry about the ship date. We will slip it if you say we need to. That is not your concern. What your concern should be: Can I find problems in this?. Let me worry about …”

-“… When you teach a tester how to test you are teaching that person to see something, to see things that you have never seen before …”

-“… Testing is an exploration and experimentation process …”

-“… A paradox of RST is that: RST tells you that you should do the right thing. Don’t do anything unnecessary, that will save you time. But how do you know what the right thing is? First you’re gonna have to do wrong things to find out what the right thing is…So the paradox is that in order to get rapid you usually have to start slow to climb the learning curve…”

-“… Testing is an intellectual activity …”

-“… Never say:I tested it and it works. That is always an untrue statement. The truth is you tested it and you haven’t seen it fail yet…I tested it and I know it can work… “

-“…Testing is not about assessing that the product is good enough, it’s about finding problems ….”

-“…Counting tests is pointless ….”

-“…Rapid Software Testing methodology which is a mindset and skillset, but it is not a set of rules, it’s not a set of templates….”

-“… New novice testers don’t need lists. They need experience. How do you give them experience? Sit with them down and test with them …”

-“… Here is what you never do: you never derive tests from requirements, just don’t say that…”

-“… It is not possible to write down a test …”

How it helped me

This course helped me realize/do a lot of things:

– I will do everything I can to have trainings with the best in the industry. With those people who create concepts. If this means paying from my own money, so be it. I prefer to wait a year or two, but to learn from the best out there;

– Online trainings are important and have their place, but they can never ever substitute a live training held by one of the top of the top of trainers;

– At least once per year I would do the Rapid Software Testing Applied training;

– As a programmer, this professional testing dimension influenced me a lot. I can no longer define myself as only a software developer and ignore the tester part in me;

– I understood how and why so much money is lost in IT with superficial testing and the “automation” trend which now is being marketed ;

– I understood how to be a better mentor;

– It helped me by giving me force to write about testing and to defend it;

– I realized that, now, in IT, a lot of concepts currently used are not understood: testing, agile, scrum, OOP, architecture…

– I was able to help/coordonate a student, an experienced developer, for a masters degree in IT regarding testing. It was the first time such a subject was presented in university and it really impressed. I was glad that, finally, the real face of testing was presented in an institution which prepares students for the IT industry. I hope that in the future I will be able to teach a course on testing, but I am not yet prepared, I need help. I think that if testing, the real one, would be taught in University it would help even more the spreading of what real testing is, which in the end will help the industry as well as the people in it.

Conclusion

I liked this training a lot. I never would have thought that it would influence me so much.

How testers can provide value for developers

On the Rapid Software Testing slack Channel(1), one month ago, an interesting question was put by David Högberg:

“Here is an interesting story and question from my friend in another Slack-group:

(translated from Swedish)

”Today was the last day of a sprint in one of the teams I coach at….

A backend developer has been very productive this sprint, he has pulled in more tickets than planned and finished those. Instead of pulling in another ticket on the last day he took the initiative to work on the unit test coverage and integration test coverage in our most important repository.

Now we have 100% code coverage for both unit and integration (API) and a branch coverage above 90%.

Besides this he always tests his own code, with both wide and deep understanding of the requirements. He always ask for feedback regarding his own testing, wants critical code reviews and practises risk-based exploratory testing.

This is what QA-coaches and testers will have to work with in the future. How do we create value for these future developers?”

….What are your thoughts regarding my friend’s question? How can testers provide value to developers who deeply care about code quality and at the same time are good at testing themselves?” (2)

The point raised by David Högberg is important and should not be ignored. Also, I saw that some testers were somehow speechless regarding this, like there was nothing to say/do/challenge/help.

There are lots of things testers can do. Testers must/can challenge things and there is a lot to challenge. So, for example:

1. The fact that what the developer does are unit checks , integration checks not tests.(3)

2. He said that he reached 100% code coverage for unit checks and integration checks. Actually code coverage is a measure and number for programers only. This is because it helps the developer to see areas not covered at all, not how well that area of code is covered. So, this number should not be exposed to management, because you can have 100% code coverage without code actually being covered. From the moment when you hear that code coverage number is exposed, there might be trouble – I know it sounds strange, but for me is a sign of exaggerated confidence. Maybe I am wrong, but for me is a heuristic for looking with care, because of the things which can’t be articulated in simple boolean confirmation statements.

I have to say that each time when I discuss with managers I try to do my best to convince them to ignore the code coverage number and the fact that it means nothing.

By the way, when I have the occasion I ask programers this simple question: So you are so confident with the unit checks? If the answer yes, then I ask: Good, are you ready to bet you salary for 3 months that I’ll not find any problems? Usually, I perceive hesitation.

A good technique to verify the actual lines/conditions, in terms of coverage, is mutation checking(4). Even with this technique, I have in my mind the difference between testing and checking! (3)

3. As a tester you can help the developer with the sampling, the data she/he is choosing. And, as you know, this is not a skill which can be learned in a day or two. Sampling at the unit and at the integration level is important and I saw a lot of times unit checks and integration checks which repeat themselves.

I do not mention that testers, need to know very well about cognitive biases, critical thinking.

4. The developer said that he/she made unit checks? Perfect. Challenge that. Take a look at this brilliant article by James Coplien: Why Most Unit Testing is Waste . You’ll notice that this article is not so easy to digest, is deep – it speaks about the theory of information, also it does not follow the current dominant thinking about unit tests. I’m sure it will help you to make that developer think. Maybe negative reactions can take place, but is OK, we want fruitful disagreements.

5. The developer said he/she is doing TDD? TDD does not help at big architecture and OOP. It helps, maybe, at the micro level, but for sure not macro. This means architecture, OOP is fully exposed in a big and nasty ways. Maybe integration checks can help to see that the parts are fitting together, but are not helping to see that they are ok built. What this means? Well… nasty things are hidden at the integration boundaries. Developers don’t do OOP, but Class Oriented Programming. OOP would mean to reflect the mental model of users, which this does not happen. All the work is split between a lot of classes and functions. I think you can imagine the dangers.

Regarding TDD, I noticed that some persons begun to make some research about it and it is interesting: A Comparative Case Study on the Impact of Test-Driven Development on Program Design and Test Coverage or Does Test-Driven Development Improve the Program Code? Alarming Results from a Comparative Case Study

By the way, I also liked this post: TDD is dead. Long live testing

So, please, go and see the research, I doubt a lot of programmers do that…

6. Unit checks and integration checks is also code, this means that is prone affected by things like: bugs, maintenance, design, analysis, architecture.

7. I love the article of Max Boisot with the patterns(5). For example: for 10 dots there are 45 possible links and 3.5 trillion possible patterns. I’m sure in applications there are more than 10 connected dots. Is not possible for a developer, no matter how smart he/she is, to be sure he/she covered all the possible patterns and also to be aware which pattern is more important than other(s).

8. Regarding “A backend developer has been very productive this sprint, he has pulled in more tickets than planned and finished those.” Maybe I’m interpreting this in a wrong way, but sorry… I’m not interested in the productivity of the developer. This is a very nasty stuff. For me is important that the team and the whole work to be ok. When I read productivity of individual, I begun to doubt the good functioning of the team and this is bloody dangerous. Is like measuring tester for the bugs introduced, sorry, but no. Were there any other members actually working on stuff? If yes, why he did not took care to help finish that? – I’m deducing this from the text, maybe I’m wrong. In Scrum the team works on a single PBI at a time.

Still the list is short. I would have added about cognitive dissonance, of how the team structure affects the code structure, but you got the idea…

Conclusion: So how testers might create value for these kind of developers? I think that when a tester will put this question then is a good sign because learning/helping continues.

(1) Michael Bolton, “RST Slack Channel”, https://www.developsense.com/blog/2017/10/rst-slack-channel/

(2) Text used with permission, thx David Högberg

(3) James Bach, Michael Bolton,”Testing and Checking Refined”, https://www.satisfice.com/blog/archives/856

(4) Robert C. Martin, “Mutation Testing”, https://blog.cleancoder.com/uncle-bob/2016/06/10/MutationTesting.html

(5) “The problem of connecting the dots”, https://sensing-ontologies.com/the-problem-of-connecting-the-dots/

Round Earth Test Strategy and earthquakes

Recently, James Bach published a nice analogy/model regarding test strategy, named “Round Earth Test Strategy“(1)

Early in 2018, in January, James Bach was kind enough to share this analogy on a special channel RST( this channel was created especially for RST alumni; it’s an active and very very very useful group to be in – another very good reason to take RST class(2)).

I liked it. But, in that moment, and days/months after, I felt that I still miss some aha moments, about those things, which confirm in a way that a certain subject is internalized in myself somehow.

Museum

In the image of this blog post is the Natural History Museum in London. It was here where I had the aha moments I knew I was missing. Suddenly, in my head, I remembered James Bach’s round earth model. In there is a section named “Volcanoes and Earthquakes”. When I began to read what was written, I was amazed how the things described there can be used as an analogy, as James thought, for testing, but maybe also for problems/risks/bugs/tools/approaches from the IT field:

● “Earthquakes can happen without warning, causing death and destruction on a massive scale. When they strike we feel a sudden, violent shaking of the ground, but they are caused by slowly moving plates on Earth surface. As these plates move, pressure builds up until it finally gives away”(3) → In reading this text I remembered bugs. How certain bugs can create a lot of mess (urgent calls, staying late in the night, missing important moments). How those bugs appear suddenly, without notice. But what stroke me is the fact of slowness and violence of earthquakes.

Slowness in:

– how functionality is written → It’s not like the functionality is developed instantly. There are meetings, mails, writing code, searching through code, testing, etc.

– execution of the program – managed memory leak, usually, takes time to be observed. But when it reaches a critical point, entire system crashes not only the programs.

The idea of moving plates made me think of integration testing.

● “Preparing for the worst; living with earthquakes: Scientists can’t predict exactly where and when an earthquake will strike but they know roughly which area around the world are at risk. It is vital for the people living in these areas to prepare for what may come and know what to do when it does. Without adequate preparation, earthquakes can cause huge suffering and destruction”(3) → It describes perfectly the role of testers and why they are searching for the worst, why they should think negatively. Testers, like these scientists, should understand that they cannot predict and also can’t be sure that the software is ok (just a black swan among thousands and thousands of white swans was enough to proof that there were not just white swans). Although they can’t predict or prove the correctness of a software program, they will use models to identify possible areas with problems guided by risks (For example, by looking at the source control metadata, weak areas within the source code can be discovered. ). Since we’re talking about people, the risks also have a psychological and sociological dimension. It is sure, problems will occur and maybe we should guide our testing also by the possible suffering we create, for the ones using our software.

● “Impact scale: There are different ways of measuring earthquakes. Unlike Richter scale, which measures magnitude of the shaking, the Mercalli scale measures the amount of damage caused – the loss of life and the damage of buildings. Generally speaking, the higher earthquake magnitude the greater the devastation, especially when it strikes near populated areas. But you also have to factor the in depth of the earthquake, and how well people have prepared. A big earthquake can have a low Mercalli value if it happens deep underground or if buildings have been properly supported”(3) → When I saw measuring, I recalled the nonsense in counting the test cases – which is very susceptible to the reification error and this is very, very dangerous. But we have something which tries to avoid the reification error and is based on events/activities: it is called Session Based Test Management.

But there is more, the fact that a big earthquake can have a low Mercalli, made me think about complexity and the fact that the relation between cause and effect is not linear. Populated areas also indicates complexity, because social systems are inherently complex. This means, for testing, that the approach is more informal (it’s about trying/probing, then make sense of it, then respond/report the possible problems which might occur), not a formal one – when thinking of testing, and more specifically the checking dimension, maybe here mutation checking makes sense.

There is also another implicit dimension here, which is the place where the earthquake happened. Even if it is a small earthquake but it happens in the ocean, it can generate a tsunami. If I relate this to testing, it makes me think of the different coverage areas like structure, platform, function, operations, data, time, interfaces(9), hazard.(thx Ionut Albescu)

● “Danger after the quake: The danger doesn’t stop once the ground has stop shaking. Fires, landslides and even liquefaction can all cause damage and loss of life…Scientists and engineers have developed ways to deal with these dangers through defenses, warning systems and building design. But even with the best plans in place some communities can still be caught off guard”(3) → How many developers, testers, scrum masters,… think that maybe a person will be fired because he/she is not working fast enough with our software product? How will a developer sleep at night when his/her code caused, even indirectly a death, or a bankruptcy? There are consequences, but a lot of people don’t get the fact that they must assume also unintended consequences, which were triggered by what their product does. How will they deal with that?

They have developed defences, but it’s interesting that they speak about them with terms like: tools (4), models(5). We, in IT, use a lot the word “automation”…

When they speak about plans it’s very serious because it’s about people’s lives. They are not using some tools/techniques as a plan, they are guided by the reality of the situation. What a test plan means, for a lot of people: automation at the unit level, integration and maybe acceptance(BDD). And they add exploratory, although they are not able to articulate what it means and how to show/do it in a professional way → this is not a test plan.

What if aftershock are the equivalent of hotfixes? And we have a flow like this:

1. A bug appears

2.A quick hotfix is made, but in a hurry

3. As a result, that hotfix can cause undesired problems(maybe inadvertently), because of the chaos created by the initial bug.

The last sentence made me think at the japanese word “hansei”. Because even though those scientists built/are building, defense and warning mechanisms(kaizen), the things can still go wrong and this is sadness/regret.(6)

● “After the earthquake, responding to disaster: earthquakes and tsunamis can destroy home and buildings, transforming lives. The hours and days that follow the disastrous events can be vital for saving anyone who has been trapped as a result….As people come to terms with the destruction they can start with the process of building resilience – changing the way they live and act to deal with the risk of an earthquake in the future. This can leave them better prepared for future earthquakes“(3) → The keyword here is resilience, but a lot of IT people want robustness. We, in IT, have chosen the wrong metaphor. Rapid Software Testing(a Context Driven methodology) is fully aware of that, that’s why it’s so different from “Factory Style” testing(7), because it sees the context as an ecology, not as a factory(8).

Conclusion: Read James Bach’s post, then read the text from the museum again. I hope you will find it as useful as it was for me.

(1) James Bach, “Round Earth Test Strategy”, http://www.satisfice.com/blog/archives/4947

(2) https://rapid-software-testing.com/

(4) “Tectonic hazards/Earthquake engineering“, https://en.m.wikiversity.org/wiki/Tectonic_hazards/Earthquake_engineering

(5) “Improving defence against earthquakes and tsunamis”, https://www.ucl.ac.uk/news/2017/mar/improving-defence-against-earthquakes-and-tsunamis

(6) James Coplien, Interview, www.infoq.com – in this interview he explained scrum and these 2 japanese words, among other things – I can’t find the text, as link, but I have it as text, tough, in my personal archive.

(7) James Bach, Michael Bolton, “RST Appendices”, http://www.satisfice.com/rst-appendices.pdf – pages 3-6

(8) Alicia Juarrero, “Safe-Fail, NOT Fail-Safe”, https://vimeo.com/95646156

(9) James Bach, Michael Bolton, SFDIPOT, http://www.satisfice.com/rst.pdf