Category: Testing

In November 2017 I attended the Rapid Software Testing course held by James Bach. Now, this course has been renamed as Rapid Software Testing Explored.

When I was on the train, going to the training, I imagined what I would have written about the course in those days. But I did not do that. After the training, I decided not to write about it and wait for a good period of time, to see how it will influence me in real life. I think this is the most important thing in the end.

I think that the time to write about this course has arrived. I hope you will find it useful.

Premises

There were some things preceding the training which, I believe, are very important and worth mentioning:

– I knew about the work of James Bach and Michael Bolton. Before going to the training I have fully read their entire blogs, and also the book “Lessons Learned in Software Testing: A Context-Driven Approach” written by James Bach;

– I knew about the work of Jerry Weinberg and have read a dozen of his books, especially the book “Perfect Software: And Other Illusions about Testing” ;

– For 2 years, before going to this training, I tried to understand the testing envisioned by Jerry Weinberg, James Bach, Michael Bolton, Cem Kaner;

– I was a software developer and team lead;

– Going to this training involved also  personal money/cost contribution. I mention this because I am strongly influenced here by the “skin in the game” concept of Nicholas Taleb;

– I have read the rst.pdf file, which was the course support material..twice;

– I have searched the internet for testimonials regarding this course with enough details, but I did not found a lot of them.

Feelings

I remember the strange feeling I had when I entered the class room. James Bach was there at his desk, but I did not dare to say anything. I can say that I was overwhelmed. It was unreal for me to be there.

After the first 2 hours of course, I realized that I knew so little about testing…

Details about course

I will try to write about this course via 3 dimensions:

-structure: the flow of the course in those 3 days;

-key words: it’s interesting to describe a training via some keywords used in the course, which can be enumerated coherently in a minute or so. I think it can have a good impact;

-pearls of wisdom: these are insights from the training, words used by James Bach. You will see that each of the texts is within quotes and preceded and followed by “….”. This is because the text has a context(story, experience, people, time, history, feelings, exercise, place). Some of those phrases took even an hour or 30 minutes to be described by James Bach for us. So there is a risk, for the reader, not to decipher the real meaning/lesson/message, but I put them there in the hope that it will convince/entice you to go at his, and Michael’s, training and find out more;

Structure

The course was centered around exercises, experiments and interesting surprises. These exercises were not easy at all, it was hard stuff(even if at the beginning, sometimes, for us it felt easy, we found out that it needs to be analysed first and we needed to ask for more information). I saw damn good testers being tricked. But James warned us.

Since these exercises and challenges were hard, he spoke to us about mistakes, about his mistakes. He told us that we should not be scared of them and that we should learn from them.

These exercises were inspired from real life difficult situations that James Bach experienced.

Each exercise was followed by a lesson/message. This was the beauty. If someone would have told me in advance the same lesson, I would have not understood it as I should – I am sure of that 100%. But you know why? Because we struggled with it, because it had put us in a delicate situation…try to imagine social pressure or a hostile environment, it is not enough to read about it, you have to live it.

The explanations were full of metaphors, so the message can be easily understood.

The interaction with James was via the Socratic method.

James intention was to make us learn more rapidly the things he learned in a much slower way.

Although I have read in advance the pdf which  accompanied the course, I realized that I have missed A LOT of insights…

Key words

Initially, I was overwhelmed by all the details from the course and I did not knew how to describe it in a short and coherent way. But, shortly after, I realized that I can use some words used in the training. So here they are:

context based risk analysis, compatibility regression testing, sanity check, test strategy, business risk, normalization analysis(from therapy), leader and its attitude toward tester, thinking outside the box, parenting, biases, process of thinking, pattern(s) recognition, efficiency and relative efficiency, sense making, common sense questioning, interview for hiring, diminishing returns principle, fallacies, design tests, cope with complexity and uncertainty, scepticism, risk mitigation, quality, agile, think negatively, tacit test procedures, normalized test sessions, cognitive science, social dynamics, psychology, social skills, qualitative methods, social pressure, functional blindness, hostile environment, data types bugs, boundary testing, heuristics, perimeter assumptions, defocusing technique, factoring, randomness, heuristic test strategy model, professionalism in software testing, hazop, inattentional blindness, audit testing, penetration testing, anchoring bias, mental testing models, adversarial images(from AI) , statistical correlation, product coverage outline, oracles, lifeboat argument, explainability/confusion oracle, person whose opinion matters oracle, defocusing, diffusion limited aggregation, deep testing, exploratory and scripted testing.

Pearl’s of wisdom

As I have already said, it is important to mention that these phrases do not have context around them. This is intentional. It is an invitation, if you like or find it problematic or… to dig more, even to go to the course. So:

-“… Sanity checking means shallow test coverage not deep test coverage. For the purposes of identifying insanities. You are looking for, not just something that is a little bit wrong, you are looking for something that is totally broken. If it’s totally broken I want to know about it …”

-“… All testing problems have a context, and if you know that context you will be able to make a sensible decision about what to do next. And if you don’t know the critical context variables then you will suggest a ridiculous answer …”

-“…. Rapid Software Testing isn’t rapid because you are typing faster than other people. It’s about thinking: Do I really need to do this? Why do I need to do this? What things can I set aside? And what things do I really need to do?  …”

-“… What a good leader should say, knowing how delicate and emotional testing is: I want to know if there is a problem. Don’t worry about the ship date. We will slip it if you say we need to. That is not your concern. What your concern should be: Can I find problems in this?. Let me worry about …”

-“… When you teach a tester how to test you are teaching that person to see something, to see things that you have never seen before …”

-“… Testing is an exploration and experimentation  process …”

-“… A paradox of RST is that: RST tells you that you should do the right thing. Don’t do anything unnecessary, that will save you time. But how do you know what the right thing is? First you’re gonna have to do wrong things to find out what the right thing is…So the paradox is that in order to get rapid you usually have to start slow to climb the learning curve…”

-“… Testing is an intellectual activity …”

-“… Never say:I tested it and it works. That is always an untrue statement. The truth is you tested it and you haven’t seen it fail yet…I tested it and I know it can work… “

-“…Testing is not about assessing that the product is good enough, it’s about finding problems ….”

-“…Counting tests is pointless ….”

-“…Rapid Software Testing methodology which is a mindset and skillset, but it is not a set of rules, it’s not a set of templates….”

-“… New novice testers don’t need lists. They need experience. How do you give them experience? Sit with them down and test with them …”

-“… Here is what you never do: you never derive tests from requirements, just don’t say that…”

-“… It is not possible to write down a test …”

How it helped me

This course helped me realize/do a lot of things:

– I will do everything I can to have trainings with the best in the industry. With those people who create concepts. If this means paying from my own money, so be it. I prefer to wait a year or two, but to learn from the best out there;

– Online trainings are important and have their place, but they can never ever substitute a live training held by one of the top of the top of trainers;

– At least once per year I would do the Rapid Software Testing Applied training;

– As a programmer, this professional testing dimension influenced me a lot. I can no longer define myself as only a software developer and ignore the tester part in me;

– I understood how and why so much money is lost in IT with superficial testing and the “automation” trend which now is being marketed ;

– I understood how to be a better mentor;

– It helped me by giving me force to write about testing and to defend it;

– I realized that, now, in IT, a lot of concepts currently used are not understood: testing, agile, scrum, OOP, architecture…

– I was able to help/coordonate a student, an experienced developer, for a masters degree in IT regarding testing. It was the first time such a subject was presented in university and it really impressed. I was glad that, finally, the real face of testing was presented in an institution which prepares students for the IT industry. I hope that in the future I will be able to teach a course on testing, but I am not yet prepared, I need help. I think that if testing, the real one, would be taught in University it would help even more the spreading of what real testing is, which in the end will help the industry as well as the people in it.

Conclusion

I liked this training a lot. I never would have thought that it would influence me so much.

Recently, James Bach published a nice analogy/model regarding test strategy, named “Round Earth Test Strategy“(1)

Early in 2018, in January, James Bach was kind enough to share this analogy on a special channel RST( this channel was created especially for RST alumni; it’s an active and very very very useful group to be in – another very good reason to take RST class(2)).

I liked it. But, in that moment, and days/months after, I felt that I still miss some aha moments, about those things, which confirm in a way that a certain subject is internalized in myself somehow.

Museum

In the image of this blog post is the Natural History Museum in London. It was here where I had the aha moments I knew I was missing. Suddenly, in my head, I remembered James Bach’s round earth model. In there is a section named “Volcanoes and Earthquakes”. When I began to read what was written, I was amazed how the things described there can be used as an analogy, as James thought, for testing, but maybe also for problems/risks/bugs/tools/approaches from the IT field:

● “Earthquakes can happen without warning, causing death and destruction on a massive scale. When they strike we feel a sudden, violent shaking of the ground, but they are caused by slowly moving plates on Earth surface. As these plates move, pressure builds up until it finally gives away”(3) → In reading this text I remembered bugs. How certain bugs can create a lot of mess (urgent calls, staying late in the night, missing important moments). How those bugs appear suddenly, without notice. But what stroke me is the fact of slowness and violence of earthquakes.

Slowness in:

– how functionality is written → It’s not like the functionality is developed instantly. There are meetings, mails, writing code, searching through code, testing, etc.

– execution of the program –  managed memory leak, usually, takes time to be observed. But when it reaches a critical point, entire system crashes not only the programs.

The idea of moving plates made me think of integration testing.

● “Preparing for the worst; living with earthquakes: Scientists can’t predict exactly where and when an earthquake will strike but they know roughly which area around the world are at risk. It is vital for the people living in these areas to prepare for what may come and know what to do when it does. Without adequate preparation, earthquakes can cause huge suffering and destruction”(3) → It describes perfectly the role of testers and why they are searching for the worst, why they should think negatively. Testers, like these scientists, should understand that they cannot predict and also can’t be sure that the software is ok (just a black swan among thousands and thousands of white swans was enough to proof that there were not just white swans). Although they can’t predict or prove the correctness of a software program, they will use models to identify possible areas with problems guided by risks (For example, by looking at the source control metadata, weak areas within the source code can be discovered. ). Since we’re talking about people, the risks also have a psychological and sociological dimension. It is sure, problems will occur and maybe we should guide our testing also by the possible suffering we create, for the ones using our software.

● “Impact scale: There are different ways of measuring earthquakes. Unlike Richter scale, which measures magnitude of the shaking, the Mercalli scale measures the amount of damage caused – the loss of life and the damage of buildings. Generally speaking, the higher earthquake magnitude the greater the devastation, especially when it strikes near populated areas. But you also have to factor the in depth of the earthquake, and how well people have prepared. A big earthquake can have a low Mercalli value if it happens deep underground or if buildings have been properly supported”(3) → When I saw measuring, I recalled the nonsense in counting the test cases – which is very susceptible to the reification error and this is very, very dangerous. But we have something which tries to avoid the reification error and is based on events/activities: it is called Session Based Test Management.

But there is more, the fact that a big earthquake can have a low Mercalli, made me think about complexity and the fact that the relation between cause and effect is not linear. Populated areas also indicates complexity, because social systems are inherently complex. This means, for testing, that the approach is more informal (it’s about trying/probing, then make sense of it, then respond/report the possible problems which might occur), not a formal one – when thinking of testing, and more specifically the checking dimension,  maybe here mutation checking makes sense.

There is also another implicit dimension here, which is the place where the earthquake happened. Even if it is a small earthquake but it happens in the ocean, it can generate a tsunami. If I relate this to testing, it makes me think of the different coverage areas like structure, platform, function, operations, data, time, interfaces(9), hazard.(thx Ionut Albescu)

● “Danger after the quake: The  danger doesn’t stop once the ground has stop shaking. Fires, landslides and even liquefaction can all cause damage and loss of life…Scientists and engineers have developed ways to deal with these dangers through defenses, warning systems and building design. But even with the best plans in place some communities can still be caught off guard”(3) → How many developers, testers, scrum masters,… think that maybe a person will be fired because he/she is not working fast enough with our software product? How will a developer sleep at night when his/her code caused, even indirectly a death, or a bankruptcy? There are consequences, but a lot of people don’t get the fact that they must assume also unintended consequences, which were triggered by what their product does. How will they deal with that?

They have developed defences, but it’s interesting that they speak about them with terms like: tools (4), models(5). We, in IT, use a lot the word  “automation”…

When they speak about plans it’s very serious because it’s about people’s lives. They are not using some tools/techniques  as a plan, they are guided by the reality of the situation. What a test plan means, for a lot of people: automation at the unit level, integration and maybe acceptance(BDD). And they add exploratory, although they are not able to articulate what it means and how to show/do it in a professional way → this is not a test plan.

What if aftershock are the equivalent of hotfixes? And we have a flow like this:

1. A bug appears

2.A quick hotfix is made, but in a hurry

3. As a result, that hotfix can cause undesired problems(maybe inadvertently), because of the chaos created by the initial bug.

The last sentence made me think at the japanese word “hansei”. Because even though those scientists built/are building, defense and warning mechanisms(kaizen), the things can still go wrong and this is sadness/regret.(6)

● “After the earthquake, responding to disaster: earthquakes and tsunamis can destroy home and buildings, transforming lives. The hours and days  that follow the disastrous events can be vital for saving anyone who has been trapped as a result….As people come to terms with the destruction they can start with the process of building resilience – changing the way they live and act to deal with the risk of an earthquake in the future. This can leave them better prepared for future earthquakes“(3) → The keyword here is resilience, but a lot of IT people want robustness. We, in IT, have chosen the wrong metaphor. Rapid Software Testing(a Context Driven methodology) is fully aware of that, that’s why it’s so different from “Factory Style” testing(7), because it sees the context as an ecology, not as a factory(8).

Conclusion: Read James Bach’s post, then read the text from the museum again. I hope you will find it as useful as it was for me.

---

(1) James Bach, “Round Earth Test Strategy”, http://www.satisfice.com/blog/archives/4947

(2) https://rapid-software-testing.com/

(3) London Natural History Museum, Earth Galleries, text used with permission under license Non-Commercial Government Licence, Copyright © The Trustees of the Natural History Museum, London

(4) “Tectonic hazards/Earthquake engineering“, https://en.m.wikiversity.org/wiki/Tectonic_hazards/Earthquake_engineering

(5) “Improving defence against earthquakes and tsunamis”, https://www.ucl.ac.uk/news/2017/mar/improving-defence-against-earthquakes-and-tsunamis

(6) James Coplien, Interview, www.infoq.com – in this interview he explained scrum and these 2 japanese words, among other things – I can’t find the text, as link, but I have it as text, tough, in my personal archive.

(7) James Bach, Michael Bolton, “RST Appendices”, http://www.satisfice.com/rst-appendices.pdf – pages 3-6

(8) Alicia Juarrero, “Safe-Fail, NOT Fail-Safe”, https://vimeo.com/95646156

(9) James Bach, Michael Bolton, SFDIPOT, http://www.satisfice.com/rst.pdf