CAST 2012 Enforced Randomization - Increasing the value of test automation

By on July 22, 2012 Comments

Last week I was at San Jose for CAST 2012 to give talks on two interesting topics. My first talk was around the limitations of acceptance criteria and second talk was focused on how we can increase value of automation by introducing randomization where it make sense.

During my conversation with participants around randomization, many people suggested that I should write a blog about this - and so here it goes.

Pretty much all the test automation projects have four main elements.

  • Navigation - How do you reach to the screen / page which to test
  • Data - How many variations of data are possible
  • Sequence - In which sequence data can be used on the page
  • Operation - What are the different ways in which operations can be performed.

For example, if we take the example of registration page for any popular site such as Twitter, this is how these four elements can be used

  • People can reach on registration page in many different ways - from header, footer, promotional pages etc.
  • On the registration page, user can use valid first name, password and email address.
  • Registration form can be filled in any sequence - top to bottom, bottom to top, random order.
  • And form data can be saved in many different ways - with mouse, keyboard or direct post.

With test automation, it is possible to build randomization in all of these activities and ensure that test runs are identical to each other. Randomization have potential to increase probability of uncovering interesting information. Testing in my opinion is a sampling exercise and with randomization, we can increase our sample size substantially.

Since this talk was recorded, pretty much everything I would like to tell / explain is already explained in the following video - so I suggest you watch this video.

By the way, if you are interested in model based testing - you might find these posts from Harry Robinson interesting.

Key point I wanted to make in the talk is - Manual testing does not guarantee variability and automated testing does not have to be repetitive. I finished my talk with bunch of questions which I will repeat here as well - Why randomization is not main stream in test automation projects? Is it our bias that test should always fail in predictable way? Is it barrier in our mind that randomization in test automation is complex?

I will let you decide. Please leave your comments if you have used randomization or have any questions on my talk or this topic.

blog comments powered by Disqus
Finished reading? Browse all posts »