Software Testing - Scripted Testing
CONTEXT - I am responsible at my organisation for reviewing test scripts for a software product in the Healthcare Arena. I have noticed a lot of repetition in the scripts.

Michael Bolton's Answer

I see a problem with repetition, but I also see big problems with scripts.

  • They prescribe the activities that a tester will perform to some degree of precision. If the tester wouldn't know what to do otherwise, or might know what to do and forget, the script might be helpful.
  • They provide some sense of reproducibility, so that if the test is interesting in some way, we can repeat the steps that it prescribes.
  • If you can demonstrate that they were followed, you can demonstrate that something happened.
  • They may provide a set of steps for developers to follow, such that the developers can test the code before releasing it to testing (this never happens, but we can dream).
  • The script might help to identify a pass/fail decision rule.
  • A set of scripts might help to co-ordinate the activities of multiple people.
  • A sufficiently simple set of actions, with a sufficiently clear decision rule, can be automated. The automation can then be programmed to incorporate a good deal of variation at high speed.

Here are some bad things about scripts:

  • They're very expensive to produce.
  • They typically take an experienced and skilled tester away from the testing task an into a documentation task.
  • Only a fraction of what an experienced and skilled tester thinks and does can be captured in a script.
  • Scripts tend to be produced at the beginning of the project, when we know less about the product than we'll ever know.
  • They go out of date quickly and easily; when the program changes, the script(s) have to change too.
  • The precision that they provide is often illusory. For example: "1. Start the program." How many ways are there to do that?--and that's just the first line.
  • While providing step-by-step instructions to follow, they tend not to provide a motivation for the test. In many ways, it's the motivation that's the important part.
  • By focusing the tester on specific observations, they tend to promote inattentional blindness, which is a well-known phenomenon in cognitive psychology. If you're paying close attention to something, you tend not to be open to other observations. That is, a test tends to orient the tester towards answering the question, "Did this assertion pass or fail?" rather than the much more important question, "Is there a problem here?"
  • They tend to be oblivious to the idea that testers really can think for themselves.
  • They constrain the variability that allows us to cover more of the program.
  • Like use cases themselves, they tend to be straightforward, single, happy paths through a program or task, where real users do all kinds of things that aren't like that. People change their minds, branch and backtrack, forget certain steps (and experience errors and error messages), do things out of order, pause and go for coffee...
  • If the program behaves in an unexpected way at some point earlier in the script, steps that follow may be invalidated.
  • They promote repetition, where variation might be a better idea if the object of the exercise is to reveal new bugs.
  • They tend to inhibit the development of skill, where supervision, mentoring, and rapid feedback tend to be more conducive to learning.
  • It's relatively easy for a tester to say, "Yeah, I did that" while pointing to a script, where a set of notes taken on the fly is somewhat harder to forge.
  • When used to guide humans, they tend to automate the behaviour of humans--but people make rotten machines. If you have a task that's simple enough to be expressed in a script, maybe it can be run (with greater speed, precision, variation, and evaluation) by a machine.

Vladimir's Answer

Scripts are a wise choice if

  • You think that the way testing is performed is as important as the checkpoint claimed in the test.
  • You believe that review of test cases is not just a formal process but a tool that helps removing badness.
  • You want to learn from each other in how to build effective tests.
  • You rely on building an effective organization when a couple of experts may supply with scripts 10-20 novices.
  • You work on an evolving product and plan on using same tests often for the regression testing purpose.
  • You have an automation engineer working next to the manual tester.
  • You work to the plans and want to have reliable estimates before the work is done, not after.

If you are working in a [somewhat] stable environment then scripting is not a waste of time but a thing that helps keeping the process (and the result) under control. Thinking ahead is not always a bad thing.

Recent Updates
Flash Objects and Selenium
Continuous Integration
Selenium Workshop
Delicious Testing
Pattern for automated testing of web applications
Exploratory Testing
 
Read More
Accessibility API Testing Article Backword BigBang Blackbox Blog Bottomup Boundary CaseStudies Certification DefectReport DistanceTest Equivalence FitNesse Geeks Graybox Guerrilla Testing Tips GUI HTA Humor Hybrid Internationalization Installation Integration Is it done? JUnit Measurement Mercury Quality Centre News One CPU better than two Patent PatternForAutomation Performace Checklist Rational Test Suite Regression Requirement Verification Research Rational Functional Tester Security Selenium Selenium Workshop SilkTest System Testing Templates TestComplete Tools Testing Types Testing Tools In News Testing Terms In News Testometer Test Plan TG Tips For Automation Top Down Integration Trait UAT UI Testing CheckList Unit Testing Usability VMWare Web Application Security Web Application Testing Checklist Whitebox Testing
Disclaimer  |  Privacy Policy  |  g e e k AT T e s t i n g G e e k DOT c o m
© Copyright 2008, www.TestingGeek.com