|
|
CONTEXT - I am responsible
at my organisation for
reviewing test scripts for a
software product in the
Healthcare Arena. I have
noticed a lot of repetition
in the scripts.
|
Michael Bolton's Answer
I see
a problem with repetition,
but I also see big problems
with scripts.
-
They prescribe the
activities that a tester
will perform to some
degree of precision. If
the tester wouldn't know
what to do otherwise, or
might know what to do
and forget, the script
might be helpful.
-
They provide some sense
of reproducibility, so
that if the test is
interesting in some way,
we can repeat the steps
that it prescribes.
-
If you can demonstrate
that they were followed,
you can demonstrate that
something happened.
-
They may provide a set
of steps for developers
to follow, such that the
developers can test the
code before releasing it
to testing (this never
happens, but we can
dream).
-
The script might help to
identify a pass/fail
decision rule.
- A
set of scripts might
help to co-ordinate the
activities of multiple
people.
- A
sufficiently simple set
of actions, with a
sufficiently clear
decision rule, can be
automated. The
automation can then be
programmed to
incorporate a good deal
of variation at high
speed.
Here
are some bad things about
scripts:
-
They're very expensive
to produce.
-
They typically take an
experienced and skilled
tester away from the
testing task an into a
documentation task.
-
Only a fraction of what
an experienced and
skilled tester thinks
and does can be captured
in a script.
-
Scripts tend to be
produced at the
beginning of the
project, when we know
less about the product
than we'll ever know.
-
They go out of date
quickly and easily; when
the program changes, the
script(s) have to change
too.
-
The precision that they
provide is often
illusory. For example:
"1. Start the program."
How many ways are there
to do that?--and that's
just the first line.
-
While providing
step-by-step
instructions to follow,
they tend not to provide
a motivation for the
test. In many ways, it's
the motivation that's
the important part.
-
By focusing the tester
on specific
observations, they tend
to promote inattentional
blindness, which is a
well-known phenomenon in
cognitive psychology. If
you're paying close
attention to something,
you tend not to be open
to other observations.
That is, a test tends to
orient the tester
towards answering the
question, "Did this
assertion pass or fail?"
rather than the much
more important question,
"Is there a problem
here?"
-
They tend to be
oblivious to the idea
that testers really can
think for themselves.
-
They constrain the
variability that allows
us to cover more of the
program.
-
Like use cases
themselves, they tend to
be straightforward,
single, happy paths
through a program or
task, where real users
do all kinds of things
that aren't like that.
People change their
minds, branch and
backtrack, forget
certain steps (and
experience errors and
error messages), do
things out of order,
pause and go for
coffee...
-
If the program behaves
in an unexpected way at
some point earlier in
the script, steps that
follow may be
invalidated.
-
They promote repetition,
where variation might be
a better idea if the
object of the exercise
is to reveal new bugs.
-
They tend to inhibit the
development of skill,
where supervision,
mentoring, and rapid
feedback tend to be more
conducive to learning.
-
It's relatively easy for
a tester to say, "Yeah,
I did that" while
pointing to a script,
where a set of notes
taken on the fly is
somewhat harder to
forge.
-
When used to guide
humans, they tend to
automate the behaviour
of humans--but people
make rotten machines. If
you have a task that's
simple enough to be
expressed in a script,
maybe it can be run
(with greater speed,
precision, variation,
and evaluation) by a
machine.
Vladimir's Answer
Scripts are a wise choice if
-
You think that the way
testing is performed is
as important as the
checkpoint claimed in
the test.
-
You believe that review
of test cases is not
just a formal process
but a tool that helps
removing badness.
-
You want to learn from
each other in how to
build effective tests.
-
You rely on building an
effective organization
when a couple of experts
may supply with scripts
10-20 novices.
-
You work on an evolving
product and plan on
using same tests often
for the regression
testing purpose.
-
You have an automation
engineer working next to
the manual tester.
-
You work to the plans
and want to have
reliable estimates
before the work is done,
not after.
If you
are working in a [somewhat]
stable environment then
scripting is not a waste of
time but a thing that helps
keeping the process (and the
result) under control.
Thinking ahead is not always
a bad thing.
|