|
|
Alan
James and Ward: You both
make my point precisely.
First of all James: Of
course we all use our own
internal processes every day
and when we do something
wrong we sometimes
understand what we did and
why we did it so that we can
avoid doing it again.
Take Ward's "a + b"
example. My assumption here
is that in some code,
somewhere, this particular
code failed. Once we know
that, we know that the
person who injected that
code into the implementation
did it for some reason.
Part of that person's
internal process. Not only
that, that coder was
following a set of rules
that that person learned
somewhere as the right set
of rules to learn. (i.e.,
it is always safe to add to
values of currency with
"+"). But somewhere that
person's internal process
broke down and a mistake was
made. So how can we avoid
that particular mistake as a
general practice? First of
all, we'll need to know who
did it. (Not likely.)
Next, we'll need to find out
why that person made that
mistake (Not likely.) Then
we would need to figure out
how to establish a rule (or
to provide training for an
existing rule) so that our
entire organization is
unlikely to make that
mistake again.
|
These problems are buried in
the software development
culture. There are some
very good efforts to change
that culture (see the list
of XP rules; my favorite is
YAGNI, violation of which is
a major source of errors),
and to provide a uniform set
of rules that everyone can
follow. We can argue
quality, but I know of no
one who respects the quality
of the software they are
using (to their knowledge).
One universal rule of
engineering practice: Sign
your work.
When the developers at
Microsoft tried doing this,
what happened and why?
There seems to be a lot of
disinformation about the
nature of
Six
Sigma. For instance
it is not about statistical
control of inputs to ensure
proper outputs but rather it
is about:
What defects appear in the
product?
How did they get there?
What can we do to prevent
that in the future?
How can we ensure that the
problem doesn't come back?
This is pretty much
Six
Sigma in a nutshell.
But I agree with James, it
ain't going to happen, but
like the TV commercial where
the guy asks about all his
women friends calling at the
same time and taking down
the phone system: "So it IS
a possibility!"
+++++++++++++++++++++++++++++++++++++++++++++++++++
James
<<First
of all James: Of course we
all use our own internal
processes every day and when
we do something wrong we
sometimes understand what we
did and why we did it so
that we can avoid doing it
again.>>
Right. And we also can talk
to each other and figure out
what went wrong. The myth
that you are helpless as a
babe if you haven't turned
yourself into an industrial
robot and your process into
some facsimile of an
assembly line is merely
something promoted by
unscrupulous, or perhaps
incompetent consultants. I
am simultaneously
re-learning how to fly and
how to sail, these days. In
both processes, my
instructors observe the
end-products of my internal
cognitive processes and
suggests ways that I can do
better. The goal is not for
me to have an identical
process to theirs, it's for
me to be able to solve
certain problems, reliably,
in a variety of
circumstances.
<<Then we
would need to figure out how
to establish a rule (or to
provide training for an
existing rule) so that our
entire organization is
unlikely to make that
mistake again.>>
This assumes that
"establishing a rule" is a
way to solve a problem.
(Sometimes that is the case,
but there are many
exceptions. Sometimes the
problem relates to the way a
basket of variables, rules,
or people interact.
Sometimes that's resolved by
training, by research, by
practicing, or by changing
the goal of the system in
question. Some problems go
away naturally. In my
experience, "establishing a
rule" often carries its own
new problems, including
creating resentment,
creating an increasing
burden of rules to learn,
the design error associated
with creating bad rules, and
subtly changing the
relationship between the
practitioner and the
practice. If the rule
designer is overreacting to
a probabilistic phenomena,
he may find himself chasing
regression effects and
control lag instead of
letting the system settle
down. This is known as
"pilot-induced
oscillation".)
This also assumes that a
rule established to solve a
particular problem will be
embraced properly by people
in another part of the
organization who may be
dealing with a different
context. Besides all
that, you were just arguing
that a personal process is
impossible to analyze. While
I disagree that it's
impossible, it is certainly
challenging, at times, to
figure out what the real
process is, because people
don't necessarily tell you
what it is, or have a clear
enough idea what it is.
Therefore, establishing a
new rule may be a rash
maneuver based on a false
understanding of the
problem.
<<These
problems are buried in the
software development
culture. There are some
very good efforts to change
that culture (see the list
of XP rules; my favorite is
YAGNI, violation of which is
a major source of errors),
and to provide a uniform set
of rules that everyone can
follow. We can argue
quality, but I know of no
one who respects the quality
of the software they are
using (to their
knowledge).>> I
would de-emphasize rules and
substitute skills,
heuristics, and intent.
Having said that, agreements
and protocols do play a
non-trivial role.
<<What
defects appear in the
product?
How did they get there?
What can we do to prevent
that in the future?
How can we ensure that the
problem doesn't come back?>>
I like that part of
it. But if that were REALLY
what it was all about, it
wouldn't be called
Six
Sigma. It would be
called engineering.
+++++++++++++++++++++++++++++++++++++++++++++++++++
David
Alan -- You say
There seems to be a lot
of disinformation about
the nature of
Six
Sigma. For
instance it is not about
statistical control of
inputs to ensure proper
outputs but rather it is
about:
What defects appear in
the product?
How did they get there?
What can we do to
prevent that in the
future?
How can we ensure that
the problem doesn't come
back?
This is pretty much
Six
Sigma in a
nutshell.
I am not sure I agree
with that. That is
process improvement in a
nutshell. While it is
true to say that
Six
Sigma seeks to
improve processes, to
say that that is
Six
Sigma in a
nutshell strikes me as
about the same as saying
Agile is Waterfall in a
nutshell...since they
both are SDLC
methodologies. Just
like Agile and Waterfall
are not the same,
neither are all process
improvement
methodologies the same.
Six
Sigma has as it's
very namesake a
mathematical reference
that is meaningless
(except as a marketing
buzzword) without at
least some study of
statistics, and
contextually meaningless
withouot some study and
understanding of
statistical process
control. Not process
improvement...process
control. Because that
IS what
Six
Sigma is all
about...process
improvement through
process control. While
Six
Sigma (tm) may
have morphed into many
things, I think if you
look at the genesis of
where it came from, you
would agree with me. I
can't see how any
six
sigma project, or
description of
six
sigma as a
process, that doesn't
reference at the very
least Design Of
Experiments, Null
Hypothesis, FMEA, Chi
Square Tests,
Statistically
Significant Differences,
Confidence Intervals,
Control Charts, CTQs,
and F-Tests is really
about
six
sigma at all.
Process improvement,
sure...but not
six
sigma.
+++++++++++++++++++++++++++++++++++++++++++++++++++
Michael Bolton
>BTW, I tried to copy
Ward's comments into
here and I couldn't do
it. I'm using
Thunderbird. Is that a
bug?
It bugs ME.
I'm not using
Thunderbird; I'm using
Outlook. There are
three workarounds that
work for me; they may
for you.
I find that if I
double-click on the text
box that encloses the
message, that activates
the text box; then I can
start copying text.
If I change the format
of the message from HTML
to plain text, I can cut
and paste to my heart's
content.
I believe that there's a
setting in Yahoo Groups,
on a per-group basis,
that allows you to have
all messages sent to you
as plain text.
Now, my question is:
what would
Six
Sigma suggest
that we do with this
problem?
+++++++++++++++++++++++++++++++++++++++++++++++++++
Alan
The first step in the
Six
Sigma process as I
learned it is "Define".
Just what is the problem,
anyway? The starting place
for me is the symptom of
what I perceive to be a
bug. I selected a section
of the display. It was
highlighted just like any
other time I wanted to do a
copy/paste operation. I
tried Alt->Edit->Copy.
Nothing bad seemed to
happen. No warning that I
haven't really copied. (No
"Garbage In, Apology Out",
and if selecting across a
portion of two text boxes
cannot be copied, or
whatever the problem is, I
expect to be told that I
can't do what I am trying to
do. Is that nuts?) When I
subsequently did a paste;
nothing happened. What did
I do wrong? (I think
Hendrickson pointed out that
that is the first human
response to a bug.) I tried
CTRL+c. Nothing I know
(that's easy) worked. In
any case, having decided
that that is a bug,
or in reality, the symptom
of a bug, somewhere there is
code doing the wrong thing.
Maybe in this case it would
be the failure to notify the
user that this particular
selection cannot be copied
or that this string cannot
be selected. Let's assume
the former. Now, in
Six
Sigma, there is a
problem: defining an
"opportunity." There is a
lot of flexibility in this
area requiring a certain
creativity. Indeed we are
not necessarily counting
apples and oranges but we do
need to select something we
can get a real handle on.
In this example, perhaps we
should select the nature of
this bug: "Notification of
an illegal user operation."
Please keep in mind that
this is only the first step,
and later steps may prove
our problem definition to be
faulty or inadequate or
overly difficult in some
way.
Then in the next step,
"Measure", we would have to
count all of the times that
our sample set should notify
of illegality and the number
of times that it does not.
This ratio is our defect
rate that we would like to
drive to zero, but to be
more practical, let's set
the limit to a nearly
impossible task, 3.4 defects per 1,000,000 opportunities. (Even more
realistically, let's simply
try to reduce the error rate
by 80% on our first attempt
and 50% thereafter until we
achieve a defect rate of
0.034 per myriad (‱),
one for which there is a
meaningful name by which
this entire process may be
identified). As in any
process, there is an art to
properly applying it.
Maybe, in order to make
"Measure" easier, we simply
want to define an
"opportunity" as a function
calling a function that
returns an error code
wherein the calling function
does not examine or
otherwise utilize the
returned error code. Now I
think that might prove to be
a very useful quality
statistic. As is often the
case in
Six
Sigma, the problem we
end up solving may not even
be the one we originally set
out to solve. But it is
still a valid quality
improvement. A fundamental
assumption of
Six
Sigma is that defects
cost real money and learning
to avoid making the same
mistakes produces tax free
income in the form of money
that does not need to be
spent (cost avoidance).
Done properly, the cost of
quality is negative, not
even counting the customer
satisfaction benefits.
Other cost avoidance
techniques: "Let the
customer test it for us."
"Let's don't fix that bug."
And yes, there is a lot of
complicated statistics
involved, but I don't need
to know why a car works in
order to drive one (but it
certainly helps,
particularly when it
doesn't, work, that is).
Companies implementing
Six
Sigma have on-call
Statistics Help Desks.
Then the next step,
"Analyze", is to find all
the reasons that we can for
the failures; root cause
analysis. We can use
Pareto, or other techniques,
to determine how many and
which of those causes we
need to fix in order to
achieve our quality
improvement goal. Then we
need to put in place fixes
for those particular
causes. For instance, maybe
the software designer didn't
know that he or she was
supposed to notify the user
when the user attempted to
do something the software
couldn't do. Maybe training
could fix that. Maybe the
code was there, but didn't
do its job correctly or
warned in a way the user
didn't notice, like
commanding a sound when the
sound is turned off. There
are lots of ways of doing
this poorly (as Ward pointed
out). Or maybe we simply
need to implant the rule:
"Every function must return
an error indicator and every
function must process every
error indicator returned by
the functions it calls."
Implementing the autonomic
requirements makes this rule
even more specific in terms
of how returned errors must be processed. Having a standard
way of doing this makes
implementation and auditing
much easier. (What?
Audit MY work?)
In any case, the next step
is "Implement" where all of
the selected fixes need to
be put into place.
The final step, "Control",
is management's method of
maintaining the fix which
usually involves a policy
that "Measurement" and
"Analysis" continue
periodically to ensure that
the quality goal continues
to be met with an action
plan when it does not (like
adding training for new
hires).
Now if your marketing
paradigm is to charge
customers for fixing bugs,
avoiding the creation of
bugs is not your goal and
Six
Sigma is not for you.
+++++++++++++++++++++++++++++++++++++++++++++++++++
Shrini
Fundamental theme in
six
sigma is to
de-humanize the whole thing
and substitute human beings
and their behavior with
outrageously simplified
mathematical models – person
X commits Y dozens of
mistakes here, person A, can
review and eliminate YY
units of waster there etc.
All human beings are
considered equivalent – like
robots. This might work in
manufacturing industry but
in software?
As Dr Kaner points out it,
it is this mathematical
model, that restricts the
utility value of
six
sigma methodology.
Alan gave a very detailed
account of DMAIC (define,
measure, analyses, Improve
and control) taking a bug in
yahoo group mails as
example. That is typical way
six
sigma oriented
process improvement is done.
The trouble starts when such
things are generalized and
glorified. The trouble
starts when people believe
that handing the folks a
check list of things to do
and not to do amounts to
improving the process.
Checking a bug database of
an application, analyzing
the reasons for those
defects, making a checklist
of things to avoid during
subsequent development (like
– during code review,
specially check for
un-initialized variables) is
a very simple and straight
forward way of "learning
from mistakes".
Instead of running a costly
six
sigma project to
improve code quality and
reduced re-work. Ask the
tester to check all old bugs
and learn from them (get
more ideas, heuristics to
test). Then pair that tester
(or group of testers) with
developers – the result will
be more "visible" and long
lasting.
Six
sigma, I believe has
two main methodologies -
DMAIC (process improvement)
and DFSS (Design for
six
sigma). When people
say "six
sigma" - they
typically talk about DMAIC
version that is about
"process quality" and
"process improvement".
DFSS -
instead is believed to
be suited for new product
design and solution.
Tools and
techniques like Quality
function deployment (QFD),
FMEA, Design Of Experiments,
Taguchi methods are known to
be used in relation with
DFSS.
Any views and
experiences of application
DFSS in software design and
testing?
+++++++++++++++++++++++++++++++++++++++++++++++++++
Dr. Cem Kaner
In the manufacturing
world, as originally
conceived and implemented, I
think
Six
Sigma is about
thoughtful continuous
quality improvement. I've
read a lot of Deming, and I
think the original
Six
Sigma philosophy--as
I understand some early
descriptions and many
discussions with Pat
Bond--was compatible with
most of Deming's key ideas.
But
Six
Sigma creates a
measurement structure.
Deming strongly recommends
driving quality improvement
through a measurement
structure. The name, "Six
Sigma" is
representative of that
structure. "Sigma"
(the Greek letter) is used
as the symbol for a standard
deviation. If a variable is
normally distributed, then
if we sample from that
variable a lot of times,
99.7% of the sampled events
will lie between -3
sigma and +3
sigma of the mean.
http://en.wikipedia.org/wiki/Normal_distribution
Manufacturing errors (and
measurement errors) are
frequently assumed to be
normally distributed. Think
of making nails. You want a
nail of a certain length and
it is defective if it is
more than x% longer or
shorter than the desired
length. So you measure the
number of defects. If that
is more than 0.3% of the
total number made, change
how you make the nails and
measure again. Track the
defect rates over time and
over the conditions you
vary, improving your process
bit by bit until you get to
0.3%. In statistical terms,
under the model, you have
narrowed the variability of
your process to the point
that errors lie outside the
6-sigma
range around your process's
mean value.
Several statistically
interesting problems arise
in applying this. The most
interesting, perhaps, is the
design-of-experiments
challenge. As you make (what
you hope are) improvements,
you have to create
circumstances that allow you
to detect improvement. This
requires you to compare
something now with something
before. And if you are
making more than one change
at a time--if variables
interact (as they often do),
then you have to
cost-effectively vary what
you do in ways that helps
you understand the practical
impact of the relationship.
Cost-effective design of
experiments in real-life
situations is very
challenging.
Statistical process control
is not fundamentally
dehumanizing. Deming is one
of the great humanists of
our field (the broad field
of quality improvement) but
he is also one of the core
advocates of statistical
process control.
The critical issue that I
have with
Six
Sigma as it applies
to software is that the
assumptions underlying
statistical process control
do not apply to what I see
as the critical questions or
situations for software. The
underlying error model, in
my view, is incompatibly
different.
The other key issue that I
have with
Six
Sigma is its
transformation to Sick
Sigma.
I was a big fan of Total
Quality Management years
ago. Back when Feigenbaum
published the book that
defined TQM, it was a
strident vision of quality
improved driven by data.
Cross-functional
collaboration was important
because it was more
efficient. (What I mean with
"efficient" here is not just
number of widgets per hour
but what it actually costs
to make all those widgets if
you include all associated
costs including failure
costs, support costs, design
maintenance costs, etc.)
More generally, a humanistic
approach to management fits
with TQM because people do
better work when they feel
secure, understand the
context of their work,
understand the processes
that lead up to the work
they do and the ways that
their work products will
impact others down the line,
etc.
TQM was a very interesting
approach to management that
required a lot of critical
thinking and a lot of
empirical checking of
assumptions.
But then the training
companies and the
consultants got to TQM and
watered it down until it
turned into pablum that
anyone could think they
understood in a few minutes.
It turned into empty
slogans, stupid metrics, and
group hugs (aka quality
circles). (Yes, some of you
old Quality Geezers will
remember "real" quality
circles that surfaced real
issues and yielded real root
cause analysis, and so on.
But after the Pablum Process
is applied, you get group
hugs.)
Even more than TQM,
Six
Sigma is a quality
improvement process that is
driven by statistical
process control.
Sick
Sigma, in contrast,
is a heavyweight
consultant-bank-account
improvement process that is
driven by mass-marketable
lectures and easy-to-get
certificates.
I have nothing against
Six
Sigma. I just don't
know how to apply it to
software and haven't met
anyone who does. Ultimately,
that's an empirical question
and it is probably
overbroad. There probably
are some aspects of software
development and testing that
are appropriate for
application of statistical
process control. I just
haven't seen any convincing
examples, yet.
I have nothing against
continuous quality
improvement, or against
efforts to find underlying
causes of problems and deal
with them, or against
efforts to figure out
whether an attempted
improvement ultimately did
more good than harm. But
that general desire, which
is certainly at the core of
Six
Sigma, is at the core
of many approaches to
management. It was around
long before
Six
Sigma, and will be
around long after the phrase
"Six
Sigma" has been
discarded as unfashionable.
I have everything against
Sick
Sigma. Like so much
in our field, I think it is
an anti-productive sham.
Sadly, I think that most of
what I have seen in advocacy
of
Six
Sigma for software
has been Sick
Sigma.
<<< Previous
|
