Mc Donald The Practical Guide To Defect Prevention 2008

User Manual:

Open the PDF directly: View PDF PDF.
Page Count: 478 [warning: Documents this large are best viewed by clicking the View PDF Link!]

BEST PRACTICES
Marc McDonald, Robert Musson, and Ross Smit
t
h
M
o
M
Do
be
and
Techniques to Meet the Demand f
for More-
qu
Te
T
T
Te
e
es
he
r M
Reliable Software
So
elia
e S
S
T
HE PRA
C
TI
C
AL
G
UIDE T
O
DEFE
CT
PREVENTIO
DEFECT
PREVENTION
THE PRACTICAL GUIDE TO
PUBLISHED BY
Microsoft Press
A Division of Microsoft Corporation
One Microsoft Way
Redmond, Washington 98052-6399
Copyright © 2008 by Marc McDonald, Robert Musson, Ross Smith, Dan Bean, David Catlett,
Lori Ada Kilty, and Joshua Williams
All rights reserved. No part of the contents of this book may be reproduced or transmitted in any form
or by any means without the written permission of the publisher.
Library of Congress Control Number: 2006931463
Printed and bound in the United States of America.
1 2 3 4 5 6 7 8 9 QWE 2 1 0 9 8 7
Distributed in Canada by H.B. Fenn and Company Ltd.
A CIP catalogue record for this book is available from the British Library.
Microsoft Press books are available through booksellers and distributors worldwide. For further infor-
mation about international editions, contact your local Microsoft Corporation office or contact Microsoft
Press International directly at fax (425) 936-7329. Visit our Web site at www.microsoft.com/mspress.
Send comments to mspinput@microsoft.com.
Microsoft, Microsoft Press, ActiveX, Encarta, Excel, Internet Explorer, MSDN, MS-DOS, MSN, SQL
Server, Visio, Visual Studio, Viva Piñata, Windows, Windows Server, Windows Vista, Xbox, and XNA
are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other
countries. Other product and company names mentioned herein may be the trademarks of their
respective owners.
The example companies, organizations, products, domain names, e-mail addresses, logos, people, places,
and events depicted herein are fictitious. No association with any real company, organization, product,
domain name, e-mail address, logo, person, place, or event is intended or should be inferred.
7KLVERRNH[SUHVVHVWKHDXWKRU¶VYLHZVDQGRSLQLRQV7KHLQIRUPDWLRQFRQWDLQHGLQWKLVERok is provided
without any express, statutory, or implied warranties. Neither the authors, Microsoft Corporation, nor its
resellers, or distributors will be held liable for any damages caused or alleged to be caused either directly
or indirectly by this book.
Acquisitions Editor: Ben Ryan
Project Editor: Lynn Finnel
Editorial Production: Waypoint Press
Peer Reviewer: Ed Triou
Illustration by John Hersey
Body Part No. X12-41751
To my incredible wife Lynsey and our four awesome kids,
Maddy, Emma, Clara, and Roo.
—Ross Smith
To my father, who gave me my love of technology, problem solving,
and a need to find the root causes of defects.
—Marc McDonald
To the members of the Windows Defect Prevention team
who make every day enjoyable.
—Robert Musson
To my wife DeAnne for all her love and support.
—Dan Bean
To my wife Erika and our amazing kids, Josh, Emilie, and Sara,
who bring me profound joy. To my parents and brother, who managed
to navigate me safely through my growing-up years so that I could
retell our adventures together on the river.
—David Catlett
For Brett, Juliana, and Brenna.
—Lori Ada Kilty
To my loving and supportive family, Melanie, Madeline,
Hannah, Sophie, and Claire.
—Joshua Williams
v
Contents at Glance
Part I Introduction to Defect Prevention
1 Defect Prevention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Defect Prevention Frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3 The Economics of Defect Prevention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Part II Defect Detection Techniques
4 Quality and the Development Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5 Using Productivity Games to Prevent Defects. . . . . . . . . . . . . . . . . . . . . . 79
6 Improving the Testability of Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Part III Defect Analysis Techniques
7 Software Measurement and Metrics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
8 Risk Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
9 Using Simulation and Modeling for Organizational Innovation . . . . . 155
10 Defect Taxonomies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
11 Root Cause Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
Part IV Defect Prevention Techniques
12 Adopting Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
13 FMEA, FTA, and Failure Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
14 Prevention Tab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
Part V A Culture of Prevention
15 Scenario Voting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
16 Creating a Quality Culture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355
17 Moving Quality Upstream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367
vi Contents at a Glance
18 Rewards, Motivation, and Incentives . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
19 Knowledge Management and Communication . . . . . . . . . . . . . . . . . . . 401
20 Pulling It All Together . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
vii
Table of Contents
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi
Part I Introduction to Defect Prevention
1 Defect Prevention
What Is a Software Defect? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Making High-Quality Software the Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Understanding Why Software Defects Occur . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Analyzing the Bases of Human Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
What Can Be Done? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Using Detection, Analysis, and Prevention Techniques . . . . . . . . . . . . . . . . . . . . . 9
What Is Different in a Prevention Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Using Defect Prevention Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Defect Detection Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Defect Analysis Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Defect Prevention Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Choosing Quality Improvement Techniques. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Factors to Consider . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Selecting a Strategy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Organizational Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Moving Quality Upstream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Learning from Mistakes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Investing for the Future . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Microsoft is interested in hearing your feedback so we can continually improve our books and learning
Microsof
resources for you. To participate in a brief online survey, please visit:
resourc
www.microsoft.com/learning/booksurvey/
What do you think of this book? We want to hear from you!
What
viii Table of Contents
2 Defect Prevention Frameworks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Examining a Sample Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Proposing a Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Defect Prevention Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Capability Maturity Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Capability Maturity Model Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Malcolm Baldrige Framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
ISO Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Other Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Comparing the Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Choosing and Using a Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Organizational Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3 The Economics of Defect Prevention. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Preventing Defects Is Good for Business. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Economic Theory and the Value of Defect Prevention . . . . . . . . . . . . . . . . . . . . 39
Profitability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Applying Marginal Cost Analysis to Software Development . . . . . . . . . . . . . . . . . . . . 41
Estimating Costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Determining the Returns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Part II Defect Detection Techniques
4 Quality and the Development Process . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
What Is Software Quality? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Development Methodology and Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
The Myth of Complete Testability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
Current Testing Methods and Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
You Can’t Test Everything . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Product Development as Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Adding Verification Steps to the Product Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Acknowledging Defects in the Original Specification. . . . . . . . . . . . . . . . . . . . . 66
Transforming Design into Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Table of Contents ix
5 Using Productivity Games to Prevent Defects . . . . . . . . . . . . . . . . . . . . . 79
What Is Game Theory? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
Games in History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
The Gamer Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Why Games Change Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
Types of Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
Games of Chance vs. Games of Skill . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Mini Games. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
Prediction Markets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
Alternate Reality Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Practical Guide to Defect Prevention Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Start with a Leader Board. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Keep It Simple . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Think Through Scoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Reward the Right Behavior. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Use Scoring to Encourage Participation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Keep the Players Coming Back to Check Their Score . . . . . . . . . . . . . . . . . . . . . 90
Youd Be Surprised at What People Will Compete For . . . . . . . . . . . . . . . . . . . . 90
Allow Room for Adjustment—Set a Time Frame . . . . . . . . . . . . . . . . . . . . . . . . . 91
Stratify to Maintain Interest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Retain a Players History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Start with a Small Pilot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Let People Play at Their Own Pace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Augment Interest by Using Cash and Prizes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Use Random Drawings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Examples of Using Games for Defect Prevention. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Tips on Game Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Game Design Checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
Suggested Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
6 Improving the Testability of Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Realizing the Benefits of Testability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Implementing Testability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
Simplicity: Creating Noncomplex Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Observability: Making Software Observable . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
Control: Increasing Control over Software Under Test . . . . . . . . . . . . . . . . . . . 104
Knowledge: Knowing What the Expected Results Are. . . . . . . . . . . . . . . . . . . . 105
xTable of Contents
Avoiding the Risks of Implementing Testability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Part III Defect Analysis Techniques
7 Software Measurement and Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
Understanding the Keys to Building a Successful Scorecard . . . . . . . . . . . . . . . . . . . 112
Creating Clear Definitions of Strategic Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Defining a Customer Strategy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Defining an Internal Business Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Defining a Financial Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Defining an Innovation Strategy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Creating Clear Definitions of Business, Process, and Improvement Objectives . . . 116
Understanding Goal Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Identifying Goals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Determining Metrics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
Prioritizing Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
Weighting Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
Avoiding Metric Manipulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
Scoping Goals Appropriately . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
Prioritizing Goals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
Creating SMART Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
Communicating Defined Goals to All Levels of Management . . . . . . . . . . . . . . . . . 123
Collecting and Presenting Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Automating the Data Collection and Reporting Processes . . . . . . . . . . . . . . . 125
Review. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Gaining Widespread Acceptance of the Defined Goals . . . . . . . . . . . . . . . . . . . . . . . 127
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
8 Risk Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
What Is Risk? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
What Is Risk Analysis? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
Applying Risk Analysis to River Rafting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Identifying Risk Analysis Phases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
Benefits of Risk Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
Understanding Risks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
Implementing Risk Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
Table of Contents xi
Creating a Risk Prediction Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
Characteristics: Identifying Code Characteristics . . . . . . . . . . . . . . . . . . . . . . . . 139
Amount: Tracking Churn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Impact: Understanding the Effect of Changes . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Reason: Understanding Why a Change Is Made . . . . . . . . . . . . . . . . . . . . . . . . 148
Ownership: Knowing Who Owns a Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
Applying a Risk Prediction Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
9 Using Simulation and Modeling for Organizational Innovation . . . . . 155
Understanding Stochastic Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
Using the Modeling Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
Defining the Goals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
Identifying the Initial Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
Determining Process Input and Outputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
Building the Espoused Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
Comparing Process Results to Organizational Results. . . . . . . . . . . . . . . . . . . . 167
Developing the Actual Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
Repeating If Necessary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
Baseline Process Model Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
Simple Planning Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
Improved Planning Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Detailed Quality Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
Process Improvement Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
Development Throughput Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
Relationship to the CMM Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
10 Defect Taxonomies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
Learning from Defects in Large Software Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
Specifying the Goals of a Defect Taxonomy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
Understanding the Organizing Principles of Defect Taxonomies . . . . . . . . . . . . . . . 200
Clarifying the Assumptions Made in a Defect Taxonomy . . . . . . . . . . . . . . . . . . . . . . 200
Assumption: We Can Make Only Certain Kinds of Changes. . . . . . . . . . . . . . . 201
Assumption: People Will Make Errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
Assumption: Defects Are Discovered Late in the Cycle . . . . . . . . . . . . . . . . . . . 202
Assumption: Defects Escape Their Phase of Creation in the Product Cycle . . 202
Assumption: Testing May Be Unbalanced . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
xii Table of Contents
Assumption: You May Have Outgrown Your Tools and Processes . . . . . . . . . 203
Assumption: You May Be Doing Late-Phase Design Correction . . . . . . . . . . . 203
Building the Example Defect Taxonomy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
Occurrence Stage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
Contributing Cause(s) Stage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
Change Stage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
Detection Stage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
Mitigation Stage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
Example of a Classified Defect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
11 Root Cause Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
Understanding How a Root Cause Analysis Study Helps Prevent Defects. . . . . . . . 227
When to Do an RCA Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
Staffing for Success. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
The Role of the RCA Study Analyst. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
Phases of an RCA Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
Phase I: Event Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
Phase II: Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
Phase III: Data Analysis and Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
Phase IV: Corrective Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
Conduct a Longitudinal Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
Phase V: Inform and Apply . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
Phase VI: Follow-up, Measurement, and Recommendations. . . . . . . . . . . . . . 244
Benefits of Root Cause Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
Risks of Root Cause Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
Part IV Defect Prevention Techniques
12 Adopting Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
Understanding the Traditional Development Process . . . . . . . . . . . . . . . . . . . . . . . . 254
Implementing Agile Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
Requirements Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
Project Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
Project Tracking and Oversight. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
Software Quality Assurance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
Software Configuration Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
Table of Contents xiii
Scrum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
Personal Software Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
Team Software Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
Encouraging Adoption of Innovative Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
Deploying a Coherent Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
13 FMEA, FTA, and Failure Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
Failure Modes and Effects Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
History of FMEA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
Implementing FMEA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
FMEA Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
Fault Tree Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .284
History. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
Implementing FTA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
Fault Tree Development Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
Fault Tree Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
Failure Modeling: Combining FMEA and FTA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
Failure Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
Comparing Threat Modeling with Failure Modeling . . . . . . . . . . . . . . . . . . . . . 300
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
14 Prevention Tab. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
How the Prevention Tab Works. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
Using the Prevention Tab Throughout the Product Cycle . . . . . . . . . . . . . . . . . . . . . 307
Writing High-Quality Preventions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
Who Can Contribute Preventive Techniques?. . . . . . . . . . . . . . . . . . . . . . . . . . . 307
Seeking Patterns of “Defect Introduction” Activity . . . . . . . . . . . . . . . . . . . . . . 310
Implementing a Prevention Tab Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
Establish Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
Identify Progress Tracking and Communication Methods . . . . . . . . . . . . . . . . 311
Establish a Location to Store Prevention Data . . . . . . . . . . . . . . . . . . . . . . . . . . 312
Provide Incentives for Prevention-Related Work . . . . . . . . . . . . . . . . . . . . . . . . 312
xiv Table of Contents
Ensure Sufficient Staff for Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
Practice Regular Reporting and Change Measurement . . . . . . . . . . . . . . . . . . 312
Taking Action on Prevention Tab Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
Categorizing Preventions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
In-Depth Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
Benefits of Using the Prevention Tab. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
Helps to Shift Individual Thinking to the Big Picture . . . . . . . . . . . . . . . . . . . . 316
Preventive Techniques and Knowledge Are Easily Shared . . . . . . . . . . . . . . . . 317
Prevention Data Is Stored with Discovery and Repair Data . . . . . . . . . . . . . . . 317
Provides Feedback Mechanism for Process Improvement . . . . . . . . . . . . . . . . 317
Eases Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
Is Applicable in All Phases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
Risks of Using the Prevention Tab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
Becomes a Platform for Blame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
Subject to Skewed Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
Easy to Overweight or Overreact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
Requires Compilation and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
Preventions May Be Too General or Too Specific. . . . . . . . . . . . . . . . . . . . . . . . 319
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
Part V A Culture of Prevention
15 Scenario Voting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
Applying the Law of Large Numbers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
Using Scenario Voting to Help Prevent Defects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324
Knowing When to Apply Scenario Voting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
Understanding the Scenario Voting Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
Operational Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
Writing High-Quality Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
Categorizing Scenarios. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
Understanding Who the Voters Are. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
Implementing a Scenario Voting Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
Understand the Appropriate Project Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
Know the Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
Develop an Experience Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
Set Clear Goals for Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332
Document and Develop the Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
Table of Contents xv
Solicit User-Developed Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335
Understand the User Base . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336
Get Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
Initiate Pilot Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
Deploy the Voting Program. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340
Keep the Program Lively . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340
Report Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340
Analyze the Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341
Encourage Ongoing Voter Participation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
Give Results to the Support Team. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344
Take Action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346
Benefits of Scenario Voting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348
Ease of Data Collection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
Ability to Gather a Large Volume of Data Across a
Wide Range of Features and Users. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
Applicability in All Phases of the Project Cycle. . . . . . . . . . . . . . . . . . . . . . . . . . 350
Risks of Scenario Voting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351
Results Are Subject to the Makeup of the Voting Population . . . . . . . . . . . . . 351
Results Provide Only a Summary View of User Opinions . . . . . . . . . . . . . . . . . 351
Incomplete Scenario Selection Can Skew Results. . . . . . . . . . . . . . . . . . . . . . . . 352
Poorly Authored Scenarios Can Skew Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 352
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
Suggested Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354
16 Creating a Quality Culture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355
Evaluating Your Current Culture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355
Common Cultural Pitfalls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
Metrics for Detecting Insufficient Designs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
Improving Your Culture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360
Influence Your Culture Toward Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365
17 Moving Quality Upstream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367
Quality and Customer Focus Are Intertwined. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367
Understanding Development as a Series of Translations . . . . . . . . . . . . . . . . . . . . . . 368
Transformations from Product Vision to Code . . . . . . . . . . . . . . . . . . . . . . . . . . 370
Avoiding Obstacles to Upstream Quality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372
Testing Does Not Create Quality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372
xvi Table of Contents
Quality Is Invisible. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372
Features Are Valued over Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
Engineering Attitude Blocks the Quality Mindset . . . . . . . . . . . . . . . . . . . . . . . 373
Task and Team Myopia Block the Big-Picture Perspective . . . . . . . . . . . . . . . . 374
Teams Avoid Due Diligence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375
Values and Rewards Do Not Promote Quality . . . . . . . . . . . . . . . . . . . . . . . . . . 377
Defects Have Different Risks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
Identifying the Causes of Poor Downstream Quality . . . . . . . . . . . . . . . . . . . . . . . . . 378
A Model of Future Product Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379
Development Is Customer Focused . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
Product Information Is Executable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383
Customer Scenarios Are Moved Upstream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384
Testing and Test Generation Are Automated . . . . . . . . . . . . . . . . . . . . . . . . . . . 384
Static Testing Is Pervasive. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385
Development Process Is Modified . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
Resulting Changes in Organization, Roles, and Career Paths. . . . . . . . . . . . . . 387
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388
18 Rewards, Motivation, and Incentives . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
Using Motivational Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
Eliminate De-motivators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392
Set SMART Goals for Defect Prevention Work . . . . . . . . . . . . . . . . . . . . . . . . . . 393
Measure Time and Effort Spent on Defect Prevention Work. . . . . . . . . . . . . . 393
Ensure That Leadership Behavior Reflects Focus on Defect Prevention. . . . . 393
Create a Culture of Defect Prevention. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394
Align Organizational Goals with Defect Prevention Efforts . . . . . . . . . . . . . . . 394
Design Organizational Processes with Defect Prevention in Mind . . . . . . . . . 395
Design the Reward System to Support Worker Diversity . . . . . . . . . . . . . . . . . 395
Incentives—It’s Not Just Money . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396
Celebrate Successes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397
Use Games and Competition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397
Understand What Motivates Individuals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397
Make an Ongoing Investment in Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . 398
Knowing What Success Looks Like. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399
Measuring Success . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400
Table of Contents xvii
19 Knowledge Management and Communication . . . . . . . . . . . . . . . . . . . 401
Problems Associated with Poor Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402
Isolated Knowledge. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402
Lack of Knowledge Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402
Inability to Discover Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403
Lack of Upward Communication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403
Techniques of Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403
Using Size as an Advantage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404
Properties of a Good Communication Model. . . . . . . . . . . . . . . . . . . . . . . . . . . 405
Taxonomies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
Organic Expertise Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406
Prevention Tab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408
Scenario Voting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
20 Pulling It All Together . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411
Understanding Standards and Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412
Trains, automobiles, and “PF” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412
Common Results Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414
Everyone Plays a Role . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415
Quality Assurance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415
Code Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 420
Program Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 428
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
Microsoft is interested in hearing your feedback so we can continually improve our books and learning
Microsof
resources for you. To participate in a brief online survey, please visit:
resourc
www.microsoft.com/learning/booksurvey/
What do you think of this book? We want to hear from you!
What
xix
Acknowledgments
After many years of studying how and why people make mistakes (and making plenty of them
ourselves), we realize how common it is for humans to make errors of omission. It’s our hope
that we have successfully prevented any errors of omission in this section. This book stems
primarily from the work of the Microsoft Windows Defect Prevention team, and therefore the
first thanks must go to Darren Muir, General Manager of Windows Core Test, for having the
vision and foresight to fund our efforts, and to his boss at the time of our inception, Brian
Valentine, Vice President of Windows Core Operating System Division, for his confidence in
this work, his faith in our ideas, and for our day jobs. None of this would be possible without
these two people recognizing the long-term benefits of an investment in researching and
developing these techniques.
This book is the work of many authors. Although only three of us are listed on the cover, we
must thank each of the contributing authors for their work on their respective chapters, as
well as for their time, effort, and contributions to the entire book, which also includes numer-
ous reviews of other authors’ chapters as well, providing valuable feedback and suggestions.
Dan Bean Chapter 1, “Defect Prevention,” and Chapter 13, “FMEA, FTA, and Failure
Modeling”
David Catlett Chapter 6, “Improving the Testability of Software,” and Chapter 8, “Risk
Analysis”
Lori Ada Kilty Chapter 7, “Software Measurement and Metrics”
Joshua Williams Introduction and Chapter 20, “Pulling It All Together”
We must also acknowledge the fantastic reviewing work of Ed Triou, Test Architect on the
Microsoft SQL Server team. His careful study, review, and feedback on each chapter has
improved this book immensely—trust us, you will enjoy it more as a result of Ed’s efforts.
We would like to thank Brendan Murphy from Microsoft Research in Cambridge for contrib-
uting lots of data to our cause, and Nachi Nagappan, Tom Ball, and Thirumalesh Bhat from
Microsoft Research in Redmond for their help with the risk analysis work. Thanks to Patrick
Wickline, another Microsoft researcher, and Chris Walker, Test Architect in the Secure
Windows Initiative, for their help with our root cause analysis efforts. Thanks to Harry Emil,
developer on the Defect Prevention team, for his help with the companion Web site
www.defectprevention.org.
Thanks also to a long list of talented developers from the Windows Core Organization for
creating a range of tools to support these techniques, including Harry, Jinu Joseph, Francie
“Viva Piñata” Emory, Tim “Forza 2” Graves, Jeff “WFH” Forde, Guru “Black Friday” Vasisht,
and James “Eat My Voltage” O’Rourke. Thanks to Jacqueline Richards, Program Manager for
xx Acknowledgments
Microsoft Windows Server Customer program, for help with scenario voting, and to William
Rios, for his patience with us and the efforts on metrics and reporting. Thanks also to Ben
Sawyer, founder of the Serious Games Initiative, for his inspiration and help with productivity
games.
Additionally, thanks to the many teams, managers, mentors, friends, and colleagues we have
worked with over the years who have helped us learn from our own mistakes. And, in no
particular order, we would like to thank the Microsoft Test Architect Group, Klaas Langhout,
MSDN Test Center, James Rodrigues, Angelika Kinneman for her work on games, Aseem
Badshah of Scriptovia.com for his help with scenario voting, Monte Christensen for support-
ing and continuing the risk analysis work, the WinSE crew Junaid Ahmed, Alex Tarvo, and
Koushik Rajaram for taking risk analysis to the next level, Craig Schertz, Kevin Burrows and
the Coverage Tools Team, folks from XNA: Michele Coady, Aaron Stebner, and Dean Johnson,
who took the time to review various chapters and provide much appreciated feedback.
Even with the help, support, and assistance of every one of the preceding people, you
would not be reading this now if it weren’t for the excellent team at Microsoft Press—the
most amazing editor we’ve ever worked with, Lynn Finnel, who totally kept us on track
(actually, we never were on track—she kept us from going too far off track), copy editor
Christina Palaia, and Stephen Sagman, who worked on production, art, and proofs.
Thanks to our editor Devon Musgrave for his help in getting things organized. Thanks
also to Microsoft Press Commissioning Editor Ben Ryan.
And thank you, our readers.
We hope you enjoy the book and find it useful.
Please send along any feedback you have—we’d love to hear from you!
xxi
Introduction
Progress, far from consisting in change, depends on retentiveness. When change is
absolute there remains no being to improve and no direction is set for possible
improvement: and when experience is not retained, as among savages, infancy is
perpetual. Those who cannot remember the past are condemned to repeat it.
—George Santayana, The Life of Reason (1905)
People have been making mistakes since the beginning of time. More important, people have
been learning from their mistakes since the beginning of time.
Charles Darwin, on page 101 of Volume I of his epic The Descent of Man (1871), quotes
Herbert Spencer: “I believe that the experiences of utility organised and consolidated through all
past generations of the human race, have been producing corresponding modifications, which, by
continued transmission and accumulation, have become in us certain faculties of moral intuition—
certain emotions responding to right and wrong conduct, which have no apparent basis in the
individual experiences of utility.”1
How many times do you lock your keys in your car before you get a Hide-A-Key or buy a car
with a combination lock? How many checks do you have to bounce before you get overdraft
protection or fudge the balance to prevent the twenty dollar fee? How often do cell phones go
off in meetings? How many times are jail inmates accidentally released? There are thousands
of simple, yet common errors that people make every day. Because some people are employed
as software developers, it’s only logical that errors appear in their work, too. For an assembly
line worker in Detroit, a human error might result in a new vehicle with a loose bolt in the
trunk. For a restaurant worker in New York City, a mistake might result in a burger without
the requested extra pickles. For a software developer in Redmond, a human error might result
in a user’s inability to print driving directions downloaded from the Web.
The goal of this book is not to help you avoid locking your keys in your car. The goal of this
book is to provide a set of techniques to help software developers, project managers, and
testers prevent human errors, or defects, in their software.
More Info This book has an associated Web site at www.defectprevention.org. Some of the
material in the book is available there, including examples, templates, and reader-generated
content.
Following the introduction, this book is organized in five sections that follow the migration of
defects through a project.
1 Letter to Mr. Mill in Bain’s Mental and Moral Science: A Compendium of Psychology and Ethics, 722; London;
Longmans, Green, and Company, 1868.
xxii Introduction
Part I, “Introduction to Defect Prevention,” explains the goals behind the book and why you
may want to implement many of these practices in your organization. These chapters intro-
duce the concept of defect prevention, outline a number of prevention frameworks, and cover
the economics of investments in prevention techniques.
Chapter 1, “Defect Prevention,” provides an overview of and introduction to the topic of pre-
venting defects. This includes describing what a software defect is, why defects occur, and
what can be done to prevent them. This chapter also introduces the concept of defect detec-
tion, analysis, and prevention techniques and the factors to consider when determining the
most appropriate defect prevention strategy for your organization.
Chapter 2, “Defect Prevention Frameworks,” contains information on selecting a process
improvement framework. The need for a framework is described and several popular frame-
works are presented. The chapter is useful for those unfamiliar with such frameworks as the
Capabilities Maturity Model or the Malcolm Baldrige Quality Framework.
Chapter 3, “The Economics of Defect Prevention,” provides information on return on invest-
ment (ROI) calculations for software. The intent is to provide the reader with enough informa-
tion to determine if a particular investment is worthwhile from a financial perspective.
Additionally, an example calculation provides a rationale for process improvement in general.
The information is useful for anyone seeking to invest in process improvements and also for
those trying to estimate the value of existing processes.
Part II, “Defect Detection Techniques,” focuses on techniques that are useful in improving
product quality.
Chapter 4, “Quality and the Development Process,” discusses what quality is for a software
product and how the development method creates opportunities for defects. Software cannot
be tested exhaustively, so it is important to make it rare for a customer to encounter a defect
and to have zero tolerance for security defects. To a customer, there is no significant difference
between a problem that is a “real” defect and a problem based on a design defect. These con-
cerns affect the type of tests you write and how you measure quality. Writing software is a
series of translations from one form to another, and, except for compilation, human review is
involved. Errors in translation and loss of information typically occur.
Chapter 5, “Using Productivity Games to Prevent Defects,” covers the use of games and com-
petition to focus individual or team effort on defect prevention activity. The challenges of soft-
ware development involve a variety of activities, and typical projects draw the focus toward
immediate, short-term activities, whereas the payoff for defect prevention investments are
often longer term. Productivity games and competition can help balance the time investments
and draw attention to the longer-term investments.
Chapter 6, “Improving the Testability of Software,” explains the importance of designing
components and systems that can be tested easily so that defects are more readily discover-
able throughout the product development life cycle. This chapter introduces the Simplicity,
Introduction xxiii
Observability, Control, and Knowledge (SOCK) model of testability. Targeted at design
engineers and testers, this chapter provides practical guidelines for improving testability.
Part III, “Defect Analysis Techniques,” showcases several important techniques to analyze
defects after they have been detected.
Chapter 7, “Software Measurement and Metrics,” discusses the importance of building the
right foundation before gathering and reporting metrics. Though this chapter may seem to
apply only to managers or marketing personnel, it emphasizes the fact that everyone working
on a project must understand what is being measured and why—how metrics are ultimately
tied to the business. For example, what does measuring defect detection rates tell you? How
will this either improve your process or the quality of the product? How does that improve-
ment turn into dollars, and is the amount saved or earned worth making the change? Building
a balanced scorecard for your business can help justify defect prevention techniques and
show how programs can affect the bottom line.
Chapter 8, “Risk Analysis,” defines risk and how it relates to the software development life
cycle. Using a Reader’s Digest, “Drama in Real Life” river rafting illustration, it provides a
unique look at how important it is to understand and predict areas of risk as the software
project moves downstream toward completion. This chapter provides a practical risk analysis
model that you can apply to any software project. The model helps project decision makers
and engineers understand the risk involved when changes are made during the software
development life cycle.
Chapter 9, “Using Simulation and Modeling for Organizational Innovation,” provides a
statistical technique for process analysis that is useful for companies implementing Capability
Maturity Model Integration (CMMI) level 4 practices. It provides a means to analyze process
improvements, estimate schedules based on historical data, and analyze details of subpro-
cesses as they relate to the organizational process. The chapter provides a process for model-
ing and develops an example from a simple model to a very detailed process that includes
dependencies between different organizational groups. The chapter is useful for anyone plan-
ning process improvements and also for project managers looking to improve estimates of
schedules and create estimates of quality.
Chapter 10, “Defect Taxonomies,” discusses how to modify your defect reporting system to
provide information for the prevention and reduction of defects as well as surface possible
best (or worst) practices. Typical defect reporting systems track a defect through the correc-
tion process but provide little data-minable information to improve the development and test-
ing processes. The learning from defect processes breaks down as a product becomes more
complex or a development team grows larger.
Chapter 11, “Root Cause Analysis,” covers the popular defect prevention technique that pro-
vides a structured methodology for analyzing the cause and effect relationships of an event or
defect in pursuit of a corrective action that eliminates the cause(s) of problems. There are six
phases to a typical root cause analysis study, and this chapter outlines how and when to apply
xxiv Introduction
this defect analysis technique and how to use it to learn from defects discovered in the
development process.
Part IV, “Defect Prevention Techniques,” begins to shift the focus to a longer-term view by
covering techniques to prevent defects.
Chapter 12, “Adopting Processes,” provides a view of several popular processes with respect
to frameworks. It provides a brief discussion of problems associated with implementation and
improvement opportunities in existing processes. The chapter is useful for those seeking to
make a change or looking for change opportunities in an existing process.
Chapter 13, “FMEA, FTA, and Failure Modeling,” introduces a couple standard defect preven-
tion techniques: Failure modes and effects analysis (FMEA) and fault tree analysis (FTA). Both
FMEA and FTA were originally introduced to improve reliability in such industries as the aero-
space and automotive industries where the risk of defects and failures can be significant and
life threatening. These same concepts can be applied to software development to help identify
potential defects and proactively address them to improve software reliability and quality.
FMEA and FTA are complementary techniques in that a failure mode identified by FMEA can
become the top failure event analyzed in FTA. Combining the use of FMEA and FTA into a uni-
fied failure model has conceptual similarities to the threat models used in security threat
modeling. The chapter concludes with a brief illustration of one possible approach.
Chapter 14, “Prevention Tab,” provides a useful technique for gathering defect prevention data
at the point of experience, when the knowledge is the richest. This technique can be imple-
mented on any project, small or large, and offers development personnel an easy way to
provide suggestions and input on how defects can be prevented.
Part V, “A Culture of Prevention,” explains how long-term organizational and cultural changes
are necessary to implement a successful defect prevention program.
Chapter 15, “Scenario Voting,” discusses a technique you can use to integrate more tightly the
voice of the customer in the development process. By using this customer feedback technique,
software developers can take advantage of the wisdom of the crowd in assessing the viability
and the quality of their software.
Chapter 16, “Creating a Quality Culture,” discusses the typical impediments to creating a
culture that values quality and methods to improve your culture. A quality culture can’t be
decreed; it is a value that everyone needs to accept and act on. Because quality and customer
perception are intertwined, customer focus is a value that helps instill a culture of quality.
Establishing a quality culture can be a long process, and it is often difficult to measure
progress.
Chapter 17, “Moving Quality Upstream,” discusses how improving quality at the start of
development reduces defects downstream. Maintaining focus on the customer from the start
is an important part of this, as is doing the due diligence during the predesign phases that
minimizes design errors surfacing late in development. Customer focus is more than just
Introduction xxv
scenarios; it is understanding a customer’s mental model of the software’s functionality. A
development process that reduces defects and provides earlier customer verification is
described as well as a future process oriented around both customer focus and reducing
translation errors in progressing from vision to code.
Chapter 18, “Rewards, Motivation, and Incentives,” outlines a variety of management tech-
niques you can use to help motivate individuals and teams to focus their efforts away from the
immediate short-term tasks and toward a bigger-picture quality view. Preventative actions
require forward thinking and the ability to act on things before they happen. Showcasing mis-
takes as a learning opportunity is not a natural human behavior, and this chapter outlines a
variety of techniques to help inspire efforts for long-term quality improvement through defect
prevention activity.
Chapter 19, “Knowledge Management and Communication,” covers how the flow of informa-
tion can affect defects. As your product becomes more complex, the team grows, or the team
becomes geographically dispersed, it is no longer possible for any one person to know every-
thing about a product. Specialization—knowing about only a limited area rather than the
entire product—is a typical approach to solving this problem. However, today’s products often
have intertwining dependencies that require broader knowledge. Methods to improve com-
munication between fields of specialization are discussed as well as how to use size as an
advantage rather than seeing it only as a problem.
Chapter 20, “Pulling It All Together,” discusses turning your development process from a
series of independent steps into an optimized workflow by analyzing each phase of the prod-
uct and how it affects others. Each role plays a part in producing a quality product, and by
coordinating quality practices throughout the development process, teams can create both
the most efficient process and the highest quality output. Successful teams rely on effective
communication, good workflows, and a mindset for continuous improvement. This chapter
discusses these practices and provides personal experiences to demonstrate them. Although
it was written primarily with the quality assurance staff in mind, project planners and
managers can also benefit from a quick read of this chapter.
Much of this book stems from the experience of the Microsoft Windows Defect Prevention
team, and although there are some specific Microsoft examples throughout the book, the goal
is to describe the techniques in a way that is applicable to all software development efforts,
large and small.
Many books have been written on the topics of software quality assurance, testing, agile
development, and other related subjects. This book is meant to augment, not replace, those.
Several successful quality assurance and defect detection and analysis techniques are not cov-
ered in this book because they are already well covered elsewhere. Where applicable, we’ve
included references to other sources, and we maintain a list on the companion Web site at
www.defectprevention.org.
xxvi Introduction
Who This Book Is For
This book is for software developers, product planners, senior managers, project managers,
and testers who want to learn practical ways to improve product quality by preventing defects
in their software. This is a practical guide, written to be understandable and applicable regard-
less of organizational size, development language, or environment. Different chapters will
make sense when you are at different phases of the development cycle; each chapter has a
slightly different audience from the others. The introductory chapters provide a good over-
view and some context for how, why, and where the techniques can be applied. We suggest
you read about techniques that may be applicable to you now, and then skim through the
other chapters rather than trying to read the book cover to cover at first. Later, you can return
to chapters that have become applicable as you move through the development process.
Support for This Book
Every effort has been made to ensure the accuracy of this book. As corrections or changes are
collected, they will be added to a Microsoft Knowledge Base article.
Microsoft Press provides support for books at the following Web site:
http://www.microsoft.com/learning/support/books/
Questions and Comments
If you have comments, questions, or ideas regarding the book or the companion content, or
questions that are not answered by visiting the preceding site, please send them to Microsoft
Press by e-mail to
mspinput@microsoft.com
Or by postal mail to
Microsoft Press
Attn: The Practical Guide to Defect Prevention Editor
One Microsoft Way
Redmond, WA 98052-6399
Please note that Microsoft software product support is not offered through the preceding
addresses.
Part I
Introduction to Defect Prevention
In this part:
Chapter 1: Defect Prevention. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Chapter 2: Defect Prevention Frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Chapter 3: The Economics of Defect Prevention . . . . . . . . . . . . . . . . . . . . . 37
3
1
Chapter 1
Defect Prevention
6QOCMGPQOKUVCMGUKUPQVKPVJGRQYGTQHOCPDWVHTQOVJGKTGTTQTUCPFOKUVCMGUVJG
YKUGCPFIQQFNGCTPYKUFQOHQTVJGHWVWTG
²2NWV CTEJ)TGGMDKQITCRJGTCPFOQTCNKUV#&±
:PVKVTUPQFOFEUIJTCPPLBOETBXUIBUUIFGJSTUQBHFPGUIFGJSTUDIBQUFSJTQSJOUFEVQTJEF
EPXO
)PXEJEZPVSFBDU 8FSFZPVTVSQSJTFE "OOPZFE "OHSZ "MMPGUIFTFFNPUJPOT %JEUIF
QSJOUJOHEFGFDUJNQBDUZPVSJOJUJBMJNQSFTTJPOPGUIFCPPLBOEJUTRVBMJUZ 
"MMQSPEVDUTDBOIBWFEFGFDUTUIBUJNQBDUUIFVTFSBOETPNFBSFNPSFBOOPZJOHUIBOPUIFST
*GUIJTQBHFPOMZJODMVEFENJOPSTQFMMJOHBOEHSBNNBSNJTUBLFTZPVNBZIBWFKVTUOPUJDFE
UIFNCVUNPWFEPO)PXFWFSGPSBNBKPSQSJOUJOHNJTUBLFZPVQSPCBCMZSFBDUFENVDIEJGGFS
FOUMZ%JEZPVDIFDLUIFOFYUQBHFUPTFFJGJUXBTQSJOUFEDPSSFDUMZBOEUIFOKVTUUVSOUIFCPPL
PWFSUPSFBEUIJTQBHF %JEUIJTNBLFZPVRVFTUJPOXIFUIFSUPCVZUIJTCPPL *GZPVIBE
BMSFBEZQVSDIBTFEUIFCPPLEJEJUNBLFZPVUIJOLBCPVUFYDIBOHJOHJGGPSBHPPEDPQZPS
BTLJOHGPSBSFGVOEQMFBTFEPOU
5IFTFBSFBMMOPSNBMSFBDUJPOTUPGJOEJOHBEFGFDUJOBOZQSPEVDUJODMVEJOHTPGUXBSF
$VTUPNFSTDBODIPPTFUPJHOPSFUIFEFGFDUBDDFQUJUDPOTJEFSBXPSLBSPVOETFFLBO
FYDIBOHFPSSFUVSOUIFQSPEVDUGPSBSFGVOE/PNBUUFSXIJDIDIPJDFJTNBEFUIFJSTBUJTGBD
UJPOXJUIUIFQSPEVDUVTVBMMZEFDSFBTFT5PNBLFNBUUFSTXPSTFDVTUPNFSTFYQFSJFODJOH
QSPEVDUEFGFDUTDSFBUFBEEJUJPOBMQSPEVDUDPTUTTVDIBTEJSFDUDPTUTGSPNSFGVOETBOEJOEJSFDU
DPTUTGSPNOFHBUJWFSFDPNNFOEBUJPOTBOEMPTUTBMFT6MUJNBUFMZUIFTFBSFBMMDPTUTPGQPPS
RVBMJUZUIBUXJMMBGGFDUDVTUPNFSTBUJTGBDUJPOBOEUIFCPUUPNMJOFPGBCVTJOFTT5PBWPJEUIFTF
DPTUTUIFCFTUPQUJPOJTUPJOWFTUJOEFGFDUQSFWFOUJPO
Figure 1-1 The 24-cent Inverted Jenny stamp at the Smithsonian National Postal Museum
Chapter 1
3
4Part I Introduction to Defect Prevention
Why should you and your software development team be concerned about defects? Because
your goal is to produce high-quality software that provides value to users and meets their
expectations. To accomplish this goal, your software, like a book with printed pages, must be
as defect free as possible. Customers expect it to be so, and if you do not provide products that
are defect free, your customers may not be your customers for long. Suffice it to say that any
time there is a defect, it will eventually cost you something. In addition to money, it may also
cost you a development team that is frustrated and worn out from continually fixing defects or
a dissatisfied customer who switches to someone else’s software product in the future. We
wrote this book to help you avoid defects and their associated costs.
Tip If you are an experienced software engineer and thoroughly understand software
defects and why they occur, you may want to skip directly to the section titled “What Can Be
Done?” later in this chapter.
If you are reading this book, there is a pretty good chance you have experienced software
problems and understand the impact they can have. Consider the last time you lost work
because of an application crash, or a time when you could not figure out how to accomplish a
task using the software. How did that make you feel? Do you want users to have these same
kinds of experiences with your software? Use your personal experiences as motivation to
learn how to prevent these types of defects in the future. Sometimes this is called “walking in
your customer’s shoes.”
On the other hand, perhaps you believe that software will always have defects. After all, soft-
ware is created by humans and “to err is human”; therefore, defects must be inevitable, right?
Many people feel this way. Following this logic, should a software development team simply
accept the inevitability of defects and the human errors that cause them? The answer must be
an emphatic “No! Defects can and must be prevented because users deserve better—they
deserve high-quality software.
Improved software quality can be accomplished in a variety of ways, but the best approach is
defect prevention because the benefits realized through prevention efforts can be reapplied in
the future. Our goal for this book is to provide you with a high-level, practical guide covering
defect prevention techniques that apply to all stages of software development and many dif-
ferent types of defects. Our hope is that the techniques outlined in this book will help soft-
ware development teams improve their software over the long term to enhance the quality of
the user experience. Although much has already been done and written about preventing
defects in software, opportunities for improvement remain.
What Is a Software Defect?
A software defect is a deficiency in a software product that causes it to perform unexpectedly.
From a software user’s perspective, a defect is anything that causes the software not to meet
their expectations. In this context, a software user can be either a person or another piece of
Chapter 1 Defect Prevention 5
software. A few examples are listed in Table 1-1. From a software developer’s perspective, a
defect is anything that must be corrected in a software work product.
Defects can be introduced during any stage of the development cycle regardless of whether
you are following the sequential requirements-design-implementation-verification-mainte-
nance phases of the waterfall model or the rapid iterations of agile software development. The
goal of this book is not to espouse any one development approach over another. However, it
is important to have a context when referring to defects and the relative point in the develop-
ment cycle when they can occur. Consequently, to provide a frame of reference the following
product life cycle will be used:
Figure 1-2 Reference software product life cycle
Table 1-1 Software Defects from the Users Perspective
Typical Examples of Software Defects
User Expectation The software will help me accomplish a task
Software Defect Desired software functionality is missing
User Expectation Clicking on the button performs the task I want to do
Software Defect Clicking on the button does nothing or not what I want it to do
User Expectation A file can be successfully copied to another location
Software Defect The file becomes corrupted during the copy process
User Expectation Calling a method in the API will perform as documented
Software Defect The API fails due to an undocumented change to the registry
Less Obvious Examples of Software Defects
User Expectation The software will hellp me avoid mistakes (for example, spelling errors)
Software Defect A spelling error caused by using a valid word incorrectly is not detected
User Expectation The software will respond quickly
Software Defect The software responds too slowly from the users perspective
User Expectation The software is secure from hackers
Software Defect Hackers are able to exploit vulnerability and attack the software
User Expectation For a “fatal error,” a return code will be received so impact can be mitigated
Software Defect No fatal error return code is sent, and the software freezes
Phases: Product Definition Product Servicing
Stages: Value Proposition Requirements Maintenance
Design
Implementation
Verification
Release
Product Development
6Part I Introduction to Defect Prevention
In the Product Definition phase, the Value Proposition stage completes an analysis of what is
happening in the marketplace. In the Product Development phase, the Requirements stage
defines the scenario requirements and features of the product. The Design stage determines
how the features will be built. The Implementation stage focuses on coding, documentation,
and internal testing. The Verification stage focuses on customer beta testing of completed fea-
tures in the product. The Release stage yields a completed product ready for customers. In the
Product Servicing phase, the Maintenance stage provides ongoing maintenance for the prod-
uct, including hotfixes, security updates, and service packs.
No matter how or when it occurs, the severity of a defect will vary based on the impact it has
on a user. A software program that crashes or stops working has a major defect that prevents
its use. A software program that presents an inconsistent user interface has a minor defect that
may lessen its value or hinder user productivity, but doesn’t actually prevent its use. Based on
project schedules, available resources, and the impact and severity of the defects, you will
need to prioritize some defects to be fixed before others. Although this prioritization can
be the reality, the ultimate goal should still be to create and release software that is defect free.
Making High-Quality Software the Goal
To have a goal of high quality, you first need to understand what high-quality software means.
Traditionally, high quality has meant a lack of defects found during inspection and testing.
Unfortunately, as pointed out in Chapter 4, “Quality and Product Development,” testing can
only indicate a lack of quality and it is not possible to test software completely. Consequently,
the number of defects found in test cannot be the only metric used to assess software quality.
In many cases, quality is in the eye of the beholder, so you also need to evaluate subjective
quality based on customer perceptions and expectations. (See Chapter 15, “Scenario Voting.”)
In terms of perceived quality, software defects caused by unmet customer expectations are
just as important as defects related to broken functionality. Therefore, they both need to be
taken into consideration. The cost of a problem found late in the development process can be
significant. Repair costs rise as the development cycle progresses because of increased rework
caused by design and coding changes, retesting, and impact on other dependent software.
Industry studies have shown that a defect that costs 1x to fix in design could cost 100x to fix
after the software is released.1 Clearly, everyone—from product designers to product develop-
ers and product stakeholders—will benefit if the development team can prevent defects early
rather than discovering and fixing them later.
Critical defects include issues such as lost data, data corruption, and inadequate security that
prevent a user from having a reliable computing experience. A solid design helps mitigate the
risk of critical defects. Fortunately, defect prevention techniques can be applied early in the
design stage to improve the quality of the design and minimize risks.
1 B. Boehm and V. Basili, “Software Defect Reduction Top 10 List,” IEEE Computer Society 34, no. 1 (January
2001): 135–137.
Chapter 1 Defect Prevention 7
Over the years, many different defect prevention techniques have evolved to manage the
“defect introduction” opportunities inherent in product development. Many of these
techniques can be applied to software development. In the case of software development, the
applicability and effectiveness of these techniques can vary dramatically across different
projects. Therefore, it is important to consider which defect prevention techniques will be the
most useful for your software project. The purpose of this book is to highlight practical and
effective defect prevention techniques to help you make those choices and select the most
beneficial techniques for your needs.
We have met the enemy and he is us.
—Walt Kelly, Pogo comic strip
Understanding Why Software Defects Occur
A defect in software results from some type of mistake. Usually these mistakes are a result of
human error, but sometimes they are caused by systemic errors in the development process.
Mistakes can also result from faulty development tools, a misunderstanding of customer
requirements, or other issues that arise in the course of software development. Fortunately,
not every mistake leads to a defect, but almost all defects can be traced back to some type of
mistake (see Table 1-2).
Table 1-2 Mistakes and Resulting Software Defects
Typical Mistakes That Cause Software Defects
Mistake Communication difficulties between customers and software developers
Software Defect Desired software functionality is missing
Mistake Developer overlooks a logic error in the code
Software Defect Clicking on the button does nothing
Mistake Developer forgets error checking in the file copy code
Software Defect A corrupted file is copied, and the software crashes
Mistake Developer does not understand the customer scenario
Software Defect The software does not meet the customers needs
Mistake Developer only tests the software using a fast computer and the performance
seems fine
Software Defect The software responds too slowly from the perspective of a user with an older
and slower computer
Mistake Developer does not review the code for software vulnerabilities
Software Defect Hackers are able to exploit vulnerabilities and attack the software
Mistake Developer does not recognize a fatal error condition, so no return code is sent
Software Defect Software does not return a code indicating a fatal error condition
8Part I Introduction to Defect Prevention
Although it may be easy, or even fashionable, to blame a computer for causing a software
defect, in reality almost all defects can be attributed to some type of human error. Because
computers are machines, there is a possibility that the computer hardware is to blame, but in
most cases software defects result from mistakes made by people. Consequently, the study of
human error offers a good framework for determining and understanding why software
defects really occur.
Analyzing the Bases of Human Error
The science of human error is a fascinating subject, and some tremendous research into clas-
sification systems, causality models, and error reduction techniques has been done. Machine
plant safety, disaster planning, and prevention of catastrophic accidents are all areas where
human error studies have led to positive results. Investigation into human errors and their
impact on software development was initiated during the 1970s. Since then, research has con-
tinued, first driven by interest in software safety and hazard analysis, and then more recently
by interest in software quality and security.
The study of human error is a broad topic, and it helps to start with an initial foundation. One
view of human errors draws its structure primarily from the work of James Reason.1 In this
view, there are three broad categories of errors that provide a framework for analysis: skill-
based errors, knowledge-based errors, and rule-based errors.
Skill-based errors are usually the obvious mistakes that occur when a person knows what to
do, has done it successfully in the past, and just “slipped up.” Knowledge-based errors are mis-
takes made because a person doesn’t know everything about a problem space and therefore
didn’t know what to do. Rule-based errors are mistakes made by following a rule; rule errors do
not include mistakes made where a rule could exist, just those made by applying or misapply-
ing an existing rule.
Applying human error research to software defects can yield interesting insights. In particular,
a clear understanding of how and why people make mistakes can be useful in improving
development processes, coding and check-in rules, compiler settings, design principles, user
research, quality assurance efforts, and many other areas that require humans to do work. See
Chapter 10, “Defect Taxonomies,” for more information.
An ounce of prevention is worth a pound of cure.
–Henry de Bracton, De Legibus, AD 1240
What Can Be Done?
Mistakes can be made in many different areas of the software development effort. Coding
defects are the most obvious, but mistakes can also occur in design, management, project
planning, and other activities throughout the organization. It often seems that the only real
1 James Reason, Human Error (Cambridge, UK: Cambridge University Press, 1990).
Chapter 1 Defect Prevention 9
guarantee is that mistakes will occur and some will cause software defects. Realizing this,
most software development teams invest heavily in testing as their primary focus for
eliminating defects before the defects can impact users.
Defect prevention requires investment with a different focus. An investment in defect
prevention emphasizes a more proactive and cost-effective way to eliminate software defects.
Rather than relying on testing to find software defects after they are introduced, defect preven-
tion techniques focus on preventing the introduction of defects in the first place. The basic
theme of this book is that defects can be prevented and that the software development pro-
cess can be improved over time, using a variety of techniques, to reduce the number of defects
in software.
Using Detection, Analysis, and Prevention Techniques
Software quality and defect rates can be addressed at three levels. The first level is detection.
This is the most common approach to improving software quality. “Test quality into the soft-
ware”—test the software until all the discoverable defects are found and fixed. (See Chapter 6,
“Improving the Testability of Software” for more information.) Although this is a goal of all
software development teams, it is not possible to completely test software. However, what is
possible is to detect many of the defects, and there are many different ways to do that. Exam-
ples include static analysis tools, automated test cases, and user beta testing. These methods
can identify defects where they exist but do little or nothing to prevent a similar defect from
recurring elsewhere.
The next level of software quality improvement is achieved through analysis. At this level, time
is spent analyzing previously discovered defects to look for trends and insights into how they
occurred and why they were not detected earlier. See Chapter 11, “Root Cause Analysis,” for
more information. Analyzing defects takes time and expertise, but the effort can yield signifi-
cant benefits when improvements are made based on the results.
Over the long term, the most effective level of quality improvement efforts is prevention. At this
level, specific techniques are used to proactively identify and eliminate potential defects. Also,
the findings from the detection and analysis levels can be used to modify processes and devel-
opment practices to eliminate the root causes of defects. These findings may also be applied to
earlier detection, thereby reducing repair and rework costs. Eliminating the defects and their
root causes improves the effectiveness of future development efforts as well. The ultimate goal
of any quality improvement effort should be to enable the development team to invest more
time in defect prevention activity.
What Is Different in a Prevention Organization
Most software development teams recognize the need to prevent defects and try a variety of
techniques to prevent them. However, most of these techniques focus on improved defect
detection (testing) and not on defect prediction and prevention. Because of issues such as
10 Part I Introduction to Defect Prevention
schedule pressures and insufficient resources, software teams are often constrained in their
ability to invest in effective quality assurance and quality improvement processes. While they
work hard to try different improvements and may even try some prediction and prevention
techniques, the sustainable results they want are never fully achieved.
To help software development teams attain sustainable results, organizations such as the
Carnegie Mellon Software Engineering Institute (SEI) have developed models like the
Capability Maturity Model (CMM) to give teams a way to assess the capabilities of their devel-
opment process and also provide a roadmap for improvement. See Chapter 2, “Defect Preven-
tion Frameworks” for more information. In the CMM five levels of process maturity, level 4
organizations predict defects and manage quality using statistical process control techniques
and level 5 organizations prevent defects by quantitatively improving processes through inno-
vative process and technological improvements.1 Because attaining this level of process matu-
rity can help teams produce better, faster and cheaper products with higher quality and better
customer satisfaction, this would seem to be a logical goal for all software development orga-
nizations. However, based on a 2006 profile of CMM appraisal results, SEI determined that
only 137 (7.6%) of the 1804 organizations undergoing CMM appraisal were operating at level
4 and only 175 (9.8%) were operating at level 5.2 The majority of the organizations were oper-
ating at a lower CMM maturity level with a focus on detection and reaction, the first level of
defect prevention and not on prediction and prevention.
Why is this? It’s primarily because a focus on analysis and prevention can initially require
more time and resources, and although the return on investment is usually high, the costs can
be significant. If a project is on a tight schedule or budget, the project manager must decide if
it is more effective to ask senior engineers to invest their time in analyzing bugs and develop-
ing preventions or to have them write new features and fix existing bugs in the current prod-
uct to accelerate the ship date. This creates a dilemma: make your boss happy by meeting a
project deadline with some defects, or make the project stakeholders and users happy by pre-
venting defects at the expense of missed deadlines and extra resource investments. Because
software projects are typically rewarded upon completion, project managers usually choose
the short-term view and opt for completing the current effort before investing in long-term
quality improvement techniques. Although this can be a prudent decision for them to make,
it can be shortsighted for their organization and company in the long run.
In contrast, a software development team that acknowledges and focuses on the potential of
defect prevention is enlightened. As a “prevention organization,” it understands that analyzing
defects and applying the learning to drive continuous improvement can achieve high quality
and significant economic benefits over the long term. Although a prevention organization
still has the normal resource and scheduling challenges, it makes a conscious effort to invest
in prevention activities. The biggest difference between a detection organization and a
1 CMMI Product Team, CMMI for Development, Version 1.2, (Pittsburg, PA, Carnegie Mellon Software
Engineering Institute, August 2006)
2 CMMI Appraisal Program, Process Maturity Profile, Software CMM 2005 End-Year Update, (Pittsburgh, PA,
Carnegie Mellon Software Engineering Institute, March 2006)
Chapter 1 Defect Prevention 11
prevention organization is that a detection organization typically is in constant “fire drill”
mode just trying to find and address all defects, whereas a prevention organization is typically
more optimistic, focused on the future, and deliberate in its approach to improving quality.
We want to help you create your own prevention organization. There is an old adage that
states, “If you always do what you’ve always done, you’ll always get what you’ve always got.”
That will not happen in a prevention organization because a prevention organization concen-
trates on changing the processes and conditions that lead to mistakes in the first place.
Stephen R. Covey in The Seven Habits of Highly Effective People,1 talks about the importance of
renewal and “sharpening the saw.” He states, “Renewal is the principle—and the process—that
empowers us to move on an upward spiral of growth and change, of continuous improve-
ment.” A prevention organization is focused on sharpening the saw—continued renewal
through improved tools, processes, and people.
Using Defect Prevention Techniques
With all due respect to F. W. Taylor,2 there is no “one best way” to prevent defects in software.
Over the years, a variety of prevention techniques have been developed for use in different
stages of the software development cycle. The purpose of all these techniques is to help pre-
dict, identify, detect, and ultimately prevent defects before they can affect users.
The techniques can be grouped into three categories based on their goals, testing focus, and
how proactive or reactive they are within the software development process. The three catego-
ries are as follows:
Defect detection techniques
Defect analysis techniques
Defect prevention techniques
(Please note that defect prevention techniques, as used here, is a subcategory of specific tech-
niques in the overall category of defect prevention.)
Defect Detection Techniques
In terms of software development, a wide variety of testing techniques can be used to identify
software defects. Functional testing, boundary testing, low-memory tests, code inspection,
and end user beta testing are all effective techniques. Defect detection techniques such as
these are used in most software development projects to discover defects and improve prod-
uct quality. Software testing is an extensive topic and the subject of many excellent books,
1 Stephen R. Covey, The Seven Habits of Highly Effective People (New York: Free Press, 1989).
2 F.W. Taylor, also known as the “father of scientific management,” advocated the analysis of work using time
and motion studies to determine “the one best way” to do something.
12 Part I Introduction to Defect Prevention
such as Hunting Security Bugs1 and the How to Break Software series,2 so these testing
techniques are not explicitly covered in this book.
However, a byproduct of software testing and subsequent error reporting is extensive
information about defects and defect patterns that can be analyzed to identify root causes and
how to eliminate or mitigate similar defects in the future. This information can feed directly
into the defect analysis techniques.
Defect Analysis Techniques
The primary goal of defect analysis techniques is to learn from defects that have already been
discovered and to use that learning to further improve the quality of software and even the
productivity of those who build it. Defect analysis techniques help by applying the data gath-
ered during traditional product development to ongoing personal and process improvements
and defect prevention. The goal is to analyze defects, determine their root causes, and then
develop ways to improve the development process to eliminate them. If the defects cannot be
eliminated, the goal must be to create awareness of the defects, identify mitigation techniques,
and then implement the mitigations to reduce the impact of these defects on users. Examples
of defect analysis techniques are defect classification using defect taxonomies, root cause anal-
ysis (RCA), and stochastic modeling. (See Chapter 9, “Stochastic Modeling,” and Chapter 11,
“Root Cause Analysis,” for more information).
Defect Prevention Techniques
The primary goal of defect prevention techniques is to anticipate and prevent defects proac-
tively before they can occur and cause failures or confuse users. This is the best approach
because a defect that does not occur is also a defect that need not be caught, fixed, and sup-
ported. The savings resulting from the successful application of defect prevention techniques
can be reapplied elsewhere in the product cycle. Examples of defect prevention techniques are
failure modes and effects analysis (FMEA), fault tree analysis (FTA) (see Chapter 13, “FMEA,
FTA, and Failure Modeling,” for more information), and use of the Prevention tab (see
Chapter 14, “Prevention Tab,” for more information).
Choosing Quality Improvement Techniques
Choosing quality improvement techniques requires careful consideration of project and orga-
nizational factors. Although each situation is different, the goal is to achieve the maximum
benefit of defect prevention in an acceptable amount of time and using the available
1 Tom Gallagher, Lawrence Landauer, and Bryan Jeffries, Hunting Security Bugs (Redmond, WA: Microsoft Press,
2006).
2 James A. Whittaker, How to Break Software (Boston, MA: Addison-Wesley, 2003); James A. Whittaker and Her-
bert H. Thompson, How to Break Software Security (Boston, MA: Addison-Wesley, 2003); Mike Andrews and
James A. Whittaker, How to Break Web Software (Boston, MA: Addison-Wesley, 2006).
Chapter 1 Defect Prevention 13
resources. Figure 1-3 provides a general effort vs. benefit comparison of some commonly used
defection prevention techniques.
Figure 1-3 Effort vs. benefit comparison chart
In general, results improve as the investment in and rigor of the techniques increase. The
downside is that formal and rigorous techniques also require more time and effort invest-
ment. As a result, with limited time and resources available, there is a tendency to use the
less rigorous techniques, including brainstorming sessions and review meetings. Although
these can be effective, the results are often highly dependent on the knowledge, skill, and
experience of the participants.
Factors to Consider
No single technique can prevent all defects. A combination of techniques, requiring varying
amounts of knowledge, experience, resources, time, and effort, can be employed to promote
the design and development of high-quality software. Because time and resources must be
determined and allocated up front, and because they are almost always in short supply,
successful defect prevention programs require strong organizational commitment.
Occasional
results
Typical
results
Brainstorming
Three smart
people in
a room”
Targeted Reviews
Design, Code
Failure Analysis
Causal Analysis
Meeting
Proactive
Failure Modeling
FMEA, FTA
Periodic
Root Causes
Analysis
Quarterly Review,
Postmortem
Continuous
Root Causes
Analysis
Daily Review
Maximum
Moderate
Minimal
None
Low High
Typical Time and Effort Involved
Defect Prevention Techniques
Effort-Benefit Comparison
Typical Benefit
14 Part I Introduction to Defect Prevention
Be sure to consider the following factors when choosing among defect prevention techniques:
People resources Which people will be available to perform the defect prevention
technique? Have they ever used this technique before? Do they have the right skills and
knowledge, or will they need to be trained?
Tool resources Are the right technique-specific tools available to help guide people
through the defect prevention effort? If the right tools are not available, how long will it
take to obtain and implement them? Is tool-specific training available to the people who
will be using the tools in a defect prevention technique?
Time resources How much time is available to the overall organization and key indi-
viduals in particular to complete the software development project? Are there conflicting
priorities on how to allocate the time that is available? What stage of the product devel-
opment life cycle is the project in, and how will that affect available time?
Organizational commitment Is management, in particular senior management, asking
for the defect prevention results? Is defect prevention part of the standard engineering
process in the organization? In product teams, how much support really exists for per-
forming defect prevention techniques, especially those that are more involved and
require more engineering rigor? When push comes to shove, will defect prevention
efforts be always postponed or cut from the schedule?
Selecting a Strategy
Defect prevention techniques can be delineated by the resource or time investment required,
by the short- or long-term effectiveness, and by the stages of the product cycle. Organizations
should determine where their greatest needs lie, what their quality goals are, and the level of
commitment to quality assurance versus ongoing quality improvement before deciding where
and how to invest in defect prevention activity. As a result of this assessment exercise, organi-
zations can set realistic goals for defect prevention investments and determine appropriate
strategies to achieve them. Depending on the level of investment, here are some alternative
strategies.
Best Strategy
Prevent future defects. Focus on techniques that help anticipate potential defects, and use
these techniques to eliminate or mitigate possible failures proactively. In this strategy, the
primary use of testing is to confirm software quality.
Good Strategy
Focus on the techniques that can help detect and analyze defects, and use analysis informa-
tion to identify underlying causes. Implement improvements to eliminate those causes and
mitigate the impact of defects before users are affected. In this strategy, the primary use of
testing is to ensure software quality.
Chapter 1 Defect Prevention 15
Bad Strategy
This strategy might represent the status quo based on how customers often perceive software
quality today: Allow defects to escape, let users find them, and then react to the complaints.
Use inadequate detection and prevention techniques or use adequate techniques half-heart-
edly. Allow indifferent organizational commitment to hinder the achievement of expected
quality benefits.
Organizational Considerations
Organizations must realistically assess their level of commitment to and expectations of the
defect prevention process. For example, many organizations want to achieve very low defect
rates like those of the NASA Space Shuttle program and the airline industry but resist the
investment, effort, and engineering rigor required to achieve such a high level of reliability
and performance. Other organizations want to eliminate defects by just putting “three smart
people in a room” to discuss an issue and quickly devise ways to resolve it. Unfortunately,
although the brainstorming approach is quick, requires minimal effort, and can be moder-
ately effective, it typically isn’t very thorough.
At the other end of the spectrum are more robust defect prevention processes such as formal
root cause analysis (RCA) and Six Sigma. These are more rigorous and typically based on gath-
ering and analyzing objective data. These formal approaches can garner results that are signif-
icantly better but can also require much more time, effort, and organizational investment.
Therefore, organizations must determine their real level of commitment to defect prevention
and set their goals and expectations accordingly. In most cases, what organizations get out of
defect prevention is determined by the effort and investment they put into it.
Moving Quality Upstream
In a typical software development project, the test team becomes involved late in the process
to find defects and “test quality into the software.” Unfortunately, as mentioned earlier in this
chapter, the later a defect is discovered, the more expensive it is to repair and the greater the
cost and impact on the overall project.
Consequently, if defects cannot be avoided altogether, a fundamental goal of a successful
defect prevention effort is to move quality verification and improvement to an earlier stage in
the software development cycle. Focusing on quality in the planning, design, and early devel-
opment stages pays big dividends later in the cycle. By moving quality assessment and
improvement “upstream” in the software development process, the test team can focus more
on the end user experience and on integration-level testing, rather than finding design or
functional errors. See Chapter 17, “Moving Quality Upstream,” for more information.
16 Part I Introduction to Defect Prevention
Learning from Mistakes
It is important to implement a process that individuals and teams can use to learn from their
mistakes. A fundamental aspect of this learning is the classification of defects using a logical
defect taxonomy. With a structured taxonomy, an organization can analyze and learn about
the types of defects that have been discovered and their relative frequencies. Focusing root
cause analysis techniques on understanding why defects have occurred provides insight into
what improvements are needed to prevent or mitigate those defects in the future.
On the other hand, without a feedback process in place to ensure that the team can benefit
from the learning that takes place during defect detection, analysis, and repair, the likelihood
of repetitive errors is high. Consequently, a goal should be that if a certain type of defect is
going to occur, it should occur only once before safeguards are deployed to prevent it from
recurring. See Chapter 10, “Defect Taxonomies,” for more information.
Investing for the Future
It is important to reiterate that organizational commitment dictates which of these defect pre-
vention techniques can be successful. An organization may desire to invest in rigorous defect
prevention techniques that promise more complete and effective results, but the desire must
be supported with a commitment to invest. If there is no willingness to allocate the necessary
time and resources, or if the company culture resists using the more rigorous techniques, the
likelihood of successful defect prevention efforts is reduced.
An important first step in securing organizational commitment and overcoming resistance is
to clearly demonstrate the potential return on investment. Each defect prevention technique
requires time and resources, and people will want to know how much benefit can be expected
from the effort. See Chapter 3, “The Economics of Defect Prevention,” for more information.
Conclusion
Consider this:
A defect prevented will never need to be fixed, saving time, resources, and money for
your organization.
A defect prevented will improve the quality of your software and help increase customer
satisfaction.
A defect prevented will not impact customers and decrease their satisfaction with your
software.
These simple statements represent why we wrote this book and the value we hope you receive
from reading it.
Chapter 1 Defect Prevention 17
Although the benefits can be significant, the effort often required to achieve high levels of
defect prevention can be significant as well. Our hope is that the practical defect prevention
techniques described in this book will help you minimize the effort, maximize the benefit, and
more rapidly become a “prevention organization.”
In this chapter, you were introduced to the topic of defect prevention including the concept of
software defects, why they occur, and what can be done to prevent them. The other chapters
in this section continue to lay a foundation for defect prevention by covering defect prevention
frameworks, economics, and the concept of software quality. The remaining sections in the
book focus on the practical detection, analysis, and prevention techniques to consider for
your own organization.
19
Chapter 2
Defect Prevention Frameworks
It is theory that decides what can be observed.
—A. Einstein
He who loves practice without theory is like the sailor who boards ship without a rudder
and compass and never knows where he may cast.
—Leonardo da Vinci
You may question why one of the opening chapters of a text targeted at practice would be
about process theory. The best answer comes from W. Edwards Deming, who is often referred
to as the father of process improvement. Deming stated that “experience alone, without the-
ory, teaches . . . nothing about what to do to improve quality and competitive position, nor
how to do it.” Deming continued, “Experience will answer a question, and a question comes
from theory.”1
A framework, like a theory, provides a means to ask questions. A framework differs in that
the incomplete portions are explicitly incomplete requiring each user to modify the
underlying theory contained in the framework. A process framework provides the skeleton
of a theory that can be filled in by the user of the framework. In 1830, Auguste Compte wrote
“A Course in Positive Philosophy” and proposed a framework for learning and the progres-
sion of science, formalizing it into the Law of Three Stages.2 The law proposes the following
hierarchy of knowledge growth for a scientific discipline:
1. The theological stage, where belief and religion dominate
2. The metaphysical stage, where philosophy dominates
3. The positive stage, where scientific reasoning dominates
Compte points out that “facts cannot be observed without the guidance of some theory” and
“no real observation of any phenomena is possible, except in so far as it is first directed, and
finally interpreted, by some theory.”
Much of the history of software development has relied on the first stage of learning, using
belief as a means to determine the correct process. This is not surprising because D. N.
Perkins found a positive correlation between intelligence and the ability to justify one’s point
of view, and a negative correlation between intelligence and the ability to consider others’
points of view.3 Thus, once a belief such as “increasing quality will increase cost” takes hold,
1 W. Edwards Deming, Out of the Crisis (Cambridge, MA: MIT Press, 1982).
2 Gertrud Lenzer, ed., Auguste Comte and Positivism: The Essential Writings (New York: Harper Press, 1975).
3 D.N. Perkins, The Mind’s Best Work (Boston: Harvard University Press, 1981)
20 Part I Introduction to Defect Prevention
it is reinforced with each subsequent project, and the ability to consider other perspectives is
reduced. Even highly intelligent software developers can find it difficult to alter such a belief.
Currently, there is a movement toward the second stage, the philosophical stage, with the so-
called Agile processes leading the way. The philosophy is codified in the principles of the
Agile manifesto.1 The Agile manifesto attempts to organize the philosophy by which Agile
processes such as Extreme Programming and Scrum operate. It defines principles such as
“Continuous attention to technical excellence and good design enhance agility.” This neces-
sary second step in the evolution of software process provides the means for dissolution of
old beliefs, but it does not result in measurable improvement to the engineering system. Thus,
this is the rationale for Deming’s statement that theory must be an integral part of experience
to eliminate dogmatic beliefs and advance the engineering system. Without theory, there is
superstition and philosophy, but no science.
Different frameworks tend to focus on different aspects of process improvement. This chapter
provides guidance on choosing and then enhancing a framework that your organization can
use to ask the right questions with respect to process improvement and defect prevention. Of
course, the right questions are those that lead to measurable improvement and the dissolu-
tion of superstition. This chapter also describes some of the basic attributes of several popular
frameworks. The purpose is to add scientific rigor to the practical application of theory.
Examining a Sample Framework
Theory need not be complex. In fact, the best theories are often deceptively simple. To
illustrate the point, consider a simple theoretical framework for questions you may ask about
a subject as described in the Introduction and used throughout this book. The questions fall
into the categories shown in Figure 2-1. The theoretical framework provides a means to exam-
ine a subject area. For example, you may expect that a practical guide provides extensive
knowledge in the How category. However, unless the topic is obvious or trivial, the model sug-
gests that there must be a justification for the subject, so it seems natural that the book must
cover Why to some extent. As was previously discussed, What is analogous to theory, and
without it, there can be only belief, so even a practical guide must provide a theoretical basis.
Therefore, according to this model, a practical guide to any subject must cover much more
than simply a list of useful techniques.
Figure 2-1 Hierarchy of questions
1 See www.agilemanifesto.org/principles.html
.
How
What
Why
Chapter 2 Defect Prevention Frameworks 21
Such a framework serves to assist in planning. Filling in the missing pieces of the framework
provides a structure for the project much as a theory provides the basis for scientific inquiry.
Notice also that it provides a means to question, that is, hypothesize, about the elements of
the model. This simple three-question model is useful as a means to assess the coverage of a
subject area. You may look at a variety of textbooks and gauge how well those books answer
these questions. You also can use the model to categorize textbooks. A theoretical subject may
include more material on What rather than How with little, if any, coverage of Why. On the
other hand, a self-help book may include extensive coverage of Why. The model provides a
means to categorize and plan research activities.
Notice that the example framework uses a subset of all possible questions. It is by no means
all encompassing and provides only partial guidance for the subject material. Kuhn, in his
examination of the structure of scientific theory, reports that a good paradigm actually is not
complete.1 The incomplete nature of the paradigm allows for scientific inquiry into the
missing pieces.
For example, after using the simple three-question paradigm, you can enhance it to include a
time-ordered progression of questions such as “Why, What, How” to provide more guidance
while at the same time restricting content. You can further enhance the model by including
more questions, such as “When” or “Who.” However, enhancement occurs only after the base
model is found to be insufficient through experiment.
If a researcher examines a number of textbooks and finds the need for more categorization,
only then would another category be added. The model provides a basic theory by which a
researcher can guide his or her thinking so that the decision to add more complexity is not
based on only the researcher’s beliefs. Model enhancement uses good science. Propose a
model, test it, and then change it only after it is found to be lacking.
Proposing a Model
It is not always necessary to create a new framework because there often is no end to the
number of models proposed by academia. Picking a reasonable process-improvement model
requires understanding the content of the model. To determine What, a basic understanding
of various existing models is required. Some models target software development, while oth-
ers are general improvement paradigms. Often, the first model used for process improvement
is a somewhat arbitrary choice to assist in the planning effort. After an organization gains a
detailed understanding of the model, it is easier for the organization to enhance or change to
another more suitable model. The next sections describe a few of the more popular models
that can be used for process improvement.
1 T. Kuhn, The Structure of Scientific Revolutions (Chicago: University of Chicago Press, 1962).
22 Part I Introduction to Defect Prevention
Defect Prevention Model
The model used in this book considers the degrees to which a defect is detected versus pre-
vented. Certainly, detection techniques prevent the customer from finding a defect, so they
qualify as a rudimentary form of defect prevention. Figure 2-2 is the conceptual hierarchy and
goals of each level. The three levels of defect activity contain a number of subactivities that
satisfy the main category goals: defect detection, defect prediction, and defect prevention.
Figure 2-2 Defect prevention model
Defect detection has the goal of finding all defects before they are found by the next phase in
the process. You may ask why such a practice is a prevention practice. In the strictest of terms,
it is not, but most organizations begin with a practice of detection, so the model begins at this
point, too. At this level, the organization is mostly reactive to defect reports. Practices focus on
testing the product after defect creation. The two-step model presented in Chapter 9, “Stochas-
tic Modeling” (see Figure 9-16), indicates such a culture. In some organizations, simple inspec-
tion techniques help to augment the test process. However, the focus is still reactive to defects,
often resulting in inspections that occur after the test process completes.
From a culture of detection, the organization moves to one of analysis. The end goal of analy-
sis is to predict areas that need attention either in the form of more removal processes or in
the form of process improvement. This level requires good measurement techniques, which
means the data is collected to answer specific questions associated with organizational goals
(see the discussion of the Goal-Question-Metric approach in Chapter 7, “Software Measure-
ment and Metrics”), and the reports are available to those who need them. Finally, this level
makes use of prediction models, such as those made possible by stochastic modeling (Chap-
ter 9). The models predict product and process quality.
The third level assumes a culture of prevention, where defects are not introduced in the first
place and where existing defects are not allowed to escape the creation process, that is, the
defects are not allowed to escape the engineer’s office. Because human beings create code, it is
Prevention
Escape Prevention Techniques
Data Modeling
Data Reporting
Data Collection
Simple Inspection
Test-centric Processes
Prevention
Prediction
Detection
Chapter 2 Defect Prevention Frameworks 23
not reasonable that it be perfect when written. It is, however, reasonable that the person
creating the defect should also remove it. Practices at this level include prevention
techniques such as formal design methods and escape prevention. An example escape
prevention technique is a formalized inspection process that uses process controls such as
review rate and predicts the number of escaped defects using a capture–recapture
methodology (see Chapter 20, “Leveraging Good Processes”).
Capability Maturity Model
One of the oldest and most thoroughly researched of all software models is the Capability
Maturity Model (CMM), first described by Watts Humphrey (1989).1 The original intent of
the model was to assess the capabilities of an organization against a common set of necessary
development processes. However, the model is also useful as a road map for improvement as
a result of the arrangement of the practices. As the model’s name implies, it describes the
degree of process maturity an organization has with respect to required elements of the
model. Here we describe the elements of the model. Literally thousands of companies use
the model to assess organizational capabilities.
Model Structure
The CMM is an extremely rich, well-researched, and hence complex improvement model.
However, conceptually, it is very simple. It consists of five levels of process maturity each of
which has a purpose. Levels build on the capabilities mastered at previous levels. Each level
has a number of key practice areas (KPAs) that together fulfill the purpose of the level. Each
KPA has a set of goals. To have a disciplined process, an organization must show commitment
to the process goals, have the ability to carry out a set of activities, and measure and verify
adherence to the processes. The commitments, abilities, activities, measurements, and
verifications form the What associated with each KPA.
As stated earlier, the model contains a wealth of information, but the downside is a temptation
to implement the activities outlined in the KPAs directly as processes. Individual KPAs must
not be thought of as processes but only as a way of assessing a process. In an attempt to imple-
ment the CMM as a How, organizations may become frustrated with the models, and the
effort will likely fail. Figure 2-3 shows the level hierarchy and target process improvement
areas for each level.
1 W. S. Humphrey, Managing the Software Process (Reading, MA: Addison-Wesley, 1989). For those readers not
familiar with Humphrey, he joined Carnegie-Mellon’s Software Engineering Institute after 27 years leading
IBM’s development organization and is a National Medal of Technology Laureate for his contributions to
software development and production methodologies.
24 Part I Introduction to Defect Prevention
Figure 2-3 Capabilities Maturity Model hierarchy
Levels of Process Maturity
Level 1 is politely referred to as Initial, or sometimes Ad Hoc. An ad hoc process is one
designed for a specific purpose and is typically intended for only one use. Once implemented,
the exact rationale for the process may be forgotten, causing other projects to adhere to an
unsuitable practice. The problem with level 1 organizations is obvious: they spend too much
time inventing working processes or using unsuitable ones. Because most of the world’s soft-
ware is developed by level 1 organizations, the level does not indicate the ability to be success-
ful in the market. However, the waste associated with using unfit processes means that most
organizations could be much more profitable.
At level 2, an organization is said to have a Repeatable process. This means that skills
mastered on one project are directly transferable to the next. At level 2, the skills are directed
Quality Management
Quantitative Process Management
Level 4
Peer Reviews
Intergroup Coordination
Product Engineering
Integrated Software Management
Training
Organizational Process Definition
Organizational Process Focus
Process Change Management
Technology Change Management
Defect Prevention
Level 5
Level 3
Configuration Management
Quality Assurance
Subcontract Management
Project Tracking and Oversight
Project Planning
Requirements Management
Level 2
Level 1
Chapter 2 Defect Prevention Frameworks 25
toward individual teams and the management of those teams. Figure 2-3 shows the practices
associated with level 2 organizations and those associated with good project management.
Organizations at level 2 establish basic project management processes “to track cost, sched-
ule, and functionality” and provide the discipline “to repeat earlier success on projects with
similar applications.” The intent at this level is to understand the capabilities associated with
the organization and the relationships in project planning. Important project management
issues such as requirements management and configuration control are planned in advance,
but a level 2 organization is still reactionary for most management issues. Although project
management skills can reduce some costs, most of the benefit is realized through an under-
standing of each team’s capabilities, hence the name Repeatable. An organization can repeat
past successes and avoid known failures at the organizational level.
Level 3 is called the Defined level. Organizations at level 3 develop standardized processes
that can be used organization-wide. By using standardized processes, level 3 organizations
can take advantage of learning of each project team and spread it to other project teams. This
does not mean that all teams use the same process but rather that all teams use a version of
the standard process proven capable for the project of interest. A standard management pro-
cess based on best practice coordinates multiple projects. The organization assesses its pro-
cesses against standard models to determine capabilities. Notice that the need to first
accomplish level 2 capabilities becomes obvious when reviewing the capabilities of level 3
organizations. Without the basic project management techniques of level 2, standardization
of best practice is not possible.
At level 4, organizations Manage projects with statistical techniques using the data gathered
by the standard organizational processes. As before, level 3 practices are prerequisite to
achieving level 4. Unless organizations gather consistent data, the ability to use techniques
such as statistical process control (SPC)1 is not possible. Deming warns that statistical tech-
niques are unusable on processes that are not “in-control,” that is, processes with too many
special cause variances.2 Although level 4 does not require that an organization use SPC, it
strongly implies that SPC is useful. Additionally, because using SPC requires processes to be
in-control, the implication is that level 4 companies have controlled special cause variance.
Again, the need for standardization at level 3 becomes obvious. Teams using ad hoc processes
are less likely to have statistical control.
Finally, level 5 companies are known as Optimizing organizations. The practices at this level
focus on improving the standard processes in a scientifically valid manner. The practices
require the use of the information obtained at level 4 and a controlled introduction of new
best practices. Once again, the progression to level 5 requires mastering the abilities of level 4.
As a side note, the practices described in this book are useful for improving level 4 and level 5
capabilities.
1 SPC is a process of determining natural process variance and distinguishing variance inherent in the process
from variance with special causes.
2 W. Edwards Deming, Out of the Crisis (Cambridge, MA: MIT Press, 1982).
26 Part I Introduction to Defect Prevention
Key Practices
We do not examine the key practice areas in detail. Instead, we describe the structure of a KPA
so that you can understand the elements of building a process model. Table 2-1 provides a list
of required practices and a brief description of the desired outcome of CMM KPAs.
Table 2-1 CMM Key Practices
Level Description Practice Area Desired Outcomes
1 Ad-hoc - Processes are
created on an as-needed
basis and often outlive their
usefulness.
None
2 Repeatable - Team
processes are defined and
past successes are repeatable
Requirements
Management
Requirements are documented
and changes are made known
to everyone affected.
Project Planning Reasonable plans are made
and plan assumptions docu-
mented. Changes find their
way into all affected teams.
Project Tracking and
Oversight
The plans are used to control
the work of the teams. Any
changes will be made known
and be agreed upon by those
affected.
Subcontract
Management
Any work subcontracted to
other organizations is man-
aged in a repeatable manner.
Quality Assurance The quality of the process used
to build the software is known.
Quality assurance activities
verify that products are built
according to the planned
standards.
Configuration
Management
Work products maintain
version control. Products are
baselined and can be rebuilt
from the change control
system.
3 Defined - Organization-wide
best practices are defined
and data are collected
Organization Process
Focus
An organizational focus exists
for improving process and
sharing best practice.
Organization Process
Definition
A set of best practices are
created and shared.
Training Engineers received necessary
training to build products.
Chapter 2 Defect Prevention Frameworks 27
Figure 2-4 shows the CMM structure as defined by the model’s authors.1 Each KPA has a set
of attributes known as common features, which define the structure of a KPA. As mentioned
earlier, a KPA has five features: commitments, abilities, activities, measurements, and
verifications. The activities clearly define the What of the process framework.
Integrated Software
Management
The engineering activities and
the management actvities are
integrated into a single
process.
Software Product
Engineering
Well-defined methods and
tools are integrated into the
process.
Intergroup Coordination All aspects of product
engineering are participate to
build a product to satisfy the
customer needs.
Peer Reviews Peer reviews are used
effectively to remove defects
from the product.
4 Managed - Processes are
controlled using the process
data
Quantitative Process
Management
The process is managed using
statistical methods.
Quality Management The quality of the product is
planned in much the same
manner as the schedule.
5 Optimizing - Process are
continuously improved
Defects Prevention Defect prevention activities are
planned. Root causes of
defects are found and
eliminated from the process.
Technology Change Man-
agement
New tools are proactively
identified and transitioned into
the organization
Process Change Manage-
ment
The software process is
continuously improved to
reduce cost, improve quality,
decrease cycle-time.
1 M. C. Paulk, C. V. Weber, B., Curtis, and M. B. Chrissis, The Capability Maturity Model: Guidelines for Improving
the Software Process (Reading, MA: Addison-Wesley, 1994).
Table 2-1 CMM Key Practices
Level Description Practice Area Desired Outcomes
28 Part I Introduction to Defect Prevention
Figure 2-4 Key practices
Measurements may also fall into a What categorization, but an examination of the actual text
leads to some doubt. The other three features clearly do not provide guidance for a software
process. Their purpose is instead to provide organizational guidance for implementation of
the process and assimilation into the organizational culture. If you are simply looking at pro-
cess improvement, the implementation features do not define what must be done with the
process.
The activities define elements of a process but not the process. The activities cannot be turned
into a step-by-step procedure even with the help of the subactivities provided. For example,
activity 7 under the Project Planning KPA has 10 steps that appear to indicate a process and
may even appear to organizations at level 1 to be a reasonable process. However, you must not
attempt to read it as a process because that would result in a very poor implementation.1
Capability Maturity Model Integration
The CMM was described first because it is a simple model that can be applied to the process
of software development. One rationale for the more complicated Capability Maturity Model
Integration (CMMI) model is to support organizations that are more complicated such as
those with separate software and system engineering functions. The CMMI adds more infor-
mation to the model as well as changes the maturity structure. Several names of levels change
in an attempt to clarify terminology. For example, level 2 is renamed Managed, and level 4 is
called Quantitatively Managed.
The CMMI renames the concept of levels as stages. In addition, there is a model representa-
tion known as Continuous that acknowledges that an organization may want to work on more
than one level simultaneously. The price paid for the added flexibility is a need to understand
the dependency between KPAs. Novice users may not fully grasp dependencies between
1 For a good implementation of the Project Planning KPA, see any of the books by Watts Humphrey referenced
throughout this book.
Commitments Abilities Activities Measurements Verification
Key Practice
Chapter 2 Defect Prevention Frameworks 29
various stages of process improvement. For example, you must have controlled processes
before attempting to use techniques such as Statistical Process Control. Implementing in the
wrong order can do more harm than good. The CMM controls the dependencies implicitly by
assuming implementation of each level until the KPA goals are fulfilled. This makes for a sim-
pler improvement road map.
The CMMI changes some of the KPAs to provide a more detailed business model. For
example, the CMMI adds a KPA for decision analysis and one for causal analysis. Although
such KPAs may be outside the scope of software development, they are highly desired
elements of a software development organization.
Malcolm Baldrige Framework
The Malcolm Baldrige (MB) award provides another process model similar in some ways to
the CMM but very different in others. The Malcolm Baldrige National Quality Improvement
Act of 1987 provides for recognition of outstanding quality in goods and services by the pres-
ident of the United States. The Malcolm Baldrige National Quality Award (MBNQA)1 requires
performance excellence along the following criteria categories:
Leadership
Strategic planning
Customer and market focus
Measurement, analysis, and knowledge management
Workforce focus
Process management
Results
Figure 2-5 shows the model associated with the MBNQA. It contains seven categories and
the associations between categories. Measurement forms a basis that all of the categories
build upon, and the Organizational Profile is the overarching description under which the
organization is assessed. Leadership, strategy, and customer focus affect the workforce
and organizational practices that lead to results.
The seven criteria categories are broken down into Items and Areas to Address, much as the
CMM levels are broken down into KPAs, and, coincidentally, there are 18 total Items, which is
the number of KPAs that exist in the CMM. The assessment of an organization uses points for
each of the Items, with a possible 1,000 total points available. The category of Results is nearly
half the total (450 out of 1,000 available points), indicating the emphasis placed on seeing
actual organizational benefits.
1 See www.quality.nist.gov/PDF_files/2007_Business_Nonprofit_Criteria.pdf.
30 Part I Introduction to Defect Prevention
The goal of Leadership is to assess the means by which senior leaders guide and sustain the
organization, the general governance process, but also the social responsibility of the organi-
zation. The organizational governance includes management accountability, fiscal account-
ability, operations transparency, independence of auditors, and protection of stakeholder
interests. The social responsibility of the organization assesses the organization’s ethical and
legal behaviors as well as the relationship between the organization and key communities
with which it interacts.
Figure 2-5 The Malcolm Baldrige framework
Strategic Planning assesses how the organization creates strategic objectives and their action
plans. Included in the planning is the means by which plans are deployed and changed if nec-
essary. The means by which information is gathered is also examined. The assessment explic-
itly examines the means for predicting performance from the data. Strategic planning may
include both formal and informal means.
The category of Customer and Market Focus examines how the organization determines
customer needs. Associated with this category is the customer relationship practice of the
organization. The assessment includes market segment analysis and use of tools such as voice-
of-the-customer. Customer relationships include the building of relationships and the
assessment of customer satisfaction.
Measurement, Analysis, and Knowledge Management considers the means used by the
organization to select data for gathering, the means used to gather the data, and the means
used to analyze the data. Additionally, the means used to improve the measurements and
the information management techniques are analyzed. The employee performance appraisal
system falls into this category, as does the information technology used by the organization.
1
Leadership
2
Strategic
Planning
5
Workforce
Focus
7
Results
6
Process
Management
3
Customer and
Market Focus
4 Measurement, Analysis, and Knowledge Management
Organizational Profile:
Environment, Relationship, and Challenges
Chapter 2 Defect Prevention Frameworks 31
Workforce Focus examines the means used to manage the workforce, engage them in the
business, and improve the workforce. Engaging the workforce refers to the means used to
align people with the business mission and strategy. Workforce Focus includes the means
used to develop new leaders.
Process Management refers to the means used to create core competencies. The assessment
includes the means for designing, managing, and improving engineering systems such that
the system delivers the required customer value and creates a sustained technology advantage
for the organization. Also assessed are risk assessment and mitigation strategies.
The Baldrige model has a number of subpractices that the practices described in this book
address. For example, in the area of Strategic Planning, business strategy requires the deploy-
ment of business goals. The Balanced Scorecard presented in Chapter 7 fulfills many of the
needs. The CMM can fulfill the need for Process Management. Stochastic modeling provides a
means for advanced analysis of measurements.
The final category, and by far most important in terms of the assessment points, is Results.
Results are examined for products, services, customer activities, financial measures, work-
force, processes, and leadership. The organization must indicate its position relative to the
market and others in its market segment.
As described in Chapter 12, “Adopting Processes,” many frameworks are not mutually
exclusive. In a sense, the Malcolm Baldrige framework provides a meta-model for many other
frameworks. Each of the categories could be considered its own framework. The difficulty
arises from the relative complexity of the model. For example, the Balanced Scorecard fulfills
some of the elements of Strategic Planning and some of the elements of Measurement,
Analysis, and Knowledge Management. The CMM provides some of the elements of Process
Management, some of Workforce Management, and some of Measurement. It is easiest to use
this model as a check step to an overall goal rather than as a means of deploying processes. In
addition, this model takes a systems approach relating causes, such as leadership, to the
results. It acknowledges the relationships between organizational components, which many
other models do not.
ISO Models
The International Organization for Standardization (ISO) publishes a wide variety of
standards such as those for the threads on screws, telephony protocols, and quality manage-
ment practices. ISO 9000 is broadly applied to any manufacturing process focusing on quality
management practices. ISO 9001 is more often applied to software. It is more general than the
CMM and provides general principles of quality management, not specifics. ISO 12207 may
be a more applicable standard, but it is less well known.
32 Part I Introduction to Defect Prevention
ISO 15504, also known as SPICE, is a maturity model much like the CMM. SPICE stands for
Software Process Improvement and Capability Determination. Processes are categorized
according to the information in Table 2-2.
SPICE provides six levels, as shown in Table 2-3, that begin with level 0, which is the equiva-
lent of the CMM Initial level. The attainment of a level uses an assessment method similar to
the continuous method of the CMMI. An organization receives a percentage score for its
attainment of the maturity at each level. The assessment method attempts to quantify process
capability along the following dimensions:
Process performance
Performance management
Process definition
Process deployment
Process measurement
Process control
Table 2-2 SPICE Process Categories
Process Category Description
Acquisition Processes performed to acquire a product or service
Supply Processes performed to propose and deliver a product or service
Engineering Processes that elicit and manage the customers requirements;
specify, implement, and maintain the software product
Operation Processes performed to provide for the correct operation and
use of the software product or service
Supporting Processes that may be used by other processes at various points
in the software life cycle
Management Processes that contain practices that may be used to manage a
project or process in the software life cycle
Process improvement Processes performed to improve the processes performed in the
organizational unit
Resource and infrastructure Processes performed to provide adequate human resources and
IT infrastructure as required by any other process
Reuse Processes performed to provide reuse opportunities in the
organization
Table 2-3 SPICE Process Category Levels
5 Optimizing process
4 Predictable process
3 Established process
2 Managed process
1 Performed process
0 Incomplete process
Chapter 2 Defect Prevention Frameworks 33
Process innovation
Process optimization
A detailed examination of the model shows its similarity to the CMM. The most noticeable
difference is an attempt to include many of the elements of the MBNQA and the ability to
assess each process for a level.
Other Models
Many other models exist and are applicable to software development organization. SPICE and
CMM are the most directly applicable because they were designed specifically for software.
However, any a good framework meant to guide quality improvement practices is applicable
to the software organization, with some modification. For example, Six Sigma methods use an
improvement model called DMAIC, which stands for define, measure, analyze, improve, and
control. The intent of the method is to provide the sequence of steps required to control any
process. DMAIC is not so much a framework for improvement as a methodology for improve-
ment and should not be confused with the CMM or SPICE frameworks.
Comparing the Models
The CMM is specifically targeted at software development organizations, as is ISO 15504. The
CMMI adds some business processes to the CMM, and SPICE attempts to identify process
dimensions not specific to software development. The Malcolm Baldrige criteria categories are
more holistic and look to solve business problems across all functional disciplines. All models
provide a means to assess competencies and compare the assessments across organizations.
Choosing and Using a Model
The first step in choosing a model is to understand the improvement goals for the
organization. One obvious goal is business need. For example, if an organization must do
business with the Department of Defense, it must use the CMMI and do periodic “assess-
ments,” which is a euphemism for process audits. The organization must get a certain rating
from the assessment. An objective of receiving a positive rating is different from an objective of
lasting process improvement.
After an organization understands its goals, choosing a model is a matter of understanding
the questions a particular model is capable of answering. For example, the CMM provides a
sequence of stages of process maturity. Each stage answers a set of questions. Level 2 answers
the following questions:
Do we have an established set of basic project management techniques?
Do we know the cost of a project?
Can we track schedule?
Can we track functionality?
Can we repeat past successes?
34 Part I Introduction to Defect Prevention
Using a model is much more difficult than choosing one. Using a model requires an
organization to assess its capabilities, looking for both strengths and weaknesses in an honest,
unbiased manner. This can be difficult for organizations new to process improvement because
they may tend to overstate their strengths and understate their weaknesses. One primary
reason for this is the lack of accurate data. Organizations that have practiced process
improvement for a reasonable period may also have difficulty accurately assessing their
capabilities against a model and may understate their strengths and overstate their weaknesses.
The primary reason for this is the abundance of data. To remove the quandary caused by this
double-edged sword of data, you must eliminate the emotional ties to the data and see it as just
data. If a weakness is a side effect of a corresponding strength, you must consider it as such.
Organizational Considerations
As mentioned, models such as the CMM specify only what must be done and do not consider
how or why. Additionally, the CMM provides only cursory organizational guidance. However,
the process model is not independent of the organizational structure in which the process is
used. H. Mintzberg provides a means of modeling the relationship between the environment
in which a business operates and appropriate organizational structures.1 Although the pro-
cess framework does not depend on the environment, the implementation of the framework
does. For this reason, it is important to consider the model provided by Mintzberg when
implementing various processes.
Mintzberg writes that two environmental factors drive various organizational structures. The
first factor is the stability of the environment, and the second is the complexity. Figure 2-6
shows a two-dimensional grid based on these attributes. The y-axis is the complexity of the
environment, and the x-axis represents the stability of the environment. Mintzberg calls the
stability attribute “dynamism” to show that the scale ranges from low to high. From this sim-
ple grid, several types of organizations can be described. For purposes of discussion, only the
extreme points of each axis are considered.
When an environment is both stable and simple, the organization is known as a Machine
Bureaucracy. The primary controlling mechanism is through standardization of work pro-
cesses, which is enforced by a central authority. At first glance, the CMM seems to rely strictly
on such coordination mechanisms leading to the misconception that it is applicable only in
large, traditional bureaucracies. However, the framework is useful in all types of structures, as
will be shown later in Chapter 12.
1 H. Mintzberg, “Structure in 5’s: A Synthesis of the Research on Organizational Design,” Management Science 26,
no. 3 (1980): 322–341.
Chapter 2 Defect Prevention Frameworks 35
Figure 2-6 Mintzberg’s model of environmental attributes
When the environment is complex but still relatively stable, the organization shifts to a state
known as a Professional Bureaucracy. Professional standards replace centrally enforced rules
and regulations. The standardization of skills causes standardized behaviors across the entire
industry. The medical profession is an example of an industry with standardized skills. The
key element of such an organization is the amount of training required to provide the stan-
dardization. The same processes used in the Machine Bureaucracy are useful in a Professional
Bureaucracy, but responsibility for the processes rests with the individual, not with a central
authority.
When the environment is simple but also dynamic, the organization forms a Simple Structure.
A single individual provides the controlling mechanism and is involved in most, if not all,
aspects of running the organization. Typically, processes are not formalized, and coordination
is through direct supervision. This environment is the most challenging for use of a frame-
work such as the CMM.
Finally, when the environment is both complex and dynamic, an organization forms what
Mintzberg terms an “Adhocracy.” The control mechanism is mutual adjustment, where indi-
viduals control their own processes much like in the Professional Bureaucracy. The difference
rests with the ability to deploy many different types of specialties. The framework in such an
organization must encompass the richness of different disciplines. An example of such a
framework is the CMMI, which attempts to merge the systems and the software work.
Adhocracies likely have many frameworks, one for each functional discipline.
Professional
Bureaucracy Adhocracy
Machine
Bureaucracy
Simple
Structure
High
Complexity
Low High
Dynamism
36 Part I Introduction to Defect Prevention
When you decide to implement a framework, you must keep in mind the environment and
the primary controlling mechanisms. It would be unreasonable to expect an organization with
a simple structure to put into place practices more suitable for a professional bureaucracy.
Although not strictly part of any of the models, the organizational structure may influence the
choice of models and certainly the implementation. An organization must also take care to
avoid the tendency of the tail wagging the dog. For example, putting into place a Machine
Bureaucracy will not cause the environment to suddenly become less complex and less
dynamic.
Conclusion
An organization must have a framework for improvement to build a plan for improving.
Established frameworks are available to guide the process but require more effort to under-
stand. Instead, an organization can begin with a simple model and add to it to provide a lower
barrier to entry; this, however, limits the potential improvement opportunity. The software
industry has a history of reinventing solutions to problems solved by other companies and
other industries. Standard models provide for sharing practices and the ability to benchmark
against other companies’ efforts.
Theoretical models, such as the CMM, CMMI, and SPICE, provide the basis for asking such
questions as the following:
“Where is our organization as benchmarked against the industry?”
“What capabilities are we lacking?”
“Have others already solved a particular problem?”
Without a framework, such questions are meaningless. The frameworks provided in this
chapter provide a starting point from which to build. Choosing a framework requires the
organization to understand its business needs and desired outcomes. You can use the tools
provided throughout the rest of this book to solve different problems using different frame-
works. Be sure to understand the context of the process before attempting to use any of the
tools. Context requires a model, even if that model is incomplete.
Einstein said, “Creating a new theory is not like destroying an old barn and erecting a
skyscraper in its place. It is rather like climbing a mountain, gaining new and wider views,
discovering unexpected connections between our starting points and its rich environment.
But the point from which we started out still exists and can be seen, although it appears
smaller and forms a tiny part of our broad view gained by the mastery of the obstacles on our
adventurous way up.” The point of a software framework is to create a view that incorporates
the map of the process environment. The framework begins as the starting point on the
improvement journey, incorporates lessons learned along the way, and provides a road map
so that others can follow.
37
Chapter 3
The Economics of Defect
Prevention
When a management team with a reputation for brilliance tackles a business with a
reputation for bad economics, it is the reputation of the business that remains intact.
—Warren Buffet
Religion and art spring from the same root and are close kin. Economics and art are
strangers.
—Nathaniel Hawthorne
In economics, the majority is always wrong.
—John Kenneth Galbraith
Chapter 2, “Defect Prevention Frameworks,” presented the idea that process improvement
activities have three perspectives: What? How? and Why? This chapter discusses why
organizations must prevent defects and presents a view on how to analyze the changes from a
profitability perspective. This book provides you with a set of tools your software develop-
ment team can use to improve the development process and prevent defects. When you make
improvements, it is important to remain focused on the bottom line: defect prevention
increases the organization’s profitability. This chapter describes the economic benefits of
defect prevention, which may be the single best investment that an organization can under-
take. Every prevention activity reaps continual benefit. Once a defect is no longer possible in
a process, all of the effort normally required to correct it is no longer necessary. Prevention
effort pays off every time a defect does not occur. As long as an organization continues to
develop software, the benefits will continue indefinitely.
Preventing Defects Is Good for Business
It is to an organization’s advantage to measure and improve the software development process
by using economic benefit as the means to gauge the efficacy of the engineering system.
Comprehending the connection between the engineering system and the balance sheet is
vital. Few organizations know the actual cost to develop and maintain software; organizations
may go years without understanding the benefits and costs of various activities in the devel-
opment process. For example, although a company may understand the concepts of budget
and headcount, often it does not know the cost of repairing defects at various phases in the
development process or the actual lifetime cost of a software feature. Some organizations
measure the cost to produce a thousand lines of code, but such measurements are rarely able
to capture the incremental value to the user. In fact, a feature’s value to the end user is often
38 Part I Introduction to Defect Prevention
unknown, unmeasured, and ill considered at design time. It is not surprising that the connec-
tion between engineering process and customer value is tenuous at best.
When an organization decides to make process changes or deploy a new tool to improve the
software engineering process, management often uses experience, a.k.a. gut feeling, as the
basis for making the decision. Sometimes the herd effect—a mentality that causes organiza-
tions to pick the latest well-marketed technology—determines which technology the company
will implement, even though there is little or no scientifically valid research to determine
whether the technology actually improves software engineering systems. (Sometimes creators
of innovations can make a profit simply by consulting with companies on how to implement
the new method.) In such cases, it is no wonder that new tools and new processes do little to
improve a company’s systems. In fact, it is to the consultant’s advantage to provide only min-
imal value to the client, thus ensuring that clients require future process improvements and
securing future consultancy business. Although such a view may seem overly cynical, it does
explain how process change after process change results in little noticeable improvement in
the quality of delivered software.
Because of a lack of process data, the value of improvement activities must be estimated.
However, enough information exists to make a good estimate of the cost of various activities
in the software development process. For example, in Winning with Software, the author, Hum-
phrey concludes that testing consumes at least half of the total cost of the development pro-
cess.1 Further, according to a National Institute of Standards and Technology (NIST) report,2
80 percent of development costs are spent identifying and fixing defects, and the cost of
inadequate testing, as defined by NIST, is estimated to be as high as $60 billion per year. The
study concludes that half of the cost of defects found by the user is borne by users and half by
the developing organizations. The NIST report estimates the direct costs incurred to test prod-
ucts using existing software development processes, which rely on defect detection, not defect
prevention. Assuming the report is accurate, we can conclude that companies that implement
defect prevention techniques can realize up to $30 billion in savings by preventing defects
instead of spending resources detecting and fixing them, not including any savings associated
with an improved development process.
So what are the true costs to organizations that do not practice defect prevention techniques?
To answer this question, a brief background in cost–benefit analysis and economic costs is
required. The next section describes one way of looking at the costs associated with the devel-
opment process and provides a method of estimating the effect of defect prevention on the
value of delivered software. The aim is to provide an economic argument for the value of
defect prevention.
1 Humphrey, W. S. Winning with Software. An Executive Strategy. (Boston: Addison-Wesley, 2003).
2 RTI. “The Economic Impact of Inadequate Infrastructure for Software Testing.” National Institute of Standards
and Technology, Program Office Strategic Planning and Economic Analysis Group. (2002)
Chapter 3 The Economics of Defect Prevention 39
Economic Theory and the Value of Defect Prevention
Software companies are fortunate in that software is a very profitable product. Essentially,
software lasts forever, never wearing out. A copy of DOS that ran a PC twenty-five years ago
works as well today as it did when first shipped. Copies of existing software can be made for
distribution at very low costs. So the nature of software and its development and distribution
leads to high margins and occasionally the view that traditional cost accounting techniques
do not apply. Although a strict adherence to traditional accounting measures do not always
apply to the software industry, they can be modified so that techniques for conducting cost–
benefit analyses and marginal cost analyses can work well to determine expected return on
investment.
For purposes of this discussion, we must define a few terms and equations to provide a brief
explanation of economic measures as they apply to the software industry. This section is
meant to provide a quick introduction to the terminology of economic profitability as used in
this chapter, which differs from its use in accounting. In addition, we do not consider subtle
nuances of economic theory. This section illustrates how you can do a cost–benefit analysis,
and you will likely need to modify it to apply to your specific organization.1
Profitability
It seems obvious that the primary purpose of a for-profit company is to make money.
Although organizations generally also desire to contribute to the community, provide for the
well-being of their employees, and improve their respective industries, these secondary
attributes merely assist in the primary purpose. Milton Friedman2 (1970) even went so far as
to propose that profitability is really the only purpose of a publicly traded company and that
long-term benefit to stockholders must drive all management decisions. Although many dis-
agree with Friedman that profitability is a company’s only concern, profitability certainly is a
powerful driving force. Therefore, the decision to prevent defects must be supportable from
the perspective of the balance sheet and corporate profits.
One model for maximizing profitability uses what is known as the marginal cost-marginal
revenue method. The concept is simple and incorporates several well-known microeconomics
concepts. Marginal revenue (MR) is the revenue realized from making one more sale. Marginal
costs (MC) are those costs incurred to deliver to the marketplace one more unit of product. An
organization’s costs are all costs associated with production, and these fall into two categories.
Variable costs are those associated with producing one more unit. For example, in manufactur-
ing, the cost of raw material that makes up the product is an example of a variable cost. Fixed
costs are those that are incurred regardless of the number of units produced. For example, in
the software industry, engineers’ salaries must be paid regardless of the number of copies of
software sold. Likewise, rent paid for facilities and utility bills are also costs independent of
1 The estimates use data from a variety of sources to illustrate how to perform such an analysis and do not
necessarily reflect data from the authors’ organization.
2 Friedman, M. “The Social Responsibility of Business Is to Increase Its Profits.” The New York Times Magazine
(1970, September 13).
40 Part I Introduction to Defect Prevention
units sold. Because fixed costs are already spent, they are not part of the marginal analysis.
Only the variable costs enter into the decision to produce one more unit of product.
Figure 3-1 shows profit maximization using the marginal cost–marginal revenue approach in
a market with perfect competition. Notice that the cost curve drops as economies of scale
allow for production efficiencies. At some point, however, the costs begin to increase as ineffi-
ciencies associated with large-scale production overwhelm the benefits. At the point where the
marginal revenue (MR) line crosses the marginal cost (MC) curve, the price paid for the unit
sold is exactly equal to its cost. Thus, marginal profit is zero. At this point, selling one more
unit results in a net loss to the organization since the revenue received will be less than the
cost to produce the item. Thus, the company maximizes its profit when it sells exactly quan-
tity Q, occurring at the point MC = MR. Profit maximization using this theory is simple and
intuitive. From the simple graph, the price for the software and the quantity sold are set.
The total profit realized must exceed not only the variable costs but also the fixed costs in
order for the product line to be profitable. The difficulty lies in determining marginal costs
and revenues and relating the net profit to determine of they cover fixed costs.
Figure 3-1 Maximum profit occurs when MR = MC
A key element of cost analysis is known as the opportunity cost of capital, or simply, opportunity
cost. By investing in a particular product line, a company must forgo the opportunity to invest
in some other area; the forgone profit of another alternative is the opportunity cost. Thus,
when calculating potential profit, the profitability of a chosen product line must exceed the
opportunity costs to determine the true economic profit; that is, the profits realized from an
investment must exceed the profits expected from the next best alternative.
To satisfy stockholders, public companies must invest in a portfolio of products that returns a
certain profit percentage. This risk–reward potential of an investment drives investment deci-
sions at the senior management level and is reflected in the stock price of a company. That is,
if the portfolio of products does not provide sufficient reward, investors sell their shares of
stock, lowering the stock price. Therefore, the opportunity cost of capital and the profitability
expectations of a company are directly contained in the price of the stock. Investments in
lower-profit opportunities lower the stock price, whereas realizing profit in excess of the
MR
MC
Q
Price
Quantity
Chapter 3 The Economics of Defect Prevention 41
expected return, as measured by the opportunity cost, raises the stock price. This is the basis
for justifying a defect prevention activity. If the cost of defect prevention can produce a return
in excess of the expected opportunity cost, it benefits the stock price and is a good investment
for the company to make.
Applying Marginal Cost Analysis to Software
Development
Both software development organizations and traditional manufacturing organizations
produce and distribute products. However, software companies differ from traditional manu-
facturing organizations in several important ways that require modification of the methods
used to analyze cost and profit. For example, when a manufacturing organization builds, say,
an appliance, the marginal costs include the costs of raw materials and labor to produce the
appliance. To sell one more unit, more production work is required, which of course increases
marginal costs. However, this is not the case with software. After a software application is cre-
ated, to sell one more unit, a software company need only burn another CD or DVD copy. The
marginal costs are the cost of the media plus the cost of salaries for those involved in the copy-
ing process, as well as any delivery costs—the total cost to produce one more unit is likely less
than one U.S. dollar. And if the user downloads the software from the Internet, the only mar-
ginal cost for the software company is the cost of maintaining the communications network.
This seems to suggest that the traditional economics of marginal costs cannot be applied to
software companies.
However, from a slightly different perspective, traditional analysis can be applied to software
companies. Marginal cost analysis focuses on production of the product. In the software
industry, engineers produce the product when they create code. Therefore, it seems reason-
able that marginal cost analysis for software should focus on the production of the final soft-
ware product and not on the costs of literally selling one more unit, that is, shipping the media
to one more user. In other words, it is more effective to analyze the creation of the code and
consider the marginal cost as the cost of including one more feature in the software. From this
perspective, marginal costs are those costs associated with adding one more unit of functional-
ity to a software product.
The marginal cost–marginal revenue method described earlier can now be applied to
software organizations. The cost associated with adding each feature to the software product
is measured by using the scorecard methods described in Chapter 7, and then the benefit of
each feature are measured using the scenario voting methods described in Chapter 15. This
method is best suited to determining marginal benefits for every feature and is a viable means
for organizations to maximize benefits to end users. However, evaluating the contribution to
profitability for implementing a defect prevention technique is slightly more difficult because
it applies to all features equally. This makes the analysis easier to conceptualize but harder
to measure.
42 Part I Introduction to Defect Prevention
Estimating Costs
From an accounting standpoint, to calculate marginal costs, you must determine the cost of
every feature, determine applicability to the feature of a particular defect prevention activity,
and then calculate the associated benefits. Recall that the goal here is not to provide the busi-
ness analysts with information; it is to determine the efficacy of a defect prevention method.
To do so, you can use a proxy for the feature and perform an analysis based on the proxy.
Because features are implemented with code, lines of code are a useful proxy for functionality.
Although software products contain other functionality, for example, help documentation,
the focus of the typical defect prevention activity is on the feature code. We can reasonably
assume that an analysis focused only on feature code can provide an understanding of the
benefits of defect prevention.
Some people argue that using lines of code as a proxy for functionality is not useful because
not every developer codes alike, not all features are alike, different programming languages
have different issues, and so on. However, all of these objections tend to focus on the difficulty
of comparing features, languages, or possibly even engineers. In the marginal cost analysis
case, two processes are compared with one another, not two features. Presumably, different
processes would result in very similar implementations of a feature, with the only difference
being the process used to create the feature. Lines of code would be a useful means to gauge
the amount of work accomplished. In such a case, the cost of the activities associated with the
process provides the means to judge the benefit of the defect prevention. That is, the time
required to produce 1000 lines of code under process A is compared with the time required
for process B. Using the techniques described in Chapter 7 of this book, you can gather data
to make a reasonably accurate judgment for cost associated with the activities of different pro-
cesses.
For discussion purposes, the method for comparing two processes is based on a simple model
of the development process. The model has five major phases: Requirements (REQ), High-
Level Design (HLD), Implementation (IMPL), Test (TEST), and Release (REL).1 Each devel-
opment phase includes more detailed activities. Table 3-1 provides phase breakdowns for two
candidate processes. The first process primarily uses test to remove the defects and will be
called a test-centric process (TCP). The second relies on defect prevention activities and
inspections so will be called a defect prevention focused process (DP). The phases represent
the work required of a software development project whether or not a process actually pro-
duces these products; that is, all systems have requirements, but not all processes produce
them. In other words, many test-centric processes spend little time in the requirements phase
and often do not produce a thorough work product. The phase exists, in that the product will
have been implemented against some loosely defined requirements. The result is that very lit-
tle time is spent in the requirements phase, and anyone trying to measure the phase may not
even notice that it existed.
1 A worksheet of the detailed cost–benefit analysis is available on the Defect Prevention Web site at
www.defectprevention.org.
Chapter 3 The Economics of Defect Prevention 43
The major activities require effort to produce the final software, even if no intermediate work
product is produced. For example, a product manager may spend time considering various
customer requirements and then verbally communicate the customer needs to the develop-
ment team. The development team uses the verbal communication to produce the code. Addi-
tionally, the work is not necessarily carried out in the order listed, and some work may be
done in parallel. The intent of this analysis is to consider the investment made in each work
product and the associated benefit in the form of quality, that is, defects in the software.
The analysis requires every phase to have a cost probability distribution function (pdf) and
an associated effect on quality pdf. The model of the development activities requires each
step in the process to estimate costs in the form of effort and benefits in the form of quality.
Chapter 9 provides effort estimates based on a process similar to that shown in Table 3-1. The
numbers provided in this chapter are averages across the probability distribution functions.
Every activity includes the possibility of introducing an error into the final product. Some
activities have a tendency to create errors, whereas others have a tendency to find existing
errors. For example, writing code creates coding defects, whereas inspecting code tends to
remove existing defects without creating many new ones. As discussed in Chapter 9, it is
possible to make changes in the process to minimize the defects that escape a product phase.
Table 3-1 Process Comparisons
Test-centric Process Defect Prevention-focused Process
Phase Detailed Process Step Detailed Process Step
Plan Plan
Requirements Write requirements Write requirements
Write system test plans Write system test plans
Inspection Inspection
High-level design High-level design High-level design
Write integration test plans Write integration test plans
Inspection Inspection
Implementation Detailed designs Detailed designs
Reviews
Test case development
Team design inspections
Code Code
Reviews
Compile Compile
Code inspections Code inspections
Unit tests Unit tests
Integration tests Integration tests
Test System tests System tests
44 Part I Introduction to Defect Prevention
A change invariably requires some effort to implement and carries with it the potential of
preventing defects from slipping to the customer release phase. Therefore, the effort required
to prevent the defect is the cost of the investment, and the time saved in later phases is the
benefit.
The next sections examine costs and benefits of two processes, a test-centric process and a
defect prevention focused process. The technique is useful for any process change associated
with defect prevention activities.
Process Cost Estimates
The primary driver of cost for software development is the salary of the engineer. The cost of
the typical computer is very much less than the cost of engineers’ salaries. Additionally, mar-
ginal cost analysis does not usually include the cost of capital equipment, only the cost of
materials and labor. As was already stated, the cost of materials for the product is insignificant
compared with the cost of labor. Determining the cost of defect prevention first requires an
estimate of engineers’ time spent in every development activity and their hourly rate. The
problem is that most software organizations do not have this information available. In the
examples used here, the data is gathered from various teams from a variety of organizations.
Table 3-2 contains a summary of the effort associated with each of the detailed activities for
producing a thousand lines of code (KLOC) for one development organization using a test-
centric process. This organization is a fairly typical example of a company that relies primarily
on testing to find and fix problems. According to Humphrey, such an organization will deliver
to the test phases on the order of 20 defects per thousand lines of code.1 Davis and Mullaney2
found that software released from a typical organization has 6 to 7 defects per KLOC when a
reactive test-centric process is used.3 In a very real sense, the engineer produces defects along
with the code. Think of these defects as a form of warranty liability: there is a chance that they
must be fixed at some future date, and a probability distribution determines the likely cost to
do so.4 As with warranty liability, the product may or may not be defective, which may or may
not generate additional cost to fix the defective product.
From the data in Table 3-2, you can see that the average cost for producing a thousand lines
of code is approximately 128 direct labor hours (DLHrs). This is an estimate of the direct
effort associated with creating the work product (the thousand lines of code).
1 Humphrey, W. S.. Winning with Software. An Executive Strategy . (Boston: Addison-Wesley, 2003).
2 Davis, N., & Mullaney, J. The Team Software Process in practice: A summary of recent results. (Pittsburgh, PA.:
Software Engineering Institute, Carnegie-Mellon University, 2003).
3 Davis and Mullaney related it to CMM level 1 organizations which tend to be test-centric.
4 Every engineer and every piece of code will have different liability curves. See chapter 9 for more on probability
distribution functions.
Chapter 3 The Economics of Defect Prevention 45
Table 3-2 Process Effort Comparisons
Phase % Total Effort Detailed Process Step % Total Effort Total Hours
Test-centric Process
1% Plan 1% 1.3
Requirements 4.20% Write requirements 2.2% 2.9
Write system test plans 0.9% 1.1
Inspection 1.1% 1.4
High-level design 4.2% High-level design 2.2% 2.9
Write integration test plans 0.9% 1.1
Inspection 1.1% 1.4
Implementation 24.6% Detailed designs 2.2% 2.9
Reviews
Test case development
Team design inspections
Code 11.7% 15
Reviews
Compile 2.2% 2.9
Code inspections 0.8% 1
Unit tests 7.8% 10
Integration tests 17.2% 22.1
Test 65.9% System tests 48.8% 62.8
Total 128.7
Defect Prevention Focused Process
4.8% Plan 4.8% 4.4
Requirements 19.9% Write requirements 9.9% 9.3
Write system test plans 5.0% 4.6
Inspection 5.0% 4.6
High-level design 18.1% High-level design 0% 8.4
Write integration test plans 4.5% 4.2
Inspection 4.5% 4.2
Implementation 41.3% Detailed designs 8.2% 7.7
Reviews 4.1% 3.8
Test case development 4.1% 3.8
Team design inspections 3.3% 3.1
Code 7.4% 6.9
Reviews 3.7% 3.5
Compile 1.3% 1.2
46 Part I Introduction to Defect Prevention
The cost for this code depends on the loaded cost of one labor hour. The loaded cost includes
salary, benefits, and other overhead expenses associated with employing a software engineer,
in this case. Most organizations calculate the loaded cost for every job title in the organization.
For discussion purposes, assume a loaded cost of $156,000 per engineering year. This may
seem high, but you must remember that it includes benefits, taxes, and so on, so the cost of an
engineer earning $100,000 per year quickly reaches $156,000. Because there are 2080 hours
in a work year, this yields a loaded cost of $75 per hour.
$156,000 ÷ 2080 = $75
Based on the number of hours required to build 1000 lines of code and the cost per hour, the
direct cost of producing 1000 lines of code is $9600.
128 DLHrs × $75 = $9600 per 1000 lines of code
However, this assumes that the engineer does nothing but develop code 8 hours a day, 5 days
a week, 52 weeks a year, which is unlikely. Some time must be excepted for attending meet-
ings, answering e-mail, taking vacations, observing holidays, and so forth. For purposes of this
discussion, assume that half of an engineer’s working time is spent developing and that other
necessary organizational activities occupy the other half. Therefore, an engineer has roughly
1000 hours available a year for developing software, which means that a typical individual
could produce just under 8000 lines of new code in a year. The cost becomes $19,500 to pro-
duce a thousand lines of code. This amortizes the unproductive time across all lines of code,
that is. $156,000 per year divided by 8000 lines gives $19,500 per 1000 lines of code, or
KLOC.
1000 hours a year ÷ 128 DLHrs per unit of code = 7.8125 × 1000 lines = 7812.5 lines of code
per year
But we must also consider that few engineers produce only new code. Most create new
features while supporting existing features, typically manifested in the form of bug fixes. As
discussed in Chapter 9 on stochastic modeling, an engineer who releases defective code has
less time available to produce new code. As pointed out earlier, the maintenance work is much
like a warranty on the work product. The probability of finding a defect in existing code is
proportional to the number of defects in the product and to the product’s usage.
Code inspections 3.3% 1
Unit tests 5.9% 5.5
Integration tests 7.2% 6.7
Test 16.1% System tests 9% 8.3
Total 93.3
Table 3-2 Process Effort Comparisons
Phase % Total Effort Detailed Process Step % Total Effort Total Hours
Chapter 3 The Economics of Defect Prevention 47
This warranty cost must be added to the cost of software production. Every defect in the
completed product is a potential liability for the engineer who created the software since the
engineer may one day need to fix the defect. The software development test processes will
find some defects, and users will find others. Calculating the liability—the cost—in hours of fix-
ing every defect is necessary, and we use average values in this discussion. Assume that to fix
the average defect detected by testing requires 1 day of an engineer’s time, whereas customer-
reported issues require 1 week to fix. This means a test-detected defect costs 4 hours of valu-
able development time, and a customer issue costs 20 hours.1
As stated earlier, the test team typically will find about 20 defects per KLOC. As reported by
Davis and Mullaney (2003)2, a typical company releases products with 6 to 7 defects in the
final code, and the customer may find about 50 percent of those defects. This means that
every 1000 lines of new code require about 14 to 14.5 test defect corrections and 3 to 3.5 user-
reported defect corrections. The engineer will release 20 defects to the test process, and 6 or
7 will escape to the customer, meaning 14 are caught by test. Of the 6 or 7 that escape, about
half will actually cause the customer a problem requiring correction.
At 4 and 20 hours per fix, respectively, the total cost for the life of the code is 254 hours, effec-
tively cutting the engineer’s productivity in half. The previous Table 3-2 process ended at the
end of system test, so the defects that escaped to the customer were not counted. This shows
up as a hidden cost to the organization that often goes unplanned. So the true rate of new
code production in the organization is about 4000 lines of code per engineer, not 8000. Of
course the numbers will vary for different types of code and different applications. The num-
bers here reflect typical application development organizations.
As an aside, this explains some of the debates on the lines of code that a typical engineer cre-
ates per year. Often, small organizations are credited with higher productivity rates. If an indi-
vidual or an organization does not need to support existing code, or if the number of users is
low, the warranty costs do not exist and engineers’ productivity can double. The more an
organization must support an existing code base and the higher the number of users, the
lower an organization’s effective productivity.
Direct Costs versus Opportunity Costs
The direct cost of quality in the example is $19,500 per thousand lines of code. The direct cost
of quality is the difference between the maximum average lines of code and the lines of code
with warranty costs added. The measurement time frame needs to be long enough to measure
usage and factor in customer-reported defects. However, this represents only the cost of the
engineer’s time and not the impact on the entire organization. That is, it measures direct cost
but not opportunity cost. To determine opportunity cost, the calculation must consider lost
1 Chapter 9 will include various other methods of finding the defect and calculate the effort savings when the
developers detect their own bugs.
2 Davis, N., & Mullaney, J. The Team Software Process in practice: A summary of recent results. (Pittsburgh, PA.:
Software Engineering Institute, Carnegie-Mellon University, 2003).
48 Part I Introduction to Defect Prevention
marginal revenue to determine actual impact. In other words, if the engineer were not fixing
bugs, how much new valuable code would be produced?
As stated earlier, when making an investment decision, an organization must consider the
next best use of the capital employed. In the case of defect prevention, simply comparing a
new process with an existing process is all that is required. We can compare making a change
to the development process with maintaining the status quo.
The test-centric development process incurs some investment costs in earlier development
phases to create the code and the bulk of the costs later in the development process to test
and fix the code. As discussed, a typical developer produces about 4000 lines of code per year.
The assumption here is that new code carries the bulk of customer value. However, existing
code must be maintained as a form of overhead, and the support costs result in even lower
productivity rates than estimated. In the example, the defect prevention focused process
results in more than double the amount of code supported and at least twice as much
valuable new code.
The next step is to determine the effect on the business strategy of doubling the amount of
valuable new code. As mentioned in the section titled “Economic Theory and the Value of
Defect Prevention” earlier in the chapter, an organization’s profit includes an opportunity
cost, which is factored into the stock price by market forces. Therefore, changing the cost
structure for producing software must have an effect on the stock price. Since software costs
half as much to produce, it must somehow increase the value of the stock price by either
increasing revenue or decreasing cost, or both. To determine the amount of the change,
calculating the opportunity cost is necessary.
It is often difficult to determine the opportunity cost for an organization, but we can use a sim-
ple proxy. Organizations typically maintain a certain ratio of developers to non-developers.
For example, Michael Cusumano writes of the Microsoft development process and describes
the organization as consisting of product managers, developers, and testers.1 The three disci-
plines work together in teams in a ratio maintained across the organization. In addition, orga-
nizations need managers, accountants, HR specialists and so on. Assuming that the functions
other than development are staffed to support the development effort, adding 100 new devel-
opers requires some amount of support staff. This fact will be used to simplify the opportunity
cost analysis. Because the number of personnel in other disciplines is a constant fixed against
the total number of software developers, only the total number of developers is germane to
the analysis. The organizational costs associated with areas outside development are some
fixed ratio of the software development cost. This assumption will become important momen-
tarily.
1 Cusumano, M. A., & Selby, R. W. Microsoft Secrets. (New York: Simon and Schuster, 1998).
Chapter 3 The Economics of Defect Prevention 49
Determining the Returns
The final step of the cost–benefit analysis is to determine the possible total organizational
benefit of a process change. Assume an organization has $100 million per year in revenue.
Further, assume that the organization requires 100 developers to produce and maintain its
software. If the organization wishes to add one more developer, revenues must increase by $1
million. The increase in revenue must cover all costs to the organization that increase as the
organization scales up to support one more developer. This includes such variable and fixed
costs as salaries for employees added in other disciplines, office space, computers, benefits,
utility bills, and so on. Of course some core operations such as HR and accounting require
only a fractional increase and adding one more employee is not a problem. However, on aver-
age, the increase in revenue must be $1 million.
In a year, the new developer will produce 4000 lines of valuable code using the test-centric
process. Therefore, each 1000 lines of code is valued at $250,000. The process improvement
associated with defect prevention results in each developer creating about 10,000 lines of
code per year, or $2.5 million worth of value. That is, a thousand lines of code required 93
hours, and 1000 hours of available time means just over 10,750 lines. Factor in post-release
bug fixes, and the number is just over 10,000.
On the surface, this looks like more than double the profitability. However, the organization
requires staffing at the old $1 million level. If the software company was making 30 percent
net profit,1 it was making $1 million on an investment of $700,000 for a net return of more
than 40 percent. That is, the software produced under the test-centric process produced $1M
in revenue, of which 30 percent was profit so $300,000 is profit, and $700,000 was required
to generate the income.2 Return on investment is $300m/$700m, or 42.9 percent.
With the same investment of $700,000 under the defect prevention focused process, the
net profit increases to $1.9 million ($2.5 million worth of valuable code less the $700,000
required to generate it). The net return is then more than 270 percent ($1.9 million/
$700,000). The result is that the new process completely changes the return on every
thousand lines of code.
Costs versus Stock Price
As stated earlier, the opportunity cost is tied closely to the current stock price. It is relatively
easy to determine the effect that improving the process has on the stock price of a firm.
Copeland, Koller, and Murrin propose discounted cash flows as a means to value a company.3
For purposes of valuation, a company is considered to provide a set of yearly cash flows. To
1 Microsoft reported more than $16B of operating income on $44B in revenue. so 30% is reasonable if not low.
2 Remember the prior assumption that most of the cost of development was salary. To validate this assumption,
one need only review the annual report of any large software company.
3 Koller, T., Goedhart, M., & Wessels, D. (1994). Valuation: Measuring and Managing the Value of Companies.
Hoboken, New Jersey: John McKinsey & Company, Inc.
50 Part I Introduction to Defect Prevention
determine the value of the cash flows, subtract the net current assets of the firm from the
current market capitalization. This provides the value of the expected future profits.
Cash Flows = Market Capitalization – Net Current Assets
Because the rate of cash flows changes by 2.7 times (from the prior profitability calculation,
this company makes 270 percent profit per engineer), the new market cap of the company is
New Value = (Old Value – Current Assets) × 2.7 + Current Assets
Divide the New Value by the existing shares outstanding to determine the new stock price.
For example, assume a company has $1 billion in market capitalization and 100 million
shares of outstanding stock. The stock price is easily calculated to be $1 billion / 100 million,
or $10 per share. Further, assume that the company has $250 million in cash and other assets,
with $50 million in debt. The value of the discounted cash flows for this company is $800 mil-
lion, or $8 per share. To get this, simply take the market value of the company and subtract its
current assets. In this case, the net assets are $200 million ($250 million in cash less the cur-
rent debt of $50 million). That leaves $800 million of residual value that must be the value of
the cash flows as determined by the market.1
Now increase the cash flows by 2.7 times. The new value of the discounted cash flows is $2.16
billion. Add in the current net assets of $200 million, and the value of the company is $2.36
billion, or $23.60 per share. Defect prevention activities at this company more than double the
stock price.
Of course it is unlikely that the company would convert all of the new productivity into reve-
nue at the old value. Instead, it is likely that some of the new productivity would be used to
enter riskier markets or accept projects that have lower returns. According to Clayton Chris-
tensen, disruptive technologies are often lower-return technologies when first discovered.2 In
fact, leading companies are often not able to create product lines with disruptive technologies
because their stock prices prevent it.3
Conclusion
Nathaniel Hawthorne said that art and economics are strangers. He was probably correct in
that view, which requires that software be turned into a science if economic analysis is to
mean anything. As an organization gathers data and builds good process frameworks, the
ability to directly connect software design activities to the balance sheet is possible. Using
1 Remember that the market is estimating future cash flows based on history. If the market values the company
above its current assets, it must be valuing the future earning power.
2 Christensen, C. M. (1997). “The innovator’s dilemma: When new technologies cause great firms to fail.” Boston:
Harvard Business School Press.
3 Using reasoning akin to that just described, although exactly the opposite because in this case the profitability
is reduced.
Chapter 3 The Economics of Defect Prevention 51
traditional cost accounting and marginal cost methods, an organization can gain tremendous
insight into processes and the ability to deliver customer value. Additionally, by changing the
focus to defect prevention, additional business strategies are made available.
Christensen describes what he calls “The Innovator’s Dilemma,” which is the dilemma that
occurs when a promising technology is dropped from consideration because of economic
constraints.1 A finding of Christensen’s work is that innovation that disrupts an entire indus-
try rarely, if ever, comes as a surprise to existing firms in the industry. In fact, many incumbent
organizations have already dismissed the new technology as impractical for their existing cus-
tomers. The reason for dismissing the new technology is simple: opportunity cost of capital.
As explained earlier, investors require a specific rate of return on their investments, which in
turn forces an organization to accept a very specific risk versus reward portfolio of projects.
Company management can invest only in projects that yield more than the expected opportu-
nity costs. As Christensen found, innovative technologies often do not provide the required
rate of return, so are not considered. Innovation is the victim of the balance sheet.
However, defect prevention changes the cost of capital requirements, allowing an organization
to accept a different mix of projects. By using defect prevention techniques, an organization
can invest in a different technology mix, including disruptive technologies that might other-
wise be valued at below the cost of capital. Therefore, defect prevention not only saves money
in the short term, it allows for long-term innovation and enables a company to invest in dis-
ruptive technologies. Defect prevention helps to prevent the Innovator’s Dilemma.
From a profitability perspective, defect prevention is possibly the single most valuable invest-
ment an organization can make. Few investments have the potential for 250 percent plus
returns, and as demonstrated in this chapter, defect prevention activities can result in at least
that amount. Additionally, no other activity short of massive reorganization can change the
opportunity cost of capital, which has the potential to change the markets in which an organi-
zation can compete. Defect prevention allows for innovation in risky technologies and pro-
tects a firm’s business strategy by opening completely different markets—no other activity
provides the flexibility to innovate in existing markets while simultaneously opening new
markets.
The natural question you may ask is, “If defect prevention is so valuable, why do so few com-
panies focus formal activities on it?” The answer probably includes fear of the unknown, risk
aversion, and perhaps even ignorance of the potential. Whatever the reason, the first company
to get it right has a competitive advantage over every other company in every other software
segment. Defect prevention itself becomes disruptive to every software segment simply based
on the opportunity cost of capital. Economic profits, that is, profit in excess of the cost of cap-
ital, are rare in a mature field. As the software industry matures, the only advantage an organi-
zation can count on will be implementing defect prevention activities.
1 Christensen, C. M. “The innovator’s dilemma: When new technologies cause great firms to fail.” (Boston:
Harvard Business School Press, 1997).
Part II
Defect Detection Techniques
In this part:
Chapter 4: Quality and the Development Process . . . . . . . . . . . . . . . . . . . . 55
Chapter 5: Using Productivity Games to Prevent Defects . . . . . . . . . . . . . . 79
Chapter 6: Improving the Testability of Software. . . . . . . . . . . . . . . . . . . . . 97
55
Chapter 4
Quality and the
Development Process
This chapter covers two issues: what software quality is and how the development process cre-
ates opportunities for defects to occur. All defects do not contribute equally to different levels
of quality. Software development is a series of transformations from the original vision.
What Is Software Quality?
How do you define software quality? Do you think it’s possible to test a sizable application
completely? How do your processes consider quality? Do you implement and then enter a
test-and-fix phase? As your application becomes larger or more complex, the answers to these
questions can make a difference between success and failure of your product, particularly if
your application is exposed to the Internet.
This section covers the typical model of achieving quality in software, which is based on a
model used for making physical products before mass production was developed, and why
this model doesn’t work well. A model of quality that does apply to software is described as
well as how your programming language allows for defects.
Development Methodology and Quality
Software development is unlike traditional manufacturing processes. In software develop-
ment, only a single instance of the product is produced rather than a single product being
mass-produced in traditional manufacturing. Improvements in modern manufacturing are
applied to the system that produces the products, and when a defect is detected changes are
made to reduce or eliminate the defect for future products. Software produces a single product
that is then mass-duplicated.
Additionally, software has no equivalent of the laws of physics to provide continuity over
different conditions, and the number of decisions made (hence, possible errors) is immense.
For example, a metal beam can be depended on not to liquefy over a wide range of tempera-
tures, whereas a software function can do anything depending on how it is coded. An error
in a single decision could have an extreme effect. As a consumer of software, you may have
wondered how seemingly random failures of software occur, and the simple answer is,
“Software can do anything.”
Physical manufacturing has evolved over time to use processes that improve productivity and
quality. Originally, production consisted of making products that were shaped and modified
56 Part II Defect Detection Techniques
until they worked (handcrafting). Then, in the early 1800s, using interchangeable parts in the
manufacture of rifles began: the quality of components was such that they could be inter-
changed between rifles. Henry Ford used a moving assembly line process for mass production
of the Model T (vastly outcompeting other manufacturers who still produced cars one at a
time—an example of parallel versus serial assembly). The Japanese used Deming’s insights
into manufacturing and instituted quality processes.
Through history, manufacturing has been moving away from the handcraft model of banging
each product into shape to achieve quality because correcting each produced product is inef-
ficient compared with improving the production process to eliminate the defect in the first
place (moving quality upstream). Additionally, the handcraft model depends on reasonably
complete testing of all the important characteristics of the product.
In physical production, quality monitoring is used to verify that the processes that instill
quality into the product are functioning correctly. If a product fails verification, it is rejected
and the production process is improved to reduce that kind of defect.
In software, using multiple layers of verification, from reviews (spec, design, and code) to
testing (unit, feature, scenario, system, stress, and customer), is one method of detecting
defects to improve quality. This type of verification operates under the premise that defects
not detected by one layer will be detected by a later layer in the gauntlet. Of course, large
numbers of defects are still detected by customers after release. You must accept the fact that it
is not possible to test software completely.
Physical products have far fewer opportunities for failure than even the simplest software
applications do. Yet, in software, we primarily use this handcraft model of production that
was supplanted back in the 1800s. But at least testing isn’t completely manual. Static testing
tools and process improvements are embodiments of the modern quality production
methodology—that is, feedback flows back into the production process.
Best Practices Software applications are incredibly more complex than any physical
product is, yet typically for software we still use quality techniques that were abandoned in
the 1800s for the testing of physical products that are many orders of magnitude less
complex and far more testable than software is.
The Myth of Complete Testability
We can’t completely test software. In fact, we can test only an infinitesimal percentage of
software functionality, yet there are many processes that depend on this false assumption of
testability and obsolete quality technique. For example, the assumption that a gauntlet of tests
can catch defects that may escape a single test is based on the tests covering a high percentage
of product functionality. Defects still escape and are discovered by customers. Such processes
are inefficient and produced low quality decades ago when applications were smaller and
Chapter 4 Quality and the Development Process 57
defects were more acceptable. Consider how your development processes, roles, and
organization reflect the myth of complete testing. Three aspects of the complete testing myth
are widespread:
Quality is a seasoning: fix bugs here and there until it tastes right.
Quality is testing’s job: testing creates quality.
Quality is a tax on development: implementation is the primary goal; fixing defects is
overhead.
Consider how your organization views quality. Do developers make the excuse, “A test should
have caught this” or “The defect is fixed; who cares what caused it? Just get a new version.”
The first excuse implies that quality is not a developer’s job. The second illustrates a lack of
concern for the customer and learning why a defect occurred so that it can be prevented in the
future.
The typical waterfall model of software development reflects the myth of testability—build it
and then bang it into shape. Even when development processes do not follow the waterfall
model, the separation of steps reinforces the attitude that testing is a tax on implementation;
this is further reinforced by having separate development and testing staffs. A postmortem
evaluation typically provides little learning because it is often viewed as another tax on imple-
mentation; it is performed after development, so many issues that occurred in the past may
have been forgotten; and unless there is honesty, important issues won’t surface. If the culture
of your team is to assess blame rather than learn from mistakes, it is natural for people to
understate issues.
How do you treat your customers during the development process? When are customers
involved—before the design process or mainly for prerelease (beta) testing? Often, customers
are used as an adjunct to the testing process to find scenario-oriented and configuration-based
defects. Customers may report usability problems, but often by the time the customer is
involved there is not enough time left in the schedule to make more than minor changes in
response to customers.
Typical defect tracking systems reflect the handcraft model—they record all sorts of infor-
mation about the processing of the defect through to resolution but really contain little
information that could be used to learn from a defect to prevent or reduce similar defects in
the future. The defect taxonomy (described in Chapter 10, “Defect Taxonomies”) and the
prevention tab (described in Chapter 14, “Prevention Tab”) are examples of the kind of
information to gather if you want to learn from defects; contrast it with your defect tracking
system.
Current Testing Methods and Quality
There’s a simple fact about testing: At best, all testing can do is indicate the lack of quality.
Testing can never provide significant coverage of an application’s functionality. If few defects
58 Part II Defect Detection Techniques
are found, it can be the result of testing areas that have already been repeatedly tested or of
testing using tests that are incomplete rather than an indication that the product has no
defects. It is not possible to tell the difference between high-quality code and poor testing.
If No One Exploits It, Is It Secure?
Security is a great example of this. Before widespread use of the Internet, software developers
were not very concerned about security exploits because the means of exploitation were lim-
ited (such as running a program you got from someone else). Because computers were iso-
lated, there was little value to be gained in terms of exploitation. The attitude was that if a
defect was hard to encounter, it wasn’t as important to fix as an easily encountered defect.
Security wasn’t an important issue.
Then the Internet arrived, Web sites proliferated, and computers became interconnected. Sud-
denly, a multitude of means to exploit vulnerabilities existed, and the value in terms of
exploits affecting other systems and theft of information such as identities or financial infor-
mation increased. Malicious people actively began to try to find security defects, so hard to
encounter was no longer a valid excuse for not fixing a security defect. Many defects, such as
null pointer references, division by zero, or memory leaks, can become a denial of service
attack if they can be triggered remotely, so they must be addressed.
There wasn’t a lot of testing for security defects before the widespread acceptance of the
Internet, hence few such defects were found—but does that mean quality was high? No, in fact
this is an example of a deceptive indicator of quality.
Deceptive Indicators of Quality
Code is inherently nonlinear: a function may work correctly for a large number of parameter
values and combinations, but a certain combination may cause a failure. Every decision in a
function is an opportunity to do something different based on a parameter or external state.
The external state can create dependencies on the execution of different bodies of code—one
function may affect external state so that another function that uses that state fails.
You really can’t determine the number of potential defects in an application from the defects
that have already been detected—even though many sampling techniques assume a reason-
able coverage of the functional space. Even if you analyze code coverage of tests, you have
verified only that a section of code worked for the parameter values tested, not that it will per-
form correctly with different parameters, configuration, or state.
So, if you have no reliable estimate of the total number of defects, how can you determine
when a product can be shipped? A slump in defects reported (a zero bug bounce) can occur
for many reasons besides actual quality (and typically does, considering defects in released
products). How many times have you had indications that quality has been achieved only for
this to be proved false when the test team changes focus?
Chapter 4 Quality and the Development Process 59
Tests have two purposes: preventing regressions and discovering new defects. After a test dis-
covers a defect, you hope the defect won’t occur again after it is fixed, but a small percentage
of regressions (defects reappearing because of changes made in correcting another defect) do
occur. Typically, the trend is that after an initial shakeout phase, regression failures become
relatively rare. If your testing staff restarts testing of a new version, you can expect initially low
defect reports because they are testing what they just tested in the previous version unless
regressions are degrading the code.
A test pass is typically a combination of automated and manual tests, and it is not instanta-
neous for large software applications. This can be because of the sheer amount of automated
tests or because some testing is manual. Automated tests are preferable to manual tests for
consistency’s sake, increased rate of testing, and portability (anyone can run these tests).
Manual tests can be script-based, in which case they are similar to automated tests, only
slower and less portable because a person must be involved in execution. Manual testing can
also be more freeform and ad hoc, which after the shakeout phase tends to discover most of
the new defects because manual tests may be testing areas not tested by the automated tests.
Often, after ad hoc testing discovers a defect, an automated test is added to detect any future
regression.
There are two other forms of manual testing: self-hosting inside the company and external
self-hosting such as beta testing (see Chapter 20, “Pulling it All Together”). Any period of test-
ing is a mix of regression (automated or script-based manual) tests and ad hoc tests, where
most new defects will be discovered. The rate of defect reporting will correlate with this mix,
and without knowledge of the mix over a time period a false zero bug bounce may occur.
For example, if a regression pass takes two weeks, involves a manual portion, and there is an
overlap of people doing the manual portion with ad hoc testing, you could expect low defect
reporting for that first week and a rise once ad hoc testing ramps up again. If your defect
counts are low, you could hit zero defects during this first week and falsely believe you’ve
turned the corner on defects. Even the ad hoc testing may have a delay in reporting because
time is involved in installing the new version, testers may not immediately switch to the new
version, and people will tend to repeat a significant amount of their previous operations
before trying new ones.
A better approach is to continue a test cycle from where it left off in the previous version rather
than starting over because it is less likely that what had been tested will fail in the new version
as opposed to what hadn’t been tested in the previous version. Ad hoc testing needs to start as
soon as possible after some level of regression testing has verified a basic level of quality. The
goal is to find defects sooner rather than to look for regressions. More negative feedback must
be given to developers for introducing and testers for not detecting regressions rather than for
introducing new bugs—regressions are far more costly.
60 Part II Defect Detection Techniques
You Can’t Test Everything
Effective testing is not impossible but requires you to embrace the fact that significant test
coverage is not possible. So how can you effectively test and know the level of quality you are
providing to your customers? Accept the idea that the basis of current testing is a myth and
think about what quality is from the customer’s point of view.
Subjective Quality: Customer Perception
To customers, quality is a subjective measure. If customers do not encounter a defect, they
may perceive quality as high, even though a defect exists. So, for software development
organizations, the quality goal is for customers not to encounter any defects.
Scenario testing, or performing tasks your customers do, is a good way to improve customer-
perceived quality. Because it is not possible to test even an insignificant portion of functional-
ity, you should focus testing on what your customers may encounter. Customer-perceived
quality is measured by mean time between failure (MTBF). This is a quality metric that is
meaningful.
Ensuring that perceived quality is high means that organizations must reverse the typical
attitude that considers fit and finish defects to be less important than other defects are. Fit and
finish defects are important contributors to a customer’s perceived quality of a product. With
software, customers make the same assumption that they do with physical products: if the
manufacturer of the product showed little concern for surface issues, that lack of concern is
assumed to follow into less apparent aspects of the product. For example, car manufacturers
have known this for decades—sloppy paint jobs, badly fitting doors, noises, and lack of ergo-
nomic design can lose a sale no matter how great the technology in the engine is. Do you give
fit and finish bugs a lower priority?
Instead of primarily concentrating on low-level tests, such as unit and feature tests, which are
useful initial shakeout and regression tests, run tests that mirror typical customer scenarios.
This pays a double dividend because determining customer usage scenarios should be an
important input into feature design. Such scenario testing does cover a wide range of per-
ceived functionality, so a value such as MTBF in testing scenarios is a directly meaningful
measure of quality.
The likelihood of encountering a defect can be used to determine where you look for defects.
Areas where code has been changed, for instance, are likely places to find new defects. A com-
bination of analysis of code coverage of scenario tests and code churn can prioritize scenario
testing on likely sources of new defects (see Chapter 8, “Risk Analysis”).
Scenario voting, as described in Chapter 15, “Scenario Voting,” gives customers a way to
describe the satisfaction and importance of scenarios as well as report defects. The satisfaction
and importance information can be used to determine a scenario testing mix. Current testing
methodology can include stress testing (see Chapter 20, “Pulling It All Together”) where a set
Chapter 4 Quality and the Development Process 61
of tests is repeated in parallel over a long period of time to uncover defects. By focusing such
stress testing on scenarios with a wide variety of configurations and cases, MTBF values can
be generated and used as a target for release quality.
Even better than using the scenarios that have been defined by scenario voting and voting
results is actually to know what your customers are doing. Rather than your depending on
scenario voting, a special version of an application can be metered to describe the operations
that a user is performing to generate scenarios and their frequency. Of course, there can be
confidentiality issues with this data. However, major customers may be quite interested
because such testing can ensure that when the product is delivered it will have a high MTBF
for their common scenarios. General beta testing could be transformed from an adjunct to
internal testing to include defining common usage patterns for scenario testing.
Objective Quality: Security
Security is an objective measure. Malicious parties are actively seeking security defects to
exploit. You can’t ignore a defect because it is unlikely to be encountered if it can be a valuable
exploit to a malicious user. You need to have a zero tolerance policy for security defects.
Security defects need to be made—as much as possible—impossible rather than unlikely. A
zero tolerance policy means implementing automatic verification of any human process and
verification that isn’t based on execution.
Catastrophic failures are important liability issues. In software, these revolve around security
and data loss. The defects targeted by scenario stress testing are those encountered in typical
operation, so such testing will not necessarily uncover many security exploits because such
exploits are usually atypical operations.
In trying to achieve zero security defects, it should be obvious that any dynamic (actual
execution of the code) testing is ineffective—you can’t test everything under all conditions.
(Such testing is useful for regression testing, not ensuring that no security defects exist.)
Zero tolerance implies static testing—validation involving scanning the source code rather
than executing it. Modern compilers are examples of static testing, such as detecting uninitial-
ized variables, dead code branches, and branches that don’t return a value.
Product Development as Transformations
Simply put, software developers make dreams into reality. A less colorful description of what
we in the software industry do is, we satisfy customer needs with our products. Meeting or
exceeding customer expectations is our goal. Our products consist of two parts: The software
and the documentation. Customers expect a product we build to have the following
characteristics:
Functionality Product improves accomplishment of user scenarios.
Reliability Product provides consistent results and does not lose or interrupt work.
62 Part II Defect Detection Techniques
High quality Minimal errors are encountered by the user.
Consistency A consistency of concepts and interfaces across product and between
product and documentation exists.
A customer evaluates these aspects in determining the value of a product.
A software defect is a discrepancy between implementation and expectations. At the highest
level, a defect is a difference between customer expectations and the product, not merely
between the product and its manual. Consider a product and customer expectations to be two
independent expressions of customer needs; unfavorable discrepancies lower the perceived
value of the product.
The stages of the product cycle typically consist of many transformations (translations) and
verifications such as reviews of written documents and testing of code. The final result deliv-
ered to a customer is the code and the documentation. During development, an additional
result is verification (tests). All three of these results can be considered translations, and they
are verified against each other. Tests verify that the code is consistent with the specification
(although the test may also be in error). Customers may report discrepancies between the
code and the documentation. A stage in the product cycle often consists of a combination of
the basic transformations:
Translation A transformation of form without loss of detail or meaning
Aggregation Collecting information from multiple sources into a single representation
Distillation Reducing information through abstraction, simplification, and/or filtering
Elaboration Expanding information through supplying or creating more detail
When transformations are done by humans, a possibility of error naturally exists. Context can
be lost, nuance can be lost, simplifications can be made, parts can be forgotten, and myriad
other types of error can be committed.
For example, when Coca-Cola was first sold in China, its erstwhile slogan “Coke adds life” was
translated into Chinese. The Chinese ideograms, when translated back into English, read,
“Coke brings your ancestors back from the dead”—the result of using reasonable synonyms in
the translation if you don’t know the context.
Adding Verification Steps to the Product Cycle
To address this possibility of transformation error, verification gates (or steps) typically are
added between stages of the product cycle, as shown in Figure 4-1. A development process
usually has a typical set of document and code review steps—feature, scenario, architecture,
design, and code. Because the reviews are also human processes, they too may contain errors.
Chapter 4 Quality and the Development Process 63
Adding a verification gate for every manual step could continue indefinitely because every
added gate is a manual step and hence may also contain errors. But this rapidly reaches a
point of diminishing returns.
Figure 4-1 Stages of a typical development process and associated verification steps
So, what is done instead to detect the defects that escape a single review of a manual step?
Additional translations independent of the code and comparison. Tests are an example of
independent translationa tester reads a specification of the expected behavior and translates
that into tests that determine whether the translations match. The assumption is that the two
different translations, code and test, have different errors—that each blocks a different class of
error and in combination they block more errors (the Swiss cheese model). But there is no
guarantee that the combination spans all possible defects; the complexity of software makes it
impossible to test any application completely. Additionally, because code and test are based
on an earlier translation, an error in the earlier translation tends to be reflected in both trans-
lations. Finally, a large amount of time passes (and a sizable implementation effort is invested)
before many defects in the earlier translation are discovered.
Scenario
Features
Design
Implement
Integrate
Plan
Plan
Plan
Unit Test
Feature Test
Scenario Test
Test Translation
Translation comparison
Code Translation
Time
Scenario Review
Feature Review
Design Reviews
Code Review
64 Part II Defect Detection Techniques
At the end of testing, defects may still exist, as shown in Figure 4-2. Scenarios represent the
expectations of the customer, what the product is expected to do. The product may not com-
pletely cover each scenario, and it may provide functionality beyond the scenarios. The prod-
uct tests may not test every scenario, and they may not test all of the code’s functionality. The
area where there is higher confidence that the product will meet customer expectations is the
area where there is code for the scenarios and there are tests for that code. There can still be
defects because the scenarios may not completely represent customer expectations, and even
if a test covers an area, it is very difficult to test it completely.
Figure 4-2 Intersections of code, tests, and scenarios and their relation to defects
The problem is the lack of independence of the translations, code and test, from each other.
Specifications often are out of date or not detailed enough, so the code is usually considered
the more accurate translation for the purpose of designing tests. The purpose of indepen-
dence is to reduce propagated translation errors in code by catching them with a parallel
translation in the form of tests. When tests use code as one of their sources for translation, as
does white box testing, both translations may have the same errors and the test can let defects
pass. (see Figure 4-3, and read the sidebar titled “White box versus black box testing,” which
follows.) Because of this lack of independence, an error in the implementation may be
reflected in the test, which can mean that an implementation error will go undetected.
Code fails test
Coded, not part
of scenario
Coded, but not tested
Not coded or
tested
Code passes
test
Scenario
Tests
Code
Chapter 4 Quality and the Development Process 65
Figure 4-3 White box testing can miss coding errors that would be detected by black box testing.
White box versus black box testing
Typically, white box testing (when tests are written with knowledge of the implementa-
tion) is considered better than black box testing (when only the specification is used)
because it can test edge conditions that are not necessarily known from the specification.
However, there are drawbacks when you consider these types of testing from the view-
point of translation.
White box testing is best used to verify compatibility from version to version. A white box
test written on an initial version can detect incompatible changes in a later version.
White box testing is a case of self-referential testing—testing an implementation using
itself as a guide. Many output values tests are also self-referential when they test against
last known good values. All of this reflects one basic problem: our chief artifact for
specification is the code because often documents are out of date and not executable.
Black box
Design
Implement
Verify Verify
White box
Code error
Code error
Code error
Design error
Design error
Design error
Design error
Design error
Design error
Test error
Test error
Test error Test error
66 Part II Defect Detection Techniques
So, what is done to reduce translation errors? Multiple levels of test translation are included
in the product cycle—function, interface, scenario, performance, stress, end-to-end scenarios,
and system testing, for example. Then there is internal self-hosting, prerelease version to cus-
tomers, and compatibility testing. Again, the assumption is that a combination of methods can
cover more usage scenarios and that there are relatively independent verifications at each
level.
But after all these verification steps are completed, customers still detect defects.
This is because, in this whole process, there is an unstated assumption in play: once trans-
lated, the original doesn’t change. This is rarely true. Discovery of defects in later stages of the
product cycle often uncovers defects in the original scenarios or design. This isn’t a fault of the
translation; the original wasn’t complete or correct. And the inaccuracy of the original isn’t an
unexpected fault because providing adequate detail for complex scenarios is a complicated
process.
Acknowledging Defects in the Original Specification
We recognize the possibility of error by providing multiple review steps as code progresses
from scenarios to implementation, but we really don’t provide for error in our original speci-
fications. In practice, a review step focuses mainly on comparing against the immediately
pevious step rather than against the original steps (far more corrections are made to the
immediately previous result).
New information tends to be reflected in code and less likely in previous stage artifacts such
as scenario or design documents. If the original isn’t changed to reflect corrections, any paral-
lel transformations may become out of sync. For example, documentation may not agree with
the code and tests may be incorrect.
There is an underlying assumption that errors in a much earlier stage can be corrected locally,
that is, by correcting the last result rather than doing a new translation from the stage of the
error. For example, a design flaw may be discovered, and instead of correcting the design and
its document, local changes that complicate the code and that don’t completely resolve the
defect are made. This can result in repeated updates being made to a flawed design. Because
of the delay inherent in defect detection, considerable time and effort may be invested in the
current design and a new design may have defects, too. Other code may be dependent on a
design, which increases the cost of a design change. Schedule pressure may manifest as a
result of the late discovery of design defects. A culture of defect prevention makes a greater
effort in preventing defects in the early stages of development and more often avoids short-
term tradeoffs. (See Chapter 17, “Moving Quality Upstream.”)
The nature of software development, as a human endeavor, includes the possibility of error.
Each stage typically involves translating the results of the previous stage, so any errors tend to
propagate forward. Tests are also a human endeavor and are based on the results of a previous
stage common with the code. Until tests are run against code, multiple manual reviews may
Chapter 4 Quality and the Development Process 67
have been done, but they can’t be guaranteed to be error-free. In a large product, everything
tends to be on a large scale—the number of developers, the number of bugs, the size of the
code, and its complexity. On this scale, you can’t afford to correct each defect in isolation; you
must learn from defects to improve development processes and tools to reduce similar defects
in the future.
Transforming Design into Code
The language you write your code in provides opportunities for error. Design concepts that
are definitional at the design level often become referential in code: rather than a single
expression of the concept, the concept is expressed everywhere it is used. The abstraction
level of programming languages is close to the underlying processor, which provides powerful
expression, including incorrect expression (for instance, unverified casting, no units enabling
byte vs. character confusion, modifying persistent state, and buffer overruns) and multiple or
complex ways of writing a particular section of code that makes the recognition of good or
bad patterns by code analysis tools more difficult and error prone.
Counterintuitive Syntax
One very simple coding error in the C family of languages is mistakenly to use an equal sign
(=) in an if rather than the equality operator (==). This mistake results in an assignment being
done rather than a comparison, and if the left-hand side is not zero, the if succeeds. Some typ-
ical examples include not using a return value and, as mentioned, using the assignment oper-
ator (=) instead of the equality operator (==) in a C/C++ if statement. (See the code listing that
follows.)
Simple Error Patterns Involving If
if (a == 2) // A valid comparison with 2
if (a = 2) // Likely semantic error, assigns a to 2
if (2 == a) // Coding convention to cause error if = used
if (2 = a) // Syntax error, reversing operands causes error
The idiom of reversing the if causes the compiler to generate an error when the left-hand side
is a constant (as in the preceding code), but of course reversing a == b will not cause an error
when both terms are variables.
The designers of C could have used another symbol for assignment, but there wasn’t an
obvious one on the keyboards of the day (the Pascal language used := for assignment).
They could have not allowed assignments in if statements, in which case the equal sign (=)
would mean equality in an if statement, but assigning a variable to a logical expression (such
as a <assignment> b <equality> c) would expose the ambiguity again. The designers of C
decided multiple assignments would be allowed, so a = b = c assigns a and b to the value of c.
68 Part II Defect Detection Techniques
This example is interesting from the point of view of human error.1 For at least a decade of
mathematics in school, you were taught that the equal sign (=) means equality, not assign-
ment. The designers of C changed the meaning of the equal sign, so the long-learned habit of
writing equals for equality is semantically incorrect but syntactically correct. To code cor-
rectly, you must break a long-held habit, turning an automatic habitual action into a con-
scious, directed one. James Reason has a term for this error, strong habit intrusion. This syntax
choice created this whole class of likely error. In the Pascal syntax, the error can’t happen.
Lost in Translation
It is difficult to infer higher-level design from code for both validation and manual code review
because there is quite a translation distance between a design and code. Compilation to
machine code is deterministic, yet the reverse translation is difficult and loses information.
And compilation is based on fixed code generation and optimization rules, unlike a developer
translating a design into code. The expansion factor of translating a design into code also
tends to be large, which makes it harder for a reviewer to validate that code matches a design.
Another major source of errors is that, in translating design to code, concepts that are
definitional (defined in one place) in the design become referential (code added for each
usage) in the code. This turns a 1:1 mapping into a 1:N mapping, which increases the possi-
bility of errors of omission (forgetting to add code for a reference) and transcription (making
a mistake in expression). The per reference instantiation of a design definitional concept is
not necessarily identical in each reference; it may need to be transposed in various ways for
the context in which it is used. This can make it hard to verify that the implementation is
present and correct (manually or automatically) and obscure the design. If the design
changes, finding and changing these instances can be error prone and difficult.
Class libraries are used to represent design-level concepts, but nothing forces a developer to
use them, and as a result of the restrictions of a class developers may need to write their own
different implementation. Developers may “roll their own,” which potentially can introduce
many kinds of error. Not only does such duplication increase the size of code, but there is also
the possibility of making a mistake in the individual instantiation, and if a change in the con-
cept is made in the future, it can be difficult to find all of the duplicates. The following exam-
ples of how language-level concepts can contribute to errors are described in the following
subsections:
Type systems
Buffer overruns
Memory allocation
Non-null object references
Use of signed rather than unsigned integers
Integer overflow
1 James Reason, Human Error (Cambridge, England: Cambridge University Press, 1990).
Chapter 4 Quality and the Development Process 69
Example 1: Type Systems In physics and other sciences, units are a method to verify the
validity of a formula (the operations on the units result in the units of the result) and also sug-
gest which formulaic transformations can be used to solve a problem or what additional data
is needed. For instance, given a speed in miles per hour, it is obvious that multiplying by time
will give distance.
The type systems of languages such as C/C++, Java, and C# are weak for scalar types. There are
no numeric units, and everything is folded into integers and floats. In some of the languages,
even Booleans are folded into integers, and all three languages fold enumerations into inte-
gers. Casting—particularly unsafe casting—provides all sorts of opportunities for errors. You
can’t even declare a variable is a pointer that is not allowed to be null. Many defects are the
result of unit conversion such as string length (excludes the terminating 0), memory length
(which, for a string, includes the 0 and is measured in bytes), character length (which is half
of string length for double byte characters), pixels, points, and others. This weakness
contributes to the loss of information in translating a design into code.
Example 2: Buffer Overruns Consider buffer overruns caused by the use of unsafe string
functions, such as copying a string into a fixed-size buffer. Because of historical compatibility,
unsafe C functions such as strcpy are in every C runtime library. You may define a safe version
of strcpy that doesn’t allow copying a string beyond the buffer size and add a static validation
that the unsafe function is not used. But why not just remove these dangerous functions and
make the error inexpressible to even the compiler?
Even when you restrict the use of the unsafe function, you cannot prevent an inline iteration
from copying a string unsafely or getting the string length and using memcpy to copy the string
unsafely. Languages provide all sorts of means of expressing unsafe operations, and it can be
difficult to detect all the variations. The original cause of buffer overruns was a performance
consideration: Passing a buffer, typically allocated on the stack, with the size to be filled in
rather than allowing the called function to allocate the memory and return it.
Example 3: Memory Allocation Consider memory allocation. There are typically only a
couple of patterns of memory management: An object is owned by another object and is cre-
ated when that object is created and deallocated with it; stack allocation, which is deallocated
when the function is exited; and everything else is “roll your own.” Everything except the first
two become individual expressions that can be miscoded, so is it any wonder that memory
leakage and dangling pointers happen?
Managed memory models with garbage collection provide another model that eliminates dan-
gling pointers but only reduces but does not eliminate memory leakage. The managed version
of memory leakage is the accidental retained reference to obsolete data; a single reference may
indirectly retain a large number of objects through indirect references. All that garbage collec-
tion does is eliminate the unreferenced objects, not the accidentally still-referenced objects.
70 Part II Defect Detection Techniques
An analysis of the memory models used in an application could be abstracted into a set of
library classes that would reduce errors both because there is only one version of the model
rather than multiple inline expressions of the model and validation could be aware of these
classes and detect more errors.
Life span is a key concept for allocated objects, so why not provide classes to implement the
various kinds of life spans objects have? For instance, a common life span is a set of objects
may live until a function that called the functions that allocated the objects returns. For exam-
ple, a class could be created and an instance passed in to the called functions that is used to
allocate any objects and retains the set of objects allocated. Freeing that object would free all
objects allocated through it.
Example 4: Non-Null Object References The limits of what can be defined about a type
can change what is conceptually associated with the type into added code for each reference
to an instance of the type. This is a case of a 1:N mapping from concept to implementation (as
shown in the following code listing).
Types Don’t Include Null Constraint
public void PlaceOrder(Product product, int quantity)
{
if (product == null)
{
throw new ArgumentException(“Product must not be null”);
}
You can’t define NonNullProduct, so instead everywhere you use Product, the body of the
method needs to have a check for a null value. An error is possible in inserting all of these
parameter checks or forgetting to do it at all in a method. In addition, a compiler or validation
tool doesn’t recognize the pattern, so it can’t detect a call such as PlaceOrder(null, 4) as an
error or detect the absence of the null check. The error can be detected by visual inspection or
execution, which assumes you test all methods with such invalid parameters. Explicitly writ-
ing tests that use invalid values and adding annotations to describe a non-null type for static
tools are two means of addressing this issue.
Example 5: Using Signed Integers Rather than Unsigned Integers If you look at the pre-
ceding example, you will see another type of common error: Quantity is defined as a signed
integer, which implies that ordering a negative number of items is permitted. That may cer-
tainly be reasonable for returning items but really shouldn’t be in an order placement method.
Imagine a Web site that allowed you to order negative quantities—you may end up with money
refunded to your credit card even if you never bought anything. Careful review looking for
improper use of signed integers can address this issue. The use of size_t for counts rather than
int is also an example.
Chapter 4 Quality and the Development Process 71
Adding a Check for Negative Quantities
public void PlaceOrder(Product product, int quantity)
{
if (product == null)
{
throw new ArgumentException(“Product must not be null”);
}
if (quantity < 0)
{
throw new ArgumentException(“Quantity must not be negative”);
}
But why add this check? The quantity parameter should have been declared as an unsigned
integer. There are many reasons for using signed integers instead of unsigned integers, includ-
ing history, habit, and sloppiness. The original C language1 did not have an unsigned integer
type or exceptions; they were added in later versions of C. The single return type of C family
languages (and for habit reasons even now where output parameters are allowed) often over-
loaded an unsigned integer return value with negative values for an error indication or other
reasons. For example, the C# String.IndexOf function returns the character index of the first
occurrence of a string and –1 if the string isn’t present.
Overloading an Index with an Error Value
public string AppendAfter(string sourceString, string subString, string appendString)
{
int index = sourceString.IndexOf(subString);
return sourceString.Substring(0, index) + subString + appendString
+ sourceString.Substring(index + substring.Length);
}
The preceding code will throw an exception if the subString is not present in the sourceString
because IndexOf will return negative one (–1) and Substring(–1) will throw an exception—all
because of overloading the return value with an error. A better solution for this issue depends
on whether the caller is supposed to ensure that the substring is present or not. In the first
case, the preceding implementation is merely misleading because the exception thrown
doesn’t clearly indicate that the problem in the substring isn’t in the source string but instead
that you specified an invalid character offset. In the second case, it would be better to use an
exception or separate error return value, as shown in the next code listing.
1 B. W. Kernighan and D. M. Ritchie, The C Programming Language (Englewood Cliffs, NJ: Prentice Hall, 1978).
72 Part II Defect Detection Techniques
Alternate Definition of IndexOf
public uint IndexOf{String sourceString, string subString)
{
int index = sourceString.IndexOf(subString);
if (index < 0)
{
throw new OperationException(“Substring not found”);
}
else
{
return (uint)index;
}
}
public string AppendAfterIfPresent(string sourceString, string subString, string
appendString)
{
try
{
uint index = IndexOf(sourceString, subString);
return sourceString.Substring(0, index) + subString +
appendString + sourceString.Substring(index +
substring.Length);
}
catch (OperationException)
{
return sourceString;
}
}
You can argue that using an exception is too expensive in terms of execution time, that it is
really a matter for improving code optimization, providing a different error model (see the
sidebar titled “Prolog error model,” which follows), or some other construct. Using a separate
error return value does not guarantee that the caller will check it first, so the issue of what to
assign the return value to exists.
Prolog error model
Prolog has an interesting model of errors and variables. Failure is an innate concept. Any
equivalent of a function call has two means of returning control to its caller:
Success, which calls the return address
Failure, which returns to the last point where there was a choice and unbinds all
variables
It’s quite a different model and has some interesting characteristics after you understand
Prolog’s concepts. It is one of the few languages where failure (error) is an innate part of
the language’s concept.
Chapter 4 Quality and the Development Process 73
You could reverse the index and the error indication by returning a value that indicates
whether the IndexOf succeeded and changing the output parameter to a reference as follows.
Definition of IndexOf Not Using Exceptions
public bool IndexOf{String sourceString, string subString, ref uint index)
{
int strIndex = sourceString.IndexOf(subString);
if (strIndex < 0)
{
return false;
}
else
{
ndex = (uint)strIndex;
return true;
}
}
public string AppendAfterIfPresent(string sourceString, string subString, string
appendString)
{
bool notFound;
unit index;
notFound = IndexOf(sourceString, subString, ref index);
if (notFound)
{
return sourceString;
}
else
{
return sourceString.Substring(0, index) + subString +
appendString + sourceString.Substring(index +
substring.Length);
}
}
}
The preceding example won’t compile because reference parameters—unlike output
parameters—are not guaranteed to be set. There is no valid index if the substring is not found,
and supplying one just allows for the error of using it without checking if notFound is false.
The compiler doesn’t know that the index variable would be set only if notFound were false.
You could initialize the index to 0, but that would open up the possibility of using index when
notFound is true. That’s why an exception is a better method to use for semantic reasons, even
though it may be slower.
Note that the name of the preceding function has been changed to AppendAfterIfPresent. The
assumption of what happens when the subString isn’t present should be explicit in the name
of the method rather than requiring the developer to look up the behavior of the method to
74 Part II Defect Detection Techniques
see what happens if the string is not present. Both AppendAfter, which throws an exception if
the subString isn’t present, and AppendAfterIfPresent, which does nothing if subString isn’t
present, may be useful methods for different callers. If the design is such that subString should
be present, AppendAfter should be used so that if some other defect causes it not to be present
an exception is thrown and the issue can be diagnosed rather than invisibly not appending.
Serious defects can be hidden by such helpful defaulting in error conditions.
Example 6: Integer Overflow Interfaces that use signed rather than unsigned parameters
are open to security attacks using negative numbers, as in the previous example of using neg-
ative quantities to get refunds. In addition, even correctly used unsigned numbers can be used
in an attack through overflow wrapping. If two numbers are multiplied together, choosing
large enough values will cause the result to exceed the resolution of the integer, resulting in a
vastly smaller result than expected or even a negative result.
By default, runtime systems at best detect errors such as divide by zero but not integer
overflow. Specific code needs to be added for each instance of a possible overflow operation
(another definitional concept changed into per reference).
C# provides a checked construct that throws an exception on integer overflow. You could
sprinkle your code with checked constructs around integer operations, but just as with the
previous example, errors can be made and places missed. Instead, at least in C#, you can
detect integer overflow centrally in a type definition. However, it is quite a few lines of code, as
follows.
Overflow Safe Positive Integers
struct PosInt
{
private uint Value;
public PosInt(uint value)
{
this.Value = value;
}
public override bool Equals(object obj)
{
return this == (PosInt)obj;
}
public override int GetHashCode()
{
return Value.GetHashCode();
}
public override string ToString()
{
return Value.ToString();
}
Chapter 4 Quality and the Development Process 75
public static PosInt operator +(PosInt a, PosInt b)
{
return new PosInt(checked(a.Value + b.Value));
}
public static PosInt operator *(PosInt a, PosInt b)
{
return new PosInt(checked(a.Value * b.Value));
}
public static PosInt operator -(PosInt a, PosInt b)
{
return new PosInt(checked(a.Value - b.Value));
}
public static PosInt operator /(PosInt a, PosInt b)
{
return new PosInt(checked(a.Value / b.Value));
}
public static PosInt operator %(PosInt a, PosInt b)
{
return new PosInt(checked(a.Value % b.Value));
}
public static bool operator ==(PosInt a, PosInt b)
{
return a.Value == b.Value;
}
public static bool operator !=(PosInt a, PosInt b)
{
return a.Value != b.Value;
}
public static bool operator >(PosInt a, PosInt b)
{
return a.Value > b.Value;
}
public static bool operator >=(PosInt a, PosInt b)
{
return a.Value >= b.Value;
}
public static bool operator <(PosInt a, PosInt b)
{
return a.Value < b.Value;
}
public static bool operator <=(PosInt a, PosInt b)
{
return a.Value <= b.Value;
}
76 Part II Defect Detection Techniques
public static PosInt operator +(PosInt a)
{
return a;
}
public static PosInt operator ++(PosInt a)
{
return new PosInt(a.Value + 1);
}
public static PosInt operator --(PosInt a)
{
return new PosInt(a.Value - 1);
}
public static bool operator !(PosInt a)
{
return a.Value != 0;
}
public static implicit operator uint(PosInt a)
{
returna.Value;
}
public static implicit operator PosInt(uint a)
{
return new PosInt(a);
}
public static implicit operator PosInt(int a)
{
return new PosInt(checked((uint)a));
}
}
Using Overflow Safe and Unsafe Integers
static void Main(string[] args)
{
int i,j,k;
Pos IntiPosInt, jPosInt, kPosInt;
i = 2000000000;
j = 2000000000;
k = i * j; // latent overflow -1651507200
Trace.WriteLine(string.Format("{0} = {1} * {2}", k, i, j));
iPosInt = 100;
jPosInt = 200;
kPosInt = iPosInt * jPosInt;
Trace.WriteLine(string.Format("{0} = {1} * {2}", kPosInt, iPosInt,
jPosInt));
Chapter 4 Quality and the Development Process 77
iPosInt = 2000000000;
jPosInt = 2000000000;
kPosInt = iPosInt * jPosInt; // Throws exception
Trace.WriteLine(string.Format("{0} = {1} * {2}", kPosInt, iPosInt,
jPosInt));
iPosInt = -1; // Throws exception
}
Output:
-1651507200 = 2000000000 * 2000000000
20000 = 100 * 200
A first chance exception of type 'System.OverflowException' occurred
Conclusion
Software quality is a combination of perceived quality based on the likelihood of a customer
encountering a defect and security defects, which are maliciously sought. You can’t test a
significant percentage of the functionality of a software product, so processes based on that
assumption can give a deceptive sense of quality.
The process of software development consists of a multitude of transformations from the
original vision into the final product. Many of the steps in this process have manual artifacts
that can’t be automatically verified; software development depends a lot on human verifica-
tion processes (reviews). Because any human process includes a possibility of error,
verification must be against something—another translation.
Each transformation is also into a different language of description. The first few are
typically written documents, but finally code and tests are written. The step of writing code
often loses a lot of information because some concepts can’t be completely expressed in
code or a definitional concept is turned into a referential concept as a result of limitations of
the language.
79
Chapter 5
Using Productivity Games to
Prevent Defects
In every job that must be done, there is an element of fun. You find the fun
and—SNAP—the job’s a game!
—Mary Poppins
Most people think of games as a leisure activity and not part of their daily work. Merriam-
Webster defines game as an activity engaged in for diversion or amusement.1 In the case of
defect prevention, you are trying to engage people for diversion—to divert or distract them
from their day-to-day tasks and encourage them to exert some effort on preventing defects.
Plenty of books and resources on game design are available. This chapter does not cover game
design in detail but shows how and why competition is a good technique to use to encourage
defect prevention effort. Most organizations do not use games and competition in the work-
place as motivational techniques, which is unfortunate because they can be incredibly effec-
tive ways to get people to do things. Consider Frederick Winslow Taylor and his “one best
way” in the famous Principles of Scientific Management. Taylor put a lot of work into studying
workers and work processes to find the optimal output rate, and then analyzing behavior to
move the median productivity rate toward the optimal rate. It’s interesting to ponder if using
games or competitions may have been a more effective approach to motivating workers. See
the sidebar titled “Using the carrot, not the stick,” which follows.
Using the carrot, not the stick
“This man we will call Schmidt. The task before us, then, narrowed itself down to getting
Schmidt to handle 47 tons of pig iron per day and making him glad to do it. This was
done as follows. Schmidt was called out from among the gang of pig-iron handlers and
talked to somewhat in this way:
“ ‘… Well, if you are a high-priced man, you will do exactly as this man tells you tomor-
row, from morning till night. When he tells you to pick up a pig and walk, you pick it up
and you walk, and when he tells you to sit down and rest, you sit down. You do that right
straight through the day. And what’s more, no back talk. Now a high-priced man does
1 Merriam-Webster Online, “Game,” http://m-w.com/dictionary/game.
80 Part II Defect Detection Techniques
just what he’s told to do, and no back talk. Do you understand that? When this man tells
you to walk, you walk; when he tells you to sit down, you sit down, and you don’t talk
back at him. Now you come on to work here tomorrow morning and I’ll know before
night whether you are really a high-priced man or not.’ ”1
Now, if Schmidt were presented with a “pig iron per day” game to play, he may have been
as productive, perhaps even more so, without the “rather rough talk,” as Taylor labels it.
The question for this chapter is whether a “high score” may have challenged Schmidt
and his cohorts to “beat the high score” and made them even more productive. Would
the “second class workman” (Taylor’s term) be motivated enough to show up the “first
class workman” in a competition that he could handle 50 tons of pig iron instead of 47?
Would Schmidt be glad to do it because he won that day’s competition? Use the carrot,
not the stick.
What Is Game Theory?
Game theory is a branch of applied mathematics and economics that studies situations
where players choose different actions in an attempt to maximize their returns. John von
Neumann was the first to expand on early mathematical analysis of probability and chance
into game theory in the 1920’s. His work was used by the military during World War II, and
then later by the RAND Corporation to explore nuclear strategy. In the 1950s, John Nash,
popularized in the film A Beautiful Mind, was an early contributor to game theory. His
“Nash Equilibrium,” which helps to evaluate player strategies in non-cooperative games.
Game theory is now used in many diverse academic fields, ranging from biology and
psychology to sociology and philosophy.2
Game theory outlines how and why people play games. Games have players, rules, strategies,
outcomes, and a reward. Games are things people do, following a set of rules, developing
strategies to create a certain outcome to receive a reward or prize. The players in the game
must make certain “moves” to determine their reward. The players must follow certain rules
while making these moves. Each player is supposed to behave rationally, which means trying
to maximize personal reward, regardless of what the other players are doing. They are trying
to “win.” Each player pursues a set of moves, within the rules, to maximize his or her reward.
See the following sidebar titled “Prisoner’s Dilemma” for a discussion of how this classic
dilemma can illuminate aspects of player strategies and motivation.
1 Frederick W. Taylor, The Principles of Scientific Management (New York: Harper and Brothers Publishers, 1911).
2 Wikipedia, “Game Theory,” http://en.wikipedia.org/wiki/Game_theory.
Chapter 5 Using Productivity Games to Prevent Defects 81
Prisoners Dilemma
The creation of the Prisoner’s Dilemma is attributed to Merrill Flood and Melvin
Dresher, who described it in their investigations of game theory as a nuclear strategy at
RAND in 1950. The Prisoner’s Dilemma is about two criminal suspects, questioned sep-
arately about a crime. If one suspect testifies against the other, and the other is silent, the
silent suspect gets the full sentence, while the testifier goes free. If both are silent, both
get minor but nonzero sentences. If both testify, they get partial—more than minor—less-
than-full sentences. Each suspect must choose whether to testify or remain silent. How-
ever, neither suspect knows what the other will do. The dilemma arises when you
assume that the suspects care only about themselves and do not know what the other
will do—and yet the outcome for each depends on the choice of the other.
Much has been written on the Prisoner’s Dilemma and its role in game theory. Since the
use of games can have a dramatic impact on defect prevention efforts, it is useful to con-
sider some of these fundamentals of game theory in the design of games to understand
player strategies and encourage participation in defect prevention activity.
Games in History
Games have been around for centuries. For example, the game of Go was played in ancient
China in 500 B.C. and is for sale along most toy aisles and in game stores, as well as online.
(See the Free Go Programs Web site at www.gnu.org/software/gnugo/free_go_software.html.)
The royal game of Ur was played in the fourth century B.C. You can play it online today on the
British Museum Web site at www.thebritishmuseum.ac.uk/explore/families_and_children/
online_tours/games/the_royal_game_of_ur.aspx. Evidence even seems to indicate that games
were used to motivate workers to build the pyramids of Egypt. (See the sidebar titled
“Productivity Games Helped Build the Pyramids,” which follows.)
Productivity games helped build the pyramids
One of the greatest testimonies to work and productivity is the ancient pyramids.
There are many theories on how these great structures were built—everything from slave
labor to aliens. What motivated the workers to accomplish these tremendous feats of
engineering?
The Great Pyramid of Giza is the largest of the pyramids and has approximately 2.3
million stone blocks, weighing an average of 2.3 metric tons each. The estimate is that
35,000 workers built this great structure. Originally, the theory was that slave labor was
82 Part II Defect Detection Techniques
used for construction, but archaeologists now think the building of the pyramids was
accomplished predominantly as a volunteer effort—farmers, masons, carpenters, and
metalworkers all contributing for the sake of national pride.
Most of the pyramids in Egypt each took decades to build. Former President Jimmy
Carter once quipped, “I’m surprised that a government organization could do it that
quickly.” So the relevant question is, How did the workers stay motivated to complete
the projects? Were productivity games used?
The answer is yes. (Otherwise, we wouldn’t have included this example here!)
Workers were organized into teams, or phyles (tribes)—“Friends of Khufu” and “Drunk-
ards of Menkaure,” left and right, green and red, and so on. The evidence seems to indi-
cate that these teams competed with one another to improve productivity. For example,
stones were tagged with color to indicate the team that placed them.1
Phyles were subdivided into divisions. And divisions were identified by names repre-
sented by single hieroglyphs with meanings like endurance, perfection, strong (see Fig-
ure 5-1). So how do we know this? When you visit the pyramids, you come to a block of
stone in the relieving chambers above the Great Pyramid. First, you see the cartouche of
a king and then some scrawls in red paint—the gang’s name—after it. In the Old Kingdom
in the time of the Pyramids of Giza, the gangs were named after kings. So, for example,
you may have a name, compounded with the name of Menkaure (a pharaoh), and it
seems to translate as “the drunks or the drunkards of Menkaure.”
It gets even more intriguing. In certain monuments, you can find the name of one gang
on one side of the monument and the name of another gang, we assume a competing
gang, on the other side of the monument. It’s as though these gangs were pitted against
each other. From this evidence, we can deduce that a labor force was assigned to respec-
tive crew gang phyles and divisions.2
1 Virginia Morell, “The Pyramid Builders,” www7.nationalgeographic.com/ngm/data/2001/11/01/html/
ft_20011101.5.fulltext.html.
2 PBS Online, Nova Online Adventures, “Who Built the Pyramids?” www.pbs.org/wgbh/nova/pyramid/explore/
builders.html; Alan Winston, “The Labors of Pyramid Building,” www.touregypt.net/featurestories/pyramidwork-
force.htm; Joyce Tyldesley, “The Private Lives of the Pyramid-Builders,” Bbc.co.uk Web site, www.bbc.co.uk/
history/ancient/egyptians/pyramid_builders_07.shtml.
Chapter 5 Using Productivity Games to Prevent Defects 83
Figure 5-1 The pyramid builders’ teams
The Gamer Generation
Although the current generation of electronic games has its roots in the 1950s and 1960s,
with Spacewar and PONG, computer and game console developments over the last few
decades have taken gaming to a new level. Perhaps it is unfair to attribute the start of a
generation to a single game, but in 1981, Nintendo published Donkey Kong, which really was
a defining moment in mainstream gaming. The popularity of video games far exceeds that of
other historical game types. According to the Entertainment Software Association (ESA), the
sales of gaming units alone have grown from 2.6 million units in 1996 to 228 million in
2005.1 Contrast those figures with the game of Monopoly, which has sold more than 200 mil-
lion units in its 65-year history.2
The point of this very brief overview of the history of electronic games is to emphasize the fact
that the workforce today is composed of people who have grown up in a world of video
games. The ESA says that the average gamer is a person who is 33 years old and has been
playing for 12 years.3 For example, any recent college computer science graduate employed in
Drunkards of
Menkaure Gang
Green
or
Fresh
AsiaticGreat Little
or
Small
Perfect,
Lowest
or Last
Green
or
Fresh
AsiaticGreat Little
or
Small
Perfect,
Lowest
or Last
Friends of
Menkaure Gang
Crew
about 2000 laborers
North
side of
temple
South
side of
temple
Phyles—about 200 laborers in each Phyles—about 200 laborers in each
1 Entertainment Software Association, “Facts and Research: Sales and Genre Data,” http://theesa.com/facts/
sales_genre_data.php.
2 Hasbro, “History,” www.hasbro.com/monopoly/default.cfm?page=history.
3 Entertainment Software Association, “Facts and Research: Game Player Data,” http://theesa.com/facts/
gamer_data.php.
84 Part II Defect Detection Techniques
software development grew up playing video games. The rise in popularity of fantasy sports
games, tribal casinos, televised poker, and gaming consoles (Xbox, PS3, and Wii) is no coinci-
dence. This is a generation that thrives on games and competition.
What does that mean for defect prevention?
It comes down to “know your customer.” In their book The Kids Are Alright, John Beck and
Mitchell Wade enumerate the basic principles that video games have taught this generation:1
If you get there first, you win.
There is a limited set of tools, and it is certain that some combination will work. If you
choose the right combination, the game will reward you.
Trial and error is the best strategy and the fastest way to learn.
Elders and their received wisdom can’t help; they don’t understand even the basics of
this new world.
You will confront surprises and difficulties that you are not prepared for. But the sum of
those risks and dangers, by definition, cannot make the quest foolish.
Once you collect the right “objects” (business plan, prototype, customers, maybe even
profits), you’ll get an infusion of gold to tide you over.
Although there may be momentary setbacks, overall the trend will be up.
It is important to pay attention to these principles when designing games to motivate the
workforce to invest time in defect prevention activities. The people closest to the work—the
individual developers working with the code—know the most about defects. Because they are
so familiar with the defects, they have the knowledge of how to prevent defects. Typically, the
majority of the individual coders are recent college graduates or those newer to the workforce.
Coincidentally, these same people grew up with video games as a significant part of their daily
environment and were taught the preceding basic principles through their heavy interaction
with video games. The result is that they identify with and will respond to the idea of using
games in their work activities.
Why Games Change Behavior
Games are compelling. Human nature has always responded to competition. Darwin’s theory
of natural selection touches on the need for members of a species to compete with one
another for food or living space or some limited resource to survive. It is well within human
1 John C. Beck and Mitchell Wade, The Kids Are Alright: How the Gamer Generation Is Changing the Workplace
(Cambridge, MA: Harvard Business School Press, 2006), 43.
Chapter 5 Using Productivity Games to Prevent Defects 85
nature for people to compete with one another, so creating a contest to drive behavioral
change targets survival instincts that are core to everyone. As Darwin says,
The struggle will generally be more severe between them [members of a species], if they
come into competition with each other.
—Microsoft Encarta, 2007.
NCAA Tournament Games
Every year, March Madness descends when the National Collegiate Athletic Association
(NCAA) hosts a basketball tournament for men’s and women’s teams. Sixty five college
basketball teams are seeded and compete against one another until one team wins the tourna-
ment. Many spectators set up brackets that list the teams competing and try to predict which
teams will win their matchups and move forward in the tournament to the final competition.
Often, employees in companies host office pools in which they compete against one another
to see who can predict the basketball team winners most accurately. The NCAA tournament is
usually preceded by a host of news articles on the loss of employee productivity brought on by
office betting pools.
According to many articles, the cost of lost productivity is more than a billion dollars. Several
articles suggest that employers suffer the losses and are happy with the “morale benefits,” say-
ing that office pools build camaraderie and foster closer relationships between staff members.
Perhaps the answer is to integrate work tasks into the pool. According to one survey, employ-
ees spend more than 13 minutes a day during the tournament checking basketball scores.
Why not put this out in the open and attach “real work” to the pool? For example, an insur-
ance company can hold a contest to allow claims processors to fill out one bracket per 10
claim forms processed. Or highway patrol officers can get a Final Four pick for every five
speeding tickets they write—just kidding!
The point of productivity games is to align the game with the goals of the organization, and if
people are spending their work time checking NCAA results, companies should integrate that
behavior into the task at hand.
Types of Games
Game theory offers a categorization of types of games that is helpful in designing productivity
games for defect prevention. As management or the “game designers” think about how to use
games to encourage defect prevention activity, an understanding of the different types of
86 Part II Defect Detection Techniques
games is tremendously helpful in making the games more fun and more effective at attracting
players to defect prevention work.
Following is a general categorization of the types of games that exist:
Noncooperative Players work independently without concern for the other players.
These are typically the best type of games for work-related competitions.
Cooperative Players may cooperate with one another. This can lead to tricky issues
when used as a motivator for defect prevention activity.
Zero Sum The sum of all rewards for all players is zero. Zero-sum games are not recom-
mended for defect prevention competition because people can do work, perform tasks,
and lose ground.
Constant Sum The sum of all rewards for all players is constant. However, a variable
reward is a more exciting and motivational incentive for defect prevention work.
Aligned Games Productivity games that are played to help achieve the goals of the task
at hand are called aligned games. For example, a janitor may see how many garbage cans
can be emptied in an hour. NCAA tournament picks that are awarded as a result of
doing a task in line with company goals can be considered aligned games.
Incidental Games Incidental games are played during the course of work and do not
contribute or directly affect the outcome or accomplishment of the task. Incidental
games probably will help improve morale, will keep people from straying too far off task,
and may foster community. However, they do not directly relate to the goals of the job.
An example might be a hotel maid who lines up pillows in certain patterns from room to
room, or a truck driver who waves to Volvo wagons for the fun of it. These incidental
games keep people thinking and keep their minds working but do not necessarily
improve output. NCAA tournament watching—as it is done today—is a great example of
an incidental game.
Games of Chance vs. Games of Skill
Games and competition used for defect prevention work can be games of either chance or
skill. The nature of the scoring and reward mechanisms determines the type of game. If the
rewards are random, these competitions are considered games of chance. On the other hand,
the competition can, for example, be about finding the most buffer overrun bugs—making it a
game of skill. It is worth noting, however, that games of skill may tend to discourage competi-
tion unless the players are all of roughly equal competence. If one or two players are far more
skilled than the rest are, they will stand out and the rest of the players may become
discouraged. A game of chance gives more players an opportunity to earn rewards, thereby
encouraging participation.
Chapter 5 Using Productivity Games to Prevent Defects 87
Mini Games
One fascinating possibility of productivity games is to make the game take the place of the
work. There are many examples in the real world where mini games alter behavior or focus
attention on a specific issue. A simple example is the “days without an accident” sign posted
at many construction sites.
Two distinct categories of mini games can be created and played within the task: games that
are aligned with the goal of the task—call these challenges—and games that are not aligned—call
these incidental mini games. Here is an example to illustrate the difference between the two.
The task is to drive a truck from New York to Seattle. The goal is to travel coast to coast in the
least amount of time. An aligned challenge can be created in which points are awarded to the
driver whenever the second hand on a watch passes the 12 at exactly the same time that the
truck passes the physical milepost, assuming the driver follows the legal speed limit, road con-
ditions are favorable, and so forth. The score is increased for each successful “hit”; that is, for
each time the second hand clicks to 12 at the exact milepost location. This game is aligned
with the overall goal of driving to Seattle, and “winning” this game contributes positively to
the goal of driving to Seattle. Most video games include mini games that are aligned with the
overall goal of the game—players can get new tools or weapons, build strength, and so forth.
An unaligned, or incidental, mini game in the truck driving scenario may a point awarded
every time the driver honks the horn at a silver SUV. The scoring for the incidental game does
not contribute positively to the overall goal of driving to Seattle. There is a third, rarely occur-
ring type, where winning the mini game penalizes the player in progress toward the overall
goal. For a truck driver, this adverse game may be to get off the highway at every even-num-
bered exit, turn around, and get back on. Winning this game is detrimental to the goal of
driving to Seattle in the least amount of time.
During the design of mini games, be sure to consider existing work and the repercussions of
overlaying a game on work that’s already being done. For example, if one person is employed
full time to find defects in the code, starting a game to encourage others to do that work may
have an adverse effect on that person. Simply starting a game to score others on the “regular”
work of another is not a good idea. Perhaps involving that person in the design of the game,
with a careful explanation and understanding of the goals, will help, but even those efforts
may not overcome the negative implications of inviting, encouraging, or rewarding others for
doing his or her job.
Prediction Markets
A prediction market is a type of betting, where the payoff is based on the outcome of some
future event. In software development, prediction markets built around product release dates
are one of the more common uses of the genre. Prediction markets can be established around
defect prevention techniques as well, and they can provide a useful alternative to common
88 Part II Defect Detection Techniques
productivity games. Again, the design and reward structure are imperative for prediction mar-
kets, so as to discourage people from doing the wrong type of work in order to win the game.
More information on prediction markets:
Prediction Markets: Wolfers and Zitzewitz - Journal of Economic Perspectives—Volume
18, Number 2—Spring 2004—Pages 107–126
Hollywood Stock Exchange - http://www.hsx.com/
http://www.ideosphere.com/
http://en.wikipedia.org/wiki/Prediction_market
Alternate Reality Games
An alternate reality game is a storyline that weaves real-life elements into the game. These
“games” involve creative writing, role playing, and puzzle solving around a common story.
Alternate reality games can be extremely useful in tying together various other productivity
games for defect prevention work. As with other types of games, significant effort must be put
into the design and the rewards to ensure that the right behavior is reinforced. The design of
an alternate reality game is significantly different from a simple productivity game, but the sto-
ryline can sit on top to help connect various productivity games around a common theme.
For more information on alternate reality games, see
http://www.argn.com
http://www.unfiction.com
Practical Guide to Defect Prevention Games
There are some simple steps you can follow to create games for defect prevention. The goal is
to take advantage of the natural attraction of games and competition and use it to encourage
people to invest time and effort in defect prevention activities.
Start with a Leader Board
The easiest way to create a competition is to start to keep score. As Peter Drucker says, “What
gets measured gets done,” and this applies directly to the use of games and competition to
encourage investment in defect prevention. As soon as you start keeping score, people sit up
and take notice. If, for example, the project could benefit from a higher defect discovery rate,
posting a running total number of defects discovered by each developer will compel people to
compete to increase their score. However, as soon as you start to keep score, people will start
to optimize their behavior for a higher score—to “game” the system—that is, the player may
enter artificial or dubious defect discovery reports just to win the game. A well-designed
game does not have this problem, but if there are holes in the scoring mechanism, it can
cause huge problems.
Chapter 5 Using Productivity Games to Prevent Defects 89
Keep It Simple
It is easy to overthink a game. The game should be simple enough that people can play easily
and score points. Simple games attract players. People tend to overanalyze and overdesign the
games at the cost of results. A simple game is easy to administer and can get people playing
immediately. There is always time to refine it as you go.
Think Through Scoring
People will quickly find any holes or improprieties in the scoring system. The famous Dilbert
response to a “bug bounty”—a competition to find the most bugs—illustrates this point exactly
by leading to completely unexpected results. It is critical to think through the scoring and
rewards to make sure they drive the right behavior. Scoring system design is a skill that
improves with practice. The best way to find and fix scoring problems is by using a small pilot
program. People are creative and will find holes immediately.
DILBERT: © Scott Adams/Dist. By United Feature Syndicate, Inc.
After participants hear the game rules, spend some time with them to understand their play-
ing strategies; this can help flush out any scoring inconsistencies. Scoring is not something to
take lightly—the success of the game depends on an equitable scoring system. Even the best
intentions can end up being misconstrued.
Reward the Right Behavior
A game will entice people to do things. The design of the scoring system and rules of the game
can encourage people to change what they do. The rewards for the game should go to those
who perform the intended actions. This may sound simple and obvious, but many times, sub-
tle nuances of the scoring system are missed until it is too late.
For example, an initial goal may be set to find bugs. A team can host a “bug bash” for an
evening and give a prize to the person who finds the most bugs during the event. However,
reviewing those bugs a week later may reveal that the winner actually entered a high number
of duplicate, insignificant, or invalid bugs and didn’t do quite as well as it looked when prizes
were distributed. In the design of the scoring system, it is important to consider the game’s
goals and the desired behavior necessary to achieve those goals. The goal may not be to find
a sheer number of bugs, but to find high-quality bugs or bugs that must be fixed. Be as clear
as possible on the exact behavior change desired when designing the scoring and rewards.
90 Part II Defect Detection Techniques
Use Scoring to Encourage Participation
This is another reminder to consider the goals of the game carefully. If the goal is to get the
entire team to perform an activity, the scoring system should account for the fact that one or
two people may jump way ahead, or people may prioritize the activity differently and fall way
behind. An example is a “consecutive days” reward when scoring is based on consecutive days
of performing the desired activity. If a team member is out of the office for the first three days
of the competition, he or she can never catch up and therefore likely will not participate for
the rest of the game. In such cases, the scoring system should allow people to catch up or offer
alternative ways to compete.
Design the scoring to encourage, not discourage, participation. Perhaps there are bonus
points to earn or alternative activities that people can do to get themselves “back in the game.”
Also, consider leaders: If one person works hard to get a big lead and that person’s effort
results in a big gain for the project, he or she should be acknowledged and rewarded for the
effort, not discouraged by being penalized and brought back to the pack through rule changes
and bonus point distribution.
Using scoring to encourage participation is harder than it looks.
Keep the Players Coming Back to Check Their Score
A good game keeps the players’ attention. The idea is to encourage players to monitor and
improve their scores. If the players are regularly checking their scores, it is more likely they are
trying to improve their scores and performing the tasks that earn them rewards. Anything that
the game designers can do to bring people back to the game helps maintain interest and keeps
people playing. Following are some activities people do when they are engaged by a game:
Check scores
Read about winners
Look up friends’ and co-workers’ scores
Learn about new games
Challenge others to one-off competitions
All these behaviors keep the interest alive and get people thinking about performing the tasks
to earn a place in the game.
Youd Be Surprised at What People Will Compete For
As mentioned, people are naturally competitive. From birth, we are taught to compete for sur-
vival—for food, attention; competition is a natural part of life. Implementing a simple game to
perform an action and receive a point can result in behavioral change. A competition to spell
a word or click a button will attract participants. At first, you may tend to think big, elaborate
games are necessary, but simple competitions are a great way to start. Model some of the
Chapter 5 Using Productivity Games to Prevent Defects 91
games after popular “analog” games, such as card games or board games. Games like Chinese
Checkers, Othello, Scrabble, Hangman, and popular board games are all based on very simple
concepts.
Allow Room for Adjustment—Set a Time Frame
A game or competition will be more successful, especially at first, if it exists for a set length
of time. If the scores are tallied daily or weekly, adjustments can be made periodically to
fine-tune the competition. When starting a game, schedule an update in a short amount of
time to allow for adjustments. If it is impossible to shorten the length of the competition—for
example, the game follows the football season—host a pilot version prior to the start of the
game period so that you can make any scoring adjustments before deployment.
Stratify to Maintain Interest
In most games, there will be serious or hardcore players and casual players. The existence and
participation of hardcore players should not discourage casual players. The scoring system
should accommodate and motivate both types of players to compete, even if they are not com-
peting with each other.
Retain a Players History
As games evolve or new games begin, try to avoid starting over. Players have earned some
value and prestige, and it is useful to retain these attributes. Players who finish second or
third, for instance, might lose interest if they are forced to start over from zero in a new game.
However, a player who finishes last wants a fresh start, so analyze the playing field and
develop the scoring and historical retention policies accordingly.
Start with a Small Pilot
By using a pilot program, the product team can monitor participation more closely. A pilot
provides the opportunity to monitor the game play and keep track of scoring rewards. With a
pilot, the product team can interview players and get a full understanding of how the game
will be played and how the players perceive the rules and rewards.
Let People Play at Their Own Pace
People work at different rates—some are slow and steady, and some work in bursts. The game
should allow players to participate at their own pace. Scoring and participation should reflect
the need for people to play at a rate that is comfortable for them.
Augment Interest by Using Cash and Prizes
To sustain interest, add cash and prize rewards. It’s probably best not to do this at the start,
but add them as the contest continues to keep interest levels high.
92 Part II Defect Detection Techniques
Use Random Drawings
When determining prizes, random drawings are sometimes a more effective way to
distribute rewards. Sometimes leaders are so far ahead of the others that the rest of the pack
loses interest. In this case, a random drawing can help to keep interest in the game while you
are still acknowledging the leaders.
Examples of Using Games for Defect Prevention
During the development of the Windows Vista operating system, the Microsoft product team
developed and hosted a number of different games to drive productivity. The size of the
project and corresponding length of the development cycle led to a repetitive set of activities,
and as the team entered the beta phases, the use of games was successful in injecting new
energy into the development effort. In some cases, there were productivity gains of as much as
300 percent using competition as a basis for encouraging task completion. As shown in
Figure 5-2, the beta 1 game asked team members to perform various tasks to “earn a letter”
and spell beta1.
Figure 5-2 Windows Vista beta 1 game
The beta 2 game (Figure 5-3) expanded on the concept and awarded points for test activity.
There were multiple levels of prizes and random drawings, and players could earn wristbands
based on participation. In some cases, the wristband became a symbol at meetings and in
hallways that spurred competition.
Chapter 5 Using Productivity Games to Prevent Defects 93
Figure 5-3 Windows Vista beta 2 game
These games culminated in a release game that was distributed company-wide. Prizes were
based on random drawings for those who completed installation and certain test activities.
Once again, the results were phenomenal, with the majority of the company participating in
the final days of testing Windows Vista.
Tips on Game Design
It is easy to spot the similarities in simple games that work well. Noncooperative games, where
each player is responsible for his or her own scoring, usually work best.
Simple leader board—do something, earn a point
Spell a word or phrase—earn letters
Hangman—earn a guess
Pick a number—earn a guess
Guess a word—earn a letter guess
Graffiti—earn letters to spell a phrase
Guess the mystery celebrity (or mystery whatever)—earn clues to guess
Scavenger hunt
Clue
Game Design Checklist
Any productivity game should accomplish the following:
Get players to be more productive (and have data to prove it)
Be easy for people to play and compete
94 Part II Defect Detection Techniques
Have well-thought-out rules and scoring
Be competitive and/or fun
Keep players engaged and looking at their scores
Keep a history of player activity
Let players play at their own pace
Let players compete against themselves
Be tested with a small group first
Evolve and be refined over time
Be careful with games that overlap existing work.
A great resource on the use of games for productivity and to change behavior is the Serious
Games Initiative (www.seriousgames.org). This organization helps to connect game designers
and organizations using games for health care, social change, education, and productivity.
Conclusion
Productivity games are an effective way to encourage investment in defect prevention tech-
niques. Games can attract interested players and motivate people to make investments in pre-
vention activities on their own. There have been many scientific studies around the use of
games in various situations. Game theory, pioneered in the 1920s by John von Neumann, is a
useful foundation to aid in the design of productivity games.There are a wide variety of games
that can be applied to help change people’s behavior. In the daily regimen of the software
development cycle, there’s usually very little thought about the “big picture” and preventing
defects. The focus is on the immediate problems, and simple competition and games can add
some fun and excitement to defect prevention work. The success of the game in improving
productivity is dependent on game and reward design. There are several key factors that can
make or break the use of games as a successful defect prevention technique. Games can be
simple or elaborate but must consider the behavioral impact on the players–games should
encourage the desired effort. Reward design is critical to a successful program. Smaller
rewards and smaller game programs are usually better because they provide an opportunity
for frequent design and reward adjustments. Productivity games provide a tremendous oppor-
tunity to attract effort toward defect prevention in a fun and exciting way.
Chapter 5 Using Productivity Games to Prevent Defects 95
Suggested Reading
GameTheory.net. Home page. www.gametheory.net.
IEEE Computer Society Task Force on Game Technologies (TFGT). “IEEE-CS Task Force on
Game Technology: Publications.” www.ucalgary.ca/~jparker/TFGT/publications.html.
“Kids Games.” www.gameskidsplay.net/.
Marks, Robert. “Strategic Game Theory for Managers.” www.agsm.edu.au/~bobm/teaching/
SGTM.html.
Serious Games Initiative. Home page. www.seriousgames.org.
“Game Theory: Types of Games,” http://library.thinkquest.org/26408/math/ gametypes.shtml.
ThinkQuest. “Game Theory: An Example game.” library.thinkquest.org/26408/math/
prisoner.shtml.
Walpole, Stuart. “Designing Games for the Wage Slave.” GameDev.net. www.gamedev.net/
reference/articles/article2121.asp.
97
Chapter 6
Improving the Testability
of Software
A fundamental part of defect prevention is to discover defects as efficiently as possible even
after they are introduced into software. Architecting, designing, and implementing software
that is relatively easy to test increase the efficiency of the testing process and make defects
more discoverable. Testability is the degree to which components and systems are designed
and implemented for test automation to achieve complete code path coverage and simulate all
usage situations in a cost-efficient manner. This chapter introduces the Simplicity, Observabil-
ity, Control, and Knowledge (SOCK) model of testability and provides practical guidelines for
improving testability.
Realizing the Benefits of Testability
Testing can be one of the most expensive parts of the product life cycle. Anything that can be
done to reduce the cost of testing is obviously a huge benefit. Architecting and designing test-
ability into a product from the start go a long way toward reducing the cost of testing and
making maintenance of the software easier (that is, more cost effective).
Software that is highly testable also tends to be well designed. In the Net Objectives Design
Patterns1 curriculum based on the book Design Patterns Explained: A New Perspective on Object-
Oriented Design,2 by Alan Shalloway and James R. Trott, the authors state that there is an
important link between testability and good design: “Easy testability is tightly correlated to
loose coupling and strong cohesion [two main pillars of good design].” In the context of unit
testing, they also assert the following:
Code that is difficult to unit test is often:
Tightly coupled: “I cannot test this without instantiating half the system.”
Weakly cohesive: “This class does so much, the test will be enormous and complex!”
Redundant: “I’ll have to test this in multiple places to ensure that it works everywhere.”
By keeping testability in mind during the architecture, design, and implementation phases,
software becomes better designed and more easily diagnosed when problems arise, which can
lead to lower maintenance costs.
1 Net Objectives, “Design Patterns Training,” www.netobjectives.com/courses/c_design_patterns.htm.
2 Alan Shalloway and James Trott, Design Patterns Explained: A New Perspective on Object-Oriented Design, 2nd ed.
(Boston: Addison-Wesley Professional, 2004).
98 Part II Defect Detection Techniques
The same methods used to make the software more testable can also be used to make the
software more manageable, for example, making it easier to control remotely or to script com-
mon actions or detect and centrally report bad behavior or significant events. The same tenets
of testability also provide the foundation for self-healing software: Software that has the ability
to observe its own state, know if the state is operationally valid, and if not, take an action to
return the software to a correct state or at the very least report on the observed operational
failure.
Testability doesn’t come free, but a good balance of testability features can reduce the overall
cost of developing, supporting, and servicing the software. Early thinking about “how are we
going to test this?” starts the discussion on testability features, which in turn leads to good
design and allows for efficient creation of test automation that fully exercises the software
under test. The same testability features can also be used to improve the diagnosability, man-
ageability, and supportability of the software, which then continue to benefit the software
throughout its support and servicing lifetime.
Implementing Testability
Testability is tied to test automation and its ability to programmatically verify whether soft-
ware is working the way it should and whether the actions of the software under test are hav-
ing the expected effect on the system on which it is running. In their 1992 paper How to
Automate Testing—the Big Picture,1 Keith Stobie and Mark Bergman describe effective test auto-
mation in terms of the acronym SEARCH: Setup, Execution, Analysis, Reporting, Cleanup,
and Help. Every automated test must include each of these elements to be a true and effective
test. Setup is the effort it takes to bring the software up to the point where the actual test oper-
ation is ready to be executed. Execution of the test is then followed by an analysis of whether
the software did what it was supposed to do and didn’t have any unintended side effects on
the system as a whole. Reporting the results of the analysis is then followed by a cleanup phase
that returns the software to a known state so that the next test can proceed. For the test to be
runnable and maintainable by more than just the person who wrote it, a decent help system
should be in place as well to describe the three basics of test documentation: How to set up
the tests, how to run the tests, and how to interpret the results.
The goal of creating more testable software is to help facilitate effective test automation by
making it easier to set up, execute, analyze, clean up, and provide straightforward help for the
tests. Software can be designed from the start to allow for efficient test automation. The key
tenets of testability described by Microsoft Test Architect David Catlett are Simplicity,
Observability, Control, and Knowledge of expected results, or SOCK.
1http://www.keithstobie.net/Documents/TestAuto_The_BigPict.PDF
Chapter 6 Improving the Testability of Software 99
Simplicity: Creating Noncomplex Software
Obviously, the more complex software is, the harder it is to test. Software that has a large
number of interrelated or stateful operations is inherently more difficult to test than are
systems that have only a handful of tasks with a limited number of options.
Adhering to the basic pillars of good design of high cohesion, loose coupling, and low
redundancy is a great way to decrease complexity, which in turn increases testability. For
example, software that is designed with high cohesion—meaning each method or function in
the software accomplishes one and only one task—simplifies the test matrix. Not having to go
through a complicated series of operations using a specific set of parameters just to get the
software to the point where the test can actually be executed is a huge testability win.
Similarly, methods that have a minimal amount of coupling, or a low number of dependen-
cies, are much easier to test. For example, using something like a SQL database instead of a
simple text file for configuration storage causes the testing to be much more complicated.
Reading and writing information over the network increase the possibility of encountering
errors, and learning how to set up and maintain a computer running Microsoft SQL Server
can greatly increase the complexity of the test matrix as well as the cost of testing. Testing
code in only one place instead of having redundant code that must be tested in many places
also simplifies testing.
Simplifying the software design to reduce the test matrix and reduce the time it takes to set up
and clean up the tests is a big win for testability. For example, if the only way to clean up the
test pass is to restart the computer, testing will become very expensive in terms of time. One
way to reduce the cost of setup to improve testability is something that many people probably
take for granted: The TCP/IP loopback functionality present in operating systems such as
Microsoft Windows. Setting up a client and server on the same computer and being able to
exercise a majority of the code paths using this simple one-computer configuration goes a
long way toward making network applications more testable. Anything that can be done to
simplify the process of getting the computer into a certain state bodes well for testability.
Another example of simplifying testing is the capability of installing multiple configurations
of the same software package on the same computer.
Componentization
Componentization, the ability to break up functionality into logical groups as separate libraries
that can easily be swapped out, is an important part of making software more testable. For
example, a contract developer working on the networking portion of a tax software package
was required to build the entire client package including his changes and install the entire
client package before he could test his changes. The process took 40 minutes. Not only that,
there were three separate types of software packages, and they could not be installed at the
same time on the same computer. So, to verify a single fix or feature addition, the developer
had at minimum two hours of setup, not including how long it took to remove the software
100 Part II Defect Detection Techniques
and restart his test computer. Needless to say, he admitted that most of the time when he
made changes he tested only one client package instead of all three.
If, however, the software had been componentized, the developer could have swapped in and
out the networking components he was working on without having to spend 40 minutes
building and reinstalling. Also, if the three client configurations could have all been installed
on the same computer, he could have easily tested all three configurations with only the cost
of the initial setup.
The other important testability feature that componentization affords is the ability to insert
test components in place of the real components. This is sometimes called fault injection and
is used to make better use of mock objects, modules that can simulate usage and error condi-
tions without implementing the entire dependent infrastructure. Separating logical tasks into
modules, such as the networking library from the tax software, makes it easier to test these
components separately, which reduces the complexity of setting up the tests, which increases
testability.
Using complexity metrics, one can also get a feeling for how testable software may be.
Complexity is also defined in terms of not only the specific methods or modules in the soft-
ware under test but also the system in which the software operates, its direct and indirect
dependencies. The more complex the software and the systems in which it interacts, the less
testable. Stephen Kan in the second edition of his book Metrics and Models in Software Quality
Engineering1 specifically mentions the usefulness of complexity metrics “To estimate program-
ming and service effort, identify troublesome code, and estimate testing effort” (p. 318, empha-
sis added). Although there is some debate about using specific metrics such as McCabe’s
Cyclomatic Complexity for estimating test effort, most agree it is a good starting point for
identifying the absolute minimum number of test cases needed to completely cover a function
or module. In practical terms, agreeing on a threshold for complexity metrics that have been
proven to be predictors of defects is important not only in reducing the number of defects but
also in maintaining the testability of software.
Another way of reducing the complexity of software is by using established design patterns.
With the use of design patterns comes predictable and inherently more testable software. The
canonical example of a design pattern that greatly improves testability is the Model-View-
Controller (MVC) pattern of user interface (UI) design. A UI becomes more testable when the
business logic is separated from the UI or the view of the model through a controller. This is
because the business logic can then be tested separately from the UI, and if (and when) the UI
changes, the core business logic tests will not have to change. For example, when the interac-
tion with a database or file system is intertwined in the UI code itself, it makes it very difficult
to automate validation of the behavior of both the UI and the business logic, and if (and
when) the UI ever changes, the tests have to be completely rewritten.
1 Kan, Stephen H., Metrics and Models in Software Quality Engineering, 2nd ed. (Addison-Wesley, 2003).
Chapter 6 Improving the Testability of Software 101
Here’s a simple example of how to make a Microsoft .NET console application written in C#
more testable. This application outputs a list of customers from a database.
static void Main1()
{
SqlConnection conn = new SqlConnection(Settings.SqlDatabaseConn);
conn.Open();
SqlCommand command = new SqlCommand(Settings.GetCustomersQuery, conn);
SqlDataReader reader = command.ExecuteReader();
while (reader.Read())
{
Console.WriteLine(reader.GetString(0));
}
reader.Close();
conn.Close();
}
This code seems simple enough until it comes time to automate the validation of the results.
In Windows, this would typically mean writing a complicated batch file to run the application,
capture the standard output, parse it, sort it, and compare it against some known list in a text
file. Then, when the customer decides they want a graphical user interface instead of a console
interface, all the tests have to be rewritten. Instead, if the developer had followed (at least
loosely) the MVC pattern, tests could be written to validate the data directly from the database
along with separate tests to validate the actual displaying of the data. In this case, by compo-
nentizing the interaction with the database into a different library, the application becomes
more testable. This also simplifies the Main1 program quite a bit:
static void Main2()
{
ArrayList results = CustomerList.GetCustomers();
foreach (String result in results)
{
Console.WriteLine(result);
}
}
The CustomerList.GetCustomers method is then in a separate library that the tester could call
directly to validate the results from the database. Then, even if the UI changes, these tests are
still valid.
This type of design for testability also enables a key aspect of testing: fault injection. Fault injec-
tion is the ability to force error conditions on software. The best candidates for fault injection
are at the boundaries of any external dependencies. In this example, it is the interaction with
the SQL database. We’re sure many a programmer is cringing at both the Main1 and Main2
code because it does not handle any error conditions. There are bugs aplenty waiting to be
102 Part II Defect Detection Techniques
uncovered when this poor defenseless application gets put into the real world. In Main1, it
would be very difficult to induce error conditions to force the code to deal with the various
exceptions that could be thrown by the interaction with the SQL database. It would be
relatively expensive to set up a testing environment that forced, for example, a connection
timeout or login failure or query timeout. In Main2, it is much easier to force these conditions
and ensure that the UI code handles them properly. The tester could easily replace the
CustomerList library with his or her own fault injection test library that returns every possible
error condition the UI should handle without needing to set up a complicated test bed.
Enabling fault injection is a key part of designing testable software, and following the key
design principle of loose coupling as well as using design patterns such as the Model-View-
Controller pattern help improve the design as well as the testability of software.
As discussed, testability is greatly improved by following good principles of design,
componentization, monitoring complexity, and using established design patterns such as the
Model-View-Controller pattern. After software is designed, other steps can be taken to
improve testability further.
Observability: Making Software Observable
During the analysis phase of a test, to determine whether a test passes or fails, the test auto-
mation must be able to observe the results of the software under test and its effect on the sys-
tem as a whole. The ability to do this in a cost-efficient manner is the degree to which the
software is observable.
Observability is a very important part of testability because it allows for efficient and accurate
test automation. The more easily the automation can observe the test execution action, the
more accurately it can judge whether the test passes or fails. Sometimes the important infor-
mation required for accurately analyzing test results is not exposed to test automation. This is
especially true in cases where an action may result in more than one reaction that may or may
not be valid based on environmental variables or a particular configuration setting.
For example, a small business version of a server application may accept only 10 connections,
but an enterprise version may accept a thousand connections. If a test application tries mak-
ing 11 connections, it will pass or fail depending on whether the small business or enterprise
version of the software is installed. Either the test suite could duplicate test connection code
and check for the boundary conditions for each server version, increasing the cost of setup,
test automation, and maintenance of the tests, or, if a method were implemented in the server
software that simply returned the version of the software, the test code could query that
method and make pass/fail decisions based on that observable piece of information. Separate
test suites would not have to be run on each version of the software because the test would be
able to automatically adjust its expectations of when a connection is refused based on query-
ing the software under test for its version.
Chapter 6 Improving the Testability of Software 103
Even better, if the queried method returns the threshold value, if the thresholds are changed
later (for example, say, the small business server version increases its acceptable client load to
20), the tests do not have to be changed because they maintain themselves, keying off the
threshold value exposed by all versions of the server application. Additionally, if a new
“medium-sized business” version of the software that supported a maximum of 200 clients
were released, the same tests could be run against the new version with no modifications. In
each of these cases, the test simply scales up connections until receiving a connection refusal,
queries to see whether the actual connections match the threshold value returned by the soft-
ware under test, and passes or fails the test accordingly.
When test automation can query for threshold values, it makes the software more observable
and thus more testable. Other examples of making software more observable include the
following:
Understand any configuration option that changes the behavior of the software under
test. By making these configuration values programmatically observable, you eliminate
the need to hard-code values in the test code, which helps reduce maintenance of the test
code, especially when the software under test changes.
Know timeout values or any “magic” number that influences the observed behavior of
the software. Knowing what these values are ahead of time and how they influence the
observed behavior is important during the analysis phase of a test. For example, a media-
streaming server may reduce or increase the quality of a video stream based on a heuris-
tic calculated from current networking conditions. To validate that the media-streaming
server algorithm is working as expected according to specification, the results of the
heuristic formula must be programmatically observable to the test software.
Discern decision formulas. Knowing which path is taken after a branch statement (such
as an if or case statement) helps make the determination of pass/fail more accurate. For
example, the XPath query language parser has a “fast path” that is activated only if the
Extensible Markup Language (XML) the query is to be run against is in a particular for-
mat. In a specific test, because the results from the XPath query were identical regardless
of whether the fast path was taken, to verify whether the code took this fast path on qual-
ified XML input, the test team had to look at an internal variable as well as the actual
XPath query results. Having programmatic access to that internal variable increased the
observability of the software.
Examine internal resource usage to find memory and handle leaks. Beaconing when
memory is allocated and subsequently deallocated or when an external resource is used
and then disposed of can be an important test and diagnostic tool. In testing for memory
leaks, the software under test can simply beacon when memory or handles are allocated
and deallocated. Then the test software can validate these claims against the actual
observed memory usage recorded by the operating system and can determine whether
there are any resource leaks. This technique can be extended to any type of interaction
with a resource that should be freed after it has been used. For example, a temporary file
104 Part II Defect Detection Techniques
that is created should be deleted after it has been used, and a network connection should
be torn down after a transaction is completed. When software exposes its internal state
to test software in an easily observable manner, this increases the testability of the
software.
Expose errors or warnings the software encounters as it goes through its operations.
This allows test software to have its own self-checking mechanism. For example, a test
may seem to pass (perhaps a method was called and it returns ERROR_SUCCESS, one
of the classic Windows error codes), but the method may have actually encountered
some internal problem, logged an error or warning somewhere else, and returned the
wrong error code, which is unfortunately very common. If there is some programmatic
way of accessing the error log file, such as through the Windows Event Log APIs, the test
will not be fooled by the bogus ERROR_SUCCESS return code.
Beacon major state changes in the software, for example, system initialization or shut-
down, establishment of a connection, or any other change a test automation program
could consume to understand the current state of the software.
Observe what happens after the software completes a task. This is particularly crucial for
certain testing methods, such as Model-Based Testing, which relies on the ability of the
test harness to programmatically evaluate the state of the software under test and com-
pare it with the “model” state, or test oracle. The more observable the software is, the
more accurate the comparison between the model and the actual state can be, thus
enabling a very powerful test methodology. Even if Model-Based Testing is not used, it is
very important to have multiple data points that the test automation can use to make an
accurate and complete analysis of whether the software is behaving correctly. For more
on Model-Based Testing, see Practical Model-Based Testing: A Tools Approach1.
Control: Increasing Control over Software Under Test
A very important aspect of testability is the capability of efficiently and reliably exercising the
software under test through all usage situations. This is considered the main pillar of testabil-
ity: Can test code easily manipulate the software in a way that is more efficient