A Practitioner's Guide To Software Design @Team LiB 11 By Lee Copeland
User Manual:
Open the PDF directly: View PDF .
Page Count: 355 [warning: Documents this large are best viewed by clicking the View PDF Link!]
- Table of Contents
- BackCover
- A Practitioner's Guide to Software Test Design
- Preface
- Chapter 1: The Testing Process
- Chapter 2: Case Studies
- Section I: Black Box Testing Techniques
- Section II: White Box Testing Techniques
- Section III: Testing Paradigms
- Section IV: Supporting Technologies
- Section V: Some Final Thoughts
APractitioner'sGuidetoSoftwareTestDesign
byLeeCopeland ISBN:158053791x
ArtechHouse©2004
Thistextpresentsalltheimportanttestdesigntechniquesinasingle
placeandinaconsistent,andeasy-to-digestformat.Itenablesyouto
choosethebesttestcasedesign,findsoftwaredefects,developoptimal
strategies,andmore.
TableofContents
APractitioner'sGuidetoSoftwareTestDesign
Preface
Chapter1 - TheTestingProcess
Chapter2 - CaseStudies
SectionI-BlackBoxTestingTechniques
Chapter3 - EquivalenceClassTesting
Chapter4 - BoundaryValueTesting
Chapter5 - DecisionTableTesting
Chapter6 - PairwiseTesting
Chapter7 - State-TransitionTesting
Chapter8 - DomainAnalysisTesting
Chapter9 - UseCaseTesting
SectionII-WhiteBoxTestingTechniques
Chapter10 - ControlFlowTesting
Chapter11 - DataFlowTesting
SectionIII-TestingParadigms
Chapter12 - ScriptedTesting
Chapter13 - ExploratoryTesting
Chapter14 - TestPlanning
SectionIV-SupportingTechnologies
Chapter15 - DefectTaxonomies
Chapter16 - WhentoStopTesting
BackCover
Here’sacomprehensive,up-to-dateandpracticalintroductiontosoftwaretest
design.Thisinvaluablebookpresentsalltheimportanttestdesigntechniquesina
singleplaceandinaconsistent,andeasy-to-digestformat.Animmediatelyuseful
handbookfortestengineers,developers,qualityassuranceprofessionals,and
requirementsandsystemsanalysts,itenablesyouto:choosethebesttestcase
design,findsoftwaredefectsinlesstimeandwithfewerresources,anddevelop
optimalstrategiesthathelpreducethelikelihoodofcostlyerrors.Italsoassistsyou
inestimatingtheeffort,timeandcostofgoodtesting.
Numerouscasestudiesandexamplesofsoftwaretestingtechniquesareincluded,
helpingyoutofullyunderstandthepracticalapplicationsofthesetechniques.From
well-establishedtechniquessuchasequivalenceclasses,boundaryvalueanalysis,
decisiontables,andstate-transitiondiagrams,tonewtechniqueslikeusecase
testing,pairwisetesting,andexploratorytesting,thebookisanindispensable
resourcefortestingprofessionalsseekingtoimprovetheirskillsandanexcellent
referenceforcollege-levelcoursesinsoftwaretestdesign.
AbouttheAuthor
LeeCopelandisaninternationallyknownconsultantinsoftwaretesting,withover
30yearsofexperienceasaninformationsystemsprofessional.Hehashelda
numberoftechnicalandmanagerialpositionswithcommercialandnonprofit
organizationsintheareasofsoftwaredevelopment,testing,andprocess
improvement.Hehastaughtseminarsandconsultedextensivelythroughoutthe
UnitedStatesandinternationally.
APractitioner'sGuidetoSoftwareTestDesign
LeeCopeland
ArtechHousePublishers
Boston•London
LibraryofCongressandBritishCIPinformationavailableonrequest685CantonStreet
Norwood,MA02062
(781)769-9750
www.artechhouse.com
46GillinghamStreet
LondonSW1V1AH
+44(0)207596-8750
Copyright©2004STQEPublishing
Allrightsreserved.Nopartofthisbookshallbereproduced,storedinaretrievalsystem,or
transmittedbyanymeans,electronic,mechanical,photocopying,recording,orotherwise
withoutwrittenpermissionfromthepublisher.
InternationalStandardBookNumber:1-58053-791-X
PrintedintheUnitedStatesofAmerica
FirstPrinting:November2003
Trademarks
Alltermsmentionedinthisbookthatareknowntobetrademarksorservicemarkshavebeen
appropriatelycapitalized.ArtechHousePublishersandSTQEPublishingcannotattesttothe
accuracyofthisinformation.Useofaterminthisbookshouldnotberegardedasaffectingthe
validityofanytrademarkorservicemark.
WarningandDisclaimer
Everyefforthasbeenmadetomakethisbookascompleteandaccurateaspossible,butno
warrantyorfitnessisimplied.Theinformationprovidedisonan"asis"basis.Theauthorsand
thepublishershallhaveneitherliabilitynorresponsibilitytoanypersonorentitywithrespectto
anylossordamagesarisingfromtheinformationcontainedinthisbook.
Dedication
TomywifeSuzanne,andourwonderfulchildrenandgrandchildren
ShawnandMartha
AndrewandCassandra
David
Cathleen
KatelynnandKiley
MelissaandJay
Ross,Elizabeth,andMiranda
BrianandHeather
CassidyandCaden
ThomasandJeni
Carrie
Sundari
Rajan
andtoWayne,Jerry,Dani,Ron,andRayannefortheirencouragementovertheyears.
LeeCopelandisaninternationallyknownconsultantinsoftwaretesting,withover30yearsof
experienceasaninformationsystemsprofessional.Hehasheldanumberoftechnicaland
managerialpositionswithcommercialandnonprofitorganizationsintheareasofsoftware
development,testing,andprocessimprovement.Hehastaughtseminarsandconsulted
extensivelythroughouttheUnitedStatesandinternationally.
AsaconsultantforSoftwareQualityEngineering,Leetravelstheworldpromotingeffective
softwaretestingtohisclients.Inaddition,heistheprogramchairforSTAREASTand
STARWEST,theworld'spremierconferencesonsoftwaretesting.
Preface
APractitioner'sGuidetoSoftwareTestDesigncontainstoday'simportantcurrenttestdesign
approachesinoneuniquebook.Untilnow,softwaretestershadtosearchthroughanumberof
books,periodicals,andWebsitestolocatethisvitalinformation.
ImportanceofTestDesign
"Theactofcareful,complete,systematic,testdesignwillcatchasmanybugsastheact
oftesting....Personally,Ibelievethatit'sfarmoreeffective."
-BorisBeizer
Thebookfocusesonlyonsoftwaretestdesign,notrelatedsubjectssuchastestplanning,test
management,testteamdevelopment,etc.Whilethoseareimportantinsoftwaretesting,they
haveoftenovershadowedwhattestersreallyneed—themorepracticalaspectsoftesting,
specificallytestcasedesign.Otherexcellentbookscanguideyouthroughtheoverallprocess
ofsoftwaretesting.OneofmyfavoritesisSystematicSoftwareTestingbyRickCraigand
StefanJaskiel.
APractitioner'sGuidetoSoftwareTestDesignillustrateseachtestdesignapproachthrough
detailedexamplesandstep-by-stepinstructions.Theseleadthereadertoaclear
understandingofeachtestdesigntechnique.
Today'sTestingChallenges
Foranysystemofinterestingsizeitisimpossibletotestallthedifferentlogicpathsandallthe
differentinputdatacombinations.Oftheinfinitenumberofchoices,eachoneofwhichisworthy
ofsomeleveloftesting,testerscanonlychooseaverysmallsubsetbecauseofresource
constraints.Thepurposeofthisbookistohelpyouanalyze,design,andchoosesuchsubsets,
toimplementthoseteststhataremostlikelytodiscoverdefects.
Itisvitaltochoosetestcaseswisely.Missingadefectcanresultinsignificantlossestoyour
organizationifadefectivesystemisplacedintoproduction.
APractitioner'sGuidetoSoftwareTestDesigndescribesasetofkeytestdesignstrategies
thatimproveboththeefficiencyandeffectivenessofsoftwaretesters.
StructureandApproach
APractitioner'sGuidetoSoftwareTestDesignexplainsthemostimportanttestdesign
techniquesinusetoday.Someofthesetechniquesareclassicsandwellknownthroughoutthe
testingcommunity.Somehavebeenaroundforawhilebutarenotwellknownamongtest
engineers.Stillothersarenotwidelyknown,butshouldbebecauseoftheireffectiveness.This
bookbringstogetherallthesetechniquesintoonevolume,helpingthetestdesignerbecome
moreefficientandeffectiveintesting.
Eachtestdesigntechniqueisapproachedfromapractical,ratherthanatheoreticalbasis.
Eachtestdesigntechniqueisfirstintroducedthroughasimpleexample,thenexplainedindetail.
Whenpossible,additionalexamplesofitsusearepresented.Thetypesofproblemsonwhich
theapproachcanbeused,alongwithitslimitations,aredescribed.Eachtestdesigntechnique
chapterendswithasummaryofitskeypoints,alongwithexercisesthereadercanusefor
practice,andreferencesforfurtherreading.Testerscanusethetechniquespresented
immediatelyontheirprojects.
ANotefromtheAuthor
Iloveagooddoubleintegralsign
asmuchasthenexttester,butwe'regoingtoconcentrateonthepractical,notthe
theoretical.
Eachtestdesignapproachisdescribedinaself-containedchapter.Becausethechaptersare
focused,concise,andindependenttheycanberead"outoforder."Testerscanreadthe
chaptersthataremostrelevanttotheirworkatthemoment.
Audience
Thisbookwaswrittenspecificallyfor:
Softwaretestengineerswhohavetheprimaryresponsibilityfortestcasedesign.This
bookdetailsthemostefficientandeffectivemethodsforcreatingtestcases.
Softwaredeveloperswho,withtheadventofExtremeProgrammingandotheragile
developmentmethods,arebeingaskedtodomoreandbettertestingofthesoftware
theywrite.Manydevelopershavenotbeenexposedtothedesigntechniques
describedinthisbook.
Testanddevelopmentmanagerswhomustunderstand,atleastinprinciple,thework
theirstaffperforms.Notonlydoesthisbookprovideanoverviewofimportanttest
designmethods,itwillassistmanagersinestimatingtheeffort,time,andcostofgood
testing.
Qualityassuranceandprocessimprovementengineerswhoarechargedwithdefining
andimprovingtheirsoftwaretestingprocess.
Instructorsandprofessorswhoaresearchingforanexcellentreferenceforacoursein
softwaretestdesigntechniques.
Appreciation
Thefollowingreviewershaveprovidedinvaluableassistanceinthewritingofthisbook:Anne
Meilof,ChuckAllison,DalePerry,DannyFaught,DorothyGraham,GeoffQuentin,James
Bach,JonHagar,PaulGerrard,RexBlack,RickCraig,RobertRose-Coutré,SidSnook,and
WayneMiddleton.Mysincerethankstoeachofthem.Anyfaultsinthisbookshouldbe
attributeddirectlytothem.(Justkidding!)
SomeFinalComments
ThisbookcontainsanumberofreferencestoWebsites.Thesereferenceswerecorrectwhen
themanuscriptwassubmittedtothepublisher.Unfortunately,theymayhavebecomebrokenby
thetimethebookisinthereaders'hands.
Ithasbecomestandardpracticeforauthorstoincludeapithyquotationonthetitlepageof
eachchapter.Unfortunately,thepracticehasbecomesoprevalentthatallthegoodquotations
havebeenused.Justforfun,Ihavechoseninsteadtoincludeoneachchaptertitlepagea
winningentryfromthe2003Bulwer-LyttonFictionContest(http://www.bulwer-lytton.com).
Since1982,theEnglishDepartmentatSanJoseStateUniversityhassponsoredthisevent,a
competitionthatchallengeswriterstocomposetheopeningsentencetotheworstofall
possiblenovels.ItwasinspiredbyEdwardGeorgeBulwer-LyttonwhobeganhisnovelPaul
Cliffordwith:
"Itwasadarkandstormynight;therainfellintorrents—exceptatoccasionalintervals,
whenitwascheckedbyaviolentgustofwindwhichsweptupthestreets(foritisin
Londonthatourscenelies),rattlingalongthehousetops,andfiercelyagitatingthescanty
flameofthelampsthatstruggledagainstthedarkness."
MyappreciationtoDr.ScottRiceofSanJoseStateUniversityforpermissiontousethese
exemplaryillustrationsofbadwriting.Hopefully,nothinginthisbookwillwinthisprestigious
award.
Acknowledgements
ThecaricatureofMickJaggerisownedandcopyrightedbyMartinO'Loughlinandusedby
permission.
ClipArtcopyrightbyCorelCorporationandusedunderalicensingagreement.
References
Beizer,Boris(1990).SoftwareTestingTechniques(SecondEdition).VanNostrand
Reinhold.
Craig,RickD.andStefanP.Jaskiel(2002).SystematicSoftwareTesting.ArtechHouse
Publishers.
Chapter1:TheTestingProcess
Overview
Theflockofgeeseflewoverheadina'V'formation—notinanold-fashioned-looking
TimesNewRomankindofa'V',branchedoutslightlyatthetwooppositearmsatthetop
ofthe'V',norinamoremodern-looking,straightandcrisp,linearArialsortof'V'
(althoughsincetheywereflying,Arialmighthavebeenappropriate),butinaslightly
asymmetric,tiltingoff-to-one-sidesortofitalicizedCourierNew-like'V'—andLaFonte
knewthathewasjustthetypeofmantoknowthedifference.[1]
—JohnDotson
[1]Ifyouthinkthisquotationhasnothingtodowithsoftwaretestingyouarecorrect.Foran
explanationpleaseread"SomeFinalComments"inthePreface.
Testing
Whatistesting?Whilemanydefinitionshavebeenwritten,atitscoretestingistheprocessof
comparing"whatis"with"whatoughttobe."AmoreformaldefinitionisgivenintheIEEE
Standard610.12-1990,"IEEEStandardGlossaryofSoftwareEngineeringTerminology"which
defines"testing"as:
"Theprocessofoperatingasystemorcomponentunderspecifiedconditions,observingor
recordingtheresults,andmakinganevaluationofsomeaspectofthesystemor
component."
The"specifiedconditions"referredtointhisdefinitionareembodiedintestcases,thesubject
ofthisbook.
Key
Point Atitscore,testingistheprocessofcomparing"whatis"with"whatoughttobe."
RickCraigandStefanJaskielproposeanexpandeddefinitionofsoftwaretestingintheirbook,
SystematicSoftwareTesting.
"Testingisaconcurrentlifecycleprocessofengineering,usingandmaintainingtestwarein
ordertomeasureandimprovethequalityofthesoftwarebeingtested."
Thisviewincludestheplanning,analysis,anddesignthatleadstothecreationoftestcasesin
additiontotheIEEE'sfocusontestexecution.
Differentorganizationsanddifferentindividualshavevariedviewsofthepurposeofsoftware
testing.BorisBeizerdescribesfivelevelsoftestingmaturity.(Hecalledthemphasesbuttoday
weknowthepoliticallycorrecttermis"levels"andtherearealwaysfiveofthem.)
Level0-"There'snodifferencebetweentestinganddebugging.Otherthaninsupportof
debugging,testinghasnopurpose."Defectsmaybestumbleduponbutthereisnoformalized
efforttofindthem.
Level1-"Thepurposeoftestingistoshowthatsoftwareworks."Thisapproach,whichstarts
withthepremisethatthesoftwareis(basically)correct,mayblindustodiscoveringdefects.
GlenfordMyerswrotethatthoseperformingthetestingmaysubconsciouslyselecttestcases
thatshouldnotfail.Theywillnotcreatethe"diabolical"testsneededtofinddeeplyhidden
defects.
Level2-"Thepurposeoftestingistoshowthatthesoftwaredoesn'twork."Thisisavery
differentmindset.Itassumesthesoftwaredoesn'tworkandchallengesthetestertofindits
defects.Withthisapproach,wewillconsciouslyselecttestcasesthatevaluatethesystemin
itsnooksandcrannies,atitsboundaries,andnearitsedges,usingdiabolicallyconstructedtest
cases.
Level3-"Thepurposeoftestingisnottoproveanything,buttoreducetheperceivedriskof
notworkingtoanacceptablevalue."Whilewecanproveasystemincorrectwithonlyonetest
case,itisimpossibletoeverproveitcorrect.Todosowouldrequireustotesteverypossible
validcombinationofinputdataandeverypossibleinvalidcombinationofinputdata.Ourgoals
aretounderstandthequalityofthesoftwareintermsofitsdefects,tofurnishtheprogrammers
withinformationaboutthesoftware'sdeficiencies,andtoprovidemanagementwithan
evaluationofthenegativeimpactonourorganizationifweshippedthissystemtocustomersin
itspresentstate.
Level4-"Testingisnotanact.Itisamentaldisciplinethatresultsinlow-risksoftwarewithout
muchtestingeffort."Atthismaturitylevelwefocusonmakingsoftwaremoretestablefromits
inception.Thisincludesreviewsandinspectionsofitsrequirements,design,andcode.In
addition,itmeanswritingcodethatincorporatesfacilitiesthetestercaneasilyuseto
interrogateitwhileitisexecuting.Further,itmeanswritingcodethatisself-diagnosing,that
reportserrorsratherthanrequiringtesterstodiscoverthem.
CurrentChallenges
WhenIaskmystudentsaboutthechallengestheyfaceintestingtheytypicallyreply:
Notenoughtimetotestproperly
Toomanycombinationsofinputstotest
Notenoughtimetotestwell
Difficultyindeterminingtheexpectedresultsofeachtest
Nonexistentorrapidlychangingrequirements
Notenoughtimetotestthoroughly
Notrainingintestingprocesses
Notoolsupport
Managementthateitherdoesn'tunderstandtestingor(apparently)doesn'tcareabout
quality
Notenoughtime
Thisbookdoesnotcontain"magicpixiedust"thatyoucanusetocreateadditionaltime,better
requirements,ormoreenlightenedmanagement.Itdoes,however,containtechniquesthatwill
makeyoumoreefficientandeffectiveinyourtestingbyhelpingyouchooseandconstructtest
casesthatwillfindsubstantiallymoredefectsthanyouhaveinthepastwhileusingfewer
resources.
TestCases
Tobemosteffectiveandefficient,testcasesmustbedesigned,notjustslappedtogether.The
word"design"hasanumberofdefinitions:
1. Toconceiveorfashioninthemind;invent:designagoodreasontoattendtheSTAR
testingconference.Toformulateaplanfor;devise:designamarketingstrategyfor
thenewproduct.
2. Toplanoutinsystematic,usuallydocumentedform:designabuilding;designatest
case.
3. Tocreateorcontriveforaparticularpurposeoreffect:agamedesignedtoappealto
allages.
4. Tohaveasagoalorpurpose;intend.
5. Tocreateorexecuteinanartisticorhighlyskilledmanner.
Key
Point
Tobemosteffectiveandefficient,testcasesmustbedesigned,notjustslapped
together.
Eachofthesedefinitionsappliestogoodtestcasedesign.Regardingtestcasedesign,Roger
Pressmanwrote:
"Thedesignoftestsforsoftwareandotherengineeringproductscanbeaschallengingas
theinitialdesignoftheproductitself.Yet...softwareengineersoftentreattestingasan
afterthought,developingtestcasesthat'feelright'buthavelittleassuranceofbeing
complete.Recallingtheobjectivesoftesting,wemustdesignteststhathavethehighest
likelihoodoffindingthemosterrorswithaminimumamountoftimeandeffort."
Welldesignedtestcasesarecomposedofthreeparts:
Inputs
Outputs
Orderofexecution
Key
Point Testcasesconsistofinputs,outputs,andorderofexecution.
Inputs
Inputsarecommonlythoughtofasdataenteredatakeyboard.Whilethatisasignificant
sourceofsysteminput,datacancomefromothersources—datafrominterfacingsystems,
datafrominterfacingdevices,datareadfromfilesordatabases,thestatethesystemisin
whenthedataarrives,andtheenvironmentwithinwhichthesystemexecutes.
Outputs
Outputshavethissamevariety.Oftenoutputsarethoughtofasjustthedatadisplayedona
computerscreen.Inaddition,datacanbesenttointerfacingsystemsandtoexternaldevices.
Datacanbewrittentofilesordatabases.Thestateortheenvironmentmaybemodifiedbythe
system'sexecution.
Alloftheserelevantinputsandoutputsareimportantcomponentsofatestcase.Intestcase
design,determiningtheexpectedoutputsisthefunctionofan"oracle."
Anoracleisanyprogram,process,ordatathatprovidesthetestdesignerwiththeexpected
resultofatest.Beizerlistsfivetypesoforacles:
KiddieOracles-Justruntheprogramandseewhatcomesout.Ifitlooksaboutright,it
mustberight.
RegressionTestSuites-Runtheprogramandcomparetheoutputtotheresultsofthe
sametestsrunagainstapreviousversionoftheprogram.
ValidatedData-Runtheprogramandcomparetheresultsagainstastandardsuchas
atable,formula,orotheraccepteddefinitionofvalidoutput.
PurchasedTestSuites-Runtheprogramagainstastandardizedtestsuitethathas
beenpreviouslycreatedandvalidated.Programslikecompilers,Webbrowsers,and
SQL(StructuredQueryLanguage)processorsareoftentestedagainstsuchsuites.
ExistingProgram-Runtheprogramandcomparetheoutputtoanotherversionofthe
program.
OrderofExecution
Therearetwostylesoftestcasedesignregardingorderoftestexecution.
Cascadingtestcases-Testcasesmaybuildoneachother.Forexample,thefirsttest
caseexercisesaparticularfeatureofthesoftwareandthenleavesthesystemina
statesuchthatthesecondtestcasecanbeexecuted.Intestingadatabaseconsider
thesetestcases:
1. Createarecord
2. Readtherecord
3. Updatetherecord
4. Readtherecord
5. Deletetherecord
6. Readthedeletedrecord
Eachofthesetestscouldbebuiltontheprevioustests.Theadvantageisthateach
testcaseistypicallysmallerandsimpler.Thedisadvantageisthatifonetestfails,the
subsequenttestsmaybeinvalid.
Independenttestcases-Eachtestcaseisentirelyselfcontained.Testsdonotbuildon
eachotherorrequirethatothertestshavebeensuccessfullyexecuted.Theadvantage
isthatanynumberoftestscanbeexecutedinanyorder.Thedisadvantageisthateach
testtendstobelargerandmorecomplexandthusmoredifficulttodesign,create,and
maintain.
TypesOfTesting
Testingisoftendividedintoblackboxtestingandwhiteboxtesting.
Blackboxtestingisastrategyinwhichtestingisbasedsolelyontherequirementsand
specifications.Unlikeitscomplement,whiteboxtesting,blackboxtestingrequiresno
knowledgeoftheinternalpaths,structure,orimplementationofthesoftwareundertest.
Whiteboxtestingisastrategyinwhichtestingisbasedontheinternalpaths,structure,and
implementationofthesoftwareundertest.Unlikeitscomplement,blackboxtesting,whitebox
testinggenerallyrequiresdetailedprogrammingskills.
Anadditionaltypeoftestingiscalledgrayboxtesting.Inthisapproachwepeekintothe"box"
undertestjustlongenoughtounderstandhowithasbeenimplemented.Thenwecloseupthe
boxanduseourknowledgetochoosemoreeffectiveblackboxtests.
TestingLevels
Typicallytesting,andthereforetestcasedesign,isperformedatfourdifferentlevels:
UnitTesting-Aunitisthe"smallest"pieceofsoftwarethatadevelopercreates.Itis
typicallytheworkofoneprogrammerandisstoredinasinglediskfile.Different
programminglanguageshavedifferentunits:InC++andJavatheunitistheclass;inC
theunitisthefunction;inlessstructuredlanguageslikeBasicandCOBOLtheunitmay
betheentireprogram.
Key
Point Theclassicaltestinglevelsareunit,integration,system,andacceptance.
IntegrationTesting-Inintegrationweassembleunitstogetherintosubsystemsand
finallyintosystems.Itispossibleforunitstofunctionperfectlyinisolationbuttofail
whenintegrated.AclassicexampleisthisCprogramanditssubsidiaryfunction:
/*mainprogram*/
voidoops(int);
intmain(){
oops(42);/*calltheoopsfunctionpassinganinteger*/
return0;
}
/*functionoops(inaseparatefile)*/
#include<stdio.h>
voidoops(doublex){/*expectsadouble,notanint!*/
printf("%f\n",x);/*Willprintgarbage(0ismostlikely)*/
}
Iftheseunitsweretestedindividually,eachwouldappeartofunctioncorrectly.Inthis
case,thedefectonlyappearswhenthetwounitsareintegrated.Themainprogram
passesanintegertofunctionoopsbutoopsexpectsadoublelengthintegerandtrouble
ensues.Itisvitaltoperformintegrationtestingastheintegrationprocessproceeds.
SystemTesting-Asystemconsistsofallofthesoftware(andpossiblyhardware,user
manuals,trainingmaterials,etc.)thatmakeuptheproductdeliveredtothecustomer.
Systemtestingfocusesondefectsthatariseatthishighestlevelofintegration.
Typicallysystemtestingincludesmanytypesoftesting:functionality,usability,security,
internationalizationandlocalization,reliabilityandavailability,capacity,performance,
backupandrecovery,portability,andmanymore.Thisbookdealsonlywithfunctionality
testing.Whiletheothertypesoftestingareimportant,theyarebeyondthescopeof
thisvolume.
AcceptanceTesting-Acceptancetestingisdefinedasthattesting,whichwhen
completedsuccessfully,willresultinthecustomeracceptingthesoftwareandgivingus
theirmoney.Fromthecustomer'spointofview,theywouldgenerallylikethemost
exhaustiveacceptancetestingpossible(equivalenttothelevelofsystemtesting).From
thevendor'spointofview,wewouldgenerallyliketheminimumleveloftestingpossible
thatwouldresultinmoneychanginghands.Typicalstrategicquestionsthatshouldbe
addressedbeforeacceptancetestingare:Whodefinestheleveloftheacceptance
testing?Whocreatesthetestscripts?Whoexecutesthetests?Whatisthepass/fail
criteriafortheacceptancetest?Whenandhowdowegetpaid?
Notallsystemsareamenabletousingtheselevels.Theselevelsassumethatthereisa
significantperiodoftimebetweendevelopingunitsandintegratingthemintosubsystemsand
thenintosystems.InWebdevelopmentitisoftenpossibletogofromconcepttocodeto
productioninamatterofhours.Inthatcase,theunit-integration-systemlevelsdon'tmakemuch
sense.ManyWebtestersuseanalternatesetoflevels:
Codequality
Functionality
Usability
Performance
Security
TheImpossibilityOfTestingEverything
InhismonumentalbookTestingObject-OrientedSystems,RobertBinderprovidesanexcellent
exampleoftheimpossibilityoftesting"everything."Considerthefollowingprogram:
intblech(intj){
j=j-1;//shouldbej=j+1
j=j/30000;
returnj;
}
Notethatthesecondlineisincorrect!Thefunctionblechacceptsanintegerj,subtractsone
fromit,dividesitby30000(integerdivision,wholenumbers,noremainder)andreturnsthe
valuejustcomputed.Ifintegersareimplementedusing16bitsonthiscomputerexecutingthis
software,thelowestpossibleinputvalueis-32768andthehighestis32767.Thusthereare
65,536possibleinputsintothistinyprogram.(Yourorganization'sprogramsareprobably
larger.)Willyouhavethetime(andthestamina)tocreate65,536testcases?Ofcoursenot.
Sowhichinputvaluesdowechoose?Considerthefollowinginputvaluesandtheirabilityto
detectthisdefect.
Input(j) ExpectedResult ActualResult
1 0 0
42 0 0
40000 1 1
-64000 -2 -2
Oops!Notethatnoneofthetestcaseschosenhavedetectedthisdefect.Infactonlyfourof
thepossible65,536inputvalueswillfindthisdefect.Whatisthechancethatyouwillchooseall
four?Whatisthechanceyouwillchooseoneofthefour?Whatisthechanceyouwillwinthe
Powerballlottery?Isyouranswerthesametoeachofthesethreequestions?
Summary
Testingisaconcurrentlifecycleprocessofengineering,using,andmaintainingtestware
inordertomeasureandimprovethequalityofthesoftwarebeingtested.(Craigand
Jaskiel)
Thedesignoftestsforsoftwareandotherengineeringproductscanbeaschallenging
astheinitialdesignoftheproductitself.Yet...softwareengineersoftentreattesting
asanafterthought,developingtestcasesthat'feelright'buthavelittleassuranceof
beingcomplete.Recallingtheobjectivesoftesting,wemustdesignteststhathavethe
highestlikelihoodoffindingthemosterrorswithaminimumamountoftimeandeffort.
(Pressman)
Blackboxtestingisastrategyinwhichtestingisbasedsolelyontherequirementsand
specifications.Whiteboxtestingisastrategyinwhichtestingisbasedontheinternal
paths,structure,andimplementationofthesoftwareundertest.
Typicallytesting,andthereforetestcasedesign,isperformedatfourdifferentlevels:
Unit,Integration,System,andAcceptance.
Practice
1. Whichfourinputstotheblechroutinewillfindthehiddendefect?Howdidyou
determinethem?Whatdoesthissuggesttoyouasanapproachtofindingother
defects?
References
Beizer,Boris(1990).SoftwareTestingTechniques(SecondEdition).VanNostrand
Reinhold.
Binder,RobertV.(2000).TestingObject-OrientedSystems:Models,Patterns,and
Tools.Addison-Wesley.
Craig,RickD.andStefanP.Jaskiel(2002).SystematicSoftwareTesting.ArtechHouse
Publishers.
IEEEStandard610.12-1990,IEEEStandardGlossaryofSoftwareEngineering
Terminology,1991.
Myers,Glenford(1979).TheArtofSoftwareTesting.JohnWiley&Sons.
Pressman,RogerS.(1982).SoftwareEngineering:APractitioner'sApproach(Fourth
Edition).McGraw-Hill.
Chapter2:CaseStudies
Theyhadbutonelastremainingnighttogether,sotheyembracedeachotherastightlyas
thattwo-flavorentwinedstringcheesethatisorangeandyellowish-white,theorange
probablybeingablandCheddarandthewhite...Mozzarella,althoughitcouldpossibly
beProvoloneorjustplainAmerican,asitreallydoesn'ttastedistinctlydissimilarfromthe
orange,yettheywouldhaveyoubelieveitdoesbycoloringitdifferently.
—MariannSimms
WhyCaseStudies?
Twocasestudiesareprovidedintheappendicesofthisbook.AppendixAdescribes"Brown&
Donaldson,"anonlinebrokeragefirm.AppendixBdescribesthe"StatelessUniversity
RegistrationSystem."Examplesfromthesecasestudiesareusedtoillustratethetestcase
designtechniquesdescribedinthisbook.Inaddition,someofthebook'sexercisesarebased
onthecasestudies.Thefollowingsectionsbrieflydescribethecasestudies.Readthedetailed
informationinAppendixAandBwhenrequired.
Brown&Donaldson
Brown&Donaldson(B&D)isafictitiousonlinebrokeragefirmthatyoucanusetopracticethe
testdesigntechniquespresentedinthisbook.B&DwasoriginallycreatedforSoftwareQuality
Engineering'sWeb/eBusinessTestingcourse(formoredetailsseehttp://www.sqe.com).
ScreenshotsofvariouspagesareincludedinAppendixA.Referencewillbemadetosomeof
thesethroughoutthebook.TheactualB&DWebsiteisfoundathttp://bdonline.sqe.com.Any
resemblancetoanyactualonlinebrokerageWebsiteispurelycoincidental.
YoucanactuallytrytheB&DWebsite.First-timeuserswillneedtocreateaBDonlineaccount.
Thisaccountisnotreal—anytransactionsrequestedorexecutedviathisaccountwillnot
occurintherealworld,onlyinthefictitiousworldofB&D.Onceyouhavecreatedanaccount,
youwillbypassthisstepandloginwithyourusernameandpassword.Whilecreatinganew
accountyouwillbeaskedtosupplyanauthorizationcode.Theauthorizationcodeiseight1s.
ThisWebsitealsocontainsanumberofdownloadabledocumentsfromtheB&Dcasestudy,
whichcanbeusedtoassistyouindevelopingtestplansforyourownWebprojects.
StatelessUniversityRegistrationSystem
Everystatehasastateuniversity.Thiscasestudydescribesanonlinestudentregistration
systemforthefictitiousStatelessUniversity.Pleasedonotattempttocashoutyourstocks
fromBrown&DonaldsontoenrollatStatelessU.
ThedocumentinAppendixBdescribestheplanneduserinterfacefortheStatelessUniversity
RegistrationSystem(SURS).Itdefinestheuserinterfacescreensintheorderinwhichtheyare
typicallyused.Itstartswiththeloginscreen.Thenitprovidesthedatabaseset-upfields,the
addition/change/deletionofstudents,theaddition/change/deletionofcourses,andthe
addition/change/deletionofclasssections.Thefinaldataentryscreenprovidestheselectionof
specificcoursesectionsforeachstudent.Additionaladministrativefunctionsarealsodefined.
Definition
Blackboxtestingisastrategyinwhichtestingisbasedsolelyontherequirementsand
specifications.Unlikeitscomplement,whiteboxtesting,blackboxtestingrequiresno
knowledgeoftheinternalpaths,structure,orimplementationofthesoftwareundertest(SUT).
Thegeneralblackboxtestingprocessis:
Therequirementsorspecificationsareanalyzed.
ValidinputsarechosenbasedonthespecificationtodeterminethattheSUTprocesses
themcorrectly.InvalidinputsmustalsobechosentoverifythattheSUTdetectsthem
andhandlesthemproperly.
Expectedoutputsforthoseinputsaredetermined.
Testsareconstructedwiththeselectedinputs.
Thetestsarerun.
Actualoutputsarecomparedwiththeexpectedoutputs.
AdeterminationismadeastotheproperfunctioningoftheSUT.
Applicability
Blackboxtestingcanbeappliedatalllevelsofsystemdevelopment—unit,integration,system,
andacceptance.
Aswemoveupinsizefrommoduletosubsystemtosystemtheboxgetslarger,withmore
complexinputsandmorecomplexoutputs,buttheapproachremainsthesame.Also,aswe
moveupinsize,weareforcedtotheblackboxapproach;therearesimplytoomanypaths
throughtheSUTtoperformwhiteboxtesting.
Disadvantages
Whenusingblackboxtesting,thetestercanneverbesureofhowmuchoftheSUThasbeen
tested.Nomatterhowcleverordiligentthetester,someexecutionpathsmayneverbe
exercised.Forexample,whatistheprobabilityatesterwouldselectatestcasetodiscover
this"feature"?
if(name=="Lee"&&employeeNumber=="1234"&&
employmentStatus=="RecentlyTerminatedForCause"){
sendLeeacheckfor$1,000,000;
}
Key
Point
Whenusingblackboxtesting,thetestercanneverbesureofhowmuchofthe
systemundertesthasbeentested.
Tofindeverydefectusingblackboxtesting,thetesterwouldhavetocreateeverypossible
combinationofinputdata,bothvalidandinvalid.Thisexhaustiveinputtestingisalmostalways
impossible.Wecanonlychooseasubset(oftenaverysmallsubset)oftheinputcombinations.
InTheArtofSoftwareTesting,GlenfordMyersprovidesanexcellentexampleofthefutilityof
exhaustivetesting:Howwouldyouthoroughlytestacompiler?Bywritingeverypossiblevalid
andinvalidprogram.Theproblemissubstantiallyworseforsystemsthatmustrememberwhat
hashappenedbefore(i.e.,thatremembertheirstate).Inthosesystems,notonlymustwetest
everypossibleinput,wemusttesteverypossiblesequenceofeverypossibleinput.
Key
Point
Eventhoughwecan'ttesteverything,formalblackboxtestingdirectsthetesterto
choosesubsetsofteststhatarebothefficientandeffectiveinfindingdefects.
Advantages
Eventhoughwecan'ttesteverything,formalblackboxtestingdirectsthetestertochoose
subsetsofteststhatarebothefficientandeffectiveinfindingdefects.Assuch,thesesubsets
willfindmoredefectsthanarandomlycreatedequivalentnumberoftests.Blackboxtesting
helpsmaximizethereturnonourtestinginvestment.
References
Myers,GlenfordJ.(1979).TheArtofSoftwareTesting.JohnWiley&Sons.
Chapter3:EquivalenceClassTesting
OnthefourthdayofhisexplorationoftheAmazon,Byronclimbedoutofhisinnertube,
checkedthelatestnewsonhispersonaldigitalassistant(hereafterPDA)outfittedwith
wirelesstechnology,andrealizedthatthegnawinghefeltinhisstomachwasnotfear
—no,hewasnotafraid,ratherelated—norwasittension—no,hewasactuallyrather
relaxed—soitwasinallprobabilityaparasite.
—ChuckKeelan
Introduction
Equivalenceclasstestingisatechniqueusedtoreducethenumberoftestcasestoa
manageablelevelwhilestillmaintainingreasonabletestcoverage.Thissimpletechniqueisused
intuitivelybyalmostalltesters,eventhoughtheymaynotbeawareofitasaformaltestdesign
method.Manytestershavelogicallydeduceditsusefulness,whileothershavediscoveredit
simplybecauseoflackoftimetotestmorethoroughly.
Considerthissituation.Wearewritingamoduleforahumanresourcessystemthatdecides
howweshouldprocessemploymentapplicationsbasedonaperson'sage.Ourorganization's
rulesare:
0–16Don'thire
16–18Canhireonapart-timebasisonly
18–55Canhireasafull-timeemployee
55–99Don'thire[*]
[*]Note:Ifyou'vespottedaproblemwiththeserequirements,don'tworry.Theyarewritten
thiswayforapurposeandwillberepairedinthenextchapter.
Observation
WiththeserulesourorganizationwouldnothavehiredDoogieHouser,M.D.orCol.Harlan
Sanders,onetooyoung,theothertooold.
Shouldwetestthemoduleforthefollowingages:0,1,2,3,4,5,6,7,8,...,90,91,92,93,
94,95,96,97,98,99?Ifwehadlotsoftime(anddidn'tmindthemind-numbingrepetitionand
werebeingpaidbythehour)wecertainlycould.Iftheprogrammerhadimplementedthis
modulewiththefollowingcodeweshouldtesteachage.(Ifyoudon'thaveaprogramming
backgrounddon'tworry.Theseexamplesaresimple.Justreadthecodeanditwillmakesense
toyou.)
If(applicantAge==0)hireStatus="NO";
If(applicantAge==1)hireStatus="NO";
…
If(applicantAge==14)hireStatus="NO";
If(applicantAge==15)hireStatus="NO";
If(applicantAge==16)hireStatus="PART";
If(applicantAge==17)hireStatus="PART";
If(applicantAge==18)hireStatus="FULL";
If(applicantAge==19)hireStatus="FULL";
…
If(applicantAge==53)hireStatus="FULL";
If(applicantAge==54)hireStatus="FULL";
If(applicantAge==55)hireStatus="NO";
If(applicantAge==56)hireStatus="NO";
…
If(applicantAge==98)hireStatus="NO";
If(applicantAge==99)hireStatus="NO";
Giventhisimplementation,thefactthatanysetoftestspassestellsusnothingaboutthenext
testwecouldexecute.Itmaypass;itmayfail.
Luckily,programmersdon'twritecodelikethis(atleastnotveryoften).Abetterprogrammer
mightwrite:
If(applicantAge>=0&&applicantAge<=16)
hireStatus="NO";
If(applicantAge>=16&&applicantAge<=18)
hireStatus="PART";
If(applicantAge>=18&&applicantAge<=55)
hireStatus="FULL";
If(applicantAge>=55&&applicantAge<=99)
hireStatus="NO";
Giventhistypicalimplementation,itisclearthatforthefirstrequirementwedon'thavetotest
0,1,2,...14,15,and16.Onlyonevalueneedstobetested.Andwhichvalue?Anyonewithin
thatrangeisjustasgoodasanyotherone.Thesameistrueforeachoftheotherranges.
Rangessuchastheonesdescribedherearecalledequivalenceclasses.Anequivalence
classconsistsofasetofdatathatistreatedthesamebythemoduleorthatshouldproduce
thesameresult.Anydatavaluewithinaclassisequivalent,intermsoftesting,toanyother
value.Specifically,wewouldexpectthat:
Ifonetestcaseinanequivalenceclassdetectsadefect,allothertestcasesinthe
sameequivalenceclassarelikelytodetectthesamedefect.
Ifonetestcaseinanequivalenceclassdoesnotdetectadefect,noothertestcasesin
thesameequivalenceclassislikelytodetectthedefect.
Key
Point
Agroupoftestsformsanequivalenceclassifyoubelievethat:
Theyalltestthesamething.
Ifonetestcatchesabug,theothersprobablywilltoo.
Ifonetestdoesn'tcatchabug,theothersprobablywon'teither.
CemKanerTestingComputerSoftware
Thisapproachassumes,ofcourse,thataspecificationexiststhatdefinesthevarious
equivalenceclassestobetested.Italsoassumesthattheprogrammerhasnotdonesomething
strangesuchas:
If(applicantAge>=0&&applicantAge<=16)
hireStatus="NO";
If(applicantAge>=16&&applicantAge<=18)
hireStatus="PART";
If(applicantAge>=18&&applicantAge<=41)
hireStatus="FULL";
//strangestatementsfollow
If(applicantAge==42&&applicantName=="Lee")
hireStatus="HIRENOWATHUGESALARY";
If(applicantAge==42&&applicantName<>"Lee")
hireStatus="FULL";
//endofstrangestatements
If(applicantAge>=43&&applicantAge<=55)
hireStatus="FULL";
If(applicantAge>=55&&applicantAge<=99)
hireStatus="NO";
Usingtheequivalenceclassapproach,wehavereducedthenumberoftestcasesfrom100
(testingeachage)tofour(testingoneageineachequivalenceclass)—asignificantsavings.
Now,arewereadytobegintesting?Probablynot.Whataboutinputvalueslike969,-42,
FRED,and&$#!@?Shouldwecreatetestcasesforinvalidinput?Theansweris,asanygood
consultantwilltellyou,"itdepends."Tounderstandthisanswerweneedtoexaminean
approachthatcameoutoftheobject-orientedworldcalleddesign-by-contract.
Note
AccordingtotheBible,theageofMethuselahwhenhediedwas969years(Gen
5:27).ThankstotheGideonswhomadethisdataeasilyaccessibleinmyhotelroom
withouttheneedforahighspeedInternetconnection.
Inlaw,acontractisalegallybindingagreementbetweentwo(ormore)partiesthatdescribes
whateachpartypromisestodoornotdo.Eachofthesepromisesisofbenefittotheother.
Inthedesign-by-contractapproach,modules(called"methods"intheobject-orientedparadigm,
but"module"isamoregenericterm)aredefinedintermsofpre-conditionsandpost-conditions.
Post-conditionsdefinewhatamodulepromisestodo(computeavalue,openafile,printa
report,updateadatabaserecord,changethestateofthesystem,etc.).Pre-conditionsdefine
whatthatmodulerequiressothatitcanmeetitspost-conditions.Forexample,ifwehada
modulecalledopenFile,whatdoesitpromisetodo?Openafile.Whatwouldlegitimate
preconditionsofopenFilebe?First,thefilemustexist;second,wemustprovidethename(or
otheridentifyinginformation)ofthefile;third,thefilemustbe"openable,"thatis,itcannot
alreadybeexclusivelyopenedbyanotherprocess;fourth,wemusthaveaccessrightstothe
file;andsoon.Pre-conditionsandpostconditionsestablishacontractbetweenamoduleand
othersthatinvokeit.
Testing-by-contractisbasedonthedesign-by-contractphilosophy.Itsapproachistocreate
testcasesonlyforthesituationsinwhichthepre-conditionsaremet.Forexample,wewould
nottesttheopenFilemodulewhenthefiledidnotexist.Thereasonissimple.Ifthefiledoes
notexist,openFiledoesnotpromisetowork.Ifthereisnoclaimthatitwillworkundera
specificcondition,thereisnoneedtotestunderthatcondition.
ForMore
Information
SeeBertrandMeyer'sbookObject-OrientedSoftwareConstructionfor
moreondesign-by-contract.
Atthispointtestersusuallyprotest.Yes,theyagree,themoduledoesnotclaimtoworkinthat
case,butwhatifthepreconditionsareviolatedduringproduction?Whatdoesthesystemdo?
Dowegetamisspelledwordonthescreenorasmokingcraterwhereourcompanyusedto
be?
Adifferentapproachtodesignisdefensivedesign.Inthiscasethemoduleisdesignedto
acceptanyinput.Ifthenormalpreconditionsaremet,themodulewillachieveitsnormal
postconditions.Ifthenormalpre-conditionsarenotmet,themodulewillnotifythecallerby
returninganerrorcodeorthrowinganexception(dependingontheprogramminglanguage
used).Thisnotificationisactuallyanotheroneofthemodule'spostconditions.Basedonthis
approachwecoulddefinedefensivetesting:anapproachthattestsunderbothnormaland
abnormalpre-conditions.
Insight
Astudentinoneofmyclasses,let'scallhimFred,saidhedidn'treallycarewhich
designapproachwasbeingused,hewasgoingtoalwaysusedefensivetesting.
WhenIaskedwhy,hereplied,"Ifitdoesn'twork,whowillgettheblame-those
responsibleorthetesters?"
Howdoesthisapplytoequivalenceclasstesting?Dowehavetotestwithinputslike-42,
FRED,and&$#!@?Ifweareusingdesign-by-contractandtesting-by-contracttheansweris
No.Ifweareusingdefensivedesignandthusdefensivetesting,theanswerisYes.Askyour
designerswhichapproachtheyareusing.Iftheyanswereither"contract"or"defensive,"you
knowwhatstyleoftestingtouse.Iftheyanswer"Huh?"thatmeanstheyarenotthinkingabout
howmodulesinterface.Theyarenotthinkingaboutpre-conditionandpost-conditioncontracts.
Youshouldexpectintegrationtestingtobeaprimesourceofdefectsthatwillbemorecomplex
andtakemoretimethananticipated.
Technique
Thestepsforusingequivalenceclasstestingaresimple.First,identifytheequivalenceclasses.
Second,createatestcaseforeachequivalenceclass.Youcouldcreateadditionaltestcases
foreachequivalenceclassifyouhavethetimeandmoney.Additionaltestcasesmaymakeyou
feelwarmandfuzzy,buttheyrarelydiscoverdefectsthefirstdoesn'tfind.
Insight
Astudentinoneofmyclasses,let'scallherJudy,feltveryuncomfortableabout
havingonlyonetestcaseforeachequivalenceclass.Shewantedatleasttwofor
thatwarmandfuzzyfeeling.Iindicatedthatifshehadthetimeandmoneythat
approachwasfinebutsuggestedtheadditionaltestswouldprobablybeineffective.
Iaskedhertokeeptrackofhowmanytimestheadditionaltestcasesfound
defectsthatthefirstdidnotandletmeknow.IneverheardfromJudyagain.
Differenttypesofinputrequiredifferenttypesofequivalenceclasses.Let'sconsiderfour
possibilities.Let'sassumeadefensivetestingphilosophyoftestingbothvalidandinvalidinput.
Testinginvalidinputsisoftenagreatsourceofdefects.
Ifaninputisacontinuousrangeofvalues,thenthereistypicallyoneclassofvalidvaluesand
twoclassesofinvalidvalues,onebelowthevalidclassandoneaboveit.ConsidertheGoofy
MortgageCompany(GMC).Theywillwritemortgagesforpeoplewithincomesbetween
$1,000/monthand$83,333/month.Anythingbelow$1,000/monthyoudon'tqualify.Anything
over$83,333/monthyoudon'tneedGMC,justpaycash.
Foravalidinputwemightchoose$1,342/month.Forinvalidswemightchoose$123/monthand
$90,000/month.
Figure3-1:Continuousequivalenceclasses Ifaninputconditiontakeson
discretevalueswithinarangeofpermissiblevalues,therearetypicallyonevalidandtwo
invalidclasses.GMCwillwriteasinglemortgageforonethroughfivehouses.(Remember,
it'sGoofy.)Zeroorfewerhousesisnotalegitimateinput,norissixorgreater.Neitherare
fractionalordecimalvaluessuchas21/2or3.14159.
Figure3-2:Discreteequivalenceclasses Foravalidinputwemight
choosetwohouses.Invalidscouldbe-2and8.
GMCwillmakemortgagesonlyforaperson.Theywillnotmakemortgagesfor
corporations,trusts,partnerships,oranyothertypeoflegalentity.
Figure3-3:Singleselectionequivalenceclasses Foravalidinputwe
mustuse"person."Foraninvalidwecouldchoose"corporation"or"trust"oranyother
randomtextstring.Howmanyinvalidcasesshouldwecreate?Wemusthaveatleast
one;wemaychooseadditionaltestsforadditionalwarmandfuzzyfeelings.
GMCwillmakemortgagesonCondominiums,Townhouses,andSingleFamily
dwellings.TheywillnotmakemortgagesonDuplexes,MobileHomes,Treehouses,or
anyothertypeofdwelling.
Figure3-4:Multipleselectionequivalenceclass Forvalidinputwe
mustchoosefrom"Condominium,""Townhouse,"or"SingleFamily."Whiletherule
sayschooseonetestcasefromthevalidequivalenceclass,amorecomprehensive
approachwouldbetocreatetestcasesforeachentryinthevalidclass.That
makessensewhenthelistofvalidvaluesissmall.But,ifthiswerealistofthefifty
states,theDistrictofColumbia,andthevariousterritoriesoftheUnitedStates,
wouldyoutesteveryoneofthem?Whatifthelistwereeverycountryintheworld?
Thecorrectanswer,ofcourse,dependsontherisktotheorganizationif,astesters,
wemisssomethingthatisvital.
Now,rarelywillwehavethetimetocreateindividualtestsforeveryseparate
equivalenceclassofeveryinputvaluethatentersoursystem.Moreoften,wewill
createtestcasesthattestanumberofinputfieldssimultaneously.Forexample,we
mightcreateasingletestcasewiththefollowingcombinationofinputs:
Key
Point
Rarelywillwehavethetimetocreateindividualtestsforevery
separateequivalenceclassofeveryinputvalue.
Table3-1:Atestcaseofvaliddatavalues.
MonthlyIncome NumberofDwellings Applicant DwellingTypes Result
$5,000 2 Person Condo Valid
Eachofthesedatavaluesisinthevalidrange,sowewouldexpectthesystemto
performcorrectlyandforthetestcasetoreportPass.
Itistemptingtousethesameapproachforinvalidvalues.
Table3-2:Atestcaseofallinvaliddatavalues.Thisisnotagood
approach.
MonthlyIncome NumberofDwellings Applicant DwellingTypes Result
$100 8 Partnership Treehouse Invalid
Ifthesystemacceptsthisinputasvalid,clearlythesystemisnotvalidatingthefour
inputfieldsproperly.Ifthesystemrejectsthisinputasinvalid,itmaydosoinsucha
waythatthetestercannotdeterminewhichfielditrejected.Forexample:ERROR:
653X-2.7INVALIDINPUT
Inmanycases,errorsinoneinputfieldmaycanceloutormaskerrorsinanother
fieldsothesystemacceptsthedataasvalid.Abetterapproachistotestoneinvalid
valueatatimetoverifythesystemdetectsitcorrectly.
Table3-3:Asetoftestcasesvaryinginvalidvaluesonebyone.
MonthlyIncome NumberofDwellings Applicant DwellingTypes Result
$100 1 Person SingleFam Invalid
$1,342 0Person Condo Invalid
$1,342 1 Corporation Townhouse Invalid
$1,342 1 Person Treehouse Invalid
Foradditionalwarmandfuzzyfeelings,theinputs(bothvalidandinvalid)couldbe
varied.
Table3-4:Asetoftestcasesvaryinginvalidvaluesonebyonebutalso
varyingthevalidvalues.
MonthlyIncome NumberofDwellings Applicant DwellingTypes Result
$100 1 Person SingleFamily Invalid
$1,342 0Person Condominium Invalid
$5,432 3 Corporation Townhouse Invalid
$10,000 2 Person Treehouse Invalid
Anotherapproachtousingequivalenceclassesistoexaminetheoutputsratherthan
theinputs.Dividetheoutputsintoequivalenceclasses,thendeterminewhatinput
valueswouldcausethoseoutputs.Thishastheadvantageofguidingthetesterto
examine,andthustest,everydifferentkindofoutput.Butthisapproachcanbe
deceiving.Inthepreviousexample,forthehumanresourcessystem,oneofthe
systemoutputswasNO,thatis,Don'tHire.Acursoryviewoftheinputsthatshould
causethisoutputwouldyield{0,1,...,14,15}.Notethatthisisnotthecomplete
set.Inaddition{55,56,...,98,99}shouldalsocausetheNOoutput.It'simportant
tomakesurethatallpotentialoutputscanbegenerated,butdon'tbefooledinto
choosingequivalenceclassdatathatomitsimportantinputs.
Examples
Example1
ReferringtotheTradeWebpageoftheBrown&DonaldsonWebsitedescribedinAppendixA,
considertheOrderTypefield.ThedesignerhaschosentoimplementthedecisiontoBuyor
Sellthroughradiobuttons.Thisisagooddesignchoicebecauseitreducesthenumberoftest
casesthetestermustcreate.Hadthisbeenimplementedasatextfieldinwhichtheuser
entered"Buy"or"Sell"thetesterwouldhavepartitionedthevalidinputsas{Buy,Sell}andthe
invalidsas{Trade,Punt,...}.Whatabout"buy","bUy","BUY"?Arethesevalidorinvalid
entries?Thetesterwouldhavetoreferbacktotherequirementstodeterminetheirstatus.
Insight Letyourdesignersandprogrammersknowwhentheyhavehelpedyou.They'll
appreciatethethoughtandmaydoitagain.
Withtheradiobuttonimplementationnoinvalidchoicesexist,sononeneedtobetested.Only
thevalidinputs{Buy,Sell}needtobeexercised.
Example2
Again,referringtotheTradeWebpage,considertheQuantityfield.Inputtothisfieldcanbe
betweenoneandfournumericcharacters(0,1,...,8,9)withavalidvaluegreaterorequalto1
andlessthanorequalto9999.Asetofvalidinputsis{1,22,333,4444}whileinvalidinputsare
{-42,0,12345,SQE,$#@%}.
Insight
VeryoftenyourdesignersandprogrammersuseGUIdesigntoolsthatcanenforce
restrictionsonthelengthandcontentofinputfields.Encouragetheiruse.Thenyour
testingcanfocusonmakingsuretherequirementhasbeenimplementedproperly
withthetool.
Example3
OntheTradepagetheuserentersatickerSymbolindicatingthestocktobuyorsell.Thevalid
symbolsare{A,AA,AABC,AAC,...,ZOLT,ZOMX,ZONA,ZRAN).Theinvalidsymbolsare
anycombinationofcharactersnotincludedinthevalidlist.Asetofvalidinputscouldbe{A,AL,
ABE,ACES,AKZOY)whileasetofinvalidscouldbe{C,AF,BOB,CLUBS,AKZAM,42,
@#$%).
ForMore
Information
ClickontheSymbolLookupbuttonontheB&DTradepagetoseethe
fulllistofstocksymbols.
Example4
Rarelywillwecreateseparatesetsoftestcasesforeachinput.Generallyitismoreefficientto
testmultipleinputssimultaneouslywithintests.Forexample,thefollowingtestscombine
Buy/Sell,Symbol,andQuantity.
Table3-5:Asetoftestcasesvaryinginvalidvaluesonebyone.
Buy/Sell Symbol Quantity Result
Buy A 10 Valid
Buy C 20 Invalid
Buy A 0 Invalid
Sell ACES 10 Valid
Sell BOB 33 Invalid
Sell ABE -3 Invalid
ApplicabilityandLimitations
Equivalenceclasstestingcansignificantlyreducethenumberoftestcasesthatmustbe
createdandexecuted.Itismostsuitedtosystemsinwhichmuchoftheinputdatatakeson
valueswithinrangesorwithinsets.Itmakestheassumptionthatdatainthesameequivalence
classis,infact,processedinthesamewaybythesystem.Thesimplestwaytovalidatethis
assumptionistoasktheprogrammerabouttheirimplementation.
Equivalenceclasstestingisequallyapplicableattheunit,integration,system,andacceptance
testlevels.Allitrequiresareinputsoroutputsthatcanbepartitionedbasedonthesystem's
requirements.
Summary
Equivalenceclasstestingisatechniqueusedtoreducethenumberoftestcasestoa
manageablesizewhilestillmaintainingreasonablecoverage.
Thissimpletechniqueisusedintuitivelybyalmostalltesters,eventhoughtheymaynot
beawareofitasaformaltestdesignmethod.
Anequivalenceclassconsistsofasetofdatathatistreatedthesamebythemodule
orthatshouldproducethesameresult.Anydatavaluewithinaclassisequivalent,in
termsoftesting,toanyothervalue.
Practice
1. ThefollowingexercisesrefertotheStatelessUniversityRegistrationSystemWeb
sitedescribedinAppendixB.Definetheequivalenceclassesandsuitabletestcases
forthefollowing:
1. ZIPCode—fivenumericdigits.
2. State—thestandardPostOfficetwo-characterabbreviationforthestates,
districts,territories,etc.oftheUnitedStates.
3. LastName—onethroughfifteencharacters(includingalphabeticcharacters,
periods,hyphens,apostrophes,spaces,andnumbers).
4. UserID—eightcharactersatleasttwoofwhicharenotalphabetic(numeric,
special,nonprinting).
5. StudentID—eightcharacters.Thefirsttworepresentthestudent'shome
campuswhilethelastsixareauniquesix-digitnumber.Validhomecampus
abbreviationsare:AN,Annandale;LC,LasCruces;RW,RiversideWest;
SM,SanMateo;TA,Talbot;WE,Weber;andWN,Wenatchee.
References
Beizer,Boris(1990).SoftwareTestingTechniques.VanNostrandReinhold.
Kaner,Cem,JackFalkandHungQuocNguyen(1999).TestingComputerSoftware
(SecondEdition).JohnWiley&Sons.
Myers,GlenfordJ.(1979).TheArtofSoftwareTesting.JohnWiley&Sons.
Chapter4:BoundaryValueTesting
ThePrincelookeddownatthemotionlessformofSleepingBeauty,wonderinghowher
supplelipswouldfeelagainsthisownandcontemplatingwhetherornotanAltoidwas
strongenoughtostandupagainstthekindofmorningbreathonlyahundredyears'nap
couldcreate.
—LynneSella
Introduction
Equivalenceclasstestingisthemostbasictestdesigntechnique.Ithelpstesterschoosea
smallsubsetofpossibletestcaseswhilemaintainingreasonablecoverage.Equivalenceclass
testinghasasecondbenefit.Itleadsustotheideaofboundaryvaluetesting,thesecondkey
testdesigntechniquetobepresented.
Inthepreviouschapterthefollowingrulesweregiventhatindicatehowweshouldprocess
employmentapplicationsbasedonaperson'sage.Theruleswere:
0–16 Don'thire
16–18Canhireonapart-timebasisonly
18–55Canhireasafull-timeemployee
55–99Don'thire
Noticetheproblemattheboundaries—the"edges"ofeachclass.Theage"16"isincludedin
twodifferentequivalenceclasses(asare18and55).Thefirstrulesaysdon'thirea16-year-
old.Thesecondrulesaysa16-year-oldcanbehiredonapart-timebasis.
Boundaryvaluetestingfocusesontheboundariessimplybecausethatiswheresomany
defectshide.Experiencedtestershaveencounteredthissituationmanytimes.Inexperienced
testersmayhaveanintuitivefeelthatmistakeswilloccurmostoftenattheboundaries.These
defectscanbeintherequirements(asshownabove)orinthecodeasshownbelow:
Key
Point
Boundaryvaluetestingfocusesontheboundariesbecausethatiswheresomany
defectshide.
If(applicantAge>=0&&applicantAge<=16)
hireStatus="NO";
If(applicantAge>=16&&applicantAge<=18)
hireStatus="PART";
If(applicantAge>=18&&applicantAge<=55)
hireStatus="FULL";
If(applicantAge>=55&&applicantAge<=99)
hireStatus="NO";
Ofcourse,themistakethatprogrammersmakeiscodinginequalitytestsimproperly.Writing>
(greaterthan)insteadof≥(greaterthanorequal)isanexample.
Themostefficientwayoffindingsuchdefects,eitherintherequirementsorthecode,is
throughinspection.GilbandGraham'sbook,SoftwareInspection,isanexcellentguidetothis
process.However,nomatterhoweffectiveourinspections,wewillwanttotestthecodeto
verifyitscorrectness.
Perhapsthisiswhatourorganizationmeant:
0–15Don'thire
16–17Canhireonapart-timebasisonly
18–54Canhireasfull-timeemployees
55–99Don'thire
Whataboutages-3and101?Notethattherequirementsdonotspecifyhowthesevalues
shouldbetreated.Wecouldguessbut"guessingtherequirements"isnotanacceptable
practice.
Thecodethatimplementsthecorrectedrulesis:
If(applicantAge>=0&&applicantAge<=15)
hireStatus="NO";
If(applicantAge>=16&&applicantAge<=17)
hireStatus="PART";
If(applicantAge>=18&&applicantAge<=54)
hireStatus="FULL";
If(applicantAge>=55&&applicantAge<=99)
hireStatus="NO";
Theinterestingvaluesonorneartheboundariesinthisexampleare{-1,0,1},{15,16,17},
{17,18,19},{54,55,56},and{98,99,100}.Othervalues,suchas{-42,1001,FRED,%$#@}
mightbeincludeddependingonthemodule'sdocumentedpreconditions.
Technique
Thestepsforusingboundaryvaluetestingaresimple.First,identifytheequivalenceclasses.
Second,identifytheboundariesofeachequivalenceclass.Third,createtestcasesforeach
boundaryvaluebychoosingonepointontheboundary,onepointjustbelowtheboundary,and
onepointjustabovetheboundary."Below"and"above"arerelativetermsanddependonthe
datavalue'sunits.Iftheboundaryis16andtheunitis"integer"thenthe"below"pointis15and
the"above"pointis17.Iftheboundaryis$5.00andtheunitis"USdollarsandcents"thenthe
belowpointis$4.99andtheabovepointis$5.01.Ontheotherhand,ifthevalueis$5andthe
unitis"USdollars"thenthebelowpointis$4andtheabovepointis$6.
Key
Point
Createtestcasesforeachboundaryvaluebychoosingonepointonthe
boundary,onepointjustbelowtheboundary,andonepointjustabovethe
boundary.
Notethatapointjustaboveoneboundarymaybeinanotherequivalenceclass.Thereisno
reasontoduplicatethetest.Thesamemaybetrueofthepointjustbelowtheboundary.
Youcould,ofcourse,createadditionaltestcasesfartherfromtheboundaries(within
equivalenceclasses)ifyouhavetheresources.Asdiscussedinthepreviouschapter,these
additionaltestcasesmaymakeyoufeelwarmandfuzzy,buttheyrarelydiscoveradditional
defects.
Boundaryvaluetestingismostappropriatewheretheinputisacontinuousrangeofvalues.
ReturningagaintotheGoofyMortgageCompany,whataretheinterestingboundaryvalues?
Formonthlyincometheboundariesare$1,000/monthand$83,333/month(assumingtheunits
tobeUSdollars).
Figure4-1:Boundaryvaluesforacontinuousrangeofinputs.
Testdatainputof{$999,$1,000,$1,001}onthelowendand{$83,332,$83,333,$83,334}on
thehighendarechosentotesttheboundaries.
BecauseGMCwillwriteamortgageforonethroughfivehouses,zeroorfewerhousesisnota
legitimateinputnorissixorgreater.Theseidentifytheboundariesfortesting.
Figure4-2:Boundaryvaluesforadiscreterangeofinputs.
Rarelywillwehavethetimetocreateindividualtestsforeveryboundaryvalueofeveryinput
valuethatentersoursystem.Moreoften,wewillcreatetestcasesthattestanumberofinput
fieldssimultaneously.
Table4-1:Asetoftestcasescontainingcombinationsofvalid(ontheboundary)
valuesandinvalid(offtheboundary)points.
MonthlyIncome NumberofDwellings Result Description
$1,000 1 Valid Minincome,mindwellings
$83,333 1 Valid Maxincome,mindwellings
$1,000 5 Valid Minincome,maxdwellings
$83,333 5 Valid Maxincome,maxdwellings
$1,000 0 Invalid Minincome,belowmindwellings
$1,000 6 Invalid Minincome,abovemaxdwellings
$83,333 0 Invalid Maxincome,belowmindwellings
$83,333 6 Invalid Maxincome,abovemaxdwellings
$999 1 Invalid Belowminincome,mindwellings
$83,334 1 Invalid Abovemaxincome,mindwellings
$999 5 Invalid Belowminincome,maxdwellings
$83,334 5 Invalid Abovemaxincome,maxdwellings
Plotting"monthlyincome"onthex-axisand"numberofdwellings"onthey-axisshowsthe
"locations"ofthetestdatapoints.
Figure4-3:Datapointsontheboundariesanddatapointsjustoutsidetheboundaries.
Notethatfouroftheinputcombinationsareontheboundarieswhileeightarejustoutside.Also
notethatthepointsoutsidealwayscombineonevalidvaluewithoneinvalidvalue(justoneunit
loweroroneunithigher).
Examples
Boundaryvaluetestingisapplicabletothestructure(lengthandcharactertype)ofinputdataas
wellasitsvalue.Considerthefollowingtwoexamples:
Example1
ReferringtotheTradeWebpageoftheBrown&DonaldsonWebsitedescribedinAppendixA,
considertheQuantityfield.Inputtothisfieldcanbebetweenoneandfournumericcharacters
(0,1,...,8,9).Asetofboundaryvaluetestcasesforthelengthattributewouldbe{0,1,4,5}
numericcharacters.
Example2
Again,ontheTradepage,considertheQuantityfield,butthistimeforvalueratherthan
structure(lengthandcharactertype).WhetherthetransactionisBuyorSell,theminimum
legitimatevalueis1souse{0,1,2}forboundarytesting.Theupperlimitonthisfield'svalueis
morecomplicated.IfthetransactionisSell,whatisthemaximumnumberofsharesthatcanbe
sold?Itisthenumbercurrentlyowned.Forthisboundaryuse{sharesOwned-1,sharesOwned,
sharesOwned+1}.IfthetransactionisBuy,themaximumvalue(numberofsharestobe
purchased)isdefinedas
shares=(accountBalance-commission)/sharePrice
assumingafixedcommission.Use{shares-1,shares,shares+1}astheboundaryvaluetest
cases.
ApplicabilityandLimitations
Boundaryvaluetestingcansignificantlyreducethenumberoftestcasesthatmustbecreated
andexecuted.Itismostsuitedtosystemsinwhichmuchoftheinputdatatakesonvalues
withinrangesorwithinsets.
Boundaryvaluetestingisequallyapplicableattheunit,integration,system,andacceptance
testlevels.Allitrequiresareinputsthatcanbepartitionedandboundariesthatcanbeidentified
basedonthesystem'srequirements.
Summary
Whileequivalenceclasstestingisuseful,itsgreatestcontributionistoleadusto
boundaryvaluetesting.
Boundaryvaluetestingisatechniqueusedtoreducethenumberoftestcasestoa
manageablesizewhilestillmaintainingreasonablecoverage.
Boundaryvaluetestingfocusesontheboundariesbecausethatiswheresomany
defectshide.Experiencedtestershaveencounteredthissituationmanytimes.
Inexperiencedtestersmayhaveanintuitivefeelthatmistakeswilloccurmostoftenat
theboundaries.
Createtestcasesforeachboundaryvaluebychoosingonepointontheboundary,one
pointjustbelowtheboundary,andonepointjustabovetheboundary."Below"and
"above"arerelativetermsanddependonthedatavalue'sunits.
Practice
1. ThefollowingexercisesrefertotheStatelessUniversityRegistrationSystemWeb
sitedescribedinAppendixB.Definetheboundaries,andsuitableboundaryvaluetest
casesforthefollowing:
1. ZIPCode—fivenumericdigits.
2. FirstconsiderZIPCodejustintermsofdigits.Then,determinethelowest
andhighestlegitimateZIPCodesintheUnitedStates.Forextracredit[1],
determinetheformatofpostalcodesforCanadaandthelowestandhighest
validvalues.
3. LastName—onethroughfifteencharacters(includingalphabeticcharacters,
periods,hyphens,apostrophes,spaces,andnumbers).Forextracredit[2]
createafewverycomplexLastNames.Canyoudeterminethe"rules"for
legitimateLastNames?Foradditionalextracredit[3]useaphonebookfrom
anothercountry—tryFinlandorThailand.
4. UserID—eightcharactersatleasttwoofwhicharenotalphabetic(numeric,
special,nonprinting).
5. CourseID—threealphacharactersrepresentingthedepartmentfollowedby
asix-digitintegerwhichistheuniquecourseidentificationnumber.The
possibledepartmentsare:
PHY-Physics
EGR-Engineering
ENG-English
LAN-Foreignlanguages
CHM-Chemistry
MAT-Mathematics
PED-Physicaleducation
SOC-Sociology
[1]Thereactuallyisnoextracredit,sodoitforfun.
[2]Thereactuallyisnoextracredit,sodoitforfun.
[3]Thereactuallyisnoextracredit,sodoitforfun.
References
Beizer,Boris(1990).SoftwareTestingTechniques.VanNostrandReinhold.
Gilb,TomandDorothyGraham(1993).SoftwareInspection.Addison-Wesley.ISBN0-
201-63181-4.
Myers,GlenfordJ.(1979).TheArtofSoftwareTesting.JohnWiley&Sons.
Chapter5:DecisionTableTesting
I'dstumbledontosolvingmyfirstmurdercase,havingfoundmyselftheonlyeyewitness,
yetnomatterhowfranticallyIpleadedwithJohnLawthattheperpwasrightinfrontof
themandtheverydamethey'dbeengrilling-thesultrybutdeviousMissKitwinkle,who
playedthegrievingpatsythewayaconcertpianistplayerplaysapiano-thecopsjust
keptsmilingandstuffingcrackersinmybeak.
—ChrisEsco
Introduction
Decisiontablesareanexcellenttooltocapturecertainkindsofsystemrequirementsandto
documentinternalsystemdesign.Theyareusedtorecordcomplexbusinessrulesthata
systemmustimplement.Inaddition,theycanserveasaguidetocreatingtestcases.
Decisiontablesareavitaltoolinthetester'spersonaltoolbox.Unfortunately,manyanalysts,
designers,programmers,andtestersarenotfamiliarwiththistechnique.
Technique
Decisiontablesrepresentcomplexbusinessrulesbasedonasetofconditions.Thegeneral
formis:
Table5-1:Thegeneralformofadecisiontable.
Rule1 Rule2 … Rulep
Conditions
Condition-1
Condition-2
…
Condition-m
Actions
Action-1
Action-2
…
Action-n
Conditions1throughmrepresentvariousinputconditions.Actions1throughnaretheactions
thatshouldbetakendependingonthevariouscombinationsofinputconditions.Eachofthe
rulesdefinesauniquecombinationofconditionsthatresultintheexecution("firing")ofthe
actionsassociatedwiththatrule.Notethattheactionsdonotdependontheorderinwhichthe
conditionsareevaluated,butonlyontheirvalues.(Allvaluesareassumedtobeavailable
simultaneously.)Also,actionsdependonlyonthespecifiedconditions,notonanypreviousinput
conditionsorsystemstate.
Perhapsaconcreteexamplewillclarifytheconcepts.Anautoinsurancecompanygives
discountstodriverswhoaremarriedand/orgoodstudents.Let'sbeginwiththeconditions.The
followingdecisiontablehastwoconditions,eachoneofwhichtakesonthevaluesYesorNo.
Table5-2:Adecisiontablewithtwobinaryconditions.
Rule1 Rule2 Rule3 Rule4
Conditions
Married? Yes Yes No No
GoodStudent? Yes No Yes No
Notethatthetablecontainsallcombinationsoftheconditions.Giventwobinaryconditions(Yes
orNo),thepossiblecombinationsare{Yes,Yes},{Yes,No},{No,Yes},and{No,No}.Eachrule
representsoneofthesecombinations.Asatesterwewillverifythatallcombinationsofthe
conditionsaredefined.Missingacombinationmayresultindevelopingasystemthatmaynot
processaparticularsetofinputsproperly.
Nowfortheactions.Eachrulecausesanactionto"fire."Eachrulemayspecifyanaction
uniquetothatrule,orrulesmayshareactions.
Table5-3:Addingasingleactiontoadecisiontable.
Rule1 Rule2 Rule3 Rule4
Conditions
Married? Yes Yes No No
GoodStudent? Yes No Yes No
Actions
Discount($) 60 25 50 0
Decisiontablesmayspecifymorethanoneactionforeachrule.Again,theserulesmaybe
uniqueormaybeshared.
Table5-4:Adecisiontablewithmultipleactions.
Rule1 Rule2 Rule3 Rule4
Conditions
Condition-1 Yes Yes No No
Condition-2 Yes No Yes No
Actions
Action-1 DoX DoY DoX DoZ
Action-2 DoA DoB DoB DoB
Inthissituation,choosingtestcasesissimple—eachrule(verticalcolumn)becomesatest
case.TheConditionsspecifytheinputsandtheActionsspecifytheexpectedresults.
Whilethepreviousexampleusessimplebinaryconditions,conditionscanbemorecomplex.
Table5-5:Adecisiontablewithnon-binaryconditions.
Rule1 Rule2 Rule3 Rule4
Conditions
Condition-1 0–1 1–10 10–100 100–1000
Condition-2 <5 5 6or7 >7
Actions
Action-1 DoX DoY DoX DoZ
Action-2 DoA DoB DoB DoB
Inthissituationchoosingtestcasesisslightlymorecomplex—eachrule(verticalcolumn)
becomesatestcasebutvaluessatisfyingtheconditionsmustbechosen.Choosingappropriate
valueswecreatethefollowingtestcases:
Table5-6:Sampletestcases.
TestCaseID Condition-1 Condition-2 ExpectedResult
TC1 0 3 DoX/DoA
TC2 5 5 DoY/DoB
TC3 50 7 DoX/DoB
TC4 500 10 DoZ/DoB
Ifthesystemundertesthascomplexbusinessrules,andifyourbusinessanalystsordesigners
havenotdocumentedtheserulesinthisform,testersshouldgatherthisinformationand
representitindecisiontableform.Thereasonissimple.Giventhesystembehavior
representedinthiscompleteandcompactform,testcasescanbecreateddirectlyfromthe
decisiontable.
Intesting,createatleastonetestcaseforeachrule.Iftherule'sconditionsarebinary,asingle
testforeachcombinationisprobablysufficient.Ontheotherhand,ifaconditionisarangeof
values,considertestingatboththelowandhighendoftherange.Inthiswaywemergethe
ideasofBoundaryValuetestingwithDecisionTabletesting.
Key
Point Createatleastonetestcaseforeachrule.
Tocreateatestcasetablesimplychangetherowandcolumnheadings:
Table5-7:Adecisiontableconvertedtoatestcasetable.
TestCase1 TestCase2 TestCase3 TestCase4
Inputs
Condition-1 Yes Yes No No
Condition-2 Yes No Yes No
ExpectedResults
Action-1 DoX DoY DoX DoZ
Action-2 DoA DoB DoB DoB
Examples
DecisionTabletestingcanbeusedwheneverthesystemmustimplementcomplexbusiness
rules.Considerthefollowingtwoexamples:Example1
ReferringtotheTradeWebpageoftheBrown&DonaldsonWebsitedescribedinAppendixA,
considertherulesassociatedwithaBuyorder.
Table5-8:AdecisiontablefortheBrown&DonaldsonBuyorder.
Rule1 Rule2 Rule3 Rule4 Rule5 Rule6 Rule7 Rule8
Conditions
ValidSymbol No No No No Yes Yes Yes Yes
ValidQuantity No No Yes Yes No No Yes Yes
SufficientFunds No Yes No Yes No Yes No Yes
Actions
Buy? No No No No No No No Yes
Admittedly,theoutcomeisreadilyapparent.Onlywhenavalidsymbol,validquantity,and
sufficientfundsareavailableshouldtheBuyorderbeplaced.Thisexamplewaschosento
illustrateanotherconcept.
Examinethefirstfourcolumns.IftheSymbolisnotvalid,noneoftheotherconditionsmatter.
Oftentableslikethisarecollapsed,rulesarecombined,andtheconditionsthatdonotaffect
theoutcomearemarked"DC"for"Don'tCare."Rule1nowindicatesthatiftheSymbolisnot
valid,ignoretheotherconditionsanddonotexecutetheBuyorder.
Table5-9:Acollapseddecisiontablereflecting"Don'tCare"conditions.
Rule1 Rule2 Rule3 Rule4 Rule5
Conditions
ValidSymbol No Yes Yes Yes Yes
ValidQuantity DC No No Yes Yes
SufficientFunds DC No Yes No Yes
Actions
Buy? No No No No Yes
NotealsothatRule2andRule3canbecombinedbecausewhetherSufficientFundsare
availabledoesnotaffecttheaction.
Table5-10:Afurthercollapseddecisiontablereflecting"Don'tCare"conditions.
Rule1 Rule2 Rule3 Rule4
Conditions
ValidSymbol No Yes Yes Yes
ValidQuantity DC No Yes Yes
SufficientFunds DC DC No Yes
Actions
Buy? No No No Yes
Whilethisisanexcellentideafromadevelopmentstandpointbecauselesscodeiswritten,itis
dangerousfromatestingstandpoint.Itisalwayspossiblethatthetablewascollapsed
incorrectlyorthecodewaswrittenimproperly.Theun-collapsedtableshouldalwaysbeused
asthebasisforourtestcasedesign.
Example2
ThefollowingscreenisfromtheStatelessUniversityRegistrationSystem.Itisusedtoenter
newstudentsintothesystem,tomodifystudentinformation,andtodeletestudentsfromthe
system.
Figure5-1:SURSStudentDatabaseMaintenanceScreen.
Toenteranewstudent,entername,address,andtelephoneinformationontheupperpartof
thescreenandpressEnter.Thestudentisenteredintothedatabaseandthesystemreturnsa
newStudentID.Tomodifyordeleteastudent,entertheStudentID,selecttheDeleteorModify
radiobuttonandpressEnter.Thedecisiontablereflectingtheserulesfollows:
Table5-11:AdecisiontableforStatelessUniversityRegistrationSystem.
Table5-11:AdecisiontableforStatelessUniversityRegistrationSystem.
Rule
1
Rule
2
Rule
3
Rule
4
Rule
5
Rule
6
Rule
7
Rule
8
Rule
9
Rule
10
Rule
11
Rule
12
Rule
13
Rule
14
Rule
Conditions
Entered
Student
data
No No No No No No No No Yes Yes Yes Yes Yes Yes Yes
Entered
Student
ID
No No No No Yes Yes Yes Yes No No No No Yes Yes Yes
Selected
Modify No No Yes Yes No No Yes Yes No No Yes Yes No No Yes
Selected
Delete No Yes No Yes No Yes No Yes No Yes No Yes No Yes
Actions
Create
new
student
No No No No No No No No Yes No No No No No
Modify
Student No No No No No No Yes No No No Yes No No No
Delete
Student No No No No No Yes No No No No No No No No
Rules1through8indicatethatnodatawasenteredaboutthestudent.Rules1through4
indicatethatnoStudentIDwasenteredforthestudent,thusnoactionispossible.Rules5
through8indicatetheStudentIDwasentered.InthesecasescreatinganewStudentisnot
proper.Rule5doesnotrequesteithermodificationordeletionsoneitherisdone.Rules6and7
requestonefunctionandsotheyareperformed.NotethatRule8indicatesthatboth
modificationanddeletionaretobeperformedsonoactionistaken.
Rules9through16indicatethatdatawasenteredaboutthestudent.Rules9through12
indicatethatnoStudentIDwasenteredsotheserulesrefertoanewstudent.Rule9createsa
newstudent.Rule10deletesthestudent.Rule11allowsmodificationofthestudent'sdata.
Rule12requeststhatbothmodificationanddeletionaretobeperformedsonoactionistaken.
Rules13through16supplystudentdataindicatinganewstudentbutalsoprovideaStudentID
indicatinganexistingstudent.Becauseofthiscontradictoryinput,noactionistaken.Often,
errormessagesaredisplayedinthesesituations.
ApplicabilityandLimitations
DecisionTabletestingcanbeusedwheneverthesystemmustimplementcomplexbusiness
ruleswhentheserulescanberepresentedasacombinationofconditionsandwhenthese
conditionshavediscreteactionsassociatedwiththem.
Summary
Decisiontablesareusedtodocumentcomplexbusinessrulesthatasystemmust
implement.Inaddition,theyserveasaguidetocreatingtestcases.
Conditionsrepresentvariousinputconditions.Actionsaretheprocessesthatshouldbe
executeddependingonthevariouscombinationsofinputconditions.Eachruledefinesa
uniquecombinationofconditionsthatresultintheexecution("firing")oftheactions
associatedwiththatrule.
Createatleastonetestcaseforeachrule.Iftherule'sconditionsarebinary,asingle
testforeachcombinationisprobablysufficient.Ontheotherhand,ifaconditionisa
rangeofvalues,considertestingatboththelowandhighendoftherange.
Practice
1. AttendingStatelessUniversityisanexpensiveproposition.Afterall,theyreceiveno
statefunding.Likemanyotherstudents,thoseplanningonattendingapplyforstudent
aidusingFAFSA,theFreeApplicationforFederalStudentAid.Thefollowing
instructionsweretakenfromthatform.Examinethemandcreateadecisiontablethat
representstheFAFSArules.(Note:Youcan'tmakeupstufflikethis.)
StepFour:Whoisconsideredaparentinthisstep?
Readthesenotestodeterminewhoisconsideredaparentforpurposesofthisform.
AnswerallquestionsinStepFouraboutthem,evenifyoudonotlivewiththem.
Areyouanorphan,orareyouorwereyou(untilage18)award/dependentofthe
court?IfYes,skipStepFour.Ifyourparentsarebothlivingandmarriedtoeach
other,answerthequestionsaboutthem.Ifyourparentiswidowedorsingle,answer
thequestionsaboutthatparent.Ifyourwidowedparentisremarriedasoftoday,
answerthequestionsaboutthatparentandthepersonwhomyourparentmarried
(yourstepparent).Ifyourparentsaredivorcedorseparated,answerthequestions
abouttheparentyoulivedwithmoreduringthepast12months.(Ifyoudidnotlive
withoneparentmorethantheother,giveanswersabouttheparentwhoprovided
morefinancialsupportduringthelast12months,orduringthemostrecentyearthat
youactuallyreceivedsupportfromaparent.)Ifthisparentisremarriedasoftoday,
answerthequestionsontherestofthisformaboutthatparentandthepersonwhom
yourparentmarried(yourstepparent).
References
Beizer,Boris(1990).SoftwareTestingTechniques(SecondEdition).VanNostrand
Reinhold.
Binder,RobertV.(2000).TestingObject-OrientedSystems:Models,Patterns,and
Tools.Addison-Wesley.
Chapter6:PairwiseTesting
AntonwasattractedtoAngelalikeamothtoaflame-notjustanymoth,butoneofthe
giantsilkmothsofthegenusHyalophora,perhapsHyalophoraeuryalus,whosegreat
red-brownwingswithwhitebasalandpostmedianlinesflapalmostlanguorouslyuntilone
ignitesintheflame,fanningtheconflagrationtoevergreaterheightsuntilburningdown
tothehirsutethoraxandabdomen,thefat-ladencontentsofwhichprovideasatisfying
sizzletoendtheagony.
—AndrewAmlen
Introduction
AstheyusedtosayonMontyPython,"Andnowforsomethingcompletelydifferent."
Considerthesesituations:
AWebsitemustoperatecorrectlywithdifferentbrowsers—InternetExplorer5.0,5.5,
and6.0,Netscape6.0,6.1,and7.0,Mozilla1.1,andOpera7;usingdifferentplug-ins
—RealPlayer,MediaPlayer,ornone;runningondifferentclientoperatingsystems—
Windows95,98,ME,NT,2000,andXP;receivingpagesfromdifferentservers—IIS,
Apache,andWebLogic;runningondifferentserveroperatingsystems—WindowsNT,
2000,andLinux.
WebCombinations
8browsers
3plug-ins
6clientoperatingsystems
3servers
3serverOS
1,296combinations.
Abankhascreatedanewdataprocessingsystemthatisreadyfortesting.Thisbank
hasdifferentkindsofcustomers—consumers,veryimportantconsumers,businesses,
andnon-profits;differentkindsofaccounts—checking,savings,mortgages,consumer
loans,andcommercialloans;theyoperateindifferentstates,eachwithdifferent
regulations—California,Nevada,Utah,Idaho,Arizona,andNewMexico.
BankCombinations
4customertypes
5accounttypes
6states
120combinations.
Inanobject-orientedsystem,anobjectofclassAcanpassamessagecontaininga
parameterPtoanobjectofclassX.ClassesB,C,andDinheritfromAsotheytoocan
sendthemessage.ClassesQ,R,S,andTinheritfromPsotheytoocanbepassed
astheparameter.ClassesYandZinheritfromXsotheytoocanreceivethemessage.
OOCombinations
4senders
5parameters
3receivers
60combinations.
Insight
Studentsinmyclassesoftenhaveaverydifficulttimethinkingofbadwaystodo
things.Cultivatetheskillofchoosingpoorly.Itwillbeinvaluableinevaluatingothers'
ideas.
CanYouBelieveThis?
Astudentinoneofmyclassessharedthisstory:Hisorganizationusesaprocesstheycall
"Post-InstallationTestPlanning."Itsoundsimpressiveuntilyoudecipherit.Whatevertests
theyhappentorunthathappentopassaredocumentedastheirTestPlan.
Whatdotheseverydifferentsituationsallhaveincommon?Eachhasalargenumberof
combinationsthatshouldbetested.Eachhasalargenumberofcombinationsthatmayberisky
ifwedonottest.Eachhassuchalargenumberofcombinationsthatwemaynothavethe
resourcestoconstructandrunallthetests,therearejusttoomany.Wemust,somehow,
selectareasonablysizedsubsetthatwecouldtestgivenourresourceconstraints.Whatare
somewaysofchoosingsuchasubset?Thisliststartswiththeworstschemesbutdoes
improve:
Don'ttestatall.Simplygiveupbecausethenumberofinput
combinations,andthusthenumberoftestcases,isjusttoogreat.
Testallcombinations[once],butdelaytheprojectsoitmissesits
marketwindowsothateveryonequitsfromstress,orthecompany
goesoutofbusiness.
Chooseoneortwotestsandhopeforthebest.
Choosetheteststhatyouhavealreadyrun,perhapsaspartof
programmer-ledtesting.Incorporatethemintoaformaltestplanand
runthemagain.
Choosetheteststhatareeasytocreateandrun.Ignorewhetherthey
provideusefulinformationaboutthequalityoftheproduct.
Makealistofallthecombinationsandchoosethefirstfew.
Makealistofallthecombinationsandchoosearandomsubset.
Bymagic,chooseaspeciallyselected,fairlysmallsubsetthatfindsa
greatmanydefects—morethanyouwouldexpectfromsuchasubset.
Thislastschemesoundslikeawinner(butitisalittlevague).Thequestionis—whatisthe
"magic"thatallowsustochoosethat"speciallyselected"subset?
Insight Randomselectioncanbeaverygoodapproachtochoosingasubsetbutmost
peoplehaveadifficulttimechoosingtrulyrandomly.
Theanswerisnottoattempttotestallthecombinationsforallthevaluesforallthevariables
buttotestallpairsofvariables.Thissignificantlyreducesthenumberofteststhatmustbe
createdandrun.Considerthesignificantreductionsintesteffortintheseexamples:
Ifasystemhadfourdifferentinputparametersandeachonecouldtakeononeof
threedifferentvalues,thenumberofcombinationsis34whichis81.Itispossibleto
coverallthepairwiseinputcombinationsinonlyninetests.
Ifasystemhadthirteendifferentinputparametersandeachonecouldtakeononeof
threedifferentvalues,thenumberofcombinationsis313whichis1,594,323.Itis
possibletocoverallthepairwiseinputcombinationsinonlyfifteentests.
Ifasystemhadtwentydifferentinputparametersandeachonecouldtakeononeof
tendifferentvalues,thenumberofcombinationsis1020.Itispossibletocoverallthe
pairwiseinputcombinationsinonly180tests.
Thereismuchanecdotalevidenceaboutthebenefitofpairwisetesting.Unfortunately,thereare
onlyafewdocumentedstudies:
InacasestudypublishedbyBrownlieofAT&Tregardingthetestingofalocal-area
network-basedelectronicmailsystem,pairwisetestingdetected28percentmore
defectsthantheiroriginalplanofdevelopingandexecuting1,500testcases(later
reducedto1,000becauseoftimeconstraints)andtook50percentlesseffort.
AstudybytheNationalInstituteofStandardsandTechnologypublishedbyWallaceand
Kuhnonsoftwaredefectsinrecalledmedicaldevicesreviewedfifteenyearsofdefect
data.Theyconcludedthat98percentofthereportedsoftwareflawscouldhavebeen
detectedbytestingallpairsofparametersettings.
KuhnandReillyanalyzeddefectsrecordedintheMozillaWebbrowserdatabase.They
determinedthatpairwisetestingwouldhavedetected76percentofthereported
errors.
Whydoespairwisetestingworksowell?Idon'tknow.Thereisnounderlying"software
physics"thatrequiresit.Onehypothesisisthatmostdefectsareeithersingle-modedefects
(thefunctionundertestsimplydoesnotworkandanytestofthatfunctionwouldfindthe
defect)ortheyaredouble-modedefects(itisthepairingofthisfunction/modulewiththat
function/modulethatfailseventhoughallotherpairingsperformproperly).Pairwisetesting
definesaminimalsubsetthatguidesustotestforallsingle-modeanddouble-modedefects.
Thesuccessofthistechniqueonmanyprojects,bothdocumentedandundocumented,isa
greatmotivationforitsuse.
Note
Pairwisetestingmaynotchoosecombinationswhichthedevelopersandtestersknow
areeitherfrequentlyusedorhighlyrisky.Ifthesecombinationsexist,usethepairwise
tests,thenaddadditionaltestcasestominimizetheriskofmissinganimportant
combination.
Technique
Twodifferenttechniquesareusedtoidentifyallthepairsforcreatingtestcases—orthogonal
arraysandtheAllpairsalgorithm.
OrthogonalArrays
Whatareorthogonalarrays?TheoriginoforthogonalarrayscanbetracedbacktoEuler,the
greatmathematician,intheguiseofLatinSquares.GenichiTaguchihaspopularizedtheirusein
hardwaretesting.AnexcellentreferencebookisQualityEngineeringUsingRobustDesignby
MadhavS.Phadke.
Considerthenumbers1and2.Howmanypaircombinations(combinationstakentwoatatime)
of'1'and'2'exist?{1,1},{1,2},{2,1}and{2,2}.Anorthogonalarrayisatwo-dimensionalarray
ofnumbersthathasthisinterestingproperty—chooseanytwocolumnsinthearray.Allthe
pairwisecombinationsofitsvalueswilloccurineverypairofcolumns.Let'sexamineanL4(23)
array:
Table6-1:L4(23)OrthogonalArray
1 2 3
11 1 1
21 2 2
32 1 2
42 2 1
Thegraycolumnheadingsandrownumbersarenotpartoftheorthogonalarraybutare
includedforconvenienceinreferencingthecells.Examinecolumns1and2—dothefour
combinationsof1and2allappearinthatcolumnpair?Yes,andintheorderlistedearlier.Now
examinecolumns1and3—dothefourcombinationsof1and2appearinthatcolumnpair?
Yes,althoughinadifferentorder.Finally,examinecolumns2and3—dothefourcombinations
appearinthatcolumnpairalso?Yestheydo.TheL4(23)arrayisorthogonal;thatis,choose
anytwocolumns,allthepairwisecombinationswilloccurinallthecolumnpairs.
Important
Note
Asatesteryoudonothavetocreateorthogonalarrays,allyoumustdois
locateoneofthepropersize.Books,Websites,andautomatedtoolswill
helpyoudothis.
Anoteaboutthecurious(butstandard)notation:L4meansanorthogonalarraywithfourrows,
(23)isnotanexponent.Itmeansthatthearrayhasthreecolumns,eachwitheithera1ora2.
Figure6-1:Orthogonalarraynotation Let'sconsideralargerorthogonal
array.Giventhenumbers1,2and3,howmanypaircombinationsof1,2,and3exist?{1,1},
{1,2},{1,3},{2,1},{2,2},{2,3},{3,1},{3,2},and{3,3}.BelowisanL9(34)array:
Table6-2:L9(34)OrthogonalArray
1 2 3 4
1 1 1 1 1
2 1 2 2 2
3 1 3 3 3
4 2 1 2 3
5 2 2 3 1
6 2 3 1 2
7 3 1 3 2
8 3 2 1 3
9 3 3 2 1
Examinecolumns1and2—dotheninecombinationsof1,2,and3allappearinthatcolumn
pair?Yes.Nowexaminecolumns1and3—dotheninecombinationsof1,2,and3appearin
thatcolumnpair?Yes,althoughinadifferentorder.Examinecolumns1and4—dothenine
combinationsappearinthatcolumnpairalso?Yestheydo.Continueonbyexaminingother
pairsofcolumns—2and3,2and4,andfinally3and4.TheL9(34)arrayisorthogonal;that
is,chooseanytwocolumns,allthecombinationswilloccurinallofthecolumnpairs.
Tool
TherdExperttoolfromPhadkeAssociatesimplementstheorthogonalarrayapproach.
Seehttp://www.phadkeassociates.com
Notethatnotallcombinationsof1s,2s,and3sappearinthearray.Forexample,{1,1,2},
{1,2,1},and{2,2,2)donotappear.Orthogonalarraysonlyguaranteethatallthepair
combinationsexistinthearray.Combinationssuchas{2,2,2}aretriples,notpairs.
ThefollowingisanL18(35)orthogonalarray.Ithasfivecolumns,eachcontaininga1,2,or3.
Examinecolumns1and2forthepair{1,1}.Doesthatpairexistinthosetwocolumns?Wait!
Don'tlookatthearray.Fromthedefinitionofanorthogonalarray,whatistheanswer?Yes,
thatpairexistsalongwitheveryotherpairof1,2,and3.Thepair{1,1}isinrow1.Notethat
{1,1}alsoappearsinrow6.Returningtotheoriginaldescriptionoforthogonalarrays,
Anorthogonalarrayisatwo-dimensionalarrayofnumbersthathasthisinteresting
property—chooseanytwocolumnsinthearray.Allthepairwisecombinationsofits
valueswilloccurineverycolumnpair.
Thisdefinitionisnottotallycomplete.Notonlywillallthepaircombinationsoccurinthe
array,butifanypairoccursmultipletimes,allpairswilloccurthatsamenumberoftimes.
Thisisbecauseorthogonalarraysare"balanced."Examinecolumns3and5—lookfor{3,2}.
Thatcombinationappearsinrows6and17.
Table6-3:L18(35)OrthogonalArray
12345
1 1 1 1 1 1
2 1 2 3 3 1
3 1 3 2 3 2
4 1 2 2 1 3
5 1 3 1 2 3
6 1 1 3 2 2
7 2 2 2 2 2
8 2 3 1 1 2
9 2 1 3 1 3
10 2 3 3 2 1
11 2 1 2 3 1
12 2 2 1 3 3
13 3 3 3 3 3
14 3 1 2 2 3
15 3 2 1 2 1
16 3 1 1 3 2
17 3 2 3 1 2
18 3 3 2 1 1
Inorthogonalarraysnotallofthecolumnsmusthavethesamerangeofvalues(1..2,1..3,
1..5,etc.).Someorthogonalarraysaremixed.ThefollowingisanL18(2137)orthogonal
array.Ithasonecolumnof1sand2s,andsevencolumnsof1s,2s,and3s.
Table6-4:L18(2137)OrthogonalArray
12345678
1 11111111
2 11222222
3 11333333
4 12112233
5 12223311
6 12331122
7 13121323
8 13232131
9 13313212
10 21133221
11 21211332
12 21322113
13 22123132
14 22231213
15 22312321
16 23132312
17 23213123
18 23321231
Reference
NeilJ.A.Sloanemaintainsaverycomprehensivecatalogoforthogonalarraysat
http://www.research.att.com/~njas/oadir/index.html
UsingOrthogonalArrays
Theprocessofusingorthogonalarraystoselectpairwisesubsetsfortestingis:
1. Identifythevariables.
2. Determinethenumberofchoicesforeachvariable.
3. Locateanorthogonalarraywhichhasacolumnforeachvariableandvalueswithinthe
columnsthatcorrespondtothechoicesforeachvariable.
4. Mapthetestproblemontotheorthogonalarray.
5. Constructthetestcases.
Ifthisseemsrathervagueatthispointit'stimeforanexample.
Web-basedsystemssuchasBrown&DonaldsonandtheStatelessUniversityRegistration
Systemmustoperateinanumberofenvironments.Let'sexecutetheprocessstep-by-step
usinganorthogonalarraytochoosetestcases.Considerthefirstexampleintheintroduction
describingthesoftwarecombinationsaWebsitemustoperatewith.
1. Identifythevariables.
ThevariablesareBrowser,Plug-in,Clientoperatingsystem,Server,andServer
operatingsystem.
2. Determinethenumberofchoicesforeachvariable.
Browser-InternetExplorer5.0,5.5,and6.0,Netscape6.0,6.1,and7.0,Mozilla1.1,
andOpera7(8choices).
Plug-in-None,RealPlayer,andMediaPlayer(3choices).
Clientoperatingsystem-Windows95,98,ME,NT,2000,andXP(6choices).
Server-IIS,Apache,andWebLogic(3choices).
Serveroperatingsystem-WindowsNT,2000,andLinux(3choices).
Multiplying8x3x6x3x3wefindthereare1,296combinations.For"complete"test
coverage,eachofthesecombinationsshouldbetested.
3. Locateanorthogonalarraythathasacolumnforeachvariableandvalues
withinthecolumnsthatcorrespondtothechoicesofeachvariable.
Whatsizearrayisneeded?First,itmusthavefivecolumns,oneforeachvariablein
thisexample.Thefirstcolumnmustsupporteightdifferentlevels(1through8).The
secondcolumnmustsupportthreelevels(1through3).Thethirdrequiressixlevels.
Thefourthandthefiftheachrequirethreelevels.Theperfectsizeorthogonalarray
wouldbe816133(onecolumnof1through8,onecolumnof1through6,andthree
columnsof1through3).Unfortunately,oneofthisexactsizedoesnotexist.When
thisoccurs,wesimplypickthenextlargerarray.
Important Asatesteryoudonothavetocreateorthogonalarrays.Allyou
Note mustdoislocateoneofthepropersizeandthenperformthe
mappingofthetestproblemontothearray.
Thefollowingorthogonalarraymeetsourrequirements.It'sanL64(8243)array.
OrthogonalarrayscanbefoundinanumberofbooksandontheWeb.Afavorite
bookisQualityEngineeringUsingRobustDesignbyMadhavS.Phadke.Inaddition,
anexcellentcatalogismaintainedontheWebbyNeilJ.A.SloaneofAT&T.See
http://www.research.att.com/~njas/oadir/index.html.
Therequirementof8161(onecolumnof1through8and1columnof1through6)is
metby82(twocolumnsof1through8).Therequirementof33(threecolumnsof1
through3)ismetby43(threecolumnsof1through4).
Thenumberofcombinationsofallthevaluesofallthevariablesis1,296andthus
1,296testcasesshouldbecreatedandrunforcompletecoverage.Usingthis
orthogonalarray,allpairsofallthevaluesofallthevariablescanbecoveredinonly
sixty-fourtests,a95percentreductioninthenumberoftestcases.
Table6-5:L64(8243)OrthogonalArray
12345
1 11111
2 14344
3 14244
4 11411
5 13533
6 12722
7 12622
8 13833
9 34133
10 31322
11 31222
12 34433
13 32511
14 33744
15 33644
16 32811
17 23121
18 22334
19 22234
20 23421
21 21543
22 24712
23 24612
24 21843
25 42143
26 43312
27 43212
28 42443
29 44521
30 41734
31 41634
32 44821
33 52142
34 53313
35 53213
36 52442
37 54524
38 51731
39 51631
40 54824
41 73124
42 72331
43 72231
44 73424
45 71542
46 74713
47 74613
48 71842
49 64132
50 61323
51 61223
52 64432
53 62514
54 63741
55 63641
56 62814
57 81114
58 84341
59 84241
60 81414
61 83532
62 82723
63 82623
64 83832
4. Mapthetestproblemontotheorthogonalarray.
TheBrowserchoiceswillbemappedontocolumn1oftheorthogonalarray.Cells
containinga1willrepresentIE5.0;cellswitha2willrepresentIE5.5;cellswitha3
willrepresentIE6.0;etc.Themappingis:
1↔IE5.0
2↔IE5.5
3↔IE6.0
4↔Netscape6.0
5↔Netscape6.1
6↔Netscape7.0
7↔Mozilla1.1
8↔Opera7
Partiallyfillinginthefirstcolumngives:
Table6-6:L64(8243)withapartialmappingofitsfirstcolumn.
Browser 2 3 4 5
1 IE5.0 1 1 1 1
2 1 4 3 4 4
3 1 4 2 4 4
4 1 1 4 1 1
5 1 3 5 3 3
6 1 2 7 2 2
7 1 2 6 2 2
8 1 3 8 3 3
9 IE6.0 4 1 3 3
10 3 1 3 2 2
11 3 1 2 2 2
12 3 4 4 3 3
13 3 2 5 1 1
14 3 3 7 4 4
15 3 3 6 4 4
16 3 2 8 1 1
17 IE5.5 3 1 2 1
18 2 2 3 3 4
19 2 2 2 3 4
20 2 3 4 2 1
21 2 1 5 4 3
22 2 4 7 1 2
23 2 4 6 1 2
24 2 1 8 4 3
25 Net6.0 2 1 4 3
26 4 3 3 1 2
27 4 3 2 1 2
28 4 2 4 4 3
29 4 4 5 2 1
30 4 1 7 3 4
31 4 1 6 3 4
32 4 4 8 2 1
33 Net6.1 2 1 4 2
34 5 3 3 1 3
35 5 3 2 1 3
36 5 2 4 4 2
37 5 4 5 2 4
38 5 1 7 3 1
39 5 1 6 3 1
40 5 4 8 2 4
41 Moz1.1 3 1 2 4
42 7 2 3 3 1
43 7 2 2 3 1
44 7 3 4 2 4
45 7 1 5 4 2
46 7 4 7 1 3
47 7 4 6 1 3
48 7 1 8 4 2
49 Net7.0 4 1 3 2
50 6 1 3 2 3
51 6 1 2 2 3
52 6 4 4 3 2
53 6 2 5 1 4
54 6 3 7 4 1
55 6 3 6 4 1
56 6 2 8 1 4
57 Opera7 1 1 1 4
58 8 4 3 4 1
59 8 4 2 4 1
60 8 1 4 1 4
61 8 3 5 3 2
62 8 2 7 2 3
63 8 2 6 2 3
64 8 3 8 3 2
Isitclearwhatishappening?Incolumn1(whichwehavechosentorepresentthe
Browser)everycellcontaininga1isbeingreplacedwith"IE5.0."Everycell
containinga2isbeingreplacedwith"IE5.5."Everycellcontainingan8isbeing
replacedwith"Opera7,"etc.
We'llcontinuebycompletingthemapping(replacement)ofallthecellsincolumn1.
Notethatthemappingbetweenthevariablevaluesandthe1s,2s,and3sistotally
arbitrary.Thereisnologicalconnectionbetween"1"andIE5.0or"7"andMozilla1.1.
But,althoughtheinitialassignmentisarbitrary,oncechosen,theassignmentsanduse
mustremainconsistentwithineachcolumn.
Table6-7:L64(8243)withafullmappingofitsfirstcolumn.
Browser 2 3 4 5
1 IE5.0 1 1 1 1
2 IE5.0 4 3 4 4
3 IE5.0 4 2 4 4
4 IE5.0 1 4 1 1
5 IE5.0 3 5 3 3
6 IE5.0 2 7 2 2
7 IE5.0 2 6 2 2
8 IE5.0 3 8 3 3
9 IE6.0 4 1 3 3
10 IE6.0 1 3 2 2
11 IE6.0 1 2 2 2
12 IE6.0 4 4 3 3
13 IE6.0 2 5 1 1
14 IE6.0 3 7 4 4
15 IE6.0 3 6 4 4
16 IE6.0 2 8 1 1
17 IE5.5 3 1 2 1
18 IE5.5 2 3 3 4
19 IE5.5 2 2 3 4
20 IE5.5 3 4 2 1
21 IE5.5 1 5 4 3
22 IE5.5 4 7 1 2
23 IE5.5 4 6 1 2
24 IE5.5 1 8 4 3
25 Net6.0 2 1 4 3
26 Net6.0 3 3 1 2
27 Net6.0 3 2 1 2
28 Net6.0 2 4 4 3
29 Net6.0 4 5 2 1
30 Net6.0 1 7 3 4
31 Net6.0 1 6 3 4
32 Net6.0 4 8 2 1
33 Net6.1 2 1 4 2
34 Net6.1 3 3 1 3
35 Net6.1 3 2 1 3
36 Net6.1 2 4 4 2
37 Net6.1 4 5 2 4
38 Net6.1 1 7 3 1
39 Net6.1 1 6 3 1
40 Net6.1 4 8 2 4
41 Moz1.1 3 1 2 4
42 Moz1.1 2 3 3 1
43 Moz1.1 2 2 3 1
44 Moz1.1 3 4 2 4
45 Moz1.1 1 5 4 2
46 Moz1.1 4 7 1 3
47 Moz1.1 4 6 1 3
48 Moz1.1 1 8 4 2
49 Net7.0 4 1 3 2
50 Net7.0 1 3 2 3
51 Net7.0 1 2 2 3
52 Net7.0 4 4 3 2
53 Net7.0 2 5 1 4
54 Net7.0 3 7 4 1
55 Net7.0 3 6 4 1
56 Net7.0 2 8 1 4
57 Opera7 1 1 1 4
58 Opera7 4 3 4 1
59 Opera7 4 2 4 1
60 Opera7 1 4 1 4
61 Opera7 3 5 3 2
62 Opera7 2 7 2 3
63 Opera7 2 6 2 3
64 Opera7 3 8 3 2
Nowthatthefirstcolumnhasbeenmapped,let'sproceedtothenextone.ThePlug-in
choiceswillbemappedontocolumn2ofthearray.Cellscontaininga1willrepresent
None(Noplug-in);cellswitha2willrepresentRealPlayer;cellswitha3willrepresent
MediaPlayer;cellswitha4willnotbemappedatthepresenttime.Themappingis:
1↔None
2↔RealPlayer
3↔MediaPlayer
4↔Notused(atthistime)
Fillinginthesecondcolumngives:
Table6-8:L64(8243)withafullmappingofitsfirstandsecondcolumns.
Browser Plug-In 3 4 5
1 IE5.0 None 1 1 1
2 IE5.0 4 3 4 4
3 IE5.0 4 2 4 4
4 IE5.0 None 4 1 1
5 IE5.0 MediaPlayer 5 3 3
6 IE5.0 RealPlayer 7 2 2
7 IE5.0 RealPlayer 6 2 2
8 IE5.0 MediaPlayer 8 3 3
9 IE6.0 4 1 3 3
10 IE6.0 None 3 2 2
11 IE6.0 None 2 2 2
12 IE6.0 4 4 3 3
13 IE6.0 RealPlayer 5 1 1
14 IE6.0 MediaPlayer 7 4 4
15 IE6.0 MediaPlayer 6 4 4
16 IE6.0 RealPlayer 8 1 1
17 IE5.5 MediaPlayer 1 2 1
18 IE5.5 RealPlayer 3 3 4
19 IE5.5 RealPlayer 2 3 4
20 IE5.5 MediaPlayer 4 2 1
21 IE5.5 None 5 4 3
22 IE5.5 4 7 1 2
23 IE5.5 4 6 1 2
24 IE5.5 None 8 4 3
25 Net6.0 RealPlayer 1 4 3
26 Net6.0 MediaPlayer 3 1 2
27 Net6.0 MediaPlayer 2 1 2
28 Net6.0 RealPlayer 4 4 3
29 Net6.0 4 5 2 1
30 Net6.0 None 7 3 4
31 Net6.0 None 6 3 4
32 Net6.0 4 8 2 1
33 Net6.1 RealPlayer 1 4 2
34 Net6.1 MediaPlayer 3 1 3
35 Net6.1 MediaPlayer 2 1 3
36 Net6.1 RealPlayer 4 4 2
37 Net6.1 4 5 2 4
38 Net6.1 None 7 3 1
39 Net6.1 None 6 3 1
40 Net6.1 4 8 2 4
41 Moz1.1 MediaPlayer 1 2 4
42 Moz1.1 RealPlayer 3 3 1
43 Moz1.1 RealPlayer 2 3 1
44 Moz1.1 MediaPlayer 4 2 4
45 Moz1.1 None 5 4 2
46 Moz1.1 4 7 1 3
47 Moz1.1 4 6 1 3
48 Moz1.1 None 8 4 2
49 Net7.0 4 1 3 2
50 Net7.0 None 3 2 3
51 Net7.0 None 2 2 3
52 Net7.0 4 4 3 2
53 Net7.0 RealPlayer 5 1 4
54 Net7.0 MediaPlayer 7 4 1
55 Net7.0 MediaPlayer 6 4 1
56 Net7.0 RealPlayer 8 1 4
57 Opera7 None 1 1 4
58 Opera7 4 3 4 1
59 Opera7 4 2 4 1
60 Opera7 None 4 1 4
61 Opera7 MediaPlayer 5 3 2
62 Opera7 RealPlayer 7 2 3
63 Opera7 RealPlayer 6 2 3
64 Opera7 MediaPlayer 8 3 2
Nowthatthefirstandsecondcolumnshavebeenmapped,let'sproceedtomapthe
nextthreecolumnssimultaneously.
ThemappingforClientoperatingsystemis:
1↔Windows95
2↔Windows98
3↔WindowsME
4↔WindowsNT
5↔Windows2000
6↔WindowsXP
7↔Notused(atthistime)
8↔Notused(atthistime)
ThemappingforServersis:
1↔IIS
2↔Apache
3↔WebLogic
4↔Notused(atthistime)
ThemappingforServeroperatingsystemis:
1↔WindowsNT
2↔Windows2000
3↔Linux
4↔Notused(atthistime)
Fillingintheremainderofthecolumnsgives:
Table6-9:L64(8243)withafullmappingofallitscolumns.
Browser Plug-in ClientOS Server ServerOS
1 IE5.0 None Win95 IIS WinNT
2 IE5.0 4 WinME 4 4
3 IE5.0 4 Win98 4 4
4 IE5.0 None WinNT IIS WinNT
5 IE5.0 MediaPlayer Win2000 WebLogic Linux
6 IE5.0 RealPlayer 7 Apache Win2000
7 IE5.0 RealPlayer WinXP Apache Win2000
8 IE5.0 MediaPlayer 8 WebLogic Linux
9 IE6.0 4 Win95 WebLogic Linux
10 IE6.0 None WinME Apache Win2000
11 IE6.0 None Win98 Apache Win2000
12 IE6.0 4 WinNT WebLogic Linux
13 IE6.0 RealPlayer Win2000 IIS WinNT
14 IE6.0 MediaPlayer 7 4 4
15 IE6.0 MediaPlayer WinXP 4 4
16 IE6.0 RealPlayer 8 US WinNT
17 IE5.5 MediaPlayer Win95 Apache WinNT
18 IE5.5 RealPlayer WinME WebLogic 4
19 IE5.5 RealPlayer Win98 WebLogic 4
20 IE5.5 MediaPlayer WinNT Apache WinNT
21 IE5.5 None Win2000 4 Linux
22 IE5.5 4 7 IIS Win2000
23 IE5.5 4 WinXP IIS Win2000
24 IE5.5 None 8 4 Linux
25 Net6.0 RealPlayer Win95 4 Linux
26 Net6.0 MediaPlayer WinME IIS Win2000
27 Net6.0 MediaPlayer Win98 IIS Win2000
28 Net6.0 RealPlayer WinNT 4 Linux
29 Net6.0 4 Win2000 Apache WinNT
30 Net6.0 None 7 WebLogic 4
31 Net6.0 None WinXP WebLogic 4
32 Net6.0 4 8 Apache WinNT
33 Net6.1 RealPlayer Win95 4 Win2000
34 Net6.1 MediaPlayer WinME IIS Linux
35 Net6.1 MediaPlayer Win98 IIS Linux
36 Net6.1 RealPlayer WinNT 4 Win2000
37 Net6.1 4 Win2000 Apache 4
38 Net6.1 None 7 WebLogic WinNT
39 Net6.1 None WinXP WebLogic 1WinNT
40 Net6.1 4 8 Apache 4
41 Moz1.1 MediaPlayer Win95 Apache 4
42 Moz1.1 RealPlayer WinME WebLogic WinNT
43 Moz1.1 RealPlayer Win98 WebLogic WinNT
44 Moz1.1 MediaPlayer WinNT Apache 4
45 Moz1.1 None Win2000 4 Win2000
46 Moz1.1 4 7 IIS Linux
47 Moz1.1 4 WinXP IIS Linux
48 Moz1.1 None 8 4 Win2000
49 Net7.0 4 Win95 WebLogic Win2000
50 Net7.0 None WinME Apache Linux
51 Net7.0 None Win98 Apache Linux
52 Net7.0 4 WinNT WebLogic Win2000
53 Net7.0 RealPlayer Win2000 IIS 4
54 Net7.0 MediaPlayer 7 4 WinNT
55 Net7.0 MediaPlayer WinXP 4 WinNT
56 Net7.0 RealPlayer 8 IIS 4
57 Opera7 None Win95 IIS 4
58 Opera7 4 WinME 4 WinNT
59 Opera7 4 Win98 4 WinNT
60 Opera7 None WinNT IIS 4
61 Opera7 MediaPlayer Win2000 WebLogic Win2000
62 Opera7 RealPlayer 7 Apache Linux
63 Opera7 RealPlayer WinXP Apache Linux
64 Opera7 MediaPlayer 8 WebLogic Win2000
Wereitnotforthefewcellsthatremainunassigned,themappingoftheorthogonal
array,andthustheselectionofthetestcases,wouldbecompleted.Whataboutthe
unassignedcells—first,whydotheyexist?;second,whatshouldbedonewiththem?
Theunassignedcellsexistbecausetheorthogonalarraychosenwas"toobig."The
perfectsizewouldbean816133array;thatis,onecolumnthatvariesfrom1to8;one
columnthatvariesfrom1to6;andthreecolumnsthatvaryfrom1to3.Unfortunately,
thatspecificsizeorthogonalarraydoesnotexist.Orthogonalarrayscannotbe
constructedforanyarbitrarysizeparameters.Theycomeinfixed,"quantum"sizes.
Youcanconstructone"thisbig";youcanconstructone"thatbig";butyoucannot
necessarilyconstructonein-between.FamousSoftwareTesterMickJaggergives
excellentadviceregardingthis,"Youcan'talwaysgetwhatyouwant,Butifyoutry
sometimes,Youjustmightfind,yougetwhatyouneed."
FamousSoftwareTester
Iftheperfectsizearraydoesnotexist,chooseonethatisslightlybiggerandapply
thesetworulestodealwiththe"excess."Thefirstruledealswithextracolumns.If
theorthogonalarraychosenhasmorecolumnsthanneededforaparticulartest
scenario,simplydeletethem.Thearraywillremainorthogonal.Thesecondruledeals
withextravaluesforavariable.Inthecurrentexample,column3runsfrom1to8but
only1through6isneeded.Itistemptingtodeletetherowsthatcontainthesecells
butDON'T.The"orthogonalness"maybelost.Eachrowinthearrayexiststoprovide
atleastonepaircombinationthatappearsnowhereelseinthearray.Ifyoudeletea
row,youlosethattestcase.Insteadofdeletingthem,simplyconverttheextracells
tovalidvalues.Someautomatedtoolsrandomlychoosefromthesetofvalidvalues
foreachcellwhileotherschooseonevalidvalueanduseitineverycellwithina
column.Eitherapproachisacceptable.Usingthissecondapproach,we'llcomplete
theorthogonalarray.Notethatitmaybedifficulttomaintainthe"balanced"aspectof
thearraywhenassigningvaluestotheseextracells.
Table6-10:L64(8243)withafullmappingofallitscolumnsincludingthe
"extra"cells.
Browser Plug-in ClientOS Server ServerOS
1 IE5.0 None Win95 IIS WinNT
2 IE5.0 None WinME IIS WinNT
3 IE5.0 None Win98 IIS WinNT
4 IE5.0 None WinNT IIS WinNT
5 IE5.0 MediaPlayer Win2000 WebLogic Linux
6 IE5.0 RealPlayer Win95 Apache Win2000
7 IE5.0 RealPlayer WinXP Apache Win2000
8 IE5.0 MediaPlayer Win98 WebLogic Linux
9 IE6.0 None Win95 WebLogic Linux
10 IE6.0 None WinME Apache Win2000
11 IE6.0 None Win98 Apache Win2000
12 IE6.0 None WinNT WebLogic Linux
13 IE6.0 RealPlayer Win2000 IIS WinNT
14 IE6.0 MediaPlayer Win95 IIS WinNT
15 IE6.0 MediaPlayer WinXP IIS WinNT
16 IE6.0 RealPlayer Win98 IIS WinNT
17 IE5.5 MediaPlayer Win95 Apache WinNT
18 IE5.5 RealPlayer WinME WebLogic WinNT
19 IE5.5 RealPlayer Win98 WebLogic WinNT
20 IE5.5 MediaPlayer WinNT Apache WinNT
21 IE5.5 None Win2000 IIS Linux
22 IE5.5 None Win95 IIS Win2000
23 IE5.5 None WinXP IIS Win2000
24 IE5.5 None Win98 IIS Linux
25 Net6.0 RealPlayer Win95 IIS Linux
26 Net6.0 MediaPlayer WinME IIS Win2000
27 Net6.0 MediaPlayer Win98 IIS Win2000
28 Net6.0 RealPlayer WinNT IIS Linux
29 Net6.0 None Win2000 Apache WinNT
30 Net6.0 None Win95 WebLogic WinNT
31 Net6.0 None WinXP WebLogic WinNT
32 Net6.0 None Win98 Apache WinNT
33 Net6.1 RealPlayer Win95 IIS Win2000
34 Net6.1 MediaPlayer WinME IIS Linux
35 Net6.1 MediaPlayer Win98 IIS Linux
36 Net6.1 RealPlayer WinNT IIS Win2000
37 Net6.1 None Win2000 Apache WinNT
38 Net6.1 None Win95 WebLogic WinNT
39 Net6.1 None WinXP WebLogic WinNT
40 Net6.1 None Win98 Apache WinNT
41 Moz1.1 MediaPlayer Win95 Apache WinNT
42 Moz1.1 RealPlayer WinME WebLogic WinNT
43 Moz1.1 RealPlayer Win98 WebLogic WinNT
44 Moz1.1 MediaPlayer WinNT Apache WinNT
45 Moz1.1 None Win2000 IIS Win2000
46 Moz1.1 None Win95 IIS Linux
47 Moz1.1 None WinXP IIS Linux
48 Moz1.1 None Win98 IIS Win2000
49 Net7.0 None Win95 WebLogic Win2000
50 Net7.0 None WinME Apache Linux
51 Net7.0 None Win98 Apache Linux
52 Net7.0 None WinNT WebLogic Win2000
53 Net7.0 RealPlayer Win2000 IIS WinNT
54 Net7.0 MediaPlayer Win95 IIS WinNT
55 Net7.0 MediaPlayer WinXP IIS WinNT
56 Net7.0 RealPlayer Win98 IIS WinNT
57 Opera7 None Win95 IIS WinNT
58 Opera7 None WinME IIS WinNT
59 Opera7 None Win98 IIS WinNT
60 Opera7 None WinNT IIS WinNT
61 Opera7 MediaPlayer Win2000 WebLogic Win2000
62 Opera7 RealPlayer Win95 Apache Linux
63 Opera7 RealPlayer WinXP Apache Linux
64 Opera7 MediaPlayer Win98 WebLogic Win2000
5. Constructthetestcases.
Now,allthatremainsistoconstructatestcaseforeachrowintheorthogonalarray.
Notethatthearrayspecifiesonlytheinputconditions.Anoracle(usuallythetester)is
requiredtodeterminetheexpectedresultforeachtest.
AllpairsAlgorithm
Usingorthogonalarraysisonewaytoidentifyallthepairs.Asecondwayistouseanalgorithm
thatgeneratesthepairsdirectlywithoutresortingtoan"external"devicelikeanorthogonal
array.
Reference
JamesBachprovidesatooltogenerateallpairscombinationsathttp://www.satisfice.com.
ClickonTestMethodologyandlookforAllpairs.
WardCunninghamprovidesfurtherdiscussionandthesourcecodeforaJavaprogramto
generateallpairscombinationsathttp://fit.c2.com/wiki.cgi?AllPairs.
JamesBachpresentsanalgorithmtogenerateallpairsinLessonsLearnedinSoftware
Testing.Inaddition,heprovidesaprogramcalled"Allpairs"thatwillgeneratetheallpairs
combinations.Itisavailableathttp://www.satisfice.com.Clickon"TestMethodology"andlook
forAllpairs.Let'sapplytheAllpairsalgorithmtothepreviousWebsitetestingproblem.
Afterdownloadingandunzipping,touseAllpairscreateatab-delimitedtableofthevariables
andtheirvalues.IfyouareaWindowsuser,theeasiestwayistolaunchExcel,enterthedata
intothespreadsheet,andthenSaveAsa.txtfile.Thefollowingtablewascreatedandsavedas
input.txt.
Table6-11:InputtotheAllpairsprogram.
Browser ClientOS Plug-in Server ServerOS
IE5.0 Win95 None IIS WinNT
IE5.5 Win98 RealPlayer Apache Win2000
IE6.0 WinME MediaPlayer WebLogic Linux
Netscape6.0 WinNT
Netscape6.1 Win2000
Netscape7.0 WinXP
Mozilla1.1
Opera7
ThenruntheAllpairsprogrambytyping:allpairsinput.txt>output.txtwhereoutput.txtwill
containthelistofallpairstestcases.Thefollowingtablewascreated:
Table6-12:OutputfromtheAllpairsprogram.
Browser ClientOS Plug-in Server ServerOS
1 IE5.0 Win95 None IIS WinNT
2 IE5.0 Win98 RealPlayer Apache Win2000
3 IE5.0 WinME MediaPlayer WebLogic Linux
4 IE5.5 Win95 RealPlayer WebLogic WinNT
5 IE5.5 Win98 None IIS Linux
6 IE5.5 WinME None Apache Win2000
7 IE6.0 Win95 MediaPlayer Apache Linux
8 IE6.0 Win98 RealPlayer IIS WinNT
9 IE6.0 WinME None WebLogic Win2000
10 Netscape6.0 WinME RealPlayer IIS Linux
11 Netscape6.0 WinNT MediaPlayer IIS Win2000
12 Netscape6.0 Win2000 None Apache WinNT
13 Netscape6.1 WinNT None WebLogic Linux
14 Netscape6.1 Win2000 MediaPlayer IIS Win2000
15 Netscape6.1 WinXP RealPlayer Apache WinNT
16 Netscape7.0 WinNT RealPlayer Apache WinNT
17 Netscape7.0 Win2000 MediaPlayer WebLogic Linux
18 Netscape7.0 WinXP MediaPlayer IIS Win2000
19 Mozilla1.1 WinXP MediaPlayer WebLogic WinNT
20 Mozilla1.1 Win98 MediaPlayer Apache Linux
21 Mozilla1.1 Win95 RealPlayer IIS Win2000
22 Opera7 WinXP None WebLogic Linux
23 Opera7 Win98 RealPlayer WebLogic Win2000
24 Opera7 WinME MediaPlayer Apache WinNT
25 IE5.5 Win2000 RealPlayer ~WebLogic ~Linux
26 IE5.5 WinNT MediaPlayer ~IIS ~WinNT
27 Netscape6.0 Win95 ~None WebLogic ~Win2000
28 Netscape7.0 Win95 None ~Apache ~Linux
29 Mozilla1.1 WinME None ~IIS ~WinNT
30 Opera7 WinNT ~RealPlayer IIS ~Linux
31 IE5.0 WinNT ~None ~Apache ~Win2000
32 IE5.0 Win2000 ~RealPlayer ~IIS ~WinNT
33 IE5.0 WinXP ~None ~WebLogic ~Linux
34 IE5.5 WinXP ~RealPlayer ~Apache ~Win2000
35 IE6.0 Win2000 ~None ~Apache ~Win2000
36 IE6.0 WinNT ~RealPlayer ~WebLogic ~WinNT
37 IE6,0 WinXP ~MediaPlayer ~IIS ~Linux
38 Netscape6.0 Win98 ~MediaPlayer ~WebLogic ~WinNT
39 Netscape6.0 WinXP ~RealPlayer ~Apache ~Linux
40 Netscape6.1 Win95 ~MediaPlayer ~Apache ~Win2000
41 Netscape6.1 Win98 ~None ~IIS ~WinNT
42 Netscape6.1 WinME ~RealPlayer ~WebLogic ~Linux
43 Netscape7.0 Win98 ~None ~WebLogic ~Win2000
44 Netscape7.0 WinME ~RealPlayer -US ~WinNT
45 Mozilla1.1 WinNT ~None ~Apache ~Linux
46 Mozilla1.1 Win2000 ~RealPlayer ~WebLogic ~Win2000
47 Opera7 Win95 ~MediaPlayer ~IIS ~WinNT
48 Opera7 Win2000 ~None ~Apache ~Win2000
Whenaparticularvalueinthetestcasedoesn'tmatter,becauseallofitspairingshavealready
beenselected,itismarkedwitha~.Bach'salgorithmchoosesthevaluethathasbeenpaired
thefewesttimesrelativetotheothersinthetestcase.Anyothervaluecouldbesubstitutedfor
oneprefixedwitha~andallpairscoveragewouldstillbemaintained.Thismightbedoneto
testmorecommonlyusedormorecriticalcombinationsmoreoften.Inaddition,Bach'sprogram
displaysinformationonhowthepairingsweredone.Itlistseachpair,showshowmanytimes
thatpairoccursinthetable,andindicateseachtestcasethatcontainsthatpair.
Becauseofthe"balanced"natureoforthogonalarrays,thatapproachrequiredsixty-fourtest
cases.The"unbalanced"natureoftheallpairsselectionalgorithmrequiresonlyforty-eighttest
cases,asavingsof25percent.
NotethatthecombinationschosenbytheOrthogonalArraymethodmaynotbethesameas
thosechosenbyAllpairs.Itdoesnotmatter.Whatdoesmatteristhatallofthepair
combinationsofparametersarechosen.Thosearethecombinationswewanttotest.
ProponentsoftheAllpairsalgorithmpointoutthatgivenaproblemwith100parameters,each
capableoftakingononeoftwovalues,101testcaseswouldberequiredusinga(balanced)
orthogonalarraywhiletheun-balancedallpairsapproachrequiresonlytentests.Sincemany
applicationshavelargenumbersofinputsthattakeononlyafewvalueseach,theyarguethe
allpairsapproachissuperior.
Tool
TheAETGtoolfromTelcordiaimplementstheall-pairstestingapproach.See
http://aetgweb.argreenhouse.com.
FinalComments
Insomesituations,constraintsexistbetweencertainchoicesofsomeofthevariables.For
example,Microsoft'sIISandApple'sMacOSarenotcompatible.Itiscertainthatthepairwise
techniqueswillchoosethatcombinationfortest.(Remember,itdoesselectallthepairs.)When
creatingpairwisesubsetsbyhand,honoringthesevariousconstraintscanbedifficult.Boththe
rdExpertandAETGtoolshavethisability.Youdefinetheconstraintsandthetoolselectspairs
meetingthoseconstraints.
Giventhetwoapproachestopairwisetesting,orthogonalarraysandtheAllpairsalgorithm,
whichismoreeffective?Oneexpert,whofavorsorthogonalarrays,believesthatthecoverage
providedbyAllpairsissubstantiallyinferior.Henotesthattheuniformdistributionoftestpoints
inthedomainofferssomecoverageagainstfaultsthataremorecomplexthandouble-mode
faults.Anotherexpert,whofavorstheAllpairsapproach,notesthatAllpairsdoes,infact,test
allthepairs,whichisthegoal.Heclaimsthereisnoevidencethattheorthogonalarray
approachdetectsmoredefects.HealsonotesthattheAllpairstoolisavailablefreeonthe
Web.Whatbothexpertsacknowledgeisthatnodocumentedstudiesexistcomparingthe
efficacyofoneapproachovertheother.
Theexcitinghopeofpairwisetestingisthatbycreatingandrunningbetween1percentto20
percentofthetestsyouwillfindbetween70percentand85percentofthetotaldefects.There
isnopromisehere,onlyahope.Manyothershaveexperiencedthissignificantresult.Trythis
technique.Discoverwhetheritworksforyou.
Cohenreportedthatinadditiontoreducingthenumberoftestcasesandincreasingthedefect
findrate,testcasescreatedbytheAllpairsalgorithmalsoprovidedbettercodecoverage.A
setof300randomlyselectedtestsachieved67percentstatementcoverageand58percent
decisioncoveragewhilethe200allpairstestcasesachieved92percentblockcoverageand
85percentdecisioncoverage,asignificantincreaseincoveragewithfewertestcases.
Onefinalcomment—itispossiblethatcertainimportantcombinationsmaybemissedbyboth
pairwiseapproaches.The80:20ruletellsusthatcombinationsarenotuniformlyimportant.Use
yourjudgmenttodetermineifcertainadditionaltestsshouldbecreatedforthosecombinations.
Inthepreviousexamplewecanbeassuredthatthedistributionofbrowsersisnotidentical.It
wouldbetrulyamazingif12.5percentofourusershadIE5.0,12.5percenthadIE5.5,12.5
percenthadIE6.0,etc.Certaincombinationsoccurmorefrequentlythanothers.Inaddition,
somecombinationsexistthat,whileusedinfrequently,absolutelypositivelymustworkproperly
—"shutdownthenuclearreactor"isagoodexample.Incasepairwisemissesanimportant
combination,pleaseaddthatcombinationtoyourtestcases.
ApplicabilityandLimitations
Likeothertestdesignapproachespreviouslypresented,pairwisetestingcansignificantly
reducethenumberoftestcasesthatmustbecreatedandexecuted.Itisequallyapplicableat
theunit,integration,system,andacceptancetestlevels.Allitrequiresarecombinationsof
inputs,eachtakingonvariousvalues,thatresultinacombinatorialexplosion,toomany
combinationstotest.
Remember,thereisnounderlying"softwaredefectphysics"thatguaranteespairwisetesting
willbeofbenefit.Thereisonlyonewaytoknow—tryit.
Summary
Whenthenumberofcombinationstotestisverylarge,donottoattempttotestall
combinationsforallthevaluesforallthevariables,buttestallpairsofvariables.This
significantlyreducesthenumberofteststhatmustbecreatedandrun.
Studiessuggestthatmostdefectsareeithersingle-modedefects(thefunctionunder
testsimplydoesnotwork)ordouble-modedefects(thepairingofthisfunction/module
withthatfunction/modulefails).Pairwisetestingdefinesaminimalsubsetthatguidesus
totestforallsingle-modeanddouble-modedefects.Thesuccessofthistechniqueon
manyprojects,bothdocumentedandundocumented,isagreatmotivationforitsuse.
Anorthogonalarrayisatwo-dimensionalarrayofnumbersthathasthisinteresting
property—chooseanytwocolumnsinthearray,allthecombinationswilloccurinevery
columnpair.
Thereisnounderlying"softwaredefectphysics"thatguaranteespairwisetestingwill
beofbenefit.Thereisonlyonewaytoknow—tryit.
Practice
1. NeithertheBrown&DonaldsonnortheStatelessUniversityRegistrationSystemcase
studiescontainhugenumbersofcombinationssuitableforthepairwisetesting
approach.Asexercises,usetheorthogonalarrayand/orallpairstechniqueonthe
othertwoexamplesinthischapter.Determinethesetofpairwisetestcasesusingthe
chosentechnique.
1. Abankhascreatedanewdataprocessingsystemthatisreadyfortesting.
Thisbankhasdifferentkindsofcustomers—consumers,veryimportant
consumers,businesses,andnon-profits;differentkindsofaccounts—
checking,savings,mortgages,consumerloans,andcommercialloans;they
operateindifferentstates,eachwithdifferentregulations—California,
Nevada,Utah,Idaho,Arizona,andNewMexico.
2. Inanobject-orientedsystem,anobjectofclassAcansendamessage
containingaparameterPtoanobjectofclassX.ClassesB,C,andD
inheritfromAsotheytoocansendthemessage.ClassesQ,R,S,andT
inheritfromPsotheytoocanbepassedastheparameter.ClassesYand
ZinheritfromXsotheytoocanreceivethemessage.
References
Brownlie,Robert,etal."RobustTestingofAT&TPMX/StarMAILUsingOATS,"AT&T
TechnicalJournal,Vol.71,No.3,May/June1992,pp.41–47.
Cohen,D.M.,etal."TheAETGSystem:AnApproachtoTestingBasedonCombinatorial
Design."IEEETransactionsonSoftwareEngineering,Vol.23,No.7,July,1997.
Kaner,Cem,JamesBach,andBretPettichord(2002).LessonsLearnedinSoftware
Testing:AContext-DrivenApproach.JohnWiley&Sons.
Kuhn,D.RichardandMichaelJ.Reilly."AnInvestigationoftheApplicabilityofDesignof
ExperimentstoSoftwareTesting,"27thNASA/IEEESoftwareEngineeringWorkshop,
NASAGoddardSpaceFlightCenter,4–6December,2002.
http://csrc.nist.gov/staff/kuhn/kuhn-reilly-02.pdf
Mandl,Robert."OrthogonalLatinSquares:AnApplicationofExperimentDesignto
CompilerTesting,"CommunicationsoftheACM,Vol.128,No.10,October1985,pp.
1054–1058.
Phadke,MadhavS.(1989).QualityEngineeringUsingRobustDesign.Prentice-Hall.
Wallace,DeloresR.andD.RichardKuhn."FailureModesInMedicalDeviceSoftware:
AnAnalysisOf15YearsOfRecallData,"InternationalJournalofReliability,Quality,
andSafetyEngineering,Vol.8,No.4,2001.
Chapter7:State-TransitionTesting
ColonelCleatusYorbvillehadbeenoneseriouslyboredastronautforthefirstfewmonths
ofhisdiplomaticmissiononthethirdplanetoftheFrangelicusXIVsystem,butallthat
hadchangedonthedayhe'ddiscoveredthathistiny,multipedalandinfinitelyhospitable
alienhostswerenotonlyediblebuttastedremarkablylikethatstuffthat'sleftonthepan
afteryou'vemadecinnamonbunsandburnedthemalittle.
—MarkSilcox
Introduction
State-Transitiondiagrams,likedecisiontables,areanotherexcellenttooltocapturecertain
typesofsystemrequirementsandtodocumentinternalsystemdesign.Thesediagrams
documenttheeventsthatcomeintoandareprocessedbyasystemaswellasthesystem's
responses.Unlikedecisiontables,theyspecifyverylittleintermsofprocessingrules.Whena
systemmustremembersomethingaboutwhathashappenedbeforeorwhenvalidandinvalid
ordersofoperationsexist,state-transitiondiagramsareexcellenttoolstorecordthis
information.
Thesediagramsarealsovitaltoolsinthetester'spersonaltoolbox.Unfortunately,many
analysts,designers,programmers,andtestersarenotfamiliarwiththistechnique.
Technique
State-TransitionDiagrams
Itiseasiertointroducestate-transitiondiagramsbyexampleratherthanbyformaldefinition.
SinceneitherBrown&DonaldsonnortheStatelessUniversityRegistrationSystemhas
substantialstate-transitionbasedrequirementslet'sconsideradifferentexample.Togetto
StatelessU,weneedanairlinereservation.Let'scallourfavoritecarrier(GraceL.Ferguson
Airline&StormDoorCompany)tomakeareservation.Weprovidesomeinformationincluding
departureanddestinationcities,dates,andtimes.Areservationagent,actingasourinterface
totheairline'sreservationsystem,usesthatinformationtomakeareservation.Atthatpoint,
theReservationisintheMadestate.Inaddition,thesystemcreatesandstartsatimer.Each
reservationhascertainrulesaboutwhenthereservationmustbepaidfor.Theserulesare
basedondestination,classofservice,dates,etc.Ifthistimerexpiresbeforethereservationis
paidfor,thereservationiscancelledbythesystem.Instate-transitionnotationthisinformation
isrecordedas:
Figure7-1:TheReservationisMade.
ThecirclerepresentsonestateoftheReservation—inthiscasetheMadestate.Thearrow
showsthetransitionintotheMadestate.Thedescriptiononthearrow,giveInfo,isanevent
thatcomesintothesystemfromtheoutsideworld.Thecommandafterthe"/"denotesan
actionofthesystem;inthiscasestartPayTimer.Theblackdotindicatesthestartingpointof
thediagram.
SometimeaftertheReservationismade,but(hopefully)beforethePayTimerexpires,the
Reservationispaidfor.ThisisrepresentedbythearrowlabeledPayMoney.Whenthe
ReservationispaidittransitionsfromtheMadestatetothePaidstate.
Figure7-2:TheReservationtransitionstothePaidstate.
Beforeweproceedlet'sdefinethetermsmoreformally:
State(representedbyacircle)—Astateisaconditioninwhichasystemiswaitingfor
oneormoreevents.States"remember"inputsthesystemhasreceivedinthepastand
definehowthesystemshouldrespondtosubsequenteventswhentheyoccur.These
eventsmaycausestate-transitionsand/orinitiateactions.Thestateisgenerally
representedbythevaluesofoneormorevariableswithinasystem.
Transition(representedbyanarrow)—Atransitionrepresentsachangefromonestate
toanothercausedbyanevent.
Event(representedbyalabelonatransition)—Aneventissomethingthatcausesthe
systemtochangestate.Generally,itisaneventintheoutsideworldthatentersthe
systemthroughitsinterface.Sometimesitisgeneratedwithinthesystemsuchas
TimerexpiresorQuantityonHandgoesbelowReorderPoint.Eventsare
consideredtobeinstantaneous.Eventscanbeindependentorcausallyrelated(event
BcannottakeplacebeforeeventA).Whenaneventoccurs,thesystemcanchange
stateorremaininthesamestateand/orexecuteanaction.Eventsmayhave
parametersassociatedwiththem.Forexample,PayMoneymayindicateCash,
Check,DebitCard,orCreditCard.
Action(representedbyacommandfollowinga"/")—Anactionisanoperationinitiated
becauseofastatechange.ItcouldbeprintaTicket,displayaScreen,turnona
Motor,etc.Oftentheseactionscausesomethingtobecreatedthatareoutputsofthe
system.Notethatactionsoccurontransitionsbetweenstates.Thestatesthemselves
arepassive.
Theentrypointonthediagramisshownbyablackdotwhiletheexitpointisshownby
abulls-eyesymbol.
ThisnotationwascreatedbyMealy.AnalternatenotationhasbeendefinedbyMoorebutis
lessfrequentlyused.Foramuchmorein-depthdiscussionofstate-transitiondiagramssee
FowlerandScott'sbook,UMLDistilled:ABriefGuideToTheStandardObjectModeling
Language.Itdiscussesmorecomplexissuessuchaspartitionedandnestedstate-transition
diagrams.
Notethatthestate-transitiondiagramrepresentsonespecificentity(inthiscasea
Reservation).Itdescribesthestatesofareservation,theeventsthataffectthereservation,
thetransitionsofthereservationfromonestatetoanother,andactionsthatareinitiatedbythe
reservation.Acommonmistakeistomixdifferententitiesintoonestate-transitiondiagram.An
examplemightbemixingReservationandPassengerwitheventsandactionscorresponding
toeach.
FromthePaidstatetheReservationtransitionstotheTicketedstatewhentheprint
command(anevent)isissued.NotethatinadditiontoenteringtheTicketedstate,aTicketis
outputbythesystem.
Figure7-3:TheReservationtransitionstotheTicketedstate.
FromtheTicketedstatewegiveTickettothegateagenttoboardtheplane.
Figure7-4:TheReservationtransitionstotheUsedstate.
Aftersomeotheractionorperiodoftime,notindicatedonthisdiagram,thestate-transition
pathendsatthebulls-eyesymbol.
Figure7-5:Thepathends.
Doesthisdiagramshowallthepossiblestates,events,andtransitionsinthelifeofa
Reservation?No.IftheReservationisnotpaidforinthetimeallotted(thePayTimer
expires),itiscancelledfornon-payment.
Figure7-6:ThePayTimerexpiresandtheReservationiscancelledfornonpayment.
Finishedyet?No.Customerssometimescanceltheirreservations.FromtheMadestatethe
customer(throughthereservationagent)askstocanceltheReservation.Anewstate,
CancelledByCustomer,isrequired.
Figure7-7:CanceltheReservationfromtheMadestate.
Inaddition,aReservationcanbecancelledfromthePaidstate.InthiscaseaRefundshould
begeneratedandleavethesystem.TheresultingstateagainisCancelledByCustomer.
Figure7-8:CancellationfromthePaidstate.
Onefinaladdition.FromtheTicketedstatethecustomercancanceltheReservation.Inthat
caseaRefundshouldbegeneratedandthenextstateshouldbeCancelledbyCustomer.
Butthisisnotsufficient.Theairlinewillgeneratearefundbutonlywhenitreceivestheprinted
Ticketfromthecustomer.Thisintroducesonenewnotationalelement—squarebrackets[]that
containaconditionalthatcanbeevaluatedeitherTrueorFalse.Thisconditionalactsasa
guardallowingthetransitiononlyiftheconditionistrue.
Figure7-9:CancellationfromtheTicketedstate.
Notethatthediagramisstillincomplete.Noarrowsandbulls-eyesemergefromtheCancelled
states.PerhapswecouldreinstateareservationfromtheCancelledNonPaystate.Wecould
continueexpandingthediagramtoincludeseatselection,flightcancellation,andother
significanteventsaffectingthereservationbutthisissufficienttoillustratethetechnique.
Asdescribed,state-transitiondiagramsexpresscomplexsystemrulesandinteractionsina
verycompactnotation.Hopefully,whenthiscomplexityexists,analystsanddesignerswillhave
createdstate-transitiondiagramstodocumentsystemrequirementsandtoguidetheirdesign.
State-TransitionTables
Astate-transitiondiagramisnottheonlywaytodocumentsystembehavior.Thediagramsmay
beeasiertocomprehend,butstate-transitiontablesmaybeeasiertouseinacompleteand
systematicmanner.State-transitiontablesconsistoffourcolumns—CurrentState,Event,
Action,andNextState.
Table7-1:State-TransitiontableforReservation.
CurrentState Event Action NextState
null giveInfo startPayTimer Made
null payMoney -- null
null print -- null
null giveTicket -- null
null cancel -- null
null PayTimerExpires -- null
Made giveInfo -- Made
Made payMoney -- Paid
Made print -- Made
Made giveTicket -- Made
Made cancel -- Can-Cust
Made PayTimerExpires -- Can-NonPay
Paid giveInfo -- Paid
Paid payMoney -- Paid
Paid print Ticket Ticketed
Paid giveTicket -- Paid
Paid cancel Refund Can-Cust
Paid PayTimerExpires -- Paid
Ticketed giveInfo -- Ticketed
Ticketed payMoney -- Ticketed
Ticketed print -- Ticketed
Ticketed giveTicket -- Used
Ticketed cancel Refund Can-Cust
Ticketed PayTimerExpires -- Ticketed
Used giveInfo -- Used
Used payMoney -- Used
Used print -- Used
Used giveTicket -- Used
Used cancel -- Used
Used PayTimerExpires -- Used
Can-NonPay giveInfo -- Can-NonPay
Can-NonPay payMoney -- Can-NonPay
Can-NonPay print -- Can-NonPay
Can-NonPay giveTicket -- Can-NonPay
Can-NonPay cancel -- Can-NonPay
Can-NonPay PayTimerExpires -- Can-NonPay
Can-Cust givelnfo -- Can-Cust
Can-Cust payMoney -- Can-Cust
Can-Cust print -- Can-Cust
Can-Cust giveTicket -- Can-Cust
Can-Cust cancel -- Can-Cust
Can-Cust PayTimerExpires -- Can-Cust
Theadvantageofastate-transitiontableisthatitlistsallpossiblestate-transitioncombinations,
notjustthevalidones.Whentestingcritical,high-risksystemssuchasavionicsormedical
devices,testingeverystate-transitionpairmayberequired,includingthosethatarenotvalid.In
addition,creatingastate-transitiontableoftenunearthscombinationsthatwerenotidentified,
documented,ordealtwithintherequirements.Itishighlybeneficialtodiscoverthesedefects
beforecodingbegins.
Key
Point
Theadvantageofastate-transitiontableisthatitlistsallpossiblestate-transition
combinations,notjustthevalidones.
Usingastate-transitiontablecanhelpdetectdefectsinimplementationthatenableinvalidpaths
fromonestatetoanother.Thedisadvantageofsuchtablesisthattheybecomeverylargevery
quicklyasthenumberofstatesandeventsincreases.Inaddition,thetablesaregenerally
sparse;thatis,mostofthecellsareempty.
CreatingTestCases
Informationinthestate-transitiondiagramscaneasilybeusedtocreatetestcases.Four
differentlevelsofcoveragecanbedefined:
1. Createasetoftestcasessuchthatallstatesare"visited"atleastonceundertest.
Thesetofthreetestcasesshownbelowmeetsthisrequirement.Generallythisisa
weakleveloftestcoverage.
Figure7-10:Asetoftestcasesthat"visit"eachstate.
2. Createasetoftestcasessuchthatalleventsaretriggeredatleastonceundertest.
Notethatthetestcasesthatcovereacheventcanbethesameasthosethatcover
eachstate.Again,thisisaweaklevelofcoverage.
Figure7-11:Asetoftestcasesthattriggeralleventsatleastonce.
3. Createasetoftestcasessuchthatallpathsareexecutedatleastonceundertest.
Whilethislevelisthemostpreferredbecauseofitslevelofcoverage,itmaynotbe
feasible.Ifthestate-transitiondiagramhasloops,thenthenumberofpossiblepaths
maybeinfinite.Forexample,givenasystemwithtwostates,AandB,whereA
transitionstoBandBtransitionstoA.Afewofthepossiblepathsare:A→B
A→B→A
A→B→A→B→A→B
A→B→A→B→A→B→AA→B→A→B→A→B→A→B→A→B
...
andsoonforever.Testingofloopssuchasthiscanbeimportantiftheymayresultin
accumulatingcomputationalerrorsorresourceloss(lockswithoutcorresponding
releases,memoryleaks,etc.).
Key
Point
Testingeverytransitionisusuallytherecommendedlevelofcoverage
forastate-transitiondiagram.
4. Createasetoftestcasessuchthatalltransitionsareexercisedatleastonceunder
test.Thisleveloftestingprovidesagoodlevelofcoveragewithoutgeneratinglarge
numbersoftests.Thislevelisgenerallytheonerecommended.
Figure7-12:Asetoftestcasesthattriggeralltransitionsatleastonce.
Testcasescanalsobereaddirectlyfromthestate-transitiontable.Thegrayrowsinthe
followingtableshowallthevalidtransitions.
Table7-2:TestingallvalidtransitionsfromaState-transitiontable.
CurrentState Event Action NextState
null giveInfo startPayTimer Made
null payMoney -- null
null print -- null
null giveTicket -- null
null cancel -- null
null PayTimerExpires -- null
Made giveInfo -- Made
Made payMoney -- Paid
Made print -- Made
Made giveTicket -- Made
Made cancel -- Can-Cust
Made PayTimerExpires -- Can-NonPay
Paid giveInfo -- Paid
Paid payMoney -- Paid
Paid print Ticket Ticketed
Paid giveTicket -- Paid
Paid cancel Refund Can-Cust
Paid PayTimerExpires -- Paid
Ticketed giveInfo -- Ticketed
Ticketed payMoney -- Ticketed
Ticketed print -- Ticketed
Ticketed giveTicket -- Used
Ticketed cancel Refund Can-Cust
Ticketed PayTimerExpires -- Ticketed
Used giveInfo -- Used
Used payMoney -- Used
Used print -- Used
Used giveTicket -- Used
Used cancel -- Used
Used PayTimerExpires -- Used
Can-NonPay giveInfo -- Can-NonPay
Can-NonPay payMoney -- Can-NonPay
Can-NonPay print -- Can-NonPay
Can-NonPay giveTicket -- Can-NonPay
Can-NonPay cancel -- Can-NonPay
Can-NonPay PayTimerExpires -- Can-NonPay
Can-Cust givelnfo -- Can-Cust
Can-Cust payMoney -- Can-Cust
Can-Cust print -- Can-Cust
Can-Cust giveTicket -- Can-Cust
Can-Cust cancel -- Can-Cust
Can-Cust PayTimerExpires -- Can-Cust
Inaddition,dependingonthesystemrisk,youmaywanttocreatetestcasesforsomeorallof
theinvalidstate/eventpairstomakesurethesystemhasnotimplementedinvalidpaths.
ApplicabilityandLimitations
State-Transitiondiagramsareexcellenttoolstocapturecertainsystemrequirements,namely
thosethatdescribestatesandtheirassociatedtransitions.Thesediagramsthencanbeused
todirectourtestingeffortsbyidentifyingthestates,events,andtransitionsthatshouldbe
tested.
State-Transitiondiagramsarenotapplicablewhenthesystemhasnostateordoesnotneedto
respondtoreal-timeeventsfromoutsideofthesystem.Anexampleisapayrollprogramthat
readsanemployee'stimerecord,computespay,subtractsdeductions,savestherecord,prints
apaycheck,andrepeatstheprocess.
Summary
State-Transitiondiagramsdirectourtestingeffortsbyidentifyingthestates,events,
actions,andtransitionsthatshouldbetested.Together,thesedefinehowasystem
interactswiththeoutsideworld,theeventsitprocesses,andthevalidandinvalidorder
oftheseevents.
Astate-transitiondiagramisnottheonlywaytodocumentsystembehavior.Theymay
beeasiertocomprehend,butstate-transitiontablesmaybeeasiertouseina
completeandsystematicmanner.
Thegenerallyrecommendedleveloftestingusingstate-transitiondiagramsistocreate
asetoftestcasessuchthatalltransitionsareexercisedatleastonceundertest.In
high-risksystems,youmaywanttocreateevenmoretestcases,approachingallpaths
ifpossible.
Practice
1. ThisexercisereferstotheStatelessUniversityRegistrationSystemWebsite
describedinAppendixB.Belowisastate-transitiondiagramforthe"enrollina
course"and"dropacourse"process.Determineasetoftestcasesthatyoufeel
adequatelycovertheenrollanddropprocess.
Thefollowingtermsareusedinthediagram:Events
create-Createanewcourse.
enroll-Addastudenttothecourse.
drop-Dropastudentfromthecourse.
Attributes
ID-Thestudentidentificationnumber.
max-Themaximumnumberofstudentsacoursecanhold.
#enrolled-Thenumberofstudentscurrentlyenrolledinthecourse.
#waiting-ThenumberofstudentscurrentlyontheWaitListforthiscourse.
Tests
isEnrolled-Answers"isthestudentenrolled(ontheSectionList)?"
onWaitList-Answers"isthestudentontheWaitList?"
Lists
SectionList-Alistofstudentsenrolledintheclass.
WaitList-Alistofstudentswaitingtobeenrolledinafullclass.
Symbols
++Incrementby1.
--Decrementby1.
Figure7-13:State-transitiondiagramforenrollanddropacourseatStatelessU.
References
Binder,RobertV.(1999).TestingObject-OrientedSystems:Models,Patterns,and
Tools.Addison-Wesley.
Fowler,MartinandKendallScott(1999).UMLDistilled:ABriefGuidetotheStandard
ObjectModelingLanguage(2ndEdition).Addison-Wesley.
Harel,David."Statecharts:avisualformalismforcomplexsystems."Scienceof
ComputerProgramming8,1987,pp231–274.
Mealy,G.H."Amethodforsynthesizingsequentialcircuits."BellSystemTechnical
Journal,34(5):1045–1079,1955.
Moore,E.F."Gedanken-experimentsonsequentialmachines,"AutomataStudies(C.E.
ShannonandJ.McCarthy,eds.),pp.129–153,Princeton,NewJersey:Princeton
UniversityPress,1956.
Rumbaugh,James,etal.(1991).Object-OrientedModelingandDesign.Prentice-Hall.
Chapter8:DomainAnalysisTesting
StandingintheconcessionscaroftheOrientExpressasithissedandlurchedawayfrom
thestation,SpecialAgentChucouldfeelenemyeyeswatchinghimfromtheinky
shadowsandknewthathewasbeingtested,foralthoughhehadnevertastedaplugof
tobaccoinhislife,hewasimpersonatinganarmsdealerknowntobeaconnoisseur,so
heknewthathe,theChosenOne,ChowChu,hadnochoicebuttochoosethechoicest
chewonthechoo-choo.
—LorenHaarsma
Introduction
InthechaptersonEquivalenceClassandBoundaryValuetesting,weconsideredthetestingof
individualvariablesthattookonvalueswithinspecifiedranges.Inthischapterwewillconsider
thetestingofmultiplevariablessimultaneously.Therearetworeasonstoconsiderthis:
Werarelywillhavetimetocreatetestcasesforeveryvariableinoursystems.There
aresimplytoomany.
Oftenvariablesinteract.Thevalueofonevariableconstrainstheacceptablevaluesof
another.Inthiscase,certaindefectscannotbediscoveredbytestingthemindividually.
Domainanalysisisatechniquethatcanbeusedtoidentifyefficientandeffectivetestcases
whenmultiplevariablescanorshouldbetestedtogether.Itbuildsonandgeneralizes
equivalenceclassandboundaryvaluetestingtonsimultaneousdimensions.Likethose
techniques,wearesearchingforsituationswheretheboundaryhasbeendefinedor
implementedincorrectly.
Key
Point
Domainanalysisisatechniquethatcanbeusedtoidentifyefficientandeffective
testcaseswhenmultiplevariablesshouldbetestedtogether.
Intwodimensions(withtwointeractingparameters)thefollowingdefectscanoccur:
Ashiftedboundaryinwhichtheboundaryisdisplacedverticallyorhorizontally
Atiltedboundaryinwhichtheboundaryisrotatedatanincorrectangle
Amissingboundary
Anextraboundary
Figure8-1isadaptedfromBinder.Itillustratesthesefourtypesofdefectsgraphically.
Figure8-1:Twodimensionalboundarydefects.
Certainlytherecanbeinteractionsbetweenthreeormorevariables,butthediagramsaremore
difficulttovisualize.
Technique
Thedomainanalysisprocessguidesusinchoosingefficientandeffectivetestcases.First,a
numberofdefinitions:
Anonpointisavaluethatliesonaboundary.
Anoffpointisavaluethatdoesnotlieonaboundary.
Aninpointisavaluethatsatisfiesalltheboundaryconditionsbutdoesnotlieona
boundary.
Anoutpointisavaluethatdoesnotsatisfyanyboundarycondition.
Choosingonandoffpointsismorecomplicatedthatitmayappear.
Whentheboundaryisclosed(definedbyanoperatorcontaininganequality,i.e.,≤,≥or
=)sothatpointsontheboundaryareincludedinthedomain,thenanonpointlieson
theboundaryandisincludedwithinthedomain.Anoffpointliesoutsidethedomain.
Whentheboundaryisopen(definedbyaninequalityoperator<or>)sothatpointson
theboundaryarenotincludedinthedomain,thenanonpointliesontheboundarybut
isnotincludedwithinthedomain.Anoffpointliesinsidethedomain.
Confused?Atthispointexamplesarecertainlyinorder.
Figure8-2:Examplesofon,off,in,andoutpointsforbothclosedandopenboundaries.
Ontheleftisanexampleofaclosedboundary.Theregiondefinedconsistsofallthepoints
greaterthanorequalto10.Theonpointhasthevalue10.Theoffpointisslightlyoffthe
boundaryandoutsidethedomain.Theinpointiswithinthedomain.Theoutpointisoutsidethe
domain.
Ontherightisanexampleofanopenboundary.Theregiondefinedconsistsofallthepoints
greaterthan(butnotequalto)10.Again,theonpointhasavalueof10.Theoffpointisslightly
offtheboundaryandinsidethedomain.Theinpointiswithinthedomain.Theoutpointis
outsidethedomain.
Havingdefinedthesepoints,the1x1("one-by-one")domainanalysistechniqueinstructsusto
choosethesetestcases:
Foreachrelationalcondition(≥,>,≤,or<)chooseoneonpointandoneoffpoint.
Foreachstrictequalitycondition(=)chooseoneonpointandtwooffpoints,one
slightlylessthantheconditionalvalueandoneslightlygreaterthanthevalue.
Notethatthereisnoreasontorepeatidenticaltestsforadjacentdomains.Ifanoffpointfor
onedomainistheinpointforanother,donotduplicatethesetests.
Bindersuggestsaveryusefultablefordocumenting1x1domainanalysistestcasescalledthe
DomainTestMatrix.
Table8-1:ExampleDomainTestMatrix.
Notethattestcases1through8testtheonpointsandoffpointsforeachconditionofthefirst
variable(X1)whileholdingthevalueofthesecondvariable(X2)atatypicalinpoint.Testcases
9through16holdthefirstvariableatatypicalinpointwhiletestingtheonandoffpointsfor
eachconditionofthesecondvariable.Additionalvariablesandconditionswouldfollowthe
samepattern.
Example
AdmissiontoStatelessUniversityismadebyconsideringacombinationofhighschoolgrades
andACTtestscores.Theshadedcellsinthefollowingtableindicatethecombinationsthat
wouldguaranteeacceptance.GradePointAverages(GPAs)areshownacrossthetopwhile
ACTscoresareshowndowntheleftside.StatelessUniversityisafairlyexclusiveschoolin
termsofitsadmissionpolicy.
Explanation
TheACTAssessmentisanexaminationdesignedtoassesshighschoolstudents'general
educationaldevelopmentandtheirabilitytocompletecollege-levelwork.
TheGradePointAverageisbasedonconvertinglettergradestonumericvaluesA=4.0
(Best)
B=3.0
C=2.0(Average)
D=1.0
Table8-2:StatelessUniversityAdmissionsMatrix.
Thistablecanberepresentedasthesolutionsetofthesethreelinearequations:ACT≤36(the
highestscorepossible)GPA≤4.0(thehighestvaluepossible)10*GPA+ACT≥71
(Thethirdequationcanbefoundbyusingthegoodoldy=mx+bformulafromelementary
algebra.Usepoints{ACT=36,GPA=3.5}and{ACT=31,GPA=4.0}andcrank—that'smath
slangforsolvethepairofsimultaneousequationsobtainedbysubstitutingeachofthesetwo
pointsintothey=mx+bequation.)
Figure8-3:StatelessUniversityAdmissionsMatrixingraphicalform.
Thefollowingtestcasescoverthesethreeboundariesusingthe1x1domainanalysisprocess.
Table8-3:1x1DomainAnalysistestcasesforStatelessUniversityadmissions.
Testcases1and2verifytheGPA≤4.0constraint.Case1checksontheGPA=4.0boundary
whilecase2checksjustoutsidetheboundarywithGPA=4.1.Bothofthesecasesusetypical
valuesfortheACTandGPA/ACTconstraints.
Testcases3and4verifytheACT≤36constraint.Case3checksontheACT=36boundary
whilecase4checksjustoutsidetheboundarywithACT=37.Bothofthesecasesusetypical
valuesfortheGPAandGPA/ACTconstraints.
Testcases5and6verifythe10*GPA+ACT≥71constraint.Case5checksontheGPA=3.7
andACT=34boundarywhilecase6checksjustoutsidetheboundarywithGPA=3.8andACT
=32.BothofthesecasesusetypicalvaluesfortheGPAandACTconstraints.
ApplicabilityandLimitations
Domainanalysisisapplicablewhenmultiplevariables(suchasinputfields)shouldbetested
togethereitherforefficiencyorbecauseofalogicalinteraction.Whilethistechniqueisbest
suitedtonumericvalues,itcanbegeneralizedtoBooleans,strings,enumerations,etc.
Summary
Domainanalysisfacilitatesthetestingofmultiplevariablessimultaneously.Itisuseful
becausewerarelywillhavetimetocreatetestcasesforeveryvariableinoursystems.
Therearesimplytoomany.Inaddition,oftenvariablesinteract.Whenthevalueofone
variableconstrainstheacceptablevaluesofanother,certaindefectscannotbe
discoveredbytestingthemindividually.
Itbuildsonandgeneralizesequivalenceclassandboundaryvaluetestington
simultaneousdimensions.Likethosetechniques,wearesearchingforsituationswhere
theboundaryhasbeenimplementedincorrectly.
Inusingthe1x1domainanalysistechniqueforeachrelationalcondition(≥,>,≤,or<)
wechooseoneonpointandoneoffpoint.Foreachstrictequalitycondition(=)we
chooseoneonpointandtwooffpoints,oneslightlylessthantheconditionalvalueand
oneslightlygreaterthanthevalue.
Practice
1. StatelessUniversitypridesitselfinpreparingnotjusteducatedstudentsbutgood
citizensoftheirnation.(That'swhattheiradvertisingbrochuresays.)Inadditionto
theirmajorandminorcoursework,StatelessU.requireseachstudenttotake(and
pass)anumberofGeneralEducationclasses.Theseare:
CollegeAlgebra(thestudentmayeithertakethecourseorshowcompetency
throughtesting).
OurNation'sInstitutions—asurveycourseofournation'shistory,government,
andplaceintheworld.
FromfourtosixteenhoursofSocialSciencecourses(numbers100–299).
FromfourtosixteenhoursofPhysicalSciencecourses(numbers100–299)
Nomorethantwenty-fourcombinedhoursofSocialScienceandPhysical
Sciencecoursesmaybecountedtowardgraduation.
Apply1x1domainanalysistotheserequirements,derivethetestcases,
anduseBinder'sDomainTestMatrixtodocumentthem.
References
Beizer,Boris(1990).SoftwareTestingTechniques.VanNostrandReinhold.
Binder,RobertV.(2000).TestingObject-OrientedSystems:Models,Patterns,and
Tools.Addison-Wesley.
Chapter9:UseCaseTesting
TheInsectKeeperGeneral,sittingastridehisgianthoveringaphid,surveyedthe
battlefieldwhichreekedwiththestenchofdecayandresonatedwiththelowdroneofthe
tatteredanddyingmutantswarmsastheirlegskickedforlornlyattheskybeforeturning
tohismasterandsaying,'MyLord,yourfliesareundone.'
—AndrewVincent
Introduction
Upuntilnowwehaveexaminedtestcasedesigntechniquesforpartsofasystem—input
variableswiththeirrangesandboundaries,businessrulesasrepresentedindecisiontables,
andsystembehaviorsasrepresentedinstate-transitiondiagrams.Nowitistimetoconsider
testcasesthatexerciseasystem'sfunctionalitiesfromstarttofinishbytestingeachofits
individualtransactions.
Definingthetransactionsthatasystemprocessesisavitalpartoftherequirementsdefinition
process.Variousapproachestodocumentingthesetransactionshavebeenusedinthepast.
Examplesincludeflowcharts,HIPOdiagrams,andtext.Today,themostpopularapproachis
theusecasediagram.Likedecisiontablesandstate-transitiondiagrams,usecasesareusually
createdbydevelopersfordevelopers.But,liketheseothertechniques,usecasesholda
wealthofinformationusefultotesters.
UsecaseswerecreatedbyIvarJacobsenandpopularizedinhisbookObject-Oriented
SoftwareEngineering:AUseCaseDrivenApproach.Jacobsendefinesa"usecase"asa
scenariothatdescribestheuseofasystembyanactortoaccomplishaspecificgoal.By
"actor"wemeanauser,playingarolewithrespecttothesystem,seekingtousethesystemto
accomplishsomethingworthwhilewithinaparticularcontext.Actorsaregenerallypeople
althoughothersystemsmayalsobeactors.A"scenario"isasequenceofstepsthatdescribe
theinteractionsbetweentheactorandthesystem.Notethattheusecaseisdefinedfromthe
perspectiveoftheuser,notthesystem.Notealsothattheinternalworkingsofthesystem,
whilevital,arenotpartoftheusecasedefinition.Thesetofusecasesmakesupthefunctional
requirementsofasystem.
TheUnifiedModelingLanguagenotionforusecasesis:
Figure9-1:SomeStatelessUniversityusecases.
Thestickfiguresrepresenttheactors,theellipsesrepresenttheusecases,andthearrows
showwhichactorsinitiatewhichusecases.
Itisimportanttonotethatwhileusecaseswerecreatedinthecontextofobject-oriented
systemsdevelopment,theyareequallyusefulindefiningfunctionalrequirementsinother
developmentparadigmsaswell.
Thevalueofusecasesisthatthey:
Capturethesystem'sfunctionalrequirementsfromtheuser'sperspective;notfroma
technicalperspective,andirrespectiveofthedevelopmentparadigmtobeused.
Canbeusedtoactivelyinvolveusersintherequirementsgatheringanddefinition
process.
Providethebasisforidentifyingasystem'skeyinternalcomponents,structures,
databases,andrelationships.
Serveasthefoundationfordevelopingtestcasesatthesystemandacceptancelevel.
Technique
Unfortunately,thelevelofdetailspecifiedintheusecasesisnotsufficient,eitherfordevelopers
ortesters.InhisbookWritingEffectiveUseCases,AlistairCockburnhasproposedadetailed
templatefordescribingusecases.Thefollowingisadaptedfromhiswork:
Table9-1:Usecasetemplate.
UseCase
Component Description
UseCaseNumber
orIdentifier Auniqueidentifierforthisusecase
UseCaseName Thenameshouldbethegoalstatedasashortactiveverbphrase
GoalinContext Amoredetailedstatementofthegoalifnecessary
Scope Corporate|System|Subsystem
Level Summary|Primarytask|Subfunction
PrimaryActor Rolenameordescriptionoftheprimaryactor
Preconditions Therequiredstateofthesystembeforetheusecaseistriggered
SuccessEnd
Conditions Thestateofthesystemuponsuccessfulcompletionofthisusecase
FailedEnd
Conditions Thestateofthesystemiftheusecasecannotexecutetocompletion
Trigger Theactionthatinitiatestheexecutionoftheusecase
MainSuccess
Scenario
Step Action
1
2
...
Extensions Conditionsunderwhichthemainsuccessscenariowillvaryanda
descriptionofthosevariations
Sub-Variations Variationsthatdonotaffectthemainflowbutthatmustbe
considered
Priority Criticality
ResponseTime Timeavailabletoexecutethisusecase
Frequency Howoftenthisusecaseisexecuted
Channelsto Interactive|File|Database|...
PrimaryActor
SecondaryActors Otheractorsneededtoaccomplishthisusecase
Channelsto
SecondaryActors Interactive|File|Database|...
DateDue Scheduleinformation
Completeness
Level
UseCaseidentified(0.1)|Mainscenariodefined(0.5)|Allextensions
defined(0.8)|Allfieldscomplete(1.0)
OpenIssues Unresolvedissuesawaitingdecisions
Example
ConsiderthefollowingexamplefromtheStatelessUniversityRegistrationSystem.Astudent
wantstoregisterforacourseusingSU'sonlineregistrationsystem,SURS.
Table9-2:Exampleusecase.
UseCaseComponent Description
UseCaseNumberor
Identifier SURS1138
UseCaseName Registerforacourse(aclasstaughtbyafacultymember)
GoalinContext
Scope System
Level Primarytask
PrimaryActor Student
Preconditions None
SuccessEndConditions Thestudentisregisteredforthecourse—thecoursehas
beenaddedtothestudent'scourselist
FailedEndConditions Thestudent'scourselistisunchanged
Trigger Studentselectsacourseand"Registers"
MainSuccessScenarioA:
ActorS:System
Step Action
1 A:Selects"Registerforacourse"
2 A:Selectscourse(e.g.Math1060)
3 S:Displayscoursedescription
4 A:Selectssection(Mon&Wed9:00am)
5 S:Displayssectiondaysandtimes
6 A:Accepts
7 S:Addscourse/sectiontostudent'scourselist
Extensions
2a Coursedoesnotexist
S:Displaymessageandexit
4a Sectiondoesnotexist
S:Displaymessageandexit
4b
Sectionisfull
S:Displaymessageandexit
6a StudentdoesnotacceptS:Displaymessageandexit
Sub-Variations
Studentmayuse
Web
Phone
Priority Critical
ResponseTime 10secondsorless
Frequency Approximately5coursesx10,000studentsovera4-week
period
ChannelstoPrimaryActor Interactive
SecondaryActors None
ChannelstoSecondary
Actors N/A
DateDue 1Feb
CompletenessLevel 0.5
OpenIssues None
Hopefullyeachusecasehasbeenthroughaninspectionprocessbeforeitwasimplemented.
Totesttheimplementation,thebasicruleistocreateatleastonetestcaseforthemain
successscenarioandatleastonetestcaseforeachextension.
Becauseusecasesdonotspecifyinputdata,thetestermustselectit.Typicallyweusethe
equivalenceclassandboundaryvaluetechniquesdescribedearlier.AlsoaDomainTestMatrix
(seetheDomainAnalysisTestingchapterforanexample)maybeausefulwayofdocumenting
thetestcases.
Itisimportanttoconsidertheriskofthetransactionanditsvariantsundertest.Lessrisky
transactionsmeritlesstesting.Moreriskytransactionsshouldreceivemoretesting.Forthem
considerthefollowingapproach.
Key
Point
Alwaysremembertoevaluatetheriskofeachusecaseandextensionandcreate
testcasesaccordingly.
Tocreatetestcases,startwithnormaldataforthemostoftenusedtransactions.Thenmoveto
boundaryvaluesandinvaliddata.Next,choosetransactionsthat,whilenotusedoften,arevital
tothesuccessofthesystem(i.e.,ShutDownTheNuclearReactor).Makesureyouhaveat
leastonetestcaseforeveryExtensionintheusecase.Trytransactionsinstrangeorders.
Violatethepreconditions(ifthatcanhappeninactualuse).Ifatransactionhasloops,don'tjust
loopthroughonceortwice—bediabolical.Lookforthelongest,mostconvolutedpaththrough
thetransactionandtryit.Iftransactionsshouldbeexecutedinsomelogicalorder,trya
differentorder.Insteadofenteringdatatop-down,trybottom-up.Create"goofy"testcases.If
youdon'ttrystrangethings,youknowtheuserswill.
FreeStuffDownloadHolodeckfromhttp://www.sisecure.com/holodeck/holodeck-
trial.aspx.
Mostpathsthroughatransactionareeasytocreate.Theycorrespondtovalidandinvaliddata
beingentered.Moredifficultarethosepathsduetosomekindofexceptionalcondition—low
memory,diskfull,connectionlost,drivernotloaded,etc.Itcanbeverytimeconsumingforthe
testertocreateorsimulatetheseconditions.Fortunately,atoolisavailabletohelpthetester
simulatetheseproblems—Holodeck,createdbyJamesWhittakerandhisassociatesatFlorida
InstituteofTechnology.Holodeckmonitorstheinteractionsbetweenanapplicationandits
operatingsystem.Itlogseachsystemcallandenablesthetestertosimulateafailureofany
callatwill.Inthisway,thediskcanbe"madefull,"networkconnectionscan"become
disconnected,"datatransmissioncan"begarbled,"andahostofotherproblemscanbe
simulated.
Amajorcomponentoftransactiontestingistestdata.BorisBeizersuggeststhat30percentto
40percentoftheeffortintransactiontestingisgenerating,capturing,orextractingtestdata.
Don'tforgettoincluderesources(timeandpeople)forthisworkinyourproject'sbudget.
Note Onetestinggroupdesignatesa"dataczar"whosesoleresponsibilityistoprovidetest
data.
ApplicabilityandLimitations
Transactiontestingisgenerallythecornerstoneofsystemandacceptancetesting.Itshouldbe
usedwheneversystemtransactionsarewelldefined.Ifsystemtransactionsarenotwell
defined,youmightconsiderpolishingupyourresumeorC.V.
Whilecreatingatleastonetestcaseforthemainsuccessscenarioandatleastoneforeach
extensionprovidessomeleveloftestcoverage,itisclearthat,nomatterhowmuchwetry,
mostinputcombinationswillremainuntested.Donotbeoverconfidentaboutthequalityofthe
systematthispoint.
Summary
Ausecaseisascenariothatdescribestheuseofasystembyanactortoaccomplish
aspecificgoal.An"actor"isauser,playingarolewithrespecttothesystem,seeking
tousethesystemtoaccomplishsomethingworthwhilewithinaparticularcontext.A
scenarioisasequenceofstepsthatdescribetheinteractionsbetweentheactorand
thesystem.
Amajorcomponentoftransactiontestingistestdata.BorisBeizersuggeststhat30
percentto40percentoftheeffortintransactiontestingisgenerating,capturing,or
extractingtestdata.Don'tforgettoincluderesources(timeandpeople)forthisworkin
yourproject'sbudget.
Whilecreatingatleastonetestcaseforthemainsuccessscenarioandatleastonefor
eachextensionprovidessomeleveloftestcoverage,itisclearthat,nomatterhow
muchwetry,mostinputcombinationswillremainuntested.Donotbeoverconfident
aboutthequalityofthesystematthispoint.
Practice
1. Giventhe"RegisterForACourse"usecasefortheStatelessUniversityRegistration
Systemdescribedpreviously,createasetoftestcasessothatthemainsuccess
scenarioandeachoftheextensionsaretestedatleastonce.Choose"interesting"
testdatausingtheequivalenceclassandboundaryvaluetechniques.
References
Beizer,Boris(1990).SoftwareTestingTechniques(SecondEdition).VanNostrand
Reinhold.
Beizer,Boris(1995).Black-BoxTesting:TechniquesforFunctionalTestingofSoftware
andSystems.JohnWiley&Sons.
Cockburn,Alistair(2000).WritingEffectiveUseCases.Addison-Wesley.
Fowler,MartinandKendallScott(1999).UMLDistilled:ABriefGuidetotheStandard
ObjectModelingLanguage(2ndEdition).Addison-Wesley.
Jacobsen,Ivar,etal.(1992).Object-OrientedSystemsEngineering:AUseCaseDriven
Approach.Addison-Wesley.
Definition
Whiteboxtestingisastrategyinwhichtestingisbasedontheinternalpaths,structure,and
implementationofthesoftwareundertest(SUT).Unlikeitscomplement,blackboxtesting,
whiteboxtestinggenerallyrequiresdetailedprogrammingskills.
Thegeneralwhiteboxtestingprocessis:
TheSUT'simplementationisanalyzed.
PathsthroughtheSUTareidentified.
InputsarechosentocausetheSUTtoexecuteselectedpaths.Thisiscalledpath
sensitization.Expectedresultsforthoseinputsaredetermined.
Thetestsarerun.
Actualoutputsarecomparedwiththeexpectedoutputs.
AdeterminationismadeastotheproperfunctioningoftheSUT.
Applicability
Whiteboxtestingcanbeappliedatalllevelsofsystemdevelopment—unit,integration,and
system.Generallywhiteboxtestingisequatedwithunittestingperformedbydevelopers.While
thisiscorrect,itisanarrowviewofwhiteboxtesting.
Whiteboxtestingismorethancodetesting—itispathtesting.Generally,thepathsthatare
testedarewithinamodule(unittesting).Butwecanapplythesametechniquestotestpaths
betweenmoduleswithinsubsystems,betweensubsystemswithinsystems,andevenbetween
entiresystems.
Disadvantages
Whiteboxtestinghasfourdistinctdisadvantages.First,thenumberofexecutionpathsmaybe
solargethantheycannotallbetested.Attemptingtotestallexecutionpathsthroughwhitebox
testingisgenerallyasinfeasibleastestingallinputdatacombinationsthroughblackboxtesting.
Second,thetestcaseschosenmaynotdetectdatasensitivityerrors.Forexample:
p=q/r;
mayexecutecorrectlyexceptwhenr=0.
y=2*x//shouldready=x2
willpassfortestcasesx=0,y=0andx=2,y=4
Third,whiteboxtestingassumesthecontrolflowiscorrect(orveryclosetocorrect).Sincethe
testsarebasedontheexistingpaths,nonexistentpathscannotbediscoveredthroughwhite
boxtesting.
Fourth,thetestermusthavetheprogrammingskillstounderstandandevaluatethesoftware
undertest.Unfortunately,manytesterstodaydonothavethisbackground.
Advantages
Whenusingwhiteboxtesting,thetestercanbesurethateverypaththroughthesoftware
undertesthasbeenidentifiedandtested.
Chapter10:ControlFlowTesting
Itwasfromtheprimevalwellspringofanantediluvianpassionthatmystoryariseswhich,
liketheroundearthflattenedonamap,isbutalinearprojectionofanotherwise
periphrasticandpolyphiloprogenitive,non-planar,non-didactic,self-invertingconstruction
whoseobscurantistgeotropicliminalityisbeyondreasonabledoubt.
—MilindaBanerjee
Introduction
Controlflowtestingisoneoftwowhiteboxtestingtechniques.Thistestingapproachidentifies
theexecutionpathsthroughamoduleofprogramcodeandthencreatesandexecutestest
casestocoverthosepaths.Thesecondtechnique,discussedinthenextchapter,focuseson
dataflow.
Key
Point
Path:Asequenceofstatementexecutionthatbeginsatanentryandendsatan
exit.
Unfortunately,inanyreasonablyinterestingmodule,attemptingexhaustivetestingofallcontrol
flowpathshasanumberofsignificantdrawbacks.
Thenumberofpathscouldbehugeandthusuntestablewithinareasonableamountof
time.Everydecisiondoublesthenumberofpathsandeveryloopmultipliesthepathsby
thenumberofiterationsthroughtheloop.Forexample:
for(i=1;i<=1000;i++)
for(j=1;j<=1000;j++)
for(k=1;k<=1000;k++)
doSomethingWith(i,j,k);
executesdoSomethingWith()onebilliontimes(1000x1000x1000).Eachuniquepath
deservestobetested.
Pathscalledforinthespecificationmaysimplybemissingfromthemodule.Anytesting
approachbasedonimplementedpathswillneverfindpathsthatwerenotimplemented.
if(a>0)doIsGreater();
if(a==0)dolsEqual();
//missingstatement-if(a<0)dolsLess();
Defectsmayexistinprocessingstatementswithinthemoduleeventhroughthecontrol
flowitselfiscorrect.
//actual(butincorrect)code
a=a+1;
//correctcode
a=a-1;
Themodulemayexecutecorrectlyforalmostalldatavaluesbutfailforafew.
intblech(inta,intb){
returna/b;
}
failsifbhasthevalue0butexecutescorrectlyifbisnot0.
Eventhoughcontrolflowtestinghasanumberofdrawbacks,itisstillavitaltoolinthetester's
toolbox.
Technique
ControlFlowGraphs
Controlflowgraphsarethefoundationofcontrolflowtesting.Thesegraphsdocumentthe
module'scontrolstructure.Modulesofcodeareconvertedtographs,thepathsthroughthe
graphsareanalyzed,andtestcasesarecreatedfromthatanalysis.Controlflowgraphs
consistofanumberofelements:
Key
Point Controlflowgraphsarethefoundationofcontrolflowtesting.
ProcessBlocks
Aprocessblockisasequenceofprogramstatementsthatexecutesequentiallyfrombeginning
toend.Noentryintotheblockispermittedexceptatthebeginning.Noexitfromtheblockis
permittedexceptattheend.Oncetheblockisinitiated,everystatementwithinitwillbe
executedsequentially.Processblocksarerepresentedincontrolflowgraphsbyabubblewith
oneentryandoneexit.
DecisionPoint
Adecisionpointisapointinthemoduleatwhichthecontrolflowcanchange.Mostdecision
pointsarebinaryandareimplementedbyif-then-elsestatements.Multi-waydecisionpointsare
implementedbycasestatements.Theyarerepresentedbyabubblewithoneentryand
multipleexits.
JunctionPoint
Ajunctionpointisapointatwhichcontrolflowsjointogether.
Thefollowingcodeexampleisrepresentedbyitsassociatedflowgraph:
Figure10-1:Flowgraphequivalentofprogramcode.
LevelsofCoverage
Incontrolflowtesting,differentlevelsoftestcoveragearedefined.By"coverage"wemeanthe
percentageofthecodethathasbeentestedvs.thatwhichistheretotest.Incontrolflow
testingwedefinecoverageatanumberofdifferentlevels.(Notethatthesecoveragelevelsare
notpresentedinorder.Thisisbecause,insomecases,itiseasiertodefineahighercoverage
levelandthendefinealowercoveragelevelintermsofthehigher.)
Level1
Thelowestcoveragelevelis"100%statementcoverage"(sometimesthe"100%"is
droppedandisreferredtoas"statementcoverage").Thismeansthateverystatement
withinthemoduleisexecuted,undertest,atleastonce.Whilethismayseemlikea
reasonablegoal,manydefectsmaybemissedwiththislevelofcoverage.Considerthe
followingcodesnippet:
if(a>0){x=x+1;}
if(b==3){y=0;}
Thiscodecanberepresentedingraphicalformas:
Figure10-2:Graphicalrepresentationofthetwo-linecodesnippet.
Thesetwolinesofcodeimplementfourdifferentpathsofexecution:
Figure10-3:Fourexecutionpaths.
Whileasingletestcaseissufficienttotesteverylineofcodeinthismodule(for
example,usea=6andb=3asinput),itisapparentthatthislevelofcoveragewillmiss
testingmanypaths.Thus,statementcoverage,whileabeginning,isgenerallynotan
acceptableleveloftesting.
Eventhoughstatementcoverageisthelowestlevelofcoverage,eventhatmaybedifficultto
achieveinpractice.Oftenmoduleshavecodethatisexecutedonlyinexceptional
circumstances—lowmemory,fulldisk,unreadablefiles,lostconnections,etc.Testersmayfind
itdifficultorevenimpossibletosimulatethesecircumstancesandthuscodethatdealswith
theseproblemswillremainuntested.
Holodeckisatoolthatcansimulatemanyoftheseexceptionalsituations.Accordingto
Holodeck'sspecificationit"willallowyou,thetester,totestsoftwarebyobservingthesystem
callsthatitmakesandcreatetestcasesthatyoumayuseduringsoftwareexecutiontomodify
thebehavioroftheapplication.Modificationsmightincludemanipulatingtheparameterssentto
functionsorchangingthereturnvaluesoffunctionswithinyoursoftware.Inaddition,youmay
alsoseterror-codesandothersystemevents.Thissetofpossibilitiesallowsyoutoemulate
environmentsthatyoursoftwaremightencounter-hencethename'Holodeck.'Insteadof
needingtounplugyournetworkconnection,createadiskwithbadsectors,corruptpacketson
thenetwork,orperformanyoutsideorspecialmanipulationofyourmachine,youcanuse
Holodecktoemulatetheseproblems.Faultscaneasilybeplacedintoanysoftwaretesting
projectthatyouareusingwithHolodeck."
Holodeck
TodownloadHolodeckvisithttp://www.sisecure.com/holodeck/holodeck-trial.aspx.
Level0
Actually,thereisalevelofcoveragebelow"100%statementcoverage."Thatlevelis
definedas"testwhateveryoutest;lettheuserstesttherest."Thecorporate
landscapeisstrewnwiththesun-bleachedbonesoforganizationswhohaveusedthis
testingapproach.Regardingthislevelofcoverage,BorisBeizerwrote"testingless
thanthis[100%statementcoverage]fornewsoftwareisunconscionableandshouldbe
criminalized....IncaseIhaven'tmademyselfclear,...untestedcodeinasystemis
stupid,shortsighted,andirresponsible."
Level2
Thenextlevelofcontrolflowcoverageis"100%decisioncoverage."Thisisalsocalled
"branchcoverage."Atthislevelenoughtestcasesarewrittensothateachdecision
thathasaTRUEandFALSEoutcomeisevaluatedatleastonce.Intheprevious
examplethiscanbeachievedwithtwotestcases(a=2,b=2anda=4,b=3).
Figure10-4:Twotestcasesthatyield100%decisioncoverage.
Casestatementswithmultipleexitswouldhavetestsforeachexit.Notethatdecision
coveragedoesnotnecessarilyguaranteepathcoveragebutitdoesguarantee
statementcoverage.
Level3
Notallconditionalstatementsareassimpleastheonespreviouslyshown.Consider
thesemorecomplicatedstatements:
if(a>0&&c==1){x=x+1;}
if(b==3||d<0){y=0;}
TobeTRUE,thefirststatementrequiresagreaterthan0andcequal1.Thesecond
requiresbequal3ordlessthan0.
Inthefirststatementifthevalueofaweresetto0fortestingpurposesthenthec==1
partoftheconditionwouldnotbetested.(Inmostprogramminglanguagesthesecond
expressionwouldnotevenbeevaluated.)Thenextlevelofcontrolflowcoverageis
"100%conditioncoverage."Atthislevelenoughtestcasesarewrittensothateach
conditionthathasaTRUEandFALSEoutcomethatmakesupadecisionisevaluated
atleastonce.Thislevelofcoveragecanbeachievedwithtwotestcases(a>0,c=1,
b=3,d<0anda≤0,c≠1,b≠3,d≥0).Conditioncoverageisusuallybetterthandecision
coveragebecauseeveryindividualconditionistestedatleastoncewhiledecision
coveragecanbeachievedwithouttestingeverycondition.
Level4
Considerthissituation:
if(x&&y){conditionedStatement;}
//note:&&indicateslogicalAND
Wecanachieveconditioncoveragewithtwotestcases(x=TRUE,y=FALSEand
x=FALSE,y=TRUE)butnotethatwiththesechoicesofdatavaluesthe
conditionedStatementwillneverbeexecuted.Giventhepossiblecombinationof
conditionssuchasthese,tobemorecomplete"100%decision/condition"coveragecan
beselected.Atthisleveltestcasesarecreatedforeveryconditionandeverydecision.
Level5
Tobeevenmorethorough,considerhowtheprogramminglanguagecompileractually
evaluatesthemultipleconditionsinadecision.Usethatknowledgetocreatetestcases
yielding"100%multipleconditioncoverage."
if(a>0&&c==1){x=x+1;}
if(b==3||d<0){y=0;}
//note:||meanslogicalOR
willbeevaluatedas:
Figure10-5:Compilerevaluationofcomplexconditions.
Thislevelofcoveragecanbeachievedwithfourtestcases:
a>0,c=1,b=3,d<0
a≤0,c=1,b=3,d≥0
a>0,c≠1,b≠3,d<0
a≤0,c≠1,b≠3,d≥0
Achieving100%multipleconditioncoveragealsoachievesdecisioncoverage,condition
coverage,anddecision/conditioncoverage.Notethatmultipleconditioncoveragedoes
notguaranteepathcoverage.
Level7
Finallywereachthehighestlevel,whichis"100%pathcoverage."Forcodemodules
withoutloopsthenumberofpathsisgenerallysmallenoughthatatestcasecan
actuallybeconstructedforeachpath.Formoduleswithloops,thenumberofpathscan
beenormousandthusposeanintractabletestingproblem.
Figure10-6:Aninterestingflowdiagramwithmany,manypaths.
Level6
Whenamodulehasloopsinthecodepathssuchthatthenumberofpathsisinfinite,a
significantbutmeaningfulreductioncanbemadebylimitingloopexecutiontoasmall
numberofcases.Thefirstcaseistoexecutetheloopzerotimes;thesecondisto
executethelooponetime,thethirdistoexecutetheloopntimeswherenisasmall
numberrepresentingatypicalloopvalue;thefourthistoexecutetheloopitsmaximum
numberoftimesm.Inadditionyoumighttrym-1andm+1.
Beforebeginningcontrolflowtesting,anappropriatelevelofcoverageshouldbechosen.
StructuredTesting/BasisPathTesting
Nodiscussiononcontrolflowtestingwouldbecompletewithoutapresentationofstructured
testing,alsoknownasbasispathtesting.Structuredtestingisbasedonthepioneeringworkof
TomMcCabe.Itusesananalysisofthetopologyofthecontrolflowgraphtoidentifytest
cases.
Thestructuredtestingprocessconsistsofthefollowingsteps:
Derivethecontrolflowgraphfromthesoftwaremodule.
Computethegraph'sCyclomaticComplexity(C).
SelectasetofCbasispaths.
Createatestcaseforeachbasispath.
Executethesetests.
Considerthefollowingcontrolflowgraph:
Figure10-7:Anexamplecontrolflowgraph.
McCabedefinestheCyclomaticComplexity(C)ofagraphas
C=edges-nodes+2
Edgesarethearrows,andnodesarethebubblesonthegraph.Theprecedinggraphhas24
edgesand19nodesforaCyclomaticComplexityof24-19+2=7.
Insomecasesthiscomputationcanbesimplified.Ifalldecisionsinthegrapharebinary(they
haveexactlytwoedgesflowingout),andtherearepbinarydecisions,then
C=p+1
CyclomaticComplexityisexactlytheminimumnumberofindependent,nonloopingpaths(called
basispaths)thatcan,inlinearcombination,generateallpossiblepathsthroughthemodule.In
termsofaflowgraph,eachbasispathtraversesatleastoneedgethatnootherpathdoes.
McCabe'sstructuredtestingtechniquecallsforcreatingCtestcases,oneforeachbasispath.
IMPORTANT
!
CreatingandexecutingCtestcases,basedonthebasispaths,guarantees
bothbranchandstatementcoverage.
Becausethesetofbasispathscoversalltheedgesandnodesofthecontrolflowgraph,
satisfyingthisstructuredtestingcriteriaautomaticallyguaranteesbothbranchandstatement
coverage.
AprocessforcreatingasetofbasispathsisgivenbyMcCabe:
1. Picka"baseline"path.Thispathshouldbeareasonably"typical"pathofexecution
ratherthananexceptionprocessingpath.Thebestchoicewouldbethemost
importantpathfromthetester'sview.
Figure10-8:ThechosenbaselinebasispathABDEGKMQS
2. Tochoosethenextpath,changetheoutcomeofthefirstdecisionalongthebaseline
pathwhilekeepingthemaximumnumberofotherdecisionsthesameasthebaseline
path.
Figure10-9:ThesecondbasispathACDEGKMQS
3. Togeneratethethirdpath,beginagainwiththebaselinebutvarytheseconddecision
ratherthanthefirst.
Figure10-10:ThethirdbasispathABDFILORS
4. Togeneratethefourthpath,beginagainwiththebaselinebutvarythethirddecision
ratherthanthesecond.Continuevaryingeachdecision,onebyone,untilthebottom
ofthegraphisreached.
Figure10-11:ThefourthbasispathABDEHKMQS
Figure10-12:ThefifthbasispathABDEGKNQS
5. Nowthatalldecisionsalongthebaselinepathhavebeenflipped,weproceedtothe
secondpath,flippingitsdecisions,onebyone.Thispatterniscontinueduntilthebasis
pathsetiscomplete.
Figure10-13:ThesixthbasispathACDFJLORS
Figure10-14:TheseventhbasispathACDFILPRS
Thus,asetofbasispathsforthisgraphare:
ABDEGKMQS
ACDEGKMQS
ABDFILORS
ABDEHKMQS
ABDEGKNQS
ACDFJLORS
ACDFILPRS
Structuredtestingcallsforthecreationofatestcaseforeachofthesepaths.Thissetoftest
caseswillguaranteebothstatementandbranchcoverage.
Notethatmultiplesetsofbasispathscanbecreatedthatarenotnecessarilyunique.Eachset,
however,hasthepropertythatasetoftestcasesbasedonitwillexecuteeverystatementand
everybranch.
Example
ConsiderthefollowingexamplefromBrown&Donaldson.Itisthecodethatdetermines
whetherB&Dshouldbuyorsellaparticularstock.Unfortunately,theinnerworkingsareahighly
classifiedtradesecretsotheactualprocessingcodehasbeenremovedandgeneric
statementslikes1;s2;etc.havesubstitutedforthem.Thecontrolflowstatementshavebeen
leftintactbuttheiractualconditionshavebeenremovedandgenericconditionslikec1andc2
havebeenputintheirplace.(Youdidn'tthinkwe'dreallyshowyouhowtoknowwhethertobuy
orsellstocks,didyou?)
Note s1,s2,...representJavastatementswhilec1,c2,...representconditions.
booleanevaluateBuySell(TickerSymbolts){
s1;
s2;
s3;
if(c1){s4;s5;s6;}
else{s7;s8;}
while(c2){
s9;
s10;
switch(c3){
case-A:
s20;
s21;
s22;
break;//EndofCase-A
case-B:
s30;
s31;
if(c4){
s32;
s33;
s34;
}
else{
s35;
}
break;//EndofCase-B
case-C:
s40;
s41;
break;//EndofCase-C
case-D:
s50;
break;//EndofCase-D
}//EndSwitch
s60;
s61;
s62;
if(c5){s70;s71;}
s80;
s81;
}//EndWhile
s90;
s91;
s92;
returnresult;
Figure10-15:JavacodeforBrown&Donaldson'sevaluateBuySellmodule.
ThefollowingflowdiagramcorrespondstothisJavacode:
Figure10-16:ControlflowgraphforBrown&Donaldson'sevaluateBuySellmodule.
Thecyclomaticcomplexityofthisdiagramiscomputedbyedges-nodes+2
or
22-16+2=8
Let'sremovethecodeandlabeleachnodeforsimplicityindescribingthepaths.
Figure10-17:ControlflowgraphforBrown&Donaldson'sevaluateBuySellmodule.
Asetofeightbasispathsis:
1. ABDP
2. ACDP
3. ABDEFGMODP
4. ABDEFHKMODP
5. ABDEFIMODP
6. ABDEFJMODP
7. ABDEFHLMODP
8. ABDEFIMNODP
Rememberthatbasispathsetsarenotunique;therecanbemultiplesetsofbasispathsfora
graph.
Thisbasispathsetisnowimplementedastestcases.Choosevaluesfortheconditionsthat
wouldsensitizeeachpathandexecutethetests.
Table10-1:Datavaluestosensitizethedifferentcontrolflowpaths.
TestCase C1 C2 C3 C4 C5
1 False False N/A N/A N/A
2 True False N/A N/A N/A
3 False True A N/A False
4 False True B False False
5 False True C N/A False
6 False True D N/A False
7 False True B True False
8 False True C N/A True
ApplicabilityandLimitations
Controlflowtestingisthecornerstoneofunittesting.Itshouldbeusedforallmodulesofcode
thatcannotbetestedsufficientlythroughreviewsandinspections.Itslimitationsarethatthe
testermusthavesufficientprogrammingskilltounderstandthecodeanditscontrolflow.In
addition,controlflowtestingcanbeverytimeconsumingbecauseofallthemodulesandbasis
pathsthatcompriseasystem.
Summary
Controlflowtestingidentifiestheexecutionpathsthroughamoduleofprogramcode
andthencreatesandexecutestestcasestocoverthosepaths.
Controlflowgraphsarethefoundationofcontrolflowtesting.Modulesofcodeare
convertedtographs,thepathsthroughthegraphsareanalyzed,andtestcasesare
createdfromthatanalysis.
CyclomaticComplexityisexactlytheminimumnumberofindependent,nonloopingpaths
(calledbasispaths)thatcan,inlinearcombination,generateallpossiblepathsthrough
themodule.
Becausethesetofbasispathscoversalltheedgesandnodesofthecontrolflow
graph,satisfyingthisstructuredtestingcriteriaautomaticallyguaranteesbothbranch
andstatementcoverage.
Practice
1. Belowisabriefprogramlisting.Createthecontrolflowdiagram,determineits
CyclomaticComplexity,chooseasetofbasispaths,anddeterminethenecessary
valuesfortheconditionstosensitizeeachpath.
if(c1){
while(c2){
if(c3){s1;s2;
if(c5)s5;
elses6;
break;//Skiptoendofwhile
else
if(c4){}
else{s3;s4;break;}
}//Endofwhile
}//Endofif
s7;
if(c6)s8;s9;
s10;
References
Beizer,Boris(1990).SoftwareTestingTechniques(SecondEdition).VanNostrand
Reinhold.
Myers,Glenford(1979).TheArtofSoftwareTesting.JohnWiley&Sons.
Pressman,RogerS.(1982).SoftwareEngineering:APractitioner'sApproach(Fourth
Edition).McGraw-Hill.
Watson,ArthurH.andThomasJ.McCabe.StructuredTesting:ATestingMethodology
UsingtheCyclomaticComplexityMetric.NISTSpecialPublication500-235availableat
http://www.mccabe.com/nist/nist_pub.php
Chapter11:DataFlowTesting
Hollyhadreachedtheageandlevelofmaturitytocomprehendtheemotionalnuancesof
ThomasWolfe'sassertion"youcan'tgohomeagain,"butinhercaseitwasevenmore
poignantbecausetherewasnohometoreturnto:herparentshadseparated,soldthe
house,euthanizedBowser,anddisownedHollyfordroppingoutofhighschooltomarry
that43-year-oldmanagerofTrailerTowninIdaho—andeventheirtrailerwasn'taplace
shecouldcallhomebecauseitwasonlyasummersublet.
—EileenOstrowFeldman
Introduction
Almosteveryprogrammerhasmadethistypeofmistake:
main(){
intx;
if(x==42){...}
}
Themistakeisreferencingthevalueofavariablewithoutfirstassigningavaluetoit.Naive
developersunconsciouslyassumethatthelanguagecompilerorrun-timesystemwillsetall
variablestozero,blanks,TRUE,42,orwhatevertheyrequirelaterintheprogram.AsimpleC
programthatillustratesthisdefectis:
#include<stdio.h>
main(){
intx;
printf("%d",x);
}
Thevalueprintedwillbewhatevervaluewas"leftover"inthememorylocationtowhichxhas
beenassigned,notnecessarilywhattheprogrammerwantedorexpected.
Dataflowtestingisapowerfultooltodetectjustsucherrors.RappsandWeyuker,
popularizersofthisapproach,wrote,"Itisourbeliefthat,justasonewouldnotfeelconfident
aboutaprogramwithoutexecutingeverystatementinitaspartofsometest,oneshouldnot
feelconfidentaboutaprogramwithouthavingseentheeffectofusingthevalueproducedby
eachandeverycomputation."
Key
Point
Dataflowtestingisapowerfultooltodetectimproperuseofdatavaluesdueto
codingerrors.
Technique
Variablesthatcontaindatavalueshaveadefinedlifecycle.Theyarecreated,theyareused,
andtheyarekilled(destroyed).Insomeprogramminglanguages(FORTRANandBASIC,for
example)creationanddestructionareautomatic.Avariableiscreatedthefirsttimeitis
assignedavalueanddestroyedwhentheprogramexits.
Inotherlanguages(likeC,C++,andJava)thecreationisformal.Variablesaredeclaredby
statementssuchas:
intx;//xiscreatedasaninteger
stringy;//yiscreatedasastring
Thesedeclarationsgenerallyoccurwithinablockofcodebeginningwithanopeningbrace{
andendingwithaclosingbrace}.Variablesdefinedwithinablockarecreatedwhentheir
definitionsareexecutedandareautomaticallydestroyedattheendofablock.Thisiscalled
the"scope"ofthevariable.Forexample:
{//beginouterblock
intx;//xisdefinedasanintegerwithinthisouterblock
...;//xcanbeaccessedhere
{//begininnerblock
inty;//yisdefinedwithinthisinnerblock
...;//bothxandycanbeaccessedhere
}//yisautomaticallydestroyedattheendof
//thisblock
...;//xcanstillbeaccessed,butyisgone
}//xisautomaticallydestroyed
Variablescanbeusedincomputation(a=b+1).Theycanalsobeusedinconditionals(if
(a>42)).Inbothusesitisequallyimportantthatthevariablehasbeenassignedavaluebefore
itisused.
Threepossibilitiesexistforthefirstoccurrenceofavariablethroughaprogrampath:
1. ~dthevariabledoesnotexist(indicatedbythe~),thenitisdefined(d)
2. ~uthevariabledoesnotexist,thenitisused(u)
3. ~k thevariabledoesnotexist,thenitiskilledordestroyed(k)
Thefirstiscorrect.Thevariabledoesnotexistandthenitisdefined.Thesecondisincorrect.A
variablemustnotbeusedbeforeitisdefined.Thethirdisprobablyincorrect.Destroyinga
variablebeforeitiscreatedisindicativeofaprogrammingerror.
Nowconsiderthefollowingtime-sequencedpairsofdefined(d),used(u),andkilled(k):
ddDefinedanddefinedagain—notinvalidbutsuspicious.Probablyaprogrammingerror.
duDefinedandused—perfectlycorrect.Thenormalcase.
dk Definedandthenkilled—notinvalidbutprobablyaprogrammingerror.
ud Usedanddefined—acceptable.
uu Usedandusedagain—acceptable.
uk Usedandkilled—acceptable.
kd Killedanddefined—acceptable.Avariableiskilledandthenredefined.
ku Killedandused—aseriousdefect.Usingavariablethatdoesnotexistorisundefinedis
alwaysanerror.
kk Killedandkilled—probablyaprogrammingerror.
Key
Point Examinetime-sequencedpairsofdefined,used,andkilledvariablereferences.
Adataflowgraphissimilartoacontrolflowgraphinthatitshowstheprocessingflowthrough
amodule.Inaddition,itdetailsthedefinition,use,anddestructionofeachofthemodule's
variables.Wewillconstructthesediagramsandverifythatthedefine-use-killpatternsare
appropriate.First,wewillperformastatictestofthediagram.By"static"wemeanwe
examinethediagram(formallythroughinspectionsorinformallythroughlook-sees).Second,we
performdynamictestsonthemodule.By"dynamic"wemeanweconstructandexecutetest
cases.Let'sbeginwiththestatictesting.
StaticDataFlowTesting
Thefollowingcontrolflowdiagramhasbeenannotatedwithdefine-use-killinformationforeach
ofthevariablesusedinthemodule.
Figure11-1:Thecontrolflowdiagramannotatedwithdefine-use-killinformation
foreachofthemodule'svariables.
Foreachvariablewithinthemodulewewillexaminedefine-use-killpatternsalongthecontrol
flowpaths.Considervariablexaswetraversetheleftandthentherightpath:
Figure11-2:Thecontrolflowdiagramannotatedwithdefine-use-killinformationforthex
variable.
Thedefine-use-killpatternsforx(takeninpairsaswefollowthepaths)are:
~define correct,thenormalcase
define-definesuspicious,perhapsaprogrammingerror
define-use correct,thenormalcase
Nowforvariabley.Notethatthefirstbranchinthemodulehasnoimpactontheyvariable.
Figure11-3:Thecontrolflowdiagramannotatedwithdefine-use-killinformation
fortheyvariable.
Thedefine-use-killpatternsfory(takeninpairsaswefollowthepaths)are:
~use majorblunder
use-defineacceptable
define-usecorrect,thenormalcase
use-kill acceptable
define-kill probableprogrammingerror
Nowforvariablez.
Figure11-4:Thecontrolflowdiagramannotatedwithdefine-use-killinformation
forthezvariable.
Thedefine-use-killpatterns(takeninpairsaswefollowthepaths)are:
~kill programmingerror
kill-use majorblunder
use-use correct,thenormalcase
use-defineacceptable
kill-kill probablyaprogrammingerror
kill-define acceptable
define-usecorrect,thenormalcase
Inperformingastaticanalysisonthisdataflowmodelthefollowingproblemshavebeen
discovered:
x:define-define
y:~use
y:define-kill
z:~kill
z:kill-use
z:kill-kill
Unfortunately,whilestatictestingcandetectmanydataflowerrors,itcannotfindthemall.
Considerthefollowingsituations:
1. Arraysarecollectionsofdataelementsthatsharethesamenameandtype.For
example
intstuff[100];
definesanarraynamedstuffconsistingof100integerelements.InC,C++,andJava
theindividualelementsarenamedstuff[0],stuff[1],stuff[2],etc.Arraysaredefined
anddestroyedasaunitbutspecificelementsofthearrayareusedindividually.Often
programmersrefertostuff[j]wherejchangesdynamicallyastheprogramexecutes.
Inthegeneralcase,staticanalysiscannotdeterminewhetherthedefine-use-killrules
havebeenfollowedproperlyunlesseachelementisconsideredindividually.
2. Incomplexcontrolflowsitispossiblethatacertainpathcanneverbeexecuted.In
thiscaseanimproperdefine-use-killcombinationmightexistbutwillneverbe
executedandsoisnottrulyimproper.
3. Insystemsthatprocessinterrupts,someofthedefine-use-killactionsmayoccurat
theinterruptlevelwhileotherdefine-use-killactionsoccuratthemainprocessing
level.Inaddition,ifthesystemusesmultiplelevelsofexecutionpriorities,static
analysisofthemyriadofpossibleinteractionsissimplytoodifficulttoperform
manually.
Forthisreason,wenowturntodynamicdataflowtesting.
DynamicDataFlowTesting
Becausedataflowtestingisbasedonamodule'scontrolflow,itassumesthatthecontrolflow
isbasicallycorrect.Thedataflowtestingprocessistochooseenoughtestcasessothat:
Every"define"istracedtoeachofits"uses"
Every"use"istracedfromitscorresponding"define"
Todothis,enumeratethepathsthroughthemodule.Thisisdoneusingthesameapproachas
incontrolflowtesting:Beginatthemodule'sentrypoint,taketheleftmostpaththroughthe
moduletoitsexit.Returntothebeginningandvarythefirstbranchingcondition.Followthat
pathtotheexit.Returntothebeginningandvarythesecondbranchingcondition,thenthethird,
andsoonuntilallthepathsarelisted.Then,foreveryvariable,createatleastonetestcaseto
covereverydefine-usepair.
ApplicabilityandLimitations
Dataflowtestingbuildsonandexpandscontrolflowtestingtechniques.Aswithcontrolflow
testing,itshouldbeusedforallmodulesofcodethatcannotbetestedsufficientlythrough
reviewsandinspections.Itslimitationsarethatthetestermusthavesufficientprogrammingskill
tounderstandthecode,itscontrolflow,anditsvariables.Likecontrolflowtesting,dataflow
testingcanbeverytimeconsumingbecauseofallthemodules,paths,andvariablesthat
compriseasystem.
Summary
Acommonprogrammingmistakeisreferencingthevalueofavariablewithoutfirst
assigningavaluetoit.
Adataflowgraphissimilartoacontrolflowgraphinthatitshowstheprocessingflow
throughamodule.Inaddition,itdetailsthedefinition,use,anddestructionofeachof
themodule'svariables.Wewillusethesediagramstoverifythatthedefine-use-kill
patternsareappropriate.
Enumeratethepathsthroughthemodule.Then,foreveryvariable,createatleastone
testcasetocovereverydefine-usepair.
Practice
1. Thefollowingmoduleofcodecomputesn!(nfactorial)givenavalueforn.Create
dataflowtestcasescoveringeachvariableinthismodule.Remember,asingletest
casecancoveranumberofvariables.
intfactorial(intn){
intanswer,counter;
answer=1;
counter=1;
loop:
if(counter>n)returnanswer;
answer=answer*counter;
counter=counter+1;
gotoloop;
}
2. Diagramthecontrolflowpathsandderivethedataflowtestcasesforthefollowing
module:
intmodule(intselector){
intfoo,bar;
switchselector{
caseSELECT-1:
foo=calc_foo_method_1();
break;
caseSELECT-2:
foo=calc_foo_method_2();
break;
caseSELECT-3:
foo=calc_foo_method_3();
break;
}
switchfoo{
caseFOO-1:
bar=calc_bar_method_1();
break;
caseFOO-2:
bar=calc_bar_method_2();
break;
}
returnfoo/bar;
}
Doyouhaveanyconcernswiththiscode?Howwouldyoudealwiththem?
References
Beizer,Boris(1990).SoftwareTestingTechniques.VanNostrandReinhold.
Binder,RobertV.(2000).TestingObject-OrientedSystems:Models,Patterns,and
Tools.Addison-Wesley.
Marick,Brian(1995).TheCraftofSoftwareTesting:SubsystemTestingIncluding
Object-BasedandObject-OrientedTesting.Prentice-Hall.
Rapps,SandraandElaineJ.Weyuker."DataFlowAnalysisTechniquesforTestData
Selection."SixthInternationalConferenceonSoftwareEngineering,Tokyo,Japan,
September13–16,1982.
Paradigms
Inhisbook,Paradigms:TheBusinessofDiscoveringtheFuture,JoelBarkerdefinesa
paradigmas"asetofrulesandregulations(writtenorunwritten)thatdoestwothings:(1)it
establishesordefinesboundaries,and(2)ittellsyouhowtobehaveinsidetheboundariesin
ordertobesuccessful."FuturistMarilynFergusonseesaparadigmas"aframeworkofthought
...aschemeforunderstandingandexplainingcertainaspectsofreality."
Paradigmsareusefulbecausetheyhelpusmakesenseofthecomplexitiesoftheworldaround
us.Inthisway,paradigmssharpenourvision.Butparadigmscanblindustorealities.
Paradigmsactaspsychologicalfilters.Datathatdoesnotmatchourparadigmsisblocked.In
thisway,paradigmscloudourvision.
Insoftwaretestingtoday,twoverydifferentparadigmsarebattlingforadherents—scripted
testingandexploratorytesting.
Scriptedtestingisbasedonthesequentialexaminationofrequirements,followedbythedesign
anddocumentationoftestcases,followedbytheexecutionofthosetestcases.Thescripted
tester'smottois,"Planyourwork,workyourplan."
Exploratorytestingisaverydifferentparadigm.Ratherthanasequentialapproach,exploratory
testingemphasizessimultaneouslearning,testdesign,andtestexecution.Thetesterdesigns
andexecutestestswhileexploringtheproduct.
WordOf
Warning!
Inthefollowingchaptersthescriptedandexploratoryparadigmsare
definedattheextremeendsofthespectrum.Rarelywilleitherbeusedas
inflexiblyasdescribed.
Thenexttwochaptersdescribetheseparadigms.Awordofwarningthough—eachparadigmis
describedattheextremeendoftheprocessspectrum.Rarelywilleitherparadigmbeusedas
inflexiblyasdescribed.Moreoften,scriptedtestingmaybesomewhatexploratoryand
exploratorytestingmaybesomewhatscripted.
TestPlanning
Planninghasbeendefinedassimply"figuringoutwhattodonext."Tobemosteffectiveand
efficient,planningisimportant.Butwhenandhowshouldthatplanningbedone?Scripted
testingemphasizesthevalueofearlytestdesignasamethodofdetectingrequirementsand
designdefectsbeforethecodeiswrittenandthesystemputintoproduction.Itsfocusison
accountabilityandrepeatability.Exploratorytestingchallengestheideathattestsmustbe
designedsoveryearlyintheproject,whenourknowledgeistypicallyatitsminimum.Itsfocus
isonlearningandadaptability.
References
Barker,JoelArthur(1992).Paradigms:TheBusinessofDiscoveringtheFuture.
HarperCollins.
Ferguson,Marilyn(1980).TheAquarianConspiracy:PersonalandSocial
TransformationinOurTime.PutnamPublishingGroup.
Chapter12:ScriptedTesting
Janewastoast,andnotthelightbutterykind,nay,shewasthekindthat'sbeencharred
andblackenedinthebottomofthetoasterandhastobethrownawaybecausenomatter
howmuchoftheburntpartyouscrapeoffwithaknife,there'salwaysmoreblackened
toastbeneath,thekindthatnotevenstarvingbirdsinwinterwilleat,thatkindoftoast.
—BethKnutson
Introduction
Forscriptedtestingtobeunderstood,itmustbeunderstoodinitshistoricalcontext.Scripted
testingemergedasoneofthecomponentpartsoftheWaterfallmodelofsoftware
development.TheWaterfallmodeldefinesanumberofsequentialdevelopmentphaseswith
specificentryandexitcriteria,taskstobeperformed,anddeliverables(tangibleworkproducts)
tobecreated.Itisaclassicexampleofthe"planyourwork,workyourplan"philosophy.
TypicalWaterfallphasesinclude:
1. SystemRequirements-Gatheringtherequirementsforthesystem.
2. SoftwareRequirements-Gatheringtherequirementsforthesoftwareportionofthe
system.
3. RequirementsAnalysis-Analyzing,categorizing,andrefiningthesoftware
requirements.
4. ProgramDesign-Choosingarchitectures,modules,andinterfacesthatdefinethe
system.
5. Coding-Writingtheprogrammingcodethatimplementsthedesign.
6. Testing-Evaluatingwhethertherequirementswereproperlyunderstood(Validation)
andthedesignproperlyimplementedbythecode(Verification).
7. Operations-Putthesystemintoproduction.
InterestingTriviaAGooglesearchfor"planyourwork"and"workyourplan"found3,570
matchesincluding:
Footballrecruiting
Businessplanning
Buildingwithconcreteblocks
Onlinemarketing
Industrialdistribution
ThePrincetonUniversity'sWomen'sWaterPoloTeam
Andthousandsmore
Thismodelwasfirstdescribedin1970inapaperentitled"ManagingtheDevelopmentofLarge
ScaleSystems"byDr.WinstonW.Royce.Roycedrewthefollowingdiagramshowingthe
relationshipsbetweendevelopmentphases:
Figure12-1:TheWaterfalllifecyclemodel.
WhatprocesswasusedbeforeWaterfall?Itisaprocessknownas"Code&Fix."
Programmerssimplycoded.Sloganslike"Requirements?Requirements?Wedon'tneedno
stinkin'Requirements!"hungonthewallsofprogrammers'offices.Developmentwaslikethe
sceneinthemovieRaidersoftheLostArk.Ourhero,IndianaJones,ishidingfromthebad
guys.Indysays,"I'mgoingtogetthattruck."Marion,ourheroine,turnstohimandasks,"How
areyougoingtogetthattruck?"Indyreplies,"Idon'tknow.I'mmakingthisupasIgo."Ifwe
substituted"buildthatsystem"for"getthattruck"we'dhavethewayrealmenandrealwomen
builtsoftwaresystemsinthegoodolddays.
CuriousHistoricalNoteToday,WinstonRoyceisknownasthefatheroftheWaterfall
modelofsoftwaredevelopment.Infact,inhispaperhewasactuallyproposinganiterative
andincrementalprocessthatincludedearlyprototyping-somethingmanyorganizations
arejustnowdiscovering.
Todaywetakeadifferentviewofscriptedtesting.Anydevelopmentmethodologyalongthe
spectrumfromWaterfalltoRapidApplicationDevelopment(RAD)mayusescriptedtesting.
Wheneverrepeatability,objectivity,andauditabilityareimportant,scriptedtestingcanbeused.
Repeatabilitymeansthatthereisadefinitionofatest(fromdesignthroughtodetailed
procedure)atalevelofdetailsufficientforsomeoneotherthantheauthortoexecuteitinan
identicalway.Objectivitymeansthatthetestcreationdoesnotdependontheextrordinary
(nearmagical)skillofthepersoncreatingthetestbutisbasedonwellunderstoodtestdesign
principles.Auditabilityincludestraceabilityfromrequirements,design,andcodetothetest
casesandbackagain.Thisenablesformalmeasuresoftestingcoverage.
"Planyourwork,workyourplan."Nophrasesoepitomizesthescriptedtestingapproachas
doesthisone,andnodocumentsoepitomizesthescriptedtestingapproachasdoesIEEEStd
829-1998,the"IEEEStandardforSoftwareTestDocumentation."
Thisstandarddefineseightdocumentsthatcanbeusedinsoftwaretesting.Thesedocuments
are:
Testplan
Testdesignspecification
Testcasespecification
Testprocedurespecification
Testitemtransmittalreport
Testlog
Testincidentreport
Testsummaryreport
Figure12-2showstherelationshipsbetweenthesedocuments.Notethatthefirstfour
documentsthatdefinethetestplan,testdesigns,andtestcasesareallcreatedbeforethe
productisdevelopedandtheactualtestingisbegun.Thisisakeyideainscriptedtesting—plan
thetestsbasedontheformalsystemrequirements.
Figure12-2:TheIEEE829TestDocuments Curiously,theIEEE829
standardstates,"Thisstandardspecifiestheformandcontentofindividualtestdocuments.
Itdoesnotspecifytherequiredsetoftestdocuments."Inotherwords,thestandarddoes
notrequireyoutocreateanyofthedocumentsdescribed.Thatchoiceislefttoyouasa
tester,ortoyourorganization.But,thestandardrequiresthatifyouchoosetowriteatest
plan,testcasespecification,etc.,thatdocumentmustfollowtheIEEE829standard.
TheIEEE829standardliststheseadvantagesforitsuse:
"Astandardizedtestdocumentcanfacilitatecommunicationbyprovidingacommon
frameofreference.
Thecontentdefinitionofastandardizedtestdocumentcanserveasacompleteness
checklistfortheassociatedtestingprocess.
Astandardizedsetcanalsoprovideabaselinefortheevaluationofcurrenttest
documentationpractices.
Theuseofthesedocumentssignificantlyincreasesthemanageabilityoftesting.
Increasedmanageabilityresultsfromthegreatlyincreasedvisibilityofeachphaseof
thetestingprocess."
IEEE829DocumentDescriptions
TheIEEE829standarddefineseightdifferentdocuments.Eachdocumentiscomposedofa
numberofsections.
TestPlan
Thepurposeofthetestplanistodescribethescope,approach,resources,and
scheduleofthetestingactivities.Itdescribestheitems(components)andfeatures
(functionality,performance,security,usability,etc.)tobetested,taskstobe
performed,deliverables(tangibleworkproducts)tobecreated,testingresponsibilities,
schedules,andapprovalsrequired.Testplanscanbecreatedattheprojectlevel
(mastertestplan)oratsubsidiarylevels(unit,integration,system,acceptance,etc.).
Thetestplaniscomposedofthefollowingsections:
1. Testplanidentifier-Auniqueidentifiersothatthisdocumentcanbe
distinguishedfromallotherdocuments.
2. Introduction-Asummaryofthesoftwaretobetested.Abriefdescriptionand
historymaybeincludedtosetthecontext.Referencestootherrelevant
documentsusefulforunderstandingthetestplanareappropriate.Definitions
ofunfamiliartermsmaybeincluded.
3. Testitems-Identifiesthesoftwareitemsthataretobetested.Theword
"item"ispurposelyvague.Itisa"chunk"ofsoftwarethatistheobjectof
testing.
4. Featurestobetested-Identifiesthecharacteristicsoftheitemstobetested.
Theseincludefunctionality,performance,security,portability,usability,etc.
5. Featuresnottobetested-Identifiescharacteristicsoftheitemsthatwillnot
betestedandthereasonswhy.
6. Approach-Theoverallapproachtotestingthatwillensurethatallitemsand
theirfeatureswillbeadequatelytested.
7. Itempass/failcriteria-Thecriteriausedtodeterminewhethereachtestitem
haspassedorfailedtesting.
8. Suspensioncriteriaandresumptionrequirements-Theconditionsunderwhich
testingwillbesuspendedandthesubsequentconditionsunderwhichtesting
willberesumed.
9. Testdeliverables-Identifiesthedocumentsthatwillbecreatedasapartof
thetestingprocess.
10. Testingtasks-Identifiesthetasksnecessarytoperformthetesting.
11. Environmentalneeds-Specifiestheenvironmentrequiredtoperformthe
testingincludinghardware,software,communications,facilities,tools,people,
etc.
12. Responsibilities-Identifiesthepeople/groupsresponsibleforexecutingthe
testingtasks.
13. Staffingandtrainingneeds-Specifiesthenumberandtypesofpeople
requiredtoperformthetesting,includingtheskillsneeded.
14. Schedule-Definestheimportantkeymilestonesanddatesinthetesting
process.
15. Risksandcontingencies-Identifieshigh-riskassumptionsofthetestingplan.
Specifiespreventionandmitigationplansforeach.
16. Approvals-Specifiesthenamesandtitlesofeachpersonwhomustapprove
theplan.
TestDesignSpecification
Thepurposeofthetestdesignspecificationistoidentifyasetoffeaturestobetested
andtodescribeagroupoftestcasesthatwilladequatelytestthosefeatures.In
addition,refinementstotheapproachlistedinthetestplanmaybespecified.Thetest
designspecificationiscomposedofthefollowingsections:
1. Testdesignspecificationidentifier-Auniqueidentifiersothatthisdocument
canbedistinguishedfromallotherdocuments.
2. Featurestobetested-Identifiesthetestitemsandthefeaturesthatarethe
objectofthistestdesignspecification.
3. Approachrefinements-Specifiesthetesttechniquestobeusedforthistest
design.
4. Testidentification-Liststhetestcasesassociatedwiththistestdesign.
Providesauniqueidentifierandashortdescriptionforeachtestcase.
5. Featurepass/failcriteria-Thecriteriausedtodeterminewhethereach
featurehaspassedorfailedtesting.
TestCaseSpecification
Thepurposeofthetestcasespecificationistospecifyindetaileachtestcaselistedin
thetestdesignspecification.Thetestcasespecificationiscomposedofthefollowing
sections:
1. Testcasespecificationidentifier-Auniqueidentifiersothatthisdocumentcan
bedistinguishedfromallotherdocuments.
2. Testitems-Identifiestheitemsandfeaturestobetestedbythistestcase.
3. Inputspecifications-Specifieseachinputrequiredbythistestcase.
4. Outputspecifications-Specifieseachoutputexpectedafterexecutingthis
testcase.
5. Environmentalneeds-Anyspecialhardware,software,facilities,etc.required
fortheexecutionofthistestcasethatwerenotlistedinitsassociatedtest
designspecification.
6. Specialproceduralrequirements-Definesanyspecialsetup,execution,or
cleanupproceduresuniquetothistestcase.
7. Intercasedependencies-Listsanytestcasesthatmustbeexecutedpriorto
thistestcase.
TestProcedureSpecification
Thepurposeofthetestprocedurespecificationistospecifythestepsforexecutinga
testcaseandtheprocessfordeterminingwhetherthesoftwarepassedorfailedthe
test.Thetestprocedurespecificationiscomposedofthefollowingsections:
1. Testprocedurespecificationidentifier-Auniqueidentifiersothatthis
documentcanbedistinguishedfromallotherdocuments.
2. Purpose-Describesthepurposeofthetestprocedureanditscorresponding
testcases.
3. Specialrequirements-Listsanyspecialrequirementsfortheexecutionofthis
testprocedure.
4. Proceduresteps-Liststhestepsoftheprocedure.Possiblestepsinclude:
Setup,Start,Proceed,Measure,ShutDown,Restart,Stop,andWrapUp.
TestItemTransmittalReport(a.k.a.ReleaseNotes)
Thepurposeofthetestitemtransmittalreportistospecifythetestitemsbeing
providedfortesting.Thetestitemtransmittalreportiscomposedofthefollowing
sections:
1. Transmittalreportidentifier-Auniqueidentifiersothatthisdocumentcanbe
distinguishedfromallotherdocuments.
2. Transmitteditems-Liststheitemsbeingtransmittedfortestingincludingtheir
versionorrevisionlevel.
3. Location-Identifiesthelocationofthetransmitteditems.
4. Status-Describesthestatusoftheitemsbeingtransmitted.Includeany
deviationsfromtheitem'sspecifications.
5. Approvals-Specifiesthenamesandtitlesofallpersonswhomustapprove
thistransmittal.
TestLog
Thepurposeofthetestlogistoprovideachronologicalrecordaboutrelevantdetails
observedduringthetestexecution.Thetestlogiscomposedofthefollowingsections:
1. Testlogidentifier-Auniqueidentifiersothatthisdocumentcanbe
distinguishedfromallotherdocuments.
2. Description-Identifiestheitemsbeingtestedandtheenvironmentunder
whichthetestwasperformed.
3. Activityandevententries-Foreachevent,liststhebeginningandendingdate
andtime,abriefdescriptionofthetestexecution,theresultsofthetest,and
uniqueenvironmentalinformation,anomalouseventsobserved,andthe
incidentreportidentifierifanincidentwaslogged.
TestIncidentReport(a.k.a.BugReport)
Thepurposeofthetestincidentreportistodocumentanyeventobservedduring
testingthatrequiresfurtherinvestigation.Thetestincidentreportiscomposedofthe
followingsections:
1. Testincidentreportidentifier-Auniqueidentifiersothatthisdocumentcanbe
distinguishedfromallotherdocuments.
2. Summary-Summarizestheincident.
3. Incidentdescription-Describestheincidentintermsofinputs,expected
results,actualresults,environment,attemptstorepeat,etc.
4. Impact-Describestheimpactthisincidentwillhaveonothertestplans,test
designspecifications,testprocedures,andtestcasespecifications.Also
describes,ifknown,theimpactthisincidentwillhaveonfurthertesting.
TestSummaryReport
Thepurposeofthetestsummaryreportistosummarizetheresultsofthetesting
activitiesandtoprovideanevaluationbasedontheseresults.Thetestsummaryreport
iscomposedofthefollowingsections:
1. Testsummaryreportidentifier-Auniqueidentifier(imaginethat!)sothatthis
documentcanbedistinguishedfromallotherdocuments.
2. Summary-Summarizestheevaluationofthetestitems.
3. Variance-Reportsanyvariancesfromtheexpectedresults.
4. Comprehensiveassessment-Evaluatestheoverallcomprehensivenessofthe
testingprocessitselfagainstcriteriaspecifiedinthetestplan.
5. Summaryofresults-Summarizestheresultsofthetesting.Identifiesall
unresolvedincidents.
6. Evaluation-Providesanoverallevaluationofeachtestitemincludingits
limitations.
7. Summaryofactivities-Summarizesthemajortestingactivitiesbytaskand
resourceusage.
8. Approvals-Specifiesthenamesandtitlesofeachpersonwhomustapprove
thereport.
AdvantagesofScriptedTesting
1. Scriptedtestingprovidesadivisionoflabor—planning,testcasedesign,testcase
implementation,andtestcaseexecutionthatcanbeperformedbypeoplewith
specificskillsandatdifferenttimesduringthedevelopmentprocess.
2. Testdesigntechniquessuchasequivalenceclasspartitioning,boundaryvaluetesting,
controlflowtesting,pairwisetesting,etc.canbeintegratedintoaformaltesting
processdescriptionthatnotonlyguidesourtestingbutthatcouldalsobeusedto
auditforprocesscompliance.
3. Becausescriptedtestsarecreatedfromrequirements,design,andcode,all
importantattributesofthesystemwillbecoveredbytestsandthiscoveragecanbe
demonstrated.
4. Becausethetestcasescanbetracedbacktotheirrespectiverequirements,design,
andcode,coveragecanbeclearlydefinedandmeasured.
5. Becausethetestsaredocumented,theycanbeeasilyunderstoodandrepeated
whennecessarywithoutadditionaltestanalysisordesigneffort.
6. Becausethetestsaredefinedindetail,theyaremoreeasilyautomated.
7. Becausethetestsarecreatedearlyinthedevelopmentprocess,thismayfreeup
additionaltimeduringthecriticaltestexecutionperiod.
8. Insituationswhereagoodrequirementsspecificationislacking,thetestcases,atthe
endoftheproject,becomethedefactorequirementsspecification,includingthe
resultsthatdemonstratewhichrequirementswereactuallyfulfilledandwhichwere
not.
9. Scriptedtests,whenwrittentotheappropriatelevelofdetail,canberunbypeople
whowouldotherwisenotbeabletotestthesystembecauseoflackofdomain
knowledgeorlackoftestingknowledge.
10. Youmayhavespecialcontractualrequirementsthatcanonlybemetbyscripted
testing.
11. Theremaybecertainteststhatmustbeexecutedinjustthesameway,everytime,in
ordertoserveasakindofbenchmark.
12. Bycreatingthetestsearlyintheprojectwecandiscoverwhatwedon'tknow.
13. Bycreatingthetestsearlywecanfocusonthe"bigpicture."
InhisbookSoftwareSystemTestingandQualityAssurance,BorisBeizersummarizesinthis
way:
"Testingislikeplayingpool.There'srealpoolandkiddiepool.Inkiddiepool,youhitthe
ballsandwhateverpockettheyhappentofallinto,youclaimastheintendedpocket.It's
notmuchofagameandalthoughsuitabletoten-year-oldsit'shardlyachallenge.The
objectofrealpoolistospecifythepocketinadvance.Similarlyfortesting.There'sreal
testingandkiddietesting.Inkiddietesting,thetestersays,afterthefact,thatthe
observedoutcomewastheintendedoutcome.Inrealtestingtheoutcomeispredictedand
documentedbeforethetestisrun."
DisadvantagesofScriptedTesting
1. Scriptedtestingisverydependentonthequalityofthesystem'srequirements.Will
therequirementsreallybecomplete,consistent,unambiguous,andstableenoughas
thefoundationforscriptedtesting?Perhapsnot.
2. Scriptedtestingis,bydefinition,inflexible.Itfollowsthescript.If,whiletesting,we
seesomethingcurious,wenoteitinaTestIncidentReportbutwedonotpursueit.
Whynot?Becauseitisnotinthescripttodoso.Manyinterestingdefectscouldbe
missedwiththisapproach.
3. Scriptedtestingisoftenusedto"de-skill"thejoboftesting.Theapproachseemsto
be,"Teachatesteraskillortwoandsendthemofftodocumentmountainsoftests.
Thesheerbulkofthetestswillprobablyfindmostofthedefects."
Summary
"Planyourwork,workyourplan."LiketheWaterfallmodel,nophrasesoepitomizes
thescriptedtestingapproachasdoesthisone,andnodocumentsoepitomizesthe
scriptedtestingapproachasdoesIEEEStd829-1998,the"IEEEStandardfor
SoftwareTestDocumentation."
TheIEEEStandard829defineseightdocumentsthatcanbeusedinsoftwaretesting.
Thesedocumentsare:testplan,testdesignspecification,testcasespecification,test
procedurespecification,testitemtransmittalreport,testlog,testincidentreport,and
testsummaryreport.
Theadvantagesofscriptedtestingincludeformaldocumentation,coverage,and
traceability.
References
Beizer,Boris(1984).SoftwareSystemTestingandQualityAssurance.VanNostrand
Reinhold.
"IEEEStandardforSoftwareTestDocumentation,"IEEEStd829-1998.TheInstituteof
ElectricalandElectronicsEngineers,Inc.ISBN0-7381-1443-X
Royce,WinstonW."ManagingtheDevelopmentofLargeSoftwareSystems,"
Proceedingsofthe9thInternationalConferenceonSoftwareEngineering,Monterey,
CA,IEEEComputerSocietyPress,LosAlamitos,CA,1987.
http://www.ipd.bth.se/uodds/pd&agile/royce.pdf
Chapter13:ExploratoryTesting
Asshecontemplatedthesettingsun,itsdyingrayscastingthelastoftheirbrilliantpurple
lightonthered-goldwatersofthelake,Debbierealizedthatsheshouldneveragainbuy
hersunglassesfromaguyparkedbythesideoftheroad.
—MalindaLingwall
Introduction
Theterm"exploratorytesting,"coinedbyCemKanerinhisbookTestingComputerSoftware,
referstoanapproachtotestingthatisverydifferentfromscriptedtesting.Ratherthana
sequentialexaminationofrequirements,followedbythedesignanddocumentationoftest
cases,followedbytheexecutionofthosetestcases,exploratorytesting,asdefinedbyJames
Bach,is"simultaneouslearning,testdesign,andtestexecution."Thetesterdesignsand
executestestswhileexploringtheproduct.
InanarticleforStickyMinds.comentitled"ExploratoryTestingandthePlanningMyth,"Bach
wrote,"ExploratoryTesting,asIpracticeit,usuallyproceedsaccordingtoaconsciousplan.
Butnotarigorousplan...it'snotscriptedindetail."Jamesadds,"Rigorrequirescertaintyand
impliescompleteness,butIperformexploratorytestingpreciselybecausethere'ssomuchI
don'tknowabouttheproductandIknowmytestingcanneverbefullycomplete."James
continues,"Totheextentthatthenexttestwedoisinfluencedbytheresultofthelasttestwe
did,wearedoingexploratorytesting.Webecomemoreexploratorywhenwecan'ttellwhat
testsshouldberun,inadvanceofthetestcycle."
ExploratoryTesting
Totheextentthatthenexttestwedoisinfluencedbytheresultofthelasttestwedid,we
aredoingexploratorytesting.Webecomemoreexploratorywhenwecan'ttellwhattests
shouldberun,inadvanceofthetestcycle.
Inexploratorytesting,thetestercontrolsthedesignoftestcasesastheyareperformedrather
thandays,weeks,orevenmonthsbefore.Inaddition,theinformationthetestergainsfrom
executingasetofteststhenguidesthetesterindesigningandexecutingthenextsetoftests.
Notethisprocessiscalledexploratorytestingtodistinguishitfromadhoctestingwhich(bymy
definition,althoughothersmaydisagree)oftendenotessloppy,careless,unfocused,random,
andunskilledtesting.Anyone,nomatterwhattheirexperienceorskilllevel,candoadhoc
testing.Thatkindoftestingisineffectiveagainstallbutthemostdefect-riddensystems,and
eventhenmaynotfindasubstantialportionofthedefects.
Bachsuggeststhatintoday'stopsy-turvyworldofincomplete,rapidlychangingrequirements
andminimaltimefortesting,theclassicalsequentialapproachofTestAnalysisfollowedbyTest
DesignfollowedbyTestCreationfollowedbyTestExecutionislikeplayingthegameof
"TwentyQuestions"bywritingoutallthequestionsinadvance.Considerthefollowing
discussionfromatestingseminardiscussingexploratorytesting:
Instructor:Let'splayagamecalled"TwentyQuestions."Iamthinkingabout
somethingintheuniverse.I'mgivingyou,theclass,twentyquestionstoidentifywhat
I'mthinkingabout.Eachquestionmustbephrasedinawaythatitcanbeanswered
"Yes"or"No."(IfIletyouphrasethequestioninanyformyoucouldask"Whatareyou
thinkingabout"andwewouldthencallthisgame"OneQuestion.")Ready?Brian,let's
beginwithyou.
TwentyQuestions:TheGame
Agameinwhichonepersonthinksofsomethingandothersaskupto20questions
todeterminewhathasbeenselected.Thequestionsmustbeanswerable"Yes"or
"No."
Whenplayedwell,eachquestionisbasedonthepreviousquestionsandtheir
answers.Writingthequestionsoutinadvancepreventsusingtheknowledge
acquiredfromeachanswer.
Brian:Doesithaveanythingtodowithsoftwaretesting?
Instructor:No,thatwouldbetooeasy.
Michael:Isitlarge?
Instructor:No,it'snotlarge.
Rebecca:Isitananimal?
Instructor:No.
Rayanne:Isitaplant?
Instructor:Yes,itisaplant.
Henry:Isitatree?
Instructor:No,itisnotatree.
Sree:Isitbig?
Instructor:No,I'vealreadysaiditisnotlarge.
Eric:Isitgreen?
Instructor:Yes,itisgreen.
Cheryl:Doesithaveleaves?
Instructor:Yes,ithasleaves.
Galina:Isitanoutdoorplant?
Instructor:Yes,itgenerallygrowsoutdoors.
Jae:Isitafloweringplant?
Instructor:No,Idon'tbelievesobutI'mnotabotanist.
Melanie:Isitashrub?
Instructor:No.
Patrick:Isitacactus?
Instructor:No,itisnotacactus.
Angel:Isitacucumber?
Instructor:No,perhapsratherthanguessingindividualplantsitwouldbemore
effectivetoidentifycategories.
Sundari:Isitaweed?
Instructor:No,goodtrythough.
Lynn:Isitaperennial?
Instructor:No,Idon'tbelieveso.Ithinkitmustbereplantedeachyear.
Julie:Doesitgrowfrombulbs?
Instructor:No.
Michelle:Isitineveryone'syard?
Instructor:No,atleastit'snotinmine.
Kristie:Isitillegal?(Laughterintheclass)
Instructor:No,it'squitelegal.Well,we'vegonethroughtheclassonce.Brian,let'sgo
backtoyou.
Brian:Isitpoisonous?
Instructor:No,althoughmychildrenthinkso.
Michael:Isiteaten?
Instructor:Yes,itiseaten.
Rebecca:Isitlettuce?
Instructor:No,notlettuce.
Rayanne:Isitspinach?
Instructor:Yes,itisspinach.Verygood.
Howsuccessfulwouldwebeatthisgameifwehadtowriteoutallthequestionsinadvance?
Whenweplaythisgamewell,eachquestiondependsonthepreviousquestionsandtheir
answers.Soitisinexploratorytesting.Eachtestprovidesuswithinformationaboutthe
product.Wemayseeevidenceoftheproduct'scorrectness;wemayseeevidenceofits
defects.Wemayseethingsthatarecurious;we'renotsurewhattheymean,thingsthatwe
wonderaboutandwanttoexplorefurther.So,aswepracticeexploratorytesting,we
concurrentlylearntheproduct,designthetests,andexecutethesetests.
Description
Inhisclassictimemanagementbook,HowtoGetControlofYourTimeandYourLife,Alan
Lakeinsuggestsweshouldconstantlyaskourselves:WhatisthemostimportantthingIcando
withmytimerightnow?Exploratorytestersaskanequivalentquestion:Whatisthemost
importanttestIcanperformrightnow?
KeyQuestion
WhatisthemostimportanttestIcanperformrightnow?
Apossibleexploratorytestingprocessis:
Creatingaconjecture(amentalmodel)oftheproperfunctioningofthesystem
Designingoneormoreteststhatwoulddisprovetheconjecture
Executingthesetestsandobservingtheoutcomes
Evaluatingtheoutcomesagainsttheconjecture
Repeatingthisprocessuntiltheconjectureisprovedordisproved
Anotherprocessmightbesimplytoexploreandlearnbeforeformingconjecturesofproper
behavior.
Exploratorytestingcanbedonewithina"timebox,"anuninterruptedblockoftimedevotedto
testing.Thesearetypicallybetweensixtyand120minutesinlength.Thisislongenoughto
performsolidtestingbutshortenoughsothatthetesterdoesnotmentallywander.Inaddition,
atimeboxofthislengthistypicallyeasiertoschedule,easiertocontrol,andeasiertoreport.
Whenperforming"charteredexploratorytesting,"acharterisfirstcreatedtoguidethetester
withinthetimebox.Thischarterdefinesaclearmissionforthetestingsession.Thechartermay
define:
Whattotest
Whatdocuments(requirements,design,usermanual,etc.)areavailabletothetester
Whattacticstouse
Whatkindsofdefectstolookfor
Whatrisksareinvolved
Thischarterisaguidelinetobeused,notascripttobefollowed.Becauseofthisapproach,
exploratorytestingmakesfulluseoftheskillsoftesters.Bachwrites,"Themorewecanmake
testingintellectuallyrichandfluid,themorelikelywewillhitupontherighttestsattheright
time."
Key
Point Thecharterisaguidelinetobeused,notascripttobefollowed.
Chartersfocustheexploratorytester'seffortswithinthetimebox.Possiblechartersinclude:
Thoroughlyinvestigateaspecificsystemfunction
Defineandthenexaminethesystem'sworkflows
Identifyandverifyalltheclaimsmadeintheusermanual
Understandtheperformancecharacteristicsofthesoftware
Ensurethatallinputfieldsareproperlyvalidated
Forceallerrorconditionstoverifyeacherrormessage
Checkthedesignagainstuserinterfacestandards
Itispossibletoperformexploratorytestingwithoutacharter.Thisiscalled"freestyle
exploratorytesting."Inthisprocesstestersusetheirskillstotheutmostastheyconcurrently
learntheproductanddesignandexecutetests.
Exploratorytestersareskilledtesters.(Ofcourse,wewanttesterstobeskillednomatter
whattestingprocessweareusing!)Theexploratorytestingapproachrespectsthoseskillsand,
infact,dependsonthem.Goodexploratorytestersare:
Goodmodelers,abletocreatementalmodelsofthesystemanditsproperbehavior.
Carefulobservers,abletosee,hear,read,andcomprehend.
Skilledtestdesigners,abletochooseappropriatetestdesigntechniquesineach
situation.Bachemphasizes,"Anexploratorytesterisfirstandforemostatest
designer."
Abletoevaluateriskandletitguidetheirtesting.
Criticalthinkers,abletogeneratediverseideas,integratetheirobservations,skills,and
experiencestoconcurrentlyexploretheproduct,designthetests,andexecutethe
tests.
Carefulreporters,abletorigorouslyandeffectivelyreporttootherswhattheyhave
observed.
Selfmanaged,abletotaketheleadintestingratherthanexecuteaplandevisedby
others.
Notdistractedbytrivialmatters.
Testerswithouttheseskillscanstillperformusefulexploratorytestingiftheyareproperly
supervisedandcoached.
Ingeneral,processesthathaveweak,slow,ornonexistentfeedbackmechanismsoftendonot
performwell.Scriptedtestingisaprimeexampleofaslowfeedbackloop.Exploratorytesting
providesatightfeedbackloopbetweenbothtestdesignandtestexecution.Inaddition,it
providestightfeedbackbetweentestersanddevelopersregardingthequalityoftheproduct
beingtested.
AdvantagesofExploratoryTesting
1. Exploratorytestingisvaluableinsituationswherechoosingthenexttestcasetobe
runcannotbedeterminedinadvance,butshouldbebasedonprevioustestsandtheir
results.
2. Exploratorytestingisusefulwhenyouareaskedtoproviderapidfeedbackona
product'squalityonshortnotice,withlittletime,offthetopofyourhead,when
requirementsarevagueorevennonexistent,orearlyinthedevelopmentprocess
whenthesystemmaybeunstable.
3. Exploratorytestingisusefulwhen,onceadefectisdetected,wewanttoexplorethe
size,scope,andvariationsofthatdefecttoprovidebetterfeedbacktoour
developers.
4. Exploratorytestingisausefuladditiontoscriptedtestingwhenthescriptedtests
become"tired,"thatis,theyarenotdetectingmanyerrors.
DisadvantagesofExploratoryTesting
1. Exploratorytestinghasnoabilitytopreventdefects.Becausethedesignofscripted
testcasesbeginsduringtherequirementsgatheringanddesignphases,defectscan
beidentifiedandcorrectedearlier.
2. Ifyouarealreadysureexactlywhichtestsmustbeexecuted,andinwhichorder,
thereisnoneedtoexplore.Writeandthenexecutescriptedtests.
3. Ifyouarerequiredbycontract,rule,orregulationtousescriptedtestingthendoso.
Consideraddingexploratorytestsasacomplementarytechnique.
Summary
Exploratorytestingisdefinedas"simultaneouslearning,testdesign,andtest
execution."Thetesterdesignsandexecutestestswhileexploringtheproduct.
Inexploratorytesting,thetestercontrolsthedesignoftestcasesastheyare
performedratherthandays,weeks,orevenmonthsbefore.Inaddition,theinformation
thetestergainsfromexecutingasetofteststhenguidesthetesterindesigningand
executingthenextsetoftests.
Exploratorytestingisvitalwheneverchoosingthenexttestcasetoberuncannotbe
determinedinadvancebutshouldbechosenbasedonprevioustestsandtheirresults.
References
Bach,James."ExploratoryTestingandthePlanningMyth."
http://www.stickyminds.com/r.asp?F=DART_2359,19March2001.
Bach,James."ExploratoryTestingExplained."v.1.316April2003.
http://www.satisfice.com/articles/et-article.pdf
Kaner,Cem,JackFalk,andHungQ.Nguyen(1999).TestingComputerSoftware.John
Wiley&Sons.
Kaner,Cem,JamesBach,andBretPettichord(2002).LessonsLearnedinSoftware
Testing:AContext-DrivenApproach.JohnWiley&Sons.
Weinberg,GeraldM.(1975).AnIntroductiontoGeneralSystemsThinking.JohnWiley
&Sons.
Chapter14:TestPlanning
JohnStevensonlivesinVancouverwithhiswifeCindyandtheirtwokidsShawnand
Cassie,whoarethesecondcousinsofMaryShaw,whoismarriedtoRichardShaw,
whosegrandmotherwasStewartWerthington'shousekeeper,whosekidsDamienand
CharliewenttotheMansfieldChristianSchoolforBoyswithDannyRobinson,whose
sisterBertaRobinsonranoffwithChrisTanner,whoridesamotorcycleandgreaseshis
hairandtheirkidChristausedtogooutwithmypalTomSlipper,whoisthemain
characterofthisstory,butnotthenarrator'causeIam(Tommycouldn'twritetosavehis
life).
—EmmaDolan
Introduction
MortSahl,thebrilliantsocialcommentatorofthe1960s,oftenbeganhisactbydividingthe
worldintothe"rightwing,"the"leftwing,"andthe"socialdemocrats."Theprevioustwo
chaptershavedescribedtherightandleftwings.Nowit'stimeforthesocialdemocrats.
Scriptedtestingisbasedonthesequentialexaminationofrequirements,followedbythe
designanddocumentationoftestcases,followedbytheexecutionofthosetestcases.The
scriptedtester'smottois,"Planyourwork,workyourplan."Exploratorytestingisavery
differentparadigm.Ratherthanasequentialapproach,exploratorytestingemphasizes
concurrentproductlearning,testdesign,andtestexecution.Thetesterdesignsandexecutes
testswhileexploringtheproduct.
Technique
Planninghasbeendefinedsimplyas"figuringoutwhattodonext."Mostofuswouldadmitthat
tobeeffectiveandefficient,planningisimportant.Butwhenandhowshouldthatplanningbe
done?Scriptedtestingemphasizesthevalueofearlytestplanninganddesignasamethodof
detectingrequirementsanddesigndefectsbeforethecodeiswrittenandthesystemputinto
production.Exploratorytestingchallengestheideathattestsmustbedesignedsoveryearlyin
theproject,whenourknowledgeistypicallyatitsminimum.Inhisarticle,"ExploratoryTesting
andthePlanningMyth,"publishedonStickyMinds.com,JamesBachdiscussestheplanningof
playsthatareruninafootballgame.Heexamineswhentheplayscanorshouldbeplanned.
Let'sconsiderthissporttolearnmoreaboutplanning.
Butfirst,anapologyorexplanation.Inthischaptertheterm"football"referstothegameofthe
samenameasplayedintheUnitedStatesandCanadaandexported,withonlymarginal
success,totherestoftheworld."Football"doesnotrefertothatmarvelousgameplayed
world-widethatNorthAmericanscall"soccer."
ForMore
Information
TolearnmoreaboutthegameoffootballasplayedinNorthAmerica
seeww2.nfl.com/basics/history_basics.html
Whenarefootballplaysplanned?Ourfirstthoughtmightbeinthehuddlejustbeforetheplay
begins,butthefollowinglistshowsmorepossibilities:
PlannedFootballPlay
Beforethegamebegins-thefirstnplaysarechosenandexecutedwithoutregardto
theirsuccessorfailuretoevaluatebothteams'abilities
Beforeeachplay-inthehuddle,basedonanoverallgameplan,fieldposition,teams'
strengthsandweaknesses,andplayerskillsandexperience
Atthelineofscrimmage-dependingonthedefensivelineup
Atthestartofaplay-playaction-runorpassdependingonthedefense
Duringtheplay-runforyourlifewhenallelsehasfailed
AdaptivePlanning
Adaptiveplanningisnotanindustrystandardterm.Otherpossibletermsare:
Dynamic
Flexible
Just-In-Time
Responsive
Pliable
Progressive
Purposefulplanning
Wecoulddefinetheterms"classicalplanning"and"adaptiveplanning"toindicatethese
differentapproaches.Therelationshipbetweenclassicalplanningandadaptiveplanningin
footballis:
Table14-1:Classicalplanningvs.Adaptiveplanning.
ClassicalPlanning Beforethegamebegins(thefirsttenplaysarescripted)
AdaptivePlanning
Beforeeachplay(inthehuddle)
Atthelineofscrimmage(dependingonthedefensivesetup)
Atthestartofaplay(playaction-runorpass)
Duringtheplay(scramblewhenallelsehasfailed)
Let'snowleavefootballandconsidersoftwaretestplanning.(Whilewe'dratherstayandwatch
thegame,we'vegotsoftwaretotest.)
Table14-2:Classicaltestplanningvs.Exploratorytestplanning.
ClassicalTest
Planning
Asrequirements,analysis,design,andcodingarebeingdone—longbefore
systemisbuiltandthetestingcanbegin
AdaptiveTest
Planning
Chooseastrategy(dependingonourcurrentknowledge)
Beforeeachscreen/function/flowistobetested
Atthestartofanindividualtest(choosedifferentstrategies)
Duringthetest(asweobservethingswedon'texpectorunderstand)
Areasonableplanningheuristicwouldbe:
Weplanasmuchaswecan(basedontheknowledgeavailable),
Whenwecan(basedonthetimeandresourcesavailable),
Butnotbefore.
Asidefromthesenewlabels,haven'tgoodplannersalwaysdonethis?Isthisconceptreally
new?
AremarkablelittlebooksimplytitledPlanning,publishedbytheUnitedStatesMarineCorpsin
1997,describestheconceptsofadaptiveplanningindetail.
TheMarineCorpsdefinesplanningasencompassingtwobasicfunctions—"envisioninga
desiredfutureandarrangingaconfigurationofpotentialactionsintimeandspacethatwillallow
ustorealizethatfuture."But,totheMarines,planningisnotsomethingdoneearlywhichthen
becomescastinconcrete."Weshouldthinkofplanningasalearningprocess—asmental
preparationwhichimprovesourunderstandingofasituation."Plansarenotconcreteeither.
"Sinceplanningisanongoingprocess,itisbettertothinkofaplanasaninterimproductbased
ontheinformationandunderstandingknownatthemomentandalwayssubjecttorevisionas
newinformationandunderstandingemerge."
TheauthorsofPlanninglisttheseplanningpitfallstoavoid:
Attemptingtoforecasteventstoofarintothefuture.Byplanningwemayfoolourselves
intothinkingwearecontrolling.Thereisadifference.
Tryingtoplanintoomuchdetail.HelmuthvonMoltke,GermanArmyChiefofStaff
duringWorldWarIsaid,"Noplansurvivescontactwiththeenemy."Inexactlythat
sameway,notestplansurvivescontactwiththedefectsinthesystemundertest.
Institutionalizingplanningmethodsthatleadustoinflexibleorlockstepthinkinginwhich
bothplanningandplansbecomerigid.Ratherthan"Planyourworkandworkyourplan"
asourmantra,weshouldconstantly"Planourwork,workourplan,re-evaluateour
work,re-evaluateourplan."
Thinkingofaplanasanunalterablesolutiontoaproblem.Rather,itshouldbeviewed
asanopenarchitecturethatallowsustopursuemanyalternatives."Wewillrarely,if
ever,conductanevolutionexactlythewayitwasoriginallydeveloped."
Ignoringtheneedforafeedbackmechanismtoidentifyshortcomingsintheplanand
makenecessaryadjustments.Thisisacomponentofplanningwhichoftendoesnot
receiveadequateemphasis."Manyplansstopshortofidentifyingthesignals,
conditions,andfeedbackmechanismsthatwillindicatesuccessfulordysfunctional
execution."
Adaptiveplanning,asdescribedabove,acknowledgesanddealswiththesepitfalls.
ThefollowingexcerptfromPlanningsummarizestheseconceptswell:
"Planningisacontinuousprocessinvolvingtheongoingadjustmentofmeansandends.
Weshouldalsoviewplanningasanevolutionaryprocessinvolvingcontinuous
adjustmentandimprovement.Wecanthinkofplanningassolution-by-evolutionrather
thansolution-by-engineering.Weshouldgenerallynotviewplanningastryingtosolvea
probleminoneiterationbecausemost...problemsaretoocomplextobesolvedthat
way.Inmanycases,itismoreadvisabletofindaworkablesolutionquicklyand
improvethesolutionastimepermits.Whatmattersmostisnotgeneratingthebest
possibleplanbutachievingthebestpossibleresult.Likewise,weshouldseeeachplan
asanevolvingratherthanastaticdocument.Likeplanning,plansshouldbedynamic;a
staticplanisofnovaluetoanadaptiveorganizationinafluidsituation."
Summary
JamesBachasks,"Whatifit[theplan]comesintoexistenceonlymomentsbeforethe
testing?"Whymusttheplanbecreatedsoveryearlyintheproject,whenour
knowledgeistypicallyatitsminimum?
Inadaptiveplanningweplanasmuchaswecan(basedontheknowledgeavailable),
whenwecan(basedonthetimeandresourcesavailable),butnotbefore.
Key
Point
Theuseofscriptedtestingdoesnotprecludetheuseofexploratory
testing.Theuseofexploratorytestingdoesnotprecludetheuseof
scriptedtesting.Smarttestersusewhatevertoolintheirtoolboxis
required.
Sinceplanningisanongoingprocess,itisbettertothinkofaplanasaninterimproduct
basedontheinformationandunderstandingknownatthemomentandalwayssubject
torevisionasnewinformationandunderstandingemerge.
Theuseofscriptedtestingdoesnotprecludetheuseofexploratorytesting.Theuseof
exploratorytestingdoesnotprecludetheuseofscriptedtesting.AsRexBlackwrote,
"Smarttestersusewhatevertoolintheirtoolboxisrequired.Noparadigmshere.No
worldviewshere.Noscrewdriversvs.hammers.Let'sdowhatevermakessensegiven
theproblemathand."
Practice
1. Inwhatareascouldyouuseadaptiveplanningwhereyounowuseclassicalplanning?
Withwhatbenefit?Whatwouldthechallengesbe?Whowouldsupportyouinthisnew
process?Whowouldopposeyourefforts?Why?
2. InwhatmoviesabouttheMarineCorpsweretheprocessofplanningandthevalueof
plansemphasizedoveraction?Canyouexplainwhy?
References
Bach,James."ExploratoryTestingandthePlanningMyth."19March2001.
http://www.stickyminds.com/r.asp?F=DART_2359
Copeland,Lee."ExploratoryPlanning."3September2001.
http://www.stickyminds.com/r.asp?F=DART_2805
"IEEEStandardforSoftwareTestDocumentation,"IEEEStd829-1998.TheInstituteof
ElectricalandElectronicsEngineers,Inc.ISBN0-7381-1443-X
Planning.MCDP5.UnitedStatesMarineCorps.
https://www.doctrine.usmc.mil/mcdp/view/mpdpub5.pdf
TheBookends
Twoquestions,likebookends,frameoursoftwaretesting:
Wheredowestart?
Whendowestop?
Wheredowestarttesting?Ofalltheplacestolookfordefects,whereshouldwebegin?One
answeriswithadefecttaxonomy.Ataxonomyisaclassificationofthingsintoorderedgroups
orcategoriesthatindicatenatural,hierarchicalrelationships.Taxonomieshelpidentifythekinds
ofdefectsthatoftenoccurinsystems,guideyourtestingbygeneratingideas,andaudityour
testplanstodeterminethecoverageyouareobtainingwithyourtestcases.Intime,theycan
helpyouimproveyourdevelopmentprocess.
Andstopping.Howdowelogicallydecidewhenwehavetestedenoughandthesoftwareis
readyfordeliveryandinstallation?BorisBeizerhaswritten,"Thereisnosingle,valid,rational
criterionforstopping."Ifheiscorrect,howdowemakethatdecision?
Thenexttwochaptersaddresstheseimportantissues.
Chapter15:DefectTaxonomies
'Failure'wassimplynotawordthatwouldevercrossthelipsofMissEvelynDuberry,
mainlybecauseEvelyn,ahaughtysocialitewithfire-redhairandacoltishgate,could
pronounceneithertheletters'f'nor'r'asaresultofanunfortunatekissinggesturemade
manyyearsearliertowardherbelovedchildhoodparrot,Snippy.
—DavidKenyon
Introduction
Whatisataxonomy?Ataxonomyisaclassificationofthingsintoorderedgroupsorcategories
thatindicatenatural,hierarchicalrelationships.ThewordtaxonomyisderivedfromtwoGreek
roots:"taxis"meaningarrangementand"onoma"meaningname.Taxonomiesnotonlyfacilitate
theorderlystorageofinformation,theyfacilitateitsretrievalandthediscoveryofnewideas.
Taxonomieshelpyou:
Guideyourtestingbygeneratingideasfortestdesign
Audityourtestplanstodeterminethecoverageyourtestcasesareproviding
Understandyourdefects,theirtypesandseverities
Understandtheprocessyoucurrentlyusetoproducethosedefects(Alwaysremember,
yourcurrentprocessisfinelytunedtocreatethedefectsyou'recreating)
Improveyourdevelopmentprocess
Improveyourtestingprocess
Trainnewtestersregardingimportantareasthatdeservetesting
Explaintomanagementthecomplexitiesofsoftwaretesting
Key
Point
Ataxonomyisaclassificationofthingsintoorderedgroupsorcategoriesthat
indicatenatural,hierarchicalrelationships.
InhisbookTestingObject-OrientedSystems,RobertBinderdescribesa"faultmodel"asalist
oftypicaldefectsthatoccurinsystems.Anotherphrasetodescribesuchalistisadefect
taxonomy.Binderthendescribestwoapproachestotesting.Thefirstusesa"non-specificfault
model."Inotherwords,nodefecttaxonomyisused.Usingthisapproach,therequirementsand
specificationsguidethecreationofallofourtestcases.Thesecondapproachusesa"specific
faultmodel."Inthisapproach,ataxonomyofdefectsguidesthecreationoftestcases.Inother
words,wecreatetestcasestodiscoverfaultsliketheoneswehaveexperiencedbefore.We
willconsidertwolevelsoftaxonomies—projectlevelandsoftwaredefectlevel.Ofmost
importanceintestdesignarethesoftwaredefecttaxonomies.Butitwouldbefoolishtobegin
testdesignbeforeevaluatingtherisksassociatedwithboththeproductanditsdevelopment
process.
Notethatnoneofthetaxonomiespresentedbelowarecomplete.Eachcouldbeexpanded.
Eachissubjectivebasedontheexperienceofthosewhocreatedthetaxonomies.
ProjectLevelTaxonomies
SEIRiskIdentificationTaxonomy
TheSoftwareEngineeringInstitutehaspublisheda"Taxonomy-BasedRiskIdentification"that
canbeusedtoidentify,classify,andevaluatedifferentriskfactorsfoundinthedevelopmentof
softwaresystems.
Table15-1:TheSEITaxonomy-BasedRiskIdentificationtaxonomy.
Class Element Attribute
ProductEngineering
Requirements
Stability
Completeness
Clarity
Validity
Feasibility
Precedent
Scale
Design
Functionality
Difficulty
Interfaces
Performance
Testability
CodeandUnitTest
Feasibility
Testing
Coding/Implementation
IntegrationandTest
Environment
Product
System
EngineeringSpecialties
Maintainability
Reliability
Safety
Security
HumanFactors
Specifications
DevelopmentEnvironment
DevelopmentProcess
Formality
Suitability
ProcessControl
Familiarity
ProductControl
DevelopmentSystem
Capacity
Suitability
Usability
Familiarity
Reliability
SystemSupport
Deliverability
ManagementProcess
Planning
ProjectOrganization
ManagementExperience
ProgramInterfaces
ManagementMethods
Monitoring
PersonnelManagement
QualityAssurance
ConfigurationManagement
WorkEnvironment
QualityAttitude
Cooperation
Communication
Morale
Resources
Schedule
Staff
Budget
Facilities
ProgramConstraints
Contract
TypesofContract
Restrictions
Dependencies
ProgramInterfaces
Customer
AssociateContractors
Subcontractors
PrimeContractor
CorporateManagement
Vendors
Politics
If,asatester,youhadconcernswithsomeoftheseelementsandattributes,youwouldwant
tostresscertaintypesoftesting.Forexample:
Ifyouareconcernedabout: Youmightwanttoemphasize:
Thestabilityoftherequirements Formaltraceability
Incompleterequirements Exploratorytesting
Impreciselywrittenrequirements Decisiontablesand/orstate-transitiondiagrams
Difficultyinrealizingthedesign Controlflowtesting
Systemperformance Performancetesting
Lackofunittesting Additionaltestingresources
Usabilityproblems Usabilitytesting
ISO9126QualityCharacteristicsTaxonomy
TheISO9126Standard"SoftwareProductEvaluation—QualityCharacteristicsandGuidelines"
focusesonmeasuringthequalityofsoftwaresystems.Thisinternationalstandarddefines
softwareproductqualityintermsofsixmajorcharacteristicsandtwenty-onesubcharacteristics
anddefinesaprocesstoevaluateeachofthese.Thistaxonomyofqualityattributesis:
Table15-2:TheISO9126QualityCharacteristicstaxonomy.
QualityCharacteristic Subcharacteristic
Suitability
Functionality
(Aretherequiredfunctionsavailableinthesoftware?)
Accuracy
Interoperability
Security
Reliability
(Howreliableisthesoftware?)
Maturity
Faulttolerance
Recoverability
Usability
(Isthesoftwareeasytouse?)
Understandability
Learnability
Operability
Attractiveness
Efficiency
(Howefficientisthesoftware?)
Timebehavior
Resourcebehavior
Maintainability
(Howeasyisittomodifythesoftware?)
Analyzability
Changeability
Stability
Testability
Portability
(Howeasyisittotransferthesoftwaretoanotheroperating
environment?)
Adaptability
Installability
Coexistence
Replaceability
Eachofthesecharacteristicsandsubcharacteristicssuggestareasofriskandthusareasfor
whichtestsmightbecreated.Anevaluationoftheimportanceofthesecharacteristicsshould
beundertakenfirstsothattheappropriateleveloftestingisperformed.Asimilar"ifyouare
concernedabout/youmightwanttoemphasize"processcouldbeusedbasedontheISO
9126taxonomy.
Theseprojectleveltaxonomiescanbeusedtoguideourtestingatastrategiclevel.Forhelpin
softwaretestdesignweusesoftwaredefecttaxonomies.
SoftwareDefectTaxonomies
Insoftwaretestdesignweareprimarilyconcernedwithtaxonomiesofdefects,orderedlistsof
commondefectsweexpecttoencounterinourtesting.
Beizer'sTaxonomy
OneofthefirstdefecttaxonomieswasdefinedbyBorisBeizerinSoftwareTestingTechniques.
Itdefinesafour-levelclassificationofsoftwaredefects.Thetoptwolevelsareshownhere.
Table15-3:AportionofBeizer'sBugTaxonomy.
1xxx Requirements
11xx Requirementsincorrect
12xx Requirementslogic
13xx Requirements,completeness
14xx Verifiability
15xx Presentation,documentation
16xx Requirementschanges
2xxx FeaturesAndFunctionality
21xx Feature/functioncorrectness
22xx Featurecompleteness
23xx Functionalcasecompleteness
24xx Domainbugs
25xx Usermessagesanddiagnostics
26xx Exceptionconditionsmishandled
3xxx StructuralBugs
31xx Controlflowandsequencing
32xx Processing
4xxx Data
41xx Datadefinitionandstructure
42xx Dataaccessandhandling
5xxx ImplementationAndCoding
51xx Codingandtypographical
52xx Styleandstandardsviolations
53xx Documentation
6xxx Integration
61xx Internalinterfaces
62XX Externalinterfaces,timing,throughput
7XXX SystemAndSoftwareArchitecture
71XX O/Scallanduse
72XX Softwarearchitecture
73XX Recoveryandaccountability
74XX Performance
75XX Incorrectdiagnostics,exceptions
76XX Partitions,overlays
77XX Sysgen,environment
8XXX TestDefinitionAndExecution
81XX Testdesignbugs
82XX Testexecutionbugs
83XX Testdocumentation
84XX Testcasecompleteness
Evenconsideringonlythetoptwolevels,itisquiteextensive.Allfourlevelsofthetaxonomy
constituteafine-grainedframeworkwithwhichtocategorizedefects.
Attheoutset,adefecttaxonomyactsasachecklist,remindingthetestersothatnodefect
typesareforgotten.Later,thetaxonomycanbeusedasaframeworktorecorddefectdata.
Subsequentanalysisofthisdatacanhelpanorganizationunderstandthetypesofdefectsit
creates,howmany(intermsofrawnumbersandpercentages),andhowandwhythese
defectsoccur.Then,whenfacedwithtoomanythingstotestandnotenoughtime,youwill
havedatathatenablesyoutomakerisk-based,ratherthanrandom,testdesigndecisions.In
additiontotaxonomiesthatsuggestthetypesofdefectsthatmayoccur,alwaysevaluatethe
impactonthecustomerandultimatelyonyourorganizationiftheydooccur.Defectsthathave
lowimpactmaynotbeworthtrackingdownandrepairing.
Kaner,Falk,andNguyen'sTaxonomy
ThebookTestingComputerSoftwarecontainsadetailedtaxonomyconsistingofover400
typesofdefects.Onlyafewexcerptsfromthistaxonomyarelistedhere.
Table15-4:AportionofthedefecttaxonomyfromTestingComputerSoftware.
UserInterfaceErrors
Functionality
Communication
Commandstructure
Missingcommands
Performance
Output
ErrorHandling
Errorprevention
Errordetection
Errorrecovery
Boundary-RelatedErrors
Numericboundaries
Boundariesinspace,time
Boundariesinloops
CalculationErrors
Outdatedconstants
Calculationerrors
Wrongoperationorder
Overflowandunderflow
InitialAndLaterStates
Failuretosetadataitemto0
Failuretoinitializealoopcontrolvariable
Failuretoclearastring
Failuretoreinitialize
ControlFlowErrors
Programrunsamok
Programstops
Loops
IF,THEN,ELSEormaybenot
ErrorsInHandlingOrInterpreting
Data
Datatypeerrors
Parameterlistvariablesoutoforderormissing
Outdatedcopiesofdata
Wrongvaluefromatable
Wrongmaskinbitfield
RaceConditions
Assumingoneeventalwaysfinishesbefore
another
Assumingthatinputwillnotoccurinaspecific
interval
Taskstartsbeforeitsprerequisitesaremet
LoadConditions Requiredresourcenotavailable
Doesn'treturnunusedmemory
Hardware Deviceunavailable
Unexpectedendoffile
SourceAndVersionControl Oldbugsmysteriouslyreappear
Sourcedoesn'tmatchbinary
Documentation None
TestingErrors
Failuretonoticeaproblem
Failuretoexecuteaplannedtest
Failuretousethemostpromisingtestcases
Failuretofileadefectreport
Binder'sObject-OrientedTaxonomy
RobertBindernotesthatmanydefectsintheobject-oriented(OO)paradigmareproblems
usingencapsulation,inheritance,polymorphism,messagesequencing,andstate-transitions.
Thisistobeexpectedfortworeasons.First,thesearecornerstoneconceptsinOO.They
formthebasisoftheparadigmandthuswillbeusedextensively.Second,thesebasicconcepts
areverydifferentfromtheproceduralparadigm.DesignersandprogrammersnewtoOO
wouldbeexpectedtofindthemforeignideas.AsmallportionofBinder'sOOtaxonomyisgiven
heretogiveyouasenseofitscontents:
Table15-5:AportionofBinder'sMethodScopeFaultTaxonomy.
MethodScope Fault
Requirements Requirementomission
Design
Abstraction LowCohesion
Refinement Featureoverridemissing
Featuredeletemissing
Encapsulation
Nakedaccess
Overuseoffriend
Responsibilities Incorrectalgorithm
Invariantviolation
Exceptions Exceptionnotcaught
Table15-6:AportionofBinder'sClassScopeFaultTaxonomy.
ClassScope Fault
Design
Abstraction Associationmissingorincorrect
Inheritanceloops
Refinement Wrongfeatureinherited
Incorrectmultipleinheritance
Encapsulation Publicinterfacenotviaclassmethods
Implicitclass-to-classcommunication
Modularity Objectnotused
Excessivelylargenumberofmethods
Implementation Incorrectconstructor
Notehowthistaxonomycouldbeusedtoguidebothinspectionsandtestcasedesign.Binder
alsoreferencesspecificdefecttaxonomiesforC++,Java,andSmalltalk.
Whittaker's"HowtoBreakSoftware"Taxonomy
JamesWhittaker'sbookHowtoBreakSoftwareisatester'sdelight.Proponentsofexploratory
testingexhortusto"explore."Whittakertellsusspecifically"wheretoexplore."Notonlydoes
heidentifyareasinwhichfaultstendtooccur,hedefinesspecifictestingattackstolocate
thesefaults.Onlyasmallportionofhistaxonomyispresented:
Table15-7:AportionofWhittaker'sFaultTaxonomy.
FaultType Attack
Inputsandoutputs
Forceallerrormessagestooccur
Forcetheestablishingofdefaultvalues
Overflowinputbuffers
Dataandcomputation Forcethedatastructuretostoretoofewortoomanyvalues
Forcecomputationresultstobetoolargeortoosmall
Filesysteminterface Fillthefilesystemtoitscapacity
Damagethemedia
Softwareinterfaces Causeallerrorhandlingcodetoexecute
Causeallexceptionstofire
Vijayaraghavan'seCommerceTaxonomy
Beizer's,Kaner's,andWhittaker'staxonomiescatalogdefectsthatcanoccurinanysystem.
Binder'sfocusesoncommondefectsinobject-orientedsystems.GiriVijayaraghavanhas
chosenamuchnarrowerfocus—theeCommerceshoppingcart.Usingthisfamiliarmetaphor,
aneCommerceWebsitekeepstrackofthestateofauserwhileshopping.Vijayaraghavanhas
investigatedthemanywaysshoppingcartscanfail.Hewrites,"Wedevelopedthelistof
shoppingcartfailurestostudytheuseoftheoutlineasatestideagenerator."Thisisoneofthe
primeusesofanydefecttaxonomy.Histaxonomylistsoversixtyhigh-leveldefectcategories,
someofwhicharelistedhere:
Performance
Reliability
Softwareupgrades
Userinterfaceusability
Maintainability
Conformance
Stability
Operability
Faulttolerance
Accuracy
Internationalization
Recoverability
Capacityplanning
Third-partysoftwarefailure
Memoryleaks
Browserproblems
Systemsecurity
Clientprivacy
Aftergeneratingthelistheconcludes,"Wethinkthelistisasufficientlybroadandwell-
researchedcollectionthatitcanbeusedasastartingpointfortestingotherapplications."His
assertioniscertainlycorrect.
AFinalObservation
Notethateachofthesetaxonomiesisalistofpossibledefectswithoutanyguidanceregarding
theprobabilitythatthesewilloccurinyoursystemsandwithoutanysuggestionofthelossyour
organizationwouldincurifthesedefectsdidoccur.Taxonomiesareusefulstartingpointsforour
testingbuttheyarecertainlynotacompleteanswertothequestionofwheretostarttesting.
YourTaxonomy
Nowthatwehaveexaminedanumberofdifferentdefecttaxonomies,thequestionarises—
whichisthecorrectoneforyou?Thetaxonomythatismostusefulisyourtaxonomy,theone
youcreatefromyourexperiencewithinyourorganization.Oftentheplacetostartiswithan
existingtaxonomy.Thenmodifyittomoreaccuratelyreflectyourparticularsituationintermsof
defects,theirfrequencyofoccurrence,andthelossyouwouldincurifthesedefectswerenot
detectedandrepaired.
Key
Point Thetaxonomythatismostusefulisyourtaxonomy,theoneyoucreate.
Justasinotherdisciplineslikebiology,psychology,andmedicine,thereisnoone,single,right
waytocategorize,thereisnoonerightsoftwaredefecttaxonomy.Categoriesmaybefuzzy
andoverlap.Defectsmaynotcorrespondtojustonecategory.Ourlistmaynotbecomplete,
correct,orconsistent.Thatmattersverylittle.Whatmattersisthatwearecollecting,analyzing,
andcategorizingourpastexperienceandfeedingitforwardtoimproveourabilitytodetect
defects.Taxonomiesaremerelymodelsand,asGeorgeBox,thefamousstatistician,reminds
us,"Allmodelsarewrong;somemodelsareuseful."
Tocreateyourowntaxonomy,firststartwithalistofkeyconcepts.Don'tworryifyourlist
becomeslong.Thatmaybejustfine.Makesuretheitemsinyourtaxonomyareshort,
descriptivephrases.Keepyourusers(that'syouandothertestersinyourorganization)inmind.
Usetermsthatarecommonforthem.Later,lookfornaturalhierarchicalrelationshipsbetween
itemsinthetaxonomy.Combinetheseintoamajorcategorywithsubcategoriesunderneath.
Trynottoduplicateoroverlapcategoriesandsubcategories.Continuetoaddnewcategories
astheyarediscovered.Revisethecategoriesandsubcategorieswhennewitemsdon'tseem
tofitwell.Shareyourtaxonomywithothersandsolicittheirfeedback.Youareonyourwaytoa
taxonomythatwillcontributetoyourtestingsuccess.
Summary
Taxonomieshelpyou:
Guideyourtestingbygeneratingideasfortestcasedesign
Audityourtestplanstodeterminethecoverageyourtestcasesareproviding
Understandyourdefects,theirtypesandseverities
Understandtheprocessyoucurrentlyusetoproducethosedefects(Always
remember,yourcurrentprocessisfinelytunedtocreatethedefectsyou're
creating)
Improveyourdevelopmentprocess
Improveyourtestingprocess
Trainnewtestersregardingimportantareasthatdeservetesting
Explaintomanagementthecomplexitiesofsoftwaretesting
Testingcanbedonewithouttheuseoftaxonomies(nonspecificfaultmodel)orwitha
taxonomy(specificfaultmodel)toguidethedesignoftestcases.
Taxonomiescanbecreatedatanumberoflevels:genericsoftwaresystem,
developmentparadigm,typeofapplication,anduserinterfacemetaphor.
References
Beizer,Boris(1990).SoftwareTestingTechniques(SecondEdition).VanNostrand
Reinhold.
Binder,RobertV.(2000).TestingObject-OrientedSystems:Models,Patterns,and
Tools.Addison-Wesley.
Carr,MarvinJ.,etal.(1993)"Taxonomy-BasedRiskIdentification."TechnicalReport
CMU/SEI-93-TR-6,ESC-TR-93-183,June1993.
http://www.sei.cmu.edu/pub/documents/93.reports/pdf/tr06.93.pdf
ISO(1991).ISO/IECStandard9126-1.SoftwareEngineering-ProductQuality-Part1:
QualityModel,ISOCopyrightOffice,Geneva,June2001.
Kaner,Cem,JackFalkandHungQuocNguyen(1999).TestingComputerSoftware
(SecondEdition).JohnWiley&Sons.
Whittaker,JamesA.(2003).HowtoBreakSoftware:APracticalGuidetoTesting.
AddisonWesley.
Vijayaraghavan,GiriandCemKaner."Bugsinyourshoppingcart:ATaxonomy."
http://www.testingeducation.org/articles/BISC_Final.pdf
Chapter16:WhentoStopTesting
Theballerinastoodonpoint,hertoescurledlikeshrimp,notdeep-friedshrimpbecause,
asbrittleastheyare,theywouldhavecrackedunderthepressure,buttenderebi-kind-of-
shrimp,pinkandlusciousasaTokyosunset,wonderingifherloverwasintheGinza,
wooingthegeishawitheyesreminiscentofroe,whichshelikedbetterthanebianyway.
—BrianTacang
TheBananaPrinciple
InhisclassicbookAnIntroductiontoGeneralSystemsThinking,GeraldWeinbergintroduces
ustothe"BananaPrinciple."Alittleboycomeshomefromschoolandhismotherasks,"What
didyoulearninschooltoday?"Theboyresponds,"Todaywelearnedhowtospell'banana'but
wedidn'tlearnwhentostop."Inthisbookwehavelearnedhowtodesigneffectiveandefficient
testcases,buthowdoweknowwhentostop?Howdoweknowwehavedoneenough
testing?
WhentoStop
InTheCompleteGuidetoSoftwareTesting,BillHetzelwroteregardingsystemtesting,
"Testingendswhenwehavemeasuredsystemcapabilitiesandcorrectedenoughofthe
problemstohaveconfidencethatwearereadytoruntheacceptancetest."Thephrases
"correctedenough"and"haveconfidence,"whilecertainlycorrect,arevague.
Regardingstopping,BorisBeizerhaswritten,"Thereisnosingle,valid,rationalcriterionfor
stopping.Furthermore,givenanysetofapplicablecriteria,howexactlyeachisweighted
dependsverymuchupontheproduct,theenvironment,thecultureandtheattitudetorisk."
Again,notmuchhelpinknowingwhentostoptesting.
EventhoughBeizersaysthereisnosinglecriterionforstopping,manyorganizationshave
chosenoneanyway.Thefivebasiccriteriaoftenusedtodecidewhentostoptestingare:
Youhavemetpreviouslydefinedcoveragegoals
Thedefectdiscoveryratehasdroppedbelowapreviouslydefinedthreshold
Themarginalcostoffindingthe"next"defectexceedstheexpectedlossfromthat
defect
Theprojectteamreachesconsensusthatitisappropriatetoreleasetheproduct
Thebosssays,"Shipit!"
CoverageGoals
Coverageisameasureofhowmuchhasbeentestedcomparedwithhowmuchisavailableto
test.Coveragecanbedefinedatthecodelevelwithmetricssuchasstatementcoverage,
branchcoverage,andpathcoverage.Attheintegrationlevel,coveragecanbedefinedinterms
ofAPIstestedorAPI/parametercombinationstested.Atthesystemlevel,coveragecanbe
definedintermsoffunctionstested,usecases(oruserstories)tested,orusecasescenarios
(mainpathplusalltheexceptionpaths)tested.Onceenoughtestcaseshavebeenexecutedto
meetthepreviouslydefinedcoveragegoals,weare,bydefinition,finishedtesting.For
example,wecoulddefineaproject'sstoppingcriteriaas:
100%statementcoverage
90%usecasescenariocoverage
Whenthisnumberoftestspass,wearefinishedtesting.(Ofcourse,therearemanyother
combinationsoffactorsthatcouldbeusedasstoppingcriteria.)Notalltestersapproveofthis
approach.GlenfordMyersbelievesthatthismethodishighlycounterproductive.Hebelieves
thatbecausehumanbeingsareverygoaloriented,thiscriterioncouldsubconsciously
encouragetesterstowritetestcasesthathavealowprobabilityofdetectingdefectsbutdo
meetthecoveragecriteria.Hebelievesthatmorespecificcriteriasuchasasetofteststhat
coverallboundaryvalues,state-transitionevents,decisiontablerules,etc.aresuperior.
DefectDiscoveryRate
Anotherapproachistousethedefectdiscoveryrateasthecriteriaforstopping.Eachweek(or
othershortperiodoftime)wecountthenumberofdefectsdiscovered.Typically,thenumberof
defectsfoundeachweekresemblesthecurveinFigure16-1.Oncethediscoveryrateisless
thanacertainpreviouslyselectedthreshold,wearefinishedtesting.Forexample,ifwehadset
thethresholdatthreedefects/week,wewouldstoptestingafterweek18.
Figure16-1:DefectDiscoveryRate
Whilethisapproachappealstoourintuition,weshouldconsiderwhatothersituationswould
produceacurvelikethis—creationofadditional,butlesseffectivetestcases;testerson
vacation;"killer"defectsthatstillexistinthesoftwarebutthathideverywell.Thisisonereason
whyBeizersuggestsnotdependingononlyonestoppingcriterion.
MarginalCost
Inmanufacturing,wedefine"marginalcost"asthecostassociatedwithoneadditionalunitof
production.Ifwe'remaking1,000donuts,whatistheadditionalcostofmakingonemore?Not
verymuch.Inmanufacturing,themarginalcosttypicallydecreasesasthenumberofunitsmade
increases.Insoftwaretesting,however,justtheoppositeoccurs.Findingthefirstfewdefects
isrelativelysimpleandinexpensive.Findingeachadditionaldefectbecomesmoreandmore
timeconsumingandcostlybecausethesedefectsareveryadeptathidingfromourtestcases.
Thusthecostoffindingthe"next"defectincreases.Atsomepointthecostoffindingthat
defectexceedsthelossourorganizationwouldincurifweshippedtheproductwiththatdefect.
Clearly,itis(past)timetostoptesting.
Noteverysystemshouldusethiscriterion.Systemsthatrequirehighreliabilitysuchas
weaponssystems,medicaldevices,industrialcontrols,andothersafety-criticalsystemsmay
requireadditionaltestingbecauseoftheirriskandsubsequentlossshouldafailureoccur.
TeamConsensus
Basedonvariousfactorsincludingtechnical,financial,political,andjust"gutfeelings,"the
projectteam(managers,developers,testers,marketing,sales,qualityassurance,etc.)decide
thatthebenefitsofdeliveringthesoftwarenowoutweighthepotentialliabilitiesandreach
consensusthattheproductshouldbereleased.
ShipIt!
Formanyofus,thiswillbetheonlystrategywewilleverpersonallyexperience.It'softenvery
dishearteningfortesters,especiallyaftermanyarduoushoursoftesting,andwithasure
knowledgethatmanydefectsarestillhidinginthesoftware,tobetold"Shipit!"Whattesters
mustrememberisthattheremaybeveryreasonableandlogicalreasonsforshippingthe
productbeforewe,astesters,thinkitisready.Intoday'sfast-pacedmarketeconomy,often
the"firsttomarket"winsasubstantialmarketshare.Eveniftheproductislessthanperfect,it
maystillsatisfytheneedsofmanyusersandbringsignificantprofitstoourorganization;profits
thatmightbelostifwedelayedshipment.
Someofthecriteriathatshouldbeconsideredinmakingthisdecisionarethecomplexityofthe
productitself,thecomplexityofthetechnologiesusedtoimplementitandourskillsand
experienceinusingthosetechnologies,theorganization'scultureandtheimportanceofrisk
aversioninourorganization,andtheenvironmentwithinwhichthesystemwilloperateincluding
thefinancialandlegalexposurewehaveifthesystemfails.
Asatester,youmaybefrustratedbythe"ShipIt"decision.Remember,ourroleastestersis
toinformmanagementoftherisksofshippingtheproduct.Theroleofyourorganization's
marketingandsalesgroupsshouldbetoinformmanagementofthebenefitsofshippingthe
product.Withthisinformation,bothpositiveandnegative,projectmanagerscanmake
informed,rationaldecisions.
SomeConcludingAdvice
Lesson185inLessonsLearnedinSoftwareTestingstates:
"Becausetestingisaninformationgatheringprocess,youcanstopwhenyou've
gatheredenoughinformation.Youcouldstopafteryou'vefoundeverybug,butitwould
takeinfinitetestingtoknowthatyou'vefoundeverybug,sothatwon'twork.Instead,
youshouldstopwhenyoureasonablybelievethattheprobabilityislowthattheproduct
stillhasimportantundiscoveredproblems.
"Severalfactorsareinvolvedindecidingthattestingisgoodenough(lowenough
chanceofundiscoveredsignificantbugs):
Youareawareofthekindsofproblemsthatwouldbeimportanttofind,ifthey
existed.
Youareawareofhowdifferentpartsoftheproductcouldexhibitimportant
problems.
Youhaveexaminedtheproducttoadegreeandinamannercommensurate
withtheserisks.
Yourteststrategywasreasonablydiversifiedtoguardagainsttunnelvision.
Youusedeveryresourceavailablefortesting.
Youmeteverytestingprocessstandardthatyourclientswouldexpectyouto
meet.
Youexpressedyourteststrategy,testresults,andqualityassessmentsas
clearlyasyoucould."
Summary
Regardingstopping,BorisBeizerhaswritten,"Thereisnosingle,valid,rationalcriterion
forstopping.Furthermore,givenanysetofapplicablecriteria,howexactlyeachis
weighteddependsverymuchupontheproduct,theenvironment,thecultureandthe
attitudetorisk."
Thefivebasiccriteriaoftenusedtodecidewhentostoptestingare:
Youhavemetpreviouslydefinedcoveragegoals
Thedefectdiscoveryratehasdroppedbelowapreviouslydefinedthreshold
Themarginalcostoffindingthe"next"defectexceedstheexpectedlossfrom
thatdefect
Theprojectteamreachesconsensusthatitisappropriatetoreleasethe
product
Thebosssays,"Shipit!"
References
Hetzel,Bill(1998).TheCompleteGuidetoSoftwareTesting(SecondEdition).John
Wiley&Sons.
Kaner,Cem,JamesBach,andBretPettichord(2002).LessonsLearnedinSoftware
Testing:AContext-DrivenApproach.JohnWiley&Sons.
Myers,Glenford(1979).TheArtofSoftwareTesting.JohnWiley&Sons.
Weinberg,GeraldM.(1975).AnIntroductiontoGeneralSystemsThinking.JohnWiley
&Sons.
YourTestingToolbox
MyoldestsonShawnisaglazier—heinstallsglass,mirrors,showerdoors,etc.Heisanartist
inglass.Asafather,Idecideditwouldbegoodtoknowwhatmysondoesforaliving,soI
rodewithhiminhistruckforafewhourswatchinghimwork.
Atthefirstjobsitehepulledoutaclipboardwithaworkorderthattoldhimwhatwasneeded.
Hehoppedoutandwalkedaroundtothebackofthetruck.There,hegrabbedhistoolbucket
(anoldfive-gallonpaintbucket)androotedaroundthroughit.Hepulledoutsometools,walked
uptothehouse,didhismagic,camebacktothetruck,putthetoolsinthebucket,andaway
wewent.Atthesecondjobsiteherepeatedtheprocess.Onceagain,hepulledoutthe
clipboard,hoppedout,walkedaroundtothebackofthetruck,grabbedhistoolbucket,and
rootedaroundthroughit.Hepulledoutsometools,butdifferenttoolsthistime,walkedupto
thehouse,didhismagic,camebacktothetruck,putthetoolsinthebucket,andawaywe
went.Aswewentfromjobtojobitoccurredtomethatallgoodcraftspeople,including
softwaretesters,needabucketoftools.Inaddition,goodcraftspeopleknowwhichtooltouse
inwhichsituation.Myintentinwritingthisbookwastohelpputmoretoolsinyourpersonal
testingtoolbucketandtohelpyouknowwhichtooltouseinwhichsituation.Remember,not
everytoolneedstobeusedeverytime.
Now,it'suptoyou.Thenextlevelofskillcomeswithpractice.FamouseducatorBenjamin
Bloomcreatedataxonomyforcategorizinglevelsofcompetencyinschoolsettings.Thefirst
threelevelsare:
Knowledge
Comprehension
Application
Thisbookhasfocusedonknowledgeandcomprehension.The"application"isuptoyou.
Bestwishesinyourtesting...
References
Bloom,BenjaminS.(1969).TaxonomyofEducationalObjectives:TheClassificationof
EducationalGoals.LongmanGroup.
AppendixA:Brown&DonaldsonCaseStudy
Introduction
Brown&Donaldson(B&D)isafictitiousonlinebrokeragefirmthatyoucanusetopracticethe
testdesigntechniquespresentedinthisbook.B&DwasoriginallycreatedforSoftwareQuality
Engineering'sWeb/eBusinessTestingcourse(seehttp://www.sqe.com).TheactualB&DWeb
siteisfoundathttp://bdonline.sqe.com.AnyresemblancetoanyactualonlinebrokerageWeb
siteispurelycoincidental.
Login
TheLoginpageisthegatewayintotheB&Dsite.Itrequiresalegitimateusernameand
password.
MarketNews
TheMarketNewspageisthemainpageoftheB&Dsite.Itcontainsnavigationbuttonsonthe
leftsideofthepage,stockperformancechartsatthetop,andnewsstoriesofinterestto
B&D'sinvestors.
Trade
TheTradepageallowsaB&Dclienttobuyandsellstocks.Itcontainsabuy/sellbutton,atext
boxforthestocktickersymbol,atextboxforthenumberofsharestobeboughtorsold
(quantity),andboxesindicatingthetypeoftrade.
SymbolLookup
TheSymbolLookuppageisreachedfromtheTradepage.ItisusedwhentheB&Dclientis
unsureofthestocktickersymbolandmustlookitup.Itcontainsonefieldwherethefirstfew
charactersoftheorganization'snameareentered.
LookupResults
TheLookupResultspageistheresultofthepreviousSymbolLookuppage.Itdisplaysthe
stocksymbolsthatmatchedtheprevioussearch.
Holdings
PerhapsthemostimportantpageontheB&Dsite,theHoldingspagedisplaysthestocks
currentlyownedbythisclient.
Glossary
TheGlossarypagecanbeusedtolookuptermsthatunfamiliartotheB&Dclient.
AppendixB:StatelessUniversityRegistrationSystemCase
Study
SystemDocumentation
StatelessUniversityRegistrationSystem(SURS)UserInterfaceSpecification
May1,2002
Version2.3
PreparedBy:
OtisKribblekoblis
SuperDuperSoftwareCompany(SDSC)
422South5thAvenue
Anytown,USA
Introduction
ThepurposeofthisdocumentistodescribetheplanneduserinterfacefortheStateless
UniversityRegistrationSystem.Itwillberevisedtoreflectthat"as-built"softwareaftersystem
testinghasbegun.ItisacustomizedversionoftheregistrationsystemdeliveredtoUniversal
OnlineUniversity(UOU)lastyear.StatelessUhasrequestedsomemajormodificationstothe
UOUversion,sothatitisessentiallyarewriteofthesoftware.Someofthemodulesfor
databasecreationandbackuphavebeenreused,butthatisnotapparentfromtheuser
interface,whichisallnew.
Thismanualhastheuserinterfacescreensdefinedintheorderinwhichtheyarecustomarily
used.Itstartswiththeloginscreen.Thenitprovidesthedatabaseset-upfields:theaddition/
change/deletionofstudents,theaddition/change/deletionofcourses,andtheaddition/
change/deletionofclasssections.Thefinaldataentryscreenprovidestheselectionof
specificcoursesectionsforeachstudent.Thereisalsoanadministrativefunctionthatis
accessibletoonlythesupervisor.Itprovidesaccesstotheadministrativefunctionsofbackup
andrestoreofthedatabases.Eachscreenisdefinedinaseparatesectionprovidingthe
followinginformation:
Functionalitysupported
Formattingrequirementsforeachdataentryfield
Asamplescreenlayout(thefinalimplementedsoftwaremaydiffer)
Thefigurebelowsummarizesthescreensandtheirnavigationoptions.
2.1Log-inandSelectFunctionScreen
2.1.1Functions
EachuserisrequiredtoenteraUserIDandaPassword.Theidentificationofthestatusof
theuser(supervisor:yesorno)ismandatoryatthetimeoflog-in.OnlyYesorNomaybe
selectedbyclickingontheappropriatebox(notboth).Afterasuccessfullog-inhasbeen
completed,thenthenextfunctiontobeexecutedcanbeselected.Onlyasupervisormay
accesstheAdministrativescreen.TheExitbuttonisactiveatalltimes.
2.1.2DataEntryFormats
Theformatsforthefieldsonthisscreenare:UserID:eightcharactersatleasttwoofwhich
arenotalphabetic(canbenumericorspecialcharacters).
Password:eightcharactersatleasttwoofwhicharenotalphabetic(canbenumericorspecial
characters).
2.1.3ScreenFormat
3.1StudentDatabaseMaintenanceScreen
3.1.1Functions
Thisscreenallowstheentryoftheidentifyinginformationforanewstudentandtheassignment
ofhis/herstudentIDnumber.AllfieldsarerequiredtobefilledinbeforetheEnterbuttonis
selected.Thefieldsmaybeenteredinanyorder.Thebackspacekeywillworkatanytimein
anyfield,buttheResetbuttonwillclearallofthefieldswhenitispressed.
IftheStudentIDisenteredfirst,thentheDelete(allowsastudenttoberemovedfromthe
database)andModify(allowsthemodificationofthestudent'scontactinformation—thedata
currentlyinthedatabasewillbedisplayed)buttonsbecomeactive.TheEnterbuttonwillcause
theDeleteortheModifytobeexecutedandthefieldsonthescreentobecleared.
3.1.2DataEntryFormats
Theformatsforthefields(allmandatory)onthisscreenare:Firstname:onetotencharacters
Middlename:onetotencharactersorNMNfornomiddlenameLastname:onetofifteen
characters(alpha,period,hyphen,apostrophe,space,numbers)Streetaddress:fourto
twentyalphanumericcharactersCity:threetotenalphacharactersState:twoalpha
charactersZip:thestandardfivenumericshyphenfournumericsPhone:telephonenumberin
thefollowingformat703.555.1212
StudentID:twocharactersrepresentingthehomecampusandasix-digitnumberwhichis
uniqueforeachstudent.Thehomecampusdesignationsare:
ANforAnnandale
LOforLoudoun
MAforManassas
WOforWoodbridge
ARforArlington
Thesix-digitnumberisgeneratedbythesystemwhentheEnterbuttonisselected.Itremains
displayeduntiltheResetbuttonisdepressed.Atthattime,allfieldsareclearedforthenextset
ofentries.
3.1.3ScreenLayout
3.2CourseDatabaseMaintenanceScreen
3.2.1Functions
Thisscreenallowstheentryoftheidentifyinginformationforanewcourseandtheassignment
ofthecourseIDnumber.AllfieldsarerequiredtobefilledinbeforetheEnterbuttonis
pressed.Thefieldsmaybeenteredinanyorder.Thebackspacekeywillworkatanytimein
anyfield,buttheResetbuttonwillclearallofthefieldswhenitisentered.TheBackbutton
causesareturntothepreviousscreen.TheExitbuttoncausesanexitfromthisapplication.
IftheCourseIDisenteredfirst,thentheDelete(allowsacoursetoberemovedfromthe
database)andModify(allowsthemodificationofanexistingcourse'sinformation—thedata
currentlyinthedatabasewillbedisplayed)buttonsbecomeactive.TheEnterbuttonwillcause
theDeleteortheModifytobeexecutedandthefieldsonthescreentobecleared.
3.2.2DataEntryFormats
Theformatsforthefields(allaremandatory)onthisscreenare:CourseID:threealpha
charactersrepresentingthedepartmentfollowedbyasix-digitintegerwhichistheunique
courseidentificationnumber.Thepossibledepartmentsare:
PHY-Physics
EGR-Engineering
ENG-English
LAN-Foreignlanguages
CHM-Chemistry
MAT-Mathematics
PED-Physicaleducation
SOC-Sociology
LIB-Libraryscience
HEC-Homeeconomics
Coursename:afreeformatalphanumericfieldofuptofortycharactersCoursedescription:
afreeformatalphanumericfieldofupto250characters
3.2.3ScreenLayout
3.3ClassSectionDatabaseMaintenanceScreen
3.3.1Functions
Thisscreenallowstheentryoftheidentifyinginformationforanewcoursesection.Allfields
arerequiredtobefilledinbeforetheEnterbuttonispressed.Thefieldsmaybeenteredinany
order.Thebackspacekeywillworkatanytimeinanyfield,buttheResetbuttonwillclearallof
thefieldswhenitisentered.TheBackbuttoncausesareturntothepreviousscreen.TheExit
buttoncausesanexitfromthisapplication.
TheCourseIDisrequiredtobeenteredfirst(allexistingsectionswillbedisplayedassoonas
itisentered),followedbythenewSection#,DatesandTimefields.TheDelete(allowsa
sectiontoberemovedfromthedatabase)andModify(allowsthemodificationofanexisting
section'sinformation)buttonsbecomeactiveaftertheSection#isentered.Ifthesectionis
alreadyinthedatabase,thecurrentinformationwillbedisplayedassoonastheSection#field
isfilledin.TheEnterbuttonwillcausetheDeleteortheModifytobeexecutedandthefields
onthescreentobecleared.
3.3.2DataEntryFormats
Theformatsforthefields(allmandatory)onthisscreenare:CourseID:threealpha
charactersrepresentingthedepartmentfollowedbyasix-digitintegerSection#:athree-digit
integer(leadingzerosarerequired)assignedbytheuserDates:thedaysoftheweekthe
classmeets(uptothreewithhyphensinbetween);theweekdaydesignationsare:
Sun
Mon
Tue
Wed
Thr
Fri
Sat
Time:thestartingandendingtimesofthesection(usingmilitarytime)withahyphenin
between,e.g.,12:00–13:30.
3.3.3ScreenLayout
3.4SectionSelectionEntryScreen
3.4.1Functions
Thisscreenallowstheentryoftheselectionofspecificcoursesectionsforanindividual
student.AllfieldsarerequiredtobefilledinbeforetheEnterbuttonispressed.Thefieldsmay
beenteredinanyorder.Thebackspacekeywillworkatanytimeinanyfield,buttheReset
buttonwillclearallofthefieldswhenitisentered.TheBackbuttoncausesareturntothe
previousscreen.TheExitbuttoncausesanexitfromthisapplication.
TheStudentIDisrequiredtobeenteredfirst,followedbytheCourseID(allavailablesections
willbedisplayedassoonasitisentered).Sectionsareselectedbyclickingonthesectionto
beassigned.TheEnterbuttonwillcausethestudenttobeaddedtotheselectedsection.
EnteringanewCourseIDwillcauseanewlistofavailablesectionstobedisplayed,allowing
anothercoursesectiontobeselectedforthesamestudent.
3.4.2DataEntryFormats
Theformatsforthefields(allmandatory)onthisscreenare:
CourseID:threealphacharactersrepresentingthedepartmentfollowedbyasix-digitinteger
StudentID:twocharactersrepresentingthehomecampusandasix-digitnumberthatis
uniqueforeachstudent
Availablesections:alistofallofthesectionsthatarenotfull
3.4.3ScreenLayout
3.5AdministrativeScreen
3.5.1Functions
Onlythesupervisormayaccesstheadministrativescreen.Itpermitsoneofthefollowingthree
activitiesatatime:
Creationofabackupofanyorallofthedatabases
Restoreofabackupofanyorallofthedatabases
Printingofareportofanyorallofthedatabases
Aftertheactivity(createorrestore)andthedatabaseshavebeenselected,thenameofthe
backupistobeentered.
TheBackandExitbuttonsareactiveatalltimes.
3.5.2DataEntryFormats
Theformatsforthefieldsonthisscreenare:
BackupID:aannnn(requiredonlyforbackups,notreports)Commentary:afreeformat
characterfield200charactersinlength(requiredonlyforbackups,notreports)
3.5.3ScreenLayout
Bibliography
WorksCited
Bach,James."ExploratoryTestingandthePlanningMyth."19March2001.
http://www.stickyminds.com/r.asp?F=DART_2359
Bach,James."ExploratoryTestingExplained."v.1.3,16April2003.
http://www.satisfice.com/articles/et-article.pdf
Beizer,Boris(1990).SoftwareTestingTechniques(SecondEdition).VanNostrandReinhold.
ISBN0-442-20672-0.
Beizer,Boris(1995).Black-BoxTesting:TechniquesforFunctionalTestingofSoftwareand
Systems.JohnWiley&Sons.ISBN0-471-12094-4.
Binder,RobertV.(2000).TestingObject-OrientedSystems:Models,Patterns,andTools.
Addison-Wesley.ISBN0-201-80938-9.
Brownlie,Robert,etal."RobustTestingofAT&TPMX/StarMAILUsingOATS,"AT&T
TechnicalJournal,Vol.71,No.3,May/June1992.
Carr,MarvinJ.,etal.(1993)Taxonomy-BasedRiskIdentification.TechnicalReport
CMU/SEI-93-TR-6,ESC-TR-93-183,June1993.
http://www.sei.cmu.edu/pub/documents/93.reports/pdf/tr06.93.pdf
Cockburn,Alistair(2000).WritingEffectiveUseCases.Addison-Wesley.ISBN0-201-
70225-8.
Cohen,D.M.,etal."TheAETGSystem:AnApproachtoTestingBasedonCombinatorial
Design."IEEETransactionsonSoftwareEngineering,Vol.23,No.7,July1997.
Copeland,Lee."ExploratoryPlanning."3September2001.
http://www.stickyminds.com/r.asp?F=DART_2805
Craig,RickD.andStefanP.Jaskiel(2002).SystematicSoftwareTesting.ArtechHouse
Publishers.ISBN1-58053-508-9.
Fowler,MartinandKendallScott(1999).UMLDistilled:ABriefGuidetotheStandard
ObjectModelingLanguage(2ndEdition).Addison-Wesley.ISBN0-201-65783X.
Gilb,TomandDorothyGraham(1993).SoftwareInspection.Addison-Wesley.ISBN0-201-
63181-4.
Harel,David."Statecharts:avisualformalismforcomplexsystems."ScienceofComputer
Programming8,1987.
Hetzel,Bill(1998).TheCompleteGuidetoSoftwareTesting(SecondEdition).JohnWiley&
Sons.ISBN0-471-56567-9.
IEEEStandardforSoftwareTestDocumentation:IEEEStandard829-1998.ISBN0-7381-
1443-X.
IEEEStandardGlossaryofSoftwareEngineeringTerminology:IEEEStandard610.12-
1990.ISBN1-55937-067-X.
ISO(1991).ISO/IECStandard9126-1.SoftwareEngineering-ProductQuality-Part1:
QualityModel.ISOCopyrightOffice,Geneva,June2001.
Jacobsen,Ivar,etal(1992).Object-OrientedSystemsEngineering:AUseCaseDriven
Approach.Addison-Wesley.ISBN0-201-54435-0.
Kaner,Cem,JackFalkandHungQuocNguyen(1999).TestingComputerSoftware(Second
Edition).JohnWiley&Sons.ISBN0-471-35846-0.
Kaner,Cem,JamesBach,andBretPettichord(2002).LessonsLearnedinSoftwareTesting:
AContext-DrivenApproach.JohnWiley&Sons.ISBN0-471-08112-4.
Kuhn,D.RichardandMichaelJ.Reilly."AnInvestigationoftheApplicabilityofDesignof
ExperimentstoSoftwareTesting,"27thNASA/IEEESoftwareEngineeringWorkshop,NASA
GoddardSpaceFlightCenter,4–6December2002.http://csrc.nist.gov/staff/kuhn/kuhn-
reilly-02.pdf
Marick,Brian(1995).TheCraftofSoftwareTesting:SubsystemTestingIncludingObject-
BasedandObject-OrientedTesting.Prentice-Hall.ISBN0-131-77411-5.
Mandl,Robert."OrthogonalLatinSquares:AnApplicationofExperimentDesigntoCompiler
Testing,"CommunicationsoftheACM,Vol.128,No.10,October1985.
Mealy,G.H."Amethodforsynthesizingsequentialcircuits."BellSystemTechnicalJournal,
34(5):1955.
Myers,Glenford(1979).TheArtofSoftwareTesting.JohnWiley&Sons.ISBN0-471-
04328-1.
Moore,E.F."Gedanken-experimentsonsequentialmachines,"AutomataStudies(C.E.
ShannonandJ.McCarthy,eds.),Princeton,NewJersey:PrincetonUniversityPress,1956.
Phadke,MadhavS.(1989).QualityEngineeringUsingRobustDesign.Prentice-Hall.ISBN
0-13-745167-9.
Planning,MCDP5.UnitedStatesMarineCorps.
https://www.doctrine.usmc.mil/mcdp/view/mpdpub5.pdf
Pressman,RogerS.(1982).SoftwareEngineering:APractitioner'sApproach(Fourth
Edition).McGraw-Hill.ISBN0-07-052182-4.
Rapps,SandraandElaineJ.Weyuker."DataFlowAnalysisTechniquesForTestData
Selection."SixthInternationalConferenceonSoftwareEngineering,Tokyo,Japan,
September13–16,1982.
Rumbaugh,James,etal.(1991).Object-OrientedModelingandDesign.Prentice-Hall.ISBN
0-13-629841-9.
Watson,ArthurH.andThomasJ.McCabe.StructuredTesting:ATestingMethodology
UsingtheCyclomaticComplexityMetric.NISTSpecialPublication500-235.
http://www.mccabe.com/nist/nist_pub.php
Wallace,DeloresR.andD.RichardKuhn."FailureModesinMedicalDeviceSoftware:An
Analysisof15YearsofRecallData,"InternationalJournalofReliability,Quality,andSafety
Engineering,Vol.8,No.4,2001.http://csrc.nist.gov/staff/kuhn/final-rqse.pdf
Weinberg,GeraldM.(1975).AnIntroductiontoGeneralSystemsThinking.JohnWiley&
Sons.ISBN0-471-92563-2.
Whittaker,JamesA.(2003).HowtoBreakSoftware:APracticalGuidetoTesting.Addison
Wesley.ISBN0-201-79619-8.
OtherUsefulPublications
Beizer,Boris(1984).SoftwareSystemTestingandQualityAssurance.VanNostrand
Reinhold.ISBN0-442-21306-9.
Black,Rex(1999).ManagingtheTestingProcess.MicrosoftPress.ISBN0-7356-0584-X.
BritishComputerSociety.StandardonSoftwareComponentTesting.BS7925-2.
http://www.testingstandards.co.ukhttp://www.testingstandards.com/BS7925_3_4.zip
Kit,Edward(1995).SoftwareTestingintheRealWorld:ImprovingtheProcess.Addison-
Wesley.ISBN0-201-87756-2.
McGregor,JohnD.andDavidA.Sykes(2001).APracticalGuidetoTestingObject-
OrientedSoftware.Addison-Wesley.ISBN0-201-32564-0.
Meyer,Bertrand(2000).Object-OrientedSoftwareConstruction(2ndEdition).Prentice-Hall.
ISBN0-136-29155-4.
Roper,Marc(1994).SoftwareTesting.McGraw-Hill.ISBN0-07-707466-1.
Tamres,Louise(2002).IntroducingSoftwareTesting.Addison-Wesley.ISBN0-201-71974–
6.
Index
A
acceptancetesting,10,135
ACTtestscores,121
action,51–52,95,97
actor,128
adhoctesting,202
adaptiveplanning,213–214,216
alleventscoverage,106
allpairstesting,64
allpairstools
AETG,88
Allpairs,85
rdExpert,68
allpathscoverage,107
allstatescoverage,105
alltransitionscoverage,108
Allison,Chuck,xvi
Allpairsalgorithm,66,85–88
comparisonwithorthogonalarrays,88–89
unbalanced,87–88
AnIntroductiontoGeneralSystemsThinking,236
automaticvariables,169
Index
B
Bach,James,xvi,85,202,206,212
bananaprinciple,236
Barker,Joel,182
baselinepath,155
basispathsets,164
basispathtesting,154–159
baselinepath,155
basispathsets,164
example,160–164
multiplesets,159
pathcreation,155
basispaths,154–155
Beizer,Boris,2,6,135,198,220,226,236
Beizer'staxonomy,226
binarydecisions,155
Binder,Robert,11,222,229
Binder'sobject-orientedtaxonomy,229
blackboxtesting,8,20–22,140
blackdotsymbol,95,97
Black,Rex,xvi
Bloom,Benjamin,246
bookends,220
boundaries,40,134
boundaryvaluetesting,40,197
boundaries,40,134
examples,45
boundaryvalues,134
Box,George,232
Brown&Donaldson,16,54,71,91,94,160,250
authorizationcode,17
bulls-eyesymbol,97
Bulwer-LyttonFictionContest,xvi
Index
C
casestudies
Brown&Donaldson,16,54,71,91,94,160,250
StatelessUniversityRegistrationSystem,17,36,47,56,71,91,94,111,131,136,260
charteredexploratorytesting,206
choosingoffpoints,118
choosingonpoints,118
choosingtestcases,119
classicalplanning,213
closedboundary,118
Cockburn,Alistair,130
codequalitytesting,11
collapsedrules,55
combinationtesting,Seepairwisetesting
combinations,63
combinedrules,55
competency,246
complexbusinessrules,50,53–54,58
conditioncoverage,151
conditional,102
conditions,50,51–53,55,58
connectionlost,134
contract,27
controlflow,141
controlflowgraphs,145–147,154–159,171–176
controlflowpaths,172
controlflowtesting
conditioncoverage,151
controlflowgraphs,145–147,154–159,171–176
decisionpoint,146
decision/conditioncoverage,151
executionpaths,144
exhaustivetesting,144
junctionpoint,146
levelsoftestcoverage,147–153
limitingloopexecution,153
missingpaths,144
multipleconditioncoverage,152
pathcoverage,153
processblock,146
statementcoverage,147,150
testcoverage,147–153
coverage,147,153
Craig,Rick,xiii,xvi,2
createdvariables,169
creatingtestcases,109–110,134
cyclomaticcomplexity,154–155
binarydecisions,155
edges,154–155
nodes,154–155
Index
D
dataflowgraphs,171–174
dataflowtesting,168
~define,170
~kill,170
~use,170
createdvariables,169
dataflowgraph,171–174
define-define,170
define-kill,170
define-use,170
define-use-kill,172–174
destroyedvariables,169
dynamicdataflowtesting,176
kill-define,170
killedvariables,169
kill-kill,170
kill-use,170
lifecycle,169
processingflow,171
static,171–176
usedvariables,169
use-define,170
use-kill,170
use-use,170
variablesincomputation,169
variablesinconditionals,169
datasensitivityerrors,141
debugging,3
decisioncoverage,150
decisionpoint,146
decisiontables
actions,51–52
collapsedrules,55
combinedrules,55
conditions,50–53,55,58
derivingtestcases,52–53
examples,54–58
expectedresults,54
"firing",51
decisiontabletesting,50–54
decision/conditioncoverage,151
defecttaxonomies,222–232
defensive-design,28
defensive-testing,28
define-define,170
define-kill,170
define-use,170
define-use-kill,172–174
design-by-contract,27
destroyedvariables,169
developmentmanagers,xv
developmentparadigm,129
diskfull,134
documentingtransactions
flowcharts,128
HIPOdiagrams,128
text,128
usecases,128
domainanalysis,116
domainanalysistesting
choosingoffpoints,118
choosingonpoints,118
choosingtestcases,119
closedboundary,118
DomainTestMatrix,120
example,121
extraboundary,116–117
inpoint,118,120
interactionsbetweenvariables,116
missingboundary,116–117
offpoint,118,120
onpoint,118,120
openboundary,118
outpoint,118
shiftedboundary,116–117
tiltedboundary,116–117
DomainTestMatrix,120,134
Index
E
edges,154–155
equivalenceclasspartitioning,197
equivalenceclasstesting,24–33
examples,33–34
inputequivalenceclasses,29
outputequivalentclasses,33
equivalenceclasstypes
continuousrangeofvalues,29
discretevalueswithinarange,29–30
multipleselection,30–31
singleselection,30
equivalenceclasses,25,28
events,94–96
executionpaths,141,144
exhaustivetesting,21,144
existingpaths,141
exploratoryplanningrigorousplan,202
exploratorytesting,182,202–208,212
chartered,206
charters,206
consciousplan,202
freestyle,207
process,205
extraboundary,116–117
Index
F
FAFSA,59
Faught,Danny,xvi
faultmodel,222
non-specific,222
specific,222
Ferguson,Marilyn,182
figures
decisioncoverage,150
annotatedcontrolflowdiagram,171–173
B&Dcontrolflowgraph,162–163
B&DJavacode,160
baselinebasispath,156
boundaryvalues,43
canceltheReservation,101
cancellationfromPaidstate,102
cancellationfromTicketedstate,103
continuousequivalenceclasses,29
dataontheboundaries,44
defectdiscoveryrate,238
discreteequivalenceclasses,29
enrollanddropacourse,113
evaluationofcomplexconditions,152
examplecontrolflowgraph,154
fifthbasispath,158
flowgraphequivalent,147
fourexecutionpaths,148
fourthbasispath,157
graphicalrepresentation,148
IEEE829testdocumentation,188
interestingflowdiagram,153
multipleselectionequivalenceclasses,30
on,off,in,andoutpoints,119
pathterminates,99
PayTimerexpires,100
ReservationinMadestate,95
ReservationinPaidstate,95
ReservationinTicketedstate,97
ReservationinUsedstate,98
secondbasispath,156
seventhbasispath,159
singleselectionequivalenceclasses,30
sixthbasispath,158
StatelessUniversityAdmissionsMatrix,121
StatelessUniversityusecases,129
SURSmaintenancescreen,56
testcasesthattriggerallevents,106
testcasesthatvisiteachstate,106
thirdbasispath,157
twodimensionalboundarydefects,117
Waterfallmodel,187
fire,51
firing,51
FloridaInstituteofTechnology,135
flowcharts,128
football,212–213
FreeApplicationforFederalStudentAid,59
freestyleexploratorytesting,207
functionalitytesting,11
Index
I
IEEE829advantages
completenesschecklist,190
evaluationoftestpractices,190
facilitatecommunication,190
increasedmanageability,190
IEEE829standard,189–190
bugreport,195
releasenotes,194
testcasespecification,188,193
testdesignspecification,188,192
testincidentreport,188,195
testitemtransmittalreport,188,194
testlog,188,194
testplan,188,190
testprocedurespecification,188,193
testsummaryreport,188,196
IEEE829testcasespecification
environmentalneeds,193
inputspecifications,193
intercasedependencies,193
outputspecifications,193
specialproceduralrequirements,193
testcasespecificationidentifier,193
testitems,193
IEEE829testdesignspecification
approachrefinements,192
featurepass/failcriteria,193
featurestobetested,192
testdesignspecificationidentifier,192
testidentification,192
IEEE829testincidentreport
impact,195
incidentdescription,195
summary,195
testincidentreportidentifier,195
IEEE829testitemtransmittalreport
approvals,194
location,194
status,194
transmittalreportidentifier,194
transmitteditems,194
IEEE829testlog
activityandevententries,195
description,195
testlogidentifier,195
IEEE829testplan
approach,191
approvals,192
environmentalneeds,192
featuresnottobetested,191
featurestobetested,191
introduction,191
itempass/failcriteria,191
responsibilities,192
risksandcontingencies,192
schedule,192
staffingandtrainingneeds,192
suspensionandresumptioncriteria,191
testdeliverables,191
testitems,191
testplanidentifier,191
testingtasks,191
IEEE829testprocedurespecification
proceduresteps,194
purpose,194
specialrequirements,194
testprocedurespecificationidentifier,194
IEEE829testsummaryreport
approvals,196
comprehensiveassessment,196
evaluation,196
summary,196
summaryofactivities,196
summaryofresults,196
testsummaryreportidentifier,196
variance,196
IEEEStandard610.12–1990,2
IEEEStandardforSoftwareTestDocumentation,188–196
IEEEStandardGlossaryofSoftwareEngineeringTerminology,2
IEEEStd829–1998,188–196
inpoint,118,120
Index
K
Kaner,Cem,202
Kaner,Falk,andNguyen'staxonomy,228
keypoints
adaptiveplanning,213
advantageofstate-transitiontable,105
bankcombinations,62
blackboxtestinghelpsefficiencyandeffectiveness,21
boundariesarewheredefectshide,40
choosingcombinations,65
comparingwhatiswithwhatoughttobe,2
controlflowgraphs,145
createtestcases,53
cultivatetheskillofchoosingpoorly,63
dataczar,135
dataflowtesting,168
domainanalysis,116
doubleintegralsign,xiv
equivalenceclasses,26
evaluatetherisk,134
executingCtestcases,155
exploratorycharter,206
exploratorytesting,202
expressyourappreciation,33
Holodeck,149
importanceoftestdesign,xiii
locateorthogonalarrays,67
Methuselah,27
mostimportanttest,205
objectorientedcombinations,62
pathdefinition,144
postinstallationtestplanning,63
randomselection,64
rarelywillwehavetime,31
rdExperttool,68
scriptedandexploratorytesting,217
scriptedandexploratoryparadigms,182
taxonomy,222
testcasesareinputs,outputs,andorder,6
testcasesateachboundary,42
testcasesmustbedesigned,5
testingeverytransition,108
testinglevels-unit,integration,system,acceptance,9
time-sequencedpairs,170
triggeralltransitions,108
TwentyQuestions,203
useGUIdesigntools,34
warm,fuzzyfeelings,29
wecanneverbesureofcoverage,21
webcombinations,62
whogetstheblame,28
yourtaxonomy,232
kiddiepool,198
kill-define,170
killedvariables,169
kill-kill,170
kill-use,170
Index
L
L18(2137)orthogonalarray,69
L18(35)orthogonalarray,68
L4(23)orthogonalarray,66
L64(8243)orthogonalarray,73
L9(34)orthogonalarray,67
Lakein,Alan,205
largenumberofcombinations,63
LessonsLearnedinSoftwareTesting,85,241
levelsofcompetency,246
lifecycle,169
limitingloopexecution,153
lowmemory,134
Index
O
O'Loughlin,Martin,xvii
Object-OrientedSoftwareEngineering:AUseCaseDrivenApproach,128
offpoints,118,120
onpoints,118,120
openboundary,118
oracles
existingprograms,7
kiddie,6
purchasedtestsuites,7
regressiontestsuites,7
validateddata,7
orderofexecution,6,7
orthogonalarrays,66–70
balanced,87
definition,68
L18(2137),69
L18(35),68
L4(23),66
L64(8243),72
L9(34),67
mappingonto,74–76
comparisonwithAllpairsalgorithm,88–89
dealingwithextracolumns,83
dealingwithextravaluesinarow,83
notation,67
unassignedcells,82
using70–85
outpoint,118
Index
P
pairwisetesting,64–65,197
additionaltests,89
allpairstesting,64
Allpairsalgorithm,66,85,88–89
constraints,88
documentedstudies,64–65
effectiveness,64–65,89
paradigms,182
cloudourvision,182
sharpenourvision,182
Paradigms:TheBusinessofDiscoveringtheFuture,182
partitionedstate-transitiondiagram,97
pathcoverage,153
pathcreation,155
pathtesting,140
performancetesting,11
Perry,Dale,xvi
Phadke,MadhavS.,66,72
planning,183,212,216
adaptive,213
classical,213
Planning,215,216
planningfunctions,215
planningheuristic,214
planningpitfalls,215
post-conditions,27–28
pre-conditions,27–28
Pressman,Roger,5
processblock,146
processingflow,171
projectleveltaxonomies,223
purchasedtestsuites,7
Index
S
scenario,128
scriptedtesting,182,186,208,212
auditability,187
objectivity,187
repeatability,187
securitytesting,11
shiftedboundary,116–117
single-modedefects,65
Sloane,NeilJ.A.,72
Snook,Sid,xvi
softwaredefecttaxonomies,223–232
Beizer'staxonomy,226
Binder'sobject-orientedtaxonomy,229
Kaner,Falk,andNguyen'staxonomy,228
Vijayaraghavan'seCommercetaxonomy,231
Whittaker'sHowToBreakSoftwaretaxonomy,230
softwaredevelopers,xv
SoftwareEngineeringInstitute,223
softwareinspection,4,41,134,164,171,176,230
SoftwareInspection,41
SoftwareQualityEngineering,16
SoftwareSystemTestingandQualityAssurance,198
softwaretestengineers,xv
SoftwareTestingTechniques,226
specificfaultmodel,222
state,51,95–96
StatelessUniversity,59,94,121,124
admisions,121
StatelessUniversityRegistrationSystem,17,36,47,56,71,91,94,111,131,136,260
statementcoverage,147
state-transitiondiagrams,94–95,128
action,95,97
alleventscoverage,106
allpathscoverage,107
allstatescoverage,105
alltransitionscoverage,108
blackdotsymbol,95,97
bulls-eyesymbol,97
conditional,102
creatingtestcases,105–110
event,94–96
guard,102
mixingdifferententities,97
nestedstate-transitiondiagrams,97
partitionedstate-transitiondiagrams,97
state,95–96
transition,95–96
state-transitiontables
action,104
advantage,105
creatingtestcases,105–110
currentstate,104
disadvantage,105
event,104
nextstate,104
staticdataflowtesting,171
StickyMinds.com,202,212
stoppingcriteria,236–240
defectdiscoveryrate,238
marginalcost,239
metcoveragegoals,237
teamconsensus,239
thebosssays"Shipit!",240
structuredtesting,154–159
system,140
systemstate,51
systemtesting,10,135
systemtransactions,135
SystematicSoftwareTesting,xiii,2
Index
T
tables
Allpairsprograminput,86
Allpairsprogramoutput,86
Beizer'sBugtaxonomy,226
Binder'sClassScopetaxonomy,230
Binder'sMethodScopetaxonomy,229
classicalvs.adaptiveplanning,214
classicalvs.exploratoryplanning,214
collapseddecisiontable,55
decisiontable,51–52,54
DomainAnalysistestcases,122
DomainTestMatrix,120
exampleusecase,132
furthercollapseddecisiontable,55
genericdecisiontable,50
invaliddatavalues,32
ISO9126QualityCharacteristics,225
Kaner'staxonomy,228
L18(2137)orthogonalarray,70
L18(35)orthogonalarray,69
L4(23)orthogonalarray,66
L64(8243)orthogonalarray,73,75,77,79,81,84
L9(34)orthogonalarray,67
sampletestcases,53
SEItaxonomy,223
sensitizingcontrolflowpaths,164
setoftestcases,44
StatelessUniversityAdmissionsMatrix,121
State-Transitiontable,104
SURSdecisiontable,57
testcasetable,54
testingallvalidtransitions,109
usecasetemplate,130
validdatavalues,31
varyingvalidandinvalidvalues,32,34
Whittaker'sFaulttaxonomy,230
taxonomies,222–223
creatingyourown,232
projectlevel,223–226
softwaredefect,226–232
taxonomy-basedriskidentification,223
templateforusecases,130
testanalysis,2
testcasecomponents
inputs,6–7
orderofexecution,7–8
outputs,6–7
testcasecreation,134
testcasedefinition,5
testcasedesign,2
testcasedesignstyles
cascading,7
independent,8
testcasespecification,SeeIEEE829testcasespecification
testcasesubsets,22
testcoverage,147–153
testdesign,Seetestcasedesign
testdesignspecification,SeeIEEE829testdesignspecification
testincidentreport,SeeIEEE829testincidentreport
testitemtransmittalreport,SeeIEEE829testitemtransmittalreport
testlog,SeeIEEE829testlog
testmanagers,xv
testmultiplevariablessimultaneously,116
testoracles,6–7
testplan,SeeIEEE829testplan
testplanning,2,212–217
testprocedurespecification,SeeIEEE829testprocedurespecification
testsuites
purchased,7
regression,7
testsummaryreport,SeeIEEE829testsummaryreport
testerskills
carefulobservers,207
carefulreporters,207
criticalthinkers,207
evaluaterisk,207
goodmodelers,207
notdistracted,207
selfmanaged,207
testdesigners,207
testing
maturitylevels,2–4
blackbox,8,20–21
exhaustive,11,21
graybox,8
whitebox,8,20
testingchallenges,4
TestingComputerSoftware,202,228
testinginteractingvariables,116
testinglevels
acceptancetesting,10
integrationtesting,9
systemtesting,10
unittesting,9
testingmaturity,2–4
testingnsimultaneousdimensions,116
TestingObject-OrientedSystems,222
testingtechniques
Allpairsalgorithm,85
allpairstesting,64
boundaryvaluetesting,40
dataflowtesting,168
decisiontables,50
domainanalysis,116
equivalenceclasstesting,24
pairwisetesting,64
testingtoolbox,246
testing,definition
aconcurrentlifecycleprocess,2
comparison,2
testing-by-contract,27
TheCompleteGuideToSoftwareTesting,236
tiltedboundary,116–117
timebox,206
tools
AETG,88
Index
U
UMLDistilled:ABriefGuideToTheStandardObjectModelingLanguage,97
UnifiedModelingLanguage,129
unit,140
unittesting,9,140
UnitedStatesMarineCorps,215
usabilitytesting,11
usecases,128–129
actor,128
example,131–133
functionalrequirements,128–129
scenario,128
template,130–131
value,129
usecasetemplate
actors,130–131
channelstoprimaryactor,131
channelstosecondaryactors,131
completenesslevel,131
datedue,131
extensions,131
failedendconditions,130
frequency,131
goalincontext,130
level,130
mainsuccessscenario,130
name,130
numberoridentifier,130
openissues,131
preconditions,130
primaryactor,130
priority,131
responsetime,131
scope,130
secondaryactors,131
sub-variations,131
successendcondition,130
trigger,130
usedvariables,169
Index
W
waterfalldevelopmentmodel,186–187
waterfalldevelopmentmodelphases
coding,186
operations,186
programdesign,186
requirementsanalysis,186
softwarerequirements,186
systemrequirements,186
waterfalldevelopmentmodeltesting,186
Webtestinglevels,11
codequality,11
functionality,11
performance,11
security,11
usability,11
Weinberg,Gerald,236
Weyuker,Elaine,168
whiteboxtesting,8,20,140–142
existingpaths,141
nonexistentpaths,141
whiteboxtestingtechniques
controlflowtesting,144–159
dataflowtesting,168–176
Whittaker,James,135,230
Whittaker'sHowToBreakSoftwaretaxonomy,230
WritingEffectiveUseCases,130
ListofFigures
Chapter7:State-TransitionTesting
Figure7-1:TheReservationisMade.
Figure7-2:TheReservationtransitionstothePaidstate.
Figure7-3:TheReservationtransitionstotheTicketedstate.
Figure7-4:TheReservationtransitionstotheUsedstate.
Figure7-5:Thepathends.
Figure7-6:ThePayTimerexpiresandtheReservationiscancelledfornonpayment.
Figure7-7:CanceltheReservationfromtheMadestate.
Figure7-8:CancellationfromthePaidstate.
Figure7-9:CancellationfromtheTicketedstate.
Figure7-10:Asetoftestcasesthat"visit"eachstate.
Figure7-11:Asetoftestcasesthattriggeralleventsatleastonce.
Figure7-12:Asetoftestcasesthattriggeralltransitionsatleastonce.
Figure7-13:State-transitiondiagramforenrollanddropacourseatStatelessU.
Chapter10:ControlFlowTesting
Figure10-1:Flowgraphequivalentofprogramcode.
Figure10-2:Graphicalrepresentationofthetwo-linecodesnippet.
Figure10-3:Fourexecutionpaths.
Figure10-4:Twotestcasesthatyield100%decisioncoverage.
Figure10-5:Compilerevaluationofcomplexconditions.
Figure10-6:Aninterestingflowdiagramwithmany,manypaths.
Figure10-7:Anexamplecontrolflowgraph.
Figure10-8:ThechosenbaselinebasispathABDEGKMQS
Figure10-9:ThesecondbasispathACDEGKMQS
Figure10-10:ThethirdbasispathABDFILORS
Figure10-11:ThefourthbasispathABDEHKMQS
Figure10-12:ThefifthbasispathABDEGKNQS
Figure10-13:ThesixthbasispathACDFJLORS
Figure10-14:TheseventhbasispathACDFILPRS
Figure10-15:JavacodeforBrown&Donaldson'sevaluateBuySellmodule.
Figure10-16:ControlflowgraphforBrown&Donaldson'sevaluateBuySellmodule.
Figure10-17:ControlflowgraphforBrown&Donaldson'sevaluateBuySellmodule.
Chapter11:DataFlowTesting
Figure11-1:Thecontrolflowdiagramannotatedwithdefine-use-killinformationforeachof
themodule'svariables.
Figure11-2:Thecontrolflowdiagramannotatedwithdefine-use-killinformationforthex
variable.
Figure11-3:Thecontrolflowdiagramannotatedwithdefine-use-killinformationforthey
variable.
Figure11-4:Thecontrolflowdiagramannotatedwithdefine-use-killinformationforthez
variable.
ListofTables
Chapter3:EquivalenceClassTesting
Table3-1:Atestcaseofvaliddatavalues.
Table3-2:Atestcaseofallinvaliddatavalues.Thisisnotagoodapproach.
Table3-3:Asetoftestcasesvaryinginvalidvaluesonebyone.
Table3-4:Asetoftestcasesvaryinginvalidvaluesonebyonebutalsovaryingthevalid
values.
Table3-5:Asetoftestcasesvaryinginvalidvaluesonebyone.
Chapter5:DecisionTableTesting
Table5-1:Thegeneralformofadecisiontable.
Table5-2:Adecisiontablewithtwobinaryconditions.
Table5-3:Addingasingleactiontoadecisiontable.
Table5-4:Adecisiontablewithmultipleactions.
Table5-5:Adecisiontablewithnon-binaryconditions.
Table5-6:Sampletestcases.
Table5-7:Adecisiontableconvertedtoatestcasetable.
Table5-8:AdecisiontablefortheBrown&DonaldsonBuyorder.
Table5-9:Acollapseddecisiontablereflecting"Don'tCare"conditions.
Table5-10:Afurthercollapseddecisiontablereflecting"Don'tCare"conditions.
Table5-11:AdecisiontableforStatelessUniversityRegistrationSystem.
Chapter6:PairwiseTesting
Table6-1:L4(23)OrthogonalArray
Table6-2:L9(34)OrthogonalArray
Table6-3:L18(35)OrthogonalArray
Table6-4:L18(2137)OrthogonalArray
Table6-5:L64(8243)OrthogonalArray
Table6-6:L64(8243)withapartialmappingofitsfirstcolumn.
Table6-7:L64(8243)withafullmappingofitsfirstcolumn.
Table6-8:L64(8243)withafullmappingofitsfirstandsecondcolumns.
Table6-9:L64(8243)withafullmappingofallitscolumns.
Table6-10:L64(8243)withafullmappingofallitscolumnsincludingthe"extra"cells.
Table6-11:InputtotheAllpairsprogram.
Table6-12:OutputfromtheAllpairsprogram.
Chapter15:DefectTaxonomies
Table15-1:TheSEITaxonomy-BasedRiskIdentificationtaxonomy.
Table15-2:TheISO9126QualityCharacteristicstaxonomy.
Table15-3:AportionofBeizer'sBugTaxonomy.
Table15-4:AportionofthedefecttaxonomyfromTestingComputerSoftware.
Table15-5:AportionofBinder'sMethodScopeFaultTaxonomy.
Table15-6:AportionofBinder'sClassScopeFaultTaxonomy.
Table15-7:AportionofWhittaker'sFaultTaxonomy.
ListofExamples