ALGORITHMS
THIRD EDITION
12BinarySearchTrees 286
12.1Whatisabinarysearchtree? 286
12.2Queryingabinarysearchtree 289
12.3Insertionanddeletion 294
?
12.4Randomlybuiltbinarysearchtrees 299
13Red-BlackTrees 308
13.1Propertiesofred-blacktrees 308
13.2Rotations 312
13.3Insertion 315
13.4Deletion 323
14AugmentingDataStructures 339
14.1Dynamicorderstatistics 339
14.2Howtoaugmentadatastructure 345
14.3Intervaltrees 348
IVAdvancedDesignandAnalysisTechniques
Introduction 357
15DynamicProgramming 359
15.1Rodcutting 360
15.2Matrix-chainmultiplication 370
15.3Elementsofdynamicprogramming 378
15.4Longestcommonsubsequence 390
15.5Optimalbinarysearchtrees 397
16GreedyAlgorithms 414
16.1Anactivity-selectionproblem 415
16.2Elementsofthegreedystrategy 423
16.3Huffmancodes 428
?
?
16.4Matroidsandgreedymethods 437
16.5Atask-schedulingproblemasamatroid 443
17AmortizedAnalysis 451
17.1Aggregateanalysis 452
17.2Theaccountingmethod 456
17.3Thepotentialmethod 459
17.4Dynamictables 463
VAdvancedDataStructures
Introduction 481
18B-Trees 484
18.1DefinitionofB-trees 488
18.2BasicoperationsonB-trees 491
18.3DeletingakeyfromaB-tree 499
19FibonacciHeaps 505
19.1StructureofFibonacciheaps 507
19.2Mergeable-heapoperations 510
19.3Decreasingakeyanddeletinganode 518
19.4Boundingthemaximumdegree 523
20vanEmdeBoasTrees 531
20.1Preliminaryapproaches 532
20.2Arecursivestructure 536
20.3ThevanEmdeBoastree 545
21DataStructuresforDisjointSets 561
21.1Disjoint-setoperations 561
21.2Linked-listrepresentationofdisjointsets 564
21.3Disjoint-setforests 568 ?
21.4Analysisofunionbyrankwithpathcompression 573
VIGraphAlgorithms
Introduction 587
22ElementaryGraphAlgorithms 589
22.1Representationsofgraphs 589
22.2Breadth-firstsearch 594
22.3Depth-firstsearch 603
22.4Topologicalsort 612
22.5Stronglyconnectedcomponents 615
23MinimumSpanningTrees 624
23.1Growingaminimumspanningtree 625
23.2ThealgorithmsofKruskalandPrim 631
24Single-SourceShortestPaths 643
24.1TheBellman-Fordalgorithm 651
24.2Single-sourceshortestpathsindirectedacyclicgraphs 655
24.3Dijkstra’salgorithm 658
24.4Differenceconstraintsandshortestpaths 664
24.5Proofsofshortest-pathsproperties 671
25All-PairsShortestPaths
684
25.1Shortestpathsandmatrixmultiplication 686
25.2TheFloyd-Warshallalgorithm 693
25.3Johnson’salgorithmforsparsegraphs 700
26MaximumFlow 708
26.1Flownetworks 709
26.2TheFord-Fulkersonmethod 714
26.3Maximumbipartitematching 732 ?
26.4Push-relabelalgorithms 736 ?
26.5Therelabel-to-frontalgorithm 748
VIISelectedTopics
Introduction 769
27MultithreadedAlgorithms 772
27.1Thebasicsofdynamicmultithreading 774
27.2Multithreadedmatrixmultiplication 792
27.3Multithreadedmergesort 797
28MatrixOperations 813
28.1Solvingsystemsoflinearequations 813
28.2Invertingmatrices 827
28.3Symmetricpositive-definitematricesandleast-squaresapproximation 832
29LinearProgramming 843
29.1Standardandslackforms 850
29.2Formulatingproblemsaslinearprograms 859
29.3Thesimplexalgorithm 864
29.4Duality 879
29.5Theinitialbasicfeasiblesolution 886
30PolynomialsandtheFFT 898
30.1Representingpolynomials 900
30.2TheDFTandFFT 906
30.3EfficientFFTimplementations 915
31Number-TheoreticAlgorithms 926
31.1Elementarynumber-theoreticnotions 927
31.2Greatestcommondivisor 933
31.3Modulararithmetic 939
31.4Solvingmodularlinearequations 946
31.5TheChineseremaindertheorem 950
31.6Powersofanelement 954
31.7TheRSApublic-keycryptosystem 958 ? 31.8Primalitytesting 965 ? 31.9Integerfactorization 975
32StringMatching 985
32.1Thenaivestring-matchingalgorithm 988
32.2TheRabin-Karpalgorithm 990
32.3Stringmatchingwithfiniteautomata 995 ?
32.4TheKnuth-Morris-Prattalgorithm 1002
33ComputationalGeometry 1014
33.1Line-segmentproperties 1015
33.2Determiningwhetheranypairofsegmentsintersects 1021
33.3Findingtheconvexhull 1029
33.4Findingtheclosestpairofpoints 1039
34NP-Completeness 1048
34.1Polynomialtime 1053
34.2Polynomial-timeverification 1061
34.3NP-completenessandreducibility 1067
34.4NP-completenessproofs 1078
34.5NP-completeproblems 1086
35ApproximationAlgorithms 1106
35.1Thevertex-coverproblem 1108
35.2Thetraveling-salesmanproblem 1111
35.3Theset-coveringproblem 1117
35.4Randomizationandlinearprogramming 1123
35.5Thesubset-sumproblem 1128
VIIIAppendix:MathematicalBackground
Introduction 1143
ASummations 1145
A.1Summationformulasandproperties 1145
A.2Boundingsummations 1149
BSets,Etc. 1158
B.1Sets 1158
B.2Relations 1163
B.3Functions 1166
B.4Graphs 1168
B.5Trees 1173
CCountingandProbability 1183
C.1Counting 1183
C.2Probability 1189
C.3Discreterandomvariables 1196
C.4Thegeometricandbinomialdistributions 1201 ?
C.5Thetailsofthebinomialdistribution 1208
DMatrices 1217
D.1Matricesandmatrixoperations 1217
D.2Basicmatrixproperties 1222
Bibliography 1231 Index 1251
Preface
Beforetherewerecomputers,therewerealgorithms.Butnowthattherearecomputers,thereareevenmorealgorithms,andalgorithmslieattheheartofcomputing.
Thisbookprovidesacomprehensiveintroductiontothemodernstudyofcomputeralgorithms.Itpresentsmanyalgorithmsandcoverstheminconsiderable depth,yetmakestheirdesignandanalysisaccessibletoalllevelsofreaders.We havetriedtokeepexplanationselementarywithoutsacrificingdepthofcoverage ormathematicalrigor.
Eachchapterpresentsanalgorithm,adesigntechnique,anapplicationarea,ora relatedtopic.AlgorithmsaredescribedinEnglishandinapseudocodedesignedto bereadablebyanyonewhohasdonealittleprogramming.Thebookcontains244 figures—manywithmultipleparts—illustratinghowthealgorithmswork.Since weemphasize efficiency asadesigncriterion,weincludecarefulanalysesofthe runningtimesofallouralgorithms.
Thetextisintendedprimarilyforuseinundergraduateorgraduatecoursesin algorithmsordatastructures.Becauseitdiscussesengineeringissuesinalgorithm design,aswellasmathematicalaspects,itisequallywellsuitedforself-studyby technicalprofessionals.
Inthis,thethirdedition,wehaveonceagainupdatedtheentirebook.The changescoverabroadspectrum,includingnewchapters,revisedpseudocode,and amoreactivewritingstyle.
Totheteacher
Wehavedesignedthisbooktobebothversatileandcomplete.Youshouldfindit usefulforavarietyofcourses,fromanundergraduatecourseindatastructuresup throughagraduatecourseinalgorithms.Becausewehaveprovidedconsiderably morematerialthancanfitinatypicalone-termcourse,youcanconsiderthisbook tobea“buffet”or“smorgasbord”fromwhichyoucanpickandchoosethematerial thatbestsupportsthecourseyouwishtoteach.
Youshouldfinditeasytoorganizeyourcoursearoundjustthechaptersyou need.Wehavemadechaptersrelativelyself-contained,sothatyouneednotworry aboutanunexpectedandunnecessarydependenceofonechapteronanother.Each chapterpresentstheeasiermaterialfirstandthemoredifficultmateriallater,with sectionboundariesmarkingnaturalstoppingpoints.Inanundergraduatecourse, youmightuseonlytheearliersectionsfromachapter;inagraduatecourse,you mightcovertheentirechapter.
Wehaveincluded957exercisesand158problems.Eachsectionendswithexercises,andeachchapterendswithproblems.Theexercisesaregenerallyshortquestionsthattestbasicmasteryofthematerial.Somearesimpleself-checkthought exercises,whereasothersaremoresubstantialandaresuitableasassignedhomework.Theproblemsaremoreelaboratecasestudiesthatoftenintroducenewmaterial;theyoftenconsistofseveralquestionsthatleadthestudentthroughthesteps requiredtoarriveatasolution.
Departingfromourpracticeinpreviouseditionsofthisbook,wehavemade publiclyavailablesolutionstosome,butbynomeansall,oftheproblemsandexercises.OurWebsite,http://mitpress.mit.edu/algorithms/,linkstothesesolutions. Youwillwanttocheckthissitetomakesurethatitdoesnotcontainthesolutionto anexerciseorproblemthatyouplantoassign.Weexpectthesetofsolutionsthat weposttogrowslowlyovertime,soyouwillneedtocheckiteachtimeyouteach thecourse.
Wehavestarred(?)thesectionsandexercisesthataremoresuitableforgraduate studentsthanforundergraduates.Astarredsectionisnotnecessarilymoredifficultthananunstarredone,butitmayrequireanunderstandingofmoreadvanced mathematics.Likewise,starredexercisesmayrequireanadvancedbackgroundor morethanaveragecreativity.
Tothestudent
Wehopethatthistextbookprovidesyouwithanenjoyableintroductiontothe fieldofalgorithms.Wehaveattemptedtomakeeveryalgorithmaccessibleand interesting.Tohelpyouwhenyouencounterunfamiliarordifficultalgorithms,we describeeachoneinastep-by-stepmanner.Wealsoprovidecarefulexplanations ofthemathematicsneededtounderstandtheanalysisofthealgorithms.Ifyou alreadyhavesomefamiliaritywithatopic,youwillfindthechaptersorganizedso thatyoucanskimintroductorysectionsandproceedquicklytothemoreadvanced material.
Thisisalargebook,andyourclasswillprobablycoveronlyaportionofits material.Wehavetried,however,tomakethisabookthatwillbeusefultoyou nowasacoursetextbookandalsolaterinyourcareerasamathematicaldesk referenceoranengineeringhandbook.
Whataretheprerequisitesforreadingthisbook?
Youshouldhavesomeprogrammingexperience.Inparticular,youshouldunderstandrecursiveproceduresandsimpledatastructuressuchasarraysand linkedlists.
Youshouldhavesomefacilitywithmathematicalproofs,andespeciallyproofs bymathematicalinduction.Afewportionsofthebookrelyonsomeknowledge ofelementarycalculus.Beyondthat,PartsIandVIIIofthisbookteachyouall themathematicaltechniquesyouwillneed.
Wehaveheard,loudandclear,thecalltosupplysolutionstoproblemsand exercises.OurWebsite,http://mitpress.mit.edu/algorithms/,linkstosolutionsfor afewoftheproblemsandexercises.Feelfreetocheckyoursolutionsagainstours. Weask,however,thatyoudonotsendyoursolutionstous.
Totheprofessional
Thewiderangeoftopicsinthisbookmakesitanexcellenthandbookonalgorithms.Becauseeachchapterisrelativelyself-contained,youcanfocusinonthe topicsthatmostinterestyou.
Mostofthealgorithmswediscusshavegreatpracticalutility.Wetherefore addressimplementationconcernsandotherengineeringissues.Weoftenprovide practicalalternativestothefewalgorithmsthatareprimarilyoftheoreticalinterest. Ifyouwishtoimplementanyofthealgorithms,youshouldfindthetranslationofourpseudocodeintoyourfavoriteprogramminglanguagetobeafairly straightforwardtask.Wehavedesignedthepseudocodetopresenteachalgorithm clearlyandsuccinctly.Consequently,wedonotaddresserror-handlingandother software-engineeringissuesthatrequirespecificassumptionsaboutyourprogrammingenvironment.Weattempttopresenteachalgorithmsimplyanddirectlywithoutallowingtheidiosyncrasiesofaparticularprogramminglanguagetoobscure itsessence.
Weunderstandthatifyouareusingthisbookoutsideofacourse,thenyou mightbeunabletocheckyoursolutionstoproblemsandexercisesagainstsolutions providedbyaninstructor.OurWebsite,http://mitpress.mit.edu/algorithms/,links tosolutionsforsomeoftheproblemsandexercisessothatyoucancheckyour work.Pleasedonotsendyoursolutionstous.
Toourcolleagues
Wehavesuppliedanextensivebibliographyandpointerstothecurrentliterature. Eachchapterendswithasetofchapternotesthatgivehistoricaldetailsandreferences.Thechapternotesdonotprovideacompletereferencetothewholefield
ofalgorithms,however.Thoughitmaybehardtobelieveforabookofthissize, spaceconstraintspreventedusfromincludingmanyinterestingalgorithms.
Despitemyriadrequestsfromstudentsforsolutionstoproblemsandexercises, wehavechosenasamatterofpolicynottosupplyreferencesforproblemsand exercises,toremovethetemptationforstudentstolookupasolutionratherthanto finditthemselves.
Changesforthethirdedition
Whathaschangedbetweenthesecondandthirdeditionsofthisbook?Themagnitudeofthechangesisonaparwiththechangesbetweenthefirstandsecond editions.Aswesaidaboutthesecond-editionchanges,dependingonhowyou lookatit,thebookchangedeithernotmuchorquiteabit.
Aquicklookatthetableofcontentsshowsthatmostofthesecond-editionchaptersandsectionsappearinthethirdedition.Weremovedtwochaptersandone section,butwehaveaddedthreenewchaptersandtwonewsectionsapartfrom thesenewchapters.
Wekeptthehybridorganizationfromthefirsttwoeditions.Ratherthanorganizingchaptersbyonlyproblemdomainsoraccordingonlytotechniques,thisbook haselementsofboth.Itcontainstechnique-basedchaptersondivide-and-conquer, dynamicprogramming,greedyalgorithms,amortizedanalysis,NP-Completeness, andapproximationalgorithms.Butitalsohasentirepartsonsorting,ondata structuresfordynamicsets,andonalgorithmsforgraphproblems.Wefindthat althoughyouneedtoknowhowtoapplytechniquesfordesigningandanalyzingalgorithms,problemsseldomannouncetoyouwhichtechniquesaremostamenable tosolvingthem.
Hereisasummaryofthemostsignificantchangesforthethirdedition:
WeaddednewchaptersonvanEmdeBoastreesandmultithreadedalgorithms, andwehavebrokenoutmaterialonmatrixbasicsintoitsownappendixchapter.
Werevisedthechapteronrecurrencestomorebroadlycoverthedivide-andconquertechnique,anditsfirsttwosectionsapplydivide-and-conquertosolve twoproblems.ThesecondsectionofthischapterpresentsStrassen’salgorithm formatrixmultiplication,whichwehavemovedfromthechapteronmatrix operations.
Weremovedtwochaptersthatwererarelytaught:binomialheapsandsorting networks.Onekeyideainthesortingnetworkschapter,the0-1principle,appearsinthiseditionwithinProblem8-7asthe0-1sortinglemmaforcompareexchangealgorithms.ThetreatmentofFibonacciheapsnolongerrelieson binomialheapsasaprecursor.
Werevisedourtreatmentofdynamicprogrammingandgreedyalgorithms.Dynamicprogrammingnowleadsoffwithamoreinterestingproblem,rodcutting, thantheassembly-lineschedulingproblemfromthesecondedition.Furthermore,weemphasizememoizationabitmorethanwedidinthesecondedition, andweintroducethenotionofthesubproblemgraphasawaytounderstand therunningtimeofadynamic-programmingalgorithm.Inouropeningexampleofgreedyalgorithms,theactivity-selectionproblem,wegettothegreedy algorithmmoredirectlythanwedidinthesecondedition.
Thewaywedeleteanodefrombinarysearchtrees(whichincludesred-black trees)nowguaranteesthatthenoderequestedfordeletionisthenodethatis actuallydeleted.Inthefirsttwoeditions,incertaincases,someothernode wouldbedeleted,withitscontentsmovingintothenodepassedtothedeletion procedure.Withournewwaytodeletenodes,ifothercomponentsofaprogram maintainpointerstonodesinthetree,theywillnotmistakenlyendupwithstale pointerstonodesthathavebeendeleted.
Thematerialonflownetworksnowbasesflowsentirelyonedges.Thisapproachismoreintuitivethanthenetflowusedinthefirsttwoeditions.
WiththematerialonmatrixbasicsandStrassen’salgorithmmovedtoother chapters,thechapteronmatrixoperationsissmallerthaninthesecondedition.
WehavemodifiedourtreatmentoftheKnuth-Morris-Prattstring-matchingalgorithm.
Wecorrectedseveralerrors.MostoftheseerrorswerepostedonourWebsite ofsecond-editionerrata,butafewwerenot.
Basedonmanyrequests,wechangedthesyntax(asitwere)ofourpseudocode. Wenowuse“ D ”toindicateassignmentand“==”totestforequality,justasC, C++,Java,andPythondo.Likewise,wehaveeliminatedthekeywords do and then andadopted“//”asourcomment-to-end-of-linesymbol.Wealsonowuse dot-notationtoindicateobjectattributes.Ourpseudocoderemainsprocedural, ratherthanobject-oriented.Inotherwords,ratherthanrunningmethodson objects,wesimplycallprocedures,passingobjectsasparameters.
Weadded100newexercisesand28newproblems.Wealsoupdatedmany bibliographyentriesandaddedseveralnewones.
Finally,wewentthroughtheentirebookandrewrotesentences,paragraphs, andsectionstomakethewritingclearerandmoreactive.
Website
YoucanuseourWebsite,http://mitpress.mit.edu/algorithms/,toobtainsupplementaryinformationandtocommunicatewithus.TheWebsitelinkstoalistof knownerrors,solutionstoselectedexercisesandproblems,and(ofcourse)alist explainingthecornyprofessorjokes,aswellasothercontentthatwemightadd. TheWebsitealsotellsyouhowtoreporterrorsormakesuggestions.
Howweproducedthisbook
Likethesecondedition,thethirdeditionwasproducedinLATEX2" .Weusedthe TimesfontwithmathematicstypesetusingtheMathTimePro2fonts.Wethank MichaelSpivakfromPublishorPerish,Inc.,LanceCarnesfromPersonalTeX, Inc.,andTimTregubovfromDartmouthCollegefortechnicalsupport.Asinthe previoustwoeditions,wecompiledtheindexusingWindex,aCprogramthatwe wrote,andthebibliographywasproducedwithB IB TEX.ThePDFfilesforthis bookwerecreatedonaMacBookrunningOS10.5.
WedrewtheillustrationsforthethirdeditionusingMacDrawPro,withsome ofthemathematicalexpressionsinillustrationslaidinwiththepsfragpackage forLATEX2" .Unfortunately,MacDrawProislegacysoftware,havingnotbeen marketedforoveradecadenow.Happily,westillhaveacoupleofMacintoshes thatcanruntheClassicenvironmentunderOS10.4,andhencetheycanrunMacDrawPro—mostly.EvenundertheClassicenvironment,wefindMacDrawProto befareasiertousethananyotherdrawingsoftwareforthetypesofillustrations thataccompanycomputer-sciencetext,anditproducesbeautifuloutput.1 Who knowshowlongourpre-IntelMacswillcontinuetorun,soifanyonefromApple islistening: PleasecreateanOSX-compatibleversionofMacDrawPro!
Acknowledgmentsforthethirdedition
WehavebeenworkingwiththeMITPressforovertwodecadesnow,andwhata terrificrelationshipithasbeen!WethankEllenFaran,BobPrior,AdaBrunstein, andMaryReillyfortheirhelpandsupport.
Weweregeographicallydistributedwhileproducingthethirdedition,working intheDartmouthCollegeDepartmentofComputerScience,theMITComputer
1 WeinvestigatedseveraldrawingprogramsthatrununderMacOSX,butallhadsignificantshortcomingscomparedwithMacDrawPro.Webrieflyattemptedtoproducetheillustrationsforthis bookwithadifferent,wellknowndrawingprogram.Wefoundthatittookatleastfivetimesaslong toproduceeachillustrationasittookwithMacDrawPro,andtheresultingillustrationsdidnotlook asgood.HencethedecisiontoreverttoMacDrawProrunningonolderMacintoshes.
ScienceandArtificialIntelligenceLaboratory,andtheColumbiaUniversityDepartmentofIndustrialEngineeringandOperationsResearch.Wethankourrespectiveuniversitiesandcolleaguesforprovidingsuchsupportiveandstimulating environments.
JulieSussman,P.P.A.,onceagainbailedusoutasthetechnicalcopyeditor.Time andagain,wewereamazedattheerrorsthateludedus,butthatJuliecaught.She alsohelpedusimproveourpresentationinseveralplaces.IfthereisaHallofFame fortechnicalcopyeditors,Julieisasure-fire,first-ballotinductee.Sheisnothing shortofphenomenal.Thankyou,thankyou,thankyou,Julie!PriyaNatarajanalso foundsomeerrorsthatwewereabletocorrectbeforethisbookwenttopress.Any errorsthatremain(andundoubtedly,somedo)aretheresponsibilityoftheauthors (andprobablywereinsertedafterJuliereadthematerial).
ThetreatmentforvanEmdeBoastreesderivesfromErikDemaine’snotes, whichwereinturninfluencedbyMichaelBender.Wealsoincorporatedideas fromJavedAslam,BradleyKuszmaul,andHuiZhaintothisedition.
Thechapteronmultithreadingwasbasedonnotesoriginallywrittenjointlywith HaraldProkop.ThematerialwasinfluencedbyseveralothersworkingontheCilk projectatMIT,includingBradleyKuszmaulandMatteoFrigo.Thedesignofthe multithreadedpseudocodetookitsinspirationfromtheMITCilkextensionstoC andbyCilkArts’sCilk++extensionstoC++.
Wealsothankthemanyreadersofthefirstandsecondeditionswhoreported errorsorsubmittedsuggestionsforhowtoimprovethisbook.Wecorrectedallthe bonafideerrorsthatwerereported,andweincorporatedasmanysuggestionsas wecould.Werejoicethatthenumberofsuchcontributorshasgrownsogreatthat wemustregretthatithasbecomeimpracticaltolistthemall.
Finally,wethankourwives—NicoleCormen,WendyLeiserson,GailRivest, andRebeccaIvry—andourchildren—Ricky,Will,Debby,andKatieLeiserson; AlexandChristopherRivest;andMolly,Noah,andBenjaminStein—fortheirlove andsupportwhilewepreparedthisbook.Thepatienceandencouragementofour familiesmadethisprojectpossible.Weaffectionatelydedicatethisbooktothem.
T HOMAS H.C ORMEN
C HARLES E.L EISERSON
RONALD L.R IVEST
Lebanon,NewHampshire
Cambridge,Massachusetts
Cambridge,Massachusetts
C LIFFORD S TEIN NewYork,NewYork
February2009
1TheRoleofAlgorithmsinComputing
Whatarealgorithms?Whyisthestudyofalgorithmsworthwhile?Whatistherole ofalgorithmsrelativetoothertechnologiesusedincomputers?Inthischapter,we willanswerthesequestions.
1.1Algorithms
Informally,an algorithm isanywell-definedcomputationalprocedurethattakes somevalue,orsetofvalues,as input andproducessomevalue,orsetofvalues,as output.Analgorithmisthusasequenceofcomputationalstepsthattransformthe inputintotheoutput.
Wecanalsoviewanalgorithmasatoolforsolvingawell-specified computationalproblem.Thestatementoftheproblemspecifiesingeneraltermsthedesired input/outputrelationship.Thealgorithmdescribesaspecificcomputationalprocedureforachievingthatinput/outputrelationship.
Forexample,wemightneedtosortasequenceofnumbersintonondecreasing order.Thisproblemarisesfrequentlyinpracticeandprovidesfertilegroundfor introducingmanystandarddesigntechniquesandanalysistools.Hereishowwe formallydefinethe sortingproblem:
Input: Asequenceof n numbers ha1 ;a2 ;:::;an i
Output: Apermutation(reordering) ha 0 1 ;a 0 2 ;:::;a 0 n i oftheinputsequencesuch that a 0 1 a 0 2 a 0 n .
Forexample,giventheinputsequence h31;41;59;26;41;58i,asortingalgorithm returnsasoutputthesequence h26;31;41;41;58;59i.Suchaninputsequenceis calledan instance ofthesortingproblem.Ingeneral,an instanceofaproblem consistsoftheinput(satisfyingwhateverconstraintsareimposedintheproblem statement)neededtocomputeasolutiontotheproblem.
Becausemanyprogramsuseitasanintermediatestep,sortingisafundamental operationincomputerscience.Asaresult,wehavealargenumberofgoodsorting algorithmsatourdisposal.Whichalgorithmisbestforagivenapplicationdepends on—amongotherfactors—thenumberofitemstobesorted,theextenttowhich theitemsarealreadysomewhatsorted,possiblerestrictionsontheitemvalues, thearchitectureofthecomputer,andthekindofstoragedevicestobeused:main memory,disks,oreventapes.
Analgorithmissaidtobe correct if,foreveryinputinstance,ithaltswiththe correctoutput.Wesaythatacorrectalgorithm solves thegivencomputational problem.Anincorrectalgorithmmightnothaltatallonsomeinputinstances,orit mighthaltwithanincorrectanswer.Contrarytowhatyoumightexpect,incorrect algorithmscansometimesbeuseful,ifwecancontroltheirerrorrate.Weshallsee anexampleofanalgorithmwithacontrollableerrorrateinChapter31whenwe studyalgorithmsforfindinglargeprimenumbers.Ordinarily,however,weshall beconcernedonlywithcorrectalgorithms.
AnalgorithmcanbespecifiedinEnglish,asacomputerprogram,orevenas ahardwaredesign.Theonlyrequirementisthatthespecificationmustprovidea precisedescriptionofthecomputationalproceduretobefollowed.
Whatkindsofproblemsaresolvedbyalgorithms?
Sortingisbynomeanstheonlycomputationalproblemforwhichalgorithmshave beendeveloped.(Youprobablysuspectedasmuchwhenyousawthesizeofthis book.)Practicalapplicationsofalgorithmsareubiquitousandincludethefollowingexamples:
TheHumanGenomeProjecthasmadegreatprogresstowardthegoalsofidentifyingallthe100,000genesinhumanDNA,determiningthesequencesofthe 3billionchemicalbasepairsthatmakeuphumanDNA,storingthisinformationindatabases,anddevelopingtoolsfordataanalysis.Eachofthesesteps requiressophisticatedalgorithms.Althoughthesolutionstothevariousproblemsinvolvedarebeyondthescopeofthisbook,manymethodstosolvethese biologicalproblemsuseideasfromseveralofthechaptersinthisbook,thereby enablingscientiststoaccomplishtaskswhileusingresourcesefficiently.The savingsareintime,bothhumanandmachine,andinmoney,asmoreinformationcanbeextractedfromlaboratorytechniques.
TheInternetenablespeopleallaroundtheworldtoquicklyaccessandretrieve largeamountsofinformation.Withtheaidofcleveralgorithms,sitesonthe Internetareabletomanageandmanipulatethislargevolumeofdata.Examples ofproblemsthatmakeessentialuseofalgorithmsincludefindinggoodroutes onwhichthedatawilltravel(techniquesforsolvingsuchproblemsappearin
Chapter24),andusingasearchenginetoquicklyfindpagesonwhichparticular informationresides(relatedtechniquesareinChapters11and32).
Electroniccommerceenablesgoodsandservicestobenegotiatedandexchangedelectronically,anditdependsontheprivacyofpersonalinformationsuchascreditcardnumbers,passwords,andbankstatements.Thecore technologiesusedinelectroniccommerceincludepublic-keycryptographyand digitalsignatures(coveredinChapter31),whicharebasedonnumericalalgorithmsandnumbertheory.
Manufacturingandothercommercialenterprisesoftenneedtoallocatescarce resourcesinthemostbeneficialway.Anoilcompanymaywishtoknowwhere toplaceitswellsinordertomaximizeitsexpectedprofit.Apoliticalcandidate maywanttodeterminewheretospendmoneybuyingcampaignadvertisingin ordertomaximizethechancesofwinninganelection.Anairlinemaywish toassigncrewstoflightsintheleastexpensivewaypossible,makingsurethat eachflightiscoveredandthatgovernmentregulationsregardingcrewschedulingaremet.AnInternetserviceprovidermaywishtodeterminewheretoplace additionalresourcesinordertoserveitscustomersmoreeffectively.Allof theseareexamplesofproblemsthatcanbesolvedusinglinearprogramming, whichweshallstudyinChapter29.
Althoughsomeofthedetailsoftheseexamplesarebeyondthescopeofthis book,wedogiveunderlyingtechniquesthatapplytotheseproblemsandproblem areas.Wealsoshowhowtosolvemanyspecificproblems,includingthefollowing:
Wearegivenaroadmaponwhichthedistancebetweeneachpairofadjacent intersectionsismarked,andwewishtodeterminetheshortestroutefromone intersectiontoanother.Thenumberofpossibleroutescanbehuge,evenifwe disallowroutesthatcrossoverthemselves.Howdowechoosewhichofall possibleroutesistheshortest?Here,wemodeltheroadmap(whichisitself amodeloftheactualroads)asagraph(whichwewillmeetinPartVIand AppendixB),andwewishtofindtheshortestpathfromonevertextoanother inthegraph.WeshallseehowtosolvethisproblemefficientlyinChapter24.
Wearegiventwoorderedsequencesofsymbols, X Dhx1 ;x2 ;:::;xm i and Y Dhy1 ;y2 ;:::;yn i,andwewishtofindalongestcommonsubsequenceof X and Y .Asubsequenceof X isjust X withsome(orpossiblyallornone)of itselementsremoved.Forexample,onesubsequenceof hA;B;C;D;E;F;G i wouldbe hB;C;E;G i.Thelengthofalongestcommonsubsequenceof X and Y givesonemeasureofhowsimilarthesetwosequencesare.Forexample, ifthetwosequencesarebasepairsinDNAstrands,thenwemightconsider themsimilariftheyhavealongcommonsubsequence.If X has m symbols and Y has n symbols,then X and Y have 2m and 2n possiblesubsequences,
respectively.Selectingallpossiblesubsequencesof X and Y andmatching themupcouldtakeaprohibitivelylongtimeunless m and n areverysmall. WeshallseeinChapter15howtouseageneraltechniqueknownasdynamic programmingtosolvethisproblemmuchmoreefficiently.
Wearegivenamechanicaldesignintermsofalibraryofparts,whereeachpart mayincludeinstancesofotherparts,andweneedtolistthepartsinorderso thateachpartappearsbeforeanypartthatusesit.Ifthedesigncomprises n parts,thenthereare nŠ possibleorders,where nŠ denotesthefactorialfunction.
Becausethefactorialfunctiongrowsfasterthanevenanexponentialfunction, wecannotfeasiblygenerateeachpossibleorderandthenverifythat,within thatorder,eachpartappearsbeforethepartsusingit(unlesswehaveonlya fewparts).Thisproblemisaninstanceoftopologicalsorting,andweshallsee inChapter22howtosolvethisproblemefficiently.
Wearegiven n pointsintheplane,andwewishtofindtheconvexhullof thesepoints.Theconvexhullisthesmallestconvexpolygoncontainingthe points.Intuitively,wecanthinkofeachpointasbeingrepresentedbyanail stickingoutfromaboard.Theconvexhullwouldberepresentedbyatight rubberbandthatsurroundsallthenails.Eachnailaroundwhichtherubber bandmakesaturnisavertexoftheconvexhull.(SeeFigure33.6onpage1029 foranexample.)Anyofthe 2n subsetsofthepointsmightbethevertices oftheconvexhull.Knowingwhichpointsareverticesoftheconvexhullis notquiteenough,either,sincewealsoneedtoknowtheorderinwhichthey appear.Therearemanychoices,therefore,fortheverticesoftheconvexhull. Chapter33givestwogoodmethodsforfindingtheconvexhull.
Theselistsarefarfromexhaustive(asyouagainhaveprobablysurmisedfrom thisbook’sheft),butexhibittwocharacteristicsthatarecommontomanyinterestingalgorithmicproblems:
1.Theyhavemanycandidatesolutions,theoverwhelmingmajorityofwhichdo notsolvetheproblemathand.Findingonethatdoes,oronethatis“best,”can presentquiteachallenge.
2.Theyhavepracticalapplications.Oftheproblemsintheabovelist,findingthe shortestpathprovidestheeasiestexamples.Atransportationfirm,suchasa truckingorrailroadcompany,hasafinancialinterestinfindingshortestpaths througharoadorrailnetworkbecausetakingshorterpathsresultsinlower laborandfuelcosts.OraroutingnodeontheInternetmayneedtofindthe shortestpaththroughthenetworkinordertorouteamessagequickly.Ora personwishingtodrivefromNewYorktoBostonmaywanttofinddriving directionsfromanappropriateWebsite,orshemayuseherGPSwhiledriving.
Noteveryproblemsolvedbyalgorithmshasaneasilyidentifiedsetofcandidate solutions.Forexample,supposewearegivenasetofnumericalvaluesrepresentingsamplesofasignal,andwewanttocomputethediscreteFouriertransformof thesesamples.ThediscreteFouriertransformconvertsthetimedomaintothefrequencydomain,producingasetofnumericalcoefficients,sothatwecandetermine thestrengthofvariousfrequenciesinthesampledsignal.Inadditiontolyingat theheartofsignalprocessing,discreteFouriertransformshaveapplicationsindata compressionandmultiplyinglargepolynomialsandintegers.Chapter30gives anefficientalgorithm,thefastFourier transform(commonlycalledthe FFT),for thisproblem,andthechapteralsosketchesoutthedesignofahardwarecircuitto computetheFFT.
Datastructures
Thisbookalsocontainsseveraldatastructures.A datastructure isawaytostore andorganizedatainordertofacilitateaccessandmodifications.Nosingledata structureworkswellforallpurposes,andsoitisimportanttoknowthestrengths andlimitationsofseveralofthem.
Technique
Althoughyoucanusethisbookasa“cookbook”foralgorithms,youmaysomeday encounteraproblemforwhichyoucannotreadilyfindapublishedalgorithm(many oftheexercisesandproblemsinthisbook,forexample).Thisbookwillteachyou techniquesofalgorithmdesignandanalysissothatyoucandevelopalgorithmson yourown,showthattheygivethecorrectanswer,andunderstandtheirefficiency. Differentchaptersaddressdifferentaspectsofalgorithmicproblemsolving.Some chaptersaddressspecificproblems,suchasfindingmediansandorderstatisticsin Chapter9,computingminimumspanningtreesinChapter23,anddetermininga maximumflowinanetworkinChapter26.Otherchaptersaddresstechniques, suchasdivide-and-conquerinChapter4,dynamicprogramminginChapter15, andamortizedanalysisinChapter17.
Hardproblems
Mostofthisbookisaboutefficientalgorithms.Ourusualmeasureofefficiency isspeed,i.e.,howlonganalgorithmtakestoproduceitsresult.Therearesome problems,however,forwhichnoefficientsolutionisknown.Chapter34studies aninterestingsubsetoftheseproblems,whichareknownasNP-complete. WhyareNP-completeproblemsinteresting?First,althoughnoefficientalgorithmforanNP-completeproblemhaseverbeenfound,nobodyhaseverproven
thatanefficientalgorithmforonecannotexist.Inotherwords,nooneknows whetherornotefficientalgorithmsexistforNP-completeproblems.Second,the setofNP-completeproblemshastheremarkablepropertythatifanefficientalgorithmexistsforanyoneofthem,thenefficientalgorithmsexistforallofthem.This relationshipamongtheNP-completeproblemsmakesthelackofefficientsolutions allthemoretantalizing.Third,severalNP-completeproblemsaresimilar,butnot identical,toproblemsforwhichwedoknowofefficientalgorithms.Computer scientistsareintriguedbyhowasmallchangetotheproblemstatementcancause abigchangetotheefficiencyofthebestknownalgorithm.
YoushouldknowaboutNP-completeproblemsbecausesomeofthemarisesurprisinglyofteninrealapplications.Ifyouarecalledupontoproduceanefficient algorithmforanNP-completeproblem,youarelikelytospendalotoftimeina fruitlesssearch.IfyoucanshowthattheproblemisNP-complete,youcaninstead spendyourtimedevelopinganefficientalgorithmthatgivesagood,butnotthe bestpossible,solution.
Asaconcreteexample,consideradeliverycompanywithacentraldepot.Each day,itloadsupeachdeliverytruckatthedepotandsendsitaroundtodelivergoods toseveraladdresses.Attheendoftheday,eachtruckmustendupbackatthedepot sothatitisreadytobeloadedforthenextday.Toreducecosts,thecompanywants toselectanorderofdeliverystopsthatyieldsthelowestoveralldistancetraveled byeachtruck.Thisproblemisthewell-known“traveling-salesmanproblem,”and itisNP-complete.Ithasnoknownefficientalgorithm.Undercertainassumptions, however,weknowofefficientalgorithmsthatgiveanoveralldistancewhichis nottoofarabovethesmallestpossible.Chapter35discussessuch“approximation algorithms.”
Parallelism
Formanyyears,wecouldcountonprocessorclockspeedsincreasingatasteady rate.Physicallimitationspresentafundamentalroadblocktoever-increasingclock speeds,however:becausepowerdensityincreasessuperlinearlywithclockspeed, chipsruntheriskofmeltingoncetheirclockspeedsbecomehighenough.Inorder toperformmorecomputationspersecond,therefore,chipsarebeingdesignedto containnotjustonebutseveralprocessing“cores.”Wecanlikenthesemulticore computerstoseveralsequentialcomputersonasinglechip;inotherwords,theyare atypeof“parallelcomputer.”Inordertoelicitthebestperformancefrommulticore computers,weneedtodesignalgorithmswithparallelisminmind.Chapter27 presentsamodelfor“multithreaded”algorithms,whichtakeadvantageofmultiple cores.Thismodelhasadvantagesfromatheoreticalstandpoint,anditformsthe basisofseveralsuccessfulcomputerprograms,includingachampionshipchess program.
1.2Algorithmsasatechnology11
Exercises
1.1-1
Giveareal-worldexamplethatrequiressortingorareal-worldexamplethatrequirescomputingaconvexhull.
1.1-2
Otherthanspeed,whatothermeasuresofefficiencymightoneuseinareal-world setting?
1.1-3
Selectadatastructurethatyouhaveseenpreviously,anddiscussitsstrengthsand limitations.
1.1-4
Howaretheshortest-pathandtraveling-salesmanproblemsgivenabovesimilar? Howaretheydifferent?
1.1-5
Comeupwithareal-worldprobleminwhichonlythebestsolutionwilldo.Then comeupwithoneinwhichasolutionthatis“approximately”thebestisgood enough.
1.2Algorithmsasatechnology
Supposecomputerswereinfinitelyfastandcomputermemorywasfree.Would youhaveanyreasontostudyalgorithms?Theanswerisyes,iffornootherreason thanthatyouwouldstillliketodemonstratethatyoursolutionmethodterminates anddoessowiththecorrectanswer.
Ifcomputerswereinfinitelyfast,anycorrectmethodforsolvingaproblem woulddo.Youwouldprobablywantyourimplementationtobewithinthebounds ofgoodsoftwareengineeringpractice(forexample,yourimplementationshould bewelldesignedanddocumented),butyouwouldmostoftenusewhichever methodwastheeasiesttoimplement.
Ofcourse,computersmaybefast,buttheyarenotinfinitelyfast.Andmemory maybeinexpensive,butitisnotfree.Computingtimeisthereforeabounded resource,andsoisspaceinmemory.Youshouldusetheseresourceswisely,and algorithmsthatareefficientintermsoftimeorspacewillhelpyoudoso.
