ElementsofMathematicalOptimization
1.1INTRODUCTION
The HandbookofGlobalOptimization by HorstandPardalos(1995) (volume1)introducedtotheoptimizationtechniques, suchasconcaveoptimization,DCoptimization,quadraticoptimization,complementaryproblems,minimax,multiplicative programmingproblems,Lipschitzoptimization,fractionalprogramming,networkflowoptimization,intervalmethods,and stochasticprogramming(two-phasemethods,randomsearchmethods,simulatedannealing,etc.).Thesecondvolumeof the HandbookofGlobalOptimization by PardalosandRomeijn(2002) includedvariousmetaheuristicssuchassimulated annealing,geneticalgorithms(GAs),neuralnetworks,taboosearch,shake-and-bakemethods,anddeformationmethods.
The HandbookofAppliedOptimization (with1095pages)by PardalosandResende(2002,pp.567–991) provided applicationsinavarietyofdomainsinagriculture(e.g.,forest),aerospace,biologyandchemistry,energy(e.g.,electrical, powersystems,oilandgas,nuclearengineering),environment(e.g.,airpollution),finance(e.g.,portfolioselection)and economics,manufacturing,mechanics,telecommunication,andtransportation.The EncyclopediaofOptimization inits fivevolumesandabout2710pagesby FloudasandPardalos(2001) collectspapersonabroadrangeofmethodsandtechnicalaspectsofnumerousapproachesandapplications.Inthesevolumes,allcontributionsareclassifiedaccordingtothe alphabeticalorderoftheirtitle.The HandbookofTestProblemsinLocalandGlobalOptimization (Floudasetal.,1999, 2010)containstestproblemsinlocalandglobaloptimizationforawiderangeofreal-worldproblems,1 forexample,quadraticprogramming,bilinear,biconvex,DC(differenceconvex)problems.
Severalbooksalsocovermostoftheproblemsandapplicationsinglobaloptimization.Thebook PracticalOptimization by Gill,Murray,andWright(1981) isonpracticaloptimization.Thebooktreatsoftheoptimalityconditions,unconstrained methodsforunivariatefunctions,multivariatenonsmoothfunctions,nonderivativemethods,methodsforlarge-scale problemsandpracticalities(useofsoftwarepackages,computedsolutionsproperties,accuracy,scaling).Thebook Global Optimization:DeterministicApproaches by HorstandTuy(1996) alsocontainsparametricconcaveprogramming,outer approximation,branch-and-boundtechnique,decompositionoflarge-scaleproblems,andparticularchallengesofconcave minimization(bilinearprogramming,complementaryproblems).Therelevanttextbookby GeigerandKanzow(2000) (in German)introducesthetheoryandpracticeofthenumericallyconstrainedoptimization.Thebookincludesoptimalityconditions,linearprogramming,nonlinearoptimization,andnonsmoothoptimization(lagrangianduality,regularizationprocessestoimprovetheoptimalityconditions,subgradientmethods,andboundedapproximation).Thetextbook Linear andNonlinearProgramming by LuenbergerandYee(2008) extendsthispresentationtoconstrainedminimizationproblems byusingprimalmethods,penaltyandbarriermethods,dual-and-cuttingplanemethods,andprimal-dualmethods.Thebook by Hastings(2006) introducesthereaderstotheextendeddomainofoperationsresearchtechniquesbyusingthesoftware package Mathematica®.Theelectronicbook(eBook)by Weise(2009) onglobaloptimizationalgorithmsfocusedonevolutionarycomputationalgorithms,includingGAs,geneticprogramming(GP),learningclassifiersystems,evolutionstrategy (ES),differentialevolution(DE),particleswarmoptimization(PSO),antcolonyoptimization.Thesecondeditionofthis eBookincludes2335references,forwhichlinksaregenerallyprovided.
Fornonconvexproblems,anumberoftechniquesofconvexificationhavebeenproposed,butotheralgorithmshave beenintroducedtosolvethiscomplexityofreal-lifeoptimizationproblems. Holland(1975) describedtwomainfactors thatpermitthedevelopmentofsuchGAs.Thefirstfactoristhecomputationpowersofparallelmachinesandthesecond aninterdisciplinarycooperationbetweenresearchers.Thebook GeneticProgramming:OntheProgrammingofComputers byMeansofNaturalSelection by Koza(1992) introducedtoESandevolutionarycomputation(seealso Jacob,2001).
Theconvexityoffunctionsandsetsinanoptimizationproblemisafundamentalconcept.Thefoundationsoftheconvex analysis(e.g.,propertiesofconvexityanddualitycorrespondences)arenotablypresentedinthebook ConvexAnalysis by Rockafellar(1970) (seealso Hiriart-Urruty&Lemarechal,2000). Thebook ConvexOptimization by Boydand Vandenberghe(2004) iscenteredonthetheoryandpracticeoftheconvexoptimization(seealso Bertsekas,2009).Infact, weknowthatprimal-dualmethodsrequireaconvexstructure(atleastlocally).Ineconomics,afundamentalproblemconsists
inallocatingscarceresourcesamongalternatingpurposes.Wethenhavetodeterminetheinstrumentswithinafeasibleset (reflectingthescarcityofresources)soastomaximizetheobjectiveprovidedin Intriligator(1971), Intriligator(1981),and ArrowandIntriligator(1981).Theconvexityassumptionisanecessaryconditionfortheexistenceofanequilibriumallocation.
1.2HISTORYOFMATHEMATICALOPTIMIZATION
Thehistoricaldevelopmentofmathematicaloptimizationconsistsofthreebroadapproaches,namelytheclassicalmethods, theevolutionaryalgorithms(EAs)andmorerecentlythehybridmethods.Theconventionalpracticemethodsfocuson optimizingasingleobjective,withorwithoutadditionalconstraints.ThepracticeofEAshasbeenmorerecentlydeveloped tosolvethemostdifficultcasesofoptimizationproblemswithseveralobjectives.Itisremarkabletoobservethatthesetwo problemsalsohaveancientoriginsdatingbacktothe19thcentury.Indeedtheoriginofvectoroptimizationgoesbackto Edgeworth(1881) and Pareto(1896).Thetwoeconomistsdevelopedthetheoryofindifferencecurvesanddefinedthebasic conceptofoptimalityinmultiobjectiveoptimization(MOO).
1.2.1OriginandEvolutionofClassicalMethods
Thefoundationofmathematicalprogrammingreliesontwomajorscientificworks:thepublicationofthe TheoryofGames andEconomicBehavior by vonNeumannandMorgenstern(1953) andthediscoveringofthesimplexmethodbyGeorgeB. Dantzigin1947(see Dantzig&Wolfe,1960).Inthesameyear,JohnvonNeumanndevelopedthetheoryofduality. Ashorthistoryby Minoux(1986) identifiedfourdecadesofdevelopmentinmathematicalprogramminguntil1987.The first10years havebeendevotedtolinearprogrammingandtheoreticalfoundationsofnonlinearprogramming.The second decade hasseentheintroductionofthefollowingtechniques,namelyintegerprogramming,networktheory,nonconvex programming,dynamicprogrammingandcontroltheory.Decompositiontechniquesweredevelopedinthesameperiodfor solvinglarge-sizesystems.The thirddecade hasseenthedevelopmentofatheoryofnondifferentiable/nonsmoothoptimizationandthecombinationofmathematicalprogrammingwithgraphtheoryleadingtocombinatorialoptimization(see Papadimitriou&Steiglitz,1982).The fourthdecade ofoptimizationwasinfluencedbytheintroductionofcomputational complexity(see Papadimitriou,1995).Morerecently,thehistoryofoptimizationtheorywasdividedinto threemajorwaves accordingto Chiang(2009).The firstwave wasattributedtolinearprogrammingandsimplexmethodinthelate1940s,the secondwave waswithconvexoptimizationandinteriorpointmethodattheendofthe1980s.The thirdwave wascharacterizedbythenonconvexoptimization.
1.2.2DevelopmentofEvolutionaryAlgorithms2
Thefirstuseofheuristicalgorithmsgoesbackto19483 when Turing(1948) wasbreakingtheGerman Enigmacode during WorldWarII(seealso Angelov,2016;Yang,2014).Thereafter,heuristicandmetaheuristicalgorithmsforsolvingprogrammingproblemswereissuedfromthedifficultieswithclassicaloptimizationmethods.
Abido(2010) mentioned fourinconveniences forsolvingMOOproblemswithconventionalalgorithms:(1)aneedfora repetitiveapplicationofanalgorithmtofindthePareto-optimalsolutions,(2)arequirementofsomeknowledgeaboutthe problem,(3)thesensitivityofanalgorithmtotheshapeofthePareto-optimalfront,and(4)thespreadofthePareto-optimal solutionsdependingonthechosenalgorithm.Heuristicalgorithmsaresuitablesolversforseverehigh-dimensional,reallifeproblems(see Tong,Chowdhury,&Messac,2014).4
Heuristicsandmetaheuristicsrefertoapproximationresolutionmethods.Heuristicsdenotetechniqueswhichseeknearoptimalsolutionsatalowcost.Metaheuristicsarecharacterizedbyamasterstrategy.Theycanguideandcorrecttheoperationsofsubordinateheuristics(see Reeves,1995).Thus,metaheuristicssuchasEAsmayrefertoahigherlevelprocedure, whichcombinesdifferentoperationsofheuristicsforexploringasearcharea.
EAsincludenotablyGAs,ES,andGP.EAsalsoinclude,butarenotlimitedto,nature-inspiredalgorithmssuchas neuralmethods,simulatedannealing,tabusearch,antcolonysystemsandotherparticleswarmintelligencetechniques. ThecapacityofsuchmethodstosolveNP-hard5 combinatorialproblemsiswell-known(e.g.,theproblemsoftraveling salesperson,scheduling,graph,andtransportation).Thebookby Michalewicz(1999) introducedmetaheuristicsforsolving numericaloptimizationproblems.AnoverviewofevolutionarytechniqueswithapplicationsisproposedbyN.Srinivasand K.Deb’ssortingGA(Srinivas&Deb,1994),C.M.FonsecaandP.J.Fleming’smultiobjectiveGA(see Fonseca&Fleming, 1993,1995),P.HajelaandL.Lee’sweighted-basedGA(see Hajela&Lee,1996;Zitzler,1999),includingSchaffer’s vector-evaluatedGA(see Schaffer,1984).
EAsaremainlybasedonprinciplesoftheDarwinianevolutioncharacterizedasfollows.Individualswithinpopulations (orspecies)differ.Traitsarepassedontooffspring.Moreoffspringareproducedthancansurviveineverygeneration.The
memberswhosurvivearenaturallyselectedwithmostfavorableperformances.Thisnaturalprocessisbasedonindividuals withconsequencesonthecorrespondingpopulation.Thisevolutionprocessisbackward,mostlydeterministic (i.e.,partiallyrandom).Itisnotperfectandcanproducenewtraitsbesidesexistingtraits.Suchalgorithmisregarded aspopulation-basedstochasticalgorithms,whichelementsincludeapopulationofindividuals,fitnessevaluation,genetic operatorsguidingevolution,andselection.
OneshouldindicatethefastdevelopmentofaMOOapproachinthemid-1980swiththehelpofEAs.6 Anearlyattempt touseGAstosolveMOOproblemswasrealizedby Ito,Akagi,andNishikawa(1983) Goldberg(1989) proposedPareto-set fitnessassignmenttosolveSchaffer’smultiobjectiveproblems.Inthesameperiod,twobooksweredevotedtothetheory andtechniquesofMOO,suchas ChangkongandHaimes(1983) andthatof Sawaragi,Nakayama,andTanino(1985).The fastexpansionofthisapproachwasstimulatedbynumerousreal-worldapplicationsfromscience,technology,management,andfinance. Rangaiah(2009) wasthefirstpublicationonMOOwithafocusonchemicalengineering.Theapplicationsinthisareaarenotablyinchemical,mineralprocessing,oilandgas,petroleum,pharmaceuticalindustries,andso on. LaiandHwang(1994) extendedtheMOOapproachtofuzzydecision-makingproblems.7
Thefirstuseofgenetic-basedsearchalgorithmstoMOOproblemsgoesbacktothepioneeringworkof Rosenberg (1967) (seealso Coello,1999).Inhisbriefhistoryofmetaheuristics, Yang(2014,pp.16–20) specifiedtherelevantdecades ofthedevelopmentofEAs.The1960sand1970sresearchersknewthedevelopmentofGAsattheUniversityofMichigan. ThecontributionofJohnHollandin1975(see Holland,1975)proposedasearchmethodbasedontheDarwinianevolution conceptsandnaturalprinciplesofbiologicalsystems.Crossover,mutation,andselectionoperatorswereusedtosolvedifficultcombinatorialproblems.Inthesameperiod,evolutionarystrategieswereinitiatedattheTechnicalUniversityof Berlin.IngoRechenbergin1971(see Rechenberg,1973)(inGerman)and Schwefel(1977) (inGerman)proposedasearch methodforsolvingoptimizationproblems. Fogel(1994) introducedtheevolutionaryprogrammingbyusingsimulatedevolutionasalearningprocess.8
FollowingYang,thedecades1980sand1990swerefruitfulstepsformetaheuristicalgorithms. Kirkpatrick,Gelatt,and Vecchi(1983) pioneeredthesimulatedannealingalgorithmin1983.Thismethodwasinspiredbytheannealingprocessof metals.In1986,theuseofmemorywasproposedbyFredGlover’stabusearchin Glover(1986).In1992,thesearchtechniquebyMarcoDorigoin Dorigo(1992) wasinspiredbytheswarmintelligenceofantcoloniesusingapheromoneto communicate.Laterin1995, KennedyandEberhart(1995) developedthePSO,inspiredbytheswarmintelligenceoffish andbirds.In1997, StornandPricee(1997) proposedthedifferentialevolution(DE)algorithm.Thisvector-basedEA provedtobemoreefficientthanageneticalgorithm.
Intherecentyears,othernature-inspiredalgorithmswereintroducedsuchasharmonysearch(HS)algorithmfordistribution,transportandscheduling2001,honeybeealgorithms2004,fireflyalgorithm(FA)2007,cuckoosearchalgorithm (CSA)2009,batalgorithm(BA)2010basedonecholocationbehavior,andflowerpollinationalgorithm(FPA)2012.
1.2.3ContemporaryEmergenceofHybridApproaches
Hybridevolutionalgorithmsarealsonamed“memeticalgorithms”(MAs)(see Ishibuchi&Yoshida,2002). Grosanand Abraham(2007) emphasizedtheneedforhybridEAsinhandlingreal-worldproblemsinvolvingcomplexitiesandvarious uncertainties(e.g.,noisyenvironment,imprecisionofdata,vaguenessinthedecisions).
KnowlesandCorne(2005) reviewedMAsforMOOproblems. Mashwani(2011) surveyedthehybridMOEAsshowing howhybridizationcanbedesigned(1)touseonealgorithmandimproveitwithothertechniques,(2)tousemultipleoperatorsinanEA,and(3)tobetterMOGAsolutionsbyimplementingeffectivelocalsearch.ThealgorithmMemetic-PAES proposedby KnowlesandCornee(2005) combinedthelocalsearchstrategyinthePareto-archivedevolutionstrategy (PAES)withtheuseofGA. Thangaraj,Pant,Abraham,andBouvry(2011) reviewedthehybridoptimizationtechnique inwhichthemainalgorithmisPSOwithcombinedalocalandaglobalsearchalgorithm. Zamuda,Brest,Boskovic,and Zumer(2009) retainedDEasanoriginalalgorithmcoupledwithalocalsearchstrategy. Wang,Cai,Guo,andZhou(2007) extendedthehybridalgorithmswithglobalandlocalsearchstrategiesforsolvingconstrainedMOOproblems. Garrettand Dasgupta(2006) analyzedtheperformancesofhybridEAsformultiobjectivequadraticassignmentproblems(QAPs).The inclusionoflocalsearchesgenerallyimprovestheperformancesofMOEAs.
Thebasicideaisapplyingalocalsearchtonewoffspring.Then,improvedoffspringcompetewiththepopulationof survivalstothenestgeneration. TangandWang(2013) reviewedthenewtrendfordevelopinghybridMOEAsbycombiningconceptsandcomponentsofdifferentmetaheuristics. Whitley,Gordon,andMathias(1994) identifiedtwoforms ofhybridgeneticsearch.Thefirsttypeuses Lamarkianevolution ,andthesecondwayintroducesan additionallocalsearch Inthisstudy,wepresenttheLamarkianstrategysearchaswiththeMOGLSandtheAbYSSalgorithms(i.e.,Adapter ScatterSearchalgorithms).TheZDT4testfunctiondemonstratestheperformancesofsuchalgorithmstogeneratethe Pareto-optimalfront.
1.3FORMULATIONOFOPTIMIZATIONPROBLEMS
Decisionproblemsusuallyinvolveonlyonegoaltoachieve.Decisionmakersmayalsohaveseveralconflictingobjectives toachieve.Aspecificformaltreatmentisnecessaryforsuchproblems.Thesetoffeasiblesolutionsisboundedbyconstraintstosatisfy.
1.3.1Single-ObjectiveOptimization
Lettheoptimizationproblem
minimize x f x ðÞ subjectto : x 2 X ℝn fg
where X isthefeasibleset(or“opportunityset”ineconomics).
Anonstrictglobalminimumissuchthatthesolutionvector x* satisfies x* 2 X and f x* ðÞ f x ðÞ forall x 2 X .Alocal minimumoftheobjectivefunction f over X is x* 2 X forwhichthereexistssomesmall e > 0suchthat f x* ðÞ f x ðÞ forall x 2 X with x x* kk < e.Moregenerally,aconvexconstrainedoptimizationprogramissuchthattheobjectivefunctionis convexandthattheconstraintsareconcave(inequalities),orbothconcaveandconvex(equalities).Formally,aminimizationproblem,with m inequalityconstraintsand p equalityconstraintsisrepresentedby
minimize x f x ðÞ subjectto : gxðÞ 0 hxðÞ¼ 0 x 2 X ℝn
where f : ℝn 7!ℝ, g : ℝn 7!ℝm ,and h : ℝn 7!ℝp .Anoptimizationproblemisoftenillustratedbyamaximizingproblem, consistingofaquadraticobjectivefunctionandlinearconstraints.Theunconstrainedandconstrainedproblemsareboth examinedhere.
Theunconstrainedproblemis
maximize x f x ðÞ cT x + 1 2 xT Qx
where x 2 ℝn , Q isa n n regularsymmetricmatrix, c isacolumnvectorofconstants.Usingthefirst-orderconditions rf ^ x ðÞ¼ c + Q^ x ¼ 0,wefindtheoptimalvalues ^ x ¼ Q 1 c and f ^ x ðÞ¼ 1 2 cT Q 1 c > 0.If Q isnegativedefinite f(x)is strictlyconcave,so ^ x isaglobalmaximum.Thequadratic-linearproblemis
maximize x f x ðÞ cT x + 1 2 x T Qx subjectto : Ax5b x 2 ℝn 8
where A isa m n matrixand c a n 1vectorofcoefficients.Let y the m 1vectorofmultipliersassociated withthe m constraints,theLagrangianforthisissueiswith n + m arguments.WehavetheLagrangian L x, y ðÞ¼ cT x + 1 2 x T Qx + y T b Ax ðÞ.Usingthe n+m first-orderconditions,wededuce x* ¼ Q 1 c AT y* and y* ¼ AQ 1 AT 1 b + AQ 1 c .Finally,wegetthetwoexpressions:x* ¼ Q 1 c AT AQ 1 AT 1 b + AQ 1 c and x* ¼ ^ x+Q 1 AT AQ 1 AT 1 b A^ x ðÞ,where ^ x istheoptimumoftheunconstrainedquadraticoptimizationproblem.9 Manyareasincludingmanufacturing,chemical,andbiologicalsciences,e ngineeringdesign,needanonconvex modeling.The nonconvexities maybeduetomultimodalobjectivefunctio ns,tointegerrequirements,andsoon.
However,thismultiplicityoflocalsolutionsmayalsobe duetononlinearitiesintheconstraintset,evenwhenthe objectivefunctionisconvex(see Tawarmalani&Sahinidis,2002,pp.1– 5).Optimizationproblemssuchasbi-level programmingaretypicalconvexandnondifferentiable.Forsuchoptimizationproblems,thestandardnonlinearprogrammingtechniques,whichmostlydependonthestartingpoint(e.g.,thesteepestdescent),willthenfailinfinding theglobaloptimumsolution.Theconsequencesofnonconvexitiesarewell-known:thei mpossibilitytodefineadual functional,theexistenceofadualitygap,andsoon(see Bazaraa,Sherali,&Shetty,2006 ,pp.257 – 314; Bertsekas, 2009,pp.216– 242).Examplesineconomicsshowthatnonconvex preferenceswillcausediscontinuitiesofthe demandfunctionsandthusthepossiblenonexistenceofequilibriumprices(see Varian,1992 ,pp.393 –394).Theglobal optimizationalgorithmsmaybe dividedintotwogroups:the deterministicapproach (e.g.,branch-and-bound,outerapproximation,cuttingplanes,decomposition)and stochasticheuristicallymethods (e.g.,randomsearch,GAs,ES,clusteringalgorithm).Atypologyofglobaloptimizationmethodscanbebasedonmathematicalstructuresasin Horstand Tuy(1996,pp.3 –51) and HendrixandToth(2010,pp.147– 159),suchasquadratic,bilinear,fractionalfunctions.The mainclassesofglobaloptimizationby HorstandTuy(1996) arethe“concaveminimization” (i.e.,aconcaveobjective functionandlinearandconvexconstraints),the“reverse convexprogramming”(i.e.,aconvexminimizationoverthe intersectionofconvexsetsandcomplementsofconvexsets),“DCprogramming”(i.e.,theobjectivefunctioncanbe expressedasadifferencebetweentwoconvexfunctions)and “Lipschitzoptimization”(i.e.,aLipschitzcontinuous objectivefunction,forwhichitsslopeisbounded).
1.3.2MultiobjectiveOptimization
AgeneralcontinuousMOOproblemstatestofind n continuousdecisionvariables x 2 ℝn thatsimultaneouslyminimize (andrespectivelymaximize) r objectivefunctions fk : ℝn 7!ℝ, k ¼ 1, …, r .Thesedecisionvariablesandobjectivesare subjecttorestrictionssuchasboundsandconstraints.
Decisionvariablestaketheirvalues10 inaclosedintervaldefinedbyalowerandanupperbound.Thereare2n bounds xi 2 x L i , xU i , i ¼ 1, , n.Theseboundsrepresentthedecisionspace.
Theobjectivesaresubjecttorestrictionsrepresentedby m inequalityfunctions gj : ℝn 7!ℝ, j ¼ 1, , m and p equalities11 hl : ℝn 7!ℝ, l ¼ 1, , p
ThebasicgenericMOOproblemtakesthefollowingform minimize x 2 X ℝn f1 x ðÞ, …, fr x ðÞ ðÞT (1.1)
Thefeasiblespaceisdefinedby X ¼ x 2 ℝn : gxðÞ 0, hxðÞ¼ 0, xi 2 x L i , xU i , i ¼ 1, , n where g : ℝn 7!ℝm and h : ℝn 7!ℝp
AfeasiblesolutiontotheMOOproblemsatisfiesallthe2n bounds,togetherwiththe m + p inequalitiesandequality constraints.
AMOO problem(1.1) maycontainavectorofparameters p.Thestandardformbecomes
minimize x;p f x; p ðÞ f1 x; p ðÞ, …, fr x; p ðÞ ðÞ subjectto : hi x; p ðÞ¼ 0, i ¼ 1, , p gj x; p ðÞ 0, j ¼ 1, , m xk 2 x L k , xU k , k ¼ 1, , n
1.4CLASSIFICATIONOFOPTIMIZATIONMETHODS
Aclassificationofoptimizationproblemscanbeperformedaccordingtotheirtechnicalcharacteristics.Thiswillbeour startingpointinspiredby SarkerandNewton(2008,pp.11–13),priortotheclassificationofsolvingmethods.Adistinction mustbemadebetweenmethodsthatsolvesingle-objectiveproblemsandprogrammingproblemswithmultipleobjectives. Inthefirstcase,werefertoS.S.Rao’sbook(Rao,2009).Inthesecondcase,wewillretaintheclassificationproposedby Miettinen(1999).
1.4.1ClassificationofOptimizationProblems
Theclassificationby SarkerandNewton(2008) illustratesaminimizationormaximizationproblem.Thisconstructionis basedontheprincipalfeaturesofanoptimizationproblem.Thecharacteristicsdifferentiatingtheoptimizationproblems arerelatedtothenumberofobjectives(i.e.,singleormultipleobjectives)andconstraints(i.e.,inequalityandequality constraints),thetypeofdesignvariables(i.e.,continuous,discreteormixedinteger),andthemathematicalproperties ofallthefunctions(i.e.,linearityornot,convexity,anddifferentiability)(see Figure1.1).
1.4.2ClassificationofSingle-ObjectiveOptimizationMethods
Twoclassificationsareproposedby Rao(2009).Oneoftheclassificationsisdevotedtomethodsforunconstrained minimizationproblems.Theotherisconstrainedminimiz ationtechniques.Inbothcases,adistinctionismade betweendirectresearchmethods(i.e.,withoutrequiringthepartialderivatives),andtechniquesofdescentwith derivatives.
The directedsearchmethods forunconstrainedoptimizationproblems(see Rao,2009,pp.309–334)includetherandom searchmethod,thegridsearchmethod,univariatemethod,patternsearchmethod(e.g.,Powell’smethod).The methodsof descent forthesametypeofoptimizationproblems(see Rao,2009,pp.335–368)consistofthesteepestdescent(orCauchy) method,theFletcher-Reevesmethod,Newton’smethod,Marquardtmethod,andthequasi-Newtonmethods(i.e.,DavidFletcher-Powellmethod,andBroyden-Fletcher-Goldfarb-Shannomethod).
Theoptimizationproblemsunderconstraintsprocessingtechniquesalsoinclude directapproaches bywhichthe constraintsarehandledexplicitly,aswellas indirectmethods,usingasequenceofunconstrainedoptimizationproblems. The directedsearchmethods forconstrainedoptimizationproblems(see Rao,2009,pp.383–428)includetherandom searchmethod,heuristicsearchmethodssuchascomplicatedmethod,objective,andconstraintapproximationmethods (i.e.,sequentiallinearorquadraticoptimizationmethods),methodsoffeasibledirection(i.e.,Zoutendijk’smethod, Rosen’sgradientprojectionalgorithm),andthegeneralizedreducedgradientalgorithm.The indirectmethods for thesametypeofoptimizationproblems(see Rao,2009,pp.428–491)oftenconsistofunconstrainedsequentialtechniques,suchastheinteriorpenaltyfunctionmethod,theexteriorpenaltyfunctionmethod,andtheaugmentedLagrange multipliermethod.
1.4.3ClassificationofMOOMethods
MOOtechniquescanbeclassifiedindifferentways.Thetypologyretainedby Miettinen(1999) and Diwekar(2008, pp.186–199) isbasedontwocriteria,thenumberofgeneratedParetosolutionsandthedecision-maker’s(DM)preferences.
FIGURE1.1 Classificationofoptimization problems. (InspiredfromSarker,R.A.,& Newton,C.S.(2008).Optimizationmodelling: Apracticalapproach.BocaRaton,FL/London, UK:CRCPress,p.12,Figure1.3.)
INumberofobjectives
IINumberofconstraints
IIITypeofdesignvariables
IVMathematicalproperties
Therearetwogroupsofmethods.Afirstgroupincludesthe generatingmethods andotherthe preference-based methods.Inthegeneratingmethods,thePareto-optimalsolutionsareobtainedwithoutanyactionoftheDMduringthe determinationprocess.TheroleoftheDMistheselectionofoptimalsolutions.Onthecontrary,preference-basedmethods integrateDM’spreferencesatsomestageoftheresolutionprocess.
Generatingmethods includethreefollowingsubgroupssuchasno-preferencemethods,aposteriorimethodsusinga scalarizationtransform,andaposteriorimethodsusingamultiobjectiveapproach.The no-preferencemethods include inparticularthemethodofglobalcriterion,12 themultiobjectiveproximalbundlemethod.13 Posteriorimethodsusinga scalarizationtransformationinclude e-constraintmethod14 andweightingmethods.15 Posteriorimethods usingamultiobjectiveapproachrefertopopulation-basedprobabilisticmethodsinspiredbynature.Thesemethodsincludethemultiobjectivesimulatedannealing16 (MOSA),thenondominatedsortingGA,17 andDE18 strategy.Thealgorithmcanfindmany Paretosolutions,andDMselectsoneofthem.19
Preference-basedmethods useDM’spreferences.Thesepreferencesarerequiredbeforetheresolutionprocessbegins. Twosubgroupsincludeapriorimethodsandinteractivemethods. Apriorimethods includethevaluefunctionmethod,20 the lexicographicorderingmethod,21 andgoalprogramming.22 InteractivemethodsformalizetheDM’spreferencesduringthe solutionprocess.
1.5DESIGNANDCHOICEOFANALGORITHM
Yang(2014,pp.23–44) developedananalysisofoptimizationalgorithm.Analgorithmisviewedasaniterativeprocess. Manyoptimizationalgorithmsareintheliterature.Foragivenapplication,thequestionarisesastoselectthebestalgorithm providingaccurateresultsusingonlyareducedcostofcomputing.Thissectioncontainssomeelementsofthispresentation.
1.5.1DesignofanAlgorithm
Newtonmethodfornonlinearprogrammingproblemsseekstoattainanoptimumfromastartingpoint x0 intheunivariate caseforwhich f(x)isoptimized.Theobjectiveistoconvergetoastationarypointwherethederivativeiszero.Intheunivariatecase,thisiterativemethodgeneratesasequenceofiterationsoftheform
Inthemultivariatecasewhere f x ðÞ, x 2 ℝn wehave x k +1 ðÞ ¼ x k ðÞ H 1 x k ðÞ rf x k ðÞ
ThealgorithmoftheNewtonmethodisillustratedbyapseudo-codein Table1.1
SolvingthenonlinearsystemofKKT(Karush-Kuhn-Tucker)necessaryconditionsforanoptimizationproblemwith n designvariablesand m constraints,wehave FxðÞ¼ 0,where F : ℝn 7!ℝn + m and x 2 ℝn .Forsolvingthisnonlinearsystem, theNewton-Raphsonmethodassumesthat x(k) whereiteration k isknownandachangelike Dx(k) iscalculated.Linearizing byusingtheTaylorexpansion,wehavetosolve rFx k ðÞ T Dx k ðÞ ¼ Fx k ðÞ
TheNewton-Raphsoniteratedprocedureiscontinueduntilastoppingcriterionissatisfied(see Arora,2012, pp.554–557).
TABLE1.1 Newtonmethod Algorithm1
1 Set k ¼ 0\*initialstep*\ [l, u]\*initialinterval*\ 2while f
6
1.5.2ChoiceofanAlgorithm
In1997,thestudyonNoFreeLunch(NFL)theoremsbyWolpertandMacready(1997)wasasignificantstepinthe developmentofbetteralgorithms.Indeed,theoremsprovedthatthereexistsnobetteruniversalalgorithmforall applications.Thus,themostefficientalgorithmshouldbefoundforagivenclassofproblems.
1.5.3BasicCycleofanEvolutionaryAlgorithm
Thebasiccycleisshownin Figure1.2.Theinitialstepconsistsofapopulationinwhichindividualsarecreatedatrandom. Intheevaluationphaseofthebasiccycle,weevaluatealltheindividualsbyusingtheobjectivefunctionsofthe
FIGURE1.2 Basiccycleofanevolutionaryalgorithm. (ReprintofFigure1.1fromKeller,A.A.(2017).Multi-objectiveoptimizationintheoryand practice.II.Evolutionaryalgorithms.BenthameBooks.)
programmingproblem.Next,fitnessvaluescanbeassignedtoindividualsonthisbasis.Then,thefittestindividualscanbe selectedforreproduction.Thereafter,newindividualsarecreatedbyusinggeneticoperators,suchaswithcrossoverand mutation.Closingthebasiccycle,thenewpopulationincludingtheselectedindividualsandoffspringistransferredtothe firststepforevaluation,andanewcyclegoeson.
ENDNOTES
1.AsupplementtotheHandbookoftestproblemsisavailableat http://titan.princeton.edu/TestProblems.Executableversionsofallthetestproblemsare downloadable.
2.Thispresentationcoversthemainhistoricalaspectsproposedbytheauthor(Keller,2017a,2017b)inhisbook“Multi-ObjectiveOptimizationin TheoryandPractice.”TheseelementsarecontainedinVolumeIIonevolutionaryalgorithms.
3.See http://www.alanturing.net/turing_archive/archive/1/132-001.html
4.Available: http://proceedings.asmedigitalcollection.asme.org/proceeding.aspx?articleid¼2090514
5.NP-hardnessofanalgorithmincludesnotablyP,NP,NP-complete,andNP-hardcategories.NPstandsfor“nondeterministicpolynomialtime.”Complexityclass“P”representsthesetofdecisionproblemsthatcanbesolvedinpolynomialtime.Complexityclass“NP”representsthesetofdecision problemsforwhichproofscanbeverifiedinpolynomialtime.Complexityclass“NP-Complete”representsallNPproblemsthatcanbereducedto polynomialtimeproblems.Complexityclass“NP-hard”aretheproblemsthatareatleastashardasNP-Completeproblems.Timecomplexityis obtainedbycountingsimpleoperationsperformedbyanalgorithm.Ifthetimerequiredonallinputsofsize n is5n3 þ 3n,theasymptotictimecomplexityisexpressedbyusingthebig O notation O(n3)(see https://en.wikipedia.org/w/index.php?title¼Time_complexity).
6.Thehistoryandbasicprinciplesofevolutionarycomputationwerespecifiedinthenewhandbookoncomputationalintelligenceeditedby Angelov (2016,pp.509–545).Ahistoricalreviewby Coello,Lamont,andVanVeldhuizen(2007) wasdevotedtothemainparadigmsofevolutionarycomputation:evolutionstrategies,evolutionaryprogramming,andgeneticalgorithm.
7.Astudyby Keller(2014) showedtheuseofmultiobjectiveheuristicoptimizationtechniquesinwaterresourcesmanagement.Inotherstudies, Keller (2009) introducedthefuzzymulti-objectivemodelingandthecomputationaltechniquestofuzzybimatrixgames.
8.Besidesmetaheuristics,otherheuristicoptimizationtechniquesweredevelopedinthelineoftheTuring’scontributions.Thesemethodsinclude notablyartificialneuralnetworks(see Yang,2014 forfurtherspecifications).
9.Seealso Intriligator(1981,pp.53–91)withadifferentconventionfromthisstudy.Intriligatorrepresentedcoefficients c byarowvector.
10.DecisionvariablesinMOOproblemscanbereal,integeroramixtureofcontinuous,binaryandintegervariables.
11.Equalityconstraintsarisefrommass-energyandmomentumbalances.Theyshouldbealgebraicanddifferentialequations.Inequalityconstraints denoteforexampletherequirementthatthetemperatureshouldstaybelowaspecifiedvalue.Theycanrepresentfailuresofthematerialusedfor equipmentfabrication,andsoon(see Miettinen,1999;Rangaiah,2009).
12.Themethodofglobalcriterionorcompromiseprogrammingminimizesthedistancebetweensomereferencepointandthefeasibleobjectiveregion (Miettinen,1999,pp.67–71).
13.Themultiobjectiveproximalbundlemethod(MPB)extendsthecorrespondingmethodforsingle-objectiveoptimization(SOO)problems.Itrelieson movinginadirectionwhereallobjectivesimprovesimultaneously(Miettinen,1999,pp.71–77).
14.Inthe e-constraintmethod,theMOOproblemisreformulatedintoaSOOproblemwithmoreconstraints.Oneobjectivefunctionisselectedtobe optimized,andallotherareconvertedintoconstraintsforwhichupperboundsareintroduced(Miettinen,1999,pp.85–94).
15.WeightingmethodsalsoconsistintransformingaMOOproblemintoaSOOproblem.Indeed,themethodminimizesaweightedsumoftheobjective functionsbyusinguser-definedweightingcoefficients(Miettinen,1999,pp.78–85).
16.Simulatedannealingisaheuristicsearchalgorithmbasedonananalogyinstatisticalmechanics.Inphysicalannealing,allatomicparticlesarrange themselvesinalatticerepresentationthatminimizesthetotalamountofenergy.Twoconditionsmustbemet:theinitialtemperatureishigh,andthe coolingisgoingslowly(Diwekar,2008,pp.100–103).
17.Geneticalgorithmsaresearchmethodsbasedonthemechanismofnaturalselectionandbiologicalevolution.Aninitialpopulationisgeneratedrandomly.Developmentisduetocrossover,mutation,andselectionoperators.Theobjectiveistomaximizethefitnessofageneration(Diwekar,2008, pp.103–107).
18.Differentialevolution(DE)algorithmappearedin1995.DEisaperformantpopulation-basedoptimizer.DEgeneratesnewpointsthatareperturbationsofexistingpoints.DEperturbsvectorswiththescaleddifferenceoftworandomlyselectedpopulationvectors.DEusesthecrossover,mutation andselectionoperators(Price,Storn,&Lampinen,2005).
19.Forreviewsonevolutionaryalgorithms,onecanreferto Coelloetal.(2007), Deb(2001),and Yang(2014)
20.Inthevaluefunctionmethod,DMproposesanexplicitmathematicalfunctionoftheobjectives(i.e.,U : ℝ r 7!ℝ for r objectivefunctions.Thismapping reflectsDM’spreferencesglobally.Thisfunctionprovidesacompleteorderingintheobjectivespace.TheoriginalMOOproblemistransformedintoa SOOproblem(Miettinen,1999,pp.115–118).
21.Inthelexicographicorderingmethod,DMarrangestheobjectivesaccordingtotheirrelativeimportance.Afterorderingtheobjectivefunctions,the primaryobjectivefunctionisminimizedsubjecttotheoriginalconstraints(Miettinen,1999,pp.118–121).
22.Ingoalprogramming,thedecisionmakerspecifiesaspirationlevelsfortheobjectivefunctionssuchthat f k ðxÞ zk ; k ¼ 1; ; r ,where x 2 ℝn.The problemistominimizethedeviationalvariables dk ¼ z f ðxÞ (Miettinen,1999,pp.121–129).
REFERENCES
Abido,M.A.(2010).Multiobjectiveparticleswarmoptimizationwithnondominatedlocalandglobalsets. NaturalComputing, 9,747–766. Angelov,P.P.(Ed.),(2016). Handbookoncomputionalintelligence.Hackensack,NJ,USA:WorldScientific. Arora,J.S.(2012). Introductiontooptimumdesign. NewYork,USA:Elsevier. Arrow,K.J.,&Intriligator,M.D.(Eds.),(1981). Handbookofmathematicaleconomics(Vol.I).Amsterdam,NL/NewYork,USA:North-Holland Publishing.
Bazaraa,M.S.,Sherali,H.D.,&Shetty,C.M.(2006). Nonlinearprogramming:Theoryandalgorithms. Hoboken,NJ,USA:JohnWiley&Sons. Bertsekas,D.P.(2009). Convexoptimizationtheory. Belmont,MA,USA:AthenaScientific. Boyd,S.,&Vandenberghe,L.(2004). Convexoptimization.Cambridge,UK-NewYork.USA:CambridgeUniversityPress. Changkong,V.,&Haimes,Y.Y.(1983). Multiobjectivedecisionmaking:Theoryandmethodology. NewYork,USA:NorthHolland. Chiang,M.(2009).Nonconvexoptimizationforcommunicationsystems.InD.Y.Gao&H.D.Sherali(Eds.), Advancesinappliedmathematicsand globaloptimization:Vol.17 (pp.137–196).NewYork,USA:SpringerScience+BusinessMedia. Coello,C.A.(1999).Acomprehensivesurveyofevolutionary-basedmultiobjectiveoptimizationtechniques. KnowledgeandInformationSystems, 1(3), 269–308.
Coello,C.A.,Lamont,G.B.,&VanVeldhuizen,D.V.(2007). Evolutionaryalgorithmsforsolvingmulti-objectiveproblems (2nded.).NewYork,USA: SpringerScience+BusinessMedia.
Dantzig,G.B.,&Wolfe,P.(1960).Decompositionprincipleoflinearprograms. OperationsResearch, 8(1),101–111. Deb,K.(2001). Multi-objectiveoptimizationusingevolutionaryalgorithms. Chichester,UK/NewYork,USA:JohnWiley&Sons. Diwekar,U.(2008). Introductiontoappliedoptimization (2nded.).NewYork,USA:SpringerScience+BusinessMedia. Dorigo,M.(1992).Optimization,learningandnaturalalgorithms.Ph.D.Thesis,PolitechnicodiMilano,Miilano,IT. Edgeworth,F.Y.(1881). Mathematicalpsychics:Anessayontheapplicationofmathematicstothemoralsciences. London,UK:PaulKegan. Floudas,C.A.,&Pardalos,P.M.(Eds.),(2001). Encyclopediaofoptimization.Dordrecht,NL:KluwerAcademicPublishers. Floudas,C.A.,Pardalos,P.M.,Adjiman,C.S.,Esposito,W.R.,Gumus,Z.H.,Harding,S.T.,etal.(2010). Handbookoftestproblemsinlocalandglobal optimization. Dordrecht,NL/Boston,MA,USA:KluwerAcademicPublishing. Floudas,C.A.,Pardalos,P.M.,Adjiman,C.S.,Esposito,W.R.,Gumus,Z.H.,Harding,S.T.,etal.(1999). Handbookoftestproblemsinlocalandglobal optimization. Dordrecht,NL/Boston,MA,USA:KluwerAcademicPublishers.
Fogel,D.B.(1994).Anintroductiontosimulatedevolutionaryoptimization. IEEETransactionsonNeuralNetworks, 5(1),3–14. Fonseca,C.M.,&Fleming,P.J.(1993).Geneticalgorithmsformultiobjectiveoptimization:Formulation,discussion,andgeneralization.InS.Forrest (Ed.), Proc.fifthint.conf.ongeneticalgorithm (pp.416–423).SanMateo,CA:MorganKauffmannPublishers. Fonseca,C.M.,&Fleming,P.J.(1995).Anoverviewofevolutionaryalgorithmsinmultiobjectiveoptimization. EvolutionaryComputation, 3(1),1–16. Garrett, D., & Dasgupta,D.(2006).Analyzingtheperformanceofhybridevolutionaryalgorithmsforthemultiobjectivequadraticassignmentproblem. RetrievedfromIEEECongressonEvolutionaryComputation: http://ais.cs.memphis.edu/files/papers/PerformanceHEA-MQAP.pdf Geiger,C.,&Kanzow,C.(2000). TheorieundNumerikRestringierterOptimierungsaufgaben. Berlin-Heidelberg,DE/NewYork,USA:SpringerVerlag. Gill,P.E.,Murray,W.,&Wright,M.H.(1981). Practicaloptimization. London,UK/NewYork,USA:AcademicPress. Glover,F.(1986).Futurepathsforintegerprogrammingandlinkstoartificialintelligence. Computers&OperationsResearch, 13(5),533–549. Goldberg,D.E.(1989). Geneticalgorithmsinsearch,optimizationandmachinelearning. Reading,MA,USA:Addison-WesleyPublishing. Grosan,C.,&Abraham,A.(2007).Hybridevolutionaryalgorithms:Methodologies,architecturesandreviews. StudiesinComputationalIntelligence, 75, 1–17.
Hajela,P.,&Lee,L.(1996).Constrainedgeneticsearchviasearchadaptation.Animmunenetworksolution. StructuralOptimization, 13(1),11–15. Hastings,K.J.(2006). Introductiontothemathematicsofoperationsresearchwithmathematica. BocaRaton,FL,USA:Chapman&Hall/CRC. Hendrix,E.M.,&Toth,B.G.(2010). Introductiontononlinearandglobaloptimization,Vol.37. NewYork,USA/Dordrecht,NL/Heidelberg,DE: Springer.
Hiriart-Urruty,J.-B.,&Lemarechal,C.(2000). Fundamentalsofconvexanalysis. Berlin-Heidelberg,DE/NewYork,USA:Springer-Verlag. Holland,J.H.(1975). Adaptationinnaturalandartificialsystems. Cambridge,MA,USA:TheMITPress. Horst,R.,&Pardalos,P.M.(Eds.),(1995). Handbookofglobaloptimization.Dordrecht,NL:KluwerAcademicPublishers. Horst,R.,&Tuy,H.(1996). Globaloptimization:Deteministicapproaches (3rded.).Berlin-Heidelberg,DE:Springer-Verlag. Intriligator,M.D.(1971). Mathematicaloptimizationandeconomictheory. EnglewoodCliffs,NJ,USA:Prentice-Hall. Intriligator,M.D.(1981).Mathematicalprogrammingwithapplicationtoeconomics.InK.J.Arrow&M.D.Intriligator(Eds.), Vol.I.Handbookof mathematicaleconomics (pp.53–91).Amsterdam,NL/NewYork,USA:North-HollandPublishing. Ishibuchi,H.,&Yoshida,T.(2002). Hybridevolutionarymultiobjectiveoptimizationalgorithm. Retrievedfrom http://citeseerx.ist.psu.edu/viewdoc/ download?doi¼10.1.1.8.5710&rep1&type¼pdf Ito,K.,Akagi,S.,&Nishikawa,M.(1983).Amultiobjectiveoptimizationapproachtoadesignproblemofheatinsulationforthermaldistributionpiping andautomation. JournalofMechanisms,TransmissionsandAutomationinDesign, 105,105–206. Jacob,C.(2001). Illustratingevolutionarycomputationwithmathematica. SanDiego,CA,USA:AcademicPress. Keller,A.A.(2009).Fuzzymultiobjectivebimatrixgame:Introductiontothecomputationaltechniques.InN.E.Mastorakis,M.Demiralp,V.Mladenov, &Boikovic(Eds.), Recentadvancesinsystemtheory&scientificcomputation (pp.148–156).Moscow,Russia:WSEASPress. Keller,A.A.(2014).Multiple-usewaterresourcesmanagementbyusingfuzzymulti-objectiveheuristicoptimizationmethods:Anoverview. InternationalJournalofFuzzySystemsandAdvancedApplications, 1(4), 36–54.
Keller,A.A.(2017a). Multi-objectiveoptimizationintheoryandpractice:I.Classicalmethods.Sharjah,UAE:BenthameBooks. Keller,A.A.(2017b). Multi-objectiveoptimizationintheoryandpractice.II.Evolutionaryalgorithms. Sharjah,UAE:BenthameBooks. Kennedy,J.,&Eberhart,R.C.(1995).Particleswarmoptimization.In IEEEinternationalconferenceonneuralnetworks,Piscataway,NJ,USA (pp.1942–1948).
Kirkpatrick,S.,Gelatt,C.D.,Jr.,&Vecchi,M.P.(1983).Optimizationbysimulatedannealing. Science, 220(4598),671–680. Knowles,J.D.,&Corne,D.W.(2005).Memeticalgorithmsformultiobjectiveoptimization:Issues,methodsandprospects.InW.E.Hart,J.E.Smith,& N.Krasnogor(Eds.), Recentadvancesinmemeticalgorithms (pp.313–352).Berlin,DE:Springer. Koza,J.R.(1992). Geneticprogramming:Ontheprogrammingofcomputersbymeansofnaturalselection. Cambridge,MA,USA/London,UK:TheMIT Press.
Lai,Y.-J.,&Hwang,C.-L.(1994). Fuzzymultipleobjectivedecisionmaking:Methodsandapplications. Berlin-Heidelberg,DE:Springer-Verlag. Luenberger,D.G.,&Ye,Y.(2008). Linearandnonlinearprogramming. NewYork,USA:SpringerScience+BusinessMedia. Mashwani,W.K.(2011).Hybridmultiobjectiveevolutionaryalgorithm:Asurveyofthestate-of-the-art. InternationalJournalofComputerScience Issues, 8(6),374–392. Michalewicz,Z.(1999). Geneticalgorithms+datastructures ¼ evolutionprograms. Berlin-Heidelberg,DE/NewYork,USA:Springer. Miettinen,K.M.(1999). Nonlinearmultiobjectiveoptimization. Boston,MA,USA/London,UK/Dordrecht,NL:KluwerAcademicPublishers. Minoux,M.(1986). Mathematicalprogramming:Theoryandalgorithms. Chichester,UK/NewYork,USA:JohnWiley&Sons. Papadimitriou,C.H.(1995). Computationalcomplexity. Reading,MA/MenloPark,CA,USA:AddisonWesleyLongman. Papadimitriou,C.H.,&Steiglitz,K.(1982). Combinatorialoptimization. EnglewoodCliffs,NJ,USA:Prentice-Hall.
Pardalos,P.M.,&Resende,M.G.(Eds.),(2002). Handbookofappliedoptimization.Oxford,UK/NewYork,USA:OxfordUniversityPress. Pardalos,P.M.,&Romeijn,H.D.(Eds.),(2002). Handbookofglobaloptimization,Vol.2.Boston,MA,USA/Dordrecht,NL:KluwerAcademic Publishers.
Pareto,V.(1896). Coursd’economiepolitique(inFrench).Lausanne,CH(EnglishtranslationbySchwierA.S.,Manualofpoliticaleconomy. AugustinM. KelleyPublishers,NewYork,NY1971. Price,K.,Storn,R.,&Lampinen,J.(2005). Differentialevolution:Apracticalapproachtoglobaloptimization. Berlin-Heidelberg,DE:Springer-Verlag. Rangaiah,G.P.(2009). Multi-objectiveoptimization:Techniquesandapplicationsinchemicalengineering. NewJersey,USA/London,UK:World Scientific. Rao,S.S.(2009). Engineeringoptimization:Theoryandpractice (4thed.).Hoboken,NJ,USA:JohnWiley&Sons. Rechenberg,I.(1973). Evolutionsstrategie:OptimierungtechnischerSystemenachPrinzipienderbiologischenevolution(inGerman). Stuttgart,DE: Fromman-HolzboogVerlag. Reeves,C.(1995). Modernheuristictechniquesforcombinatorialproblems. London,UK:McGrawHill. Rockafellar,R.T.(1970). Convexanalysis. Princeton,NJ,USA:PrincetonUniversityPress. Rosenberg,R.S.(1967). Simulationofgeneticpopulationswithbiochemicalproperties. AnnArbor,MI,USA:UniversityofMichigan. Sarker,R.A.,&Newton,C.S.(2008). Optimizationmodelling:Apracticalapproach. BocaRaton,FL/London,UK:CRCPress. Sawaragi,Y.,Nakayama,H.,&Tanino,T.(1985). Theoryofmultiobjectiveoptimization. NewYork,USA/London,UK:AcademicPress. Schaffer,J.D.(1984). Multiple objective optimizationwithvectorevaluatedalgorithms. Nashville,TH,USA:VanderbiltUniversity. Schwefel,H.-P.(1977). NumerischeOptimierungvonComputer-ModellenMittelsderEvolutionsstrategie. Basel:Birkhauser. Srinivas,N.,&Deb,K.(1994).Multi-objectivefunctionoptimizationusingnon-dominatedsortinggeneticalgorithms. EvolutionaryComputation, 2(3), 221–248.
Storn,R.,&Price,K.(1997).Differentialevolution—Asimpleandefficientheuristicforglobaloptimizationovercontinuousspaces. JournalofGlobal Optimization, 11,341–359.
Tang,L.,&Wang,X.K.(2013).Ahybridmultiobjectiveevolutionaryalgorithmformultiobjectiveoptimizationproblems. IEEETransactionsonEvolutionaryComputation, 17(1),20–46. Tawarmalani,M.,&Sahinidis,N.V.(2002). Convexificationandglobaloptimizationincontinuousandmixed-integernonlinearprogramming:Theory, algorithms,software,andapplications. Dordrecht,NL:KluwerAcademicPublishers.
Thangaraj,R.,Pant,M.,Abraham,A.,&Bouvry,P.(2011).Particleswarmoptimization:Hybridizationperspectivesandexperimentalillustrationspectives. AppliedMathematicsandComputation, 217,5208–5226.
Tong,W.,Chowdhury,S.,&Messac,A.(2014).Anewmulti-objectivemixed-discreteparticleswarmoptimizationalgorithm.In ASME2014internationaldesignengineeringtechnicalconferencesandcomputersandinformationinengineering.vol.2A:40thdesignautomationconference.Buffalo, NY,USA:ASME.
Turing,A.M.(1948). Intelligentmachinery. London,UK:Report,NationalPhysicalLaboratory. Varian,H.R.(1992). Microeconomicanalysis (3rded.).NewYork,USA/London,UK:W.W.Norton. vonNeumann,J.,&Morgenstern,O.(1953). Theoryofgamesandeconomicbehavior (3rded.).NewYork,USA/London,UK:JohnWiley&Sons. Wang,Y.,Cai,Z.,Guo,G.,&Zhou,Y.(2007).Multiobjectiveoptimizationandhybridevolutionaryalgorithmtosolveconstrainedoptimization problems. IEEETransactionsonSystems,ManandCybernetics—PartBCybernetics, 37(3),560–575. Weise,T.(2009). Globaloptimizationalgorithms—Theoryandapplications. Retrievedfrom http://www.it-weise.de/projects/ Whitley,D.,Gordon,V.S.,&Mathias,K.(1994).Lamarkianevolution,theBaldwineffectandfunctionoptimization.In Parallelproblemsolvingfrom naturePPSNIII (pp.5–15).Berlin,DE:Springer. Wolpert,D.H.,&Macready,W.G.(1997).Nofreelunchtheoremsforoptimization. IEEETransactionsonEvolutionaryComputation, 1(1),67–82.
Yang,X.S.(2014). Nature-inspiredoptimizationalgorithms. Waltham,MA,USA:Elsevier.
Zamuda,A.,Brest,J.,Boskovic,B.,&Zumer,V.(2009).Differentialevolutionwithself-adaptationandlocalsearchforconstrainedmultiobjective optimization.In IEEEcongressonevolutionarycomputation (pp.195–202).
Zitzler,E.(1999). Evolutionaryalgorithmsformultiobjectiveoptimization:Methodsandapplication [Doctordegreedissertation].Zurich,CH:ETHEidgenossischeTechnischeHochschule,InstitutfurTechnischeInformatikundKommunikationsnetze(TIK).
GlossaryofMathematicalOptimization
Terminology
2.1INTRODUCTION
Informationonthemathematicaloptimizationavailabletousersfortheirapplicationsisparticularlyrichandabundant. Thisliteratureisdividedintospecificworksandnumerousarticlesofqualitygivinganup-to-datereview(surveys)in adomainoronaparticulartechnique.Theneedforuserstohavemoreglobalandcomprehensiveinformationcandirect themtoothertypesofcontributions,suchasthehandbooks,encyclopediaonoptimizationandoperationalResearch,online glossaries,andsitesofferingtoolsdirectlyoperativesuchasalgorithms,pseudocodes,codesinC,andtestproblems.Inthe remainderofthisoverview,weprovidesomeclarificationontheseavailabledocumentaryaspectsusefultotheuser.Subsequently,weproposeourownprojectdefinedasapracticaluser-orientedguidetowardsreal-worldapplications.Inthe following,wespecifythefollowingfeaturesofdocumentationonoptimizationandoperationsresearch(OR).Thefirst aspectisontheessentialhandbooksthatcouldcomposeapersonallibraryonoptimizationandOR.Thesecondpoint isamorepracticalinstrumentforusers,suchaswithencyclopedia,dictionary,andglossariesonoptimizationandOR. Thethirdpointdescribestheprojectofthisbooktobeapracticaluser’sguideinoptimizationandOR.
2.1.1PrimaryHandbooks
Letussupposethatwewanttobuildathree-shelvepersonallibraryon classicaloptimization, evolutionaryoptimization, andapplicationto networksandgraphs
Thefirstshelfreservedfor traditionalmethodsofglobaloptimization shouldincludethefollowingbooks,namelythe Handbookof globaloptimization by ReinerandPardalos(1995),theHandbookofthetheoryandalgorithmsof combinatorialoptimization by KorteandVygen(2012),theHandbookofthetheory,algorithms,andapplicationsin semidefinite programming by Wolkowicz,Saigal,andVandenberghe(2000),thebookwith algorithmsinC by Sedgewick(2002), anotherbookconsistingof numericalrecipesinC by Press,Teulkolsky,Vetterling,andFlannery(1992),thebookoncomputationalcomplexityby Papadimitriou(1995),thecollectionof testproblems forconstrainedglobaloptimizationalgorithmsby FloudasandPardalos(1990),andthelastHandbookoftheshelfwouldalsobeon testproblemsinlocaland globaloptimization by Floudasetal.(2010)
Thesecondshelfreservedfor evolutionarytechniquesforoptimization shouldincludethefollowingbooks,namelythe Handbookofgeneticalgorithmsby Davis(1991),Handbookoncomputationalintelligence(withvolume1 infuzzylogic, systems,artificialneuralnetworks,andlearning;andvolume2 inevolutionarycomputation,hybridsystems,andapplications)by Angelov(2016a,2016b),andthelastHandbookoftheshelfwouldbethetwoHandbooksofmetaheuristics by GloverandKochenberger(2003) andby GendreauandPotvin(2010)
Thethirdshelfreservedfor networkandgraphapplications shouldincludecontainthefollowingbooks,namelythe bookon ModelsandMethodsofOperationsResearch by JensenandBard(2003),thebookon ApplicationsandAlgorithms ofOperationsResearch by Winston(2014),the HandbookofOperationsResearchinNaturalResources by Weintraub, Romero,Bjornd,andEpstein(2007),thebookon NeuralNetworksforOptimizationandSignalProcessing by Cichocki andUnbehauen(1993),the HandbookofGraphTheory by GrossandYellen(2004),andthelastbookoftheshelfwouldbe abookon Graphs,Algorithms,andOptimization by KocayandKreher(2005).
2.1.2EncyclopediasandGlossaries
Theliteratureonoptimizationandoperationsresearch(OR)alsocomprisesencyclopediasandglossariesthatofferauseful overviewofthedomainandtendtobeexhaustive.Encyclopediasincludesurveyarticlesonavarietyofsubjectsrelatedto optimization.Glossariescontaingeneraldefinitionsoftermsforwhichsomeformulascanbeproposed.
Encyclopediascanbefoundonthefollowingdomains:optimization,optimizationandOR,combinatorialoptimization, andgametheory.The“HandbooksinOperationsResearchandManagement”in1989to2007includevolumesinthe methodsandapplicationsofoptimization.Optimizationmethodsare“Optimization”(Volume1)1 (see Nemhauser, Rinnooy,&Todd,1989),“StochasticProgramming”(Volume10),“DiscreteOptimization”(Volume12).Applicationsof optimizationare“LogisticsofProductionandInventory”(Volume4),“Marketing”(Volume5),“Finance”(Volume9), “SupplyChainManagement;Design,CoordinationandOperation”(Volume11),“Transportation”(Volume14),and “FinancialEngineering”(Volume15).2 Thesecondeditionof“EncyclopediaofOptimization”by FloudasandPardalos (2009) consistsofacollectionofadaptedsurveyarticles.Thereare570contributorsworldwide,mostlyfromtheUnited States.Thedescriptionsareunifiedandincludethefollowingelements: ●atitle, ●AMS200classificationcodes, ●anarticle outline, ●keywords, ●synonyms, ●textofthearticlewhereimportantconceptsandtermsareinitalics, ●crossreferences, and ●bibliographicreferences.Entriesarearrangedinalphabeticalorderandmayincludefiguresandtables.Thiscollection ofreviewarticlesconsistingofsevenvolumes3 contains4626pages.AnEncyclopediaby Schrijver(2003) withthreevolumes AtoC(83Chapterswith1881pages)iscenteredon“CombinatorialOptimization”whereVolumeA ison“Paths,Flows, Matchings,”VolumeB on“Matroids,Trees,StableSets,”andVolumeC on“DisjointPaths,Hypergraphs.”Another Encyclopediaby Derigs(2009) includesinfourvolumes(with1536pages)on“OptimizationandOperationsResearch”. Volume1 focusesonthefoundationsofmathematicalprogramming(i.e.,linearandnonlinearprogramming,dynamic programming,anddiscreteoptimization).Volume2 discussestheproblemsandtechniquesofacontemporaryoptimization (e.g.,dualitytheory,combinatorialoptimization,scheduling,androutingproblems)dealingwithpracticaldifficulties(e.g., large-scaleoptimization,nonsmoothoptimization,globaloptimizationandheuristics,andapproximationalgorithms). Volume3 focusesonsystemdynamicsandcontrol(e.g.,calculusofvariations,maximumprincipleofPontryagin,dynamic programming,andBellman’sprinciple),andintroducesgametheory.Finally,Volume4 focusesondecisionprocessesmainly inanuncertainenvironment(e.g.,Markovmodelsandprocesses,queuingsystems,expectedutilitytheory,andstochastic games).WemustalsopointouttheEncyclopediainfourvolumesI—IVby Varoufakis(2001) on“GameTheory:Critical ConceptsintheSocialScience.”Thevolumesbringtogetherthefoundingarticlesofthecontemporarytheoryofgames. VolumeI ison“Foundations,”VolumeII on“Refinements,”VolumeIII on“EconomicApplications GameTheoryand SocialSciences,”andVolumeIV on“Discontents.”
Themostcomprehensiveglossaryonoptimizationistheonedevelopedintheperiod1999–2006,andactuallydistributedbyINFORMSComputingSocietyat http://glossary.computing.society.informs/2006-14.Thetermsalsorelate tothestrictdomainofoptimizationintheirtechnical(mostlymathematical)context.4 Figure2.1 describesforagiven conceptthedifferentlevelsofdefinitionsthatcanbereached(underlinedtermsorbookmarks).Theinterestofthispractice fortheuseristoallowthecreationofdocumentconsistingofthemaindefinedconceptaccompaniedwiththechoiceofthe userofadditionalrelateddefinitions.SeeMathematicalProgrammingGlossary(Holder,2014).
Othertools online giveaccesstoready-to-usealgorithms.Thus,adescriptionofDijkstra’salgorithm5 forfindingall shortestpathsfromonenodetoallothernodesofaweightedgraphisavailableat http://www.programming-algorithms.net/ article/45514/Dijkstra’sl-algorithm
2.1.3PracticalUser’sGuide
Thisglossaryincludesthetermsandexpressionsin“optimization”(or“programming”)withfamiliaracronyms.Forentry, thedefinition,formulation,illustrativeexamples,andapplicationareasarespecified.Referencestosurveys,books,and crossreferencesarementioned.Thislistofselectedtermsdoesnotaimtobeexhaustiveatthepublicationinstant.
Thisuserguidepresentstheinterestofseparatingspecificoptimizationterms(Chapter2)fromothertechnicalterms relatingtothecontextinwhichoptimizationcanbeapplied.ThetechnicaltermsdefinedinChapter3mayrefertothe conceptsandnotionsofmathematics,statisticsandprobabilities,andoperationsresearch.Forexample,mathematicalprogrammingmethodscomeunderthischapter,whilemethodsforsolvingsystemsofequations(oftheNewtonmethodtype) arepresentedinChapter3.Theentriesofthischaptercoverthemajorfeaturesofoptimizationtoday.Thetermsandexpressionsconcernvariouscategoriesofmodels,suchas ●discrete,continuous,andmixedinteger-continuousprogramming,
Residuals Step sizeTolerance Diophantine equations
FIGURE2.1 Firstthreelevelsofthetreeof termsdefiningtheconceptof“ABSalgorithm.”
DichotomousFibonacciGolden sectionLattice
●deterministicandstochasticprogramming, ●classicalandevolutionaryalgorithms, ●singleandmultipleobjectiveoptimizationproblems, ●hierarchicalprograms, ●combinatorymodelsusinggraphsandnetworks,and ●modelsofthegame theory.Thisglossaryissufficientlycompleteandupdatedtoprovideausefulandpreciseguideforreadersworkingwith optimization.Thisguideintroducestothevarietyofaspectsandmethodsinoptimizationpractice.Itpresentsanoverview thatmayguidetheapplications.
Matrixnotationsarepreferredtopresentmathematicalspecifications.Keyresultsareboxedandhighlightedintext. A blockofinformation foreachentryconsistsof ●referencesin chronologicalorder, ●primarytosecondarysubjectcategoriesfromtheAMSclassificationMSC2010,6 ●cross-references(inalphabeticalorder)toothertermsofthischapter (withbookmark“●”)andcross-referencestoothertermsofChapter3(withbookmark“u”),and ● informationsfrom onlinesites.Theseindicationsarecollectedinablockofinformation. Table2.1 providesanillustrationforanentrynamed “WarehouseProblem.”
TABLE2.1 Blockofinformationofentry“WarehouseProblem”
2.2GLOSSARYOFTERMSALPHABETA
●ABSAlgorithm. ThisalgorithmbyAbaffy-Broyden-Spedicatoisforsolving fullrankordeficientrank,determinedor underdetermined linearsystems.Letthefollowinglinearsystem Ax ¼ b , where AT ¼(a1, a2,...,am)T2 ℝm n with m n where a1 am arerowvectorsin ℝn,and b 2 ℝm , x 2 ℝn.Supposethatrank(A)isarbitrary.TheABSalgorithmisafinite proceduretaking m stepsthataredescribedin Table2.2
ABSalgorithmischaracterizedbythepropertythatthe kthiterate x(k) isasolutiontothefirst k equations.ABSalgorithmcanalsosolvelinearleast-squaresproblems,diophantineequations,andnonlinearalgebraicequations.
▸See: Abaffy,Broyden,andSpedicato(1984) and Spedicato(2009).
✓SubjectCategory (MSC2010): 65K05/Mathematicalprogrammingmethods, 65K10/Optimizationandvariationaltechniques.
●ActiveInequalityConstraint. Letaninequalityconstraintproblem(ICP)with m inequalityconstraintsbeminimize {f(x): x 2 ℝn ; g(x) 0},where f : ℝn 7! ℝ and g : ℝn 7! ℝm.Foranyfeasiblepoint, thesetofactiveinequalityconstraints is7 A(x) ¼ {j : gj(x) ¼ 0}.An ithconstraintissaid“active”(or“tight,”or“binding”)at x if gi(x) ¼ 0,and“inactive”if gi(x) 6¼ 0.An“activeset”A(x)at x consistsofallactiveconstraints.
▸See: Bertsekas (1999,p.314).
✓SubjectCategory (MSC2010): 90C05/Linearprogramming, 90C30/Nonlinearprogramming.
✓Alsorefersto ●ActiveSetMethod.
●ActiveSetMethod. If ^ x isalocalminimumofanICP(inequalityconstraintproblem),then ^ x isalsoalocaloptimumfor thisICPexcepttheinactiveconstraints.Theactivesetmethodpartitionsinequalityconstraintsinto activeconstraintsand inactiveconstraints.Theinactiveconstraintsareignoredfortheiteration.Theactivesetforthisiterationisthe workingset. Thenewpointisselectedbymovingontheworkingsurface.
▸See: NocedalandWright(2006) and Murty(1997)
✓SubjectCategory (MSC2010): 90C05/Linearprogramming, 90C30/Nonlinearprogramming.
✓Alsorefersto ●ActiveInequalityConstraint.
✓Retrievedfrom https://en.wikipedia.org/w/index.php?title ¼Active_set_method
TABLE2.2 BasicABSalgorithm
Algorithm2.1
BasicABSAlgorithm
1.Let x(1) bearbitrary,abaffian H(1) anarbitrarynonsingular n n matrix,and v(1) 2 ℝm anarbitrarynonzero.Set i ¼ 1.
2.Computetheresidual r(i ) ¼ Ax(i ) b
●IF r(i ) ¼ 0,THENSTOP x(i ) solvestheproblem
ELSEcompute s(i ) ¼ H(i )AT v(i )
●IF s(i ) 6¼ 0,THENGOTO3
●IF s(i ) ¼ 0and v(i )T r(i ) ¼ 0,THENset x(i+1) ¼ x(i ) , H(i+1) ¼ H(i ),andGOTO6.
ELSESTOP:nosolution
3.Computethesearchvectorby p(i ) ¼ H(i )T z(i ) where z(i ) 2 ℝn isarbitrarysavefor v(i )TAH(i )T z(i ) 6¼ 0
4.Updatethesolutionby x(i+1) ¼ x(i ) aip(i ),wherethestepwiseis ai ¼ v i ðÞT rðÞ r i ðÞT Ap i ðÞ
5.Updatethematrixby H i + 1 ðÞ ¼ H i ðÞ H i ðÞ AT v i ðÞ w i ðÞT H i ðÞ w i ðÞT H i ðÞ AT v i ðÞ ,where w(i ) 2 ℝn isarbitrarysavefor w(i )TH(i )AT v(i ) 6¼ 0.
6.IF i ¼ m THENSTOP: x(m+1) solvesthesystemELSEdefine v(i+1) asanarbitraryvectorlinearlyindependentfrom v(1) , , v(i ) GOTO2.
●AdaptativeConvexificationAlgorithm. Anadaptativeconvexificationalgorithmisamethodtosolve semi-infiniteprogramming (SIP)problemsbyusingasequenceoffeasibleiterates.ItconstructsadaptativeconvexrelaxationsofthelowerlevelproblemandsolvesthemathematicalprogramwithcomplementaryconditionsdrawnfromtheKKTconditions. SupposethattheSIPis
minimize x2X ℝ n f x ðÞ subjectto : g x, y ðÞ 0, y 2 0; 1 ½ ,
where f 2 C2(ℝn , ℝ)and g 2 C2(ℝn ℝ, ℝ)aretwicedifferentiablefunctions.Thelower-levelproblemofthis equivalent Stackelberggame is
maximize y 2Y ℝ g x, y ðÞ¼ Q x ðÞ subjectto : 0 y 1
Supposethat Q(x)denotesaconvexoptimizationproblem,thatis, g(x,.)isconcaveon Y ¼ [0,1].Replacingthelower level-problembyitsKKTconditions,weobtainthefollowing equivalentprogrammingproblem withadditionalcomplementaryconstraint.
minimize x, y , l, m f x ðÞ subjectto : g x, y ðÞ 0, ry g x, y ðÞ + l m ¼ 0, l y ¼ 0, m 1 y ðÞ¼ 0, y ,1 y , l, m 0,
where l , m denotethedualvariables.
▸See: Stein(2009).
✓SubjectCategory (MSC2010): 90C25/Convexprogramming, 97P50/Programmingtechniques.
✓Alsorefersto ●Semi-InfiniteProgramming; uStackelbergGame.
●AerodynamicOptimizationAlgorithm(AOA). Shape’soptimizationmethods areappliedtobetheaerodynamic design.Inaircraftdesign,AOAtechniquesuseComputationalFluidDynamicssimulations.Thedesignerspecifies,inparticular,asetofparametersthatdefinetherangeofpossiblegeometries.AOAfindsthevalueoftheseparametersthatminimizeanobjectivefunctionwhilesatisfyingasetofconstraints. Fastalgorithms arepreferredfordeterminingthegeometric designofanaircraft.
▸See: ZinggandElias(2006), HickenandZingg(2010),and Likeng,Zhenghong,andDehu(2013)
✓SubjectCategory (MSC2010): 90C90/Applicationofmathematicalprogramming, 76G25/Generalaerodynamicsandsubsonic flows.
●AffineRankMinimizationProblem(ARM). ARMproblemisanequality-constrainedminimizationproblem.Weare searchingfora matrixofminimumnuclearnorm belongingtoanaffinesubspace.Theproblemhasthefollowing convex formulation:
minimize X kk∗ subjectto: A X ðÞ¼ b,
where X 2 ℝm n and A : ℝ m n 7!ℝ p alinearmapping.Thedualconvexformulationis
maximize bT z subjectto: A∗ z ðÞ kk 1,
where A∗ : ℝ p 7!ℝ m n istheadjointof A
▸See: Recht,Fazel,andParrilo(2010)
✓SubjectCategory (MSC2010): 90C90/Applicationofmathematicalprogramming; 15A60/Normsofmatrices,numericalrange, andapplicationoffunctionalanalysistomatrixtheory.
✓Alsorefersto ●RankMinimizationProblem.
●AffineScalingAlgorithm. Anaffinescalingalgorithmreferstothe KarmarkarAlgorithm forLPsoftheform minimizex2ℝ n cT x : Ax ¼ b, x 0,where A hasfullrowrank.Thebasicstepsareshownin Table2.3 Thescalingoperationconsistsinmultiplyingby X suchaswith AX and Xc.
▸See: Holder(2014)
✓SubjectCategory (MSC2010):90C5/Linearprogramming, 65K05/Mathematicalprogrammingmethods.
✓Alsorefersto ●KarmarkarAlgorithm.
●AirlineIndustryOptimization. Problemareasofairlineindustrylargelyuse operationsresearch methodology.These areasincludenotablythefollowingproblems:flightscheduleconstruction,fleetassignment,aircraftroutingprocess,crew scheduling,revenuemanagementairlinere-routing,andaircraftgrounddelay.Themethodsusedtosolvetheseproblems are integer-programming (e.g.,aircraftrouting,crewpairing), mixedintegerprogramming (e.g.,fleetassignment), network model (e.g.,crewpairing,crewscheduling,irregularevents,andresources), probabilisticdecisionmodel (e.g.,revenue management), stochasticlinearprogramming (e.g.,aircraftgrounddelay), dynamicprogramming (revenuemanagement), and heuristicmethods
▸See: YuandThengvall(2009)
✓SubjectCategory (MSC2010): 90C06/Large-scaleproblem, 90C08/Specialproblemsoflinearprogramming, 90C90/Applications ofmathematicalprogramming, 90C08/Specialproblemsoflinearprogramming, 90C35/Programminginvolvinggraphsor networks.
✓Alsorefersto ●CrewSchedulingProblem.
●Algorithm. Analgorithmreferstoa step-by-step descriptionofasolutiontechniquetoaproblem.
TABLE2.3 Affinescalingalgorithm
Algorithm2.2
Affinescalingalgorithm
1.Given x > 0,let X ¼ diag(x).
2.Estimatedual y ¼ (AX2AT) 1AX2c, y 2 ℝm , and d ¼ c AT y, d 2 ℝn
3.Move x ¼ x aX2 d Xd kk∞ ,where a 2 (0,1).
FIGURE2.2 Iterationpathfromvalue1tothesquare rootexactvalue3.31662ofnumber11.
Example2.1.
Asimpleexample forcomputingthesquarerootofanypositivenumber(see Yang,2014,pp.1–2) is
where k denotestheiterationcounterand N ¼ 11thepositivenumberforwhichweseektoachievetheexactsquarerootat 3.31662.SolvingtherecurrenceEquation (2.1) yieldsthecorrectexpressionoftheiterationpath ak ½ ¼ 11p coth2k coth 1 1 11p .Thefastconvergenceisshownin Figure2.2
Analgorithmmapreferstoasequenceoftheform x(k+1) ¼ Sk(Ak(x(k))),withaninitialpoint x0.Thenotation Ak isan algorithmmapandnotation Sk isaselectionfunction.
Example2.2.
Wemayhave x(k+1) ¼ x(k) + skd(k),where sk isascalarparameter,and d(k) thedirectionofchange.
▸See: Holder(2014) and Yang (2014,pp.1–21)
✓SubjectCategory (MSC2010): 68Q25/Analysisofalgorithmsandproblemcomplexity.
✓Retrievedfrom http://glossary.computing.society.inform.org/ver2/mpgwiki/index.php?title ¼Main_Page
●AlgorithmicComplexity.Algorithmcomplexityisalsocalled timecomplexity ofanalgorithm.Itreferstothe numberof stepsneededtoexecuteanalgorithm.Itisameasureoftheefficiencyofanalgorithm.Thecomputationaltimecanbetaken asafunctionoftheproblemsize n.Itisdenotedbytheordernotation O(n)inthecaseofalinearcomplexity,while O(n2) referstoaquadraticcomplexity.Forexample,theinverseofan n n matrixoftenhas O(n3)complexity.Aquick-sort algorithmcanalsorequire O(n log n)calculationsabout3000calculationsfor n ¼ 1000.Inparticular,asolutiontoanoptimizationproblemcanbeexpressedinpolynomialtime.Anoptimizationproblem,whichcanbesolvableinpolynomial(P) timehas P-complexity.Anefficientalgorithmhas P-complexity.Aproblemwhichcannotbesolvableinpolynomialtime has NP-complexity.Noknownefficientalgorithmexisttosolve NP-hardproblems.Onlyapproximatesolutionsarepossiblefortheseproblems.
▸See: Yang (2010b,pp.24–25)
✓SubjectCategory (MSC2010): 68Q15/Complexityclasses.
●AnchorPoint. Anchorpoints(oroptimumvertices)areobtainedwhilesolvinga multiobjectiveoptimumproblem with r objectives,equalityandinequalityconstraints,andlowerandupperboundeddecisionvariables.Theanchorpoints(i.e., endpointsofthe Paretofrontier)areobtainedbysolving
minimize x2X ℝ n f1 x ðÞ, …, fr x ðÞ ðÞT where X ¼ {x 2 ℝn : gj(x) 0, j ¼ 1, …,m; hk(x) ¼ 0, k ¼ 1, …,p;and xi 2 [xi L , xi U], i ¼ 1,…,n}. Thereareexactly r anchorsforaMOPproblemwith r objectives.Anchorscanbeusedtoestimatethe Paretofrontier
▸See: Messac (2015,p.407).
✓SubjectCategory (MSC2010): 90C29/Multiobjectiveandgoalprogramming.
●AnnealingSchedule.Annealingschedulereferstoaprocedureofvaryingthetemperatureparameterinsimulated annealing(SA)algorithms tolowertheenergyofthesystemuntiltoconvergence NouraniandAndresen(1998) compared several annealingcoolingstrategies.Investigatedcoolingscheduleswereconstantthermodynamicspeed,exponential,logarithmic,andlinear.Theauthorsseekedtoidentifyacoolingschedule,which minimizesthetotalentropyproduction during theannealingprocess.The constantthermodynamicspeedschedule wasshowntobethebest.
▸See: NouraniandAndresen(1998)
✓SubjectCategory (MSC2010): 74N05/Crytals, 90C59/Approximatemethodsandheuristics.
✓Alsorefersto ●SimulatedAnnealing.
●AntColonyOptimization(ACO). ACOisaclassofsearchalgorithmsusingthebehaviorofanantcolony.TheACO algorithmwasdevelopedby Dorigo(1992) 8 Theswarmintelligenceofantsisused.Incombinatorialoptimization,the routesaremarkedbypheromenadepositedbyants.Anantwillpreferablychoosearoutewithaprobability,whichcan beproportionaltothepheromenaconcentration.Consideranetworkroutingproblemsuchas(i, j) 2 {1,2, …, n}.Theprobabilityofantsataparticularnodedependsonpheromoneconcentration fij anddesirability dij.Theprobabilityofantsata node i tochoosetheroutefromnode i tonode j canbeexpressedby pij ¼ fij a dij b/Pi,jfij a dij b,where a , b > 0areinfluence parameters(bothequalto1inthecaseofproportionality).Thepheromoneconcentrationchangesduetotheevaporation ataconstantrate g 2 [0,1].Supposetheexponentiallytimevariation f(t) ¼ f0 exp( gt).For g ≪ 1,wemayretainthe approximation f(t)(1 gt)f0.Thepheromoneconcentrationcanbeupdatedas
f(t) (1 gt)f0
Theshortestrouteswillbeselectedsince dij ∝ 1 sij ,where sij denotesthedistancebetween i and j
In Figure2.3 route2istheshortestroute.Theantsaredividedequallyonbothroutesin(a)atthefirstiteration.Ants choosemostlytheshortestroute2in(b)atafurtheriteration.
▸See: Dorigo(1992,2001) and Yang (2010b,pp.189–196)
✓SubjectCategory (MSC2010): 90C59/Approximatemethodsandheuristics, 92D50/Animalbehavior.
✓Alsorefersto ●VirtualAntAlgorithm.
●Antioptimization. Thepurposeofantioptimizationistolookfortheworstscenarioforanoptimum.Thisoptimization algorithmiscapableof introducingprocessuncertainties.Theantioptimizationmethodconsistsof twolevels.Atthe upper level,thegoalisastandardminimizationproblem.Atthe lowerlevel,theworstcaseforalltheconstraintsissearched.
▸See: Elishakoff,Haftka,andFang(1994), LombardiandHaftka(1998), McWilliam(2001), Guo,Bai,Zhang,andGao(2009),and Chevallier,Genty,Fressengeas,andJacquet(2013).
✓SubjectCategory (MSC2010): 90C30/Nonlinearprogramming, 90C29/Multiobjectiveandgoalprogramming.
●AppliedOptimization. Appliedoptimizationreferstotheapplicationoftheoryandmethodsin continuousandcombinatorialoptimization formodelingandsolvingoptimizationproblems.Vastareasofreal-worldproblemsareconcerned suchastransportationandcommunication,locationtheory,economicsandmarket,waterresourceplanning,manufacturing
Another random document with no related content on Scribd:
before (date I cannot recollect), also one from Genl. Ferguson, whom I shall be most happy to see, though I rejoice that he has escaped this winter campaign. I never wish to serve another, particularly for such a morose uncivil set, who will only talk. Adieu, may God bless you all, and may I soon have the happiness of embracing you. Remember me most kindly to the Adamsons, and believe me, ever most affectionately yours in the greatest haste,
W. W.
P.S.—Everything should be moved from Porto, I think. I will write by first opportunity
Our cavalry have distinguished themselves. This letter in perfect confidence from Yrs.
W. W.
We have had tremendous weather, particularly during our march over the mountains. As long as I have health, however, I do not care for myself, though I am not yet really hardened enough to misery and wretchedness, not to be unhappy at contemplating the miseries of war in our men and the wretched inhabitants of the country. May our beloved country never be a scene of warfare. Better ½ of its men should die on the beach.
Barfleur, S , Jany 18, 1809
M D F ,
I have just time to say I am quite well, and happy in the prospect of soon seeing all my beloved friends, after our disastrous and most harassing retreat from Lugo. We arrived at Coruña and found no transports, they arrived a few days after, but before we could embark the French attacked us on the 16th, with all their force, in our most disadvantageous position. They were repulsed by a valour which only English troops can possess, though exposed to a tremendous commanding fire of cannon. Poor Sir John Moore was killed. Sir
David Baird lost his arm. Our loss in killed and wounded is very great, though not so much as that of the enemy.
Our Brigade, which was in the Town to cover the embarkation, moved to cover a road to the right of the position, but were not attacked, or engaged at all, as was expected. We were therefore contemplators only of the gallant and astonishing firmness of our comrades. The 50th and the 42nd suffered most.
During the night most of our army embarked. Genl. Beresford’s Brigade covered the embarkation, having retired into the works of the town. The French approached in the morning close to us. We gave them a warm reception with our 24 prs. assisted by the Spaniards, who on this occasion behaved very well. The enemy fired on our transports most, and several went on shore and were lost in the confusion. Our situation was most critical all the next day and night, till (we) embarked the whole, about one in the morning.
Fortunately the enemy did not fire on the town, and suffered us to embark, (or were totally ignorant of it), without annoying us. We were very weak, just enough to man the works, and dreaded an assault, the boats being able to take only 500 at a time, and weather very bad. However we not only got ourselves but most of the wounded in safety, though all most overcome with fatigue.
Adieu, in hopes of soon seeing you, My dearest Parents. Kindest love to all my friends, from your most affectionate Son, W . W .
(Note, in Henry Warre’s writing, “Received 24th Jany. at night.”)
P , Jany 23, 1809
At last, my Dearest Mother, I have the happiness to tell you of our safe arrival at this place. I wrote a few lines in a great hurry from off Coruña, which I hope you received. I long to reach town, and shall set off as soon as possible in a chaise, with Col. Douglas of Wycombe. We go by Bath, where we shall shake the Hardies by the
hand, and in 4 days shall, I hope, embrace all my beloved family I am very far from well, and most in need of rest. A constant bowel complaint, occasioned by fatigue and being constantly wet, has pulled me down very much. I am a mere skeleton, but rest and the happiness of seeing all that is dearest to me will soon, I think, recover me.
To describe our anxiety, and what we went through at Coruña the last day and night, is not easy. Suffice it to say, we had (but for Mr Samuel How) been left behind, and now instead of being in our dear native country, should have been marching prisoners to France. The thought even now makes me shudder. Nearly exhausted and harassed to death, we were in a bad state to undertake such a journey. We, however, were more fortunate and brought off all our sick and wounded except very few.
Don’t forget our last conversation. I have indulged in it in my most distressing moments. What a spur it has been to exertion I leave you to guess.
Adieu; kindest love to my dear Father, Emily, Uncle Wm., etc., etc., etc., from, Dearest Mother, Your most truly Affectionate W . W .
P.S.—Pray buy me some worsted socks very long in the feet, I am almost naked as to foot, having worn my present pair at least ten days.
L , March 3, 1809
M D F ,
We arrived here yesterday, safe and well, after a very pleasant voyage of 8 days. The Portuguese are in high spirits, and promise well. They have had some skirmishing on the Minho, and repulsed the French, whose numbers we know nothing certain of. Of course these accounts are much exaggerated, but if they can be made to think they can resist, and stand fire, it is a great point.
As to our own destination I as yet know nothing. The Portuguese army is on the frontier towards Monte Rey. I suppose we shall join them. Romana is near there, and, I hear, has collected a considerable force, and is in spirits as is the Marquez de Valliadacen, who is with them. It is, it seems, the general opinion that the French under Thomier, about 10,000 men, will endeavour to penetrate by the Minho, and that the Portuguese are determined to give them fight. By the last accounts from the frontier not a Frenchman had passed it. Something may yet be done.
The Spaniards under Cuesta and the Duke of Infantado have advanced towards the border of the Sierra Morena nearest Madrid, and at least our official accounts tell us that they speak with confidence, and are in high spirits. Romana wants nothing but ammunition, which has been sent, and we spoke at sea and brought into this with us a Spanish schooner with 105,000 dollars for the Asturians, who has proceeded.
Every thing that I hear confirms my opinion that our retreat from Spain, etc., etc., etc., was inconsiderate, and I fear will place us in rather a disgraceful light. This entre nous The French after they entered Coruña acknowledged having lost 1000 men killed on the 16th, and of course more than as many wounded. They spoke highly of the bravery of our men. This we have from the General’s Italian servants, whom we left there, and who were in Gurèa’s house when Laborde took up his Quarters there. This I believe certain, that Buonaparte has returned to Paris, and taken his Imperial Guards with him.
The Brest fleet 16 S. L. and 3 frigates is out. We were becalmed off Cape Finisterre only a few hours before they came up bringing the breeze. It was a narrow escape. Yesterday Sir John Duckworth was off here with 11 S. L. and 2 frigates, and was joined from here by 2 S. L., the Norge and Conqueror, and is in pursuit of them. God send he may come up with them. The issue is not doubtful.
So much for Public News. I send on mere reports, though I do not entirely vouch for the whole being true. The Portuguese are very anxious for Sir Arthur Wellesley. They think he would do everything
that is possible. Nothing can exceed the high idea they have of him, and they are right.
I am very sorry to tell you that I hear Alvez had not shipped any of your wines, and had near 340 pipes on hand. They complain of want of instructions from you, but could, I believe, if he had exerted himself, have got freight for most. Ignorant as I am of business, and particularly of the instructions you may have given him, I feel great delicacy in writing and giving any orders. He never, I think, can have received my letter from Coruña. Croft certainly did not. I shall write to him by to-morrow post, desiring him to give me an account of how your affairs are, at the same time taking upon myself to desire him, if not contrary to any instructions he may have received from you, to charter at all events a vessel to ship off all your wines (if he can get one), but to wait for convoy unless the business presses very much. Though things look brighter than I expected, the fate of war is so uncertain, and the odds are against us, so that I think no time should be lost. I should have chartered a vessel here, but on consulting with our worthy friend James Butler, he seems not to think it worth while till I hear from Alvez, and there is no English ship at present in this port. I have felt much distressed at this apparent want of foresight, but suppose the last packet must have brought him your instructions. Nothing can be kinder than the interest that Wm. Naylor and Butler take in your concerns, but with great delicacy. It was said on Change at Porto that several Packets only brought 2 letters from my Uncle to John Benito, which caused a smile. I write this to you, my dearest Friend, because these sort of smiles, I fear, do much harm in business. This I heard from other quarters. Croft is here, but I have not yet met him. My heavy baggage, which was left here, I will send by the Amazon to England. I write this by the Peacock sloop of war, though in great haste. I am quite well. I have so many things to do and think of that I hope I shall not have time to be sick. I will write again by the first opportunity. In meantime may God bless and preserve you all. Give my kindest and warmest love to my dearest Mother, etc., etc., and from Yr. ever most affectionate Son, W . W .
The weather is most delightful though very warm. The change from England is very striking. Adieu.
I hope the wines I ordered from Spain, have or will be sent.
M D F ,
L , April 1, 1809.
In addition to what I wrote to my mother by this Conveyance, the Diligent Gun Brig, which has been delayed by the Bar, I have merely time to communicate the very disastrous news of the taking of Porto by the French. We have as yet received no particulars, and only know that the Bishop, and one British officer, Captn. Arenschild of Artillery, the G. Legion, were arrived at Coimbra.
From the complete state of insubordination of the populace of that city, this event we have for some time foreseen, and in the state of indiscipline and insubordination of the Portuguese army, any assistance we could have sent would have, I much fear, only added to our loss, as they would have been also hurried away by, and as intractable as, the mob, who, cruel and sanguinary to an excess against themselves or prisoners, are always timid and cowardly. They have assassinated many people there, amongst others Oliveira the former Governor, who was in gaol. They also murdered nine or ten French prisoners, and let all the felons loose. Such was their wretched state that they would obey no one, and rendered it highly dangerous to attempt any plan to secure a retreat, in case of accidents, as you would risk being murdered. I therefore fear our loss in men and arms very great, but we have no details.
The Mob, some days before, broke into a magazine of arms, which they plundered, and then seized the Fort of St Johns, allowing no ship to go out. I have therefore every reason to fear your ship with wine, which was loaded, was unable to get out. The Captain had moved over to the other side of the water, which is however within shot. The wines in the Lodges, if they have, as I hope, destroyed the
Bridge, are still safe, for bad as this news is, I have still hopes that Soult and his division are in a bad scrape, and weaker in numbers than generally supposed. The Provinces behind him are in a state of insurrection, and I trust Silveira will get into their rear, as he is now disembarrassed by the taking (by him) of the Fort of St Francisco at Chaves, with 870, added to 200 in Chaves, when he before entered, who were sick, upwards of a thousand, and 300 killed. He has also taken more mules, horses, artillery, etc., etc., than were in the place when taken by the French, and his own loss trifling.
Galicia is certainly in a state of complete insurrection and full of enthusiasm and spirit. They have summoned Vigo, and given the French garrison only 24 hours to decide in. Tuy is also surrounded and expected to fall. Thus the retreat or communications of Soult’s Corps (of whose numbers we are ignorant, but cannot believe exceed 15,000 men) are pretty well cut off, and, unless supported by the Corps, which threatens us by the banks of the Tagus, and at present besieges Ciudad Rodrigo, will, I hope, be destroyed.
We have also a report here to-day that the Duke of Albuquerque and Cuesta have joined and given Victor a beating, which we give little credit to, as we knew of Victor’s precipitate retreat from pursuing Cuesta towards the South, and being followed by that General, whose retreat was a very masterly movement, and I suspect had really drawn the French into a cul de sac, which they discovered before it was intended they should, but late enough to enable their rear to be turned. Urbina, it is said, is advancing towards Madrid. If Austria would but declare, everything might yet go well.
Our friend Whittingham has distinguished himself very much, and been thanked in orders by the Spanish General, I am not sure which. He was quite well with the Carolina army. The people here are quiet at present, though not much pleased with the inactivity of the English force. They are great fools, and know nothing about the matter, though I myself wish our people would make a movement. Adieu, in great haste, with kindest love to all at home, ever yr. most affectionate Son,
W . W .
Major P.F., Aide-de-Camp.
M D F ,
April 7, 1809.
I take the opportunity of Fred Crofts going in the Amazon to send you receipts for my Staff pay. I also yesterday drew on you, dated 5th inst., to Dr Deane or order £56, 10s, amount of a Spanish horse bought of him, and which by providing me with two horses renders it unnecessary you should be at any trouble about buying me any, as the General having given me one, I have now 3, which is enough.
Croft will tell you all the news and all about me, which it is out of my power to do now myself, as Croft will tell you. We leave this tomorrow for the army at Thomar, which the Marshal is going to take Command of.
By the Amazon I have sent all my heavy baggage, five cases, etc., and some sweetmeats, which pray send to Ferguson to present to Mrs Ferguson with my best respects. I have also sent some chains directed to my Mother, which she will be so good as to distribute as directed. I am most anxious to hear from you and will write myself on the first opportunity.
As I have only time now to beg my kindest love, and assure you I am ever most Affectionately Yours,
W . W .
H ., T , April 27, 1809.
M D F ,
Many thanks for your very affectionate letters of the 10th, 7th, 5th April, which I received all together, and which were most pleasing to
me, whose happiness so much depends upon your approval of my conduct.
You will long before this arrives have heard of the melancholy fate of Oporto. It did not in the least surprise me. I was sure it would be taken the moment it was attacked in earnest; the inevitable consequence of insubordination and anarchy. I hope you had ensured your property.
I was of course delighted to use every exertion in my power, and am very much indebted to Mr Villiers for his friendly assistance. Long before the crisis he offered me a transport or more to go round and bring away the property, which I refused in consequence of letters from Pedro Alvez stating that one ship was arrived, and another daily expected, and fearing that the expense of chartering them would be lost. At the same time I was unaware of how little resistance would be made at Braga, and the Passes of Salamonde, etc. Since that Mr Villiers wrote, as did also Noble, very strong letters to Capt. Loring of the Niobe to render every assistance, but these were too late and have since been returned to me, as also one you wrote to Chiappe with some accounts, which I opened, and have ready to deliver when an opportunity, I trust not very remote, shall enable me, as also those you send me now.
We expect to march immediately to drive that miscreant Soult out of Porto. The General went two days ago to Lisbon to meet Sir A. Wellesley, and as soon as he returns this evening or to-morrow, we shall all advance. I was left here to continue to form the Algarve Brigade, the finest in the service, and who march to-morrow morning.
I have every hope that Soult has committed himself by his rapid advance, and since detached Corps, one of which 7 to 8000 have attacked Silveira at Amarante two days successively. He has defended himself bravely as did the Regt. No. 9 (Peña Macor) commanded by Major Patrick, who came over with us, and who is, poor fellow, I fear, badly wounded, after distinguishing himself very much. Silveira expected to be attacked next morning, and will, I fear,
not be able to resist, as the Militias and Ordenança had abandoned him.
Victor has called everything to him near Merida, from Salamanca, and even Zamora, which looks as if he was close pressed, and leaves our Eastern frontier unmenaced for the present.
Cuesta has certainly reassembled 20,000 Infantry and 5500 horse and has pushed forward his advanced guards. If the Spaniards can reassemble their armies in so short a time after being dispersed they must in the end destroy the French, unless they receive great succours, which I believe impossible.
My friend Col. D’Urban, who was in the battle of Medellin, assures me he never saw any troops behave better than the Infantry, or worse than the Cavalry, of Cuesta’s army. And I think this was, as well as the loss of the army, in a great measure owing to Cuesta’s bad order of battle, in the extended line without any reserve whatever, his Cavalry in the first line advancing with the Infantry at their pace, and his having allowed the enemy to pass the Bridge of Medellin and deploy before he attacked them. He committed the same fault at Rio Seco, and suffered for it. It appears an infatuation, and as unaccountable as Victor’s not attempting to pursue the Spaniards, who fled in confusion, even with his Cavalry, which leads me to suppose he must have suffered more than we are aware of.
The enemy have occupied Valença de Tuy without resistance, Vianna, Ponte de Lima, Penafiel, and desolated these unhappy countries. On this side their posts are at Ovar and on the Vouga, and our advance on this side of that river, under Col. Trant. They have constant skirmishes which signify nothing except wasting ammunition. In Porto itself there are not above 800 or 1000 men, and they are organising a Portuguese Legion, for which they have got some men.
I was in a state of the greatest anxiety about poor dear Clara, to whom I had written several letters without receiving an answer, till yesterday, when, by a letter from my worthy friend Bettrão, I heard she with the rest of the ladies had quitted the convent on the news of the approach of the French, and their entering Porto, and had
travelled on foot over the mountains to Mesão Frio, and then to Ancede, where she now is with another nun, a friend of hers, with some of Frè Bernardo’s relations who have afforded her every protection, and he has written to them to give her every assistance. She was quite well, he tells me. I immediately despatched John Benito by the extra post, with a letter to her and 15 Pieces, besides an order for 15 more, in case of necessity, desiring him to stay with her as servant, and to remove her as a guard, in case of absolute necessity towards Lisbon, where I intend to place her with Sʳ Lucas de Siabra’s family, if she is forced to fly, till I can make some proper arrangement. If danger should not press, she is to stay where she is, till I can get away to see her myself and make other arrangements. At present the chances of war are so uncertain that I think she is better out of the convent, the marked objects of vengeance to these unprincipled invaders. Frè Bernardo, to whom I have sent John Benito, (in whom I have every confidence from his attachment to our family and honesty), will give him the orders he thinks necessary, and he will stay with her as her servant, and in case of removal guard, till I can make any other arrangement.
I have got three pretty good horses and therefore, unless you have already sent them, do not think it worth while being at the expense of sending out any more.
The Portuguese troops immediately under the instruction of British officers are coming on very well. I could have wished we had been allowed more time, but even now have great hopes of some corps. The men may be made anything we please of, with proper management, and, wherever I have had authority, I have soon settled the little mean jealousies and tricks of the officers, and without, I hope, gaining much ill will. I endeavour to combine inflexible firmness with politeness of manner. I know it is the only way to make these fellows respect you, and the mass of officers is miserable indeed. This, however, will in time be altered. Merit is the great recommendation with the General, not grey hairs and number of years service, however much to be respected, for these Subalterns, some of whom should be anything but soldiers.
I am very happy to hear the 23rd are coming out to this country, and should like much to join them, if I could with propriety. It is a fine dashing service, but this I fear is impossible, and I begin to learn the necessity of commanding my wishes and feelings. At all events I completely agree with you that it would be folly to quit the Dragoons, when I have two years longer to serve as Captn., and God knows what changes may occur in that period.
Every officer I have heard speak on the subject is much dissatisfied with the new C. in C., particularly those who most know him; and, setting H. R. Highness’s morality aside, he did incalculable good to the army, and I am sure we cannot have a better, at least that I know of, and this is the opinion of, I believe, the majority of the army.
By the new regulations of service we shall have Brigadiers at 60 years of age, and Generals in night cap and slippers, prudent and inactive as they formerly were, and as the Portuguese are. It is surprising that people can suppose a man unfit to command, till he has attained an age at which enterprise and activity generally cease. I should not be surprised to see some years hence advertised in the papers of the day restorative cordials for Generals taking commands, or patent easy-chairs for foreign service, addressed to the Generals of the British army.
I am much obliged to you for your kind attention to Custine’s letter, [12] and the advance of 10£. I would not wish you to commit yourself in cashing his bills to any considerable amount. He was once in Germany very civil to me, and I am happy to be able to repay him. I should have been better satisfied with the parole d’honneur of a gentleman, than that of a French officer, which goes very little way in my opinion. He is a prisoner, and in distress, poor fellow. I therefore in moderation will be very happy to afford him some assistance, and I hope he will not deceive the idea I have formed of him.
I have just heard that the 3rd and 4th heavy Dragoons are arrived and landed at Lisbon.
My Boots, etc., will be a valuable acquisition to me, and which, as well as the plans you are so good as to send, are arrived.
Pray give my kindest love to all at Home, from, my Dearest Father, your ever Affectionate and Dutiful Son,
W . W
I wrote two days ago to my mother, and suppose the letter will go by the same conveyance as this. Adieu.
[12] See Memoir, and p. 101.
M D F ,
L , 13th July, 1809.
Though I wrote to you a very long letter by last Packet, and am now somewhat prest for time, I will not delay thanking you for your affectionate letters of 20th of May and 1st of June, which did not reach me till yesterday, having travelled to Porto and back again after me, and in it my uncle Wm’s. very kind letter of 20th May, for which pray thank him with my kindest love, and tell him I will answer very shortly, as also Hardy’s, whose entire recovery gives me the sincerest pleasure, and I hope soon to hear that he has got a ship. At a time when so much is doing in all parts of the world, I know it must be irksome to him to be unemployed.
We were to have left this place yesterday to join the army assembled about Guarda, etc., and to advance into Spain as an army of observation, but business has prevented the General, and we only set off to-morrow morning, and proceed direct to Guarda, where we shall remain but a few days, I suppose.
Most of the English officers who came over to join the Portuguese army have accepted the Pay. I have however refused it, as I cannot see any credit in serving them for the pittance of Pay, particularly when I know they are so poor they cannot pay their own Officers. Besides, I consider that receiving Pay invalidates in some measure my claims on future promotion in my own service, and in some
degree deprives me, I consider, of the right of quitting this when I choose. I am ready, as I told the Marshal, to exert myself for the service of this Country without being any weight or charge to them. They have certainly some claims to my service from the kindness my family has for a long series of years experienced, and if H.R.H. hereafter chooses to reward me in the end, he can do so, without my being an expense to his Government, and, if he does not, I am pretty tolerably indifferent, and shall be satisfied with having done my duty. I certainly very much dislike this service and their mean intrigue and absurd presumption, which shades their good qualities, and would therefore avoid any possible reason for my being kept with it longer than suits my convenience and I consider my duty requires. I hope you therefore will approve of my having declined any emolument for my services.
The conduct of the English Government in refusing the step of rank to those Officers who have come out, or, being here appointed, have joined the Army, is very extraordinary. They now have only an additional step in the Portuguese, and the pay of both. I am astonished any British officer will come out on these terms.
I will write to you whenever an opportunity occurs. In the meantime, my Dearest Father, give my kindest love to all at Home, and believe me Affectionately Your Dutiful Son, W . W .
I do not send the certificate of horses lost at Coruña, as we have written home for the printed form, when I shall know how it is to be filled up. Adieu.
L , 10th August, 1809
M D F ,
An unfortunate accident of having dislocated the knuckles of my right hand, and having broke one of the small bones, obliges me to apply to my friend Captn. Souza to serve as an amanuensis. It being
now nearly a month since the accident happened, I am afraid you will be very anxious to hear from me. I am in other respects perfectly well, and so far recovered from this, that I but yesterday returned from travelling night and day to the English Hd. Quarters post and back again. I have not yet, however, quite the use of my hand.
You will long before this have heard of the battle of Talavera perhaps the most glorious ever gained, if we consider the disproportion of numbers. Not having had the good fortune to be present I can give no further particulars than you will have seen by Sir Arthur’s despatch. The attacks were most vigorous and repeated by upwards of 40,000 men in heavy columns, first against the left, then the right, and afterwards along the whole British line which was occupied by about 19,000 men. Nothing however could overcome the steadiness and gallantry of our troops. After having been engaged the 26th and 27th, the greatest part of the night between the 27th and 28th, and from daybreak till night that day, the enemy was completely repulsed, leaving 11,000 killed and wounded on the field, and the next morning retired 4 leagues to Sebola. Our loss was also very considerable, about 4500 killed and wounded. You will be sorry to hear that the 23rd lost half their men in a charge, and among a great many officers wounded are Capt. Howard badly, Drake ditto. He was taken and afterwards released by the enemy, Allen wounded and taken, D. W. Russell slightly, Frankland slightly, Lieut. Anderson badly, and 226 men killed and wounded. I saw Col. Seymour and Dance, who are quite well. The Regt. was ordered to charge two columns of the enemy, who were deploying, but who unfortunately had time to form square without there being time for the order being revoked, and they unfortunately persevered in attempting an attack which it was impossible should succeed.
The British army as usual has been deprived of the fruits of their glorious victory; for Soult, Ney, and Mortier, having penetrated from Castille to Placencia with 34,000 men, added to the impossibility of placing any dependance upon the Spaniards, who during and after the battle of Talavera had remained, except their Artillery, entirely spectators, with 20,000 men, exposing the British army to finding itself between two fires, besides entirely cutting off its retreat and
communications with Portugal, obliged Sir Arthur to retire by the bridge of Arçobispo to the other side of the Tagus; that of Almaraz was already occupied by the enemy. Cuesta, who was left at Talavera to keep the army of Victor in check, I suppose not feeling very confident in his troops, set off after Sir Arthur, thus abandoning all such of our wounded, who could not crawl along the road, to the enemy, who however, it must be confessed, on all occasions have treated the English prisoners with great humanity.
We have moved forward with the Portuguese army to occupy the strong passes near this place, and assist, as far as we may be able with our small force of 12,000 men, the British and Spanish armies, the former of which occupies a position on the South Bank of the Tagus at Almaraz, the latter at Arçobispo. The names of these passes are Perales and Gata and are at four leagues distance from Coria. The French have advanced towards Talavera from Placencia. Our army are in very good spirits, and will, I have no doubt, maintain their character better than their neighbours, in whom, you know, I never had much confidence.
I am happy to tell you that Jack Prince is well, also Genl. Fane. Poor Milman is badly wounded, as is Sir W. Sheridan.
I am much obliged to you for the boots and my glass, which I have received, and which I was in great want of.
I will write to you again the moment I am able, and in the meantime I have only to add that I remain, my dear Father, with love to all at home, Your very affectionate Son,
W . W , A.D.C.
I beg you will believe my hand is really of no consequence and nearly well, nor do I find it a bit the worse for a ride of fifty hours de suite to the British Head Quarters and 36 back, and I am otherwise in as good health as I ever was.
L H , August 13
I have been unable to forward this letter before to-day and have merely to add that my hand is much better. We continue near these passes, though we made the other day a movement to Salteros, but
retired again to the same position, and established our Head Quarters at this place. I believe Soult’s, Ney’s, etc., army are moving again into Castille by Baños without deigning to take notice of us. The cowardly Spaniards have suffered the enemy to pass the bridge at Arçobispo with very little resistance, and now occupy the passes in which I left the British army, on its right flank.
Every day convinces me more strongly that the fate of these countries depends entirely upon Austria, of which, you may well imagine, we are most anxious for positive accounts. We have had a French bulletin with accounts of an armistice, and other rumours of a peace. But as they have all come from the French, I trust unfounded. I hope you will let all your arrangements, with regard to Portugal, depend upon the successes in Germany.
Yrs. most affectionately, W . W .
M D M ,
S , August 18, 1809
I take the opportunity of being able to write to give you some account of myself and our proceedings. My hand, as you will see by my being able to write, is nearly well, though still weak. I suffered a good deal from it, from not applying the proper remedies, and supposing I had merely dislocated two of my knuckles, for my hand and arm had swelled so much, from travelling day after day in the excessive heat, that it was not till I arrived at the English Head Quarters express a month after, and consulted an English surgeon, that I discovered that one of the small bones in the back of the hand was broken. Nature, however, has joined it, and I trust in a few weeks I shall be entirely as strong as ever. It has been a serious inconvenience, particularly when near the enemy, and expecting to be engaged. Except in writing, however, it never has prevented my
duty, though I confess sometimes, after a sleepless night, I could almost have cried from pain and vexation.
I dictated a letter to my father from Acebo and Los Hoyos, fearing you would be very anxious at not hearing from me, which I hope has been received. We have now made a forward movement to Moralega in order to straiten the enemy in his foraging. They constantly dislodged a post we had at Coria, where they came for provisions, nor was it in our power to prevent them, and the inhabitants, who had not fled, either from fear or treason, seemed more ready to supply them than us, so much so in every direction, added to the ignorance and want of arrangement in our Commissaries, that our troops have suffered greatly from want of provisions, particularly bread. The selfish unfriendly conduct of the Spaniards high and low, not giving us any hopes of a supply, Marshal Beresford has been forced to retreat towards this place, on his way to Castello Branco, in order to feed his troops, who are in great distress, without even seeing the enemy, or his making the least forward movement towards us, except in small foraging parties, to Coria, near where they have caught a valuable convoy of English hospital stores, I cannot help thinking, in a great measure from the excessive ignorance and want of energy in the Purveyor, who was seven days considering whether the French would come there or not.
As to the conduct of the Spaniards, both to the English and this army, it has been most shameful. I shall not enlarge on this disagreeable subject. It is enough to say both armies are very much irritated. They have every wish that we should fight for them, but do not deign to treat us with common civility, or our men, when sick or wounded, with common humanity. They conceal their provisions, drive away their cattle, and when possible escape themselves, leaving either friends or foes to subsist as well as they can, complaining however most loudly and bitterly if a single cabbage is taken without leave. When our men have been starving they have refused to sell even a loaf, and if they did, at a most exorbitant price. They will rob your very stores almost in your sight, and, though every town and village expects you are to stay for its defence, they will not,
except forced, contribute in any way to assist. This is the complaint, and universal in both English and Portuguese armies, and as for their soldiers fighting, I never thought they would. They never have. The French treat them with the utmost contempt. 5300 and odd brave soldiers of the British were killed or wounded at Talavera without 45,000 Spaniards, who were present, moving in any way to their support; and since, 3000 wounded of these were abandoned by that old brute Cuesta in Talavera, contrary to Sir A. Wellesley’s orders or intention, and without any attack on the part of the enemy.
This obstinate surly old ignorant fellow is, thank God, removed. He was, to say the best of him, quite superannuated, and so violent and obstinate that everybody feared him but his enemies.
There never was such folly as sending an army into Spain again. The character of the Spaniards is so selfish, jealous, and proud, with all the surliness of Englishmen, and not a spark of their good qualities, that a foreign army in their country must always risk being abandoned. They, besides, will not fight for themselves, and it is impossible England alone can defend them. This picture is perhaps strong, and I really feel much irritated against them, but I am sure it is the opinion of almost every individual. The inhabitants fly in all directions at the approach of the enemy, and whenever your army comes, they fancy the enemy are coming also. You are therefore unable to procure subsistence, and of course equally so to defend them. The magistrates fly, to avoid the trouble of providing you, as everything is concealed. All the towns we have been in are nearly abandoned, and we have been forced to break into empty houses for a lodging. In short, war in any shape is a horrid scourge to the inhabitants.
We are in very low spirits at the bad accounts from Austria. A peace in that country will decide the fate of these most undoubtedly We may prolong the war and sacrifice many lives, but I am convinced that it will be to no purpose, and even should Sir A.W., who, it is reported, is to be made Commander in Chief in Spain, and a most clever fine fellow as ever existed, be able to avert their ultimate destruction, another brilliant victory, or even more, if the Tyrant overruns Germany, and Austria falls, cannot alter my opinion,
and I shall doubly regret every British life that is lost after that country makes peace.
Poor Whittingham, who is a Brigadier in the Spanish Service, was shot through the cheek and hurt severely, while endeavouring most gallantly to rally a Spanish regiment of cavalry. He is however doing well. I am much annoyed at not being able to get any account of Harvey. Milman is badly wounded. These are the only officers I have heard of that you know. Fremantle is well.
C B , August 20th.
We arrived here yesterday, and will, I hope, remain some days to refresh our poor patient half-starved soldiers, and observe the enemy’s motions. A strong corps of theirs forced the Pass of Baños defended by Sir R. Wilson and about 3000 men, Portuguese and Spaniards. They resisted the whole day, but had no guns, and were forced to retreat to avoid being surrounded.
It is impossible to judge yet of what the plans of the French can be, particularly this Corps, which has re-entered Castile and marched towards Salamanca, leaving 10 or 12,000 men at Placencia; nor have I the least idea of what Sir A. Wellesley’s intentions are. I over and over again wish I was with his brave army. It is wretched unsatisfactory work being with this; nothing but constant vexation and disgust, particularly of their Officers. The men, poor devils, are patient and obedient, voilà tout, I think, yet the British Officers with the regiments think they would fight. I am convinced this would depend entirely on circumstances, and if they do unfortunately get beaten, I fear they will at any rate not hazard it again. What a different army I was with a year ago! How gloriously employed where with such soldiers! If Austria makes peace, I shall soon have the happiness again of embracing my beloved family, for the game will be soon settled in these countries.
I think the French will move towards Zamora, and threaten Portugal immediately, to draw away our army from this quarter, and Sir A. W., if possible, out of Spain, to protect it.
Adieu, my dearest Mother, kindest love to my Father, etc., etc., etc., from Your Ever Affectionate Son,
