DESIGNFOREMBEDDED IMAGEPROCESSING ONFPGAs
SecondEdition
DonaldG.Bailey
MasseyUniversity,PalmerstonNorth,NewZealand
Thissecondeditionfirstpublished2024 ©2024JohnWiley&Sons,Ltd
EditionHistory
JohnWiley&Sons(Asia)PteLtd(1e,2011)
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmitted,inanyformorbyany means,electronic,mechanical,photocopying,recordingorotherwise,exceptaspermittedbylaw.Adviceonhowtoobtain permissiontoreusematerialfromthistitleisavailableat http://www.wiley.com/go/permissions
TherightofDonaldG.Baileytobeidentifiedastheauthorofthisworkhasbeenassertedinaccordancewithlaw.
RegisteredOffice(s)
JohnWiley&Sons,Inc.,111RiverStreet,Hoboken,NJ07030,USA
JohnWiley&SonsLtd,TheAtrium,SouthernGate,Chichester,WestSussex,PO198SQ,UK
Fordetailsofourglobaleditorialoffices,customerservices,andmoreinformationaboutWileyproductsvisitusat www.wiley.com
Wileyalsopublishesitsbooksinavarietyofelectronicformatsandbyprint-on-demand.Somecontentthatappearsinstandard printversionsofthisbookmaynotbeavailableinotherformats.
Trademarks
WileyandtheWileylogoaretrademarksorregisteredtrademarksofJohnWiley&Sons,Inc.and/oritsaffiliatesintheUnited Statesandothercountriesandmaynotbeusedwithoutwrittenpermission.Allothertrademarksarethepropertyoftheirrespective owners.JohnWiley&Sons,Inc.isnotassociatedwithanyproductorvendormentionedinthisbook.
LimitofLiability/DisclaimerofWarranty
Whilethepublisherandauthorshaveusedtheirbesteffortsinpreparingthiswork,theymakenorepresentationsorwarrantieswith respecttotheaccuracyorcompletenessofthecontentsofthisworkandspecificallydisclaimallwarranties,includingwithout limitationanyimpliedwarrantiesofmerchantabilityorfitnessforaparticularpurpose.Nowarrantymaybecreatedorextendedby salesrepresentatives,writtensalesmaterialsorpromotionalstatementsforthiswork.Thisworkissoldwiththeunderstandingthat thepublisherisnotengagedinrenderingprofessionalservices.Theadviceandstrategiescontainedhereinmaynotbesuitablefor yoursituation.Youshouldconsultwithaspecialistwhereappropriate.Thefactthatanorganization,website,orproductisreferred tointhisworkasacitationand/orpotentialsourceoffurtherinformationdoesnotmeanthatthepublisherandauthorsendorsethe informationorservicestheorganization,website,orproductmayprovideorrecommendationsitmaymake.Further,readersshould beawarethatwebsiteslistedinthisworkmayhavechangedordisappearedbetweenwhenthisworkwaswrittenandwhenitisread. Neitherthepublishernorauthorsshallbeliableforanylossofprofitoranyothercommercialdamages,includingbutnotlimitedto special,incidental,consequential,orotherdamages.
LibraryofCongressCataloging-in-PublicationData
Names:Bailey,DonaldG.(DonaldGraeme),1962-author.
Title:DesignforembeddedimageprocessingonFPGAs/DonaldG.Bailey.
Description:Secondedition.|Hoboken,NJ:Wiley,2024.|Includesindex.
Identifiers:LCCN2023011469(print)|LCCN2023011470(ebook)|ISBN 9781119819790(cloth)|ISBN9781119819806(adobepdf)|ISBN 9781119819813(epub)
Subjects:LCSH:Embeddedcomputersystems.|Fieldprogrammablegate arrays.
Classification:LCCTK7895.E42B32642024(print)|LCCTK7895.E42 (ebook)|DDC621.39/5–dc23/eng/20230321
LCrecordavailableathttps://lccn.loc.gov/2023011469
LCebookrecordavailableathttps://lccn.loc.gov/2023011470
CoverDesign:Wiley
CoverImage:©bestfoto77/Shutterstock
Setin9/11ptTimesLTStdbyStraive,Chennai,India
1.2.2Colour 5
1.3ImageProcessingOperations6
1.4Real-timeImageProcessing8 1.5EmbeddedImageProcessing9
1.6ComputerArchitecture10
1.7Parallelism 11
1.7.1TemporalorTaskParallelism12
1.7.2SpatialorDataParallelism13
1.7.3LogicalParallelism14
2Field-programmableGateArrays19
2.1HardwareArchitectureofFPGAs19
2.1.1Logic 21
2.1.2DSPBlocks22
2.1.3Memory23
2.1.4EmbeddedCPU23
2.1.5Interconnect24
2.1.6InputandOutput24
2.1.7Clocking26
2.1.8Configuration26
2.1.9FPGAsvs.ASICs27
2.2ProgrammingFPGAs28
2.2.1RegisterTransferLevel30
2.2.2HardwareDescriptionLanguages32
2.2.3High-levelSynthesis33
2.3FPGAsandImageProcessing38
2.3.1ChoosinganFPGAorDevelopmentBoard39 2.4Summary 40 References 41
3DesignProcess 45
3.1ProblemSpecification45
3.2AlgorithmDevelopment47
3.2.1AlgorithmDevelopmentProcess47
3.2.2AlgorithmStructure48
3.2.3FPGADevelopmentIssues51
3.3ArchitectureSelection51
3.3.1SystemArchitecture52
3.3.2PartitioningBetweenHardwareandSoftware53
3.3.3ComputationalArchitecture55
3.4SystemImplementation60
3.4.1MappingtoFPGAResources60
3.4.2AlgorithmMappingIssues62
3.5TestingandDebugging63
3.5.1Design63
3.5.2Implementation64
3.5.3CommonImplementationBugs64
3.5.4Timing66
3.5.5SystemDebugging68
3.5.6AlgorithmTuning70
3.5.7In-fieldDiagnosis71
3.6Summary 72 References 73
4DesignConstraints 77
4.1TimingConstraints77
4.1.1Low-levelPipelining78
4.1.2ProcessSynchronisation80
4.1.3SynchronisingBetweenClockDomains82
4.1.4I/OConstraints83
4.2MemoryBandwidthConstraints84
4.2.1MemoryArchitectures84
4.2.2Caching86
4.2.3RowBuffering87
4.3ResourceConstraints88
4.3.1Bit-serialComputation89
4.3.2ResourceMultiplexing89
4.3.3Arbitration92
4.3.4ResourceControllers94
4.3.5Reconfigurability95
4.4PowerConstraints97
4.5PerformanceMetrics98
4.5.1Speed 99
4.5.2Resources99
4.5.3Power99
4.5.4Cost 100
4.5.5ApplicationMetrics100
4.6Summary 101 References 102
6.3.5Ethernet150
6.3.6PCIExpress151
6.4Off-chipMemory151
6.4.1StaticRAM152
6.4.2DynamicRAM152
6.4.3FlashMemory155
6.5Processors 155
6.5.1AXIInterface155
6.5.2AvalonBus156
6.5.3OperatingSystems157
6.5.4ImplicationsforSystemDesign157
6.6Summary 157 References 158
7PointOperations 161
7.1PointOperationsonaSingleImage161
7.1.1ContrastandBrightnessAdjustment161
7.1.2GlobalThresholdingandContouring164
7.1.3LookupTableImplementation166
7.2PointOperationsonMultipleImages167
7.2.1ImageAveraging168
7.2.2ImageSubtraction170
7.2.3BackgroundModelling172
7.2.4IntensityScaling175
7.2.5Masking175
7.2.6HighDynamicRange(HDR)Imaging177
7.3Colour 179
7.3.1FalseColour179
7.3.2ColourSpaceConversion180
7.3.3ColourThresholding192
7.3.4ColourEnhancement193
7.3.5WhiteBalance194
7.4Multi-spectralandHyperspectralImaging197
7.4.1HyperspectralImageAcquisition197
7.4.2ProcessingSteps198
7.5Summary 199 References 199
8HistogramOperations203
8.1GreyscaleHistogram203
8.1.1BuildingtheHistogram203
8.1.2DataGathering205
8.1.3HistogramEqualisation209
8.1.4AutomaticExposure214
8.1.5ThresholdSelection215
8.1.6HistogramSimilarity220
8.2MultidimensionalHistograms220
8.2.1TriangularArrays221
8.2.2MultidimensionalStatistics222
8.2.3ColourSegmentation225
10.4ImageRegistration302
11.1DiscreteFourierTransform322
11.1.1FastFourierTransform(FFT)323 11.1.2Goertzel’sAlgorithm331 11.1.3Applications332
11.2DiscreteCosineTransform(DCT)336 11.3WaveletTransform338 11.3.1FilterImplementations340 11.3.2Applications344 11.4Summary
12ImageandVideoCoding349 12.1CompressionTechniques350 12.1.1ColourConversion350 12.1.2PredictionandTransformation350 12.1.3MotionEstimationandCompensation351 12.1.4Quantisation352 12.1.5Run-lengthCoding353 12.1.6EntropyCoding354
12.2DCT-basedCodecs357 12.2.1DCTBlockProcessing357
12.2.2JPEG357 12.2.3VideoCodecs358 12.3Wavelet-basedCodecs359 12.4LosslessCompression360 12.5PerceptualCoding361 12.6CodingHyperspectralImages362 12.7Summary 362 References
13BlobDetectionandLabelling367 13.1BoundingBox367 13.2Run-lengthCoding369 13.3ChainCoding369
13.3.1SequentialImplementation370 13.3.2Single-passStreamProcessingAlgorithms370 13.3.3FeatureExtraction372
13.4ConnectedComponentLabelling(CCL)374 13.4.1RandomAccessAlgorithms374 13.4.2MultiplePassAlgorithms374 13.4.3Two-passAlgorithms375 13.4.4ParallelAlgorithms377
13.4.5HysteresisThresholding377
13.5ConnectedComponentAnalysis(CCA)377
15.3Real-timeProduceGrading448 15.3.1SoftwareAlgorithm448
15.4.1Rectification454
15.4.2CalculatingtheDepth456
Preface
Imageprocessing,andinparticularembeddedimageprocessing,facesmanychallenges,fromincreasingresolution,increasingframerates,andtheneedtooperateatlowpower.Theseoffersignificantchallengesforimplementationonconventionalsoftware-basedplatforms.Thisleadsnaturallytoconsideringfield-programmable gatearrays(FPGAs)asanimplementationplatformforembeddedimagingapplications.Manyimageprocessingoperationsareinherentlyparallel,andFPGAsprovideprogrammablehardware,alsoinherentlyparallel. Therefore,itshouldbeassimpleasmappingoneontotheother,right?Well,yes andno.
Imageprocessingistraditionallythoughtofasasoftwaredomaintask,whereasFPGA-baseddesignisfirmly inthehardwaredomain.Therearealotoftricksandtechniquesrequiredtocreateanefficientdesign.Perhaps thebiggesthurdletoanefficientimplementationistheneedforahardwaremindset.Tobridgethegapbetween softwareandhardware,itisnecessarytothinkofalgorithmsnotontheirownbutmoreintermsoftheirunderlyingcomputationalarchitecture.Implementinganimageprocessingalgorithm(orindeedanyalgorithm)onan FPGAthereforeconsistsofdeterminingtheunderlyingarchitectureofanalgorithm,mappingthatarchitecture ontotheresourcesavailablewithinanFPGA,andfinallymappingthealgorithmontothehardwarearchitecture. Whilethemechanicsofthisprocessismostlyautomatedbyhigh-levelsynthesistools,theunderlyingdesign isnot.Alow-qualitydesigncanonlygosofar;itisstillimportanttokeepinmindthehardwarethatisbeing impliedbythecodeanddesignthealgorithmfortheunderlyinghardware.
Unfortunately,thereislimitedmaterialavailabletohelpthosenewtotheareatogetstarted.Whilethere aremanyresearchpaperspublishedinconferenceproceedingsandjournals,thereareonlyafewthatfocus specificallyonhowtomapimageprocessingalgorithmsontoFPGAs.Theresearchpapersfoundintheliterature canbeclassifiedintoseveralbroadgroups.
ThefirstfocusesontheFPGAarchitectureitself.Mostoftheseprovideananalysisofarangeoftechniques relatingtothestructureandgranularityoflogicblocks,theroutingnetworks,andembeddedmemories.As wellastheFPGAstructure,awiderangeoftopicsarecovered,includingunderlyingtechnology,powerissues, theeffectsofprocessvariability,anddynamicreconfigurability.Manyofthesepapersarepurelyproposals,or relatetoprototypeFPGAsratherthancommerciallyavailablechips.Whiletheyprovideinsightsastosomeof thefeatureswhichmightbeavailableinthenextgenerationofdevices,mostofthetopicswithinthisgroupare attoolowalevel.
Asecondgroupofpapersinvestigatesthetopicofreconfigurablecomputing.Here,thefocusisonhowan FPGAcanbeusedtoacceleratesomecomputationallyintensivetaskorrangeoftasks.Whileimageprocessing isonesuchtaskconsidered,mostoftheresearchrelatesmoretohigh-performancecomputingratherthan low-powerembeddedsystems.Topicswithinthisgroupincludehardwareandsoftwarepartitioning,hardware andsoftwareco-design,dynamicreconfigurability,communicationsbetweenanFPGAandcentralprocessing unit(CPU),comparisonsbetweentheperformanceofFPGAs,graphicsprocessingunits(GPUs)andCPUs, andthedesignofoperatingsystemsandspecificplatformsforbothreconfigurablecomputingapplicationsand research.Importantprinciplesandtechniquescanbegleanedfrommanyofthesepaperseventhoughthismay notbetheirprimaryfocus.
ThenextgroupofpapersconsiderstoolsforprogrammingFPGAsandapplications,withafocusonimprovingtheproductivityofthedevelopmentprocess.Awiderangeofhardwaredescriptionlanguageshavebeen proposed,withmanymodelledaftersoftwarelanguagessuchasC,Java,andevenProlog.Manyoftheseare developedasresearchtools,withveryfewmakingitoutofthelaboratorytocommercialavailability.Therehas
Againstthis,therehasbeenanincreasingawarenessofpowerandsustainabilityissues.Asalow-power computingplatform,FPGAsarewellplacedtoaddressthepowerconcernsinmanyapplications.
ThecapabilitiesofFPGAshaveimprovedsignificantlyastechnologyimprovementsenablemoretobe packedontothem.Notonlyhastherebeenanincreaseintheamountofprogrammablelogicandon-chipmemoryblocks,butFPGAsarebecomingmoreheterogeneous.ManyFPGAsnowincorporatesignificanthardened logicblocks,includingmoderatelypowerfulreducedinstructionsetcomputing(RISC)processors,external memoryinterfacing,andawiderangeofcommunicationinterfaces.Digitalsignalprocessing(DSP)blocks arealsoimproving,withthemovetowardssupportingfloating-pointinhigh-enddevices.Technologyimprovementshaveseensignificantreductionsinthepowerrequired.
EventheFPGAmarkethaschanged,withthetakeoverofbothAlteraandXilinxbyIntelandAMD,respectively.ThisisanindicationthatFPGAsareseenasaseriouscontenderforhigh-performancecomputingand acceleration.Thecompetitionhasnotstoodstill,withbothCPUsandGPUsincreasingincapability.Inparticular,anewgenerationoflow-powerGPUshasbecomeavailablethataremoreviableforembeddedimage processing.
High-levelsynthesistoolsarebecomingmorematureandaddressmanyofthedevelopmenttimeissues associatedwithconventionalregistertransferleveldesign.Theabilitytocompiletobothsoftwareandhardware enablesmorecomplexalgorithmstobeexplored,withfasterdebugging.Theyalsoallowfasterexplorationofthe designspace,enablingefficientdesignstobedevelopedmorereadily.However,theuseofhigh-levelsynthesis doesnoteliminatetheneedforcarefulalgorithmdesign.
WhiletheuseofFPGAsforimageprocessinghasnotbecomemainstream,therehasbeenalotofactivityin thisspaceasthecapabilitiesofFPGAshaveimproved.Theresearchliteratureonprogrammingandapplying FPGAsinthecontextofimageprocessinghasgrownsignificantly.However,itisstillquitediffuse,withmost papersfocusingononespecificaspect.Asresearchershavelookedatmorecompleximageprocessingoperations,thedescriptionsoftheimplementationhavebecomehigherlevel,requiringalotofreadingbetweenthe lines,andadditionaldesignworktobeabletoreplicateadesign.
Onesignificantareathathasbecomemainstreaminimageprocessingistheuseofdeeplearningmodels. Deeplearningwasnotaroundwhenthepreviouseditionwaswrittenandonlystartedbecomingsuccessfulin imageprocessingtasksintheearly2010s.Theirsuccesshasmadethemadrivingapplication,notonlyfor FPGAsandFPGAarchitecturebutalsowithincomputingingeneral.However,deeplearningmodelsposea hugecomputationaldemandonprocessing,especiallyfortraining,butalsofordeployment.Inanembedded visioncontext,thishasmadeFPGAsatargetplatformfortheirdeployment.Deeplearningisabigtopiconits own,sothisbookisunabletodomuchmorethanscratchthesurfaceandconcentrateonsomeoftheissues associatedwithFPGA-basedimplementation.
OutlineoftheContents
Thisbookaimstoprovideacomprehensiveoverviewofalgorithmsandtechniquesforimplementingimage processingalgorithmsonFPGAs,particularlyforlow-andintermediate-levelvision.However,aswithdesign inanyfield,thereismorethanonewayofachievingaparticulartask.Muchoftheemphasishasbeenplacedon stream-basedapproachestoimplementingimageprocessing,asthesecanefficientlyexploitparallelismwhen theycanbeused.Thisemphasisreflectsmybackgroundandexperienceintheareaandisnotintendedtobe thelastwordonthetopic.
AbroadoverviewofimageprocessingispresentedinChapter1,withabriefhistoricalcontext.Manyof thebasicimageprocessingtermsaredefined,andthedifferentstagesofanimageprocessingalgorithmare identified.Theproblemofreal-timeembeddedimageprocessingisintroduced,andthelimitationsofconventionalserialprocessorsfortacklingthisproblemareidentified.High-speedimageprocessingmustexploitthe parallelisminherentintheprocessingofimages;thedifferenttypesofparallelismareidentifiedandexplained. FPGAscombinetheadvantagesofbothhardwareandsoftwaresystems,byprovidingreprogrammable (henceflexible)hardware.Chapter2providesanintroductiontoFPGAtechnology.Whilesomeofthiswill bemoredetailedthanisnecessarytoimplementalgorithms,abasicknowledgeofthebuildingblocksand underlyingarchitectureisimportanttodevelopingresourceefficientsolutions.ThesynthesisprocessforbuildinghardwareonFPGAsisdefined,withparticularemphasisonthedesignflowforimplementingalgorithms.
ImagewarpingandrelatedtechniquesarecoveredinChapter10.Theforwardandreversemapping approachestogeometrictransformationarecomparedinsomedetail,withparticularemphasisontechniques forstreamprocessingimplementations.Interpolationisfrequentlyassociatedwithgeometrictransformation. Hardware-basedalgorithmsforbilinear,bicubic,andspline-basedinterpolationaredescribed.Related techniquesofimageregistrationarealsodescribedattheendofthischapter,includingadiscussionoffeature pointdetection,description,andmatching.
Chapter11introduceslineartransforms,withaparticularfocusonthefastFouriertransform(FFT),the discretecosinetransform(DCT),andthewavelettransform.Bothparallelandpipelinedimplementationsof theFFTandDCTaredescribed.Filteringandinversefilteringinthefrequencydomainarediscussedinsome detail.Lifting-basedfilteringisdevelopedforthewavelettransform.Thiscanreducethelogicrequirementsby uptoafactorof4overadirectfiniteimpulseresponseimplementation.
Imagecodingisimportantforimagestorageortransmission.Chapter12discussesthestageswithinimage andvideocodingandoutlinessomeofthetechniquesthatcanbeusedateachstage.Severalofthestandards forbothstillimageandvideocodingareoutlined,withanoverviewofthecompressiontechniquesused.
Aselectionofintermediate-leveloperationsrelatingtoregiondetectionandlabellingispresentedin Chapter13.Standardsoftwarealgorithmsforchaincodingandconnectedcomponentlabellingareadapted togiveefficientstreamedimplementation.Thesecansignificantlyreduceboththelatencyandmemory requirementsofanapplication.Hardwareimplementationsofthedistancetransform,thewatershedtransform, andtheHoughtransformarealsopresented,discussingsomeofthekeydesigndecisionsforanefficient implementation.
Machinelearningtechniquesarecommonlyusedwithincomputervision.Chapter14introducesthekey techniquesforregressionandclassification,withaparticularfocusonFPGAimplementation.Deeplearningtechniquesareincreasinglybeingusedinmanycomputervisionapplications.Arangeofdeepnetwork architecturesisintroduced,andsomeoftheissuesforrealisingtheseonFPGAsarediscussed.
Finally,Chapter15presentsaselectionofcasestudies,showinghowthematerialandtechniquesdescribed inthepreviouschapterscanbeintegratedwithinacompleteapplication.Theseapplicationsbrieflyshowthe designstepsandillustratethemappingprocessatthewholealgorithmlevelratherthanpurelyattheoperation level.Manygainscanbemadebycombiningoperationstogetherwithinacompatibleoverallarchitecture.The applicationsdescribedarecolouredregiontrackingforagesture-baseduserinterface,calibratingandcorrecting barreldistortioninlenses,developmentofafovealimagesensorinspiredbysomeoftheattributesofthehuman visualsystem,amachinevisionsystemforreal-timeproducegrading,stereoimagingfordepthestimation,and facedetection.
ConventionsUsed
ThecontentsofthisbookareindependentofanyparticularFPGAorFPGAvendor,oranyparticularhardwaredescriptionlanguage.Thetopicisalreadysufficientlyspecialisedwithoutnarrowingtheaudiencefurther! Asaresult,manyofthefunctionsandoperationsarerepresentedinblockschematicform.Thisenablesa language-independentrepresentationandplacesemphasisonaparticularhardwareimplementationofthealgorithminawaythatisportable.ThebasicelementsoftheseschematicsareillustratedinFigureP.1. I isgenerally usedastheinputofanimageprocessingoperation,withtheoutputimagerepresentedby Q
Withsomemathematicaloperations,suchassubtractionandcomparison,theorderoftheoperandsisimportant.Insuchcases,thefirstoperandisindicatedwithablobratherthananarrow,asshownonthebottomin FigureP.1.
Considerarecursivefilteroperatingonstreameddata:
wherethesubscriptinthisinstancereferstothe nthpixelinthestreamedimage.Atahighlevel,thiscanbe consideredasanimageprocessingoperation,andrepresentedbyasingleblock,asshowninthetop-leftof FigureP.1.Thelow-levelimplementationisgiveninthemiddle-leftpanel.Theinputandoutput, I and Q,are
1 ImageProcessing
Visionisarguablythemostimportanthumansense.Theprocessingandrecordingofvisualdatathereforehas significantimportance.Theearliestimagesarefromprehistoricdrawingsoncavewallsorcarvedonstone monumentscommonlyassociatedwithburialtombs.(Itisnotsomuchthemediumthatisimportanthere;anythingelsewouldnothavesurvivedtotoday.)Suchimagesconsistofamixtureofbothpictorialandabstract representations.Improvementsintechnologyenabledimagestoberecordedwithmorerealism,suchaspaintingsbythemasters.Imagesrecordedinthismannerareindirectinthesensethatthelightintensitypatternis notuseddirectlytoproducetheimage.Thedevelopmentofchemicalphotographyintheearly1800senabled directimagerecording.Thistrendhascontinuedwithelectronicrecording,firstwithanaloguesensors,and subsequentlywithdigitalsensors,whichincludeanaloguetodigital(A/D)conversiononthesensorchip,to directlyproducedigitalimages.
Imagingsensorshavenotbeenrestrictedtotheportionoftheelectromagneticspectrumvisibletothehuman eye.Sensorshavebeendevelopedtocovermuchoftheelectromagneticspectrumfromradiowavesthrough togammarays.Awidevarietyofotherimagingmodalitieshavealsobeendeveloped,basedonultrasound, electrons,atomicforce(Binnigetal.,1986),magneticresonance,andsoon.Inprinciple,anyquantitythatcan besensedcanbeusedforimaging,evendustrays(Auer,1982).
Sincevisionissuchanimportantsense,theprocessingofimageshasbecomeimportanttoo,toaugmentor enhancehumanvision.Imagescanbeprocessedtoenhancetheirsubjectivecontentortoextractusefulinformation.Whileitispossibletoprocesstheopticalsignalsassociatedwithvisualimagesusinglensesandoptical filters,thisbookfocusesondigitalimageprocessing,thenumericalprocessingofimagesbydigitalhardware.
Oneoftheearliestapplicationsofdigitalimageprocessingwasfortransmittingdigitisednewspaperpictures acrosstheAtlanticOceanintheearly1920s(McFarlane,1972).However,itwasonlywiththeadventofcomputerswithsufficientmemoryandprocessingpowerthatdigitalimageprocessingbecamewidespread.Theearliest recordedcomputer-basedimageprocessingwasfrom1957whenascannerwasaddedtoacomputerattheUSA NationalBureauofStandards(Kirsch,1998).Itwasusedfortheearlyresearchonedgeenhancementandpattern recognition.Inthe1960s,theneedtoprocesslargenumbersoflargeimagesobtainedfromsatellitesandspace explorationstimulatedimageprocessingresearchatNASA’sJetPropulsionLaboratory(Castleman,1979).At thesametime,researchinhigh-energyparticlephysicsrequireddetectinginterestingeventsfromlargenumbersofcloudchamberphotographs(Duff,2000).Ascomputersgrewinpowerandreducedincost,therange ofapplicationsfordigitalimageprocessingexploded,fromindustrialinspectiontomedicalimaging.Image sensorsarenowubiquitous,inmobilephones,laptops,andinvideo-basedsecurityandsurveillancesystems.
1.1BasicDefinitions
Moreformally,an image isaspatialrepresentationofanobject,scene,orotherphenomenon(Haralickand Shapiro,1991).Examplesofimagesincludeaphotograph,whichisapictorialrecordformedfromthelight
DesignforEmbeddedImageProcessingonFPGAs,SecondEdition.DonaldG.Bailey. ©2024JohnWiley&SonsLtd.Published2024byJohnWiley&SonsLtd. CompanionWebsite:www.wiley.com/go/bailey/designforembeddedimageprocessc2e
intensitypatternonanopticalsensor;aradiograph,whichisarepresentationofdensityformedthroughexposure toX-raystransmittedthroughanobject;amap,whichisaspatialrepresentationofphysicalorculturalfeatures; andavideo,whichisasequenceoftwo-dimensionalimagesthroughtime.Morerigorously,animageisany continuousfunctionoftwoormorevariablesdefinedonsomeboundedregionofspace.
A digitalimage isanimageindigitalformat,sothatitissuitableforprocessingbycomputer.Thereare twoimportantcharacteristicsofdigitalimages.Thefirstisspatialquantisation.Computersareunabletoeasily representarbitrarycontinuousfunctions,sothecontinuousfunctionissampled.Theresultisaseriesofdiscretepictureelements,or pixels (for2-Dimages)orvolumeelements, voxels (for3-Dimages).Samplingdoes notnecessarilyhavetobespatiallyuniform,forexample pointclouds fromaLiDARscanner.Samplingcan representacontinuousimageexactly(inthesensethattheunderlyingcontinuousfunctionmayberecovered exactly),givenaband-limitedimageandasufficientlyhighsamplerate(Shannon,1949).Thesecondcharacteristicofdigitalimagesissamplequantisation.Thisresultsindiscretevaluesforeachpixel,enablinganinteger representation.Commonbitwidthsperpixelare1(binaryimages),8(greyscaleimages),and24(3 × 8bitsfor colourimages).Modernhighdynamicrangesensorscanprovide12–16bitsperpixel.Unlikesampling,value quantisationwillalwaysresultinanerrorbetweentherepresentationandtruevalue.Inmanycircumstances, however,this quantisationerror or quantisationnoise maybemadesmallerthantheuncertaintyinthetrue valueresultingfrominevitablemeasurementnoise.
Initsbasicform,adigitalimageissimplyatwo(orhigher)-dimensionalarrayofnumbers(usuallyintegers),whichrepresentsanobjectorscene.Onceinthisform,animagemaybereadilymanipulatedbya digitalcomputer.Itdoesnotmatterwhatthenumbersrepresent,whetherlightintensity,reflectance,attenuation,distancetoapoint(range),temperature,populationdensity,elevation,rainfall,oranyothernumerical quantity.
Digitalimageprocessing canthereforebedefinedassubjectingsuchanimagetoaseriesofmathematical operationsinordertoobtainadesiredresult.Thismaybeanenhancedimage;thedetectionofsomecritical featureorevent;ameasurementofanobjectorkeyfeaturewithintheimage;aclassificationorgradingof objectswithintheimageintooneoftwoormorecategories;oradescriptionofthescene.
Imageprocessingtechniquesareusedinanumberofrelatedfields.Whiletheprincipalfocusofthefields oftendiffers,manyofthetechniquesremainthesameatthefundamentallevel.Someofthedistinctivecharacteristicsarebrieflyoutlinedhere.
Imageenhancement involvesimprovingthesubjectivequalityofanimageorthedetectabilityofobjectswithin theimage(HaralickandShapiro,1991).Theinformationthatisenhancedisusuallyapparentintheoriginal imagebutmaynotbeclear.Examplesofimageenhancementincludenoisereduction,contrastenhancement, edgesharpening,andcolourcorrection.
Imagerestoration goesonestepfurtherthanimageenhancement.Itusesknowledgeofthecausesofhowan imageisdegradedtocreateamodelofthedegradationprocess.Thismodelisthenusedtoderiveaninverse processthatisusedtorestoretheimage.Inmanycases,theinformationintheimagehasbeendegradedto theextentofbeingunrecognisable,forexamplesevereblurring.
Imagereconstruction involvesrestructuringthedatathatisavailableintoamoreusefulform.Examplesare imagesuper-resolution(reconstructingahigh-resolutionimagefromaseriesoflow-resolutionimages), imagefusion(combiningimagesfrommultiplesources),andtomography(reconstructingacrosssection ofanobjectfromaseriesofprojections).
Imageanalysis refersspecificallytousingcomputerstoextractdatafromimages.Theresultisusually someformofmeasurement.Inthepast,thiswasalmostexclusively2-Dimaging,althoughwiththe adventofconfocalmicroscopyandotheradvancedimagingtechniques,thishasextendedtothree dimensions.
Patternrecognition isconcernedwithidentifyingobjectsbasedonpatternsinthemeasurements(Haralickand Shapiro,1991).Thereisastrongfocusonstatisticalapproaches,althoughsyntacticandstructuralmethods arealsoused.
Computervision tendstouseamodel-basedapproachtoimageprocessing.Mathematicalmodelsofboththe sceneandtheimagingprocessareusedtoderivea3-Drepresentationbasedononeormore2-Dimagesof ascene.Theuseofmodelsimplicitlyprovidesaninterpretationofthecontentsoftheimagesobtained.
