Design Donald G. Bailey Visit to download the full and correct content document: https://ebookmass.com/product/design-for-embedded-image-processing-on-fpgas-2n d-edition-donald-g-bailey/
More products digital (pdf, epub, mobi) instant download maybe you interests ...
Design Recipes for FPGAs using Verilog and VHDL [2nd ed.] Peter Wilson
https://ebookmass.com/product/design-recipes-for-fpgas-usingverilog-and-vhdl-2nd-ed-peter-wilson/
Feature extraction and image processing for computer vision Fourth Edition Aguado
https://ebookmass.com/product/feature-extraction-and-imageprocessing-for-computer-vision-fourth-edition-aguado/
Graph Spectral Image Processing Gene Cheung
https://ebookmass.com/product/graph-spectral-image-processinggene-cheung/
Machine Learning Algorithms for Signal and Image Processing Suman Lata Tripathi
https://ebookmass.com/product/machine-learning-algorithms-forsignal-and-image-processing-suman-lata-tripathi/
Image Processing for Automated Diagnosis of Cardiac Diseases Rajeev Kumar Chauhan (Editor)
https://ebookmass.com/product/image-processing-for-automateddiagnosis-of-cardiac-diseases-rajeev-kumar-chauhan-editor/
Digital Image Processing Using MATLAB Rafael C. Gonzalez
https://ebookmass.com/product/digital-image-processing-usingmatlab-rafael-c-gonzalez/
Engineering Economic Analysis 14th Edition Donald G. Newnan
https://ebookmass.com/product/engineering-economic-analysis-14thedition-donald-g-newnan/
Signal processing for neuroscientists 2nd ed Edition Drongelen
https://ebookmass.com/product/signal-processing-forneuroscientists-2nd-ed-edition-drongelen/
Methods and techniques for fire detection : signal, image and video processing perspectives 1st Edition Çetin
https://ebookmass.com/product/methods-and-techniques-for-firedetection-signal-image-and-video-processing-perspectives-1stedition-cetin/
DESIGNFOREMBEDDED IMAGEPROCESSING ONFPGAs DESIGNFOREMBEDDED IMAGEPROCESSING ONFPGAs SecondEdition DonaldG.Bailey
MasseyUniversity,PalmerstonNorth,NewZealand
Thissecondeditionfirstpublished2024 ©2024JohnWiley&Sons,Ltd
EditionHistory
JohnWiley&Sons(Asia)PteLtd(1e,2011)
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmitted,inanyformorbyany means,electronic,mechanical,photocopying,recordingorotherwise,exceptaspermittedbylaw.Adviceonhowtoobtain permissiontoreusematerialfromthistitleisavailableat http://www.wiley.com/go/permissions
TherightofDonaldG.Baileytobeidentifiedastheauthorofthisworkhasbeenassertedinaccordancewithlaw.
RegisteredOffice(s)
JohnWiley&Sons,Inc.,111RiverStreet,Hoboken,NJ07030,USA
JohnWiley&SonsLtd,TheAtrium,SouthernGate,Chichester,WestSussex,PO198SQ,UK
Fordetailsofourglobaleditorialoffices,customerservices,andmoreinformationaboutWileyproductsvisitusat www.wiley.com
Wileyalsopublishesitsbooksinavarietyofelectronicformatsandbyprint-on-demand.Somecontentthatappearsinstandard printversionsofthisbookmaynotbeavailableinotherformats.
Trademarks
WileyandtheWileylogoaretrademarksorregisteredtrademarksofJohnWiley&Sons,Inc.and/oritsaffiliatesintheUnited Statesandothercountriesandmaynotbeusedwithoutwrittenpermission.Allothertrademarksarethepropertyoftheirrespective owners.JohnWiley&Sons,Inc.isnotassociatedwithanyproductorvendormentionedinthisbook.
LimitofLiability/DisclaimerofWarranty
Whilethepublisherandauthorshaveusedtheirbesteffortsinpreparingthiswork,theymakenorepresentationsorwarrantieswith respecttotheaccuracyorcompletenessofthecontentsofthisworkandspecificallydisclaimallwarranties,includingwithout limitationanyimpliedwarrantiesofmerchantabilityorfitnessforaparticularpurpose.Nowarrantymaybecreatedorextendedby salesrepresentatives,writtensalesmaterialsorpromotionalstatementsforthiswork.Thisworkissoldwiththeunderstandingthat thepublisherisnotengagedinrenderingprofessionalservices.Theadviceandstrategiescontainedhereinmaynotbesuitablefor yoursituation.Youshouldconsultwithaspecialistwhereappropriate.Thefactthatanorganization,website,orproductisreferred tointhisworkasacitationand/orpotentialsourceoffurtherinformationdoesnotmeanthatthepublisherandauthorsendorsethe informationorservicestheorganization,website,orproductmayprovideorrecommendationsitmaymake.Further,readersshould beawarethatwebsiteslistedinthisworkmayhavechangedordisappearedbetweenwhenthisworkwaswrittenandwhenitisread. Neitherthepublishernorauthorsshallbeliableforanylossofprofitoranyothercommercialdamages,includingbutnotlimitedto special,incidental,consequential,orotherdamages.
LibraryofCongressCataloging-in-PublicationData
Names:Bailey,DonaldG.(DonaldGraeme),1962-author.
Title:DesignforembeddedimageprocessingonFPGAs/DonaldG.Bailey.
Description:Secondedition.|Hoboken,NJ:Wiley,2024.|Includesindex.
Identifiers:LCCN2023011469(print)|LCCN2023011470(ebook)|ISBN 9781119819790(cloth)|ISBN9781119819806(adobepdf)|ISBN 9781119819813(epub)
Subjects:LCSH:Embeddedcomputersystems.|Fieldprogrammablegate arrays.
Classification:LCCTK7895.E42B32642024(print)|LCCTK7895.E42 (ebook)|DDC621.39/5–dc23/eng/20230321
LCrecordavailableathttps://lccn.loc.gov/2023011469
LCebookrecordavailableathttps://lccn.loc.gov/2023011470
CoverDesign:Wiley
CoverImage:©bestfoto77/Shutterstock
Setin9/11ptTimesLTStdbyStraive,Chennai,India
1.2.2Colour 5
1.3ImageProcessingOperations6
1.4Real-timeImageProcessing8 1.5EmbeddedImageProcessing9
1.6ComputerArchitecture10
1.7Parallelism 11
1.7.1TemporalorTaskParallelism12
1.7.2SpatialorDataParallelism13
1.7.3LogicalParallelism14
2Field-programmableGateArrays19
2.1HardwareArchitectureofFPGAs19
2.1.1Logic 21
2.1.2DSPBlocks22
2.1.3Memory23
2.1.4EmbeddedCPU23
2.1.5Interconnect24
2.1.6InputandOutput24
2.1.7Clocking26
2.1.8Configuration26
2.1.9FPGAsvs.ASICs27
2.2ProgrammingFPGAs28
2.2.1RegisterTransferLevel30
2.2.2HardwareDescriptionLanguages32
2.2.3High-levelSynthesis33
2.3FPGAsandImageProcessing38
2.3.1ChoosinganFPGAorDevelopmentBoard39 2.4Summary 40 References 41
3DesignProcess 45
3.1ProblemSpecification45
3.2AlgorithmDevelopment47
3.2.1AlgorithmDevelopmentProcess47
3.2.2AlgorithmStructure48
3.2.3FPGADevelopmentIssues51
3.3ArchitectureSelection51
3.3.1SystemArchitecture52
3.3.2PartitioningBetweenHardwareandSoftware53
3.3.3ComputationalArchitecture55
3.4SystemImplementation60
3.4.1MappingtoFPGAResources60
3.4.2AlgorithmMappingIssues62
3.5TestingandDebugging63
3.5.1Design63
3.5.2Implementation64
3.5.3CommonImplementationBugs64
3.5.4Timing66
3.5.5SystemDebugging68
3.5.6AlgorithmTuning70
3.5.7In-fieldDiagnosis71
3.6Summary 72 References 73
4DesignConstraints 77
4.1TimingConstraints77
4.1.1Low-levelPipelining78
4.1.2ProcessSynchronisation80
4.1.3SynchronisingBetweenClockDomains82
4.1.4I/OConstraints83
4.2MemoryBandwidthConstraints84
4.2.1MemoryArchitectures84
4.2.2Caching86
4.2.3RowBuffering87
4.3ResourceConstraints88
4.3.1Bit-serialComputation89
4.3.2ResourceMultiplexing89
4.3.3Arbitration92
4.3.4ResourceControllers94
4.3.5Reconfigurability95
4.4PowerConstraints97
4.5PerformanceMetrics98
4.5.1Speed 99
4.5.2Resources99
4.5.3Power99
4.5.4Cost 100
4.5.5ApplicationMetrics100
4.6Summary 101 References 102
5ComputationalTechniques105
5.1NumberSystems105
5.1.1BinaryIntegers105
5.1.2ResidueSystems106
5.1.3RedundantRepresentations107
5.1.4Fixed-pointNumbers107
5.1.5Floating-pointNumbers108
5.1.6LogarithmicNumberSystem110
5.1.7PositNumbers110
5.2ElementaryFunctions111
5.2.1SquareRoot111
5.2.2TrigonometricFunctions112
5.2.3LinearCORDIC116
5.2.4HyperbolicFunctions117
5.2.5LogarithmsandExponentials118
5.2.6LookupTables118
5.2.7PolynomialApproximations122
5.2.8IterativeTechniques123
5.3OtherComputationTechniques124
5.3.1IncrementalUpdate124
5.3.2Separability124
5.4MemoryStructures124
5.4.1FIFOBuffer124
5.4.2ZigzagBuffers126
5.4.3Stacks126
5.4.4LinkedLists127
5.4.5Trees128
5.4.6Graphs129
5.4.7HashTables129
6.1.1AnalogueVideo136
6.1.2DirectDigitalInterface137
6.1.3MIPICameraSerialInterface138
6.1.4CameraLink139
6.1.5USBCameras139
6.1.6GigEVision139
6.1.7CameraProcessingPipeline140
6.2DisplayOutput143
6.2.1DisplayDriver143
6.2.2DisplayContent146
6.3SerialCommunication147
6.3.1RS-232147
6.3.2I2 C 148
6.3.3SerialPeripheralInterface(SPI)149
6.3.4UniversalSerialBus(USB)150
6.3.5Ethernet150
6.3.6PCIExpress151
6.4Off-chipMemory151
6.4.1StaticRAM152
6.4.2DynamicRAM152
6.4.3FlashMemory155
6.5Processors 155
6.5.1AXIInterface155
6.5.2AvalonBus156
6.5.3OperatingSystems157
6.5.4ImplicationsforSystemDesign157
6.6Summary 157 References 158
7PointOperations 161
7.1PointOperationsonaSingleImage161
7.1.1ContrastandBrightnessAdjustment161
7.1.2GlobalThresholdingandContouring164
7.1.3LookupTableImplementation166
7.2PointOperationsonMultipleImages167
7.2.1ImageAveraging168
7.2.2ImageSubtraction170
7.2.3BackgroundModelling172
7.2.4IntensityScaling175
7.2.5Masking175
7.2.6HighDynamicRange(HDR)Imaging177
7.3Colour 179
7.3.1FalseColour179
7.3.2ColourSpaceConversion180
7.3.3ColourThresholding192
7.3.4ColourEnhancement193
7.3.5WhiteBalance194
7.4Multi-spectralandHyperspectralImaging197
7.4.1HyperspectralImageAcquisition197
7.4.2ProcessingSteps198
7.5Summary 199 References 199
8HistogramOperations203
8.1GreyscaleHistogram203
8.1.1BuildingtheHistogram203
8.1.2DataGathering205
8.1.3HistogramEqualisation209
8.1.4AutomaticExposure214
8.1.5ThresholdSelection215
8.1.6HistogramSimilarity220
8.2MultidimensionalHistograms220
8.2.1TriangularArrays221
8.2.2MultidimensionalStatistics222
8.2.3ColourSegmentation225
8.2.4ColourIndexing228
8.2.5TextureAnalysis229 8.3Summary 231 References 231
9LocalFilters 235
9.1WindowCaching235
9.1.1BorderHandling237
9.1.2FilterLatency239
9.2LinearFilters 239
9.2.1FilterTechniques240
9.2.2NoiseSmoothing243
9.2.3EdgeDetection246
9.2.4EdgeEnhancement248
9.3NonlinearFilters249
9.3.1GradientMagnitude249
9.3.2EdgeOrientation250
9.3.3PeakDetectionandNon-maximalSuppression251
9.3.4Zero-crossingDetection252
9.3.5BilateralFilter252
9.3.6AdaptiveThresholding253
9.3.7HighDynamicRangeToneMapping255
9.4RankFilters 256
9.4.1SortingNetworks258
9.5AdaptiveHistogramEqualisation262
9.6MorphologicalFilters262
9.6.1BinaryMorphology262
9.6.2GreyscaleMorphology266 9.7ColourFiltering268
9.7.1ColourMorphologyandVectorMedian269
9.7.2EdgeEnhancement269
9.7.3BayerPatternDemosaicing271
9.7.4WhiteBalancing272 9.8Summary 273 References 274
10GeometricTransformations281
10.1ReverseMapping282
10.1.1Anti-aliasFiltering283
10.1.2Interpolation284 10.2ForwardMapping291
10.2.1SeparableMapping292
10.2.2HybridApproach296 10.3CommonMappings297
10.3.1AffineTransformation297
10.3.2PerspectiveMapping297
10.3.3PolynomialMapping298
10.3.4LensDistortion299
10.3.5Non-parametricMappings302
10.4ImageRegistration302
11.1DiscreteFourierTransform322
11.1.1FastFourierTransform(FFT)323 11.1.2Goertzel’sAlgorithm331 11.1.3Applications332
11.2DiscreteCosineTransform(DCT)336 11.3WaveletTransform338 11.3.1FilterImplementations340 11.3.2Applications344 11.4Summary
12ImageandVideoCoding349 12.1CompressionTechniques350 12.1.1ColourConversion350 12.1.2PredictionandTransformation350 12.1.3MotionEstimationandCompensation351 12.1.4Quantisation352 12.1.5Run-lengthCoding353 12.1.6EntropyCoding354
12.2DCT-basedCodecs357 12.2.1DCTBlockProcessing357
12.2.2JPEG357 12.2.3VideoCodecs358 12.3Wavelet-basedCodecs359 12.4LosslessCompression360 12.5PerceptualCoding361 12.6CodingHyperspectralImages362 12.7Summary 362 References
13BlobDetectionandLabelling367 13.1BoundingBox367 13.2Run-lengthCoding369 13.3ChainCoding369
13.3.1SequentialImplementation370 13.3.2Single-passStreamProcessingAlgorithms370 13.3.3FeatureExtraction372
13.4ConnectedComponentLabelling(CCL)374 13.4.1RandomAccessAlgorithms374 13.4.2MultiplePassAlgorithms374 13.4.3Two-passAlgorithms375 13.4.4ParallelAlgorithms377
13.4.5HysteresisThresholding377
13.5ConnectedComponentAnalysis(CCA)377
13.5.1BasicSingle-passAlgorithm378
13.5.2ReducingMemoryRequirements379
13.5.3EliminatingEnd-of-rowOverheads379
13.5.4ParallelAlgorithms380
13.5.5FurtherConsiderationsandOptimisations381 13.6DistanceTransform381
13.6.1MorphologicalApproaches381
13.6.2ChamferDistance382
13.6.3EuclideanDistance384
13.6.4Applications386
13.6.5GeodesicDistanceTransform386 13.7WatershedTransform387
13.7.1FlowAlgorithms388 13.7.2ImmersionAlgorithms389 13.8HoughTransform391
13.8.1LineHoughTransform391 13.8.2CircleHoughTransform394
13.8.3GeneralisedHoughTransform395 13.9Summary 396 References 396
14MachineLearning 403 14.1Training 403
14.1.1LossandCostFunctions404 14.1.2ModelOptimisation405 14.1.3Considerations406 14.2Regression 409
14.2.1LinearRegression409 14.2.2NonlinearRegression409 14.2.3NeuralNetworks409 14.3Classification 411
14.3.1DecisionTrees411
14.3.2RandomForests412
14.3.3BayesianClassification412
14.3.4QuadraticDiscriminantAnalysis414
14.3.5LinearDiscriminantAnalysis414
14.3.6SupportVectorMachines415
14.3.7NeuralNetworks416
14.3.8Clustering417 14.4DeepLearning418
14.4.1BuildingBlocks419 14.4.2ArchitecturesandApplications421 14.4.3Training427
14.4.4ImplementationIssues428 14.5Summary 433 References 433
15ExampleApplications441 15.1ColouredRegionTracking441 15.2FovealSensor443
15.2.1FovealMapping444 15.2.2UsingtheSensor447
15.3Real-timeProduceGrading448 15.3.1SoftwareAlgorithm448
15.4.1Rectification454
15.4.2CalculatingtheDepth456
Preface Imageprocessing,andinparticularembeddedimageprocessing,facesmanychallenges,fromincreasingresolution,increasingframerates,andtheneedtooperateatlowpower.Theseoffersignificantchallengesforimplementationonconventionalsoftware-basedplatforms.Thisleadsnaturallytoconsideringfield-programmable gatearrays(FPGAs)asanimplementationplatformforembeddedimagingapplications.Manyimageprocessingoperationsareinherentlyparallel,andFPGAsprovideprogrammablehardware,alsoinherentlyparallel. Therefore,itshouldbeassimpleasmappingoneontotheother,right?Well,yes andno.
Imageprocessingistraditionallythoughtofasasoftwaredomaintask,whereasFPGA-baseddesignisfirmly inthehardwaredomain.Therearealotoftricksandtechniquesrequiredtocreateanefficientdesign.Perhaps thebiggesthurdletoanefficientimplementationistheneedforahardwaremindset.Tobridgethegapbetween softwareandhardware,itisnecessarytothinkofalgorithmsnotontheirownbutmoreintermsoftheirunderlyingcomputationalarchitecture.Implementinganimageprocessingalgorithm(orindeedanyalgorithm)onan FPGAthereforeconsistsofdeterminingtheunderlyingarchitectureofanalgorithm,mappingthatarchitecture ontotheresourcesavailablewithinanFPGA,andfinallymappingthealgorithmontothehardwarearchitecture. Whilethemechanicsofthisprocessismostlyautomatedbyhigh-levelsynthesistools,theunderlyingdesign isnot.Alow-qualitydesigncanonlygosofar;itisstillimportanttokeepinmindthehardwarethatisbeing impliedbythecodeanddesignthealgorithmfortheunderlyinghardware.
Unfortunately,thereislimitedmaterialavailabletohelpthosenewtotheareatogetstarted.Whilethere aremanyresearchpaperspublishedinconferenceproceedingsandjournals,thereareonlyafewthatfocus specificallyonhowtomapimageprocessingalgorithmsontoFPGAs.Theresearchpapersfoundintheliterature canbeclassifiedintoseveralbroadgroups.
ThefirstfocusesontheFPGAarchitectureitself.Mostoftheseprovideananalysisofarangeoftechniques relatingtothestructureandgranularityoflogicblocks,theroutingnetworks,andembeddedmemories.As wellastheFPGAstructure,awiderangeoftopicsarecovered,includingunderlyingtechnology,powerissues, theeffectsofprocessvariability,anddynamicreconfigurability.Manyofthesepapersarepurelyproposals,or relatetoprototypeFPGAsratherthancommerciallyavailablechips.Whiletheyprovideinsightsastosomeof thefeatureswhichmightbeavailableinthenextgenerationofdevices,mostofthetopicswithinthisgroupare attoolowalevel.
Asecondgroupofpapersinvestigatesthetopicofreconfigurablecomputing.Here,thefocusisonhowan FPGAcanbeusedtoacceleratesomecomputationallyintensivetaskorrangeoftasks.Whileimageprocessing isonesuchtaskconsidered,mostoftheresearchrelatesmoretohigh-performancecomputingratherthan low-powerembeddedsystems.Topicswithinthisgroupincludehardwareandsoftwarepartitioning,hardware andsoftwareco-design,dynamicreconfigurability,communicationsbetweenanFPGAandcentralprocessing unit(CPU),comparisonsbetweentheperformanceofFPGAs,graphicsprocessingunits(GPUs)andCPUs, andthedesignofoperatingsystemsandspecificplatformsforbothreconfigurablecomputingapplicationsand research.Importantprinciplesandtechniquescanbegleanedfrommanyofthesepaperseventhoughthismay notbetheirprimaryfocus.
ThenextgroupofpapersconsiderstoolsforprogrammingFPGAsandapplications,withafocusonimprovingtheproductivityofthedevelopmentprocess.Awiderangeofhardwaredescriptionlanguageshavebeen proposed,withmanymodelledaftersoftwarelanguagessuchasC,Java,andevenProlog.Manyoftheseare developedasresearchtools,withveryfewmakingitoutofthelaboratorytocommercialavailability.Therehas
alsobeenconsiderableresearchoncompilationtechniquesformappingstandardsoftwarelanguagestohardware(high-levelsynthesis).Techniquessuchasloopunrolling,stripmining,andpipeliningtoproduceparallel hardwareareimportantprinciplesthatcanresultinmoreefficienthardwaredesigns.
Thefinalgroupofpapersfocusesonarangeofapplications,includingimageprocessingandtheimplementationofbothimageprocessingoperationsandsystems.Unfortunately,asaresultofpagelimitsandspace constraints,manyofthesepapersgivetheresultsoftheimplementationofvarioussystemsbutpresentrelativelyfewdesigndetails.Thisisespeciallysointhecaseofmanypapersthatdescribedeeplearningsystems. Oftenthefinalproductisdescribed,withoutdescribingmanyofthereasonsordecisionsthatledtothatdesign. Manyofthesedesignscannotberecreatedwithoutacquiringthespecificplatformandtoolsthatwereusedor inferringalotofthemissingdetails.Whilesomeofthesedetailsmayappearobviousinhindsight,withoutthis knowledge,manyarefarfromobviousjustfromreadingthepapers.Thebetterpapersinthisgrouptendedto haveatighterfocus,consideringtheimplementationofasingleimageprocessingoperation.
So,whiletheremaybeareasonableamountofmaterialavailable,itisquitediffuse.Inmanycases,itis necessarytoknowexactlywhatyouarelookingfor,orjustbeluckytofindit.Theintentionofthisbook, therefore,istobringtogethermuchofthisdiverseresearch(onbothFPGAdesignandimageprocessing)and presentitinasystematicwayasareferenceorguide.
IntendedAudience Thisbookiswrittenprimarilyforthosewhoarefamiliarwiththebasicsofimageprocessingandwantto considerimplementingimageprocessingusingFPGAs.Perhapsthebiggesthurdleisswitchingfromasoftware mindsettoahardwarewayofthinking.Whenweprograminsoftware,agoodcompilercanmapthealgorithmin theprogramminglanguageontotheunderlyingcomputerarchitecturerelativelyefficiently.Whenprogramming hardwarethough,itisnotsimplyamatterofportingthesoftwareontohardware.Theunderlyinghardware architectureneedstobedesignedaswell.Inparticular,programminghardwareusuallyrequirestransforming thealgorithmintoanappropriateparallelarchitecture,oftenwithsignificantchangestothealgorithmitself.This requiressignificantdesign,ratherthanjustdecompositionandmappingofthedataflow(asisaccomplishedby agoodhigh-levelsynthesistool).Thisbookaddressesthisissuebynotonlyprovidingalgorithmsforimage processingoperationsbutalsodiscussesboththedesignprocessandtheunderlyingarchitecturesthatcanbe usedtoimplementthealgorithmsefficiently.
Thisbookwouldalsobeusefultothosewithahardwarebackground,whoarefamiliarwithprogramming andapplyingFPGAstootherproblems,andareconsideringimageprocessingapplications.Whilemanyofthe techniquesarerelevantandapplicabletoawiderangeofapplicationareas,mostofthefocusandexamplesare takenfromimageprocessing.Sufficientdetailisgiventomakemanyofthealgorithmsandtheirimplementation clear.However,learningimageprocessingismorethanjustcollectingasetofalgorithms,andthereareany numberofexcellentimageprocessingtextbooksthatprovidethese.
ItisthedomainofembeddedimageprocessingwhereFPGAscomeintotheirown.Anefficient,low-power designrequiresthatthetechniquesofboththehardwareengineerandthesoftwareengineerbeintegratedtightly withinthefinalsolution.
ChangesSincetheFirstEdition Althoughmanyoftheunderlyingdesignprincipleshavenotchanged,theenvironmenthaschangedquitesignificantlysincethefirstedition.Ingeneral,therehasbeenanincreaseintherequirementswithinapplications. Resolutionsareincreasing,withhigh-definitiontelevision(HDTV)becomingubiquitous,andgrowingdemand within4Kand8Kresolutions(the8Kformathas33.2Mpixels).Sensorresolutionsarealsoincreasingsteadily, withsensorshavingmorethan100Mpixelsbecomingavailable.Framerateshavealsobeenincreasing,withup to120framespersecondbeingpartofthestandardforultra-high-definitiontelevision(UHDTV).Thedynamic rangeisalsoincreasingfromcommonlyused8bitsperpixelto12–16bits.Allofthesefactorsleadtomore datatobeprocessed,atafasterrate.
Againstthis,therehasbeenanincreasingawarenessofpowerandsustainabilityissues.Asalow-power computingplatform,FPGAsarewellplacedtoaddressthepowerconcernsinmanyapplications.
ThecapabilitiesofFPGAshaveimprovedsignificantlyastechnologyimprovementsenablemoretobe packedontothem.Notonlyhastherebeenanincreaseintheamountofprogrammablelogicandon-chipmemoryblocks,butFPGAsarebecomingmoreheterogeneous.ManyFPGAsnowincorporatesignificanthardened logicblocks,includingmoderatelypowerfulreducedinstructionsetcomputing(RISC)processors,external memoryinterfacing,andawiderangeofcommunicationinterfaces.Digitalsignalprocessing(DSP)blocks arealsoimproving,withthemovetowardssupportingfloating-pointinhigh-enddevices.Technologyimprovementshaveseensignificantreductionsinthepowerrequired.
EventheFPGAmarkethaschanged,withthetakeoverofbothAlteraandXilinxbyIntelandAMD,respectively.ThisisanindicationthatFPGAsareseenasaseriouscontenderforhigh-performancecomputingand acceleration.Thecompetitionhasnotstoodstill,withbothCPUsandGPUsincreasingincapability.Inparticular,anewgenerationoflow-powerGPUshasbecomeavailablethataremoreviableforembeddedimage processing.
High-levelsynthesistoolsarebecomingmorematureandaddressmanyofthedevelopmenttimeissues associatedwithconventionalregistertransferleveldesign.Theabilitytocompiletobothsoftwareandhardware enablesmorecomplexalgorithmstobeexplored,withfasterdebugging.Theyalsoallowfasterexplorationofthe designspace,enablingefficientdesignstobedevelopedmorereadily.However,theuseofhigh-levelsynthesis doesnoteliminatetheneedforcarefulalgorithmdesign.
WhiletheuseofFPGAsforimageprocessinghasnotbecomemainstream,therehasbeenalotofactivityin thisspaceasthecapabilitiesofFPGAshaveimproved.Theresearchliteratureonprogrammingandapplying FPGAsinthecontextofimageprocessinghasgrownsignificantly.However,itisstillquitediffuse,withmost papersfocusingononespecificaspect.Asresearchershavelookedatmorecompleximageprocessingoperations,thedescriptionsoftheimplementationhavebecomehigherlevel,requiringalotofreadingbetweenthe lines,andadditionaldesignworktobeabletoreplicateadesign.
Onesignificantareathathasbecomemainstreaminimageprocessingistheuseofdeeplearningmodels. Deeplearningwasnotaroundwhenthepreviouseditionwaswrittenandonlystartedbecomingsuccessfulin imageprocessingtasksintheearly2010s.Theirsuccesshasmadethemadrivingapplication,notonlyfor FPGAsandFPGAarchitecturebutalsowithincomputingingeneral.However,deeplearningmodelsposea hugecomputationaldemandonprocessing,especiallyfortraining,butalsofordeployment.Inanembedded visioncontext,thishasmadeFPGAsatargetplatformfortheirdeployment.Deeplearningisabigtopiconits own,sothisbookisunabletodomuchmorethanscratchthesurfaceandconcentrateonsomeoftheissues associatedwithFPGA-basedimplementation.
OutlineoftheContents Thisbookaimstoprovideacomprehensiveoverviewofalgorithmsandtechniquesforimplementingimage processingalgorithmsonFPGAs,particularlyforlow-andintermediate-levelvision.However,aswithdesign inanyfield,thereismorethanonewayofachievingaparticulartask.Muchoftheemphasishasbeenplacedon stream-basedapproachestoimplementingimageprocessing,asthesecanefficientlyexploitparallelismwhen theycanbeused.Thisemphasisreflectsmybackgroundandexperienceintheareaandisnotintendedtobe thelastwordonthetopic.
AbroadoverviewofimageprocessingispresentedinChapter1,withabriefhistoricalcontext.Manyof thebasicimageprocessingtermsaredefined,andthedifferentstagesofanimageprocessingalgorithmare identified.Theproblemofreal-timeembeddedimageprocessingisintroduced,andthelimitationsofconventionalserialprocessorsfortacklingthisproblemareidentified.High-speedimageprocessingmustexploitthe parallelisminherentintheprocessingofimages;thedifferenttypesofparallelismareidentifiedandexplained. FPGAscombinetheadvantagesofbothhardwareandsoftwaresystems,byprovidingreprogrammable (henceflexible)hardware.Chapter2providesanintroductiontoFPGAtechnology.Whilesomeofthiswill bemoredetailedthanisnecessarytoimplementalgorithms,abasicknowledgeofthebuildingblocksand underlyingarchitectureisimportanttodevelopingresourceefficientsolutions.ThesynthesisprocessforbuildinghardwareonFPGAsisdefined,withparticularemphasisonthedesignflowforimplementingalgorithms.
Traditionalhardwaredescriptionlanguagesarecomparedwithhigh-levelsynthesis,withthebenefitsandlimitationsofeachoutlinedinthecontextofimageprocessing.
TheprocessofdesigningandimplementinganimageprocessingapplicationonanFPGAisdescribedin detailinChapter3.ParticularemphasisisgiventothedifferencesbetweendesigningforanFPGA-based implementationandastandardsoftwareimplementation.Thecriticalinitialstepistoclearlydefinetheimage processingproblemthatisbeingtackled.Thismustbeinsufficientdetailtoprovideaspecificationthatmay beusedtoevaluatethesolution.Theprocedurefordevelopingtheimageprocessingalgorithmisdescribedin detail,outliningthecommonstageswithinmanyimageprocessingalgorithms.Theresultingalgorithmmust thenbeusedtodefinethesystemandcomputationalarchitectures.Themappingfromanalgorithmismore thansimplyportingthealgorithmtoahardwaredescriptionlanguage.Itisnecessarytotransformthealgorithm tomakeefficientuseoftheresourcesavailableontheFPGA.Thefinalstageistoimplementthealgorithm bymappingitontothecomputationalarchitecture.Severalchecklistsprovideaguideandhintsfortestingand debugginganalgorithmonanFPGA.
Fourtypesofconstraintsonthemappingprocessarelimitedprocessingtime,limitedaccesstodata,limitedsystemresources,andlimitedsystempower.Chapter4describesseveraltechniquesforovercomingor alleviatingtheseconstraints.Timingexploreslow-levelpipelining,processsynchronisation,andworkingwith multipleclockdomains.Arangeofmemoryandcachingarchitecturesarepresentedforalleviatingmemory bandwidth.Resourcesharingandassociatedarbitrationissuesarediscussed,alongwithreconfigurability.The chapterfinisheswithasectionintroducingcommonlyusedperformancemetricsintermsofbothsystemand applicationperformance.
Chapter5focusesonthecomputationalaspectsofimageprocessingdesigns.Thesehelptobridgethegap betweenasoftwareandhardwareimplementation.Differentnumberrepresentationandnumbersystemsare described.Techniquesforthecomputationofelementaryfunctionsarediscussed,withaparticularfocuson thosethatarehardwarefriendly.Manyofthesecouldbeconsideredthehardwareequivalentofsoftwarelibraries forefficientlyimplementingcommonfunctions.PossibleFPGAimplementationsofarangeofdatastructures commonlyfoundincomputervisionalgorithmsarepresented.
Anyembeddedapplicationmustinterfacewiththerealworld.Arangeofcommonperipheralsisdescribed inChapter6,withsuggestionsonhowtheymaybeinterfacedtoanFPGA.Particularattentionisgivento interfacingcamerasandvideooutputdevices.Interfacingwithotherdevicesisdiscussed,includingserialcommunications,off-chipmemory,andserialprocessors.
Thenextsectionofthisbookdescribestheimplementationofmanycommonimageprocessingoperations. SomeofthedesigndecisionsandalternativewaysofmappingtheoperationsontoFPGAsareconsidered.While reasonablycomprehensive,particularlyforlow-levelimage-to-imagetransformations,itisimpossibletocover everypossibledesign.Theexamplesdiscussedareintendedtoprovidethefoundationformanyotherrelated operations.
Chapter7considerspointoperations,wheretheoutputdependsonlyonthecorrespondinginputpixelinthe inputimage(s).Bothdirectcomputationandlookuptableapproachesaredescribed.Withmultipleinputimages, techniquessuchasimageaveragingandbackgroundmodellingarediscussedindetail.Thefinalsectionsinthis chapterconsidertheprocessingofcolourandhyperspectralimages.Colourprocessingincludescolourspace conversion,colourbalancing,andcoloursegmentation.
Theimplementationofhistogramsandhistogram-basedprocessingarediscussedinChapter8.Thetechniquesofaccumulatingahistogram,andthenextractingdatafromthehistogram,aredescribedinsomedetail. Particulartasksarehistogramequalisation,thresholdselection,andusinghistogramsforimagematching.The conceptsofstandard1-Dhistogramsareextendedtomulti-dimensionalhistograms.Theuseofclusteringfor coloursegmentationandclassificationisdiscussedinsomedetail.Thechapterconcludeswiththeuseoffeatures extractedfrommulti-dimensionalhistogramsfortextureanalysis.
Chapter9considersawiderangeoflocalfilters,bothlinearandnonlinear.Particularemphasisisgiven tocachingtechniquesforastream-basedimplementationandmethodsforefficientlyhandlingtheprocessing aroundtheimageborders.Rankfiltersaredescribed,andaselectionofassociatedsortingnetworkarchitecturesreviewed.Morphologicalfiltersareanotherimportantclassoffilters.Statemachineimplementationsof morphologicalfilteringprovideanalternativetotheclassicfilterimplementation.Separabilityandbothserial andparalleldecompositiontechniquesaredescribedthatenablemoreefficientimplementations.
ImagewarpingandrelatedtechniquesarecoveredinChapter10.Theforwardandreversemapping approachestogeometrictransformationarecomparedinsomedetail,withparticularemphasisontechniques forstreamprocessingimplementations.Interpolationisfrequentlyassociatedwithgeometrictransformation. Hardware-basedalgorithmsforbilinear,bicubic,andspline-basedinterpolationaredescribed.Related techniquesofimageregistrationarealsodescribedattheendofthischapter,includingadiscussionoffeature pointdetection,description,andmatching.
Chapter11introduceslineartransforms,withaparticularfocusonthefastFouriertransform(FFT),the discretecosinetransform(DCT),andthewavelettransform.Bothparallelandpipelinedimplementationsof theFFTandDCTaredescribed.Filteringandinversefilteringinthefrequencydomainarediscussedinsome detail.Lifting-basedfilteringisdevelopedforthewavelettransform.Thiscanreducethelogicrequirementsby uptoafactorof4overadirectfiniteimpulseresponseimplementation.
Imagecodingisimportantforimagestorageortransmission.Chapter12discussesthestageswithinimage andvideocodingandoutlinessomeofthetechniquesthatcanbeusedateachstage.Severalofthestandards forbothstillimageandvideocodingareoutlined,withanoverviewofthecompressiontechniquesused.
Aselectionofintermediate-leveloperationsrelatingtoregiondetectionandlabellingispresentedin Chapter13.Standardsoftwarealgorithmsforchaincodingandconnectedcomponentlabellingareadapted togiveefficientstreamedimplementation.Thesecansignificantlyreduceboththelatencyandmemory requirementsofanapplication.Hardwareimplementationsofthedistancetransform,thewatershedtransform, andtheHoughtransformarealsopresented,discussingsomeofthekeydesigndecisionsforanefficient implementation.
Machinelearningtechniquesarecommonlyusedwithincomputervision.Chapter14introducesthekey techniquesforregressionandclassification,withaparticularfocusonFPGAimplementation.Deeplearningtechniquesareincreasinglybeingusedinmanycomputervisionapplications.Arangeofdeepnetwork architecturesisintroduced,andsomeoftheissuesforrealisingtheseonFPGAsarediscussed.
Finally,Chapter15presentsaselectionofcasestudies,showinghowthematerialandtechniquesdescribed inthepreviouschapterscanbeintegratedwithinacompleteapplication.Theseapplicationsbrieflyshowthe designstepsandillustratethemappingprocessatthewholealgorithmlevelratherthanpurelyattheoperation level.Manygainscanbemadebycombiningoperationstogetherwithinacompatibleoverallarchitecture.The applicationsdescribedarecolouredregiontrackingforagesture-baseduserinterface,calibratingandcorrecting barreldistortioninlenses,developmentofafovealimagesensorinspiredbysomeoftheattributesofthehuman visualsystem,amachinevisionsystemforreal-timeproducegrading,stereoimagingfordepthestimation,and facedetection.
ConventionsUsed ThecontentsofthisbookareindependentofanyparticularFPGAorFPGAvendor,oranyparticularhardwaredescriptionlanguage.Thetopicisalreadysufficientlyspecialisedwithoutnarrowingtheaudiencefurther! Asaresult,manyofthefunctionsandoperationsarerepresentedinblockschematicform.Thisenablesa language-independentrepresentationandplacesemphasisonaparticularhardwareimplementationofthealgorithminawaythatisportable.ThebasicelementsoftheseschematicsareillustratedinFigureP.1. I isgenerally usedastheinputofanimageprocessingoperation,withtheoutputimagerepresentedby Q
Withsomemathematicaloperations,suchassubtractionandcomparison,theorderoftheoperandsisimportant.Insuchcases,thefirstoperandisindicatedwithablobratherthananarrow,asshownonthebottomin FigureP.1.
Considerarecursivefilteroperatingonstreameddata:
wherethesubscriptinthisinstancereferstothe nthpixelinthestreamedimage.Atahighlevel,thiscanbe consideredasanimageprocessingoperation,andrepresentedbyasingleblock,asshowninthetop-leftof FigureP.1.Thelow-levelimplementationisgiveninthemiddle-leftpanel.Theinputandoutput, I and Q,are
Counter
block
Single bit signal
Multi-bit signal (a number)
Signal concatenation
Signal splitting Frame buffer
Multiplexer
FigureP.1 Conventionsusedinthisbook.Top-left:representationofanimageprocessingoperation; middle-left:ablockschematicrepresentationofthefunctiongivenbyEq.(P.1);bottom-left:representation ofoperatorswheretheorderofoperandsisimportant;right:symbolsusedforvariousblockswithinblock schematics.
representedbyregisters(darkblocks,withoptionalregisternamesinwhite);thesubscriptshavebeendropped becausetheyareimplicitwithstreamedoperation.Insomeinstancesadditionalcontrolinputsmaybeshown,for exampleCEforclockenable,RSTforreset.Constantsarerepresentedasmid-greyblocks,andotherfunction blockswithlight-greybackground.
Whenrepresentinglogicfunctionsinequations, ∨ isusedforlogicalOR,and ∧ forlogicalAND.Thisisto avoidconfusionwithadditionandmultiplication.
DonaldG.Bailey MasseyUniversity PalmerstonNorth,NewZealand
Acknowledgments IwouldliketoacknowledgeallthosewhohavehelpedmetogetmewhereIcurrentlyaminmyunderstandingof field-programmablegatearray(FPGA)-baseddesign.Inparticular,Iwouldliketothankmyresearchstudents (DavidJohnson,KimGribbon,ChrisJohnston,AaronBishell,AndreasBuhler,NiMa,AnoopAmbikumar, TariqKhan,andMichaelKlaiber)whohelpedtoshapemythinkingandapproachtoFPGAdevelopmentaswe struggledtogethertoworkoutefficientwaysofimplementingimageprocessingalgorithms.Thisbookisas muchareflectionoftheirworkasitisofmine.
MostofourearlyworkusedHandel-CandwastestedonboardsprovidedbyCeloxica.Iwouldliketo acknowledgethesupportprovidedbyRogerGookandhisteam,firstwithCeloxicaandlaterwithAgility DesignSolutions.LaterworkwasonboardssuppliedbyTerasic.IwouldliketoacknowledgeSeanPengand histeamfortheiron-goingsupportandencouragement.
MasseyUniversityhasprovidedasupportiveenvironmentandthefreedomformetoexplorethisfield.In particular,SergeDemidenkogavemetheencouragementandthepushtobeginplayingwithFPGAs;hehas beenasourceofbothinspirationandchallengingquestions.Othercolleagueswhohavebeenofparticular encouragementareGourabSenGupta,RichardHarris,AmalPunchihewa,andStevenLeMoan.Iwouldalso liketoacknowledgePaulLyons,whoco-supervisedseveralofmystudents.
EarlyversionsofsomeofthematerialinthisbookwerepresentedashalfdaytutorialsattheIEEERegion 10Conference(TENCON)in2005inMelbourne,theIEEEInternationalConferenceonImageProcessing (ICIP)in2007inSanAntonioTexas,andthe2010AsianConferenceonComputerVision(ACCV)inQueenstown,NewZealand.Itwasthenadaptedtothree-dayworkshops,whichwereheldinAustralia,NewZealand, Japan,andasaMastersCourseinGermany.Iwouldliketothankattendeesattheseworkshopsandcoursesfor providingvaluablefeedbackandstimulatingdiscussion.
During2008,IspentasabbaticalwiththeCircuitsandSystemsGroupatImperialCollegeLondon,where Ibeganwritingthefirstedition.IwouldliketothankPeterCheung,ChristosBouganis,PeterSedcole,and GeorgeConstantinidesfordiscussionsandopportunitiestobounceideasoff.
Mywife,Robyn,hashadtoputupwithmyabsencemanyeveningsandweekendswhileworkingon themanuscriptsforboththefirstandsecondeditions.Iamgratefulforbothherpatienceandhersupport. Thisbookisdedicatedtoher.
DonaldG.Bailey
AbouttheCompanionWebsite Thisbookisaccompaniedbyacompanionwebsite. www.wiley.com/go/bailey/designforembeddedimageprocessc2e ThiswebsiteincludesPowerPointslides.
1 ImageProcessing Visionisarguablythemostimportanthumansense.Theprocessingandrecordingofvisualdatathereforehas significantimportance.Theearliestimagesarefromprehistoricdrawingsoncavewallsorcarvedonstone monumentscommonlyassociatedwithburialtombs.(Itisnotsomuchthemediumthatisimportanthere;anythingelsewouldnothavesurvivedtotoday.)Suchimagesconsistofamixtureofbothpictorialandabstract representations.Improvementsintechnologyenabledimagestoberecordedwithmorerealism,suchaspaintingsbythemasters.Imagesrecordedinthismannerareindirectinthesensethatthelightintensitypatternis notuseddirectlytoproducetheimage.Thedevelopmentofchemicalphotographyintheearly1800senabled directimagerecording.Thistrendhascontinuedwithelectronicrecording,firstwithanaloguesensors,and subsequentlywithdigitalsensors,whichincludeanaloguetodigital(A/D)conversiononthesensorchip,to directlyproducedigitalimages.
Imagingsensorshavenotbeenrestrictedtotheportionoftheelectromagneticspectrumvisibletothehuman eye.Sensorshavebeendevelopedtocovermuchoftheelectromagneticspectrumfromradiowavesthrough togammarays.Awidevarietyofotherimagingmodalitieshavealsobeendeveloped,basedonultrasound, electrons,atomicforce(Binnigetal.,1986),magneticresonance,andsoon.Inprinciple,anyquantitythatcan besensedcanbeusedforimaging,evendustrays(Auer,1982).
Sincevisionissuchanimportantsense,theprocessingofimageshasbecomeimportanttoo,toaugmentor enhancehumanvision.Imagescanbeprocessedtoenhancetheirsubjectivecontentortoextractusefulinformation.Whileitispossibletoprocesstheopticalsignalsassociatedwithvisualimagesusinglensesandoptical filters,thisbookfocusesondigitalimageprocessing,thenumericalprocessingofimagesbydigitalhardware.
Oneoftheearliestapplicationsofdigitalimageprocessingwasfortransmittingdigitisednewspaperpictures acrosstheAtlanticOceanintheearly1920s(McFarlane,1972).However,itwasonlywiththeadventofcomputerswithsufficientmemoryandprocessingpowerthatdigitalimageprocessingbecamewidespread.Theearliest recordedcomputer-basedimageprocessingwasfrom1957whenascannerwasaddedtoacomputerattheUSA NationalBureauofStandards(Kirsch,1998).Itwasusedfortheearlyresearchonedgeenhancementandpattern recognition.Inthe1960s,theneedtoprocesslargenumbersoflargeimagesobtainedfromsatellitesandspace explorationstimulatedimageprocessingresearchatNASA’sJetPropulsionLaboratory(Castleman,1979).At thesametime,researchinhigh-energyparticlephysicsrequireddetectinginterestingeventsfromlargenumbersofcloudchamberphotographs(Duff,2000).Ascomputersgrewinpowerandreducedincost,therange ofapplicationsfordigitalimageprocessingexploded,fromindustrialinspectiontomedicalimaging.Image sensorsarenowubiquitous,inmobilephones,laptops,andinvideo-basedsecurityandsurveillancesystems.
1.1BasicDefinitions Moreformally,an image isaspatialrepresentationofanobject,scene,orotherphenomenon(Haralickand Shapiro,1991).Examplesofimagesincludeaphotograph,whichisapictorialrecordformedfromthelight
DesignforEmbeddedImageProcessingonFPGAs,SecondEdition.DonaldG.Bailey. ©2024JohnWiley&SonsLtd.Published2024byJohnWiley&SonsLtd. CompanionWebsite:www.wiley.com/go/bailey/designforembeddedimageprocessc2e
intensitypatternonanopticalsensor;aradiograph,whichisarepresentationofdensityformedthroughexposure toX-raystransmittedthroughanobject;amap,whichisaspatialrepresentationofphysicalorculturalfeatures; andavideo,whichisasequenceoftwo-dimensionalimagesthroughtime.Morerigorously,animageisany continuousfunctionoftwoormorevariablesdefinedonsomeboundedregionofspace.
A digitalimage isanimageindigitalformat,sothatitissuitableforprocessingbycomputer.Thereare twoimportantcharacteristicsofdigitalimages.Thefirstisspatialquantisation.Computersareunabletoeasily representarbitrarycontinuousfunctions,sothecontinuousfunctionissampled.Theresultisaseriesofdiscretepictureelements,or pixels (for2-Dimages)orvolumeelements, voxels (for3-Dimages).Samplingdoes notnecessarilyhavetobespatiallyuniform,forexample pointclouds fromaLiDARscanner.Samplingcan representacontinuousimageexactly(inthesensethattheunderlyingcontinuousfunctionmayberecovered exactly),givenaband-limitedimageandasufficientlyhighsamplerate(Shannon,1949).Thesecondcharacteristicofdigitalimagesissamplequantisation.Thisresultsindiscretevaluesforeachpixel,enablinganinteger representation.Commonbitwidthsperpixelare1(binaryimages),8(greyscaleimages),and24(3 × 8bitsfor colourimages).Modernhighdynamicrangesensorscanprovide12–16bitsperpixel.Unlikesampling,value quantisationwillalwaysresultinanerrorbetweentherepresentationandtruevalue.Inmanycircumstances, however,this quantisationerror or quantisationnoise maybemadesmallerthantheuncertaintyinthetrue valueresultingfrominevitablemeasurementnoise.
Initsbasicform,adigitalimageissimplyatwo(orhigher)-dimensionalarrayofnumbers(usuallyintegers),whichrepresentsanobjectorscene.Onceinthisform,animagemaybereadilymanipulatedbya digitalcomputer.Itdoesnotmatterwhatthenumbersrepresent,whetherlightintensity,reflectance,attenuation,distancetoapoint(range),temperature,populationdensity,elevation,rainfall,oranyothernumerical quantity.
Digitalimageprocessing canthereforebedefinedassubjectingsuchanimagetoaseriesofmathematical operationsinordertoobtainadesiredresult.Thismaybeanenhancedimage;thedetectionofsomecritical featureorevent;ameasurementofanobjectorkeyfeaturewithintheimage;aclassificationorgradingof objectswithintheimageintooneoftwoormorecategories;oradescriptionofthescene.
Imageprocessingtechniquesareusedinanumberofrelatedfields.Whiletheprincipalfocusofthefields oftendiffers,manyofthetechniquesremainthesameatthefundamentallevel.Someofthedistinctivecharacteristicsarebrieflyoutlinedhere.
Imageenhancement involvesimprovingthesubjectivequalityofanimageorthedetectabilityofobjectswithin theimage(HaralickandShapiro,1991).Theinformationthatisenhancedisusuallyapparentintheoriginal imagebutmaynotbeclear.Examplesofimageenhancementincludenoisereduction,contrastenhancement, edgesharpening,andcolourcorrection.
Imagerestoration goesonestepfurtherthanimageenhancement.Itusesknowledgeofthecausesofhowan imageisdegradedtocreateamodelofthedegradationprocess.Thismodelisthenusedtoderiveaninverse processthatisusedtorestoretheimage.Inmanycases,theinformationintheimagehasbeendegradedto theextentofbeingunrecognisable,forexamplesevereblurring.
Imagereconstruction involvesrestructuringthedatathatisavailableintoamoreusefulform.Examplesare imagesuper-resolution(reconstructingahigh-resolutionimagefromaseriesoflow-resolutionimages), imagefusion(combiningimagesfrommultiplesources),andtomography(reconstructingacrosssection ofanobjectfromaseriesofprojections).
Imageanalysis refersspecificallytousingcomputerstoextractdatafromimages.Theresultisusually someformofmeasurement.Inthepast,thiswasalmostexclusively2-Dimaging,althoughwiththe adventofconfocalmicroscopyandotheradvancedimagingtechniques,thishasextendedtothree dimensions.
Patternrecognition isconcernedwithidentifyingobjectsbasedonpatternsinthemeasurements(Haralickand Shapiro,1991).Thereisastrongfocusonstatisticalapproaches,althoughsyntacticandstructuralmethods arealsoused.
Computervision tendstouseamodel-basedapproachtoimageprocessing.Mathematicalmodelsofboththe sceneandtheimagingprocessareusedtoderivea3-Drepresentationbasedononeormore2-Dimagesof ascene.Theuseofmodelsimplicitlyprovidesaninterpretationofthecontentsoftheimagesobtained.
Thefieldsaresometimesdistinguishedbasedonapplication:
Machinevision isusingimageprocessingaspartofthecontrolsystemforamachine(Schaffer,1984).Images arecapturedandanalysed,andtheresultsareuseddirectlyforcontrollingthemachinewhileperforminga specifictask.Real-timeprocessingisoftencritical.
Remotesensing usuallyreferstotheuseofimageanalysisforobtaininggeographicalinformation,eitherusing satelliteimagesoraerialphotography(includingfromdrones).
Medicalimaging encompassesawiderangeofimagingmodalities(X-ray,ultrasound,magneticresonance, positronemission,andothers)concernedprimarilywithmedicaldiagnosisandothermedicalapplications.It involvesbothimagereconstructiontocreatemeaningfulimagesfromtherawdatagatheredfromthesensors andimageanalysistoextractusefulinformationfromtheimages.
Imageandvideocoding focusesonthecompressionofanimageorvideo,sothatitoccupieslessstoragespace ortakeslesstimetotransmitfromonelocationtoanother.Compressionispossiblebecausemanyimages containsignificantredundantinformation.Inthereversestep,imagedecoding,thefullimageorvideois reconstructedfromthecompresseddata.
1.2ImageFormation Whiletherearemanypossiblesensorsthatcanbeusedforimaging,thefocusinthissectionwillbeonoptical imagescapturedfromvisiblepartoftheelectromagneticspectrum.Whilethesensingtechnologymaydiffer significantlyforothertypesofimaging,manyoftheimagingprincipleswillbesimilar.
Thefirstrequirementtoobtaininganimageissomeformofsensortodetectandquantifytheincominglight. Thiscouldbeasinglesensingelement,withlightfromdifferentdirectionsreflectedtowardsthesensorusing ascanningmirror.Thisapproachiscommonlyusedwithtime-of-flightlaserrangescannersorLiDAR.The outputofaLiDARistypicallyathree-dimensionalpointcloud.
Morecommonly,a2-Dsensorarrayisused,whichmeasuresthelightintensityateachsensingelement andproducesa2-Dimage.Incapturinganimage,itisimportanttosamplewithatleasttwopixelsacross thehighestspatialfrequency(theNyquistsamplingcriterion)toavoidaliasing.Thetwomostcommon solid-statesensortechnologiesarechargecoupleddevices(CCDs)andcomplementarymetaloxidesemiconductor(CMOS)activepixelsensors(Fossum,1993).Thebasiclightsensingprincipleisthesame:an incomingphotonliberatesanelectronwithinthesiliconsemiconductorthroughthephotoelectriceffect. Thesephotoelectronsarethenaccumulatedduringtheexposuretimebeforebeingconvertedintoavoltagefor readingout.
AlthoughtheactivepixelsensortechnologywasdevelopedbeforeCCDs,theneedforlocaltransistorsmade earlyCMOSsensorsimpracticalbecausethetransistorsconsumedmostoftheareaofthedevice(Fossum, 1993).Therefore,CCDsensorsgainedearlydominanceinthemarket.However,thecontinualreductionoffeaturesizeshasmeantthatCMOSsensorsbecamepracticalfromtheearly1990s.EarlyCMOSsensorshadlower sensitivityandhighernoisethanasimilarformatCCDsensor,althoughwithrecenttechnologicalimprovementsthereisnowverylittledifferencebetweenthetwofamilies(Litwiller,2005).SinceCMOSsensorsuse thesameprocesstechnologyasstandardCMOSdevices,theyalsoenableotherfunctionssuchasA/Dconversiontobedirectlyintegratedonthesamechip.MostmodernsensorsareCMOSdevices,withintegrateddigital conversion,providingadigitaloutputstream.
Arelativenewcomerisanimagingsensorbasedonsingle-photonavalanchediodes(SPAD)(Morimotoetal., 2020).Eachsensorelementcandetectsinglephotons,whichareamplifiedtogiveapulse.Thesepulsescan becounted,givingadirectdigitalmeasureoftheintensity.Theindividualpulsescanalsobeaccuratelytimed, givingdirecttime-of-flightmeasurementfor3-Dapplications.
1.2.1Optics Acameraconsistsofasensorcombinedwithalens.Thelensfocusesaparallelbeamoflight(effectivelyfrom infinitedistance)fromaparticulardirectiontoapointonthesensor.Foranobjectatafinitedistance,asseen
Figure1.1 Opticalimageformationusingalensusinggeometricoptics.
inFigure1.1,geometricopticsgivestherelationshipbetweenthelensfocallength, f ,theobjectdistance, d o , andimagedistance, di as
Sincetheraythroughthecentreofthelensisastraightline,thisraymaybeusedtodeterminewherean objectwillbeimagedtoonthesensor.This pinholemodel ofthecamerawillmapa3-Dpoint (x, y, z) incamera coordinates(theoriginisthepinholelocation;the z-axiscorrespondstotheprincipalaxisofthecamera;and the x-and y-axesarealignedwiththesensoraxes)tothepoint
onthesensorplane. di isalsocalledtheeffectivefocaldistanceofthelens.Notethattheoriginonthesensor planeisinthecentreoftheimage(theintersectionoftheprincipalaxis).Thisprojectivemappingeffectively scalestheobjectininverseproportiontoitsdistancefromthecamera(z = d o ).Fromsimilartriangles,theoptical magnificationis
(1.3)
Theimageisgenerallyfocusedbyadjustingtheimagedistance, di ,bymovingthelensawayfromthesensor (althoughinmicroscopy, di isfixed,andtheimageisfocusedbyadjusting d o ).Thiscaneitherbemanualor, foranelectronicallycontrolledlens,anautofocusalgorithmautomaticallyadjusts di tomaximisetheimage sharpness(seeSection6.1.7.2).
Objectsnotattheoptimalobjectdistance, d o ,willbecomeblurred,dependingonboththeoffsetindistance frombestfocus, Δd o ,andthediameterofthelensaperture, D.Thesizeoftheblurredspotiscalledthe circle ofconfusion andisgivenby
with N beingthenumericalapertureofthelens(N = f D ).
Thedepthoffieldofacameraistherangeofdistancesoverwhichthedegreeofblurringisacceptable.This isoftendefinedastherangewherethesizeofthecircleofconfusionislessthanonepixel.Iftheimagesare forhumanuse,itisalsonecessarytotakeintoaccounthowtheimagesaredisplayed(especiallythedegree ofmagnification),theviewingdistance,andvisualacuity.Thisisusuallytakenas0.2mmat250mmdistance (Conrad,2006).
Asecondfactorwhichaffectsimagesharpness(ofanideallens)isdiffraction.Ratherthanfocusingaparallel beamtoanidealpoint,thefocuspatternintheimageplanewithacircularapertureisanAirydisk.TheRayleigh criterion(Rayleigh,1879)indicatesthattwopointscanberesolvedifthepeakofthediffractionpatternfrom onepointisoutsidethefirstnullofthediffractionpatternoftheotherpoint.Thesizeoftheblurspotfrom diffractionofanobjectatfinitedistanceistherefore
where �� isthewavelengthofthelight.
Equations(1.4)and(1.5)maybecombinedtogivetheeffectivesizeofthebluras(Hansma,1996)
Ofcourse,anyotherlensaberrationsorimperfectionswillfurtherincreasetheblur.
1.2.2Colour Humansareabletoseeincolour.Therearethreetypesofcolourreceptors(cones)inthehumaneyethatrespond differentlytodifferentwavelengthsoflight.Ifthewavelengthdependenceofareceptoris Sk (��),andthelight fallingonthereceptorcontainsamixoflightofdifferentwavelengths, C (��),thentheresponseofthatreceptor willbegivenbythecombinationoftheresponsesofalldifferentwavelengths:
Whatisperceivedascolouristheparticularcombinationofresponsesfromthethreedifferenttypesofreceptors.Manydifferentwavelengthdistributionsproducethesameresponsesfromthereceptorsandthereforeare perceivedasthesamecolour(thisisknownas metamerism).Inpractice,itisalittlemorecomplicatedthan thisbecauseofchromaticadaptation.
Theincominglightdependsonthewavelengthdistributionofthesourceoftheillumination, I (��),andthe wavelength-dependentreflectivityoftheobjectwearelookingat, ��(��).So,Eq.(1.7)becomes
Toreproduceacolour,itisonlynecessarytoreproducetheassociatedcombinationofreceptorresponses. Thisisaccomplished,forexampleinatelevisionmonitor,byproducingthecorrespondingmixtureofred, green,andbluelight.
Tocaptureacolourimageforhumanviewing,itisnecessarytohavethreedifferentcolourreceptorsinthe sensor.Ideally,theseshouldcorrespondtothespectralresponsesofthecones.However,sincemostofwhatis seenhasbroadspectralcharacteristics,aprecisematchisnotcriticalexceptwhentheilluminanthasanarrow spectralcontent(forexamplesodiumvapourlampsandLED-basedilluminationsources).Siliconhasabroad spectralresponse,withapeakinthenearinfrared.Therefore,toobtainacolourimage,thisresponsemustbe modifiedthroughappropriatecolourfilters.
Twoapproachesarecommonlyusedtoobtainafullcolourimage.Thefirstistoseparatethered,green, andbluecomponentsoftheimageusingaprismandwithdichroicfilters.Thesecomponentsarethencaptured usingthreeseparatesensors,oneforeachcomponent.Theneedforthreechipswithprecisealignmentmakes suchcamerasrelativelyexpensive.Analternativeapproachistouseasinglechipsensor,withamosaicofsmall filters,withonecolourfilterintegratedwitheachpixel.Sinceeachpixelsensesonlyonecolourchannel,the effectivespatialresolutionofthesensorisdecreased.Afullcolourimageiscreatedbyinterpolatingthemissing pixelsofeachcomponent.ThemostcommonlyusedcolourfilterarrayistheBayerpattern(Bayer,1976),with filterstoselectthered,green,andblueprimarycolours.Otherpatternsarealsopossible,forexamplefilteringthe cyan,magenta,andyellowsecondarycolours(Parulski,1985).Camerasusingthesecondarycolourshavebetter lowlightsensitivitybecauseeachfilterabsorbslessoftheincominglight.However,theprocessingrequired toproducetheoutputgivesabettersignal-to-noiseratiofromtheprimaryfiltersthanfromsecondaryfilters (Parulski,1985;Baeretal.,1999).
Onefurtherpossibilitythathasbeenconsideredistostackthered,green,andbluesensorsoneontopofthe other(LyonandHubel,2002;Gilblometal.,2003).TheFoveonsensorreliesonthefactthatlongerwavelengths oflightpenetratefurtherintothesiliconbeforebeingabsorbed.Thus,usingphotodiodesatdifferentdepths,a fullcolourpixelmaybeobtainedwithoutexplicitfiltering.
Thecameraoutputisusuallystreamedfollowingarasterscan.Forananaloguecamera,thisisavoltage waveform,oftenwithcombinedsynchronisationsignalstoindicatetheendofeachrowandframe.Acolour cameracanoutputeithercomponentvideo(eachcolourchannelhasaseparatesignal)orcompositevideo
(withchrominancecomponentsmodulatingacoloursub-carrier).A framegrabber preprocessesthesignal fromthecamera,amplifiesit,separatesthesynchronisationsignals,andifnecessarydecodesthecomposite coloursignalintoitscomponents.TheanaloguevideosignalisdigitisedbyanA/Dconverter(orthreeA/D convertersforcolour)andstoredinmemoryforlaterprocessing.AdigitalcamerahasintegratedA/Dconversion anddirectlyprovidesadigitalpixelstream.
1.3ImageProcessingOperations Aftercapturingtheimage,thenextstepistoprocesstheimagetoachievethedesiredresult.Thesequenceof imageprocessingoperationsusedtoprocessanimagefromonestatetoanotheriscalledan imageprocessingalgorithm.Thetermalgorithmcansometimescauseconfusion,sinceeachoperationisalsoimplemented throughanalgorithminsomeprogramminglanguage.Thisdistinctionbetweenapplication-andoperation-level algorithmsisillustratedinFigure1.2.Usually,thecontextwillindicatethesenseinwhichthetermisused. Astheinputimageisprocessedbytheapplication-levelalgorithm,itundergoesaseriesoftransformations. TheseareillustratedforasimpleexampleinFigure1.3.Theinputimageconsistsofanarrayofpixels.Each pixel,onitsown,carriesverylittleinformation,buttherearealargenumberofpixels.Theindividualpixel valuesrepresenthighvolumebutlowvaluedata.Astheimageisprocessed,collectionsofpixelsaregrouped together.Atthisintermediatelevel,thedatamaystillberepresentedintermsofpixels,butthepixelvalues usuallyhavemoremeaning;forexample,theymayrepresentalabelassociatedwitharegion.Intermediate-level representationsmaydepartfromanexplicitimagerepresentation,withregionsrepresentedbytheirboundaries (forexampleasachaincodeorasetoflinesegments)ortheirstructure(forexampleasaquad-tree).Fromeach region,asetoffeatures(afeaturevector)maybeextractedthatcharacterisetheregion.Generally,thenumber offeaturesissmall(relativetothenumberofpixels),witheachfeaturecontainingsignificantinformationthat canbeusedtodistinguishitfromotherregionsorotherobjectsthatmaybeencountered.Intermediate-level databecomessignificantlylowerinvolumebuthigherinquality.Finally,atthehighlevel,thefeaturedatais usedtoclassifytheobjectorsceneintooneofseveralcategoriesortoderiveadescriptionoftheobjectorscene. Ratherthanfocusingonthedata,itisalsopossibletofocusontheimageprocessingoperations.The operationscanbegroupedaccordingtothetypeofdatathattheyprocess(Weems,1991).Thisgrouping issometimesreferredtoasanimageprocessingpyramid(DowntonandCrookes,1998)asrepresented
