Compression, Coding and Protection of Images and Videos, Subject Head – Christine Guillemot
Graph Spectral Image Processing
Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms and licenses issued by the CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the undermentioned address:
ISTE Ltd
John Wiley & Sons, Inc.
27-37 St George’s Road 111 River Street London SW19 4EU Hoboken, NJ 07030
The rights of Gene Cheung and Enrico Magli to be identified as the author of this work have been asserted by them in accordance with the Copyright, Designs and Patents Act 1988.
Library of Congress Control Number: 2021932054
British Library Cataloguing-in-Publication Data
A CIP record for this book is available from the British Library
Agraph G (V , E , W ) containsaset V of N nodesandaset E of M edges.While directedgraphsarealsopossible,inthisbookwemorecommonlyassumean undirectedgraph,whereeachexistingedge (i,j ) ∈E isundirectedandcontainsan edgeweight wi,j ∈ R,whichistypicallypositive.Alargepositiveedgeweight wi,j wouldmeanthatsamplesatnodes i and j areexpectedtobesimilar/correlated.
where li ∈ R2 isthelocationofpixel i onthe2Dimagegrid, xi ∈ R istheintensity ofpixel i,and σ 2 l and σ 2 x aretwoparameters.Hence, 0 ≤ wi,j ≤ 1.Largergeometric and/orphotometricdistancesbetweenpixels i and j wouldmeanasmallerweight wi,j .Edgeweightscanalternativelybedefinedbasedonlocalpixelpatches,features, etc.(Milanfar2013b).Toalargeextent,theappropriatedefinitionofedgeweightis applicationdependent,aswillbediscussedinvariousforthcomingchapters.
A graphsignal x on G isadiscretesignalofdimension N –onesample xi ∈ R foreachnode1 i in V .Assumingthatnodesareappropriatelylabeledfrom 1 to N ,we cansimplytreatagraphsignalasavector x ∈ RN .
I.3.Graphspectrum
Denoteby W ∈ RN ×N an adjacencymatrix,wherethe (i,j )thentryis Wi,j = wi,j .Next,denoteby D ∈ RN ×N adiagonal degreematrix,wherethe (i,i)thentryis Di,i = j Wi,j .A combinatorialgraphLaplacianmatrix L is L = D W (Shuman etal.2013).Because L isrealandsymmetric,onecanshow, viathespectraltheorem,thatitcanbeeigen-decomposedinto:
L = UΛU [I.2] where Λ isadiagonalmatrixcontainingrealeigenvalues λk alongthediagonal,and U isaneigen-matrixcomposedoforthogonaleigenvectors ui ascolumns.Ifalledge
weights wi,j arerestrictedtobepositive,thengraphLaplacian L canbeproventobe positivesemi-definite (PSD)(Chung1997)2,meaningthat λk ≥ 0, ∀k and x Lx ≥ 0, ∀x.Non-negativeeigenvalues λk canbeinterpretedas graph frequencies,andeigenvectors U canbeinterpretedascorrespondinggraphFourier modes.Togethertheydefinethe graphspectrum forgraph G .
Thesetofeigenvectors U for L collectivelyformtheGFT(Shuman etal.2013), whichcanbeusedtodecomposeagraphsignal x intoitsfrequencycomponentsvia α = U x.Infact,onecaninterpretGFTasageneralizationofknowndiscrete transformslikethe DiscreteCosineTransform (DCT)(seeShuman etal.2013for details).
Notethatifthemultiplicity mk ofaneigenvalue λk islargerthan1,thenthe setofeigenvectorsthatspanthecorrespondingeigen-subspaceofdimension mk is non-unique.Inthiscase,itisnecessarytospecifythegraphspectrumasthecollection ofeigenvectors U themselves.
Closelyrelatedtothecombinatorialgraph,Laplacian L,areothervariantsof Laplacianoperators,eachwiththeirownuniquespectralproperties.A normalized graphLaplacian Ln = D 1/2 LD 1/2 isasymmetricnormalizedvariantof L.In contrast,a randomwalkgraphLaplacian Lr = D 1 L isanasymmetricnormalized variantof L.A generalizedgraphLaplacian Lg = L + diag(D) isagraphLaplacian withself-loops di,i atnodes i –calledthe loopygraphLaplacian inDörflerand Bullo(2013)–resultinginageneralsymmetricmatrixwithnon-positive off-diagonalentriesforapositivegraph(Biyikoglu etal. 2005). Eigen-decompositioncanalsobeperformedontheseoperatorstoacquireasetof graphfrequenciesandgraphFouriermodes.Forexample,normalizedvariants Ln and Lr (whicharesimilaritytransformsofeachother)sharethesameeigenvalues between 0 and 2.While L and Ln arebothsymmetric, Ln doesnothavetheconstant vectorasaneigenvector.Asymmetric Lr canbesymmetrizedvialeftandright diagonalmatrixmultiplications(Milanfar2013a).Differentvariationoperatorswill beusedthroughoutthebookfordifferentapplications.
2Onecanprovethatagraph G withpositiveedgeweightshasPSDgraphLaplacian L viathe Gershgorincircletheorem:eachGershgorindisccorrespondingtoarowin L islocatedinthe non-negativehalf-space,andsincealleigenvaluesresideinsidetheunionofalldiscs,theyare non-negative.
I.5.Graphsignalsmoothnesspriors
Traditionally,forgraph G withpositiveedgeweights,signal x isconsidered smooth ifeachsample xi onnode i issimilartosamples xj onneighboringnodes j withlarge wi,j .Inthegraphfrequencydomain,itmeansthat x mostlycontainslow graphfrequencycomponents,i.e.coefficients α = U x arezeros(ormostlyzeros) forhighfrequencies.Thesmoothestsignalistheconstantvector–thefirst eigenvector u1 for L,correspondingtothesmallesteigenvalue λ1 =0
Mathematically,wecandeclarethatasignal x issmoothifits graphLaplacian regularizer (GLR) x Lx issmall(PangandCheung2017).GLRcanbeexpressed as:
Because L isPSD, x Lx islowerboundedby 0 andachievedwhen x = cu1 for somescalarconstant c.OnecanalsodefineGLRusingthenormalizedgraph Laplacian Ln insteadof L,resultingin x Ln x.Thecaveatsisthattheconstant vector u1 –typicallythemostcommonsignalinimaging–isnolongerthefirst eigenvector,andthus u1 Ln u1 =0
InChen etal. (2015),theadjacencymatrix W isinterpretedasashiftoperator, andthus,graphsignalsmoothnessisinsteaddefinedasthedifferencebetweenasignal x anditsshiftedversion Wx.Specifically, graphtotalvariation (GTV)basedon lp -normis:
where λmax istheeigenvalueof W withthelargestmagnitude(alsocalledthe spectralradius),and p isachoseninteger.Asavarianttoequation[I.4],aquadratic smoothnesspriorisdefinedinRomano etal.(2017),usingarow-stochasticversion Wn = D 1 W oftheadjacencymatrix W :
Toavoidconfusion,wewillcallequation[I.5]the graphshiftvariation (GSV) prior.GSViseasiertouseinpracticethanGTV,sincethecomputationof λmax is requiredforGTV.NotethatGSV,asdefinedinequation[I.5],canalsobeusedfor signalsondirectedgraphs.
Supposethatanimpulseresponseofafilter hn isgiven apriori.Thediscrete-time filteredsignal yn intheLTIsystemiscalculatedfrom xn and hn byconvolutionas follows:
Thisequationisbasedonthe shift ofthesignalorimpulseresponse.InLTI systems,we(implicitly)assumethattheshiftofadiscrete-timesignaliswell defined,i.e. xn k isuniqueandtimeinvariant.Therefore,equation[1.1]is equivalentlyrepresentedas
Inequation[1.2],theimpulseresponse hk isinvariantfor n,i.e.thesamefilteris usedfordifferentvaluesof n.Instead,wecanusedifferentfiltersfordifferentvalues of n toyield yn ,whoseimpulseresponse hk [n] isoftendefinedinasignal-dependent manner,i.e. hk [n] = hk [m] for m = n.Itisformulatedas
1Here,weassumeboth x and y arefinitelengthsignalsandtheirboundariesareextendedor filteredbyaboundaryfiltertoensurethattheequationisvalid.
Inthischapter,weconsider linear graphfilters.Readerscanfindnonlineargraph filters,likeoneusedindeeplearning,inthefollowingchapters,specifically Chapter10.
Letusdenoteagraphfilteras H ∈ RN ×N ,whereitselementsaretypicallyderived from G and x.AsintheLTIsystem,thefilteredsignalisrepresentedas
Therepresentationofitselement yn issimilartothatobservedinequation[1.3], i.e.
where [ ]n,k isthe n,k -elementinthematrix.Similartodiscrete-timesignals,graph signalfilteringmaybedefinedinthevertexandgraphfrequencydomains.Theseare describedinthefollowing.
1.3.1. Vertexdomainfiltering
Vertexdomainfilteringisananalogoffilteringinthetimedomain.However, GSPsystemsarenotshift-invariant:Thismeansnodeindicesdonothaveany physicalmeaning,ingeneral.Therefore,theshiftofgraphsignalsbasedonthe indicesofnodes,similartothatusedfordiscrete-timesignals,wouldbe inappropriate.Moreover,theunderlyinggraphwillexhibitahighlyirregular connectivity,i.e.thedegreeineachnodewillvarysignificantly.Forexample,thestar graphshowninFigure1.1hasonecenternodeand N 1 surroundingnodes.Itis clearthat N 1 edgesareconnectedtothecenternode,i.e.thecenternodehas degree N 1,whereasallofthesurroundingnodeshavedegree 1.Inanimage processingperspective,suchanirregularitycomesfromtheedgeandtextureregions. AnexampleisprovidedontherightsideofFigure1.1.Supposethatweconstructa graphbasedonpixelintensity,i.e.pixelsarenodes,andtheyareconnectedbyedges withhigherweightswhentheirpixelvaluesarecloser.Inthissituation,pixels along edge/texturedirectionswillbeconnectedtoeachotherstronglywithalargedegree, whereasthose across edge/texturedirectionsmayhaveweakeredges,ormayevenbe disconnected.Filteringbasedonsuchagraphwillthereforereflectstructuresinthe vertex(i.e.pixel)domain.
Figure1.1. Left:Stargraphwith N =7.Right:TexturedregionofBarbara
Vertexdomainfilteringcanbedefinedmoreformallyasfollows.Let Nn,p beaset of p-hopneighborhoodnodesofthe nthnode.Clearly |Nn,p | variesaccordingto n
Vertexdomainfilteringmaybetypicallydefinedasalocallinearcombinationofthe neighborhoodsamples yn := k ∈Nn,p [H]n,k xk
Since Nn,p variesaccordingto n, [H]n,k shouldbeappropriatelydeterminedfor all n.Thematrixformofequation[1.9]mayberepresentedas
y =(diag([H]0,0 ,..., [H]N 1,N 1 )+ h(W ))x,
where h(W ) isamatrixcontainingfiltercoefficients h[n,k ](n = k ) asafunctionof theadjacencymatrix W ,inwhich [h(W )]n,k =0 if k ∈Nn,p .
Thevertexdomainfilteringinequations[1.9]and[1.10]requiresthe determinationof n |Nn,p | filtercoefficients,ingeneral;moreover,itsometimes needsincreasedcomputationalcomplexity.Typically, [H]n,k maybeparameterized inthefollowingform:
where hp isarealvalueand Wp ∈ RN ×N isamaskedadjacencymatrixthatonly contains p-hopneighborhoodelementsof W .Itisformulatedas
Thenumberofparametersrequiredinequation[1.12]is P ,whichissignificantly smallerthanthatrequiredinequation[1.10].
Onemayfindasimilaritybetweenthetimedomainfilteringinequation[1.2]and theparameterizedvertexdomainfilteringinequation[1.11].Infact,iftheunderlying graphisacyclegraph,equation[1.11]coincideswithequation[1.2]withaproper definitionof Wp .However,theydonotcoincideingeneralcases:Itiseasily confirmedthatthesumofeachrowofthefiltercoefficientmatrixinequation[1.11] isnotconstantduetotheirregularnatureofthegraph,whereas k hk isaconstant intime-domainfiltering.Therefore,theparametersofequation[1.11]shouldbe determinedcarefully.
1.3.2. Spectraldomainfiltering
Thevertexdomainfilteringintroducedaboveintuitivelyparallelstime-domain filtering.However,ithasamajordrawbackinafrequencyperspective.Asmentioned insection1.2,time-domainfilteringandfrequencydomainfilteringareidenticalup totheDTFT.Unfortunately,ingeneral,suchasimplerelationshipdoesnotholdin GSP.Asaresult,thenaïveimplementationofthevertexdomainfiltering equation[1.10]doesnotalwayshaveadiagonalresponseinthegraphfrequency domain.Inotherwords,thefiltercoefficientmatrix H isnotalwaysdiagonalizable bytheGFTmatrix U,i.e. U HU isnotdiagonalingeneral.Therefore,thegraph frequencyresponseof H isnotalwaysclearwhenfilteringisperformedinthevertex domain.Thisisacleardifferencebetweenthefilteringofdiscrete-timesignalsand thatofthegraphsignals.
Asshowninequation[1.5],theconvolutionof hn and xn inthetimedomain isamultiplicationof ˆ h(ω ) and ˆ x(ω ) intheFourierdomain.Filteringinthegraph frequencydomainutilizessuchananalogtodefinegeneralizedconvolution(Shuman etal. 2016b):
where ˆ xi = ui , x isthe ithGFTcoefficientof x andtheGFTbasis ui isgivenbythe eigendecompositionofthechosengraphoperatorequation[I.2].Furthermore, ˆ h(λi ) isthegraphfrequencyresponseofthegraphfilter.Thefilteredsignalinthevertex domain, y [n],canbeeasilyobtainedbytransforming ˆ y backto y
where [ui ]n isthe nthelementof ui .Thisisequivalentlywritteninthematrixform as
where du isthedegreeofthevertex u.Severalotherimprovementsonthebound arealsofoundinliterature.AlthoughthegraphLaplaciansmentionedabovehave aboundofthelargesteigenvalue,suchboundsarenotapplicabletotheadjacency matrix.Consideringthis,appropriatecareofthegraphfrequencyrangemustbetaken whiledesigninggraphfilters.
Asmentioned,graphfrequencydomainfilteringisananalogofFourierdomain filtering.However,thisdoesnotmeanwealwaysobtainavertexdomainexpression ofthissimilartoequation[1.9].Hence,weneedtocomputetheGFToftheinput signal,whichraisesacomputationalissuedescribedasfollows.FortheGFT,the eigenvectormatrix U hastobecalculatedfromthegraphoperator.The eigendecompositionrequires O (N 3 ) complexityforadensematrix2.This
2Whilethecomputationcostforeigendecompositionofasparsematrixisgenerallylowerthan O (N 3 ),itstillrequiresahighcomputationalcomplexity,especiallyforlargegraphs.
Typically,graphspectralimageprocessingvectorizesimagepixels.Letusassume thatwehaveagrayscaleimageofsize W × H pixels.Itsvectorizedversionis x ∈ RWH anditscorrespondinggraphoperatorwouldbe RWH ×WH .Forexample, 4Kultra-high-definitionresolutioncorrespondsto W =3, 840 and H =2, 160, whichleadsto WH> 8 × 106 :thisistoolargetoperformeigendecomposition, evenforarecenthigh-speccomputer.Insection1.6,severalfastcomputation methodsofgraphspectralfilteringwillbediscussedtoalleviatethisproblem.
where η isthenormalizingfactorand φ(pi , pj ) isanon-negativefunction,which assignsalargeweightif pi and pj areclose.Thetypicalchoiceof φ(pi , pj ) willbe pi pj 1 .
Thematrix W issymmetric.Ifwechoose φ(pi , pi )=0,itsdiagonalelements wouldbecomezero,andasaresult, W couldbeviewedasanormalizedadjacency matrix.Thecoordinatesarethensmoothedbyagraphlow-passfilter,after computingtheGFTbasis U.Similarapproachestothismethodhavebeenusedin severalcomputergraphics/visiontasks(Zhang etal. 2010;ValletandLévy2008; Desbrun etal. 1999;Fleishman etal. 2003;KimandRossignac2005).
Inthegraphsetting,weneedtodefinepixel-wiseorpatch-wiserelationshipsasa distancebetweenpixelsorpatches,anditisusedtoconstructagraph.Thefollowing distancesareoftenconsidered(Milanfar2013b),where i and j arepixelorpatch indicesand φ(·) issomenonnegativefunction:
1)Geometricdistance: dg (i,j )= φ( pi pj ),where pi isthe ithpixel coordinate.
Supposethatthefiltercoefficientsaredeterminedbasedontheabovefeatures,and thattheyaresymmetric,i.e.theoutputpixelvalue yi isrepresentedas
Here, dk ( , ) isoneofthedistancemetricsmentionedearlierand K isthe numberoffeaturesweconsidered.Thescalingfactor Di normalizesthefilterweights as Di = j Wi,j .Forexample,thebilateralfilteris K =2 for dg ( , ) and dp ( , ).
TheFourierdomainrepresentationofsuchpixel-dependentfilterscannotbe calculatedinaclassicalsensebecauseitisnolongershift-invariant:thefiltermatrix W cannotbediagonalizedbytheDFTmatrix.Incontrast,GSPprovidesa frequency-likenotioninthegraphfrequencydomain.Ingeneral,theweightmatrix W inequation[1.24]canberegardedasanadjacencymatrixbecauseall dk ( , ) are assumedtobedistancesbetweenpixels.Supposethatthereisnoself-loopin W ,for simplicity.Ingeneral,thesmoothedimageinequation[1.24]isrepresentedinthe followingmatrixform:
= D 1 Wx
where D = diag(D0 ,...,DN 1 ).Thiscanberewrittenbyusingtherelationship L = D W as(Gadde etal.2013):
where x := D1/2 x isadegree-normalizedsignal.Letusdenotethe eigendecompositionof Ln as Ln := UΛU .Theabovefilteringinequation[1.29] isfurtherrewrittenas:
= U(I Λ)U x
= U ˆ h(Λ)U x [1.31] = h(Ln )x, [1.32]
where y := D1/2 y andthegraphspectralfilterisdefinedas ˆ h(λ):=1 λ.Moreover, λ ∈ [0, 2] forthesymmetricnormalizedgraphLaplacian;therefore,itactsasalinear decaylow-passfilterinthegraphfrequencydomain.
Thisgraphspectralrepresentationofapixel-dependentfiltersuggeststhat the pixel-dependentfilter W implicitlyandsimultaneouslydesignstheunderlyinggraph (andtherefore,theGFTbasis)andthespectralresponseofthegraphfilter.Inother words,theGSPexpressionofthepixel-dependentfilterisfreetodesignthespectral response ˆ h(λ),apartfromthelineardecayone,oncewedetermine W .Forexample, letusconsiderthefollowingspectralresponse:
