Solution Manual For Image Analysis, Classification and Change Detection in Remote Sensing With Algor

Morton John Canty mort.canty@gmail.com

Image Analysis, Classification and Change Detection in Remote Sensing Fifth Revised Edition

Solutions to Selected Exercises

CRC PRESS Boca Raton Ann Arbor

London

Tokyo

Contents

1 Images, Arrays and Matrices

1

2 Image Statistics

7

3 Transformations

21

4 Convolutions, Filters and Fields

29

5 Image Enhancement and Correction

35

6 Supervised Classification Part 1

39

7 Supervised Classification Part 2

45

8 Unsupervised Classification

49

9 Change Detection

55

i

1 Images, Arrays and Matrices

Exercise 1 The components of x and y are x1 = ∥x∥ cos θx ,

x2 = ∥x∥ sin θx

y1 = ∥y∥ cos θy ,

y2 = ∥y∥ sin θy .

The angle between the two vectors is θx − θy . The inner product is therefore x⊤ y = ∥x∥∥y∥ cos(θx − θy ) = ∥x∥∥y∥(cos θx cos θy + sin θx sin θy ) = x1 y1 + x2 y2 .

Exercise 2 The outer product of x and y is ⊤

xy =

x1 y1 x2 y1

x1 y2 x2 y2

,

the determinant of which is |xy ⊤ | = x1 y1 x2 y2 − x1 y2 x2 y1 = 0.

We can express the second column as in terms of the first:

x1 x1 y2 = cy1 x2 x2 by choosing c = y2 /y1 , so thr rank is 1.

Exercise 3 Verifying the identity with random 2 × 2 matrices: import numpy as np A = np . mat ( np . random . rand (2 ,2)) B = np . mat ( np . random . rand (2 ,2)) print ( A * B ). T

1

2

Images, Arrays and Matrices print B . T * A . T [[0.58298781 [0.26627182 [[0.58298781 [0.26627182

0.40052227] 0.18432174]] 0.40052227] 0.18432174]]

Exercise 4 Let the colinear vectors be x, y and z. Then clearly the difference vectors z − x and y − x point in the same direction. Therefore z − x = ρ(y − x) or (1 − ρ)x + ρy + (−1)z = 0 for some value of ρ and the vectors are linearly dependent.

Exercise 5 The determinant of

A=

a11 a12

a12 a22

is |A| = a11 a22 − a212 .

The eigenvalues are

q 1 λ(1) = a11 + a22 + (a11 + a22 )2 − 4(a11 a22 − a212 ) 2

q 1 λ(2) = a11 + a22 − (a11 + a22 )2 − 4(a11 a22 − a212 ) .

2 with product λ(1) λ(2) =

1 ((a11 + a22 )2 − ((a11 + a22 )2 − 4(a11 a22 − a212 ))) = |A|.

4

Exercise 6 Let A be symmetric. Its inverse satisfies A−1 A = I. Transposing both sides: A⊤ (A−1 )⊤ = I. But A⊤ = A and therefore A(A−1 )⊤ = I or (A−1 )⊤ = A−1 , so A−1 is also symmetric.

3

Exercise 7 Let the eigenvalues and eigenvectors of A be λk and uk , respectively, for k = 1, 2. Then Auk = λk uk .

Multiply both sides with A−1 . Then uk = λk A−1 uk or A−1 uk = λ−1 k uk .

Exercise 8 Starting from y = x⊤ Ax, we can write xyx⊤ = xx⊤ Axx⊤ , and therefore, since y is scalar, tr(xx⊤ )y = tr(xx⊤ Axx⊤ ).

With the second of Equations (1.30), tr(xx⊤ )y = tr(Axx⊤ xx⊤ ).

But since (x⊤ x) is also scalar, tr(xx⊤ )y = tr(Axx⊤ )(x⊤ x).

The identity then follows from tr(xx⊤ ) = (x⊤ x).

Exercise 9 Let A be a Hermitian matrix, λ be an eigenvalue of A, and v be the corresponding eigenvector. We’ll prove that λ is real.

1. By definition of an eigenvector: Av = λv 2. Take the conjugate transpose of both sides: (Av)H = (λv)H v H AH = λ ∗ v H Where λ∗ denotes the complex conjugate of λ.

4

Images, Arrays and Matrices 3. Since A is Hermitian, we know that AH = A

. Substituting this: v H A = λ∗ v H 4. Now, multiply both sides of the original equation by v H from the left: v H Av = v H λv = λv H v 5. The left side of this equation is equal to λ∗ v H v (from step 3), so we have: λ∗ v H v = λv H v 6. v H v is a positive real number (it’s the dot product of v with itself), so we can divide both sides by it: λ∗ = λ 7. For a complex number to be equal to its own conjugate, it must be real.

Exercise 10 Let the eigenvectors of A be u1 and u2 . Then ⊤ u⊤ 2 Au1 = λ1 u2 u1 ⊤ u⊤ 1 Au2 = λ2 u1 u2 . ⊤ Subtracting gives, since u⊤ 2 u1 = u1 u2 , ⊤ ⊤ (λ1 − λ2 )u⊤ 1 u2 = u2 Au1 − u1 Au2 = 0,

since A is symmetric. Hence u1 and u2 are orthogonal.

Exercise 11 For a 2 × 2 symmetric matrix A, Equation (1.45) can be written

⊤ λ1 0 u1 A = (u1 , u2 ) .

0 λ2 u⊤ 2 Multiplying this out gives Equation (1.46).

Exercise 12 With Equation (1.46),

X X X X X (i) (i) λi uj uj = λi u(i)⊤ u(i) = λi .

Tr(A) = λi Tr u(i) u(i)⊤ = i

i

j

i

i

5

FIGURE 1.1 Plot of f (x) = x21 − x22 .

Exercise 13 ∂ ∂y

1 ⊤ x Ay

=−

1

∂

(x⊤ Ay)2 ∂y

(x⊤ Ay) = −

Exercise 14 This calculates the right eigenvectors: import numpy as np A = [[1 ,2 ,3] ,[4 ,5 ,6] ,[7 ,8 ,9]] print np . linalg . eig ( A )[1] [[ -0.23197069 -0.78583024 0.40824829] [ -0.52532209 -0.08675134 -0.81649658] [ -0.8186735 0.61232756 0.40824829]]

Exercise 15 The following code plots the function, see Figure 1.1. from mpl_toolkits . mplot3d import Axes3D import matplotlib . pyplot as plt import numpy as np fig = plt . figure () ax = fig . gca ( projection = ’3 d ’) X1 = range ( -10 ,10)

1 (x⊤ Ay)2

A⊤ x.

6

Images, Arrays and Matrices X2 = range ( -10 ,10) X1 , X2 = np . meshgrid ( X1 , X2 ) Z = X1 **2 - X2 **2 surf = ax . plot_surface ( X1 , X2 , Z )

The Lagrange function is L = x21 − x22 + λ(x21 + x22 − 1).

Differentiating: ∂L = 2x1 + 2λx1 = 0 ⇒ x1 (1 + λ) = 0 ∂x1 ∂L = −2x2 + 2λx2 = 0 ⇒ x2 (−1 + λ) = 0 ∂x2 ∂L = x21 + x22 = 0. ∂λ Possible values for λ are therefore λ = 1 and λ = −1. If λ = 1, then necessarily x1 = 0 and therefore x2 = ±1. Similarly if λ = −1 then x2 = 0, x1 = ±1.

It is clear from the plot that the constrained maxima are at (±1, 0) and the minima at (0, ±1).

2 Image Statistics

Exercise 1 From the definition of variance, var(G) = ⟨(G − ⟨G⟩)2 ⟩ = ⟨G2 − 2G⟨G⟩ + ⟨G⟩2 ⟩ = ⟨G2 ⟩ − 2⟨G⟩2 + ⟨G⟩2 = ⟨G2 ⟩ − ⟨G⟩2 .

Using this result, var(a0 + a1 G) = var(a1 G) = ⟨a21 G2 ⟩ − ⟨a1 G⟩2 = a21 (⟨G2 ⟩ − ⟨G⟩2 ) = a21 var(G).

Exercise 2 Let Y = |X| where X has the distribution Φ(x). For y > 0 we have P (y) = Pr(Y ≤ y) = Pr(|X| ≤ y) = Pr(−y ≤ X ≤ y) = Φ(y) − Φ(−y).

Hence

d P (y) = ϕ(y) − ϕ(−y) = 2ϕ(y), dy and clearly for y ≤ 0, p(y) = 0. p(y) =

Exercise 3 (a) The chi square distribution is 1 Pχ2 ;m (z) = m/2 2 Γ(m/2)

Z z

x(m−2)/2 ex/2 dx.

0

Let x → 2t, Z z/2 1 Pχ2 ;m (z) = m/2 (2t)(m−2)/2 e−t 2dt 2 Γ(m/2) 0 Z z/2 1 = tm/2−1 e−t dt = γ(m/2, z/2). Γ(m/2) 0 7

8

Image Statistics (b) In the GEE Python API, import ee def chi2cdf ( chi2 , df ): ’’’ Chi square cumulative distribution function ’’’ return ee . Image ( chi2 . divide (2)). \ gammainc ( ee . Number ( df ). divide (2))

Exercise 4 Let X be standard normally distributed, i.e. with density function 2 1 p(x) = ϕ(x) = √ e−x /2 .

2π

We can’t apply Theorem 2.1 directly to get the density function for X 2 because the function u(x) = x2 is not monotonic wherever p(x) ̸= 0. But consider the random variable Y = Z 2 , where Z = |X|.

We saw in Exercise 2 that Z has the density function n f (z) = 2ϕ(z) for z > 0 0 elsewhere.

The function y = u(z) = z 2 is monotonic wherever f (z) > 0, so Theorem 2.1 holds. Inverting, z = w(y) = y 1/2 . So for y > 0 g(y) = f (y 1/2 )

1 −1/2 2 1 1 y = √ e−y/2 y −1/2 = √ y −1/2 e−y/2 , 2 2 2π 2π

which is the chi-square distribution, Equation (2.38), for m = 1.

Exercise 5 Given X1 and X2 independent and ∼ N (0, 1), we want the distribution of Y = X1 + X2 . The joint density function for the random vector X = (X1 , X2 )⊤ is, from independence, 2 2 1 1 −x21 /2−x22 /2 1 e f (x1 , x2 ) = √ e−x1 /2 √ e−x2 /2 = 2π 2π 2π

or

1 −(x1 +x2 )2 /2+x1 x2 e .

2π For fixed X1 , we can write x2 = w(y) = y − x1 . Applying Theorem 2.1, Y has the density f (x1 , x2 ) =

g(y, x1 ) = f (x1 , y − x1 )

∂w 1 −y2 /2+(y−x1 )x1 = e . ∂y 2π

9 The complete density for Y is then obtained by integrating over x1 . Dropping the subscript on x1 , this is Z Z ∞ 1 −y2 /2 ∞ (y−x)x e e p(y) = g(y, x)dx = dx.

2π −∞ −∞ The integral is Z ∞ Z ∞ Z ∞ 2 xy−x2 +y 2 /4−y 2 /4 y 2 /4 (y−x)x e−(x−y/2) dx. e dx = e e dx = Let u =

√

−∞

−∞

−∞

2(x − y/2). Then the last expression above is Z ∞ √ 2 2 2 1 ey /4 e−u /2 √ du = ey /4 π.

2 −∞

Therefore p(y) = √

2 1 √ e−y /4 , 2π 2

that is, Y ∼ N (0, 2).

Exercise 6 We have, with Theorem 2.3, * ⟨Z̄⟩ = and

var(Z̄) =

1 m

m

1 X Zi m i=1

2 var

m X i=1

+ =

1 mµ = µ, m

!

Zi

=

1 1 mσ 2 = σ 2 .

2 m m

Exercise 7 The gamma function is Z ∞ Γ(α) =

xα−1 e−x dx.

0

R R Integrating by parts ( vdu = uv − udv) with v = xα−1 , we have

du = e−x dx ⇒ u = −e−x ,

Z ∞ ∞ Γ(α) = −e−x xα−1 + e−x (α − 1)xα−2 dx 0 0 Z ∞ α−2 −x = (α − 1) x e = (α − 1)Γ(α − 1).

0

10

Image Statistics

Exercise 8 (a) From Equation (2.33) Z ∞ ⟨X⟩ =

x 0

1 xα−1 e−x/β dx. β α Γ(α)

Let y = x/β. Then Z ∞

1 β α−1 y α−1 e−y βdy βy α β Γ(α) 0 Z ∞ 1 1 =β y α e−y dy = β Γ(α + 1) = αβ. Γ(α) 0 Γ(α)

⟨X⟩ =

Similarly 2

Z ∞

1 β α−1 y α−1 e−y βdy (βy)2 α β Γ(α) 0 Z ∞ 1 = β2 y α+1 e−y dy Γ(α) 0 1 1 Γ(α + 2) = β 2 (α + 1)Γ(α + 1) = α(α + 1)β 2 . = β2 Γ(α) Γ(α)

⟨X ⟩ =

Therefore var(X 2 ) = ⟨X 2 ⟩ − ⟨X⟩2 = α(α + 1)β 2 − (αβ)2 = αβ 2 . (b) For exponentially distributed Z1 and Z2 , let Z = Z1 + Z2 . Then the distribution function for Z is Z z Z z−z2 1 z1 /β z2 /β P (z) = Pr(Z1 + Z2 < z) = e e dz1 dz2 β2 0 0 Z z Z z−z2 1 = 2 ez2 /β ez1 /β dz1 dz2 β 0 0

Z z z−z2 1 z2 /β z1 /β dz2 = 2 e −βe β 0 0 Z z h i 1 = 2 ez2 /β −βe−z/β ez2 /β + β dz2 β 0 Z zh i 1 = 2 −βe−z/β + βe−z2 /β dz2 β 0 h iz i 1 h = 2 −βze−z/β + β −βez2 /β β 0 h ii 1 h = 2 −βze−z/β + β −β(e−z/β − 1) β 1 = − ze−z/β − e−z/β + 1. β

11 Differentiating,

d 1 −z/β z −z/β 1 P (z) = − e − e + e−z/β dz β β β 1 z ze−z/β = 2 e−z/β = 2 β β Γ(2)

p(z) =

since Γ(2) = 1.

Exercise 9 From the inverse transformations x = su, y = s(1 − u) we get with Theorem 2.2 the Jacobian J=

∂x ∂s ∂y ∂s

∂x ∂u ∂y ∂u

=

u s = −s 1 − u −s

and |J| = s. Therefore the joint density function of S and U is p(s, u) =

1 β −m−n (su)n−1 (s(1 − u))m−1 e−su/β e−s(1−u)/β s. Γ(m)Γ(n)

This can be manipulated into the form p(s, u) =

Γ(m + n) n−1 1 1 u (1 − u)m−1 sn+m−1 e−sβ . Γ(m)Γ(n) Γ(m + n) β m+n

This is just the product of the beta and gamma density functions. We get the marginal distributions for s by integrating over u, so the random variable S is (m + n, β)-gamma distributed. Similarly U is (m, n)-beta distributed.

Since the joint probability density function of S and U is the product of the separate density functions, they are independent.

Exercise 10 For two dimensions, the covariance is cov(a⊤ G, b⊤ G) = cov(a1 G1 + a2 G2 , b1 G1 + b2 G2 ) = a1 b1 var(G1 ) + a1 b2 cov(G1 G2 ) + a2 b1 cov(G2 , G1 ) + a2 b2 var(G2 ), which by inspection is the same as

var(G1 ) cov(G1 , G2 ) b1 (a1 , a2 ) = a⊤ Σb. cov(G2 , G1 ) var(G2 ) b2 More generally, cov(a⊤ G, b⊤ G) = ⟨a⊤ G(b⊤ G)⊤ ⟩ = a⊤ ⟨GG⊤ ⟩b = a⊤ Σb.

12

Image Statistics

Exercise 11 For Σ = σ 2 I, |Σ| = (σ 2 )N , and the multivariate density simplifies as follows

1 1 ⊤ −1 p exp − (g − µ) Σ (g − µ) p(g) = 2 (2π)N/2 |Σ|

1 exp −∥g − µ∥2 /2σ 2 = √ N ( 2πσ) =

N Y

√

i=1

1 exp −(gi − µi )2 /2σ 2 .

2πσ

For N = 1, therefore, p(g) = √

1 exp −(g − µ)2 /2σ 2 .

2πσ

The mean of G is given by 1 ⟨G⟩ = √ 2πσ

Z ∞

g−µ

2

g · e− 2 ( σ ) dg 1

−∞

which we can write in the form Z ∞ Z ∞ 2 1 g−µ 2 1 µ g − µ − 21 ( g−µ ) σ ⟨G⟩ = √ dg + √ e e− 2 ( σ ) dg. σ 2π −∞ 2πσ −∞ R∞ The second term is just µ −∞ p(g)dg = µ. The first term vanishes. That is, making the change of variable substitution

1 g−µ y= , 2 σ we have

Z ∞ µ

and similarly

2 g − µ − 21 ( g−µ σ ) dg = σ e σ

Z ∞

e−y dy = σ

0

Z µ

2 g − µ − 21 ( g−µ σ ) dg = −σ. e σ −∞

Thus ⟨G⟩ = µ.

Exercise 12 The covariance matrix estimate can be written as s=

m m 1 X · (z i − z̄)(z i − z̄)⊤ . m − 1 m i=1

13 Expanding, " # m 1 X 1 ⊤ ⊤ ⊤ s= z i z 1 − 2z̄z̄ + · mz̄z̄ m−1 m i m

m 1 ⊤ ⊤ = Z Z − z̄z̄ m−1 m With Equation (2.52) we can write

1 ⊤ m 1 ⊤ ⊤ s= Z Z − 2 Z 1m 1m Z m−1 m m

1 1 ⊤ ⊤ ⊤ = Z Z − Z 1m 1m Z m−1 m

1 1 1 ⊤ ⊤ = Z I mm − 1m 1m Z = ZHZ. m−1 m m−1 Clearly, H ⊤ = H, so H is symmetric. Also

1 1 ⊤ 1 1 HH = I mm − 1m 1⊤ I − m m mm m m m 1 1 1 ⊤ 1m 1⊤ = I mm − 1m 1⊤ 1m 1⊤ m− m+ m 1m 1m m m m2 1 = I mm − 1m 1⊤ m = H, m since 1⊤ m 1m = m. Let x be any m-component vector. Then with the above result 1 1 x⊤ sx = x⊤ Z ⊤ HHZx = y ⊤ y ≥ 0, m−1 m−1 where y = HZx. Therefore s is positive semi-definite.

Exercise 13 With si =

√

sii Equation (2.58) is 

1/s1  d−1/2 sd−1/2 =  ...

0   =

··· .. . ···

s11 /s21

.. . sm1 /sm s1

0 .. .

1/sm ··· .. . ···



s11   ..  . sm1

··· .. . ··· 

  1/s s1m 1 ..   ..  . . smm

s1m /s1 sm  ..  = r. . smm /s2m

0

··· .. . ···

0 .. .

1/sm

  

14

Image Statistics

Exercise 14 Let Ai represent the situation “auto is behind door i”. The a priori probabilities are Pr(Ai ) = 1/3, i = 1, 2, 3. Let Oi be the observation “quizmaster opens door i”. Suppose the contestant chooses door 2 and the quizmaster opens door 1. Then we have the following conditional probabilities: Pr(O1 | A1 ) = 0

quizmaster won’t give the car away.

Pr(O1 | A2 ) = 1/2

quizmaster is indifferent.

Pr(O1 | A3 ] = 1

quizmaster has no choice.

Now apply Bayes’ Theorem to find the a posteriori probability that the auto is behind door 2, given the observation: Pr(O1 | A2 )Pr(A2 ) Pr(O1 | A2 )Pr(A2 ) + Pr(O1 | A3 )Pr(A3 ) (1/2)(1/3) = 1/3, = (1/2)(1/3) + (1)(1/3)

Pr(A2 | O1 ) =

whereas, with the same argument, Pr(A3 | O1 ) = 2/3.

The contestant would therefore be well-advised to switch to door 3.

Exercise 15 From the monotonicity of F0 , Pr(Q ≥ q | H0 ) = Pr(F0 (Q) ≥ F0 (q)) = 1 − Pr(F0 (Q) < F0 (q)).

Therefore, Pr(F0 (Q) < F0 (q)) = 1 − Pr(Q ≥ q | H0 ) = Pr(Q < q | H0 ) = F0 (q).

Exercise 16 import numpy as np import scipy . stats as stats s1 = np . random . randn (100) s2 = np . random . randn (100) stat , p = stats . ttest_ind ( s1 , s2 ) print p stat , p = stats . bartlett ( s1 , s2 ) print p 0.35275438625619526 0.42889408182589617

15

Exercise 17 X L(µ0 ) = exp − (z(ν) − µ0 )2 /2σ 2 L(µ̂) ν

!,

! exp −

X

2

(z(ν) − z̄) /2σ

2

ν

1 X (z(ν) − µ0 )2 − (z(ν) − z̄)2 = exp − 2 2σ ν

1 = exp − 2 (z̄ − µ0 )2 ≤ k 2σ /m

!

Exercise 18 Differentiating Equation (2.88): ∂z 2 X =− 2 (y(ν) − a − bx(ν)) ∂a σ ν 2 X ∂z x(ν)(y(ν) − a − bx(ν)). =− 2 0= ∂b σ ν

0=

From the first equation we get immediately a = ȳ − bx̄, where x̄ and ȳ are given by Equation (2.90). Substituting for a in the second equation, we have X X X 0= x(ν)y(ν) − (ȳ − bx̄) x(ν) − b x(ν)2 . ν

ν

ν

Re-arranging: ! b

X

x(ν)2 − x̄

ν

X

x(ν)

=

ν

X ν

x(ν)y(ν) − ȳ

X

x(ν).

ν

The expression in brackets on the left is just X X x(ν)2 − mx̄2 = (x(ν) − x̄)2 = msxx , ν

ν

as can easily be seen by expansion. Similarly the right hand side is X (x(ν) − x̄)(y(ν) − ȳ) = msxy . ν

Hence b=

sxy . sxx

16

Image Statistics

FIGURE 2.1 Regression of band 2 on band 1.

To show that these values minimize Equation (2.88), we require the Hessian matrix: ∂2z 2m = 2 ∂a2 σ 2 X ∂2z = 2 x(ν)2 2 ∂b σ ν ∂2z ∂2z 2 X = = 2 x(ν) ∂b∂a ∂a∂b σ ν The Hessian matrix is thus H=

2m σ2

1 x̄

x̄ sxx + x̄2

.

For any vector y ̸= 0 we then have y ⊤ Hy = (y1 + x̄y2 )2 + sxx y22 > 0, so H is positive definite and (a, b) is a minimum of Equation (2.88).

17

Exercise 19 We require the derivative of a with respect to yi , where a = ȳ − bx̄ =

sxy 1 X y(ν) − x̄. m ν sxx

We have ∂a 1 x̄ ∂sxy = − ∂y(ν) m sxx ∂y(ν) and   X 1  ∂ ȳ  ∂sxy = x(ν) − x̄ − (xj − x̄) ∂y(ν) m ∂y(ν) j =

1 (x(ν) − x̄). m

Hence ∂a 1 = ∂y(ν) m

1−

x̄ (x(ν) − x̄) sxx

and

∂a ∂y(ν)

2 =

1 m2

2x̄ x̄2 1− (x(ν) − x̄) + 2 (x(ν) − x̄)2 . sxx sxx

Summing over i gives

X ∂a 2 1 x̄2 = 2 m + 0 + 2 msxx ∂y(ν) m sxx ν P

x(ν)2 1 sxx + x̄2 = ν2 , = m sxx m sxx from which the expression for σa2 in Equation (2.91) readily follows.

Exercise 20 From Equation (2.98), z(w) =

1 ⊤ (y y − y ⊤ X w − w⊤ X y + w⊤ X ⊤ X w), σ2

so that ∂z(w) 1 = 0 = 2 (−X ⊤ y − X ⊤ y + 2X ⊤ X w) ∂w σ from which Equation (2.99) follows.

18

Image Statistics

Exercise 21 import numpy as np from osgeo import gdal from osgeo . gdalconst import GA_ReadOnly import matplotlib . pyplot as plt import scipy . stats as stats gdal . AllRegister () inDataset = gdal . Open ( ’ imagery / AST_20010409 ’ , GA_ReadOnly ) cols = inDataset . RasterXSize rows = inDataset . RasterYSize bands = inDataset . RasterCount image = np . zeros (( bands , rows * cols )) f o r b in range ( bands ): band = inDataset . GetRasterBand ( b +1) image [b ,:]= band . ReadAsArray (0 ,0 , cols , rows ). ravel () inDataset = None b , a ,_ ,_ , _ = stats . linregress ( image [0 ,:] , image [1 ,:]) plt . plot ( image [0 ,:] , image [1 ,:] , ’. ’ , [50 ,250] , [ a + b *50 , a + b *250] , ’r ’)

Exercise 22 The numerator of the right hand side of Equation (2.94) becomes X

(y(ν) + ŷ(ν) − ŷ(ν) − ȳ)(ŷ(ν) − ȳ)

ν

=

X

=

X

(y(ν) − ŷ(ν))(ŷ(ν) − ȳ) +

ν

X

(ŷ(ν) − ȳ)2

ν 2

(ŷ(ν) − ȳ) ,

ν

P since the term ν (y(ν)− ŷ(ν))(ŷ(ν)− ȳ) vanishes, see Section 2.6.2. Including the denominator then gives the desired result.

Exercise 23 From Equations (2.107) and (2.108), ŵ =

1 (X ⊤ y − X ⊤ X w) λ

Solving for ŵ gives Equation (2.109).

19

Exercise 24 We want to show that

Z −

p(x) ln

q(x) dx ≥ 0 p(x)

when q(x) ̸= p(x). From Jensen’s inequality, Equation (2.121), identify f (x) with the convex function − ln(x) and g(x) with the ratio q(x)/p(x) to get

Z

Z q(x) dx ≥ − ln q(x)dx . − p(x) ln p(x) But, since q(x) is also a probability density function, the integral on the right hand side is one, and its logarithm vanishes.

Exercise 25 This is simply because KL(p, q) ̸= KL(q, p), whereas a distance measure must be symmetric.

3 Transformations

Exercise 1 Substituting ĝ(−1) = −1/2i, ĝ(1) = 1/2i with ĝ(f ) = 0 elsewhere into (3.1), Z ∞ 1 1 ĝ(f )e−i2πf x dx = − e− i2πx + ei2πx 2i 2i −∞ 1 = (−(cos 2πx − i sin 2πx) + (cos 2πx + i sin 2πx)) 2i = sin(2πx).

Exercise 2 ′

Multiply Equation (3.5) by e−i2πk j and sum over j to get, with (3.7), X X X X ′ ′ e−i2πk j g(j) = ĝ(k) ei2π(k−k )j/c = ĝ(k)cδk,k′ = cĝ(k ′ ), j

k

j

k

which gives Equation (3.6).

Exercise 3 From Equation (3.5) we have the equations j=0:

2 = ĝ(0) + ĝ(1) + ĝ(2) + ĝ(3)

j=1:

4 = ĝ(0) + iĝ(1) − ĝ(2) − iĝ(3)

j=2:

6 = ĝ(0) − ĝ(1) + ĝ(2) − ĝ(3)

j=3:

8 = ĝ(0) − iĝ(1) − ĝ(2) + iĝ(3)

which we can solve by substitution to get ĝ(0) = 5, ĝ(1) = −1 + i, ĝ(2) = −1, ĝ(3) = −1 − i.

In Python: >>> import numpy . fft as fft >>> fft . fft ([2 ,4 ,6 ,8])/4 array ([ 5.+0. j , -1.+1. j , -1.+0. j , -1. -1. j ])

21

22

Transformations

Exercise 4 From Equation (3.8) g(i, j) exp (i2π(ik0 /c + jℓ0 /r)) X = ĝ(k, ℓ) exp (i2π(ik/c + ik0 /c + jℓ/r + jℓ0 /r)) . k,ℓ

Let k ′ = k + k0 and ℓ′ = ℓ + ℓ0 . Then g(i, j) exp (i2π(ik0 /c + jℓ0 /r)) X = ĝ(k ′ − k0 , ℓ′ − ℓ0 ) exp (i2π(ik ′ /c + jℓ′ /r)) , k′ ,ℓ′

which is the translation property g(i, j) exp (i2π(ik0 /c + jℓ0 /r)) ⇔ ĝ(k ′ − k0 , ℓ′ − ℓ0 ).

Exercise 5 Starting with Equation (3.6) we have ′ 1 X g(j)g ∗ (j ′ )e−i2πk(j−j )/c , |ĝ(k)|2 = ĝ(k)ĝ ∗ (k) = 2 c ′ j,j

where the ∗ denotes the complex conjugate. Therefore, with Equation (3.7), X X ′ 1 X |ĝ(k)|2 = 2 g(j)g ∗ (j ′ ) e−i2πk(j−j )/c c ′ k

j,j

k

1 X 1X = 2 g(j)g ∗ (j ′ )cδj,j ′ = |g(j)|2 . c c j ′ j,j

Exercise 6 The standard Haar basis for V2 is {ϕ2,0 , ϕ2,1 , ϕ2,2 , ϕ2,3 }. According to the Orthogonal Decomposition Theorem, ϕ2,0 can be expanded as ϕ2,0 (x) =

⟨ϕ2,0 , ϕ1,0 ⟩ ⟨ϕ2,0 , ϕ1,1 ⟩ 1 ϕ1,0 (x) + ϕ1,1 (r) + r(x) = ϕ1,0 (x) + r(x), ⟨ϕ1,0 , ϕ1,0 ⟩ ⟨ϕ1,1 , ϕ1,1 ⟩ 2

where r(x) is in the residual space V1⊥ . Hence 1 1 1 r(x) = ϕ2,0 (x) − ϕ1,0 (x) = ϕ(4x) − (ϕ(4x) + ϕ(4x − 1)) = (ϕ(4x) − ϕ(4x − 1)) 2 2 2 1 = ψ1,0 (x).

2 Similarly the residuals for other V2 basis functions can be expressed in terms of ψ1,0 or ψ1,1 . Therefore {ψ1,0 , ψ1,1 } is an orthogonal basis for V1⊥ .

23

FIGURE 3.1 Cascade algorithm approximation to the B-spline scaling function.

Exercise 7 The dilation equation for ϕ(x − j) is ϕ(x − j) =

X

ck ϕ(2x − 2j − k),

k

so we have, for j ̸= 0,

0 = ⟨ϕ(x), ϕ(x − j)⟩ =

* X

+ ck ϕ(2x − k),

k

=

X

=

X

=

1 ck ck′ δk,k′ +2j 2 ′

1X ck ck−2j .

2 k

′

ck ϕ(2x − 2j − k )

k′

ck ck′ ⟨ϕ(2x − k)ϕ(2x − 2j − k ′ )⟩

k,k′

k,k

X

24

Transformations

Exercise 8 From the dilation equation, Z Z X Z X Z 1X du = ck ϕ(x)dx, ϕ(x)dx = ck ϕ(2x − k)dx = ck ϕ(u) 2 2 k

k

and since the integral is not zero, X

k

ck = 2.

k

Exercise 9 The modified program is (see Figure 3.1): def F (x ,i , c ): i f i ==0: i f x ==0: return 1.0 else : return 0.0 else : return c [0]* F (2* x ,i -1 , c )+ c [1]* F (2* x -1 ,i -1 , c ) \ + c [2]* F (2* x -2 ,i -1 , c )+ c [3]* F (2* x -3 ,i -1 , c ) \ + c [4]* F (2* x -4 ,i -1 , c ) # B - spline wavelet refinement coefficients c = np . zeros (5) c [0]=1./8; c [4]=1./8; c [1]=1./2; c [3]=1./2; c [2]=3./4 # fourth order approximation n = 4 x = np . array ( range (4*2** n ))/ f l o a t (2** n ) FF = np . zeros (4*2** n ) f o r i in range (4*2** n ): FF [ i ] = F ( x [ i ] ,n , c ) plt . plot (x , FF ) plt . ylim (0 ,1)

Exercise 10 (a) This can be seen from the cascade algorithm, which approximates a scaling function to any degree of accuracy: ϕ(x) = limi→∞ fi (x), where f0 (x) = δx,0 fi (x) = c0 fi−1 (2x) + c1 fi−1 (2x − 1) + . . . + cn fi−1 (2x − n).

25 Clearly no iteration will lead to a value of f (x) ̸= 0 for x < 0. For i = 0, f0 (x) ̸= 0 only for x = 0. For i = 1 the largest value at which f1 (x) can be i . non-vanishing is x = n/2, for i = 2 it is x = 3n/4, and for any i it is (2 −1)n 2i Hence the scaling function vanishes outside the interval [0, n]. (b) We know from the above that the scaling function ϕ(x), which appears on the left hand side of the dilation Equation (3.25), is supported on a finite interval [a, b] included within [0, n]. The shifted functions ϕ(2x − k) on the right hand side are supported on [(a + k)/2, (b + k)/2] and the k ranges from 0 to n. Equating the intervals for left and right hand sides,

a b+n , [a, b] = , 2 2 leads to the solution a = 0, b = n.

Exercise 11 After line 40 insert the code G_r = reshape ( G_r ,( rows , cols , r )) outfile , fmt = auxil . select_outfilefmt () driver = gdal . GetDriverByName ( fmt ) outDataset = driver . Create ( ’ imagery / recon ’ , cols , rows , bands , GDT_Float32 ) projection = inDataset . GetProjection () geotransform = inDataset . GetGeoTransform () i f geotransform i s not None : outDataset . SetGeoTransform ( geotransform ) i f projection i s not None : outDataset . SetProjection ( projection ) f o r k in range ( r ): outBand = outDataset . GetRasterBand ( k +1) outBand . WriteArray ( G_r [: ,: , k ] ,0 ,0) outBand . FlushCache () outDataset = None

Exercise 12 The SVD of the m × N data matrix G is G = U W V ⊤, Then

U ⊤ U = V ⊤ V = I, W is diagonal.

(m − 1)Σ = G ⊤ G = V W U ⊤ U W V ⊤ = V W 2V ⊤

or V ⊤ ΣV =

1 W 2 is diagonal, m−1

26

Transformations

so V is the matrix of eigenvectors and the eigenvalues are proportional to the squares of the diagonal elements of W .

Exercise 13 (a) The difference of the class means of the projections v is v̄1 − v̄2 =

1 X ⊤ 1 X ⊤ w g− w g = w⊤ (m1 − m2 ). n1 n2 g ∈B1 g ∈B2

Hence (v̄1 − v̄2 )2 = ∥w⊤ (m1 − m2 )∥2 = ∥(m1 − m2 )⊤ w∥2 = w⊤ C B w and, since the overall variance is var(v) = w⊤ C 1 w + w⊤ C 2 w, we get Equation (3.81). (b) By differentiating J(w) with respect to w and setting the result equal to zero, we get the generalized eigenvalue problem C B w = λ(C 1 + C 2 )w.

But C B w = (m1 − m2 )[(m1 − m2 )⊤ w] ∝ (m1 − m2 ) from which the result follows. (c) The following script calculates and plots Fisher’s linear discriminant for the simulated data: % matplotlib inline import numpy as np import matplotlib . pyplot as plt # generate a 2 - band image with 2 classes n1 = np . random . randn (1000) n2 = n1 + np . random . randn (1000) B1 = np . stack (( n1 , n2 ) ,1) B2 = np . stack (( n1 +4 , n2 ) ,1) image = np . zeros ((2000 ,2)) image [:1000 ,:] = B1 image [1000: ,:] = B2 center_x = np . mean ( image [: ,0]) # determine direction W m1 = np . mat ([ np . mean ( n1 ) , np . mean ( n2 )]) m2 = np . mat ([ np . mean ( n1 )+4 , np . mean ( n2 )]) C1 = np . mat ( np . cov ( B1 , rowvar = False )) C2 = np . mat ( np . cov ( B2 , rowvar = False ))

27 W = ( C1 + C2 ). I *( m1 - m2 ). T a = W [1 ,0]/ W [0 ,0] # scatterplot and Fischer discriminant plt . xlim (( -8 ,8)) plt . ylim (( -8 ,8)) plt . axes (). set_aspect (1) plt . scatter ( image [: ,0] , image [: ,1]) plt . plot ([ center_x -10 , center_x +10] ,[ -10* a ,10* a ] , ’r ’)

Exercise 14 The code is available from the author on request.

Exercise 15 We have w = G ⊤ α = λ−1/2 G ⊤ v. Therefore var(w⊤ g) = w⊤ Cw = w⊤ G ⊤ Gw/(m − 1) = λ−1 v ⊤ GG ⊤ GG ⊤ v/(m − 1) = λ−1 λ2 v ⊤ v/(m − 1) = λ/(m − 1).

Exercise 16 Primal formulation: with Equation (3.54), we maximize a⊤ G ⊤ Ga −1 a⊤ G ⊤ N GN a with respect to a. This leads to the generalized eigenvalue problem ⊤ G⊤ N G N a = λG Ga.

Dual formulation: with a ∝ G ⊤ b, we maximize b⊤ GG ⊤ GG ⊤ b − 1. ⊤ b⊤ GG ⊤ N GN G b Using the notation K = GG ⊤ and KN = GG ⊤ N we get the generalized eigenvalue problem 2 KN K⊤ N b = λK b.

Since the rank of the m × m Gram matrix K (and hence of K2 ) is N < m, it is singular, so it is not positive definite. Therefore solution of the dual problem with Cholesky decomposition (see, e.g., IDL’s function CHOLDC()) is not possible.

4 Convolutions, Filters and Fields

Exercise 1 For f = g ∗ h we have f (0) =

2 X

h(ℓ)g(−ℓ) = h(0)g(0) + h(1)g(5) + h(2)g(4)

ℓ=0

f (1) =

2 X

h(ℓ)g(1 − ℓ) = h(0)g(1) + h(1)g(0) + h(2)g(5)

ℓ=0

.. . f (5) =

2 X

h(ℓ)g(5 − ℓ) = h(0)g(5) + h(1)g(4) + h(2)g(3).

ℓ=0

Exercise 2 % matplotlib inline import numpy as np import scipy . signal as signal from numpy import fft from osgeo import gdal from osgeo . gdalconst import GA_ReadOnly import matplotlib . pyplot as plt # get an image band gdal . AllRegister () infile = ’ imagery / AST_20070501 ’ inDataset = gdal . Open ( infile , GA_ReadOnly ) cols = inDataset . RasterXSize rows = inDataset . RasterYSize # pick out the middle row of pixels band = inDataset . GetRasterBand (3) G = band . ReadAsArray (0 , 0 , 200 ,200) # define a 3 x3 FIR kernel h = np . array ([[1 ,2 ,1] ,[2 ,4 ,2] ,[1 ,2 ,1]])/16.0 # convolve in the spatial domain Gs = signal . convolve2d (h , G ) hp = G *0

29

30

Convolutions, Filters and Fields hp [0:3 ,0:3] = h # convolve in the frequency domain Gf = fft . ifft2 ( fft . fft2 ( G )* fft . fft2 ( hp )) plt . imshow ( Gs ) plt . show () plt . imshow ( np . real ( Gf ))

Exercise 3 def butterworthfilter ( d0 ,n ,k , l ): dst = dist (k , l ) result = [] f o r d in dst : result . append ( 1./(1+( d / d0 )**(2* n )) ) return np . reshape ( np . array ( result ) ,(k , l ))

Exercise 4 The row of pixels f = (f0 , f1 . . . fc−1 ) defines the function fn (x) in Vn given by c−1 X fj ϕn,j (x) fn (x) = j=0

in terms of the standard basis. The basis functions are normalized and orthogonal: ⟨ϕn,j (x), ϕn,j ′ (x)⟩ = δj,j ′ .

Now project fn (x) onto the residual subspace Vn⊥ spanned by the basis functions ψn−1,k (x) = ψ(2n−1 x − k), where, from Equation (3.34), ψ(2n−1 x − k) =

X

(−1)k h1−k′ ϕ(2n x − 2k − k ′ ).

(4.1)

k′

(Due to normalization, the factor tions ϕn,k .) We get

√

2 has been absorbed into the basis func-

c/2−1

rn−1 (x) =

X

⟨fn (x), ψn−1,k (x)⟩ ψn−1,k (x)

k=0 c/2−1

=

X k=0

(Gf )k ψn−1,k (x),

31 where (Gf )k = ⟨fn (x), ψn−1,k (x)⟩ =

c−1 X

fj ⟨ϕ(2n x − j), ψn−1,k (x)⟩.

j=0

Substituting from Equation (4.1) above then gives

(Gf )k =

c−1 X

fj

c−1 X j=0

=

c−1 X

gk′ ⟨ϕ(2n x − j), ϕ(2n x − 2k − k ′ )⟩

k′

j=0

=

X

fj

X

gk′ δj,2k+k′

k′

fj gj−2k ,

j=0

which is Equation (4.13).

Exercise 5 (a) Consider the inner product of the first row of W with the first column of W ⊤ . With (3.35), h20 + h21 + h22 + h23 =

1 1 2 (c + c21 + c22 + c23 ) = · 2 = 1 2 0 2

and similarly for the other diagonal elements of the product. The inner product of the first row of W with the second column of W ⊤ is h0 h3 − h1 h2 + h2 h1 − h3 h0 = 0 and similarly for all other non-diagonal elements. (b) You will have to install the discrete wavelet package PyWavelets (see its documentation), then run the script % matplotlib inline import pywt import numpy as np import matplotlib . pyplot as plt a = np . zeros (1024 , dtype = np . float64 ) a [4] = 1.0 r = pywt . idwt (a , None , ’ db4 ’) plt . plot ( r [:10])

32

Convolutions, Filters and Fields

Exercise 7 To see this, let γ = 1/2 and expand ∥g − g ′ ∥2 = g ⊤ g + g ′⊤ g ′ − 2g ⊤ g ′ , so that

′⊤ ′ ⊤ g g g g exp(g ⊤ g ′ ) exp − . k(g, g ′ ) = exp − 2 2

Now expand the central factor in a Taylor series: 1 exp(g ⊤ g ′ ) = 1 + g ⊤ g ′ + (g ⊤ g ′ )2 + . . . .

2 Then the function in Equation (4.2) is of the form ϕ(g)⊤ ϕ(g ′ ), where ⊤ ⊤ g g g2 g 2 g1 g2 ϕ(g) = exp − 1, g1 . . . gn , √1 . . . √n , √ ... .

2 2 2 2

Exercise 8 def kernelMatrix (X , Y = None , gma = None , nscale =10 , bias =1 , degree =2 , kernel =0): i f Y i s None : Y = X i f kernel == 0: # linear kernel X = np . mat ( X ) Y = np . mat ( Y ) return ( X *( Y . T ) ,0) e l i f kernel == 1: # Gaussian kernel m = X [: ,0]. size n = Y [: ,0]. size onesm = np . mat ( np . ones ( m )) onesn = np . mat ( np . ones ( n )) K = np . mat ( np .sum( X *X , axis =1)). T * onesn K = K + onesm . T * np . mat ( np .sum( Y *Y , axis =1)) K = K - 2* np . mat ( X )* np . mat ( Y ). T i f gma i s None : scale = np .sum( np . sqrt ( abs ( K )))/( m **2 - m ) gma = 1/(2*( nscale * scale )**2) return ( np . exp ( - gma * K ) , gma ) else : # polynomial kernel X = np . mat ( X ) Y = np . mat ( Y ) i f gma i s None : scale = np .sum( np . sqrt ( abs ( K )))/( m **2 - m ) gma = 1/(2*( nscale * scale )**2) return (( gma * X * Y . T + bias )** degree , gma )

(4.2)

33

Exercise 9 For m = 10 and N = 5 a program to do this is pro solution4_9 G = randomu ( seed ,5 ,10) K = gausskernel_ matrix ( G ) print , ’ Average ␣ eiv ␣ pre - centering : ␣ ’+ $ strtrim ( total ( eigenql ( K ))/10 ,2) print , ’ Average ␣ eiv ␣ post - centering : ␣ ’+ $ strtrim ( total ( eigenql ( center ( K )))/10 ,2) K = polykernel_matrix ( G ) print , ’ Average ␣ eiv ␣ pre - centering : ␣ ’+ $ strtrim ( total ( eigenql ( K ))/10 ,2) print , ’ Average ␣ eiv ␣ post - centering : ␣ ’+ $ strtrim ( total ( eigenql ( center ( K )))/10 ,2) end

Exercise 10 First note that, since P ⊤ P = I r , we have B 1/2 B 1/2 = P Λ1/2 P ⊤ P Λ1/2 P ⊤ = P Λ1/2 Λ1/2 P ⊤ = P ΛP ⊤ = B. Moreover B −1/2 = (P Λ1/2 P ⊤ )−1 = P Λ−1/2 P ⊤ , where Λ−1/2 is an r × r diagonal matrix of the inverse square roots of the (nonzero) eigenvalues, so the inverse exists and Equation (4.42) can be written in the form (B −1/2 AB −1/2 )(B 1/2 x) = λ(B 1/2 x), a symmetric, ordinary eigenvalue problem.

Exercise 11 The answer is no, since the pixels in a clique must be mutual neighbors.

5 Image Enhancement and Correction

Exercise 1 See the code for the function lin2pcstr() in the auxil.auxil1.py module in the accompanying software.

Exercise 2 The decorrelation stretch can be implemented with a simple modification of the script pca.py. After the principal components of three bands have been determined in the variable pcs, stretch each component import auxil . auxil1 as auxil f o r k in range ( bands ): pcs [: , k ] = auxil . histeqstr ( pcs [: , k ])

and then proceed with the reconstruction, choosing all three of the components.

Exercise 3 The power spectrum of ∇1 is generated by import numpy as np from numpy import fft import matplotlib . pyplot as plt from matplotlib import cm import auxil . auxil1 as auxil # create filter g = np . zeros ((512 ,512) , dtype = f l o a t ) g [0:2 ,0:2] = np . array ([[1 ,0] ,[0 , -1]]) # a i j g

= = = =

shift Fourier transform to center np . reshape ( range (512**2) ,(512 ,512)) a % 512 a / 512 ( -1)**( i + j )* g

# compute power spectrum and display p = np . abs ( fft . fft2 ( g ))**2

35

36

Image Enhancement and Correction plt . imshow ( auxil . linstr ( p ) , cmap = cm . jet )

Exercise 5 The transformation generates the new coordinates x′ = αx,

y ′ = αy.

The centralized moment transforms to Z Z µ′pq = (x′ − x′c )p (y ′ − yc′ )q f ′ (x′ , y ′ )dx′ dy ′ .

But necessarily f ′ (x′ , y ′ ) = f (x, y), so we get Z Z ′ µpq = αp (x − xc )p αq (y − yc )q f (x, y)d(αx)d(αy) = αp+q+2 µpq .

Therefore ηpq is scale invariant: ′ ηpq =

µ′pq µ′00 (p+q)/2+1

=

αp+q+2 µpq (p+q)/2+1 (α2 )(p+q)/2+1 µ00

=

µpq = ηpq . (p+q)/2+1 µ00

Exercise 6 The noise reduction program is given in Listing ??. Pre- and post-reduction noise covariance matrices for the last three MNF bands of a LANDSAT TM image are given below.

Noise covariance matrix , f i l e [ Memory6 ] (600 x600x3 ) 1.0915236 -0.0063084596 -0.0029047414 -0.0063084596 1.0775507 -0.0031050553 -0.0029047414 -0.0031050553 1.0520645 Noise covariance matrix , f i l e [ Memory7 ] (600 x600x3 ) 0.28040335 -0.021618183 -0.0067004804 -0.021618183 0.22010893 -0.0018719031 -0.0067004804 -0.0018719031 0.17734220

Exercise 7 The renormalized panchromatic wavelet coefficients are of the form aC + b,

b = mms − ampan , a = σms /σpan .

Therefore, ⟨aC + b⟩ = a⟨C⟩ + b = ampan + mms − ampan = mms and var(aC + b) = a2 var(C) =

2 σms 2 2 σpan = σms .

2 σpan

37

Exercise 9 (a) We have pu (u) = e−u/2 /2 and g = xu/2. Hence 2 du = dg x and, with Theorem 2.1, pg (g) =

1 1 −u/2 du 2 1 = e−g/x · = e−g/x , e · 2 dg 2 x x

g ≥ 0.

(b) We have pg (g) =

1 g m−1 e−gm/x (x/m)m Γ(m)

and v = g/x. Hence dg = x. dv With Theorem (2.1), pv (v) =

1 mm m−1 −vm xm−1 v m−1 e−mv · x = v e . m (x/m) Γ(m) Γ(m)

Exercise 10 (a) For single polarization the ENL estimate reduces to ⟨Ḡ⟩2 ⟨Ḡ⟩2 = . ⟨Ḡ2 ⟩ − ⟨Ḡ⟩2 var(Ḡ) (b) ¡to be done¿ (c) The following Python script will simulate an m-look quad polSAR image: def make_simimage ( fn , m =5 , sigma =1 , alpha =0.2 , beta =0.2): simimage = np . zeros ((100**2 ,9)) ReSigma = np . zeros ((3 ,3)) ImSigma = np . zeros ((3 ,3)) f o r i in range (3): f o r j in range (3): i f i == j : ReSigma [i , j ]= sigma **2 e l i f i<j: ReSigma [i , j ] = alpha * sigma **2 ImSigma [i , j ] = beta * sigma **2 else : ReSigma [i , j ] = alpha * sigma **2 ImSigma [i , j ] = - beta * sigma **2 Sigma = np . mat ( ReSigma +1 j * ImSigma )

38

Image Enhancement and Correction C = np . linalg . cholesky ( Sigma ) f o r i in range (100**2): X = np . mat ( np . random . randn (m ,3)) Y = np . mat ( np . random . randn (m ,3)) Wr = X . T * X + Y . T * Y Wi = X . T * Y - Y . T * X W = ( Wr - 1 j * Wi )/2 W = C*W*C.H simimage [i ,0] = np . real ( W [0 ,0]) simimage [i ,1] = np . real ( W [0 ,1]) simimage [i ,2] = np . imag ( W [0 ,1]) simimage [i ,3] = np . real ( W [0 ,2]) simimage [i ,4] = np . imag ( W [0 ,2]) simimage [i ,5] = np . real ( W [1 ,1]) simimage [i ,6] = np . real ( W [1 ,2]) simimage [i ,7] = np . imag ( W [1 ,2]) simimage [i ,8] = np . real ( W [2 ,2]) driver = gdal . GetDriverByName ( ’ GTiff ’) outDataset = driver . Create ( fn ,100 ,100 ,9 , GDT_Float32 ) f o r i in range (9): outband = outDataset . GetRasterBand ( i +1) outband . WriteArray ( np . reshape ( simimage [: , i ] , (100 ,100)) ,0 ,0) outband . FlushCache () outDataset = None

Exercise 12 From the definition of gradient, we get ∇g(x1 , x2 ) = i

∂g(x1 , x2 ) ∂g(x1 , x2 ) +j . ∂x1 ∂x2

Therefore s

2 2 ∂g(x1 , x2 ) ∂g(x1 , x2 ) ∥∇g(x1 , x2 )∥ = + ∂x1 ∂x1 2 s 2 2 ∆x ∆y ≈ + .

2 2

6 Supervised Classification Part 1

Exercise 1 (a) For the one-dimensional case we have

Z p Z 1 1 1 p(g | 1)p(g | 2) = √ exp − 2 (g − µ1 )2 − 2 (g − µ2 )2 dg 4σ 4σ2 2πσ1 σ2

12 Z 2 1 1 µ1 µ =√ exp − + 22 exp(ag − bg 2 )dg, 4 σ12 σ2 2πσ1 σ2 where a=

1 2

µ1 µ2 + 2 σ12 σ2

,

b=

1 4

1 1 + 2 σ12 σ2

.

Therefore, using the definite integral given in the hint,

r

Z p 1 µ22 1 µ21 π p(g | 1)p(g | 2) = √ exp(a2 /4b) exp − 2 + σ2 4 σ b 2πσ1 σ2 1 2 v

u 1 1 µ21 µ22 u 1

=√ exp − + exp(a2 /4b) t 4 σ12 σ22 2σ1 σ2 1 12 + 12 4

=

σ1 σ2 2

σ1

1 1 + 2 σ12 σ2

σ2

−1/2

1 µ21 µ22 exp − + exp(a2 /4b).

4 σ12 σ22

The first factor may be written in the form

2

1 (σ1 + σ22 )/2 exp − log .

2 σ1 σ2 The exponents in the second two terms can be combined to give

1 µ21 µ22 1 1 µ1 µ2

− + 2 + , 2 + σ2 1 4 σ12 σ2 2 + σ12 4b σ1 σ2 1

2

which simplifies straightforwardly to 1 2 − (µ1 − µ2 )2 2 .

8 σ1 + σ22 39

40

Supervised Classification Part 1

Combining, we get the one-dimensional Bhattacharyya distance 2

1 (σ1 + σ22 )/2 1 2 + log .

B = (µ1 − µ2 )2 2 8 σ1 + σ22 2 σ1 σ2 (b) For the multivariate case, we had Z ∞p p(g | 1)p(g | 2)dg = e−B . −∞

Therefore, the Jeffries-Matusita distance is Z ∞ 2 J= p(g | 1)1/2 − p(g | 2)1/2 dg −∞ Z ∞

p = p(g | 1) + p(g | 2) − 2 p(g | 1)p(g | 2) dg −∞

= 2 − 2e−B . (c) This is a satisfactory separability measure since d12 vanishes when the distributions are identical and is positive otherwise. Moreover it is symmetric, i.e., d12 = d21 . (d) From the definition of the Kullback-Leibler divergence we have Z p(g | 2) KL(p(g | 1), p(g | 2)) = p(g | 1) log dg p(g | 1)

Z 1 1 σ1 − 2 (g − µ2 )2 + 2 (g − µ1 )2 dg = p(g | 1) log σ2 2σ2 2σ1 Z 1 σ1 1 p(g | 1)(g − µ2 )2 dg + . = log − 2 σ2 2σ2 2 The remaining integral can be expanded as follows: Z Z p(g | 1)(g − µ2 )2 dg = p(g | 1)(g − µ1 + µ1 − µ2 )2 dg Z

= p(g | 1) (g − µ1 )2 + 2(g − µ1 )(µ1 − µ2 ) + (µ1 − µ2 )2 dg Z = σ12 + 2(µ1 − µ2 ) p(g | 1)(g − µ1 )dg + (µ1 − µ2 )2 = σ12 + (µ1 − µ2 )2 .

This gives KL(p(g | 1), p(g | 2)) = log

σ1 (µ1 − µ2 )2 σ2 − σ2 , + 2 2 1 + σ2 2σ2 2σ12

and, from symmetry, KL(p(g | 2), p(g | 1)) = log

σ2 σ2 − σ2 (µ1 − µ2 )2 + 1 2 2 + . σ1 2σ1 2σ22

Adding the two expressions gives the result.

41

Exercise 2 (a) This follows directly by expanding Equation (6.17) and dropping terms that do not involve the class label k. (b) The classifier will choose k = 1 when 1 1 −1 −1 −1 log(Pr(1)) − µ⊤ g + µ⊤ µ1 > log(Pr(2)) − µ⊤ g + µ⊤ Σ−2 µ2 , 1Σ 1Σ 2Σ 2 2 2 which simplifies to ⊤

−1

h = (µ1 − µ2 ) Σ

µ + µ2 g− 1 2

> log

Pr(2) Pr(1)

.

(c) The random variable H is

µ + µ2 = a⊤ G − b, H = (µ1 − µ2 )⊤ Σ−1 G − 1 2 where a⊤ = (µ1 − µ2 )⊤ Σ−1 ,

b=

1 (µ − µ2 )⊤ Σ−1 (µ1 + µ2 ).

2 1

If the observation in in class 1, then with H = H1 , ⟨H1 ⟩ = a⊤ µ1 − b = (µ1 − µ2 )⊤ Σ−1

(µ1 − µ2 ) 1 = d2 .

2 2

Similarly, if the observation is in class 2, 1 ⟨H2 ⟩ = − d2 2 In either case, the variance is var(H1 ) = var(H2 ) = a⊤ Σa = (µ1 − µ2 )⊤ Σ−1 Σ (µ1 − µ2 )⊤ Σ−1

⊤

= (µ1 − µ2 )⊤ Σ(µ1 − µ2 ) = d2 .

Thus H1 ∼ N (d2 /2, d2 ) and H2 ∼ N (−d2 /2, d2 ). (d) The probability of misclassification is given by the weighted sum

Pr(2) Pr(2) + Pr(2) · Pr H2 > log .

Pr(1) · Pr H1 < log Pr(1) Pr(1) For the first term we can standardize H1 and write

 Pr(2) − d2 /2 

log 2 Pr(2) H1 − d /2 Pr(1)  Pr H1 < log = Pr  < Pr(1) d d

1 1 Pr(2) = Φ − d + log 2 d Pr(1)

42

Supervised Classification Part 1

and similarly, for H2 ,

 Pr(2) + d2 /2 

log 2 Pr(2) H2 + d /2 Pr(1)  Pr H2 > log = Pr  > Pr(1) d d

1 Pr(2) 1 d + log =1−Φ 2 d Pr(1)

1 1 Pr(2) = Φ − d − log .

2 d Pr(1) (e) With a = Pr(1) = 1 − Pr(2) the misclassification probability is

d 1 a d 1 a aΦ − + log + (1 − a)Φ − − log .

2 d 1−a 2 d 1−a The first term increases monotonically from 0 to 1 and the second term decreases monotonically from 1 to 0 as a increases from 0 to 1. Therefore the minimum in their sum must occur at their intersection, when both terms are equal. This is at a = 1/2. Hence the minimum misclassification probability is Φ − d2 .

Exercise 3 (a) Let g 0 and g be any two points in the hyperplane defined by w and w0 .

Then w⊤ (g − g 0 ) = 0, so that w is orthogonal to any vector g − g 0 which lies in the hyperplane. (b) Since the 3 points are not colinear, a straight line (oriented hyperplane in two dimensions) can always be found which separates any one from the other two. This is not possible for 4 non-colinear points in two dimensions, for example the four corners of a square with two diagonally opposite points belonging to class 1 and the other two to class 2. Therefore the VC dimension for the hyperplane classifier in two dimensions is 3. (c) See Listing ?? and Figure ??.

Exercise 4 Taking the logarithm of the likelihood function, we get X X log p(ℓ(ν) | g(ν)) − log p(g(ν). ν

ν

The first term involves the posterior probability density, which is modelled by the network and hence depends on the synaptic weights. The second term doesn’t involve the network weights and so can be ignored. Maximizing the likelihood is thus equivalent to minimizing the cost function X E=− log p(ℓ(ν) | g(ν)). ν

43 Since the components of ℓ(ν) are statistically independent, we can write this as XX E=− log p(ℓk (ν) | g(ν)). ν

k

Now substituting from Equation (6.80) gives E=

nK 1 XX 2 (mk (ν) − ℓk (ν)) + nK log σ + log(2π), 2 2σ ν 2 k

where we have replaced the unknown functions hk with the network outputs mk which model them. Dropping the terms which don’t depend on the synaptic weights and the factor 1/σ 2 , E=

1X 1 XX 2 (mk (ν) − ℓk (ν)) = ∥m(ν) − ℓ(ν)∥2 , 2 ν 2 ν k

which is the quadratic cost function.

Exercise 6 The third step in the backpropagation algorithm is o o wjk (ν + 1) → wjk (ν) − η

∂E(ν) o , ∂wjk

where, with (6.36), ∂E(ν) = −δko (ν)nj (ν). o ∂wjk Therefore

o o o wjk (ν + 1) → wjk (ν) + ηδk nj (ν) = wjk (ν) + η n(ν)δ o (ν)⊤ jk .

Exercise 7 (a) This follows easily from f (x) = 1/(1 + e−x ). (b) Consider the jth neuron in the hidden layer of Figure 6.10 in the text. Its output signal can be written as ! wh⊤ 1 1 j g nj (g) = tanh + 2 2 2 Its contribution to the activation of the kth output neuron is therefore ! ! wh⊤ 1 1 j g o o nj (g)wkj = tanh + wkj .

2 2 2

44

Supervised Classification Part 1

If we reverse the sign of the hidden weight vector wh⊤ and also that of the j o output weight wkj , then this contribution becomes o nj (g)wkj =

1 tanh 2

wh⊤ j g 2

!

1 − 2

! o wkj .

o The difference between the two is just wkj . This difference can be exactly o compensated by the bias weight wk0 so that the network output is unchanged.

There are thus two equivalent configurations of the network for each of the L hidden neurons, or 2L in all. It follows that every minimum in the cost function corresponds to at least 2L different synaptic weight combinations.

Exercise 8 With ∆ independent of ν, for the first n updates, ∆1 → ∆ + α∆ ∆2 → ∆ + α(∆ + α∆) = ∆(1 + α + α2 ) .. . ∆n → ∆(1 + α + α2 + . . . + αn−1 ) = ∆(1 − αn )/(1 − α) Thus for n → ∞ and α < 1 ∆→

1 ∆.

1−α

Exercise 9 Combining Equations (6.73) and (6.76) gives X Pr(ℓ = i | g) j̸=i

µij

= (K − 2)Pr(ℓ = i | g) + 1.

Solving for Pr(ℓ = i | g) gives Equation (6.77).

7 Supervised Classification Part 2

Exercise 1 From Equation (7.13) with Y = y the limits on θ occur when y/n − θ = ±s, σ where from Equation (7.11) r σ=

θ(1 − θ) . n

Therefore the limits for θ are the solutions of (y − nθ)2 = s2 nθ(1 − θ), a quadratic equation for θ. The solutions are p y + s2 /2 ± s2 /4 + y(n − y)/n θ= s2 + n which gives (7.14) for s = 1.960.

Exercise 4 The log discriminant function for x = mc̄, where c̄ is a multi-look polarimetric SAR image pixel in complex covariance matrix format, is dk (x) = log(P r(k) + log(pWc (x|Σk )) = log(P r(k)) − m log |Σk | − tr(Σ−1 k x) + terms independent of k.

In terms of c̄, therefore, dk (c̄) = log(P r(k)) − m log |Σk | − tr(Σ−1 k mc̄)

= log(P r(k)) − m log |Σk | + tr(Σ−1 k c̄) , which is Equation (7.34).

45

46

Supervised Classification Part 2

Exercise 5 We have ⊤ L = (G − M α)⊤ Σ−1 R (G − M α) + 2λ(α 1K − 1).

Expanding, ⊤ −1 ⊤ −1 ⊤ ⊤ ⊤ L = G⊤ Σ−1 R G + α M ΣR M α − 2α M ΣR G + 2λ(α 1K − 1)

and applying the rules for vector differentiation, ∂L ⊤ −1 = 2M ⊤ Σ−1 R M α − 2M ΣR G + 2λ1k = 0. ∂α Therefore,

⊤ −1 −1 α = (M ⊤ Σ−1 M ) M Σ G − λ1 .

K R R Together with the constraint α⊤ 1K = 1, the substitution G → g gives the result.

Exercise 6 (a) The matrix P = I − U (U ⊤ U )−1 U ⊤ “projects out” the undesirable components, since P G = P dαd + IU β − U β + P R = P dαd + P R. (b) A suitable function for OSP is def osp (G , U ): N = G . shape [1] P = np . eye ( N ) - U *( U . T * U ). I * U . T return P * G . T

(c) Assuming the mean to be close to zero, we can write ⟨(w⊤ G)2 ⟩ ≈ ⟨(w⊤ G − 0)2 ⟩ = var(w⊤ G).

Therefore we must minimize var(w⊤ G) = w⊤ ΣR w under the constraint w⊤ d = 1. The corresponding Lagrange function is L = w⊤ ΣR w − 2λ(w⊤ d − 1).

We have

∂L = 2ΣR w − 2λd = 0, ∂w

47 so that w = λΣ−1 R d.

The constraint requires ⊤ λ(Σ−1 R d) d = 1

or λ=

1 d Σ−1 R d

w=

Σ−1 R d . d Σ−1 R d

and therefore

⊤

(7.1)

⊤

(d) The SAM angle θ is given by θ = cos−1

g⊤ d ∥g∥∥d∥

so we project g along d. In CEM with ΣR = σ 2 I we do precisely the same thing, i.e., we project g along w given by, see Equation (7.1) above, w = d/∥d∥2 .

Exercise 7 With Equation (7.38) we can write # " 1 X ⊤ 1 X ⊤ ⊤ ′ µϕ Φ̃ = ϕ(ν ) ϕ (ν) (ϕ(1) . . . ϕ(m)) − m ν m ′ ν

1 X ⊤ 1 X ⊤ = ϕ (ν)ϕ(1) . . . ϕ⊤ (ν)ϕ(m) − 2 ϕ (ν)ϕ(ν ′ ). m ν m ′ νν

Replacing inner products ϕ⊤ (ν)ϕ(ν ′ ) by the kernels k(g(ν), g(ν ′ )) gives Equation (7.52).

8 Unsupervised Classification

Exercise 1 (a) Consider only the kth term in the cost function Equation (8.12) and write n = nk and ui = uki . The term is then !⊤ !

X 1X 1X ui g i − ui′ g i′ gi − ui′′ g i′′ . n ′ n ′′ i i

i

Expanding, we get X 2X 1X ui ∥g i ∥2 − ui ui′ g ⊤ ui′ ui′′ g ⊤ i g i′ + i′ g i′′ , n n ′ ′ ′′ i i,i

i ,i

which simplifies to X

ui ∥g i ∥2 −

i

1X ui ui′ g ⊤ i g i′ . n ′ i,i

This can be rewritten in the equivalent form   X X X 1   n ∥g i ∥2 + n ∥g i′ ∥2 − 2 ui ui′ g ⊤ i g i′ 2n i i′ i,i′   X X X 1   = ui ui′ ∥g i ∥2 + ui ui′ ∥g i ∥2 − 2 ui ui′ g ⊤ i g i′ 2n ′ ′ ′ i,i

=

1 X 2n

i,i

i,i

ui ui′ ∥g i − g i′ ∥2 .

i,i′

Reintroducing the index k and summing over it establishes Equation (8.62). (b) The expression D = G ⊤ − G ⊤ U ⊤ M U = G ⊤ (I m − U ⊤ M U ) is a N × m matrix whose νth column is the distance of observation g(ν) from its associated mean. Therefore the diagonal elements of D ⊤ D are the squares of those distances, and the sum of squares is its trace:

E(C) = tr(D ⊤ D) = tr (I m − U ⊤ M U )GG ⊤ (I m − U ⊤ M U )

49

50

Unsupervised Classification

or, expanding and using the identity tr(A) = tr(A⊤ ),

E(C) = tr GG ⊤ − 2U ⊤ M U GG ⊤ + U ⊤ M U GG ⊤ U ⊤ M U .

Now, with the identity tr(AB) = tr(BA), we can write

E(C) = tr GG ⊤ − 2U ⊤ M U GG ⊤ + U ⊤ M U U ⊤ M U GG ⊤ .

But m1 0  U U ⊤M =   ...

0 m2 .. .

... ... .. .

  1/m 0 1 0  0  . ..  .   ..

0

0

...

mK



0

0 1/m2 .. .

... ... .. .

0 0 .. .

0

...

1/mK

   = Ik 

so we have finally

E(C) = tr GG ⊤ − 2U ⊤ M U GG ⊤ + U ⊤ M U GG ⊤

= tr GG ⊤ − U ⊤ M U GG ⊤ .

Exercise 2 (b) This is a matter of experimentation. A good discrimination can be obtained with an NSCALE value of around 0.3.

Exercise 3 (a) The symmetry of L follows from the symmetry of D and K. For any m-dimensional vector x, x⊤ Lx = x⊤ Dx − x⊤ Kx X X = di x2i − xi xj (K)ij i

ij

  X X X 1 =  di x2i − 2 xi xj (K)ij + dj x2j  2 i ij j   X X X 1 =  (K)ij x2i − 2 xi xj (K)ij + (K)ij x2j  2 ij ij ij =

1X (K)ij (xi − xj )2 ≥ 0, 2 ij

so that L is positive semi-definite. In particular, 1X 1⊤ (K)ij (1 − 1)2 = 0, m L1m = 2 ij

51 so that L1m = 01m and 1m is an eigenvector with eigenvalue 0. Since L is symmetric positive semi-definite, all the other eigenvalues must be ≥ 0. (c) Generalization could for example be accomplished by using the clustered samples to train a supervised classifier such as a neural network.

Exercise 7 Starting from the first equality in Equations (8.27): X

∆(k, j) =

∥g i − m̄∥2 −

i∈Ck ∪Cj

= =

X

X

∥g i − mk ∥2 −

i∈Ck

X

∥g i − mj ∥2

i∈Cj

2

2

(∥g i − m̄∥ − ∥g i − mk ∥ ) +

X

(∥g i − m̄∥2 − ∥g i − mj ∥2 )

i∈Ck

i∈Cj

X

2 2 (2g ⊤ i (mk − m̄) − ∥m̄∥ − ∥mk ∥ )

i∈Ck

X

+

2 2 (2g ⊤ i (mj − m̄) − ∥m̄∥ − ∥mj ∥ )

i∈Cj 2 2 = 2nk m⊤ k (mk − m̄) + nk (∥m̄∥ − ∥mk ∥ ) 2 2 + 2nj m⊤ j (mj − m̄) + nj (∥m̄∥ − ∥mj ∥ )

= nk (∥mk ∥2 + ∥m̄∥2 ) + nj (∥mj ∥2 + ∥m̄∥2 ) − 2m̄⊤ (nk mk + nj mj ).

Now multiply the last term above by (nk + nj )/(nk + nj ) to give ∆(k, j) = nk (∥mk ∥2 + ∥m̄∥2 ) + nj (∥mj ∥2 + ∥m̄∥2 ) − 2∥m̄∥2 (nk + nj ) = nk ∥mk ∥2 + nk ∥mj ∥2 − ∥m̄∥2 (nk + nj ) = nk ∥mk ∥2 + nk ∥mj ∥2 −

∥nk mk + nj mj ∥2 nk + nj

nk nj 2 (∥mk ∥2 − 2m⊤ k mj + ∥mj ∥ ) nk + nj nk nj = ∥mk − mj ∥2 . nk + nj =

Exercise 8 The proposed cost function is E(C) =

Xp |C k |. k

Under the linear transformation T , the covariance matrix becomes C k → C ′k = T C k T ⊤ .

52

Unsupervised Classification

From the properties of the determinant, Chapter 1, |C ′k | = |T |2 |C k |, so that E ′ (C) =

X Xq p |C ′k | = |T | |C k | = |T |E(C). k

k ′

Therefore E (C) has the same minimum as E(C).

Exercise 9 (a) Differentiating the log-likelihood with respect to µk , !

X ∂ X ∂L ′ ′ 0= = log p(gi | k )Pr(k ) ∂µk ∂µk i k′ X 1 ∂ = p(gi | k)Pr(k) p(g ) ∂µ i k i X 1 gi − µk Pr(k)p(gi | k) =− p(gi ) σk2 i X gi − µk =− Pr(k | gi ) (using Bayes’ Theorem). σk2 i Solving for µk gives the required result, namely P Pr(k | gi )gi µk = Pi . i Pr(k | gi ) The expression for σk2 follows in the same way by setting the derivative of L Pwith respect to σk equal to zero. The parameter Pr(k) is constrained by k Pr(k) = 1, so to get it we must use the Lagrange function L=

X

X X log( p(gi | k)Pr(k)) − λ( Pr(k) − 1).

i

k

k

Thus 0=

X 1 ∂L = p(gi | k) − λ ∂Pr(k) p(gi ) i X Pr(k | gi ) = − λ, Pr(k) i

from which Pr(k) =

1X Pr(k | gi ). λ i

53 Finally, summing the last equation over k, 1=

1 XX 1X 1 Pr(k | gi ) = 1 = n, λ i λ i λ k

so that λ = n. (b) In the FMLE algorithm for one-dimensional data, replace the class membership uki by Pr(k | gi ) and it becomes identical to the EM algorithm.

Exercise 10 For a cubic array of pixels in a Kohonen network, d2 (k ∗ , k) = [(k ∗ − 1) mod m − (k − 1) mod m]2 + [((k ∗ − 1)/m − (k − 1)/m) mod m]2 + [(k ∗ − 1)/m2 − (k − 1)/m2 ]2 .

Exercise 12 (a) The mean shift preserves edges because each pixel is assigned to the basin of attraction of the nearest mode and the basins are unambiguous, that is, they have well-defined edges. (b) Writing the Gaussian kernel density estimate as p(w) =

1 X −∥w−y ∥2 /2σ2 ν e , Z ν

where Z is the normailization, we have 2 2 ∂p(w) 1 X =− 2 (w − y ν )e−∥w−y ν ∥ /2σ ∂w Zσ ν # "P −∥w −y ν ∥2 /2σ 2 y e 1 X −∥w−y ∥2 /2σ2 ν ν ν = e −w P −∥w −y ν ∥2 /2σ 2 Zσ 2 ν νe "P # −∥w −y ν ∥2 /2σ 2 1 ν yν e = 2 p(w) P −∥w−y ∥2 /2σ2 − w . ν σ νe

The expression in square brackets is the difference between the weighted mean as calculated with the kernel and the center of the window, i.e., the mean shift.

Thus the mean shift is proportional to the estimated gradient of the density.

9 Change Detection

Exercise 1 (a) Multiply the first equation in Equations (9.7) from the left by Σ−1 11 and the second by ρ to give Σ−1 11 Σ12 b − ρa = 0 2 ρΣ⊤ 12 a − ρ Σ22 b = 0.

Eliminate a by multiplying the first equation above from the left by Σ⊤ 12 and adding. This gives Equation (9.11).

Exercise 2 (a) The vector of canonical variates can be written as U = A⊤ G. Therefore the covariance matrix with the original variates G is 

··· .. . cov(GN , U1 ) · · ·

cov(G1 , U1 )  ..  .

 cov(G1 , UN )  .. ⊤ ⊤  = ⟨GU ⟩ = ⟨GG ⟩A = Σ11 A. . cov(GN , UN )

Multiplying this expression from the left with the matrix D will generate the matrix p p   cov(G1 , U1 )/ var(G1 ) · · · cov(G1 , UN )/ var(G1 )   .. .. ..

DΣ11 A =  , . . p . p cov(GN , U1 )/ var(GN ) · · · cov(GN , UN )/ var(GN ) but since var(Ui ) = 1, i = 1 . . . N , this is just the correlation matrix 

··· .. . corr(GN , U1 ) · · ·

corr(G1 , U1 )  ..

DΣ11 A =  .

 corr(G1 , UN )  .. . . corr(GN , UN ) 55

56

Change Detection

(b) Multiplying the above correlation matrix from the right with the diagonal matrix S gives   N corr(G1 , U1 ) |ss11 | · · · corr(G1 , UN ) |ssN |   .. .. ..

DΣ11 AS =   . . .

N corr(GN , U1 ) |ss11 | · · · corr(GN , UN ) |ssN | with column sums

s2 s21 ... N , |s1 | |sN |

all of which are positive as required.

Exercise 3 The answer is of course no. The MAD transformation is invariant under any linear transformation.

Exercise 5 (a) For the null hypothesis log(L(x)) = −2m log x − (g1 + g2 )

m + terms independent of x. x

Differentiating, d log(L(x)) 1 m = −2m + (g1 + g2 ) 2 = 0. dx x x Therefore

g1 + g2 .

2 The result for the alternative hypothesis follows just as easily. (b) Taking logarithms: x̂ =

i

ln L0 = −mj ln x −

mX gi + terms indpendent of x. x i=1

Setting ∂ ln0 /∂x = 0 gives the result.

Exercise 6 The exponent in L0 is obviously −mj. For L1 we have Pj−1 gj i=1 gi −m 1 P − m = −m(j − 1) − m = −mj j−1 gj i=1 gi j−1

as well.

57

Exercise 8 Setting N = 2 the likelihood ratio test becomes Q = 24m

|x1 |m |x2 |m ≤ k. |x1 + x2 |2m

The quantity −2 log Q = −2m(4 log 2 + log |x1 | + log |x2 | − 2 log |x1 + x2 |) is chi-square distributed with 4 degrees of freedom as m → ∞.

Exercise 10 The second of Equations (9.50) follows from the scaling of the y-axis: YR − ỹ LR = .

YT − y LT The x-axis scales as

and hence

LR /aR , LT /aT x̃ − XR LR aT = x − XT LT /aR

from which we get the first of Equations (9.50).