Machine Learning: A Bayesian and Optimization Perspective 2nd Edition By Sergios Theodoridis INSTRU

INSTRUCTOR'S SOLUTION MANUAL

1

Solutions To Problems of Chapter 2

2.1. Derive the mean and variance for the binomial distribution.

Solution: For the mean value we have that,

E[x]

= =

n X k=0 n X k=1

= np

kn! pk (1 − p)n−k (n − k)!k! n! pk (1 − p)n−k (n − k)!(k − 1)!

n X

(n − 1)!

pk−1 (1 − p)(n−1)−(k−1) (n − 1) − (k − 1) !(k − 1)!

k=1

= np

n−1 X l=0

(n − 1)! pl (1 − p)(n−1)−l (n − 1) − l !l!

= np(p + 1 − p)n−1 = np.

(1)

where the formula for the binomial expansion has been employed. For the variance we have,

σx2

= =

n X

(k − np)2

k=0 n X k=0 n X

k2

n! pk (1 − p)n−k + (n − k)!k!

(np)2

k=0

2np

n! pk (1 − p)n−k (n − k)!k!

n X k=0

n! pk (1 − p)n−k − (n − k)!k!

k

n! pk (1 − p)n−k , (n − k)!k!

(2)

or

σx2 =

n X k=0

k2

n! pk (1 − p)n−k + (np)2 − 2(np)2 , (n − k)!k!

(3)

2

However, n X

n! pk (1 − p)n−k = (n − k)!k!

k2

k=0

np

n X

k

k=1

np

(n − 1)!

pk−1 (1 − p)(n−1)−(k−1) = (n − 1) − (k − 1) !(k − 1)!

n−1 X

(l + 1)

l=0

(n − 1)! pl (1 − p)(n−1)−l = (n − 1) − l !l!

np + np(n − 1)p,

(4)

which finally proves the result.

2.2. Derive the mean and the variance for the uniform distribution.

Solution: For the mean we have Z b µ = E[x]

= =

1 xdx b − a a 1 b b b+a = . b−a2 a 2

(5)

For the variance, we have σx2

= = =

1 b−a

Z b

(x − µ)2 dx =

a

1 b−a

Z b−µ

y 2 dy

a−µ

1 y 3 b−µ b − a 3 a−µ 1 (b − a)2 .

12

(6)

2.3. Derive the mean and covariance matrix of the multivariate Gaussian.

Solution: Without harming generality, we assume that µ = 0, in order to simplify the discussion. We have that 1 (2π)l/2 |Σ|1/2

Z +∞

1 x exp − xT Σ −1 x dx, 2 −∞

(7)

which due to the symmetry of the exponential results in E[x] = 0.

For the covariance we have that Z +∞

1 exp − xT Σ −1 x dx = (2π)l/2 |Σ|1/2 .

2 −∞

(8)

3

Following similar arguments as for the univariate case given in the text, we are going to take the derivative on both sides with respect to matrix Σ. Recall from linear algebra the following formulas. ∂trace{AX −1 B} ∂|X k | = −(X −1 BAX −1 )T , = k|X k |X −T . ∂X ∂X Hence, taking the derivatives of both sides in (8) with respect to Σ we obtain, Z 1

T 1 1 +∞ exp − xT Σ −1 x Σ −1 xxT Σ −1 dx = (2π)l/2 |Σ|1/2 Σ −T , 2 −∞ 2 2 (9) which then readily gives the result.

2.4. Show that the mean and variance of the beta distribution with parameters a and b are given by a , E[x] = a+b and ab σx2 = . (a + b)2 (a + b + 1) Hint: Use the property Γ(a + 1) = aΓ(a).

Solution: We know that Γ(a + b) a−1 x (1 − x)b−1 . Γ(a)Γ(b)

Beta(x|a, b) = Hence E[x] =

Γ(a + b) Γ(a)Γ(b)

Z 1

xxa−1 (1 − x)b−1 dx =

0

Γ(a + b) Γ(a + 1)Γ(b) , Γ(a)Γ(b) Γ(a + 1 + b)

which, using the property Γ(a + 1) = aΓ(a), results in a .

E[x] = a+b

(10)

For the variance we have E[(x − E[x])2 ] =

Γ(a + b) Γ(a)Γ(b)

Z 1 x− 0

a a+b

2

xa−1 (1 − x)b−1 dx,

(11)

or Γ(a + b) σx2 =

Γ(a)Γ(b)

Z 1

xa+1 (1 − x)b−1 dx

0

Z a2 Γ(a + b) 1 a−1 x (1 − x)b−1 dx (a + b)2 Γ(a)Γ(b) 0 Z a Γ(a + b) 1 a −2 x (1 − x)b−1 dx, a + b Γ(a)Γ(b) 0 +

(12)

4

and following a similar path as the one adopted for the mean, it is a matter of simple algebra to show that σx2 =

ab (a + b)2 (a + b + 1)

.

2.5. Show that the normalizing constant in the beta distribution with parameters a, b is given by Γ(a + b) . Γ(a)Γ(b) Solution: The beta distribution is given by Beta(x|a, b) = Cxa−1 (1 − x)b−1 , 0 ≤ x ≤ 1.

(13)

Hence C

−1

Z 1 =

xa−1 (1 − x)b−1 dx.

(14)

0

Let x = sin2 θ ⇒ dx = 2 sin θ cos θdθ.

Hence C −1 = 2

Z π2

(15)

(sin θ)2a−1 (cos θ)2b−1 dθ.

(16)

0

Recall the definition of the gamma function Z ∞ Γ(a) = xa−1 e−x dx, 0

and set x = y 2 ⇒ dx = 2ydy, hence

Z ∞ Γ(a) = 2

2

y 2a−1 e−y dy.

(17)

0

Thus

Z ∞Z ∞ Γ(a)Γ(b) = 4 0

2

2

x2a−1 y 2b−1 e−(x +y ) dxdy.

(18)

0

Let x = r sin θ, y = r cos θ ⇒ dxdy = rdrdθ.

Hence Z π2 Z ∞ Γ(a)Γ(b) = 4 0

0

2

r2(a+b)−1 e−r (sin θ)2a−1 (cos θ)2a−1 drdθ.

(19)

5

where integration over θ is in the interval 0, π2 to guarantee that x remains non-negative. From (19) we have !

Z ∞

Z π2 2 Γ(a)Γ(b) = 2 r2(a+b)−1 e−r dr 2 (sin θ)2a−1 (cos θ)2b−1 dθ 0

0 −1

= Γ(a + b)C

,

which proves the claim.

2.6. Show that the mean and variance of the gamma pdf Gamma(x|a, b) =

ba a−1 −bx x e , a, b, x > 0. Γ(a)

are given by a , b a σx2 = 2 . b

E[x] =

Solution: We have that ba E[x] = Γ(a)

Z ∞

xa e−bx dx.

0

Set bx = y. Then Z ∞ ba 1 y a e−y dy Γ(a) ba+1 0 aΓ(a) a 1 Γ(a + 1) = = . = bΓ(a) bΓ(a) b

E[x] =

For the variance, the following is valid Z a ba n ∞ a+1 −bx σx2 = E[(x − )2 ] = x e dx b Γ(a) 0 Z Z a ∞ a −bx o a2 ∞ a−1 −bx x e dx − 2 x e dx , + 2 b 0 b 0 and following a similar path as before we obtain a σx2 = 2 . b 2.7. Show that the mean and variance of a Dirichlet pdf with K variables, xk , k = 1, 2, . . . , K and parameters ak , k = 1, 2, . . . , K, are given by ak E[xk ] = , k = 1, 2, . . . , K a ak (a − ak ) , k = 1, 2, . . . , K, σk2 = 2 a (1 + a) ai aj cov[xi xj ] = − 2 , i 6= j, a (1 + a)

6

where a =

PK

k=1 ak .

Solution: Without harm of generality, we will derive the mean for xK .

The others are derived similarly. To this end, we have

p(x1 , x2 , . . . , xK−1 ) = C

K−1 Y

xakk −1

1−

k=1

K−1 X

!aK −1 xk

k=1

where C=

Γ(a1 + a2 + . . . + aK ) . Γ(a1 )Γ(a2 ) . . . Γ(aK )

Z 1 "Z 1−PK−1 k=1 xk

Z 1 ···

E[xK ] = C

xK p(x1 , . . . , xK−1 , xK )dxK dxK−1 . . . dx1

0

Z 1 =C

0

0

 Z 1 Z 1−PK−1 K−1 Y a −1 k=1 xk  ··· xK x k

0

0

Z 1 ···

=C

#

Z 1 K−1 Y

0

0

1−

k

0

k=1

xakk −1

1−

k=1

K−1 X

!aK "Z

K−1 X

!aK −1 xk

k=1 PK−1 1− k=1 xk

xk

 dxK  dxK−1 . . . dx1

# dxK dxK−1 . . . dx1 ,

0

k=1

or Z 1 ···

E[xK ] = C 0

Z 1 K−1 Y 0

xakk −1

k=1

1−

K−1 X

!aK xk

dxK−1 . . . dx1

k=1

Γ(a1 ) . . . Γ(aK + 1) Γ(a1 + a2 + . . . + aK + 1) aK Γ(a1 ) . . . Γ(aK ) =C (a1 + a2 + . . . + aK )Γ(a1 + a2 + . . . + aK ) aK = . a =C

In the sequel, we will show that ai aj E[xi xj ] = − 2 , i 6= j. a (a + 1) We derive it for the variables xK and xK−1 , since any of the variables can

7

be taken in place of xK and xK−1 . Hence, Z 1 "Z 1−PK−1 k=1 xk

Z 1 ···

E[xK−1 xK ] = C

0

0

Z 1 ···

=C

Z 1 K−2 Y

0

C = aK + 1

0

0

···

aK−1 xakk −1 xK−1

Z 1 K−2 Y 0

0

! xakk −1

# aK−1 aK xK−1 xK dxK

dxK−1 . . . dx1

k=1

k=1

Z 1

K−2 Y

"Z

P 1− K−1 k=1 xk

0

aK−1 xakk −1 xK−1

k=1

1−

# xaKK dxK

K−1 X

dxK−1 . . . dx1

!aK +1 xk

dxK−1 . . . dx1

k=1

C Γ(a1 ) . . . Γ(aK−2 )Γ(aK−1 + 1)Γ(aK + 2) aK + 1 Γ(a1 + . . . + aK−2 + aK−1 + aK + 2) aK aK−1 Γ(a1 ) . . . Γ(aK )(aK + 1) C = aK + 1 (1 + a1 + . . . + aK )(a1 + . . . + aK )Γ(a1 + . . . + aK ) =

or E[xK−1 xK ] =

aK aK−1 . a(1 + a)

Thus in general, E[xi xj ] =

ai aj . a(1 + a)

For the covariance, we have

cov[xi xj ] = E xi − E[xi ] E xj − E[xj ] = E[xi xj ] − E[xi ] E[xj ], or ai aj ai aj − 2 a(1 + a) a ai aj a − ai aj (1 + a) ai aj = =− 2 . a2 (1 + a) a (1 + a)

cov[xi xj ] =

2.8. Show that the sample mean, using N i.i.d drawn samples, is an unbiased estimator with variance that tends to zero asymptotically, as N −→ ∞.

Solution: From the definition of the sample mean we have N N 1 X 1 X E[µ̂N ] = E[xn ] = E[x] = E[x].

N n=1 N n=1

(20)

8

For the variance we have,  σµ̂2 N

N 1 X

!

N 1 X



xi − µ  xj − µ N i=1 N j=1    N N X X 1 = E  2  (xi − µ) (xj − µ) N i=1 j=1 = E

=

N N 1 XX E [(xi − µ)(xj − µ)] .

N 2 i=1 j=1

(21)

(22)

However, since the samples are i.i.d. drawn, the expected value of the product is equal to the product of the mean values, hence it is zero except for i = j, which then results in σµ̂2 N =

1 2 σ , N x

which proves the claim.

2.9. Show that for WSS processes r(0) ≥ |r(k)|, ∀k ∈ Z, and that for jointly WSS processes, ru (0)rv (0) ≥ |ruv (k)|2 , ∀k ∈ Z. Solution: Both properties are shown in a similar way. So, we are going to focus on the first one. Consider the obvious inequality, E[|un − λun−k |2 ] ≥ 0, or E[|un |2 ] + |λ|2 E[|un−k |2 ] ≥ λ∗ r(k) + λr∗ (k), or r(0) + |λ|2 r(0) ≥ λ∗ r(k) + λr∗ (k).

This is true for any λ, thus it will be true for λ = r(k) r(0) . Substituting, we obtain |r(k)|2 r(0) ≥ , r(0) which proves the claim.

Similar steps are adopted in order to prove the property for the crosscorrelation.

9

2.10. Show that the autocorrelation of the output of a linear system, with impulse response, wn , n ∈ Z, is related to the autocorrelation of the input process, via, ∗ rd (k) = ru (k) ∗ wk ∗ w−k .

Solution: We have that rd (k)

  X X = E[dn d∗n−k ] = E  wi∗ un−i wj u∗n−k−j  i

=

XX i

j

=

X

wj

j

j

wi∗ wj E[un−i u∗n−k−j ]

X

wi∗ ru (k + j − i).

(23)

h(n) := wn ∗ ru (n).

(24)

i

Set Then we can write, rd (k)

= =

X

wj h(k + j) =

X

j ∗ w−k ∗ wk ∗ ru (k),

∗ wj h − ((−k) − j) = w−k ∗ h(−(−k))

j

which proves the claim.

2.11. Show that ln x ≤ x − 1.

Solution: Define the function f (x) = x − 1 − ln x. then f 0 (x) = 1 −

1 1 , and f 00 (x) = 2 . x x

Thus x = 1 is a minimum, i.e., f (x) ≥ f (1) = 1 − 1 − 0 = 0. or ln x ≤ x − 1.

2.12. Show that I(x; y) ≥ 0.

10

Hint: Use the inequality of Problem 2.11.

Solution: By the respective definition, we have that −I(x; y)

= −

XX x

=

log e

P (x, y) log

y

XX x

P (x|y) P (x)

P (x, y) ln

y

P (x) , P (x|y)

where we have used only terms where P (x, y) 6= 0. Taking into account the inequality, we have that

XX P (x) −1 = −I(x; y) ≤ log e P (x, y) P (x|y) x y XX log e {P (x)P (y) − P (x, y)} . x

y

Note that the summation over the terms in the brackets is equal to zero, which proves the claim.

Note that if the random variables are independent, then P (x) = P (x|y) and I(x; y) = 0.

2.13. Show that if ai , bi , i = 1, 2, . . . , M are positive numbers, such as M X

ai = 1, and

M X

bi ≤ 1,

i=1

i=1

then −

M X

ai ln ai ≤ −

M X

ai ln bi .

i=1

i=1

Solution: Recalling the inequality from Problem 2.11, that ln

bi bi ≤ − 1, ai ai

or M X i=1

ai ln

M X bi ≤ (bi − ai ) ≤ 0, ai i=1

which proves the claim and where the assumptions concerning ai and bi have been taken into account.

2.14. Show that the maximum value of the entropy of a random variable occurs if all possible outcomes are equiprobable.

11

Solution: Let pi , i = 1, 2, . . . , M , be the corresponding probabilities of the M possible events. According to the inequality in Problem 2.13 for bi = 1/M , we have, −

M X

pi ln pi ≤

i=1

or −

M X

pi ln M,

i=1

M X

pi ln pi ≤ ln M.

i=1

Thus the maximum value of the entropy is ln M , which is achieved if all probabilities are equal to 1/M .

2.15. Show that from all the pdfs which describe a random variable in an interval [a, b] the uniform one maximizes the entropy.

Solution: The Lagrangian of the constrained optimization task is Z +∞ Z +∞

L(p(·), λ) = − p(x) ln p(x)dx + λ p(x)dx − 1 . −∞

−∞

According to the calculus of variations (for the unfamiliar reader, treat p(x) as a variable and take derivatives under the integrals as usual) we take the derivative and set it equal to zero, resulting in ln p(x) = λ − 1.

Plugging it in the constrain equation, and performing the integration results in 1 , p(x) = b−a which proves the claim.

1

Solutions To Problems of Chapter 3 3.1. Prove the least squares optimal solution for the linear regression case given in Eq. (3.13).

Solution: The cost function is J(θ)

=

N X

(yn − θ T xn )2

n=1

=

N X

(yn − θ T xn )(yn − xTn θ),

n=1

or J(θ) =

N X

yn2 − 2θ T

n=1

N X

! xn yn

+ θT

n=1

N X

! xn xTn

θ.

n=1

Taking the gradient of the cost with respect to θ and equating to zero, we finally get, !

N N X X T xn xn θ = xn yn n=1

n=1

3.2. Let θ̂i , i = 1, 2, . . . , m, be unbiased estimators of a parameter vector θ, i.e., E[θ̂i ] = θ, i = 1, . . . , m. Moreover, assume that the respective estimators are uncorrelated to each other and that all have the same (total) variance, σ 2 = E[(θi − θ)T (θi − θ)]. Show that by averaging the estimates, e.g., m

θ̂ =

1 X θ̂i , m i=1

1 2 the new estimator has total variance σc2 := E[(θ̂ − θ)T (θ̂ − θ)] = m σ .

Solution: First, it is easily checked out that the new estimator is also unbiased. By the definition of the total variance (which is the trace of the respective covariance matrix), we have σc2

= E[(θ̂ − θ)T (θ̂ − θ)]   !  m m T

X X 1 1  = E θ̂i − θ θ̂j − θ  m i=1 m j=1 =

m T

1 X 1 E θ̂i − θ θ̂j − θ = σ 2 , m2 i,j=1 m

since E[(θ̂i − θ)T (θ̂j − θ)] = δij σ 2 .

2

3.3. Let a random variable x being described by a uniform pdf in the interval [0, θ1 ], θ > 0. Assume a function1 g, which defines an estimator θ̂ := g(x) of θ. Then, for such an estimator to be unbiased, the following must hold: Z θ1 g(x)dx = 1.

0

However, such a function g does not exist.

Solution: Necessarily, the pdf of x must be ( p(x) =

θ, x ∈ [0, θ1 ], 0, otherwise.

For the estimator to be unbiased, Z ∞ E[θ̂]

=

g(x)p(x)dx −∞

Z θ1 =

g(x)θdx = θ,

∀θ > 0.

g(x)dx = 1,

∀θ > 0.

0

Hence, Z θ1 G(θ) :=

(1)

0

However, such a function g cannot exist. Indeed, one can easily verify by the basic integral theory that limθ→∞ G(θ) = 0, and limθ→0 G(θ) = 1, by (1). Clearly, these results contradict each other.

3.4. A family {p(D; θ) : θ ∈ A} is called complete if, for any vector function h(D) such that ED [h(D)] = 0, ∀θ, then h = 0.

Show that if {p(D; θ) : θ ∈ A} is complete, and there exists an MVU estimator, then this estimator is unique.

Solution: Assume two MVU estimators θ1 , θ2 . Then, ED [θ1 − θ2 ] = θ − θ = 0. Hence, by the definition of completeness, θ1 = θ2 .

3.5. Let θ̂u be an unbiased estimator, i.e., E[θ̂u ] = θ0 . Define a biased one by θ̂b = (1 + α)θ̂u . Show that the range of α where the MSE of θ̂b is smaller than that of θ̂u is −2 < −

2MSE(θ̂u ) MSE(θ̂u ) + θ02

< α < 0.

1 To avoid any confusion, let g be Lebesgue integrable on intervals of R.

3

Solution: The MSE for the new estimator is

i 2 h = E (1 + α)θ̂u − θ0 E (θ̂b − θ0 )2

2 = E (1 + α)(θ̂u − θ0 ) + αθ0 =

(1 + α)2 MSE(θ̂u ) + α2 θ02 .

To obtain smaller MSE for the unbiased estimator we must have (1 + α)2 MSE(θ̂u ) + α2 θ02 < MSE(θ̂u ), or, after using elementary algebra, " # 2 var(θ̂u ) α α+ < 0, θ02 + var(θ̂u ) where var(·) denotes the variance, and clearly MSE(θ̂u ) = var(θ̂u ). The solution of the previous inequality results to the desired interval: −

2 var(θ̂u ) < α < 0.

2 θ0 + var(θ̂u )

3.6. Show that for the setting of the Problem 3.5, the optimal value of α is equal to 1 α∗ = − , θ2 1 + var(0θ̂ ) u

where, of course, the variance of the unbiased estimator is equal to the corresponding MSE. Solution: The minimum value of h i MSE(θ̂b ) = E (θ̂b − θ0 )2 = (1 + α)2 MSE(θ̂u ) + α2 θ02 , with respect to α occurs when the derivative becomes zero, that is when 2(1 + α) var(θ̂u ) + 2αθ02 = 0, or, equivalently, when α∗ = −

var(θ̂u ) 1 =− . θ2 2 θ0 + var(θ̂u ) 1 + var(0θ̂ ) u

3.7. Show that the regularity condition for the Cramér-Rao bound holds true if the order of integration and differentiation can be interchanged.

4

Solution: By the definition of the expectation we have

Z Z ∂ ln p(X ; θ) ∂ ln p(X ; θ) E = · · · p(X ; θ) dx1 · · · dxN ∂θ ∂θ Z Z ∂p(X ; θ) dx1 · · · dxN = ··· ∂θ Z Z ∂ · · · p(X ; θ)dx1 · · · dxN = ∂θ ∂1 = = 0. ∂θ This is in general true, unless the domain where the pdf is nonzero depends on the unknown parameter θ.

3.8. Derive the Cramér-Rao bound for the LS estimator, when the training data result from the linear model yn = θxn + ηn ,

n = 1, 2, . . . ,

where xn and ηn are observations of i.i.d random variables, drawn from a zero mean random process, with variance σx2 , and a Gaussian white noise process, with zero mean and variance ση2 , respectively. Assume, also, that x and η are independent. Then, show that the LS estimator achieves the CR bound only asymptotically.

Solution: First, notice that in this case X = {(xn , yn )}N n=1 . That is, both yn as well as xn change as we change the training set. Define here the quantities xN := [x1 , x2 , . . . , xN ]T , yN := [y1 , y2 , . . . , yN ]T , and recall, also, the elementary relations p(X ; θ) = p(xN , yN ; θ) = p(yN |xN ; θ)p(xN ; θ) = p(yN |xN ; θ)p(xN ).

Hence, ln p(X ; θ) = ln p(yN |xN ; θ) + ln p(xN ), and eventually, ∂ ln p(yN |xN ; θ) ∂ ln p(X ; θ) = . ∂θ ∂θ Now, notice that by our original assumptions on the data model, !

N 1 1 X 2 p(yN |xN ; θ) = (yn − θxn ) , N/2 exp − 2σ 2 η n=1 2πσ 2 η

or N

1 X N 2 (yn − θxn )2 . ln p(yN |xN ; θ) = − ln 2πση − 2 2 2ση n=1

(2)

5

Thus, by (2), N N ∂ ln p(X ; θ) 1 X 1 X (yn − θxn ) xn = 2 ηn xn . = 2 ∂θ ση n=1 ση n=1

(3)

It can be readily verified by (3) that the regularity condition of the CramérRao Theorem is satisfied.

Now, N 1 X 2 ∂ 2 ln p(X ; θ) = − x . ∂θ2 ση2 n=1 n

Therefore

E

∂ 2 ln p(X ; θ) σ2 = −N x2 , 2 ∂θ ση

and the Cramér-Rao bound is given by var(θ̂) ≥

1 ση2 .

N σx2

We will now show that this bound cannot be achieved by any unbiased estimator. The necessary and sufficient condition for the existence of an MVU estimator that achieves the Cramér-Rao bound translates, for this case, to the existence of a function g(X ) such that ∂ ln p(X ; θ) σ2 = N x2 (g(X ) − θ), ∂θ ση However, looking at (3), it becomes apparent that such a factorization is not possible.

Let us now rewrite (3) as ∂ ln p(X ; θ) ∂θ

=

N

N

σx2 1 X 2 y x − θx n n n ση2 N σx2 n=1

N

N N σx2 1 X 1 X 2 y x − θ( x ) n n ση2 N σx2 n=1 N σx2 n=1 n

N

N σx2 1 X 2 g(X ) − θ( x ) ση2 N σx2 n=1 n

(4)

N 1 X yn xn .

N σx2 n=1

(5)

where g(X ) :=

For a large number the training data set, we assume the following PN N of 2 approximation: x ≈ N σx2 . By embedding this into (4) we obtain n=1 n

6

a form that allows for an unbiased estimator to attain the Cramér-Rao bound, and the corresponding estimate is given by θ̂ = g(X ) =

N 1 X yn xn .

N σx2 n=1

(6)

However, (6) is the LS estimator for large values of N . Indeed, by the definition of the LS estimator we have that !

N N 1 X 2 1 X xn θ̂ = xn yn , (7) N n=1 N n=1 which results in (6). It is easy to verify that (7) corresponds to an unbiased estimator.

Let us do it for the sake of an exercise. First of all, let us examine if the LS estimator for this more general case is unbiased. We have " # # " 1 X 1 X E[θ̂] = E P 2 xn yn = E P 2 xn (θxn + ηn ) n xn n n xn n " !# " # X X X 1 1 x2n + xn ηn xn ηn = E P 2 θ =θ+E P 2 n xn n xn n n n " " ## X 1 = θ + Ex P 2 Eη|x xn ηn n xn n = θ + 0 = θ.

In other words, the LS estimator is unbiased even for this case, where both output as well as input samples change in the training set and this is true independent of the number of measurements. The corresponding variance is given by  !2  h i X 1 E (θ̂ − θ)2 = E P xn ηn  2 ( n x2n ) n    X XX 1  = Ex  P x2n η2n + xi xj ηi ηj  2 Eη|x ( n x2n ) n i j6=i " P #

2 1 2 2 n xn P 2 . = ση Ex P 2 = ση E ( n x2n ) n xn Asymptotically, this provides the bound that we have previously derived.

However, for finite N , this is different.

3.9. Let us consider the regression model yn = θ T xn + ηn ,

n = 1, 2, . . . , N,

7

where the noise vector η := [η1 , . . . , ηN ]T comprises samples from zero mean Gaussian random variable, with covariance matrix Ση . If X := [x1 , . . . , xN ]T stands for the input matrix, and y = [y1 , . . . , yN ]T , the vector of the observations, then show that the corresponding estimator, θ̂ = X T Ση−1 X

−1

X T Ση−1 y,

is an efficient one.

Notice, here, that the previous estimator coincides with the Maximum Likelihood (ML) one. Moreover, bear in mind that in the case where the noise process is considered to be white, i.e., Ση = σ 2 IN , then the ML estimate becomes equal to the LS one.

Solution: In the case where the parameter θ becomes a k-dimensional vector, the Cramér-Rao bound takes a more general form than the one we have met previously, i.e., the case where the parameter θ is a scalar.

For any unbiased estimator g(X ) of the unknown parameter vector θ, the Cramér-Rao bound becomes as follows:

E (g(X ) − θ)(g(X ) − θ)T I −1 (θ), ∀θ, where I(θ) is the Fisher information matrix defined as

2 ∂ ln p(X ; θ) , I(θ) := − E ∂θ 2 and which is known to be a positive semidefinite matrix. Given any matrices A, B, of the same dimensions, the inequality A B means that the matrix A − B is positive semidefinite. A necessary and sufficient condition for g to be efficient is for the equation ∂ ln p(X ; θ) = I(θ) (g(X ) − θ) . ∂θ

(8)

For the present model, we have that X = y and

1 1 T −1 p(y; θ) = exp − (y − Xθ) Σ (y − Xθ) .

N 1 η 2 (2π) 2 (det Ση ) 2 T

Hence, ln p(y; θ) = − 12 (y − Xθ) Ση−1 (y − Xθ) + constant, and ∂ ln p(y; θ) = X T Ση−1 y − X T Ση−1 Xθ ∂θ

= X T Ση−1 X (X T Ση−1 X)−1 X T Ση−1 y − θ .

The second derivative is equal to ∂ 2 ln p(y; θ) = −X T Ση−1 X, ∂θ 2

(9)

8

so that the Fisher information matrix becomes I(θ) = X T Ση−1 X. By this, and by a simple inspection of (9), we can readily verify that (8) is −1 T −1 satisfied, if g(y) = X T Ση−1 X X Ση y. This establishes the claim.

3.10. Assume a set of i.i.d X := {x1 , x2 , . . . , xN } Gaussian random variables, with mean µ and variance σ 2 . Define also the quantities N

Sµ :=

1 X xn , N n=1

N

Sσ2 :=

1 X (xn − Sµ )2 , N n=1

N

S̄σ2 :=

1 X (xn − µ)2 .

N n=1

Show that if µ is considered to be known, a sufficient statistic for σ 2 is S̄σ2 .

Moreover, in the case where both (µ, σ 2 ) are unknown, then a sufficient statistic is the pair (Sµ , Sσ2 ).

Solution: The joint pdf of X is obviously 1 X − 2 (xn − µ)2 p(X ) = N exp 2σ n (2πσ 2 ) 2 1

! .

If only σ 2 is considered to be unknown, then notice that

N 1 2 exp − S̄ ≡ g1 (S̄σ2 , σ 2 ), p(X ; σ 2 ) = σ N 2σ 2 (2πσ 2 ) 2 where g1 is a function that depends only on S̄σ2 and the unknown σ 2 .

According to the Fisher-Neyman factorization theorem, the statistic S̄σ2 is sufficient.

Assume now the case where both (µ, σ 2 ) are unknown. Notice that by N X

(xn − µ)2 =

n=1

N X

(xn − Sµ )2 + N (Sµ − µ)2

n=1

= N Sσ2 + N (Sµ − µ)2 , the previous joint pdf becomes

1 2 p(X ; µ, σ ) = − 2 N Sσ2 + N (Sµ − µ) N exp 2σ (2πσ 2 ) 2

:= g2 (Sµ , Sσ2 ), (µ, σ 2 ) .

2

1

It can be readily verified that g2 depends only on the statistic (Sµ , Sσ2 ) and the unknowns (µ, σ 2 ). Hence, once again, by the Fisher-Neyman factorization theorem, the statistic (Sµ , Sσ2 ) is sufficient.

9

3.11. Show that solving the task

minimize

N X

L(θ, λ) =

yn − θ0 −

n=1

l X

!2 θi xni

+λ

i=1

l X

|θi |2 ,

i=1

is equivalent with minimizing

minimize

L(θ, λ) =

N X

l X

(yn − ȳ) −

n=1

!2 θi (xni − x̄i )

+λ

i=1

l X

|θi |2 ,

i=1

and the estimate of θ0 is given by θ̂0 = ȳ −

l X

θ̂i x̄i .

i=1

Solution: We have that L(θ0 , θ1:l ) =

N X

l X

yn − θ 0 −

n=1

θi xni

2

i=1

+

l X

θi2 .

i=1

Taking first the derivative with respect to θ0 and setting it equal to zero we obtain N l

X X ∂L = − 2(yn − θ0 ) + 2 θi xni = 0 ∂θ0 n=1 i=1 or N θ0 =

N X

yn −

n=1

l X i=1

θi

N X

xni ,

n=1

which results in θ0 = ȳ −

l X

θi x̄i .

i=1

That is, the optimum value for θ0 , is given in terms of the rest components.

Thus, optimizing with respect to θi , i = 1, 2, . . . , l, this has to be taken into account. Substituting the above in the Lagrangian, we get L(θ̂0 , θ1:l ) =

N X n=1

yn − ȳ −

l X i=1

l 2 X θi (xni − x̄i ) + θi2 , i=1

which proves the claim.

Note that the exact form of the regularizer does not enter into the game, since does not depend on θ0 . Hence, this technique of centering the data is also applicable to other forms of regularization.

10

3.12. This problem refers to Example 3.4, where a linear regression task with a real valued unknown parameter θ is considered. Show that MSE(θ̂b (λ)) < MSE(θ̂MVU ), i.e., the ridge regression estimate shows a lower MSE performance than the one for the MVU estimate, if  ση2 2  λ ∈ (0, ∞), θ ≤ N ,

 λ ∈

0,

2ση2

2 ση

θ2 − N

,

σ2

θ2 > Nη .

Moreover, the minimum MSE performance for the ridge regression estimate is attained at λ∗ = ση2 /θ2 .

Solution: Theory suggests that our estimate θ̂b is the solution of the task of minimizing the following loss function with respect to θ ∈ R: L(θ, λ) =

N X

(yn − θ)2 + λθ2 ,

λ ≥ 0.

n=1

The minimizer θ̂b will be obtained if we set the gradient dL(θ, λ)/dθ equal to zero, or equivalently, θ̂b (λ) =

N N N 1 X yn := θ̂MVU , N + λ N n=1 N +λ

where we used the notation θ̂b (λ) in order to highlight the dependence of the estimate θ̂b on the parameter λ. Notice here that E[θ̂b (λ)] = NN+λ θ0 , where θ0 is the estimandum.

Elementary calculus helps us to express MSE θ̂b (λ) as

2

2 + E[θ̂b (λ)] − θ0 MSE θ̂b (λ) = E θ̂b (λ) − E[θ̂b (λ)]

N 2 MSE θ̂MVU + λ2 θ02 = , (N + λ)2

(10)

and

2θ02 λ(N + λ)2 − 2 N 2 MSE θ̂MVU + λ2 θ02 (N + λ) d MSE θ̂b (λ) = . dλ (N + λ)4 (11) Let us first examine the range of values of λ > 0 which guarantee that MSE θ̂b (λ) < MSE θ̂MVU . The case of λ = 0 is excluded from the discussion, since in such a case, θ̂b (0) = θ̂MVU . It is easy to verify by (10)

11

that

MSE θ̂b (λ) < MSE θ̂MVU

⇔ λ θ02 − MSE θ̂MVU < 2N MSE θ̂MVU .

In the case where θ02 > MSE θ̂MVU , then the desired λ belongs to the interval (0, 2ση2 /(θ02 − ση2 /N )), where we have used the fact that

σ2 η .

MSE θ̂MVU = N

In the case where θ02 ≤ MSE θ̂MVU , notice that ∀λ > 0, we have that

λ θ02 − MSE θ̂MVU ≤ 0 < 2N MSE θ̂MVU , i.e., the desired λ belongs to the interval (0, ∞).

It is also easy to verify by equating the numerator of (11) to zero that

the λ∗ which minimizes MSE θ̂b (λ) becomes equal to ση2 /θ02 . To leave 2 no place for ambiguity, we remark here that in the case where θ0 > 2 2 2 2 MSE θ̂MVU = ση /N , this λ∗ belongs to the interval (0, 2ση /(θ0 −ση /N )), since ση2 ση2 2ση2 0 < λ∗ = 2 < 2 2 < . σ2 θ0 θ0 θ2 − η 0

N

3.13. Consider, once more, the same regression model as that of Problem 3.9, but with Ση = IN . Compute the MSE of the predictions E[(y−ŷ)2 ], where y is the true response and ŷ is the predicted value, given a test point x and using the LS estimator, θ̂ = X T X

−1

X T y.

The LS estimator has been obtained via a set of N measurements, collected in the input matrix X and y, where the notation has been introduced previously in this chapter. The expectation E[·] is taken with respect to to y, the training data, D and the test points x. Observe the dependence of the MSE on the dimensionality of the space.

Hint: Consider, first, the MSE, given the value of a test point x, and then take the average over all the test points.

Solution: From the theory, we have that given a point x, the LS estimator of the output is given by ŷ = yT X(X T X)−1 x.

12

Moreover, MSE(θ̂(x))

i h

T = E (y − ŷ)2 = E (θ T x + η − θ̂ x)2 h i h i = ση2 + ED xT (θ − θ̂)(θ − θ̂)T x + 2 ED|η (θ − θ̂)T x Eη [η] h i = ση2 + ED xT (θ − θ̂)(θ − θ̂)T x

= ση2 + xT Σθ̂ x = ση2 + ση2 xT (X T X)−1 x ,

where in place of the covariance matrix of the LS estimator we used Σθ̂ = ση2 (X T X)−1 .

Indeed, we have that (X T X)−1 X T y,

θ̂ =

(X T X)−1 X T (Xθ + η) = θ + (X T X)−1 X T η,

=

(12)

or θ̂ − θ = (X T X)−1 X T η.

Hence, Σθ̂

=

h i ED (X T X)−1 X T ηηT X(X T X)−1

=

(X T X)−1 X T Eη [ηηT ]X(X T X)−1

=

ση2 (X T X)−1 ,

(13)

where we used the fact that the variation in the data sets is solely due to noise variation (thus only to the output variables) and not to the input, X, which is a major assumption underlying the LS method.

We can now make the following approximation, for large values of N :

1 Σ := Ex xxT ≈ X T X, N where Σ is the covariance matrix of the (zero mean) input vectors. Then, we have MSE(θ̂) ≈ = = =

ση2 Ex xT Σ −1 x N

ση2 Ex trace Σ −1 xxT N

ση2 trace Σ −1 Ex xxT N

ση2 ση2 trace Σ −1 Σ = l.

N N

ση2 +

13

In other words, the MSE is proportional to the dimensionality of the space as well as the variance of the noise (which for the case of the problem is taken equal to one), and inversely proportional to the number of data points. That is, for given number of points and noise variance, the error depends on the dimensionality, which is a manifestation of the curse of dimensionality.

3.14. Assume that the model that generates the data is

yn = A sin

2π kn + φ + ηn , N

(14)

where A > 0, and k ∈ {1, 2, . . . , N − 1}. Assume that ηn are samples from a Gaussian white noise, of variance ση2 . Show that there is no unbiased estimator for the phase, φ, based on N measurement points, yn , n = 0, 1, . . . N − 1, that attains the Cramér-Rao bound.

Solution: The joint pdf of the measurements y := [y0 , y1 , . . . , yN −1 ]T is given by 2

N −1 1 1 X 2π p(y; φ) = kn + φ exp − y − A sin n 2σ 2 n=0 N (2πσ 2 )N/2

! .

The two derivatives of the ln of this function can be easily shown to be

N −1 ∂ ln p(y; φ) 2π 4π A X A = 2 yn cos kn + φ − sin kn + 2φ , ∂φ σ n=0 N 2 N (15) and

N −1 ∂ 2 ln p(y; φ) A X 4π 2π = − kn + φ + A cos kn + 2φ , y sin n ∂φ2 σ 2 n=0 N N and by substituting the value of yn from (14), the mean value becomes

E

N −1 ∂ 2 ln p(y; φ) A2 X 1 1 4π = − + cos kn + 2φ ∂φ2 σ 2 n=0 2 2 N =−

N A2 .

2σ 2

To derive the above, we used the trigonometric formula sin2 α = (1 −

14

cos(2α))/2, and also the fact N −1 X n=0

cos

4π kn + 2φ N

N −1

4π 1 X j ( 4π e N kn+2φ) + e−j ( N kn+2φ) = 2 n=0 =

N −1 N −1 1 2jφ X j 4π kn 1 −2jφ X −j 4π kn e e N + e e N 2 2 n=0 n=0 4π

4π

1 2jφ 1 − ej N kN 1 −2jφ 1 − e−j N kN e + e 4π 4π 2 2 1 − ej N k 1 − e−j N k = 0, √ 4π since e−j N kN = 1, where j := −1. =

Hence, if φ̂ stands for an unbiased estimator of φ, then var(φ̂) ≥

2σ 2 .

N A2

However, looking back at (15), we can verify that there does not exist a function g such that ∀y ∈ RN , N −1

g(y) − φ =

2 X N A n=0

yn cos

2π A 4π kn + φ − sin kn + 2φ .

N 2 N

Thus, even if an unbiased estimator exists, this cannot achieve the CramérRao bound.

3.15. Show that if (y, x) are two jointly distributed random vectors, with values in Rk × Rl , then the MSE optimal estimator of y given the value x = x is the regression of y conditioned on x, i.e., E[y|x].

S olution: The proof follows a similar line as the scalar case. Let f (x) := [f1 (x), . . . , fk (x)]T be the vector estimator. Then the MSE optimal one should minimize the sum of square errors per component, i.e., " k # k X X

2 E (yi − fi (x)) = E (yi − fi (x))2 . i=1

i=1

This is equivalent with minimizing k scalar terms individually, which can be carried out as in the text in the Chapter. The result of the ith problem is that the respective ith component of the MSE optimal estimator is given by, ĝi (x) = E[yi |x], or ĝ(x) = E[y|x].

15

Note that minimizing the sum of square errors per component is equivalent with minimizing the trace of the error covariance,

E eeT = E (y − f (x))(y − f (x))T .

3.16. Assume that x, y are jointly Gaussian random vectors, with covariance matrix

x − µx Σx Σxy Σ := E (x − µx )T , (y − µy )T = . y − µy Σyx Σy Assuming also that the matrices Σx and Σ̄ := Σy − Σyx Σx−1 Σxy are non-singular, then show that the optimal MSE estimator E[y|x] takes the following form, E[y|x] = E[y] + Σyx Σx−1 (x − E[x]).

Notice that E[y|x] is an affine function of x. In other words, for the case where x and y are jointly Gaussian, the optimal estimator of y, in the MSE sense, which is in general a non-linear function, becomes an affine function of x.

In the special case where x, y are scalar random variables, then E[y|x] = µy +

ασy (x − µx ) , σx

where α stands for the correlation coefficient, defined as α :=

E [(x − µx )(y − µy )] , σx σy

with |α| ≤ 1. Notice, also, that the previous assumption on the nonsingularity of Σx and Σ̄ translates, in this special case, to σx 6= 0 6= σy , and |α| < 1.

T Solution: First, it is easy to verify that Σyx = Σxy . Moreover, since Σx and Σ̄ are assumed to be non-singular, then it can be verified, e.g., [Magn 99], that the determinant det Σ = det Σx det Σ̄, and that

Σ

−1

−1 Σx + Σx−1 Σxy Σ̄ −1 Σyx Σx−1 = −Σ̄ −1 Σyx Σx−1

−Σx−1 Σxy Σ̄ −1 . Σ̄ −1

Observe that Σ̄ is the Schur complement of Σx in Σ. Also, the previous formula is the matrix inversion formula in terms of the Schur complement, as provided in the Appendix A of the book. To save space, let x̄ := x−µx

16

and ȳ := y − µy . Then, the joint pdf of x and y becomes

1 1 T T −1 x̄ p(x, y) = − x̄ , ȳ Σ 1 exp ȳ 2 (2π)l (det Σ) 2

1 1 1 − x̄T Σx−1 x̄ − x̄T Σx−1 Σxy Σ̄ −1 Σyx Σx−1 x̄ = 1 exp l 2 2 (2π) (det Σ) 2

1 +x̄T Σx−1 Σxy Σ̄ −1 ȳ − ȳ T Σ̄ −1 ȳ 2

1 1 exp − x̄T Σx−1 x̄ = 1 1 2 (2π)l (det Σx ) 2 (det Σ̄) 2

T −1 1 −1 −1 ȳ − Σyx Σx x̄ . − ȳ − Σyx Σx x̄ Σ̄ 2 As a result, the marginal pdf p(x) becomes

Z 1 1 T −1 exp − x̄ Σ x̄ p(x) = p(x, y)dy = 1 x 2 (2π)l/2 (det Σx ) 2

Z T 1 1 × exp − y − µy − Σyx Σx−1 x̄ 1 2 (2π)l/2 (det Σ̄) 2

−1 −1 × Σ̄ y − µy − Σyx Σx x̄ dy

1 T −1 1 − x̄ Σx x̄ . = 1 exp 2 (2π)l/2 (det Σx ) 2 Using the previous relations, we can easily see that p(y|x) =

p(x, y) p(x)

T ! ȳ − Σyx Σx−1 x̄ Σ̄ −1 ȳ − Σyx Σx−1 x̄ . = − l 1 exp 2 (2π) 2 (det Σ̄) 2 1

A simple inspection of this relation shows that the conditional pdf p(y|x) is Gaussian with covariance matrix Σ̄ and conditional mean E[y|x] = µy + Σyx Σx−1 (x − µx ).

3.17. Assume a number l of jointly Gaussian random variables {x1 , x2 , . . . , xl }, and a non-singular matrix A ∈ Rl×l . If x := [x1 , x2 , . . . , xl ]T , then show that the components of the vector y, obtained by y = Ax, are also jointly Gaussian random variables.

A direct consequence of this result is that any linear combination of jointly Gaussian variables is also Gaussian.

Solution: The Jacobian matrix of a linear transform y = Ax is easily shown to be J := J(y; x) = A.

17

Also, since A is non-singular, we have that x = A−1 y. Without any loss of generality, assume that E[x] = 0, which results into E[y] = 0. Hence,

Σy = E yyT = E AxxT AT = AΣx AT .

Clearly, det Σy = (det A)2 det Σx . Then, by the theorem of transformation for random variables, e.g., [Papo 02], we have the following: p(y) =

p(A−1 y) p(x) = | det J| | det A| 1

1 T −T −1 −1 exp − y A Σ A y 1 x 2 (2π)l/2 | det A|(det Σx ) 2

1 1 − y T Σy−1 y , = 1 exp l/2 2 2 (2π) (det Σy ) =

which establishes the first claim.

For the second claim, assume a non-zero vector a ∈ Rl , and define the linear combination of {x1 , x2 , . . . , xl } as y = aT x. Elementary linear algebra guarantees that there always exists a set of non-zero l-dimensional vectors {a1 , . . . , al−1 } such that the collection {a, a1 , . . . , al−1 } constitutes a basis of Rl [Magn 99]. Thus, the matrix A := [a, a1 , . . . , al−1 ]T ∈ Rl×l is non-singular, and the first component of the vector y = Ax is the quantity y = aT x. We have already seen by the first claim that the components of y are jointly Gaussian random variables. Moreover, a classical result states that if a number of random variables are jointly Gaussian, then each one of them, and thus y, is also Gaussian (the opposite is not always true) [Papo 02]. This establishes the second claim of the problem.

3.18. Let x ∈ Rl be a vector of jointly Gaussian random variables, of covariance matrix Σx . Consider the general linear regression model y = Θx + η, where Θ ∈ Rk×l is a parameter matrix and η is the vector of noise samples, which are considered to be Gaussian, with zero mean, and with covariance matrix Ση , independent of x. Then show that y and x are jointly Gaussian, with covariance matrix given by

ΘΣx ΘT + Σeta ΘΣx Σ= . Σx ΘT Σx Solution: The combined vector is given by

y Θx + η Θ Ik x = = . x x Il 0l×k η | {z } A

18

However, since x and η are both Gaussian vector variables, and mutually independent, then they are also jointly Gaussian. Notice also that the matrix A is non-singular; h indeed, i a simple permutation of the columns of Ik Θ A leads to the matrix 0l×k Il , whose determinant can be easily seen to be equal to 1.

Therefore, according to Problem 3.17, [yT , xT ]T is also jointly Gaussian.

The covariance matrix is a straightforward result following the definitions of the involved variables.

3.19. Show that a linear combination of Gaussian independent variables is also Gaussian.

Solution: This is a direct consequence of Problem 3.17, since independent Gaussian variables can be readily checked out that they are also jointly Gaussian.

3.20. Show that if a sufficient statistic T (X ) for a parameter estimation problem exists, then T (X ) suffices to express the respective ML estimate.

Solution: This is direct consequence of the Fisher-Neyman factorization theorem. Indeed, recall that T (X ) is sufficient iff the respective joint pdf can be factored as: p(X ; θ) = h(X )g(T (X ), θ), where h and g are appropriate functions. Hence, by the definition of the ML estimate, θ̂ML ≡ arg maxθ p(X ; θ) = arg maxθ g(T (X ), θ).

In other words, T (X ) is sufficient, via g, to obtain the ML estimate.

3.21. Show that if an efficient estimator exists then it is also optimal in the ML sense.

Solution: Assume the existence of an efficient estimator, i.e., a function g which achieves the Cramér-Rao bound. A necessary and sufficient condition for g to be efficient, is for (8) to hold true for all values of θ. Since (8) holds for all values of θ, then it holds true for θ = θ̂ML . However, for this value, the left-hand-side of (8) becomes zero, and since I(θ̂ML ) is non-singular, we obtain that g(X ) = θ̂ML . This establishes the claim.

3.22. Let the observations resulting from an experiment be xn , n = 1, 2, . . . , N .

Assume that they are independent and that they originate from a Gaussian PDF N (µ, σ 2 ). Both, the mean and the variance, are unknown. Prove that the ML estimates of these quantities are given by N

µ̂ML =

1 X xn , N n=1

N

2 σ̂ML =

1 X (xn − µ̂ML )2 .

N n=1

Solution: The log-likelihood function is given by N N N 1 X 2 L(µ, σ ) = − ln(2π) − ln σ − 2 (xn − µ)2 .

2 2 2σ n=1 2

19

Taking the gradient with respect to µ, σ 2 , and equating it to zero we obtain the following system of equations N

−

1 X (xn − µ) σ 2 n=1

=

0

N 1 X N 1 + (xn − µ)2 2 σ2 2σ 4 n=1

=

0.

The solution of this system leads trivially to the required result.

3.23. Let the observations xn , n = 1, 2, . . . , N, come from the uniform distribution ( 1 , 0 < x ≤ θ, p(x; θ) = θ 0, otherwise.

Obtain the ML estimate of θ.

Solution: The likelihood function is given by L(x; θ) =

N Y 1 n=1

θ

=

1 . θN

We know that θ ≥ xn , n = 1, . . . , N , or equivalently, θ ≥ maxn=1,...,N xn .

Hence, the likelihood function is maximized by taking the minimum value of θ, which is θ̂ML = max{x1 , x2 , . . . , xN }.

3.24. Obtain the ML estimate of the parameter λ > 0 of the exponential distribution ( λ exp(−λx), x ≥ 0, p(x) = 0, x < 0, based on a set of measurements, xn , n = 1, 2, . . . , N .

Solution: The log-likelihood function is L(x; λ) = N ln λ − λ

N X

xn .

n=1

Taking the derivative and equating to zero we obtain N N X − xn = 0, λ n=1

which leads to

N λ̂ML = PN

n=1 xn

.

20

3.25. Assume an µ ∼ N (µ0 , σ02 ), and a stochastic process {xn }∞ n=−∞ , consisting of i.i.d random variables, such that p(xn |µ) = N (µ, σ 2 ). Consider N observations so that X ≡ {x1 , x2 , . . . , xN }, and prove that the posterior p(x|X ), of any x = xn0 conditioned on X , turns out to be Gaussian with 2 mean µN and variance σ 2 + σN , where µN ≡

N σ02 x̄ + σ 2 µ0 , N σ02 + σ 2

2 σN ≡

σ 2 σ02 .

N σ02 + σ 2

Solution: From basic theory we have that N

p(µ|X ) = R

Y p(X |µ)p(µ) = αp(µ) p(xk |µ), p(X |µ)p(µ)dµ k=1

or

N

1 (µ − µ0 )2 Y 1 1 (xk − µ)2 α √ exp − exp − 2 σ02 2 σ2 2πσ0 2πσ k=1 2 2 !!

N µ − µ0 1 X µ − xk + = α1 exp − 2 σ σ0 k=1 ! !!

N X µ N 1 1 1 0 xk + 2 µ + 2 µ2 − 2 = α2 exp − 2 σ2 σ0 σ2 σ0 k=1

1 N σ02 + σ 2 N σ02 x̄ + σ 2 µ0 = α2 exp − µ2 − 2µ 2 σ 2 σ02 σ 2 σ02

2 2 2 N σ0 + σ N σ0 x̄ + σ 2 µ0 2 = α2 exp − µ − 2µ 2σ 2 σ02 N σ02 + σ 2

2 (µ − µN ) , = α3 exp − 2 2σN

p(µ|X ) = √

where α, α1 , α2 , α3 are factors independent of µ, and N

1 X x̄ := xk , N k=1

N σ02 x̄ + σ 2 µ0 µN := , N σ02 + σ 2 σ 2 σ02 2 σN := .

N σ02 + σ 2 Since p(µ|X ) is a PDF, then necessarily α3 = √

1 .

2πσN

21 2 Hence, limN →∞ σN = 0, and for large N , p(µ|X ) behaves like a δ function centered around µN . Thus,

Z 1 (x − µN )2 . p(x|X ) = p(x|µ)p(µ|X )dµ ' p(x|µN ) = √ exp − 2σ 2 2πσ

Therefore, p(x|X ) tends to a Gaussian pdf with mean µN and variance σ 2 . Furthermore, limN →∞ µN = x̄.

For the general case of any value of N , and not only the case of large N , we have Z p(x|X ) = p(x|µ)p(µ|X )dµ

Z 1 (µ − µN )2 (x − µ)2 = exp − dµ.

(16) exp − 2 2πσσN 2σ 2 2σN Hence, in order to obtain p(x|X ), the previous integration has to take place. Here we will follow another path, which avoids any direct integration. Assume a random variable y defined as y := ξ + ν, where 2 ), independent of each other. It is wellξ ∼ N (0, σ 2 ) and ν ∼ N (µN , σN known [Papo 02] that the PDF of y is given by the joint PDF of ξ and ν as follows: Z p(y) = pξν (y − ν, ν)dν.

However, since ξ and ν are assumed to be independent, then pξν (ξ, ν) = pξ (ξ)pν (ν), and Z p(y) = pξ (y − ν)pν (ν)dν

Z 1 (y − ν)2 (ν − µN )2 = dν, exp − exp − 2 2πσσN 2σ 2 2σN which is identical to (16). However, recall from basic statistics [Papo 02] that y, being the sum of two independent Gaussians is also Gaussian (see, also, Problem 3.19), with mean the sum of the mean values and variance the sum of the variances. Therefore, (16) becomes

1 (x − µN )2 p(x|X ) = p exp − 2 ) .

2 ) 2(σ 2 + σN 2π(σ 2 + σN 3.26. Show that for the linear regression model, y = Xθ + η, the a-posteriori probability p(θ|y) is a Gaussian one, if the prior distribution probability is given by p(θ) = N (θ, Σ0 ), and the noise samples

22

follow the multivariate Gaussian distribution p(η) = N (0, Ση ). Compute the mean vector and the covariance matrix of the posterior distribution.

Solution: It can be easily checked that p(θ|y) = const×exp − 21 Ψ , where Ψ = (y − Xθ)T Ση−1 (y − Xθ) + (θ − θ0 )T Σ0−1 (θ − θ0 ) = y T Ση−1 y − 2y T Ση−1 Xθ + θ T X T Ση−1 Xθ + (θ − θ0 )T Σ0−1 (θ − θ0 ).

From now on, all terms that will be independent of θ will be collected in constant terms. Hence Ψ = α1 − 2y T Ση−1 Xθ + (θ − θ0 )T Σ0−1 (θ − θ0 ) + (θ − θ0 )T X T Ση−1 X(θ − θ0 ) − θ0T X T Ση−1 Xθ0 + 2θ0T X T Ση−1 Xθ.

As a result,

Ψ = α2 + (θ − θ0 )T Σ0−1 + X T Ση−1 X (θ − θ0 ) + 2θ0T X T Ση−1 Xθ − 2y T Ση−1 Xθ

= α3 + (θ − θ0 )T Σ0−1 + X T Ση−1 X (θ − θ0 ) T

− 2 (y − Xθ0 ) Ση−1 X (θ − θ0 ) .

(17)

In the sequel, we will follow a standard trick that we do in situations like that. We introduce an auxiliary variable θ̄, whose value is to be determined so that to make the following to be true,

Ψ = α4 + (θ − θ0 − θ̄)T Σ0−1 + X T Ση−1 X (θ − θ0 − θ̄)

= α4 + (θ − θ0 )T Σ0−1 + X T Ση−1 X (θ − θ0 )

+ θ̄ T Σ0−1 + X T Ση−1 X θ̄

T −1 − 2θ̄ T Σ−1 (18) 0 + X Ση X (θ − θ0 ).

Inspection of (17) and (18) indicates that this can happen if we choose θ̄ = Σ0−1 + X T Ση−1 X

−1

X T Ση−1 (y − Xθ0 ) .

Then, we can finally write that T

−1 Ψ = α4 + (θ − E[θ|y]) Σθ|y (θ − E[θ|y]) ,

where E[θ|y] = θ0 + Σ0−1 + X T Ση−1 X

−1

X T Ση−1 (y − Xθ0 ) ,

and Σθ|y = Σ0−1 + X T Ση−1 X

−1

.

23

3.27. Assume that xn , n = 1, 2 . . . , N , are i.i.d observations from a Gaussian N (µ, σ 2 ). Obtain the MAP estimate of µ, if the prior follows the exponential distribution p(µ) = λ exp (−λµ) ,

λ > 0, µ ≥ 0.

Solution: Upon defining X := {x1 , x2 , . . . , xN }, the posterior distribution is given by

N λ exp (−λµ) Y (xn − µ)2 . p(µ|X ) ∝ p(X |µ)p(µ) = exp − 2σ 2 (2π)N/2 σ N n=1 Taking the ln, differentiating with respect to µ, and equating to zero we obtain

PN ∂ −λµ − 2σ1 2 n=1 (xn − µ)2 = 0, ∂µ or N 1 X −λ + 2 (xn − µ) = 0. σ n=1 Finally, PN µ̂MAP =

n=1 xn − λσ

N for nonnegative values of the numerator.

2

,

24

Bibliography [Magn 99] Magnus, J. R., and Neudecker, H. Matrix Differential Calculus with Applications in Statistics and Econometrics. John Wiley & Sons, revised Ed., 1999.

[Papo 02] Papoulis, A., and Unnikrishna, P. Probability, Random Variables, and Stochastic Processes. McGraw Hill, 4th Ed., 2002.

25

1

Solutions To Problems of Chapter 4 4.1. Show that the set of equations Σθ = p has a unique solution if Σ > 0 and infinite many if Σ is singular.

Solution: a) Let Σ > 0. Then the linear system of equations has a unique solution. The converse is also true. Let the linear system has a unique solution, θ∗ . Then Σ > 0. Indeed, if this was not the case, then there exists a nonzero vector, a 6= 0, such that Σa = 0.

Then θ = θ∗ + a is also a solution, which contradicts the uniqueness assumption. b) Let Σ is singular and let θ∗ be a solution. Then, any vector a ∈ N (Σ), i.e., the null subspace, will also be a solution. For the converse, since there are infinite many solutions, let two of them, θ1 6= θ2 . Since, Σθ1 6= Σθ2 ⇒ Σ(θ1 − θ2 ) = 0, hence Σ is singular.

We still have to show that there always exists a solution. This is proved in the next problem.

4.2. Show that the set of equations Σθ = p has always a solution.

Solution: The existence of solution when Σ > 0 is obvious. Let Σ be singular. Then, in order to guarantee a solution, we have to show that p lies in the range space of Σ. For this, it suffices to show that p ⊥ a, ∀a ∈ N (Σ).

Since L the range spaces and null spaces are orthogonal to each other and R(Σ) N (Σ) = Rl , then if p is orthogonal to N (Σ), it will necessary lie in the column space (range) of matrix Σ.

Let us now assume that ∃ a ∈ N (Σ), such as aT p 6= 0.

This means that, Σa = 0 ⇒ aT Σa = 0 ⇒ aT E[xxT ]a = 0 or 2 E[ aT x ] = 0 ⇒ aT x = 0.

Hence, aT p = aT E[xy] = E[(aT x)y] = 0, which contradicts the claim aT p 6= 0.

2

4.3. Show that the shape of the isovalue contours of the mean-square error (J(θ)) surface, J(θ) = J(θ∗ ) + (θ − θ∗ )T Σ(θ − θ∗ ), are ellipses whose axes depend on the eigenstructure of Σ.

Hint: Assume that Σ has discrete eigenvalues.

Solution: Since Σ is symmetric, we know that it can be diagonalized, Σ = QΛQT , where Λ is the diagonal matrix with eigenvalues along the main diagonal and Q the unitary matrix (QQT = I) comprising the respective eigenvectors as its columns. Then we have J(θ)

= J(θ∗ ) + (θ − θ∗ )T QΛQT (θ − θ∗ ) = J(θ∗ ) + v T Λv = J(θ∗ ) +

l−1 X

λi vi2 ,

(1)

i=0

where v = QT (θ − θ∗ ).

For a fixed value of J(θ), Eq. (1) corresponds to an (hyper)ellipse centered at v = 0, (or equivalently at θ = θ∗ ). Its principle axes are parallel to those obtained by rotating the original coordinates ([1, 0, ..., 0]T , [0, 1, ..., 0]T , [0, 0, ..., 1]T ) by Q; that is, parallel to the eigenvectors of Σ.

4.4. Prove that if the true relation between the input x and the true output y is linear, i.e., y = θoT x + v, θo ∈ Rl where v is independent of x, then the optimal MSE θ∗ satisfies θ∗ = θo .

Solution: The optimal parameter vector is given by Σθ∗

=

E[xy] = E[x xT θo + v ]

=

Σθo ,

or θ∗ = θo .

The noise variance is equal to σv2

=

E[ y − θoT x y − xT θo ]

=

σy2 − 2θoT E[xy] + θoT Σθo

=

σy2 − θoT Σθo = M SE(θ∗ ).

3

4.5. Show that if y = θoT x + v, θo ∈ Rk where v is independent of x, then the optimal MSE θ∗ ∈ Rl , l < k is equal to the top l components of θo , if the components of x are uncorrelated.

Solution: Let θo =

1 θo , θo1 ∈ Rl , θo2 ∈ Rk−l . θo2

Then y=

h

T θo1

T θo2

i x l

φ 

,

 x(l + 1)   .. φ :=  . . x(k) The optimal θ∗ ∈ Rl is given by Σθ∗

= E[xl y]

T T = E xl θo1 xl + θo2 φ + v = Σθo1 + E[xl φT ]θo2 .

If the input variables are mutually uncorrelated, θ∗ = θo1 . If not θ∗ = θo1 + Σ −1 E[xl φT ]θo2 , that is, it is equal to θo1 plus a weighted combination of the components of θo2 .

4.6. Derive the normal equations by minimizing the cost in (4.15).

Hint: Express the cost in terms of the real part θr and its imaginary part θi of θ and optimize with respect to θr , θi .

Solution: The cost function is

J(θ) := E |y − θ H x|2 .

Let x = xr + jxi and θ = θr + jθi .

Then we obtain J(θ) ≡ J(θr , θi ) = E[|(yr + jyi ) − (θrT − jθiT )(xr + jxi )|2 ]

= E[| yr − θrT xr − θiT xi + j yi − θrT xi + θiT xr |2 ] 2 2 = E[ yr − θ T x + yi − θ T x̃ ],

(2)

4

where

θr xr xi θ = , x = , x̃ = . θi xi −xr

(3)

The cost in (2) can now be written as J(θr , θi )

= E[yr2 ] − 2 E[xT yr ]θ + θ T E[x xT ]θ + + E[yi2 ] − 2 E[x̃T yi ]θ + θ T E[x̃ x̃T ]θ .

Taking the derivative with respect to θ we obtain

θr T T E[x x ] + E[x̃ x̃ ] = E[x yr ] + E[x̃ yi ] θi and after some algebra we obtain

θr E[xr yr ] + E[xi yi ] Θ̃ = , θi E[xi yr ] − E[xr yi ] where

E[xr xTr ] + E[xi xTi ] E[xr xTi ] − E[xi xTr ] . Θ̃ := E[xi xTr ] − E[xr xTi ] E[xi xTi ] + E[xr xTr ]

Verify now that this is the same set of equations as the normal equations, after equating the real and imaginary parts on both sides, in Σθ = p.

4.7. Consider the multichannel filtering task

ŷr x ŷ = =Θ r . ŷi xi Estimate Θ so that to minimize the error norm: E[||y − ŷ||2 ].

Solution: The error norm is equal to

2

2 J(Θ) = E yr − ŷr + E yi − ŷi

2

2 T T T T = E yr − θ11 xr − θ12 xi + E yi − θ21 xr − θ22 xi , where Θ :=

T θ11 T θ21

T θ12 T . θ22

Hence, one can separately optimize the two terms in the cost with respect to (θ11 , θ12 ) and (θ21 , θ22 ), respectively.

From the first term, working similarly as in Problem 4.6 we obtain

E[xr xTr ] E[xr xTi ] θ11 E[xr yr ] = E[xi yr ] E[xi xTr ] E[xi xTi ] θ12 and a similar set of equations, as in theory, results from the second term.

5

4.8. Show that (4.34) is the same as (4.25).

Solution: By the respective definitions, we have ŷr + jŷi

=

(θrT − jθiT )(xr + jxi ) + (vrT − jviT )(xr − jxi ),

or after some trivial algebra T ŷr θr + vrT = ŷi −θiT − viT

θiT − viT θrT − vrT

xr , xi

(4)

which results in (4.25) after substituting in (4) the definitions in (4.32) and (4.33).

4.9. Show that the MSE achieved by a linear complex-valued estimator is always larger than that obtained by a widely linear one. Equality is achieved only under the circularity conditions.

Solution: The minimum MSE for the linear filter is M SEl

= E[(y − θ∗H x)(y∗ − xH θ∗ )] = E[y2 ] + θ∗H E[xxH ]θ∗ − θ∗H E[xy∗ ] − E[yxH ]θ∗ = E[y2 ] − E[xH y]θ∗ = E[y2 ] − pH Σx−1 p,

where Σx = E[xxH ], p = E[xy∗ ].

Similarly we can show that M SEwl = E[y2 ] − p̃H Σ̃x−1 p̃, where

p Σx p̃ = ∗ , Σ̃x = q P∗

P , Σx∗

with P = E[xxT ], q = E[xy].

Using the matrix inversion lemma and after a bit of algebra, we obtain M SEl − M SEwl = (q ∗ − P ∗ Σx−1 q)H (Σx∗ − P ∗ Σx−1 P )−1 (q ∗ − P ∗ Σx−1 q).

However, Σx∗ − P ∗ Σx−1 P is the Schur compliment of Σ̃x and since we know that is positive definite, the Schur compliment is also positive definite (as well as Σx ). This is a well known result from linear algebra. Thus M SEl − M SEwl ≥ 0. Note that under circularity conditions, P = 0 and q = 0, leading to M SEl = M SEwl .

6

4.10. Show that under the second order circularity assumption, the conditions in (4.39) hold true.

Solution: By the second order circularity condition we have E[xxT ] = 0, or E[(xr + jxi )(xTr + jxTi )] = 0, or E[xr xTr ] − E[xi xTi ] = 0 ⇒ Σr = Σi , and E[xr xTi ] + E[xi xTr ] = 0 ⇒ Σri = −Σir .

The rest are shown in a similar way.

4.11. Show that if f : C −→ R, then the Cauchy-Riemann conditions are violated.

Proof: Let f (x + jy) = u(x, y) ∈ R. Then by assumption, the imaginary part v(x, y) is identically zero. Hence the Cauchy-Riemann conditions, i.e., ∂v ∂u ∂v ∂u = , =− , ∂x ∂y ∂y ∂x are violated, since this would only hold true if ∂u ∂u = = 0, ∂x ∂y which is the trivial case of a constant function.

4.12. Derive the optimality condition in (4.45).

Solution: We will show that any other filter, hi , i ∈ Z, results in a larger MSE compared to the filter wi , i ∈ Z, which satisfies the condition. Indeed, we have that X X

A := E (dn − hi un−i )2 = E (dn − (hi − wi + wi )un−i )2 . i

i

After expanding the term in squares, we obtain that X

X 2 A = E (dn − wi un−i )2 + E (hi − wi )un−i − i

i

X X

2 (hj − wj ) E (dn − wi un−i )un−j . j

i

7

However, by assumption the last term is zero. Hence, the error associated with hi is always larger than that of the filter associated with the optimality condition.

4.13. Show Equations (4.50) and (4.51).

Solution: a) Eq. (4.51): We know from Chapter 2 that the power spectrum of the output process, y(n, m) is related to that of the input, d(n, m), by Sy (ω1 , ω2 ) = |H(ω1 , ω2 )|2 Sd (ω1 , ω2 ).

Let now, u(n, m) = y(n, m) + η(n, m), where η(n, m) is independent of y(n, m). The autocorrelation of u is now given as h

i ru (i, j) = E y(n, m)+η(n, m) y(n−i, m−i)+η(n−i, m−j) = ry (i, j)+rη (i, j), and taking the Fourier transform the claim is proved. b) Eq. (4.50): By the respective definition we have that,

rdu (k, l) = E d(n, m)u(n − k, m − l) , and also

+∞ X

u(n, m) =

+∞ X

h(i, j)d(n − i, m − j).

i=−∞ j=−∞

Combining the previous two equations, we obtain rdu (k, l) =

+∞ X

+∞ X

h(i, j) E d(n, m)d(n − i − k, m − j − l) ,

i=−∞ j=−∞

or rdu (k, l) =

+∞ X

+∞ X

h(i, j)rd (i + k, j + l).

i=−∞ j=−∞

Setting i0 = i + k, j 0 = j + l, we obtain +∞ X

rdu (k, l) =

+∞ X

rd (i0 , j 0 )h(i0 − k, j 0 − l)

i0 =−∞ j 0 =−∞

or rdu (k, l) =

+∞ X

+∞ X

i0 =−∞ j 0 =−∞

rd (i0 , j 0 )h(−(k − i0 ), −(l − j 0 )).

8

Taking into account that if H is the Fourier transform of h(n, m) then the sequence h(−n, −m) has as Fourier transform H ∗ , then we finally obtain that Sdu (ω1 , ω2 ) = H ∗ (ω1 , ω2 )Sd (ω1 , ω2 ).

4.14. Derive the normal equations for Example 4.2.

Solution: We have E[un un ]

=

E

0.5sn + sn−1 + ηn 0.5sn + sn−1

+ηn =

0.25rs (0) + rs (0) + rη (0)

1.25σs2 + ση2 ,

= E 0.5sn + sn−1 + ηn 0.5sn−1 + sn−2

+ηn−1

= E[un un−1 ]

=

E un un−2 ]

0.5σs2 ,

= E[ 0.5sn + sn−1 + ηn 0.5sn−2 + sn−3

+ηn−2 =

0.

Similarly,

= E un sn−1

= E (0.5sn + sn−1 + ηn ) sn−1 = σs2 ,

E un−1 dn = E un−1 sn−1

= E (0.5sn−1 + sn−2 + ηn−1 ) sn−1 = 0.5σs2 ,

E un−2 dn = E un−2 sn−1 = 0.

E[un dn ]

4.15. The input to the channel is a white noise sequence sn of variance σs2 . The output of the channel is the AR processes yn = a1 yn−1 + sn .

(5)

The channel also adds white noise ηn of variance ση2 . Design an optimal equalizer of order two, which at its output recovers an approximation of sn−L . Sometimes, this equalization task is also known as whitening, since in this case the action of the equalizer is to “whiten” the AR process.

Solution: The input to the equalizer is un = yn + ηn .

The desired response is dn = sn−L .

9

Thus, the elements of the input covariance/autocorrelation matrix are given by • ru (0) = E[un un ] =ry (0) + ση2

= E[(yn + ηn )(yn + ηn )] =

σs2 + ση2 1 − a21

• ru (1) = E[(yn + ηn )(yn−1 + ηn−1 )] a1 σs2 .

1 − α12 For the cross-correlation, we have

• p(0) := E[un sn−L ] = E[ yn + ηn sn−L ] = E[yn sn−L ].

Applying the recursion in (5), we have that =ry (1) =

• p(0) := E[ α1L yn−L +

L−1 X

α1j sn−j sn−L ]

j=0

=α1L E[yn−L sn−L ],

L ≥ 1.

Due to stationarity, it suffices to compute the following:

• E[yn−L sn−L ] = E[yn sn ] = E[ a1 yn−1 + sn sn ] = σs2 . Hence, 2 p(0) = aL 1 σs .

Note that the last equation is also valid for L = 0, as a direct consequence of (5).

• p(1) := E[un−1 sn−L ] = E[ α1L−1 yn−L +

L−2 X

αj sn−1−j s(n − L)] = α1L−1 σs2 , L ≥ 2.

j=0

The latter one is also valid for L = 1. However, for L = 0 it is readily seen from (5), that p(1) = 0. Hence the equalizer is obtained via the following normal equations  2 

L 2 σs a1 σs2 2 + σ 2 2 α1 σs w∗ (0) η 1−a1  1−a1 2  = L−1 a1 σs σs2 2 w (1) α σs2 ∗ 1 2 2 + σ 1−a1

1−a1

η

4.16. Show that the forward and backward MSE optimal predictors are conjugate reverse of each other.

Solution: By the respective definitions we have −1 ∗ am = Σm rm

and −1 bm = Σm Jm rm .

10

Since Σm is a Hermitian Toeplitz matrix, it is easily checked out that ∗ Jm Σm Jm = Σm

or −1 −1 ∗ Jm Σm Jm = (Σm ) .

Hence, since Jm Jm = Im , −1 −1 ∗ bm = Jm Jm Σm Jm rm = Jm (Σm ) rm or

b∗m = Jm am .

Hence for the minimum MSE backward error we have b αm

=

H −1 H −1 ∗ r(0) − rm Jm Σm Jm rm = r(0) − rm (Σm ) rm

=

T −1 ∗ f r(0) − rm Σm rm = αm .

b f ) are updated according = αm 4.17. Show that the MSE prediction errors (αm to the recursion b b αm = αm−1 (1 − |κm−1 |2 ).

Solution: By the respective definition we have b αm

H −1 H H ∗ = r(0) − rm Jm Σm Jm rm = r(0) − rm Jm bm = r(0) − rm am ( )

H

a∗m−1 −b∗m−1 ∗ r∗ (m) = r(0) − rm−1 + κm−1 0 1

H H = r(0) − rm−1 a∗m−1 + r∗ (m) − rm−1 b∗m−1 κ∗m−1 H b = r(0) − rm−1 a∗m−1 − |κm−1 |2 αm−1

=

H b r(0) − rm−1 Jbm−1 − |κm−1 |2 αm−1

=

b αm−1 (1 − |κm−1 |2 ).

4.18. Derive the BLUE in the Gauss-Markov theorem.

Solution: The optimization task is H∗ s.t.

Let

:=

arg minH trace{HΣη H T }, HX = I.

 T h1 hT2 

  H :=  .  , X = x1 , x2 , · · · , xl .  ..  hTl

11

Observe that trace{HΣη H T } =

l X

hTi Ση hi .

(6)

i=1

Also, HX = I is equivalent with hTi xi = 1,

(7)

hTi xj = 0, j 6= i.

(8)

and Since Ση is positive definite, our optimization task is equivalent with l constrained minimization problems, one for each hi , i = 1, 2, ..., l. Using Lagrange multipliers, the Lagrangian for each one of the tasks is written as l X (i) (i) min hTi Ση hi − λi (hTi xi − 1) − λj hTi xj . hi

j6=i

Taking the gradient we obtain, 2Ση hi −

l X

(i)

λj xj = 0 ⇒

j=1

 (i)  λ1  (i)  1 λ2  1 Ση hi = Xλ(i) := X  .  , 2 2   ..  (i) λl or hi

=

hTi

=

1 −1 Σ Xλ(i) ⇒ 2 η 1 (i) T T −1 λ X Ση , i = 1, 2, .., l, 2

or H = ΛX T Ση−1 , where

 T λ(1)  (2) T   λ 1 , Λ=  .  2  ..   λ(l)

T

is computed from the constraints, i.e., ΛX T Ση−1 X = I Λ = (X

T

⇒

Ση−1 X)−1

or H∗ = (X T Ση−1 X)−1 X T Ση−1 .

12

4.19. Show that the mean-square error (which in this case coincides with the variance of the estimator) of any linear unbiased estimator is higher than that associated with the BLUE. Solution: The mean-square error of any H is given by M SE(H) = trace{HΣη H T }.

However, (H − H∗ )Ση (H − H∗ )T = HΣη H T − HΣη H∗T − H∗ Ση H T + H∗ Ση H∗T , or HΣη H T = (H − H∗ )Ση (H − H∗ )T + HΣη H∗T + H∗ Ση H T − H∗ Ση H∗T .

Recall that H∗ = (X T Ση−1 X)−1 X T Ση−1 and also HX = H∗ X = I. Hence, HΣη H∗T = HΣη Ση−1 X(X T Ση−1 X)−1 = (X T Ση−1 X)−1 .

Similarly, H∗ Ση H T = (X T Ση−1 X)−1 and H∗ Ση H∗T

=

(X T Ση−1 X)−1 X T Ση−1 Ση Ση−1 X(X T Ση−1 X)−1

=

(X T Ση−1 X)−1 .

Thus, HΣη H T = (H − H∗ )Ση (H − H∗ )T + H∗ Ση H∗T .

Hence, trace{HΣη H T }

=

X

hTi Ση hi =

i

X (hTi − hT∗,i )Ση (hi − h∗,i )

+

trace{H∗ Ση H∗T }.

i

Since Ση is positive definite, all terms are positive and the minimum occurs if hi = h∗,i . Moreover, the minimum MSE is M SE(H∗ ) = trace{H∗ Ση H∗T }.

However, H∗ Ση H∗T = (X T Ση−1 X)−1 or M SE(H∗ ) = trace{(X T Ση−1 X)−1 }.

13

4.20. Show that if Ση is positive definite, then X T Ση−1 X is also positive definite if X is full rank.

Solution: Recall from linear algebra that if X is full rank, then X T X is positive definite and vice versa. Indeed, assume that this is not the case. Then, there will be a 6= 0, such that X T Xa = 0, or ||Xa||2 = αT X T Xa = 0 ⇔ Xa = 0, which contradicts the assumption that X is full rank.

For our case, we know that Ση is symmetric and positive definite. Thus, Ση = U ΛU T , where Λ is the diagonal matrix with the eigenvalues of Ση and U the unitary matrix with the orthonormal eigenvectors as its columns. Hence, X T Ση−1 X = X(U ΛU T )−1 X = XU Λ−1 U T X = B T B, where B = Λ−1/2 U T X. Note that if X is full rank, then B is also full rank, hence B T B is positive definite.

4.21. Derive a MSE optimal linearly constrained widely linear beamformer.

Solution: The output of the widely linear beamformer is given by ŝ(t)

wH u(t) + v H u∗ (t)

=

:= w̃eH (t)ũe (t), where

ũe (t) =

u(t) w , w̃ = . e u∗ (t) v

The constraint must guarantee the distortionless response for signals impinging on the array from the direction φ, associated with the steering vector x. That is, ŝ(t) = wH xs(t) + v H x∗ s∗ (t) = s(t).

Thus, wH x = 1 and v H x∗ = 0.

14

The error signal is s(t) − ŝ(t)

=

s(t) − wH xs(t) − wH η(t) − v H x∗ s∗ (t) − v H η∗ (t)

=

−wH η(t) − v H η∗ (t) = −w̃eH η̃(t),

where

η(t) η̃(t) = ∗ , η (t)

and the constraints have been taken into account. Thus, the error variance w̃eH Ση̃ w̃e must be minimized, where

Ση Pη η(t)ηH (t) η(t)ηT (t) = E Ση̃ = .

Pη∗ Ση∗ η∗ (t)ηH (t) η∗ (t)ηT (t) Let us write the two constraints in a more compact form to facilitate the optimization. To this end, we have that

1 H X w̃e = , 0 where

x X := 0

0 x 0 = x̃1 x̃2 , x̃1 := , x̃2 := ∗ . x∗ 0 x

Thus, our task now becomes minimize w.r. w̃e s.t.

w̃eH Ση̃ w̃e w̃eH X = [1, 0],

or using the Lagrange multipliers, L(w̃e ) := w̃eH Ση̃ w̃e − λ1 (w̃eH x̃1 − 1) − λ2 w̃eH x̃2 .

Minimization of the Lagrangian results in

1 λ Ση̃ w̃e = X 1 . λ2 2 Hence, plugging the previous into the constraint, we get

1 H H w̃e X = [1, 0] ⇒ X w̃e = ⇒ 0

λ /2 1 X H Ση̃−1 X 1 = λ2 /2 0 or

1 λ1 1 = (X H Ση̃−1 X)−1 0 2 λ2

15

and finally w̃e = Ση̃−1 X(X H Ση̃−1 X)−1

1 .

0

Note that, if the noise vector is circular, i.e., Pη = 0, the results become identical to the linear beamformer.

4.22. Prove that the Kalman gain that minimizes the error variance matrix Pn|n = E[(xn − x̂n|n )(xn − x̂n|n )T ], is given by Kn = Pn|n−1 HnH (Rn + Hn Pn|n−1 HnT )−1 .

Hint: Use the following formulas ∂ trace{AB} ∂A ∂ trace{ACAT } ∂A

=

B T (AB a square matrix)

=

2AC, (C = C T ).

Solution: We know that x̂n|n = x̂n|n−1 + Kn (yn − Hn x̂n|n−1 ).

Thus, Pn|n

= E (xn − x̂n|n )(xn − x̂n|n )T

T = E (xn − x̂n|n−1 ) − Kn en (xn − x̂n|n−1 ) − Kn en ,

where en = yn − Hn x̂n|n−1 .

Hence, Pn|n = Pn|n−1 − Kn E[en eTn|n−1 ] − E[en|n−1 eTn ]KnT + Kn E[en eTn ]KnT .

However, E[en eTn|n−1 ]

=

E[(Hn xn + vn − Hn x̂n|n−1 )eTn|n−1 ]

=

Hn Pn|n−1 + E[vn (xn − x̂n|n−1 )T ]

=

Hn Pn|n−1 .

Similarly, E[en|n−1 eTn ] = Pn|n−1 HnT .

Moreover, E[en eTn ]

=

E[(yn − Hn x̂n|n−1 )(yn − Hn x̂n|n−1 )T ]

T E[ Hn (xn − x̂n|n−1 ) + vn Hn (xn − x̂n|n−1 ) + vn ]

=

Rn + Hn Pn|n−1 HnT .

=

16

Hence, we have that Pn|n = Pn|n−1 −Kn Hn Pn|n−1 −Pn|n−1 HnT KnT +Kn (Rn +Hn Pn|n−1 HnT )KnT , (9) or trace{Pn|n }

=

trace{Pn|n−1 } − 2 trace{Kn Hn Pn|n−1 } +

trace Kn (Rn + Hn Pn|n−1 HnT )KnT .

(10)

Taking the derivative (gradient) with respect to Kn , we obtain ∂ trace{Pn|n } = −2Pn|n−1 HnT + 2Kn (Rn + Hn Pn|n−1 HnT ) = 0 ∂Kn or Kn = Pn|n−1 HnT (Rn + Hn Pn|n−1 HnT )−1 .

4.23. Show that in Kalman filtering, the prior and posterior error covariance matrices are related as Pn|n = Pn|n−1 − Kn Hn Pn|n−1 .

Solution: Recall from the solution of the Problem 4.22 that Pn|n = Pn|n−1 −Kn Hn Pn|n−1 −Pn|n−1 HnT KnT +Kn (Rn +Hn Pn|n−1 HnT )KnT .

Plug in the above the optimum Kalman gain Kn = Pn|n−1 HnT (Rn + Hn Pn|n−1 HnT )−1 , which results in the desired update.

4.24. Derive the Kalman algorithm in terms of the inverse state-error covariance −1 matrices, Pn|n . In statistics, the inverse error covariance matrix is related to Fisher’s information matrix, hence the name of the scheme.

Solution: To build the Kalman algorithm around the inverse state-error −1 −1 covariance matrices Pn|n , Pn|n−1 , we need to apply the following matrix inversion Lemmas, P B T (R + BP B T )−1 = (P −1 + B T R−1 B)−1 B T R−1 (A + BD

−1

−1

C)

=A

−1

−A

−1

−1

B(D + CA

−1

B)

−1

CA

(11)

.

According to the text we have: −1 • Sn−1 = (Rn +Hn Pn|n−1 HnT )−1 = Rn−1 −Rn−1 Hn (Pn|n−1 +HnT Rn−1 Hn )−1 HnT Rn−1 , −1 • Kn = Pn|n−1 HnT (Rn +Hn Pn|n−1 HnT )−1 = (Pn|n−1 +HnT Rn−1 Hn )−1 HnT Rn−1 ,

• Pn|n = Pn|n−1 − Pn|n−1 HnT (Rn + Hn Pn|n−1 HnT )−1 Hn Pn|n−1 ,

17

which gives −1 −1 • Pn|n = (Pn|n−1 + HnT Rn−1 Hn )−1 , T • Pn+1|n = Fn+1 Pn|n Fn+1 + Qn+1 ,

and the last one results in −1 −1 −1 −1 T −1 T Pn+1|n = Q−1 Fn+1 Q−1 n+1 − Qn+1 Fn+1 (Pn|n + Fn+1 Qn+1 Fn+1 ) n+1 .

1

Solutions To Problems of Chapter 5 5.1. Show that the gradient vector is perpendicular to the tangent at a point of an isovalue curve.

Solution: The differential of the cost function, J(θ), at a point θ (i) , is given by dJ(θ (i) )

= =

∂J(θ (i) )

(i)

∂J(θ (i) )

(i) dθl (i) (i) ∂θ1 ∂θl (i) (i) T (i) [dθ1 , . . . , dθl ] ∇J(θ ) = dθ (i)T ∇J(θ (i) ).

dθ1 + . . . +

However, moving along the isovalue curve crossing θ (i) makes the differential value zero. Hence, the gradient vector is perpendicular to dθ (i) , whose direction defines the respective tangent plane.

5.2. Prove that if

∞ X

∞ X

µ2i < ∞,

i=1

µi = ∞,

i=1

the steepest descent scheme, for the MSE loss function and for the iterationdependent step size case, converges to the optimal solution.

Solution: The basic iteration update is given by θ (i) = θ (i−1) − µi ∇J(θ (i−1) ), or θ (i) − θ∗ = θ (i−1) − θ∗ − µi ∇J(θ (i−1) ), where θ∗ is the optimal point. Thus, (θ (i) − θ∗ )T (θ (i) − θ∗ )

(θ (i−1) − θ∗ )T (θ (i−1) − θ∗ ) +

=

µ2i ∇T J(θ (i−1) )∇J(θ (i−1) ) − 2µi (θ (i−1) − θ∗ )T ∇J(θ (i−1) ), and applying the above recursively, we get (θ (i) − θ∗ )T (θ (i) − θ∗ )

=

(θ (0) − θ∗ )T (θ (0) − θ∗ ) + i X

µ2k ∇T J(θ (k−1) )∇J(θ (k−1) ) −

k=1

2

i X

µk (θ (k−1) − θ∗ )T ∇J(θ (k−1) ).

k=1

However, ∇T J(θ (k−1) )∇J(θ (k−1) ) ≥ 0.

2

Also, since the MSE loss function is a convex one (hence the current proof carries on to any other convex loss function) we have that1 J(θ∗ ) ≥ J(θ (k) ) + ∇T J(θ (k) )(θ∗ − θ (k) ), ∀k, from which we have that (θ (k−1) − θ∗ )T ∇J(θ (k−1) ) ≥ 0.

Hence, assuming that the gradient is bounded at every point, then we can write, (θ (i) − θ∗ )T (θ (i) − θ∗ ) − (θ (0) − θ∗ )T (θ (0) − θ∗ ) ≤ A

i X

µ2k −

k=1

2

i X

µk (θ (k−1) − θ∗ )T ∇J(θ (k−1) ),

k=1

for some A > 0. However, the term on the left hand side is bounded from below, since P the first term is always positive and if it diverges it will go to i +∞. Also, k=1 µ2k tends to zero, by assumption. Thus, the only way, Pi if we assume that k=1 µk diverges, that the right hand side does not diverge to −∞, is when θ (i) −→ θ∗ .

Otherwise the left hand side becomes unbounded form below, which is a contradiction. Hence the claim has been proved.

5.3. Derive the steepest gradient descent direction for the complex-valued case.

Solution: We know from the text that = J(θr , θi ) + ∆θrT ∇r J(θr , θi ) +

J(θ + ∆θ)

∆θiT ∇i J(θr , θi ).

Simplifying a bit the notation and taking into account that the real and imaginary parts can be expressed in terms of the complex number and its conjugate we obtain, J(θ + ∆θ)

=

J(θ) +

1 T ∆θ + ∆θ H ∇r J(θ) + 2

1 T ∆θ − ∆θ H ∇i J(θ), 2j or J(θ + ∆θ)

1 T ∆θ ∇r J(θ) − j∇i J(θ) + 2

1 H ∆θ ∇r J(θ) + j∇i J(θ) , 2

= J(θ) +

1 This is the definition of convexity, and it is valid for any points in place of θ

∗ and θ

(k) .

3

and finally, taking into account that J(θ) is a real function, J(θ + ∆θ) = J(θ) +

1 T ∆θ (∇θ∗ J(θ))∗ + ∆θ H ∇θ∗ J(θ) , 2

which proves the claim, since the addition of a number with its complex conjugate gives its real part.

5.4. Let θ, x be two jointly distributed random variables. Let also the function (regression) f (θ) = E[x|θ], assumed to be an increasing one. Show that under the conditions in (5.29) the recursion θn = θn−1 − µn xn converges in probability to the root of f (θ).

Solution: Let us divide xn into two parts, i.e.,

Figure 1: Figure for Problem 5.4.

xn = f (θn ) + ηn .

Then, E[ηn |θn ] = E[xn |θn ] − f (θn ) = 0.

Moreover, we will assume that the respective variance is finite, E[η2n ] = σ 2 < ∞,

4

and also we assume that ηn and θn are independent. Let θ∗ be a root.

Then, we form θn − θ∗ = θn−1 − θ∗ − µn f (θn−1 ) − µn ηn .

Taking the expectation of the square we get E[(θn − θ∗ )2 ]

=

E[(θn−1 − θ∗ )2 ] + µ2n E[f 2 (θn−1 )] + µ2n E[η2n ] − 2µn E[(θn − θ∗ )f (θn−1 )],

and applying the previous one iteratively we obtain 2

2

E[(θn − θ∗ ) ] − E[(θ0 − θ∗ ) ]

=

n X

µ2i E[f 2 (θi )] + E[η2i ] −

i=1 n X

2

µi E[(θi − θ∗ )f (θi )].

i=1

Assume now that E[f 2 (θn−1 )] ≤ b < ∞.

Then we obtain the following, E[(θn − θ∗ )2 ] − E[(θ0 − θ∗ )2 ≤ (b + σ 2 )

n X

µ2i −

i=1

2

n X

µi E[(θi − θ∗ )f (θi )].

(1)

i=1

However, from Figure 1, it is readily seen that f (θ) > 0 if (θ − θ∗ ) > 0 f (θ) < 0 if (θ − θ∗ ) < 0 f (θ) = 0 if (θ − θ∗ ) = 0.

As a matter of fact, the above are the crucial conditions for convergence.

Note that even if a function is decreasing, one can consider its negative form and still find the root. Thus, these conditions are quite general, provided a single root exists.

Thus, f (θ)(θ − θ∗ ) ≥ 0,

(2)

and hence, E[(θi − θ∗ )f (θi )] ≥ 0.

If both conditions for µn hold true, then the first term on the right hand side of (1) tends to zero. Also, the left hand side is bounded from below.

5

The only way that the P second term remains bounded from below, and having assumed that i µi diverges to ∞ is the following one to be true, lim E[(θn−1 − θ∗ )f (θn )] = 0.

n−→∞

However, since (2) is valid for all θ, the last condition is equivalent to lim Pr {θn = θ∗ } = 1,

n−→∞

which proves the claim.

5.5. Show that the LMS algorithm is a nonlinear estimator.

Solution: Consider the basic LMS recursion θn

= θn−1 + µxn en

en

= yn − xTn θn−1 .

Applying it recursively and assuming that we start from θ−1 = 0, we obtain n X θn = µ x i ei . i=0

Note that ei depends on xi , and θi−1 depends on xi−1 , which shares common elements with xi , which are multiplied. Moreover, the output of the filter n X ei xTi xn , ŷn = θnT xn = µ i=0

which also verifies the nonlinearity.

5.6. Show equation (5.42).

Solution: Our starting point is Σc,n

=

Σc,n−1 − µΣx Σc,n−1 − µΣc,n−1 Σx

+

2µ2 Σx Σc,n−1 Σx + µ2 Σx trace{Σx Σc,n−1 } + µ2 ση2 Σx

(3)

Take into account that QQT = QT Q = I and also that QT trace{A}Q = trace{QT AQ}I, for any matrix A. Hence, QT Σc,n Q = QT Σc,n−1 Q − µQT Σx QQT Σc,n−1 Q − µQT Σc,n−1 QQT Σx Q +

2µ2 QT Σx QQT Σc,n−1 QQT Σx Q

+ µ2 QT Σx QQT trace{Σx QQT Σc,n−1 }Q + µ2 ση2 QT Σx Q. which then proves the claim.

(4)

6

5.7. Derive the bound in (5.45).

Hint: Use the well known property from linear algebra, that the eigenvalues of a matrix, A ∈ Rl×l , satisfy the following bound, max |λi | ≤ max

1≤i≤l

1≤i≤l

l X

|aij | := kAk1 .

j=1

Solution: The elements (entries) of the matrix 2

A := (I − µΛ) + µ2 Λ2 + µ2 λλT are:

Aij =

(1 − µλi )2 + µ2 λ2i + µ2 λ2i , i = j µ2 λi λj , i 6= j.

Note that all the entries are positive, since Σx is positive definite and has positive eigenvalues. Hence, it is required that   l   X max (1 − µλi )2 + µ2 λ2i + µ2 λi λj < 1. i   j=1

This is a sufficient condition for stability and guarantees that all eigenvalues of A have magnitude less than one, or 1 − 2µλi + 2µ2 λ2i + µ2 λi

l X

λj < 1 or

j=1

 µ 2λi +

l X

 λj  < 2,

j=1

or, 0<µ<

2λi +

2 Pl

j=1 λj

≤ <

2 Pl 2λmin + j=1 λj 2 2 = , Pl trace{Σ x} j=1 λj

which proves the claim.

5.8. Gershgorin circle theorem. Let A be an l × l matrix, with entries aij , Pl i, j = 1, 2, . . . , l. Let Ri := j=1 |aij |, be the sum of absolute values of j6=i

the non-diagonal entries in row i. Then show that if λ is an eigenvalue of A, then there exists at least one row i, such that the following is true, |λ − aii | ≤ Ri .

7

The last bound defines a circle, which contains the eigenvalue λ.

Solution: By the definition of an eigenvalue we have that Ax = λx.

Let |xi | be the maximum entry, i.e., i : |xi | ≥ |xj |, j 6= i.

Then, we can write that l X

aij xj = λxi

j=1

or −aii xi + λxi =

l X

aij xj

j6=i

or |λ − aii | =

|

Pl

j6=i aij xj |

|xi |

≤

l X j6=i

l

|xj | X |aij | ≤ |aij | := Ri , |xi | j6=i

which proves the claim. The same is true if we add the elements columnwise. It suffices to apply the theorem on AT .

5.9. Apply the Gershgorin circle theorem to prove the bound in (5.45).

Solution: First note that the matrix A = (I − µΛ)2 + µ2 Λ2 + µ2 λλT is non-negative definite. Indeed ∀x ∈ Rl , we have that xT (I − µΛ)T (I − µΛ)x + µ2 xT ΛT Λx + µ2 xT λλT x ≥ 0.

Hence all its eigenvalues are non-negative; recall that the eigenvalues are real, since A = AT . Hence, there must be a row, i, such that |λmax | − |aii | ≤ |λmax − aii | ≤

l X

|aij | = µ2 λi

j6=i

l X

λj ,

j6=i

since the off-diagonal elements are µ2 λi λj . Hence, 0 ≤ λmax

≤

aii + µ2 λi

l X

λj

j=1

=

(1 − µλi )2 + µ2 λ2i + µ2 λi

l X

λj

j=1

which then proves the claims following the steps in Problem 5.7.

8

5.10. Derive the misadjustment formula given in (5.52).

Solution: Taking into account that in steady state, sn = sn−1 := s in (5.42), for the ith element we can write si = (1 − 2µλi + 2µ2 λ2i )si + µ2 λi

X

λj sj + µ2 ση2 λi .

j

It is common to neglect the third term in the parenthesis on the right hand side, for very small values of µ, hence, si ' (1 − 2µλi )si + µ2 λi

X

λj sj + µ2 ση2 λi .

j

Summing up both sides we obtain, X i

si '

X

si − 2µ

i

X

λi si +

i

X

µ2 λi

X

i

λj sj + µ2 ση2

j

X

λi

i

and using (5.48), we get !

Jexc,∞

2−µ

X

λi

= µση2

i

X

λi .

i

Note that the sum of the eigenvalues is equal to the trace of a matrix.

This completes the proof.

5.11. Derive the APA iteration scheme.

Solution: The optimization task is θn = arg min kθ − θn−1 k2 s.t.

θ T xn−i θ = yn−i ,

i = 0, . . . , q − 1.

Using Lagrange multipliers, the Lagrangian becomes L(θ, λ) = (θ − θn−1 )T (θ − θn−1 ) +

q−1 X

λi (yn−i − θ T xn−i ),

i=0

Taking the derivatives and equating to zero, we get q−1

θ

=

1X λi xn−i θn−1 + 2 i=0

=

1 θn−1 + XnT λ, 2

(5)

9

where

 xTn   Xn =  ...  , XnT = [xn , . . . , xn−q+1 ]. xTn−q+1 

Substituting (5) into the constraints, written as yn = Xn θ or

−1 1 λ = Xn XnT (yn − Xn θn−1 ).

2

Hence θn

= θn−1 + XnT Xn XnT

en

= yn − Xn θn−1 ,

−1

en

5.12. Given a value x, define the hyperplane comprising all values of θ such as xT θ − y = 0.

Then x is perpendicular to the hyperplane .

Solution: Let two points that belong to the hyperplane, θ1 and θ2 . Then by definition we have x T θ1 − y T

x (θ1 − θ2 )

= x T θ2 − y ⇒ =

0.

Since this is true for any θ1 , θ2 in the hyperplane, x is perpendicular to it.

5.13. Derive the recursions for the widely-linear APA. Solution: Define en = yn − φH n−1 x̃n , where

x x̃n = n∗ . xn

Let also

 x̃H n   X̃n =  ...  . x̃H n−q+1 

For the APA, we require ϕH x̃n−i = yn−i , i = 0, . . . , q − 1.

10

or (ϕ∗ )T x̃n−i = yn−i , or ∗ x̃H n−i ϕ = yn−i , i = 0, 1, . . . , q − 1,

or X̃n ϕ = yn∗ ,

(6)

with yn = [yn , . . . , yn−q+1 ]T .

The Lagrangian becomes (ϕ − ϕn−1 )H (ϕ − ϕn−1 ) +

q−1 X

∗ λi (yn−i − x̃H n−i ϕ)

i=0

or H H ϕH ϕ + φH n−1 ϕn−1 − ϕn−1 ϕ − ϕ ϕn−1 +

q−1 X

∗ λi (yn−1 − x̃H n−i ϕ).

i=0

Taking the gradient with respect to ϕ and treating ϕ∗ as a constant and equating to 0 we obtain ϕ∗ = ϕ∗n−1 + X̃nT λ,

(7)

ϕ = ϕn−1 + X̃nH λ∗ .

(8)

or Plugging (8) into (6) we obtain λ∗ = (X̃n X̃nH )−1 e∗n where e∗n = yn∗ − X̃n ϕn−1 and the widely linear APA has been obtained.

5.14. Show that a similarity transformation of a square matrix, via a unitary matrix, does not affect the eigenvalues.

Solution: Let Σ be a matrix and let Σ´ = T H ΣT. Let λ be an eigenvalue of Σ´. Then Σ´q

= λq ⇒

T ΣT q

= λq ⇒

ΣT q

= λT q

H

Hence λ is also an eigenvalue of Σ. This is true for all eigenvalues of Σ´.

Thus, both matrices share the same eigenvalues.

11

5.15. Show that if x ∈ Rl is a Gaussian random vector, then F := E[xxT SxxT ] = Σx trace{SΣx } + 2Σx SΣx and if x ∈ Cl , F := E[xxH SxxH ] = Σx trace{SΣx } + Σx SΣx Solution: Let  λ1  ..

H Q Σx Q = Λ =  .

0

··· .. . ···

 0 ..  , . λl

where we know that λi ≥ 0, i = 1, 2, . . . , l. Define x0 = QH x.

Then Σx0 = E[x0 x0H ] = QH Σx Q = Λ.

Thus the entries of x0 are uncorrelated, hence independent since they are Gaussians. Define F 0 = QH F Q = =

E[QH xxH QQH SQQH xxH Q] E[x0 x0H S 0 x0 x0H ],

where S 0 = QH SQ. • For real data, the matrix under the expectation will comprise entries of the form 0 E[(x0k1 x0k2 x0k3 x0k4 )Sij ].

Since the variables are independent, the only non zero entries will be those such as k1 = k2 = k3 = k4 , or k1 = k2 and k3 = k4 , k1 = k4 and k2 = k3 or k1 = k3 and k2 = k4 . Then it can be checked out (use e.g., a 3 × 3 matrix example) that F 0 = Σx0 trace{S 0 Σx0 } + 2Σx0 S 0 Σx0 which results in F = Σx trace{SΣx } + 2Σx SΣx .

• For complex valued data, we assume circular symmetric variable, thus E[x2k ] = E[x02 k ] = 0, which results in F = Σx Trace{SΣx } + Σx SΣx .

12

5.16. Show that if a l × l matrix C is right stochastic, then all its eigenvalues satisfy |λi | ≤ 1, i = 1, 2, . . . , l.

The same holds true for left and doubly stochastic matrices.

Solution: By the definition of a stochastic matrix, its elements are nonnegative and each row adds to 1. Recalling the Gershgorin circle theorem, we know that for any matrix, all the eigenvalues satisfy ∃i : |λ − cii | ≤

l X

|cij | =

j=1 j6=i

l X

cij

j=1 j6=i

for at least one i. That is ∃i : −

l X

cij ≤ λ − cii ≤

l X

cij

j=1 j6=i

j=1 j6=i

or −1 ≤ λ ≤ 1 or |λ| ≤ 1.

Moreover note that λ = 1 is an eigenvalue, since C1 = 1.

For the left and doubly stochastic, use the fact that C, C T share the same eigenvalues.

5.17. Prove Theorem 5.2 Solution: Collect all estimates from all nodes at iteration, i, together in a common vector, T (i)

θ (i) = [θ1

T (i)

, θ2

T (i)

, . . . , θK ]T ; θk ∈ Rl ; k = 1, 2, . . . , K.

Then the consensus iteration can be compactly written (check it) as, θ (i) = AT θ (i−1) ,

(9)

where AT is defined as the Kronecker product AT = AT ⊗ Il with θ (0) = [xT1 , . . . , xTk ]T , xK ∈ Rl , k = 1, 2, . . . , K.

(10)

13

Let

K

x :=

1 X xk .

K k=1

Then, by the definition of the Kronecker product it can be written as x=

1 1T ⊗ Il θ (0) .

K

Define (i)

cik := θk − x, that is, the error vector. Then, we can write that, T (i)

c(i) := [c1

T (i) T

, . . . , ck

] = θ (i) − (1 ⊗ Il )x.

• Sufficiency. Iterating (9) we obtain θ (i) = AT

i

θ (0) ;

or c(i)

i

θ (0) − (1 ⊗ Il )x i 1 = AT θ (0) − (1 ⊗ Il )(1T ⊗ Il )θ (0) K

1 T i A − (1 ⊗ Il )(1T ⊗ Il ) θ (0) . = K AT

=

Using the identity (B ⊗ C)(D ⊗ E) = (BD ⊗ CE), we obtain c(i)

= =

1 11T ⊗ Il Il θ (0) K

1 i 11T ⊗ Il θ (0) .

AT − K

AT

i

−

Recalling the definition of A in (10) and the properties (B + C) ⊗ D = B ⊗ D + C ⊗ D, (B ⊗ C)i = B i ⊗ C i , we obtain c

(i)

1 T T i = A ) − 11 ⊗ Il θ (0) K

However, note that

i 1 1 AT − 11T = (AT )i − 11T .

K K

(11)

14

This is because of the doubly stochastic assumption on A. Indeed

AT −

1 T 11 K

2

1 T T 1 11 A − AT 11T K K

=

(AT )2 −

+

1 11T 11T , K2

and since AT 1 = 1, and A1 = 1, and 1T 1 = K we get

2 1 1 AT − 11T = (AT )2 − 11T , K K which generalizes to higher powers by induction. Hence ! i

1 T (i) T c ⊗ Il θ (0) = A − 11 K !i

1 T T = A − 11 ⊗ Il θ (0) , K due to the property (B ⊗ C)i = B i ⊗ C i .

It is known from the linear algebra that the eigenvalues of the Kronecker product B ⊗ C are given by all combinations of the products of the eigenvalues of B and C. Since A is a consensus matrix, this guarantees that all eigenvalues are less than one and the iteration converges to zero.

• Necessary. Assume that θ (i) = AT

i

θ (0)

converges, i.e., lim AT

i→∞

i

θ (0)

=

(1 ⊗ Il )x

=

1 (1 ⊗ Il )(1T ⊗ Il )θ (0) K

for any θ (0) . Hence, this implies that lim AT

i→∞

i

= =

1 (1 ⊗ Il )(1T ⊗ Il ) K 1 (11T ⊗ Il ).

K

15

However, AT

i

= (AT ⊗ Il )i = AT

i

⊗ Il .

Hence, combining lim AT

i→∞

i

1 T 11 .

K

=

(12)

This in turn implies that lim AT AT

i

i→∞

= =

Thus

1 T 11 K i+1 lim AT .

AT

i→∞

1 T T 1 A 11 = 11T , K K

or AT 1 = 1, that is, it is left stochastic. By considering lim AT

i

i→∞

AT =

1 T T 11 A .

K

we can prove that it is right stochastic as well, A1 = 1.

Since we have proved that A is doubly stochastic, then (11) holds true. Thus, from (12), we get

lim

i→∞

AT

i

−

1 T 11 K

= lim

i→∞

1 AT − 11T K

i = 0,

which can only be true if the respective eigenvalues are less than one.

This concludes the proof.

1

Solutions To Problems of Chapter 6 6.1. Show that if A ∈ Cm×m is nonnegative definite, its trace is nonnegative.

Solution: By the definition of a positive semidefinite matrix, ∀x ∈ Cm , xH Ax ≥ 0.

Hence, this will also be true for x = [1, 0, . . . , 0]. Thus, [A]11 ≥ 0. In the same way, we can prove that any element in the diagonal is a nonnegative real number.

6.2. Show that under a) the independence assumption of successive observation vectors and b) the presence of white noise independent of the input, then the LS estimator is asymptotically distributed according to the normal distribution, i.e., √ N (θ − θ0 ) −→ N (0, σ 2 Σx−1 ), where σ 2 is the noise variance and Σx the covariance matrix of the input observation vectors, assuming that it is invertible.

Solution: Recall that according to the law of large numbers, we have N

1 X xn xTn −→ Σx , N n=1 where the limit indicates convergence in probability. Also, E[xn ηn ] = E[xn ] E[ηn ] = 0.

Moreover, we have that, Cov(xn ηn ) = E[η2n xn xTn ] = Ex [E[ηn2 |xn ]xn xTn ] = σ 2 Σx .

Thus, the covariance matrix of the sum of independent terms in N

1 X √ xn ηn N n=1

(1)

is given as, Σxη = σ 2 Σx .

Also, due to the independence of the summands and from the central limit theorem, we have that N

1 X √ xn ηn −→ N (0, σ 2 Σx ).

N n=1

2

Combining the previous findings and using the result from the text, we get !−1 !

N N √ 1 X 1 X T √ xn xn N (θ − θ0 ) = xn ηn N n=1 N n=1 −→

Σx−1 N (0, σ 2 Σx ) = N (0, σ 2 Σx−1 ),

(2)

where the limit is meant to be convergent in distribution. To prove the above we have made use of the so called Slutsky’s theorem, which states the following: If xn −→ x, in distribution and Yn −→ Y , in probability, where Y is a constant, then Yn−1 xn −→ Y −1 x, where the last limit is in distribution.

6.3. Let X ∈ Cm×l . Then show that the two matrices, XX H and X H X, have the same nonzero eigenvalues.

Solution: Let λi be an eigenvalue of X H X. Then (X H X)vi

=

λi vi ⇒

X(X H X)vi

=

λi Xvi ⇒

(XX H )Xvi

=

λi Xvi ,

or (XX H )qi = λi qi .

That is, λi is also an eigenvalue of XX H .

6.4. Show that if X ∈ Cm×l , then the eigenvalues of XX H (X H X) are real and nonnegative. Moreover, show that if λi 6= λj , vi ⊥ vj .

Solution: By the definition we have, XX H vi

=

λi vi ⇒

viH XX H vi kX H vi k2

=

λi kvi k2 ⇒

=

λi kvi k2 ⇒

λi

≥

0.

Let now λi 6= λj . Then we have, XX H vi H vj XX H vi

= λi vi ⇒ = λi vjH vi ,

3

and since XX H is Hermitian and λi real, viH XX H vj = λi viH vj .

(3)

Similarly, XX H vj

= λj vj ⇒

viH XX H vj

= λj viH vj

(4)

Subtracting (3) from (4) we obtain (λi − λj )(viH vj ) = 0 ⇒ vi ⊥ vj .

Note that it can be shown that for Hermitian matrices, if λi = λj , still one can construct orthogonal eigenvectors.

6.5. Let X ∈ Cm×l . Then show that if vi is the normalized eigenvector of X H X, corresponding to λi 6= 0, then the corresponding normalized eigenvector ui of XX H is given by, 1 ui = √ Xvi λi Solution: From Problem 6.3, we know that the eigenvectors XX H and X H X corresponding to λi are related as, qi = Xvi .

By the respective definition we have XX H vi H vi XX H vi

= λi vi ⇒ = λi kvi k2 = λi

for normalized vi , or kXvi k2 = kqi k2 = λi .

Thus the normalized ui will be 1 1 ui = √ qi = √ Xvi λi λi 6.6. Show Eq. (6.19).

Solution: From the respective definitions, we get ŷ

=

X(X T X)−1 X T y = Ul DVlT (Vl DUlT Ul DVlT )−1 Vl DUlT y

=

Ul DVlT (Vl D2 VlT )−1 Vl DUlT y

=

Ul DVlT (Vl D−2 VlT )Vl DUlT y

=

Ul UlT y,

where we have used the definition of the unitary (V V T = V T V = I) .

4

6.7. Show that the right singular vectors, v1 , . . . , vr , corresponding to the r singular values of a rank-r matrix, X, solve the following iterative optimization task: compute vk , k = 2, 3, . . . , r, such as, 1 ||Xv||2 , 2 ||v||2 = 1,

minimize subject to

v ⊥ {v1 , . . . , vk−1 }, k 6= 1, where || · || denotes the Euclidean norm.

Solution: We start with k = 1, to solve the (Rayleigh ratio) task 1 min ||Xv||2 , v 2 ||v||2 = 1.

v1 : s.t.

The corresponding Lagrangian becomes L(v, µ) =

1 T T v X Xv − µ(v T v − 1), 2

which, after taking the gradient w.r. to v and equating to zero gives, X T Xv = µv, which shows that the solution v1 , is an eigenvector of X T X and in order to maximize the cost, it must necessarily correspond to the largest singular value. We now proceed to k = 2, to solve the task, vk : s.t.

1 min ||Xv||2 , k = 1, 2, . . . , r, v 2 ||v||2 = 1, v ⊥ v1 .,

The Lagrangian is now given by, L(v, µ1 , µ2 ) =

1 T T v X Xv − µ1 (v T v − 1) − µ2 v T v1 , 2

which results in X T Xv − µ1 v = µ2 v1 .

However, since v ⊥ v1 , µ2 has to be equal to zero, which again leads to the fact that v2 must also be an eigenvector of X T X. Since it cannot be equal to v1 , the next eigenvector that maximises the cost is the one which corresponds to the second largest singular value. The same rationale carries on for larger values of k.

5

6.8. Show that projecting the rows of X onto the k-rank subspace, Vk = span{v1 , . . . , vk }, results in the largest variance, compared to any other k-dimensional subspace, Zk .

Solution: The projection of a row xn of X onto Vk is given by x̂n =

k X (xTn vi )vi , i=1

and the respective square norm ( (scaled) variance of its elements) is ||x̂n ||2 =

k X (xTn vi )2 . i=1

Then, projecting all the rows of the matrix onto Vk results in a total variance, k k N X N X X X ||Xvi ||2 .

(5) (xTn vi )2 = ||x̂n ||2 = n=1

n=1 i=1

i=1

Hence, it is readily observed that, in general, the variance of the matrix elements, after the projection onto a subspace, depends on the specific subspace. Our goal here is to show that the maximum variance is attained when the subspace is Vk . To this end, let us consider any other subspace Zk . We will prove the claim inductively.

• k = 1: From Problem 6.7, it is readily concluded that ||Xz1 ||2 ≤ ||Xv1 ||2 , where, z1 is a unit vector in the one-dimensional Z1 .

• k = 2: In the two-dimensional subspace Z2 , define a basis around z2 so that, z2 ⊥ v1 . This can always be done, by projection v1 onto Z2 and selecting z2 to be perpendicular to the projection. Then, from Problem 6.7, we have that ||Xz1 ||2 ≤ ||Xv1 ||2 , and also, since both v2 , z2 are perpendicular to v1 , ||Xz2 ||2 ≤ ||Xv2 ||2 .

Adding by parts the two inequalities, proves the claim that V2 is the two-dimensional subspace which maximizes the total variance, due to (5).

• Assume that Vk−1 is the optimal (k −1)-dimensional subspace. Build an orthonormal basis in any Zk , around a zk so that zk ⊥ {v1 , . . . , vk−1 }.

6

This can always be done. Indeed, let a basis ei , i = 1, 2, . . . , k be a basis in Zk . Then zk can be found by solving the linear system of equations, zk =

k X

K X

ai ei :

i=1

ai eTi vj = 0, j = 1, 2, . . . k − 1,

i=1

which is a system with k − 1 equations and k unknowns and which, in general, has always a solution.

Then, we have ||Xzk ||2 ≤ ||Xvk ||2 , and also by the assumption of the induction, k−1 X

||Xzi ||2 ≤

i=1

k−1 X

||Xvi ||2 .

i=1

Adding the two inequalities by part, the claim is proved. Note that if k is equal to the rank of the matrix, r, then the total variance of the matrix, which also defines the Frobenius norm, is given by (Problem 6.13), r r X X σi2 = ||Xvi ||2 . trace{X T X} = i=1

i=1

6.9. Show that the squared Frobenius norm is equal to the sum of the squared singular values.

Solution: By the definition of the Frobenius norm, XX kXk2F := |X(i, j)|2 = kUr DVr kF . i

j

However, it is known from linear algebra that if Q and U are orthogonal matrices then kXk2F = kQXU k2F , and since Ur and Vr are orthogonal matrices, we have that kXk2F = kDk2F =

r X

σi2 .

i=1

The previously used property, concerning orthogonal matrices, is easily shown from the fact that, XX kXk2F = |X(i, j)|2 = trace{X T X}. i

j

However, we know that multiplication with unitary matrices does not change the trace. Indeed, kXk2F = trace{X T X} = trace{X T QT QX} = kQXk2F .

7

6.10. Show that the best k rank approximation of a matrix X of rank r > k, in the Frobenius norm sense, is given by: X̂ =

k X

σi ui viT ,

i−1

where σi are the singular values and vi , ui , i = 1, 2, . . . , r, are the right and left singular vectors of X, respectively. Then show that the approximation error is given by: v u X u r t σi2 . i=k+1

Solution: Let, X̂ :=

k X

σi ui viT =

k X (Xvi )viT .

(6)

i=1

i=1

From (6), it is readily deduced that the rows of X̂, x̂Tn , n = 1, 2, . . . , N , are projections of the respective rows of X onto the subspace spanned by Vk = span{v1 , . . . , vk }, i.e., x̂Tn =

k X

(xTn vi )viT .

i=1

Thus, we can write that xn = x̂n + en , where en is the error vector. Hence, the respective norms are related (due to the orthogonality) via the Pythagoras theorem, ||xn ||2 = ||x̂n ||2 + ||en ||2 .

However, we know form Problem 6.8, that projecting onto the Vk results in the largest norm of ||x̂n ||2 , compared to any other subspace; this is equivalent with the smallest error norm ||en ||. Hence, the corresponding Frobenius norm, N X ||X − X̂||2F = ||en ||2 , n=1

is the smallest one. We have proved that from all possible approximations of X, obtained by projecting its rows onto any rank k subspace, the best one is X̂.

Let us now assume that there is another matrix B, of rank k, which results in a better approximation. However, this is not possible. Let XB be the approximation of X with rows equal to the projections of the rows of X

8

onto the row space of B, which is of rank k. Then, if B 6= XB , then XB will be a better approximation compared to B, i.e., ||X − X̂||2F < ||X − XB ||2F < ||X − B||2F , which proves the claim.

For the error matrix we have that, r X

E=

σi ui viT .

i=k+1

Thus, ||E||2F

=

r r n X o X trace{E T E} = trace σi vi uTi uj vjT i=k+1

=

trace

nXX i

=

r X

σi σj vi uTi uj vjT

j=k+1

o

j

σi2 ,

(7)

i=k+1

due to the orthonormality of the involved vectors and the properties of the trace.

6.11. Show that X̂, as given in Problem 6.10, also minimizes the spectral norm and that, ||X − X̂||2 = σk+1 .

Solution: We first show that, ||X − X̂||2 = σk+1 .

By the definition of the spectral norm, it is equal to the maximum singular value of the respective matrix; thus, it suffices to compute the singular value of the error matrix.

By the definition of the maximum singular value, we know that it is the value that maximizes the square norm, ||(X − X̂)v||2 , subject to the constrain that ||v|| = 1. Let v=

l X i=1

a i vi .

9

Then, we have that ||(X − X̂)v||2

= ||

= ||

r X

(σi ui viT )

i=k+1 r X

l X

aj vj ||2

j=1

ai σi ui viT vi ||2

i=k+1 r X

=

a2i σi2 .

(8)

i=k+1

Pl The maximum value of the above, taking into account that i=1 a2i = 1, occurs if ak+1 = 1 and the rest are zero, which proves the claim that ||X − X̂||2 = σk+1 .

Let us now turn our focus in showing the best k-rank approximation. Let B be another matrix that results in lower error, ||X − B||2 < σk+1 . Since B is a rank k matrix, its null space will be of dimension l − k. Thus, by basic dimension arguments, S := N (B) ∩ span{v1 , . . . , vk , vk+1 } = 6 ∅.

Hence, ∃ z 6= 0 ∈ S, ||z|| = 1. Then, by the definition of the spectral norm, and taking into account that Bz = 0, we get ||X − B||22

= =

max ||(X − B)v||2 ≥ ||(X − B)z||2 = ||Xz||2

||v||=1

||

r X

σi ui viT z||2

i=1

=

||

r X i=1

=

||

k+1 X

σi ui viT

k+1 X

(vjT z)vj ||2

j=1

σi ui (viT z)||2

i=1

=

k+1 X i=1

2 σi2 (viT z)2 ≥ σk+1

k+1 X

2 (viT z)2 = σk+1 ,

i=1

since ||z|| = 1. This contradicts the assumption that B provides a smaller error norm and proves the claim.

6.12. Show that the Frobenius and spectral norms are unaffected by multiplication with unitary matrices, i.e., kXkF = kQXU kF

10

and kXk2 = kQXU k, T

if QQ = U U

T

= I.

Solution: The proof for the Frobenius was given in Problem 6.9. From the definition of the the spectral norm, we have that kXk2 = σ1 , where λ1 is the largest eigenvalue of XX T (X T X). However, multiplication by a unitary matrix does not affect the respective eigenvalues. The latter is known from linear algebra and it is trivially shown.

6.13. Show that the null and range spaces of a m × l matrix, X, of rank r are given by, N (X)

=

span{vr+1 , . . . , vl },

R(X)

=

span{u1 , . . . , ur },

where

D X = [u1 , . . . , um ] O

Solution: Recall that X=

r X

  v1T O  .  . .

O  .  vlT

σi ui viT .

i=1 l

Hence, ∀a ∈ R , Xa =

r X

σi ui (viT a),

i=1

and R(X) = span{u1 , . . . , ur }.

From linear algebra we know that if the dimensionality of the range space is r, then the dimensionality of the the null space will be l − r, and since Xvj =

r X

σi ui (viT vj ) = 0,

j = r + 1, . . . , l ,

i=1

due to the orthonormality of vi , i = 1, 2, . . . , l, then N (X) = span{vr+1 , . . . , vl }.

6.14. Show that for the ridge regression ŷ =

l X

σi2 T 2 (ui y)ui λ + σ i i=1

11

Solution: We have that ŷ

−1 = Ul DVlT Vl DUlT Ul DVlT + λI Vl DUlT y −1 = Ul DVlT Vl D2 VlT + λVl VlT Vl DUlT y

−1 = Ul DVlT Vl (D2 + λI)VlT Vl DUlT y = Ul D(D2 + λI)−1 DUlT y =

l X

σi2 T 2 (ui y)ui . λ + σ i i=1

6.15. Show that the normalized steepest descent direction of J(θ) at a point θ0 , for the quadratic norm kvkP is given by v=

1 P −1 ∇J(θ0 ). ||P −1 ∇J(θ0 )||P

Solution: The task is minimize

v T ∇J,

s.t.

v T P v = 1.

v

Using Lagrangian multipliers we obtain L(v) = v T ∇J + λ(v T P v − 1) or ∇v L = ∇J + 2λP v = 0 or

1 −1 P ∇J. 2λ Substituting the above in the constraint we get v=−

1 (∇J)T P −1 P P −1 ∇J = 1 4λ2 or

1 2λ = (∇J)T P −1 ∇J 2 .

6.16. Justify why the convergence of Newton’s iterative minimization method is relatively insensitive on the Hessian matrix.

Hint: Let P be a positive definite matrix. Define a change of variables, 1

θ̃ = P 2 θ,

12

and carry gradient descent minimization based on the new variable.

Solution: Let

1

θ̃ = P 2 θ.

Note that kθk2p := θ T P θ. Also, define, ˜ θ̃) := J(P − 21 θ̃) = J(θ).

J( We have that

˜ θ̃)) = P − 21 ∇θ J(θ). ∇θ̃ (J(

(9)

˜ θ̃) will be Then the gradient descent step for J( 1

∆θ̃ = −P − 2 ∇θ J(θ).

However,

1

∆θ̃ = P 2 ∆θ. or ∆θ = −P −1 ∇θ J(θ).

Note that the latter is the step for the steepest descent direction with respect kθkp norm. Set now P = ∇2 J(θ∗ ), where θ∗ is the optimum, and assume that close enough to the optimum ˜ θ̃) will the Hessian does not vary much. Then from 9, the Hessian of J( be equal to 1 1 P − 2 ∇2 J(θ∗ )P − 2 ' I. Thus the step for the Newton’s method can equivalently been seen as a gradient descent with a cost whose Hessian matrix is of low condition number.

6.17. Show that the steepest descent direction, v, of J(θ) at a point, θ0 , constrained to kvk1 = 1, is given by ek , where ek is the standard basis vector in the direction, k, such that |(∇J(θ0 ))k | > |(∇J(θ0 ))j |, k 6= j.

Solution: We will use two ways to prove it. The second one is lengthier, but it is an exercise for the reader to get familiar with optimizing with respect to nondifferential functions via the use of the subdifferential notion. a) It is known from linear algebra and the properties of norms, that |v T a| ≤ kvk1 kak∞ ,

13

where kak∞ = maxi |ai |, i = 1, 2, . . . , l. Indeed, T

|v a| =

l X

l X

≤

v i ai

i=1

|vi ||ai | ≤

l X

i=1

! |vi | kak∞

i=1

= kvk1 kak∞ .

Hence, −kak∞ ≤ v T a ≤ kak∞ , since kvk1 = 1. Thus the minimum is achieved if v = − sgn(ai )ei where ei the direction corresponding to the component associated with kak∞ . b) Since the `1 norm is not differentiable, the notion of subgradient will be mobilized. We have minimize s.t.

v T ∇J kvk1 = 1.

The Lagrangian is L(v) = v T ∇J + λ(kvk1 − 1) or ∂L(v) ∂kvk1 = ∇J + λ ∂v ∂v or 0∈

∇J + λ

∂kvk1 , ∂v

(10)

1 where ∂kvk is the subdifferential set for kvk1 . Then, taking into account ∂v the subgradient of the absolute value, (10) can be written component wise as  vj > 0  {1}, {−1}, vj < 0 (∇J)j = −λ j = 1, 2, . . . , l.

(11)  a ∈ [−1, 1], vj = 0

Since v 6= 0, λ has to be chosen so as to satisfy one of these equations, otherwise there is no solution. We choose j → k corresponding to the largest component |(∇J)k | > |(∇J)j |, j 6= k. Thus v : vj = 0, j 6= k, and vk > 0.

Then λ = −(∇J)k ,

14

which guarantees that v T ∇J = −|(∇J)k |2 , which for all available choices, corresponds to the steepest descent.

Note that this choice also guarantees that all equations in (11) are satisfied, since in this case there is always a value aj , such as (∇J)j = −aj λ, j 6= k. with aj ∈ [−1, 1].

6.18. Show that the TLS solution is given by, 2 θ̂ = X T X − σ̄l+1 I

−1

X T y,

. where σ̄l+1 is the smallest singular value of [X .. y].

Solution: By the respective definition we have that, T

..

X 2 v̄l+1 = σ̄l+1 v̄l+1 , X . y yT or

XT X yT X

XT y 2 v̄ = σ̄l+1 v̄l+1 , y T y l+1

(l+1)

and after dividing by −v̄l+1 , T X X yT X

XT y yT y

θ̂T LS θ̂T LS 2 = σ̄l+1 , −1 −1

or 2 X T X θ̂T LS − X T y = σ̄l+1 θ̂T LS

or

2 X T X − σ̄l+1 I θ̂T LS = X T y, which proves the claim.

6.19. Given a set of centered data points, (yn , xn ) ∈ Rl+1 , derive a hyperplane aT x + y = 0, which crosses the origin, such as the total square distance of all the points from it to be minimum.

Solution: Let wT = [aT , 1]T .

15

and zn = (xn , yn ) ∈ Rl+1 , n = 1, 2, . . . , N. Then we will search for the w given by N X

minimize w

|wT zn |2

n=1

kwk2 = 1.

s.t.

The constraint is used since we are only interested in the direction. If kwk 6= 1, then the square distance of a point from the hyperplane would be equal to, |wT zn |2 . kwk2 Let  T   T x 1 y1 z1  ..   . ..  =  ...  Z= . . xTN

yN

T zN

Then, N X

|wT zn |2 = wT Z T Zw.

n=1

Hence, the Lagrangian becomes L(w, λ̄) = wT Z T Zw − λ̄(wT w − 1) or ∂L(w, λ̄) ∂w

=

2(Z T Zw − λ̄w) = 0

⇒ Z T Zw = λ̄w.

Thus, λ̄ is an eigenvalue of ZZ T . Let λ̄1 ≥ λ̄2 ≥ . . . ≥ λ̄l > λ̄l+1 > 0.

That is, we assume that ZZ T is full rank and also the smallest eigenvalue is single. Then, w = v̄l+1 .

Note that in this case, wT Z T Zw = λ̄min kv̄l+1 k2 = λ̄min , since v̄l+1 is the normalized eigenvector, in order to guarantee the constrain. For any other choice of λ̄, the value of loss would be larger, hence not minimum. Thus, the hyperplane is defined by the eigenvector corresponding to the minimum single singular value of [X|y].

Note that, in general, there is not a single solution.

1

Solutions To Problems of Chapter 7 7.1. Show that the Bayesian classifier is optimal, in the sense that it minimizes the probability of error.

Hint: Consider a classification task of M classes and start with the probability of correct label prediction, P (C). Then the probability of error will be P (e) = 1 − P (C).

Solution: Let P (C) be the probability of correct classification. Then P (C) =

M X

P (x ∈ Ri , ωi ) =

i=1

M X

P (ωi )P (x ∈ Ri |ωi ),

i=1

or P (C) =

M X i=1

Z P (ωi )

P (x|ωi )dx = Ri

M Z X i=1

P (ωi )p(x|ωi )dx,

Ri

or P (C) =

M Z X i=1

P (ωi |x)p(x)dx.

Ri

For minimum classification error P (e), P (C) must be maximum (P (C) + P (e) = 1). Thus P (C) is maximized if the regions Ri are chosen so that in each region the corresponding integrals, which are all positive, have the maximum possible value. That is, Ri : P (ωi |x)p(x) > P (ωj |x)p(x)

∀ i 6= j

or Ri : P (ωi |x) > P (ωj |x)

∀ i 6= j

7.2. Show that if the data follow the Gaussian distribution in an M class task, with equal covariance matrices in all classes, the regions formed by the Bayesian classifier are convex.

Solution: Consider two points, x1 , x2 , lying in Ri . Then any point lying on the line segment, which connects these two points, can be written as x = ax1 + (1 − a)x2 , a ∈ [0, 1].

In this case, the decision is taken according to a linear discriminant function, i.e,

θiT x + θ0i = a θiT x1 + θ0i + (1 − a) θiT x2 + θ0i .

2

However, since by the definition of the partition, θiT x1 + θ0i > θjT x1 + θ0j , j 6= i, and θiT x2 + θ0i > θjT x2 + θ0j , j 6= i, it turns out that x ∈ Ri , which proves the claim.

7.3. Derive the form of the Bayesian classifier for the case of two equiprobable classes, when the data follow the Gaussian distribution of the same covariance matrix. Furthermore, derive the equation that describes the LS linear classifier. Compare and comment on the results.

Solution: For the scenario of this problem, we already know that the Bayesian classifier is equivalent to the minimum Mahalanobis distance classifier, i.e., (x − µ1 )T Σ −1 (x − µ1 ) > (<)(x − µ2 )T Σ −1 (x − µ2 ).

Expanding and rearranging the terms, we obtain that the decision surface is of the following form g(x) = −xT Σ −1 x + 2xT Σ −1 µ1 − µT1 Σ −1 µ1 + xT Σ −1 x − 2xT Σ −1 µ2 + µT2 Σ −1 µ2 = 0.

Rearranging the terms one can easily obtain that g(x) = θ T (x − θ0 ) = 0, where

1 (µ1 + µ2 ).

2 Let us now turn our attention to the LS solution. By extending the involved training feature vectors by one, in order to accommodate the bias term, the LS classifier is given by the solution of the following set of linear equations: !

N N 1 X xn T 1 X θ x xn , 1 = yn n , 1 θ0 1 N N θ := Σ −1 (µ1 − µ2 ),

n=1

θ0 =

n=1

where we have divided both sides by 1/N . For large enough N and for equiprobable classes, we can assume that both classes are represented with equal number of points N/2 and by the law of large numbers we can equate sample means with expectations. Then the previous formula can be written as

1

R E[x] θ 2 (µ1 − µ2 ) = (1) θ0 E[xT ] 1 0

3

P where the zero comes from n yn , in which half of the terms are +1 and the rest −1. In the previous formula we have that R := E[xxT ] = Σb + µµT , 1 µ := (µ1 + µ2 ) = E[x] = P (ω1 ) E[x|ω1 ] + P (ω2 ) E[x|ω2 ], 2 and Σb is the covariance matrix for the two classes together; that is, around the overall mean value µ. Substituting the previous definitions in (1), and solving the system of equations, it is trivial to see that θ0 = −θ T µ, 1 θ = Σb−1 (µ1 − µ2 ).

2 Combining the previous finding we obtain the LS optimal linear classifier as g(x) = θ T x + θ0 = θ T (x − θ0 ) = 0, where θ = Σb−1 (µ1 − µ2 ),

θ0 ≡ µ =

1 (µ1 + µ2 ).

2

However,

Σb = P (ω1 ) E (x − µ1 )(x − µ1 )T + P (ω2 ) E (x − µ2 )(x − µ2 )T = Σ.

That is, under the adopted assumptions, the LS classifier tends asymptotically to the optimal Bayesian classifier.

7.4. Show that the ML estimate of the covariance matrix of a Gaussian distribution, based on N i.i.d, observations, xn , n = 1, 2, . . . , N , is given by, N 1 X Σ̂M L = (xn − µ̂M L )(xn − µ̂M L )T , N n=1 where

N

1 X µ̂M L = xn .

N n=1 Solution: We shall focus on the simplest case where Σ = σ 2 I. The loglikelihood function is L(µ, σ 2 ) =

N X

ln p(xn ; µ, σ 2 ),

n=1 N N 1 X 2 = − ln σ − 2 (xn − µ)T (xn − µ).

2 2σ n=1

4

where constants have been omitted. The unknown set of parameters is now θ T = [µT , σ 2 ], hence, " # " # PN ∂L(θ) 1 (x − µ) ∂L(θ) 2 n n=1 σ P ∂µ N T = = ∂L(θ) = 0.

N n=1 (xn −µ) (xn −µ) ∂θ − 2σ 2 + 2σ 4 ∂ σ¯2 Solving the above system w.r. to µ and σ 2 results in, N

µ̂ =

1 X xn , N k=1

and

N

σ̂ 2 =

1 X (xn − µ)T (xn − µ).

N n=1

7.5. Prove that the covariance estimate N 1 X Σ̂ = (xk − µ̂)(xk − µ̂)T N −1 k=1

corresponds to an unbiased estimator, where, N

µ̂ =

1 X xk .

N k=1

Solution: We have that N i 1 X h T E[Σ̂] = E ((xk − µ) − (µ̂ − µ)) ((xk − µ) − (µ̂ − µ)) N −1 k=1

=

N N

1 X 1 X E (xk − µ)(xk − µ)T + E (µ̂ − µ)(µ̂ − µ)T − N −1 N −1 k=1

1 N −1

N X

k=1

E (xk − µ)(µ̂ − µ)T −

k=1

N

1 X E (µ̂ − µ)(xk − µ)T .

N −1

(2)

k=1

However,   N N X X

1 1 E (µ̂ − µ)(µ̂ − µ)T = E (xi − µ) (xj − µ)T  N i=1 N j=1 =

N

1 X E (xi − µ)(xi − µ)T N 2 i=1

=

1 1 N Σ = Σ, 2 N N

(3)

5

where independence among the samples has been assumed, i.e.,

E (xi − µ)(xj − µ)T = δij Σ.

Following a similar path, we end up with

1 E (xk − µ)(µ̂ − µ)T = E (µ̂ − µ)(xk − µ)T = Σ.

N

(4)

E (xk − µ)(xk − µ)T = Σ.

(5)

Also, Combining Eqs (2)-(5), we get E[Σ̂] =

N 1 1 1 Σ+ Σ− Σ− Σ = Σ.

N −1 N −1 N −1 N −1

Hence the estimator is an unbiased one.

7.6. Show that the derivative of the logistic link function is given by

dσ(t) = σ(t) 1 − σ(t) . dt

Solution: We have that σ(t) =

1 .

1 + exp(−t)

Thus dσ(t) exp(−t) exp(−t) = = σ(t) (1 − σ(t)) .

2 = σ(t) dt (1 + exp(−t)) 1 + exp(−t) 7.7. Derive the gradient of the negative log-likelihood function associated with the two-class logistic regression.

Solution: The log-likelihood is given by L(θ) = −

N X

yn ln sn + (1 − yn ) ln(1 − sn ) ,

n=1

where sn = σ(θ T xn ).

Let an := yn ln sn + (1 − yn ) ln(1 − sn ).

Hence, ∇θ an = yn

1 1 (sn (1 − sn )) ∇θ tn − (1 − yn ) (sn (1 − sn )) ∇θ tn , sn 1 − sn

6

where tn := θ T xn , or ∇θ an = yn (1 − sn )xn − (1 − yn )sn xn = (yn − sn )xn .

Hence, ∇θ L(θ) =

N X

(sn − yn )xn = X T (s − y).

n=1

7.8. Derive the Hessian matrix of the negative log-likelihood function associated with the two-class logistic regression.

Solution: From the previous problem, we have already derived the gradient, i.e., ∇θ an = (yn − sn )xn .

Hence, the i, j element of the respective Hessian matrix is given by,

2 ∂ ∇θ an ij = (sn − yn )xn (j) = (sn (1 − sn )) xn (i)xn (j), ∂θi which can equivalently be written as, 2

∇θ an ij = xn rn xTn ij , where rn = sn (1 − sn ).

Hence, ∇2θ an = xn rn xTn , which leads to ∇2θ L(θ) =

N X

xn rn xTn = X T Rn X,

n=1

where Rn is the diagonal matrix comprising, rn , n = 1, 2, . . . , N .

7.9. Show that the Hessian matrix of the negative log-likelihood function of the two-class logistic regression is a positive definite matrix.

Solution. We have seen that ∇2θ L(θ) = X T Rn X. Thus for any vector x ∈ Rl , the following is true, xT X T Rn Xx = y T Rn y > 0, due to the positive definite nature of Rn , where y := Xx. This is the definition of a positive definite matrix.

7

7.10. Show that if

exp(tm ) φm = PM , j=1 exp(tj )

the derivative with respect to tj , j = 1, 2, . . . , M is given by, ∂φm = φm (δmj − φj ). ∂tj Solution: a) Let j = m. Then we have exp(tm ) ∂φm = ∂tm or

PM

j=1 exp(tj ) − exp(tm ) exp(tm ) , P 2 M j=1 exp(tj )

∂φm = φm (1 − φm ). ∂tm

Now, let tj 6= tm . Then, in this case, ∂φm − exp(tj ) exp(tm ) = P 2 = −φm φj . ∂tj M exp(t ) j j=1 Thus, ∂φm = φm (δmj − φj ), j = 1, 2, . . . , M. ∂tj 7.11. Derive the gradient of the negative log-likelihood for the multiclass logistic regression case.

Solution: Let an :=

M X

ynm ln φnm .

m=1

Then we have that, ∇θj an

=

M X

ynm

1 ∂φnm ∂tj φnm ∂tj ∂θj

ynm

1 φnm (δmj − φnj )xn φnm

m=1

=

M X m=1

= since

(ynj − φnj )xn , M X m=1

ynm = 1.

8

Hence, ∇θj L(θ1 , . . . , θM ) =

N X

(φnj − ynj )xn .

n=1

7.12. Derive the j, k block element of the Hessian matrix of the negative loglikelihood function for the multiclass logistic regression.

Solution: From the previous problem, we have already obtained that, ∇θj L(θ1 , . . . , θM ) =

N X

(φnj − ynj )xn .

n=1

Hence, ∇θk ∇θj L(θ1 , . . . , θM )

=

N X

xn ∇Tθk (φnj − ynj )

n=1

=

N X

φnj (δkj − φnk )xn xTn ,

n=1

where the result of Problem 7.10 has been used.

7.13. Consider the so-called Rayleigh ratio, R=

θ T Aθ , ||θ||2

where A is a symmetric matrix. Show that R is maximized, with respect to θ, if θ is the eigenvector corresponding the maximum eigenvalue of A. Solution: The maximization task is equivalent to maximize w.r. θ

: θ T Aθ,

s.t

||θ||2 = 1,

since we are only interested in the direction and the ratio does not depend on the norm of θ. Using Lagrange multipliers, the Lagrangian becomes, L(θ, λ) = θ T Aθ − λ(||θ||2 − 1), and taking the gradient with respect to θ and equating to zero, we obtain, Aθ = λθ.

In other words, θ is an eigenvector of A. In order to maximize the ratio, we choose the one which corresponds to the maximum eigenvalue, leading to R = θ T Aθ = λ||θ||2 = λ.

9

7.14. Consider the generalized Rayleigh quotient, Rg =

θ T Bθ . θ T Aθ

where A and B are a symmetric matrices. Show that Rg is maximized with respect to θ, if θ is the eigenvector which corresponds to the maximum eigenvalue of A−1 B, assuming that the inversion is possible.

Solution: For the case of our problem (generalized Rayleigh quotient), let y := A1/2 θ ⇒ θ = A−1/2 y.

Then the problem becomes equivalent with maximizing max y

y T A−1/2 BA−1/2 y , yT y

where the symmetry of A has been taken into account. From the previous problem, the above is maximized if y is the eigenvector corresponding to the largest eigenvalue λ, i.e., A−1/2 BA−1/2 y = λy, and finally, by replacing y by θ, if θ is chosen to satisfy Bθ = λAθ.

Equivalently we can solve the following eigenvalue task A−1 Bθ = λθ.

The corresponding maximum value is θ T Bθ θ T Aθ = λ =λ θ T Aθ θ T Aθ which justifies the choice of the maximum eigenvalue.

7.15. Show that the between-classes scatter matrix Σb for an M class problem is of rank M − 1.

Solution: We will show it for M = 2 and M = 3 and the result is easily generalized. We use P (ωi ) = Pi . For M = 2, we have P1 + P2 = 1. Thus, µ1 − µ0 = µ1 − P1 µ1 − P2 µ2 = P2 (µ1 − µ2 ).

Similarly we can show that µ2 − µ0 = −P1 (µ1 − µ2 ).

10

Thus Σb

=

P1 (µ1 − µ0 )(µ1 − µ0 )T +P2 (µ2 − µ0 )(µ2 − µ0 )T

=

P1 P22 (µ1 − µ2 )(µ1 − µ2 )T +P2 P12 (µ1 − µ2 )(µ1 − µ2 )T

=

P1 P2 (µ1 − µ2 )(µ1 − µ2 )T .

(6)

Thus Σb is a rank one matrix.

Also, for M = 3, the vectors entering into the summation are, µ1 − µ0 , µ2 − µ0 , µ3 − µ0 , or µ1 − µ0 = (1 − P1 )µ1 − P2 µ2 − P3 µ3 = P2 (µ1 − µ2 ) + P3 (µ1 − µ3 ), µ2 − µ0 = (1 − P2 )µ2 − P1 µ1 − P3 µ3 = P1 (µ2 − µ1 ) + P3 (µ2 − µ3 ), µ3 − µ0 = (1 − P3 )µ3 − P1 µ1 − P2 µ2 = P1 (µ3 − µ1 ) + P2 (µ3 − µ2 ), From the above, it is obvious that P1 (µ1 − µ0 ) + P2 (µ2 − µ0 ) = −P3 (µ3 − µ0 ).

Thus, again two of them produce the third. This is easily generalized to M, that is, P1 (µ1 − µ0 ) + . . . + PM −1 (µM −1 − µ0 ) = −PM (µM − µ0 ) PM Hence i=1 Pi (µi − µ0 )(µi − µ0 )T is of rank M − 1

7.16. Derive the arithmetic rule for combination, by minimizing the average KL divergence.

Solution: Let the output from each classifier be Pj (ωi |x) := Pj (ωi ) and let P (ωi |x) := P (ωi ) be the output of the combiner to be estimated.

Minimizing the average Kullback-Leibler divergence and employing the obvious constraint M X P (ωi ) = 1, (7) i=1

the corresponding Lagrangian becomes !

L M L M M X 1 XX 1 XX Pj (ωi ) ln Pj (ωi )− Pj (ωi ) ln P (ωi )−λ P (ωi ) − 1 .

L j=1 i=1 L j=1 i=1 i=1 (8)

11

Taking the derivative with respect to P (ωi ) and equating to zero, it turns out that L 1 X P (ωi ) = − Pj (ωi ).

(9) λL j=1 Substituting into the constraint equation, we obtain M

λ=−

L

1 XX Pj (ωi ), L i=1 j=1

(10)

and finally, L

1X Pj (ωi ).

P (ωi ) = L j=1

(11)

7.17. Derive the product rule via the minimization of the Kullback-Leibler divergence, as pointed out in the text.

Solution: We will estimate P (ωi ) by minimizing the Kullback-Leibler average divergence, i.e, L

M

P (ωi ) 1 XX P (ωi ) ln , L j=1 i=1 Pj (ωi )

(12)

subject to the constraint M X

P (ωi ) = 1.

(13)

i=1

Employing Lagrange multipliers, the above problem becomes equivalent with minimizing the Lagrangian !

L M M X P (ωi ) 1 XX P (ωi ) ln −λ P (ωi ) − 1 .

(14) L j=1 i=1 Pj (ωi ) i=1 Taking the derivative with respect to P (ωi ) we get L L 1X 1X ln P (ωi ) + 1 − ln Pj (ωi ) − λ = 0, L j=1 L j=1

(15)

and solving the above we have L

P (ωi )

= =

1X ln Pj (ωi )} L j=1 Y 1 exp{λ − 1} exp ln Pj (ωi ) L exp{λ − 1} exp{

j

=

exp{λ − 1}

Y j

1 Pj (ωi ) L .

(16)

12

Substituting the above in the constraint equation it turns out that, 1

exp{λ − 1} = P

M QL i=1 j=1

1 Pj (ωi ) L

(17)

Hence, 1 L

QL

j=1 (Pj (ωi )) P (ωi ) = PM QL .

1 L i=1 j=1 (Pj (ωi ))

(18)

7.18. Show that the error rate on the training set of the final classifier, obtained by boosting, tends to zero exponentially fast.

Solution The error rate on the training data set is given by PeN =

N N 1 X 1 X I(1 − yn f (xn )) ≤ exp(−yn F (xn )), N n=1 N n=1

which by the definition of the combined classifier is written as N

1 X exp(−yn F (xn )) N n=1

=

!

N K X 1 X exp −yn ak φ(xn ; θ k ) N n=1 k=1

=

N K X 1 Y n=1

N

exp(−yn ak φ(xn ; θ k )).

(19)

k=1

However, (K)

wn(K+1)

=

wn

exp(−yn aK φ(xn ; θ K )) ZK

(K−1)

=

wn

exp(−yn aK−1 φ(xn ; θ K−1 )) exp(−yn aK φ(xn ; θ K )) ZK−1 ZK

QK =

k=1 exp(−yn ak φ(xn ; θ k )) .

QK N k=1 Zk

(20)

However, we know that the weights sum up to one. Thus N QK X k=1 exp(−yn ak φ(xn ; θ k )) = 1.

QK N k=1 Zk n=1

(21)

Combining (19) and (21) results in, PeN ≤

K Y k=1

Zk .

(22)

13

By the respective definition we have, Zk

=

N X

wn(k) exp(−yn ak φ(xn ; θ k ))

n=1

X

=

X

wn(k) exp(ak ) +

yn φ(xn ;θ k )<0

wn(k) exp(−ak ).

yn φ(xn ;θ k )>0

Using the respective formulas, (7.93) and (7.94) from the proof in the book, the above becomes Zk = Pk exp(ak ) + (1 − Pk ) exp(−ak ).

Also we know that ak =

(23)

1 1 − Pk ln .

2 Pk

(24)

Combining (23) and (24), we obtain p Zk = 2 Pk (1 − Pk ).

(25)

Hence (22) can now be written as PeN ≤

K n p K q K o Y Y X 2 Pk (1 − Pk ) = 1 − 4γk2 ≤ exp(−2 γk2 ), k=1

k=1

k=1

where by definition γk ≡ 1/2 − Pk . Since the base classifiers do better than a random guessing, we can write that γk ≥ γ > 0. Thus the error is bounded by exp(−2Kγ 2 ) and drops exponentially fast with K.

1

Solutions To Problems of Chapter 8 1. Prove the Cauchy - Schwartz’s inequality in a general Hilbert space.

Solution: We have to show that ∀x, y ∈ H, |hx, yi| ≤ kxkkyk, and that equality holds only if y = ax, a ∈ C. Solution: The inequality holds true for x and/or y = 0. Let now y, x 6= 0.

We have 0

≤ kx − λyk2 = hx − λy, x − λyi = kxk2 + |λ|2 kyk2 − λ∗ hx, yi − λhy, xi.

Since the last inequality is valid for any λ ∈ C, let λ=

hx, yi . kyk2

Thus 0 ≤ kxk2 −

|hx, yi|2 , kyk2

from which a) the inequality results and b) the fact that the equality holds iff x = ay.

Indeed, if x = ay, then equality is trivially shown. Let us now assume that equality holds true. Then hx, xihy, yi = hx, yi∗ hx, yi and from the properties of the inner product in a Hilbert space we have hx, ||y||2 xi = hx, hx, yiyi, from which it is readily seen that x=

hx, yi y, ||y||2

which proves the claim.

2. Show a) that the set of points in a Hilbert space H, C = {x : kxk ≤ 1} is a convex set, and b) the set of points C = {x : kxk = 1}

2

is a nonconvex one.

Solution: From the definition of a Hilbert space (see Appendix of this chapter) the norm is the induced by the inner product norm, i.e., 1

kxk = (hx, xi) 2 . a) Let us now consider two points, x1 , x2 ∈ H, such that kx1 k ≤ 1, kx2 k ≤ 1, and let x = λx1 + (1 − λ)x2 , λ ∈ [0, 1].

Then, by the triangle inequality property of a norm kxk = kλx1 + (1 − λ)x2 k ≤ kλx1 k + k(1 − λ)x2 k, and since λ ∈ [0, 1] kxk ≤ λkx1 k + (1 − λ)kx2 k ≤ (λ + 1 − λ)1 ≤ 1. b) Let two points such that, kx1 k = 1, kx2 k = 1, and x = λx1 + (1 − λ)x2 .

Then we have that kxk2

= hλx1 + (1 − λ)x2 , λx1 + (1 − λ)x2 i = λ2 kx1 k2 + (1 − λ)2 kx2 k2 + 2λ(1 − λ)hx1 , x2 i = λ2 + (1 − λ)2 + 2λ(1 − λ)hx1 , x2 i.

(1)

From the Schwartz inequality (Problem 1), we have that |hx1 , x2 i| ≤ kx1 kkx2 k,

(2)

− 1 ≤ hx1 , x2 i ≤ 1.

(3)

or From (1) and (3) is readily seen that kxk2 ≤ 1.

As a matter of fact, the only way for kxk2 = 1, is that hx1 , x2 i = 1 = kx1 kkx2 k.

However, this is not possible. Equality in (2) is attained if x1 = ax2 , and since kx1 k = kx2 k = 1, this can only happen in the trivial case of x1 = x2 .

3

3. Show the first order convexity condition.

Solution: Assume that f is convex. Then

f λy + (1 − λ)x ≤ λf (y) + (1 − λ)f (x) or

f x + λ(y − x) − f (x) ≤ λ f (y) − f (x) .

Taking λ −→ 0, we can employ the Taylor expansion and get

f x+λ(y −x) −f (x) ≈ f (x)+λ∇T f (x)(y −x)−f (x) ≤ λ f (y)−f (x) , from which, in the limit we obtain f (y) ≥ f (x) + ∇T f (x)(y − x).

(4)

b) Assume that f (y) ≥ f (x) + ∇T f (x)(y − x), is valid ∀x, y ∈ X, where X is the domain of definition of f . Then, we have f (y1 ) ≥ f (x) + ∇T f (x)(y1 − x), (5) and f (y2 ) ≥ f (x) + ∇T f (x)(y2 − x).

(6)

Combining the previous two inequalities together, we obtain λf (y1 ) + (1 − λ)f (y2 ) ≥ λf (x) + (1 − λ)f (x) + λ∇T f (x)(y1 − x) + (1 − λ)∇T f (x)(y2 − x),

(7)

for λ ∈ (0, 1). Since this is true for any x, it will also be true for x = λy1 + (1 − λ)y2 , which results in

f λy1 + (1 − λ)y2 ≤ λf (y1 ) + (1 − λ)f (y2 ), which proves the claim.

4. Show that a function f is convex, iff the one-dimensional function, g(t) := f (x + ty), is convex, ∀x, y in the domain of definition of f .

Solution: Observe that, g(λt1 + (1 − λ)t2 )

= f (x + λt1 y + (1 − λ)t2 y) = f (λx + (1 − λ)x + λt1 y + (1 − λ)t2 y)

= f λ(x + t1 y) + (1 − λ)(x + t2 y) .

(8)

4

Also note that g(t1 ) = f (x + t1 y), g(t2 ) = f (x + t2 y), and taking the definition of convexity, the claim is now straightforward to be shown.

5. Show the second order convexity condition.

Hint: Show the claim first for the one-dimensional case and then use the result of the previous problem, for the generalization.

Solution: We start with the one-dimensional case. Let a function f (x) be convex. Then we know from the first order convexity condition that f 0 (x)(y − x) ≤ f (y) − f (x) ≤ f 0 (y)(y − x), and dividing both sides by the positive quantity (y − x)2 , we get f 0 (y) − f 0 (x) ≥ 0, y−x and taking the limit y −→ x we obtain 00

f (x) ≥ 0.

(9)

Assume now that the second derivative is non-negative everywhere. Then select y > x and we get Z y 00 0 ≤ f (z)(y − z)dz x Z y = f 0 (z)(y − z) |z=y + f 0 (z)dz z=x x

=

−f 0 (x)(y − x) + f (y) − f (x).

(10)

The above is true for y > x. Note that we can also show that f (x) ≥ f 0 (y)(x − y) + f (y), by using the identity Z y 0≤

00

f (z)(z − x)dz. x

Thus we proved f is convex. For the more general case, consider g(t) = f (x + ty), form which we get

00

g (t) = y T ∇2 f (x + ty)y.

Since this is true for any x, y and t and using the previously obtained results, the claim is readily shown.

5

6. Show that a function f : Rl 7−→ R is convex iff its epigraph is convex.

Solution: a) Assume f to be convex. We have to show that its epigraph is a convex set. Let two points, x1 , x2 . From the convexity of f we have f (λx1 + (1 − λ)x2 ) ≤ λf (x1 ) + (1 − λ)f (x2 ).

(11)

Consider two points in the epigraph y1 = (x1 , r1 ) and y2 = (x2 , r2 ). Then we have λy1 + (1 − λ)y2 := y = (x, r) with x = λx1 + (1 − λ)x2 ,

(12)

r = λr1 + (1 − λ)r2 ,

(13)

f (x1 ) ≤ r1 , f (x2 ) ≤ r2 .

(14)

and since y1 , y2 ∈ epi(f )

Combining (11) and (14) we get f (x) ≤ λf (x1 ) + (1 − λ)f (x2 ) ≤ λr1 + (1 − λ)r2 = r, hence y = (x, r) ∈ epi(f ) and the epigraph is convex. b) Assume the epigraph to be convex. Then y = λy1 + (1 − λ)y2 ∈ epi(f ), hence r = λr1 + (1 − λ)r2 ≥ f (λx1 + (1 − λ)x2 )

(15)

for any r1 ≥ f (x1 ), r2 ≥ f (x2 ).

Thus (15) is also valid for r1 = f (x1 ), r2 = f (x2 ) and therefore f (λx1 + (1 − λ)x2 ) ≤ λf (x1 ) + (1 − λ)f (x2 ).

7. Show that if a function is convex, then its lower level set is convex for any ξ.

Solution: Let the function f be convex and two points, x, y, which lie in the lev≤ξ (f ). Then, f (x) ≤ ξ, f (y) ≤ ξ.

Hence, by the definition of convexity, f (λx + (1 − λ)y) ≤ λf (x) + (1 − λ)f (y) ≤ λξ + (1 − λ)ξ ≤ ξ, which proves the claim, that λx + (1 − λ)y ∈ lev≤ξ (f ).

6

8. Show that in a Hilbert space, H, the parallelogram rule,

kx + yk2 + kx − yk2 = 2 kxk2 + kyk2 , ∀x, y ∈ H. holds true.

Solution: The proof is straightforward from the respective definitions and the properties of the inner product relation. kx + yk2

= kxk2 + 2hx, yi + kyk2 ,

kx − yk2

= kxk2 − 2hx, yi + kyk2 ,

from which the parallelogram rule is obtained.

9. Show that if x, y ∈ H, where H is a Hilbert space, then the induced by the inner product norm satisfies the triangle inequality, as required by any norm, i.e., kx + yk ≤ kxk + kyk

Solution: By the respective definitions we have kx + yk2 := hx + y, x + yi = kxk2 + hx, yi + hy, xi + kyk2 , or kx + yk2

= kxk2 + 2Real(hx, yi) + kyk2 ≤

kxk2 + 2|hx, yi| + kyk2

≤

kxk2 + kyk2 + 2kxkkyk = (kxk + kyk)2 .

where the Cauchy-Schwartz inequality has been used.

10. Show that if a point x∗ is a local minimizer of a convex function, it is necessarily a global one. Moreover, it is the unique minimizer if the function is strictly convex.

Solution: Let x∗ be a local minimizer. Then, there exists > 0 and y∗ ∈ / B[0, ] such that f (x∗ ) ≤ f (x∗ + ∆), ∀∆ ∈ B[0, ], and f (y∗ ) < f (x∗ ).

Let λ :=

.

2||y∗ − x∗ ||

Then, λ(y∗ − x∗ ) ∈ B[0, ].

7

Hence,

f x∗ + λ(y∗ − x∗ ) ≤ (1 − λ)f (x∗ ) + λf (y∗ ) < f (x∗ ), which is not possible, since x∗ is a local minimizer.

Assume now that f is strictly convex and that there exist two minimizers, x∗ 6= y∗ . Then, by the definition of strict convexity, we have that 1 1 1 1 f ( x∗ + y∗ ) < f (x∗ ) + f (y∗ ) = f (x∗ ), 2 2 2 2 which is not possible, since x∗ is a global minimizer.

11. Let C be a closed convex set in a Hilbert space, H. Then show that ∀x ∈ H, there exists a point, denoted as PC (x) ∈ C, such that kx − PC (x)k = min kx − yk. y∈C

Solution: Let x 6∈ C, otherwise it is trivial. Let ρ be the largest lower bound of kx − yk, y ∈ C, i.e., ρ := inf kx − yk > 0. y∈C

Consider the sequence, 1 . n By the definition of the infimum, for each n, there will be at least one element xn ∈ C, such that ρn = ρ +

kx − xn k < ρn , which then defines a sequence, {xn }, of points for which we have that ρ ≤ kx − xn k < ρn , or ρ ≤ lim kx − xn k < lim ρn = ρ, n→∞

n→∞

which necessarily leads to lim kx − xn k = ρ

n→∞

(16)

From the parallelogram law, we can write

k(x−xm )+(x−xn )k2 +k(x−xm )−(x−xn )k2 = 2 kx−xm k2 +kx−xn k2 , or 1 kxn − xm k2 = 2(kx − xm k2 + kx − xn k2 ) − 4kx − (xn + xm )k2 .

2

8

However, since C is convex, the point 12 (xn + xm ) ∈ C and we deduce that

kxn − xm k2 ≤ 2 kx − xm k2 + kx − xn k2 − 4kρk2 .

Taking the limit for n, m → ∞ on both sides we get lim kxm − xn k2 ≤ 0 ⇒

n,m→∞

lim kxm − xn k = 0.

n,m→∞

That is, xn is a Cauchy sequence and since H is Hilbert the sequence converges to a point x∗ . Moreover it converges in C, since C is closed, i.e., x ∈ C. Hence we have kx − x∗ k

= kx − xn + xn − x∗ k ≤

kx − xn k + kxn − x∗ k.

Taking the limit and using (16) we obtain kx − x∗ k ≤ ρ.

However, since x∗ ∈ C, kx − x∗ k ≥ ρ, which means that kx − x∗ k = ρ, that is the infimum is attained, which proves the claim. Uniqueness has been established in the text.

12. Show that the projection of a point x ∈ H onto a non-empty closed convex set, C ⊂ H, lies on the boundary of C. Solution: Assume that PC (x) is an interior point of C. By the definition of interior points, ∃δ > 0 such that Sδ := {y : ky − PC (x)k < δ} ⊂ C. Let z := PC (x) +

x − PC (x) δ · , 2 kx − PC (x)k

where by assumption kx − PC (x)k > 0, since x 6∈ C. Obviously z ∈ Sδ .

Hence,

δ . kx − zk = |x − PC (x) 1 − 2kx − PC (x)k However, δ can be chosen arbitrarily small, thus choose δ < kx − PC (x)k.

9

However, this is not possible, since in this case kx − zk < kx − PC (x)k, which violates the definition of projection. Thus, PC (x) lies onto the boundary of C. 13. Derive the formula for the projection onto a hyperplane in a (real) Hilbert space, H. Solution: Let us first show that a hyperplane, H, is a closed convex set.

Convexity is shown trivially. To show closeness, let yn ∈ H −→ y∗ . We will show that y∗ ∈ H. Let 0 ≤ khθ, y∗ i + θ0 k2

=

khθ, y∗ i + hθ, yn i − hθ, yn i + θ0 k2

=

khθ, y∗ − yn ik2 .

Hence, 0 ≤ khθ, y∗ i + θ0 k2 = lim khθ, y∗ − yn ik2 = 0, n→∞

or hθ, y∗ i + θ0 = 0, which proves the claim.

Let now z ∈ H be the projection of x ∈ H, i.e., z : PC (x). Then by the definition z := arg min hx − z, x − zi. hθ,zi+θ0 =0

Using Lagrange multipliers, we obtain the Lagrangian

L(z, λ) = hx − z, x − zi − λ hθ, zi + θ0 .

For those not familiar with infinite dimensional spaces, it suffices to say that similar rules of differentiation apply, although the respective definitions are different (more general).

After differentiation of the Lagrangian, we obtain 2z − 2x − λθ = 0 or

1 (2x + λθ).

2 Plugging into the constraint, we obtain z=

λ = −2 which then results in the solution.

hθ, xi + θ0 , kθk2

10

14. Derive the formula for the projection onto a closed ball, B[0, ρ].

Solution: The closed ball is defined as B[0, ρ] = {y : kyk ≤ ρ}.

We have already seen (Problem 2) that it is convex. Let us show the closeness. Let yn ∈ B[0, ρ] → y ∗ .

We have to show that y ∗ ∈ B[0, ρ]. ky ∗ k2

= ky ∗ − yn + yn k2 ≤ kyn k2 + ky ∗ − yn k2 ≤ ρ2 + ky ∗ − yn k2 ,

or ky ∗ k2 ≤ ρ2 + lim ky ∗ − yn k2 , n→∞

or ky ∗ k2 ≤ ρ2 , which proves the claim.

To derive the projection, we follow similar steps as in Problem 13, by replacing the constraint by kzk2 = ρ2 , since the projection is on the boundary. Taking the gradient of the Lagrangian we get 2(z − x) + 2λz = 0, or z=

1 x.

1+λ

Plugging into the constraint, we get |1 + λ| =

1 kxk. ρ

When 1 + λ = ρ1 kxk, we get z=

ρ x, kxk

and when 1 + λ = − ρ1 kxk z=−

ρ x. kxk

11

From the two possible vectors, we have to keep the one that has the smaller distance from x. However, x−

ρ ρ x < x+ x , kxk kxk

ρ since kxk < 1. Thus,

1+λ=

1 kxk, ρ

and the projection is equal to

PB[0,ρ] (x) =

ρ kxk x,

x

||x|| > ρ otherwise.

15. Find an example of a point whose projection on the `1 ball is not unique.

Solution: The `1 ball of radius ρ = 1 in Rl is defined as X S1 [0, ρ] = {y : |yi | ≤ ρ}. i=1

Let the point x = [1, 1]T ∈ R2 , which obviously does not lie inside the `1 ball of radius ρ = 1, since kxk1 = 2 > 1. For any point y ∈ S1 [0, 1] we have kx − yk1 = |1 − y1 | + |1 − y2 | ≥ 1 − |y1 | + 1 − |y2 | or kx − yk1 ≥ 2 − kyk1 ≥ 1.

That is, the `1 norm of x from any point in the set S1 [0, 1] is bounded below by 1. Consider the two points y1 = [1, 0]T and y1 = [0, 1]T .

For both of them the lower bound is achieved, i.e., kx − y1 k = kx − y2 k = 1.

Moreover, one can easily check out that all points on the line segment y : y1 + y2 = 1 can be projection points of x.

16. Show that if C ⊂ H, is a closed convex set in a Hilbert space, then ∀x ∈ H and ∀y ∈ C, the projection PC (x) satisfies the following properties:

12

• Real{hx − PC (x), y − PC (x)i} ≤ 0.

• kPC (x) − PC (y)k2 ≤ Real{hx − y, PC (x) − PC (y)i}.

Solution: We know that PC (x) ∈ C. Hence, due to the convexity of C, ∀λ ∈ [0, 1] λy + (1 − λ)PC (x) ∈ C. Hence kx − PC (x)k2

≤

kx − (λy + (1 − λ)PC (x)) k2

≤

k(x − PC (x)) − λ(y − PC (x))k2 ,

which by the definition of the norm, gives kx − PC (x)k2

≤

kx − PC (x)k2 + λ2 ky − PC (x)k2 − 2λReal{hx − PC (x), y − PC (x)i}.

or

λ ky − PC (x)k2 .

2 Taking the limit λ → 0, we prove the first property.

Real{hx − PC (x), y − PC (x)i} ≤

To prove the second property, since PC (y) ∈ C, we apply the previous property with PC (y) in place of y, i.e., Real {hx − PC (x), PC (y) − PC (x)i} ≤ 0.

Similarly Real {hy − PC (y), PC (x) − PC (y)i} ≤ 0.

After adding the above inequalities together and rearranging the terms we obtain the second property.

17. Prove that if S is a closed subspace S ⊂ H in a Hilbert space H, then ∀x, y ∈ H, hx, PS (y)i = hPS (x), yi = hPS (x), PS (y)i. and PS (ax + by) = aPS (x) + bPS (y).

Hint: Use the result of Problem 18.

Solution: We have that hx, PS (y)i = hPS (x) + (x − PS (x)), PS (y)i = hPS (x), PS (y)i, since x − PS (x) ⊥ PS (y)

13

from Problem 18. Similarly, we can show that hPS (x), yi = hPS (x), PS (y)i.

Hence hx, PS (y)i = hPS (x), yi.

For the linearity, we have x

= PS (x) + (x − PS (x)),

y

= PS (y) + (y − PS (y)),

where PS (x), PS (y) ∈ S and x−PS (x) ∈ S ⊥ and y −PS (y) ∈ S ⊥ . Hence ax + by = (aPS (x) + bPS (y)) + (a(x − PS (x)) + b(y − PS (y))) , and since the term in the second parenthesis on the right hand side lies in S ⊥ we readily obtain that PS (ax + by) = aPS (x) + bPS (y).

18. Let S be a closed convex subspace in a Hilbert space H, S ⊂ H. Let S ⊥ be the set of all elements x ∈ H which are orthogonal to S. Then show that, a) S ⊥ is also a closed, subspace, b) S ∩ S ⊥ = {0}, c) H = S ⊕ S ⊥ ; that is, ∀x ∈ H, ∃x1 ∈ S and x2 ∈ S ⊥ : x = x1 +x2 , where x1 , x2 are unique.

Solution: a) We will first prove that S ⊥ is a subspace. Indeed if x1 ∈ S ⊥ and x2 ∈ S ⊥ then hx1 , yi = hx2 , yi = 0, ∀y ∈ S. or hax1 + bx2 , yi = 0 ⇒ ax1 + bx2 ∈ S ⊥ .

Also, 0 ∈ S ⊥ since hx, 0i = 0. Hence S ⊥ is a subspace.

We will prove that S ⊥ is also closed. Let {xn } ∈ S ⊥ and lim xn = x∗ .

n→∞

We will show that x∗ ∈ S ⊥ . By the definition hxn , yi = 0, ∀y ∈ S. Moreover, |hx∗ , yi| = =

|hxn , yi − hx∗ , yi| |hxn − x∗ , yi| ≤ kxn − x∗ kkyk → 0,

14

where the Cauchy-Schwartz inequality has been used. The last inequality leads to hx∗ , yi = 0 ⇒ x∗ ∈ S ⊥ . b) Let x ∈ S ∩ S ⊥ . By definition, since it belongs to both subspaces, hx, xi = 0 ⇒ x = 0, c) Let x ∈ H. We have that x = PS (x) + (x − PS (x)).

We will first show that x − PS (x) ∈ S ⊥ . Then we will show that this decomposition is unique. We already know that Real{hx − PS (x), y − PS (x)i} ≤ 0, ∀y ∈ S. Also, since S is a subspace, ay ∈ S, ∀a ∈ R, hence Real{hx − PS (x), ay − PS (x)i} ≤ 0, or aReal{hx − PS (x), yi} ≤ Real{hx − PS (x), PS (y)i}, which can only be true if Real{hx − PS (x), yi} = 0.

We apply the same for jy. Then we have that hx − PS (x), jyi = −jhx − PS (x), yi.

Recall that if c ∈ C Imag{c} = Real{−jc}, Hence, Real{−jhx − PS (x), yi} = 0 = Imag{hx − PS (x), yi}.

Thus, hx − PS (x), yi} = 0, ∀y ∈ S, and x − PS (x) ∈ S ⊥ .

Thus x = x1 + x2 , x1 = PS (x) ∈ S, x2 = x − PS (x) ∈ S ⊥ .

Let us now assume that there is another decomposition, x = x3 + x4 , x3 ∈ S, x4 ∈ S ⊥ .

15

Then x1 + x2 = x3 + x4 or S → x1 − x3 = x4 − x2 ∈ S ⊥ , which necessarily implies that they are equal to the single point comprising S ∩ S ⊥ , i.e., x1 − x3 = 0 = x4 − x2 , hence the decomposition is unique and we have proved the claim.

Let us elaborate a bit more. we will show that, PS ⊥ (x) = x − PS (x).

Indeed, PS ⊥ (x) is unique. Also, x = PS (x) + (x − PS (x)). or = 0 + PS ⊥ (x − PS (x))

PS ⊥ (x)

= x − PS (x), since x − PS (x) ∈ S ⊥ . Note that we used the fact that if y ∈ S then PS ⊥ (y) = 0.

Indeed, ky − 0k2 = kyk2 < ky − ak2 , ∀a 6= 0 ∈ S ⊥ since ky − ak2

= kyk2 + kak2 − 2Realhy, ai = kyk2 + kak2

19. Show that the relaxed projection operator is a non-expansive mapping.

Solution: By the respective definitions we have, kTC (x) − TC (y)k

=

kx + µ(PC (x) − x) − y − µ(PC (y) − y)k

= k(1 − µ)(x − y) + µ(PC (x) − PC (y))k. a) µ ∈ (0, 1]. Recalling the triangle property of a norm (Appendix of Chapter 8), we get kTC (x) − TC (y)k

≤

|1 − µ|kx − yk + µkPC (x) − PC (y)k

≤

|1 − µ|kx − yk + µkx − yk

≤

kx − yk.

16

b) µ ∈ (1, 2). In this case kTC (x) − TC (y)k2

=

(1 − µ)2 kx − yk2 + µ2 kPC (x) − PC (y)k2

+

2µ(1 − µ)Real{hx − y, PC (x) − PC (y)i}

≤ (1 − µ)2 kx − yk2 + µ2 kPC (x) − PC (y)k2 +

2µ(1 − µ)kPC (x) − PC (y)k2

=

(1 − µ)2 kx − yk2 + µ(2 − µ)kPC (x) − PC (y)k2

≤

(1 − µ)2 kx − yk2 + µ(2 − µ)kx − yk2

≤

kx − yk2 .

To derive the bounds we used that 1 − µ < 0 and 2 − µ > 0, for µ ∈ (1, 2).

20. Show that the relaxed projection operator is a strongly attractive mapping.

Solution: By the respective definition we have, kTC (x) − yk2

=

kx + µ(PC (x) − x) − yk2

=

kx − yk2 + µ2 kPC (x) − xk2 + 2µReal{hx − y, PC (x) − xi}

=

kx − yk2 + µ2 kPC (x) − xk2

+

2µReal{hx − PC (x) + PC (x) − y, PC (x) − xi}

=

kx − yk2 + µ2 kPC (x) − xk2 − 2µkPC (x) − xk2

+

2µReal{hPC (x) − y, PC (x) − xi}

=

kx − yk2 + µ2 kPC (x) − xk2 − 2µkPC (x) − xk2

+

2µReal{hx − PC (x), y − PC (y)i}

≤

kx − yk2 − µ(2 − µ)kPC (x) − xk2 .

where (8.15) has been used. Thus kTC (x) − yk2

µ(2 − µ) kTC (x) − xk2 µ2 (2 − µ) = kx − yk2 − kTC (x) − xk2 , µ ≤

kx − yk2 −

where we used that PC (x) − x =

1 (TC (x) − x). µ

21. Give an example of a sequence in a Hilbert space H, which converges weakly but not strongly.

Solution: Define the sequence of points xn ∈ l2 , xn = {0, 0, . . . , 1, 0, 0, . . .} := {δni }, i = 0, 1, 2, . . .

17

That is, each point, xn is itself a sequence, with zeros everywhere, except at time index, n, where it is 1. For every point (sequence) y ∈ l2 , we have that ∞ ∞ X X kyk2 := |yn |2 = |hxn , yi|2 < ∞, n=1

n=1

2

by the definition of l space (Appendix of Chapter 8). The previous inequality implies that hxn , yi −−−−→ 0 = h0, yi. n→∞

On the other hand, xn 6−→ 0, since kxn − 0k = kxn k = 1.

22. Prove that if C1 . . . CK are closed convex sets in a Hilbert space H, then the operator T = TCK · · · TC1 , is a regular one; that is, kT n−1 (x) − T n (x)k −→ 0, n −→ ∞, where T n := T T . . . T is the application of T n successive times.

Solution: Fact 1: T = TCK TCK−1 · · · TC1 := TK · · · T1 is a non-expansive mapping.

Indeed, ∀x, y ∈ H kT (x) − T (y)k

= kTK (TK−1 · · · T1 )(x) − TK (TK−1 · · · T1 )(y)k ≤

kTK−1 · · · T1 (x) − TK−1 · · · T1 (y)k

≤ .................................... ≤

kT1 (x) − T1 (y)k ≤ kx − yk.

Fact 2: Fix(T ) =

K \

Ck := C.

k=1

Indeed, if x ∈ C then TK TK−1 · · · T1 (x)

=

TK TK−1 · · · T2 (x) = . . .

=

TK (x) = x.

18

Moreover, let us assume that ∃ x 6∈ C: TK (x) = x.

Then ∀y ∈ C we have kx − yk = kT (x) − T (y)k ≤ kT1 (x) − T1 (y)k, as shown before. Thus, kx − yk = kT1 (x) − T1 (y)k = kT1 (x) − yk ≤ kx − yk or kT1 (x) − yk = kx − yk, ∀y ∈ C. which can only be true if x ∈ C and hence T1 (x) = x. Note that the previous two facts are valid for general Hilbert spaces.

Fact3: If C is a closed subspace, then Tk , k = 1, . . . , K, and T = TK TK−1 · · · T1 are linear operators. The proof is trivial from the respective linearity of the projection operators, Pk , k = 1, . . . , K. This is also true for general Hilbert spaces.

Fact 4: The operator T is a regular one, i.e., kT n (x) − T n−1 (x)k −→ 0. n−→∞

Recall from Problem 20 that ∀x ∈ H, kT1 (x) − yk2 ≤ kx − yk2 − µ1 (2 − µ1 )kx − P1 (x)k2 or kx − P1 (x)k2 ≤

1 kx − yk2 − kT1 (x) − yk2 , µ1 (2 − µ1 )

and by the definition T1 (x) = x + µ1 (P1 (x) − x), we get kx − T1 (x)k = µ21 kx − P1 (x)k2 ≤

µ1 kx − yk2 − kT1 (x) − yk2 .

2 − µ1

Let now kx − T2 T1 (x)k2

= kx − T1 (x) + T1 (x) − T2 T1 (x)k2 ≤ (kx − T1 (x)k + kT2 T1 (x) − T1 (x)k)

2

≤ 2 kx − T1 (x)k2 + kT2 T1 (x) − T1 (x)k2 ,

19

or kx − T2 T1 (x)k2

2µ1 (kx − yk2 − kT1 (x) − yk2 ) 2 − µ1 2µ2 (kT1 (x) − yk2 − kT2 T1 (x) − yk2 ).

2 − µ2

≤ +

Let

b2 = max

2µ1 2µ2 , 2 − µ1 2 − µ2

.

Then obviously, we can write kx − T2 T1 (x)k2 ≤ 2b2 (kx − yk2 − kT12 (x) − yk2 ), where

T12 (x) = T2 T1 (x) .

Following a similar rationale and by induction we can show that kx − T (x)k2 ≤ bK 2K−1 (kx − yk2 − kT (x) − yk2 ),

(17)

where, T = TK TK−1 · · · T1 , and

bK = max

1≤k≤K

µk 2 − µk

.

Now by induction, kT (x) − T 2 (x)k2 ≤ bK 2K−1 (kT (x) − y)k2 − kT 2 (x) − yk

(18)

......................................................... kT n−1 (x) − T n (x)k2 ≤ bK 2K−1 (kT n−1 (x) − y)k2 − kT n (x) − yk (19) Summing by parts (17)–(19), we obtain ∞ X

kT n−1 (x) − T n (x)k2 ≤ bK 2K−1 kx − yk2 < +∞.

n=1

Hence, lim kT n−1 (x) − T n (x)k = 0.

n−→∞

Note that till now, everything is valid for general Hilbert spaces.

23. Show the fundamental POCS theorem for the case of closed subspaces in a Hilbert space, H. Solution: Fact 1: The relaxed projection operator is self adjoint, i.e., hx, TCi (y)i = hTCi (x), yi, ∀x, y ∈ H.

20

This is a direct consequence of the self-adjoint property of the projection, when Ck is a closed subspace, i.e., hx, PCk (y)i = hPCk (x), yi = hPCk (x), PCk (y)i.

Fact 2: For a closed subspace, Ck , the respected relaxed projection operator is linear, i.e., TCk (ax + by) = aTCk (x) + bTCk (y), ∀x, y ∈ H. This is also a direct consequence of the linear property of the projection operator onto subspaces. This property is easily checked out that it is readily transferred to T := TCK · · · TC1 .

Fact 3: hx, T (y)i =

hx, TCK · · · TC1 (y)

=

hTCK (x), TCK−1 · · · TC1 (y)

=

...

=

hTC1 · · · TCK (x), yi

=

hT ∗ (x), yi,

where T ∗ = TC1 TC2 · · · TCK .

Fact 4: Let the operator T = TCK · · · TC1 , with Fix(T ) = C, where C is a closed subspace. Then, the set S := {y : y = (I − T )(z), ∀z ∈ H} is also a (closed) subspace and it is the orthogonal complement of C, i.e., S = C ⊥.

The proof that S is a subspace is trivial, from the linearity of T . Also, let x ∈ S ⊥ . Then, by the respective definition, 0

= hx, (I − T )(z)i = hx, zi − hx, T (z)i = hx, zi − hT ∗ (x), zi = h(I − T ∗ )(x), zi, ∀z ∈ H.

Hence, (I − T ∗ )(x) = x − T ∗ (x) = 0, or T ∗ (x) = x, and since T ∗ and T have the same fixed point set (the proof trivial), S ⊥ ⊆ C.

21

Let now x ∈ C. Then hx, (I − T )zi =

hx, zi − hx, T (z)i

=

hx, zi − hT ∗ (x), z, i

=

hx − T ∗ (x), zi = 0,

since T ∗ (x) = x, which proves that S ⊥ = C. Note that what we have said so far is a generalization of Problem 18. We are now ready to establish strong convergence.

The repeated application of T on any x ∈ H leads to T n (x) = (T T T )n (x).

We know that ∀x ∈ H there is a unique decomposition into two orthogonal complement (closed) subspaces, i.e., x = y + z, y ∈ C and z ∈ C ⊥ , ∀x ∈ H and that y = PC (x).

Hence, due to the linearity of T n (C subspace in H) T n (x)

= T n (y) + T n (z) = y + T n (z),

since C = Fix(T n ). However, T n (z)

= T n (I − T )w, for somew ∈ H = T n (w) − T n+1 (w)

and we know that kT n (z)k = kT n (w) − T n+1 (w)k −→ 0.

Thus, kT n (x) − PC (x)k −→ 0. which proves the claim.

24. Derive the subdifferential of the metric distance function dC (x), where C is a closed convex set C ⊆ Rl and x ∈ Rl .

Solution: By definition we have ∂dC (x) = {g : g T (y − x) + dC (x) ≤ dC (y), ∀y ∈ Rl }.

Thus let g be a subgradient, then g T (y − x) ≤ dC (y) − dC (x) ≤ ky − xk.

22

The above is easily shown by the respective definition dC (y) = min ky − zk z∈C

≤

min(ky − xk + kx − zk)

≤

ky − xk + min kx − zk,

z∈C

z∈C

or dC (y) − dC (x) ≤ ky − xk.

Hence, g T (y − x) ≤ ky − xk.

Since this is true ∀y, let y : y − x = g ⇒ kgk2 ≤ kgk ⇒ kgk ≤ 1, or g ∈ B[0, 1]. a) Let x ∈ / C and g any subgradient. For any y ∈ Rl , g T (y + PC (x) − x) ≤ dC (y + PC (x)) − dC (x).

However, dC (y + PC (x))

=

min ky + PC (x) − zk

≤

kyk + min kPC (x) − zk,

z∈C

z∈C

and letting z = PC (x) dC (y + PC (x)) ≤ kyk.

Hence, we can write that g T (y + PC (x) − x) ≤ kyk − dC (x), ∀y ∈ Rl .

Set y = 0. Then, −g T (x − PC (x)) ≤ −kx − PC (x)k, or g T (x − PC (x)) ≥ kx − PC (x)k.

However, kgk ≤ 1, and recalling the Cauchy-Schwartz inequality, we obtain kx − PC (x)kkgk ≥ kx − PC (x)k ⇒ kgk = 1, and g T (x − PC (x)) = kx − PC (x)k,

23

which implies (recall condition for equality in the Cauchy-Schwartz theorem) (x − PC (x)) g= , k(x − PC (x))k which proves the claim. b) Let x ∈ C. Then by definition, we have g T (y − x) ≤ dC (y) − dC (x), and for any y ∈ C g T (y − x) ≤ 0, kgk ≤ 1.

(20)

If in addition x is an interior point, there will be ε > 0 : ∀z ∈ Rl g T (x − ε(z − x) − x) ≤ 0, since x − ε(z − x) ∈ C and the condition (20) has been used. Thus, g T (z − x) ≤ 0, ∀z ∈ Rl .

Set z − x = g, which leads to g = 0.

This completes the proof.

25. Derive the bound in (8.55).

Solution: Subtracting θ∗ from both sides of the recursion, squaring and taking into account the definition of the subgradient, it is readily shown that,

||θ (i) − θ∗ ||2 ≤ ||θ (i−1) − θ∗ ||2 − 2µi J(θ (i−1) ) − J(θ∗ ) + µ2i ||J 0 (θ (i−1) )||2 .

Applying the previous recursively, we obtain ||θ (i) − θ∗ ||2

≤

||θ (0) − θ∗ ||2 − 2

i X

µk J(θ (k−1) ) − J(θ∗ ) +

k=1 i X

µ2k ||J 0 (θ (k−1) )||2 .

k=1

Taking into account the bound of the subgradient and the fact that the left hand side of the inequality is a non-negative number, we obtain 2

i X k=1

i X

µk J(θ (k−1) ) − J(θ∗ ) ≤ ||θ (0) − θ∗ ||2 + µ2k G2 . k=1

(21)

24

However, by the respective definition we get (i)

J(θ (k−1) ) − J(θ∗ ) ≥ J∗ − J(θ∗ ), k = 1, . . . , i.

Employing the previous bound in (21), the claim is readily obtained, i.e., Pi µ2k 2 ||θ (0) − θ∗ ||2 (i) J∗ − J(θ) ≤ + Pk=1 G .

Pi i 2 k=1 µk 2 k=1 µk 26. Show that if a function is γ-Lipschitz, then any of its subgradients is bounded.

Solution: By the definition of the subgradient we have that ∀u, v we have that 0 f (u) − f (v) ≥< f (v)(u − v) > .

Since this is true for all, u, v, I can always select a u so that u − v to be 0 parallel to f (v), Then 0

0

0

< f (v)(u − v) >= | < f (v)(u − v) > | = ||f (v)||||u − v||.

0

Then, if we plug in the Lipschitz condition, we show that ||f (v)|| is bounded.

27. Show the convergence of the generic projected subgradient algorithm in (8.61).

Solution: Let us break the iteration into two steps, z (i) θ

(i)

= θ (i−1) − µi J 0 (θ (i−1) ), = PC (z

(i)

).

(22) (23)

Then, following the same arguments as the ones adopted in Problem 25, we get ||z (i) − θ∗ ||2

≤

||θ (0) − θ∗ ||2 − 2

i X

µk J(θ (k−1) ) − J(θ∗ ) +

k=1 i X

µ2k ||J 0 (θ (k−1) )||2 .

(24)

k=1

However, from the non-expansive property of the projection operator, and taking into account that θ∗ ∈ C, since it is a solution, ||θ (i) − θ∗ ||2 = ||PC (z (i) ) − PC (θ∗ )||2 ≤ ||z (i) − θ∗ ||2 .

(25)

Combining the last two formulas the proof proceeds as in Problem 25.

25

28. Derive equation (8.100).

Solution: By the definition n

Jn (θ) =

1X L(yk , xk , θ). n k=1

Hence, Jn (θ)

1 n

=

n−1 X

!

L(yk , xk , θ) + L(yn , xn , θ)

k=1

n−1 1 Jn−1 (θ) + L(yn , xn , θ), n n

= or ∇Jn (θ) =

n−1 1 ∇Jn−1 (θ) + ∇L(yn , xn , θ). n n

Hence

1 ∇L(yn , xn , θ∗ (n − 1)). n Expanding the left hand side to a first order Taylor approximation we get, ∇Jn (θ∗ (n − 1)) = 0 +

∇Jn (θ∗ (n)) = 0 = ∇Jn (θ∗ (n−1))+∇2 Jn (θ∗ (n−1)) (θ∗ (n) − θ∗ (n − 1)) , or ∇Jn (θ∗ (n − 1)) = ∇2 Jn (θ∗ (n − 1)) (θ∗ (n) − θ∗ (n − 1)) , which finally proves the claim.

29. Consider the online version of PDMb in (8.64), i.e.,

(

n−1 ) 0 n−1 ) , If J 0 (θ n−1 ) 6= 0, PC θn−1 − µn ||JJ(θ 0 (θ n−1 )||2 J (θ θn = PC (θ n−1 ), If J 0 (θ n−1 ) = 0,

(26)

where we have assumed that J∗ = 0. If this is not the case, a shift can accommodate for the difference. Thus we assume that we know the minimum. For example, this is the case for a number tasks, such as the hinge loss function, assuming linearly separable classes, or the linear insensitive loss function, for bounded noise. Assume that n X

Ln (θ) =

ωk dCk (θn−1 ) dCk (θ) k=n−q+1 ωk dCk (θn−1 )

Pn k=n−q+1

Then derive that APSM algorithm of (8.39).

Solution: Let the loss function be Ln (θ) =

n X

ωk dCk (θn−1 ) dCk (θ) = k=n−q+1 ωk dCk (θn−1 )

n X

Pn k=n−q+1

k=n−q+1

βk dCk (θ),

26

where n X

βk = 1.

k=n−q+1

Then for the recursion (26), we need to compute the subgradient of Ln (θ), which by Example 8.5 (and for θ ∈ / Ck ) becomes n X

L´n (θn−1 ) =

βk d0Ck (θ)

= θ=θn−1

k=n−q+1

n X

βk

k=n−q+1

θn−1 − PCk (θn−1 ) , dCk (θn−1 )

or L´n (θn−1 )

=

1 L

n X

ωk (θn−1 − PCk (θn−1 )), with

k=n−q+1 n X

ωk dCk (θn−1 ).

L =

k=n−q+1

Hence, (26) now becomes (using µ0n instead, for reasons to become apparent soon), θn

=

Pn 1 2 k=n−q+1 ωk dCk (θn−1 ) L 0 θn−1 − µn 2 Pn 1 2 k=n−q+1 ωk (θn−1 − PCk (θn−1 )k L k n X 1 L

ωk (θn−1 − PCk (θn−1 ))

k=n−q+1

 =

θn−1 + µ0n M 

n X

 ωk (PCk (θn−1 ) − θn−1 ))

k=n−q+1

 =

θn−1 + µ0n M 

n X

 ωk PCk (θn−1 ) − θn−1 

k=n−q+1

and Pn M :=

2 k=n−q+1 ωk dCk (θn−1 )

k

Pn

2 k=n−q+1 ωk PCk (θn−1 ) − θn−1 k

.

Setting µn = µ0n M ∈ (0, 2M ) the APSM algorithm results.

30. Derive the regret bound for the subgradient algorithm in (8.83).

27

Solution: From the text, we have that Ln (θn−1 ) − Ln (h) ≤ gnT (θn−1 − h)

1 ||θn−1 − h||2 − ||θn − h||2 + ≤ 2µn µn 2 G .

2 Summing up both sides, results in N X

Ln (θn−1 ) −

n=1

N X

Ln (h) ≤

n=1

(27)

N

X 1 ||θn−1 − h||2 − ||θn − h||2 + 2µn n=1 N

G2 X µn .

2 n=1

(28)

Carrying out the summations on the left hand side we get A

1 1 ||θ0 − h||2 − ||θ1 − h||2 + 2µ1 2µ1 1 1 ||θ1 − h||2 − ||θ2 − h||2 + 2µ2 2µ2 .........

1 1 ||θN −1 − h||2 − ||θN − h||2 + 2µN 2µN N G2 X µn , 2 n=1

:=

or A

≤

1 1 ||θ0 − h||2 − ||θ1 − h||2 + 2µ1 2µ1 1 1 ||θ1 − h||2 − ||θ2 − h||2 + 2µ2 2µ2 .........

N G2 X 1 µn , ||θN −1 − h||2 + 2µN 2 n=1

Taking into account the bound ||θn −h||2 ≤ F 2 , and selecting the step-size to be a decreasing sequence, we readily get A ≤ F2

N

1 2µ1

+

N

1X 1 1 G2 X − + µn , 2 n=2 µn µn−1 2 n=1

(29)

N 1 G2 X F2 + µn , .

2µN 2 n=1

(30)

which then easily leads to A≤

Combining the above with (28), the claim is proved.

28

31. Show that a function f (x) is σ-strongly convex if and only if the function f (x) − σ2 ||x||2 is convex.

Solution: a) Assume that

σ ||x||2 , 2 is convex. Then, by the definition of the subgradient at x, we have f (x) −

f (y) −

σ σ ||y||2 − f (x) + ||x||2 ≥ g T (y − x) − σxT (y − x), 2 2

(31)

which readily implies that f (y) − f (x) ≥ g T (y − x) +

σ ||y − x||2 , 2

(32)

from which the strong convexity of f (x) is deduced. b) Assume that f (x) is strongly convex. Then by its definition we have, σ σ ||y||2 + ||x||2 − σxT y 2 2 +σ||x|||2 − σ||x||2 ,

f (y) − f (x) ≥ g T (y − x) +

or σ σ ||y||2 − ||x||2 − σxT y + σ||x||2 , 2 2

(33)

σ σ ||y||2 − f (x) + ||x||2 ≥ g T (y − x) − σxT (y − x), 2 2

(34)

f (y) − f (x) ≥ g T (y − x) + from which we obtain, f (y) −

which proves the claim that f (x) − σ2 ||x||2 is convex.

1 32. Show that if the loss function is σ-strongly convex, then if µn = σn , the regret bound for the subgradient algorithm becomes N N 1 X 1 X G2 (1 + ln N ) Ln (θn−1 ) ≤ Ln (θ∗ ) + .

N n=1 N n=1 2σN

(35)

Solution: Taking into account the strong convexity we have that, Ln (θn−1 ) − Ln (θ∗ ) ≤ gnT (θn−1 − θ∗ ) −

σ ||θn−1 − θ∗ ||2 , 2

(36)

and following similar arguments as for Problem 30, we get

1 Ln (θn−1 ) − Ln (θ∗ ) ≤ ||θn−1 − θ∗ ||2 − ||θn − θ∗ ||2 − 2µn σ µn 2 ||θn−1 − θ∗ ||2 + G .

(37) 2 2

29 1 Using µn = σn , results in

2 Ln (θn−1 ) − Ln (θ∗ ) ≤

σn ||θn−1 − θ∗ ||2 − ||θn − θ∗ ||2 − σ||θn−1 − θ∗ ||2 +

1 2 G . σn

Summing up both sides we obtain 2

N X

Ln (θn−1 ) − Ln (θ∗ ) ≤

n=1

σ ||θ0 − θ∗ ||2 − ||θ1 − θ∗ ||2 − σ||θ0 − θ∗ ||2 +

2σ ||θ1 − θ∗ ||2 − ||θ2 − θ∗ ||2 − σ||θ1 − θ∗ ||2 + ..................................................................

N σ ||θN −1 − θ∗ ||2 − ||θN − θ∗ ||2 − σ||θN −1 − θ∗ ||2 + G2

N X 1

σn

n=1

,

or 2

N X

Ln (θn−1 ) − Ln (θ∗ )||2 ≤

n=1

σ ||θ0 − θ∗ ||2 − ||θ1 − θ∗ − σ||θ0 − θ∗ ||2 +

2σ ||θ1 − θ∗ ||2 − ||θ2 − θ∗ − σ||θ1 − θ∗ ||2 + ..................................................................

N σ ||θN −1 − θ∗ ||2 − σ||θN −1 − θ∗ ||2 + 2

G

N X 1 n=1

≤ G2

σn

N X 1 n=1

σn

.

Using now the bound N X 1

n n=1 the claim is proved.

Z N ≤1+ 1

1 dt = (1 + ln N ), t

(38)

30

33. Consider a batch algorithm that computes the minimum of the empirical loss function, θ∗ (N ), having a quadratic convergence rate, i.e., ln ln

1 ∼ i. ||θ (i) − θ∗ (N )||2

Show that an online algorithm, running for n time instants so that to spend the same computational processing resources as the batch one, achieves for large values of N better performance than the batch algorithm, i.e., [12], ||θn − θ∗ ||2 ∼

1 1 << ∼ ||θ∗ (N ) − θ∗ ||2 .

N ln ln N N

Hint: Use the fact that ||θn − θ∗ ||2 ∼

1 , n

and ||θ∗ (N ) − θ∗ ||2 ∼

1 .

N

Solution: Let K be the number of operations per iteration for the online algorithm. This amounts to a total of Kn operations. The batch algorithm, in order to make sense, should perform O(ln ln N ) operations, so that to get close to ||θ (i) −θ∗ (N )||2 ∼ 1/N . Assuming that at each iteration it performs, approximately, K1 N operations, this amounts to a total of K1 N ln ln N operations. To keep the same load for both algorithms, it should be, Kn = K1 N ln ln N. This leads to the following approximate accuracies, ||θn − θ∗ ||2 ∼

1 1 << ∼ ||θ∗ (N ) − θ∗ ||2 , N ln ln N N

which proves the claim. Note that in practice, the values of K and K1 play an important role as well.

34. Show property (8.111) for the proximal operator.

Solution: Assume first that p = Proxλf (x). By definition, f (p) +

1 1 2 2 kx − pk ≤ f (v) + kx − vk , 2λ 2λ

∀v ∈ Rl .

Since the previous inequality holds true for any v ∈ Rl , it also holds true for αv + (1 − α)p, where v is any vector in Rl , and α any real number

31

within (0, 1). Hence, 1 1 2 2 kx − αv − (1 − α)pk − kx − pk 2 2 1 2 ≤ λαf (v) + λ(1 − α)f (p) + kx − pk 2 1 1 2 2 + α2 kv − pk − α hx − p, v − pi − kx − pk 2 2 1 2 = λαf (v) + λ(1 − α)f (p) + α2 kv − pk − α hx − p, v − pi .

2

λf (p) ≤ λf (αv + (1 − α)p) +

After re-arranging terms in the previous relation, 1 2 λf (p) ≤ λf (v) + α kv − pk − hx − p, v − pi , 2

∀α ∈ (0, 1).

Application of limα→0 on both sides of the previous inequality results in the desired hv − p, x − pi ≤ λ f (v) − f (p) .

Conversely, assume that hv − p, x − pi /λ ≤ f (v) − f (p). Then, f (p) +

1 1 1 2 2 kx − pk ≤ f (v) + kx − pk − hv − p, x − pi 2λ 2λ λ 1 1 2 = f (v) + k(x − v) + (v − p)k − hv − p, x − pi 2λ λ 1 1 2 2 = f (v) + kx − vk + kv − pk 2λ 2λ 1 1 + hv − p, x − vi − hv − p, x − pi λ λ 1 1 1 2 2 2 = f (v) + kx − vk + kv − pk − kv − pk 2λ 2λ λ 1 1 2 2 = f (v) + kx − vk − kv − pk 2λ 2λ 1 2 ≤ f (v) + kx − vk , ∀v ∈ Rl .

2λ

The previous inequality clearly suggests that p = Proxλf (x).

35. Show property (8.112) for the proximal operator.

Solution: For compact notations, define pj := Proxλf (xj ), j = 1, 2. Then,

hp2 − p1 , x1 − p1 i ≤ λ f (p2 ) − f (p1 ) ,

hp1 − p2 , x2 − p2 i ≤ λ f (p1 ) − f (p2 ) .

Adding the previous inequalities results into hp1 − p2 , (p1 − p2 ) − (x1 − x2 )i ≤ 0, 2

which in turn leads to the desired kp1 − p2 k ≤ hp1 − p2 , x1 − x2 i.

32

36. Prove that the recursion in (8.118) converges to a minimizer of f .

Solution: Define the mapping R := 2 Proxλf −I. Then, (8.118) takes the following form:

µk xk+1 = xk + R(xk ) − xk 2

µk µk xk + = 1− R(xk ).

2 2 Notice that R is non-expansive: ∀x1 , x2 ∈ Rl ,

2 2 kR(x1 ) − R(x2 )k = 2 Proxλf (x1 ) − Proxλf (x2 ) − (x1 − x2 ) 2

= 4 kProxλf (x1 ) − Proxλf (x2 )k + kx1 − x2 k

2

− 4 hProxλf (x1 ) − Proxλf (x2 ), x1 − x2 i 2

≤ 4 kProxλf (x1 ) − Proxλf (x2 )k + kx1 − x2 k

2

2

− 4 kProxλf (x1 ) − Proxλf (x2 )k 2

= kx1 − x2 k .

In turn, let z be a fixed point, then 2

2 µk µk (xk − z) + R(xk ) − z 2 2

µk µk 2 2 = 1− kxk − zk + kR(xk ) − zk 2 2 µk µk 2 1− kR(xk ) − xk k − 2 2

µk µk 2 2 = 1− kxk − zk + 2 Proxλf (xk ) − z − (xk − z) 2 2 2 − µk (2 − µk ) kProxλf (xk ) − xk k

µk µk 2 2 = 1− kxk − zk + kxk − zk 2 2 2 + 2µk kProxλf (xk ) − zk − 2µk hProxλf (xk ) − z, xk − zi

kxk+1 − zk =

1−

2

− µk (2 − µk ) kProxλf (xk ) − xk k 2

2

2

≤ kxk − zk + 2µk kProxλf (xk ) − zk − 2µk kProxλf (xk ) − zk 2

− µk (2 − µk ) kProxλf (xk ) − xk k 2

2

= kxk − zk − µk (2 − µk ) kProxλf (xk ) − xk k .

Hence, ∀k, 2

2

2

µk (2 − µk ) kProxλf (xk ) − xk k ≤ kxk − zk − kxk+1 − zk .

Given any non-negative integer k0 , the previous telescoping inequality is

33

utilized for all k ∈ {0, . . . , k0 } to produce k0 X

2

2

2

µk (2 − µk ) kProxλf (xk ) − xk k ≤ kx0 − zk − kxk0 +1 − zk

k=0 2

≤ kx0 − zk .

Since the previous relation holds for any k0 , applying limk0 →∞ on both sides of the inequality results into +∞ X

2

µk (2 − µk ) kProxλf (xk ) − xk k < +∞.

(39)

k=0

Moreover, notice that 1 kR(xk+1 ) − xk+1 k 2

1 µk = R(xk+1 ) − R(xk ) + 1 − R(xk ) − xk 2 2 1 µk 1 1− kR(xk ) − xk k ≤ kR(xk+1 ) − R(xk )k + 2 2 2

1 µk ≤ kxk+1 − xk k + 1 − kProxλf (xk ) − xk k 2 2 µk µk = kProxλf (xk ) − xk k + 1 − kProxλf (xk ) − xk k 2 2 = kProxλf (xk ) − xk k .

kProxλf (xk+1 ) − xk+1 k =

Since (kProxλf (xk ) − xk k)k∈N is monotonically non-increasing, and bounded 2 from below, it converges. Necessarily, limk→∞ kProxλf (xk ) − xk k = 0.

Otherwise, there exists an > 0 and a subsequence (km )m∈N such that 2

kProxλf (xkm ) − xkm k ≥ , This together with the fact that limm→∞ imply that

+∞ >

≥

+∞ X

∀m ∈ N.

Pkm

i=0 µi (2 − µi ) = +∞, and (39)

2

µk (2 − µk ) kProxλf (xk ) − xk k

k=0 +∞ X

2

µkm (2 − µkm ) kProxλf (xkm ) − xkm k ≥

m=0

which is clearly absurd.

+∞ X m=0

µkm (2 − µkm ) = +∞,

34

Let x∗ be an arbitrary cluster point. Notice that kx∗ − Proxλf (x∗ )k

2 2

= kx∗ − xkm + xkm − Proxλf (x∗ )k 2

2

= kx∗ − xkm k + kxkm − Proxλf (xkm ) + Proxλf (xkm ) − Proxλf (x∗ )k + 2 hx∗ − xkm , xkm − Proxλf (x∗ )i 2

2

2

= kx∗ − xkm k + kxkm − Proxλf (xkm )k + kProxλf (xkm ) − Proxλf (x∗ )k + 2 hxkm − Proxλf (xkm ), Proxλf (xkm ) − Proxλf (x∗ )i + 2 hx∗ − xkm , xkm − x∗ + x∗ − Proxλf (x∗ )i 2

2

2

≤ kx∗ − xkm k + kxkm − Proxλf (xkm )k + kxkm − x∗ k + 2 hxkm − Proxλf (xkm ), Proxλf (xkm ) − Proxλf (x∗ )i 2

− 2 kx∗ − xkm k + 2 hx∗ − xkm , x∗ − Proxλf (x∗ )i ≤ kxkm − Proxλf (xkm )k

2

+ 2 kxkm − Proxλf (xkm )k kProxλf (xkm ) − Proxλf (x∗ )k + 2 kx∗ − xkm k kx∗ − Proxλf (x∗ )k 2

≤ kxkm − Proxλf (xkm )k + 2 kxkm − Proxλf (xkm )k kxkm − x∗ k + 2 kx∗ − xkm k kx∗ − Proxλf (x∗ )k .

Applying limm→∞ on both sides of the previous inequality results into x∗ = Proxλf (x∗ ) ⇔ x∗ ∈ Fix(Proxλf ). Since x∗ was chosen arbitrarily within the set of all cluster points of (xk )k∈N , then it can be readily seen that all cluster points belong to Fix(Proxλf ).

2

We have already seen that the sequence (kxn − xk )n∈N converges for any x ∈ Fix(Proxλf ). Moreover, any cluster point of (xk )k∈N belongs to Fix(Proxλf ). Let us show now that (xk )k∈N possesses only one cluster point. To this end, assume two cluster points x, y of (xk )k∈N . This means that there exist subsequences (xkm )m∈N and (xlm )m∈N which converge to x and y, respectively. Moreover, notice that

1 2 2 kxk − yk − kxk − xk + kxk2 − kyk2 . hxk , x − yi = 2 2

2

Since both (kxk − xk )k∈N and (kxk − yk )k∈N converge, so does also the sequence (hxk , x − yi)k∈N . Hence, D E hx, x − yi = lim xkm , x − y = lim hxkm , x − yi m→∞

m→∞

= lim hxk , x − yi = lim hxlm , x − yi m→∞ k→∞ D E = lim xlm , x − y = hy, x − yi , m→∞

2

and in turn, kx − yk = 0 ⇒ x = y. To conclude, (xk )k∈N converges to a point in Fix(Proxλf ) = arg minv∈Rl f (v).

35

37. Derive (8.122) from (8.121).

Solution: Use the matrix inversion lemma (A + BD−1 C)−1 = A−1 − A−1 B(D + CA−1 B)−1 CA−1 , with B = C = I, D = A−1 and A = I, which gives (A + I)−1

= = =

which finally leads to the result.

−1 1 1 1 I − 2 I + A−1

−1 1 1 1 I − 2 A+I A

1 1 I − A + I)−1 A,

1

Solutions To Problems of Chapter 9 9.1. Show that if xi , yi , i = 1, 2, . . . , l, are real numbers, then prove the Cauchy-Schwarz inequality: !2 ! ! l l l X X X 2 2 ≤ x i yi xi yi . i=1

i=1

i=1

Solution: Consider the identity l X (zxi + yi )2 ≥ 0, ∀z ∈ R. i=1

Expanding the previous we get z2

l X

x2i + 2z

i=1

l X

x i yi +

i=1

l X

yi2 ≥ 0, ∀z.

i=1

Since the previous quadratic expansion is nonnegative for every z, we know from our early college mathematics that its discriminant must be non-positive, i.e., !2 ! ! l l l X X X 2 2 x i yi − xi yi ≤ 0, i=1

i=1

i=1

which proves the required inequality.

9.2. Prove that the l2 (Euclidean) norm is a true norm, i.e., it satisfies the four conditions that define a norm.

Hint: To prove the triangle inequality, use the Cauchy-Schwarz inequality.

Solution: To prove the three first conditions, i.e., • ||x||2 ≥ 0 • ||x||2 = 0 ⇔ x = 0 • ||αx||2 = |α|||x||2 is straightforward. For the triangle inequality, recall the CauchySchwartz inequality !2 ! ! l l l X X X x i yi ≤ x2i yi2 , i=1

i=1

i=1

2

or l X

v v u l u l uX uX t 2 xy ≤ x t y2.

i=1

i=1

i i

i

i

i=1

Multiplying both terms by two and adding the same nonnegative factors in both sides we get l X i=1

x2i +

l X

yi2 + 2

i=1

l X i=1

x i yi ≤

l X

x2i +

i=1

l X

v v u l u l uX uX 2 t 2 x t y2, y +2 i

i

i=1

i=1

i

i−1

or v 2 v u l u l l uX X uX x2i + t yi2  , (xi + yi )2 ≤ t i=1

i=1

i−1

and taking the square root of both sides we prove the inequality.

9.3. Prove that any function that is a norm is also a convex function.

Solution: Let a function f : Rl 7−→ [0, ∞), which is a norm. Then by the definition of a norm we have f (ax + by) ≤ |a|f (x) + |b|f (y), and using λ and 1 − λ in place of a and b, we prove the claim.

9.4. Show Young’s inequality for nonnegative real numbers a and b, ab ≤

ap bq + , p q

for ∞ > p > 1 and ∞ > q > 1 such that 1 1 + = 1. p q Solution: First, it is obvious that the inequality holds if a and/or b are zero. Then, we turn our attention to the case that both of them are positive. We know that the ln(·) function is a concave one. Then for t = p1 and 1 − t = 1q we have that, ln (tap + (1 − t)bq ) ≥ t ln(ap ) + (1 − t) ln(bq ) = ln a + ln b = ln(ab). where the claim is proved by removing the ln and keeping in mind that ln is a monotonically increasing function.

3

9.5. Prove Holder’s inequality for lp norms, T

||x y||1 =

l X

|xi yi | ≤ ||x||p ||y||q =

i=1

l X

! p1 |xi |

p

q X

!1 q

q

|yi |

,

i=1

i=1

for p ≥ 1 and q ≥ 1 such that 1 1 + = 1. p q Hint: Use Young’s inequality.

Solution: First note that if p = 1 and q = ∞ then the inequality is obvious, since in this case ||y||∞ = max{|y1 |, . . . , |yl |}. Thus we concentrate in the case where are ∞ > p > 1 and ∞ > q > 1. We use |xi | |yi | Young’s inequality with ||x|| , and ||y|| in place of a and b and take p q the sum. This results in l

l

l

i=1

i=1

i=1

X 1 1 X |yi |q 1 1 1 X |xi |p + + = 1, |xi yi | ≤ p q = ||x||p ||y||q p p q ||x||p q ||y||q which proves the claim.

9.6. Prove Minkowski’s inequality, l X (|xi | + |yi |)p

! p1 ≤

i=1

l X

! p1 |xi |p

l X

+

i=1

! p1 |yi |p

,

i=1

for p ≥ 1.

Hint: Use Holder’s inequality together with the identity: (|a| + |b|)p = (|a| + |b|)p−1 |a| + (|a| + |b|)p−1 |b|.

Solution: The result is obvious for p = 1.We will prove it for p > 1.

In the given identity, put in place of a xi and in the place of b yi and add to get l X i=1

(|xi | + yi |)p =

l X

(|xi | + |yi |)p−1 |xi | +

i=1

l X

(|xi | + |yi |)p−1 |yi |.

i=1

Using Holder’s inequality we have l X i=1

(|xi | + |yi |)p−1 |xi | ≤

l X i=1

! 1q (|xi | + |yi |)q(p−1)

l X i=1

! p1 |xi |p

,

4

and similarly l X

(|xi | + |yi |)p−1 |yi | ≤

i=1

l X

! 1q (|xi | + |yi |)q(p−1)

i=1

l X

! p1 |yi |p

.

i=1

If we add both inequalities together and take into account that q(p − 1) = p, we get ! 1q  l ! p1 ! p1  l l l X X X X  (|xi |+|yi |)p ≤ + (|xi | + |yi |)p |xi |p |yi |p  . i=1

Dividing by

i=1

i=1

P

l p i=1 (|xi | + |yi |)

1 q

i=1

, Minkowski’s inequality results.

9.7. Prove that for p ≥ 1 the lp norm is a true norm.

Solution: The first three conditions for a function to be a norm are obvious. The triangle inequality is a direct consequence of Minkowski’s inequality, if we recall that |a + b| ≤ |a| + |b|.

9.8. Use a counterexample to show that the lp norm for 0 < p < 1 is not a true norm and it violates the triangle condition.

Solution: Consider the two vectors in the l-dimensional space x = [1, 0, . . . , 0]T , y = [0, 0, . . . , 1]T .

We will show that for these two vectors the triangle inequality is violated for p < 1. Indeed, we have 1

||x + y||p = 2 p ≤ ||x||p + ||y||p = 1 + 1 = 2, which is not possible for p < 1.

9.9. Show that the null space of a full rank N × l matrix X is a subspace of dimensionality l − N , for N < l.

Solution: By the definition of a null space, i.e., N = {z : Xz = 0} , it is straightforward to see that the three conditions that define a linear space are met, i.e., • 0∈N • If x1 and x2 belong to the null subspace so does their sum

5

• If x belongs to the space so does cx, c ∈ R To show that its dimensionality is l − N , recall from linear algebra (and it is readily established by the respective definitions) that the dimensionality of the range space is N , which is equal to the rank of the matrix, since it has been assumed to be full rank. The null space, being orthogonal to the range space, will necessarily have dimension equal to l − N 9.10. Show, using Lagrange multipliers, that the l2 minimizer in (9.18) accepts the closed form solution θ̂ = X T XX T

−1

y.

Solution: The `2 minimizer task is given by minimize s.t.

||θ||22 xTn θ = yn , n = 1, 2, . . . , N,

(1)

where by assumption N < l. The equivalent Lagrangian is given by T

L(θ) = θ θ +

N X

λn (yn − xTn θ).

n=1

Taking the gradient w.r. to θ and equating to zero we obtain θ = X T λ, where λ = [λ1 , λ2 , . . . , λN ]T , and X being the input matrix  T  x1  ..  X =  . . xTN

Plugging the solution into the set of constraints, which can be written as y = Xθ, we obtain the equation Xθ = y = XX T λ ⇒ λ = XX T which finally results in the solution θ = X T XX T

−1

y.

−1

y,

6

9.11. Show that the necessary and sufficient condition for a θ to be a minimizer of ||θ||1

minimize : s.t.

Xθ = y,

is the following X

sign(θi )zi ≤

i:θi 6=0

X

|zi |, ∀z ∈ N ,

i:θi =0

where N is the null space of X. Moreover, if the minimizer is unique the previous inequality becomes a strict one.

Solution: (a) Sufficiency Let θ be a minimizer, then for any z ∈ N and ∀t ∈ R we have l X

|θi + tzi | ≥

l X

i=1

|θi |,

i

by the definition of the `1 norm. Fix z. Then, for sufficiently small value of t, θi and θi + tzi will have the same sign, whenever θi 6= 0.

Hence, the previous inequality becomes X X X |tzi | + sign(θi )(θi + tzi ) ≥ sign(θi )θi , or i:θi 6=0

i:θi =0

t

X

i:θi 6=0

sign(θi )zi +

i:θi 6=0

X

|tzi | ≥ 0, or

i:θi =0

−t

X

sign(θi )zi ≤ |t|

i:θi 6=0

X

|zi |.

i:θi =0

Since the previous is valid ∀t, sufficiently small, we can choose the sign of t so that to obtain X

sign(θi )zi ≤

i:θi 6=0

X

|zi |.

i:θi =0

Obviously, if θ is a strict minimizer the previous is true with the strict inequality. (b) Sufficiency Let X i:θi 6=0

sign(θi )zi ≤

X i:θi =0

|zi |

(2)

7

be true. Then the following are valid, l X

X

|θi | =

sign(θi )θi

i:θi 6=0

i=1

X

=

X

sign(θi )(θi + zi ) −

i:θi 6=0

sign(θi )zi .

(3)

|zi |.

(4)

i:θi 6=0

However, (2) implies that −

X

|zi | ≤

X

sign(θi )zi ≤

i:θi 6=0

i:θi =0

X i:θi =0

Combining (4), and (3) we get l X

|θi | ≤

X

sign(θi )(θi + zi ) +

i:θi 6=0

i=1

≤

X

X

|zi |

i:θi =0

X

|θi + zi | +

i:θi 6=0

|zi | =

l X

|θi + zi |.

i

i:θi =0

Moreover, if the strict inequality holds in (2), then the minimizer is unique.

9.12. Prove that if the `1 norm minimizer is unique, then the number of its components, which are identically zero, must be at least as large as the dimensionality of the null space of the corresponding input matrix.

Solution: Assume that ξ := card{i : θ̂i = 0} < dim(null(X)) := m.

There exists a set of vectors {u1 , . . . , um } which constitutes a basis for null(X). Hence, for any vector in the solutions set, θ ∈ Θ, there exist nonzero vectors z ∈ null(X) and a ∈ Rm , such that θ = z + θ̂ =

m X

aj uj + θ̂.

j=1

As a result, ∀i such that θ̂i = 0, we have that θi = zi = which generates the following system of equations:      θ i1 u1i1 · · · umi1 a1  ..   ..   . . .. ..   ...   . = . . θiξ

u1iξ

···

umiξ

am

Pm

j=1 aj uji ,

8

· · = θiξ = Since ξ < m, there exists a nonzero a such that θi1 = ·P 0. This implies also that the nonzero z, defined as z = m j=1 aj uj , satisfies zi1 = · · · = ziξ = 0. If we plug this last result into Lemma ?? then we end up with the following absurd relation: X

0≤

sgn(θ̂i )zi <

i: θ̂i 6=0

X

|zi | = 0.

i: θ̂i =0

9.13. Show that the l1 norm is a convex function (as all norms), yet it is not strictly convex. In contrast, the squared Euclidean norm is a strictly convex function.

Solution: From the definition of convexity we have that f (λx + (1 − λ)y) ≤ λf (x) + (1 − λ)f (y), and for strict convexity the strict inequality is valid. For the l1 norm we have X X X (|λxi + (1 − λ)yi |) ≤ λ |xi | + (1 − λ) |yi |, i

i

i

but there is no reason for strict inequality. Take for example the case where all xi and yi to be nonnegative.

However, things are different with the square Euclidean norm. Indeed, for this case we have.

X X X (λxi + (1 − λ)yi )2 < λ x2i + (1 − λ) yi2 . i

i

i

By expanding the term on the left hand side we obtain that X X X X λ2 x2i + (1 − λ)2 yi2 + 2λ(1 − λ)xi yi < λ x2i + (1 − λ) yi2 , i

i

i

i

or

!

2λ(1 − λ)xi yi < λ(1 − λ)

X

x2i +

X

i

yi2

,

i

and finally X

(xi − yi )2 > 0, for x 6= y.

i

Following a similar path, and exploiting the Cauchy-Schwartz inequality (which becomes equality for co-linear vectors) show that the `2 norm function is not strictly convex.

9

9.14. Construct in the 5-dimensional space a matrix that has a) rank equal to five and spark equal to four, b) rank equal to five and spark equal to three and c) rank and spark equal to four.

Solution: What we do here can obviously be applied in any Rl space.

Consider a basis in the space, e.g., 

         1 0 0 0 0  0   1   0   0   0                     e1 =   0  , e2 =  0  , e3 =  1  , e4 =  0  , e5 =  0  .  0   0   0   1   0  0 0 0 0 1 a) Combine the by taking three of them at a time. Obviously previous

5 we can form more vectors. Take, for example, the following 3 combinations          1 1 0 0 1  0   0   0   1   1             , e234 =  1  , e345 =  1  , e134 =  1  , e135 =  1  .

1 e123 =             0   1   1   1   0  1 0 1 0 0 

The array that has as columns ALL the previous vectors, from e1 to e135 , has rank equal to five and spark equal to four. Every combination of three of them are linearly independent. For the case b), combine the vectors of the basis in pairs, then follow the same procedure and it is straightforward to see that the resulting matrix has rank five and spark three. For the case c), take four of the five basis vector together with the ones that result from the combinations formed in a) and the corresponding matrix will be or rank 4 and spark 4.

9.15. Let X be a full row rank N × l matrix, with l > N . Derive the Welch bound for the mutual coherence µ(X), s µ(X) ≥

l−N .

N (l − 1)

(5)

The bound for N = l is obviously zero, since the matrix can be orthogonal.

Solution: Without harming generality, let us assume that the columns

10

xci , i = 1, 2, . . . , l, are normalized to unit norm. Form the Gram matrix   cT c x1 x1 . . . xc1 T xcl  xc T xc . . . xc T xc  2 l   2 1 G = XT X =  (6) . .. .. . .   . . . xcl T x1 . . . xcl T xcl

We know that the trace of G is equal to the sum of its eigenvalues. Moreover, since G is of rank N and semidefinite, only N of its eigenvalues are positive and the rest are zero. Let the nonzero eigenvalues be λ1 , . . . , λN . Form the two vectors 1 = [1, 1, . . . , 1]T and λ = [λ1 , . . . , λN ]T . Apply the Cauchy-Schwartz inequality on these two vectors to obtain !2 N N X X (trace(G))2 = λi (7) ≤N λ2i , i=1

i=1

and taking into account that the trace of G is equal to l (due to the normalization of the columns of X) we obtain N X

λ2i ≥

i=1

l2 .

N

(8)

However from linear algebra, we know that the Frobenius norm satisfies the following: ([Golu 83]. This is true for any matrix and it is easily shown by the definition of the Frobenius norm and the fact that the sum of the square singular values is equal to the trace of the respective Gram) ||G||2F ≡

l X l X

|xci T xcj |2 =

i=1 j=1

N X

σi2 ,

(9)

i=1

where σi , i = 1, 2, . . . , N , are the singular values of G. In addition, we know that the square singular values of a matrix are the eigenvalues of its Gram, [Golu 83]. The Gram of G is equal to GT G = GG, since G is symmetric. Hence the eigenvalues of GT G are equal to the square of the eigenvalues of G. Thus σi2 = λ2i , i = 1, 2, . . . , l. Therefore, we get that l X l N X X |xci T xcj |2 = λ2i .

(10) i=1 j=1

i=1

Combining (8) and (10) results to l X l X i=1 j=1

|xci T xcj |2 ≥

l2 .

N

(11)

11

Taking into account that all the elements in the diagonal of G are equal to one, we get l+

l l X X

l2 , N

(12)

l(l − N ) .

N

(13)

|xci T xcj |2 ≥

i=1 j=1,j6=i

or

l l X X

|xci T xcj |2 ≥

i=1 j=1,j6=i

Also, by the definition of the mutual coherence, it becomes obvious that (the maximum is greater than the average value) 2

(µ(X))

1 := (max(|xci T xcj |2 ) ≥ 2 i6=j l −l

l X

|xci T xcj |2 ,

(14)

j=1,j6=i

and combining it with (13), we finally obtain that s l−N . µ(X) ≥ N (l − 1)

(15)

9.16. Let X be a N × l matrix. Then prove that its spark is bounded as spark(X) ≥ 1 +

1 , µ(X)

where µ(X) is the mutual coherence of the matrix.

Hint: Consider the Gram matrix X T X and the following theorem, concerning positive definite matrices: An m × m matrix A is positive definite if |A(i, i)| >

m X

|A(i, j)|, ∀i = 1, 2, . . . , m,

j=1,j6=i

see, e.g., [Horn 85].

Solution: Let the matrix X be X = [xc1 , xc2 , . . . , xcl ] .

Assume that the columns are normalized to unity. Otherwise, we have to normalize them, which does not affect the spark and the mutual coherence. Then the corresponding Gram matrix is  cT c  x1 x1 . . . xc1 T xcl  xc T xc . . . xc T xc  2 l   2 1 T G=X X= (16) . .. .. . .   . . . xcl T x1 . . . xcl T xcl

12

Observe that all diagonal elements are equal to one. Moreover, all the off-diagonal elements have absolute value less than or equal to the mutual coherence, by its definition. Let 1 ≤ p ≤ N be the maximum number for which the following holds: (p − 1)µ(X) < 1.

(17)

This means that any submatrix G of X T X, that results by combining any p columns of X, will be positive definite, according to the matrix theorem, stated in the hint, since the diagonal elements are 1 and the p − 1 off-diagonals are bounded by µ(X) in absolute value. Moreover, (17) implies that this will also be true for any m ≤ p. Thus, if this is valid, then any m ≤ p columns are linearly independent, hence spark(X) ≥ p + 1,

(18)

However, since p is the maximum value for which (17) is valid, then 1 ≤ pµ(X).

(19)

Combining (19) with (17) results in spark(X) ≥ 1 +

1 . µ(X)

Note that if for p = N (17) is valid, then spark attains its maximum value of N + 1. If p = 1, this necessarily means that the mutual coherence is 1, hence the spark is bounded by two (only one column can be guaranteed to be linearly independent) 9.17. Show that if the underdetermined system of equations y = Xθ accepts a solution such that

1 1 ||θ||0 < 1+ 2 µ(X) then the l1 minimizer is equivalent to the l0 one. Assume that the columns of X are normalized.

Solution: Recall that in this case, the l0 minimizer is unique, so any other solution is bound to have larger l0 norm. We only have to check that if this condition is valid, then any other solution will also have larger l1 norm.

Let us assume that there exists a solution with smaller `1 norm. Then, this means that there will be a vector z in the null space of X such that ||θ + z||1 − ||θ||1 ≤ 0, Xz = 0,

13

where θ is the solution with l0 norm that satisfies the given condition.

Let us assume, without loss of generality, that the k = ||θ||0 nonzero elements are the first k elements of the vector. Then we obtain k X

(|θj + zj | − |θj |) +

j=1

X

|zj | ≤ 0.

j>k

Taking into consideration that |a + b| − |b| ≥ −|a|, the previous inequality, if it is true, should also satisfy the following −

k X

|zj | +

X

j=1

|zj | ≤ 0,

(20)

j>k

P or adding and subtracting the term kj=1 |zj |, the previous becomes equivalent to k X ||z||1 − 2 |zj | ≤ 0, Xz = 0. j=1

The second of the two requirements ( i.e., that z lies in the null space) implies that

X T Xz = 0 ⇒ −z = X T X − I z.

Hence the following is true −zi =

l X

xci T xcj zj − zi =

j=1

l X

xci T xcj zj ,

j=1, j6=i

where we have used the definition of X T X in terms of its columns and the fact that X is normalized, hence the diagonal elements xci T xci = 1.

Taking the absolute values of the previously obtained relationship we have |zi | = |

l X

xci T xcj zj | ≤

j=1, j6=i

l X

|xci T xjj ||zj | ≤ µ(X)

j=1, j6=i

l X j=1, j6=i

µ(X)||z||1 − µ(X))|zi |, i = 1, 2, . . . , l or k

X µ(X) µ(X) |zi | ≤ ||z||1 ⇒ −2 |zi | ≥ −2k ||z||1 .

1 + µ(X) 1 + µ(X) i=1

The last one combined with (20) results in

||z||1

µ(X) 1 − 2k 1 + µ(X)

≤ ||z||1 − 2

k X i=1

|zi | ≤ 0.

|zj | =

14

From the last one we obtain that µ(X) µ(X) ≤ 0 ⇒ 1 ≤ 2k , 1 + µ(X) 1 + µ(X)

1 − 2k and finally that

1 k≥ 2

1 1+ µ(X)

,

which is obviously a contradiction to the condition which we have

1 1 initially assumed to hold true, i.e., k < 2 1 + µ(X) . Hence this proves the claim.

9.18. Prove that if the RIP of order k is valid for a matrix X and δk < 1 , then any m < k columns of X are necessarily linearly independent.

Solution: Recall from the definition of RIP that it is valid for any k sparse vector, that is, for any vector that has up to k nonzero elements. Assume, now, that there exist m ≤ k columns that are linearly dependent. Without loss of generality assume these to be x1 , . . . , xm .

Then, there will be a sparse vector θ = [θ1 , . . . , θm , 0, 0, . . . , 0]T , θi 6= 0, i = 1, 2, . . . , m, such that m X

θi xi = Xθ = 0 ⇒ ||Xθ||22 = 0,

i=1

which is a contradiction, since ||Xθ||22 ≥ (1 − δk )||θ||22 > 0.

9.19. Show that if X satisfies the RIP of order k and some isometry constant δk so does the product XΨ if Ψ is an orthonormal matrix.

Solution: This is a direct consequence of the properties of the matrix orthonormality.

Bibliography [Horn 85] Horn R.A., Johnson C.R. Matrix Analysis, Cambridge University Press, New York 1985.

[Golu 83] Golub G.H., Van Loan C.F. Matrix Computations, The John Hopkins University Press, Baltimore, 1983.

15

1

Solutions To Problems of Chapter 10 10.1. Show that the step, in a greedy algorithm, that selects the column of the sensing matrix, so that to maximize the correlation between the column and the currently available error vector e(i−1) , is equivalent with selecting the column that reduces the l2 norm of the error vector.

Hint: All the parameters obtained in previous steps are fixed, and the optimization is with respect to the new column as well as the corresponding weighting coefficient in the estimate of the parameter vector.

Solution: Let e(k−1) = y − Xθ (k−1) be the current estimate of the residual vector. We know that e(k−1) ⊥ xci1 , xci2 , . . . , xcik−1 .

Let xcik be the new column to be selected and θik the associated coefficient in θ (k) . Then, if we fix all the previously selected columns and their associated coefficients in θ (k) , the square `2 norm of the error becomes ||y − Xθ (k−1) − xcik θik ||22 = ||e(k−1) − xcik θik ||22 .

Minimizing the previous norm w.r. to θik , we easily obtain that θ̂ik =

xcik T e(k−1) xcik T xcik

.

Plugging this optimal value into the error norm, it is easily shown that the error norm, corresponding to the optimal value, is equal to ||e(k−1) − xcik θ̂ik ||22 = ||e(k−1) ||22 − θ̂i2k xcik T xik .

Hence, the error norm is minimized if we select the column to satisfy ik = arg max i

|xci T e(k−1) | . ||xci ||2

10.2. Prove the proposition stating that if there is a sparse solution to the linear system y = Xθ such that

1 1 k0 = ||θ||0 < 1+ , 2 µ(X) where µ(X) is the mutual coherence of X, then the column selection procedure in a greedy algorithm will always select a column among

2

the active columns X, that correspond to the support of θ; that is, the columns that take part in the representation of y in terms of the columns of X. Hint: Assume that k0 X y= θi xci . i=1

Solution: Let m ≤ k0 be the index for which |θm |||xcm || ≥ |θi |||xci ||, i 6= m, i ≤ k0 , where the vector norms are assumed to be the `2 norms. Then if we show that |xcj T y| |xcm T y| > , ∀j > k0 , (1) ||xcm || ||xcj || then there is no way for any column with index outside the support to be selected in the first iteration; there is at least one column, with index within the support, whose correlation with y is greater than that of any column whose index is outside the support.

Combining (1) with the expansion equation leads to k0 X i=1

k0 X xcj T xci xcm T xci > θi , j > k0 . θi ||xcm || ||xcj ||

(2)

i=1

Using known identities of the absolute values, we have k0 X i=1

xc T xc θi m c i ||xm ||

≥

|θm |||xcm || −

k0 X

|θi |

i=1,i6=m k0 X

xcm T xci ||xcm ||

≥

|θm |||xcm || −

|θm |||xcm ||µ(X)

≥

i=1,i6=m c |θm |||xm || (1 − µ(X)(k0 − 1)) .

On the other hand, we have that k0 X i=1

xcj T xci θi ||xcj ||

≤

≤

k0 X i=1 k0 X

|θi |

xcj T xci ||xcj ||

|θi |||xci ||µ(X)

i=1

≤ |θm |||xcm ||µ(X)k0 .

Hence, in order (1) to hold true it suffices that 1 1 − µ(X)k0 + µ(X) ≥ µ(X)k0 ⇒ k0 < 2

1+

1 µ(X)

.

3

For the second iteration step, we have that e(1) = y − θi1 xci1 =

k0 X

θ̃i xci ,

i=1

and we can repeat the previous arguments in a similar way.

10.3. Give an explanation to justify why in step 4 of the CoSaMP algorithm the value of t is taken to be equal to 2k Solution: The estimates, θ (i) , obtained at each iteration are always k-sparse vectors, due to the hard thresholding operation in step 7.

Hence X T e(i−1) = X T (y−Xθ (i−1) ) = X T (Xθ−Xθ (i−1) ) = X T X(θ−θ (i−1) ) and for near orthogonal matrices, X T X ≈ I, the difference between two k-sparse vectors, the true one and the currently available estimate, will be 2k-sparse, approximately.

10.4. Show that if 1 1 J(θ, θ̃) = ||y − Xθ||22 + λ||θ||1 + d(θ, θ̃), 2 2 where d(θ, θ̃) := c||θ − θ̃||22 − ||Xθ − X θ̃||22 , then minimization results to

1 T θ̂ = Sλ/c X (y − X θ̃) + θ̃ . c Solution: Expanding the surrogate function, we obtain J(θ, θ̃) =

1 T 1 y y + θ T X T Xθ − y T Xθ + λ||θ||1 + 2 2 1 T c(θ θ − 2θ̃ T θ + θ̃ T θ̃) − 2 1 T T 1 1 1 θ X Xθ + θ̃ T X T Xθ + θ T X T X θ̃ − θ̃ T X T X θ̃.

2 2 2 2

Note that the term that involves θ T X T Xθ is canceled out. When dealing with the LASSO in the theory, in order to come up with an analytic solution, we had to assume orthogonality on X. With the surrogate function, this term is canceled out by the extra term that we have added. Taking the subgradient of the previous function we have cθ − cθ̃ − X T y + X T X θ̃ + λ∂||θ||1 = 0, and following the same procedure as we did in the theory for the LASSO, for the orthogonal case, we end up with the required solution.

4

10.5. Prove the basic recursion of the parallel coordinate descent algorithm.

Hint: Assume that at the ith iteration, it is the turn of the jth component to be updated, so that the following is minimized 1 (i−1) J(θj ) = ||y − Xθ (i−1) + θj xj − θj xj ||22 + λ|θj |.

2 Solution: Basically in the cost, one removes the contribution of the jth component, as this was estimated by the previous step and attempts the find a new update of the corresponding coefficient that minimizes the function, by keeping the rest of the elements unchanged. Expanding the cost function we get 1 1 J(θj ) = (y − a)T (y − a) − θj (y − a)T xj + θj2 xTj xj + λ|θj |, 2 2 (i−1)

where a = Xθ (i−1) − θj xj . Taking the subgradient with respect to θj and equating to zero, we obtain −(y − a)T xj + θj xTj xj + λ∂|θj | = 0, and following the familiar procedure, as the one used to solve LASSO for the case of an orthogonal matrix X, we obtain ! (i−1) )T x + xT x θ (i−1) (y − Xθ j j j j (i) . θj = Sλ/||xj ||22 xTj xj This completes the proof.

10.6. Derive the iterative scheme to minimize the weighted Pl `1 ball, using a majorization-minimization procedure to minimize i=1 ln(|θi | + ), subject to the measurements set, y = Xθ.

Hint: Use the linearization of the logarithmic function to bound it from above, since it is a concave function and its graph is located below its tangent.

Solution: Our task is minimize w.r. θ

l X

ln(|θj | + ),

j=1

s.t.

Xθ = y.

It is known from optimization theory, and it is easily checked out, that the above task is equivalent to minimize w.r. (θ, u)

l X

ln(uj + ).

j=1

s.t.

Xθ = y |θj | ≤ uj , j = 1, 2, . . . , l.

5

Instead, at each iteration step, we will try to minimize the linearization of the function, around its previous estimate. We have seen that this is also an alternative way to derive the gradient descent, albeit there, we use regularized local optimization. Hence the task now becomes   l l X X (i−1) θ (i) = arg min  ln(uj + ) + (u − u(i−1) )T ∇ (ln(uj + )|u =u(i−1)  θ,u

s.t.

j

j=1

j=1

j

Xθ = y |θj | ≤ uj ,

which is equivalent with θ (i)

 l X  = arg min

s.t.

Xθ = y



uj  (i−1) +

u j=1 j

θ,u

|θj | ≤ uj , which is finally equivalent to θ (i) = arg min

l X

θ

s.t.

|θj | (i−1) |+ j=1 |θj

Xθ = y.

10.7. Show that the weighted `1 ball, used in SpAPSM, is upper bounded by the `0 norm of the target vector.

Solution: Assume that θ(n) converges to the true vector. Moreover assume that ´n ≥ ´ > 0. Then l X

wj (n)|θj (n)| ≤

j=1

l X j=1

|θj (n)| . |θj (n)| + ´

Hence the following is true lim sup n→∞

l X

wj (n)|θj (n)| ≤ lim sup n→∞

j=1

=

l X j=1

X j∈Support(θ∗ )

<

l

X |θj (n)| |θj (n)| = lim |θj (n)| + ´ n→∞ |θj (n)| + ´

X j∈Support(θ∗ )

j=1

|θ∗,j | + |θ∗,j | + ´

X j∈ / Support(θ∗ )

|θ∗,j | = ||θ∗ ||0 . |θ∗,j |

|θ∗,j | |θ∗,j | + ´

6

10.8. Show that the canonical dual frame minimizes the total `2 norm of the dual frame, i.e., X 2 ψ̃i .

2

i∈I

Hint: Use the result of Problem 9.10.

Solution: We know from the theory that the matrices corresponding to the frame and its dual satisfy the following condition Ψ̃ΨH = ΨΨ̃H = I. Let us derive from the above the columns of Ψ̃H one by one. We will work for the first and the rest can be obtained in a similar way. Let ψ̃1∗ denote the first column vector of Ψ̃. Then we have Ψψ̃1∗ = [1, 0, . . . , 0]T .

Then we know that the minimum norm solution for the previous system is given by ψ̃1∗ = ΨH (ΨΨH )−1 [1, 0, . . . , 0]T .

Repeating the above for all columns we get Ψ̃H = ΨH (ΨΨH )−1 =⇒ Ψ̃ = (ΨΨH )−1 Ψ 10.9. Show that Parseval’s tight frames are self dual.

Solution: Let the dimensionality of the frame’s matrix Ψ be N × p.

From the definition of the Parseval tight frame (PTFP) we have that ∀s,

p X

2

|hψi , si| =

i=1

p X

sH ψi ψiH s = sH s.

i=1

Hence p X

sH ψi ψiH s = sH

i=1

p X

ψi ψiH )s = sH ΨΨH s = sH s.

i=1

Hence ΨΨH = I, which obviously proves the claim.

10.10. Prove that the bounds A, B of a frame coincide with the maximum and minimum eigenvalues of the matrix product ΨΨH .

Solution: From the definition of a frame we have A ksk22 ≤

p X i=1

|hψi , si|2 = sH ΨΨH s ≤ B ksk22 .

7

However, we know from linear algebra that the minimum and maximum values for the Rayleigh quotient sH P s , sH s for any Hermitian matrix P , are the maximum and minimum eigenvalues of P . This proves the claim.

8

Bibliography [Horn 85] Horn R.A., Johnson C.R. Matrix Analysis, Cambridge University Press, New York 1985.

[Golu 83] Golub G.H., Van Loan C.F. Matrix Computations, The John Hopkins University Press, Baltimore, 1983.

9

1

Solutions To Problems of Chapter 11 11.1. Derive the formula for the number of groupings O(N, l) in Cover’s theorem.

Hint: Show first the following recursion O(N + 1, l) = O(N, l) + O(N, l − 1).

To this end, start with N points and add an extra one. Show that the extra number of linear dichotomies is solely due to those, for the N data point case, which could be drawn via the new point.

Solution: Let us assume that we start with N points in the l-dimensional space, and we add an extra point P . Then, the O(N, l) old dichotomies fall into either of the two categories.

• For those dichotomies where the dichotomizing plane can not pass through P , the new point P can only belong to the old dichotomies.

• For those dichotomies for which the plane could pass through P , then to each one of the old dichotomies correspond two new dichotomies.

This is because by shifting infinitesimally the plane, P can change class ownership.

Thus, the total number of dichotomies with the N + 1 points will be O(N + 1, l) = O(N, l) + O(N, l − 1).

(1)

The second term is the number of dichotomies of N points in the ldimensional space that are constrained to pass through a specific point P . This is equivalent with drawing dichotomies of N points in the (l − 1)dimensional space.

In the sequel, we will show by induction that

k k O(N, l) = O(N − k, l) + O(N − k, l − 1) + . . .

0 1

k + O(N − k, l − k) for k = 1, 2, . . . , N − 1.

(2) k The above is true for k = 1. Indeed

1 1 = 1 = .

0 1 Then, assume it is true for some k. We will show that it is also true for k + 1. We have that O(N − k, l) = O(N − k − 1, l) + O(N − k − 1, l − 1).

2

Thus

k O(N, l) = {O(N − k − 1, l) + O(N − k − 1, l − 1)} 0

k + {O(N − k − 1, l − 1) + O(N − k − 1, l − 2)} + 1

k ··· + {O(N − k − 1, l − k) + O(N − k − 1, l − k − 1)}. k However,

k r

+

k r+1

=

k+1 r+1

.

Indeed,

k r

Also

k r

k! k! ,= , r!(k − r)! r!(k − r − 1)!(k − r)

k! k = , r+1 r!(r + 1)(k − r − 1)!

k r+1

+

k 0

=

(r + 1)k! + (k − r)k! = r!(r + 1)(k − r − 1)!(k − r)

(k + 1)k! k+1 = . r+1 (r + 1)!(k + 1 − r − 1)!

=

=

k+1 0

and

k k

=

k+1 k+1

.

Hence,

O(N, l) =

k+1 0

k+1 O(N − k − 1, l) + O(N − k − 1, l − 1) + 1

k+1 ... + O(N − k − 1, l − k − 1). k+1

Thus, (2) is true. By convention O(N, l) = 0 for l < 0. Then, setting in (2) k = N − 2 and taking into account that O(1, l) = 2,

l ≥ 0,

we obtain O(N, l) = 2

l X N −1 i=0

i

.

Note that the convention O(1, 0) = 2 is in line with the recursion in (1).

Indeed, for l = 1 and N = 1 we have O(2, 1) = O(1, 1) + O(1, 0).

3

However, O(2, 1) = 4, since 2 points in a line can be separated by a point in four possible groupings. Also O(1, 1) = 2, since a point can belong to either of two classes. Then O(1, 0) = 2 is in agreement with the recursion.

11.2. Show that if N = 2(l + 1), the number of linear dichotomies in Cover’s theorem is equal to 22l+1 .

Hint: Use the identity j X j = 2j , i i=1 and recall that

2n + 1 n−i+1

=

2n + 1 . n+i

Solution: From the theory, we have that for N = 2l + 2, 2l+1 X

2l + 1 i

i=0

= 22l+1 .

Then, O(2l + 2, l) = 2

l X 2l + 1 i=0

i

# " l 2l+1 X 2l + 1 1 X 2l + 1 + = 2 i i 2 i=0

i=l+1

= 2

1 2l+1 2 = 2N −1 .

2

11.3. Show that the reproducing kernel is a positive definite one.

Solution: Consider N > 0, the real numbers a1 , . . . , aN , and the elements x1 , . . . , xN ∈ X . Then N N X X

an am κ(xn , xm )

=

n=1 m=1

N N X X

an am hκ(·, xn ), κ(·, xm )i

n=1 m=1

=

=

N X

* an

κ(·, xn ),

am κ(·, xm )

n=1

m=1

*N X

N X

an κ(·, xn ),

n=1

=

+

N X

N X

an κ(·, xn )

+ am κ(·, xm )

m=1 2

≥ 0.

n=1

11.4. Show that if κ(·, ·) is the reproducing kernel in a RKHS, H, then H = span{κ(·, x), x ∈ X }.

4

Solution: We will show that the only function in H which is orthogonal to A = span{κ(·, x), x ∈ X } is the zero function. Let f ∈ H be a function that is orthogonal to A. Then f (x) = hf, κ(·, x)i = 0, ∀x ∈ X, which holds true for f = 0. Thus A⊥ = A⊥ = {0}.

Let now f ∈ H such that f ∈ / Ā. By its definition, Ā is closed and also is convex, being a subspace. Note also that Ā ⊆ H. This is because, H by its definition, is complete and hence it contains all the limit points of sequences in H. Also, by its definition, any κ(·, x) ∈ H, x ∈ X. Thus according to what we have said in Chapter 8, there is a g ∈ Ā which is the projection of f on Ā. Also, since Ā is a closed subspace, we know from (8.20) H = Ā + Ā⊥ .

Hence f − g ∈ Ā⊥ , which means that f − g is orthogonal to any element in Ā, hence in A. This leads to f − g = 0.

In other words, if there exists such an f , it coincides with its projection on Ā, hence f ∈ Ā.

This contradicts our assumption that f ∈ / Ā.

11.5. Show the Cauchy-Schwarz inequality for kernels, that is, kκ(x, y)k2 ≤ κ(x, x)κ(y, y).

Solution: Let φ(x) := κ(·, x), and φ(y) := κ(·, y).

Then, khφ(x), φ(y)ik ≤ kφ(x)k · kφ(y)k, or κ(x, y) ≤

p p κ(x, x) κ(y, y).

5

11.6. Show that if κi (·, ·) : X × X 7−→ R, i = 1, 2 are kernels then: • κ(x, y) = κ1 (x, y) + κ2 (x, y) is also a kernel.

• aκ(x, y), a > 0 is also a kernel.

• κ(x, y) = κ1 (x, y)κ2 (x, y) is also a kernel. .

Solution: Recall that it suffices to show that the respective kernel matrices of the new constructed functions are positive semidefinite.

• For the addition, the i, j element of K is given by [K]i,j = κ1 (xi , xj ) + κ2 (xi , xj ) = [K1 ]i,j + [K2 ]i,j or K = K1 + K2 .

Since K1 and K2 are positive semidefinite, so is their sum.

• For the case κ(x, y) = aκ(x, y), a > 0 it is trivial that K is also positive semidefinite for a > 0.

• The proof for the product is slightly more “clever’’. Note that the kernel matrix consists of respective products, e.g.,

κ (x , y )κ (x , y ) κ1 (x1 , y2 )κ2 (x1 , y2 ) K= 1 1 1 2 1 1 κ1 (x2 , y1 )κ2 (x2 , y1 ) κ1 (x2 , y2 )κ2 (x2 , y2 ) for N = 2. Observe that K results as a principal submatrix of Kronecker product K1 ⊗ K2 , which is the N 2 × N 2 matrix, that results if each element of K1 is replaced by K2 multiplied by the respective element. K results from K1 ⊗ K2 by keeping a set of columns and the same set of rows. Verify it for the case N = 2. Hence 2 ∀a ∈ RN , ∃b ∈ RN : aT Ka = bT (K1 ⊗ K2 )b ≥ 0.

The latter inequality is true since we know from linear algebra that if K1 , K2 are positive semidefinite, so their Kronecker product is.

11.7. Derive Equation (11.25).

Solution: The starting point is J(θ) =

N X n=1

yn −

N X m=1

!2 θm κ(xn , xm )

+ hf, f i.

6

Substitute in the regularizer the form of f to get *N + N X X hf, f i = θn κ(·, xn ), θm κ(·, xm ) n=1

=

N X

m=1

* θn

κ(·, xn ),

n=1

=

N X

=

N X n=1 T

+ θm κ(·, xm )

m=1

θn

n=1

=

N X

N X

θm hκ(·, xn ), κ(·, xm )i

m=1

θn

N X

θm κ(xn , xm )

m=1 T

θ Kθ = θ KT θ.

The first term can be written as N X

yn − θ T k(xn )

2

= ky − Kθk2 ,

n=1

where k(·) := [κ(·, x1 ) · · · κ(·, xN )]T , with K being the kernel matrix. This proves the claim.

11.8. Show that the solution for the parameters, θ̂, for the kernel ridge regression, if a bias term, b, is present, is given by

y K + CI 1 θ = , b yT 1 1T K N where 1 is the vector with all its elements being equal to one. Invertibility of the kernel matrix has been assumed.

Solution: In this case, the unknown coefficients are estimated by minimizing J(θ, b), where !

N N X X J(θ, b) = yn − θm κ(xn , xm ) − b + Chf, f i.

(3) n=1

m=1

Equation (3) can be recast in a more compact form as follows: J(θ, b) = (y − Kθ − b1)T (y − Kθ − b1) + Cθ T Kθ.

(4)

As J(θ, b) is a strictly convex function, the unique minimum can be obtained by solving ∂J ∂J = 0, = 0. ∂θ ∂b

7

Hence, we obtain the system of equations (K2 + CK)θ + bK1 = Ky, 1T Kθ + N b = y T 1.

Assuming that K is invertible, we obtain the result.

11.9. Derive Equation (11.56).

Solution: The dual representation of the Lagrangian in (11.55) can be written as, 1 T 1 L(λ) = y T λ − λ Kλ − λT λ, 4C 4 where K is the kernel matrix and λ = [λ1 , . . . , λN ]T .

Taking the gradient with respect to λ and equating to zero we obtain −1

λ = 2C (K + CI)

y.

11.10. Derive the dual cost function associated with the linear -insensitive loss function.

Solution: From the text, the Lagrangian is given by

L(θ, θ0 , ξ̃, ξ, λ, µ)

=

+

1 kθk2 + C 2 N X

N X

ξn +

n=1

N X

! ξ˜n

n=1

λ̃n (yn − θ T xn − θ0 − − ξ˜n )

n=1

+

N X

λn (θ T xn + θ0 − yn − − ξn )

n=1

−

N X n=1

µ̃n ξ˜n −

N X

µn ξn .

n=1

Let us plug in place of θ the corresponding KKT condition, i.e.,

θ=

N X

(λ̃n − λn )xn .

n=1

8

Then we obtain N

L =

N

1XX (λ̃n − λn )(λ̃m − λm )xTn xm + C 2 n=1 m=1 −

N X n=1

+

N X

λ̃n

X

(λ̃n − λn )xTm xn +

m=1

λ̃n yn −

n=1

+

N X n=1

−

N X

n=1 N X

λ̃n θ0 −

n=1

λn θ0 −

N X

n=1

N X

N X

λ̃n −

n=1

λn yn −

n=1

µ̃n ξ˜n −

N X

N X

λn

X

N X

ξn +

n=1

N X

! ξ˜n

n=1

(λ̃m − λm )xTm xn

m=1 N X

λ̃n ξ˜n

n=1

λn −

n=1

N X

λn ξn

n=1

µn ξn .

n=1

Taking into account, from the KKT conditions given in the text, that C − λ̃n − µ̃n = C − λn − µn = 0, n = 1, 2, . . . , N, and that N X

λ̃n =

n=1

N X

λn ,

n=1

we obtain L =

N X

(λ̃n − λn )yn −

n=1 N

+

N X

(λ̃n + λn )

n=1 N

1XX (λ̃n − λn )(λ̃m − λm )xTn xm .

2 n=1 m=1

11.11. Derive the dual cost function for the separable class SVM formulation.

Solution: The Lagrangian is given by L(θ, θ0 , λ) =

N X

1 kθk2 − λn yn (θ T xn + θ0 ) − 1 .

2 n=1

From the KKT condition, given in the text, we have θ=

N X n=1

λn yn xn .

9

Plugging it into the Lagrangian we obtain L(λ)

=

−

N N N X 1XX λn λm yn ym xTn xm − λn yn 2 n=1 m=1 n=1 N X

λn yn θ0 +

n=1

N X

! λm ym xTm

xn

m=1

λn .

n=1

Taking into account from the text that L(λ) =

N X

N X n=1

N

λn −

PN

n=1 λn yn = 0, we finally get

N

1XX λn λm yn ym xTn xm .

2 n=1 m=1

11.12. Derive the kernel approximation in Eq. (11.91) Solution: The formula comes as a direct manipulation by plugging in the definition Euler’s formula.

11.13. Derive the subgradient for the Huber loss function.

Solution: The Huber loss function is given by

2 if |y − z| > , |y − z| − 2 , L(y, z) = 1 2 |y − z| , if |y − z| ≤ , 2 where z = f (x). Hence, for |y − z| > ∂ L(y, z) = − sgn(y − z). ∂z For |y − z| < ∂ L(y, z) = −(y − z). ∂z At the discontinuities either of the two can be a subgradient, and we will choose the latter. Thus, ( − sgn(y − z), |y − z| > ∂L(y, z) = ∂z −(y − z), |y − z| ≤ .

1

Solutions To Problems of Chapter 12 12.1. Show that if p(z) = N (z|µz , Σz ), and p(t|z) = N (t|Az, Σt|z ), then −1 −1 E[z|t] = (Σz−1 + AT Σt|z A)−1 (AT Σt|z t + Σz−1 µz )

Solution: We have shown in the Appendix of the chapter that, −1 −1 E[z|t] := µz|t = µz + (Σz−1 + AT Σt|z A)−1 AT Σt|z (t − Aµz )

(1)

or −1 −1 µz|t =(Σz−1 + AT Σt|z A)−1 AT Σt|z t+

−1 −1 I − (Σz−1 + AT Σt|z A)−1 AT Σt|z A µz .

(2)

Recall the matrix inversion identity (P −1 + B T R−1 B)−1 B T R−1 = P B T (R + BP B T )−1 .

(3)

Then the following is true −1 −1 (Σz−1 + AT Σt|z A)−1 AT Σt|z = Σz AT (Σt|z + AΣz AT )−1 .

(4)

Hence −1 −1 I − (Σz−1 + AT Σt|z A)−1 AT Σt|z A=

I − Σz AT (Σt|z + AΣz AT )−1 A = (Σz − Σz AT (Σt|z + AΣz AT )−1 AΣz )Σz−1 Recall now the other matrix inversion identity (A + BD−1 C)−1 = A−1 − A−1 B(D + CA−1 B)−1 CA−1 .

Then we obtain (Σz − Σz AT (Σt|z + AΣz AT )−1 AΣz )Σz−1 = −1 (Σz−1 + AT Σt|z A)−1 Σz−1

Combining (2) and (5) proves the claim.

(5)

2

12.2. Let x ∈ Rl be a random vector following the normal N (x|µ, Σ). Consider xn , n = 1, 2, . . . , N , to be i.i.d. observations. If the prior for µ follows N (µ|µ0 , Σ0 ), show that the posterior p(µ|x1 , . . . , xN ) is normal N (µ|µ̃, Σ̃) with Σ̃ −1 = Σ0−1 + N Σ −1 , and µ̃ = Σ̃(Σ0−1 µ0 + N Σ −1 x̄) where x̄ = N1

PN

n=1 xn .

Solution: We have that p(µ|x1 , . . . , xN ) ∝ p(µ|µ0 , Q−1 0 )

N Y

p(xn |µ; Q−1 ),

(6)

n=1

where Q0 := Σ0−1 and Q := Σ −1 . In turn, (6) is written as !

N 1X T p(µ|x1 , . . . , xN ) ∝ − (xn − µ) Q(xn − µ) × N l exp 2 n=1 (2π) 2

|Q0 |1/2 1 T (µ − µ ) Q (µ − µ ) . exp − 0 0 0 l 2 (2π) 2 |Q|N/2

Keeping only the terms that depend on µ, we get

1 T T p(µ|x1 , . . . , xN ) ∝ exp − µ Q0 µ + µ Q0 µ0 × 2

1 T T exp − µ (N Q)µ + µ (N Q)x̄ , 2 or 1 p(µ|x1 , . . . , xN ) ∝ exp − µT (Q0 + N Q)µ 2

T + µ (Q0 µ0 + N Qx̄) .

Note that the exponent in the above is of a quadratic form, i.e., µT Q̃µ + µT p̃, which proves the claim.

In words, the posterior mean is a weighted average (by the respective precision matrices) of the prior mean and the sample mean. Moreover the posterior precision matrix is the sum of the prior precision and the conditional precision matrix (weighted by N ). Observe that as N −→ ∞ then 1 Σ̃ −→ Σ, µ̃ −→ x̄.

N

3

12.3. If X is the set of observed variables and X l the set of the corresponding latent ones, show that

∂ ln p(X ; ξ) ∂ ln p(X , X l ; ξ) =E , ∂ξ ∂ξ where E[·] is with respect to p(X l |X ; ξ) and ξ is an unknown vector parameter. Note that if one fixes the value of ξ in p(X l |X ; ξ), then one has obtained the M-step of the EM algorithm.

Solution: we have that ∂ ln p(X ; ξ) ∂ξ

= =

R +∞ ∂ ln −∞ p(X , X l ; ξ)dX l ∂ξ 1 R +∞ −∞ Z +∞

= −∞

Z +∞ = −∞

Z +∞

R +∞ ∂ −∞ p(X , X l ; ξ)dX l

p(X , X l ; ξ)dX l

∂ξ

1 ∂p(X , X l ; ξ) dX l p(X ; ξ) ∂ξ p(X , X l ; ξ) ∂ ln p(X , X l ; ξ) dX l p(X ; ξ) ∂ξ

∂ ln p(X , X l ; ξ) dX l ∂ξ −∞

∂ ln p(X , X l ; ξ) = E . ∂ξ =

p(X l |X ; ξ)

(7)

12.4. Show equation (12.42).

Solution: By the definition of Eq. (12.40), in case the hyperparameters vector is considered to be random, we have,

Q(ξ, ξ (j) ) = E ln p X , X l , ξ which by the product rule of probabilities is written as

Q(ξ, ξ (j) ) = E ln p X , X l |ξ p(ξ) or

Q(ξ, ξ (j) ) = E ln p X , X l |ξ + ln p(ξ) which gives the (12.42).

12.5. Let y ∈ RN , θ ∈ Rl and Φ a matrix of appropriate dimensions. Derive the expected value of ky − Φθk2 with respect to θ, given E[θ] and the corresponding covariance matrix Σθ .

4

Solution: Let φ = y − Φθ. By definition we have Σφ = E[(φ − E[φ])(φ − E[upφ])T ] = E[φφT ] − E[φ] E[φT ],

(8)

µφ := E[φ] = y − Φµθ .

(9)

where Elaborating on (8) we get Σφ = E[(y − Φθ)(y − Φθ)T ] − (y − Φµθ )(y − Φµθ )T = yy T − Φµθ y T − yµTθ ΦT + Φ E[θθT ]ΦT − yy T + Φµθ y T + yµTθ ΦT − Φµθ µTθ ΦT = Φ E[(θ − µθ )(θ − µθ )T ]ΦT = ΦΣθ ΦT .

(10)

Hence, E[(y − Φθ)T (y − Φθ)] := E[φT φ] = E[trace{φφT }] = trace{µφ µTφ } + trace{Σφ } = kµφ k2 + trace{Σφ } = ky − Φµθ k2 + trace{Σφ }.

12.6. Derive recursions (12.60)-(12.62).

Solution: Recall from Eq. (12.59) of the text that (j)

Q(Ξ, P ; Ξ

,P

(j)

)

N X

=

h i E ln p(xn |kn ; ξkn )Pkn

n=1 K N X X

l P (k|xn ; Ξ(j) , P (j) ) ln Pk − ln σk2 2 n=1 k=1

1 − 2 kxn − µk k2 + C. (11) 2σk

:=

• Iteration for the mean value: The above can be rewritten as Q(Ξ, P ; Ξ(j) , P (j) = −

N K 1 XX P (k|xn ; Ξ(j) , P (j) )||xn −µk ||2 +constant 2σk2 n=1 k=1

where constant included all terms which are independent of µk . Taking the derivative with respect to a specific k, we obtain N X n=1

γkn µk =

N X n=1

γkn xn ,

5

which leads to the recursion.

• Recursion for the variance: Eq. (12.59) is now written as (j)

Q(Ξ, P ; Ξ

,P

(j)

N X K X

=−

(j)

P (k|xn ; Ξ

,P

(j)

)

l ln σk2 +

n=1 k=1

1 2 ||xn − µk || +constant.

2σk2

Taking the derivative with respect to σk2 , for a specific values k, obtain N X

γkn

n=1

N X l 1 = γkn 4 ||xn − µk ||2 , 2 σk σk n=1

which gives the recursion.

• Recursion for the probabilities: Eq. (12.59) is now written as Q(Ξ, P ; Ξ(j) , P (j) =

K N X X

P (k|xn ; Ξ(j) , P (j) ) ln Pk + constant.

n=1 k=1

Also since we refer to probabilities, they have to sum up to one, K X

Pk = 1.

k−1

Thus, we have an constrained optimization task. Using Lagrange multipliers, the following Lagrangian results !

K K N X X X (j) (j) P (k|xn ; Ξ , P ) ln Pk + λ Pk − 1 .

L(λ, Pk ) = n=1 k=1

k=1

Taking the derivative of the Lagrangian with respect to Pk , for a specific value of k, we get N

Pk =

1X γkn . λ n=1

Substituting in the constraint equation, it turns out that λ = N and finally the recursion results.

12.7. Show that the Kullback-Leibler divergence KL(p k q) is a nonnegative quantity.

Hint: Recall that ln(·) is a concave function and use Jensen’s inequality, that is, Z Z f

g(x)p(x)dx

≤

f (g(x))p(x)dx,

where p(x) is a pdf and f is a convex function

6

Solution: By definition of the KL(q k p) divergence and the fact that − ln(·) is a convex function, we have Z q(x) KL(q k p) = − p(x) ln dx p(x) Z Z q(x) p(x)dx = − ln q(x)dx = 0. ≥ − ln p(x) 12.8. Prove that the binomial and beta distributions are conjugate pairs with respect to the mean value.

Solution: The binomial distribution

N P (x|µ) = µx (1 − µ)N −x , x is a special case of the multinomial one, when only two events are possible.

On the other hand, the beta distribution p(µ|a, b) =

Γ(a + b) a−1 µ (1 − µ)b−1 Γ(a)Γ(b)

is a special case of the Dirichlet distribution when only two events are possible. Hence, since the multinomial and Dirichlet are conjugate pairs, the binomial and beta are conjugate pairs, too.

12.9. Show that the normalizing constant C in the Dirichlet pdf Dir(x|a) = C

K Y

xakk −1 ,

k=1

K X

xk = 1,

k=1

is given by C=

Γ(a1 + a2 + . . . + aK ) Γ(a1 )Γ(a2 ) . . . Γ(aK )

Hint: Use the property Γ(a + 1) = aΓ(a). a) Use induction. Since the proposition is true for k = 2 (beta distribution), assume that it is true for k = K − 1, and prove that it will be true for k = K. PK b) Note that due to the constraint k=1 xk = 1, only K −1 of the variables are independent. So, basically the Dirichlet pdf implies that p(x1 , x2 , . . . , xK−1 ) = C

K−1 Y k=1

xakk −1

1−

K−1 X k=1

!aK −1 xk

.

7

Solution: We will integrate xK−1 out. Then we get p(x1 , x2 , . . . , xK−2 ) = P K−2 Y a −1 Z 1− K−2 k=1 xk aK−1 −1 k C xk xK−1

1−

0

k=1

K−1 X

!aK −1 xk

dxK−1 .

(12)

k=1

The upper limit in the integration takes into account that xk are probabilities (sum to one); hence, given the values x1 , . . . , xK−2 , the limits in PK−2 which xK−1 can lie range from zero to 1 − k=1 xk . Set !

K−2 X xK−1 = t 1 − xk .

(13) k=1

Hence, 1−

dxK−1 =

K−2 X

! xk

dt,

(14)

k=1

and 1−

K−1 X k=1

K−2

X 1−t = (1 − t) 1 − xk xk = xK−1 t

! .

(15)

k=1

From (12)-(15), we obtain p(x1 , x2 , . . . , xK−2 ) = C

K−2 Y k=1 Z 1

xakk −1

1−

K−2 X

!(aK−1 +aK −1) xk

k=1

taK−1 −1 (1 − t)ak −1 dt.

(16)

0

However, recalling the result from beta pdf in Problem 12.8 the integral K−1 )Γ(aK ) is equal to Γ(a Γ(aK−1 +aK ) . Hence, (16) can be rewritten as p(x1 , x2 , . . . , xK−2 ) =C

K−2 Y

xakk −1

1−

k=1

K−2 X

!(aK−1 +aK −1) xk

k=1

Γ(aK−1 )Γ(aK ) . Γ(aK−1 + aK )

(17)

Looking at (17) it is readily seen that p(x1 , x2 , . . . , xK−2 ) is a Dirichlet pdf, hence by assumption C

Γ(aK−1 )Γ(aK ) Γ(a1 + a2 + . . . + aK−2 + aK−1 + aK ) = , Γ(aK−1 + aK ) Γ(α1 ) . . . Γ(αK−2 )Γ(aK−1 + aK )

or C=

Γ(a1 + a2 + . . . + aK ) . Γ(a1 ) . . . Γ(aK )

8

12.10. Show that N (x|µ, Σ) for known Σ is of an exponential form and that its conjugate prior is also Gaussian.

Solution: p(x|µ)

= =

1 1 T −1 exp − (x − µ) Σ (x − µ) 2 (2π)l/2 |Σ|1/2

1 T −1 1 1 T −1 exp − x Σ x exp − µ Σ µ × 2 2 (2π)l/2 |Σ|1/2 exp(µT Σ −1 x).

Hence φ(µ)

= µ

u(x)

= Σ −1 x

1 1 , = exp − xT Σ −1 x 2 (2π)l/2 |Σ|1/2

f (x)

and from the respective definition and some trivial algebraic manipulation

1 T −1 g(µ) = exp − µ Σ µ .

2 For the conjugate prior, we have

λ T −1 p(µ|λ, v) = exp − µ Σ µ h(λ, v) exp(µT v) 2 !

−1 1 T 1 T = h(λ, v) exp − µ Σ µ+µ v .

2 λ This is of an exponential form and quadratic with respect to µ, hence Gaussian.

12.11. Show that the conjugate prior of the multivariate Gaussian with respect to the precision matrix, Q, is a Wishart distribution.

Solution: We have that,

|Q|1/2 1 T p(x|Q) = exp − (x − µ) Q(x − µ) 2 (2π)l/2

|Q|1/2 1 T = exp − Trace{Q(x − µ)(x − µ) } , 2 (2π)l/2

(18)

where we used the identity from matrix theory trace{AB} = trace{BA}.

We will now bring (18) into the standard form of the exponential family,

|Q|1/2 1 T T T p(x|Q) = exp − [q , q , . . . , q ]u(x) , l 2 1 2 (2π)l/2

9

where qiT , i = 1, 2, . . . , l, are the row vectors of Q, i.e.,  T  q1  q2T    Q :=  .   ..  qlT and u(x) comprises the columns of (x − µ)(x − µ)T , stacked in sequence one below the other. Hence p(x|Q) is of an exponential form. Its conjugate prior will be given by

λ 1 T T p(Q; λ, v) = h(λ, v)|Q| 2 exp [q1 , q2 , . . . , qlT ]v .

(19) 2 Eq. (19) can be brought in a slightly different form, which will facilitate the computation of h(λ, v). Consider v as comprising the column vectors of a matrix W −1 , and set λ = ν − l − 1 then (19) is written as

ν−l−1 1 −1 2 exp − trace{W Q} , p(Q; ν, W ) = h(ν, W )|Q| 2 which is a Wishart distribution and the normalizing constant then necessarily becomes  

−1 l l(l−1) Y νl ν ν + 1 − j  . Γ h(ν, W ) = |W |− 2 2 2 π 4 2 j=1 12.12. Show that the conjugate prior of the univariate Gaussian N (x|µ, σ 2 ) with respect to the mean and the precision β = σ12 , is the Gaussian-gamma product

v λ + 1 v1 v22 2 −1 , (λβ) Gamma β , − p(µ, β; λ, v) = N µ λ 2 2 2λ where v := [v1 , v2 ]T .

Solution: We have shown in the respective section in the book that

λ −λβµ2 β v1 p(µ, β; λ, v) = h(λ, v)β 2 exp exp [− , βµ] , v2 2 2 which leads to

λβ v2 β v2 β p(µ, β; λ, v) = h(λ, v) exp − µ2 + v2 βµ − 2 + 2 × 2 2λ 2λ λ v1 β 2 exp(− β) 2

λ v2 2 v1 v22 λβ 2 µ− β exp −β − = h(λ, v) exp − 2 λ 2 2λ

λ−1 1 λβ v2 2 v1 v22 = h(λ, v)β 2 exp − µ− β 2 exp −β − .

2 λ 2 2λ

10

The last formula is of the form p(µ, β) = p(µ|β)p(β), which readily suggests that h(λ, v) should be such that to lead to

p(µ, β; λ, v) = N

µ

v2 β −1 , λ λ

λ + 1 v1 v2 Gamma β , − 2 .

2 2 2λ

12.13. Show that the multivariate Gaussian N (x|µ, Q−1 ) has as a conjugate prior, with respect to the mean and the precision matrix, Q, the GaussianWishart product.

Solution: Let

1 T 1 T |Q|1/2 T exp − x Qx − µ Qµ + µ Qx p(x|µ, Q) = 2 2 (2π)l/2

1/2 1 T |Q| 1 T T µ Qµ exp − x Qx + µ Qx . = exp − 2 2 (2π)l/2 Let also

p(µ, Q; µ0 , λ, W, ν) = N µ|µ0 , (λQ)−1 W (Q|W, ν) .

Then, the posterior is of the form

|Q|1/2 1 T 1 T T p(µ, Q|x) ∝ exp − µ Qµ exp − x Qx + µ Qx × 2 2 (2π)l/2

1/2 1 T |λQ| 1 T T µ (λQ)µ − µ (λQ)µ + µ (λQ)µ exp − 0 0 × 2 2 0 (2π)l/2

ν−λ−1 1 h|Q| 2 exp − trace{W −1 Q} , 2 which after some trivial manipulations becomes

1 p(µ, Q|x) ∝ exp − µT (λ + 1)Qµ + µT (Qx + λQµ0 ) × 2 1 1 exp − trace{W −1 Q} − trace{xxT Q}− 2 2 ν−l 1 T trace{λµ0 µ0 Q} |Q| 2 , 2 which is rewritten as

1 p(µ, Q|x) ∝ exp − (µ − µ̃0 )T (λ̃Q)(µ − µ̃0 ) × 2

ν̃−l−1 1 |Q| 2 exp − trace{W̃ −1 Q} , 2

11

where ν̃ = ν + 1 λ̃ = λ + 1 µ̃0 = (λ̃Q)−1 (Qx + λQµ0 ) W̃ −1 = W −1 + xxT + λµ0 µT0 .

Since p(µ, Q|x) = p(µ|Q, x)p(Q|x) must integrate to one, then the missing constants must be such that

p(µ, Q|x) = N µ|µ̃0 , (λ̃Q)−1 W Q|W̃ , ν̃ .

12.14. Show that the distribution P (x|µ) = µx (1 − µ)1−x , x ∈ {0, 1}, is of an exponential form and derive its conjugate prior with respect to µ.

Solution: We have

P (x|µ) = exp ln(µx (1 − µ)1−x ) = exp (x ln µ + (1 − x) ln(1 − µ)) = exp (ln(1 − µ)) exp(x(ln µ − ln(1 − µ))

µ , = (1 − µ) exp x ln 1−µ from which we obtain g(µ) = 1 − µ, φ(µ) = ln

µ , u(x) = x.

1−µ

Hence, the conjugate prior will be given by

p(µ|λ, v) = h(1 − µ)λ exp v ln = h(1 − µ)λ

µ (1 − µ)

µv = h(1 − µ)λ−v µv , (1 − µ)v

which is of a beta distribution form Beta(µ|a, b) =

Γ(a + b) a−1 µ (1 − µ)b−1 , Γ(a)Γ(b)

with a = v + 1, b = λ − v + 1. Hence h=

Γ(λ + 2) . Γ(v + 1)Γ(λ − v + 1)

12

12.15. Show that estimating an unknown pdf by maximizing the respective entropy, subject to a set of empirical expectations, results in a pdf that belongs to the exponential family.

Solution: The problem is cast as Z maximize w.r.t p(x)

p(x) ln p(x)dx, Z E[ui (x)] = p(x)ui (x)dx, i ∈ I. Ax

s.t.

Ax

Using Lagrange multipliers, the Lagrangian becomes Z L(p(x), θ) = p(x) ln p(x)dx Ax

Z X Z − θi p(x)ui (x)dx − µ̂i − θ0 Ax

i∈I

p(x)dx − 1 .

Ax

Taking the variational derivative with respect to p(x) and equating to zero we get Z Z ∂L = ln p(x)dx + dx ∂p(x) Ax Ax Z Z X − θi ui (x)dx − θ0 dx = i∈I

Ax

Ax

!

Z ln p(x) −

X

Ax

θi ui (x) − (θ0 − 1) dx = 0,

i∈I

which results in ln p(x) =

X

θi ui (x) + (θ0 − 1)

i∈I

and finally ! p(x) = C exp

X i∈I

θi ui (x) .

1

Solutions To Problems of Chapter 13 13.1. Show Eq. (13.5).

Solution: The functional F(q) is defined as Z p(X , X l , θ) F(q) = q(X l , θ) ln dX l dθ. q(X l , θ)

(1)

Plugging in the mean field approximation, q(X l , θ) = qX l (X l )qθ (θ), the previous equation becomes Z p(X , X l , θ) dX l dθ, (2) F(q) = qX l (X l )qθ (θ) ln qX l (X l )qθ (θ) or Z

qθ (θ) ln p(X , X l , θ)dθ dX l qX l (X l ) Z

Z qθ (θ)dθ dX l − qX l (X l ) ln qX l (X l ) Z

Z − qθ (θ) ln qθ (θ) qX l (X l )dX l dθ, Z F(q)

=

(3)

which then leads trivially to the claim.

13.2. Show equation (13.38).

Solution: From Eq. (13.37) in the text we have " K−1 # K−1 X X 1 1 ln qα(j+1) (α) = Eq(j+1) ln αk − αk θ2k θ 2 2 k=0

−

K−1 X

ln Γ(a) + a

k=0

−b

K−1 X

(4)

k=0

K−1 X

ln b + (a − 1)

k=0

K−1 X

ln αk

(5)

k=0

αk + constant

(6)

k=0

=

K−1

K−1 X 1 1 X a−1+ ln αk − b + Eq(j+1) [θ2k ] αk (7) 2 2 θ k=0

k=0

− K ln Γ(a) + aK ln b + constant, or qα(j+1) (α) ∝

K−1 Y k=0

1 (a−1+ 12 ) αk exp − b + Eq(j+1) [θ2k ] αk , 2 θ

(8)

2

or qα(j+1) (α) =

K−1 Y k=0

1 1 Gamma αk ; a + , b + Eq(j+1) [θ2k ] , 2 2 θ

which proves the claim.

13.3. Show equations (13.43)-(13.45).

Solution: From the text we have (j+1)

ln qβ

(β) = Eq(j+1) q(j+1) [ln p(y|θ, β) + ln p(β)] + constant α θ

N 1 2 = Eq(j+1) ln β − βky − Φθk − ln Γ(c) θ 2 2 + c ln b + (c − 1) ln β − dβ

N 1 2 = c−1+ ln β − d + Eq(j+1) ky − Φθk β 2 2 θ + constant,

which then results to (j+1)

qβ

˜ (β) ∝ β c̃ exp(−dβ),

with N 2 1 d˜ = d + Eq(j+1) [ky − Φθk2 ].

2 θ c̃ = c +

Thus, 1 ˜c̃ (c̃−1) ˜ d β exp(−dβ). Γ(c̃) The proportionality constant has to be equal to (j+1)

qβ

(β) =

d˜c̃ Γ(c̃) (j+1)

in order qβ

(β) to integrate to one.

13.4. Show that if

1 , x then the random variable z = ln x follows a uniform distribution. p(x) ∝

Solution: We know that pz (z) ∝ px (x(z))

dx = exp(−z) exp(z) = 1, dz

where we used the fact that x = exp(z).

(9)

3

13.5. Derive the lower bound after convergence of the variational Bayesian EM for the linear regression task which is modeled as in Section 13.3.

Solution: The lower bound after convergence to q̃θ (θ), q̃α (α), q̃β (β), which are defined by µ̃θ , Σ̃θ for the Gaussian q̃θ , by (ã, b̃i ), i = 1, 2, . . . , l for the gamma q̃α (α) and c̃, d˜ for the gamma q̃β (β), will be F(q̃θ ,q̃α , q̃β ) = Eθ [Eα [Eβ [ln p(y, θ, α, β)]]] − Eθ [ln qθ (θ)] − Eα [ln qα (α)] − Eβ [ln qβ (β)] .

We have that ln p(y, θ, α, β) = ln p(y|θ, β) + ln p(θ|α) + ln p(α) + ln p(β), where N

(a) ln p(y|θ, β) = ln β 2 − N2 ln(2π) − β2 ky − Φθk2 .

PK−1 PK−1 1 2 (b) ln p(θ|α) = 21 k=0 ln αk − K k=0 αk θk .

2 ln(2π) − 2 PK−1 PK−1 (c) ln p(α) = −K ln Γ(a) + Ka ln b + (a − 1) k=0 ln αk − b k=0 αk . (d) ln p(β) = − ln Γ(c) + c ln d + (c − 1) ln β − dβ.

Using identities from the Appendix of the chapter and the independence among q̃θ , q̃α , q̃β , we get: (a) N 1 N Eβ [ln β] − ln(2π) − Eβ [β] Eθ [ky − Φθk2 ] 2 2 2 n o N ˜ − N ln(2π) − 1 c̃ y − Φµ̃θ k2 + trace{ΦΣ̃θ ΦT } , = [ψ(c̃) − ln d] 2 2 2 d˜ where the results of Problem 12.13 have been used.

A1 := Eθ Eβ [ln p(y|θ, β)] =

(b) K−1

A2 := Eθ Eα [ln p(θ|α)] =

K−1

K 1 X 1 X Eα [ln αk ] − ln(2π) − Eα [αk ] Eθ [θ2k ] 2 2 2 k=0

=

1 2

K−1 X

[ψ(ã) − ln b̃k ] −

k=0

k=0

K 1 ln(2π) − 2 2

K−1 X

ã

k=0

b̃k

Σ̃θ + µ̃θ µ̃Tθ

. kk

(c) A3 := Eα [ln p(α)] = −K ln Γ(a) − Ka ln b − (a − 1)

K−1 X k=0

= K ln Γ(a) − Ka ln b − (a − 1)

K−1 X

[ψ(ã) − ln b̃k ]

k=0

−b

K−1 X

ã . b̃ k=0 k

Eα [ln αk ] − b

K−1 X k=0

Eα [αk ]

4

(d) A4 := Eβ [ln p(β)] = − ln Γ(c) + c ln d + (c − 1) Eβ [ln β] − d Eβ [β] ˜ − d c̃ . = − ln Γ(c) + c ln d + (c − 1)(ψ(c̃) − ln d) d˜ In the sequel the respective entropies have to be computed (a) h 1 K B1 := Eθ [ln q̃θ (θ)] = Eθ − ln |Σ̃θ | − ln(2π)− 2 2 i 1 (θ − µ̃θ )T Σ̃θ−1 (θ − µ̃θ ) 2 K 1 1 1 ln(2π) − Eθ [θT Σ̃θ−1 θ] − µ̃Tθ Σ̃θ−1 µ̃θ = − ln |Σ̃θ | − 2 2 2 2 + µ̃Tθ Σ̃θ−1 E[θ] K 1 1 ln(2π) + µ̃Tθ Σ̃θ−1 µ̃θ = − ln |Σ̃θ | − 2 2 2 h i 1 T −1 − Eθ trace{Σ̃θ θθ } 2 1 K 1 = − ln |Σ̃θ | − ln(2π) + µ̃Tθ Σ̃θ−1 µ̃θ 2 2 2 1 −1 − trace{Σ̃θ (Σ̃θ + µ̃θ µ̃Tθ )} 2 K 1 1 ln(2π) + µ̃Tθ Σ̃θ−1 µ̃θ = − ln |Σ̃θ | − 2 2 2 1 − Trace{I + Σ̃θ−1 µ̃θ µ̃Tθ }, 2 where Eq. 12.48 from the text has been used. (b) ˜ + (c̃ − 1) ln β] B2 := Eβ [ln q̃β (β)] = Eβ [− ln Γ(c̃) + c̃ ln d˜ − dβ c̃ ˜ = − ln Γ(c̃) + c̃ ln d˜ − d˜ + (c̃ − 1)(ψ(c̃) − ln d). d˜ (c) B3 := Eα [ln q̃α (α)] = Eα [−K ln Γ(ã) + ã = −K ln Γ(ã) + ã

= −K ln Γ(ã) + ã

K−1 X

ln b̃k + (ã − 1)

K−1 X

ln αk −

k=0

k=0

K−1 X

K−1 X

k=0

k=0

k=0

K−1 X

K−1 X

K−1 X

k=0

ln b̃k + (ã − 1) ln b̃k + (ã − 1)

(ψ(ã) − ln b̃k ) −

(ψ(ã) − ln b̃k ) − Kã,

k=0

K−1 X k=0

b̃k

ã b̃k

b̃k αk ]

5

which concludes the derivation.

13.6. Consider the Gaussian mixture model

p(x) =

K X

Pk N (x|µk , Q−1 k ),

k=1

with priors p(µk ) = N (µk |0, β −1 I),

(10)

and p(Qk ) = W(Qk |ν0 , W0 ).

Given the set of observations X = {x1 , . . . , xN }, x ∈ Rl , derive the respective variational Bayesian EM algorithm, using the mean field approximation for the involved posterior pdfs. Consider Pk , k = 1, 2, . . . , K, as deterministic parameters and optimize the respective lower bound of the evidence with respect to the Pk ’s.

Solution: Consider q(Z, µ1:K , Q1:K ) = q(Z)q(µ1:K )q(Q1:K ), where the notation has been introduced in Section 13.4. From the theory we have: Step 1a: ln qz(j+1) (Z) = Eq(j) q(j) [ln p(X , Z, µ1:K , Q1:K )] + constant µ

Q

= Eq(j) q(j) [ln p(X |Z, µ1:K , Q1:K )]+ µ

Q

ln P (Z|P (j) )] + constants.

Recall that p(X |Z, µ1:K , Q1:K ) =

N Y K Y

N xn |µk , Q−1 k

n=1 k=1

and P (Z|P ) =

N Y K Y n=1 k=1

zn

Pk k .

znk

,

6

Hence, we have that ln qz(j+1) (Z) =

N X K X n=1 k=1

1 1 (j) znk Eq(j) q(j) ln Pk + ln |Qk | − (xn − µk )T Qk (xn − µk ) µ Q 2 2

+ constants N X K X

1 (j) znk ln Pk + Eq(j) [ln |Qk |] 2 Q n=1 k=1 n 1 − trace Eq(j) [Qk ](xn xTn − Eq(j) [µk ]xTn − µ Q 2 o T T xn Eq(j) [µk ] + Eq(j) [µk µk ]) + constants, =

µ

µ

or if we set 1 (j) E (j) [ln |Qk |] πnk = Pk exp 2 qQ n o 1 − trace Eq(j) [Qk ](xn xTn − Eq(j) [µk ]xTn − xn Eq(j) [µTk ] + Eq(j) [µk µTk ]) , µ µ µ Q 2 then we have that qz(j+1) (Z) =

N Y K Y

zn

ρnkk ,

n=1 k=1

where we have taken care for the normalization of the respective probabilities, hence πn . ρnk = PK k k=1 πnk Also note that Eq(j+1) [znk ] = ρnk , by the binary nature of znk . z

Step 1b: ln qµ(j+1) (µ1:K ) = Eq(j+1) q(j) [ln p(X |Z, µ1:K , Q1:K ) + ln p(µ1:K )] + constants z

=

Q

K X N X k=1

1 Eq(j+1) [znk ] E (j) [ln |Qk |]− z 2 qQ n=1

1 1 Eq(j) [xTn Qk xn ] + µTk Eq(j) [Qk ]xn − µTk Eq(j) [Qk ]µk Q Q 2 Q 2 1 T − βµk µk + constants, 2 or ln qµ(j+1) (µ1:K ) =

K X k=1

1 − µTk 2

+ µTk Eq(j) [Qk ] Q

βI + Eq(j) [Qk ] Q

N X n=1

N X

! ρ nk

n=1

ρnk xn + constants,

µk

7

and since the exponent is of quadratic form we get, qµ(j+1) (µ1:K ) =

K Y

N (µk |µ̃k , Q̃k ),

k=1

where Q̃k = βI + Eq(j) [Qk ] Q

K X

ρ nk

k=1

and µ̃k = Q̃−1 k Eq (j) [Qk ] Q

K X

ρ nk x n .

k=1

Moreover, following (12.48) we have that Eq(j+1) [µk µTk ] = Σ̃k + µ̃k µ̃Tk . µ

Hence, we have now obtained the recursions for all the statistics that are required by the previous step.

Step 1c: (j+1)

ln qQ

h (Q1:K ) = Eq(j+1) q(j+1) ln p(X |Z, µ1:K , Q1:K )+ z µ i ln p(Q1:K ) + constants = Eq(j+1) q(j+1) z

K X N hX

µ

k=1

z nk

n=1

1 2

ln |Qk |−

n o 1 trace (xn − µk )(xn − µk )T Qk + 2 n o i 1 ν0 − l − 1 ln |Qk | − trace W0−1 Qk + constants 2 2 or (j+1)

ln qQ

(Q1:K ) =

K X 1 k=1

2

ln |Qk |

N X

ρ nk −

n=1 N

nX 1 trace ρnk xn xTn − µ̃k xTn − xn µ̃Tk 2 n=1 o + Eq(j+1) [µk µTk ] Qk + µ

ν0 − l − 1 1 ln |Qk | − trace{W0−1 Qk } + constants, 2 2 which results in (j+1)

qQ

(Q1:K ) =

K Y k=1

W(Qk |ν̃k , W̃k ),

8

where ν̃k = ν0 +

N X

ρ nk

n=1

W̃k−1 = W̃0−1 +

N X

ρnk (xn xTn − µ̃k xTn − xn µ̃Tk +

n=1

Eq(j+1) [µk µTk ]). µ

(j+1)

Since qQ (Q1:K ) is a product of Wisharts, then t is known from statistics that the respective mean values are given by, Eq(j+1) [Qk ] = ν̃k W̃k Q

Eq(j+1) [ln |Qk |] =

l X

Q

ψ

i=1

ν̃k + 1 − i 2

+ l ln 2 + ln |W̃k |.

where ψ(·) is the digamma function defined in the text.

Step 2: The lower bound is given by F(q; P ) = Eq(j+1) q(j+1) q(j+1) [ln p(X , Z, µ1:K , Q1:K )] z

µ

Q

− Eq(j+1) [ln qz(j+1) (Z)] − Eq(j+1) [ln qµ(j+1) (µ1:K )] z

µ

(j+1)

− Eq(j+1) [ln qQ (Q1:K )] Q # "N N XX znk ln Pk + constants, = Eq(j+1) z

n=1 k=1

where constants are independent on P . Hence, F(q; P ) =

N X N X

ρnk ln Pk + constants.

(11)

n=1 k=1

P now results by maximizing F(q; P ) with respect to P subject to the constraint K X Pk = 1.

(12) k=1

Thus, ∂ ∂P or

"K X

ln Pk (

N X

n=1

k=1

ρnk ) − λ

K X

# Pk = 0,

k=1

N

1X Pk = ρn , k = 1, 2, . . . , K. λ n=1 k

(13)

9

Plugging (13) into (12) λ=

N X K X

ρnk = N.

n=1 k=1

or

N

Pk =

1 X ρn , k = 1, 2, . . . , K. N n=1 k

13.7. Consider the Gaussian Mixture model of Problem 13.6, with the following priors imposed on µ, Q, and P: p(µ, Q) = p(µ|Q)p(Q) =

K Y

N µk |0, (λQk )−1 W(Qk |ν0 , Wo ),

k=1

that is, a Gaussian-Wishart product and p(P ) = Dir(P |a) ∝

K Y

Pka−1 ,

k=1

i.e., a Dirichlet prior. That is, P is treated as a random vector. Derive the E algorithmic steps of the variational Bayesian approximation adopting the mean field approximation for the involved posterior pdfs. We have adopted the notation µ in place of µ1:K and Q in place of Q1:K , for notational simplicity.

Solution: If Z is the set of latent variables associated with the mixture indices, we have q(Z, P , µ, Q) = q(Z)q(P )q(µ, Q).

Step 1a: We have that ln qz(j+1) (Z) = Eq(j) q(j) [ln p(X , Z, P, µ, Q)] P

µ,Q

= Eq(j) q(j) [ln p(X |Z, µ, Q) + ln p(Z|P)] + constants, P

µ,Q

or ln qz(j+1) (Z) = Eq(j) [ln p(Z|P )] + Eq(j) [ln p(X |Z, µ, Q)] P

µ,Q

N X K X

N X K X

1 znk Eq(j) ln |Qk | P µ,Q 2 n=1 k=1 n=1 k=1

1 − Eq(j) (xn − µk )T Qk (xn − µk ) , µ,Q 2

= Eq(j) [

znk ln Pk ] +

10

or if we define ln πnk = Eq(j) [ln Pk ] + P

1 1 Eq(j) [ln |Qk |] − Eq(j) [(xn − µk )T Qk (xn − µk )], µ,Q 2 2 µ,Q (14)

then

N Y N Y

qz(j+1) (Z) =

zn

ρnkk ,

(15)

n=1 k=1

with

πn , ρnk = PK k k=1 πnk

(16)

with vq(j+1) [znk ] = ρnk , by definition and the binary nature of znk . z

Step 1b: (j+1)

ln qP

h i (P ) = Eq(j+1) q(j) ln p(X |Z, µ, Q) + ln p(Z|P ) + ln(P ) + constants z µ,Q "N K # K XX X = Eq(j+1) znk ln Pk + (a − 1) ln Pk + constants z

n=1 k=1

=

K X

N X

k=1

n=1

k=1

! ρnk + a − 1 ln Pk + constants,

from which we obtain (j+1)

qP

(P ) ∝

K Y

Pkak −1 ,

k=1

with ak = a +

N X

ρ nk ,

n=1

that is, it remains Dirichlet.

Step 1c: h i (j+1) lnqµ,Q (µ, Q) = Eq(j+1) q(j+1) ln p(X |Z, µ, Q, P ) + ln p(Z|P ) + ln p(P ) + ln p(µ, Q) z P

N X K X 1 1 T = Eq(j+1) [znk ] ln |Qk | − (xn − µk ) Qk (xn − µk ) 2 2 n=1 k=1

+

−

K 1X

2 1 2

k=1 K X k=1

ln(|λQk |) −

K

K

k=1

k=1

1X T ν0 − l − 1 X µk (λQk )µk + ln |Qk | 2 2

trace{W0−1 Qk } + constants,

11

or (j+1)

ln qµ,Q

=

K X 1 k=1

2 |

N X

N

X 1 ρnk − trace{Qk ρnk xn xTn } 2 n=1 k=1 {z } {z } |

ln |Qk |

C

A

N X

1 − µTk ] Qk ρ nk 2 n=1 {z |

! µk + µTk Qk

ρ nk x n

n=1

}

B

+

N X

|

{z

}

B

1 1 ν0 − l − 1 ln |λQk | − µTk (λQk )µk + ln |Qk | 2 {z |2 {z } | 2 {z } | } A

B

A

1 − trace{W0−1 Qk } +constants. | 2 {z } C

Combining all the terms A, B, C respectively together we obtain PN ν0 + 1 + n=1 ρnk − l − 1 ln |Qk | A: 2 N N X X 1 B : − µTk (λ + ρnk )Qk µk + µTk Qk ρ nk x n , 2 n=1 n=1 which is quadratic with respect to µk and it is rewritten, according to the appendix of the chapter, as 1 1 B : − (µk − µ̂k )T (λ̃Qk )(µk − µ̂k ) + µ̂Tk (λ̃Qk )µ̂k , 2 2 where λ̃ = λ +

N X

ρnk

n=1 N

µ̂k =

C:

1X ρnk xn λ̃ n=1

N X 1 trace{(W0−1 + ρnk xn xTn )Qk }.

2 n=1

Combining A, B and C, we finally obtain (j+1)

ln qµ,Q (µ, Q) =

K X

1 − (µk − µ̂k )T (λ̃Qk )(µk − µ̂k )+ 2 k=1

1 ν̃k − l − 1 ln |Qk | − trace{W̃k−1 Qk } , 2 2

12

where ν̃k = ν0 +

N X

ρnk + 1,

n=1

and W̃k−1 = W̃0−1 + λ̃µ̂k µ̂Tk −

N X

ρnk xn xTn .

n=1

Hence, (j+1)

qµ,Q

=

K Y

N (µk |µ̂k , (λ̃Qk )−1 )W(Qk |ν̃k , W̃k ).

k=1

Thus, it is a Gaussian-Wishart product. Note that for the algorithm to be complete the expected values of ln |Qk | and (xn − µk )T Qk (xn − µk ), (j+1) w.r. to qµ,Q have to be computed. The former one is obtained from the Wishart properties, as in Problem 13.6 13.8. If µ and Q are distributed according to a Gaussian-Wishart product p(µ, Q) = N (µ|µ̂, (λQ)−1 )W(Q|ν, W ), then compute the expectation E[µT Qµ].

Solution: We have that E[µT Qµ] = E[trace{QµµT }] = EQ Eµ|Q [trace{QµµT }] = trace{EQ [Q Eµ|Q [µµT ]]}, which from Eq. (12.74) in the text becomes T

E[µT Qµ] = trace{EQ [Q((λQ)−1 + µ̂µ̂ )]}

1 T I + EQ [Qµ̂µ̂ ] = trace λ l T = + trace{νW µ̂µ̂ } λ l T = + ν µ̂ W µ̂, λ where l is the respective dimensionality.

13.9. Derive the Hessian matrix w.r. to θ of the cost function J(θ) =

N X

[yn ln σ(φT (xn )θ) + (1 − yn ) ln(1 − σ(φT (xn )θ))]

n=1

1 − θ T Aθ, 2

13

where σ(z) =

1 .

1 + exp(−z)

Solution: Define t = σ(z).

Then we have ∂t ∂ 1 = = ∂z ∂z 1 + exp(−z) exp(−z) = = (1 + exp(−z))2 = σ(z)(1 − σ(z)) = t(1 − t).

(17)

Jn (t) = yn ln t + (1 − yn ) ln(1 − tn ).

(18)

∂Jn (t) yn − t 1 (1 − yn ) = yn − = . ∂t t 1−t t(1 − t)

(19)

Now let We have

Combining (17) and (19) and letting zn = φT (xn )θ, we obtain N ∂J(θ) X ∂Jn (tn ) ∂tn ∂zn = − Aθ ∂θ ∂tn ∂zn ∂θ n=1

=

=

N X yn − t n tn (1 − tn )φ(xn ) t (1 − tn ) n n=1 N X

(yn − tn )φ(xn ) − Aθ

n=1

=

N X

(yn − σ(φT (xn )θ))φ(xn ) − Aθ.

n=1

Hence, for the Hessian we have 2

N X ∂ J(θ) ∂tn ∂zn = −[A]ij − φi (xn ) ∂θ ∂zn ∂θj ij n=1 = −[A]ij −

N X

φj (xn )(tn (1 − tn ))φi (xn )

n=1

= −[A]ij −

N X n=1

φj (xn )σ(φT (xn )θ)(1 − σ(φT (xn )θ)φi (xn ),

14

or

∂ 2 J(θ) = −A − ΦT T Φ, ∂θ

where T = diag{t1 , t2 , . . . , tN }.

13.10. Show that the marginal of a Gaussian pdf with a gamma prior on the variance, after integrating out the variance, is the student’s-t pdf, given by 1/2 Γ( ν+1 λ 1 2 ) .

(20) st(x|µ, λ, ν) = ν

ν+1 Γ( 2 ) πν 2 2 1 + λ(x−µ) ν Solution: From the text in the chapter and for the one dimensional (parameter) case, we have that Z p(θ; a, b) = p(θ|α)p(α)dα Z = N (θ|0, α−1 )Gamma(α|a, b)dα Z =

1 2π

12

1 α exp − α(x − µ)2 2 1 2

1 a a−1 b α exp (−bα) dα, Γ(a) (21)

or 12

1 2 b α exp −(b + (x − µ) )α dα 2 12 −a− 21 1 1 1 = ba b + (x − µ)2 . Γ(a) 2π 2 a+ 21

Z 1 1 1 b + (x − µ)2 αa− 2 exp −(b + (x − µ)2 )α dα.

2 2 (22)

1 p(θ; a, b) = Γ(a)

1 2π

a

Z

a− 21

Note that the quantity under the integral is a gamma distribution with parameters α + 21 and b + 12 (x − µ)2 and its integration results in the corresponding normalization constant, hence p(θ; a, b) =

1 Γ(a)

1 2π

21

−a− 21 1 1 ba b + (x − µ)2 Γ(a + ), 2 2

(23)

and the standard form of the student’s-t pdf results from the previous in a trivial way by introducing the change of variables, ν = 2a and a = λb.

13.11. Derive the pair of recursions (13.62)-(13.63).

15

Solution: Our starting point is Eq. (13.60).

L(α, β)

N 1 ln(2π) − ln |β −1 I + ΦA−1 ΦT | 2 2 −1 1 T −1 − y β I + ΦA−1 ΦT y.

2

= −

(24)

However, using the identity concerning determinants from the matrix theory (Appendix A), we have C

:=

|A|−1 |I||A + βΦT Φ| β = − ln |Σ| − ln |A| − N ln β, ln |β −1 I + ΦA−1 ΦT | = ln

(25)

where, Σ := A + βΦT Φ

−1

.

(26)

Also, from the matrix inversion lemma we have, D := β −1 + ΦA−1 ΦT

−1

= βI − βΦ A + βΦT Φ

−1

ΦT β.

(27)

Thus, E

:= y β −1 I + ΦA−1 ΦT

−1

y = βy T y −

βy T ΦΣΦT βy =

βy T (y − Φµ) ,

where µ = βΣΦT y.

(28)

From the previous definitions and after some algebra, one can show that E = βy T (y − Φµ) = β||y − Φµ||2 + µT Aµ.

Indeed T

(y − Φµ) (y − Φµ)

=

y T y − y T Φµ + µT ΦT Φµ − µT ΦT y,

or βy T (y − Φµ)

=

β||y − Φµ||2 + βµT ΦT y − βµT ΦT ΦT µ,

or taking into account that βΦT Φ = Σ −1 − A,

(29)

16

we get βy T (y − Φµ)

=

β||y − Φµ||2 + βµT ΦT y −

µT Σ −1 − A µ

=

β||y − Φµ||2 + βµT ΦT y − µT Σ −1 µ + µT Aµ.

Taking into account that Σ −1 µ = βΣ −1 ΣΦT y, proves the claim.

Combining, the formulae for C, D, E, we obtain that L(α, β)

=

N 1 1 1 ln(2π) − ln |Σ −1 | + ln |A| + N ln β − 2 2 2 2 1 1 β||y − Φµ||2 − µT Aµ.

2 2

−

(30)

Using standard formulae for the differentiation for matrices and determinants, we obtain ∂ ln |Σ −1 | ∂αk

= =

1 ∂|Σ −1 | |Σ −1 | ∂αk

∂Σ −1 1 −1 |Σ |trace Σ |Σ −1 | ∂αk

(31)

However, Σ −1 = A + βΦT Φ, hence

∂Σ −1 = diag{0, . . . , 1, 0, . . . , 0}, ∂αk

(32)

with an 1 at the kth position. Thus

Also

∂ ln |Σ −1 | = Σkk . ∂αk

(33)

∂ ln |A| 1 ∂ Y 1 = αj = . ∂αk |A| ∂αk j αk

(34)

∂ µT Aµ = µ2k . ∂αk

(35)

L(α, β) 1 1 1 1 = − Σkk + − µ2k . ∂αk 2 2 αk 2

(36)

Also,

Thus we have

17

Equating to zero and setting γk = 1 − αk Σkk , we obtain αk =

1 γk = 2. µ2k + Σkk µk

We now turn into the derivation with respect to β. We have that ∂ ln |Σ −1 | ∂β

1 ∂Σ −1 |Σ −1 |trace{Σ } −1 |Σ | ∂β

1 = trace{ΣΦT Φ} = trace{ Σ βΦT Φ + A − A } β

1 −1 = trace{ Σ Σ − A } β 1 1X = trace{(I − ΣA)} = γk .

(37) β β =

k

Thus finally we have 1 1N 1X ∂L(α, β) = − ||y − Φµ||2 + − γk , ∂β 2 2β 2

(38)

k

which after setting it to zero leads to, β=

P N − k γk , ||y − Φµ||2

(39)

which concludes the proof.

13.12. Consider a two class classification task and assume that the feature vectors in each one of the two classes, ω1 , ω2 , are distributed according to the Gaussian pdf. Both classes share the same covariance matrix Σ, and the mean values are µ1 and µ2 , respectively. Prove that, given an observed feature vector, x ∈ Rl , the posterior probabilities for deciding in favor of one of the classes is given by the logistic function, i.e., P (ω2 |x) =

1 , 1 + exp (−θ T x + θ0 )

where θ := Σ −1 (µ2 − µ1 ), and θ0 =

P (ω1 ) 1 (µ2 − µ1 )T Σ −1 (µ2 + µ1 ) + ln 2 P (ω2 )

Solution: We have that p(x|ωi ) =

1 1 T −1 exp − (x − µ ) Σ (x − µ ) , i = 1, 2. i i 2 (2π)l/2 |Σ −1 |1/2

18

Hence, P (ω1 |x)p(x) = p(x|ω1 )P (ω1 ), and P (ω2 |x)p(x) = p(x|ω2 )P (ω2 ), where P (ω1 ), P (ω2 ) are the respective prior probabilities. Dividing the previous equations by part, substituting the Gaussian pdf in place of p(x), it is a matter of simple algebra to see that,

1 P (ω1 ) P (ω1 |x) = exp −θ T x + (µ2 − µ1 )T Σ −1 (µ2 + µ1 ) exp ln , P (ω2 |x) 2 P (ω2 ) where θ = Σ −1 (µ2 − µ1 ).

Taking into account that P (ω2 |x) = 1 − P (ω1 |x), we prove the claim.

13.13. Derive equation (13.74).

Solution: Our starting point is the cost J :=

N X

yn ln σ(θ T φn ) + (1 − yn ) ln 1 − σ(θ T φn )

n=1

1 − θ T Aθ, (40) 2

where φn := φ(xn ). Taking the gradient with respect to θ, we get ∇θ J

=

N X n=1

yn

1 σ(θ T φ

(1 − yn )

0

n)

φn σ (θ T φ) +

0 1 T φ − σ (θ φ ) − n n 1 − σ(θ T φn )

Aθ

(41)

Taking into account that (see also next problem)

0 σ (z) = σ(z) 1 − σ(z) , and after some trivial algebra we end up with ∇θ J

= =

N X

yn − σ(θ T φn ) φn − Aθ

n=1 T

Φ (y − s) − Aθ,

which is set equal to zero and proves the claim.

(42)

19

13.14. Show Equation (13.75).

Solution: By the respective definition we have that σ(t) =

1 , 1 + exp(−t)

hence ∂σ(t) = σ(t)(1 − σ(t)). ∂t Also, we have ∂ T (θ φ(xn )) = φ(xn ). ∂θ Let now, sn := σ(θ T φ(xn )).

Then, by combining the previous formulae we get ∂sn = sn (1 − sn )φ(xn ). ∂θ Define now an := yn ln sn + (1 − yn ) ln(1 − sn ).

Then after some simple algebra and taking into account well known differentiation rules concerning the logarithm, we readily obtain that ∂an = yn φ(xn ) − sn φ(xn ). ∂θ For the needs of the proof, we need to consider the second derivative. The respective (i, j) element is readily obtained by h ∂2a i n

∂θ 2

ij

= −sn (1 − sn )φi (xn )φj (xn ) = −φi (xn )tn φj (xn ) h i = φ(xn )tn φ(xn )T . ij

where tn := sn (1 − sn ).

Recalling the definition,  φT (x1 )   .. Φ=  . 

φT (xN )

20

as well as that ln (P (y|θ)p(θ|α))

=

N h X i yn ln σ(θ T φ(xn )) + (1 − yn ) ln 1 − σ(θ T φ(xn )) − n=1

1 T θ Aθ + constant, 2

(43)

it is now straightforward to write Σ −1 = A + ΦT T Φ,

(44)

where we have used that ∂ 2 (θ T Aθ) = A. ∂θ 2 13.15. Derive the recursion (13.77).

Solution: Taking the logarithm of P (y|α) in (13.75) and keeping only the terms which depend on α, we have, ln P (y|α) = ln p(θ̂MAP |α) −

1 ln |Σ −1 | + constants.

2

Taking the derivative of the first term on the right hand side is trivial, since the pdf is Gaussian and we have done it many times already. The emphasis will be given on computing the determinant term. Using the formula form the linear algebra, ∂Σ −1 ∂ ln |Σ −1 | 1 = |Σ −1 |trace Σ . −1 ∂αk |Σ | ∂αk However, from the text we know that Σ −1 = A + ΦT T Φ, and since A is diagonal (see, also, Problem 13.11), we get that −

1 ∂ ln |Σ −1 | 1 = − Σkk , 2 ∂αk 2

which is the last term in (13.77).

13.16. Show that if f is a convex function f : Rl → R, then it is equal to the conjugate of its conjugate, i.e., (f ∗ )∗ = f .

Solution: Recall from the theory that f ∗ (ξ) = ξ T x∗ − f (x∗ ),

(45)

21

where x∗ : x∗ (ξ) : ∇f (x∗ ) = ξ,

(46)

where we have suppressed the dependence of x∗ on ξ. Hence, combining (45) and (46) we get xT ξ − f ∗ (ξ) = f (x∗ ) + (x − x∗ )T ∇f (x∗ ).

(47)

However, we know that if f is convex then xT ξ − f ∗ (ξ) = f (x∗ ) + (x − x∗ )T ∇f (x∗ ) ≤ f (x), ∀x, x∗ ∈ Rl , and equality, that is the maximum is achieved for x = x∗ .

Hence, due to the dependence of x∗ on ξ (Eq. (46)), we can write f (x) : maxξ (xT ξ − f ∗ (ξ)).

13.17. Prove that f (x) = ln

√ λ − λ x, x ≥ 0, 2

is a convex function.

2

f (x) Solution: It is known from the theory of convex functions that if d dx ≥ 2 0, then f (x) is convex ([Boyd 04]). Hence,

λ 1 λ 3 f 0 (x) = − x− 2 , f 00 (x) = x− 4 ≥ 0, x ≥ 0.

2 4 13.18. Derive variational bounds for the logistic regression function σ(x) =

1 , 1 + e−x

one of them in terms√of a Gaussian function. For the latter case, use the transformation, t = x.

Solution: This is a typical case where the function itself is neither convex or concave. We will first transform it, by taking the logarithm, which leads to a concave one. a)

f (x) = ln σ(x) = − ln(1 + e−x ). This is a concave function since f 0 (x) =

e−x −e−x 00 , f (x) = < 0.

1 + e−x (1 + e−x )2

Now, f ∗ (ξ) = min(ξx + ln(1 + e−x )), x

22

and taking the derivative we obtain e−x =0⇒ 1 + e−x ξ 1−ξ e−x∗ = ⇒ x∗ = ln , 0 < ξ < 1.

1−ξ ξ

x∗ : ξ −

Hence f ∗ (ξ) = ξ ln

ξ 1−ξ + ln 1 + ξ 1−ξ

= −(1 − ξ) ln(1 − ξ) − ξ ln ξ, which is the binary entropy function. Hence. ln σ(x) ≡ f (x) ≤ ξx − f ∗ (ξ), or σ(x) ≤ exp(ξx − f ∗ (ξ)), 0 < ξ < 1, and this bounds the sigmoid regression by exponential functions. b) Following [Jaak 97] , we write, ln σ(x)

= =

− ln 1 + exp(−x) = − ln exp(−x/2) exp(x/2) + exp(−x/2)

x − ln exp(x/2) + exp(−x/2) .

(48) 2

Let us now define

√ √ f (x) := − ln exp( x/2) + exp(− x/2) , x ≥ 0.

(49)

We will first show that f (x) is a convex function. To this end we will prove that the second derivative is nonnegative, in the respective domain of definition. To this end, set t= Hence,

√

x, or

dt 1 = √ . dx 2 x

√ df 1 dg 1 x = √ = − √ tanh( ), dx 2 2 x dt 4 x

where

g(t) = ln exp(t/2) + exp(−t/2) . and tanh(t) =

exp(t) − exp(−t) . exp(t) + exp(−t)

23

Following similar arguments as before, the second derivative turns out to be √ √ x d2 f 1 tanh( 2 ) 1 x 2 √ = − 1 − tanh ( ) , dx2 8x 2 2 x where we have used the chain differentiation rule as well as the property, 1 d tanh(y) = 1 − tanh2 (y) = . dy cosh2 (y) Let now y=

√

x/2.

It suffices to show that 1 tanh(y) ≥ , 2y 2 cosh2 (y) or recalling the known properties form the analysis, tanh(y) =

sinh(y) , cosh(y)

and 2 sinh(y) cosh(y) = sinh(2y), we have to show that

sinh(2y) ≥ 1.

2y

However, this is always true for y ≥ 0, from the known from the analysis expansion y5 y3 + + ..., sinh(y) = y + 3!

5! which proves the claim about convexity.

From the convexity property we can now write that

f ∗ (ξ) = max ξx − f (x) , ξ < 0. x

Note that the constraint is necessary in order to guarantee that the conjugate function remains finite. Otherwise, as x tends to infinity, the conjugate function also tends to infinity. The conjugate function can be obtained by the differentiation and finding the optimal value x∗ . However, there is no need for it, since the exact form is of no interest for us.

Thus we can now write that f ∗ (ξ) ≥ ξx − f (x), or f (x) ≥ ξx − f ∗ (ξ).

24

Hence, ln σ(x) = x/2 + f (x2 ) ≥ x/2 + ξx2 − f ∗ (ξ), or σ(x) ≥ exp(−f ∗ (ξ)) exp

x + ξx2 , 2

which provides a bound in terms of a Gaussian function.

13.19. Prove equation (13.100).

Solution: We know that N β N ln β − ln(2π) − E[ky − Φθk2 ] 2 2 2 K−1 K−1 X 1 1 X E[θ2k ] K ln(2π) − ln |Ξ| − . + ln φ(ξk ) − 2 2 2 ξk

Q(ξ, β; ξ (j) ,β (j) ) =

k=0

k=0

Taking the derivative with respect to ξk , and recalling that |Ξ| =

K−1 Y

ξk ,

k=0

φ(ξk ) =

2 λp λ 2πξk exp − ξk , 2 2

we have • K−1

1 ∂ X ln φ(ξk ) = φ0 (ξk ) ξk φ(ξk ) k=0 2 2 exp λ2 ξk λ √ λ −1/2 = λ√ 2πξk exp − ξk 4 2 2πξ k 2 2 2 p λ λ λ 2πξk − exp − ξk + 2 2 2 2 1 λ = ξk−1 − .

2 2 • −

1 ∂ 1 ln |Ξ| = − ξk−1 .

2 ∂ξk 2

• " # K−1 ∂ 1 X E[θ2k ] E[θ2k ] − = . ∂ξk 2 ξk ξk2 k=0

25

Combining the previous steps and equating to zero we obtain r E[θ2k ] . ξk = λ2 13.20. Derive the mean and variance of G(Tk ) for a DP process.

Solution: Recall the mean and variance values of a Dirichlet distribution from Chapter 2. Also, for the case of DPs, the parameters of the associated Dirichlet distribution are ak = αG0 (Tk ). Then we get, E[G(Tk )] =

αG0 (Tk ) = G0 (Tk ).

PK α k=1 G0 (Tk )

For the variance we get, var[G(Tk )]

=

P

K αG0 (Tk ) α k=1 G0 (Tk ) − αG0 (Tk ) P P

K K α2 G (T ) α G (T ) + 1 0 k 0 k k=1 k=1

=

G0 (Tk ) (1 − G0 (Tk )) 1+α

13.21. Show that the posterior DP, after having obtained n observations from the set Θ, is given by, !! n X 1 δθi (θ) G|θ1 , . . . , θn ∼ DP α + n, αG0 + α+n i=1 Solution: By definition we have that 0

0

α G0 (Tk ) = αG0 (Tk ) + nk .

Moreover, the above is true for all finite (measurable) partitions, e.g., for disjoint Tk s. Adding over k and taking into account that probabilities add to one, we get, !

K K K X X 0 X 0 0 α G0 (Tk ) = α = α G0 (Tk ) + nk = α + n. k=1

k=1

k=1

Hence we can now write that 0

(α + n)G0 (Tk ) = αG0 (Tk ) + nk = αG0 (Tk ) +

n X

δθi (θ).

i=1:θi ∈Tk

The above is true for any partition, hence we can write that. ! n X 0 1 G0 = αG0 + δθi (θ) . α+n i=1

26

13.22. The stick breaking construction of a DP is built around the following rule: P1 = β1 ∼ Beta(β|1, α) and, βi

∼ Beta(β|1, α),

Pi

= βi

i−1 Y

(1 − βj ), i ≥ 2.

j=1

Show that if the number of steps is finite, i.e., we assume that Pi = 0, i > T , for some T , then βT = 1.

Solution: If we stop at a step T , then we should have, T X

Pi = 1

i=1

or PT

=

1 − P1 −

T −1 X

Pi = 1 − β 1 −

T −1 X i=2

i=2

=

βi

i−1 Y

(1 − βj )

j=1

TY −1

(1 − βj ), T ≥ 2.

j=1

The last equality can be shown recursively. First, it holds true for T = 2, because P2 = 1 − P1 = 1 − β1 . In the sequel, assume that it is true for some T > 2, i.e., 1 − β1 −

T −1 X i=2

βi

i−1 Y

TY −1

(1 − βj ) =

(1 − βj ).

j=1

j=1

We will show that it is true for T + 1. Indeed, 1 − β1 −

T X i=2

βi

i−1 Y

(1 − βj )

=

1 − β1 −

j=1

T −1 X

βi

i=2

βT

i−1 Y

(1 − βj ) −

j=1

TY −1

(1 − βj )

j=1

=

=

TY −1

TY −1

j=1

j=1

(1 − βj ) − βT

T Y

(1 − βj ),

j=1

and the claim has been proved.

(1 − βj )

27

Hence, PT = β T

TY −1

TY −1

j=1

j=1

(1 − βj ) =

(1 − βj ) =⇒ βT = 1.

13.23. Show that in CRP, the cluster assignments are exchangeable and do not depend on the sequence that customers arrive, up to a permutation of the labels of the tables.

Solution: Let us assume that n customers have arrived and Kn tables (clusters) have been formed. Associate with each customer a label, yi , where yi = k means that the ith customer sits at table k ∈ {1, 2, . . . , Kn }.

According to the chain rule of probabilities, we get P (y1 , y2 , . . . , yn ) = P (y1 )P (y2 |y1 ).....P (yn |yn−1 , yn−2 , . . . , 1).

(50)

In the above product on the right hand side, collect all the terms where the corresponding label, yi , takes the value k; that is, we consider together all customers that sit at table k. Note that the corresponding number of terms is equal to the number of customers sitting at the kth table, denoted as nk . Each of these terms represents the respective probability, which has been assigned according to the rule in Eq. (13.119). Thus, we can write that this product is equal to 1 2 nk − 1 α × × ×. . .× . n(k, 1) − 1 + α n(k, 2) − 1 + α n(k, 3) − 1 + α n(k, nk ) − 1 + α Indeed, one of the terms will be related to the first customer sitting at table k. This corresponds to the first term in the above product, where n(k, 1) denotes the order in which the aforementioned customer arrived.

For example, if he/she were the first customer, n(k, 1) = 1, if he/she were, say, the 9th customer in the queue entering the restaurant, then n(k, 1) = 9, etc. The rest of the terms are similarly defined. Now, the term of the product in Eq. (50) that corresponds to the second customer sitting at the kth table is equal to the second term above, and so on up to the nk th term. Thus, we can now write the product in Eq. (50) as, α Kn P (y1 , y2 , . . . , yn ) = Qn

QKn

i=1 (ni − 1)! . (i i=1 − 1 + α)

(51)

The numerator is straightforward, since we have Kn groups of terms in Eq. (50). For the denominator, recall that each customer arrives only once.

Hence, each one of the terms n(k, j), j = 1, 2, . . . , nk , and k = 1, 2, . . . , Kn , has a unique and distinct, from all the others, value in the set {1, 2, . . . , n}.

Thus, each one of the terms in the denominators in the last product is equal to i − 1 + α for a specific value of i in the previous set of integers up to n.

28

Observe that the product in Eq. (51) depends only on the number of tables/clusters, Kn , the number of customers, n, the number of customers sitting at the tables, and not on the sequence in which customers have arrived and sat at the tables.

13.24. Show that in an IBP, the probabilities for P (Z) and the equivalence classes, P ([Z]), are given by the formulae,

K α Y Γ (N − mk + 1) α Γ mk + K

.

P (Z) = α K Γ N +1+ K k=1 and

K α Y Γ (N − mk + 1) α Γ mk + K

P ([Z]) = Q2N −1 , α K Γ N + 1 + K ! h K k=1 h=0 K!

respectively. Note that Kh , h = 1, 2, . . . , 2N − 1, is the number of times the row vector associated with the hth nonzero binary number appears in Z. Solution: From the text and the definition of P (Z), taking into account that the probabilities are beta distributed, we get P (Z)

=

K Z 1 Y k=1

=

0

α

Pkmk (1 − Pk )N −mk

K Y

1

B k=1

α K,1

Z 1

0

α −1 m +K

Pk k

−1

PkK dPk α B K ,1 N −mk

(1 − Pk )

dPk .

α The above integrand is the normalizing constant of a Beta Pk |mk + K , N − mk + 1 and we know that this is equal to

Γ m + α Γ (N − m + 1) α k k K

B mk + , N − mk + 1 = . α K Γ N +1+ K

Thus, P (Z)

=

=

K Y Γ

α K + 1 Γ α K Γ(1)

α mk + K Γ (N − mk + 1)

α Γ Γ N +1+ K k=1

K α Y Γ (N − mk + 1) α Γ mk + K

. α K Γ N +1+ K k=1

For P ([Z]), we have that, P ([Z]) =

X Z∈[Z]

P (Z).

29

Let K be the number of total rows in Z. Let K0 be the number of times the 0th row vector is present, K1 is the respective number for the row vector associated with the binary 1, and so on. The total number of permutations of K objects, grouped in K0 , K1 ,. . . ,K2N −1 groups is known from combinatorics to be equal to

K! K .

(52) = Q2N −1 K0 , K1 , . . . , K2N −1 h=0 Kh !

Hence,

K α Y Γ (N − mk + 1) α Γ mk + K

P ([Z]) = Q2N −1 . α K Γ N + 1 + K ! h K k=1 h=0 K!

The previously used combinatorics formula in (52) can be shown by the following arguments. Out of the K rows, K0 are the zero ones. Then the total number of permutations of these zero rows are,

K! K = .

K0 K0 !(K − K0 )!

Now, for each one of the above permuted matrices, we make all possible permutations for the row vectors associated with K1 . This number amounts to,

(K − K0 )!

K − K0 . = K1 K1 !(K − K0 − K1 )!

Thus, the total number of permutations involving K0 and K1 amounts to, K! .

K0 !K1 !(K − K0 − K1 )!

Taking the above rationale recursively, the total number of permutations will be given by involving successive products as before, which it proves formula in Eq. (52).

13.25. Show that the discarded pieces, πk , in the stick-breaking construction of an IBP are equal to the sequence of probabilities produced in a DP stickbreaking construction.

Solution: The sequence of the discarded segments is equal to π1

=

1 − β1 ,

π2

=

β1 − β1 β2 = (1 − β2 )β1 ,

...

=

πk

=

... k−1 Y j=1

βj −

k Y j=1

βj = (1 − βk )

k−1 Y j=1

βj

30 0

Set β := 1 − β. Then, the previous recursions can be written as 0

π1

= β1 ,

π2

= β2 (1 − β1 ),

...

= ...

πk

=

0

0

0

= βk

k−1 Y

0

(1 − βj ).

j=1

The above sequence is the same with that in the DP construction. It 0 suffices to show that β follow a Beta(1, α) distribution. Indeed, β ∼ Beta(α, 1) =

β α−1 .

B(α, 1)

Hence, 0

0

β ∼ which proves the claim.

(1 − β )α−1 = Beta(1, α), B(α, 1)

1

Solutions To Problems of Chapter 14 14.1. Show that if Fx (x) is the cumulative distribution function of a random variable x, then the random variable u = Fx (x) follows the uniform distribution in [0, 1].

Solution: Let u = Fx (x), 0 ≤ u ≤ 1.

Then, we have that Fu (u)

:=

Prob{u ≤ u} = Prob{Fx (x) ≤ u} = Prob{x ≤ Fx−1 (u)}

=

Prob{x ≤ x} = Fx (x) = u.

Hence, Fu (u) = u which implies that u is uniform in [0, 1].

14.2. Show that if u follows the uniform distribution and x = Fx−1 (u) := g(u), then indeed x is distributed according to Fx (x) =

(1) Rx −∞

p(x)dx.

Solution: Due to the monotonicity of Fx (x), for each x there exists a unique u. Let p̃(x) be the pdf of the generated x by (1). We have that x = Fx−1 (u) or u = Fx (x).

Hence, 1 = p(u)

=

p̃(x) p̃(x) ⇒ = |Fx0 (x)| p(x) p̃(x) = p(x),

where we have used the rule of transforming random variables and that Z 1 cdu = 1 ⇒ c = 1.

0

14.3. Consider the random variables r and φ with exponential and uniform distributions r 1 pr (r) = exp − , r≥0 2 2 and 1 0 ≤ φ ≤ 2π 2π pφ (φ) = 0 otherwise,

2

respectively. Show that the transformation √ x = r cos φ = gx (r, φ), √ y = r sin φ = gy (r, φ), renders both x and y to follow the normalized Gaussian N (0, 1).

Solution: We have that r

x2 + y2 := gr (x, y), y arctan( ) := gφ (x, y). x

=

φ = The respective Jacobian is " J(x, y; r, φ) =

∂gx ∂r ∂gy ∂r

∂gx ∂φ ∂gy ∂φ

# =

1 −1 1 2 2 2 r 1 cos φ −r1 sin φ 1 −2 sin φ r 2 cos φ 2r

and |J(x, y; r, φ)| = Note that

 2x J(r, φ; x, y) =  2y

1 .

2

−y 1 x2 1+( y )2 x  1 1 y x 1+( )2 x



and |J(r, φ; x, y)| = 2. Hence, px,y (x, y)

1 = prφ (gr (x, y), gφ (x, y)) 1/2 2

x + y2 1 1 exp − = 2 2π 2 2 2 2 1 x 1 y √ exp − = √ exp − .

2 2 2π 2π

14.4. Show that if px (x) = N (x|0, I), then y given by the transformation y = Lx + µ is distributed according to py (y) = N (y|µ, Σ), where Σ = LLT .

Solution: We have that, y = Lx + µ,

3

is a linear transformation and it is readily checked out that the Jacobian matrix  ∂y1  1 . . . ∂y ∂x1 ∂xl  .. ..  = L. J(y; x) =  ... . .  ∂yl ∂x1

∂yl ∂xl

...

Also, x = L−1 (y − µ), where the inverse is assumed to exist (Σ is invertible). Then

px L−1 (y − µ) , py (y) = det J(y; x or py (y)

= = =

T −1

1 −1 L (y − µ) L (y − µ) exp − l 2 | det L|(2π) 2

1 1 T −T −1 (y − µ) L L (y − µ) exp − l 2 | det L|(2π) 2

1 1 T −T −1 − (y − µ) L L (y − µ) , l 1 exp 2 (2π) 2 |Σ| 2 1

where |Σ| is the determinant of Σ and we used that |Σ| =

|L||LT | = |L|2 ⇒

|L| =

|Σ| 2 .

1

Recall that |Σ| > 0 since Σ is positive definite.

14.5. Consider two Gaussians p(x) = N (x|0, σp2 I), σp2 = 0.1 and q(x) = N (x|0, σq2 I), σq2 = 0.11 x ∈ Rl . In order to use q(x) for drawing samples from p(x), via the rejection sampling method, a constant c has to be computed so that cq(x) ≥ p(x).

Show that

c≥

σq σp

l ,

and compute the probability of accepting samples.

4

Solution: The maximum values for the two distributions occur at x = 0 and they are 1 1 = p(0) = l , l l l (2π) 2 σp (2π) 2 (0.10) 2 and q(0) =

1 l 2

(2π)

σql

1

=

l

l 2

(2π) (0.11) 2

.

Note that p(x) and q(x) are very close, since σp2 ' σq2 . If σq2 σp2

c≥

! 2l l

= (1.1) 2 ,

then cq(x) ≥ p(x), due to the broader shape of q(x) (draw the graphs to see it). Let us choose c to have the smallest possible value, l

c = (1.1) 2 .

For large l, this can be a very large value indeed. The corresponding probability for acceptance is equal to Z 1 1 p(x)dx = .

Prob{acceptance} = c c which for large values of l, can be very small.

14.6. Show that using importance sampling leads to an unbiased estimator for the normalizing constant of the desired distribution, p(x) =

1 φ(x).

Z

However, the estimator E[f (x)], of a function f (·) is a biased one.

Solution: We know that N

Ẑ =

1 X w(xi ) N i=1

or

N

E[Ẑ] =

1 X E[w(xi )].

N i=1

However, Z E[w(x)]

=

φ(x) q(x)

q(x)dx

Z =

φ(x)dx = Z.

5

Hence E[Ẑ] = N1 N Z = Z. Also, since Ẑ is the sum of N unbiased variables σ2

divided by N , we know (Chapter 3) that its variance will be Nw , where 2 2 σw is the variance of w(x), provided σw is finite. Thus, Ẑ converges to Z as N −→ ∞.

On the other hand, we have PN \ E[f (x)] =

i=1 f (xi )w(xi ) .

PN i=1 w(xi )

PN For the denominator, we know that E[ i=1 w(xi )] = N Z. For the numerator, we get "N #

N Z X X φ(x) q(x)dx E f (xi )w(xi ) = f (x) q(x) i=1 i=1 =

N Z · E[f (x)]

However, for finite number of N , this does not mean that although the mean values of the numerator and denominator are N Z · E[f (x)] and N Z, respectively, their ratio is necessarily an unbiased estimator of E[f (x)].

Only for N → ∞, we obtain the unbiased estimator, when we divide by N Z, since Ẑ converges to Z. 14.7. Let p(x) = N (x|0, σ12 I). Choose the proposal distribution for importance sampling as q(x) = N (x|0, σ22 I).

The weights are computed as w(x) =

p(x) . q(x)

If w(0) is the weight at x = 0, then the ratio w(x) w(0) is given by w(x) 1 = exp w(0) 2

l

σ12 − σ22 X 2 x σ12 σ22 i=1 i

! .

Observe that even for very good match between q(x) and p(x) (σ12 ' σ22 ), for large values of l, the values of the weights can change significantly, due to the exponential dependence.

Solution: We have that

l Y 1 x2i p(x) = exp − 2 , 2σ1 (2πσ12 )l/2 i=1

6

and l Y

x2i q(x) = exp − 2 . l/2 2σ2 (2πσ 2 ) i=1 1

2

Hence, w(x) w(0)

l l

1 σ2 Y 1 − x2i exp σ1 i=1 2σ22 2σ12

l Y 1 σ12 − σ22 2 x = exp 2 σ12 σ22 i i=1 ! l 1 σ12 − σ22 X 2 = exp x .

2 σ12 σ22 i=1 i

=

σ1 σ2

l

14.8. Show a stochastic matrix P has always the value λ = 1 as its eigenvalue.

Solution: By definition we have P a = λa, and λ is computed so that the matrix P −λI to be singular, i.e., |P −λI| = 0. However, since the entries in each column of P add to one we have that adding all the rows of P − I together leads to zeros. Hence the rows are linearly dependent, P − I is singular and λ = 1 is an eigenvalue.

14.9. Show that if the eigenvalue of a transition matrix is not equal to one, its magnitude can not be larger than one, i.e., |λ| ≤ 1.

Solution: Let λ be an eigenvalue such that |λ| > 1, with a its corresponding eigenvector, P a = λa.

Then we know that P n a = λn a, and P n diverges, as n → ∞. However, this is not possible, since P n is a stochastic matrix such that pn = P n p0 . and all its elements are between 0 and 1.

14.10. Prove that if P is a stochastic matrix and λ 6= 1, then the elements of the corresponding eigenvector add to zero.

Solution: Let P a = λa, λ 6= 1,

7

then P11 a1 + . . . + P1K aK = λa1 PK1 a1 + . . . + PKK aK = λaK Adding together we have that (a1 + . . . + aK ) = λ(a1 + . . . + aK ).

Since the entries in each column of P add to one. Hence, either λ = 1 or P K k=1 ak = 0.

14.11. Prove the square root dependence of the distance travelled by a random walk, with infinite many integer states, on the time, n.

Solution: We have that X X

E[x2n ] = x2 pn (x) = x2 qpn−1 (x) + ρpn−1 (x − 1) + ρpn−1 (x + 1) x

=

q

x

X

2

x pn−1 (x) + ρ

X X (x + 1)2 pn−1 (x) + ρ (x − 1)2 pn−1 (x)

x

= = = E[x2n ]

=

x x 2 2 q E[xn−1 ] + ρ E[(xn−1 + 1) ] + ρ E[(xn−1 − 1)2 ] q E[x2n−1 ] + 2ρ E[x2n−1 ] + 2ρ E[x2n−1 ] + 2ρ or 2ρn + E[x20 ] = 2ρn.

14.12. Prove, using the detailed balance condition, that the invariant distribution associated with the Markov chain implied by the Metropolis-Hastings algorithm is the desired distribution, p(x).

Solution: By the definition of the respective kernel density we have

p(x|y)p(y) = q(x|y)α(x|y) + δ(x − y)r(x) p(y)

q(y|x)p(x) p(y) + δ(x − y)r(x)p(y) = q(x|y) min 1, q(x|y)p(y) = min {q(x|y)p(y), q(y|x)p(x)} + δ(x − y)r(x)p(y)

q(x|y)p(y) = q(y|x) min 1, p(x) + δ(y − x)r(y)p(x) q(y|x)p(x)

= q(y|x)α(y|x) + δ(y − x)r(y) p(x) = κ(y|x), which proves the claim.

14.13. Show that in Gibbs sampling, the desired joint distribution is invariant with respect to each one of the base transition pdfs.

8

Solution: By the definition of the base transition pdfs we have Y Bd (x|y) = p(x(d)|{y(i)} : i 6= d) δ(y(i) − x(i)). i6=d

Hence Z Bd (x|y)p(y)dy

Z p(x(d)|{y(i)} : i 6= d)

=

Y

δ(y(i) − x(i))p(y)dy

i6=d

Z =

p(x(d)|{y(i)} : i 6= d)p(y(d)|{y(i)} : i 6= d) × Y p({y(i)} : i 6= d) δ(y(i) − x(i))dy i6=d

Z =

p(x(d)|{x(i)} : i 6= d)p(y(d)|{x(i)} : i 6= d) × p({x(i)} : i 6= d)d(y(d))

= p(x(d)|{x(i)} : i 6= d)p({x(i)} : i 6= d) × Z p(y(d)|{x(i)} : i 6= d)d(y(d)) = p(x) × 1 = p(x), which proves the claim.

14.14. Show that the acceptance rate for the Gibbs sampling is equal to one.

Solution: We know that the acceptance ratio for the Metropolis-Hastings algorithm is given by

q(xn−1 |x)p(x) α(x|xn ) = min 1, . q(x|xn−1 )p(xn−1 ) For the Gibbs sampling, the second term in the min operator becomes Q p(xn−1 (d)|{x(i)} : i 6= d) i6=d δ(xn−1 (i) − x(i)) Q p(x(d)|{xn−1 (i)} : i 6= d) i6=d δ(x(i) − xn−1 (i))

= =

p(x(d)|{x(i)} : i 6= d)p({x(i)} : i 6= d) p(xn−1 (d)|{xn−1 (i)} : i 6= d)p({xn−1 (i)} : i 6= d) p(xn−1 (d)|{xn−1 (i)} : i 6= d)p(x(d)|{x(i)} : i 6= d)p({xn−1 (i)} : i 6= d) p(x(d)|{x(i)} : i 6= d)p(xn−1 (d)|{xn−1 (i)} : i 6= d)p({xn−1 (i)} : i 6= d) 1,

which proves the claim.

14.15. Derive the formulae for the conditional distributions of Section 14.11

9

Solution: The joint distribution of n0 , λ1 , λ2 , x1:N is given by n0 Y

p(n0 , λ1 , λ2 , x1:N ) =

N Y

P (xn |λ1 )

n=1

P (xn |λ2 )p(λ1 )p(λ2 )

n=n0 +1

1 .

N

Taking the logarithm, we have A := ln p(n0 , λ1 , λ2 , x1:n )

=

n0 X

ln P (xn |λ1 ) +

n=1

+

N X

ln P (xn |λ2 )

n=n0 +1

ln p(λ1 ) + ln p(λ2 ) − ln(N )

or A=

n0 X

N X

(xn ln λ1 − λ1 − ln(xn !)) +

n=1

(xn ln λ2 − λ2 − ln(xn !)) +

n=n0 +1

(a − 1) ln λ1 − bλ1 + a ln b − ln Γ(a) + (a − 1) ln λ2 − bλ2 + a ln b − ln Γ(a) − ln N. To get the conditionals, it suffices to freeze the values of the rest of variables and consider them constants. For example, ! n0 X xn ln λ1 − (n0 + b)λ1 + c1 , ln p(λ1 |n0 , λ2 , x1:n ) = a − 1 + n=1

where c1 is a constant (normalizing). It is straightforward to see that the same result would occur by applying the Bayes theorem p(λ1 |n0 , λ2 , x1:n ) =

p(n0 , λ1 , λ2 , x1:n ) . p(n0 , λ2 , x1:n )

Hence p(λ1 |n0 , λ2 , x1:n ) = Gamma(λ1 |a1 , b1 ) where a1 = a +

n0 X

xn , b1 = n0 + b.

n=1

Similarly p(λ2 |n0 , λ1 , x1:n ) a2

=

Gamma(λ2 |a2 , b2 )

= a+

N X

xn , b2 = N − n0 + b.

n=n0 +1

For n0 , we have P (n0 |λ1 , λ2 , x1:n ) = ln λ1

n0 X n=1

xn − n0 λ1 + ln λ2

N X

xn − (N − n0 )λ2 + c3

n0 +1

for n0 = 1, 2, . . . , N . Note that this is a discrete distribution, providing the probability for each n0 , given the observations and λ1 , λ2 .

1

Solutions To Problems of Chapter 15 15.1. Show that in the product

n Y

(1 − xi )

i=1

the number of cross product terms, x1 x2 · · · xk , 1 ≤ k ≤ n, for all possible combinations of x1 , · · · , xn is equal to 2n − n − 1.

Solution: The claim is true for n = 1. Also, the total number of nonproduct terms is equal to n, excluding the term 1.

Assume Qn−1that the claim is true for n − 1. That is, the number of products in i=1 (1 − xi ) is 2n−1 − (n − 1) − 1 = 2n−1 − n. Then the number of products in ! n n−1 Y Y (1 − xi ) = (1 − xi ) (1 − xn ) i=1

n−1

i=1

n

will be 2(2 − n) + n − 1 = 2 − n − 1, which proves the claim.

Take as examples: • (1 − x1 )(1 − x2 ) = 1 − x1 − x2 + x1 x2 : 22 − 2 − 1 = 1.

• (1 − x1 )(1 − x2 )(1 − x3 ) = (1 − x1 − x2 + x1 x2 ) (1 − x3 ) = 1 − x1 − x2 + x1 x2 − x3 + x1 x3 + x2 x3 − x1 x2 x3 : 23 − 3 − 1 = 4.

15.2. Prove that if a probability distribution p satisfies the Markov condition, as implied by a BN, then p is given as the product of the conditional distributions given the values of the parents.

Solution: We will prove it for the case of discrete probabilities, but similar is the proof for pdfs, as well. Let the involved random variables be x1 , x2 , · · · , xl . We order them in the so called ancestral ordering. That is, every descendant variable comes after its parents. Let x1 , x2 , · · · , xl be the resulting ordering. We will show that P (xl , xl−1 , · · · , x1 ) = P (xl |Pal )P (xl−1 |Pal−1 ) · · · P (x1 |Pa1 ).

For the proof, induction will be used. (a) It is true for the first node for which Pa1 = 0 P (x1 |Pa1 ) = P (x1 ) (b) Assume that it is true up to the value l − 1, i.e., P (xl−1 , · · · , x1 ) =

l−1 Y

i=1

P (xi |Pai )

2

(c) Using Bayes theorem we get P (xl , · · · x1 ) = =

P (xl |xl−1 , · · · x1 )P (xl−1 , · · · , x1 ) P (xl |xl−1 , · · · , x1 )

l−1 Y

P (xi |Pai ),

i=1

=

l Y

i=1

P (xl |Pal )

l−1 Y

P (xi |Pai ),

i=1

which proves the claim.

15.3. Show that if a probability distribution factorizes according to a Bayesian network structure, then it satisfies the Markov condition.

Solution: The proof is the same with that in the second part of the next problem.

15.4. Consider a DAG and associate each node with a random variable. Define for each node the conditional probability of the respective variable given the values of its parents. Show that the product of the conditional probabilities yield a valid joint probability and that the Markov condition is satisfied.

Solution: To prove that the product, P , is a valid probability, we have to prove that 0 ≤ P ≤ 1 and that X X P (xl |xl−1 , · · · , x1 ) = 1 ··· A := x1

xl

Ordering the variables in ancestral order, we have X X P (x1 |Pa1 )P (x2 |Pa2 ) · · · P (xl |Pal ) ··· A := =

x1

xl

X

X

···

X x1

=

1

···

X

X

P (xl |Pal )

xl

xl−1

x1

=

P (x1 |Pa1 ) · · · P (xl−1 |Pal−1 ) P (x1 |Pa1 ) · · · P (xl−2 |Pal−1 )

X

P (xl−1 |Pal−1 )

xl−1

xl−2

since the most right hand side summation always results in 1, due to the ancestral ordering. Also P (xl |Pal )P (xl−1 |Pal−1 ) · · · P (x1 |Pa1 ) ≥ 0 since it is the product of non negative terns. To prove that the Markov condition holds, consider a node xk . Order the variables so that all (and only) non-descendants precede xk in the ordering. Thus N Dk = {x1 , x2 , · · · , xk−1 }

3

and the set of descendants is given by Dk = {xk+1 , · · · , xl } Then we have P (xk |N Dk ) = = The numerator A becomes A

P (xk , N Dk ) P (N Dk ) P P Ql A xk+1 · · · i=1 P (xi |Pai ) l := P P Ql B xk · · · l i=1 P (xi |Pai )

= P (xk |Pak ) · · · P (x1 |Pa1 )

X

xk+1

= P (xk |Pak ) · · · P (x1 |P a1 )

···

X

P (xk+1 |Pak+1 ) · · · P (xl |Pal )

xl

since all nodes involved in the summation form a subgraph which is also DAG and according to the first part of the problem the product of the conditional probabilities is the joint pdf of the involved variables. Following the same reasoning, we can show that B = P (xk−1 |Pak−1 ) · · · P (x1 |P a1 ).

Hence, P (xk |N Dk ) = P (xk |Pak ), which proves the claim.

15.5. Consider the graph in Figure 1. The r.v. x has two possible outcomes, with probabilities P (x1 ) = 0.3 and P (x2 ) = 0.7. Variable y has three possible outcomes with conditional probabilities, P (y1 |x1 ) = 0.3, P (y2 |x1 ) = 0.2, P (y3 |x1 ) = 0.5, P (y1 |x2 ) = 0.1, P (y2 |x2 ) = 0.4, P (y3 |x2 ) = 0.5.

Finally, the conditional probabilities for z are

Figure 1: Graphical Model for Problem 15.5 P (z1 |y1 ) = 0.2, P (z2 |y1 ) = 0.8, P (z1 |y2 ) = 0.2, P (z2 |y2 ) = 0.8, P (z1 |y3 ) = 0.4, P (z2 |y3 ) = 0.6.

4

Show that this probability distribution, which factorizes over the graph, has x and z independent. However, x and z in the graph are not d– separated, since y is not instantiated.

Solution: It must be shown that P (z|x) = P (z) for all combinations of the values z1 , z2 , x1 , x2 . We will sketch the derivation. Let P (z1 |x1 ) = = = = =

P (z1 , x1 ) P (x1 ) P (z1 , y1 , x1 ) + P (z1 , y2 , x1 ) + P (z1 , y3 , x1 ) P (x1 ) P (y1 |x1 )P (z1 |y1 ) + P (y2 |x1 )P (z1 |y2 ) +P (y3 |x1 )P (z1 |y3 ) 0.06 + 0.04 + 0.20 0.3.

Also, P (z1 ) =

XX x

y

P (x, y, z1 ) =

XX x

P (z1 |y)P (y|x)P (x).

y

Carrying on the computations as before we obtain again 0.3. The same is true for all the other combinations.

Figure 2: DAG for Problem 15.6. Nodes in red have been instantiated.

15.6. Consider the DAG in Figure 2. Detect the d–separations and d–connections in the graph.

Solution: Nodes x11 and x12 are d–separated from all the others, due

5

to instantiation of x10 and the diverging connection. Then the rest of the uninstantiated nodes are d–connected. From x1 evidence flows from x3 and then to x4 (unblocked serial connection the other path is blocked by x2 ). Then it passes via x5 to x7 (converging node but the descendant of x5 is instantiated). Then to x8 and x9 (serial unblocked connection). Recall from the theory that a chain of nodes is active if there are no instantiated nodes in it, and whenever a converging connection along the chain is present, xi−1 → xi ← xi+1 , then a descendant node of xi is instantiated.

15.7. Consider the DAG of Figure 3. Detect the blanket of node x5 and verify that if all the nodes in the blanket are instantiated, then the node becomes d–separated from the rest of the nodes in the graph.

Solution: The blanket of x5 comprises the nodes x1 ,x2 , x8 ,x9 (children) and x4 ,x6 (share children with x5 ). Check that any chain that connects x5 with any node outside the blanket, once nodes in the blanket have been instantiated, is blocked.

Figure 3: The graph structure for Problem 15.7 15.8. In a linear Gaussian Bayesian network model, derive the mean values and the respective covariance matrices for each one of the variables in a recursive manner Solution: Since P (xi |Pai ) = N

xi

X

θik xk + θi0 , σi2

k:xk ∈Pai

!

,

we get that E[xi ] =

X

θik E[xk ] + θi0 .

k:xk ∈pai

Thus, we can arrange our variables in ancestral order and compute the mean values recursively.

6

Figure 4: Network for Problem 15.9 For the computation of the covariance matrix, recall that X xi = θik xk + θi0 + σi ni , k:xk ∈pai

where ni is a white (zero-mean) noise Gaussian source. Having adopted ancestral ordering and considering, say, xi , we are certain that xi is not involved in any one of the parent sets, Pa1 , Pa2 , · · · , Pai−1 . We will derive the covariances inductively.

• We know σi2 • Assume that we have computed all the covariances cov(xj , xr ), ∀r, j : j < i and r < i • Based on the previous computations, we will obtain all the covariances cov(xi , xj ), j ≤ i.

For j = 1, 2, · · · , i − 1 we have, E [(xi − E[xi ])(xj − E[xj ])] = " X E (xj − E[xj ])( θik xk + θi0 − k:xk ∈Pai

"

E (xj − E[xj ])( X

X

X

#

θik E[xk ] − θi0 + σi ni ) =

k:xk ∈Pai

#

θik (xk − E[xk ]) + σi ni ) =

k:xk ∈Pai

θik E [(xj − E[xj ])(xk − E[xk ])] + 0, i 6= j,

k:xk ∈Pai

and finally cov(xi , xj ) =

X

θik cov(xj , xk ) + δij σi2 , j ≤ i,

k:xk ∈Pai

which results in a hierarchical computation of the covariances.

15.9. Assuming the variables associated with the nodes of the Bayesian structure of Figure 4 to be Gaussian, find the respective mean values and covariances.

7

Solution: Following the recursive manner of computation, as suggested in Problem 15.8, we have E[x1 ] = θ10 , E[x2 ] = θ21 E[x1 ] + θ20 = θ21 θ10 + θ20 , E[x3 ] = θ32 E[x2 ] + θ30 , = θ32 (θ21 θ10 + θ20 ) + θ30 , = θ32 θ21 θ10 + θ32 θ20 + θ30 .

For the covariances we have, • cov(x1 , x1 ) = σ12 2 2 • cov(x2 , x1 ) = θ21 σ12 , cov(x2 , x2 ) = θ21 σ1 + σ22

• cov(x3 , x1 ) = θ32 cov(x2 , x1 ) = θ32 θ21 σ12 $ 2 2

• cov(x3 , x2 ) = θ32 cov(x2 , x2 ) = θ32 θ21 σ1 + σ22 $ 2 2

2 • cov(x3 , x3 ) = θ32 cov(x3 , x2 ) + σ32 = θ32 θ21 σ1 + σ22 + σ32 ,   σ12 θ21 σ12 θ32 θ21 σ12 $ 2 2 2 2 . θ$21 σ1 + σ22 θ$32 θ21 σ1 + σ22 • Σ =  θ21 σ12 2 2 2 2 2 θ32 θ21 σ12 θ32 θ21 σ1 + σ22 θ32 θ21 σ1 + σ22 + σ32

15.10. Prove that if p is a Gibbs distribution that factorizes over a MRF H, then H is an I-map for p.

Solution: Let X,Y ,Z be any three disjoint sets such as Z separates X and Y . Then we will show that for any x ∈ X and y ∈ Y , x ⊥ y|Z. Let us start by assuming that X ∪ Y ∪ Z = X , where X is the set of all points. Thus, any clique in H will be contained either in X ∪ Z or Y ∪ Z, since there are no edges connecting X and Y directly, without passing via Z. Then, since p factorizes over H, it will be given as a product of potential functions over cliques such as p (x1 , x2 , . . . , xl ) =

1 Z

Y

φci (xci )

ci ∈X∪Z

Y

φcj (xcj ).

cj ∈Z∪Y

Thus, we can write that p (x1 , x2 , . . . , xl ) =

1 f (X, Z) g (Y, Z) , Z

where the first factor contains variables in X,Z and the second one variables in Y ,Z. This implies that p(x, y|z) = p(x|z)p(y|z), i.e., independence among the variables in X and Y .

8

If X ∪ Y ∪ Z ⊂ X , let Ψ = X − (X ∪ Y ∪ Z). Partition Ψ into two disjoint groups, such as Z separates X ∪Ψ1 from Y ∪Ψ2 . Apply the previous result on X ∪ Ψ1 and Y ∪ Ψ2 , and then prove that ∀x ∈ X ∪ Ψ1 and ∀y ∈ Y ∪ Ψ2 , then x ⊥ y|Z. Then we readily have the claim.

15.11. Show that if H is the moral graph that results from moralization of a BN structure then I(H) ⊆ I(G) Solution: In an undirected graph, independencies are guaranteed by respective separations among nodes, given a set of other nodes. Take any node in the moral graph. The only separations that can be guaranteed of this node from any other node are those imposed by the nodes comprising the respective Markov blanket of the node in the original DAG. However, these independencies, is a subset of the set of d–separations of the original DAG. This proves the claim.

15.12. Consider a Bayesian network structure and a probability distribution p.

Then show that if I(G) ⊆ I(p), then p factorizes over G. Solution: The proof consists of a combination of two theorems. We know that, given a BN structure the set of all d–separations does include the local independencies, on which the Bayesian structure is built upon. Since I(G) ⊆ I(p), then these local independencies imply conditional independencies on p. Hence from Theorem 15.1, we have that p factorizes over G. 15.13. Show that in a undirected chain graphical model, the marginal probability P (xj ) of a node, xj , is given by 1 µf (xj )µb (xj ), Z where µf (xj ) and µb (xj ) are the received by the node forward and backward messages.

P (xj ) =

Solution: We have that X X X l−1 Y 1 X P (xj ) = ··· ··· ψi,i+1 (xi , xi+1 ), Z x x x x i=1 1

j−1

j+1

l

assuming l nodes, or P (xj ) =

X l−1 X j−1 Y X Y 1 X ··· ψi,i+1 (xi , xi+1 ) ··· ψi,i+1 (xi , xi+1 ) Z x x x x i=j i=1 1

=

j−1

1 µf (xj )µb (xj ), Z

j+1

l

9

by the respective definitions.

15.14. Show that the joint distribution of two neighboring nodes in a undirected chain graphical model is given by P (xj , xj+1 ) =

1 µf (xj )ψj,j+1 (xj , xj+1 )µb (xj+1 ).

Z

Solution: Following the same steps as for the problem 15.13, we have X X X l−1 Y 1 X ··· ··· ψi,i+1 (xi , xi+1 ) Z x x x x i=1

P (xj , xj+1 ) =

j−1

1

1 X ··· Z x x

=

1

X

···

xj+2

j+2

X j−1 Y

l

ψi,i+1 (xi , xi+1 )ψj,j+1 (xj , xj+1 )

j−1 i=1

X l−1 Y

ψi,i+1 (xi , xi+1 )

xl i=j+1

1 µf (xj )ψj,j+1 (xj , xj+1 )µb (xj+1 ).

Z

=

15.15. Using Figure 15.26, prove that if there is a second message passing, starting from x, towards the leaves, then any node will have the available information for the computation of the respective marginals.

Solution: Let us consider node x4 . The proof is similar for all the other nodes. In order to compute P (x4 ), we need µfd →x4 (x4 ), µfe →x4 (x4 ) and µfa →x4 (x4 ), as Equation (15.55) suggests. Messages µfd →x4 (x4 ) and µfe →x4 (x4 ) have already been computed during the first pass. When message passing starts from x1 , then x1 will propagate: µx1 →fa (x1 ) = µfg →x1 (x1 )µfh →x1 (x1 ) to factor node A. Although not of interest here, note that x1 will also propagate µx1 →fg (x1 ) = µfa →x1 (x1 )µfh →x1 (x1 ) to factor node G and µx1 →fh (x1 ) = µfg →x1 (x1 )µfa →x1 (x1 ) to node H. Thus, factor node A will propagate messages to x2 , x3 and x4 and so on.

Focusing on x4 , the received message will be XXX µfa →x4 (x4 ) = fa (x1 , x2 , x3 , x4 )µx2 →fa (x2 )µx3 →fa (x3 ) x1

x2

x3

µx1 →fa (x1 ).

10

However, µx2 →fa (x2 ) and µx3 →fa (x3 ) have been computed during the first pass. Hence, all information for P (x4 ) is now available. The same is true for all nodes.

15.16. Consider the tree graph of Figure 15.26. Compute the marginal probability P (x1 , x2 , x3 , x4 ).

Solution: Let VA′ be all the nodes of VA excluding x2 , x3 and x4 . Then following similar reasoning as for (15.46) 1 X X X P (x1 , x2 , x3 , x4 ) = ΨA (x1 , x2 , x3 , x4 , xA′ ) · Z x∈VA′ x∈VF x∈VG

ΨH (x1 , xH )ΨG (x1 , xG ),

or P (x1 , x2 , x3 , x4 ) =

X 1 X ΨA (x1 , x2 , x3 , x4 , xA′ ) ΨH (x1 , xH ) Z xA′ ∈VA′ xH ∈VH X ΨG (x1 , xG ).

xG ∈VG

Then, following exactly the same steps as the ones in order to prove Equation (15.54), we have that X ΨF (x1 , xH ) = µfh →x1 (x1 ), xH ∈VH

X

ΨG (x1 , xG ) =

µfg →x1 (x1 ).

xG ∈VG

Furthermore, note that µx1 →fa (x1 ) = µfh →x1 (x1 )µfg →x1 (x1 ).

Also, due to (15.48) (by neglecting the sum over x2 , x3 , x4 ) X ΨA (x1 , x2 , x3 , x4 , xA′ ) = fa (x1 , x2 , x3 , x4 )µx2 →fa (x2 ) xA′ ∈VA′

µx3 →fa (x3 )µx4 →fa (x4 ).

Thus, finally we obtain P (x1 , x2 , x3 , x4 )

=

1 fa (x1 , x2 , x3 , x4 ) Z µx1 →fa (x1 )µx2 →fa (x2 )µx3 →fa (x3 )µx4 →fa (x4 ).

15.17. Repeat the message procedure to find the optimal combination of variables for Example 15.4 using the logarithmic version and the max–sum algorithm

1

Solutions To Problems of Chapter 16 16.1. Prove that an undirected graph is triangulated if and only if its cliques can be organized into a join tree.

Solution: The proof follows [Jens 01]. a) Let the cliques be organized in a join tree. Let V be a leaf clique and let F be its unique neighbor. From the definition of a join tree and the running intersection property, if V has common nodes with any other clique, these nodes must also belong to F . Hence, there must be at least one node x ∈ V , such that x ∈ / F , otherwise V ⊂ F . Then we can eliminate this variable node, x, without adding any fill-ins, since x is a part of a single clique, namely V . We keep removing variable nodes such as x. Once we have completed it, we left with a graph, without V , which is also a join tree. Then the process continues and in this way we have generated a perfect elimination sequence. Hence the graph is triangulated. b) Assume a triangulated graph. Then we will obtain a join tree by construction. We follow the algorithmic procedure of subsection 16.2.1. We must first prove that the generated structure by such a construction is a join tree. First, it is a tree, since every clique has at most one parent, and there cannot be multiple paths. To prove that it is a join tree, we have to show the running intersection property. Let Vi and Vj (i < j) be two clique nodes sharing a set of variable nodes, X. We have proved that it is a tree, hence there is a unique path from Vi to Vj . The nodes in X cannot be eliminated from Vi (since they are shared with other cliques) and must be part of Si and, by construction, part of Vi ’s parent, say, Vk . IF j = k, the proof is complete. If not, we continue the reasoning as before, with the min (k, j) clique.

16.2. For the graph of Figure 16.3a give all possible perfect elimination sequences and draw the resulting sequence of graphs.

Solution: Alternative perfect elimination sequences are: x5 , x6 , x3 , x1 , x2 , x4 , or x1 , x5 , x6 , x3 , x2 , x4 , or x6 , x1 , x3 , x5 , x2 , x4 .

The sequence of the resulting graphs for the first one, is shown in Figure 1 16.3. Derive the formulas for the marginal probabilities of the variables in a) a clique node and b) in a separator node in a junction tree.

2

(a)

(b)

(c)

(d)

(e)

Figure 1: The resulting graphs associated with the elimination sequence: x5 , x6 , x3 , x1 , x2 , x4 , for Problem 16.2.

Solution: The proof is direct consequence of the definition of messages.

Let ψV (xV ) be a clique of variable (nodes) which comprises xV . Some of them are not shared by any other clique, and some of them are. The shared ones will appear in the separators with whom V is connected to.

We have 1 X Y P (xV ) = ψc (xc ), Z c x\xV

where x is the set of all variables and c runs over all cliques. We can rewrite as X Y 1 P (xV ) = ψV (xV ) ψc (xc ).

Z x\xV c\V

However, the summation over the products is nothing but the product of the messages sent by the separators (which V is connected to) to V .

Indeed, the message passed to a separator is the marginalization over all variables (except those in the separator) in the products of the factors along the path from a leaf to the separator. Similar arguments hold for the joint probability of the variables in a separator.

16.4. Prove that in a junction tree the joint pdf of the variables is given by (16.8).

Solution: We will show it first for a junction tree comprising two clique

3

Figure 2: The graph structures for Problem 16.4. The arrows indicate two possible message flows (black and red) in different directions, which are required by the message passing algorithm. nodes and one separator and then three clique nodes. Then the generalization is straightforward. The respective graphs are shown in Figure 2. a) We have that 1 (1) P (x) = ψ1 (xv1 )ψ2 (xv2 ) Z However, from (16.6) we get P (xv1 ) =

1 ψ1 (xv1 )µs→v1 (xs ), Z

(2)

P (xv2 ) =

1 ψ2 (xv2 )µs→v2 (xs ).

Z

(3)

1 µv →s (xs )µv2 →s (xs ) Z 1

(4)

and

Also, from (16.7) we get P (xs ) =

Moreover, take into account that µv1 →s (xs ) = µs→v2 (xs ) and µv2 →s (xs ) = µs→v1 (xs ).

(5)

Combining Equations (1)–(5) we obtain the result. The reason Z is eliminated, is that we have one separator and two clique nodes. b) For the three clique nodes we have 1 ψ1 (xv1 )ψ2 (xv2 )ψ3 (xv3 ), Z

(6)

1 ψ1 (xvi )µs→vi (xs ), i = 1, 2, 3.

Z

(7)

P (x) = and similarly P (xvi ) =

4

Also,

1 µv →s (xs )µv2 →s (xs )µv3 →s (xs ) Z 1

P (xs ) =

(8)

where µs→v1 (xs ) = µv3 →s (xs )µv2 →s (xs ),

(9)

µs→v2 (xs ) = µv1 →s (xs )µv3 →s (xs ),

(10)

µs→v3 (xs ) = µv2 →s (xs )µv1 →s (xs ).

(11)

Combining (6)–(11), we obtain

1 ZP (xv1 ) ZP (xv2 ) P (x) = Z µv3 →s (xs )µv2 →s (xs ) µv1 →s (xs )µv3 →s (xs )

ZP (xv3 ) µv2 →s (xs )µv1 →s (xs ) P (xv1 )P (xv2 )P (xv3 ) = .

(Ps (xs ))2 16.5. Show that obtaining the marginal over a single variable is independent of which one from the cliques/separators nodes, which contain the variable, the marginalization is performed.

Hint : Prove it for the case of two neighboring clique nodes in the junction tree.

Solution: We will consider that the two neighboring nodes V1 , V2 share one variable x, which also comprises their common separator, as shown in Figure 3. Assume for simplicity that V1 is a leaf node, without harming generality. We have that X P (x) = ψ1 (xV1 , x)µV2 →S (x) xV1

=

X

ψ1 (xV1 , x)

xV1

X xV2

ψ2 (xV2 , x)

Y

µs→V2 (xs ),

s

where xs ⊂ (xV2 , x). Now let Y X µs→V2 (xs )µS→V2 (x) ψ2 (xV2 , x) P ′ (x) = =

xV2

s

X

Y

ψ2 (xV2 , x)

xV2

= P (x).

s



µs→V2 (xs ) 

X xV1



ψ1 (xV1 , x)

The proof is similar if one considers a separator node to compute the marginal.

5

Figure 3: The set up for Problem 16.5

(a)

(b)

Figure 4: a) The graph for the Problem 16.6. b) A triangulated graph.

16.6. Consider the graph in Figure 4a. Obtain a triangulated version of it.

Solution: The triangulated version is shown in Figure 4b 16.7. Consider the Bayesian network structure given in Figure 5. Obtain a equivalent join tree.

Solution: First, we moralize the network, and the structure of Figure 6 results. Then we peel it off to form cliques and separators; as shown in Figure 7. Note that, at each time, nodes that belong to exclusively one clique can be removed. The resulting join tree is shown in Figure 8.

16.8. Consider the random variables A, B, C, D, E, F, G, H, I, J and assume that the joint distribution is given by the product of the following potential functions p=

1 ψ1 (A, B, C, D)ψ2 (B, E, D)ψ3 (E, D, F, I)ψ4 (C, D, G)ψ5 (C, H, G, J) Z

Construct an undirected graphical model on which the previous joint probability factorizes and in the sequence derive an equivalent junction tree.

Solution: The corresponding graph is shown in Figure 6. The nodes are eliminated in the following sequence: F, I, H, J, G, E, D, C, B, A with the following associated cliques and separators V2 = (D, E, F, I), S2 = (D, E), V4 = (C, H, G, J), S4 = (C, G), V5 = (C, G, D), S5 = (C, D), V6 =

6

Figure 5: The Bayesian network structure for Problem 16.7

Figure 6: The moralized counterpart of the graph in 5

Figure 7: The set of cliques and separators after the sequence of nodes’ elimination from Figure 6.

(B, E, D), S6 = (B, D), V10 = 9A, B, C, D). The resulting junction tree is shown in Figure 10

7

Figure 8: The Join tree for Problem 16.7.

Figure 9: The undirected graph on which the joint Probability of Problem 16.8 factorizes.

16.9. Prove that the function g(x) = 1 − exp(−x), x > 0, is log-concave.

Solution: Let f (x) = ln(1 − exp(−x)).

Taking the derivative we have f ′ (x) =

exp(−x) , 1 − exp(−x)

8

Figure 10: The resulting junction tree for Problem 16.8. and the second derivative f ′′ (x)

− exp(−x)(1 − exp(−x)) − exp(−x) exp(−x) (1 − exp(−x))2 − exp(−x) < 0, (1 − exp(−x))2

= =

hence the function is concave.

16.10. Derive the conjugate function of f (x) = ln(1 − exp(−x)).

Solution: By the definition and since f (x) is concave we have

f ∗ (ξ) = min ξx − ln(1 − exp(−x) . x

Taking the derivative with respect to x and equating to zero, we have ξ−

exp(−x) 1 − exp(−x)

=

exp(−x)

=

x =

0⇒ ξ or 1+ξ − ln ξ + ln(1 + ξ).

Also, 1 − exp(−x) = 1 −

ξ 1 = .

1+ξ 1+ξ

9

Hence, f ∗ (ξ)

1 1+ξ = −ξ ln ξ + (1 + ξ) ln(1 + ξ).

= ξ(ln(1 + ξ) − ln ξ) − ln

16.11. Show that minimizing the bound in Eq. (16.12) is a convex optimization task.

Solution: Instead of minimizing the bound itself we can minimize its logarithm, i.e., X −f ∗ (ξi ) + ξi di θij . j∈Pai However, f ∗ (ξ) is concave, (Section 13.8) being the result of point-wise minimization of a set of affine functions. Hence −f ∗ (ξi ) is convex. Since the cost is a sum of convex functions, it is also convex.

16.12. Show that the function 1 + exp(−x), x ∈ R is log-convex and derive the respective conjugate one.

Solution:

f (x) = ln 1 + exp(−x) ,

thus,

f ′ (x) = −

exp(−x) , 1 + exp(−x)

and f ′′ (x)

=

− exp(−x)(1 + exp(−x)) + exp(−x) exp(−x) (1 + exp(−x))2 exp(−x) > 0.

(1 + exp(−x))2

−

=

The conjugate function is

f (ξ) = max ξx − ln(1 + exp(−x) , ∗

x

or taking the derivative w.r. to x and equating to zero, ξ− or x = − ln

− exp(−x) = 0, 1 + exp(−x)

−ξ 1 and 1 + exp(−x) = .

1+ξ 1+ξ

Hence, f ∗ (ξ) = =

1+ξ 1 − ln −ξ 1+ξ (ξ + 1) ln(ξ + 1) − ξ ln(−ξ), − 1 < ξ < 0.

ξ ln

10

16.13. Derive the KL divergence between P (X l |X ) and Q(X l ) for the mean field Boltzmann machine and obtain the respective variational l parameters.

Solution: The KL divergence is given by   KL =

−

X

xi ∈X l

+

ln Z̃ +





X  X       Q(X l |X ; µ) ln  θij xi xj + θ̃i0 xi  exp −   i:xi ∈X l

X

j>i j:xj ∈X l

Q(X l |X ; µ) ln Q(X l |X ; µ)

xi ∈X l

=

X

xi ∈X l

+





 X  X    Q(X l |X ; µ)  θij xi xj + θ̃i0 xi     i:xi ∈X l

ln Z̃ +

X

l

Q(X |X ; µ) ln

xi ∈X l

=

X

i:xi ∈X l

+

X



!

X

j>i j:xj ∈X l

 

X

xi ∈X l

j>i j:xj ∈X l

!

Y

µxi i (1 − µi )(1−xi )

i:xi ∈X l

Q(X l |X ; µ)θij xi xj  +

l

Q(X |X ; µ) (xi ln µi + (1 − xi ) ln(1 − xi ))

xi ∈X l

=

X

i:xi ∈X l

+



!

X

X

" Q(X l |X ; µ)θ̃i0 xi

xi ∈X l

"

+ ln Z̃ "

θij µi µj + θ̃i0 µi + µi ln µi + (1 − µi ) ln(1 − µi )

j>i j:xj ∈X l

ln Z̃,

where we assumed that under Q, xi and xj are independent. Also, due to the binomial assumption of each individual xi , we have that E[xi ] = µi .

Taking the derivative with respect to each µi and setting it equal to zero we get ∂KL X θij µj + θ̃i0 + ln µi + 1 − ln(1 − µi ) − 1 = 0, = ∂µi j6=i

or µi = where

1 = σ(z), 1 + exp(−z)

  X z = − θij µj + θ̃i0  . j6=i

11

16.14. Given a distribution in the exponential family,

p(x) = exp θ T u(x) − A(θ) ,

show that A(θ) generates the respective mean parameters which define the exponential family ∂A(θ) = E[ui (x)] = µi . ∂θi Also, show that A(θ) is a convex function.

Solution: By the definition of A(θ) we have Z

∂A(θ) ∂ = ln exp θ T u(x) dx, ∂θi ∂θi or R

ui (x) exp θ T u(x) dx ∂A(θ) R = ∂θi exp (θ T u(x)) dx

R ui (x) exp θ T u(x) − A(θ) dx R = exp (θ T u(x) − A(θ)) dx Z = ui (x)p(x; θ)dx = E[ui (x)].

For the convexity we have R

∂ ui (x) exp θ T u(x) exp − A(θ) dx ∂2A = ∂θj ∂θi θj Z = ui (x)uj (x)p(x)dx − E[ui (x)]E[uj (x)]

= E ui (x) − E[ui (x)] uj (x) − E[uj (x)] , and finally,

T ∂2A = Σ := E u(x) − E[u(x)] (u(x) − E[u(x)] , 2 ∂θ

(12)

which is a nonnegative matrix. Hence convexity has been shown.

16.15. Show that the conjugate function of A(θ), associated with an exponential distribution, such as that in Problem 16.14, is the corresponding negative entropy function. Moreover, if µ and θ(µ) are doubly coupled, then µ = E[u(x)], where E[·] is with respect to p(x; θ(µ)).

Solution:

A∗ (µ) = max θ T µ − A(θ) , θ

12

or θ(µ) : µ = ∇A(θ) = E[u(x)], where we used the result of Problem 16.14.

Thus, the negative entropy is given by Z H = p(x; θ(µ)) ln p(x; θ(µ))dx

= θ T (µ)E[u(x)] − A(θ) = A∗ (µ).

16.16. Derive a recursion for updating γ(xn ) in HMMs independent of β(xn ) Solution: We know from the text and the respective definition that X γ(xn ) = P (xn |Y ) = P (xn , xn+1 |Y ) xn+1

=

X

P (xn |xn+1 , Y[1:n] , Y[n+1,N ] )P (xn+1 |Y ).

xn+1

However, by the d–separation properties of the graph representing an HMM, P (xn |xn+1 , Y[1:n] , Y[n+1,N ] ) = P (xn |xn+1 , Y[1:n] ).

Thus, we can write γ(xn ) =

X

P (xn |xn+1 , Y[1:n] )γ(xn+1 ).

xn+1

Also, P (xn |xn+1 , Y[1:n] ) ∝ =

P (xn , xn+1 |Y[1:n] ) P (xn+1 |xn )P (xn |Y[1:n] ),

where again we have used that by the d–separation properties of the Bayesian network structure P (xn+1 |xn , Y[1:n] ) = P (xn+1 |xn ).

Thus, since P (xn |Y[1:n] ) ∝ α(xn ), we can write P (xn |xn+1 , Y[1:n] ) ∝ P (xn+1 |xn )α(xn ), and the proportionality constant is found by normalization. Note that γ(xN ) ∝ α(xN ).

Thus, one computes α(xn ) and then goes backwards to obtain γ(xn ), n = N − 1, N − 2, . . . , 1.

13

16.17. Derive an efficient scheme for prediction in HMM models; that is, to obtain p(yN +1 |Y ) where Y = {y1 , y2 , . . . , yN }.

Solution: We have that X

p(yN +1 |Y ) =

p(yN +1 , xN +1 |Y )

xN +1

X

=

p(yN +1 |xN +1 )P (xN +1 |Y )

xN +1

X

=

p(yN +1 |xN +1 )

xN +1

X

P (xN +1 , xN |Y ).

xN

Note that the second sum is over ξ(xN +1 , xN ). However, to obtain ξ(·, ·) we need two passes. This is not necessary if prediction is our focus. Let us carry on the previous analysis to obtain, p(yN +1 |Y )

=

X

p(yN +1 |xN +1 )

X

p(yN +1 |xN +1 )

xN +1

=

X

P (xN +1 |xN )P (xN |Y )

X

P (xN +1 |xN )

xN

xN +1

xN

p(xN , Y ) p(Y )

X 1 X P (xN +1 |xN )α(xN ), p(yN +1 |xN +1 ) p(Y ) x x

=

N +1

N

where p(Y ) =

X

α(xN ).

xN

Thus for prediction, the forward message passing is sufficient.

16.18. Prove the estimation formulas for the probabilities Pk , k = 1, 2, . . . , K, and Pij , i, j = 1, 2, . . . , K, in the context of the forward-backward algorithm for training HMM. Solution: From the text, we have Q(Θ, Θ(t) ) =

K X

γ(x1,k = 1; Θ(t) ) ln Pk

k=1

+

N X K X K X

ξ(xn−1,j = 1, xn,i = 1; Θ(t) ) ln Pij

n=2 i=1 j=1

+

constant,

where constant involves parameters independent of Pk , Pij . Since Pk and Pij are decoupled, they can be solved independently. We will solve for

14

Pij . The Lagrangian becomes L(Pij , λ) =

N X K X K X

ξ(xn−1,j = 1, xn,i = 1; Θ(t) ) ln Pij −λ

n=2 i=1 j=1

!K X

"

Pkj − 1 .

k=1

Taking the derivative with respect to Pij and equating to zero we get, N 1 X ξ(xn−1,j = 1, xn,i = 1; Θ(t) ) = λ, Pij n=2

and plugging into the constraint, in order to compute λ, we obtain PN ξ(xn−1,j = 1, xn,i = 1; Θ(t) ) (t+1) Pij = PN n=2 .

PK (t) n=2 k=1 ξ(xn−1,j = 1, xn,k = 1; Θ ) (t+1)

The reestimation rule for the probabilities Pk way.

is obtained in a similar

16.19. Consider the Gaussian Bayesian network of Section 15.3.5 defined by the local conditional pdfs   X p(xi |Pai ) = N xi | θik xk + θi0 , σ 2  , i = 1, 2, . . . , l. k:xk ∈Pai

Assume a set of N observations, xi (n), n = 1, 2, . . . , N , i = 1, 2, . . . , l, and derive a maximum likelihood estimate of the parameters θ; assume the common variance σ 2 to be known.

Solution: The likelihood is given by L(X; θ) =

N X l X

n=1 i=1

1 T 2 − 2 (xi (n) − θi hi (n)) , 2σ

where hi (n) is the vector with values of xk : k ∈ Pai , extended by one dimension to include 1 as its last element, so that θi includes θi0 as well.

That is, θi = [θi1 , . . . , θiK , θi0 ]T , where K is the number of parents of xi and hi = [hi1 , hi2 , . . . , hiK , 1]T . Since the log-likelihood is obviously decomposable, we will optimize each factor

N X 1 T 2 li (X; θi ) = − 2 (xi (n) − θi hi (n)) , i = 1, 2, . . . , l, 2σ n=1 individually. Taking the gradient with respect to θi and equating to zero, we obtain N N X X T (hi (n)hi (n))θi = xi (n)hi (n), n=1

n=1

15

or θ̂i =

! N X

n=1

"−1 ! N X

hi (n)hTi (n)

n=1

"

xi (n)hi (n) .

In other words, θ̂i is nothing else but the LS solution if one tries to predict xi (n) via the values of its parents.

1

Solutions To Problems of Chapter 17 17.1. Let

Z µ := E[f (x)] =

f (x)p(x)dx

and q(x) be the proposal distribution. Show that if w(x) := and

p(x) , q(x)

N

µ̂ =

1 X w(xi )f (xi ), N i=1

then the variance Z 2

h i 1 f (x)p2 (x) 2 2 2 σf = E (µ̂ − E [µ̂]) = dx − µ .

N q(x) Observe that if f 2 (x)p2 (x) goes to zero slower than q(x), then for fixed N σf2 −→ ∞.

Solution: Following the solution of Problem 14.6, using p in place of φ, we have that µ̂ is unbiased, hence E[µ̂] = µ. where E[·] is with respect to q. Hence,

E (µ̂ − µ)2 = E[(µ̂)2 ] + µ2 − 2µ E[µ̂] = E[(µ̂)2 ] − µ2 .

However  E[(µ̂)2 ]

= E

=

N 1 X

N i=1 

!2  w(xi )f (xi )

 

N N X N X X  1  2 2  E[w(xi )f (xi )w(xj )f (xj )] E[w (x )f (x )] + i i   2 N i=1 i=1 j=1 i6=j

=

1 N2

Z N

2

2

p (x)f (x) 2 2 q(x)dx + (N − N ) (E[w(x )f (x )]) , i i q 2 (x)

due to independence of xi and xj . Thus, Z 2 !

Z 2 1 p (x)f 2 (x) p(x)f (x) 2 2 E[(µ̂) ] = N dx + (N − N ) q(x)dx N2 q(x) q(x) Z 2

1 p (x)f 2 (x) = dx + (N − 1)µ2 N q(x)

2

Hence σf2 =

1 N

f 2 (x)p2 (x) dx − µ2 . q(x)

Z

17.2. In Importance sampling, with weights defined as φ(x) , q(x)

w(x) = where

1 φ(x), Z we know from Problem 14.6 that the estimator p(x) =

N

Ẑ =

1 X w(xi ).

N i=1

is an unbiased estimator of the normalizing constant, Z. Show that the respective variance is given by

Z 2 Z2 p (x) dx − 1 . var[Ẑ] = N q(x)

Solution: The proof proceeds in exactly the same line as Problem 17.1.

This is easily checked out if in the previous problem one sets f (x) = 1 and Zw(x) instead of w(x).

17.3. Show that using resampling in importance sampling, then as the number of particles tends to infinity the approximating, by the respective discrete random measure, distribution, p̄, tends to the true (desired) one, p.

Hint: Consider the one-dimensional case.

Solution: Recall from the text that after resampling each resampled particle appears N (i) times out of N . Hence for large enough N , every particle (i) has a probability of occurrence equal to NN ≈ p(i) = w(i) . Thus we can write, P (x ≤ z)

=

X

W (i) =

i:xi ≤z

=

1 PN

φ(xi )/q(xi ) PN i=1 φ(xi )/q(xi ) i:xi ≤z X

N X

φ(xi ) i=1 q(xi ) i=1

χ[0,z] (xi )

φ(xi ) . q(xi )

where χ[·,·] (·) is the characteristic function, defined in the text. For large values of N , and assuming all the required regularity conditions are valid,

3

the summation tends to R P (x ≤ z)

=

χ[0,z] (x) φ(x) q(x) q(x)dx R φ(x) q(x) q(x)dx

Z z =

p(x)dx, −∞

which is the cumulative desired distribution and the claim is proved.

17.4. Show that in sequential importance sampling, the proposal distribution that minimizes the variance of the weight at time n, conditioned on x1:n−1 , is given by qnopt (xn |x1:n−1 ) = pn (xn |x1:n−1 )

Solution: Note from the text that we have w(x1:n )

φn (x1:n ) φn−1 (x1:n−1 )qn (xn |x1:n−1 ) φn (x1:n ) := a , qn (xn |x1:n−1 ) =

w(x1:n−1 )

since we assume the past (depending on x1:n−1 ) is known and fixed. Then, Z E[w(x1:n )] = a φn (x1:n )dxn := aZn pn (x1:n−1 ), from the respective definition pn (x1:n ) =

φn (x1:n ) , Zn

and where all means are with respect to the unknown qn (xn |x0:n−1 ).

Hence, var[w(x1:n )] = E[w2 (x1:n )] − a2 Zn2 p2n (x1:n−1 ).

Also, 2

E[w (x1:n )]

φ2n (x1:n ) qn (xn |x1:n−1 )dxn qn2 (xn |x1:n−1 ) Z φ2n (x1:n ) dxn . = a2 qn (xn |x1:n−1 ) = a

2

Z

Using a) calculus of variations, b) the constraint Z qn (xn |x1:n−1 )dxn = 1

(1)

4

and c) Lagrange multipliers, we obtain φ2n (x1:n ) = λ⇒ qn2 (xn |x1:n−1 ) qn (xn |x1:n−1 ) = λφn (x1:n ) ⇒ φn (x1:n ) φn (x1:n ) qn (xn |x1:n−1 ) = R = .

Zn pn (x1:n−1 ) φn (x1:n )dxn Then, E[w2 (x1:n )]

= a2 Zn pn (x1:n−1 )

Z φn (x1:n )dxn

= a2 Zn2 p2n (x1:n−1 ), 1:n ) leading to zero variance. Note that one could put qn (xn |x1:n−1 ) = Znφpnn(x (x1:n−1 ) straight away and since this makes the variance zero, it is the optimal value without having to use calculus of variation. Thus,

qn (xn |x1:n−1 ) =

φn (x1:n ) pn (x1:n ) = = pn (xn |x1:n−1 ).

Zn pn (x1:n−1 ) pn (x1:n−1 )

17.5. In a sequential importance sampling task, let pn (x1:n )

n Y

=

φn (x1:n )

k=1 n Y

=

k=1

N (xk |0, 1) 2 x exp − k 2

and let the proposal distribution be qn (x1:n ) =

n Y

N (xk |0, σ 2 ).

k=1 n

Let the estimator of Zn = (2π) 2 , be N

Ẑn =

1 X (i) w(x1:n ).

N i=1

Show that the variance of the estimator is given by: !

n2 Zn2 σ4 var[Ẑn ] = −1 .

N 2σ 2 − 1 Observe that for σ 2 > 1/2, which is the range of values for which the above formula makes sense and guarantees a finite value for the variance,

5

the variance exhibits an exponential increase with respect to n. To keep the variance small one has to make N very large; that is, to generate a very large number of particles ([15]).

Solution: From the Problem 17.2 we have that Z 2

pn (x1:n ) Z2 dx1:n − 1 . var[Ẑn ] = n N qn (x1:n ) However, n

p2n (x1:n ) =

1 Y exp −x2k 2 Zn k=1

and

n Y x2k 1 exp − 2 . qn (x1:n ) = n n 2σ (2π) 2 (σ 2 ) 2 k=1

Hence, Z

p2n (x1:n ) dx1:n qn (x1:n )

=

=

n n n Z (2π) 2 (σ 2 ) 2 Y 2σ 2 − 1 2 exp − xk dxk Zn2 2σ 2 k=1

n2 n2 σ4 (2π)n+1 σ4 = .

Zn2 2σ 2 − 1 2σ 2 − 1

or Z2 var[Ẑn ] = n N

σ4 2σ 2 − 1

n2

! −1 .

17.6. Prove that the use of the optimal proposal distribution in particle filtering leads to wn (x1:n ) = wn−1 (x1:n−1 )p(yn |xn−1 ).

Solution: Recall that the general update recursion is given by wn (x1:n ) = wn−1 (x1:n−1 )

p(yn |xn )p(xn |xn−1 ) . qn (xn |x1:n−1 , y1:n )

(2)

We know that the optimal proposal distribution is given by q(xn |x1:n−1 , y1:n )

= p(xn |xn−1 , yn ) p(yn |xn , xn−1 )p(xn |xn−1 ) = p(yn |xn−1 ) p(yn |xn )p(xn |xn−1 ) = p(yn |xn−1 )

Combining (2) and (3), proves the claim.

(3)

1

Solutions To Problems of Chapter 18 18.1. Prove that the perceptron algorithm, in its pattern-by-pattern mode of operation, converges in a finite number of iteration steps. Assume that θ (0) = 0.

Hint: Note that since classes are assumed to be linearly separable, there exists a normalized hyperplane, θ∗ , and a γ > 0, so that γ ≤ yn θ∗T xn , n = 1, 2, . . . , N, where, yn is the respective label, being +1 for ω1 and −1 for ω2 . By the term normalized hyperplane, we mean that, θ∗T = [θ̂∗ , θ0∗ ]T , with ||θ̂∗ || = 1.

In this case, yn θ∗T xn is the distance of xn from the hyperplane θ∗ ([164]).

Solution: Let us assume that we are currently at iteration i. This means that so far i − 1 update corrections have been performed by the algorithm. The ith update will be performed only if the current sample, x(i) , is misclassified, i.e., only if y(i) xT(i) θ (i−1) ≤ 0, and the update will be θ (i) = θ (i−1) + µy(i) x(i) .

(1)

R := max ||xn ||,

(2)

Let us now define, n

and let θ∗ be a normalized solution. Then from (1) we obtain, θ∗T θ (i)

= θ∗T θ (i−1) + µy(i) θ∗T x(i) ≥ θ∗T θ (i−1) + µγ,

which by induction and starting from a zero initial condition, implies that θ∗T θ (i) ≥ iµγ.

(3)

Also from (1), we have that ||θ (i) ||2

=

||θ (i−1) ||2 + 2µy(i) xT(i) θ (i−1) + 2 µ2 y(i) ||x(i) ||2

≤

2 ||θ (i−1) ||2 + µ2 y(i) ||x(i) ||2

≤

||θ (i−1) ||2 + µ2 R2 ,

(4)

2

which by induction leads to ||θ (i) ||2 ≤ iµ2 R2 .

(5)

Thus combining (3) and (5) and using the Schwartz inequality, we obtain √ ||θ∗ || iµR ≥ ||θ∗ ||||θ (i) || ≥ θ∗T θ (i) ≥ iµγ, (6) or

i≤

R γ

2

||θ∗ ||2 ,

(7)

which proves that i will remain finite.

18.2. The derivative of the sigmoid functions has been computed in Problem 7.6. Compute the derivative of the hyperbolic tangent activation function and show that it is equal to,

f 0 (z) = ac 1 − f 2 (z) .

Solution: We have that, f (z) = a tanh(cz) ⇒ f 0 (z) = (ac) cosh−2 cz) or f 0 (z) = ac(1 − tanh2 (cz)) or f 0 (z) = ac(1 − f 2 (z)) 18.3. Show that the effect of the momentum term in the gradient descent backpropagation scheme is to effectively increase the learning convergence rate of the algorithm.

Hint: Assume that the gradient is approximately constant over I successive iterations.

Solution Let i denote the current iteration, i.e., ∆θjr (i) = a∆θjr (i − 1) − µg,

(8)

since the gradient has been assumed to be constant. Repeating the iteration over I successive steps, we get ∆θjr (I) = −µ

I−1 X

ai g + aI ∆θjt (0).

(9)

i=0

Since a is less than one, the rightmost term in the previous recursion tends to zero and the remaining term, for large values of I, becomes ∆θjr (I) ≈ −µ(1 + a + a2 + . . . + aI−1 )g ≈ −

µ g.

1−a

(10)

3

18.4. Show that if a) the activation function is the hyperbolic tangent, b) the input variables are normalized to zero mean and unit variance, then in order to guarantee that all the outputs of the neurons are zero mean and unit variance, the weights must be drawn from a distribution of zero mean and standard deviation equal to σ = m−1/2 , where m is the number of synaptic weights associated with the corresponding neuron.

Hint: For simplicity, consider the bias to be zero, and also that the inputs to each neuron are mutually uncorrelated.

Solution: Let us focus on the weights of a specific neuron, and denote by yi , i = 1, 2, . . . , m, the respective inputs. Then, the input to the activation functions will be given by z=

m X

θ i yi .

(11)

i=1

Assume that the inputs to the neuron are zero mean and unit variance and are also uncorrelated, then σi2 = E[yi2 ]

(12)

E[yi yj ] = δij .

(13)

and

Recall that our goal is to produce a variable z of zero mean and of unit variance, since for the selected activation function, this guarantees values which are small but not very small, and correspond to the linear operation region of it. Let us select all weights to be of zero mean. Since the weights are generated independently of the inputs, then we have that the mean value of z is zero. Hence, hXX i σz2 = E[z2 ] = E θi θj yi yj i

=

=

XX i m X

j

E[θi θj ] E[yi yj ]

j

E[θ2i ] = mσθ2 .

i=1

Since we want the variance of z to be one, then σθ = m−1/2 .

(14)

4

18.5. Consider the sum of error squares cost function k

N

L 1XX (ŷnm − ynm )2 .

2 n=1 m=1

J=

(15)

Compute the elements of the Hessian matrix ∂2J 0

r ∂θ r ∂θkj k0 j 0

.

(16)

Near the optimum, show that the second order derivatives can be approximated by kL N X X ∂2J ∂ ŷnm ∂ ŷnm = .

(17) 0 0 r r r ∂θkj ∂θkj ∂θ 0 0 ∂θr 0 0 n=1 m=1 k j

k j

In other words, the second order derivatives can be approximated as products of the first order derivatives. The derivatives can be computed by following similar arguments as the gradient descent backpropagation scheme, [71].

Solution: k

N

J =

L 1XX (ŷnm − ynm )2 2 n=1 m=1

kL N X X ∂J ∂ ŷnm = r r (ŷnm − ynm ) ∂θkj ∂θkj n=1 m=1 kL N X X ∂ 2 ŷnm ∂2J = (ŷnm − ynm ) 0 0 r r r ∂θk0 j 0 ∂θkj ∂θkr 0 j 0 ∂θkj n=1 m=1

+

kL N X X ∂ ŷnm ∂ ŷnm 0

n=1 m=1

r ∂θkj ∂θkr 0 j 0

If we are near a good minimum, ŷnm ' ynm thus, kL N X X ∂ ŷnm ∂ ŷnm ∂2J ' 0 0 r r r ∂θkj ∂θk0 j 0 ∂θkj ∂θkr 0 j 0 n=1 m=1

18.6. It is common when computing the Hessian matrix, to assume that it is diagonal. Show, that under this assumption, the quantities ∂2E , ∂(θkj )2 where E :=

kL X

2 L f (zm ) − ym ,

m=1

propagate backwards according to the following,

5 2

2

kj

j

r−1 2 ∂ E ∂ E • ∂(θ ) . r )2 = ∂(z r )2 (yk

•

∂2E 00 L 0 L 2 r )2 = f (zj )ej + (f (zj )) . ∂(zL 2

E = (f 0 (zjr−1 ))2 • ∂(z∂r−1 )2

Pkr r r ∂2E r 2 00 r−1 θkl δk . ) k=1 r )2 (θkj ) + f (zj k ∂(zk

P

k

Solution:

∂2E ∂ ∂E ∂ ∂E ∂zjr = [ ] = r )2 r ∂θ r r [ ∂z r ][ ∂θ r ] = ∂(θkj ∂θkj ∂θkj j kj kj

(18)

∂δjr r−1 ∂ r r−1 δ y . = j k r r yk ∂θkj ∂θkj However,

∂δjr ∂δjr ∂zjr ∂δjr r−1 = = y . k r r ∂θkj ∂zjr ∂θkj ∂zjr

(19)

∂2E ∂2E = (y r−1 )2 . r 2 ∂(θkj ) ∂(zjr )2 k

(20)

Hence,

For the backpropagation of the involved second derivatives, we have I: r = L:

∂E = f 0 (zjL )ej ∂zjL

and ∂2E r )2 ∂(zL

= f 00 (zjL )ej + (f 0 (zjL )

∂ej ∂zjL

= f 00 (zjL )ej + (f 0 (zjL ))2 II r < L: We know from the text that, δjr−1 =

kr X k=1

δkr

kr X ∂zkr r = δkr θkj f 0 (zjr−1 ) ∂zjr−1 k=1

(21)

and kl X ∂δjr−1 ∂2E ∂δkr 0 r−1 r = = θkj f (zj ) r−1 2 r−1 ∂(zj ) ∂zj ∂zjr−1 k=1

+

kl X k=1

r r θkj δk

∂f 0 (zjr−1 ) ∂zjr−1

,

or kl kl X X ∂δkr 0 r−1 ∂2E 00 r−1 r r r = f (v ) θ δ + θkj f (zj ) kj k j r−1 2 ∂(zj ) ∂zjr−1 k=1 k=1

(22)

6

also ∂δkr ∂zjr−1

=

X ∂δ r ∂z r0 X ∂2E r 0 r−1 k k = ) r−1 r r ∂z r θk0 j f (zj ∂z ∂z ∂z 0 0 k k k j k0 k0

=

∂ 2 E r 0 r−1 θ f (zj ). ∂(zkr )2 kj

Thus, finally we get, kr X ∂2E X ∂2E 0 r−1 2 r 2 00 r−1 r r = (f (z )) (θ ) + f (z ) θkl δk j j ∂(zkr )2 kj ∂(zkr−1 )2 k k=1

18.7. Show that if the activation function is the logistic sigmoid and the loss L function in (18.44) is used, then δnj in (18.21) becomes L δnj = a(ŷnj − ynj ).

If the cross-entropy in Eq. (18.41) is used, then it is equal to L δnj = aynj (ŷnj − 1).

Solution: The cost function is J=

N X

Jn ,

n=1

and Jn = −

kL X

(ynk ln ŷnk + (1 − ynk ) ln(1 − ŷnk )) .

k=1

Hence, ∂Jn ∂Jn ∂ ŷnk = . ∂znk ∂ ŷnk ∂znk However, ∂Jn ynk 1 − ynk ŷnk − ynk =− + = . ∂ ŷnk ŷnk 1 − ŷnk ŷnk (1 − ynk ) Also, 0 ∂ ŷnk = σ (znk ) = aσ(znk )(1 − σ(znk )) = aŷnk (1 − ŷnk ). ∂znk

Combining the above we finally obtain that L δnk :=

∂Jn = a(ŷnk − ynk ). ∂znk

The proof for the cross-entropy is straightforward from the above.

7

18.8. Show that if the activation function is the logistic sigmoid and the relative L entropy loss function is used, then δnj in (18.21) becomes, L δnj = a(ŷnj − 1)ynj .

Solution: From the theory, we have that L δnj :=

∂Jn , L ∂znj

and L ŷnj = f (znj ).

The cost function is J =

N X

Jn ,

n=1

with Jn = −

kL X

ynk ln

k=1

= −

kL X k=1

ynk ln

ŷnk ynk

L f (znk ) ynk

∂Jn ynj 0 L = − f (znj ) L L ) ∂znj f (znj ynj L L = − af (znj )(1 − f (znj )) L ) f (znj

L δnj :=

or L δnj = a(ŷnj − 1)ynj

18.9. Show that the cross-entropy loss function depends on the relative output errors.

Solution: We will prove it for one of the versions, and similar is the proof for the other. Taking into consideration the limiting value 0 ln 0 = 0, we can write the cross-entropy cost as, J = −

kL N X X 1 − ŷnk ŷnk + (1 − ynk ) ln . ynk ln ynk 1 − ynk n=1 k=1

Let, 0

ŷnk = ynk + nk , and (1 − ŷnk ) = (1 − ynk ) + nk

8 0

where nk and nk are the respective errors. Then, J = −

kL N X X

0

ynk ln

n=1 k=1

ynk + nk 1 − ynk + nk + (1 − ynk ) ln . ynk 1 − ynk

Thus J depends on the relative errors.

18.10. Show that if the activation function is the softmax and the loss function L is the cross-entropy (or the loss in (18.44)), then δnj in (18.21) does not depend on the derivatives of the nonlinearity.

Solution: We will show it for Eq. (18.44) and for the cross-entropy it is just an obvious variant. We have that Jn = −

kL X

(ynk ln ŷnk + (1 − ynk ) ln(1 − ŷnk )) ,

k=1

where L exp(znk ) . ŷnk = PkL L m=1 exp(znm )

Thus, ∂Jn ∂Jn ∂ ŷnj = .

L L ∂ ŷnj ∂znj ∂znj Now, ∂ ∂ ŷnj = L L ∂znj ∂znj

"

#

L exp(znj )

PkL

L m=1 exp(znm )

,

or ∂ ŷnj L ∂znj

=

= =

L exp(znj )

PkL

L L m=1 exp(znm ) − exp(2znj ) 2 kL L m=1 exp(znm )

P L exp(znj )

PkL

L m=1 exp(znm )

−

L exp(znj)

!2

PkL

L m=1 exp(znm )

ŷnj (1 − ŷnj ).

(23)

Also, ynj 1 − ynj ŷnj − ynj ∂Jn =− + = . ∂ ŷnj ŷnj 1 − ŷnj ŷnj (1 − ŷnj ) Combining Eqs. (23) and (24), we finally obtain that ∂Jn = ŷnj − ynj .

L ∂znj

(24)

9

18.11. As in the previous problem, use the relative entropy as the cost function and the softmax activation function. Then show that, L δnj = ŷnj − ynj .

Solution: Jn = −

kL X

ynk ln

k=1

ŷnk , ynk

L exp(znk ) , L ) exp(z nm m

ŷnk = P

k

L δnj

=

L ∂Jn ∂ X L ynk znk − = − L L ∂znj ∂znj k=1

kL X

ynk ln

m

k=1

= − ynj +

L exp(znm ) ,

X

kL X k=1

= −ynj + ŷnj

L exp(znj ) , L m exp(znm )

ynk P

kL X

ynk = ŷnj − ynj

k=1

18.12. Derive the backpropagation through time algorithm for training recurrent neural networks.

Solutions: The starting set of equations are those defining an RNN, i.e., hn

=

f (U xn + W hn−1 + b) ,

ŷn

=

g (V hn + c) .

Let us start with the gradient of the cost J w.r. to the elements of matrix W . By definition we have,  ∂J  . . . ∂w∂J ∂w11 1kh   ∂J .. .. .. , :=  . . .   ∂W ∂J . . . ∂w∂J ∂wk 1 k k h

h h

where kh is the dimensionality of the state vector.

For the computation of the above, we have the following chain rule, N X ∂J ∂J ∂hni = . ∂wij ∂h ni ∂wij n=1

(25)

10

Note from the defining RNN equations, all the wij , j = 1, 2, . . . , kh , elements affect only the corresponding ith component of hn . Also, 0 ∂hni = f (z)h(n−1)j , z := (U xn + W hn−1 + b)i . ∂wij

Let us focus on the tanh nonlinearity. Then, 0

f (z) = 1 − (tanh(z))2 = 1 − h2ni , or

∂hni = 1 − h2ni h(n−1)j . ∂wij

(26)

Combining Eqs. (25) and (26) in a vector form, we get N X ∂J T ∂J = diag 1 − h2ni:1,kh h , ∂W ∂hn n−1 n=1

(27)

where diag 1 − h2ni:1,kh is a diagonal matrix with its (i, i) element being equal to h2ni , i = 1, 2, . . . , kh . In the above formula, the gradient with respect to the state vectors are computed via the chain rule, in the backward pass, as it is explained in the text, i.e., ∂J = ∂hn

∂hn+1 ∂hn

T

∂J + ∂hn+1

∂ ŷn ∂hn

T

∂J . ∂ ŷn

(28)

∂J (where the reFor the above to be complete, we need to compute ∂h N ∂ ŷn n+1 cursion starts), ∂h and ∂h ∂hn . The gradient of the cost w.r. to the n outputs is straightforward, and depends on the specific loss function, see, e.g., Problem 18.10.

Before we proceed further, let us introduce the variable, zn = V hn +c, i.e., prior to the output nonlinearity. We will express derivatives w.r. to the latter, since it makes the formulae simpler and more general, independent of the specific form of the output nonlinearity. (a) For n = N , we have, ∂J ∂J =VT . ∂hN ∂zN where the computation of the latter gradient is straightforward and it depends on the loss function and the output nonlinearity, e.g., Problem 18.10. (b) Also, following similar arguments as before, ∂hn+1 = diag(1 − h2(n+1)i:1,kh )W. ∂hn

11

(c) Taking into account that the output nonlinearity is the softmax and recalling the results from Problem 18.10, we get

∂ ŷn = diag ŷi (1 − ŷi ) V. ∂hn Note that Eq. (28) can also be expressed in terms of gradients w.r. ∂ ŷn ∂zn we would have to compute ∂h , which to zn . Then, instead of ∂h n n is straightforward.

The above have completed the calculations of the gradient of J w.r. to W. • For the gradient w.r. to V , we get, N X ∂J T ∂J = hn ∂V ∂z n n=1

• For the gradient w.r. to U and following similar arguments as for W we obtain N X ∂J T ∂J diag 1 − h2ni:1,kh = x , ∂U ∂hn n−1 n=1 • For the gradient w.r. to c, we get T N N X X ∂zn ∂J ∂J ∂J = = ∂c ∂c ∂z ∂z n n n=1 n=1 • The gradient w.r. to b is equal to, ∂J ∂b

=

=

T N X ∂hn ∂J ∂b ∂h n n=1 N X

diag 1 − h2ni:1,kh

n=1

∂J ∂hn

We have now derived the required gradients of the cost J w.r. to all parameters and matrices.

18.13. Derive the gradient of the log-likelihood in (18.74).

Solutions: To simplify slightly the notation, we will consider only one term of the sum. Thus we have, X

∂ ∂P (v; Θ) = ln exp (−E) − (29) ∂θij ∂θij h XX

∂ ln exp (−E) , ∂θij v h

12

or ∂P (v; Θ) ∂θij

X 1 ∂E exp (−E) + exp (−E) ∂θ ij h h 1 XX ∂E exp (−E) , Z v ∂θij

= −P

(30)

h

or ∂P (v; Θ) ∂θij

exp (−E) P hi vj − h exp (−E) h XX P (v, h)hi vj .

X

=

v

(31)

h

Taking into account that exp (−E) , P (h|v) = P h exp (−E) the desired formula is derived.

18.14. Prove that for the case of RBMs, the conditional probabilities are given by the following factorized form, P

J I exp Y j=1 θij vj + bi hi P i.

P (h|v) = P h 0 J 0 i=1 j=1 θij vj + bi hi h exp i

Solution: Our starting point is P

PM M PK exp θ h v + b h ij i j i i i=1 j=1 i=1 P .

P (h|v) = P PM M PK exp θ h v + b h ij i j i i h i=1 j=1 i=1 The numerator is written as, A :=

M Y

exp

i=1

K X

θij vj + bi hi .

j=1

The denominator becomes B:

=

XX

M XY

h2

hM i=1

M X Y

K X

h1

=

...

i=1 hi

exp

exp

K X

θij vj + bi hi

j=1

θij vj + bi hi .

j=1

Combining A and B we obtain the result.

13

18.15. Derive Eq. (18.83).

Solution: The integrand is equal to

pr (x) ln(D(x)) + pg (x) ln 1 − D(x) .

Taking the derivative w.r. to D and setting it equal to zero, the claim is readily obtained.

1

Solutions To Problems of Chapter 19 19.1. Show that the second principal component in PCA is given as the eigenvector corresponding to the second largest eigenvalue.

Solution As pointed out in the text, the following optimization task is in order, v T Σ̂v,

maximize s.t.

(2)

T

(3)

v v = 1,

s.t.

(1)

T

v v1 = 0.

Forming the Lagrangian, taking the gradient and setting it equal to zero we readily obtain

Σ̂ − aI v = bv1 , (4) where we used a and b for the Lagrange multipliers, to avoid confusion with the eigenvalues. Left multiplying both sides with v1T , using the second of the constraints in the last equation and recalling that v1 is an eigenvector of the sample covariance matrix, results in b = 0, or Σ̂v = av.

Thus the maximizing vector is an eigenvector. Plugging it into the cost and taking into account the second of the constraints we have that v T Σ̂v = a which is maximized if a is equal to the second largest eigenvalue. Note that since the matrix is symmetric, eigenvectors corresponding to different eigenvalues are orthogonal.

For the general case, using induction, the proof follows similar arguments.

19.2. Show that the pair of directions, associated with CCA, which maximize the respective correlation coefficient, satisfy the following pair of relations, Σxy uy

= λΣxx ux ,

Σyx ux

= λΣyy uy .

Solution: The corresponding Lagrangian is given by L(ux , uy , λx , λy ) = uTx Σxy uy −

λy

λx T ux Σxx ux − 1 − uTy Σyy uy − 1 , 2 2

and taking the gradients and equating to zero we get Σxy uy

= λx Σxx ux ,

Σyx ux

=

λy Σyy uy ,

2 T where we used the fact that Σxy = Σyx . Multiplying the first one by uTx T and the second one by uy and taking into account the constraints, it is readily seen that λx = λy := λ,

which proves the claim.

19.3. Establish the arguments that verify the convergence of the k-SVD. Solution: Let us first assume that we can perform the sparse coding stage perfectly; that is, we can retrieve the best approximation to xn , n = 1, 2, . . . , N . that contains no more than T0 nonzero entries. In this case, and assuming a fixed dictionary, each sparse coding step decreases the total representation error; this is the way that any greedy-like algorithm works.

During the second stage, since the Frobenius error norm is minimized, the error is guaranteed not to increase, and at the same time the sparsity constraint is not violated. Executing a series of such steps ensures a monotonic MSE reduction, and therefore the algorithm converges in a local minimum.

19.4. Prove that (19.83) and (19.89) are the same.

Proof: We have that ẑ =

−1 T 1 σ2 2 T Σ A x = σ I + AT A A x, z|x 2 2 σ σ

or ẑ =

1 σ2

I + AT

A σ2

−1

AT x.

Using the matrix inversion lemma (I + AB)−1 A = A(I + BA)−1 , we obtain ẑ =

1 T A σ2

−1 A I + 2 AT x = AT Σx−1 x, σ

where Σx = σ 2 I + AAT .

19.5. Show that the ML PPCA tends to PCA as σ 2 → 0.

Solution: Recall that PCA relies on the transformation y = ÃT x,

3

where à comprises the principal eigenvectors of the covariance (for zero mean) matrix and that in this case Σy = ÃT Σx à = Λ.

If we impose on y to have identity covariance matrix as the latent variables in PPCA, we make the transformation z = Λ−1/2 y = Λ−1/2 ÃT x, since this results in Σz = I. Recall now that the ML estimation in PPCA, for σ 2 = 0 results in Σx = 0 + AAT = ÃΛÃT .

Thus, taking into account that A is a factor of Σx and recalling from the theory that ẑ = AT Σx−1 x = Λ1/2 ÃT (ÃΛ−1 ÃT )x = Λ−1/2 ÃT x. hence ẑ = z. Note that we have used the orthogonality property Ã−1 = ÃT .

19.6. Show Eqs. (19.91)-(19.92) Solution: Our starting point are the relations given in the text, N X

1 (j) 1 l (j) ln β + kµz|x (n)k2 + trace{Σz|x } 2 2 2 n=1

β β (j) (j) + kxn − Aµz|x (n)k2 + trace{AΣz|x AT } + C, 2 2 (5)

Q(A, β; A(j) , β (j) ) = −

−

where C is a constant and (j)

(j)

(j)

µz|x (n) = β (j) Σz|x A(j)T xn , Σz|x = (I + β (j) A(j)T A(j) )−1 .

Eq. (5) is rewritten as Q(A, β; A(j) , β (j) ) = −

N lN ln β βX − (xn − Aµ(n))T (xn − Aµ(n)) 2 2 n=1

Nβ trace{AΣAT } + constants, 2

4 (j)

where for notational simplicity we have used µ(n) instead of µz|x (n) and (j)

Σ in place of Σz|x . Elaborating on the previous equation we get ( ) N N X lN ln β βX T (j) (j) T T Q(A, β; A , β ) = − x xn + βtrace A xn µ (n) 2 2 n=1 n n=1 ( ! ) N X β Nβ T T − trace A µ(n)µ (n) A − trace{AΣAT } 2 2 n=1 + constant.

(6)

Recalling now the following identities ∂ trace{ABAT } = A(B + B T ), ∂A ∂ trace{AT B} = B, ∂A ∂ trace{AB} = B T , ∂A and using them to compute the derivative with respect to A of the right hand side of (6) and equating to zero, we get β

N X

xn µT (n) − βA

n=1

N X

µ(n)T µ(n) − βN AΣ = O

n=1

which then results in Eq. (19.102).

Eq. (19.88) results in a straightforward way by taking the derivative w.r to β and equating to zero.

19.7. Show equation (19.102).

Solution: Our starting point is the eigenvalue/eigenvector equation Σu = λu, where u=

N X

an φ(xn ) = ΦT a,

n=1

where ΦT := [φ(x1 ), . . . , φ(xN )].

Employing the last definition and (8), we can rewrite (7) as 1 T Φ ΦΦT a = λΦT a, N or finally

1 ΦΦT a − λa = 0.

N If Φ is full rank, then the claim has been proved. ΦT

(7)

(8)

5

19.8. Show that the number of degrees of freedom of a rank r matrix is equal to r(l1 + l2 ) − r2 .

Solution: A rank r matrix can be expressed via SVD as in (19.119). The first column of U has l1 parameters whereas the second one, since it is orthogonal to the first one, it is fully described with l1 − 1 parameters. In the same manner, the third one is fully described with l1 − 2 parameters because it is orthogonal the the first two, etc. The same is true for V and r extra parameters are needed for the description of the singular values. So in total, the number of degrees of freedom is r(l1 +l2 )−2(1+, . . . , l1 −1)+r which directly leads to the answer.