Instructor Solution Manual Probabilistic Machine Learning An Introduction

SOLUTION MANUAL

Full Solution Manual for “Probabilistic Machine Learning: An Introduction” Kevin Murphy November 21, 2021

1

1

Solutions

2

Part I

Foundations

3

2

Solutions

2.1

Conditional independence

PRIVATE 1. Bayes’ rule gives P (H|E1 , E2 ) =

P (E1 , E2 |H)P (H) P (E1 , E2 )

(1)

Thus the information in (ii) is sufficient. In fact, we don’t need P (E1 , E2 ) because it is equal to the normalization constant (to enforce the sum to one constraint). (i) and (iii) are insufficient.

2. Now the equation simplifies to P (H|E1 , E2 ) =

P (E1 |H)P (E2 |H)P (H) P (E1 , E2 )

(2)

so (i) and (ii) are obviously sufficient. (iii) is also sufficient, because we can compute P (E1 , E2 ) using normalization.

2.2

Pairwise independence does not imply mutual independence

We provide two counter examples.

Let X1 and X2 be independent binary random variables, and X3 = X1 ⊕ X2 , where ⊕ is the XOR operator. We have p(X3 |X1 , X2 ) 6= p(X3 ), since X3 can be deterministically calculated from X1 and X2 . So the variables {X1 , X2 , X3 } are not mutually independent. However, we also have p(X3 |X1 ) = p(X3 ), since without X2 , no information can be provided to X3 . So X1 ⊥ X3 and similarly X2 ⊥ X3 . Hence {X1 , X2 , X3 } are pairwise independent.

Here is a different example. Let there be four balls in a bag, numbered 1 to 4. Suppose we draw one at random. Define 3 events as follows: • X1 : ball 1 or 2 is drawn.

• X2 : ball 2 or 3 is drawn.

• X3 : ball 1 or 3 is drawn.

We have p(X1 ) = p(X2 ) = p(X3 ) = 0.5. Also, p(X1 , X2 ) = p(X2 , X3 ) = p(X1 , X3 ) = 0.25. Hence p(X1 , X2 ) = p(X1 )p(X2 ), and similarly for the other pairs. Hence the events are pairwise independent.

However, p(X1 , X2 , X3 ) = 0 6= 1/8 = p(X1 )p(X2 )p(X3 ).

2.3

Conditional independence iff joint factorizes

PRIVATE Independency ⇒ Factorization. Let g(x, z) = p(x|z) and h(y, z) = p(y|z). If X ⊥ Y |Z then p(x, y|z) = p(x|z)p(y|z) = g(x, z)h(y, z)

4

(3)

Factorization ⇒ Independency. If p(x, y|z) = g(x, z)h(y, z) then X X X X 1= p(x, y|z) = g(x, z)h(y, z) = g(x, z) h(y, z) x,y

p(x|z) =

X

x,y

p(x, y|z) =

X

y

p(y|z) =

x

g(x, z)h(y, z) = g(x, z)

y

X

p(x, y|z) =

X

x

X

X

h(y, z)

(5)

g(x, z)

(6)

y

g(x, z)h(y, z) = h(y, z)

x

p(x|z)p(y|z) = g(x, z)h(y, z)

X x

g(x, z)

X

x

(7)

g(y, z)

y

(8)

= g(x, z)h(y, z) = p(x, y|z)

2.4

(4)

y

Convolution of two Gaussians is a Gaussian

We follow the derivation of [Jay03, p221]. Define φ(x − µ|σ) = N (x|µ, σ 2 ) =

1 x−µ φ( ) σ σ

(9)

where φ(z) is the pdf of the standard normal. We have (10)

p(y) = N (x1 |µ1 , σ12 ) ⊗ N (x2 |µ2 , σ22 ) Z = dx1 φ(x1 − µ1 |σ1 )φ(y − (x1 − µ2 )|σ2 )

(11)

Now the product inside the integral can be written as follows " ( 2 2 #) 1 x1 − µ1 1 y − x1 − µ2 φ(x1 − µ1 |σ1 )φ(y − (x1 − µ2 )|σ2 ) = exp − + 2πσ1 σ2 2 σ1 σ2

(12)

We can bring out the dependency on x1 by rearranging the quadratic form inside the exponent as follows

x − µ1 σ1

2

+

y − (x1 − µ2 ) σ2

2 = (w1 + w2 )(x1 − x̂) +

w1 w2 (y − (µ1 − µ2 ))2 w1 + w2

(13)

where we have defined the precision or weighting terms w1 = 1/σ12 , w2 = 1/σ22 , and the term x̂ = Note that

w1 µ1 + w2 y − w2 µ2 w1 + w2

(14)

1 w1 w2 σ2 σ2 1 = 2 2 21 2 2 = 2 w1 + w2 σ1 σ2 σ1 + σ2 σ1 + σ22

(15)

Hence p(y) =

1 2πσ1 σ2

Z

1 1 w1 w2 exp − (w1 + w2 )(x1 − x̂)2 − (y − (µ1 − µ2 ))2 dx1 2 2 w1 + w2

(16)

The integral over x1 becomes one over the normalization constant for the Gaussian: Z

1 1 exp[− (w1 + w2 )(x1 − x̂)2 ] dx1 = (2π) 2 2

5

σ22 σ22 σ12 + σ22

21

(17)

Hence 21 1 σ22 σ22 (y − µ1 − µ2 )2 ] exp[− σ12 + σ22 2(σ12 + σ22 ) − 1 1 (y − µ1 − µ2 )2 ] σ12 + σ22 2 exp[− 2 2(σ1 + σ22 )

1

1

p(y) = (2π)−1 (σ12 σ22 )− 2 (2π) 2 1

= (2π)− 2

(19) (20)

= N (y|µ1 + µ2 , σ12 + σ22 )

2.5

(18)

Expected value of the minimum of two rv’s

PRIVATE Let Z = min(X, Y ). We have P (Z > z) = P (X > z, Y > z) = P (X > z)p(Y > z) = (1 − z)2 = 1 + z 2 − 2z.

Hence the cdf of the minimum is P (Z ≤ z) = 2z − z 2 and the pdf is p(z) = 2 − 2z. Hence the expected value is Z 1

(21)

z(2 − 2z) = 1/3

E [Z] = 0

2.6

Variance of a sum

We have (22)

V [X + Y ] = E[(X + Y )2 ] − (E[X] + E[Y ])2 2

2

2

2

= E[X + Y + 2XY ] − (E[X] + E[Y ] + 2E[X]E[Y ])

(23)

= E[X ] − E[X] + E[Y ] − E[Y ] + 2E[XY ] − 2E[X]E[Y ]

(24)

= V [X] + V [Y ] + 2Cov [X, Y ]

(25)

2

2

2

2

If X and Y are independent, then Cov [X, Y ] = 0, so V [X + Y ] = V [X] + V [Y ].

2.7

Deriving the inverse gamma density

PRIVATE We have

dx | dy

(26)

dx 1 = − 2 = −x2 dy y

(27)

ba a−1 −xb x e Γ(a)

(28)

py (y) = px (x)| where So py (y) = x2

ba a+1 −xb x e Γ(a) ba −(a+1) −b/y = y e = IG(y|a, b) Γ(a) =

6

(29) (30)

2.8

Mean, mode, variance for the beta distribution

For the mode we can use simple calculus, as follows. Let f (x) = Beta(x|a, b). We have df dx = (a − 1)xa−2 (1 − x)b−1 − (b − 1)xa−1 (1 − x)b−2

0=

= (a − 1)(1 − x) − (b − 1)x a−1 x= a+b−2

(31) (32) (33) (34)

For the mean we have Z 1 E [θ|D] =

θp(θ|D)dθ Z Γ(a + b) = θ(a+1)−1 (1 − θ)b−1 dθ Γ(a)Γ(b) Γ(a + b) Γ(a + 1)Γ(b) a = = Γ(a)Γ(b) Γ(a + 1 + b) a+b

(35)

0

where we used the definition of the Gamma function and the fact that Γ(x + 1) = xΓ(x).

We can find the variance in the same way, by first showing that Z

Γ(a + b) 1 Γ(a + 2 + b) (a+2)−1 θ (1 − θ)b−1 dθ E θ2 = Γ(a)Γ(b) 0 Γ(a + 2)Γ(b) a a+1 Γ(a + b) Γ(a + 2 + b) = = Γ(a)Γ(b) Γ(a + 2)Γ(b) a+ba+1+b 2 2 Now we use V [θ] = E θ − E [θ] and E [θ] = a/(a + b) to get the variance.

2.9

(36) (37)

(38) (39)

Bayes rule for medical diagnosis

PRIVATE Let T = 1 represent a positive test outcome, T = 0 represent a negative test outcome, D = 1 mean you have the disease, and D = 0 mean you don’t have the disease. We are told P (T = 1|D = 1) = 0.99

(40)

P (T = 0|D = 0) = 0.99

(41)

P (D = 1) = 0.0001

(42)

We are asked to compute P (D = 1|T = 1), which we can do using Bayes’ rule: P (T = 1|D = 1)P (D = 1) P (T = 1|D = 1)P (D = 1) + P (T = 1|D = 0)P (D = 0) 0.99 × 0.0001 = 0.99 × 0.0001 + 0.01 × 0.9999 = 0.009804

P (D = 1|T = 1) =

(43) (44) (45)

So although you are much more likely to have the disease (given that you have tested positive) than a random member of the population, you are still unlikely to have it.

7

2.10

Legal reasoning

Let E be the evidence (the observed blood type), and I be the event that the defendant is innocent, and G = ¬I be the event that the defendant is guilty.

1. The prosecutor is confusing p(E|I) with p(I|E). We are told that p(E|I) = 0.01 but the relevant quantity is p(I|E). By Bayes rule, this is p(I|E) =

p(E|I)p(I) 0.01p(I) = p(E|I)p(I) + p(E|G)p(G) 0.01p(I) + (1 − p(I))

(46)

since p(E|G) = 1 and p(G) = 1 − p(I). So we cannot determine p(I|E) without knowing the prior probability p(I). So p(E|I) = p(I|E) only if p(G) = p(I) = 0.5, which is hardly a presumption of innocence.

To understand this more intuitively, consider the following isomorphic problem (from http://en. wikipedia.org/wiki/Prosecutor’s_fallacy): A big bowl is filled with a large but unknown number of balls. Some of the balls are made of wood, and some of them are made of plastic. Of the wooden balls, 100 are white; out of the plastic balls, 99 are red and only 1 are white. A ball is pulled out at random, and observed to be white.

Without knowledge of the relative proportions of wooden and plastic balls, we cannot tell how likely it is that the ball is wooden. If the number of plastic balls is far larger than the number of wooden balls, for instance, then a white ball pulled from the bowl at random is far more likely to be a white plastic ball than a white wooden ball — even though white plastic balls are a minority of the whole set of plastic balls.

2. The defender is quoting p(G|E) while ignoring p(G). The prior odds are 1 p(G) = p(I) 799, 999

(47)

p(G|E) 1 = p(I|E) 7999

(48)

The posterior odds are

So the evidence has increased the odds of guilt by a factor of 1000. This is clearly relevant, although perhaps still not enough to find the suspect guilty.

2.11

Probabilities are sensitive to the form of the question that was used to generate the answer

PRIVATE 1. The event space is shown below, where X is one child and Y the other.

X Y Prob.

G G 1/4 G B 1/4 B G 1/4 B B 1/4

8

Let Ng be the number of girls and Nb the number of boys. We have the constraint (side information) that Nb + Ng = 2 and 0 ≤ Nb , Ng ≤ 2. We are told Nb ≥ 1 and are asked to compute the probability of the event Ng = 1 (i.e., one child is a girl). By Bayes rule we have p(Nb ≥ 1|Ng = 1)p(Ng = 1) p(Nb ≥ 1) 1 × 1/2 = = 2/3 3/4

p(Ng = 1|Nb ≥ 1) =

(49) (50)

2. Let Y be the identity of the observed child and X be the identity of the other child. We want p(X = g|Y = b). By Bayes rule we have p(Y = b|X = g)p(X = g) p(Y = b) (1/2) × (1/2) = 1/2 = 1/2

p(X = g|y = b) =

(51) (52)

Tom Minka [Min98] has written the following about these results: This seems like a paradox because it seems that in both cases we could condition on the fact that "at least one child is a boy." But that is not correct; you must condition on the event actually observed, not its logical implications. In the first case, the event was "He said yes to my question."

In the second case, the event was "One child appeared in front of me." The generating distribution is different for the two events. Probabilities reflect the number of possible ways an event can happen, like the number of roads to a town. Logical implications are further down the road and may be reached in more ways, through different towns. The different number of ways changes the probability.

2.12

Normalization constant for a 1D Gaussian

Following the first hint we have Z 2π Z ∞

r2 r exp − 2 drdθ 2σ 0 0 Z 2π Z ∞

r2 = dθ r exp − 2 dr 2σ 0 0

Z2 =

(54) (55)

= (2π)I where I is the inner integral

(53)

Z ∞ I= 0

r2 r exp − 2 2σ

(56)

Following the second hint we have I = −σ

2

Z − h

r −r2 /2σ2 e dr σ2 i ∞ 2 2

= −σ 2 e−r /2σ

= −σ 2 [0 − 1] = σ

0 2

(57) (58) (59)

Hence Z 2 = 2πσ 2 p Z = σ (2π)

9

(60) (61)

3

Solutions

3.1

Uncorrelated does not imply independent

PRIVATE We have (62)

E[X] = 0

(63)

2

V [X] = (1 − −1) /12 = 1/3 2

(64)

2

E[Y ] = E[X ] = V [X] + (E[X]) = 1/3 + 0 Z 0

Z 1 Z 1 −( x3 dx) + ( x3 dx) E[XY ] = p(x)x3 dx = 2 −1 0

1 −1 4 0 1 41 1 = [x ]−1 + [x ]0 = [−1 + 1] = 0 2 4 4 8

(65) (66)

where we have split the integral into two pieces, since x3 changes sign in the interval [−1, 1]. Hence ρ=

3.2

Cov [X, Y ] E[XY ] − E[X]E[Y ] 0 = =0 σX σY σX σY σX σY

(67)

Correlation coefficient is between -1 and +1

We have

X Y + σX σY

X Y X Y =V +V + 2Cov , σX σY σX σY

X Y V [X] V [Y ] + + 2Cov , = 2 σX σY2 σX σY

(68)

0≤V

(69) (70) (71)

= 1 + 1 + 2ρ = 2(1 + ρ) Hence ρ ≥ −1. Similarly,

0≤V

X Y − σX σY

(72)

= 2(1 − ρ)

so ρ ≤ 1.

3.3

Correlation coefficient for linearly related variables is ±1

PRIVATE Let E [X] = µ and V [X] = σ 2 , and assume a > 0. Then we have (73)

E [Y ] = E [aX + b] = aE [X] + b = aµ + b 2

2 2

V [Y ] = V [aX + b] = a V [X] = a σ

E [XY ] = E [X(aX + b)] = E aX 2 + bX = aE X 2 + bE [X] = a(µ2 + σ 2 ) + bµ 2

2

Cov [X, Y ] = E [XY ] − E [X] E [Y ] = a(µ + σ ) + bµ − µ(aµ + b) = aσ 2

Cov [X, Y ] aσ ρ= p =√ √ =1 2 V [X] V [Y ] σ a2 σ 2 √ If Y = aX + b, where a < 0, we find ρ = −1, since a/ a2 = −a/|a| = −1.

10

2

(74) (75) (76) (77)

3.4

Linear combinations of random variables

1. Let y = Ax. Then

Cov [y] = E (y − E [y])(y − E [y])T

= E (Ax − Am)(Ax − Am)T

= AE (x − m)(x − m)T AT = AΣA

(78) (79) (80) (81)

T

P P 2. C = AB has entries cij = k aik bkj . The diagonalPelements of C (when i = j) are given by k aik bki .

So the sum of the diagonal elements is tr(AB) = ik aik bki which is symmetric in A and B. 3. We have

E xT Ax = E tr(xT Ax) = E tr(AxxT )

= tr(AE xxT ) = tr(A(Σ + mmT )) T

= tr(AΣ) + m Am

3.5

(82) (83) (84)

Gaussian vs jointly Gaussian

1. For the mean, we have E [Y ] = E [W X] = E [X] E [X] = 0

(85)

V [Y ] = E [V [Y |W ]] + V [E [Y |W ]]

(86)

For the variance, we have

= E [W [V [X]]W ] + V [W E [X]]

= E W2 + 0 = 1

(87) (88)

To show it’s Gaussian, we note that Y is a linear combination of Gaussian rv’s.

2. To show that Cov [X, Y ] = 0, we use the rule of iterated expectation. First we have E [XY ] = E [E [XY |W ]] X = p(w)E [XY |w]

(89) (90)

w∈{−1,1}

= −1 · 0.5 · E [X · −X] + 1 · 0.5 · E [X · X]

(91)

== 0

(92)

E [Y ] = E [E [Y |W ]] X = p(w)E [Y |w]

(93)

Then we have

(94)

w∈{−1,1}

= 0.5 · E [−X] + 0.5 · E [X]

(95)

=0

(96)

Hence Cov [X, Y ] = E [XY ] − E [X] E [Y ] = 0 So X and Y are uncorrelated even though they are dependent.

11

(97)

3.6

Normalization constant for a multidimensional Gaussian

Let Σ = UΛUT , so Σ

−1

=U

−T

−1

Λ

U

−1

−1

= UΛ

U=

p X 1 i=1

λi

ui uTi

(98)

(x − µ)

(99)

Hence T

(x − µ) Σ

−1

T

(x − µ) = (x − µ)

p X 1 i=1

=

p X 1

λ i=1 i

(x − µ)

T

λi

! ui uTi

ui uTi (x − µ) =

p X y2 i

λ i=1 i

(100)

where yi , uTi (x − µ). The y variables define a new coordinate system that is shifted (by µ) and rotated (by U) with respect to the original x coordinates: y = U(x − µ). Hence x = UT y + µ.

The Jacobian of this transformation, from y to x, is a matrix with elements Jij = so J = UT and |J| = 1.

So Z

∂xi = Uji ∂yj

1 exp(− (x − µ)T Σ−1 (x − µ))dx = 2 =

(101)

Z Y

exp(−

i

1 X yi2 )dyi |J| 2 i λi

Yp 2πλi = |2πΣ|

(102) (103)

i

3.7

Sensor fusion with known variances in 1d

Define the sufficient statistics as y1 =

n1 n2 1 X 1 X (1) (2) yi , y 2 = y , n1 i=1 n2 i=1 i

(104)

Define the prior as

(105)

µµ = 0, Σµ = ∞ Define the likelihood as

v1 1 0 A= , µy = , Σy = n 1 1 0 0

0

v2 n2

Now we just apply the equations. The posterior precision is a sum of the precisions of each sensor:

nv1 0 n1 n2 1 −1 T −1 1 Σµ|y = A Σy A = 1 1 = + 0 nv22 1 v1 v2 The posterior mean is a weighted sum of the observed values from each sensor:

nv1 0 n1 y 1 n2 y 2 y1 −1 −1 1 µµ|y = Σµ|y 1 1 = Σµ|y + 0 nv22 y2 v1 v2

12

(106)

(107)

(108)

3.8

Show that the Student distribution can be written as a Gaussian scale mixture Z ∞ p(w|µ, a, b) =

(109)

N (w|µ, α−1 )Ga(α|a, b)dα

0

Z ∞

α ba e−bα αa−1 α 21 exp − (w − µ)2 dα Γ(a) 2π 2 0 1 Z

∞ 1 ba 1 1 2 = αa− 2 exp −α b + (w − µ)2 dα Γ(a) 2π 2 0

=

(110) (111)

Let us define = b + (w − µ)2 /2 and z = α∆. Then, using dz = ∆dα, and the definition of the Gamma R∞ ∆ x−1 −u function, 0 u e du = Γ(x), the integral becomes Z ∞ 0

1 1 z a+ 12 −1 −z −1 e ∆ dz = ∆−a− 2 Γ(a + ) ∆ 2

(112)

so we have Γ(a + 1/2) a b p(w|µ, a, b) = Γ(a)

1 2π

12

1

∆−a− 2

(113)

Let us define a = ν/2 and b = ν/(2λ). Then we have 1 −(ν+1)/2 (w − µ)2 Γ((ν + 1)/2) ν ν/2 1 2 ν + Γ(ν/2) 2λ 2π 2λ 2 −(ν+1)/2 12

Γ((ν + 1)/2) ν ν/2 1 ν λ = 1 + (w − µ)2 Γ(ν/2) 2λ 2π 2λ ν −(ν+1)/2 12 Γ((ν + 1)/2) λ λ = 1 + (w − µ)2 Γ(ν/2) νπ ν

p(w|µ, a, b) =

(114) (115) (116) (117)

= Tν (w|µ, λ−1 )

13

4

Solutions

4.1

MLE for the univariate Gaussian

PRIVATE Starting with the mean, we have 2 X ∂` = (xi − µ) = 0 ∂µ 2σ 2 i

(118)

N

µ̂ =

1 X xi = x N i=1

(119)

We can derive σ̂ 2 as follows.

X ∂` 1 n = σ −4 (xi − µ̂)2 − 2 = 0 2 ∂σ 2 2σ i

(120)

N

1 X (xi − µ̂)2 N i=1 " # X 1 X 2 X 2 1 X 2 = xi + µ̂ − 2 xi µ̂ = [ xi + N µ̂2 − 2N µ̂2 ] N N i i i i !

N 1 X 2 x − (x)2 = N i=1 i

σ̂ 2 =

4.2

(121) (122) (123)

MAP estimation for 1D Gaussians

PRIVATE 1. Since the mean and mode of a Gaussian are the same, we can just write down the MAP estimate using the standard equations for the posterior mean of a Gaussian, which is

ns2 σ2 m (124) E[µ|D] = x+ ns2 + σ 2 ns2 + σ 2 However, we can also derive the MAP estimate explicitly, as follows. The prior distribution over the mean is (µ − m)2 p(µ) = (2πs2 )−1/2 exp[− ] (125) 2s2 Since the samples xi are taken to be independent, we have: 1

p(D|θ) = e− 2σ2

Pn

2 i=1 (xi −µ)

(126)

And combining them: n 1 (µ − m)2 n 1 X 2 log p(µ)p(D|µ) = − log(2πs2 ) − − log(2πσ ) − (xi − µ)2 2 2s2 2 2σ 2 i=1

14

(127)

Taking the derivative with respect to µ : n 1 X ∂ log(p(µ)p(D|µ)) 2(µ − m)(1) − 0 − =0− (2)(xi − µ)(−1) ∂µ 2s2 2σ 2 i=1

(128)

n µ−m 1 X =− 2 + 2 (xi − µ) s σ i=1

(129)

n m 1 X nµ µ xi − 2 =− 2 + 2 + 2 s s σ i=1 σ

(130)

Setting this derivative to zero in order to find the maximum: 0=−

n µ̂M AP m 1 X nµ̂M AP + + xi − s2 s2 σ 2 i=1 σ2

(131)

n

µ̂M AP (

n 1 1 X m + 2) = 2 xi + 2 2 σ s σ i=1 s Pn Pn 1 m s2 i=1 xi + mσ 2 i=1 xi + s2 σ2 µ̂M AP = = n 1 ns2 + σ 2 σ 2 + s2 P

n ns2 σ2 i=1 xi = + m ns2 + σ 2 n ns2 + σ 2

(132) (133) (134)

2

2

2. Notice that as n increases, while s, m and σ remain constant, we have ns2σ+σ2 → 0 while nsns 2 +σ 2 → 1, P

x

yielding µ̂M AP → ni i = µ̂M LE . That is, when there are many samples, the prior knowledge becomes less and less relevant, and all MAP estimators (for any prior) converge to the maximum likelihood estimator µ̂M LE . This is not surprising: as we have more data, it outweighs our prior speculations.

2

3. As the prior variance s2 increases, even for a small number of samples n, we have ns2σ+σ2 → 0 while ns2 ns2 +σ 2 → 1, again yielding µ̂M AP →

P

i xi

= µ̂M LE . That is, when we have an uninformative, almost uniform, prior, we are left only with our data to base out estimation on. n

2

4. On the other hand, if the prior variance is very small, s2 → 0, then we have ns2σ+σ2 → 1 while ns2 ns2 +σ 2

→ 0, yielding µ̂M AP → m. That is, if our prior is very concentrated, we essentially already know the answer, and can ignore the data.

4.3

Gaussian posterior credible interval

We want an interval that satisfies p(` ≤ µn ≤ u|D) ≥ 0.95

(135)

` = µn + Φ−1 (0.025)σn = µn − 1.96σn

(136)

where

u = µn + Φ

−1

(0.9755)σn = µn + 1.96σn

(137)

where Φ is the cumulative distribution function for the standard normal N (0, 1) distribution, and Φ−1 (0.025) is the value below which 2.5% of the probability mass lies (in Matlab, norminv(0.025)=-1.96). and Φ−1 (0.975) is the value below which 97.5% of the probability mass lies (in Matlab, norminv(0.975)=1.96). We want to find n such that u−`=1 (138) 15

Hence we solve 2(1.96)σn = 1 σn2 =

1 4(1.96)2

(139) (140)

where σ 2 σ02 nσ02 + σ 2

(141)

nσ02 + σ 2 = (σ 2 σ02 )4(1.96)2

(142)

σn2 = Hence

2

σ (σ02 4(1.96)2 − 1) n= σ02 2 =

4(9 × (1.96) − 1) = 61.0212 9

(143) (144)

Hence we need at least n ≥ 62 samples.

4.4

BIC for Gaussians

PRIVATE 1. Using the fact that Σ̂M L = S, we have N −1 N tr Σ̂ S − log(|Σ̂|) 2 2 N N = − tr(ID ) − log(|Σ̂|) 2 2

N D + log |Σ̂| =− 2

log p(D|θ̂M L ) = −

(145) (146) (147)

There are D free parameters for the mean and D(D + 1)/2 for the covariance, so the BIC score is BIC = −

d N D + log |Σ̂| − log N 2 2

(148)

where d = D + D(D + 1)/2.

2. Let Sdiag = diag(diag(S)) = Σ̂diag . Then −1 tr(Σ̂−1 diag S) = tr(Σ̂diag Sdiag ) = tr(ID ) = D

because the off-diagonal elements of the matrix product get ignored. Hence BIC = −

d N D + log |Σ̂| − log N 2 2

where d = D + D parameters (for the mean and diagonal).

16

(149)

4.5

BIC for a 2d discrete distribution

1. The joint distribution is p(x, y|θ) = p(x|θ1 )p(y|x, θ2 ): x=0 x=1

y=0 (1 − θ1 )θ2 θ1 (1 − θ2 )

y=1 (1 − θ1 )(1 − θ2 ) θ1 θ2

2. The log likelihood is log p(D|θ) =

X

log p(xi |θ1 ) +

X

i

(150)

log p(yi |xi , θ2 )

i

Hence we can optimize each term separately. For θ1 , we have P I(xi = 1) N (x = 1) 4 = = = 0.5714 θ̂1 = i n N 7 For θ2 , we have

P θ̂2 =

i I(xi = yi )

n

=

(151)

N (x = y) 4 = N 7

(152)

The likelihood is 3 4 3 4 p(D|θ̂, M2 ) = ( )N (x=1) ( )N (x=0) ( )N (x=y) ( )N (x6=y) 7 7 7 7 4 4 3 3 4 4 3 3 =( ) ( ) ( ) ( ) 7 7 7 7 4 8 3 6 = ( ) ( ) ≈ 7.04 × 10−5 7 7

(153) (154) (155)

3. The table of joint counts is y=0 y=1 x=0 2 1 x=1 2 2 We can think of this as a multinomial distribution with 4 states. Normalizing the counts gives the MLE: y=0 y=1 x = 0 2/7 1/7 2/7 x = 1 2/7 The likelihood is 2 1 2 2 N (x=0,y=0) N (x=0,y=1) N (x=1,y=0) N (x=1,y=1) θ01 θ10 θ11 = ( )2 ( )1 ( )2 ( )2

p(D|θ̂, M4 ) = θ00

7

7

7

2 1 = ( )6 ( )1 ≈ 7.77 × 10−5 7 7

7

(156) (157)

Thus is higher than the previous likelihood, because the model has more parameters.

4. For M4 , when we omit case 7, we will have θ̂01 = 0, so p(x7 , y7 |m4 , θ̂) = 0, so L(m4 ) = −∞. However, L(m2 ) will be finite, since all counts remain non zero when we leave out a single case. Hence CV will prefer M2 , since M4 is overfitting.

5. The BIC score is BIC(m) = log p(D|θ̂, m) −

dof(m) log n 2

(158)

where n = 7. For M2 , we have dof = 2, so 4 3 2 BIC(m2 ) = 8 log( ) + 6 log( ) − log 7 = −11.5066 7 7 2 17

(159)

For M4 , we have dof = 3 because of the sum-to-one constraint, so 2 1 3 BIC(m4 ) = 6 log( ) + 1 log( ) − log 7 = −12.3814 7 7 2

(160)

So BIC also prefers m2 .

4.6

A mixture of conjugate priors is conjugate

PRIVATE We now show that, if the prior is a mixture, so is the posterior: p(D|θ)p(θ) p(D) P p(D|θ) k p(Z = k)p(θ|Z = k) P =R p(D|θ0 ) k0 p(Z = k 0 )p(θ0 |Z = k 0 )dθ0 P k)p(D, θ|Z = k) k p(Z = R =P 0 p(D, θ0 |Z = k 0 )dθ0 0 p(Z = k ) Pk p(Z = k)p(θ|D, Z = k)p(D|Z = k) = k P 0 0 k0 p(Z = k )p(D|Z = k )

X p(Z = k)p(D|Z = k) P = p(θ|D, Z = k) 0 0 k0 p(Z = k )p(D|Z = k ) k X = p(Z = k|D)p(θ|D, Z = k)

(161)

p(θ|D) =

(162) (163) (164) (165) (166)

k

where p(Z = k) = πk are the prior mixing weights, and p(Z = k|D) are the posterior mixing weights given by p(Z = k)p(D|Z = k) 0 0 k0 p(Z = k )p(D|Z = k )

(167)

p(Z = k|D) = P

4.7

2 is biased ML estimator σmle

Because the variance of any random variable R is given by var(R) = E[R2 ]−(E[R])2 , the expected value of the square of a Gaussian random variable Xi with mean µ and variance σ 2 is E[Xi2 ] = var(Xi )+(E[Xi ])2 = σ 2 +µ2 . n

1X (Xi − EX1 ,...,Xn ∼N (µ,σ) [σ (X1 , . . . , Xn )] = E[ n i=1

Pn

2

=

1X nE[(Xi − n i=1

1X = nE[(Xi − n i=1

j=1 Xj 2

) ]

n Pn

j=1 Xj 2

) ]

n Pn

j=1 Xj

n

Pn )(Xi −

j=1 Xj

n

)]

n n n 1X 2 X 1 XX 2 = nE[Xi − Xi Xj + 2 Xj Xk ] n i=1 n j=1 n j=1 k=1

n

n

n

n

n

2 XX 1 XXX 1X nE[Xi2 ] − 2 E[Xi Xj ] + 3 [Xj Xk ] = n i=1 n i=1 j=1 n i=1 j=1 k=1

18

Pn Pn Pn Pn Consider the two summations i=1 j=1 E[Xi Xj ] and j=1 k=1 [Xj Xk ]. Of the n2 terms in each of these summations, n of them satisfy i = j or j = k, so these terms are of the form E[Xi2 ]. By linearity of expectation, these terms contribute nE[Xi2 ] to the sum. The remaining n2 − n terms are of the form E[Xi Xj ] or E[Xj Xk ] for i = 6 j or j = 6 k. Because the Xi are independent samples, it follows from linearity of expectation that these terms contribute (n2 − n)E[Xi ]E[Xj ] to the summation. n X n X i=1 j=1

E[Xi Xj ] =

n X n X

E[Xj Xk ]

j=1 k=1

= nE[Xi2 ] + (n2 − n)E[Xi ][Xj ] = n(σ 2 + µ2 ) + (n2 − n)µµ = nσ 2 + nµ2 + n2 µ2 − nµ2 = nσ 2 + n2 µ2

EX1 ,...,Xn ∼N (µ,σ) [σ 2 (X1 , . . . , Xn )] = n 2 1 X 1X = 1n (σ 2 + µ2 ) − 2 (nσ 2 + n2 µ2 ) + 3 (nσ 2 + n2 µ2 ) n i n n i=1 1 σ2 1 (nσ 2 + nµ2 ) − 2 − 2µ2 + 3 (n2 σ 2 + N 3 µ2 ) n n n σ2 σ2 − 2µ2 + + µ2 = σ 2 + µ2 − 2 n n σ2 n−1 2 = σ2 − = σ n n =

Since the expected value of σˆ2 (X1 , . . . , Xn ) is not equal to the actual variance σ 2 , σ 2 is not an unbiased estimator. In fact, the maximum likelihood estimator tends to underestimate the variance. This is not surprising: consider the case of only a single sample: we will never detect any variance. If there are multiple samples, we will detect variance, but since our estimate for the mean will tend to be shifted from the true mean in the direction of our samples, we will tend to underestimate the variance.

4.8

Estimation of σ 2 when µ is known

PRIVATE 2 We can re-do the derivation of σ̂M LE , but instead of taking the maximum over µ , we just use the known µ. We get: n 1X σ̂ 2 = (xi − µ)2 n i=1

In calculating the expected value of the estimator, note that we get exactly the definition of a variance of a

19

random variable: n

EX1 ,...,Xn ∼N (µ,σ) [σ̂ 2 (X1 , X2 , . . . , Xn ) =

1X EXi ∼N (µ,σ) [(Xi − µ)2 ] n i=1 n

=

1X V ar[N (µ, σ)] n i=1

=

1X 2 σ n i=1

n

= σ2 So the estimator is unbiased.

4.9

Variance and MSE of the unbiased estimator for Gaussian variance

PRIVATE We have

N −1 2 σ unb = 2(N − 1) σ2 (N − 1)2 2 V σunb = 2(N − 1) σ4 2 2σ 4 V σunb = N −1

V

Hence the variance of the ML estimator is

2 2 N −1 2 N −1 2σ 4 2(N − 1) 4 V σ̂M L = V σ̂N −1 = = σ N N N −1 N2

(168) (169) (170)

(171)

Finally, we have 2 2 2 MSE(σmle ) = bias(σmle )2 + V σmle 2

(172)

−σ 2 2(N − 1) 4 ) + σ N N2 2N − 1 4 σ = N2

(173)

2 2 2 MSE(σunb ) = bias(σunb )2 + V σunb

(175)

2σ 4 =0+ N −1

(176)

=(

(174)

and

20

5

Solutions

5.1

Reject option in classifiers

1. We have to choose between rejecting, with risk λr , and choosing the most probable class, jmax = arg maxj p(Y = j|x), which has risk X λs p(Y = j|x) = λs (1 − p(Y = jmax |x)) (177) j6=jmax

Hence we should pick jmax if λr ≥ λs (1 − p(Y = jmax |x)) λr ≥ (1 − p(Y = jmax |x)) λs λr p(Y = jmax |x) ≥ 1 − λs

(178) (179) (180)

otherwise we should reject.

For completeness, we should prove that when we decide to choose a class (and not reject), we always pick the most probable one. If we choose a non-maximal category k 6= jmax , the risk is X λs p(Y = j|x) = λs (1 − p(Y = k|x)) ≥ λs (1 − p(Y = jmax |x)) (181) j6=k

which is always bigger than picking jmax .

2. If λr /λs = 0, there is no cost to rejecting, so we always reject. As λr /λs → 1, the cost of rejecting increases. We find p(Y = jmax |x) ≥ 1 − λλrs is always satisfied, so we always accept the most probable class.

5.2

Newsvendor problem

PRIVATE

Z ∞

Z Q (P − C)Qf (D)dD +

Eπ(Q) = Q

Z Q (P − C)Df (D) −

0

C(Q − D)f (D)dD

(182)

0

Z Q = (P − C)Q(1 − F (Q)) + P

Z Q Df (D) − CQ

0

f (D)dD

(183)

0

Z Q = (P − C)Q(1 − F (Q)) + P

Df (D) − CQF (Q)

(184)

0

Hence d Eπ(Q) = (P − C)(1 − F (Q)) − (P − C)Qf (Q) + P Qf (Q) − CF (Q) − CQf (Q) dQ = (P − C) − P F (Q)

(185) (186) (187)

=0 So F (Q) =

P −C P

21

(188)

5.3

Bayes factors and ROC curves

PRIVATE p(H1 |D) is a monotonically increasing function of B, and therefore cannot affect the shape of the ROC curve. This makes sense intuitively, since, in the two class case, it should not matter whether we threshold the ratio p(H1 |D)/p(H0 |D), or the posterior, p(H1 |D), since they contain the same information, just measured on different scales.

5.4

Posterior median is optimal estimate under L1 loss

To prove this, we expand the posterior expected loss as follows: Z Z ρ(a|x) = Eθ|x |θ − a| = (θ − a)p(θ|x)dθ + (a − θ)p(θ|x)dθ θ≥a θ≤a Z ∞ Z a = (θ − a)p(θ|x)dθ + (a − θ)p(θ|x)dθ

(189) (190)

−∞

a

Now recall the rule to differentiate under the integral sign: d da

Z B(a)

Z B(a) φ(a, θ)dθ =

A(a)

φ0 (a, θ)dθ + φ(a, B(a))B 0 (a) + φ(a, A(a))A0 (a)

(191)

A(a)

d where φ0 (a, θ) = da φ(a, θ). Applying this to the first integral in Equation 190, with A(a) = a, B(a) = ∞, φ(a, θ) = (θ − a)p(θ|x)dy, we have Z ∞ Z ∞ (θ − a)p(θ|x)dθ = −p(θ|x)dθ + 0 + 0 (192) a

a

Analogously, one can show

Z a

Z a (a − θ)p(θ|x)dθ =

−∞

p(θ|x)dθ

(193)

−∞

Hence Z ∞

Z a p(θ|x)dθ

(194)

= −P (θ ≥ a|x) + P (θ ≤ a|x) = 0

(195)

0

ρ (a|x) = −

p(θ|x)dθ + −∞

a

So the value of a that makes ρ0 (a|x) = 0 satisfies P (θ ≥ a|x) = P (θ ≤ a|x) Hence the optimal a is the posterior median.

22

(196)

6

Solutions

6.1

Expressing mutual information in terms of entropies

PRIVATE

I(X, Y ) =

X

p(x, y) log

p(x, y) p(x)p(y)

(197)

p(x, y) log

p(x|y) p(x)

(198)

x,y

=

X x,y

=−

X

X

p(x, y) log p(x) +

x,y

(199)

p(x, y) log p(x|y)

x,y

! =−

X

p(x) log p(x) −

−

X

x

x,y

X

X

(200)

p(x, y) log p(x|y) !

=−

p(x) log p(x) −

−

x

p(y)

X

y

p(x|y) log p(x|y)

(201)

x

(202)

= H(X) − H(X|Y ) We can show I(X, Y ) = H(Y ) − H(Y |X) by symmetry.

For the second. Using H(Y |X) = H(X, Y ) − H(X) we have I(X; Y ) = H(X) + H(Y ) − H(X, Y )

(203)

Hence adding and subtracting H(Y ) we have H(X, Y ) = H(X, Y ) + H(X, Y ) − H(X, Y ) = (H(X, Y ) − H(Y )) + H(X, Y ) + (H(Y ) − H(X, Y ))

(204) (205)

Now adding and subtracting H(X) we have H(X, Y ) = (H(X, Y ) − H(Y )) + (H(X, Y ) − H(X)) + (H(Y ) + H(X) − H(X, Y ))

(206)

Simplifying yields H(X, Y ) = H(X|Y ) + H(Y |X) + I(X, Y )

23

(207)

6.2

Relationship between D(p||q) and χ2 statistic

We have D(p||q) =

X

p(x) q(x)

p(x) log

x

(208)

∆(x) = (Q(x) + ∆(x)) log 1 + q(x) x

X ∆(x) ∆(x)2 − + ··· = (Q(x) + ∆(x)) q(x) 2q(x) x X

=

X

∆(x) +

x

=0+

X ∆(x)2 x

since

X

2q(x)

∆(x) =

x

6.3

∆(x)2 ∆(x)2 − + ··· q(x) 2q(x) + ···

X

p(x) − q(x) = 0

(209) (210) (211) (212)

(213)

x

Fun with entropies

PRIVATE Here is p(x, y) and the marginals p(x) and p(y) written in the margins (hence the name). x p(y) 1 2 3 4 1 1/8 1/16 1/32 1/32 1/4 y 2 1/16 1/8 1/32 1/32 1/4 3 1/16 1/16 1/16 1/16 1/4 4 1/4 0 0 0 1/4 p(x) 1/2 1/4 1/8 1/8 Here is p(x|y) and H(X|y) x H(X|y) 1 2 3 4 1 1/2 1/4 1/8 1/8 7/4 y 2 1/4 1/2 1/8 1/8 7/4 3 1/4 1/4 1/4 1/4 2 4 1 0 0 0 0 1. H(X, Y ) = 27/8 = 3.375 2. H(X) = 7/4 = 1.75, H(Y ) = 2.0.

3. H(X|y) = 1.75, 1.75, 2.0, 0.0. Note that H(X|Y = 4) = 0 < H(X), so observing Y = 4 decreases our entropy, but H(X|Y = 3) = 2.0 > H(X), so observing Y = 3 increases our entropy. Also, H(X|Y = 1) = H(X|Y = 2) = H(X), so observing Y = 1 or Y = 2 does not change our entropy (but does change our posterior distribution!).

4. H(X|Y ) = 11/8 = 1.375 < H(X), so on average, we are less uncertain after seeing Y .

5. I(X, Y ) = H(X) − HX|Y ) = 3/8 = 0.375

24

6.4

Forwards vs reverse KL divergence

1. We have KL (pkq) =

X

(214)

p(x, y)[log p(x, y) − log q(x) − log q(y)]

xy

=

X

p(x, y) log p(x, y) −

X

xy

p(x) log q(x) −

x

X

p(y) log q(y)

(215)

y

We can P optimize wrt q(x) and q(y) separately. Imposing a Lagrange multiplier to enforce the constraint that x q(x) = 1 we have the Lagrangian L(q, λ) =

X

p(x) log q(x) + λ(1 −

x

X

q(x))

(216)

x

Taking derivatives wrt q(x) (thinking of the function as a finite length vector, for simplicity), we have ∂L p(x) = −λ=0 ∂q(x) q(x) p(x) q(x) = λ

(217) (218)

Summing both sides over x we get λ = 1 and hence (219)

q(x) = p(x) Analogously, q(y) = p(y).

2. We require q(x, y) = 0 whenever p(x, y) = 0, otherwise log q(x, y)/p(x, y) = ∞. Since q(x, y) = qx (x)qy (y), it must be that qx (x) = qy (y) whenever x = y, and hence qx = qy are the same distribution.

There are only 3 possible distributions that put 0s in the right places and yet sum to 1. The first is: x 1 2 3 4 q(y) 1 0 0 0 0 0 0 0 0 0 0 y 2 3 0 0 1 0 1 4 0 0 0 0 0 q(x) 0 0 1 0 The second one is

y

1 2 3 4 q(x)

1 0 0 0 0 0

x 2 0 0 0 0 0

3 0 0 0 0 0

4 0 0 0 1 1

q(y) 0 0 0 1

1 For both of these, we have KL (qkp) = 1 × log 1/4 = log 4. Furthermore, any slight perturbation of these probabilities away from the designated values will cause the KL to blow up, meaning these are local minima.

The final local optimum is

25

y

1 2 3 4 q(x)

1 1/4 1/4 0 0 1/2

x 2 1/4 1/4 0 0 1/2

3 0 0 0 0 0

4 0 0 0 0 0

q(y) 1/2 1/2 0 0

This has KL (qkp) = 4( 14 log 1/4 1/8 ) = log 2, so this is actually the global optimum.

To see that there are no other solutions, one can do a case analysis, and see that any other distribution will not put 0s in the right places. For example, consider this: x 1 2 3 4 q(y) 1 1/4 0 1/4 0 1/2 y 2 0 0 0 0 0 1/4 0 1/4 0 1/2 3 4 0 0 0 0 0 q(x) 1/2 0 1/2 0 Obviously if we set q(x, y) = p(x)p(y) = 1/16, we get KL (qkp) = ∞.

26

7

Solutions

7.1

Orthogonal matrices

1. Let c = cos(α) and s = sin(α). Using the fact that c2 + s2 = 1, we have     2   1 c s 0 c −s 0 c + s2 + 0 −cs + sc + 0 0 RT R = −s c 0 s c 0 = −sc + sc + 0 c2 + s2 + 0 0 = 0 0 0 1 0 0 1 0 0 1 0

0 1 0

 0 0 1

(220)

2. The z-axis v = (0, 0, 1) is not affected by a rotation around z. We can easily check that (0, 0, 1) is an eigenvector with eigenvalue 1 as follows:      c s 0 0 0 −s c 0 0 = 0 (221) 0 0 1 1 1 Of course, (0, 0, −1) is also a valid solution. We can check this using the symbolic math toolbox in Matlab: syms c s x y z S=solve(’c*x-s*y=x’,’s*x+c*y=y’,’x^2+y^2+z^2=1’) >> S.x ans = 0 0 >> S.y ans = 0 0 >> S.z ans = 1 -1 If we ignore the unit norm constraint, we find that (0, 0, z) is a solution for any z ∈ R. We can see this as follows:      c s 0 x x −s c 0 y  = y  (222) 0 0 1 z z becomes cx − sy = x

(223)

sx + cy = y

(224)

z=z

(225)

Solving gives x(c − 1) s cx(x − 1) sx + cy = sx + s x(c − 1) y= s and hence x = y = 0. We can check this in Matlab: y=

27

(226) (227) (228)

S=solve(’c*x-s*y=x’,’s*x+c*y=y’) S = x: [1x1 sym] y: [1x1 sym] >> S.x 0 >> S.y 0

7.2

Eigenvectors by hand

PRIVATE In matlab, we find >> [V,D]=eig([2 0; 0 3]) V = 1 0 0 1 D = 2 0 0 3 so λ1 = 2, λ2 = 3, v1 = (1, 0)T , v2 = (0, 1)T . Let us check this by hand. We want to solve Avi = λi vi (A − λi I)vi = 0

(229) (230)

The characteristic equation is

|A − λI| = 0

0 =0 3 − λ2

(231)

(2 − λ1 )(3 − λ2 ) = 0

(233)

2 − λ1 0

so we see λ1 = 2,λ2 = 3. Solving for the eigenvectors

2 0 vi1 v = λi i1 0 3 vi2 vi2

(232)

(234)

yields 2v11 = 2(v11 + v12 )

(235)

3v22 = 3(v21 + v22 )

(236)

so v11 = 1, v12 = 0, v21 = 0, v22 = 1.

28

8

Solutions

8.1

Subderivative of the hinge loss function

PRIVATE At x = 0 and x = 2 the function is differentiable and ∂f (0) = {−1} and ∂f (2) = {0}. At x = 1 there is a kink, and ∂f (1) = [−1, 0].

8.2

EM for the Student distribution

At first blush, it might not be apparent why EM can be used, since there is no missing data. The key idea is to introduce an “artificial” hidden or auxiliary variable in order to simplify the algorithm. In particular, we will exploit the fact that a Student distribution can be written as a Gaussian scale mixture [AM74; Wes87] as follows: Z ν ν T (xn |µ, Σ, ν) = N (xn |µ, Σ/zn )Ga(zn | , )dzn (237) 2 2 This can be thought of as an “infinite” mixture of Gaussians, each one with a slightly different covariance matrix.

Treating the zn as missing data, we can write the complete data log likelihood as `c (θ) =

N X

[log N (xn |µ, Σ/zn ) + log Ga(zn |ν/2, ν/2)]

(238)

n=1 N X 1 zn ν ν ν D − log(2π) − log |Σ| − δn + log − log Γ( ) 2 2 2 2 2 2 n=1

ν D + (log zn − zn ) + ( − 1) log zn 2 2

=

(239) (240)

where we have defined the Mahalanobis distance to be δn = (xn − µ)T Σ−1 (xn − µ)

(241)

We can partition this into two terms, one involving µ and Σ, and the other involving ν. We have, dropping irrelevant constants, (242)

`c (θ) = LN (µ, Σ) + LG (ν) N X

1 1 LN (µ, Σ) , − N log |Σ| − zn δn 2 2 n=1 N 1 1 X LG (ν) , −N log Γ(ν/2) + N ν log(ν/2) + ν (log zn − zn ) 2 2 n=1

(243) (244)

Let us first derive the algorithm with ν assumed known, for simplicity. In this case, we can ignore the LG term, so we only need to figure out how to compute E [zn ] wrt the old parameters.

One can show that p(zn |xn , θ) = Ga(zn |

ν + D ν + δn , ) 2 2

(245)

Now if zn ∼ Ga(a, b), then E [zn ] = a/b. Hence the E step at iteration t is h i ν (t) + D (t) z (t) = n , E zn |xn , θ (t) ν (t) + δn 29

(246)

The M step is obtained by maximizing E [LN (µ, Σ)] to yield (t+1)

µ̂

P

(t)

n = P

z n xn

(247)

(t) n zn

1 X (t) z (xn − µ̂(t+1) )(xn − µ̂(t+1) )T N n n ! # " N X 1 X (t) z xn xTn − z (t) µ̂(t+1) (µ̂(t+1) )T = n N n n n=1

Σ̂(t+1) =

(248) (249)

These results are quite intuitive: the quantity z n is the precision of measurement n, so if it is small, the corresponding data point is down-weighted when estimating the mean and covariance. This is how the Student achieves robustness to outliers.

To compute the MLE for the degrees of freedom, we first need to compute the expectation of LG (ν), which involves zn and log zn . Now if zn ∼ Ga(a, b), then one can show that h i (t) `n , E log zn |θ (t) = ψ(a) − log b (250) d where ψ(x) = dx log Γ(x) is the digamma function. Hence, from Equation (245), we have (t)

(t)

ν (t) + D ν (t) + δn ) − log( ) 2 2 ν (t) + D ν (t) + D = log(z (t) ) − log( ) n ) + Ψ( 2 2

`n = Ψ(

(251) (252)

Substituting into Equation (244), we have E [LG (ν)] = −N log Γ(ν/2) +

Nν ν X (t) log(ν/2) + (` − z (t) n ) 2 2 n n

(253)

The gradient of this expression is equal to N N N 1 X (t) d E [LG (ν)] = − Ψ(ν/2) + log(ν/2) + + (` − z (t) n ) dν 2 2 2 2 n n

(254)

This has a unique solution in the interval (0, +∞] which can be found using a 1d constrained optimizer.

Performing a gradient-based optimization in the M step, rather than a closed-form update, is an example of what is known as the generalized EM algorithm.

30

Part II

Linear models

31

9

Solutions

9.1

Derivation of Fisher’s linear discriminant

We have f = wT SB w

(255)

0

(256)

f = 2SB w

(257)

T

g = w SW w

(258)

0

g = 2SW w T

T

dJ(w) (2sB w)(w SW w) − (w SB w)(2SW w) = =0 dw (wT SW w)T (wT SW w)

(259)

(sB w) (wT SW w) = (wT SB w)(SW w) | {z } | {z }

(260)

aSB w = bSW w b SB w = SW w a

(261)

Hence

a

b

32

(262)

10

Solutions

10.1

Gradient and Hessian of log-likelihood for multinomial logistic regression

P 1. Let us drop the i subscript for simplicity. Let S = k eηk be the denominator of the softmax.

∂ ηk −1 ∂µk ∂ = e S] · −S −2 S + e ηk · [ (263) ∂ηj ∂ηj ∂ηj = (δij eηk )S −1 − eηk eηj S −2

e ηj e ηk δij − = S S

(264)

= µk (δkj − µj )

(266)

(265)

2. We have X X yik X X ∂` ∂µik ∂ηij = µik (δjk − µij )xi ∂µik ∂ηij ∂wj µik i i k k XX X XX = yik (δjk − µij )xi = yij xi − ( yik )µij xi

∇wj ` =

i

=

i

k

X

i

(267) (268)

k

(269)

(yij − µij )xi

i

3. We consider a single term xi in the log likelihood; we can sum over i at the end. Using the Jacobian expression from above, we have ∇wc0 (∇wc `)T = ∇wc0 ((yic − µic )xTi )

10.2

(270)

= −(∇wc0 µic )xTi

(271)

= −(µic (δc,c0 − µi,c0 )xi )xTi

(272)

Regularizing separate terms in 2d logistic regression

PRIVATE 1. Unregularized solution: see Figure 1(a). There is a unique ML decision boundary which makes 0 errors, but there are several possible decision boundaries which all makes 0 errors.

2. Heavily regularizing w0 sets w0 = 0, so the line must go through the origin. There are several possible lines with different slopes, but all will make 1 error. see Figure 1(b).

3. Regularizing w1 makes the line horizontal since x1 is ignored. This incurs 2 errors. see Figure 1(c).

4. Regularizing w2 makes the line vertical, since x2 is ignored. This incurs 0 errors. see Figure 1(d).

10.3

Logistic regression vs LDA/QDA

PRIVATE 1. GaussI ≤ LinLog. Both have logistic (sigmoid) posteriors p(y|x, w) = σ(ywT x), but LinLog is the logistic model which is trained to maximize p(y|x, w). (GaussI may have high joint p(y, x), but this does not necessarily mean p(y|x) is high; LinLog can achieve the maximum of p(y|x), so will necessarily do at least as well as GaussI.) 33

(a)

(b)

(c)

(d)

Figure 1: (a) Unregularized. (b) Regularizing w0 . (c) Regularizing w1 . (d) Regularizing w2 .

2. GaussX ≤ QuadLog. Both have logistic posteriors with quadratic features, but QuadLog is the model of this class maximizing the average log probabilities.

3. LinLog ≤ QuadLog. Logistic regression models with linear features are a subclass of logistic regression models with quadratic functions. The maximum from the superclass is at least as high as the maximum from the subclass.

4. GaussI ≤ QuadLog. Follows from above inequalities.

5. Although one might expect that higher log likelihood results in better classification performance, in general, having higher average log p(y|x) does not necessarily translate to higher or lower classification error. For example, consider linearly separable data. We have L(linLog) > L(GaussI), since maximum likelihood logistic regression will set the weights to infinity, to maximize the probability of the correct labels (hence p(yi |xi , ŵ) = 1 for all i). However, we have R(linLog) = R(gaussI), since the data is linearly separable. (The GaussI model may or may not set σ very small, resulting in possibly very large class conditional pdfs; however, the posterior over y is a discrete pmf, and can never exceed 1.)

As another example, suppose the true label is always 1 (as opposed to 0), but model M always predicts p(y = 1|x, M ) = 0.49. It will always misclassify, but it is at least close to the decision boundary.

By contrast, there might be another model M 0 that predicts p(y = 1|x, M 0 ) = 1 on even-numbered inputs, and p(y = 1|x, M 0 ) = 0 on odd-numbered inputs. Clearly R(M 0 ) = 0.5 < R(M ) = 1, but L(M 0 ) = −∞ < L(M ) = log(0.49).

34

11

Solutions

11.1

Multi-output linear regression

PRIVATE We have Ŵ = (ΦT Φ)−1 ΦT Y

1 1 1 0 0 T Φ = 0 0 0 1 1

3 0 ΦT Φ = 0 3   −1 −1 −1 −2   −2 −1  Y= 1 1   1 2 2 1

−4 −4 ΦT Y = 4 4

(273) 0 1

(274) (275)

(276)

(277) (278)

(ΦT Φ)−1 = (1/3)I2 Ŵ = (ΦT Φ)−1 ΦT Y =

11.2

−4/3 4/3

−4/3 4/3

(279)

Centering and ridge regression

Suppose X is centered, so x = 0. Then (280)

J(w, w0 ) = (y − Xw − w0 1)T (y − Xw − w0 1) + λwT w T

T

T

T

T

T

T

T

= y y + w (X X)w − 2y (Xw) + λw w + −2w0 1 y + 2w0 1 Xw + w0 1 1w0

(281)

Consider the terms in brackets: w0 1T y = w0 ny X w0 1T Xw = w0 xTi w = nxT w = 0

(282) (283)

i

(284)

w0 1T 1w0 = nw02 Optimizing wrt w0 we find ∂ J(w, w0 ) = −2ny + 2nw0 = 0 ∂w0 ŵ0 = y

(285) (286)

Optimizing wrt w we find ∂ J(w, ŵ0 ) = [2XT Xw − 2XT y] + 2λw = 0 ∂w w = (XT X + λI)−1 XT y

35

(287) (288)

11.3

Partial derivative of the RSS

PRIVATE 1. We have RSS(w) =

X (ri − wk xik )(ri − wk xik )

(289)

i

=

X [ri2 − wk2 x2ik − 2wk xik ri ]

(290)

i

So X ∂ RSS(w) = [−2wk x2ik − 2xik ri ] ∂wk i

(291)

Let us check this matches the standard expression (292)

∇w = 2XT Xw − 2XT y Recall that [XT X]jk =

i xij xik . For simplicity, suppose there are two features. Then

P

X X X X X [∇w ]1 = 2[ x2i1 , xi1 xi2 ]T w − 2[ xi1 yi ] = 2w1 x2i1 + 2 xi1 (w2 xi2 − yi ) = a1 w1 − c1 (293) i

i

i

i

i

2. Setting the derivative to zero, we have X X wk ( x2ik ) = (xik ri ) i

so wk =

11.4

xT:,k r xT:,k x:,k

(295)

Reducing elastic net to lasso

We have and

11.5

(294)

i

J1 (w) = y T y + (Xw)T (Xw) − 2y T (Xw) + λ2 wT w + λ1 |w|1

(296)

J2 (w) = y T y + c2 (Xw)T (Xw) − 2c2 y T (Xw) + λ2 c2 wT w + cλ1 |w|1 = J1 (cw)

(297)

Shrinkage in linear regression

PRIVATE 1. From the book, we have that ŵkOLS = xTk y = ck /2. Hence the solid line (1) is OLS and has slope 1/2. ck Also, ŵkridge = ŵkOLS /(1 + λ2 ) = 2(1+λ . Hence the dotted line (2) is ridge and has slope 1/4. Finally, 2) lasso must be the dashed line (3), since it sets coefficients to zero if −λ1 ≤ ck ≤ λ1 .

2. λ1 = 1 3. λ2 = 1 36

11.6

EM for mixture of linear regression experts

In the E step, we compute the conditional responsibilities rnk = p(zn = k|xn , yn ) =

p(yn |xn , zn = k)p(zn = k|xn ) p(yn |xn )

(298)

In the M step, we update the parameters of the gating funtion by maximizing the weighted likelihood XX `(θg ) = rnk log p(zn = k|xn , θg ) (299) n

k

and the parameters of the k’th expert by maximizing the weighted likelihood X `(θk ) = rnk log p(yn |xn , zn = k, θk )

(300)

n

If the gating function and experts are linear models, these M steps correspond to convex subproblems that can be solved efficiently.

For example, consider a mixture of linear regression experts using logistic regression gating functions. In the M step, we need to maximize Q(θ, θ old ) wrt wk , σk2 and V. For the regression parameters for model k, the objective has the form

N X 1 (301) Q(θk , θ old ) = rnk − 2 (yn − wTk xn ) σk n=1 We recognize this as a weighted least squares problem, which makes intuitive sense: if rnk is small, then data point n will be downweighted when estimating model k’s parameters. From the weighted least square section, we can immediately write down the MLE as wk = (XT Rk X)−1 XT Rk y

(302)

where Rk = diag(r:,k ). The MLE for the variance is given by σk2 =

PN

T 2 n=1 rnk (yn − wk xn ) PN n=1 rnk

(303)

We replace the estimate of the unconditional mixing weights π with the estimate of the gating parameters, V. The objective has the form XX `(V) = rnk log πn,k (304) n

k

We recognize this as equivalent to the log-likelihood for multinomial logistic regression, except we replace the “hard” 1-of-C encoding yi with the “soft” 1-of-K encoding ri . Thus we can estimate V by fitting a logistic regression model to soft target labels.

37

12

Solutions

38

Part III

Deep neural networks

39

13

Solutions

13.1

Backpropagation for a 1 layer MLP

To compute δ 1 , let L = CrossEntropyWithLogits(y, a). Then ∂L = (p − y)T ∂a

(305)

∂L ∂a ∂h ∂a ∂h ∂L = = δ1 ∂z ∂a ∂h ∂z ∂h ∂z ∂h = δ1 U since a = Uh + b2 ∂z = δ 1 U ◦ ReLU0 (z) since h = ReLU(z)

(306)

= δ 1 U ◦ H(h)

(309)

δ1 = where p = S(a).

To compute δ 2 , we have δ2 =

(307) (308)

Now we compute the gradients wrt the parameters: ∂L ∂a ∂L = = δ 1 hT ∂U ∂a ∂U ∂L ∂L ∂a = = δ1 ∂b2 ∂a ∂b2 ∂L ∂z ∂L = = δ 2 xT ∂W ∂z ∂W ∂L ∂L ∂z = = δ2 ∂b1 ∂z ∂b1 ∂L ∂z ∂L = = δ2 W ∂x ∂z ∂x

40

(310) (311) (312) (313) (314)

14

Solutions

41

15

Solutions

42

Part IV

Nonparametric models

43

16

Solutions

44

17

Solutions

17.1

Fitting an SVM classifier by hand

PRIVATE 1. The perpendicular to the decision boundary is a line through φ(x1 ) to φ(x2 ), and hence is parallel to φ(x2 ) − φ(x1 ) = (1, 2, 2) − (1, 0, 0) = (0, 2, 2). Of course, any scalar multiple of this is acceptable, e.g., (0, 1, 1).

2. There are 2 support vectors, namely the two data points. The decision boundary will be half way between them. This midpoint is m = (1,2,2)+(1,0,0) = (1, 1, 1). The distance of each of the training 2 points to this midpoint is p √ ||φ(x1 ) − m|| = ||φ(x2 ) − m|| = ||(0, 1, 1)|| = 02 + 12 + 12 = 2 (315) √ Hence the margin is 2. p √ 3. We have w = (0, 1/2, 1/2), which is parallel to (0, 2, 2) and has ||w|| = (1/2)2 + (1/2)2 = 1/ 2 as required.

4. Using the equations we have (316)

−1([1, 0, 0]T w + w0 ) = 1

(317)

T

+1([1, 2, 2] w + w0 ) = 1 Hence w0 = −1.

5. We have T

T

√

√ 2

w0 + w φ(x) = −1 + (0, 1/2, 1/2) (1, 2x, x ) = −1 +

45

1 2 x + x2 2 2

(318)

18

Solutions

46

Part V

Beyond supervised learning

47

19

Solutions

19.1

Information gain equations

To see this, let us define pn , p(θ|D) as the current belief state, and pn+1 , p(θ|D, x, y) as the belief state after observing (x, y). Then Z pn+1 KL (pn+1 kpn ) = pn+1 log dθ (319) pn Z Z = pn+1 log pn+1 dθ − pn+1 log pn dθ (320) Z = − H pn+1 − pn+1 log pn dθ (321) Next we need to take expectations wrt p(y|x, D). We will use the following identity: Z X p(y|x, D) pn+1 log pn dθ

(322)

y

=

X

Z p(y|x, D)

p(θ|D, x, y) log p(θ|D)dθ

(323)

y

=

Z X

p(y, θ|x, D) log p(θ|D)dθ

(324)

y

Z =

(325)

p(θ|D) log p(θ|D)dθ

Using this, we have Z

0

U (x) = Ep(y|x,D) [− H pn+1 ] −

pn log pn dθ

= Ep(y|x,D) [− H p(θ|x, y, D) + H p(θ|D)] = U (x)

48

(326) (327)

20

Solutions

20.1

EM for FA

In the E step, we can compute the posterior for zn as follows: (328)

p(zn |xn , θ) = N (zn |mn , Σn ) T

−1

T

−1

Σn , (I + W Ψ mn , Σn (W Ψ

−1

(329)

(xn − µ))

(330)

W)

We now discuss the M step. We initially assume µ = 0. Using the trace trick we have i X X h T −1

E (x̃i − Wzi )T Ψ−1 (x̃i − Wzi ) = x̃i Ψ x̃i + E ziT WT Ψ−1 Wzi − 2x̃Ti Ψ−1 WE [zi ] i

(331)

i

=

Xh

tr(Ψ−1 x̃i x̃Ti ) + tr(Ψ−1 WE zi ziT WT )

(332)

i

−tr(2Ψ−1 WE [zi ] x̃Ti )

i

(333)

, tr(Ψ−1 G(W))

(334)

Hence the expected complete data log likelihood is given by Q=

1 N log |Ψ−1 | − tr(Ψ−1 G(W)) 2 2

(335)

∂ ∂ Using the chain rule and the facts that ∂W tr(WT A) = A ∂W WT AW = (A + AT )W we have

1 ∇W Q(W) = − Ψ−1 ∇W G(W) = 0 2 X X

∇W G(W) = 2W E zi ziT − 2( E [zi ] x̃Ti )T i

Wmle =

(336) (337)

i

" X

#" #−1 X

T x̃i E [zi ] E zi zi

i

i

T

(338)

Using the facts that ∇X log |X| = X−T and ∇X tr(XA) = AT we have 1 N Ψ − G(Wmle ) = 0 2 2 1 Ψ = diag(G(Wmle )) N

∇Ψ−1 Q =

(339) (340)

We can simplify this as follows, by plugging in the MLE (this simplification no longer holds if we use MAP estimation). First note that X X T E zi ziT Wmle = E [zi ] x̃Ti (341) i

i

so

1 X x̃i x̃Ti + Wmle E [zi ] x̃Ti − 2Wmle E [zi ] x̃Ti N i !

X 1 X T T x̃i x̃i − Wmle = E [zi ] x̃i N i i

Ψ=

49

(342) (343)

To estimate µ and W at the same time, we can define W̃ = (W, µ) and z̃ = (z, 1), Also, define bn , E [z̃|xn ] = [mn ; 1] h i E zzT |x E [z|x ] n n T Ωn , E z̃z̃ |xn = T E [z|xn ] 1

(344) (345)

Then the M step is as follows: " #" #−1 X X ˆ T W̃ = x n bn Ωn n

(346)

n

( )

X 1 ˆ T xn − W̃b Ψ̂ = diag n xn N n

(347)

It is interesting to apply the above equations to the PPCA case in the limit where σ 2 → 0. This provides an alternative way to fit PCA models, as shown by [Row97]. Let Z̃ be a L × N matrix storing the posterior means (low-dimensional representations) along its columns. Similarly, let X̃ = XT be an D × N matrix storing the original data along its columns. From the PPCA posterior, when σ 2 = 0, we have (348)

Z̃ = (WT W)−1 WT X̃ This constitutes the E step. Notice that this is just an orthogonal projection of the data.

From Equation (346), the M step is given by Ŵ =

" X

T

xn E [zn ]

#" X

n

#−1 T

E [zn ] E [zn ]

(349)

n

where we exploited the fact that Σ = Cov [zn |xn , θ] = 0I when σ 2 = 0. It P is worth comparing this expression P to the MLE for multi-output linear regression, which has the form W = ( n yn xTn )( n xn xTn )−1 . Thus we see that the M step is like linear regression where we replace the observed inputs by the expected values of the latent variables.

In summary, here is the entire algorithm: • E step: Z̃ = (WT W)−1 WT X̃ T

T

• M step: W = X̃Z̃ (Z̃Z̃ )−1 [TB99] showed that the only stable fixed point of the EM algorithm is the globally optimal solution.

That is, the EM algorithm converges to a solution where W spans the same linear subspace as that defined by the first L eigenvectors. However, if we want W to be orthogonal, and to contain the eigenvectors in descending order of eigenvalue, we have to orthogonalize the resulting matrix (which can be done quite cheaply). Alternatively, we can modify EM to give the principal basis directly [AO03].

20.2

EM for mixFA

PRIVATE We state the results below without proof (the derivation can be found in [GH96]).

In the E step, we compute the posterior responsibility of cluster c for data point n using rnc , p(sn = c|xn , θ) ∝ πc N (xn |µc , Wc WTc + Ψ)

50

(350)

The conditional posterior for zn is given by p(zn |xn , sn = c, θ) = N (zn |mnc , Σnc ) −1 Σnc , (I + WTc Ψ−1 c Wc ) mnc , Σnc (WTc Ψ−1 c (xn − µc ))

(351) (352) (353)

For the M step, we define W̃c = (Wc , µc ), and z̃ = (z, 1). Also, define bnc , E [z̃|xn , cn = c] = [mnc ; 1] h i E zzT |x , c = c E [z|x , c = c] n n n n T Ωnc , E z̃z̃ |xn , cn = c = T E [z|xn , cn = c] 1

(354) (355)

Then the M step is as follows: " #" #−1 X X ˆ T W̃c = rnc xn bc rnc Ωnc n

( )

X 1 ˆ T rnc xn − W̃c bnc xn Ψ̂ = diag N nc 1 X rnc π̂c = N n

20.3

(356)

n

(357) (358)

Deriving the second principal component

1. Dropping terms that no not involve z2 we have J=

n n

1 X 1 X 2 T 2 −2zi2 v2T (xi − zi1 v1 ) + zi2 v2 v2 = −2zi2 v2T xi + zi2 n i=1 n i=1

(359)

since v2T v2 = 1 and v1T v2 = 0. Hence

so 2. We have

∂J = −2v2T xi + 2zi2 = 0 ∂zi2

(360)

zi2 = v2T xi

(361)

∂ J˜ = −2Cv2 + 2λ2 v2 + λ12 v1 = 0 ∂v2

(362)

0 = −2v1T Cv2 + 2λ2 v1T v2 + λ12 v1T v1

(363)

Premultiplying by v1T yields

Now v1T Cv2 = v1T (λ1 v2 ) = 0, and v1T v2 = 0, and v1T v1 = 1, so λ12 = 0. Hence 0 = −2Cv2 + 2λ2 v2 Cv2 = λ2 v2

(364) (365)

So v2 is an eigenvector of C. Since we want to maximize the variance, we want to pick the eigenvector with the largest eigenvalue, but the first one is already taken. Hence v2 is the evector with the second largest evalue.

51

20.4

Deriving the residual error for PCA

PRIVATE 1. From the previous exercise we have z z * i1

* i2 2

2 (xi − zi1 v1 − zi2 v2 )T (xi − zi1 v1 − zi2 v2 ) = xTi xi − 2zi1 xTi v1 − 2zi2 xTi v2 + zi1 + zi2 2 2 = xTi xi − zi1 − zi2 = xTi xi − v1T xTi xTi v1 − v2T xTi xTi v2

(366) (367) (368)

This generalizes to K > 2 in the obvious way.

2. Hence   n K X 1 X T xi xi − vjT xi xTi vj  JK = n i=1 j=1 =

(369)

n n X 1X 1X T xi xi − vjT ( xi xTi )vj n i=1 n j i=1

(370)

n

X 1X T xi xi − = vjT Cvj n i=1 j

(371)

n K X 1X T xi xi − λj n i=1 j=1

(372)

=

3. If K = d there is no truncation and Jd = 0. Hence 0=

n K d d X X X 1X T xi xi − λj − λj = JK − λj n i=1 j=1 j=K+1

(373)

j=K+1

Hence the residual error arising from only using K < d terms is JK =

d X

λj

(374)

j=K+1

20.5

PCA via successive deflation

1. We have

1 (I − v1 v1T )XT X(I − v1 v1T ) n

1 T = (X X − v1 v1T XT X)(I − v1 v1T ) n

1 T = X X − v1 (v1T XT X) − (XT Xv1 )v1T + v1 (v1T XT Xv1 )v1T n

1 T = X X − v1 (nλ1 v1T ) − (nλ1 v1 )v1T + v1 (v1T nλ1 v1 )v1T n

1 T = X X − nλ1 v1 v1T − nλ1 v1 v1T + nλ1 v1 v1T n 1 = XT X − λ1 v1 v1T n

C̃ =

52

(375) (376) (377) (378) (379) (380)

2. Since X̃ lives in the d − 1 subspace orthogonal to v1 , the vector u must be orthogonal to v1 . Hence uT v1 = 0 and uT u = 1, so u = v2 .

3. We have function [V, lambda] = simplePCA(C, K, f) d = length(C); V = zeros(d,K); for j=1:K [lambda(j), V(:,j) = f(C); C = C - lambda(j)*V(:,j)*V(:,j)’; % deflation end

20.6

PPCA variance terms 1

Define A = (ΛK − σ 2 I) 2 .

1. We have v T Cv = v T (UARRT AT UT + σ 2 I)v = v T σ 2 Iv = σ 2

(381)

2. We have v T Cv = uTi (UARRT AT UT + σ 2 I)ui = uTi U(Λ − σ 2 I)UT ui + σ 2 = eTi (Λ − σ 2 I)ei + σ 2 = λi − σ 2 + σ 2

20.7

(382) (383) (384)

Posterior inference in PPCA

PRIVATE Using Bayes rule for Gaussians with the substitutions Λ = I, L−1 = σ 2 I, A = W, b = µ, µ = 0, we have

−1 1 2 T 1 −1 T Cov [z|y] = (I + W 2 w) = (σ I + W W) = σ 2 M−1 (385) σ σ2 and

E[z|y] = Cov [z|y] WT σ −2 (x − µ) = M−1 WT (x − µ)

20.8

(386)

Imputation in a FA model

PRIVATE Letting C = WWT + Ψ, we have −1 p(xh |xv , θ) = N (xh |µh + Chv C−1 vv (xv − µv ), Chh − Chv Cvv Cvh )

20.9

(387)

Efficiently evaluating the PPCA density

Since C is not full rank, we can use matrix inversion lemma to invert it efficiently: C−1 =

1 I − W WT W + σ 2 I WT 2 σ 53

(388)

Plugging in the MLE we find 1

W = UK (ΛK − σ 2 I) 2

1 T C−1 = 2 I − UK (ΛK − σ 2 I)Λ−1 K UK σ

1 = 2 I − UK JUTK σ J = diag(1 − σ 2 /λj )

(389) (390) (391) (392)

Similarly it can be shown that log |WWT + σ 2 I| = (d − K) log σ 2 +

K X i=1

54

log λi

(393)

21

Solutions

55

22

Solutions

56

23

Solutions

57