Machine Learning: A Bayesian and Optimization Perspective 2nd Edition By Sergios Theodoridis INSTRU

INSTRUCTOR'S SOLUTION MANUAL

1

Solutions To Problems of Chapter 2

2.1. Derive the mean and variance for the binomial distribution.

Solution: For the mean value we have that,

E[x]

= =

n X k=0 n X k=1

= np

kn! pk (1 − p)n−k (n − k)!k! n! pk (1 − p)n−k (n − k)!(k − 1)!

n X

(n − 1)!

pk−1 (1 − p)(n−1)−(k−1) (n − 1) − (k − 1) !(k − 1)!

k=1

= np

n−1 X l=0

(n − 1)! pl (1 − p)(n−1)−l (n − 1) − l !l!

= np(p + 1 − p)n−1 = np.

(1)

where the formula for the binomial expansion has been employed. For the variance we have,

σx2

= =

n X

(k − np)2

k=0 n X k=0 n X

k2

n! pk (1 − p)n−k + (n − k)!k!

(np)2

k=0

2np

n! pk (1 − p)n−k (n − k)!k!

n X k=0

n! pk (1 − p)n−k − (n − k)!k!

k

n! pk (1 − p)n−k , (n − k)!k!

(2)

or

σx2 =

n X k=0

k2

n! pk (1 − p)n−k + (np)2 − 2(np)2 , (n − k)!k!

(3)

2

However, n X

n! pk (1 − p)n−k = (n − k)!k!

k2

k=0

np

n X

k

k=1

np

(n − 1)!

pk−1 (1 − p)(n−1)−(k−1) = (n − 1) − (k − 1) !(k − 1)!

n−1 X

(l + 1)

l=0

(n − 1)! pl (1 − p)(n−1)−l = (n − 1) − l !l!

np + np(n − 1)p,

(4)

which finally proves the result.

2.2. Derive the mean and the variance for the uniform distribution.

Solution: For the mean we have Z b µ = E[x]

= =

1 xdx b − a a 1 b b b+a = . b−a2 a 2

(5)

For the variance, we have σx2

= = =

1 b−a

Z b

(x − µ)2 dx =

a

1 b−a

Z b−µ

y 2 dy

a−µ

1 y 3 b−µ b − a 3 a−µ 1 (b − a)2 .

12

(6)

2.3. Derive the mean and covariance matrix of the multivariate Gaussian.

Solution: Without harming generality, we assume that µ = 0, in order to simplify the discussion. We have that 1 (2π)l/2 |Σ|1/2

Z +∞

1 x exp − xT Σ −1 x dx, 2 −∞

(7)

which due to the symmetry of the exponential results in E[x] = 0.

For the covariance we have that Z +∞

1 exp − xT Σ −1 x dx = (2π)l/2 |Σ|1/2 .

2 −∞

(8)

3

Following similar arguments as for the univariate case given in the text, we are going to take the derivative on both sides with respect to matrix Σ. Recall from linear algebra the following formulas. ∂trace{AX −1 B} ∂|X k | = −(X −1 BAX −1 )T , = k|X k |X −T . ∂X ∂X Hence, taking the derivatives of both sides in (8) with respect to Σ we obtain, Z 1

T 1 1 +∞ exp − xT Σ −1 x Σ −1 xxT Σ −1 dx = (2π)l/2 |Σ|1/2 Σ −T , 2 −∞ 2 2 (9) which then readily gives the result.

2.4. Show that the mean and variance of the beta distribution with parameters a and b are given by a , E[x] = a+b and ab σx2 = . (a + b)2 (a + b + 1) Hint: Use the property Γ(a + 1) = aΓ(a).

Solution: We know that Γ(a + b) a−1 x (1 − x)b−1 . Γ(a)Γ(b)

Beta(x|a, b) = Hence E[x] =

Γ(a + b) Γ(a)Γ(b)

Z 1

xxa−1 (1 − x)b−1 dx =

0

Γ(a + b) Γ(a + 1)Γ(b) , Γ(a)Γ(b) Γ(a + 1 + b)

which, using the property Γ(a + 1) = aΓ(a), results in a .

E[x] = a+b

(10)

For the variance we have E[(x − E[x])2 ] =

Γ(a + b) Γ(a)Γ(b)

Z 1 x− 0

a a+b

2

xa−1 (1 − x)b−1 dx,

(11)

or Γ(a + b) σx2 =

Γ(a)Γ(b)

Z 1

xa+1 (1 − x)b−1 dx

0

Z a2 Γ(a + b) 1 a−1 x (1 − x)b−1 dx (a + b)2 Γ(a)Γ(b) 0 Z a Γ(a + b) 1 a −2 x (1 − x)b−1 dx, a + b Γ(a)Γ(b) 0 +

(12)

4

and following a similar path as the one adopted for the mean, it is a matter of simple algebra to show that σx2 =

ab (a + b)2 (a + b + 1)

.

2.5. Show that the normalizing constant in the beta distribution with parameters a, b is given by Γ(a + b) . Γ(a)Γ(b) Solution: The beta distribution is given by Beta(x|a, b) = Cxa−1 (1 − x)b−1 , 0 ≤ x ≤ 1.

(13)

Hence C

−1

Z 1 =

xa−1 (1 − x)b−1 dx.

(14)

0

Let x = sin2 θ ⇒ dx = 2 sin θ cos θdθ.

Hence C −1 = 2

Z π2

(15)

(sin θ)2a−1 (cos θ)2b−1 dθ.

(16)

0

Recall the definition of the gamma function Z ∞ Γ(a) = xa−1 e−x dx, 0

and set x = y 2 ⇒ dx = 2ydy, hence

Z ∞ Γ(a) = 2

2

y 2a−1 e−y dy.

(17)

0

Thus

Z ∞Z ∞ Γ(a)Γ(b) = 4 0

2

2

x2a−1 y 2b−1 e−(x +y ) dxdy.

(18)

0

Let x = r sin θ, y = r cos θ ⇒ dxdy = rdrdθ.

Hence Z π2 Z ∞ Γ(a)Γ(b) = 4 0

0

2

r2(a+b)−1 e−r (sin θ)2a−1 (cos θ)2a−1 drdθ.

(19)

5

where integration over θ is in the interval 0, π2 to guarantee that x remains non-negative. From (19) we have !

Z ∞

Z π2 2 Γ(a)Γ(b) = 2 r2(a+b)−1 e−r dr 2 (sin θ)2a−1 (cos θ)2b−1 dθ 0

0 −1

= Γ(a + b)C

,

which proves the claim.

2.6. Show that the mean and variance of the gamma pdf Gamma(x|a, b) =

ba a−1 −bx x e , a, b, x > 0. Γ(a)

are given by a , b a σx2 = 2 . b

E[x] =

Solution: We have that ba E[x] = Γ(a)

Z ∞

xa e−bx dx.

0

Set bx = y. Then Z ∞ ba 1 y a e−y dy Γ(a) ba+1 0 aΓ(a) a 1 Γ(a + 1) = = . = bΓ(a) bΓ(a) b

E[x] =

For the variance, the following is valid Z a ba n ∞ a+1 −bx σx2 = E[(x − )2 ] = x e dx b Γ(a) 0 Z Z a ∞ a −bx o a2 ∞ a−1 −bx x e dx − 2 x e dx , + 2 b 0 b 0 and following a similar path as before we obtain a σx2 = 2 . b 2.7. Show that the mean and variance of a Dirichlet pdf with K variables, xk , k = 1, 2, . . . , K and parameters ak , k = 1, 2, . . . , K, are given by ak E[xk ] = , k = 1, 2, . . . , K a ak (a − ak ) , k = 1, 2, . . . , K, σk2 = 2 a (1 + a) ai aj cov[xi xj ] = − 2 , i 6= j, a (1 + a)

6

where a =

PK

k=1 ak .

Solution: Without harm of generality, we will derive the mean for xK .

The others are derived similarly. To this end, we have

p(x1 , x2 , . . . , xK−1 ) = C

K−1 Y

xakk −1

1−

k=1

K−1 X

!aK −1 xk

k=1

where C=

Γ(a1 + a2 + . . . + aK ) . Γ(a1 )Γ(a2 ) . . . Γ(aK )

Z 1 "Z 1−PK−1 k=1 xk

Z 1 ···

E[xK ] = C

xK p(x1 , . . . , xK−1 , xK )dxK dxK−1 . . . dx1

0

Z 1 =C

0

0

 Z 1 Z 1−PK−1 K−1 Y a −1 k=1 xk  ··· xK x k

0

0

Z 1 ···

=C

#

Z 1 K−1 Y

0

0

1−

k

0

k=1

xakk −1

1−

k=1

K−1 X

!aK "Z

K−1 X

!aK −1 xk

k=1 PK−1 1− k=1 xk

xk

 dxK  dxK−1 . . . dx1

# dxK dxK−1 . . . dx1 ,

0

k=1

or Z 1 ···

E[xK ] = C 0

Z 1 K−1 Y 0

xakk −1

k=1

1−

K−1 X

!aK xk

dxK−1 . . . dx1

k=1

Γ(a1 ) . . . Γ(aK + 1) Γ(a1 + a2 + . . . + aK + 1) aK Γ(a1 ) . . . Γ(aK ) =C (a1 + a2 + . . . + aK )Γ(a1 + a2 + . . . + aK ) aK = . a =C

In the sequel, we will show that ai aj E[xi xj ] = − 2 , i 6= j. a (a + 1) We derive it for the variables xK and xK−1 , since any of the variables can