SOLUTIONS MANUAL to A First Course in Machine Learning 2nd Edition by Simon Rogers & Mark Girolami

SOLUTION MANUAL

SOLUTION MANUAL

solutions MANUAL FOR A First Course In Machine Learning (2Nd Edition) Exercise Solutions by

Simon Rogers and Mark Girolami

K26591_SM_Cover.indd 1

05/04/16 3:37 pm

K26591_SM_Cover.indd 2

05/04/16 3:37 pm

solutionS MANUAL FOR A First Course In Machine Learning (2Nd Edition) Exercise Solutions by

Simon Rogers and Mark Girolami

Boca Raton London New York

CRC Press is an imprint of the Taylor & Francis Group, an informa business

K26591_SM_Cover.indd 3

05/04/16 3:37 pm

CRC Press Taylor & Francis Group 6000 Broken Sound Parkway NW, Suite 300 Boca Raton, FL 33487-2742 © 2017 by Taylor & Francis Group, LLC CRC Press is an imprint of Taylor & Francis Group, an Informa business No claim to original U.S. Government works Printed on acid-free paper Version Date: 20160404 International Standard Book Number-13: 978-1-4987-3859-0 (Ancillary) This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint.

Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers.

For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged.

Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe.

Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com and the CRC Press Web site at http://www.crcpress.com

K26591_SM_Cover.indd 4

05/04/16 3:37 pm

Chapter 1

EX 1.1. A high positive value of w0 and a small negative value for w1 . These reflect the high intercept on the t axis (corresponding to the theoretical time winning time at x = 0 and the small decrese in winning time over the years.

EX 1.2. The following would do the job: 1 2 3 4 5 6 7 8 9 10 11 12 13

% Attributes are stored in Nx1 vector x % Targets are stored in Nx1 vector t xb = mean(x); tb = mean(t); x2b = mean(x.*x); xtb = mean(x.*t); w1 = (xtb − xt*xb)/(x2b−xbˆ2); w0 = tb−w1*xb; % Plot the data plot(x,t,'b.','markersize',25); % Plot the model hold on; plot(x,w0+w1*x,'r','linewidth',2);

EX 1.3. We need to find wT XT Xw. We’ll start with XT X. Multiplying XT by X gives:

N x2 N n=1 n1 n=1 xn2 xn1

N N w0 n=1 x2n1 + w1 n=1 xn1 xn2 N N w0 n=1 xn2 xn1 + w1 n=1 x2n2

T

X X=

N xn1 xn2 n=1 N 2 n=1 xn2

Multiplying this by w gives:

XT Xw =

1

K26591_SM_Cover.indd 5

05/04/16 3:37 pm

2

CHAPTER 1.

Finally, pre-multiplying this by wT gives:

N N

T T 2 w X Xw = w0 w0 xn1 + w1 xn1 xn2 + n=1

w1 w0 =

w02

N

n=1

xn2 xn1 + w1

n=1

N

n=1

x2n1 + 2w0 w1

n=1

N

N

x2n2

xn1 xn2 + w12

n=1

N

x2n2

n=1

as required.

EX 1.4. Let’s first work out Xw: 

Therefore

  Xw =  

w0 x11 + w1 x12 w0 x21 + w1 x22 .. . w0 xN 1 + w 1 xN 2

    

(Xw)T = [w0 x11 + w1 x12 , w0 x21 + w1 x22 , . . . , w0 xN 1 + w1 xN 2 ] Finally, work out wT XT : wT XT = [w0 x11 + w1 x12 , w0 x21 + w1 x22 , . . . , w0 xN 1 + w1 xN 2 ] as required.

EX 1.5. Starting with n xn tn . The result of this is a column vector of the same size as x (2 × 1). Now, using the definition of X,

x11 , x21 , . . . , xN 1 T X = x12 , x22 , . . . , xN 2 (which is a 2 × N vector). Multiplying this by t gives a 2 × 1 vector that looks like this:

N xn1 tn T n=1 X t = N n=1 xn2 tn

which is n xn tn as required. The second example, XT Xw. We already know what XT Xw is (Exercise 1.3)

N N w0 n=1 x2n1 + w1 n=1 xn1 xn2 T N N X Xw = w0 n=1 xn2 xn1 + w1 n=1 x2n2 Now, xn xT n is the following matrix:

= xn xT n

K26591_SM_Cover.indd 6

x2n1 xn2 xn1

xn1 xn2 x2n2

05/04/16 3:37 pm

3 Multiplying this by w gives: xn xT nw =

w0 x2n1 + w1 xn1 xn2 w0 xn2 xn1 + w1 x2n2

Summing over the N terms leads us to the matrix we derived previously.

EX 1.6. Code below: 1 2 3 4 5 6 7 8 9 10

%% Women's 100m data % Load all Olympic data load olympics; % Copy the necessary variables x = female100(:,1); % Olympic year t = female100(:,2); % Winning time % Augment x X = [repmat(1,size(x)) x]; % Get solution w = inv(X'*X)*X'*t;

The fitted model is: t = 40.9242 − 0.0151x EX 1.7. Plugging 2012 and 2016 into the above expression yields winning times of 10.5997 and 10.5394 respectively.

EX 1.8. The men’s model is: t = 36.4165 − 0.0133x The women’s model is: t = 40.9242 − 0.0151x The women’s time is decreasing faster than the men’s. Therefore, the women will be faster at the first Olympics after the x that gives identical winning times: 40.9242 − 0.0151x x

= =

36.4165 − 0.0133x 2589

The next Olympic year after 2589 is (assuming they continue to be held every four years) is the year 2592. The winning times are the unrealistically fast 1.8580 seconds and 1.8628 seconds for women and men respectively.

EX 1.9. Code below (synthdata cv.m): 1 2

clear all;close all; load synthdata

3 4 5 6 7 8

% Augment x X = repmat(1,size(x)); for k = 1:4 X = [X x.ˆk]; end

9

K26591_SM_Cover.indd 7

05/04/16 3:37 pm

4

CHAPTER 1.

10 11

% Fit the model w = inv(X'*X)*X'*t;

12 13 14 15 16 17 18

% Randomise the data order N = size(X,1); order = randperm(N); sizes = repmat(floor(N/10),1,10); sizes(end) = sizes(end) + N−sum(sizes); sizes = [0 cumsum(sizes)];

19 20

X = repmat(1,size(x));

21 22 23 24 25 26 27 28 29 30 31 32 33

loss = zeros(4,10); for poly order = 1:4 % Augment x X = [X x.ˆpoly order]; for k = 1:10 % 10−fold CV % Extract the train and test data traindata = X(order,:); traint = t(order); testdata = X(order(sizes(k)+1:sizes(k+1)),:); testt = t(order(sizes(k)+1:sizes(k+1))); traindata(sizes(k)+1:sizes(k+1),:) = []; traint(sizes(k)+1:sizes(k+1)) = [];

34

% Fit the model w = inv(traindata'*traindata)*traindata'*traint;

35 36 37 38 39 40

end

41 42

% Compute loss on test data predictions = testdata*w; loss(poly order,k) = sum((predictions − testt).ˆ2);

end

43 44 45

% Plot the loss plot([1:4],mean(loss,2));

EX 1.10. The total loss is L=

N

n=1

(tn − wT xn )2 .

Writing this in matrix form, differentiating and solving gives us: L ∂L ∂w w

= =

(t − Xw)T (t − Xw)

tT t − 2wT XT t + wT XT Xw

=

−2XT t + 2XT Xw = 0

=

(XT X)−1 XT t.

This is identical to the value obtained for the average loss. It is not surprising as all we are doing is multiplying the loss by a constant and this will not change the value of w at the minimum.

K26591_SM_Cover.indd 8

05/04/16 3:37 pm

5 EX 1.11. The loss is given by N

L= If we define the matrix:

1 αn (tn − wT xn )2 N n=1 

α1  0  A= .  ..

0

0 α2 .. .

... ... .. .

0 0 .. .

0

...

αN

we can write the loss in vector/matrix form as:

    

N

L=

1 (t − Xw)T A(t − Xw) N n=1

Multiplying out, differentiating, equating to zero and solving: L ∂L ∂w w

= = =

1 T t At − 2wT XT At + wT XT AXw N 2 2 − XT At + XT AXw = 0 N N (XT AX)−1 XT At.

Try this out in Matlab (set some αn very low and some very high) to see the effect on the solution.

EX 1.12. Code below (regls100m.m): 1 2 3 4 5

clear all;close all; load olympics; % Extract men's 100m data x = male100(:,1); t = male100(:,2);

6 7 8

% Choose number of folds K = 5;

9 10 11 12 13 14 15

% Randomise the data order N = size(x,1); order = randperm(N); sizes = repmat(floor(N/K),1,K); sizes(end) = sizes(end) + N−sum(sizes); sizes = [0 cumsum(sizes)];

16 17 18 19

% Rescale x x = x − x(1); x = x./4;

20 21 22 23

K26591_SM_Cover.indd 9

X = [repmat(1,size(x)) x]; % Comment out the following line for linear X = [X x.ˆ2 x.ˆ3 x.ˆ4];

05/04/16 3:37 pm

6

CHAPTER 1.

24 25 26

% Scan a wide range of values of the regularisation perameter regvals = 10.ˆ[−12:1:12];

27 28 29 30 31 32 33 34 35 36

for r = 1:length(regvals) for k = 1:K % Extract the train and test data traindata = X(order,:); traint = t(order); testdata = X(order(sizes(k)+1:sizes(k+1)),:); testt = t(order(sizes(k)+1:sizes(k+1))); traindata(sizes(k)+1:sizes(k+1),:) = []; traint(sizes(k)+1:sizes(k+1)) = [];

37

% Fit the model w = inv(traindata'*traindata + regvals(r)*eye(size(X,2)))*... traindata'*traint;

38 39 40 41 42 43 44

end

45 46

K26591_SM_Cover.indd 10

% Compute loss on test data predictions = testdata*w; loss(r,k) = sum((predictions − testt).ˆ2);

end

05/04/16 3:37 pm

Chapter 2

EX 2.1. The errors are real valued and hence a continuous random variable would be more appropriate.

EX 2.2. If all outcomes are equally likely, they have the same probability of occurring.

Defining Y to be the random variable taking the value shown on a die, we can state the following: P (Y = y) = r, where r is a constant. From the definition of probabilities, we know that: 6

P (Y = y) = 1.

y=1

Substituting r into this gives us the following: 6

r = 1, 6r = 1, r = 1/6.

y=1

EX 2.3. (a) Y is a discrete random variable that can take any value from 0 to inf. The probability that Y ≤ 4 is equal to the sum of all of the probabilities that satisfy Y ≤ 4, Y = 0, Y = 1, Y = 2, Y = 3, Y = 4: P (Y ≤ 4) =

4

P (Y = y).

y=0

When λ = 5, we can compute these probabilities as: P (Y ≤ 4) = 0.0067379 + 0.0336897 + 0.0842243 + 0.1403739 + 0.1754674 = 044049. (b) Because Y has to satisfy either P (Y | ≤ 4) or P (Y > 4), we know that P (Y > 4) = 1 − P (Y ≤ 4): P (Y > 4) = 0.5591.

EX 2.4. We require Ep(y) {sin(y)} where p(y) = U(a, b). The uniform density is given by: 1 a≤y≤b b−a p(y) = 0 otherwise 7

K26591_SM_Cover.indd 11

05/04/16 3:37 pm

8

CHAPTER 2.

The required expectation is given by: Ep(y) {sin(y)}

= =

sin(y)p(y) dy

b

y=a

= = When a = 0, b = 1, this is equal to

sin(y)

1 dy b−a

1 b [− cos(y)]a b−a cos(a) − cos(b) . b−a

cos(0) − cos(1) = 0.45970.

1 Code to compute a sample-based approximation below (sampleexpect.m): Ep(y) {sin(y)} =

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

clear all; close all; % Compute a sample based approximation to the required expectation u = rand(10000,1); % Take 10000 samples su = sin(u); % Plot how the approximation changes as more samples are used ns = 10:100:10000; stages = zeros(size(ns)); for i = 1:length(ns) stages(i) = mean(su(1:ns(i))); end plot(ns,stages) % Plot the true value hold on plot([0 ns(end)],[0.4597 0.4597],'k−−')

EX 2.5. The multivariate Gaussian pdf is given by:

1 1 T −1 exp − (w − µ) Σ (w − µ) . p(w) = 2 (2π)D/2 |Σ|1/2 Setting Σ = σ 2 I gives: p(w) =

1 1 T −1 exp − (w − µ) I (w − µ) .

2σ 2 (2π)D/2 |σ 2 I|1/2

Because it only has elements on the diagonal, the determinant of σ 2 I is given by the product of these diagonal elements. As they are all the same, |σ 2 I|1/2 = 1/2

D 2 σ = (σ 2 )D/2 . I−1 = I and multiplying a vector/matrix by I leaves the d=1 matrix/vector unchanged. Therefore, the argument within the expectation can be written as − 2σ1 2 (w − µ)T (w − µ) and recalling that bT b = i b2i , we can rewrite the pdf as:

D 1 1 2 p(w) = exp − 2 (wd − µd ) .

2σ (2π)D/2 (σ 2 )D/2 d=1

K26591_SM_Cover.indd 12

05/04/16 3:37 pm

9 Where wd and µd are the dth elements of w and µ respectively. The exponential of a sum is the same as a product of exponentials. Hence, p(w)

=

=

D

1 1 2 exp − 2 (wd − µd ) 2σ (2π)D/2 (σ 2 )D/2 d=1

D

1 1 2 exp − 2 (wd − µd ) 2σ (2π)1/2 σ

d=1

=

D

d=1

p(wd |µd , σ 2 ),

2

where p(wd |µd , σ ) = N (µd , σ 2 ). Hence, the diagonal covariance is equivalent to assuming that the elements of w are distributed as independent, univariate Gaussians with mean µd and variance σ 2 .

EX 2.6. Using the same methods as in the previous exercise, we see that the determinant of D the covariance matrix is given by d=1 σd2 and we have the following:

D 1 (wd − µd )2 1 p(w) =

1/2 exp − 2 σd2 D 2 d=1 (2π)D/2 σ d=1 d Changing the sum to a product leaves us with p(w)

= (2π) =

1

D D/2

2 d=1 σd

D

1/2

d=1

1 exp − 2 (wd − µd )2 2σd

1 1 2 exp − 2 (wd − µd ) .

2σd (2π)1/2 σd d=1 D

This is the product of D independent univariate Gaussian densities.

EX 2.7. The Hessian for a general model of our form is given by: −

1 T X X σ2

For the linear model, X is defined as: 

1  1  X= .  ..

1

Therefore − σ12 XT X is: XT X =

K26591_SM_Cover.indd 13

xN

N

N



x1 x2 .. .

n=1 xn

   

N x n n=1 N 2 n=1 xn

05/04/16 3:37 pm

10

CHAPTER 2.

N The diagonal elements are −N/σ 2 and −(1/sigma2 ) n=1 x2n which are equivalent (they differ only by multiplication with a negative constant) the expressions obtained in Chapter 1.

EX 2.8. We have N values, x1 , . . . , xN . Assuming that these values came from a Gaussian, we want to find the maximum likelihood estimate of the G and want to find the maximum likelihood estimates of the mean and variance of the Gaussian. The Gaussian pdf is:

1 1 2 √ exp − 2 (xn − µ) 2σ 2πσ Assuming the IID assumption, the likelihood of all N points is given by a product over the N objects:

N

1 1 √ exp − 2 (xn − µ)2 .

2σ 2πσ n=1 We’ll work with the log of the likelihood:

N

1 1 1 2 2 log L = − log(2π) − log(σ ) − 2 (xn − µ) 2 2 2σ n=1 To find the maximum likelihood estimate for µ, we differentiate with respect to µ, equate to zero and solve: ∂ log L ∂µ

=

0

=

0

=

N

1

n=1

σ2

(x − µ)

N

1 (xn − µ) σ 2 n=1

N

n=1

=

N

n=1

µ

=

xn −

N

µ

n=1

xn − N µ N

1 xn N n=1

Similarly, for σ 2 , ∂ log L ∂σ 2 Nσ

2

=

N

n=1

=

N

n=1

σ2

K26591_SM_Cover.indd 14

1 1 (xn − µ)2 − 2+ 2σ 2(σ 2 )2

=

(xn − µ)2

=0 (2.1)

N

1 (xn − µ)2 N n=1

(2.2)

05/04/16 3:37 pm

11 EX 2.9. The Bernoulli distribution is defined as: P (Xn = x|r) = rx (1 − r)1−x where x is either 0 or 1. Using the IID assumption, we have:

L=

N

n=1

rxn (1 − r)1−xn

and the log likelihood is:

log L =

N

n=1

xn log r + (1 − xn ) log(1 − r)

Differentiating with respect to r gives us: ∂ log L ∂r N

xn

n=1 N

n=1

xn − r

N

r

xn

=

N

xn

r

n=1

=

n=1

=

=0

1−r

rN − r N

r

N

1 − xn

n=1

=

1 − xn 1−r

−

N

xn

n=1

1 xn .

N n=1

EX 2.10. The Fisher information is defined as the expectation of the negative second derivative. From the above expression, we can see that the second derivative of the Gaussian likelihood (assuming N observations, x1 , . . . , xN is: N ∂ 2 log L = − 2.

2 ∂µ σ Hence the Fisher information is equal to N/σ 2 .

EX 2.11. Starting from the second expression, we have 2 = 1 σ N

K26591_SM_Cover.indd 15

N

n=1

t2n − 2

N

n=1

+ tn x T nw

N

n=1

2

(xn w)

.

05/04/16 3:37 pm

12

CHAPTER 2.

Concentrating on the final term, N

n=1

2 (xT w)

=

N

n=1

w T xn xT nw

=

w T XT ) Tr(Xw Tr(X(XT X)−1 XT ttT X(XT X)−1 XT )

=

Tr(XT ttT X(XT X)−1 )

= =

Tr(ttT X(XT X)−1 XT ) Tr(tT X(XT X)−1 XT t)

= =

Tr(XT X(XT X)−1 XT ttT X(XT X)−1 )

tT X(XT X)−1 XT t)

= tT Xw N

= tn x T n w.

=

n=1

Therefore,

Now,

N

2 = 1 σ N

N

n=1

t2n −

N

n=1

2 T n=1 tn = t t and we already know that

as required.

tn x T nw

.

N

So, = tT Xw. n=1 tn xn w

2 = 1 tT t − tT Xw , σ N

EX 2.12. Code below (predvar.m): 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

clear all;close all; % Relevant code extraced from predictive variance example.m x = rand(50,1)*10−5; x = sort(x); % Compute true function values f = 5*x.ˆ3 − x.ˆ2 + x; % Generate some test locations testx = [min(x):0.2:max(x)]'; % Add some noise t = f+randn(50,1)*sqrt(1000); % Remove all training data between −1.5 and 1.5 pos = find(x>−1.5 & x<1.5); x(pos) = []; f(pos) = []; t(pos) = [];

16 17 18

% Choose model order K = 5;

19 20 21

K26591_SM_Cover.indd 16

X = repmat(1,size(x)); testX = repmat(1,size(testx));

05/04/16 3:37 pm

13 22 23 24 25

for k = 1:K X = [X x.ˆk]; testX = [testX testx.ˆk]; end

26 27 28 29 30 31 32 33 34 35

w hat = inv(X'*X)*X'*t; ss hat = mean((t − X* w hat).ˆ2); pred va = ss hat *diag(testX*inv(X'*X)*testX'); % Make a plot figure(1);hold off plot(x,t,'b.'); hold on errorbar(testx,testX* w hat,pred va,'r');

EX 2.13. The Bernoulli distribution for a binary random variable x is: p(x|θ) = θx (1 − θ)1−x The Fisher information is defined as the negative expected value of the second derivative of the log density evaluated at some paramater value:

2 ∂ log p(x|θ) F = −Ep(x|θ)

∂θ2 θ Differentiating log p(x|θ) twice gives: ∂ log p(x|θ) ∂θ ∂ 2 log p(x|θ) ∂θ2

= =

x 1−x − θ 1−θ x 1−x − 2− . θ (1 − θ)2

The Fisher information is therefore: F

=

1 1 Ep(x|θ) {x} + Ep(x|θ) {1 − x} . θ2 (1 − θ)2

Substituing in the expectations (θ and 1 − θ respectively gives: F=

1−θ 1 θ + = θ2 (1 − θ)2 θ(1 − θ)

EX 2.14. The multivariate Gaussian pdf is given by:

1 1 T −1 (x − µ) p(x|µ, Σ) = exp − Σ (x − µ) .

2 (2π)D/2 |Σ|1/2 Logging and removing terms not including µ: 1 log p(x|µ, Σ) ∝ µT Σ−1 x − µT Σ−1 µ.

2

K26591_SM_Cover.indd 17

05/04/16 3:37 pm

14

CHAPTER 2.

First and second derivatives are: ∂ log p(x|µ, Σ) ∂µ ∂ 2 log p(x|µ, Σ) ∂µ∂µT

=

Σ−1 x − Σ−1 µ

=

−Σ−1 .

Therefore, the Fisher information is: F = Σ−1 .

K26591_SM_Cover.indd 18

05/04/16 3:37 pm

Chapter 3

EX 3.1. Our Beta prior (with α = β = 1) is defined as: p(r) = 1 (0 ≤ r ≤ 1) The binomial likelihood is given by: P (Y = y|r, N ) =

N y

ry (1 − r)N −y

We know that the posterior density for r is proportional to the likelihood multiplied by the prior, and we also know, because our prior is a particular Beta density, and the Beta prior is conjugate to the binomial likelihood that the posterior must also be a Beta density: p(r|Y, N )

∝ ∝

=

p(Y = y|r, N )p(r) ry (1 − r)N −y × 1

rα −1 (1 − r)β −1

suggesting that the posterior is a Beta density with parameters α = y + 1 and β = N − y + 1.

EX 3.2. Using the same steps as the previous exercise: p(r|Y, N )

∝ ∝

∝

=

p(Y = y|r, N )p(r) ry (1 − r)N −y × 2r

ry+1 (1 − r)N −y

rα −1 (1 − r)β −1

which suggests a Beta density with parameters α = y + 2 and β = N − y + 1.

To find the prior parameters corresponding to the prior p(r) = 2r, consider the form of the Beta density (ignoring the constant term): p(r) ∝ rα−1 (1 − r)β−1

∝ ∝

2r r1 (1 − r)0

From this, it is clear that the parameters are α = 2, β = 1.

15

K26591_SM_Cover.indd 19

05/04/16 3:37 pm

16

CHAPTER 3.

EX 3.3. Using the same steps as the previous exercise: p(r|Y, N )

∝ p(Y = y|r, N )p(r) ∝ ry (1 − r)N −y × 3r2 ∝ ry+2 (1 − r)N −y =

rα −1 (1 − r)β −1

which suggests a Beta density with parameters α = y + 3 and β = N − y + 1.

To find the prior parameters corresponding to the prior p(r) = 3r2 , consider the form of the Beta density (ignoring the constant term): p(r) ∝ rα−1 (1 − r)β−1

∝ ∝

3r2 r2 (1 − r)0

From this, it is clear that the parameters are α = 3, β = 1.

EX 3.4. The effective sample size is α − 1 heads and β − 1 tails. In the first example, this is 0 of each. In the second, we have 1 head and 0 tails and in the third, 2 heads and 0 tails.

EX 3.5. From the definition of expectations, Ep(r) {r}

=

r=1

rp(r) dr

r=0

= = = where

Γ(α + β) Γ(α)Γ(β) Γ(α + β) Γ(α)Γ(β)

r=1 r=0 r=1 r=0 r=1

r × rα−1 (1 − r)β−1 dr rα (1 − r)β−1 dr

Γ(α + β) rα −1 (1 − r)β−1 dr Γ(α)Γ(β) r=0 α = α + 1.

Now, the integrand is an unnormalised Beta density so its integral must be the inverse of the Beta normalisation constant. Therefore, Ep(r) {r}

= =

Γ(α + β) Γ(α )Γ(β) Γ(α)Γ(β) Γ(α + β) Γ(α + β) Γ(α + 1)Γ(β) . Γ(α)Γ(β) Γ(α + β + 1)

Now we require the following Gamma identity, Γ(n + 1) = nΓ(n).

Ep(r) {r}

= =

K26591_SM_Cover.indd 20

Γ(α + β) Γ(α)Γ(β) α Γ(α)Γ(β) Γ(α + β) α + β α . α+β

05/04/16 3:37 pm

17 EX 3.6. We require

2

var{r} = Ep(r) r2 − Ep(r) {r} .

From the previous Exercise, the second term is 2

2 α Ep(r) {r} = . α+β

The first term is computed as follows: r=1 2 Ep(r) r = r2 p(r) dr r=0

=

= = where

Γ(α + β) Γ(α)Γ(β) Γ(α + β) Γ(α)Γ(β)

r=1 r=0 r=1 r=0

r2 × rα−1 (1 − r)β−1 dr rα+1 (1 − r)β−1 dr

Γ(α + β) r=1 α −1 r (1 − r)β−1 dr Γ(α)Γ(β) r=0 α = α + 2.

As in the previous Exercise, the integrand is just an unnormalised Beta density.

Therefore,

Γ(α + β) Γ(α )Γ(β) = Ep(r) r2 Γ(α)Γ(β) Γ(α + β) Γ(α + β) Γ(α + 1 + 1)Γ(β) = Γ(α)Γ(β) Γ(α + β + 1 + 1) Γ(α + β) Γ(α + 1)Γ(β) α + 1 = Γ(α)Γ(β) Γ(α + β + 1) α + β + 1 Γ(α + β) Γ(α)Γ(β) α + 1 α = Γ(α)Γ(β) Γ(α + β) α + β + 1 α + β α(α + 1) = , (α + β)(α + β + 1) where we used the Gamma identity twice. Combining this with the expression for

2 Ep(r) {r} gives

2 α(α + 1) α − var{r} = (α + β)(α + β + 1) α+β α2 α(α + 1) − = (α + β)(α + β + 1) (α + β)2 α2 (α + β + 1) α(α + 1)(α + β) − = 2 (α + β) (α + β + 1) (α + β)2 (α + β + 1) α(α + 1)(α + β) − α2 (α + β + 1) = (α + β)2 (α + β + 1) αβ . = 2 (α + β) (α + β + 1)

K26591_SM_Cover.indd 21

05/04/16 3:37 pm

18

CHAPTER 3.

EX 3.7. Assuming that the probability of heads is given by r, we observe y heads in N tosses, and r has a Beta prior with parameters α and β, the posterior density is a Beta density with parameters δ = α + y and γ = β + N − y. The marginal likelihood is given by:

Γ(α + β) Γ(α + yN )Γ(β + N − yN ) N p(yN |α, β) = . yN Γ(α)Γ(β) Γ(α + β + N ) The probability of winning is given by: Ep(r|yN ) {P (Ynew ≤ 6|r)} = 1 −

ynew =10

ynew =7

Ep(r|yN ) {P (Ynew = ynew |r)}

where Ep(r|yN ) {P (Ynew = ynew |r)} =

Nnew ynew

Γ(δ + γ) Γ(δ + ynew )Γ(γ + Nnew − ynew ) Γ(δ)Γ(γ) Γ(δ + γ + Nnew )

From the question, yN = 9, N = 20.

Scenario 1: α = β = 1. The posterior density has parameters δ = α + yN = 10, γ = β + N − yN = 1 + 20 − 9 = 12. The marginal likelihood comes out as: 0.0476.

The probability of winning as: 0.84812.

Scenario 2: α = β = 50. The posterior density has parameters δ = α + yN = 59, γ = β + N − yN = 50 + 20 − 9 = 61. The marginal likelihood comes out as: 0.1486.

The probability of winning as: 0.83162.

Scenario 3: α = 5, β = 1. The posterior density has parameters δ = α + yN = 14, γ = β + N − yN = 1 + 20 − 9 = 12. The marginal likelihood comes out as: 0.0135.

The probability of winning as: 0.7211.

EX 3.8. Code below (scenarios.m): 1 2 3 4 5

clear all; close all; % Define an array to hold the three scenarios % Each row is alpha,beta for one scenario hypers = [1,1;50,50;5,1];

6 7

scenario = 1; % Change this to look at different ones

8 9 10 11 12 13 14 15 16 17 18 19

% Generate the 100 tosses N = 100; toss = rand(N,1)<0.7; Nheads = sum(toss); Ntails = N − Nheads; % Compute the posterior alpha = hypers(scenario,1); beta = hypers(scenario,2); postalpha = alpha + Nheads; postbeta = beta + Ntails;

20

K26591_SM_Cover.indd 22

05/04/16 3:37 pm

19 21 22 23 24 25 26 27

% Plot the posterior and prior figure(1);hold off x = [0:0.01:1]; plot(x,betapdf(x,hypers(scenario,1),hypers(scenario,2))); hold on plot(x,betapdf(x,postalpha,postbeta),'r') legend('Prior','Posterior');

28 29 30 31 32 33 34

% Compute the marginal likelihood (Equation 3.14 in book) % Note we do it in log space first, because of numerical issues ml = log(factorial(N)/(factorial(Nheads)*factorial(Ntails)))+... gammaln(alpha+beta)−gammaln(alpha)−gammaln(beta)+... gammaln(postalpha)+gammaln(postbeta)−gammaln(postalpha+postbeta+N); ml = exp(ml);

35 36 37 38 39 40 41 42 43 44 45 46 47

% Probability of winning % Compute prob of y=7,8,9,10 tot = 0; for i = 7:10 tot = tot + ... exp(... log(factorial(10)/(factorial(i)*factorial(10−i))) + ... gammaln(postalpha+postbeta) − gammaln(postalpha) − gammaln(postbeta) + ... gammaln(postalpha+i) + gammaln(postbeta + 10 − i) − ... gammaln(postalpha+postbeta+10)... ) end

48 49 50

% Compute probability of winning prob = 1 − tot;

EX 3.9. Firstly equate the two expressions and then re-arrange to find Σ0 :

−1 1 T −1 X X + Σ0 XT t = σ2

−1 1 1 T −1 X X + Σ = 0 σ2 σ2

−1 T 1 1 T −1 X X + N λI 2 X X + Σ0 = σ σ2 1 T X X + N λI 2 = σ Nλ 1 T X X+ 2 I = σ2 σ 1 σ2

Σ0

=

XT X + N λI

XT X + N λI

I

−1

XT t

−1

1 T X X + Σ−1 0 σ2 1 T X X + Σ−1 0 σ2 2 σ I Nλ

(3.1)

EX 3.10. See figure below:

K26591_SM_Cover.indd 23

05/04/16 3:37 pm

20

CHAPTER 3.

µw

Σw

w

σ2

tn

xn N

EX 3.11. Code below (marglike.m): 1 2 3 4 5 6 7 8

clear all;close all; % Generate the data N = 100; x = rand(N,1)*10−5; x = sort(x); ss = 100; f = 5*x.ˆ3 − x.ˆ2 + x; t = f + randn(size(f))*sqrt(ss);

9 10 11

% Choose \sigma 0ˆ2 ss0 = 0.3; % Try increasing this

12 13 14 15 16 17 18 19

% Loop over model orders order = [1:7]; X = repmat(1,size(x)); for i = 1:length(order) mu0 = zeros(order(i)+1,1); s0 = ss0*eye(order(i)+1); X = [X x.ˆorder(i)];

20 21 22

mlmean = X*mu0; mlcov = ss*eye(N) + X*s0*X';

23 24 25

K26591_SM_Cover.indd 24

logml(i) = −(N/2)*log(2*pi) − (N/2)*log(det(mlcov)) − ...

0.5*(t−mlmean)'*inv(mlcov)*(t−mlmean);

05/04/16 3:37 pm

21 26

% Note that the non log value may be numerically instable. ml(i) = exp(logml(i));

27 28 29

end

EX 3.12. Assuming w is fixed, and the following inverse gamma prior on σ 2 :

βα β 2 2 −α−1 p(σ |α, β) = (σ ) exp − 2 . Γ(α) σ The likelihood is the standard Gaussian likelihood:

1 1 2 T p(t|w, X, σ ) = exp − 2 (t − Xw) (t − Xw) .

2σ (2π)N/2 (σ 2 )N/2 The posterior density over σ 2 is therefore proportional to the product of these two densities. Collecting similar terms:

1 1 T 2 2 −α−N/2−1 p(σ |w, X, α, β) ∝ (σ ) exp − 2 β + (t − Xw) (t − Xw) . σ 2 This is another inverse gamma with parameters α∗ and β ∗ given by:

K26591_SM_Cover.indd 25

α∗

=

β∗

=

N 2 1 β + (t − Xw)T (t − Xw).

2 α+

05/04/16 3:37 pm

22

K26591_SM_Cover.indd 26

CHAPTER 3.

05/04/16 3:37 pm

Chapter 4

EX 4.1. Assuming the following prior density for w: p(w) = N (0, ID ), the posterior is: p(w|t1 , . . . , tN , x1 , . . . , xN ) ∝ N (0, ID )

N

n=1

p(tn |w, xn )

Combining all of the D−dimensional data objects into an N × D matrix X and the labels into a vector t allows us to write this as the product of two multi-variate Gaussians:

1 1 T T p(w|t, X) ∝ exp − w w exp − (t − Xw) (t − Xw) .

2 2 Re-arranging and equating coefficients gives the posterior:

1 T T T T T T p(w|t, X) ∝ exp − w w + t t + w X Xw − w X t 2 = N (µ, Σ) Σ = (XT X + ID )−1 µ

=

ΣXT t.

The Laplace approximation approximates the posterior with a Gaussian with mean equal to the posterior mode and a covariance matrix equal to the negative inverse of the second derivate of the log posterior. Removing constants, the log posterior is equal to: 1 1 g = − wT w − (t − Xw)T (t − Xw).

2 2 To find the maximum value, we take partial derivatives with respect to w:

1 ∂g = − 2(I + XT X)w − 2XT t . ∂w 2

Setting to zero and solving for w gives:

w = (I + XT X)−1 XT t.

23

K26591_SM_Cover.indd 27

05/04/16 3:37 pm

24

CHAPTER 4.

This will be the mean of the Gaussian approximation and we can already see that it is equal to µ, the posterior mean.

To find the covariance matrix, we take second derivatives of g: ∂2g = −(ID + XT X). ∂w∂wT The covariance of the Laplace approximation is therefore: (I + XT X)−1 .

This is identical to the posterior covariance matrix thus confirming that the approximation is, in this case, exact.

EX 4.2. We are interested in the Laplace approximation to the posterior p(r|N, y, α, β). The first step is to find the MAP value of r. The log posterior is (removing constants): g = (α − 1) log r + (β − 1) log(1 − r) + y log r + (N − y) log(1 − r).

Taking partial derivatives with respect to r leaves: ∂g α+y−1 β+N −y−1 = − . ∂r r 1−r Re-arranging this expression gives the MAP solution for r: r =

α+y−1 . α+β+N −2

This will be the mean of the Gaussian approximation. The variance is given by: 2 −1 ∂ g σ2 = − ∂r2

−1 α+y−1 β+N −y−1 = − − − r2 (1 − r)2 EX 4.3. The plot below shows an example where α = β = 2, y = 2, and N = 10.

3.5

Exact posterior Laplace approximation

3

p(r| . . .)

2.5 2 1.5 1 0.5 0 0

K26591_SM_Cover.indd 28

0.2

0.4

r

0.6

0.8

1

05/04/16 3:37 pm

25

EX 4.4. This is a nice example to illustrate the power of sampling. Imagine sampling pairs of values from uniform distributions between -0.5 and 0.5. These values would be uniformly distributed in the square shown below: The probability that one of these points lies within the circle must be equal to the ratio of the area of the circle to the area of the square. We can obtain an empirical estimate of this value by say generating N pairs (i.e. points) and then computing the number of pairs that lie within the circle (s). Then: πr2 s ≈ = 0.52 π.

N 1×1

Re-arranging this expression will give an approximate value of π: s . π≈ 0.52 N For example, one random sample of 100 points yields a value of: π ≈ 3.240.

As we would expect, this approximation will, on average, get better as we take more samples. An example after 10000 is: π ≈ 3.129.

EX 4.5. Re-arrange as follows:

P (Tnew = 1|xnew , w)

0.5

=

2

=

1

=

therefore EX 4.6. Assuming a Gaussian prior: and:

=

1 T xnew ) 1 + exp(−w 1 1 1 T xnew ) = 1 + exp(−w 2 1 T xnew ) 1 + exp(−w T xnew ) exp(0) = exp(−w T xnew = 0. w

p(w) = N (0, ID ), vn = exp{wT xn }

the log posterior is proportional to:

1 (tn log vn − vn ) . g = − wT w + 2 n The gradient is obtained by taking partial derivatives with respect to w: tn ∂vn ∂vn ∂g = −w + − . ∂w v ∂w ∂w n n

K26591_SM_Cover.indd 29

05/04/16 3:37 pm

26

CHAPTER 4.

Now,

∂vn = xn v n ∂w

and therefore:

∂g = −w + xn (tn − vn ). ∂w n

The Hessian is obtained by differentiating a second time. You might find it worthwhile to work out each element of the matrix individually for (say) D = 2 if this step appears difficult:

∂2g = −I − vn x n x T n ∂w∂wT n as Given some initial value of w, the optimisation scheme repeatadly updates w follows until it converges to a maximum: w←w−

−I −

vn x n x T n

n

−1

−w +

n

xn (tn − vn )

EX 4.7. Given the result of the previous exercise, this is straightforward. The Laplace ap the maxiproximation is a Gaussian with mean µ and covariance Σ where µ = w, mum obtained using the Newton-Raphson scheme derived in the previous exercise and the covariance is:

−1

T Σ = − −I − v , n xn xn n

where:

T xn }. v n = exp{w

EX 4.8. Example code below (poisscount.m): 1 2 3 4

%% Laplace and Metropolis−Hastings for a Poisson Counts model % Simon Rogers, August 2011 clear all; close all;

5 6 7 8 9

%% Generate the 'true' data.

N = 200; % Number of data objects x = [repmat(1,N,1) randn(N,1)]; w = randn(2,1);

10 11

temp = exp(x*w);

12 13

t = poissrnd(temp);

14 15 16 17 18 19

K26591_SM_Cover.indd 30

truew = w; %% Newton Raphson % Initialise w w = rand(2,1)*5; wall = [];

05/04/16 3:38 pm

27 20 21 22 23 24 25 26 27 28 29 30 31 32 33

change = inf; temp = exp(x*w); ll = −0.5*w'*w + sum(t.*log(temp) − temp); while change > 1e−3 % Compute gradient temp = exp(x*w); gra = −w + sum(repmat(t,1,2).*x − repmat(temp,1,2).*x,1)'; % Compute Hessian hess = −eye(2) − x'*(x.*repmat(temp,1,2)); % Update w neww = w − inv(hess)*gra; % Compute change change = sum((neww−w).ˆ2); w = neww;

34 35 36 37 38 39 40

end

% Store all of the w wall = [wall;w']; % Compute the likelihood (should always increase) temp = exp(x*w); ll = [ll;−0.5*w'*w + sum(t.*log(temp) − temp)];

41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62

%% Metropolis−Hastings % Initialise w wm = randn(2,1); temp = exp(x*wm); wall m = []; % Compute old log posterior oldll = −0.5*wm'*wm + sum(t.*log(temp) − temp); for s = 1:10000 % Generate 10000 samples % Propose a new candidate wnew = wm + randn(2,1)*0.1; % Gaussian proposal, std = 0.1 % Compute new log likelihood temp = exp(x*wnew); newll = −0.5*wnew'*wnew + sum(t.*log(temp) − temp); % Accept of reject if rand<exp(newll−oldll) wm = wnew; oldll = newll; end % Store all w wall m = [wall m;wm']; end

63 64 65 66 67 68

% Plot samples % No thinning or ignoring of burn−in figure(1); hold off plot(wall m(:,1),wall m(:,2),'k.','color',[0.6 0.6 0.6])

69 70 71 72 73 74 75 76 77 78

K26591_SM_Cover.indd 31

%% Plot the approximate posterior % Compute posterior (negative inverse of Hessian) gcov = −inv(−eye(2) − x'*(x.*repmat(temp,1,2))); % Evaluate over a set of points [W1,W2] = meshgrid([−1:0.01:1],[−1:0.01:1]); gli = 1./(sqrt(2*pi)*det(gcov)).*repmat(1,size(W1)); for i = 1:size(W1,1) for j = 1:size(W1,2)

05/04/16 3:38 pm

28

CHAPTER 4.

79 80 81 82 83 84 85

wv = [W1(i,j);W2(i,j)]; gli(i,j) = gli(i,j).*exp(−0.5*(wv−w)'*inv(gcov)*(wv−w));

end end % Plot the contours hold on contour(W1,W2,gli,'k')

86 87 88 89

K26591_SM_Cover.indd 32

% Plot the true value plot(truew(1),truew(2),'ko','markersize',20,'linewidth',2)

05/04/16 3:38 pm

Chapter 5

EX 5.1. The class conditional density is given by: p(xn |tn = c, µc , Σc ) = N (µc , Σc ).

Assuming a Gaussian prior on µc : p(µc |µ0 , Σ0 ), the posterior density for µ is given by: p(µc |x1 , . . . , xNc , µ0 , Σ0 , Σc ) ∝ N (µ0 , Σ0 )

Nc

n=1

p(xn |tn = c, µc , Σc ).

Removing constant terms and re-arranging:

Nc 1 1 T −1 T p(µc | . . .) ∝ exp − (µc − µ0 ) Σ0 (µc − µ0 ) exp − (xn − µc ) (xn − µc ) .

2 2 n=1

Keeping only terms dependent on µc :

Nc

1 T −1 T T −1 T p(µc | . . .) ∝ exp µc Σ0 µc + Nc µc µc − 2µc Σ0 µ0 − 2µc xn .

2 n=1 Therefore, the posterior is a Gaussian:

p(µc | . . .) = N (m, S), where: S m

= =

−1

Σ−1 0 + Nc I

S

Σ−1 0 µ0 +

Nc

n=1

xn

.

EX 5.2. This requires the expectation of a Gaussian with respect to another Gaussian: p(xnew |Tnew = c, X, t)

= = =

Ep(µ |m,S) {p(xnew |µc , Σc )} c Nµc (m, S)Nxnew (µc , Σc ) dµc

Nµc (m, S)Nµc (xn , Σc ) dµc .

29

K26591_SM_Cover.indd 33

05/04/16 3:38 pm

30

CHAPTER 5.

We now make use of the following standard result: Nx (a, A)Nx (b, B) = KNx (d, D), where: K = Na (b, A + B).

The quantities d and D are not important as this density will disappear in our integral. Therefore, our expectation becomes: p(xnew |Tnew = c, X, t) = Nxnew (m, S + Σc ).

EX 5.3. The total likelihood for class c is: L=

1 1 T −1 (x − µ ) Σ (x − µ ) . exp − n n c c c 2 (2π)D/2 |Σc |1/2 n=1 Nc

As normal, it is easiest to maximise the log likelihood. Logging and multiplying out gives: log L = −

Nc

T −1 Nc 1 DNc −1 T −1 log(2π)− log |Σc |− xn Σc xn − 2µT c Σc x n + µ c Σc µ c .

2 2 2 n=1

Taking the partial derivative with respect to µc , equating to zero and solving gives a solution for µc : ∂ log L =0 ∂µc Σ−1 c

Nc

= Σ−1 c

Nc

n=1

xn

=

µc

=

xn − Nc Σ−1 c µc

Nc Σ−1 c µc

n=1 N

Similarly, for Σc : ∂ log L =0 ∂Σc

= −

c 1 xn .

Nc n=1

Nc Nc −1 1 −1 Σ c + Σc (xn − µc )(xn − µc )T Σ−1 c , 2 2 n=1

where we have used the following two results (for symmetric A): ∂ log |A| = A−1 ∂A ∂aT A−1 a = −A−1 aaT A−1 . ∂A Re-arranging gives the expression for Σc : N

Σc =

c 1 (xn − µc )(xn − µc )T Nc n=1

where µc is the maximum likelihood estimate derived above.

K26591_SM_Cover.indd 34

05/04/16 3:38 pm

31 EX 5.4. The data consists of Nc M -dimensional vectors of integer counts. The multinomial likelihood for the Nc data objects is: p(x1 , . . . , xNc |qc1 , . . . , qcM ) = L ∝

Nc M

xnm qcm ,

n=1 m=1

(where the constant that does not depend on qcm has been omitted for brevity).

Once again, we work with the log likelihood and take partial derivatives with respect to qcm . We must also add a Lagrangian term due to the constraing that m qcm = 1.

Calling this new objective function g gives: log L g

=

Nc M

xnm log qcm

n=1 m=1

=

Nc M

n=1 m=1

∂ log L =0 ∂qcm

=

qcm

=

xnm log qcm − λ

Nc

xnm

q

m

qcm − 1

−λ

n=1 cm N c n=1 xnm

λ

.

To compute λ we sum both sides over m:

qcm

=

1

=

λ

=

m

1 xnm λ m,n 1 xnm λ m,n

1

n,m xnm

.

Therefore, the maximum likelihood estimate of qcm is:

n xnm

qcm = .

n,m xnm

EX 5.5. The likelihood (ignoring terms not involving qcm ) is: p(x1 , . . . , xNc |qc ) ∝

Nc M

xnm . qcm

n=1 m=1

The prior over qc is a Dirichlet with constant parameter α (again, ignoring terms not involving qcm ): M

α−1 qcm . p(qc |α) ∝ m=1

K26591_SM_Cover.indd 35

05/04/16 3:38 pm

32

CHAPTER 5.

The Dirichlet is the conjugate prior to the multinomial likelihood. Therefore, we know the posterior will be another Dirichlet. Multiplying the prior and likelihood together and combing qcm terms gives: p(qc |α, x1 , . . . , xNc ) ∝

M

α−1+

qcm

m=1

n xnm

.

This is the form of a Dirichlet, where the mth parameter is βm = α + EX 5.6. The required expectation is:

n xnm .

p(xnew |Tnew = c, X, t) = Ep(qc |Xc ,α) {p(xnew |qc } .

Writing out the expectation in full gives:

Γ( m βm ) βm −1 ( m xnew,m )! xnew,m

qcm qcm dqc p(xnew |Tnew = c, X, t) = m Γ(βm ) m m (xnew,m !) m

( xnew,m )! Γ( m βm ) βm +xnew,m −1

= m dqc qcm m (xnew,m !) m Γ(βm ) m

( m xnew,m )! Γ( m βm ) m Γ(βm + xnew,m )

. = m (xnew,m !) m Γ(βm ) Γ( m βm + xnew,m )

EX 5.7. The MAP estimate is obtained by maximising the likelihood multiplied by the prior.

Extracting only the terms that include qcm , and logging give us:

log L ∝ (α − 1) log qcm + xnm log qcm . m

m

n

Introducing the necessary Lagrangian (to ensure m qcm = 1) we end up with the following objective function:

g= xnm + α − 1 log qcm − λ qcm − 1 . m

n

m

Taking partial derivatives, equating to zero and solving gives:

α − 1 + n xnm ∂g =0 = −λ ∂qcm qcm

α − 1 + n xnm

qcm = , M (α − 1) + n,m xnm

where we summed both sides over m to obtain the value for λ.

EX 5.8. The starting point is: argmin w

K26591_SM_Cover.indd 36

1 T w w+C ξn 2 n

05/04/16 3:38 pm

33 subject to: ξn ≥ 0, tn (wT xn + b) ≥ 1 − ξn .

Adding the constraints as Lagrangian terms gives:

1 T w w+C ξn − αn (tn (wT xn + b) − 1 + ξn ) − ξn γ n 2 n n n subject to αn ≥ 0, γn ≥ 0, which must be minimised with respect to w, b and ξn and maximised with respect to αn and γn . At the solution: ∂ ∂w

=

∂ ∂b

=

∂ ∂ξn

=

w− −

α n tn = 0

n

α n tn = 0

n

C − αn − γn = 0.

Substituing the first and third of these expressions back into the objective function and re-arranging gives: −

1 α n α m t n tm x T αn n xm + 2 n,m n

which must be maximised with respect to α subject to:

αn tn = 0, 0 ≤ αn ≤ C. n

Where the first constraint comes from the partial derivative with respect to b and the second from the partial derivative with respect to ξn and the fact that γn ≥ 0.

K26591_SM_Cover.indd 37

05/04/16 3:38 pm

34

K26591_SM_Cover.indd 38

CHAPTER 5.

05/04/16 3:38 pm

Chapter 6

EX 6.1. The log likelihood for all N data objects is:

2 L= log πk p(xnd |µkd , σkd ). n

k

d

Insersting a set of variational parameters, qnk such that k qnk = 1 and qnk ≥ 0 and subsequently applying Jensen’s inequality results in: qnk

2 log πk p(xnd |µkd , σkd ) L = q nk n d k

2 log p(xnd |µkd , σkd ) − qnk log qnk . qnk log πk + qnk ≥ n

d

k

2 . Writing this term out in full leaves: Only the second term involves σkd

1 1 1 2 2 qnk − log 2π − log σkd − 2 (xnd − µkd ) .

2 2 2σkd n k

d

2 results in: Taking partial derivatives with respect to σkd

qnk qnk ∂ 2 = − n2 + 2 2 )2 (xnd − µkd ) . ∂σkd 2σkd 2(σ kd n

Equating to zero and re-arranging results in:

qnk (xnd − µkd )2 2 σkd = n . n qnk

EX 6.2. Starting from the log likelihood above and applying Jensen’s inequality:

qnk L = log πk p(xnd |µkd , σk2 ) q nk n k d

qnk log πk + qnk log p(xnd |µkd , σk2 ) − qnk log qnk . ≥ n

k

d

σl2 only appears in the second term. Expanding:

n

k

qnk

D 1 D (xnd − µkd )2 − log 2π − log σk2 − 2 2 2 2σk d

.

35

K26591_SM_Cover.indd 39

05/04/16 3:38 pm

36

CHAPTER 6.

Taking partial derivatives with respect to σk2 gives: D 1 ∂ =− 2 qnk + (xnd − µkd )2 .

2 ∂σk 2σk n 2(σk2 )2 d,n

Equating to zero and re-arranging gives:

2 2 n qnk d (xnd − µkd ) σk = D n qnk

EX 6.3. The log likelihood is given by: x

log πk pkdnd (1 − pkd )1−xnd .

L= n

k

d

Applying Jensen’s inequality gives:

(xnd log pkd + (1 − xnd ) log(1 − pkd )) − qnk log qnk qnk log πk + qnk L≥ n

k

d

Taking partial derivatives with respect to pkd setting to zero and solving gives:

∂ xnd 1 − xnd =0 = qnk − qnk ∂pkd p 1 − pkd kd n n

xnd 1 − xnd qnk = qnk p 1 − pkd kd n n

qnk xnd = pkd qnk − pkd qnk xnd qnk xnd − pkd n

n

n

pkd

=

n qnk xnd

n qnk

n

EX 6.4. This proceeds in exactly the same manner as the previous exercise except that after applying Jensen’s inequality, the bound will also include a term from the Beta prior.

Including only terms that involve qnk (and removing summations over k and d):

(α − 1) log pkd + (β − 1) log(1 − pkd ) + qnk (xnd log pkd + (1 − xnd ) log(1 − pkd )) . n

Taking partial derivatives with respect to pkd , equating to zero and solving gives: ∂ =0 ∂pkd

α−1 xnd + p pkd kd n

(α − 1) − pkd (α − 1) + qnk xnd qnk

= = =

n

pkd

K26591_SM_Cover.indd 40

=

α−1 β−1 xnd 1 − xnd − + qnk − qnk pkd 1 − pkd pkd 1 − pkd n n

β−1 1 − xnd qnk + 1 − p 1 − pkd kd n

pkd (β − 1) + pkd qnk n

α − 1 + n qnk xnd

. α + β − 2 + n qnk

05/04/16 3:38 pm

37 EX 6.5. The log likelihood (after applying Jensen’s ineuqality and omitting terms not featuring µkd ) is: L≥−

1 1 2 (µkd − m)2 − qnk 2 (xnd − µkd ) + . . .

2 2s 2σ kd n k

d

Taking partial derivatives with respect to µkd , equating to zero and re-arranging gives: ∂L ∂µkd

=

µkd

=

1 1 − 2 (µkd − m) + 2 qnk (xnd − µkd ) = 0 s σkd n

1 1 2 n qnk xnd s2 m + σkd

.

1 1 n qnk s2 + σ 2 kd

EX 6.6. The log likelihood is given by: L=

log

n

λxkn exp{−λk } xn !

πk

k

Incorporating variational parameteres qnk and applying Jensen’s inequality gives: L≥

n

qnk log πk +

n

k

qnk log

k

λxkn exp{−λk } − qnk log qnk . xn ! n k

We shall first derive an update for π k . The relevant terms in the bound, incorporating a Lagrangian term to ensure k πk = 1 are:

qnk log πk − λ πk − 1 . n

k

k

Taking partial derivatives with respect to πk gives: 1 ∂ = qnk − λ. ∂πk πk n Equating to zero: λπk =

qnk .

n

To compute λ we sum both sides over k:

πk = λ k

λ =

Therefore: πk =

K26591_SM_Cover.indd 41

n

qnk

k

N.

1 qnk .

N n

05/04/16 3:38 pm

38

CHAPTER 6.

Now qnk . Each term is relevant, and adding the Lagrangian:

n

qnk log πk +

k

n

k

qnk log p(xn |λk ) −

n

k

qnk log qnk − λ

qnk

k

.

Taking partial derivatives with respect to qnk : ∂ = log πk + log p(xn |λk ) − (log qnk + 1) − λ. ∂qnk Equating to zero, taking exponetials and grouping all constants into γ: γqnk = πk p(xn |λk ).

To find γ, we sum both sides over k:

γ qnk = k

γ

=

k

k

Therefore:

πk p(xn |λk ) πk p(xn |λk ).

πk p(xn |λk ) . qnk = j πj p(xn |λj )

Finally, the update for λk . Expanding the relevant component of the bound:

qnk xn log λk − qnk λk . n

k

n

k

Taking partial derivatives with respext to λk :

1 ∂ = qnk xn − qnk . ∂λk λk n n Equating to zero and solving for λk gives:

n qnk xn . λk = n qnk

Example code to run this mixture is given below (poissmix.m): 1

clear all;close all;

2 3 4 5 6 7 8 9 10

K26591_SM_Cover.indd 42

% Generate data from 3 poissons N = 20; x = poissrnd(10,[N 1]); x = [x;poissrnd(2,[N,1])]; x = [x;poissrnd(20,[N,1])]; % Fit a two component mixture K = 3; % Initialise

05/04/16 3:38 pm

39 11 12 13 14 15 16 17 18 19 20

pr = repmat(1/K,1,K); mu = rand(K,1); N = size(x,1); allmu = []; % Do 100 iterations for it = 1:100 % Update q temp = poisspdf(repmat(x,1,K),repmat(mu',N,1)); q = temp.*repmat(pr,N,1); q = q./repmat(sum(q,2),1,K);

21

% Update pr pr = mean(q,1);

22 23 24

% Update means mu = (sum(repmat(x,1,K).*q,1)./sum(q,1))'; allmu = [allmu;mu'];

25 26 27 28

% Plot the evolution of the means, and show the q subplot(121);plot(allmu); subplot(122);imagesc(q);drawnow

29 30 31 32

K26591_SM_Cover.indd 43

end

05/04/16 3:38 pm

40

K26591_SM_Cover.indd 44

CHAPTER 6.

05/04/16 3:38 pm

Chapter 7

EX 7.1. The bound is given by: log p(Y ) ≥

Q(θ) log

p(Y, θ) dθ.

Q(θ)

To show that the bound is maximised if Q(θ) is the true posterior, we will substitute the posterior p(θ|Y ) for Q(θ) and re-arrange the right hand side such that the inequality becomes an equality. Firstly, substituting and expanding:

log p(Y ) ≥ p(θ|Y ) log p(Y, θ) dθ − p(θ|Y ) log p(θ|Y ) dθ.

Next we expand the joint density: p(Y, θ) = p(θ|Y )p(Y ):

log p(Y ) ≥ p(θ|Y ) log p(θ|Y ) dθ + p(θ|Y ) log p(Y ) dθ − p(θ|Y ) log p(θ|Y ) dθ.

The first and third terms cancel. The second term is an expectation of a constant and hence the expectation disappears:

p(θ|Y ) log p(Y ) dθ = log p(Y ) p(θ|Y ) dθ = log p(Y ).

EX 7.2. We shall go through each term in the order they are given in Section 7.5.5 starting with EQτ (τ ) {log p(τ |a, b)}: EQτ (τ ) {log p(τ |a, b)} Now

EQτ (τ ) {a log b − log Γ(a) + (a − 1) log τ − bτ }

a log b − log Γ(a) + (a − 1) log τ − b τ .

n EQxn (xn ) {log p(xn )}. Recall that p(xn ) = N (0, I) and Qxn (xn ) = N (µxn , Σxn ):

n

= =

EQxn (xn ) {log p(xn )}

= =

1 D EQxn (xn ) − log 2π − (xT x ) n 2 2 n n

1 ND log 2π − Tr(Σxn ) + µT − xn µ xn .

2 2 n

(Note that ENa (m,S) aT Ba = Tr(SB) + mT Bm).

41

K26591_SM_Cover.indd 45

05/04/16 3:38 pm

42

CHAPTER 7.

The expression for m EQwm (wm ) {log p(wm )} is arrived at in exactly the same manner as that for n EQxn (xn ) {log p(xn )}.

Next, n m EQ(·) {log p(ynm |xn , wm , τ )} (where the expectation is over all parameters):

1 1 1 T = xn )2 EQ(·) − log 2π + log τ − τ (ynm − wm 2 2 2 n m

NM 1 NM T log 2π + log τ − τ xn )2 . (ynm − wm = − 2 2 2 n m

Now, we shall look at EQτ (τ ) {log Qτ (τ )}. Qτ (τ ) is Gamma with parameters e and f (see Equation 7.12). Therefore it follows exactly the same form as EQτ (τ ) {log p(τ |a, b)}: EQτ (τ ) {log Qτ (τ )} = e log f − log Γ(e) + (e − 1) log τ − f τ .

The final two expressions are derived in exactly the same way and we will only show

the one for xn , n EQxn (xn ) {log Qxn (xn )}. Recall that Qxn (xn ) = N (µxn , Σxn ):

n

EQxn (xn ) {log Qxn (xn )}

=

1 ND log 2π − log |Σxn | − . . .

2 2 n

1 EQxn (xn ) (xn − µxn )T Σ−1 xn (xn − µxn ) .

2 n

−

To progress further, we need the following identity:

ENa (b,C) (a − d)T E(a − d) = (b − d)T E(b − d) + Tr(EC).

Matching this to our expression, we can see that b = d = µxn and so the first term on the RHS is zero. The second term is Tr(Σ−1 xn Σxn ) = Tr(I) = D (for Ddimensional xn . Therefore, our expectation becomes:

n

EQxn (xn ) {log Qxn (xn )} = −

The expression for

ND 1 ND log 2π − − log |Σxn |.

2 2 2 n

m EQwm (wm ) {log Qwm (wm )} follows in exactly the same way.

EX 7.3. Starting with Qxn (xn ):

Qxn (xn ) ∝ exp EQW (W)Qτ (τ ) {log p(Y, W, X, τ |Z)}

znm log p(ynm |xn , wm , τ ) ∝ exp EQW (W)Qτ (τ ) log p(xn ) + m

1 T 1 T 2 ∝ exp − xn xn − znm EQW (W)Qτ (τ ) τ (ynm − wm xn ) 2 2 m

1 T 1 T T ∝ exp − xn xn − τ znm (xT n wm wm xn − 2ynm xn wm ) .

2 2 m

K26591_SM_Cover.indd 46

05/04/16 3:38 pm

43 This has the form of a Gaussian, Qxn (xn ) = N (µxn , Σxn ). Equating coefficients, we get the following: Σ xn

=

µ xn

=

I + τ

τ Σxn

znm wm wm

m

m

−1

T

znm ynm wm .

The expression for Qwm (wm ) follows an almost identical argument so we omit it here. Finally, the expression for Qτ (τ ): Qτ (τ )

∝ ∝ ∝ ∝

exp EQW (W)QX (X) {log p(Y, W, X, τ |Z)}

exp EQW (W)QX (X) log Γ(a, b) + znm log p(ynm |xn , wm , τ )

n,m

1 T znm EQW (W)QX (X) (ynm − wm xn ) 2 ) τ 2 2 n,m

1 n,m znm T 2 exp −bτ + (a − 1) log τ − znm (ynm − wm xn ) . log τ − τ 2 2 n,m

exp −bτ + (a − 1) log τ −

n,m znm

log τ −

This has the form of another Gamma distribution, with parameters:

K26591_SM_Cover.indd 47

e

=

f

=

1 znm 2 n,m

1 T b+ znm (ynm − wm xn )2 .

2 n,m a+

05/04/16 3:38 pm

44

K26591_SM_Cover.indd 48

CHAPTER 7.

05/04/16 3:38 pm

Chapter 8

EX 8.1. The following code would do it: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

x = [0:0.01:1]; gam = 10.0; N = length(x); K = zeros(N); % Create the kernel matrix for n = 1:N for m = 1:N K(n,m) = exp(−gam*(x(n)−x(m))ˆ2); end end % Add a small constant to the diagonal for numerical stability K = K + 1e−6*eye(N); % Generate 10 samples f samps = gausssamp(repmat(0,N,1),K,10) plot(x,f samps)

EX 8.2. Use the code above, adding a linear term to K. EX 8.3. The following code gives a function to compute the marginal likelihood. Note that we optimise the log of the parameters to avoid having to use a constrained optimisation: 1 2 3 4 5 6 7 8 9 10 11

function ML = marg like(hyp,y,x) % Function to compute the marginal likelihood % hyp contains log of the hyperparameters N = length(x); % Compute covariance function CSS = exp(−exp(hyp(1))*(repmat(x,1,N) − repmat(x',N,1)).ˆ2); CSS = CSS + exp(hyp(2))*eye(N); % Compute marginal likelihood ML = −(N/2)*log(2*pi) − 0.5*log(det(CSS)) − 0.5*y'*inv(CSS)*y; % Return negative as optimiser minimises ML = −ML;

and the following code calls Matlab’s fminunc function for some generated data: 1

x = [0:0.05:1];

45

K26591_SM_Cover.indd 49

05/04/16 3:38 pm

46

CHAPTER 8.

2 3 4

gam = 10.0; N = length(x); K = zeros(N);

5 6 7 8 9 10 11

% Create the kernel matrix for n = 1:N for m = 1:N K(n,m) = exp(−gam*(x(n)−x(m))ˆ2); end end

12 13 14 15

K = K + 1e−6*eye(N); % Generate a sample f samps = gausssamp(repmat(0,N,1),K,1)

16 17 18 19

% Add some noise ss = 0.1; true f = f samps(1,:) + randn(1,N)*sqrt(ss);

20 21 22 23

true f = true f'; x = x';

24 25 26

% Initial hyperparameters inithyp = [0;0]

27 28 29

% Call optimisation code hyp = fminunc(@(h) marg like(h,true f,x),inithyp)

EX 8.4. The function value at the nth data point if fn . This function is then exponentiated to give the rate of a Poisson λn = exp(fn ). The integer count sampled from the Poisson is zn . The gradient of the log posterior is given by: δ = z − exp(f ) − C−1 f δf δ2 = −C−1 − E δf δf T where E is a matrix with zeros everywhere except the diagonal where the nth diagonal value is exp(fn ). This is demonstrated with the following code: 1 2 3 4 5

% Generate some data x = [0:0.01:1]; gam = 10.0; N = length(x); C = zeros(N);

6 7 8 9 10 11 12

% Create the kernel matrix for n = 1:N for m = 1:N C(n,m) = exp(−gam*(x(n)−x(m))ˆ2); end end

13 14

K26591_SM_Cover.indd 50

C = C + 1e−6*eye(N);

05/04/16 3:38 pm

47 15 16

% Generate a sample f samps = gausssamp(repmat(0,N,1),C,1)

17 18 19 20 21

% Plot the function figure(1); hold off plot(x,f samps);

22 23 24 25 26 27

% Generate the counts rate = exp(f samps); n = poissrnd(rate)'; hold on plot(x,n,'ro')

28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

% Use Newton−Raphson to optimise f f = zeros(N,1); iK = inv(C); oldf = f; for it = 1:100 gr = n − exp(f) − iK*f; he = −iK − diag(exp(f)); f = f − inv(he)*gr; ch = sum((f − oldf).ˆ2); oldf = f; if ch < 1e−6 break end end

43 44

plot(x,f,'g')

EX 8.5. The following code generates some data, uses M-H to sample from the posterior and uses the posterior samples to compute the predictions on a grid, in order to show the predictive contours.

1 2 3

% GP classification using M−H x = [randn(20,2)−repmat(4,20,2);randn(20,2)+repmat(4,20,2)]; t = [repmat(0,20,1);repmat(1,20,1)];

4 5

N = size(x,1);

6 7 8 9 10 11 12 13 14

% Compute the covariance matrix gam = 1;alp = 5; C = zeros(N); for n = 1:N for m = 1:N C(n,m) = alp*exp(−gam*(sum((x(n,:)−x(m,:)).ˆ2))); end end

15 16

iC = inv(C);

17 18 19 20

nSamps = 10000; pos0 = find(t==0); pos1 = find(t==1);

21 22

K26591_SM_Cover.indd 51

% Initialise with f being −3 for the −ve class and +3 for the positive

05/04/16 3:38 pm

48

CHAPTER 8.

23 24 25 26 27 28 29 30 31

%\ Could use anything, but this gets us to a good place a bit faster! f = [repmat(−3,20,1);repmat(3,20,1)]; oldProb = −0.5*f'*iC*f − sum(t.*log(1+exp(−f))); oldProb = oldProb + sum((1−t).*(−f − log(1+exp(−f)))); % Make a grid of prediction points and compute the % test covariance matrix [X,Y] = meshgrid(min(x(:,1))−0.5:0.1:max(x(:,1)),... min(x(:,2))−0.5:0.1:max(x(:,2)));

32 33 34 35 36 37 38 39 40

testData = [X(:),Y(:)]; n test = size(testData,1); testC = zeros(n test,size(x,1)); for n = 1:n test for m = 1:size(x,1) testC(n,m) = alp*exp(−gam*(sum((testData(n,:)−x(m,:)).ˆ2))); end end

41 42 43 44

% This term is used for predictions and doesn't change so we can just % compute it once testCovTerm = diag(testC*iC*testC');

45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64

% Run the sampler testProbs = zeros(n test,1); allf = zeros(N,nSamps/100); for s = 1:nSamps % Propose a new sample for each observation % Note that we could do this for all samples at once, but % it becomes very hard to get a sample accepted order = randperm(N); for n = 1:N newf = f; newf(order(n)) = f(order(n)) + randn(1,1)*0.1; newProb = −0.5*newf'*iC*newf − sum(t.*log(1+exp(−newf))); newProb = newProb + sum((1−t).*(−newf − log(1+exp(−newf)))); u = rand; if u <= exp(newProb − oldProb) f = newf; oldProb = newProb; end end

65

% Do the predictions testVar = alp − testCovTerm; testMu = testC*iC*f;

66 67 68 69

f samp = randn(n test,1).*sqrt(testVar) + testMu; testProbs = testProbs + 1./(1+exp(−f samp));

70 71 72 73

end

74 75 76 77 78 79 80 81

K26591_SM_Cover.indd 52

% Plot the output figure(4) hold off pos = find(t==0); plot(x(pos,1),x(pos,2),'ko'); pos = find(t==1);

05/04/16 3:38 pm

49 82 83 84 85

K26591_SM_Cover.indd 53

hold on plot(x(pos,1),x(pos,2),'ko','markerfacecolor','k') testContour = reshape(testProbs./nSamps,size(X)); contour(X,Y,testContour,5);

05/04/16 3:38 pm

50

K26591_SM_Cover.indd 54

CHAPTER 8.

05/04/16 3:38 pm

Chapter 9

EX 9.1. The following code does both methods of sampling and then plots the autocorrelation: 1 2 3

% Generate some points data in 1−D x = [0:0.1:1]; alp = 5;gam = 1;

4 5 6 7 8 9 10 11

N = length(x); C = zeros(N); for n = 1:N for m = 1:N C(n,m) = alp*exp(−gam*(x(n)−x(m))ˆ2); end end

12 13

C = C + 1e−6*eye(N);

14 15

nSamps = 10000;

16 17 18 19

% Sample 1000 values from the Gaussian f full = gausssamp(zeros(N,1),C,nSamps);

20 21 22

% Samples using Gibbs sampling f = zeros(N,1);

23 24

all f = zeros(nSamps,N);

25 26 27 28 29 30 31 32 33 34 35 36 37 38

for s = 1:nSamps for n = 1:N % Compute the conditional mean and covariance subC = C; subC(n,:) = []; subC(:,n) = []; subc = C(n,:); subc(n) = []; subf = f; subf(n) = []; % Note that the mean is zero so these % expressions are a bit simpler than in % the book

39

51

K26591_SM_Cover.indd 55

05/04/16 3:38 pm

52

CHAPTER 9. % Also − the following expression is % much more accurate than subc*inv(subC)*subc' co = C(n,n) − subc/subC*subc'; mu = subc*inv(subC)*subf; f(n) = randn.*sqrt(co) + mu;

40 41 42 43 44

end all f(s,:) = f';

45 46 47

end

48 49 50 51

% Compute the autocorrelation of the nth point n = 5;

52 53 54 55

k vals = [0:1:200]; ac = zeros(length(k vals),2); N = nSamps;

56 57 58 59 60 61 62

full vals = f full(:,n); full vals = full vals − mean(full vals); all vals = all f(:,n); all vals = all vals − mean(all vals); ss = [var(full vals), var(all vals)];

63 64 65 66 67

for k = 1:length(k vals) ac(k,1) = sum(full vals(1:end−k vals(k)).*full vals(1+k vals(k):end)) ac(k,1) = ac(k,1)/(ss(1)*(N−k vals(k)));

68 69 70 71

end

ac(k,2) = sum(all vals(1:end−k vals(k)).*all vals(1+k vals(k):end)) ac(k,2) = ac(k,2)/(ss(2)*(N−k vals(k)));

72 73 74 75 76

figure(1) plot(ac) xlabel('Lag'); ylabel('Autocorrelation');

EX 9.2. See the solution to Exercise 8.5. The only difference is changing the sigmoid likelihood to the probit likelihood.

EX 9.3. Code available on accompanying website.

EX 9.4. Code available on accompanying website.

EX 9.5. The following code will sample a regression dataset and then obtain an approximate posterior via ABC: 1 2

% ABC for GP regression x = [0:0.1:1];

3 4 5

alp = 5; gam = 1;

6 7 8

N = length(x); C = zeros(N);

9

K26591_SM_Cover.indd 56

05/04/16 3:38 pm

53 10 11 12 13 14

for n = 1:N for m = 1:N C(n,m) = alp*exp(−gam*(x(n)−x(m))ˆ2); end end

15 16 17

C = C + 1e−6*eye(N); noise var = 0.1;

18 19

t = gausssamp(repmat(0,N,1),C,1)' + randn(N,1).*sqrt(noise var);

20 21

% ABC

22 23 24

nSamps = 5000; accepted samps = [];

25 26 27 28 29 30 31 32 33

err thresh = 5.0; figure(1);hold off plot(x,t,'k','linewidth',2);hold on for s = 1:nSamps % Propose a value proposal = gausssamp(repmat(0,N,1),C,1)' + randn(N,1).*sqrt(noise var); err = sum((proposal − t).ˆ2); if err < err thresh accepted samps = [accepted samps proposal]; plot(x,proposal,'k','color',[0.6 0.6 0.6]);drawnow end

34 35 36 37 38 39

end

40 41

plot(x,t,'k','linewidth',2);

42 43

%% Compare with the true posterior

44 45 46

postCov = inv((eye(n)*(1/noise var) + inv(C))); postMu = (1/noise var) * postCov * t;

47 48 49 50 51 52 53

K26591_SM_Cover.indd 57

% Generate samples samp f = gausssamp(postMu,postCov,1000); figure(2);hold off plot(x,samp f','k','color',[0.6 0.6 0.6]); hold on plot(x,t,'k','linewidth',2);

05/04/16 3:38 pm

54

K26591_SM_Cover.indd 58

CHAPTER 9.

05/04/16 3:38 pm

Chapter 10

EX 10.1. We have N observations, x1 , . . . , xN . Our model is defined as a set of K Gaussian components, with means µk and unit variances (σk2 = 1). At any point within the Gibbs sampling procedure, each of the N observations is assigned to one of these components (znk = 1, znj = 0 ∀j = k). Each mean value has a Gaussian prior density with its own mean and variance µ0 , σ02 . The conditional density for µk can therefore be computed as: p(µk | . . .)

N

z

∝

p(µk |µ0 , σ02 )

∝

N 1 1 2 2 exp − 2 (µk − µ0 ) exp − znk (xn − µk ) 2σ0 2 n=1

= µ2k b2

=

b2

=

n=1

[p(xn |µk )] nk

N (a, b2 )

µ2k + µ2k znk 2 σ0 n

−1

−2 σ0 + znk n

aµk b2

=

a

=

N

µk µ 0 + µ znk xn k σ02 n=1

N µ0 2 b + znk xn σ02 n=1

EX 10.2. The following code performs Gibbs sampling for the one-dimensional Gaussian mixture with fixed known variance: 1 2 3

% 1D Gaussian mixture x = [randn(20,1);randn(20,1)+3]; x = sort(x);

4 5

N = length(x);

6 7

% Define the prior parameters

55

K26591_SM_Cover.indd 59

05/04/16 3:38 pm

56

CHAPTER 10.

8 9

mu0 = 0; ss0 = 1;

10 11 12

% Component variance ss = 1;

13 14 15

% Number of components K = 2;

16 17 18

nSamps = 1000;

19 20 21 22 23

% Parameters for the prior Dirichlet alp = repmat(1,K,1); prk = gamrnd(alp,1); prk = prk./sum(prk);

24 25

Z = zeros(N,K);

26 27 28 29 30

% Structures to hold the output allZZ = zeros(N); allMu = zeros(nSamps,K); allPrk = zeros(nSamps,K);

31 32 33 34 35 36 37

% Outer loop to run multiple chains for computing Rhat nChain = 5; samps = zeros(nSamps,nChain); for chain = 1:nChain % Randomly initialise the means mu = randn(K,1).*sqrt(ss0) + mu0;

38 39 40 41 42 43 44 45 46 47 48 49 50

for s = 1:nSamps % Update the Z for n = 1:N like = −0.5*(x(n) − mu).ˆ2; like = like + log(prk); prob = exp(like − max(like)); prob = prob./sum(prob); pos = find(rand<=cumsum(prob),1); Z(n,:) = 0; Z(n,pos) = 1; end

51 52

% Update the mu

53 54 55 56 57 58

for k = 1:K ssk = 1/(1/ss0 + sum(Z(:,k))); muk = ssk * (mu0/ss0 + sum(Z(:,k).*x)); mu(k) = randn*sqrt(ssk) + muk; end

59 60

% Update the prior

61 62 63

prk = gamrnd(alp + sum(Z,1)',1); prk = prk./sum(prk);

64 65 66

K26591_SM_Cover.indd 60

allPrk(s,:) = prk'; allMu(s,:) = mu';

05/04/16 3:38 pm

57 67

end

68

allZZ = allZZ + Z*Z';

69

figure(1); plot(allMu); figure(2); plot(allPrk);

70 71 72 73 74 75

allZZ = allZZ./nSamps; figure(3); imagesc(allZZ);

76 77 78 79

% Compute the autocorrelation of the kth mu lag vals = [0:1:200]; ac = zeros(length(lag vals),1);

80 81 82 83

k = 1;

84 85

full vals = allMu(:,k); full vals = full vals − mean(full vals);

86 87 88

samp ss = var(full vals);

89 90 91

for l = 1:length(lag vals) ac(l) = sum(full vals(1:end−lag vals(l)).*full vals(1+lag vals(l):end)) ac(l) = ac(l)/(samp ss *(nSamps−lag vals(l))); end

92 93 94 95 96

figure(4); plot(lag vals,ac,'ro');drawnow

97 98 99

% Store the values of the lower mean (they switch in different % instances) m = mean(allMu); pos = find(m==min(m)); samps(:,chain) = allMu(:,pos);

100 101 102 103 104 105

end

106 107 108 109 110 111 112 113 114 115 116 117 118

% Compute Rhat muc = zeros(nSamps,nChain); vc = zeros(nSamps,nChain); for s = 1:nSamps muc(s,:) = mean(samps(1:s,:),1); vc(s,:) = var(samps(1:s,:),1); end W = mean(vc,2); count = [1:nSamps]'; B = (count./(nChain−1)).*sum((muc − repmat(mean(muc,2),1,nChain)).ˆ2,2); V = ((count−1)./count).*W + (1./count).*B; Rhat = sqrt(V./W);

119 120 121

figure(5); plot(count,Rhat);

EX 10.3. We observe N binary vectors, each of which has D dimensions. We model them as a mixture model with K components, each of which is parameterised by a a vector

K26591_SM_Cover.indd 61

05/04/16 3:38 pm

58

CHAPTER 10. of D probabilities, pk = [pk1 , . . . , pkD ]T . For each component of these vectors we assume a Beta prior with parameters α and β: p(pkd |α, β) =

Γ(α + β) α−1 p (1 − pkd )β−1 . Γ(α)Γ(β) kd

The vector of prior probabilities for each component π = [π1 , . . . , πK ]T is given a uniform Dirichlet prior with parameter γ. The likelihood of an observation xn = [xn1 , . . . , xnD ]T being in component k is: P (xn |pk ) =

D

d=1

pxkdnd (1 − pkd )1−xnd .

The posterior over all parameters is given by: K N K

znk p(pk |α, β) (πk p(xn |pk )) . p(Z, π, p1 , . . . , pK |X, α, β, γ) ∝ p(π|γ) n=1 k=1

k=1

To create a Gibbs sampler we require the conditional distributions for znk , pkd and πk . The conditional density N for π k is the same as in the Gaussian case – a Dirichlet with parameters γ + n=1 znk :

N N

p(π| . . .) = Dir γ + zn1 , . . . , γ + znK . n=1

n=1

The conditional distribution for znk is: P (znk = 1| . . .) ∝ πk p(xn |pk ), where the normalisation constant will be the sum of these terms over j = 1 . . . K. Finally, the conditional distribution for pkd is derived as follows: p(pkd | . . .)

∝ ∝ ∝

β−1 pα−1 kd (1 − pkd )

N

xnd znk pkd (1 − pkd )1−xnd

n=1

z x α−1 β−1 pkd (1 − pkd ) pkdn nk nd (1 − pkd ) n znk (1−xnd )

α−1+ n znk xnd pkd (1 − pkd )β−1+ n znk (1−xnd )

which is a Beta density with parameters α∗ = α +

n znk (1 − xnd ).

n znk xnd

and β ∗ = β +

EX 10.4. The conditional density of the mean parameter given in the book is: p(µk | . . .) = N (a, B) where: B=

K26591_SM_Cover.indd 62

Σ−1 0 +

n

znk I

−1

,

05/04/16 3:38 pm

59 and a=B

Σ−1 0 µ0 +

znk xn .

n

We are interested in computing the conditional probability of znk = 1 with the mean parameter µk collapsed. The posterior over µk not including the nth observation is identical to that given above, with the summations omitting the nth component.

From now on, assume that a and B do not include the contribution from the nth observation. The likelihood is given by: p(x|µk ) = N (µk , I).

We therefore need to compute the following:

p(x| . . .) = N (µk , I)p(µk |a, B) dµk .

As both components are Gaussian, this is another Gaussian: p(x| . . .) = N (a, B + I).

Therefore, the conditional probability for znk = 1 is proportional to this term multiplied by the term derived in the book from collapsing π. It is normalised by dividing by the same term summed over j = 1 . . . K. EX 10.5. The following code will perform collapsed Gibbs sampling for a Gaussian mixture with 1-dimensional data.

1 2 3

% 1D Gaussian mixture with collapsed Gibbs sampling x = [randn(20,1);randn(20,1)+3]; x = sort(x);

4 5

N = length(x);

6 7 8 9

% Define the prior parameters mu0 = 0; ss0 = 1;

10 11 12

% Component variance ss = 1;

13 14 15

% Number of components K = 2;

16 17 18

nSamps = 1000;

19 20 21 22 23

% Parameters for the prior Dirichlet alp = repmat(1,1,K); prk = gamrnd(alp,1); prk = prk./sum(prk);

24 25 26

K26591_SM_Cover.indd 63

% Randomly initialise Z Z = rand(N,K);

05/04/16 3:38 pm

60

CHAPTER 10.

27

Z = (Z == repmat(max(Z,[],2),1,K));

28 29 30 31 32

% Structures to hold the output allZZ = zeros(N); allMu = zeros(nSamps,K); allPrk = zeros(nSamps,K);

33 34 35 36 37

% Outer loop to run multiple chains for computing Rhat nChain = 5; samps = zeros(nSamps,nChain); for chain = 1:nChain

38 39 40 41 42 43 44 45 46 47 48

for s = 1:nSamps % Update the Z for n = 1:N Z(n,:) = 0; sumZ = sum(Z,1); sumXZ = sum(Z.*repmat(x,1,K),1); postvar = (1./ss0 + sumZ).ˆ(−1); postmean = postvar.*(mu0/ss0 + sumXZ); predvar = postvar + ss;

49

like = −(1./(2*postvar)).*(x(n) − postmean).ˆ2; like = like − 0.5*log(2*pi*postvar);

50 51 52

% Add the collapsed prior term like = like + log(sumZ + alp);

53 54 55

prob = exp(like − max(like)); prob = prob./sum(prob); pos = find(rand<=cumsum(prob),1); Z(n,pos) = 1;

56 57 58 59

end

60 61

% Sample the mu (even though we don't need it)

62 63

for k = 1:K ssk = 1/(1/ss0 + sum(Z(:,k))); muk = ssk * (mu0/ss0 + sum(Z(:,k).*x)); mu(k) = randn*sqrt(ssk) + muk; end

64 65 66 67 68 69 70 71 72 73 74

end

allMu(s,:) = mu'; Z = 1.0*Z; allZZ = allZZ + Z*Z';

75 76 77 78 79

figure(1); plot(allMu); figure(2); plot(allPrk);

80 81 82 83 84

allZZ = allZZ./nSamps; figure(3); imagesc(allZZ);

85

K26591_SM_Cover.indd 64

05/04/16 3:38 pm

61 % Compute the autocorrelation of the kth mu lag vals = [0:1:200]; ac = zeros(length(lag vals),1);

86 87 88 89

k = 1;

90 91

full vals = allMu(:,k); full vals = full vals − mean(full vals);

92 93 94

samp ss = var(full vals);

95 96 97

for l = 1:length(lag vals) ac(l) = sum(full vals(1:end−lag vals(l)).*full vals(1+lag vals(l):end)) ac(l) = ac(l)/(samp ss *(nSamps−lag vals(l))); end

98 99 100 101 102

figure(4); plot(lag vals,ac,'ro');drawnow

103 104 105

% Store the values of the lower mean (they switch in different % instances) m = mean(allMu); pos = find(m==min(m)); samps(:,chain) = allMu(:,pos);

106 107 108 109 110 111

end

112 113 114 115 116 117 118 119 120 121 122 123 124

% Compute Rhat muc = zeros(nSamps,nChain); vc = zeros(nSamps,nChain); for s = 1:nSamps muc(s,:) = mean(samps(1:s,:),1); vc(s,:) = var(samps(1:s,:),1); end W = mean(vc,2); count = [1:nSamps]'; B = (count./(nChain−1)).*sum((muc − repmat(mean(muc,2),1,nChain)).ˆ2,2); V = ((count−1)./count).*W + (1./count).*B; Rhat = sqrt(V./W);

125 126 127

figure(5); plot(count,Rhat);

EX 10.6. The following code samples from a CRP. Vary alpha to see the effect to the histogram: 1 2

% Sample from a CRP N = 100; % 100 customers

3 4

Z = ones(N,1); % Initialise with all customers at one table

5 6

nSamps = 1000;

7 8 9

alp = 0.1; % Concentration parameter to vary nK = zeros(nSamps,1);

10 11

K26591_SM_Cover.indd 65

for s = 1:nSamps

05/04/16 3:38 pm

62

CHAPTER 10. for n = 1:N thistable = find(Z(n,:)); Z(n,:) = 0; sumZ = sum(Z,1); if sumZ(thistable) == 0 sumZ(thistable) = []; Z(:,thistable) = []; end probs = [sumZ alp]; probs = probs./sum(probs); pos = find(rand<=cumsum(probs),1); Z(n,pos) = 1; end nK(s) = size(Z,2);

12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

end

27 28

hist(nK,unique(nK));

EX 10.7. It will be most efficient to use a collapsed Gibbs sampling scheme for this by marginalising the probabilities pkd . As shown above, the conditional posterior density over these parameters is a Beta distribution with parameters:

znk xnd , β ∗ = β + znk (1 − xnd ). α∗ = α + n

n

(remember that when considering the collapsed update for the nth observation we would omit the nth data point from the summation). The update for znk is: P (znk = 1| . . .) ∝ (

m =n

znk )

D

d=1

P (xnd |znk = 1, pkd )p(pkd | . . .) dpkd .

Because of the conjugacy of the Beta prior and the binomial likelihood, we can analytically evaluate the integral:

∗ Γ(α∗ + β ∗ ) α∗ −1 p (1 − pkd )β −1 dpkd = pxkdnd (1 − pkd )1−xnd Γ(α∗ )Γ(β ∗ ) kd

∗ Γ(α∗ + β ∗ ) α∗ +xnd −1 pkd = (1 − pkd )β +(1−xnd )−1 dpkd Γ(α∗ )Γ(β ∗ ) Γ(α∗ + β ∗ ) Γ(α∗ + xnd )Γ(β ∗ + (1 − xnd )) = Γ(α∗ )Γ(β ∗ ) Γ(α∗ + β ∗ + 1) (α∗ )xnd (β ∗ )1−xnd . = α∗ + β ∗ Where in the final step we have used the identity Γ(z + 1) = zΓ(z). This completes the definition of the sampler. The update for znk is proportional to the product of these terms over the dimensions (D) multiplied by the number of objects in the cluster k. The probability of a new cluster is proportional to the concentration parameter multiplied by the product of these terms where α∗ = α and β ∗ = β.

EX 10.8. The following code samples from a DP with a Gaussian base distribution:

K26591_SM_Cover.indd 66

05/04/16 3:38 pm

63

1 2 3 4

% DP sample basemu = 0; basess = 1; alp = 10; % Concentration parameter

5 6

N = 100; % Number of samples to draw

7 8

samples = [];

9 10 11 12 13 14 15 16 17 18 19 20 21 22

for n = 1:N if n==1 samples = randn.*sqrt(basess) + basemu; else if rand < alp/(alp + n−1) % Sample a new value samples(end+1) = randn.*sqrt(basess) + basemu; else % Copy a previous one samples(end+1) = samples(randi(length(samples))); end end end

EX 10.9. See script on accompanying webpage.

EX 10.10. We will derive the collapsed sampler. Assume that we observe m = 1 . . . M datasets, each of which has n = 1 . . . Nm observations, xm n . There are k = 1 . . . K top-level components and the top-level concentration parameter is α. The concentration parameter in the mth file is γm . In the mth file there are v = 1 . . . Vm components, m = 1 if the nth each associated with one of the top-level components. Let znv observation in the mth file is associated with the vth component in that file. Let m = 1 if the vth component in the mth file is associated with the kth top-level wvk component. m For the collapsed sampler, we require the distributions for re-sampling znv . When we re-sample this assignment, there are three possibilities. 1: We put it into a current component in file m. 2: We put it into a new component that uses a preexisting top-level component. 3: We put it into a new component that has a new top-level component. We will consider each of these in turn. Firstly, though we need the posterior density over µk (the mean of the kth top-level component). This is Gaussian and given by:

p(µk | . . .)

b

2

a

K26591_SM_Cover.indd 67

∝

N (µ0 , σ02 )

=

N (a, b2 )

= =

σ02 + σ12

b2

Vm M N m

m=1 n=1 v=1

1

m

m

2 znv wvk N (xm n |µk , σ )

(10.1)

−1

m m m,n,v znv wvk

m m m µ0 m,n,v znv wvk xn + σ2 σ02

05/04/16 3:38 pm

64

CHAPTER 10.

The predictive density is therefore (where summations will not include the data point in question): m m 2 2 m p(xm n |znv = 1, wvk = 1) = N (a, b + σ ) = L(xn |k)

Note that the expression for a brand new top-level component is identical but all the summations will equate to zero.

The probability of placing the object in component v is therefore proportional to:

m m m = 1|wvk = 1, . . .) ∝ L(xm znv P (znv n |k) n

To compute the probability of placing the point in a new component in file m we have to marginalise over the top level components (admitting the possibility of a new one). Let sk be the number of components across all files that are assigned to top-level component k:

m m ∗ αL(x |k ) + s L(x |l) l n n m l P (znv ∗ = 1| . . .) ∝ γm α + j sj

We can therefore use these two probabilities to decide if the nth object ought to go into one of the current components in the mth file, or a new component. If we choose a new component, we then need to choose which top-level component it ought to be associated with (or a new one). The probability of it being a current one is proportional to sl L(xm n |l) and the probability of a new one is proportional to ∗ αL(xm |l ). n

This completes the sampler, although it might not be particularly efficient. To m for each file-level component. improve its efficiency, one could re-sample the wvk It is fairly straightforward to compute the required distribution but, be careful: if, as in this case, the mean parameters have all been marginalised, the joint density of all of the objects assigned to this component cannot be assumed to be independent in the standard way (they are all dependent via the marginalisation of the mean parameters). This can be overcome by, for example, computing p(x1 , x2 , x3 , . . . | . . .) = p(x1 | . . .)p(x2 |x1 , . . .)p(x3 |x1 , x2 , . . .) etc.

EX 10.11. The equation we are trying to obtain is the probability that the ith word in the nth document comes from topic k. This will be the product of two components: the probability of the kth topic in document n multiplied by the probability of word w (assuming that the ith word is word w) in topic k. Based on the definitions in the text, in the un-collapsed world, this would be given by: P (zni = k|xni = w, βkw , θnk ) = θnk βkw .

To obtain the collapsed version, we need to marginalise the parameters θ n and β k . Both are multinomials and have Dirichlet priors with parameters α and γ respectively. We know from our work in mixture models that if we have one sample from a multinomial (say the kth value has been sampled) whose parameters are Dirichlet distributed, the probability of that sample is equal to the kth Dirichlet

K26591_SM_Cover.indd 68

05/04/16 3:38 pm

65 parameter divided by the sum of the parameters. We also know that the posterior Dirichlet for a Dirichlet prior and multinomial likelihood has parameters equal to the prior parameters plus the observation counts. Therefore, defining c−i nk as the count of the number of words (excluding the ith one) in the nth document currently assigned to topic k, we know that the term associated with the marginalisation of θ n is: c−i + αk nk−i j cnj + αj and the term associated with the marginalisation of β k is:

vkw + γw

w vkw + γw

Multiplying the two together gives the update given in the text.

K26591_SM_Cover.indd 69

05/04/16 3:38 pm

The title documents are available in full on the following links Title: EBOOK to A First Course in Machine Learning 2nd Edition by Simon Rogers & Mark Girolami - Complete Elaborated & Latest. ALL Chapters (1-10) Included & Updated

Link I: https://drive.google.com/file/d/1657iW3dPWDtDBfJXtLGDqQBVpxyARsoZ/view?usp=sharing