Exercises
493
15.17 (?) Show that if we maximize the expected complete-data log likelihood function (15.45) for a mixture of Bernoulli distributions with respect to µk , we obtain the M-step equation (15.49). 15.18 (?) Show that if we maximize the expected complete-data log likelihood function (15.45) for a mixture of Bernoulli distributions with respect to the mixing coefficients πk , and use a Lagrange multiplier to enforce the summation constraint, we obtain the M-step equation (15.50). 15.19 (?) Show that as a consequence of the constraint 0 6 p(xn |µk ) 6 1 for the discrete variable xn , the incomplete-data log likelihood function for a mixture of Bernoulli distributions is bounded above and hence that there are no singularities for which the likelihood goes to infinity. 15.20 (? ? ?) Consider a D-dimensional variable x each of whose components i is itself a multinomial variable of degree M so that x is a binary vector with components xij P where i = 1, . . . , D and j = 1, . . . , M , subject to the constraint that j xij = 1 for all i. Suppose that the distribution of these variables is described by a mixture of the Section 3.1.3 discrete multinomial distributions so that p(x) =
K X
πk p(x|µk )
(15.66)
k=1
where p(x|µk ) =
D Y M Y
x
ij µkij .
(15.67)
i=1 j =1
The parameters µkij represent the probabilities P p(xij = 1|µk ) and must satisfy 0 6 µkij 6 1 together with the constraint j µkij = 1 for all values of k and i. Given an observed data set {xn }, where n = 1, . . . , N , derive the E-step and M-step equations of the EM algorithm for optimizing the mixing coefficients πk and the component parameters µkij of this distribution by maximum likelihood. 15.21 (?) Verify the relation (15.52) in which L(q, θ) and KL(qkp) are defined by (15.53) and (15.54), respectively. 15.22 (?) Show that the lower bound L(q, θ) given by (15.53), with q(Z) = p(Z|X, θ (old) ), has the same gradient with respect to θ as the log likelihood function ln p(X|θ) at the point θ = θ (old) . 15.23 (? ?) Consider the incremental form of the EM algorithm for a mixture of Gaussians, in which the responsibilities are recomputed only for a specific data point xm . Starting from the M-step formulae (15.16) and (15.17), derive the results (15.60) and (15.61) for updating the component means. 15.24 (? ?) Derive M-step formulae for updating the covariance matrices and mixing coefficients in a Gaussian mixture model when the responsibilities are updated incrementally, analogous to the result (15.60) for updating the means.