Computer Vision: Models, Learning, and Inference By Prince, Simon J. D. INSTRUCTORS SOLUTIONS MANUAL

INSTRUCTORS SOLUTIONS MANUAL

Chapter 2

Introduction to probability Problem 2.1 Give a real-world example of a joint distribution P r(x, y) where x is discrete and y is continuous.

Answer Consider the joint distribution of the time y > 0 taken before a vehicle passes in front of your house (continuous) and the type of vehicle x ∈ {car, truck, motorbike, van} (discrete).

Problem 2.2 What remains if I marginalize a joint distribution P r(v, w, x, y, z) over five variables with respect to variables w and y? What remains if I marginalize the resulting distribution with respect to v?

Answer Assuming the variables are continuous, when we marginalize out w and y we get Z Z P r(v, w, x, y, z)dwdy = P r(v, x, z).

When we further marginalize out the variable v we get Z P r(v, x, z)dv = P r(x, z).

Problem 2.3 Show that the following relation is true: P r(w, x, y, z) = P r(x, y)P r(z|w, x, y)P r(w|x, y).

Answer We use the relation P r(a, b) = P r(a|b)P r(b) twice: Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

6

2

P r(x, y)P r(z|w, x, y)P r(w|x, y)

Introduction to probability

=

P r(w|x, y)P r(x, y)P r(z|w, x, y)

=

P r(w, x, y)P r(z|w, x, y)

=

P r(z, w, x, y)

=

P r(w, x, y, z),

as required.

Problem 2.4 In my pocket there are two coins. Coin 1 is unbiased, so the likelihood P r(h = 1|c = 1) of getting heads is 0.5 and the likelihood P r(h = 0|c = 1) of getting tails is also 0.5. Coin 2 is biased, so the likelihood P r(h = 1|c = 2) of getting heads is 0.8 and the likelihood P r(h = 0|c = 2) of getting tails is 0.2. I reach into my pocket and draw one of the coins at random. There is an equal prior probability I might have picked either coin. I flip the coin and observe a head. Use Bayes’ rule to compute the posterior probability that I chose coin 2.

Answer Applying Bayes’ rule we have

P r(c = 2|h = 1)

= = =

P r(h = 1|c = 2) · P r(c = 2) P2 k=1 P r(h = 1|c = k) · P r(c = k) 0.8 · 0.5 0.5 · 0.5 + 0.8 · 0.5 8/13 = 0.6153.

There is 61.5% probability that it I have drawn the second coin.

Problem 2.5 If variables x and y are independent and variables x and z are independent, does it follow that variables y and z are independent?

Answer No, it does not follow. Consider any general distribution P r(y, z) where y and z are NOT independent. Now consider the marginal distributions P r(y) and P r(z). It is perfectly possible to have a third distribution P r(x) which does not provide any information about y or z and hence is independent of each and P r(x, y) = P r(x)P r(y) and P r(x, z) = P r(x)P r(z).

If you are unsure about this, then construct a counter example where x, y and z are all discrete variables with two entries. Construct a non-independent 2 × 2 distribution between y and z, marginalize it with respect to each and then construct 2 × 2 independent joint distributions between x and z and x and y.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

7 Problem 2.6 Use equation 2.3 to show that when x and y are independent, the marginal distribution P r(x) is the same as the conditional distribution P r(x|y = y ∗ ) for any y ∗ .

Answer Starting with equation 2.3 we have

P r(x|y = y ∗ ) =

P r(x, y = y ∗ ) .

P r(y = y ∗ )

Now we use the definition of independence P r(x, y = y ∗ ) = P r(x) · P r(y = y ∗ ) in the numerator to yield

P r(x|y = y ∗ )

= =

P r(x)P r(y = y ∗ ) P r(y = y ∗ ) P r(x).

In other words the distribution of x does not depend on the particular value of y ∗ and the marginal and conditional distributions are the same. You can see this property in Figure 2.6. If you cut the distribution in any place (compute the conditional probability) then it has the same shape. Consequently, if you marginalize (sum all of the profiles with the same shape) then the result has the same profile again.

Problem 2.7 The joint probability P r(w, x, y, z) over four variables factorizes as P r(w, x, y, z) = P r(w)P r(z|y)P r(y|x, w)P r(x).

Demonstrate that x is independent of w by showing that P r(x, w) = P r(x)P r(w).

Answer We compute the distribution P (x, w) by marginalizing the joint distribution P r(w, x, y, z) with respect to the unwanted variables y and z:

Z Z P r(x, w)

=

P r(w, x, y, z)dydz Z Z

= =

P r(w)P r(z|y)P r(y|x, w)P r(x)dydz P r(w)P r(x) = P r(x)P r(w)

where in the third line we have simply integrated over the unwanted variables y and z, removing them from the equations.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

8

2

Introduction to probability

Problem 2.8 Consider a biased die where the probabilities of rolling sides {1, 2, 3, 4, 5, 6} are {1/12, 1/12, 1/12, 1/12, 1/6, 1/2}, respectively. What is the expected value of the die?

If I roll the die twice, what is the expected value of the sum of the two rolls?

Answer Let the variable x represent the value of the die. Using the definition of expectation we have

E[x]

=

6 X

k · P r(x = k)

k=1

1 1 1 1 1 1 +2· +3· +4· +5· +6· 12 12 12 12 6 2

=

1·

=

56 = 4.6667 12

To compute the expected value of the sum of two rolls, we use the second expectation relation (Equation 2.15): E[x1 + x2 ] = E[x1 ] + E[x2 ] = 2 · 4.6667 = 9.3333

Problem 2.9 Prove the four relations for manipulating expectations.

E[κ]

=

κ,

E[κf [x]]

=

κE[f [x]],

E[f [x] + g[x]]

=

E[f [x]] + E[g[x]],

E[f [x]g[y]]

=

E[f [x]]E[g[y]],

if x, y independent.

Answer Relation 1: Z E[κ]

= =

κP r(x)dx Z κ P r(x)dx

=

κ.

Relation 2: Z E[κf [x]]

= =

κf [x]P r(x)dx Z κ f [x]P r(x)dx

=

κE[f [x]].

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

9 Relation 3:

Z E[f [x] + g[x]]

=

(f [x] + g[x])P r(x)dx Z

= Z = =

(f [x]P r(x) + g[x]P r(x)) dx Z f [x]P r(x)dx + g[x]P r(x)dx

E[f [x]] + E[g[x]].

Relation 4:

Z Z E[f [x] · g[y]]

f [x] · g[y]P r(x, y)dxdy

= Z Z = Z = =

f [x] · g[y]P r(x)P r(y)dxdy Z f [x]P r(x)dx g[y]P r(y)dy

E[f [x]]E[g[y]],

where we have used the definition of independence between the first two lines.

Problem 2.10 Use the relations from problem 2.9 to prove the following relationship between the second moment around zero and the second moment about the mean (variance):

E (x − µ)2 = E x2 − E[x]E[x].

Answer Using the third relation we have

E (x − µ)2

=

E x2 − 2µx + µ2

=

E[x2 ] − E[2µx] + E[µ2 ]

The second term term here is a constant multiplied by a function of x and so rule 1 can be applied. The third term is the expectation of a constant which, by rule 1, is just a constant itself. Applying these manipulations gives

E (x − µ)2

=

E[x2 ] − 2µE[x] + µ2 .

We now note that the mean µ = E[x] and substitute in to give Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

10

2

E (x − µ)2

Introduction to probability

=

E[x2 ] − 2µE[x] + µ2

=

E[x2 ] − 2E[x]E[x] + E[x]E[x]

E x2 − E[x]E[x],

= as required.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

Chapter 3

Common probability distributions Problem 3.1 Consider a variable x which is Bernoulli distributed with parameter λ. Show that the mean E[x] is λ and the variance E[(x − E[x])2 ] is λ(1 − λ).

Answer

E[x]

=

X

=

0 · (1 − λ) + 1 · λ

=

λ.

xP r(x)

To compute the variance, we first compute the expectation E[x]2 as

E[x2 ]

=

X

=

02 · (1 − λ) + 12 · λ

=

λ.

x2 P r(x)

Then we use the relation from problem 2.10.

E (x − µ)2

=

E x2 − E[x]E[x]

=

λ − λ2

=

λ(1 − λ).

Problem 3.2 Calculate an expression for the mode (position of the peak) of the beta distribution with α, β > 1 in terms of the parameters α and β.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

12

3

Common probability distributions

Answer The beta distribution is given by

P r(λ)

=

Γ[α + β] α−1 λ (1 − λ)β−1 , Γ[α]Γ[β]

We want to maximize this expression with respect to λ. Since the logarithm is a monotonic transformation, the maximum of the logarithm of this expression will in the same place, so we alternatively maximize: L = log[P r(λ)] = κ + (α − 1) log[λ] + (β − 1) log[(1 − λ)], where we have subsumed the terms that do not depend on λ into the constant κ. To find the maximum, we compute the derivative: ∂L α−1 β−1 = − = 0. ∂λ λ 1−λ Multiplying by the denominators of each fraction we get (1 − λ)(α − 1) − λ(β − 1) = 0.

Collecting together all of the terms in λ , we get α − 1 − λ(α + β − 2) = 0, which can be re-arranged to give λ̂ =

α−1 . α+β−2

Problem 3.3 The mean and variance of the beta distribution are given by the expressions

E[λ] = µ

=

α α+β

E[(λ − µ)2 ] = σ 2

=

αβ . (α + β)2 (α + β + 1)

We may wish to choose the parameters α and β so that the distribution has a particular mean µ and variance σ 2 . Derive suitable expressions for α and β in terms of µ and σ 2 .

Answer We first re-arrange the first expression to get the relation α=

βµ 1−µ

Then we substitute this into the second expression to get Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

13

σ2

=

αβ (α + β)2 (α + β + 1)

=

βµ β 1−µ βµ βµ ( 1−µ + β)2 ( 1−µ + β + 1)

=

β 2 µ(1 − µ)2 (βµ + β(1 − µ))2 (βµ + β(1 − µ) + (1 − µ))

=

β 2 µ(1 − µ)2 β+1−µ

This gives us the quadratic equation

β 2 µ(1 − µ)2 − βσ 2 − (1 − µ)σ 2 , which can be solved using the standard formula

β

p σ 4 + 4µ(1 − µ)2 (1 − µ)σ 2 2µ(1 − µ)2 p σ 2 ± σ σ 2 + 4µ(1 − µ)3 .

2µ(1 − µ)2

σ2 ±

= =

Substituting this expression into the original expression for α we have

α

=

p σ 2 ± σ σ 2 + 4µ(1 − µ)3 βµ = , 1−µ 2(1 − µ)3

where we choose the root such that α, β > 0.

Problem 3.4 All of the distributions in this chapter are members of the exponential family and can be written in the form P r(x|θ) = a[x] exp[b[θ]T c[x] − d[θ]], where a[x] and c[x] are functions of the data and b[θ] and d[θ] are functions of the parameters. Find the functions a[x], b[θ], c[x] and d[θ] that allow the Beta distribution to be represented in the generalized form of the exponential family.

Answer The beta distribution is given by:

P r(x)

=

Γ[α + β] α−1 x (1 − x)β−1 . Γ[α]Γ[β]

We choose functions Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

14

3

a[x]

=

b[θ]

=

1

α−1 β−1

Common probability distributions

c[x]

=

log[x] log[1 − x]

d[θ]

=

log[Γ[α]] + log[Γ[β]] − log[Γ[α + β],

These functions now conform to the multivariate version of the exponential family.

P r(x|θ) = a[x] exp[b[θ]T c[x] − d[θ]],

Problem 3.5 Use integration by parts to prove that if Z ∞ Γ[z] =

tz−1 e−t dt,

0

then Γ[z + 1] = zΓ[z].

Answer We’ll start with the left hand side Z ∞ Γ[z + 1] =

tz e−t dt.

0

Denoting tz by u and e−t by dv, we use the relation Z

Z u · dv = u · v −

v · du.

Noting that du = ztz−1 and v = −e−t , we get

Γ[z + 1]

=

Z z −t ∞ t · e 0 − −ze−t · tz−1 dt Z 0 − 0 + tz−1 e−t dt

=

zΓ[z],

=

as required.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

15 Problem 3.6 Consider a restricted family of univariate normal distributions where the variance is always 1, so that

1 P r(x|µ) = √ exp −0.5(x − µ)2 .

2π Show that a normal distribution over the parameter µ P r(µ) = Normµ [µp , σp2 ] has a conjugate relationship to the restricted normal distribution.

Answer Taking the product of the two distributions we get

P r(x|µ) · P r(µ)

= = = =

(µ − µp )2 1 1 2 √ exp −0.5(x − µ) · √ exp −0.5 σp2 2π 2πσ 2

µ2p 1 µ2 2µµp exp −0.5 x2 − 2µx + µ2 − 2 − 2 2 + 2 2πσ σp σp σp

2

2 µ σ + 1 σp2 x + µp 1 p p 2 exp −0.5 x2 + 2 exp −0.5 µ − 2 µ 2πσ σp σp2 σp2

2

2 σp + 1 2 σp x + µp κ1 · exp −0.5 µ −2 µ σp2 σp2

The constant and the first exponential terms in the third line do not depend on µ so we denote them by the constant κ1 for short. The second term is a quadratic function in µ.

We now complete the square of the second term by multiply and dividing by a suitable factor:

P r(x|µ) · P r(µ)

= =

=

2 σp2 x + µp σp + 1 2 µ − 2 µ κ1 · exp −0.5 σp2 σp2 "

2 !# σp2 σp2 x + µp κ1 · exp −0.5 − 2 σp + 1 σp2 " 2 2 !# σp2 + 1 2 σp2 x + µp σp2 σp x + µp · exp −0.5 µ −2 µ+ 2 σp2 σp2 σp + 1 σp2 "

2 !# σp2 + 1 2 σp2 x + µp σp2 σp2 x + µp µ −2 µ+ 2 κ2 · exp −0.5 σp2 σp2 σp + 1 σp2 

2  σ 2 x+µp

=

µ − pσ2 +1  p  κ2 · exp −0.5 2 2 σp /(σp + 1) 

   

where the extraneous factor that we multiplied by in the second line is incorporated into a second constant κ2 between the second and third lines. The final result is a constant multiplied by a normal distribution and so we conclude that this distribution was conjugate as was originally claimed in the question.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

16

3

Common probability distributions

Problem 3.7 For the normal distribution, find the functions a[x], b[θ], c[x] and d[θ] that allow it to be represented in the generalized form of the exponential family (see problem 3.4).

Answer The normal distribution has the form

P r(x)

= =

(x − µ)2 1 exp −0.5 σ2 2πσ 2

2

x 1 µ2 µx √ exp −0.5 + 2 −2 2 σ2 σ σ 2πσ 2 √

a[x]

=

b[θ]

=

1

− 2σ1 2

µ σ2

c[x]

=

2 x x

d[θ]

=

h√ i µ2 + log 2πσ 2 2 2σ

These functions now conform to the multivariate version of the exponential family.

P r(x|θ) = a[x] exp[b[θ]T c[x] − d[θ]],

Problem 3.8 Calculate an expression for the mode (position of the peak in µ, σ 2 space) of the normal scaled inverse gamma distribution in terms of the parameters α, β, γ, δ.

Answer The normal-scaled inverse gamma distribution has the pdf: P r(µ, σ 2 ) =

α+1

√ γ βα 2β + γ(δ − µ)2 1 √ exp − .

2σ 2 σ 2π Γ[α] σ 2

To compute the peak of the distribution with respect to µ and σ 2 we want to find the position where the derivatives of the pdf with respect to these parameters is zero. However, to make this process easier, we first pass the pdf through the logarithmic function. Since this function is monotonic, the peak will remain in the same place. The log likelihood L is given by L = κ − (α + 3/2) log[σ 2 ] −

γ(δ − µ)2 β − , 2 σ 2σ 2

where we have subsumed the log terms which do not depend on the parameters into a single constant term κ.

The derivative with respect to µ is Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

17

γ(δ − µ) ∂L = = 0. ∂µ σ2 Setting this to zero and re-arranging gives the relation µ = δ.

The derivative with respect to σ 2 is α + 3/2 ∂L β =− + 4 = 0, ∂σ 2 σ2 σ where the last term has disappeared because at the maximum µ = δ. Re-arranging this equation we get the expression σ2 =

β . α + 3/2

Problem 3.9 Show that the more general form of the conjugate relation in which we multiply I Bernoulli distributions by the conjugate beta prior is given by I Y

Bernxi [λ] · Betaλ [α, β] = κ · Betaλ [α̃, β̃],

i=1

where

κ

=

α̃

=

β̃

=

P P Γ[α + β]Γ[α + xi ]Γ[β + (1 − xi )] Γ[α + β + I]Γ[α]Γ[β] X α+ xi X β+ (1 − xi ).

Answer The answer is achieved by multiplying out the terms to create an expression that is proportional to a new Beta distribution. We then multiply and divide by the constant associated with the new distribution.

I Y

Bernxi [λ]Betaλ [α, β]

i=1

=

I Y i=1 P

= λ

λxi (1 − λ)1−xi . i xi

(1 − λ)

P

Γ[α + β] α−1 λ (1 − λ)β−1 Γ[α]Γ[β]

i (1−xi )

.

Γ[α + β] α−1 λ (1 − λ)β−1 Γ[α]Γ[β]

P Γ[α + β] Pi xi +α−1 λ (1 − λ)I− i xi +β−1 Γ[α]Γ[β] " # P P X X Γ[α + β] Γ[ i xi + α]Γ[I − i xi + β] P . P Betaλ xi + α, I − xi + β = Γ[α]Γ[β] Γ[ xi + α + I − xi + β] i i " # P P X X Γ[α + β]Γ[ i xi + α]Γ[I − i xi + β] = Betaλ xi + α, I − xi + β Γ[α + β + I]Γ[α]Γ[β] i i

=

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

18

3

Common probability distributions

as required.

Problem 3.10 Prove the conjugate relation I Y

Catxi [λ1...K ] · Dirλ1...K [α1...K ] = κ · Dirλ1...K [α̃1...K ],

i=1

where

κ̃

=

P QK Γ[ K j=1 αj ] j=1 Γ[αj + Nj ] . QK PK Γ[I + j=1 αj ] j=1 Γ[αj ]

α̃1...K

=

[α1 + N1 , α2 + N2 , . . . , αK + NK ].

and Nk is the total number of times that the variable took the value k.

Answer I Y

Catxi [λ1...K ]Dirλ1...K [α1...K ]

i=1

=

I Y K Y i=1 j=1

! x λj ij

P K Y Γ[ K α −1 j=1 αj ] λj j QK Γ[α ] j j=1 j=1

P K Y Γ[ K α −1+Nj j=1 αj ] = QK λj j j=1 Γ[αj ] j=1 ! ! K PK P QK Y αj +Nj −1 Γ[ j=1 αj ] Γ[I + K j=1 αj ] j=1 Γ[αj + Nj ] λj = QK . .

PK QK Γ[I + j=1 αj ] j=1 Γ[αj ] j=1 Γ[αj + Nj ] j=1 !

P QK Γ[ K j=1 αj ] j=1 Γ[αj + Nj ] = QK Dirλ1...K [α1 + N1 , α2 + N2 , . . . , αk + NK ] .

P Γ[I + K j=1 αj ] j=1 Γ[αj ] P QK Γ[ K j=1 αj ] j=1 Γ[αj + Nj ] = . QK Dirλ1...K [α1 + N1 , α2 + N2 , . . . , αk + NK ] PK Γ[I + j=1 αj ] j=1 Γ[αj ] P P where Nj = i xij and j Nj = I. The new parameters and constant can easily be read off from this expression.

Problem 3.11 Show that the conjugate relation between the normal and normal inverse gamma is given by I Y

Normxi [µ, σ 2 ] · NormInvGamµ,σ2 [α, β, γ, δ] = κ · NormInvGamµ,σ2 [α̃, β̃, γ̃, δ̃],

i=1

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

19 where

κ

=

α̃

=

β̃

=

γ̃

=

δ̃

=

√ α γβ Γ[α̃] 1 √ α̃ I/2 (2π) γ̃ β̃ Γ[α] α + I/2 P 2 P (γδ + i xi )2 γδ 2 i xi +β+ − 2 2 2(γ + I) γ+I P (γδ + i xi ) . γ+I

Answer I Y i=1

Normxi [µ, σ 2 ]NormInvGamx [α, β, γ, δ] = √ α (α+1) γβ 1 1 √ (2π)I/2 σ I σ 2πΓ[α] σ 2 " !# X 1 2 2 exp − 2 (xi − µ) + 2β + γ(δ − µ) 2σ i √ α (α+1) γβ 1 1 √ = (2π)I/2 σ I σ 2πΓ[α] σ 2 " !# X 2 X 1 2 2 2 exp − 2 xi + Iµ − 2µ xi + 2β + γδ + γµ − 2µγδ 2σ i i √ α (α+1) γβ 1 1 √ = (2π)I/2 σ I σ 2πΓ[α] σ 2 " !# P P

X 2 (γδ + i xi ) 2 (γδ + i xi )2 1 2 exp − 2 xi + 2β + γδ + (γ + I) µ − − 2σ γ+I γ+I i √ α (α+1) γβ 1 1 √ = (2π)I/2 σ I σ 2πΓ[α] σ 2 " P P P 2

!# (γδ + i xi ) 2 (γδ + i xi )2 1 γδ 2 i xi exp − 2 2 +β+ − + (γ + I) µ − , 2σ 2 2 2(γ + I) γ+I

=

from which the parameters and the constant can easily be read off.

Problem 3.12 Show that the conjugate relationship between the multivariate normal and the normal inverse Wishart is given by I Y

h i Normxi [µ, Σ] · NorIWisµ,Σ [α, Ψ, γ, δ] = κ · NorIWis α̃, Ψ̃, γ̃, δ̃ ,

i=1

where Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

20

3

κ

=

α̃

=

Ψα/2 ΓD [α̃/2] γ D/2 π ID/2 Ψ̃α̃/2 ΓD [α/2] γ̃ D/2 α+I

Ψ̃

=

Ψ + γδδ T +

γ̃

=

γ+I

δ̃

=

P γδ + Ii=1 xi . γ+I

Common probability distributions

1

I X i=1

xi xTi −

1 (γ + I)

γδ +

I X i=1

! xi

γδ +

I X

!T xi

i=1

You may need to use the relation Tr zzT A−1 = zT A−1 z.

Answer TO DO

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

Chapter 4

Fitting probability models Problem 4.1 Show that the maximum likelihood solution for the variance σ 2 of the normal distribution is given by σ2 =

I X (xi − µ̂)2 i=1

I

.

Answer The likelihood of a set of data {xi }Ii=1 under the normal probability distribution is

P r(x) =

I Y

1 √ exp −0.5(xi − µ)2 /σ 2 , 2 2πσ i=1

To find the maximum likelihood solution for the variance σ 2 we work with the log likelihood which has the form I X (xi − µ)2 I I L = − log[2π] − log[σ 2 ] − , 2 2 2σ 2 i=1

Now we compute the derivative with respect to σ 2 which is I X (xi − µ)2 ∂L I = + .

2 2 ∂σ 2σ 2σ 4 i=1

Setting this equation to zero and solving we find that

σ2

=

I X (xi − µ̂)2 i=1

I

.

as required.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

22

4

Fitting probability models

Problem 4.2 Show that the MAP solution for the mean µ and variance σ 2 of the normal distribution are given by PI µ̂ =

i=1 xi + γδ

and

I +γ

σˆ2 =

PI

+ 2β + γ(δ − µ̂)2 , I + 3 + 2α

i=1 (xi − µ̂)

2

when we use the conjugate normal-scaled inverse gamma prior P r(µ, σ 2 ) =

α+1

√ γ βα 2β + γ(δ − µ)2 1 √ exp − .

2σ 2 σ 2π Γ[α] σ 2

Answer In the MAP solution we aim to maximizing the expression L=

I X

log [P r(xi |θ)] + log [P r(θ)] .

i=1

Substituting in the normal and normal scaled inverse gamma terms we have

L

=

=

I X (xi − µ)2 I 1 1 I + log[γ] − log[σ 2 ] − log[2π] − log[σ 2 ] − 2 2 2 2σ 2 2 i=1

2β + γ(δ − µ)2 1 − log[2π] + α log[β] − log[Γ[α]] − (α + 1) log[σ 2 ] − 2 2σ 2 I 2 X (xi − µ) I +1 I + 3 + 2α 1 − log[2π] − log[σ 2 ] − + log[γ] 2 2 2 2σ 2 i=1 +α log[β] − log[Γ[α]] −

2β + γ(δ − µ)2 2σ 2

Taking derivatives with respect to µ we get

∂L ∂µ

=

I X (xi − µ) i=1

=

1 σ2

σ2 I X

+

γ(δ − µ) σ2 !

xi − Iµ + γδ − γµ

i=1

Setting this equation to zero and solving for µ we get PI µ̂ =

i=1 xi + γδ

I +γ

,

as required.

Taking derivatives with respect to σ 2 we get

∂L ∂σ 2

=

−

I 2β + γ(δ − µ)2 I + 3 + 2α X (xi − µ)2 + + .

2 4 2σ 2σ 2σ 4 i=1

Setting this expression to zero and re-arranging gives us Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

23

σˆ2 =

PI

+ 2β + γ(δ − µ̂)2 , I + 3 + 2α

i=1 (xi − µ̂)

2

as required.

Problem 4.3 Taking equation 4.29 as a starting point, show that the maximum likelihood parameters for the categorical distribution are given by Nk

λ̂k = P6

m=1 Nm

,

where Nk is the number of times that category K was observed in the training data.

Answer From equation 4.29 the log likelihood for the categorical distribution is given by

L=

6 X

Nk log[λk ] + ν

k=1

6 X

! λk − 1 ,

k=1

where ν is a Lagrange multiplier that was added to enforce the constraint Taking the derivative with respect to λk we get

P6

k=1 λk = 1.

∂L Nk = + ν, ∂λk λk Setting this to zero and re-arranging we have Nk + νλk = 0.

Now if we sum the six equations involving λ1 . . . λk together we get 6 X m=1

Nm + ν

M X

λm

=

m=1

6 X

Nm + ν = 0,

m=1

P6 where P we have used the constraint k=1 λk = 1. From this we can establish that ν = − m Nm . Substituting this back into the original derivative we have 6 X ∂L Nk = − Nm . ∂λk λk m=1

Setting this to zero and solving for λk yields the desired relation λ̂k = P6

Nk

m=1 Nm

.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

24

4

Fitting probability models

Problem 4.4 Show that the MAP estimate for the parameters {λ}K k=1 of the categorical distribution is given by Nk + αk − 1 , m=1 (Nm + αm − 1)

λ̂k = P6

under the assumption of a Dirichlet prior with hyperparameters {αk }K k=1 . The terms Nk again indicate the number of times that category k was observed in the training data.

Answer The criterion which we wish to optimize

L=

I X

log[P r(xi |λ1...6 )] + log[P r(λ1...6 )],

i=1

and we wish to do this with the constraint that of

L=

6 X

P

m λm = 1.

This gives a final criterion

" "K ## 6 6 X X X Nk log[λk ]+log Γ αk − log [Γ[αk ]]+ (αk − 1) log[λk ]+ν

k=1

m=1

k=1

m=1

6 X

! λk −1 ,

k=1

where ν is a Lagrange multiplier that is used to enforce this constraint.

Taking the derivative with respect to λk we get Nk α−1 ∂L = + + ν, ∂λk λk λk Setting this to zero and re-arranging we have Nk + αk − 1 + νλk = 0.

Now if we sum the six equations involving λ1 . . . λk together we get 6 X

(Nm + αm − 1) + ν

m=1

M X m=1

λm

=

6 X

(Nm + αm − 1) + ν = 0,

m=1

P6 where P we have used the constraint k=1 λk = 1. From this we can establish that ν = − m (Nm + αm − 1). Substituting this back into the original derivative we have 6 X Nk αk − 1 ∂L = + − (Nm + αm − 1). ∂λk λk λk m=1

Setting this to zero and solving for λk yields the desired relation Nk + αk − 1 . m=1 (Nm + αm − 1)

λ̂k = P6

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

25 Problem 4.5 The denominator of Bayes’ rule

P r(x1...I ) =

Z Y I

P r(xi |θ)P r(θ) dθ

i=1

is known as the evidence. It is a measure of how well the distribution fits regardless of the particular values of the parameters. Find an expression for the evidence term for (i) the normal distribution and (ii) the categorical distribution assuming conjugate priors in each case.

Answer i) Using the result from problem 4.11, the term in the integral is I Y

Normxi [µ, σ 2 ] · NormInvGamµ,σ2 [α, β, γ, δ] = κ · NormInvGamµ,σ2 [α̃, β̃, γ̃, δ̃],

i=1

where

κ

=

α̃

=

β̃

=

γ̃

=

δ̃

=

√ α γβ Γ[α̃] 1 √ (2π)I/2 γ̃ β̃ α̃ Γ[α] α + I/2 P 2 P (γδ + i xi )2 γδ 2 i xi +β+ − 2 2 2(γ + I) γ+I P (γδ + i xi ) . γ+I

Clearly this has the form of a constant multiplied by a probability distribution. The probability distribution integrates to one, and so when we apply the integral, we are just left with the constant, so P r(x1...I ) = κ =

√ α γβ 1 √ (2π)I/2 γ̃ β̃ α̃

ii) Using the result of problem 4.10, the term in the integral is I Y

Catxi [λ1...K ] · Dirλ1...K [α1...K ] = κ · Dirλ1...K [α̃1...K ],

i=1

where

κ̃

=

P QK Γ[ K j=1 αj ] j=1 Γ[αj + Nj ] .

P QK Γ[I + K α ] j=1 j j=1 Γ[αj ]

α̃1...K

=

[α1 + N1 , α2 + N2 , . . . , αK + NK ].

By the same logic as in part (i) of this question, the integral is just equal to the constant so we have Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

26

4

P r(x1...I ) = κ =

Fitting probability models

P QK Γ[ K j=1 αj ] j=1 Γ[αj + Nj ] . QK PK Γ[I + j=1 αj ] j=1 Γ[αj ]

Problem 4.6 The evidence term can be used to compare models. Consider two sets of data S1 = {0.1, −0.5, 0.2, 0.7} and S2 = {1.1, 2.0, 1.4, 2.3}. Let us pose the question of whether these two data sets came from the same normal distribution or from two different normal distributions.

Let model M1 denote the case where all of the data comes from the one normal distribution. The evidence for this model is Z

Y

P r(S1 ∪ S2 |M1 ) =

P r(xi |θ)P r(θ) dθ,

i∈S1 ∪S2

where θ = {µ, σ 2 } contains the parameters of this normal distribution. Similarly, we will let M2 denote the case where the two sets of data belong to different normal distributions P r(S1 ∪ S2 |M2 ) =

Z Y

P r(xi |θ 1 )P r(θ 1 ) dθ 1

i∈S1

Z Y

P r(xi |θ 2 )P r(θ 2 ) dθ 2 ,

i∈S2

where θ 1 = {µ1 , σ12 } and θ 2 = {µ2 , σ22 }.

Now it is possible to compare the probability of the data under each of these two models using Bayes’ rule P r(S1 ∪ S2 |M1 )P r(M1 ) n=1 P r(S1 ∪ S2 |Mn )P r(Mn )

P r(M1 |S1 ∪ S2 ) = P2

Use this expression to compute the posterior probability that the two datasets came from the same underlying normal distribution. You may assume normal-scaled inverse gamma priors over θ, θ 1 , and θ 2 with parameters α = 1, β = 1, γ = 1, δ = 0.

Answer From Problem 4.5 part one, we have that P r(x1...I ) = κ =

√ α γβ 1 √ (2π)I/2 γ̃ β̃ α̃

where

α̃

=

β̃

=

γ̃

=

α + I/2 P 2 P (γδ + i xi )2 γδ 2 i xi +β+ − 2 2 2(γ + I) γ+I

Now we simple compute this expression for the data S1 , S2 and S1 ∪ S2 and substitute into Bayes’ rule.

The resulting probabilities are Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

27

P r(S1 )

=

4.4 × 10−3

P r(S2 )

=

6.4 × 10−4

P r(S1 ∪ S1 )

=

9.7 ∗ 10−8

and the posterior probability if the two sets of data coming from the same normal is P r(M1 |S1 ∪ S2 ) = 0.0344,

Problem 4.7 In the Bernoulli distribution, the likelihood P r(x1...I |λ) of the parameter λ under the data {xi }Ii=1 where xi ∈ {0, 1} is P r(x1...I |λ) =

I Y

λxi (1 − λ)1−xi .

i=1

Find an expression for the maximum likelihood estimate of the parameter λ.

Answer To find the maximum likelihood solution, we write an expression for the log likelihood, then take the derivative with respect to the parameters λ and set the resulting expression to zero. The log likelihood in this case is

L=

I X

xi log[λ] +

i=1

I X (1 − xi ) log[1 − λ]. i=1

The derivative with respect to λ is I I X ∂L xi X 1 − xi = − = 0. ∂λ λ 1−λ i=1 i=1

We now multiply by both λ and (1 − λ) to remove the quantities from the denominators which gives us (1 − λ)

I X i=1

xi − λ

I X (1 − xi ) = 0. i=1

This can easily be re-arranged to give the answer PI PI xi i=1 xi λ̂ = Pi=1 = , I I i=1 1 which has the natural interpretation of being the number of times that a ’1’ was seen in the data.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

28

4

Fitting probability models

Problem 4.8 Find an expression for the MAP estimate of the Bernoulli parameter λ (see problem 4.7) assuming a beta distributed prior P r(λ) = Betaλ [α, β].

Answer In the MAP approach, we maximize the criterion L=

I X

log[P r(xi |λ)] + log[P r(λ)]

i=1

Substituting in the Bernoulli likelihood and Beta terms we have

L

=

I X

xi log[λ] +

i=1

I X

(1 − xi ) log[1 − λ] + log[P r(λ)] +

i=1

log[Γ[α + β]] − log[Γ[α]] − log[Γ[β]] + (α − 1) log[λ] + (β − 1) log(1 − λ) Taking the derivative of this expression with respect to λ we get I I X xi + α − 1 X 1 − xi + β − 1 ∂L = − = 0. ∂λ λ 1−λ i=1 i=1

We now multiply by both λ and (1 − λ) to remove the quantities from the denominators which gives us (1 − λ)

I X

! (xi ) + α − 1

I X −λ (1 − xi ) + β − 1

i=1

! = 0.

i=1

This can easily be re-arranged to give the answer PI λ̂ =

i=1 (xi ) + α − 1 .

I +α+β−2

Problem 4.9 Now consider the Bayesian approach to fitting Bernoulli data, using a beta distributed prior. Find expressions for (i) the posterior probability distribution over the Bernoulli parameters given observed data {xi }Ii=1 and (ii) the predictive distribution for new data x∗ .

Answer (i) We compute the posterior distribution over the Bernoulli distribution using Bayes’ rule.

QI P r(λ|x1...I

=

i=1 P r(xi |λ)P r(λ)

P r(x1...I ) QI

=

i=1 Bernxi [λ]Betaλ [α, β]

P r(x1...I )

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

29 Using the result from problem 3.9, the numerator of this expression is given by I Y

Bernxi [λ] · Betaλ [α, β] = κ · Betaλ [α̃, β̃],

i=1

where

κ

=

α̃

=

β̃

=

P P Γ[α + β]Γ[α + xi ]Γ[β + (1 − xi )] Γ[α + β + I]Γ[α]Γ[β] X α+ xi X β+ (1 − xi ).

So, the numerator is proportional to a Beta distribution. However, the posterior probability distribution must be a valid distribution itself, so it must be that the constant κ and the denominator P r(x1...I ) exactly cancel out and it follows that P r(λ|x1...I ) = Betaλ [α̃, β̃]. (ii) The predictive distribution for a new data example x∗ is calculated as P r(x∗ ) =

Z

P r(x∗ |λ)P r(λ|x1...I )dλ.

Using the conjugacy relation again, we have that

∗

P r(x )

Z Bernx∗ [λ] · Betaλ [α̃, β̃]dλ

= Z = =

κ0 · Betaλ [ᾰ, β̆]dλ

κ0

where we have used the result that the integral of a probability distribution is one and so only the constant remains:

P r(x∗ ) = κ0 =

Γ[α̃ + β̃]Γ[α̃ + x∗ ]Γ[β̃ + 1 − x∗ ] Γ[α̃ + β̃ + 1]Γ[α̃]Γ[β̃]

Problem 4.10 Staying with the Bernoulli distribution, consider observing data 0, 0, 0, 0 from four trials. Assuming a uniform beta prior (α = 1, β = 1), compute the predictive distribution using the (i) maximum likelihood, (ii) maximum a posteriori and (iii) Bayesian approaches. Comment on the results.

Answer i) The maximum likelihood solution is PI λ̂ =

i=1 (xi )

I

=0

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

30

4

Fitting probability models

and so the predictive distribution is P r(x∗ ) = Bernx∗ [0].

In other words the maximum likelihood solution predicts zero with certainty. ii) The MAP solution is PI λ̂ =

i=1 (xi ) + α − 1 =0 I +α+β−2

and so the predictive distribution is P r(x∗ ) = Bernx∗ [0].

In other words the MAP solution also predicts zero with certainty. iii) In the Bayesian solution we first compute the posterior probability distribution over λ based on the four observations. Using the result of the previous question we have P r(λ|x1...4 ) = Betaλ [α, β + 4].

Then we compute the predictive distribution which is equal to

P r(x∗ )

= = =

Γ[α + β + 4]Γ[α + x∗ ]Γ[β + 5 − x∗ ] Γ[α + β + 5]Γ[α]Γ[β + 4] Γ[6]Γ[1 + x∗ ]Γ[6 − x∗ ] Γ[7]Γ[1]Γ[5] 5!x∗ !(6 − x∗ )!

6!0!4!

Evaluating this for x∗ = 0 and x∗ = 1 we see that P r(x∗ = 0)

=

0.83

P r(x∗ = 1)

=

0.17.

This is clearly equivalent to P r(x∗ ) = Bernx∗ [0.17].

The Bayesian prediction is more moderate - it accounts for the possibility that a ’1’ will be seen in the future. This make intuitive sense, if we flip a coin 4 times and see tails every time, it is still not quite possible that we will see a head in the future.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

Chapter 5

The normal distribution Problem 5.1 Consider a multivariate normal distribution in variable x with mean µ and covariance Σ. Show that if we make the linear transformation y = Ax + b then the transformed variable y is distributed as: h i P r(y) = Normy Aµ + b, AΣAT .

Answer The multivariate normal distribution is defined as P r(x) =

1 exp −0.5 (x − µ) Σ−1 (x − µ) (2π)D |Σ|1/2

To transform the distribution as y = Ax + b, we substitute in x = A−1 (y − b) and divide by the determinant of the Jacobian of the transformation, which for this case is just |A| giving: P r(x) =

1 exp −0.5 A−1 (y − b) − µ Σ−1 A−1 (y − b) − µ (2π)D |Σ|1/2 |A|

We’ll first work with the constant κ from outside the exponential to show that it is the constant for a distribution with variance AΣAT . We have

κ

= = = = =

1 (2π)D |Σ|1/2 |A| 1 (2π)D |Σ|1/2 |A|1/2 |A|1/2 1 (2π)D |Σ|1/2 |A|1/2 |AT |1/2 1 (2π)D |A|1/2 |Σ|1/2 |AT |1/2 1 (2π)D |AΣAT |1/2

which is exactly the required constant.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

32

5

The normal distribution

Now we’ll work with the quadratic in the exponential term to show that it corresponds to a normal distribution in y with variance AΣAT and mean Aµ + b.

A−1 (y − b) − µ Σ−1 A−1 (y − b) − µ = yT A−T Σ−1 A−1 y − 2(bT A−T + µT )Σ−1 A−1 y + (A−1 b + µ)T Σ−1 (A−1 b + µ) = yT A−T Σ−1 A−1 y − 2(bT + µT AT )A−T Σ−1 A−1 y +(A−1 b + Aµ)T A−T Σ−1 A−1 (b + Aµ)

−1 = (y − (Aµ + b))T AΣAT (y − (Aµ + b)) This is clearly the quadratic term from a normal distribution in y with variance AΣAT and mean Aµ + b as required.

Problem 5.2 Show that we can convert a normal distribution with mean µ and covariance Σ to a new distribution with mean 0 and covariance I using the linear transformation y = Ax + b where

A

=

Σ−1/2

b

=

−Σ−1/2 µ.

This is known as the whitening transform.

Answer Using the transformation relation:

P r(y)

=

h i Normy Aµ + b, AΣAT h i Normy Σ−1/2 µ − Σ−1/2 µ, Σ−1/2 ΣΣ−1/2

=

Normy [0, I]

=

as required.

Problem 5.3 Show that for multivariate normal distribution

x1 µ1 Σ11 P r(x) = P r = Normx , x2 µ2 Σ21

ΣT21 Σ22

,

the marginal distribution in x1 is

P r(x1 )

=

Normx1 [µ1 , Σ11 ] .

Hint: apply the transformation y = [I, 0]x.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

33

Answer Using the transformation rule: h i P r(y) = Normy Aµ + b, AΣAT , with b = 0 and A = [I, 0], we get

P r(x1 )

=

µ Σ11 Normx1 [I, 0] 1 + 0, [I, 0] µ2 Σ21

=

Normx1 [µ1 , Σ11 ]

ΣT21 [I, 0]T Σ22

as required.

Problem 5.4 The Schur complement identity states that inverse of a matrix in terms of its sub-blocks is

A C

B D

−1

=

(A − BD−1 C)−1 −D C(A − BD−1 C)−1 −1

D

−1

−(A − BD−1 C)−1 BD−1 −1 −1 −1 −1 . + D C(A − BD C) BD

Show that this relation is true.

Answer To show that this is true, we simply multiply the Schur matrix by the original matrix and show that the product equals the identity. In other words we compute

V A = X C

U W

B D

(A − BD−1 C)−1 −1 −D C(A − BD−1 C)−1

−(A − BD−1 C)−1 BD−1 .

D−1 + D−1 C(A − BD−1 C)−1 BD−1

Now we need to show that U = I, V = 0, W = 0 and X = I, so that the left hand side as a whole is the identity matrix. Solving for U we have

U

=

A(A − BD−1 C)−1 + B − D−1 C(A − BD−1 C)−1

=

(A + B − D−1 C)(A − BD−1 C)−1

=

I,

as required. Solving for V we have

V

=

A(−(A − BD−1 C)−1 BD−1 ) + B D−1 + D−1 C(A − BD−1 C)−1 BD−1

=

BD−1 − (A − BD−1 C)(A − BD−1 C)−1 BD−1

=

BD−1 − BD−1

=

0,

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

34

5

The normal distribution

as required. Solving for W we get

W

=

C(A − BD−1 C)−1 + D(−D−1 C(A − BD−1 C)−1 )

=

C(A − BD−1 C)−1 + −C(A − BD−1 C)−1 )

=

0,

as required. Solving for X we get

X

=

C(−(A − BD−1 C)−1 BD−1 ) + D(D−1 + D−1 C(A − BD−1 C)−1 BD−1 )

=

−C(A − BD−1 C)−1 BD−1 + I + C(A − BD−1 C)−1 BD−1 )

=

I,

as required.

Problem 5.5 Prove the conditional distribution property for the normal distribution: if

x1 µ1 Σ11 ΣT12 , P r(x) = P r = Normx , x2 µ2 Σ12 Σ22 then h i T −1 P r(x1 |x2 ) = Normx1 µ1 + ΣT12 Σ−1 22 (x2 − µ2 ), Σ11 − Σ12 Σ22 Σ12 .

Answer We can write out the joint distribution as " P r(x) = κ1 exp −0.5

T x1 µ U − 1 x2 µ2 W

V X

# x1 µ − 1 x2 µ2

where κ1 is the standard constant associated with the normal distribution and

U W

−1 V Σ11 ΣT12 = X Σ12 Σ22

−1 (Σ11− ΣT12 Σ−1 22 Σ12 ) = −1 −1 T −Σ22 Σ12 (Σ11 −Σ12 Σ22 Σ12 )−1

−1 T −(Σ11 −ΣT12 Σ−1 Σ12 Σ−1 22 Σ12 ) 22 . −1 −1 T −1 T Σ−1 Σ12 Σ−1 22 +Σ22 C(Σ11 −Σ12 Σ22 Σ12 ) 22

Now we exploit Bayes’ rule P r(x1 |x2 ) = P r(x1 , x2 )/P r(x2 ) and note that P r(x1 |x2 ) ∝ P r(x1 , x2 ). We hence take the approach of reformulating the joint distribution in terms of a normal distribution in x1 alone. Multiplying out the matrices in the exponent, we get

P r(x)

=

h

i κ2 exp −0.5 (x1 − µ1 )T U(x1 − µ1 ) + 2(x1 − µ1 )T V(x2 − µ2 )

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

35 where we have subsumed the terms that do not depend on x1 to create a new constant κ2 . Now we both add and subtract a term to complete the square,

P r(x)

=

h

κ3 exp −0.5 (x1 −µ1 )T U(x1 −µ1 )+2(x1 −µ1 )T V(x2 −µ2 ) i +(x2 −µ2 )T VT U−1 V(x2 −µ2 )

where we have subsumed the term we must subtract in the exponential to balance the one we have added into the new constant κ3 . Re-factoring, we see that this is a normal distribution

P r(x)

=

h

i κ3 exp −0.5 (x1 − µ1 + U−1 V(x2 −µ2 ))T U(x1 − µ1 + U−1 V(x2 −µ2 ))

The conditional distribution P r(x1 |x2 ) is proportional to this expression, and the constant of proportionality must be 1/(κ3 |U|1/2 2π D/2 ) so that it is a proper normal distribution.

Hence the conditional distribution is given by

P r(x1 |x2 )

= =

Normx1 µ1 − U−1 V(x2 −µ2 ), U−1 h i T −1 Normx1 µ1 + ΣT12 Σ−1 22 (x2 − µ2 ), Σ11 − Σ12 Σ22 Σ12 .

Problem 5.6 Use the conditional probability relation for the normal distribution to show that the conditional distribution P r(x1 |x2 = k) is the same for all k when the covariance is diagonal and the variables are independent (see figure 5.5b).

Answer The conditional probability relation is: h i T −1 P r(x1 |x2 ) = Normx1 µ1 + ΣT12 Σ−1 22 (x2 − µ2 ), Σ11 − Σ12 Σ22 Σ12 .

When the covariance is diagonal Σ12 which contains the off-diagonal elements relating x1 and x2 will be zero. Hence, in this case, we have P r(x1 |x2 ) = Normx1 [µ1 , Σ11 ] , and the distribution is the same regardless of the value of x2 .

Problem 5.7 Show that Normx [a, A]Normx [b, B] ∝ Normx [(A−1 + B−1 )−1 (A−1 a + B−1 b), (A−1 + B−1 )−1 ].

Answer Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

36

5

The normal distribution

We first write out the expression for the product of the normals: h i Normx [a, A] Normx [b, B] = κ1 exp −0.5(x − a)T A−1 (x − a) − 0.5(x − b)T B−1 (x − b) h i = κ2 exp −0.5(xT (A−1 + B−1 )x + 2(A−1 a + B−1 b)x) , where we have multiplied out the terms in the exponential and subsumed the terms that are constant in x into a new constant κ2 .

Now we complete the square by multiplying and dividing by a new term, creating a new constant κ3 : h i Normx [a, A] Normx [b, B] = κ2 exp −0.5(xT (A−1 +B−1 )x+2(A−1 a+B−1 b)x) h = κ3 exp −0.5(xT (A−1 +B−1 )x+2(A−1 a+B−1 b)x i +(A−1 a+B−1 b)T (A−1 + B−1 )−1 (A−1 a + B−1 b)) .

Finally, we re-factor the exponential term into the form of a standard normal distribution, and divide and multiply by the associated constant term for this new normal distribution, creating a new constant κ4 : h T

Normx [a, A] Normx [b, B] = κ3 exp −0.5 x−(A−1 +B−1 )−1 (A−1 a+B−1 b) A−1 +B−1

x − (A−1 +B−1 )−1 (A−1 a+B−1 b)

= κ4 Normx (A−1 +B−1 )−1 (A−1 a+B−1 b), (A−1 +B−1 )−1 .

Problem 5.8 For the 1D case, show that when we take the product of the two normal distributions with means µ1 , µ2 and variances σ12 , σ22 , the new mean lies between the original two means and the new variance is smaller than either of the original variances.

Answer For the 1D case, the mean is given by

µ=

σ1−2 µ1 + σ2−2 σ1−1 + σ2−2

This has the form µ = aµ1 + bµ2 where a > 0, b > 0 and a + b = 1 and so the new mean must be somewhere between the original two means.

The new variance is given by

σ2 =

1 σ1−1 + σ2−2

To show that the new variance σ 2 is smaller than σ12 we start with the uncontroversial proposition Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

37

σ1−2 + σ2−2 > σ1−2 , where this must be true because all the terms are inverse variances and so are guaranteed to be positive. Now we take the reciprocal of both sides of the equation to yield 1 σ1−2 + σ2−2

< σ12 ,

and since the left hand side is the equal to the new variance σ 2 we conclude that σ 2 < σ12 .

A similar argument produces the symmetric result σ 2 < σ22 .

Problem 5.9 Show that the constant of proportionality κ in the product relation in problem 5.7 is also a normal distribution where

κ = Norma [b, A + B].

Answer Using the answer from problem 5.7 for reference, we will compute each constant in turn.

κ1

=

κ2

=

κ3

=

1 (2π)D |A|0.5 |B|0.5 h i κ1 · exp −0.5aT A−1 a − 0.5bT B−1 b h i κ2 · exp 0.5(A−1 a+B−1 b)T (A−1 + B−1 )−1 (A−1 a + B−1 b)

κ4

=

κ3 · 2π D/2 |(A−1 + B−1 )−1 |0.5

Working first with the constant, we have

κ

= = =

1 · 2π D/2 |(A−1 + B−1 )−1 |0.5 exp[•] (2π)D |A|0.5 |B|0.5 1 exp[•] 2π D/2 |A|0.5 ||A−1 + B−1 |0.5 |B|0.5 1 exp[•] 2π D/2 |A + B|0.5

which is the appropriate constant for the normal distribution.

Now let’s work on the exponential term. We have Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

38

5

The normal distribution

h

i κ ∝ exp −0.5 aT A−1 a + bT B−1 b − (A−1 a+B−1 b)T (A−1 + B−1 )−1 (A−1 a + B−1 b) h

∝ exp −0.5 aT A−1 a + bT B−1 b − aT A−1 (A−1 + B−1 )−1 A−1 a i −bT B−1 (A−1 + B−1 )−1 B−1 b − 2aT A−1 (A−1 + B−1 )−1 B−1 b h

∝ exp −0.5 aT A−1 a + bT B−1 b − aT A−1 (A−1 + B−1 )−1 A−1 a i −bT B−1 (A−1 + B−1 )−1 B−1 b − 2aT (A + B)−1 b h

∝ exp −0.5 aT A−1 a + bT B−1 b − aT (A + B)−1 BA−1 a i −bT (A + B)−1 AB−1 b − 2aT (A + B)−1 b Now we use the identities aT A−1 a

=

aT (A + B)−1 (A + B)A−1 a

=

aT (A + B)−1 a + aT (A + B)−1 BA−1 a

=

bT (A + B)−1 (A + B)B−1 b

=

bT (A + B)−1 b + bT (A + B)−1 AB−1 b

and bT B−1 b

Substituting these into the main equation we get h

κ ∝ exp −0.5 aT A−1 a + bT B−1 b − aT (A + B)−1 BA−1 a i −bT (A + B)−1 AB−1 b − 2aT (A + B)−1 b h

i ∝ exp −0.5 aT (A + B)−1 a + bT (A + B)−1 b − 2aT (A + B)−1 b h i ∝ exp −0.5(a − b)T (A + B)−1 (a − b) Putting this together with the constant term we have κ = Norma [b, A + B]. as required.

Problem 5.10 Prove the change of variable relation. Show that Normx [Ay + b, Σ] = κ · Normy [A0 x + b0 , Σ0 ], and derive expressions for κ, A0 , b0 and Σ0 . Hint: write out the terms in the original exponential, extract quadratic and linear terms in y, and complete the square.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

39

Answer h

i Normx [Ay + b, Σ] = κ1 exp −0.5 (x − b − Ay)T Σ−1 (x − b − Ay) h

i = κ2 exp −0.5 yT AT Σ−1 Ay − 2yT AT Σ−1 (x − b) where we have expanded the quadratic term and absorbed the term that does not depend on y into a new constant κ3 . Now we complete the square to make a new distribution in the variable y h

i Normx [Ay + b, Σ] = κ2 exp −0.5 yT AT Σ−1 Ay − 2yT AT Σ−1 (x − b) h

= κ3 exp −0.5 yT AT Σ−1 Ay − 2yT AT Σ−1 (x − b) −2(x − b)T Σ−1 AT (AT Σ−1 A)−1 AΣ−1 (x − b)

T = κ3 exp −0.5 y − (AT Σ−1 A)−1 AΣ−1 (x − b) (AT Σ−1 A)

i y − (AT Σ−1 A)−1 AΣ−1 (x − b) =

i

κ4 Normy [(AT Σ−1 A)−1 AΣ−1 (x − b), (AT Σ−1 A)−1 ]

as required.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

40

5

The normal distribution

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

Chapter 6

Learning and inference in vision Problem 6.1 Consider the following problems. i Determining the gender of an image of a face. ii Determining the pose of the human body given an image of the body. iii Determining which suit a playing card belongs to based on an image of that card. iv Determining whether two images of faces match (face verification). v Determining the 3D position of a point given the positions to which it projects in two cameras at different positions in the world (stereo reconstruction).

For each case, try to describe the contents of the world state w and the data x. Is each discrete or continuous? If discrete, then how many values can it take? Which are regression problems and which are classification problems?

Answer i) The world state is a binary variable w ∈ {0, 1} that is zero when the face is male and one when it is female. The data x consists of the RGB image values. Since the world is discrete, this is a classification problem. ii) The world state is a continuous variable w containing the joint angles. The data x consists of the RGB image values. Since the world is continuous, this is a regression problem. iii) The world state is a discrete variable w ∈ {1, 2, 3, 4} where each possible value of the world variable represents one of the suits. The data x consists of the RGB image values.

Since the world is discrete, this is a classification problem. iv) The world state is a discrete variable w ∈ {0, 1} that is zero when the faces don’t match and one when they do match. The data x consists of the RGB values from the two images. Since the world state is discrete, this is a classification problem. v) The world state is a continuous variables w = [u, v, w]T containing the position (u, v, w) in the world. The data x consists of the positions (x1 , y1 ) and (x2 , y2 ) where the points appear in the two images. Since the world state is continuous, this is a regression problem.

Problem 6.2 Describe a classifier that relates univariate discrete data x ∈ {1 . . . K} to a univariate discrete world state w ∈ {1 . . . M } for both discriminative and generative model types.

Answer Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

42

6

Learning and inference in vision

For the discriminative case, we choose a distribution over the world state and make its parameters a function of the data. The world state is discrete and multivalued and so a suitable choice is the categorical distribution. This has a M × 1 parameter vector λ which contains the probability of observing each of the M possible states.. Since the data is also discrete, we will have a different set of parameters λk for each possible observation x and the model becomes: P r(w = m|x = k) = Catw [λk ] .

For the discriminative case, we choose a distribution over the data and make its parameters a function of the world state. By symmetry, an appropriate model is P r(x = k|w = m) = Catx [λm ] .

Problem 6.3 Describe a regression model that relates univariate binary discrete data x ∈ {0, 1} to a univariate continuous world state w ∈ [−∞, ∞]. Use a generative formulation in which P r(x|w) and P r(w) are modeled.

Answer In the generative formulation we choose a distribution over the data and make its parameters depend on the world state. In this case, the data is binary and discrete so a Bernoulli distribution is appropriate. The Bernoulli has a single parameters λ ∈ [0, 1].

Hence, we form a linear function φ0 + φ1 w of the world state and pass this through the logistic sigmoid to ensure that it obeys the constraint 0 ≤ λ ≤ 1. The resulting likelihood is

P r(x|w) = Bernx

1 .

1 + exp[−φ0 − φ1 w]

Since the world state is continuous and defined on an infinite domain, a suitable prior would be the univariate normal distribution P r(w) = Normw [µp , σp2 ].

Note the symmetry of this model with logistic regression, where the data x was continuous, but the world state w discrete, but we modelled P r(w|x) (i.e., it was a discriminative model, not generative).

Problem 6.4 Describe a discriminative regression model that relates a continuous world state w ∈ [0, 1] to univariate continuous data x ∈ [−∞, ∞]. Hint: Base your classifier on the Beta distribution. Ensure that the constraints on the parameters are obeyed.

Answer In a discriminative model we choose a probability distribution over the world state and make its parameters a function of the data. In this case, the world state is continuous and lies in the range w ∈ [0, 1] so a suitable distribution is the beta distribution. This has two parameters α and β each of which must be non-negative. To make these two parameters Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

43 functions of the data, we could model them as two linear functions of the data, which are subsequently passed through an exponential function to ensure positivity. The complete model is hence P r(w|x) = Betaw [exp[φ0 + φ1 x], exp[ψ0 + ψ1 x]] .

Problem 6.5 Find expressions for the maximum likelihood estimates of the parameters in the discriminative linear regression model (section 6.3.1). In other words find the parameters {φ0 , φ1 , σ 2 } that satisfy

φ̂0 , φ̂1 , σ̂

2

=

argmax

" I Y

φ0 ,φ1 ,σ 2

=

argmax

argmax

P r(wi |xi , φ0 , φ1 , σ )

i=1

" I X

φ0 ,φ1 ,σ 2

=

# 2

#

log P r(wi |xi , φ0 , φ1 , σ 2 )

i=1

" I X

φ0 ,φ1 ,σ 2

#

log Normw φ0 + φ1 x, σ 2 ,

i=1

where {wi , xi }Ii=1 are paired training examples.

Answer The log likelihood L that we wish to maximize can be written out as I X (wi − φ0 − φ1 xi )2 I I L = − log[σ 2 ] − log[2π] − .

2 2 2σ 2 i=1

To maximize this function we take the derivative with respect to each parameters and set the resulting equation to zero. This gives three equations to solve for the three parameters.

We’ll start with the offset parameters φ0 for which the associated derivative is: I X ∂L w i − φ 0 − φ 1 xi = , ∂φ0 σ2 i=1

which can be re-arranged to give the relation P φ0 =

i w i − φ1

I

P

i xi

= µw − φ 1 µx ,

where µw and µx are the mean of the world states and data point respectively.

Substituting back into the original equation we have I X (wi − µw + µx − φ1 xi )2 I I .

L = − log[σ 2 ] − log[2π] − 2 2 2σ 2 i=1

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

44

6

Learning and inference in vision

Taking the derivative with respect to φ1 we get I X ∂L wi − µw + µx − φ1 xi = . xi ∂φ0 σ2 i=1

We now set this equations to zero and solve for φ1 which gives us the relation PI φ1 =

i=1 xi (wi − µw + µx ) PI 2 i=1 xi

Finally, we take the derivative of the log likelihood with respect to σ 2 which gives I X (wi − φ0 − φ1 xi )2 ∂L I =− 2 + , 2 ∂σ 2σ 2σ 4 i=1

which can be re-arranged to get σ2 =

I X (wi − φ0 − φ1 xi )2

I

i=1

Collecting these results together we have: PI φ1

=

φ0

=

σ2

=

i=1 xi (wi − µw + µx ) PI 2 i=1 xi

= µw − φ 1 µx I X (wi − φ0 − φ1 xi )2

I

i=1

.

Problem 6.6 Consider a regression model that models the joint probability P r(x, w) between the world w and the data x as

Pr

wi xi

= Norm[wi ,xi ]T

2 µw σ , ww 2 µx σxw

2 σxw σxx

.

Use the relation in section 5.5 to compute the posterior distribution P r(wi |xi ). Show that it has the form P r(wi |xi ) = Normwi [φ0 + φ1 x, σ 2 ], and compute expressions for φ0 and φ1 in terms of the training data {wi , xi }Ii=1 by substi2 2 2 tuting in explicit maximum likelihood estimates of the parameters {µw , µx , σww , σxw , σxx }.

Answer Using the relation from section 5.5, we have

2 −2 2 2 −2 2 P r(w|x = x∗ ) = Normw µw +σxw σxx (x∗ −µx ), σww −σxw σxx σxw Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

45 This clearly has the above form with

φ0

=

2 −2 µw −σxw σxx µx

φ1

=

2 −2 µw +σxw σxx

2

=

2 2 −2 2 σww −σxw σxx σxw

σ

The maximum likelihood solutions for the various quantities are PI µ̂w

i=1 xi

=

I PI

µ̂x 2 σ̂xx 2 σ̂ww 2 σ̂xw

i=1 wi

=

I PI

i=1 (xi − µx )

=

2

I PI

i=1 (wi − µw )

=

2

I (x i − µx )(wi − µw ) i=1 I

PI =

Substituting these expressions into the solutions for the parameters we get PI φ0

=

µw −

i=1 (xi − µx )(wi − µw )

PI

i=1 (xi − µx )

2

µx I I PI (xi − µx )(wi − µw ) i=1 (xi − µx )2 µw + i=1 I I PI PI PI 2 2 PI i=1 (wi − µw ) i=1 (xi − µx )(wi − µw ) i=1 (xi − µx ) i=1 (xi − µx )(wi − µw ) − I I I I PI

φ1

=

σ2

=

Problem 6.7 For a two-class problem, the decision boundary is the locus of world values w where the posterior probability P r(w = 1|x) is equal to 0.5. In other words, it represents the boundary between regions that would be classified as w = 0 and w = 1. Consider the generative classifier from section 6.4.2. Show that with equal priors P r(w = 0) = P r(w = 1) = 0.5 points on the decision boundary (the locus of points where P r(w = 0|x) = P r(w = 1|x)) obey a constraint of the form ax2 + bx + c = 0, where and {a, b, c} are scalars. Does the shape of the decision boundary for the logistic regression model from section 6.4.1have the same form?

Answer Using Bayes’ rule we have P r(w = 1|x) =

P r(x|w = 1)P r(w = 1) + P r(x|w = 1)P r(w = 1).

P r(x|w = 0)P r(w = 0

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

46

6

Learning and inference in vision

Substituting in P r(w = 1|x) = P r(w = 0) = P r(w = 1) = 0.5 we can re-arrange to get the relation P r(x|w = 0) = P r(x|w = 1) on the boundary. So, on the decision boundary Normx [µ0 , σ02 ] = Normx [µ1 , σ12 ] from which it follows that

log Normx [µ0 , σ02 ] = log[Normx µ1 , σ12 ] .

Substituting in the expressions for the log normals we have (x − µ0 )2 (x − µ1 )2 1 1 1 1 2 = − log[2π] − log[σ ] − − log[2π] − log[σ02 ] − 1 2 2 2σ02 2 2 2σ12 Cancelling out the first terms on each side and multiplying both sides by minus two gives: log[σ02 ] +

(x − µ0 )2 (x − µ1 )2 = log[σ12 ] + .

2 σ0 σ12

Taking the terms of the left hand side to the right hand side gives log[σ02 ] +

(x − µ0 )2 (x − µ1 )2 − log[σ12 ] − = 0, 2 σ0 σ12

which can then be simplified to

1 1 − 2 σ02 σ1

x2 − 2

µ0 µ0 − 2 σ02 σ0

x+

µ20 µ21 2 2 − + log[σ ] − log[σ ] = 0, 0 1 σ02 σ12

which clearly is of the form ax2 + bx + c = 0 as required.

Problem 6.8 Consider a generative classification model for 1D data with likelihood terms

P r(xi |wi = 0)

=

P r(xi |wi = 1)

=

Normxi 0, σ 2

Normxi 0, 1.5σ 2 .,

What is the decision boundary for this classifier with equal priors P r(w = 0) = P r(w = 1) = 0.5? Develop a discriminative classifier that can produce the same decision boundary.

Hint: base your classifier on a quadratic rather than a linear function.

Answer Using the result of problem 6.8, we have that the decision boundary is given by

1 1 − 2 σ02 σ1

x2 − 2

µ0 µ0 − 2 σ02 σ0

x+

µ20 µ2 − 12 + log[σ02 ] − log[σ12 ] = 0, 2 σ0 σ1

Substituting in our values for µ0 , µ1 , σ0 , σ1 we get the decision boundary

1 1 − σ2 1.5σ 2

x2 + log[σ 2 ] − log[1.5σ 2 ] = 0.

You can solve for the roots of this equations using the standard quadratic formula.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

47 This generative classification model has two roots - there are two different places where the posterior crosses 0.5, and these are symmetrically either side of the origin. The discriminative logistic regression model, can only produce sigmoid function which cross 0.5 in a single place. However, we can create a function that crosses 0.5 in two places by forming a quadratic function of the data and then passing this through the logistic sigmoid function. The full model would be

P r(w|x) = Bernw

1 .

1 + exp [−φ0 − φ1 x − φ2 x2 ]

Notice that we have managed to create a more complicated decision boundary here by passing the data x through a non-linear function to form the vector [x, x2 ] and then taken a linear sum of these terms. This trick is the basis for creating more complex classifiers.

Problem 6.9 Consider a generative binary classifier for multivariate data based on multivariate normal likelihood terms

P r(xi |wi = 0)

=

Normxi [µ0 , Σ0 ]

P r(xi |wi = 1)

=

Normxi [µ1 , Σ1 ]

and a discriminative classifier based on logistic regression for the same data h i P r(wi |xi ) = Bernwi sig[φ0 + φT xi ] . where there is one entry in the gradient vector φ for each entry of xi .

How many parameters does each model have as a function of the dimensionality of xi ?

What are the relative advantages and disadvantages of each model as the dimensionality increases?

Answer The generative classifier has D parameters for each mean and D(D + 1)/2 parameters for each covariance (recall that the covariance is symmetric). This gives a total of 2D + D(D + 1) parameters. The discriminative classifier has D + 1 parameters.

The generative classifier forms a quadratic decision boundary (see question 6.7). However, the discriminative classifier only creates a linear decision boundary.

In conclusion, the number of parameters in the generative classifier increases faster as a function of the dimensionality of the space but it produces a more complicated decision boundary and can hence correctly classify a larger family of data configurations.

Problem 6.10 One of the problems with the background subtraction method described is that it erroneously classifies shadows as foreground. Describe a model that could be used to classify pixels into three categories (foreground, background, and shadow).

Answer Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

48

6

Learning and inference in vision

The key thing to note is that in a shadow, a pixel maintains the same colour, but its intensity is dimmed. Hence, a reasonable way to model this situation is with a generative classifier with likelihoods for the nth pixel of

P r(xn |w = 0)

=

Normxn [µn0 , Σn0 ]

P r(xn |w = 1)

=

1/κ

P r(xn |w = 2)

=

Normxn [λµn0 , λ0 Σn0 ]

where the worlds states 0,1,and 2 denote background, foreground, and shadow regions respectively. The first two equations are exactly as described in the original background subtraction algorithm. The last equation explains the shadow region as being normally distributed with a mean that is a fraction λ < 1 of the background mean and a covariance that is a fraction λ0 of the background covariance. These likelihoods can be combined with suitable priors and the posterior over the world state can be computed using Bayes’ rule.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

Chapter 7

Modeling complex data densities Problem 7.1 Consider a computer vision system for machine inspection of oranges in which the goal is to tell if the orange is ripe. For each image we separate the orange from the background and calculate the average color of the pixels, which we describe as a 3 × 1 vector x. We are given training pairs {xi , wi } of these vectors, each with an associated binary variable w ∈ {0, 1} that indicates that this training example was unripe (w = 0) or ripe (w = 1). Describe how to build a generative classifier that could classify new examples x∗ as being ripe or unripe.

Answer We model the ripe and unripe likelihoods with normal distributions so that

P r(xi |wi = 0)

=

Normxi [µ0 , Σ0 ]

P r(xi |wi = 1)

=

Normxi [µ1 , Σ1 ]

We model the prior probability over the world state as a Bernoulli distribution P r(wi ) = Bernwi [λ].

We learn the unknown parameters of this system using the maximum likelihood method.

In particular, we learn µ0 and Σ0 from the subset of data where the world state is zero and µ1 and Σ1 from the subset of data where the world state is one. We learn the Bernoulli parameter λ from the world states {wi }Ii=1 .

In the inference stage, we compute the posterior probability P r(w∗ |x∗ ) for a new data examples x∗ using Bayes’ rule: P r(w∗ |x∗ ) =

P r(x∗ |w∗ )P r(w∗ ) .

P r(x∗ )

Problem 7.2 It turns out that a small subset of the training labels wi in the previous example were wrong. How could you modify your classifier to cope with this situation?

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

50

7

Modeling complex data densities

Answer One way to cope with this noise in the training data would be to replace the normal distribution with a robust probability model such as a the student t-distribution. This will be less affected by the outliers. The new likelihoods become

P r(xi |wi = 0)

=

Studxi [µ0 , Σ0 , ν0 ]

P r(xi |wi = 1)

=

Studxi [µ1 , Σ1 ν1 ]

Problem 7.3 Derive the M-step equations for the mixtures of Gaussians model (equation 7.19).

Answer From equation 7.18, the criterion that we wish to maximize is

L=

I X K X

rik log [λk Normxi [µk , Σk ]] + ν

i=1 k=1

K X

! λk − 1 ,

k=1

where ν is a Lagrange multiplier that has been added to enforce the constraint that the sum of the Gaussian weights {λk }K k=1 is one.

Substituting in the expression for the normal distribution we have

L

=

I X K X

(xi − µk )T Σ−1 D 1 k (xi − µk ) rik log[λk ] − log 2π − log[|Σk |] − 2 2 2 i=1 k=1 !

K X +ν λk − 1 , k=1

Now we take derivatives with respect to µk , making use of the relations in Appendix C.6: I X ∂L = rik Σ−1 k (xi − µk ). ∂µk i=1

Multiplying out the terms and setting the result to zero, we get the relation

Σ−1 k

I X i=1

rik xi − Σ−1 k µk

I X

rik = 0

i=1

We now multiply both sides by Σk and re-arrange to get PI rik xi µk = Pi=1 , I i=1 rik Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

51 as required.

Now we take derivatives with respect to Σk , again making use of the relations in Appendix C.6: I I X Σ−2 (xi − µk )(xi − µk )T ∂L 1X =− rik Σk −T + rik ∂Σk 2 i=1 2 i=1

Now we note that Σ = ΣT because the covariance is always symmetric, multiply through by Σ2 /2 and set the resulting expression to zero to get the result −

I X i=1

rik Σk +

I X

rik (xi − µk )(xi − µk )T = 0,

i=1

which can be re-arranged to give PI T i=1 rik (xi − µk )(xi − µk ) , Σk = PI i=1 rik as required.

Finally, we take the derivative with respect to λk to get I X rik ∂L = +ν ∂λk λk i=1

Setting this to zero and re-arranging, we get the relation I X

rik + νλk = 0

i=1

Now we sum all K equations of this type to get K X I X

rik + ν

k=1 i=1

Noting that

K X

λk = 0.

k=1

P

k λk = 1, we get an expression for ν:

ν=−

K X I X

rik .

k=1 i=1

Now we substitute this back into the original expression for the derivative with respect to λk to give I K I X ∂L rik X X = − rik . ∂λk λk i=1 i=1 k=1

Setting this to zero and re-arranging gives the relation PI rik λk = PK i=1 PI k=1

i=1 rik

,

as required.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

52

7

Modeling complex data densities

Problem 7.4 Consider modeling some univariate continuous visual data x ∈ [0, 1] using a mixture of beta distributions. Write down an equation for this model. Describe in words what will occur in (i) the E-step and (ii) the M-step.

Answer The equation for the mixture of beta distributions is

P r(x) =

K X

λk Betax [αk , βk ],

k=1

where αk and βk are the parameters of the kth component of the model and λk is the weight of the kth component, where the weights are constrained to be positive and sum to one. (i) In the E-Step, we find a posterior probability distribution over the hidden variable for each data point. This will determine the responsibility of each component for the data point. (ii) In the M-Step, we update the expected log likelihood with respect to the unknown parameters {λk , αk , βk }K k=1 .

Problem 7.5 Prove that the student t-distribution over x is the marginalization with respect to h of the joint distribution P r(x, h) between x and a hidden variable h where Studx [µ, σ 2 , ν] =

Z

Normx [µ, σ 2 /h]Gamh [ν/2, ν/2]dh.

Answer Substituting in the appropriate terms, we get:

Studx [µ, σ 2 , ν]

Z ∞ =

Normx [µ, σ 2 /h] · Gamh [ν/2, ν/2]dh

0

Z ∞

h ν i (x − µ)2 h1/2 ν ν/2 √ exp − h · ν/2 exp − h hν/2−1 dh 2 2σ 2 2 Γ[ν/2] 2πσ 2 0

Z ∞ 2 ν/2 (x − µ) ν ν √ exp − + h h(ν−1)/2 dh.

2 2σ 2 2ν/2 Γ[ν/2] 2πσ 2 0

= =

The integral in this expression has the form Z ∞

exp[−ah]hb dh =

0

∞ b! exp[−ah] = b+1 ab+1 a 0 b!

where this was solved by using integration by parts b times in a row.

Applying this result to our problem we get Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

53

Studx [µ, σ 2 , ν]

=

ν ν/2 ν √ ν/2 2 Γ 2 2πσ 2

=

ν ν/2 Γ[ ν+1 ] ν √2 ν/2 2 Γ 2 2πσ 2

=

=

(x − µ)2 ν + 2 2σ 2

−(ν+1)/2 ((ν − 1)/2)!

−(ν+1)/2 (x − µ)2 ν + 2 2σ 2 −(ν+1)/2 ν −(ν+1)/2 ν ν/2 Γ[ ν+1 ] (x − µ)2 1+ ν √2 νσ 2 2ν/2 Γ 2 2πσ 2 2

− ν+1 2 Γ ν+1 (x − µ)2 2 √ , ν 1 + 2 νσ νπσ 2 Γ

2

as required.

Problem 7.6 Show that the peak of the Gamma distribution Gamz [α, β] is at ẑ =

α−1 . β

Answer The equation for the Gamma distribution is Gamz [α, β] =

βα exp[−βz]z α−1 . Γ[α]

To find the peak, we can equivalently work with the log L of the Gamma distribution.

This has the peak in the same place as the original function because the logarithm is a monotonic transformation.

L = α log[β] − log[Γ[α]] − βz + (α − 1) log[z] Now we compute the derivative of this function with respect to the main variable z to get α−1 ∂L = −β + ∂z z We now set this expression to zero and solve for z which gives ẑ =

α−1 , β

as required.

Problem 7.7 Show that the Gamma distribution is conjugate to the inverse scaling factor of the variance in a normal distribution so that

Normxi [µ, Σ/hi ]Gamhi [ν/2, ν/2] =

ν ν/2 2ν/2 Γ[ν/2]

h ν i exp − h hν/2−1 , 2

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

54

7

Modeling complex data densities

and find the constant of proportionality κ and the new parameters α and β.

Answer Normxi [µ, Σ/hi ] · Gamhi [ν/2, ν/2] h ν i h i ν ν/2 hD/2 = exp − h hν/2−1 exp −0.5h(x − µ)T Σ−1 (x − µ) · D/2 1/2 Γ[ν/2] 2 (2π) |Σ|

ν + (x − µ)T Σ−1 (x − µ) 1 ν ν/2 = exp − h hν/2+D/2−1 2 (2π)D/2 |Σ|1/2 Γ[ν/2] This has the general form of a Gamma distribution but to make it complete we multiply and divide by the appropriate constants, which gives us Normxi [µ, Σ/hi ] · Gamhi [ν/2, ν/2]

=

κBetah [α̃, β̃],

where

κ

=

α̃

=

β̃

=

ν ν/2 Γ[α̃] (2π)D/2 |Σ|1/2 β̃ α̃ ν + (x − µ)T Σ−1 (x − µ) 2 ν/2 + D/2

Normxi [µ, Σ/hi ] · Gamhi [ν/2, ν/2] =

ν + (x − µ)T Σ−1 (x − µ) ν ν/2 Beta , h 2 (2π)D/2 |Σ|1/2 Γ[ν/2] 1

Problem 7.8 The model for factor analysis can be written as xi = µ + Φhi + i , where hi is distributed normally with mean zero and identity covariance and i is distributed normally with mean zero and covariance Σ. Determine expressions for 1. E[xi ], 2. E[(xi − E[xi ])E[xi ])T ].

Answer i) Applying the four expectation rules from Section 2.7, we have E[xi ]

=

E[µ] + E[Φhi ] + i

=

µ + ΦE[hi ] + 0

=

µ+Φ·0+0

=

µ.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

55 ii) Applying the expectation rules to the second case, we have

E[(xi − E[xi ])2 ]

=

h i E (µ + Φhi + i − µ)(µ + Φhi + i − µ)T h i E (Φhi + i )(Φhi + i )T h i E Φhi hTi ΦT + 2Φhi Ti + i i h i h i E Φhi hTi ΦT + E 2Φhi Ti + E [ i i ] h i h i ΦΦT E hi hTi + 2ΦE [hi ] E Ti + E [ i i ]

=

ΦΦT · I + 2Φ · 0 + Σ

=

ΦΦT + Σ

= = = =

So, factor analysis model has a mean of µ and a covariance of ΦΦT + Σ which is exactly what we claimed when we marginalized out the hidden variable:

Z P r(x)

P r(xi |hi )P r(hi )dhi

= Z = =

Normxi [µ + Φhi , Σ]Normhi [0, I]dhi Normxi [µ, ΦΦT + Σ].

Problem 7.9 Derive the E-step for factor analysis (equation 7.34).

Answer We begin by applying the change in variables relation (section 5.7) which gives us

q̂(hi )

∝

Normxi [µ + Φhi , Σ]Normhi [0, I]

=

Normhi [(ΦT Σ−1 Φ)−1 ΦT Σ−1 (xi − µ), (ΦT Σ−1 Φ)−1 ]Normhi [0, I]

Now we apply the product of two normal distributions rule (section 5.6) which gives us

q̂(hi )

∝

Normhi [(ΦT Σ−1 Φ)−1 ΦT Σ−1 (xi − µ), (ΦT Σ−1 Φ)−1 ]Normhi [0, I]

=

Normhi [(ΦT Σ−1 Φ + I)−1 ΦT Σ−1 (xi − µ), (ΦT Σ−1 Φ + I)−1 ]

as required. Note that the constant must be one to make the posterior a valid probability distribution.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

56

7

Modeling complex data densities

Problem 7.10 Derive the M-step for factor analysis (equation 7.38).

Answer The log likelihood is given by

L

=

=

# " I

X 1 T −1 − E 2D log 2π + 2 log[|Σ|] + (xi − µ − Φhi ) Σ (xi − µ − Φhi ) 2 i=1 " I X 1 − E 2D log 2π + 2 log[|Σ|] + µT Σ−1 µ + xTi Σ−1 xi + hi ΦT Σ−1 Φhi 2 i=1 i −2µT Σ−1 xi − 2µT Σ−1 Φhi − 2xTi Σ−1 Φhi

Taking the derivative with respect to µ we get

∂L ∂µ

" I # X

1 −1 −1 −1 − E 2Σ µ − 2Σ xi − 2Σ Φhi 2 i=1

=

I X

=

−Σ−1 µ + 2Σ−1 xi = 0

i=1

which we can re-arrange to get PI µ̂ =

i=1 xi

I Taking the derivative with respect to Φ we get

∂L ∂Φ

=

=

" I #

X −1 1 T −1 T −1 T − E 2Σ Φhi hi − 2Σ µhi − 2Σ xi hi 2 i=1 I X

2Σ−1 ΦE[hi hTi ] − 2Σ−1 µE[hTi ] − 2Σ−1 xi E[hTi ] = 0

i=1

This can be re-arranged to get

Φ̂ =

I X (xi − µ)E[hi ]T i=1

!

I X

!−1 E[hi hTi ]

i=1

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

Chapter 8

Regression models Problem 8.1 Consider a regression problem where the world state w is known to be positive. To cope with this we could construct a regression model in which the world state is modeled as a gamma distribution. We could constrain both parameters α, β of the gamma distribution to be the same so that α = β and make them a function of the data x. Describe a maximum likelihood approach to fitting this model.

Answer This model can be written as h i P r(w|x) = Gamw exp[φ0 + φT x], exp[φ0 + φT x] , where we have introduced the exponential function to ensure that the two parameters of the Gamma distribution are non-negative. To learn this model using the maximum likelihood approach, we use the criterion φˆ0 , φ̂ = argmax φ0 ,φ

" I X

h

h

T

T

log Gamwi exp[φ0 + φ xi ], exp[φ0 + φ xi ]

ii

# .

i=1

We could fit the parameters φ0 and φ using a non-linear optimization approach using this cost function.

Problem 8.2 Consider a robust regression problem based on the t-distribution rather than the normal distribution. Define this model precisely in mathematical terms and sketch out a maximum likelihood approach to fitting the parameters.

Answer The model is given by h i P r(w|x) = Studw φ0 + φT x, σ 2 , ν] , In the maximum likelihood approach, we would choose parameters using the criterion φˆ0 , φ̂, σˆ2 , ν̂ = argmax φ0 ,φ,σ 2 ,ν

" I X

h

h ii log Studwi φ0 + φT xi , σ 2 , ν]

#

i=1

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

58

8

Regression models

The parameters could be fit either by direct optimization using a non-linear algorithm such as the Gauss Newton approach, or by writing the t-distribution as a marginalization of a joint density with a hidden variable and using the EM algorithm.

Problem 8.3 Prove that the maximum likelihood solution for the gradient in the linear regression model is φ̂ = (XXT )−1 Xw. Answer To fit the parameters, we optimize the criterion

φ̂

=

h i argmax log[Normw [XT φ, σ 2 I φ

= = = = =

D D 1 argmax − log[σ 2 ] − log[2π] − 2 (w − XT φ)T (w − XT φ) 2 2 2σ φ

1 T T T argmax − 2 (w − X φ) (w − X φ) 2σ φ

1 argmax − 2 wT w + φT XXT φ − 2wXT φ 2σ φ

1 argmax − 2 φT XXT φ − 2wXT φ 2σ φ h i argmax −φT XXT φ + 2wXT φ φ

Denoting this criterion by L and then taking the derivative with respect to φ and setting the result to zero we get ∂L = −2XXT φ + 2Xw = 0 ∂φ Re-arranging this equation gives us the desired relation: φ̂ = (XXT )−1 Xw.

Problem 8.4 For the Bayesian linear regression model (section 8.2), show that the posterior distribution over the parameters φ is given by

1 −1 −1 P r(φ|X, w) = Normφ A Xw, A , σ2 where

A=

1 1 XXT + 2 I. σ2 σp

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

59

Answer The posterior is computed via Bayes rule so

P r(φ|X, w)

∝

P r(w|X, φ)P r(φ)

=

Normw [XT φ, σ 2 I]Normφ [0, σp2 I]

Using the change of variables relation from section 5.7 we see that Normφ [(σ −2 XXT )−1 Xσ −2 w, (σ −2 XXT )−1 ]Normφ [0, σp2 I]

∝

P r(φ|X, w)

Using the product of tow normals rule form section 5.6 we get

P r(φ|X, w)

∝

Normφ [(σ −2 XXT + σp−2 I)−1 Xσ −2 w, (σ −2 XXT )−1 ]

It is now easy to see that this has the desired form. Note that the constant of proportionality must be one since the posterior distribution must be a valid probability distribution.

Problem 8.5 For the Bayesian linear regression model (section 8.2) show that the predictive distribution for a new data example x∗ is given by

∗

∗

P r(w |x , X, w) = Norm

w∗

1 ∗T −1 ∗T −1 ∗ 2 x A Xw, x A x + σ . σ2

Answer

P r(w∗ |x∗ , X, w)

Z = =

P r(w∗ |x∗ , φ)P r(φ|X, w)dφ

Z 1 −1 −1 Normw∗ [φT x∗ , σ 2 ]Normφ A Xw, A dφ σ2

It is quite arduous to prove this the straightforward way, so we’ll take a shortcut. First, we note that the predictive distribution will be normal. The normal distribution is selfconjugate with respect to its own mean and after reformulating the two terms in the integral as a single distribution in φ, the remaining constant will also be normal.

Our goal now is to compute the mean and covariance of this normal. To do this, we’ll write both distributions from the original integral as generative equations: w∗

=

φ

=

x∗T φ + i 1 −1 A Xw + αi σ2

where i is a noise term with mean 0 and variance σ 2 and α is a second noise term with mean 0 and covariance A−1 .

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

60

8

Regression models

Now we compute the expected value of w∗ which is E[w∗ ]

=

x∗T E[φ] + E[ i ]

=

x∗T E[φ] 1 ∗T −1 x A Xw σ2

=

which is indeed the claimed mean of the predictive distribution.

Now we compute the second moment about zero E[w∗2 ]

=

x∗T E[φφT ]x∗ + E[ i Ti ]

=

x∗T E[φφT ]x∗ + σ 2

We note that E[φφT ]

= =

E[(φ − E[φ])T (φ − E[φ])] + E[φ]E[φT ] 1 A−1 + 4 A−1 XwwT XT A−1 σ

and so the covariance is given by E[(w − E[w])2 ]

=

E[w∗2 ] − E[w∗ ]2

=

x∗T E[φφT ]x∗ + σ 2 −

=

x∗T A−1 x∗ + σ 2 ,

1 ∗T −1 x A XwwT XT A−1 x∗ σ4

as required.

Problem 8.6 Use the matrix inversion lemma (appendix C.8.4) to show that

A−1 =

1 1 XXT + 2 ID σ2 σp

−1

−1 σ2 = σp2 ID − σp2 X XT X + 2 II XT . σp

Answer The matrix inversion lemma states that (F−1 + BT C−1 B)−1 = F − FBT (BFBT + C)−1 BF In this case, we have: B

=

XT

C

=

σ2 I

F

=

σp2 I

Substituting these values in gives the correct answer.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

61 Problem 8.7 Compute the derivative of the marginal likelihood

P r(w|X, σ 2 )

=

Normw [0, σp2 XT X + σ 2 I],

with respect to the variance parameter σ 2 .

Answer Writing out the expression in full, we have l=

1 (2π)D/2 |A|1/2

h i exp −0.5wT A−1 w

where A = σp2 XT X + σ 2 I. Taking the derivative we have

∂l ∂σ 2

= = = = =

h i ∂0.5wT A−1 w ∂|A| T −1 exp −0.5w A w ∂σ 2 2(2π)D/2 |A|3/2 ∂σ 2

h i ∂0.5wT A−1 w ∂A 1 tr A−1 2 exp −0.5wT A−1 w D/2 1/2 ∂σ ∂σ 2 2(2π) |A|

h i ∂A−1 1 ∂A tr A−1 2 exp −0.5wT A−1 w wwT ∂σ ∂σ 2 4(2π)D/2 |A|1/2

h i ∂A −1 ∂A tr A−1 2 exp −0.5wT A−1 w wwT A−1 2 A−1 ∂σ ∂σ 4(2π)D/2 |A|1/2 h i

−1 tr A−1 exp −0.5wT A−1 w wwT A−1 A−1 D/2 1/2 4(2π) |A| 1

Problem 8.8 Compute a closed form expression for the approximated t-distribution used to impose sparseness.

q(h) = max Normφ [0, h−1 ]Gamh [ν/2, ν/2] . h

Plot this function for ν = 2. Plot the 2D function [h1 , h2 ] = q(h1 )q(h2 ) for ν = 2.

Answer First, we note that the inverse variance parameter of the normal distribution is conjugate to the gamma distribution so that

q(h)

= = = =

max Normφ [0, h−1 ]Gamh [ν/2, ν/2] h

max Gamh [(ν + 1)/2, (φ2 + ν)/2] h

max log[Gamh [(ν + 1)/2, (φ2 + ν)/2] h

max −(φ2 + ν)h/2 + (ν − 1)/2 log[h] h

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

62

8

Regression models

Denoting this function by l we now take the derivative with respect to h and set the result to zero

∂l = −(φ2 + ν)/2 + (ν − 1)/2h = 0 ∂h Re-arranging, we get the expression

h=

ν−1 φ2 + ν

The 2D function resembles the sparse prior from figure 8.10.

Problem 8.9 Describe maximum likelihood learning and inference algorithms for a nonlinear regression model based on polynomials where P r(w|x) = Normw [φ0 + φ1 x + φ2 x2 + φ3 x3 , σ 2 ].

Answer We make the transformation z = [1, x, x2 , x3 ]T and then run the standard maximum likelihood and inference algorithms, substituting z for x.

Problem 8.10 I wish to learn a linear regression model in which I predict the world w from I examples of D × 1 data x using the maximum likelihood method. If I > D is it more efficient to use the dual parameterization or the original linear regression model?

Answer It is more efficient to use the original model.

Problem 8.11 Show that the maximum likelihood estimate for the parameters ψ in the dual linear regression model (section 8.7) is given by ψ̂ = (XT X)−1 w.

Answer To fit the parameters, we optimize the criterion Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

63

φ̂

=

h i argmax log[Normw [XT Xψ, σ 2 I ψ

= = = = =

D D 1 argmax − log[σ 2 ] − log[2π] − 2 (w − XT Xψ)T (w − XT Xψ) 2 2 2σ ψ

1 argmax − 2 (w − XT Xψ)T (w − XT Xψ) 2σ ψ

1 argmax − 2 wT w + ψ T XT XXT Xψ − 2wXT Xψ 2σ ψ

1 argmax − 2 ψ T XT XXT Xψ − 2wXT Xψ 2σ ψ i h argmax −ψ T XT XXT Xψ + 2wXT Xψ ψ

Denoting this criterion by L and then taking the derivative with respect to ψ and setting the result to zero we get ∂L = −2XT XXT Xψ + 2XT Xw = 0 ∂ψ Re-arranging this equation gives us the desired relation: ψ̂ = (XT X)−1 w.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

64

8

Regression models

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

Chapter 9

Classification models Problem 9.1 The logistic sigmoid function is defined as sig[a] =

1 .

1 + exp[−a]

Show that (i) sig[−∞] = 0, (ii) sig[0] = 0.5, (iii) sig[∞] = 1.

Answer i) sig[−∞] =

1 1 = = 0.

1 + exp[∞] 1+∞

ii) sig[0] =

1 1 = = 0.5.

1 + exp[0] 1+1

sig[∞] =

1 1 = = 1.

1 + exp[−∞] 1+0

iii)

Problem 9.2 Show that the derivative of the log posterior probability for the logistic regression model L=

I X i=1

wi log

X

I exp[−φT xi ] 1 + (1 − wi ) log T T 1 + exp[−φ xi ] 1 + exp[−φ xi ] i=1

with respect to the parameters φ is given by I X ∂L =− (sig[ai ] − wi ) xi . ∂φ i=1

Answer Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

66

9

Classification models

First, we’ll simplify the criterion by sorting out the log terms

L =

I h i h i h i X −wi log 1+exp[−φT xi ] +(1−wi ) log exp[−φT xi ] −(1−wi ) log 1+exp[−φT xi ] i=1

=

I X

h i h i (1 − wi ) log exp[−φT xi ] − log 1 + exp[−φT xi ]

i=1

=

I X

h i −(1 − wi )φT xi − log 1 + exp[−φT xi ]

i=1

∂L ∂φ

=

I X

−(1 − wi )xi −

i=1

=

I X

1 · exp[−φT xi ] · −xi 1 + exp[−φT xi ]

((1 − wi )xi + (1 − sig[ai ])xi )

i=1

=

−

I X

(sig[ai ] − wi ) xi

i=1

as required.

Problem 9.3 Show that the second derivative of the log likelihood of the logistic regression model is given by I X ∂2L = − sig[ai ](1 − sig[ai ])xi xTi . ∂φ2 i=1

Answer

∂2L ∂φ2

=

# " I X ∂ − (sig[ai ] − wi ) xi ∂φ i=1

=

−

=

−

=

−

I X

1 · exp[−φT xi ]xi xTi T x ])2 (1 + exp[−φ i i=1

I X

exp[−φT xi ] 1 · x xT T Tx ] i i 1 + exp[−φ x 1 + exp[−φ i] i i=1

I X

sig[ai ](1 − sig[ai ])xi xTi ,

i=1

as required.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

67 Problem 9.4 Consider fitting a logistic regression data to 1D data x where the two classes are perfectly separable. For example, perhaps all the data x where the world state w = 0 takes values less than 0 and all the data x where the world state is w = 1 takes values greater than 1. Hence it is possible to classify the training data perfectly. What will happen to the parameters of the model during learning? How could you rectify this problem?

Answer In this case, the sigmoid profile will be fit between the classes and will become infinitely steep until it becomes a step function. In practice, this means that the gradient parameters φ will tend towards infinity. This can cause numerical problems and cause the actual position of the step to become very unstable. The usual way of dealing with this is to us maximum a posteriori estimation with a normally distributed prior with mean zero over the parameter vector φ. This stops the parameter values getting out of hand.

Problem 9.5 Compute the Laplace approximation to a beta distribution with parameters α = 1.0, β = 1.0.

Answer The mode of the beta distribution can be found at λ̂ =

α−1 , α+β−2

See problem 3.2 for the details of this calculation. If we substitute in the values α = 0, β = 0 then we find that the answer is undefined - this distribution is flat, so there is no unique mode and hence no unique Laplace approximation.

For general α, β though, we would calculate the second derivative ∂2L α−1 β−1 =− 2 − ∂λ2 λ (1 − λ)2 and then set

µ

=

λ̂

σ2

=

−

∂2L ∂λ2

−1 .

(9.1)

λ=λ̂

Problem 9.6 Show that the Laplace approximation to a univariate normal distribution with mean µ and variance σ 2 is the normal distribution itself.

Answer The peak of the univariate normal is at the mean µ. The log likelihood and derivatives are Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

68

9

L

=

∂L ∂x ∂2L ∂x2

= =

Classification models

(x − µ)2 1 1 − log[2π] − log[σ 2 2 2 2σ 2 x−µ − 2 σ −1 σ2

To form the Laplace approximation, we create a new normal distribution µ0

=

σ 02

=

x̂ = µ 2 −1 ∂ L − ∂x2

= σ2 .

(9.2)

x=x̂

which just recreates the original normal distribution (as one might have initially suspected!).

Problem 9.7 Show that the second derivative of the logarithm L = log[Normx [µ, Σ]] of a normal distribution evaluated at the mean µ is given by ∂L = −Σ−1 . ∂φ2 µ Answer

L ∂L ∂x ∂2L ∂x2

D 1 log[2π] − log[|Σ|] − 0.5(x − µ)Σ−1 (x − µ) 2 2

=

−

=

−Σ−1 (x − µ)

=

−Σ−1

as required.

Problem 9.8 Devise a method to choose the scale parameter λ0 in the radial basis function in kernel logistic regression (equation 9.33).

Answer The log likelihoods for kernel logistic regression is given by:

L=

I X

h h ii log Bernwi sig[ψ T K[X, xi ] ,

(9.3)

i=1

To find the best scale parameters λ0 using maximum likelihood, we simply optimize with respect to this parameter in the fitting stage (together with the model parameter, ψ).

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

69 b) 1

a) 1

c) 1

Expert 2

Expert 1

Mixture of experts

Mixing weights

0

0

0

Figure 9.1 Mixture of two experts model for 1D data. Pink circles indicate positive examples. Green circles indicate negative examples. a) Two expert is specialized to model the left and right sides of the data, respectively. b) The mixing weights change as a function of the data. c) The final output of the model is mixture of the two constituent experts and fits the data well.

Problem 9.9 A mixture of experts (Jordan and Jacobs, 1994) divides space into different regions, each of which receives specialized attention (figure 9.1) . For example, we could describe the data as a mixture of logistic classifiers so that P r(wi |xi ) =

K X

i h λk [xi ]Bernwi sig[φTk xi ] .

k=1

Each logistic classifier is considered as an expert and the mixing weights decide the combination of experts that are applied to the data. The mixing weights, which are positive and sum to one, depend on the data x: for a two-component model they could be based on a second logistic regression model with activation ω T x. This model can be expressed as the marginalization of a joint distribution between wi and a hidden variable hi so that K X

∀ xi

λk [xi ] = 1

k=1

For example, to divide the data into two regions we could model the values λk based on the outputs of a second logistic regression model with parameters ω.

P r(wi |xi ) =

K X

P r(wi , hi = k|xi ) =

k=1

K X

P r(wi |hi = k, xi )P r(hi = k|xi ),

k=1

where P r(wi |hi = k, xi )

=

P r(hi = k|xi )

=

h i Bernwi sig[φTk xi ] h i Bernhi sig[ω T xi ] .

How does this model differ from branching logistic regression (section 9.8)? Devise a learning algorithm for this model.

Answer Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

70

9

Classification models

The mixtures of experts is very closely related to the branching logistic regression model.

In the former case the prediction is based on a linear sum of logistic regression models associated with each expert. In the latter the prediction is the result of a single logistic regression model where the activation is a linear sum of terms associated with each expert.

As for the branching logistic regression model, the mixtures of experts can be generalized so the mixing weights or the experts themselves contain a nonlinear activation term. They may also be built hierarchically to form a tree structure.

The model can either be learnt by direct optimization of the log posterior probability, or using the EM algorithm. In the latter case, a hidden variable is associated with each data examples. This can be interpreted as indicating which of the logistic regression models a data example is assigned to. In the E-step we calculate a posterior distribution over the I hidden variables {hi }Ii=1 associated with each of the data examples. In the M-step we maximize the bound with respect to the parameters θ = {φ0 , φ1 , ω}.

Problem 9.10 The softmax[•, •, . . . , •] function is defined to return a multivariate quantity where the kth element is given by exp[ak ] sk = softmaxk [a1 , a2 , . . . aK ] = PK . j=1 exp[aj ] Show that 0 < sk < 1 and that

PK

k=1 sk = 1.

Answer It must be true that 0 < sk < 1 since all terms in the numerator and denominator are positive and the numerator is a component of the denominator.

P To prove that K k=1 sk = 1 we compute K X

sk =

k=1

K X

PK exp[ak ] exp[ak ] = =1 PK Pk=1 K exp[a ] j j=1 j=1 exp[aj ] k=1

Problem 9.11 Show that the first derivative of the log-probability of the multi-class logistic regression model is given by equation 9.61.

Answer The cost function is given by

L

=

=

# exp[φTn xi δ[wi − n] log PK T j=1 exp[φj xi i=1 n=1 " # !

I J N X X X T T − log exp[φj xi ] + δ[wi − n]φn xi "

I X K X

i=1

j=1

n=1

Now we compute the derivative Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

71

∂L ∂φn

=

I X i=1

=

−

exp[φTn xi xi + δ[wi − n]x − PK T j=1 exp[φj xi

!

I X (yin − δ[wi − n])xi i=1

Problem 9.12 The classifiers in this chapter have all been based on continuous data x.

Devise a model that can distinguish between M world states w ∈ {1 . . . M } based on a discrete observation x ∈ {1 . . . K} and discuss potential learning algorithms.

Answer For a discriminative classifier, we choose a distribution over the world state and make the parameters of this distribution dependent on the input data. In this case, the world state is discrete, so a categorical distribution is suitable. Since the data is also discrete, we’ll just choose a separate parameters λk for each possible data value. This gives the model P r(wi |xi = k) = Catwi [λk ] In learning, we must estimate the parameters {λk }K k=1 . The simplest approach is to use maximum likelihood in which we simply maximize the log of the posterior probability with respect to the parameters. Alternatively, we could use the maximum a posteriori or Bayesian approaches. In each case this means fitting a separate categorical distribution for each possible data value. This can be done in closed form for all three techniques, using the methods described in chapter 4.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

72

9

Classification models

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

Chapter 10

Graphical models Problem 10.1 The joint probability model between variables {xn }7n=1 factorizes as

P r(x1 , x2 , x3 , x4 , x5 , x6 , x7 )

=

P r(x1 )P r(x3 )P r(x7 )P r(x2 |x1 , x3 )P r(x5 |x7 , x2 )P r(x4 |x2 )P r(x6 |x5 , x4 ).

Draw a directed graphical model relating these variables. Which variables form the Markov blanket of variable x2 ?

Answer The associated graphical model is illustrated in figure 10.1. The Markov blanket of x2 includes variables x1 , x3 , x4 , x5 and x7 .

Problem 10.2 Write out the factorization corresponding to the directed graphical model in figure 10.2a.

Answer

Figure 10.1 Graphical model corresponding to distribution in problem 10.1 Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

74

10

P r(x1...15 ) =

Graphical models

P r(x1 )P r(x2 )P r(x3 )P r(x4 |x1 , x2 ) ×P r(x5 |x2 , x3 )P r(x6 ×P r(x7 )P r(x8 |x4 , x5 )P r(x9 |x3 , x5 , x6 ) ×P r(x10 |x7 )P r(x11 |x10 , x8 )P r(x12 |x8 , x9 )P r(x13 |x9 ) ×P r(x14 |x11 )P r(x15 |x12 ).

a)

b)

Figure 10.2 a) Graphical model for problem 10.2. b) Graphical model for problem 10.4

Problem 10.3 An undirected graphical model has the form P r(x1 . . . x6 ) =

1 Φ1 [x1 , x2 , x5 ]Φ2 [x2 , x3 , x4 ]Φ3 [x1 x5 ]Φ4 [x5 , x6 ].

Z

Draw the undirected graphical model that corresponds to this factorization.

Answer The associated graphical model is illustrated in figure 10.3.

Problem 10.4 Write out the factorization corresponding to the undirected graphical model in figure 10.14b from the book.

Answer Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

75

Figure 10.3 Graphical model corresponding to distribution in problem 10.3

1 φ1 [x1 , x4 ]φ2 [x2 , x4 , x8 ]φ3 [x3 ]φ4 [x2 , x8 , x5 ] Z φ5 [x5 , x9 ]φ6 [x6 , x9 ]φ7 [x7 , x10 ]

P r(x1...15 ) =

φ8 [x7 , x11 ]φ9 [x8 , x11 ]φ10 [x8 , x12 ] φ11 [x9 , x12 , x13 , x15 ]φ12 [x11 , x14 ]

Problem 10.5 Consider the undirected graphical model defined over binary values {xi }4i=1 ∈ {0, 1} defined by

P r(x1 , x2 , x3 , x4 ) =

1 φ(x1 , x2 )φ(x2 , x3 )φ(x3 , x4 )φ(x4 , x1 ), Z

where the function φ is defined by

φ(0, 0) = 1

φ(1, 1) = 2

φ(0, 1) = 0.1

φ(1, 0) = 0.1

Compute the probability of each of the 16 possible states of this system.

Answer The probabilities are given by Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

76

10

P r(x1 = 0, x2 = 0, x3 = 0, x4 = 0)

=

P r(x1 = 0, x2 = 0, x3 = 0, x4 = 1)

=

P r(x1 = 0, x2 = 0, x3 = 1, x4 = 0)

=

P r(x1 = 0, x2 = 0, x3 = 1, x4 = 1)

=

P r(x1 = 0, x2 = 1, x3 = 0, x4 = 0)

=

P r(x1 = 0, x2 = 1, x3 = 0, x4 = 1)

=

P r(x1 = 0, x2 = 1, x3 = 1, x4 = 0)

=

P r(x1 = 0, x2 = 1, x3 = 1, x4 = 1)

=

P r(x1 = 1, x2 = 0, x3 = 0, x4 = 0)

=

P r(x1 = 1, x2 = 0, x3 = 0, x4 = 1)

=

P r(x1 = 1, x2 = 0, x3 = 1, x4 = 0)

=

P r(x1 = 1, x2 = 0, x3 = 1, x4 = 1)

=

P r(x1 = 1, x2 = 1, x3 = 0, x4 = 0)

=

P r(x1 = 1, x2 = 1, x3 = 0, x4 = 1)

=

P r(x1 = 1, x2 = 1, x3 = 1, x4 = 0)

=

P r(x1 = 1, x2 = 1, x3 = 1, x4 = 1)

=

Graphical models

1 ×1 Z 1 × 0.01 Z 1 × 0.01 Z 1 × 0.02 Z 1 × 0.01 Z 1 × 0.0001 Z 1 × 0.02 Z 1 × 0.04 Z 1 × 0.01 Z 1 × 0.02 Z 1 × 0.0001 Z 1 × 0.04 Z 1 × 0.02 Z 1 × 0.04 Z 1 × 0.04 Z 1 × 16 Z

where Z = 17.2802 is given by the sum of all of the unnormalized values.

Problem 10.6 What is the Markov blanket for each of the variables in figures 10.7 and 10.8 from the book?

Answer Figure 10.7a: x1 :

x3

x2 :

x3

x3 :

x1 , x 2

x1 :

x3

x2 :

x3

x3 :

x1 , x 2

Figure 10.7b:

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

77 Figure 10.7c: x1 :

x2 , x3

x2 :

x1 , x3

x3 :

x1 , x2

x1 :

x2 , x3

x2 :

x1 , x4

x3 :

x1 , x4

x4 :

x2 , x3

Figure 10.8a:

Figure 10.8b: x1 :

x2 , x3

x2 :

x1 , x3 , x4

x3 :

x1 , x 2 , x 4

x4 :

x2 , x 3

Problem 10.7 Show that the stated patterns of independence and conditional independence in figure 10.7 and figure 10.8 from the book are true.

Answer Using the rules given in the text In figure 10.7a x2 is conditionally independent of x1 given x3 because the path between x2 and x1 them is blocked at x3 .

In figure 10.7b x2 is conditionally independent of x1 given x3 because there is no route from x2 to x1 that does not include x3 In figure 10.7c x1 and x2 are independent because they share no ancestors.

In figure 10.8a x1 is conditionally independent of x4 given x2 and x3 because there is no route from x1 to x4 that does not include x2 or x3 .

In figure 10.8a x2 is conditionally independent of x3 given x1 and x4 because there is no route from x2 to x3 that does not include x1 or x4 .

In figure 10.8b x1 is conditionally independent of x4 given x2 and x3 because all routes from x1 to x4 are blocked at either x2 or x3 .

In figure 10.8b x2 is conditionally independent of x3 given x1 because there is the route from x2 to x3 via x1 is blocked at x1 .

Problem 10.8 A factor graph is a third type of graphical model that depicts the factorization of a joint probability. As usual it contains a single node per variable, but it also contains one node per factor (usually indicated by a solid square). Each factor variable is connected to all of the variables that are contained in the associated term in the factorization by undirected links. For example, the factor node corresponding to the term P r(x1 |x2 , x3 ) in a directed model would connect to all three variables x1 , x2 and x3 .

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

78

10

a)

Graphical models

b)

Figure 10.4 Factor graphs contain one node (square) per factor in the joint pdf as well as one node (circle) per variable. Each factor node is connected to all of the variables that belong to that factor. This type of graphical model can distinguish between the undirected graphical models a) P r(x1 , x2 , x3 ) = 1 φ [x , x2 , x3 ] and b) P r(x1 , x2 , x3 ) = Z1 φ12 [x1 , x2 ]φ23 [x2 , x3 ]φ13 [x1 , x3 ].

Z 123 1

Figure 10. 7a

Figure 10. 7c

Figure 10. 7d

Figure 10. 7e

Figure 10. 7b

Figure 10.5 Factor graphs for problem 10.8

Similarly, the factor node corresponding to the term φ12 [x1 , x2 ] in an undirected model would connect variables x1 and x2 . Figure 10.4 shows two examples of factor graphs.

Draw the factor graphs corresponding to the graphical models in figures 10.7 and 10.8 from the book. You must first establish the factorized joint distribution associated with each graph.

Answer The associated factor graphs are illustrated in figure 10.5.

Problem 10.9 What is the Markov blanket of variable w2 in figure 10.9c from the book?

Answer w1 , w2 , x2 , w4 and w5

Problem 10.10 What is the Markov blanket of variable w8 in figure 10.9e from the book?

Answer Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

79 w5 , w7 , x8 , w9 and w11

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

80

10

Graphical models

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

Chapter 11

Models for chains and trees Problem 11.1 Compute by hand the lowest possible cost for traversing the graph in figure 11.1 using the dynamic programming method.

Answer The best cost is 16.4, which corresponds to the solution w = [5, 4, 4, 4, 4, 4]T .

2.0

1.1 0.0 2.0

2.8

4.3

6.4

2.3

2.0 0.0 2.0 2.0 0.0 2.0 2.0 0.0 2.0 2.0 0.0

6.8

2.3

0.0

2.2

1.5

5.7 0.0 2.0 2.0 0.0 2.0 2.0 0.0 2.0 2.0 0.0 2.0 2.0 0.0

1.9

2.4

2.1

4.9

2.0 0.0 2.0 2.0 0.0 2.0 2.0 0.0 2.0 2.0 0.0

6.0 0.0 2.0

0.0 2.0 6.0

2.4

0.8

8.9

2.0 0.0 2.0 2.0 0.0 2.0 2.0 0.0 2.0 2.0 0.0

3.1 0.0 2.0

2.9

6.6

7.1

5.0

2.0 0.0 2.0 2.0 0.0 2.0 2.0 0.0 2.0 2.0 0.0

3.3

6.2

2.1

9.8

Figure 11.1 Dynamic programming example for problem 11.1.

Problem 11.2 MAP inference in chain models can also be performed by running Djikstra’s algorithm on the graph in figure 11.2, starting from the node on the left hand side and terminating when we first reach the node on the right hand side. If there are N variables, each of which takes K values, what is the best and worst case complexity of the algorithm?

Describe a situation where Djikstra’s algorithm outperforms dynamic programming.

Answer In the worst case, Djikstra’s algorithm will visit all N K nodes. Every time it reaches a new node, it must consider which is the minimum possible cost of the remaining options (of which the maximum is K. If we implement this minimization by storing a linked list and searching through it, then this will have complexity K and so the cost will be O(N K 2 ).

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

82

11

Models for chains and trees

Figure 11.2 Graph construction for problem 11.2. This is the same as the dynamic programming graph (figure 11.3) except that: (i) there are two extra nodes at the start and the end of the graph. (ii) There are no vertex costs. (iii) The costs associated with the left-most edges are U1 (k) and the costs associated with the right-most edges are 0. The general edge cost for passing from label a and node n to label b at node n + 1 is given by Pn,n+1 (a, b) + Un+1 (b).

In the best case, Dijkstra’s algorithm will only visit N nodes, greedily choosing a path straight from left to right and then terminating. Here, the cost will be O(N K).

Problem 11.3 Consider the graphical model in figure 11.3a. Write out the cost function for MAP estimation in the form of equation 11.17. Discuss the difference between your answer and equation 11.17.

Answer In this case, the prior has the form P r(w6 )P r(w5 |w6 )P r(w2 |w5 )P r(w4 |w5 )P r(w3 |w4 )P r(w1 |w2 ) Notice that there is no three-wise term here, and the cost function will have the form

ŵ1...6 = argmin w1...6

" 6 X

# Un (wn ) + P5 (w5 , w4 ) + P2 (w2 , w5 ) + P4 (w4 , w5 ) + P3 (w3 , w4 )P1 (w1 , w2 )

n=1

Problem 11.4 Compute the solution (minimum cost path) to the dynamic programming problem on the tree in figure 11.4.

Answer We apply the rules Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

83 a)

b)

Figure 11.3 a) Graphical model for problem 11.3. b) Graphical model for problem 11.10. The unknown variables w3 , w4 . . . in this model receive connections from the two preceding variables and so the graph contains loops.

b)

a)

0.0 2.0 1.8

0.4 1.7

0.0 1.7 3.7

1.3 0.2

1.7 1.3 1.0 2.3 10.4 0.2

1.1

2.4

0.3

3.4

2.0 0.0 2.0 2.0 0.0 2.0 2.0 0.0

0.8

5.3 0.0 2.0 7.0

3.4 6.8

1.2 0.0 2.0 3.8

3.7

0.3 5.2 2.3 0.7 0.4 1.4

0.4 1.7

1.7 1.3 0.4

2.3 3.0

0.7

2.0 0.0 2.0 2.0 0.0 2.0 2.0 0.0

1.5

5.7 6.0

6.1 4.8

2.0 0.0 2.0 2.0 0.0 2.0 2.0 0.0

2.0

5.4

0.8

2.1

4.0

Figure 11.4 Dynamic programming example for problem 11.4.

S1,k

=

U1 (w1 = k)

S2,k

=

U2 (w2 = k) + min [S1,l + P2 (w2 = k, w1 = l)]

S3,k

=

U3 (w3 = k)

S4,k

=

U4 (w4 = k) + min [S3,1 + P4 (w4 = k, w3 = l)]

S5,k

=

U5 (w5 = k) + min [S2,l + S4,m + T5 (w5 = k, w2 = l, w4 = m)]

S6,k

=

U6 (w6 = k) + min [S5,l + P6 (w6 = k, w5 = l)] .

l

l

l,m l

The cumulative sums and the overall solution are shown in figure 11.5.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

84

11 a)

2.4

3.5

1.8

2.6

0.3

5.6

b)

8.3

1.2

9.0

3.8

2.4

3.5

1.8

2.6

0.3

5.6

3.4

8.3

18.9

18.3 3.4

Models for chains and trees

15.4

17.0

13.0

18.4 1.2

9.0

8.0

3.8

8.0

3.7

4.8

3.7

4.8

0.7

4.7

0.7

4.7

15.8

15.9

18.3

18.9

15.4

17.0

13.0

18.4

15.9

15.8

Figure 11.5 Solution for problem 11.4. a) Cumulative sums. b) Minimum cost solution.

Problem 11.5 MAP inference for the chain model can be expressed as "

"

"

ŵN = argmax max max . . . max w1

wN

w2

wN −1

" N X

log[P r(xn |wn )] +

n=1

N X

#

###

log[P r(wn |wn−1 )] . . .

n=2

Show that it is possible to compute this expression piecewise by moving the maximization terms through the summation sequence in a manner similar to that described in section 11.4.1.

Answer We simply move the maximizations through the sequence so that they only address the terms that are relevant

ŵN

=

max log[P r(xN |wn ] + max [log[P r(wN |wN −1 )] + log[P r(xN −1 |wN −1 ] + . . . + wN wN −1

max log[P r(w3 |w2 )] + log[P r(x2 |w2 ] + max [log[P r(w2 |w1 )] + log[P r(x1 |w1 ]] w2

w1

Problem 11.6 Develop an algorithm that can compute the marginal distribution for an arbitrary variable wn in a chain model.

Answer This is a special case of the forward backward algorithm where there is no need to compute all of the terms.

First, we observe (equation 11.1) that the marginal distribution can be computed as P r(wn |x1...N ) ∝ P r(wn , x1...n )P r(xn+1...N |wn ) = fn [wn ]bn [wn ].

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

85 We recursively compute the forward terms (up to variable n) using the relation fn [wn ] = P r(xn |wn )

X

P r(wn |wn−1 )fn−1 [wn−1 ],

wn−1

where we set f1 [w1 ] = P r(x1 |w1 )P r(w1 ). We recursively compute the backward terms from N to n using the relation bn−1 [wn−1 ] =

X

P r(xn |wn )P r(wn |wn−1 )bn [wn ],

wn

where we set bN [wN ] to the constant value 1/K. Then we take the product fn [wn ]bn [wn ] and normalize.

Problem 11.7 Develop an algorithm that computes the joint marginal distribution of any two variables wm and wn in a chain model.

Answer Let’s assume that m < n. We can write out the joint probability as

P r(wm , wn |x1...N )

∝

P r(wm , wn , x1...N )

=

P r(wm , x1...m )P r(wn , xm+1...n |wm , x1...m )P r(xn+1...N |wm , wn , x1...n )

=

P r(wn , x1...n )P r(wn , xm+1...n |wm )P r(xn+1...N |wn ),

The first and last of these terms can be computed via a forward and backward recursion as in the original forward backward algorithm. The middle term admits a new recursion that starts at position m + 1 and moves to position n

P r(wn , xm+1...n |wm )

=

X

P r(wn , wn−1 , xm+1...n |wm )

wn−1

=

X

P r(wn , xn |wm , wn−1 , xm+1...n−1 )P r(wn−1 , xm+1...n−1 |wm )

wn−1

=

X

P r(xn |wn )P r(wn |wn−1 )P r(wn−1 , xm+1...n−1 |wm )

wn−1

Problem 11.8 Consider the following two distributions over three variables x1 , x2 , and x3 :

P r(x1 , x2 , x3 )

=

P r(x1 , x2 , x3 )

=

1 φ12 [x1 , x2 ]φ23 [x2 , x3 ]φ31 [x3 , x1 ] Z1 1 φ123 [x1 , x2 , x3 ].

Z2

Draw (i) an undirected model and (ii) a factor graph for each distribution. What do you conclude?

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

86

11

Models for chains and trees

Answer The graphical models are illustrated in figure 11.6. The undirected models do not distinguish between these two cases. However, the factor graphs do demonstrate the difference.

We conclude that factor graphs are more expressive than undirected models.

i)

ii)

Figure 11.6 Graphical models for problem 11.8. i) The undirected models for the two distributions are identical. iii) The factor graphs for the two models are distinct.

Problem 11.9 Convert each of the graphical models in figure 11.7 into the form of a factor graph. Which of the resulting factor graphs take the form of a chain? a)

b)

c)

d)

Figure 11.7 Answers for problem 11.9.

Answer The models in (a) and (d) take the form of chains.

Problem 11.10 Figure 11.3b shows a chain model in which each unknown variable w depends on its two predecessors. Describe a dynamic programming approach to finding the MAP solution. (hint: you need to combine variables.) If there are N variables in the chain and each takes K values, what is the overall complexity of your algorithm?

Answer Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

87 a)

c)

b)

d)

Figure 11.8 Graphical models for problem 11.9.

Combine adjacent variables to create a new set of nodes. For example new node 1 contains joint states of w1 and w2 and new node 2 contains joint states of w2 and w3 . Set up graph with appropriate costs, where now each new node depends on the previous pair of states.

Ensure that the duplicate entries for each original variable are constrained to be the same (i.e. infinite costs for differing). If there were K states each for N variables, then the cost will be O(N K 4 ).

Problem 11.11 In the stereo vision problem, the solution was very poor when the pixels are treated independently. Suggest some improvements to this method (while keeping the pixels independent).

Answer The simplest thing to do is to associate a larger pixel region with each state - for example, we might assume that the local 3 × 3 pixel region is a Gaussian corruption of an offset 3 × 3 region in the other image. This gives more reliable unary terms, but copes less well in regions where the disparity is changing fast.

Problem 11.12 Consider a variant on the segmentation application in which we update all of the contour positions at once. The graphical model for this problem is a loop (i.e., a chain where there is also a edge between wN and w1 ). Devise an approach to finding the exact MAP solution in this model. If there are N variables each of which can take K values, what is the complexity of your algorithm?

Answer Set up the usual dynamic programming graph, but duplicate w1 at the right hand side as wN +1 . If there are K possible values for w1 and wN +1 then compute the dynamic programming solution K times, in each case, ensuring that w1 and wN +1 take the SAME value, k. This can be achieved by setting the costs for other solutions to infinite. Compute the minimum cost of these K solutions and for the best trace back the route as in a normal dynamic programming problem. This algorithm has cost O(N K 3 ).

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

88

11

Models for chains and trees

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

Chapter 12

Models for grids Problem 12.1 Consider a Markov random field with the structure P r(x1 , x2 , x3 , x4 ) =

1 φ[x1 , x2 ]φ[x2 , x3 ]φ[x3 , x4 ]φ[x4 , x1 ] Z

but where the variables x1 , x2 , x3 , and x4 are continuous and the potentials are defined as

φ[a, b] = exp −(a − b)2 .

This is known as a Gaussian Markov random field. Show that the joint probability is a normal distribution, and find the information matrix (inverse covariance matrix).

Answer Filling in the missing elements of the equation we have:

P r(x1 , x2 , x3 , x4 )

= = =

=

1 φ[x1 , x2 ]φ[x2 , x3 ]φ[x3 , x4 ]φ[x4 , x1 ] Z

1 exp −(x1 − x2 )2 exp −(x2 − x3 )2 exp −(x3 − x4 )2 exp −(x4 − x1 )2 Z

1 exp −(x1 − x2 )2 − (x2 − x3 )2 − (x3 − x4 )2 − (x4 − x1 )2 Z      x1 2 −1 0 −1   x2  −1 1 2 −1 0     exp  − x1 x2 x3 x4  0 −1 2 −1 x3  Z −1 0 −1 2 x4

Problem 12.2 Compute the MAP solution to the three-pixel graph cut problem in figure 12.1 by (i) computing the cost of all eight possible solutions explicitly and finding the one with the minimum cost (ii) running the augmenting paths algorithm on this graph by hand and interpreting the minimum cut.

Answer Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

90

12

Models for grids

The costs are given by:

C[0, 0, 0]

=

1.1 + 4.5 + 7.1 = 12.7

C[0, 0, 1]

=

1.1 + 4.5 + 3.2 + 4.1 = 12.9

C[0, 1, 0]

=

1.1 + 6.8 + 7.1 + 4.1 + 4.0 = 23.1

C[0, 1, 1]

=

1.1 + 6.8 + 3.2 + 4.1 = 15.2

C[1, 0, 0]

=

8.3 + 4.5 + 7.1 + 1.1 = 21.0

C[1, 0, 1]

=

8.3 + 4.5 + 7.2 + 1.1 + 4.1 = 25.2

C[1, 1, 0]

=

8.3 + 6.8 + 7.2 + 4.0 = 26.3

C[1, 1, 1]

=

8.3 + 6.8 + 3.2 = 18.3

so the best solution is to choose C[0, 0, 0].

Source

s

Source 1.1

a

b

c

a

1.1 4.1

s 4.5

b

7.1

4.0 4.1

c

6.8 8.3

Sink

t

Sink

3.2

t

Figure 12.1 Graph for problem 12.2.

Problem 12.3 Explicitly compute the costs associated with the four possible minimum cuts of the graph in figure 12.10 from the book.

Answer The costs are as follows:

C[0, 0]

=

Ua (0) + Ub (0) + Pab (0, 0)

C[0, 1]

=

Ua (0) + Ub (1) + Pab (0, 1)

C[1, 0]

=

Ua (1) + Pa,b (1, 1) + Ub (0) + Pab (1, 1) + Pab (1, 0) − Pab (1, 1) − Pab (0, 0)

=

Ua (1) + Ub (1) + Pab (1, 0)

C[1, 1]

=

Ua (1) + Pa,b (1, 1) + Ub (1)

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

91 Problem 12.4 Compute the cost for each the four possible cuts of the graph in figure 12.11c from the book.

Answer C[0, 0]

=

Ua (0) + Ub (0) + β

C[0, 1]

=

Ua (0) + Ub (1) + Pab (0, 1) + β

C[1, 0]

=

Ua (1) + β + Ub (0) + β + Pab (1, 0) − β

=

Ua (1) + Ub (1) + Pab (1, 0) + β

=

Ua (1) + β + Ub (1)

C[1, 1]

Problem 12.5 Consider the graph construction in figure 12.2a, which contains a number of constraint edges of infinite cost (capacity). There are 25 possible minimum cuts on this graph, each of which corresponds to one possible labeling of the two pixels. Write out the cost for each labeling. Which solutions have finite cost for this graph construction?

Answer The costs are as follows: C[1, 1]

=

Ua (1) + Ub (1)

C[1, 2]

=

Ua (1) + Ub (1) + Pab (1, 2)

C[1, 3]

=

∞

C[1, 4]

=

∞

C[1, 5]

=

∞

C[2, 1]

=

Ua (2) + Ub (1) + Pab (2, 1)

C[2, 2]

=

Ua (2) + Ub (2)

C[2, 3]

=

Ua (2) + Ub (3) + Pab (2, 3)

C[2, 4]

=

∞

C[2, 5]

=

∞

C[3, 1]

=

∞

C[3, 2]

=

Ua (3) + Ub (2) + Pab (3, 2)

C[3, 3]

=

Ua (3) + Ub (3)

C[3, 4]

=

Ua (3) + Ub (4) + Pab (3, 4)

C[3, 5]

=

∞

C[4, 1]

=

∞

C[4, 2]

=

∞

C[4, 3]

=

Ua (4) + Ub (3) + Pab (4, 3)

C[4, 4]

=

Ua (4) + Ub (4)

C[4, 5]

=

Ua (4) + Ub (5) + Pab (4, 5)

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

92

12 C[5, 1]

=

∞

C[5, 2]

=

∞

C[5, 3]

=

∞

C[5, 4]

=

Ua (5) + Ub (4) + Pab (5, 4)

C[5, 5]

=

Ua (5) + Ub (5)

Models for grids

This graph construction only assigns finite costs to solutions where the neighboring pixels are the same or differ by one value, and hence ensures smooth solutions.

a)

Source

b)

s

Source

s

a1

b1

a1

b1

c1

a2

b2

a2

b2

c2

a3

b3

a3

b3

c3

a4

b4

a4

b4

c4

Sink

t

Sink

t

Figure 12.2 Alternative multi-label graph constructions. Each of these these graphs has extra constraint links with infinite weight. These have the effect of giving an infinite cost to a subset of the possible solutions.

Problem 12.6 Which of the possible minimum cuts of the graph in figure 12.2b have a finite cost?

Answer This graph construction only assigns finite costs to solutions where pixels a and b differ by one value of less, pixels b and c differ by one value or less and pixels a and c differ by one value or less. Adding the diagonal links between pixels a and c introduces this last constraint, which can be used to ensure that the label field changes even more smoothly.

Problem 12.7 Confirm that the costs of the cuts in figure 12.14b from the book are as claimed by explicitly performing the summation over the relevant terms Cij .

Answer Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

93 For the case in figure 12.14a the cost M is:

M

=

Ua (1) + Ub (3) +

5 1 X X

Cab

i=1 j=4

=

Ua (1) + Ub (3) + C1,4 + C1,5

=

Ua (1) + Ub (3) + Pa,b (1, 3) + Pab (0, 4) − Pab (1, 4) − Pab (0, 3) +Pa,b (1, 4) + Pab (0, 5) − Pab (1, 5) − Pab (0, 4)

=

Ua (1) + Ub (3) + Pa,b (1, 3) + Pab (0, 4) − Pab (0, 3) + Pab (0, 5) − Pab (1, 5) − Pab (0, 4)

=

Ua (1) + Ub (3) + Pa,b (1, 3) + 0 + 0 − 0 − 0

=

Ua (1) + Ub (3) + Pa,b (1, 3)

as required.

For the case in figure 12.14b the cost M is:

M

=

Ua (1) + Ub (3) +

4 X 5 X

Cab

i=1 j=5

=

Ua (4) + Ub (4) + C1,5 + C2,5 + C3,5 + C4,5

=

Ua (4) + Ub (4) + Pa,b (1, 4) + Pab (0, 5) − Pab (1, 5) − Pab (0, 4) +Pa,b (2, 4) + Pab (1, 5) − Pab (2, 5) − Pab (1, 4) +Pa,b (3, 4) + Pab (2, 5) − Pab (3, 5) − Pab (2, 4) +Pa,b (4, 4) + Pab (3, 5) − Pab (4, 5) − Pab (3, 4)

=

Ua (4) + Ub (4) + Pab (0, 5) − Pab (0, 4) + Pa,b (4, 4) − Pab (4, 5)

=

Ua (4) + Ub (4) + 0 − 0 + Pa,b (4, 4) − 0

=

Ua (4) + Ub (4) + Pa,b (4, 4)

as required.

Problem 12.8 Show that the Potts model (figure 12.17c) is not submodular by providing a counter-example to the required criterion: Pab (β, γ) + Pab (α, δ) − Pab (β, δ) − Pab (α, γ) ≥ 0.

Answer Let the cost for a non-zero label difference be c. Now choose β = γ and the β > α and δ>γ

Pab (β, γ) + Pab (α, δ) − Pab (β, δ) − Pab (α, γ)

=

0 + c − c − c = −c

This clearly disobeys the inequality.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

94

12

Models for grids

Source States before

a

b

c

d

e

Sink Figure 12.3 Graph construction for problem 12.9

Problem 12.9 An alternative to the alpha-expansion algorithm is the alpha-beta swap.

Here, a multi-label MRF with non-convex potentials is optimized by repeatedly choosing pairs of labels α, β and performing a binary graph cut that allows them to swap in such a way that the overall cost function decreases. Devise a graph structure that can be used to perform this operation. Hint: consider separate cases for neighboring labels α, α, β, β, β, γ, α, γ and γ, γ where γ is a label that is neither α or β.

Answer The graph construction is shown in figure 12.3. Nodes which are neither α or β do not need to be included in the graph. The pairwise term that encodes the cost for one of the nodes in the graph being next to a node with a value that is neither α or β can be included in the links connecting this node to the source and sink. Notice that this graph construction does not require the triangle inequality to be true.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

Chapter 13

Image processing and feature extraction Problem 13.1 Consider an 8-bit image in which the pixel values are evenly distributed in the range 0to 127, with no pixels taking a value of 128 or larger. Draw the cumulative histogram for this image. What will the histogram of pixel intensities look like after applying histogram equalization?

Answer Both the cumulative histogram and the histogram of pixel intensities after equalization are shown in figure 13.1. The histogram is not quite as you might have imagined it. The original value ‘1’ gets mapped to ‘2’ and the original value ‘2’ gets mapped to ‘4’ etc, and consequently, there no entry at ‘3’ in the new histogram. The new histogram hence takes the form of a comb with 128 entries.

Problem 13.2 Consider a continuous image f [i, j] and a continuous filter f [m, n] In the continuous domain, the operation f ⊗ p of convolving an image with the filter is defined as: Z ∞ Z ∞ f ⊗p=

p[i − m, j − n]f [m, n] dmdn. −∞

−∞

Now consider two filters f and g. Prove that convolving the image first with f and then with g has the same effect as convolving f with g and then convolving the image with the result. In other words: g ⊗ (f ⊗ p) = (g ⊗ f ) ⊗ p.

Does this result extend to discrete images?

Answer Let’s define images a and b as Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

96

13

Image processing and feature extraction b)

a)

0

Frequency

Cumulative Proportion

1

0

Gray-level Intensity

Gray-level Intensity

255

Figure 13.1 a) Cumulative histogram for problem 13.1. b) Histogram after histogram equalization for problem 13.1

Z ∞ Z ∞ a

=f ⊗p=

p[i − m, j − n]f [m, n] dmdn −∞

−∞

Z ∞ Z ∞ b

=g⊗f =

f [i − m, j − n]g[m, n] dmdn. −∞

−∞

Now, we use the same formula again to compute the left hand side

g ⊗ (f ⊗ p)

=

g⊗a Z ∞ Z ∞

=

a[k − i, l − j]g[i, j] didj −∞ −∞ Z ∞ Z ∞ Z ∞ Z ∞ p[k − i − m, l − j − n]f [m, n]g[i, j] didjdmdn

=

−∞

−∞

−∞

−∞

and the right hand side

(g ⊗ f ) ⊗ p

=

b⊗p Z ∞ Z ∞

=

p[k − m0 , l − n0 ]b[m0 , n0 ] dm0 dn0

−∞ −∞ Z ∞ Z ∞ Z ∞ Z ∞

=

p[k − m0 , l − n0 ]f [m0 − i, n0 − j]g[i, j] didjdm0 dn0

−∞ −∞ −∞ −∞ Z ∞ Z ∞ Z ∞ Z ∞

p[k − i − m, l − j − n]f [m, n]g[i, j] didjdmdn

= −∞

−∞

−∞

−∞

where we have made the changes of variables m0 = m + i and n0 = n + j between the last two lines. We notice that both expressions are the same as required.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

97 This extends to infinite discrete images, but we can get different results for finite sized images, depending on how we deal with the boundaries.

Problem 13.3 Describe the series of operations that would be required to compute the Haar-like filters in figures 13.6a-d from an integral image. How many points from the integral image are needed to compute each?

Answer Denoting the areas with the letters given by figure 13.2 we have: a) 6 points

T

=

(−a + d − b − c) + (c + f − d − e)

=

−a + b + 2c − 2d − e + f

=

−(a + e − b − c) + b + f − c − e

=

−a + 2b − c + d − 2e + f

b) 6 points

T

c) 8 points

T

=

−(a − b − c + d) + 2(c + f − d − e) − (e + h − f − g)

=

−a + b + 3c − 3d − 3e + 3f − h − g

d) 9 points

T

=

−(a + e − b − d) + (b + f − c − e) + (d + h − e − g) − (e + i − h − f )

=

−a + 2b − c + 2d − 4e + 2f − g + 2h − i

Problem 13.4 Consider a blurring filter where each pixel in an image is replaced by a weighted average of local intensity values, but the the weights decrease if these intensity values differ markedly from the central pixel. What effect would this bilateral filter have when applied to an image?

Answer It will smooth the image, but it will respect edges - close to an step, it mainly averages pixels that are on the same side of the edge, so that the edge does not blur.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

98

13

a

b

c

d

e

f

a)

a

b

Image processing and feature extraction

c

a

b

c

d

e

f

a

b

d

e

g

b)

d

e

c)

f

h

c f i

d)

Figure 13.2 Haar features for problem 13.3. In each case, the letters indicate the total image intensity in the area above and to the left of the marked point.

Problem 13.5 Define a 3 × 3 filter that is specialized to detecting luminance changes at a 45o angle and gives a positive response where the image intensity increases from the bottom left to the bottom right of the image.

Answer 

0 F = −1 0

1 0 −1

 0 1 0

Problem 13.6 Define a 3 × 3 filter that responds to the second derivative in the horizontal direction, but is invariant to the gradient and absolute intensity in the horizontal direction and invariant to all changes in the vertical direction.

Answer 

−1 F = −1 −1

2 2 2

 −1 −1 −1

Problem 13.7 Why are most local binary patterns in a natural image typically uniform or near-uniform?

Answer Most of the local binary patterns are near-uniform because most surfaces are gradually changing in intensity due to lighting effects. When there is a smooth gradient, half of the values will be greater than the center and half less and these will be spatially coherent It is actually rare to find uniform examples (where all the pixels are greater than or less than the center one).

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

99

Figure 13.3 An example of a dataset where K means will fail to cluster the data properly because it only models spherical clusters whereas mixtures of Gaussians (which describes clusters as ellipsoid) will succeed.

Figure 13.4 Clustering with the Kmeans algorithm in the presence of outliers (problem 13.9). This data set contains two clusters and a single outlier (the point on the right hand side).

The outlier causes problems for the Kmeans algorithm when K = 2 clusters are used due to the implicit assumption that the clusters can be modeled as normal distributions with spherical covariance.

Problem 13.8 Give one example of a 2D data set where the mixtures of Gaussians model will succeed in clustering the data, but the K-means algorithm will fail.

Answer An example is illustrated in figure 13.3.

Problem 13.9 Consider the data in figure 13.4. What do you expect to happen if we run the K-means algorithm with two clusters on this data set? Suggest a way to resolve this problem.

Answer The outlying point will drag the mean of one of the clusters into the large blank space and the result will be that the central data cluster will wind up being shared by the two components. One way to handle this would be to use a mixture of two t-distributions which are robust to outliers.

Problem 13.10 An alternative approach to clustering the data would be to find modes (peaks) in the density of the points. This potentially has the advantage of also automatically selecting the number of clusters. Propose an algorithm to find these modes.

Answer One simple approach is the ‘mean-shift’ algorithm. We determine a starting position and place a spheroid (circle in 2D) centered at this position. The radius should be such that Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

100

13

Image processing and feature extraction

it includes a number of data points, but not all of them. Then we move the center of the spheroid to a new position that is the average of the points that fall within the radius.

We iterate this process. The result is that the spheroid gradually moves towards regions of greater density and will eventually stop moving when it reaches a mode. We can repeat this from many starting positions to find all of the modes.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

Chapter 14

The pinhole camera Problem 14.1 A pinhole camera has a sensor that is 1cm × 1cm and a horizontal field of view of 60o . What is the distance between the optical center and the sensor? The same camera has a resolution of 100 pixels in the horizontal direction and 200 pixels in the vertical direction (i.e., the pixels are not square). What are the focal length parameters fx and fy from the intrinsic matrix?

Answer The geometry of this system is shown in figure 14.1. Using basic trigonometry, we can calculate the distance d to the focal plane as d=

0.5 = 0.866cm. tan[30o ]

To compute the focal length parameters, we use the relations

x

=

y

=

fx U W fx V W

We take the bottom left corner of the camera sensor as our world point so that [U, V, W ]T = [0.5, 0.5, d]T . This is projected to position [x, y]T = [50, 100] (assuming that pixels counted form the center). Now we can re-arrange to get

fx

= = xW U

28.87

fy

= yW U

57.74

=

Problem 14.2 We can use the pinhole camera model to understand a famous movie effect.

Dolly zoom was first used in Alfred Hitchcock’s Vertigo. As the protagonist looks down a stairwell, it appears to deform (figure 14.2) in a strange way. The background seems to move away from the camera, while the foreground remains at a constant position.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

102

14

The pinhole camera

Focal length, d

Optical center

Figure 14.1 lem 14.1.

Geometry

for

prob-

w = 0.5 cm Image plane

In terms of the camera model, two things occur simultaneously during the dolly zoom sequence: the camera moves along the w-axis, and the focal distance of the camera changes. The distance moved and the change of focal length are carefully chosen so that objects in a pre-defined plane remain at the same position. However, objects out of this plane move relative to one another (figures 14.2d-e).

I want to capture two pictures of a scene at either end of a ‘dolly zoom’. Before the zoom, the camera is at w = 0, the distance between the optical center and the image plane is 1cm, and the image plane is 1cm × 1cm. After the zoom, the camera is at w = 100cm. I want the plane at w = 500cm to be stable after the camera movement. What should the new distance between the optical center and the image plane be?

Answer For simplicity, we’ll assume that the pixels are counted from the center. Before the change, the position of points in the image at the desired plane is governed by the equation fU 500 After moving the camera, the position x should be the same, but now the distance is 400cm and so we have x=

x=

f 0U 400

Combining these equations we have 400 f 500 Since, we know that the focal length parameter is proportional to the distance d between the camera and focal plane, it follows that f0 =

d0 =

400 d = 0.8cm 500

Problem 14.3 Figure 14.3 shows two different camera models: the orthographic and weak perspective cameras. For each camera, devise the relationship between the homogeneous world points and homogeneous image points. You may assume that the world coordinate system and the camera coordinate system coincide, so there is no need to introduce the extrinsic matrix.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

103 a)

d)

Optical center Camera plane

b)

Constant plane

e)

Camera moves forward

c)

Optical center Camera plane Constant plane

Figure 14.2 Dolly Zoom. (a-c) Three frames from ‘Vertigo’ sequence in which the stairwell appears to distort. Nearby objects remain in roughly the same place whereas object further away systematically move through the sequence.

To see this, consider the red and green circles that are at the same (x, y) position in each frame. The red circle remains on the near bannister, but the green circle is on the floor of the stairwell in the first image but halfway up the stairs in the last image. d) To understand this effect consider a camera viewing a scene which consists of several green points at the same depth and some other surfaces (colored lines). e) We move the camera along the w-axis but simultaneously change the focal length so that the green points are imaged at the same position. Under these changes, objects in the plane of the green points are static, but other parts of the scene move and may even occlude one another.

Answer Each camera can be described by 4 × 3 projection matrix. We’ll describe the equivalent of the normalized camera in each case, but it is easy to add the focal length and offset parameters if required.

For the orthographic camera model we have Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

104

14 a)

The pinhole camera

b)

Optical center Image plane

Image plane

Figure 14.3 Alternative camera models. a) Orthographic camera. Rays are parallel and orthogonal to image plane. b) Weak perspective model. Points are projected orthogonally onto a reference plane at distance zr from the camera and then pass to the image plane by perspective projection.

   x 1 λ y  = 0 1 0



0 1 0

0 0 0

 U 0 V   0 =  W  1 1

0 1 0

0 0 0

   U 0 V    0 =  W zr 1



For the weak perspective camera we have    x 1 λ y  = 0 1 0

Problem 14.4 Find the point where the homogeneous lines l̃1 and l̃2 join where: 1. l̃1 = [3, 1, 1], and l̃2 = [−1, 0, 1] 2. l̃1 = [1, 0, 1], and l̃2 = [3, 0, 1] Hint: the (3 × 1) homogeneous point vector x̃ must satisfy both l̃T1 x̃ = 0 and l̃T2 x̃ = 0. In other words it should be orthogonal to both l̃1 and l̃2 .

Answer 1. To find a vector that is orthogonal to both l̃1 and l̃2 we take the cross product i x̃ = l̃1 × l̃2 = 3 −1

j 1 0

  k −1 1 = 4  1 −1

j 0 0

  k 0 1 = 2 1 0

2. By the same process, we get i x̃ = l̃1 × l̃2 = 1 3

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

105 Notice that these two lines were parallel, so the point where the converge is at infinity (last component of homogeneous representation is zero).

Problem 14.5 Find the line joining the homogeneous points x̃1 and x̃2 where x̃1 = [2, 2, 1]T , x̃2 = [−2, −2, 1]T .

Answer By the same logic as the previous question we have i l̃ = x̃1 × x̃2 = 2 −2

j 20 −2

  k 4 1 =  4 1 0

Problem 14.6 A conic C is a geometric structure that can represent ellipses and circles in the 2D image. The condition for a point to lie on a conic is given by

x

y

 a 1 b c

b d e

  c x e  y  = 0, f 1

or x̃T Cx̃ = 0.

Describe an algorithm to estimate the parameters a, b, c, d, e, f given several points x1 , x2 , . . . , xn that are known to lie on the conic. What is the minimum number of points that your algorithm requires to be successful?

Answer Writing out this equation in full we get

2 x

2xy

2x

y2

2y

  a b   c  1   d  = 0.   e f

Now we can take a number of points {xi }Ii=1 and form the equations  2 x1 x22  .  .. x2I

2x1 y1 2x2 y1 .. .

2xI yI

2x1 2x2 .. .

2xI

y12 y22 .. . yI2

2y1 2y2 .. .

2yI

   a 1 b   1  c  ..     = 0. .  d e 1 f

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

106

14

The pinhole camera

This has the form Aw = 0 of a minimum direction problem (see Appendix C of book) and can be solved using the singular value decomposition. The six unknowns are ambiguous up to scale, so we need at least 5 points to find a unique solution.

Problem 14.7 Devise a method to find the intrinsic matrix of a projector using a camera and known calibration object.

Answer We have a calibration object with points {wi }Ii=1 on its surface. We need to find the set of points {xi } in the projector matrix that project onto these world points. Then we can proceed exactly as if this were a camera.

One way to do this would be to place a camera close to the projector that is also pointing at the surface. Then we sweep a horizontal line down through the projector image, one pixel per frame. We use the camera to determine exactly when this line touches each of the points {wi }Ii=1 and hence determine the y coordinates. We can repeat this procedure with vertical line to determinethe x coordinates.

Problem 14.8 What is the minimum number of binary striped light patterns of the type illustrated in figure 14.13 required to estimate the camera-projector correspondences for a projector image of size H × W ?

Answer For an projector image of dimensions H × W we can determine the correspondences with only 2(log2 [W ] + log2 [H]) patterns.

Problem 14.9 There is a potential problem with the shape from silhouette algorithm as described; the point that we have found on the surface of the object may be occluded by another part of the object with respect to the nearest camera. Consequently, when we copy the color, we will get the wrong value. Propose a method to circumvent this problem.

Answer One approach is to run the depth-finding algorithm again from the pixel in the real image that we are going to copy. If the depth that we find corresponds to the same point as the one in the new image then we know it is not occluded. If it is occluded, we should copy the pixel colour from another camera.

Problem 14.10 In the augmented reality application (figure ??), the realism might be enhanced if the object had a shadow. Propose an algorithm that could establish whether a point on the desktop (assumed planar) is shadowed by the object with respect to a point light source at a known position.

Answer Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

107 For each pixel on the surface of the plane, run the main depth finding algorithm along the ray that points from this pixel to the light source. If there is no point which is inside all of the object silhouettes, then the point is not occluded and we should not add a shadow.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

108

14

The pinhole camera

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

Chapter 15

Models for transformations Problem 15.1 The 2D point x2 is created by a rotating point x1 using the rotation matrix Ω1 and then translating it by the translation vector τ 1 so that x 2 = Ω1 x 1 + τ 1 .

Find the parameters Ω2 and τ 2 of the inverse transformation x 1 = Ω2 x 2 + τ 2 in terms of the original parameters Ω1 and τ 1 .

Answer Subtracting τ1 from both sides of the original equation we have x 2 − τ 1 = Ω1 x 1 .

Now we pre-multiply by ΩT1 to get the relation ΩT1 x2 − ΩT1 τ 1 = x1 , where we have used the fact that the inverse of a rotation matrix is its transpose.

Reading off the new rotation and translation we see that

Ω2

=

ΩT1

τ2

=

−ΩT1 τ 1

Problem 15.2 A 2D line can be as expressed as ax + by + c = 0 or in homogeneous terms lx̃ = 0, where l = [a, b, c]. If points are transformed so that x̃0 = Tx̃, Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

110

15

Models for transformations

what is the equation of the transformed line?

Answer It must be true that for the new line l0 x0 = 0 and so l0 Tx = 0 By inspection we can see that to make this true we can set l0 = lT−1 .

Problem 15.3 Using your solution from problem 15.2, develop a linear algorithm for estimating a homography based on a number of matched lines between the two images (i.e., the analogue of the DLT algorithm for matched lines).

Answer Using the result of the previous question, the relation between the two sets of lines is lΦ−1 = l0 or, re-arranging by post-multiplying by Φ and taking the transpose we have lT = ΦT lT .

We can now use the relation lT × ΦT lT = 0. to create two independent equations for the elements of Φ and proceed as in the DLT algorithm.

Problem 15.4 A conic (see problem 14.6) is defined by x̃T Cx̃ = 0, where C is a 3 × 3 matrix. If the points in the image undergo the transformation x̃0 = Tx̃, then what is the equation of the transformed conic?

Answer For the new conic, it must be true that T

x̃0 C0 x̃0 = 0, Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

111 and so substituting in we get x̃T TT C0 Tx̃0 = 0, By inspection, we can see that to make this true, we can choose C0 = T−T CT−1 .

Problem 15.5 All of the 2D transformations in this chapter (Euclidean, similarity, affine, projective) have 3D equivalents. For each class write out the 4 × 4 matrix that describes the 3D transformation in homogeneous coordinates. How many independent parameters does each model have?

Answer Euclidean (6 parameters): 

 τx τy   τz  1

ω11 ω21 Teuc =  ω31 0

ω12 ω22 ω32 0

ω13 ω23 ω33 0



ω12 ρω22 ω32 0

ω13 ω23 ρω33 0



φ11 φ21  Taf f =  φ31 0

φ12 φ22 φ32 0

φ13 φ23 φ33 0

 φ14 φ24   φ34  1

 φ11 φ21 Tproj =  φ31 φ41

φ12 φ22 φ32 φ42

φ13 φ23 φ33 φ43

 φ14 φ24   φ34  φ44

Similarity (6 parameters): ρω11  ω21  Tsim =  ω31 0

 τx τy   τz  1

Affine (12 parameters):

Projective (15 parameters):

Problem 15.6 Devise an algorithm to estimate a 3D affine transformation based on two sets of matching 3D points. What is the minimum number of points required to get a unique estimate of the parameters of this model?

Answer The three-dimensional affine transform has 12 parameters. We get three parameters per point for the x, y, and z coordinates, and so we need a minimum of four matches equations.

The equation is given by Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

112

15



φ11 φ21 Taf f =  φ31 0

φ12 φ22 φ32 0

φ13 φ23 φ33 0

Models for transformations

 φ14 φ24   φ34  1

The learning problem can be stated as

Φ̂, τ̂

=

" I # X T argmin (xi − aff[wi , Φ, τ ]) (xi − aff[wi , Φ, τ ]) Φ,τ

=

i=1

" I # X T argmin (xi − Φwi − τ ) (xi − Φwi − τ ) . Φ,τ

(15.1)

i=1

To solve this problem, observe that we can re-express Φwi + τ as a linear function of the unknown elements of Φ and τ





ui Φwi + τ =  0 0

vi 0 0

wi 0 0

1 0 0

0 ui 0

0 vi 0

0 wi 0

0 1 0

0 0 ui

0 0 vi

0 0 wi

 φ11 φ12    φ13     τx     φ21   0  φ22   0  φ23  = Ai b, (15.2)  1   τy    φ31    φ32    φ33  τz

where Ai is a 3×12 matrix based on the point wi , and b contains the unknown parameters.

The problem can now be written as

b̂ = argmin b

" I X

# (xi − Ai b)T (xi − Ai b) ,

(15.3)

i=1

which is a linear least squares problem and can be solved easily (appendix C.7.1).

Problem 15.7 A 1D affine transformation acts on 1D points x as x0 = ax + b. Show that the ratio of two distances is invariant to a 1D affine transformation so that I=

x1 − x2 x0 − x02 = 10 . x2 − x3 x2 − x03

Answer Starting with the right hand side we have Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

113

x01 − x02 x02 − x03

= = =

ax1 + b − ax2 − b ax2 + b − ax3 − b ax1 − ax2 ax2 − ax3 x1 − x2 x2 − x3

as required.

Problem 15.8 A 1D projective transform acts on 1D points x as x0 = (ax + b)/(cx + d).

Show that the cross-ratio of distances is invariant to a 1D projective transformation so that I=

(x3 − x1 )(x4 − x2 ) (x0 − x01 )(x04 − x02 ) . = 03 (x3 − x2 )(x4 − x1 ) (x3 − x02 )(x04 − x01 )

Answer Starting with the right hand side, we have

I

=

=

=

(x03 − x01 )(x04 − x02 ) (x03 − x02 )(x04 − x01 )

ax3 +b ax4 +b 1 +b 2 +b − ax − ax cx3 +d cx1 +d cx4 +d cx2 +d

ax3 +b ax4 +b 2 +b 1 +b − ax − ax cx3 +d cx2 +d cx4 +d cx1 +d ((ax3 + b)(cx1 + d) − (ax1 + b)(cx3 + d)) ((ax4 + b)(cx2 + d) − (ax2 + b)(cx4 + d)) ((ax3 + b)(cx2 + d) − (ax2 + b)(cx3 + d)) ((ax4 + b)(cx1 + d) − (ax1 + b)(cx4 + d))

Now, consider the first term in the denominator. We have ((ax3 +b)(cx1 +d)−(ax1 +b)(cx3 +d)) = acx3 x1 +bcx1 +bd+adx3 −acxx x1 −bcx3 −bc−adx1 = bcx1 − adx1 + adx3 − bcx3 = (ad − bc)(x3 − x1 ) Applying this same method to all four terms, we get

I

= = =

((ax3 + b)(cx1 + d) − (ax1 + b)(cx3 + d)) ((ax4 + b)(cx2 + d) − (ax2 + b)(cx4 + d)) ((ax3 + b)(cx2 + d) − (ax2 + b)(cx3 + d)) ((ax4 + b)(cx1 + d) − (ax1 + b)(cx4 + d)) (ad − bc)(x3 − x1 )(ad − bc)(x4 − x2 ) (ad − bc)(x3 − x2 )(ad − bc)(x4 − x1 ) (x3 − x1 )(x4 − x2 ) (x3 − x2 )(x4 − x1 )

as required.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

114

15

Models for transformations

Problem 15.9 Show that equation 15.36 follows from equation 15.35.

Answer We start with the premise x̃ × Φw̃ = 0 We can write Φw̃ as a vector  φ•1 w̃ Φw̃ = φ•2 w̃ φ•3 w̃ 

where φ•,k represents the kth row of the matrix Φ.

Now we take the cross product to give

x̃ × Φw̃

i x

=

=

=

j y

k 1

φ•1 w̃ φ•2 w̃ φ•3 w̃   yφ•3 w̃ − φ•2 w̃  φ•1 w̃ − xφ•3 w̃  xφ•2 w̃ − yφ•1 w̃   y(φ31 u + φ32 v + φ33 ) − (φ21 u + φ22 v + φ23 )  (φ11 u + φ12 v + φ13 ) − x(φ31 u + φ32 v + φ33 )  = 0. x(φ21 u + φ22 v + φ23 ) − y(φ11 u + φ12 v + φ13 )

as required.

Problem 15.10 A camera with intrinsic matrix λ and extrinsic parameters Ω = I, τ = 0 takes an image and then rotates to a new position Ω = Ω1 , τ = 0 and takes a second image. Show that the homography relating these two images is given by Φ = ΛΩ1 Λ−1 .

Answer For the first image, we have

λ1 x˜1

=

Λ[I0]w̃

=

Λw

Similarly, for the second image we have

λ2 x˜2

=

Λ[Ω0]w̃

=

ΛΩw

From the first image we get the relation Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

115

w = λ1 Λ−1 x1 We now substitute into the relation in the second camera to get λ3 x2 = ΛΩΛ−1 x1 as required.

Problem 15.11 Consider two images of the same scene taken from a camera that rotates, but does not translate between taking the images. What is the minimum number of point matches required to recover a 3D rotation between two images taken using a camera where the intrinsic matrix is known?

Answer There are three unknowns in a 3D rotation matrix. We get two constraints per point, so we need two points to estimate the rotation.

Problem 15.12 Consider the problem of computing a homography from point matches that include outliers. If 50% of the initial matches are correct, how many iterations of the RANSAC algorithm would we expect to have to run in order to have a 95% chance of computing the correct homography?

Answer P r(all four correct) = 0.54 P r(don’t compute correct) = (1 − 0.54 )N = 1 − 0.95 Solving for N we get N=

log[0.05] = 46.4 log[1 − 0.54 ]

We must run it at least 47 times to have a 95% chance.

Problem 15.13 A different approach to fitting transformations in the presence of outliers is to model the uncertainty as a mixture of two Gaussians. The first Gaussian models the image noise, and the second Gaussian, which has a very large variance, accounts for the outliers. For example, for the affine transformation we would have

P r(x|w) = λNormx aff[w, Φ, τ ], σ 2 I + (1 − λ)Normx aff[w, Φ, τ ], σ02 I + where λ is the probability of being an inlier, σ 2 is the image noise and σ02 is the large variance that accounts for the outliers. Sketch an approach to learning the parameters σ 2 , Φ, τ , and λ of this model. You may assume that σ02 is fixed. Identify a possible weakness of this model.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

116

15

Models for transformations

Answer This problem is suitable for the EM algorithm. We associated a binary hidden variable with each pair of matches indicating whether they belong to the foreground or the outlier model. In the E-Step we compute the probability distribution over each of the values of the hidden variable. In the M-Step we re-estimate the model parameters based on these assignments. One possible disadvantage is that the fitting process is prone to local minima.

Problem 15.14 In the description of how to compute the panorama (section ??), it is suggested that we take each pixel in the central image and transform it into the other images and then copy the color. What is wrong with the alternate strategy of taking each pixel from the other images and transforming them into the central image?

Answer To see why this is a bad idea, consider doing this for the simple case where new image was expanded by a factor of two in each dimension. Now there are four times as many pixels as before, and so if we start with the old image and transform each pixel in turn, 3/4 of the new pixels will remain empty! Using the inverse transformation guarantees that a value is placed at each new pixel.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

Chapter 16

Multiple cameras Problem 16.1 Sketch the pattern of epipolar lines on the images in figure 16.3a.

Answer The pattern of epipolar lines is shown in figure 16.1. The epipole in the first camera is at infinity and so the lines are parallel. However, the epipole in the second camera is at a finite position (where the plane strikes the line between the two camera centers) and so the epipolar lines converge.

Problem 16.2 Show that the cross product relation can be written in terms of a matrix multiplication so that 

0 a × b =  a3 −a2

−a3 0 a1

  a2 b1 −a1  b2  .

0 b3

Answer

a×b

= = =

=

i a1 b1

j a2 b2

k a3 b3

i(a2 b3 − a3 b2 ) + j(a3 b1 − a1 b3 ) + k(a1 b2 − a2 b1 )   a2 b3 − a3 b2 a3 b1 − a1 b3  a1 b2 − a2 b1    0 −a3 a2 b1  a3 0 −a1  b2  . −a2 a1 0 b3

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

118

16

Multiple cameras

Ray 2 Ray 1

Camera plane 1

Camera plane 2

Optical center 1

Optical center 2 Figure 16.1 Geometry for problem 16.1.

Problem 16.3 Consider figure 16.2. Write the direction of the three 3D vectors O1 O2 , O1 w, and O2 w in terms of the observed image positions x1 ,x2 and the rotation Ω and translation τ between the cameras. The scale of the vectors is unimportant.

The three vectors that you have found must be coplanar. The criterion for three 3D vectors a, b, c being coplanar can be written as a.(b × c) = 0. Use this criterion to derive the essential matrix.

Answer From the perspective of camera 2, the vector O1 w = κ1 Ωx1 , the vector O1 O2 = κ2 τ and the vector O2 w = x2 . Combining these using the coplanarity condition we get

xT2 (τ × Ωx1 ) = 0 as required.

Problem 16.4 A clueless computer vision professor writes: “The essential matrix is a 3×3 matrix that relates image coordinates between two images of the same scene. It contains 8 independent degrees of freedom (it is ambiguous up to scale). It has rank 2. If we know the intrinsic matrices of the two cameras, we can use the essential matrix to recover the rotation and translation between the cameras exactly.”

Edit this statement to make it factually correct.

Answer “The essential matrix is a 3×3 matrix that relates image coordinates between two images of the same scene. It contains 5 independent degrees of freedom (it is ambiguous up to scale). It has rank 2. If we know the intrinsic matrices of the two cameras, we can use the Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

119 essential matrix to recover the rotation between the cameras exactly and the translation up to an unknown scale factor”

Problem 16.5 The essential matrix relates points in two cameras so that

xT2 Ex1 = 0 is given by 

0 E= 0 −10

0 0 0

 10 0 .

0

What is the epipolar line in image 2 corresponding to the point x1 = [1, −1, 1]? What is the epipolar line in image 2 corresponding to the points x1 = [−5, −2, 1]? Determine the position of the epipole in image 2. What can you say about the motion of the cameras?

Answer The epipolar line corresponding to point x1 = [1, −1, 1] is 

0 l2 =  0 −10

  10 1

0  −1 = 10 0 1

0 0 0

0

−10

The epipolar line corresponding to point x1 = [−5, −2, 1] is 

0 l20 =  0 −10

0 0 0

  10 5

0  −2 = 100 0 1

−5

The epipole must be at [0, 1, 0] as this is the null space of this matrix.

The first line is x = 1 and the second line is x = 0.5. These are both parallel vertical lines. This means that the cameras point in the same direction and have moved vertically relative to one another orthogonal to this direction.

Problem 16.6 Show that we can retrieve the essential matrix by multiplying together the expressions from the decomposition (equations 16.19) as E = τ × Ω.

Answer τ×

=

Ω

=

ULWUT UW−1 VT

=

ULWW−1 VT

=

ULVT

=

E

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

120

16

Multiple cameras

Problem 16.7 Derive the fundamental matrix relation: −1 x̃T2 Λ−T 2 EΛ1 x̃1 = 0.

Answer We will assume that the world coordinate system is centered on the first camera so that the extrinsic parameters (rotation and translation) of the first camera are {I, 0}. The second camera may be in any general position {Ω, τ }. We will further assume that camera 1 has intrinsic matrix Λ1 an that camera 2 has intrinsic matrix Λ2 .

λ1 x̃1

=

Λ1 [I, 0]w̃

λ2 x̃2

=

Λ2 [Ω, τ ]w̃.

where x̃1 is the observed position in the first camera, x̃2 is the observed position in the second camera, and both are expressed in homogeneous coordinates.

These relations simplify to

λ1 x̃1

=

Λ1 w

λ2 x̃2

=

Λ2 Ωw + τ .

Substituting one equation into the other yields λ2 x̃2 = λ1 Λ2 ΩΛ−1 1 x̃1 + τ .

We now premultiply by Λ2−1 to get −1 λ2 Λ−1 2 x̃2 = λ1 ΩΛ1 x̃1 + τ .

We first take the cross product of both sides with the translation vector τ . This removes the last term as the cross product of any vector with itself is zero. Now we have −1 λ2 τ × Λ−1 2 x̃2 = λ1 τ × ΩΛ1 x̃1 .

Then we take the inner product of both sides with Λ−1 2 x̃2 . The left hand side disappears −1 since τ × Λ−1 2 x̃2 must be perpendicular to Λ2 x̃2 , and so we have −1 x̃T2 Λ−1 2 τ × ΩΛ1 x̃1 = 0,

as required.

Problem 16.8 I intend to compute the fundamental matrix using the eight-point algorithm.

Unfortunately, my data set is polluted by 30% outliers. How many iterations of the RANSAC algorithm will I need to run to have a 99% probability of success (i.e., computing the fundamental matrix from eight inliers at least once)? How many iterations will I need if I use an algorithm based on seven points?

Answer Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

121 The probability of success is 0.78 . So the probability of failure is 1 − 0.78 . The probability of failing N times in a row is (1 − 0.78 )N . So our criterion should be that (1 − 0.78 )N = 1 − 0.99 Re-arranging this equation we get N = log[1 − 0.99]/ log[1 − 0.78 ] which gives N = 77.55, so the minimum number of iterations is 78 to be sure.

If we only needed 7 points then we would have N = log[1 − 0.99]/ log[1 − 0.77 ] which gives N = 53.58, so the minimum number of iterations if 54.

Problem 16.9 We are given the fundamental matrix F13 relating images 1 and 3 and the fundamental matrix F23 relating images 2 and 3. I am now given corresponding points x1 and x2 in images 1 and 2, respectively. Derive a formula for the position of the corresponding point in image 3.

Answer The epipolar line in image 3 due to the point in image 1 is given by l = F13 x̃1 . The epipolar line in image 3 due to the point in image 1 is given by l0 = F23 x̃2 .

The point in the third image must be where these two epipolar lines meet. By the logic of problem 14.4, this can be found by taking the cross product and so the final expression is x̃3 = F13 x̃1 × F23 x̃2

Problem 16.10 Tomasi-Kanade factorization. In the orthographic camera (figure 14.3), the projection process can be described as

x y

φx 0

π11 π21

=

=

 ω ω12 ω13 γ δx  11 ω21 ω22 ω23 φy δy 0 0 0   u 0 π12 π13   τ v + x0 , π22 π23 τy w

   u τx   v τy   w  1 1

or in matrix form x = Πw + τ 0 .

Now consider a data matrix X containing the positions {xij }IJ i,j=1 of J points as seen in I images so that Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

122

16

Multiple cameras

Camera plane 1

Camera plane 2

Optical center 1

Optical center 2 Figure 16.2 Figure for problem 16.3.

a)

b)

Ray 2 Ray 1

Camera plane 1

Camera plane 2

Optical center 1

Optical center 2

Figure 16.3 a) Figure for problem 16.1. c) Figure for problem 16.11. Gray regions represent non-zero entries in this portrait Jacobian matrix.



x11 x21  X= .  .. xI1

x12 x22 .. . xI2

... ... .. . ...

 xiJ x2J   ..  .  xIJ ,

and where xij = [xij , yij ]T .

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

123 (i) Show that the matrix X can be written in the form X = PW + T where P contains all of the I 3 × 2 projection matrices {Πi }Ii=1 , W contains all of the J 3D world positions {wj }Jj=1 and T contains the translation vectors {τ 0i }Ii=1 . (ii) Devise an algorithm to recover the matrices P, W and T from the measurements X. Is your solution unique?

Answer i) We can define the matrices 

P

=

W

=

T

=

 π1 π 2     .   ..  πI

w1 w2 . . . wJ   τ1 τ1 ... τ1 τ 2 τ 2 . . . τ 2    .. ..   ..  .  . . . . .   .. τI τI . τi

and then the condition X = PW + T will be fulfilled. ii) A simple algorithm is to re-arrange the equation to get X − T = PW and then observe that PW is the product of a 2I × 3 matrix and a 3 × J matrix. To find the best approximation to these matrices, we can compute the svd [U, L, V] = svd[X − T] and then set all but the first 3 values of L to zero. We can now choose P = UL and W = VT .

Problem 16.11 Consider a Jacobian that has a structure of non-zero entries as shown in figure 16.3b. Draw an equivalent image that shows the structure of the non-zero entries in the matrix JT J. Describe how you would use the Schur complement relation to invert this matrix efficiently.

Answer The structure is shown in figure 16.4. The Schur relation could be used to exploit divide the matrix into four quadrants and compute the full inverse via two smaller inversions.

The first would be to invert the block diagonal matrix in the bottom right quadrant. The second would be to invert a full matrix that was the size of the upper right quadrant.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

124

16

Multiple cameras

= Figure 16.4 Solution to problem 16.11.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

Chapter 17

Models for shape Problem 17.1 A conic is defined as the set of points where

x

 α 1 β γ

y

  γ x  y  = 0, ζ 1

β δ

or x̃T Cx̃ = 0.

Use MATLAB to draw the 2D function x̃T Cx̃ and identify the set of positions where this function is zero for the following matrices:  3 C1 = 0 0

 0 0 −1

0 2 0



0 C 2 = 0 1

 1 0 −2

0 0 0



−1 C3 =  0 0

 0 1 .

0

0 0 1

Answer These conics are shown in figure 17.1. The three examples show the cases where the conic takes the form of an ellipse, a straight line and a parabola respectively.

1 -0

10

10

8

8

8

6

6

6

4

4

4

2

2

2

0

0

0

+2

−2

−2

−4

−4

−4

−6

−6

−6

−8

−8

−10 −10

−8

−6

−4

−2

0

2

4

6

8

10

−10 −10

−8

−8

−6

−4

−2

0

2

4

6

8

10

−10 −10

−8

−6

−4

−2

0

2

4

6

8

10

Figure 17.1 Solution to problem 17.1.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

126

17

Models for shape

Problem 17.2 Devise an efficient algorithm to compute the distance transform. The algorithm should take a binary image and return at each pixel the city block distance to the nearest non-zero element of the original image. The city block distance d between pixels (x1 , y1 ) and pixel (x2 , y2 ) is defined as d = |x1 − x2 | + |y1 − y2 |.

Answer A simple algorithm proceeds as follows. First move through each pixel from top-left to bottom right of the image, computing the city block distance to the nearest pixel that is above and to the left of this pixel. This can be computed recursively. Then repeat this procedure in the other four directions to get the distances to the nearest pixel above and to the right, below and to the left, and below and to the right. Then take the minimum of these four values at each point to compute the city block distance. Other algorithms are available!

Problem 17.3 Consider a prior that is based in the curvature term: curve[w, n] = −(wn−1 − 2wn + wn+1 )T (wn−1 − 2wn + wn+1 ).

If landmark point w1 = [100, 100] and landmark point w3 is at position w3 = [200, 300], what position w2 will minimize the function Curve[w, 2]?

Answer This prior penalizes curvature and will give the best answer when the curve is a straight line. Consequently, the best position w2 = [150, 100].

Problem 17.4 If the snake as described in section 17.2 is initialized in an empty image, how would you expect it to evolve during the fitting procedure?

Answer The points will tend to be the same distance from each other and the curvature will tend to decrease. The former factor will tend to make the contour form a circle. The latter factor (rather subtly) makes the circle smaller; the measure of curvature is not normalized by the scale and so it will tend to make the circle smaller.

Problem 17.5 The spacing element of the snake prior 17.5 encourages all of the control points of the snake to be the equidistant. An alternative approach is to give the snake a tendency to shrink (so that it collapses around objects). Write out an alternative expression for the spacing term that accomplishes this goal.

Answer The following criterion penalizes any difference between adjacent points and so creates a shrinking tendency.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

127

space[w, n] = −(wn − wn−1 )T (wn − wn−1 ),

Problem 17.6 Devise a method to find the ‘best’ weight vector h given a new vector w and the parameters {µ, Φ, σ 2 } of the PPCA model (see figure ??).

Answer This can be accomplished by finding the posterior distribution over the weight vector, which is given by

P r(h|w)

=

P r(w|h)P r(h) P r(w)

=

Normw [(ΦT Φ + σ 2 I)−1 ΦT (w − µ), σ 2 (ΦT Φ)−1 ]

This was calculated using the relations in sections 5.6 and 5.7 (see also problem 7.9).

One definition of the ’best’ weight vector is to take the maximum a posteriori estimate, which is simply the mean of this distribution.

Problem 17.7 Show that if the singular value decomposition of a matrix W can be written as W = ULVT , then it follows that WWT

=

UL2 UT

T

=

VL2 VT .

W W

Answer

WWT

=

ULVT (ULVT )T = ULVT VLU2 = UL2 UT

WT W

=

(ULVT )T ULVT = VLUT ULV2 = VL2 V .

T

Problem 17.8 Devise a method to learn the PPCA model using the EM algorithm, giving details of both the E- and M-steps. Are you guaranteed to get the same answer as the method based on the SVD? Answer In the E-step we compute the posterior distribution over the hidden variable: Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

128

17

P r(h|w)

Models for shape

=

P r(w|h)P r(h) P r(w)

=

Normw [(ΦT Φ + σ 2 I)−1 ΦT (w − µ), σ 2 (ΦT Φ)−1 ]

This was calculated using the relations in sections 5.6 and 5.7 (see also problem 7.9).

In the M-Step, we update the EM criterion with respect to the parameters (see also problem 7.10) PI µ̂

=

Φ̂

=

i=1 xi

I ! !−1 I I X X T T (xi − µ̂)E[hi ] E[hi hi ] i=1

σˆ2

=

i=1

I h h ii 1 X Trace diag (xi − µ̂)(xi − µ̂)T − Φ̂E[hi ](xT − µ) , ID i=1

You will not in general get the same values in the matrix Φ - the SVD guarantees that the principal components will be ordered in terms of their magnitude, whereas if we learn them with the EM algorithm, no particular ordering is guaranteed and they may be permuted by an arbitrary rotation matrix.

Problem 17.9 Show that the maximum a posteriori solution for the hidden weight variable h is as given in equation 17.32.

Answer

ĥ

=

argmax h

=

argmin h

" N X

# 2

2

T

− (yn − b − Aµn − AΦn h, Ψ]) /σ − log[h h]

(17.1)

n=1

" N X

# T

2

T

(yn − b − Aµn − AΦn h, Ψ]) (yn − b − Aµn − AΦn h, Ψ]) σ + log[h(17.2) h]

n=1

This now has the form of a regularized least squares problem. The proof proceeds very similarly to the standard least squares problem (see appendix C.7.1). We take the derivative with respect to the unknown quantity h equate the result to zero and re-arrange.

Problem 17.10 You are given a set of 100 male faces and 100 female faces. By hand you mark 50 landmark points on each image. Describe how to use this data to develop a generative approach to gender classification based on shape alone. Describe both the training process and how you would infer the gender for a new face that does not contain landmark points.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

129

Answer We build a separate statistical shape model for male and female faces based on probabilistic PCA. For a new face, we fit both the male and female model. We can calculate the likelihood for the each model as

P r(w|m)

=

2 Normw [µm , Φm ΦTm + σm I]

where m = 0 denotes a female face is present and m = 1 denotes a male face. See problem 7.8 for more information on the derivation of this expression. These likelihoods can be combined with suitable priors and the posterior can be computed using Bayes’ rule.

Problem 17.11 Imagine that we have learned a point distribution for the shape of the human face. Now we see a new face where everything below the nose is occluded by a scarf. How could you exploit the model to estimate both the positions of the landmark points in the top half of the face and the landmark points in the (missing) bottom half of the face?

Answer The marginal density of the landmark points after compensating for the overall translation is P r(w) = Normw [µ, ΦΦT + σ 2 I] If part of the data is missing, we can estimate it from the remaining data by computing the conditional distribution of the missing parts of the normal distribution given the observed parts (see sections 5.5 and 7.9.5).

Problem 17.12 An alternative approach to building a nonlinear model of shape is to use a mixture model. Describe an approach to training a statistical shape model based on the mixture of probabilistic principal component analyzers. How would you fit this model to a new image?

Answer The new model of shape has equations:

P r(wi |hi , µ, Φ, σ 2 , c)

=

Normwi [µc + Φc hi , σc2 I]

P r(hi )

=

Normhi [0, I]

P r(c) = Catc [λ] where c denotes the index of the cluster used. The general strategy to learn this model might involve two nested instances of the EM algorithm. At the top-level, we alternate between assigning the responsibilities of each of the models for each data point and then updating the parameters of each model based on these weights. The parameters of each Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

130

17

Models for shape

model might themselves depend on a loop we update the parameter of each PPCA model based on the assignment of the data to each cluster.

Problem 17.13 One way to warp one image to another is to implement a piecewise affine warp. Assume that we have a number of points in image 1 and their corresponding points in image 2. We first triangulate each set of points in the same way. We now represent the position x1 in image 1 as a weighted sum of the three vertices of the triangle a1 , b1 , c1 that it lies in so that x1 = αa1 + βb1 + γc1 , where the weights are constrained to be positive with α + β + γ = 1. These weights are known as barycentric coordinates.

To find the position in the second image, we then compute the position relative to the three vertices a2 , b2 , c2 of the warped triangle so that x2 = αa2 + βb2 + γc2 .

How can we compute the weights α, β, γ? Devise a method to warp the whole image in this manner.

Answer We can rewrite the mappings above in matrix form as   aT α x = bT  β  , γ cT 

and then compute the unknowns by inverting this system of equations.

A strategy for warping the entire image is to triangulate the original image using Delaunay triangulation of the landmark points. These triangles are transferred to the second image in which the landmark points have been moved. We now have a set of connected triangles in the first and the second image. For each pixel in the second image, we find which triangle it belongs, to and compute the barycentric coordinates. From these, we can figure out where this point is in the first image and sample the intensity using bilinear interpolation or similar.

Problem 17.14 Consider an ellipsoid in 3D space that is represented by the quadric w̃T

A bT

b w̃ = 0, c

where A is a 3 × 3 matrix, b is a 3 × 1 vector, and c is a scalar.

For a normalized camera we can write the world point w̃ in terms of the image point x̃ as w̃ = [x̃T , s]T where s is a scaling factor that determines the distance along the ray x̃. (i) Combine these conditions to produce a criterion that must be true for an image point x̃ to lie within the projection of the conic.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

131 (ii) The edge of the image of the conic is the locus of points for which there is a single solution for the distance s. Outside the conic there is no real solution for s and inside it there are two possible solutions corresponding to the front and back face of the quadric. Use this intuition to derive an expression for the conic in terms of A, b and c. If the camera has intrinsic matrix Λ, what would the new expression for the conic be?

Answer i) Combining these equations together, we have cs2 + 2bT x̃s + x̃T Ax̃ = 0 ii) This is a quadratic in s which has a unique solution when the discriminant is zero, so that x̃T (bbT − cA)x = 0 This is a conic with parameters C = bbT − cA. If the camera has intrinsic matrix Λ, then the points w now have the form w̃ = [Λ−1 x̃T , s]T .

Following this through the equations we get a conic with parameters C = Λ−T (bbT − cA)Λ−1 .

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

132

17

Models for shape

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

Chapter 18

Models for style and identity Problem 18.1 Prove that the posterior distribution over the hidden variable in the subspace identity model is as given in equation 18.9.

Answer

P r(hi |xi• )

∝

J Y

P r(xij |hi )P r(hi )

j=1

∝

J Y

Normxij [µ + Φhi , Σ]Normhi [0, I]

j=1

First we apply the change in variables relation (section 5.7) which gives us

q̂(hi )

∝

J Y

Normxij [µ + Φhi , Σ]Normhi [0, I]

j=1

=

J Y

Normhi [(ΦT Σ−1 Φ)−1 ΦT Σ−1 (xij − µ), (ΦT Σ−1 Φ)−1 ]Normhi [0, I]

j=1

Now we apply the product of normal distributions rule (section 5.6) which gives us

q̂(hi )

∝

Normhi [(JΦT Σ−1 Φ)−1 ΦT Σ−1

J X (xij − µ), (JΦT Σ−1 Φ)−1 ]Normhi [0, I] j=1

Finally, we apply the product of normal distributions rule (section 5.6) again which gives us

q̂(hi )

∝

Normhi [(JΦT Σ−1 Φ + I)−1 ΦT Σ−1

J X (xij − µ), (JΦT Σ−1 Φ)−1 ] j=1

as required. Note that the constant must be one to make the posterior a valid probability distribution.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

134

18

Models for style and identity

Problem 18.2 Show that the M-step updates for the subspace identity model are as given in equation 18.11.

Answer The log likelihood is given by

L

=

=

" I J #

XX 1 T −1 − E 2D log 2π + 2 log[|Σ|] + (xij − µ − Φhi ) Σ (xij − µ − Φhi ) 2 i=1 j=1 " I J XX 1 2D log 2π + 2 log[|Σ|] + µT Σ−1 µ + xTij Σ−1 xih + hi ΦT Σ−1 Φhi − E 2 i=1 j=1 i −2µT Σ−1 xij − 2µT Σ−1 Φhi − 2xTij Σ−1 Φhi

Taking the derivative with respect to µ we get

∂L ∂µ

=

=

# " I J XX

1 −1 −1 −1 2Σ µ − 2Σ xij − 2Σ Φhi − E 2 i=1 j=1 I X J X

−Σ−1 µ + 2Σ−1 xi = 0

i=1 j=1

which we can re-arrange to get PI µ̂ =

i=1

PJ

j=1 xi

IJ

Taking the derivative with respect to Φ we get

∂L ∂Φ

=

=

" I J #

X X −1 1 − E 2Σ Φhi hTi − 2Σ−1 µhTi − 2Σ−1 xij hTi 2 i=1 j=1 I X J

X 2Σ−1 ΦE[hi hTi ] − 2Σ−1 µE[hTi ] − 2Σ−1 xij E[hTi ] = 0 i=1 j=1

This can be re-arranged to get

Φ̂ =

I X J X (xij − µ)E[hi ]T i=1 j=1

!

J

I X

!−1 E[hi hTi ]

i=1

as required.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

135 Problem 18.3 Develop a closed form solution for learning the parameters {µ, Φ, σ 2 } of a subspace identity model where the noise is spherical: P r(xij ) = Normxij [µ, ΦΦT + σ 2 I].

Hint: Assume you have exactly J = 2 examples of each of the I training images and base your solution on probabilistic PCA. Answer I’m still working on this one! Should be hints in Ioffe 2006 though.

Problem 18.4 In a face clustering problem, how many possible models of the data are there with 2,3,4,10, and 100 faces?

Answer In each case, the answer is the number of partitions of a set containing N faces. It is also known as the Bell number. For 2,3,4,10 and 100 faces, there are respectively 2,5,15, 115975 and 101 15 possible models.

Problem 18.5 An alternative approach to face verification using the identity subspace model is to compute the probability of the observed data x1 and x2 under the models: P r(x1 , x2 |w = 0)

=

P r(x1 )P r(x2 )

P r(x1 , x2 |w = 1)

=

P r(x1 )P r(x2 |x1 ).

Write down expressions for the marginal probability terms P r(x1 ), P r(x2 ) and the conditional probability P r(x2 |x1 ). How could you use these expressions to compute the posterior P r(w|x1 , x2 ) over the world state?

Answer The marginal probabilities are given by the normal distributions

P r(x1 )

=

Normx1 [µ, ΦΦT + Σ]

P r(x2 )

=

Normx2 [µ, ΦΦT + Σ]

An expression for the conditional probability can be computed from the joint distribution

P r(x1 , x2 )

=

Norm[x1 ,x2 ]

µ Φ+Σ , µ Φ

Φ Φ+Σ

using the result in section 5.5 of the book i h P r(x1 |x2 ) = Normx1 µ+ΦΦT (ΦΦT + Σ)−1 (x2 −µ), ΦΦT −ΦΦT (ΦΦT + Σ)−1 ΦΦT

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

136

18

Models for style and identity

The posterior distribution is computed using Bayes’ rule as usual P r(w|x1 , x2 ) =

P r(x1 , x2 |w)P r(w) P r(x1 , x2 )

Problem 18.6 Propose a version of the subspace identity model that is robust to outliers in the training data.

Answer One possibility is to base the model on the t-distribution, so that the final density is given by P r(xij ) = Studxij [µ, ΦΦT + σ 2 I, ν]. where ν is the degrees of freedom.

Problem 18.7 Moghaddam et al. (2000) took a different probabilistic approach to face verification. They took the difference x∆ = x2 − x1 and modeled the likelihoods of this vector P r(x∆ |w = 0) and P r(x∆ |w = 1) when the two faces match or don’t. Propose expressions for these likelihoods and discuss learning and inference in this model. Identify one possible disadvantage of this model.

Answer The likelihoods of the difference image in each situation can be described as factor analyzers 2 P r(xδ |w) = Normxδ [µw , Φw ΦTw + σw I].

Inference can be achieved using Bayes’ rule P r(w|xδ ) =

P r(xδ |w)P r(w) P r(xδ )

where P r(w) is a Bernoulli distribution that assigns a prior probability to each of the two world states.

Problem 18.8 Develop a model that combines the advantages of PLDA and the asymmetric bilinear model; it should be able to model the within-individual covariance with a subspace, but also be able to compare data between disparate styles. Discuss learning and inference in your model.

Answer The model would have the form P r(xijk |hi , wi , s = k) = Normxijk [µk + Φk hi + Ψk wi , Σk ] Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

137 As in the bilinear model there is one set of parameters {µ, Φ, Ψ, Σ} for each style. However, as in the PLDA model, there is also a continuous style variable, which we have denoted by wi .

Learning proceeds using the EM algorithm - In the E-Step, we create a compound system containing all of the data for individual i and invert this to find a probability distribution over the hidden variables hi and wi . In the M-Step we update each set of parameters {µk , Φk , Ψk , Σk } based on the images that were in the kth style. Inference can be done in the same style as before by forming a compound set of equations for all of the data and using the standard formula. More details can be found in Prince and Elder (2007).

Problem 18.9 In the asymmetric bilinear model, how would you infer whether the style of two examples is the same or not, regardless of whether the images matched?

Answer This is done with Bayes’ rule. Let’s denote the variable t as the indicator for the style being different (t=1) or the same (t=2). We will compute the probability that the faces have the same style using Bayes’ rule

P r(t|x1 , x2 ) =

P r(x1 , x2 |t)P r(t) P r(x1 , x2 )

where we have defined a prior P r(t) over the possibility they are the same or different.

The associated likelihoods are

P r(x1 , x2 |t = 0) =

Z X 1

P r(x1 , x2 , w, s1 = 1, s2 = 1)dh +

w=0

P r(x1 , x2 |t = 1) =

Z X 1 w=0

Z X 1

P r(x1 , x2 , w, s1 = 2, s2 = 2)dh

w=0

P r(x1 , x2 , w, s1 = 1, s2 = 2)dh +

Z X 1

P r(x1 , x2 , w, s1 = 2, s2 = 1)dh

w=0

In other words, the likelihoods after integrating out the hidden variable, and marginalizing over whether the images match or not and taking into account the possible permutations of the styles matching and the styles not matching.x

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

138

18

Models for style and identity

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

Chapter 19

Temporal models Problem 19.1 Prove the Chapman-Kolmogorov relation: Z P r(wt |x1...t−1 )

P r(wt |wt−1 )P r(wt−1 |x1...t−1 ) dwt−1

= Z = =

Normwt [µp + Ψwt−1 , Σp ]Normwt−1 [µt−1 , Σt−1 ] dwt−1 Normwt [µp + Ψµt−1 , Σp + ΨΣt−1 ΨT ]

Answer It is possible to solve this integral directly, using the relations in chapter 5. However, taking on trust that the result will be a normal distribution, we can directly compute the mean and covariance. We first write the expression as a generative equation wt = µp + Ψwt−1 + p where p is noise with mean µt−1 and covariance Σt−1 .

The mean is given by E[wt ]

=

E[µp + Ψwt−1 + p ]

=

µp + Ψwt−1 + 0

=

µp + Ψwt−1

as required.

The covariance is given by E[(wt − wt )(wt − wt )T ]

=

E[Ψ(wt−1 − µt−1 ) + p )(Ψ(wt−1 − µt−1 ) + p )T ] T

T

=

E[Ψ[(wt−1 − µt−1 )(wt−1 − µt−1 ) ]Ψ

=

ΨΣt−1 ΨT + Σp

(19.1)

+ E[ p Tp ] + ΨT E[(wt−1 − µt−1 (19.2) ) Tp

as required.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

(19.3)

140

19

Temporal models

Problem 19.2 Derive the measurement incorporation step for the Kalman filter. In other words, show that P r(xt |wt )P r(wt |w1...t−1 , x1...t ) P r(x1...t ) Normxt [µm + Φwt , Σm ]Normwt [µ+ , Σ+ ] = P r(x1...t )

−1

T −1 −1 = Normwt Φ Σ−1 ΦT Σ−1 m Φ + Σ+ m (xt −µm ) + Σ+ µ+ ,

P r(wt |x1...t ) =

−1 −1 . ΦT Σ−1 m Φ + Σ+

Answer We begin by applying the change in variables relation (section 5.7) which gives us

P r(wt |x1...t )

∝

Normxt [µm + Φwt , Σm ]Normwt [µ+ , Σ+ ]

=

−1 T T −1 −1 Normwt [(ΦT Σ−1 Φ Σ−1 )]Normwt [µ+ , Σ+ ] m Φ) m (xt − µm , (Φ Σm Φ)

Now we apply the product of two normal distributions rule (section 5.6) which gives us

P r(wt |x1...t )

∝

−1 T T −1 −1 Normwt [(ΦT Σ−1 Φ Σ−1 )]Normwt [µ+ , Σ+ ] m Φ) m (xt − µm , (Φ Σm Φ)

=

−1 T −1 −1 Normwt [(ΦT Σ−1 (ΦT Σ−1 (xi − µ) + Σ−1 ] m Φ + Σ+ ) + µ+ ), (Φ Σm Φ + Σ+ )

as required. Note that the constant must be one to make the posterior a valid probability distribution.

Problem 19.3 Consider a Kalman filter type system where the prior based on the previous time step is a mixture of K Gaussians P r(wt |x1...t−1 ) =

K X

λk Normwt [µ+k , Σ+k ].

k=1

What will happen in the subsequent measurement incorporation step? What will happen in the next time update step?

Answer In the measurement incorporation step, the mixtures of K Gaussians prior will be multiplied by the Gaussian likelihood to create a posterior distribution that is also a mixture of Gaussians. In the temporal update step, the mixtures of Gaussians posterior will be convolved with the normal temporal update and this will create a new mixture of K Gaussians. So the mixture will propagate through time and retain it’s form.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

141 Problem 19.4 Consider a model where there are two possible temporal update equations, represented by state transition matrices Ψ1 and Ψ2 and the system periodically switches from one regime to the other. Write a set of equations that describe this model and discuss a strategy for max-marginals inference.

Answer Let’s denote the binary variable ct as representing the index of the regime that we are in.

A sensible model would be:

P r(ct |ct−1 = 0)

=

Bernct [λ0 ]

P r(ct |ct−1 = 1)

=

Bernct [λ1 ]

P r(wt |wt−1 , ct = 0)

=

Normwt [µp + Ψ1 wt−1 , Σp ]

P r(wt |wt−1 , ct = 1)

=

Normwt [µp + Ψ2 wt−1 , Σp ]

P r(xt |wt )

=

Normxt [µm + Φwt , Σm ]

where the parameters λ0 and λ1 represent the probability of choosing regime ct = 1 when the regime at the previous time was zero or one respectively.

There are a number of ways to perform inference in this model, but one possibility is to use a combination of dynamic programming to establish the most likely regime at each time and the Kalman filter (to establish the posterior marginals from which the max-marginal state can be established).

Problem 19.5 In a Kalman filter model, discuss how you would compute the joint posterior distribution P r(w1...T |x1...T ) over all of the unknown world states. What form will this posterior distribution take? In the Kalman filter we choose to compute the marginal posteriors instead. Why is this?

Answer The marginal posterior takes the form of a joint normal distribution between all of the variables. It can be solved for in closed form, by writing out the joint likelihood of all of the data conditioned on the hidden variables, and the joint prior distribution of all of the hidden variables and then using Bayes’ rule. However, in practice, we usually have a large number of measurements so this is a very large system of equations and solving it is computationally intensive.

Problem 19.6 Apply the sum-product algorithm (section ??) to the Kalman filter model and show that the result is equivalent to applying the Kalman filter recursions.

Answer The graph for the Kalman filter looks just like figure 11.9 from the main book. We can apply the rules in the same order as the worked example in section 14.4.

We start by passing a message mx1 →g1 from node x1 to the function node g1 . Using rule 2, this message is a delta function at the observed value x∗1 , so that Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

142

19

Temporal models

mx1 →g1 = δ[x∗1 ].

Now we pass a message from function g1 to node w1 . Using rule 3, we have Z

P r(x1 |w1 )δ[x∗1 ]dx1

=

mg1 →w1

=

P r(x1 = x∗1 |w1 )

=

N ormx1 [µm + Φwt , Σm ]

By rule 1, the message from node w1 to function g1,2 is simply the product of the incoming nodes, and since there is only one incoming node, this is just

mw1 →g12 =

=

P r(x1 = x∗1 |w1 )

=

N ormx1 [µm + Φwt , Σm ].

Assuming a uniform prior in the initial case, the marginal posterior will be proportional to this likelihood term P r(w1 |x1 ) ∝ N ormx1 [µm + Φwt , Σm ].

By rule 3, the message from function g1,2 to node w2 is computed as Z mg1,2 →w2

=

P r(w2 |w1 )P r(x1 = x∗1 |w1 )dw1

=

Normwt [µp + Ψµt−1 , Σp + ΨΣt−1 ΨT ]

=

Normwt [µ+ , Σ+ ],

which is clearly the Chapman-Kolmogorov relation.

Continuing this process, the messages from x2 to g2 and g2 to w2 are

mx2 →g2

=

δ[x∗2 ]

mg2 →w2

=

P r(x2 = x∗2 |w2 ),

and the message from w2 to g2,3 is mw2 →g2,3 = P r(x2 = x∗2 |w2 )Normwt [µ+ , Σ+ ].

This is the numerator of Bayes’ rule, so the result is proportional to the result of Bayes’ rule.

The forward pass of belief propagation is hence alternately applying the ChapmanKolmogorov equation and Bayes’ rule and is hence equivalent to the Kalman filter equations.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

143 Problem 19.7 Prove the Kalman smoother recursions:

µt|T

=

µt + Ct (µt+1|T − µ+|t )

Σt|T

=

Σt + Ct (Σt+1|T − Σ+|t )CTt ,

where Ct = Σt|t ΨT Σ−1 +|t .

Hint: It may help to examine the proof of the forward-backward algorithm for HMMs (section 11.4.2).

Answer TBD

Problem 19.8 Discuss how you would learn the parameters of the Kalman filter model given training sequences consisting of (i) both the known world state and the observed data, and (ii) just the observed data alone.

Answer When we know both the world state and the observed data, we can learn the parameters using a standard maximum likelihood approach. We learn the parameters of the measurement model Pr(xt |wt ) from the pairs {xt , wt }tt=1 and the prior model P r(wt |wt−1 ) from the pairs {wt , wt−1 }Tt=2 .

When the world states are not observed, they can be considered as hidden variables and the EM algorithm can be used. In the E-Step, we compute the marginal posterior distribution over the world states by running the Kalman filter and smoother. In the MStep, we update the parameters of the model using the expected means and covariances of these hidden variables.

Problem 19.9 In the unscented Kalman filter we represented a Gaussian with mean µ and covariance Σ with a set of weights ŵ[0]

=

µt−1

ŵ[j]

=

µt−1 +

ŵ[Dw +j]

r r =

µt−1 −

Dw 1/2 Σ ej 1 − a0 t−1

for all j ∈ {1 . . . Dw }

Dw 1/2 Σ ej 1 − a0 t−1

for all j ∈ {1 . . . Dw },

with associated weights

aj

=

1 − a0 .

2Dw

Show that the mean and covariance of these points are indeed µt−1 and Σt−1 so that Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

144

19

µt−1

=

2D w X

Temporal models

aj ŵ[j]

j=0

Σt−1

=

2D w X

aj (ŵ[j] − µt−1 )(ŵ[j] − µt−1 )T .

j=0

Answer Dealing with the mean first, we have:

2D w X

aj ŵ

[j]

=

a0 µt−1 +

j=0

D X 1 − a0 j=1

+

D X 1 − a0 j=1

=

a0 µt−1 +

2Dw

D X 1 − a0 j=1

=

2Dw

2Dw

!

Dw 1/2 µt−1 + Σ ej 1 − a0 t−1 ! r Dw 1/2 µt−1 − Σ ej 1 − a0 t−1 r

µt−1 +

D X 1 − a0 j=1

2Dw

µt−1

µt−1

as required.

Now turning our attention to the covariance we have Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

145

2D w X

aj (ŵ[j] − µt−1 )(ŵ[j] − µt−1 )T

=

j=0

a0 (µ−1 − µt−1 )T (µ−1 − µt−1 )  !T ! r r D X 1 − a D D 0 w w 1/2 1/2  µt−1 − Σ ej − µt−1 µt−1 + Σ ej − µt−1  + 2Dw 1 − a0 t−1 1 − a0 t−1 j=1  !T ! r r D X 1 − a D D 0 w w 1/2 1/2  + µt−1 − Σ ej − µt−1 µt−1 − Σ ej − µt−1  2Dw 1 − a0 t−1 1 − a0 t−1 j=1  !T r ! r D X 1 − a D D 0 w w 1/2 1/2  = Σ ej Σ ej  2Dw 1 − a0 t−1 1 − a0 t−1 j=1  !T ! r r D X 1 − a D D 0 w w 1/2 1/2  + − − Σ ej Σ ej  2Dw 1 − a0 t−1 1 − a0 t−1 j=1  !T r ! r D X 1 − a D D 0 w w 1/2 1/2  = Σ ej Σ ej  Dw 1 − a0 t−1 1 − a0 t−1 j=1 =

D X

1/2

Σt−1 ej

T

1/2

Σt−1 ej

j=1

= Σt−1

D X

(ej )T (ej )

j=1

= Σt−1 I = Σt−1 . as required.

Problem 19.10 The extended Kalman filter requires the Jacobian matrix describing how small changes in the data create small changes in the measurements. Compute the Jacobian matrix for the measurement model for the pedestrian-tracking application (equation 19.50).

Answer The Jacobian is 

1 1+w

 0  J=  0 0

0 1 1+w

0 0

0 0 1 1+w

0 0 0

0

1 1+w

−u1  (1+w)2 −v1  (1+w)2  −u2  .

(1+w)2  −v2 (1+w)2

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

146

19

Temporal models

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

Chapter 20

Models for visual words Problem 20.1 The bag of words method in this chapter uses a generative approach to model the frequencies of the visual words. Develop a discriminative approach that models the probability of the object class as a function of the word frequencies.

Answer In a discriminative model, we choose a probability distribution over the world state and parameterize it by the measurements. In this case, the world state is discrete and multivalued and discrete and so we choose the categorical distribution. We make the parameters of this categorical distribution a function of the word frequencies. For example, we could use a linear function so that P r(w) = Catw [φ01 + φT1 f , φ02 + φT2 f , . . . , φ0N + φTN f ] where f is the normalized vector of word frequencies.

Problem 20.2 Prove the relations in equation 20.8, which show how to learn the latent i Dirichlet allocation model in the case where we do know the part labels {pij }I,J i=1,j=1 .

Answer From equation 4.35 in the book, we have that the predictive density for a categorical model trained with data with frequencies {Nk }K k=1 is given by Nk + αk j=1 (Nj + αj )

P r(x∗ = k|x1...I ) = P6

where α is hyperparameter of the Dirichlet prior.

Both of the terms in equation 20.8 take this form.

Problem 20.3 Show that the likelihood and prior terms in Latent Dirichlet Allocation are given by equations 20.10 and 20.11 respectively.

Answer Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

148

20

Models for visual words

From problem 3.10 we have: I Y

Catxi [λ1...K ] · Dirλ1...K [α1...K ] = κ · Dirλ1...K [α̃1...K ],

i=1

where

κ̃

=

P QK Γ[ K j=1 αj ] j=1 Γ[αj + Nj ] .

P QK α ] Γ[I + K j=1 j j=1 Γ[αj ]

α̃1...K

=

[α1 + N1 , α2 + N2 , . . . , αK + NK ].

and Nk is the total number of times that the variable took the value k.

It follows that if we integrate over the unknown variables, the distribution integrates to one and we are left with the constant P QK Γ[ K j=1 αj ] j=1 Γ[αj + Nj ] Catxi [λ1...K ] · Dirλ1...K [α1...K ] = κ = . QK PK Γ[I + j=1 αj ] j=1 Γ[αj ] i=1

Z Y I

It can now easily be seen that both equations 20.10 and 20.11 take this form.

Problem 20.4 Li and Perona (2005) developed an alternative model to the single authortopic model in which the hyperparameter α was different for each value of the object label w. Modify the graphical model for latent Dirichlet allocation to include this change.

Answer The associated graphical model is shown in figure 20.1.

Problem 20.5 Write out generative equations for the author-topic model in which multiple authors are allowed for each document. Draw the associated graphical model.

Answer

P r(wij )

=

Catwij [ω i ]

P r(pij |wij = n)

=

Catpij [π n ]

P r(fij |pij )

=

Catfij [λpij ].

To complete the model, we add Dirichlet priors to the unknown parameters {ω i }Ii=1 , M {π n }N n=1 and {λm }m=1 :

P r(ω i ) = Dirωi [γ] P r(π n ) = Dirπn [α] P r(λm ) = Dirλm [β], Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

149

Figure 20.1 Graphical model corresponding problem 20.4

Figure 20.2 Graphical model for problem 20.5

Here there is (potentially) a different author wij for each word, and the proportion of words written by each author is different for each document and depends on the categorical parameters ω i .

The associated graphical model is shown in figure 20.2.

Problem 20.6 In real objects, we might expect visual words f that are adjacent to one another to take the same part label. How would you modify the author topic model to Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

150

20

Models for visual words

Figure 20.3 Graphical model for problem 20.7

encourage nearby part labels to be the same. How would the Gibbs sampling procedure for drawing samples from the posterior probability over parts be affected?

Answer To encourage nearby labels to be the same, we could model them as a Markov random field by adding undirected horizontal connections between neighbors. The Gibbs’ sampling procedure will now be slightly more complex as we cannot sample each of the part labels independently. In practice, a new term will be added to the expression for the posterior probability (i.e., equation 20.19 from the book) and which will involve the neighbors of the current part label.

Problem 20.7 Draw a graphical model for the scene model described in section 20.6.

Answer The associated graphical model is shown in figure 20.3.

Problem 20.8 All of the models in this chapter have dealt with classification; we wish to infer a discrete variable representing the state of the world based on discrete observed features {fj }. Develop a generative model that can be used to infer a continuous variable based on discrete observed features (i.e., a regression model that uses visual words).

Answer Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.

151 For a generative model, we choose a probability distribution over the observations and make the parameters a function of the world state. In this case, the measurements might consist of a vector containing the normalized histogram of N svisual features. The Dirichlet distribution is a suitable distribution to describe this type of measurement and so we might have P r(x) = Dirx [α1 , α2 , . . . , αN ] where the parameters {αn }N n=1

are all positive. The world parameter is a continuous value w. We model each parameters α as a linear function an = φn0 + φn1 w and pass this function through the exponential function to ensure that the result is positive. The final model is P r(x|w) = Dirx [exp[φ10 + φ11 w], exp[φ20 + φ21 w], . . . , exp[φN 0 + φN 1 w]] .

To be complete, the model also needs a prior over the world state which might be modeled as a normal distribution.

Copyright c 2012 by Simon Prince. This latest version of this document can be downloaded from http://www.computervisionmodels.com.