The Bayesian Choice From Decision-Theoretic Foundations to Computational Implementation Second Editi

SOLUTION MANUAL

Solution Manual for Selected Problems from The Bayesian Choice 2nd Edition and Paperback Edition Christian P. Robert c °2007 Springer Science+Business Media This manual has been compiled by Roberto Casarin, Université Paris Dauphine and Universitá di Brescia, under the loose supervision of the author. Therefore, errors may (and should) remain: use at your own risk! Any difficulty with a solution should be signaled to the authors, who are also interested in collecting solutions to other problems to include them in this manual.

— Third Version, October 19, 2008

Chapter 1 - Introduction Exercise 1.1 (Kelker (1970)) A vector x ∈ Rp is distributed according to a spherically symmetric distribution if e.x has the same distribution than x for every orthogonal transform e. a. Show that, when a spherically symmetric distribution has a density, it is a function of xt x only. b. Show that, if the density of x is ϕ(xt x), the density of r = ||x|| is proportional to rp−1 ϕ(r2 ), and give the proportionality coefficient. ′ c. Show that, if x = (x′1 , x′2 ) with x1 ∈ Rq and x2 ∈ Rp−q , and ||x||2 = ||x1 ||2 + ||x2 ||2 , the density of (r1 , r2 ) = (||x1 ||, ||x2 ||) is proportional to ¢ ¡ r1q−1 r2p−q−1 ϕ r12 + r22 . d. Deduce that U =

||x1 ||2 + ||x2 ||2

||x1 ||2

is distributed according to a beta distribution Be(q/2, (p − q)/2).

2

Solution Manual

e. Conclude that

p − q ||x1 ||2 q ||x2 ||2

is distributed according to the F -distribution Fp−q,q independently of the spherically symmetric distribution of x. Deduce that the F -ratio is a robust quantity in the sense that its distribution is constant on a range of spherically symmetric distributions. a. Let x, y ∈ Rp be such that ||x|| = ||y||, then there exists an orthogonal transform such that: ex = y. By definition of spherically symmetric distribution it follows: f (y) = f (ex) = f (x). We conclude that f is constant on the sphere centered at 0 and thus it depends only on ||x||. b. Let f (x) = ϕ(xt x) = ϕ(||x||2 ) and ψ a function defined on R+ then E (ψ(r)) = E (ψ(||x||)) Z = ψ(||x||)f (x)dx Z = ψ(||x||)ϕ(||x||2 )dx.

Consider the following transform in hyperspherical coordinates  x ∈ Rp 7→ (r, ϑ1 , . . . , ϑp−1 ) ∈ R+ × [0, 2π]p−1  x1 = r cos(ϑ1 )   x2 = r sin(ϑ1 ) cos(ϑ2 ) h:  x3 = r sin(ϑ1 ) cos(ϑ2 )  ... xp = r sin(ϑ1 ) . . . sin(ϑp ) then

Z E(ψ(r)) = =

Z

−1 |drdϑ1 . . . dϑp−1 ψ(r)ϕ(r2 )|Jh(r,ϑ 1 ,...,ϑp−1 )

ψ(r)ϕ(r2 )rp−1 drS(sp ).

where S(sp ) is the surface of the unitary sphere in Rp+ with S(sp ) = ′

′

(2π)P/2 Γ (p/2)

c. Let h : (x1 , x2 ) 7→ (r1 , ϑ1 , . . . , ϑq−1 , r2 , α1 , . . . , αp−q−1 ) then Z E (ψ(r1 , r2 )) = ψ(r1 , r2 )ϕ(r12 , r22 )r1q−1 r2p−q−1 dr

Bayesian Choice

3

−1 |dϑ1 . . . ϑp−1 dα1 . . . dαp−q−1 |Jh(r,ϑ 1 ,...,ϑq−1 ,α1 ,...,αp−q−1 )

thus we conclude that (r1 , r2 ) ∼ r1q−1 r2p−q−1 ϕ(r12 r22 ) d. Note that r2 ||x1 ||2 = 2 1 2 U = 2 2 ||x1 || + ||x2 || r1 + r2 and let

µ h : (r1 , r2 ) 7→

then

u=

r12 r22 , v = r12 + r22 r12 + r22

¶

¶ r22 r12 , r1q−1 r2p−q−1 ϕ(r12 + r22 )dr1 dr2 r12 + r22 r12 + r22 ¶ p−q−1 µ ¶ µ Z ³ u ´ q−1 2 1 1−u 2 ∝ ψ(u, v) (v −4 u−1/2 (1 − u)−1/2 )dvdu ψ v v v µ ¶ Z 1 ∝ ψ(u, v)uq/2−1 (1 − u)(p−q)/2−1 ψ v −3−p/2 dvdu v µ

Z

E(ψ(u, v)) ∝

ψ

thus fU =

Γ

¡q¢ 2

¡ ¢ Γ p−q 2 uq/2−1 (1 − u)(p−q)/2−1 . Γ (p)

Exercise 1.3 Show that, if x1 , . . . , xn are known explanatory variables and y1 , . . . , yn are distributed as E[yi ] = bxi , the least-squares estimator of b, solution of n X min (yi − bxi )2 , b

i=1

is also a maximum likelihood estimator under a normality assumption.

Under the normality assumption the probability density function of yi is: ¶ µ −(yi − bxi )2 −1 −1/2 . f (yi |b) = σ (2π) exp 2σ 2 The likelihood and the log-likelihood functions of the sample y1 , . . . , yn are µ ¶¸ n · Y −(yi − bxi )2 −1 −1/2 σ (2π) exp ℓ(b|yi ) = 2σ 2 i=1 à ! n 1 X 2 −n −n/2 (yi − bxi ) = σ (2π) exp − 2 2σ i=1

n n 1 X L(b|yi ) = −n log σ − log(2π) − 2 (yi − bxi )2 2 2σ i=1

4

Solution Manual

respectively. From the of the maximum log-likelihood Pnfirst order condition P n 1 (y − bx )x = 0 ⇔ = − problem: ∂L i i i=1 (yi − bxi )xi = 0, it follows ∂b σ 2 ¡ i=1 i ¢ P P n n 2 −1 that b̂ = ( i=1 yi xi ) xi is the solution of the minimum least squares Pi=1 n (MLS) problem: minb i=1 (yi − bxi )2 because Pn one can easily verify that b̂ satisfies the MLS first order condition: −2 i=1 (yi − bxi )xi = 0.

We conclude that the minimum least square estimator of b is the maximum likelihood estimator under the normality assumption.

Exercise 1.4 In Example 1.3, give the expectation of n. Does that mean that 20 × 30/n is an unbiased estimator of N ?

In Example 1.3, n follows the hypergeometric distribution H(30, N, 20/N ).

The expected value of n is E(n) = (30 × 20/N ). N̂ = 20 × 30/n is not an unbiased estimator of N because E(N̂ ) = E(30 × 20/n) > 30 × 20/E(n) = N .

Exercise 1.5 In Example 1.6, show that the moments of x ∼ f (x) can be written as E[xk ] = pE[xk1 ] + (1 − p)E[xk2 ]. Deduce a moment estimator of (p, µ1 , µ2 , σ12 , σ22 ).

Let Xi ∼ N (µi , σi2 ), i = 1, 2. The r.v. ½ X1 with probability p X= X2 with probability 1 − p has density

½ ½ ¾ ¾ (x − µ1 )2 (x − µ2 )2 1−p p √ exp − exp − + . f (x) = √ 2σ12 2σ22 2πσ1 2πσ2

Let U ∼ U[0,1] be a uniform r.v. ¡ ¢ ¡ ¡ ¢¢ E X k = EU E X k |U ¡ ¡ ¢¢ ¡ ¡ ¢¢ = EU E X k I(U ≤ p)|U + EU E X k I(U > p)|U ¡ ¡ ¢¢ ¡ ¡ ¢¢ = EU I(U ≤ p)E X k |U + EU I(U > p)E X k |U ¢ ¢ ¡ ¡ = EU I(U ≤ p)X1k + EU I(U > p)X2k ¢ ¢ ¡ ¡ = EX1 EU (I(U ≤ p)|X1 ) X1k + EX2 EU (I(U > p)|X2 ) X2k ¡ ¢ ¡ ¢ = pEX1 X1k + (1 − p)EX2 X2k ¡ ¢ ¡ ¢ = pE X1k + (1 − p)E X2k If we apply the method of moments to estimate the 5 unknown parameters p, µ1 , µ2 , σ1 and σ2 , we need the moments up to the 5th order. Remember that, for a standard normal distribution, the even moments¡are:¢m2k =¡ (2k)! and the¢ 2k k! k odd moments are zero. Let Yi = (Xi −µi )/σi . Then E Xi = E (σi Yi + µi )k and

Bayesian Choice

5

¢ ¢ ¡ E Xi3 = E (σi Yi )3 + 3µi (σi Yi )2 + 3µ2i (σi Yi ) + µ3i ¡

= 3µi σi2 + µ2i ¢ ¢ ¡ E Xi4 = E (σi Yi )4 + 4µi (σi Yi )3 + 6µ2i (σi Yi )2 + 4µ3i (σi Yi ) + µ4i ¡

= 3σi4 + 6σi2 µi 2 + µ4i ¢ ¢ ¡ E Xi5 = E (σi Yi )5 + 5µi (σi Yi )4 + 10µ2i (σi Yi )3 + 10µ3i (σi Yi )2 + 5µ4i (σi Yi ) + µ5i ¡

= 15σi4 µi + 10σi2 µi 3 + µ5i .

The parameter estimators are obtained by solving the following system of equations E (X)= pµ1 + (1 − p)µ2 ¡ ¢ E X 2 = p(σ12 + µ21 ) + (1 − p)(σ22 + µ22 ) ¡ ¢ E X 3 = p(3σ12 µ1 + µ21 ) + (1 − p)(3σ22 µ2 + µ22 ) ¡ ¢ E X 4 = p(3σ14 + 6σ12 µ21 + µ41 ) + (1 − p)(3σ24 + 6σ22 µ22 + µ42 ) ¡ ¢ E X 5 = p(15σ14 µ1 + 10σ12 µ31 + µ51 ) + (1 − p)(15σ24 µ2 + 10σ22 µ32 + µ52 ) where the moments E(X k ) are replaced by their empirical counterparts.

Exercise 1.6 Derive the probabilities of Example 1.11 from the approximation 2 1 e−x /2 , Φ(−x) ≃ √ 2πx

which is valid when x is large.

Let x be the probability of a male birth and let • Ng be the random number of male births; • ng the observed number of male births; • n the total number of observed births.

Assume a uniform prior for x: x ∼ U[0,1] and a binomial likelihood for n: n ∼ B(n, x). The posterior distribution of x is π(x|Ng = ng ) = R 1 0

= R1 0

= R1 0

π(N = n|x)π(x) π(N = n|x)π(x)dx n

Cn g xng (1 − x)n−ng n

Cn g xng (1 − x)n−ng dx xng (1 − x)n−ng xng (1 − x)n−ng dx

which is the density of the beta distribution: Be(ng + 1, n − ng + 1). We have R1 n x g (1 − x)n−ng dx 1 1/2 . π(x ≥ |N = ng ) = R 1 2 xng (1 − x)n−ng dx 0

6

Solution Manual

According to Stigler (1986), the estimation of the numerator represented a challenging problem for Bayes. Laplace proposed a solution based on the following approximation. Consider xng (1 − x)n−ng = eng ln(x) e(n−ng ) ln(1−x) ´´ ³ ³n ³ ng ´ g ln(1 − x) = exp n ln(x) + 1 − n n ¡ ng ¢ ng and let h(x) = n ln(x) + 1 − n ln(1 − x), then for x 6= 0 and x 6= 1 ng ³ 1 ng ´ h′ (x) = − 1− =0 nx n (1 − x) ³ ng ´ ng x=0 (1 − x) − 1 − ⇔ n n ng ⇔x= . n Considering the Taylor expansion around ng /n, truncated at the second order ¡n ¢ ¡n ¢ 1 ¡ n ¢2 x − ng h′′ ng we have term: h(x) ≈ h ng + 2!

Z 1 Z 1 ng ng 2 ′′ ng 1 enh(x) dx ≈ enh( n ) en 2 (x− n ) h ( n ) dx.

1/2

1/2

The integrand function is¡ the density of a normal distribution with mean p n ¢ ng /n and variance −nh′′ ng multiplied by (2π). Let Ψ be the c.d.f. of a standard normal then s ¶ ½ µr Z 1 ³n ´ ³ n ng ´ 2π g nh( ng ) nh(x) ′′ ¡n ¢ × Ψ 1− e dx ≈ e −nh n n −nh′′ ng 1/2 µr ¶¶¾ µ ³n ´ 1 n g g +Ψ − −nh′′ n 2 n where ng = 251257, n = 493473 and ng /n ≈ 0.5097088.

Exercise 1.7 An examination has 15 questions, each with 3 possible answers.

Assume that 70% of the students taking the examination are prepared and answer correctly each question with probability 0.8; the remaining 30% answer at random. a. Characterize the distribution of S, score of a student if one point is attributed to each correct answer. b. Eight correct answers are necessary to pass the examination. Given that a student has passed the examination, what is the probability that she was prepared? a. The score S of each student depends on the number of right answers to the 15 questions. For unprepared students, the probability to give the

Bayesian Choice

7

right answer to each question is 1 over 3, since 3 represents the number of possible answers to each question. For those students, the total number of right answers, S, is a sum of independent Bernoulli r.v. and it follows the binomial distribution: B(15, 1/3). Similarly, for prepared students, the total number of right answers is a binomial random variable: B(15, 0.8).

The score of an unidentified student is therefore a rv that follows a mixture of those two binomial distributions, .7B(15, 0.8) + .3B(15, 1/3). b. Let R denote the event ”the student passes the exam”, i.e. S ≥ 8 and P the event ”the student is well prepared”. By applying Bayes’ formula: P (R|P ) × P (P ) P (R) P (S ≥ 8|P ) × P (P ) = P (S ≥ 8|P ) × P (P ) + P (S ≥ 8|P c ) × P (P c ) P15 k 0.7 × k=8 C15 0.8k 0.215−k = P15 P15 k 0.8k 0.215−k + 0.3 × k k 15−k 0.7 × k=8 C15 k=8 C15 0.33 0.67 = 0.9867

P (P |R) =

Exercise 1.8 Prove the discrete and continuous versions of Bayes’ Theorem.

Let A and B be two events and let Ac denote the complementary event of A. Let us define the conditional probability of A given B as P (A|B) = P (A ∩ B)/P (B), then P (A ∩ B) = P (A|B)P (B) and P (B) = P (B ∩ (A ∪ Ac )) = P ((B ∩ A) ∪ (B ∩ Ac )) = P (B|A) × P (A) + P (B|Ac ) × P (Ac ). The conditional probability of A given B: P (B ∩ A) P (B) P (B|A) × P (A) = P (B) P (B|A) × P (A) = P (B|A) × P (A) + P (B|Ac ) × P (Ac )

P (A|B) =

which is the result of Bayes theorem. The continuous version of the theorem follows by considering at the numerator the joint density of (θ, x): ϕ(θ, x) = R f (x|θ)π(θ) Rand at the denominator the marginal density of x: m(x) = ϕ(θ, x)dθ = f (x|θ)dθ. The posterior density of θ is: π(θ|x) = R

f (x|θ)π(θ) . f (x|θ)π(θ)dθ

Exercise 1.9 (Romano and Siegel (1986)) The Simpson paradox provides an illustration of the need for a conditional approach in Statistics. Consider two medical treatments, T1 and T2 , T1 being applied to 50 patients and T2 to 50

8

Solution Manual

others. The result of the experiment gives the following survival percentages: 40% for treatment T1 , 32% for treatment T2 . Therefore, treatment T1 seems better because it leads to an higher survival rate. However, if age is taken into account, dividing the subjects between juniors (50) and seniors (50), the success rates are described in the following table: T1 T2 junior 40 50 senior 10 35 and T1 is worse than T2 in both cases. Explain the paradox in terms of Bayes’s Theorem.

Let J be the event: ”the patient is junior”, S the event: ”the patient is senior” and G the event ”the patient survives”. It follows that 16 40 1 P (G|S ∩ T1 ) = 10 24 P (G|J ∩ T2 ) = 40 1 P (G|S ∩ T2 ) = 10 P (G|J ∩ T1 ) =

Then by applying Bayes’s theorem the probability, P (J|T1 ), for the patient to be junior given he is in T1 is not 1/2 and this represents a paradox.

Exercise 1.10 Show that the quantity δ that minimizes (1.6) is the median of the distribution of ξ. Give the quantity δ that minimizes the average squared error Eξ [(ξ − δ)2 ].

Eπ [|ξ − δ|] = = ∂ π E [|ξ − δ|] = ∂δ thus

Z |ξ − δ| π(ξ)dξ Z δ −∞ Z δ −∞

Z δ −∞

(δ − ξ) π(ξ)dξ + π(ξ)dξ −

π(ξ)dξ =

Z +∞ δ

Z +∞ δ

Z +∞ δ

(ξ − δ) π(ξ)dξ

π(ξ)dξ = 0

π(ξ)dξ

which is the definition of the median of π(ξ). The value of δ that minimises Eξ (ξ − δ)2 is

Eξ (ξ − δ)2 =

Bayesian Choice

Z

9

(ξ − δ)2 π(ξ)dξ Z Z ∂ ξ 2 E (ξ − δ) = −2 ξπ(ξ)dξ + 2δ π(ξ)dξ = 0 ∂δ Z Z ⇔ ξπ(ξ)dξ = δ π(ξ)dξ = δ

and we conclude that the expected value of ξ is the value of δ that minimizes Eξ (ξ − δ)2 .

Exercise 1.11 Find the median of the posterior distribution associated with the sampling distribution (1.7) and a flat prior π(ξ) = 1 on ξ. [Note: See Stigler (1986) for a resolution.]

The likelihood is ϕξ (x) = 2ξ exp {−ξ|x|}. The posterior density is π(ξ|x) =

1 ξ exp {−ξ|x|} R +∞ ξ 2 exp {−ξ|x|} dξ 0

2

ξ 1 = exp {−ξ|x|} ¯ ¯+∞ R +∞ ¯ ¯ 2 ξ 1 + 2|x| exp{−ξ|x|}dx ¯ − 2|x| exp {−ξ|x|} ¯ 0 = x2 ξ exp{−ξ|x|}

0

The median is the value of t such that the integral ½ ¾ Z t t 1 ¯¯ exp{−ξ|x|} ¯¯t 2 π(ξ|x)dξ = x − exp{−|x|t} + ¯− ¯ |x| |x| |x| 0 0 ½ ¾ t 1 − exp{−|x|t} = x2 − exp{−|x|t} + |x| |x|2 is equal to 1/2. It is therefore obtained by solving −t|x| exp{−|x|t} + 1 − exp{−|x|t} = 1/2.

Exercise 1.13 Consider x ∼ P(λ). Show that δ(x) = I0 (x) is an unbiased estimator of e−λ which is null with probability 1 − e−λ .

E(δ(x)) = P (x = 0) × I0 (x) + P (x 6= 0) × I0 (x) = e−λ + 0 then: E(δ(x)) = e−λ . Consider P (δ(x) = 0) = P (I0 (x) = 0) = 1 − P(I0 (x) 6= 0)

10

Solution Manual

= 1 − P (x = 0) = 1 − E(δ(x)) it follows that: P (δ(x) = 0) = 1 − e−λ .

Exercise 1.14 A statistic S is said to be ancillary if its distribution does not depend on the parameter θ and it is said to be complete if Eθ [g(S)] = 0 for every θ implies g(s) ≡ 0. Show that, if S is complete and minimal sufficient, it is independent of every ancillary statistic. [Note: This result is called Basu’s Theorem. The reverse is false.]

Let T : X → T be an ancillary statistic and S : X → T a sufficient and complete statistic for θ, with θ ∈ Θ. From our assumptions it follows that for all measurable sets B ⊂ T , Pθ (T ∈ B) = P (T ∈ B) that is, Pθ (T ∈ B) does not depend on θ. Moreover the conditional probability Pθ (T ∈ B|S) = P (T ∈ B|S). We prove that P (T ∈ B|S) = P (T ∈ B). The relation between marginal and the conditional probabilities is Z P (T ∈ B) = P (T ∈ B|S = y)dS(Pθ )(y).

Y

Let f (y) = P (T ∈ B) − P (T ∈ B|S = y), ∀y ∈ Y then by applying the definition of conditional probability Z f (y)dS(Pθ )(y) = 0.

Y

By construction f is bounded and by assumption the statistic S is quasicomplete, thus the above equality implies that P (T ∈ B) − P (T ∈ B|S = y) = 0, for all y ∈ Y and S(Pθ )− a.s.. We conclude that S and T are two independent statistics under Pθ .

Exercise 1.15 Consider a sample x1 , . . . , xn of i.i.d. variables with c.d.f. F . a. Give the density of the order statistic. b. Show that O = (X(1) , ..., X(n) ) is sufficient. What is the conditional distribution of (X1 , ..., Xn ) given O? c. Consider X1 , ..., Xn i.i.d. with totally unknown density. Show that O is then complete. a. Let s be one of the permutation of {1, . . . , n}, Sn the space of all permutations and φs the function which associates to each sample the sample with the permuted indexes: · n R → Rn ¡ ¢ φs : x = (x1 , . . . , xn ) 7→ xs = xs(1) , . . . , xs(n) We apply the random variable transform theorem for local diffeomorphisms. The domain and codomain must be open sets and the domain

Bayesian Choice

11

must be represented as the union of disjoints open sets. The order statistic φ is the image of (X1 , . . . , Xn ) through φ. φs takes values in O = {(x1 , . . . , xn )/x1 ≤ . . . ≤ xn } and λ-a.s. in

Ω = {(x1 , . . . , xn )/x1 < . . . < xn } .

Thus the codomain is an open set.

S Note that the set N = i6=j {x = (x1 , . . . , xn )/xi = xj } is a null set. The statistic (X1 , . . . , Xn ) takes values a.s. on the complementary set, which is an open set. x ∈ N c ⇔ ∀ (i, j) i 6= j xi 6= xj ⇔ ∃ s ∈ Sn φs ∈ Ω ⇔ ∃ s ∈ Sn x ∈ Os with Os = φ−1 s (Ω) [ Os ⇔x∈ s∈Sn

S Thus Os is an open set and N c = s∈Sn Os .

Let L denote the likelihood associated to (X1 , . . . , Xn ). We apply the change of variable theorem for local diffeomorphisms to the restriction of φs to the open set Os . In Os , the absolute value of the Jacobian of φs is 1 X L(φ−1 | ∀x ∈ Ω, L(X1 ,...,Xn ) (x, θ) = s (x), θ)|Jφ−1 s s∈Sn

X

=

L(φs−1 (x), θ)

s∈Sn

X

=

L(xs−1 , θ)

s∈Sn

and due to the independence assumption for (X1 , . . . , Xn ): ∀x ∈ Ω,

L(X1 ,...,Xn ) (x, θ) = n!L(x, θ)

b. Let O be the order statistic L(x|θ)LO (o|x, θ) LO (o) L(x|θ)LO (o|x, θ) = n!L(x|θ) LO (o|x, θ) = n!

L(X1 ,...,Xn ) (x|O = o, θ) =

12

Solution Manual

which is 1/n! for all o corresponding to the increasing values of x and 0 otherwise. The likelihood of (X1 , . . . , Xn ) conditionally on O is independent of θ. We conclude that the order statistic O is a sufficient statistic for θ. c. Let φ be measurable and assume ∀, θ E(φ(O(X))) = 0. We prove that φ(O(X)) = 0 a.s.: ∀ θ, ⇒

Z +∞ −∞

Z +∞ −∞

n!

n Y

f (xi , θ)φ(O(x))dx = 0

i=1

φ(O(x))dx = 0

⇒ φ(O(x)) = 0 a.s.

Exercise 1.16 Show that a statistic T is sufficient if and only if ℓ(θ|x) ∝ ℓ(θ|T (x)).

By applying the Neyman-Pearson factorization, T is sufficient iff f (x|θ) = g(T (x)|θ)h(x|T (x)) ⇔ l(θ|x) = l(θ|T (x))h(x|T (x)) ⇔ l(θ|x) ∝ l(θ|T (x)) where f and g are the p.d.f. of x and T (x) respectively and h is a positive measurable function.

Exercise 1.17 (Berger and Wolpert (1988, p. 21)) Consider x with support {1, 2, 3} and distribution f (· | 0) or f (· | 1), where x 1 2 3 f (x|0) 0.9 0.05 0.05 f (x|1) 0.1 0.05 0.85 Show that the procedure that rejects the hypothesis H0 : θ = 0 (to accept H1 : θ = 1) when x = 2, 3 has a probability 0.9 to be correct (under H0 as well as under the alternative). What is the implication of the Likelihood Principle when x = 2?

We rejects the hypothesis H0 : θ = 0 under the alternative H1 : θ = 1 with probability PrH1 ({x = 2} ∪ {x = 3}) = PrH1 ({x = 2}) + PrH1 ({x = 3}) = 0.05 + 0.85 = 0.9. We accept the hypothesis H0 : θ = 0 under H0 : θ = 1 with probability PrH0 ({x = 1}) = PrH0 ({x = 1}) = 0.9. Let x1 and x2 be observations of x such that x1 = x2 = 2 and let ℓ1 (θ|x1 ) and ℓ2 (θ|x2 ) be the

Bayesian Choice

13

likelihood functions associated to f (·|0) and f (·|1) for the observations x1 and x2 respectively. There does not exist a constant c such that ℓ1 (θ|x1 ) = cℓ2 (θ|x2 ) for each θ. Note that if θ = 0, then ℓ1 (θ|x1 ) = 0.05 but ℓ2 (θ|x2 ) = 0 and if θ = 1, then ℓ1 (θ|x1 ) = 0 and ℓ2 (θ|x2 ) = 0.05. Thus the Likelihood Principle does not allow to conclude that x1 and x2 bring to the same inference on θ.

Exercise 1.24 (Berger (1985a)) In the setting of Example 1.16, show that, for the UMPU test of H0 : p = 1/2, the null hypothesis will be accepted or rejected at level 5%, depending on the distribution considered. Deduce that the frequentist theory of tests is not compatible with the Likelihood Principle.

(Hint: See Chapter 5 for definitions.)

Let us test the audience level, 0 ≤ θ ≤ 1. With 9 spectators and 3 persons who did not watch at the program, we have: (1) x ∼ B(12, θ) with likelihood: µ ¶ 12 x f1 (x, θ) = θ (1 − θ)n−x = 22θ9 (1 − θ)3 x with x = 9 and n = 12.

(2) N ∼ N eg(3, 1 − θ), with likelihood: µ ¶ µ+x−1 x f (x, θ) = θ (1 − θ)n−x = 55θ9 (1 − θ)3 . x In both cases the likelihood is proportional to θ∗ (1−θ)n . Due to the likelihood principle the inference result should not depend on the choice of the model.

Let us consider the UMPU test: H0 : θ = 1/2 and H1 : θ > 1/2 at the confidence level 5%. The probability associated to the critical region H0 is : Xi = PH0 (X ≥ 9) = P (X ≥ 9|θ = 1/2) = fi (−9/(1/2)) + fi (10/(1/2)) + fi (11/(1/2)) +fi (12/(1/2)) = 0.075 accept X2 = PH0 (x ≥ 9) = fx (9/(1/2))+fx (10/(1/2))+fx′ (11/(1/2))+fx (12/(1/2)) = 0.0325 reject.

At the level 5%, we reject H0 for the negative binomial and accept for the binomial. The classical test theory is not compatible with the Likelihood Principle.

Exercise 1.27 Consider a Bernoulli random variable y ∼ B([1 + eθ ]−1 ). a. If y = 1, show that there is no maximum likelihood estimator of θ.

14

Solution Manual

b. Show that the same problem occurs when y1 , y2 ∼ B([1 + eθ ]−1 ) and y1 = y2 = 0 or y1 = y2 = 1. Give the maximum likelihood estimator in the other cases. a. Let ℓ denote the likelihood and L the log-likelihood: ³ £ ¤−1 ´l=y ℓ(θ|y) = P [Y = y] = [1 + eθ ]−y 1 − 1 + eθ µ θ ¶ e L(θ|y) = −y log(1 + eθ ) + (l − y) log 1 + eθ = −y log(l + eθ ) + (1 − y)(θ − log(1 + eθ )) −y log(1 + eθ ) + θ − log(1 + eθ ) − yθ + y log(1 + eθ ) = (1 − y)θ − log(1 + eθ ) The first order condition for the maximum likelihood estimator of θ is ∂L eθ eθ = =0⇔ =1−y θ ∂θ 1+e 1 + eθ If y = 1 then

eθ = 0 ⇔ eθ = 0 ⇔ θ = log 0 1 + eθ

The log function is not defined in 0, thus we conclude that if y = 1 the maximum likelihood estimator of θ does not exist. b. The likelihood and log-likelihood functions are l(θ|yi ) =

2 Y i=1

=

2 Y

P [Yi = yi ] ³ £ ¤−1 ´1−yi [1 + eθ ]−yi 1 1 + eθ

i=1 θ −

= [1 + e ]

L(θ|yi ) = −

2 X

P2

i=1 yi

³

P £ ¤ ´2− 2i=1 yi θ −1 1− 1+e

à θ

yi log(1 + e ) +

i=1

= −

2 X i=1

−

2 X i=1

2− Ã

θ

yi log(1 + e ) +

2−

2 X

!

i=1 2 X

µ log

yi ! yi

eθ 1 + eθ

¶

(θ − log(1 + eθ ))

i=1

yi log(1 + eθ ) + 2θ − 2 log(1 + eθ )

Bayesian Choice 2 X

− Ã = ∂L = ∂θ

yi θ +

à 2

yi log(1 + eθ )

i=1

i=1

2−

2 X

15

2 X i=1

2 X

yi

!

θ − 2 log(1 + eθ )

yi

! −

i=1

2 X 2eθ 2eθ yi = 0 ⇔ = 2 − 1 + eθ 1 + eθ i=1

2θ 2eθ = 2 ⇔ = 0 ⇔ θ = log 0. y1 = y2 = 1 ⇒ 1 + eθ 1 + eθ The log function is not defined in 0, thus the maximum likelihood estimator of θ does not exist if y1 = y2 = 0. If y1 = y2 = 1 then 2eθ eθ = 0 ⇔ = 0 ⇔ eθ ⇔ θ = log 0.

1 + eθ 1 + eθ The log function is not defined in 0, thus the maximum likelihood estimator of θ does not exist if y1 = y2 = 1.

In the other cases 2 X 2eθ yi = 2 1 + eθ i=1 2 eθ 1X = 1 − yi 1 + eθ 2 i=1 ! ! Ã Ã 2 2 X X 1 1 ⇔ eθ = 1 − yi + eθ 1 − yi 2 i=1 2 i=1

⇔

!# Ã !

2 2 1X 1X θ e = 1− yi yi ⇔ 1− 1− 2 i=1 2 i=1 Ã 2 ! ! Ã 2 1X θ 1X ⇔ e = 1− yi 2 i=1 2 i=1 Ã 2 !−1 1X θ ⇔e = yi −1 2 i=1  Ã !−1 2 X 1 − 1 ⇔ θ = log  yi 2 i=1 "

Ã

The maximum likelihood estimator is

16

Solution Manual

Ã

2

1X θ̂ = log  yi 2 i=1

!−1

 − 1 .

Exercise 1.29 The Weibull distribution W e(α, c) is widely used in engineering and reliability. Its density is given by c

f (x|α, c) = cα−1 (x/α)c−1 e−(x/α) . a. Show that, when c is known, this model is equivalent to a gamma model. b. Give the likelihood equations in α and c and show that they do not allow for explicit solutions. c. Consider an i.i.d. x1 , . . . , xn sample from W e(α, c) censored from the right in y0 . Give the corresponding likelihood function when α and c are unknown and show that there is no explicit maximum likelihood estimators in this case either. a. The p.d.f. of the Gamma distribution is f (x|a, b) =

ba a−1 −bx x e , x ∈ R+ Γ (a)

and its first order moment is

Z ba xa e−bx dx Γ (a) R+ Z 1 z a e−z dz = bΓ (a) R+

E(x) =

For a Weibull distribution we find Z xc −( αx )c E(x) = cα−1 e dx c−1 R+ α Z x c −c = cα xc e−( α ) dx R+ Z c =c y c e−y αdy R Z+ 1 =α z c e−z dz R+

both models are equivalent for α = 1/(bΓ (a)) and 1/c = a − 1, with c known. b. Since ³ x ´c log L(α, c(x)) = log c − log α + (c − 1) log x − (c − 1) log α − α

Bayesian Choice

17

then the first order conditions are ∂ log L 1 (c − 1) cxc =− − + c+1 = 0 ∂α α α α ³ x ´ ³ x ´c 1 ∂ log L =0 = + log x − log ∂c c α α and they do not allow for an explicit solution. c. Let y1 , . . . , yn be an iid sample. The likelihood and log-likelihood are L(α, c|y1 , . . . , yn ) = =

n Y i=1 n Y

f (yi |α, c) cα−1

³ y ´c−1

i=1

i

α

log L = log c − log α +

n ³ y ´c o i exp − α

n ³ X

(c − 1) log

³y ´

i=1

i

α

−

³ y ´c ´ i

α

and the first order conditions are n ∂L 1 Xc−1 yic =0 =− − + c c+1 ∂α α i=1 α α

n ³ y ´ ³ y ´c ³y ´ 1 X ∂L i i i − log =0 log = + ∂c c i=1 α α α

that do not allow for an explicit solution.

Exercise 1.32 (Olkin et al. (1981)) Consider n observations x1 , . . . , xn from B(k, p) where both k and p are unknown. a. Show that the maximum likelihood estimator of k, k̂, is such that (k̂(1 − p̂))n ≥

n Y

(k̂ − xi )

and

((k̂ + 1)(1 − p̂))n <

n Y

(k̂ + 1 − xi ),

i=1

i=1

where p̂ is the maximum likelihood estimator of p. b. If the sample is 16, 18, 22, 25, 27, show that k̂ = 99. c. If the sample is 16, 18, 22, 25, 28, show that k̂ = 190 and conclude on the stability of the maximum likelihood estimator. a. The likelihood and log likelihood are L(x1 , . . . , xn |k, p) =

n Y i=1

Ckxi pxi (1 − p)k−xi

18

Solution Manual

log L =

n X i=1

log Ckxi +

n X

xi log p +

n X

(k − xi ) log(1 − p)

i=1

i=1

and from the first order condition Pn Pn xi nk − i=1 xi ∂ log L = i=1 − =0 ∂p p 1−p we have p̂ =

x̄ k̂

.

Consider the first inequality, we have n

k̂(1 − p̂) = k̂ − x̄ =

1X (k̂ − xi ). n i=1

pQn Pn √ Note that ∀ x, y ∈ R+ : 21 (x + y) > xy and n1 i=1 xi > n i=1 xi due to the fact that x 7→ log x is concave. By applying the previous inequality to the ML estimator it follows v un n X uY 1 n (k̂ − xi ) ≥ t (k̂ − xi ) n i=1 i=1 and k̂(1 − p̂)n ≥

n Y

(k̂ − xi ).

i=1

Consider the second inequality, the first order condition is ¡k¢ n ∂ log L X ∂ xi = + n log(1 − p) = 0. ∂k ∂k i=1 Note that

¡ ¢ ∂ xki ∂k

and

1 ≥ k−j

Z k−j+1 k−j−1

=

xX i −1

1 k − j j=0

1 dt = log(k − j + 1) − log(k − j − 1) t

thus µ ¶ xX i −1 k+1 1 log(k − j + 1) − log(k − j − 1) = log . ≥ k−j k + 1 − xj j=1 j=1

xX i −1

Bayesian Choice L Note that ∂ log ∂k |k=k̂ = 0, then n log(1−p̂) = Take the exponential and obtain

µ

1 1 − p̂

¶n ≥

n Y

Pn

j=1

x ∂Ck i

∂k |k=k̂ ≥ log

³

19 k̂+1 k̂+1−xj

k̂ + 1

i=1 k̂ + 1 − xj

and the desired inequality follows. b-c.Both examples show the instability of the ML estimator for samples of small sizes.

Exercise 1.34 (Basu (1988)) An urn contains 1000 tickets; 20 are tagged θ and 980 are tagged 10θ. A ticket is drawn at random with tag x. a. Give the maximum likelihood estimator of θ, δ(x), and show that P (δ(x) = θ) = 0.98. b. Suppose now there are 20 tickets tagged θ and 980 tagged ai θ (i ≤ 980), such that ai ∈ [10, 10.1] and ai 6= aj (i 6= j). Give the new maximum likelihood estimator, δ ′ , and show that P (δ ′ (x) < 10θ) = 0.02. Conclude about the appeal of maximum likelihood estimation in this case. a. Let

  0.02 if x = θ f (x|θ) = 0.98 if x = 10θ  0 otherwise

then the likelihood is ¡ ¢ L(x|θ) = 0.02I{x=θ} + 0.98I{x=10θ} , which is maximized for x = 10θ. Thus the ML estimator δ(x), of θ is δ(x) = Note that P (δ(x) = θ) = P

³x 10

x .

10

´ = θ + P (x = 10θ) = 0.98

b. Let 980 tickets be tagged ai θ, with (i ≤ 980), ai ∈ [10, 10.1] and ai 6= aj for i 6= j. The likelihood 980

L(x|θ) = 0.02I{x=θ} + has a maximum for

1 X I{x=ai θ} 980 i=1

δ ′ (x) = x

´

.

20

Solution Manual

and

P (δ ′ (x) < 10θ) = P (x < 10θ) = P (x = θ) = 0.02

We conclude that if 980 tickets are tagged 10θ we have a good estimator of θ. If 980 tickets are tagged ai θ then the estimator of θ is no more appealing. A confidence interval estimator should behave better than a pointwise estimator.

Exercise 1.35 (Romano and Siegel (1986)) Given " µ ¶2 # 1 1 exp −50 − 1 f (x) = x x

(x > 0),

show that f is integrable and that there exist a, b > 0 such that Z b 0

af (x)dx = 1 and

Z b 1

af (x)dx = 0.99.

For the distribution with density p(y|θ) = aθ−1 f (yθ−1 )I[0,bθ] (y), give the maximum likelihood estimator, δ(y), and show that P (δ(y) > 10θ) = 0.99. n ¡ ¢2 o Let α > 0 and hα : x 7→ x1 exp −50 x1 − 1 , then hα =

n αo −2α x exp − 2 2 −2α {z x } |x | {z } integrable in 0 null in 0

is integrable in 0 because it is the product of two functions, that are integrable in 0. Consider that ∀x < 12 , x1 − 1 > 1/2 x , then µ

and

¶2 1 (1/2)2 −1 > x x2

( ¾ µ ¶2 ) ½ 50 1 1 1 1 −1 < exp − .

0 ≤ exp −50 x x x 4 x24 | {z } h 50 (x) 4

Note that h 50 (x) is integrable in a neighbourhood of 0, thus f too is integrable 4 in a neighbourhood of 0. f is continuous on [0, A] and thus it is integrable.

The quantity

Bayesian Choice

a = R1

21

0.1

f (x)dx 0

is well defined. If b exists, then Z b 0

thus

f (x)dx = Rb R01

1 = 10 a

f (x)dx

f (x)dx 0

Now let

Rt g : t 7→ R 01 0

Z 1 0

f (x)dx

= 10.

f (x)dx f (x)dx

.

Note that g is strictly increasing with g ′ > 0, g(1) = 1 and lim g(t) = +∞, t→+∞

thus by the mean value theorem g(t) = 10 has a solution in R+ and then b exists. The first order condition is 1 y f ′ (yθ−1 ) ∂ log p(y|θ) =− − 2 =0 ∂θ θ θ f (yθ−1 ) where

( µ ¶2 ) µ µ ¶ ¶ 1 1 1 f (x) = 2 exp −50 −1 100 −1 −1 . x x x ′

The solution of the ML equation is given by 1 y f ′ (yθ−1 ) − − 2 =0⇔ θ θ f (yθ−1 ) µ ¶ 1 y f ′ (yθ−1 ) ⇔ − 1+ =0 θ θ f (yθ−1 ) µ µ µ µ ¶¶¶¶ 1 y θ θ θ ⇔ − 1+ −1 + 100 −1 =0 θ θ y y y Exercise 1.36 (Romano and Siegel (1986)) Consider x1 , x2 , x3 i.i.d. N (θ, σ 2 ). a. Give the maximum likelihood estimator of σ 2 if (x1 , x2 , x3 ) = (9, 10, 11) or if (x1 , x2 , x3 ) = (29, 30, 31). b. Given three additional observations x4 , x5 , x6 , give the maximum likelihood estimator if (x1 , . . . , x6 ) = (9, 10, 11, 29, 30, 31). Does this result contradict the Likelihood Principle?

22

Solution Manual

a. We have ℓ(x1 , x2 , x3 , σ) =

3 · Y

σ −1 (2π)−1/2 exp

i=1

=σ

−3

µ

Ã

−3/2

(2π)

ℓ(x1 , x2 , x3 , σ) = −3 log σ −

−(x1 − θ)2 2σ 2

¶¸

2 1 X (xi − θ)2 exp − 2 2σ i=1

!

3 3 1 X log(2π) − 2 (xi − θ)2 2 2σ i=1

3 X ∂L 3 1 2 (xi − θ)2 = − + × 2× ∂σ σ 2 σ i=1 3 3 1 X = − + 3 (xi − θ)2 σ σ i=1 3

⇔

1 X 3 (x1 − θ)2 = σ 3 i=1 σ

⇔ σ2 =

3

1X (xi − θ)2 3 i=1

The ML estimator of σ 2 is: σ̂ 2 = 31

P3

2 i=1 (xi − θ) .

P3 1 If (x1 , x2 , x3 ) = (9, 10, 11) then θ = 3 i=1 xi = 31 × (9 + 10 + 11) = 10 2

thus an estimates of σ is: ¤ 2 1£ σ̂ 2 = (9 + 10)2 + (10 − 10)2 + (11 − 10)2 = ≈ 0.66 3 3 P 3 If (x1 , x2 , x3 ) = (29, 30, 31) then θ = 13 i=1 ×(29 + 30 + 31) = 30 and an 2 estimate of σ is: ¤ 2 1£ 2(9 − 30)2 + (30 − 30)2 + (31 − 30)2 ) = ≈ 0.66 σ̂ 2 = 3 3 b. We have µ ¶¸ 6 · Y −(x1 − θ)2 σ −1 (2π)−1/2 exp 2σ 2 i=1 à !

6 1 X −6 −6/2 2 = σ (2π) exp − 2 (xi − θ) .

2σ i=1

L(x1 , ..., x6 , σ) =

Then

Bayesian Choice

log L(x1 , ..., x6 , σ) = −6 log σ −

23

6 6 1 X log(2π) − 2 (x1 − θ)2 = 0 2 2σ i=1

6 X 6 1 2 ∂ log L (xi − θ)2 =− + × 2 × 2∂ σ 2 σ i=1

⇔−

6 6 1 X + 3 (xi − θ)2 = 0 σ σ i=1 6

⇔

1 X 6 (xi − θ)2 = σ 3 i=1 σ

⇔ σ2 =

6

1X (xi − θ)2 .

6 i=1

P6 Thus the ML estimator of σ 2 is σ̂ 2 = 61 i=1 (xi = θ)2 . If (x1 , x2 , P6 x3 , x4 , x5 , x6 ) = (9, 10, 11, 29, 30, 31) then θ = 61 i=1 xi = 16 × (9 + 10 + 2 11 + 29 + 30 + 31) = 20 and an estimate of σ is: σ̂ 2 =

1£ (9 − 20)2 + (10 − 20)2 + (11 − 20)2 + (29 − 20)2 6 ¤ 302 +(30 − 20)2 + (31 − 20)2 = ≃ 100.66.

3

Due to the Likelihood principle the whole information on θ from x is contained in the likelihood function L(θ, x). If x1 and x2 are such that a constant c exists such that for all θ, L(θ, x1 ) = cL(θ, x2 ), then they give the same information on θ and the same inference result should be obtained.

If we exploit the relation between the estimator based on 6 observations and the ones based on 3 observations, we have L(σ 2 |x1 , ..., x6 ) = ¤ 1£ = (x1 − θ)2 + (x2 − θ)2 + (x3 − θ)2 + (x4 − θ)2 + (x5 − θ)2 + (x6 − θ)2 6 ¤ 1£ 3L(σ 2 |x1 , x2 , x3 ) + (x4 − θ)2 + (x5 − θ)2 + (x6 − θ)2 = 6 6 1X 1 (xi − θ)2 . = L(σ 2 |(x1 , x2 , x3 ) + 2 6 i=4 Thus a constant c such that L(σ 2 |x1 , ..., X6 ) = cL(σ 2 |x1 , x2 , x3 ) does not exist. The result contradicts the likelihood principle.

24

Solution Manual

Exercise 1.37 If x ∼ N (θ, σ 2 ), y ∼ N (̺x, σ 2 ), as in an autoregressive model, with ̺ known, and π(θ, σ 2 ) = 1/σ 2 , give the predictive distribution of y given x.

We have

½ ½ ¾ ¾ (y − ̺x)2 (x − θ)2 1 1 1 √ √ exp − exp − m(y|x) = dθdσ 2σ 2 2σ 2 σ2 2πσ 2 2πσ 2 R2 ½ ¾ Z (y − ̺x)2 1 1 √ exp − dσ = 2σ 2 σ2 2πσ 2 R ¾ ½ Z (y − ̺x)2 1 1 √ exp − ds = 1 2s 2 s +1 R+ 2 2π ¾ ½ Z 1 1 1 β = √ exp − ds α+1 s s 2 2 R+ Γ (α) Z

with β = ((y − ̺x)2 )/2 and α = 1/2. The last quantity is proportional to a inverse gamma distribution, thus ¾ ½ Z 1 1 β 1 exp − ds m(y|x) = √ α+1 Γ (α) s s 2 2 R+ 1 1 = √ α 2 2β √ 2 1 = √ 2 2 |y − ̺x| 1 . = 2|y − ̺x| Exercise 1.38 If y ∼ B(n, θ), x ∼ B(m, θ), and θ ∼ Be(α, β), give the predictive distribution of y given x.

We have π(θ|x) = R where

φ(x, θ) φ(x, θ)dθ

¡M ¢ α+x−1 (1 − θ)m−x+β−1 x θ B(α, β) ¡M ¢ Z 1 x θα+x−1 (1 − θ)m−x+β−1 dθ m(x) = B(α, β) 0 ¡M ¢

φ(x, θ) =

=

x

B(α, β)

B(α + x, m − x + β).

Bayesian Choice

25

It follows

¡M ¢ α+x−1 θ (1 − θ)m−x+β−1 B(α, β) ¡m¢ B(α + x, m − x + β) π(θ|x) = x B(α, β) x B(α, β) =

θα+x−1 (1 − θ)m−x+β−1 B(α + x, m − x + β)

thus θ|x ∼ B(α + β, m − x + β). Moreover Z g(y|x) = g(y|θ, x)π(θ|x)dθ Z 1µ ¶ θα+x−1 (1 − θ)m−x+β−1 N y θ (1 − θ)n−y = dθ y B(α + x, m − x + β) 0 ¡N ¢ Z =

1

y

B(α + x, m − x + β)

0

θy+x+α−1 (1 − θ)n+m−x+β−x−y−1 dθ

thus

µ ¶ N B(y + x + α, n + m + β − x − y) y|x ∼ y B(α + x, m − x + β)

( Beta-Binomial distribution).

Exercise 1.39 Given a proper distribution π(θ) and a sampling distribution f (x|θ), show that the only case such that π(θ|x) and π(θ) are identical occurs when f (x|θ) does not depend on θ.

Assume that f (x|θ) does not depend on θ, then f (x|θ)π(θ) f (x|θ)π(θ)dθ θ

π(θ|x) = R =

f (x|θ)π(θ) R f (x|θ) θ π(θ)dθ π(θ) π(θ)dθ θ

= R

and π(x|θ) = π(θ). Conversely, assume that π(θ|x) = π(θ) and that f (x|θ) depends on θ, then R

f (x|θ)π(θ) = π(θ) f (x|θ)π(θ)dθ θ

f (x|θ) =1 f (x|θ)π(θ)dθ θ Z ⇔ f (x|θ) = f (x|θ)π(θ)dθ

⇔R

θ

⇔ f (x|θ) = c ∈ R

(false).

26

Solution Manual

We conclude that f (x|θ) does not depend on θ.

Exercise 1.43 Give the posterior and the marginal distributions in the following cases: (i) x|σ ∼ N (0, σ 2 ), 1/σ 2 ∼ G (1, 2); (ii) x|λ ∼ P(λ), λ ∼ G (2, 1); (iii) x|p ∼ N eg(10, p), p ∼ Be(1/2, 1/2). (i) We have f (x|σ) = σ −1 (2π)−1/2 exp and

µ π

1 σ2

¶

µ

−x2 2σ 2

¶

¶ µ 2 2 = exp − 2 Γ (1) σ

The posterior of λ is: µ π

1 |x σ2

¶

¶ 1 σ2 ¶ µ 2 −x − 4 ∝ σ −1 exp 2σ 2 µ

∝ f (x|σ)π

Thus the posterior of σ −2 is a G (3/2, (−x2 − 4)/2) distribution.

The marginal of x is: µ ¶ Z 1 m(x) = f (x|σ)π dσ σ2 ¶ µ µ 2¶ Z 2 −2 −x dσ exp = σ −1 (2π)−1/2 exp 2σ 2 Γ (1) σ2 ¶ µ 2 Z 2(2π)−1/2 −x − 4 −1 = dσ σ exp Γ (1) 2σ 2 (ii) We have f (x|λ) = exp(−λ) and m(λ) =

1 λ exp(−λ) Γ (2)

The posterior of λ is: π(λ|x) = R

λx x!

f (x|λ)π(λ) f (x|λ)π(λ)dλ

Bayesian Choice x

= R = R =

1 λ exp(−λ) exp(−λ) λx! Γ (2) x

1 λ exp(−λ)dλ exp(−λ) λx! Γ (2)

λx+1 exp(−2λ) λx+1 exp(−2λ)dλ

λx+1 exp(−2λ) .

2−(x+2) Γ (x + 2)

Thus the posterior of λ is a G (x + 2, 2) distribution.

The marginal of x is: Z m(x) = f (x|λ)π(λ)dλ Z λ2 1 = exp(−λ) λ exp(−λ)dλ x! Γ (2) Z 1 λx+1 exp(−2λ)dλ = x!Γ (2) =

2−(x+2) Γ (x + 2) x!Γ (2)

(iii)We have

µ f (x|p) =

and π(p) =

¶ n−x n p (1 − p)x x

p(1/2)−1 (1 − p)(1/2)−1 .

B(1/2, 1/2)

The posterior of p is: f (x|λ)π(λ) f (x|λ)π(λ)dλ ¡n−x+1¢ n p (1 − p)(p(1/2)−1 /B(1/2, 1/2)) = R ¡n−x+1¢ n x p (1 − p)x (p(1/2)−1 (1 − p)(1/2)−1 /B(1/2, 1/2))dp n

π(λ|x) = R

= R =

pn−(1/2) (1 − p)x−(1/2) pn−(1/2) (1 − p)c−(1/2) dp

pn−(1/2) (1 − p)x−(1/2) .

B(n + 1/2, x + 1/2)

That is a B(n + 1/2, x + 1/2) distribution.

The marginal of x is: Z m(x) = f (x|λ)π(λ)dλ

27

28

Solution Manual

Z µ

¶ 1 n−x+1 n )dp p (1 − p)x (p(1/2)−1 (1 − p)(1/2)−1 = B(1/2, 1/2) x ¡n−x+1¢ Z x = pn−(1/2) (1 − p)x−(1/2) dp B(1/2, 1/2) ¡n−x+1¢

x B(, +1/2, x + 1/2) B(1/2, 1/2) µ ¶ n−x+1 Γ (1) Γ (n + 1/2)Γ (x + 1/2) = x Γ (1/2)Γ (1/2) Γ (n + x + 1) µ ¶ n−x+1 1 B (n + 1/2, x + 1/2) . = π x

=

Exercise 1.49 (Fernandez and Steel (1999)) Consider n i.i.d. observations x1 , . . . , xn from the mixture pN (µ0 , σ02 ) + (1 − p)N (µ0 , σ12 ) , where p, µ0 and σ0 are known. The prior on σ1 is a beta Be(α, β) distribution.

Show that, if r ≥ 1 observations are equal to µ0 , the posterior distribution is only defined when α > r. [Note: From a measure theoretical point of view, the set of xi ’s equal to µ0 is of measure zero. If one (or more) observation is exactly equal to µ0 , it means that the continuous mixture model is not appropriate.]

Let us introduce the allocation variable zi such that zi = 1 if xi ∼ N (µ0 , σ12 ) and 0 otherwise. The posterior distribution is ( ¾ ¾) ½ ½ n Y p (1 − p) (xi − µ0 )2 (xi − µ0 )2 p π(σ1 |x) ∝ +p exp − exp − 2σ02 2σ12 2πσ02 2πσ12 i=1

1 σ α−1 (1 − σ12 )β−1 I[0,1] (σ1 ) B(α, β) 1 ( ( ½ ¾)1−zi Y ¾)zi ½ n n Y (xi − µ0 )2 1 (xi − µ0 )2 1 exp − pp ∝ exp − (1 − p) p 2σ02 2σ12 2πσ02 2πσ12 i=1 i=1 1 σ α−1 (1 − σ12 )β−1 I[0,1] (σ1 ) B(α, β) 1 ½ ¾¾ n ½ Y zi (xi − µ0 )2 1 σ1α−1 (1 − σ12 )β−1 I[0,1] (σ1 ) ∝ zi exp − 2 σ 2σ 1 1 i=1 ( n ) P X zi (xi − µ0 )2 α−1− n i=1 zi ∝ exp − (1 − σ12 )β−1 I[0,1] (σ1 ). σ 1 2 2σ 1 i=1

Now let the first r ≥ 1 observations be equal to µ0 , then the posterior can be written as follow

( π(σ1 |x) ∝ exp −

n X zi (xi − µ0 )2 2σ12 i=r+1

)

Bayesian Choice P α−1− n i=1 zi

σ1

29

(1 − σ12 )β−1 I[0,1] (σ1 ).

The posterior is defined if ( ) Z n P X zi (xi − µ0 )2 α−1− n i=1 zi π(σ1 |x) ∝ exp − (1−σ12 )β−1 dσ1 < ∞. σ 1 2 2σ [0,1] 1 i=r+1 Exercise 1.50 (Exercise 1.49 cont.) Consider an observation x from a normal distribution N (0, σ 2 ). a. If the prior distribution on σ is an exponential distribution E xp(λ), show that the posterior distribution is not defined for x = 0. b. If the prior distribution on σ is the improper prior π(σ) = σ −1 exp(−ασ −2 ), with α > 0, show that the posterior distribution is always defined. a. The normalizing constant is Z f (x|θ)π(θ)dθ m(x) = Θ ¾ ½ Z +∞ x2 1 √ exp − 2 λ exp {−σλ} dσ = 2σ 2πσ 0 Z +∞ 1 ⇒ m(0) ∝ exp {−σλ} dσ σ 0 By applying a convergence criteria for generalized integrals1 we conclude that m(0) diverges. b. The normalizing constant is Z m(x) = f (x|θ)π(θ)dθ Θ ½ ¾ Z +∞ x2 1 √ exp − 2 σ −1 exp(−ασ −2 )dσ = 2σ 2πσ 0 ½ ¾ Z +∞ 1 1 √ = exp − 2 (x2 + 2α) dσ 2σ 2πσ 2 0 ¾ ½ Z +∞ 2 1 λ(x + 2α) 1 √ exp − dλ with λ = 2 . = 2 σ λ 0 By applying the same convergence criteria as above, we find that m(x) converges.

1

Rb Let I = a+ f (x)dx if xβ f (x) −→ l 6= 0, for x −→ a+ then I converges if β < 1 and diverges otherwise.

30

Solution Manual

Exercise 1.52 (Musio and Racugno (1999)) Consider the Poisson P(θ) model Pθ (X = x) =

θx −θ e , x!

x = 0, 1, . . . ,

θ > 0,

and the prior distribution π(θ) = 1/θ. Show that for x = 0, the posterior distribution is not defined.

The normalizing constant is

Z

m(x) = = ⇒ m(0) =

θ

f (x|Θ)π(θ)dθ

Z +∞ Y n 0

θxi −θ 1 e dθ x! θ i=1 i

0

1 e−nθ dθ θ

Z +∞

By applying a convergence criteria we find that the posterior distribution is not defined in x = 0.

Exercise 1.54 (Bauwens (1991)) Consider x1 , . . . , xn i.i.d. N (θ, σ 2 ) and π(θ, σ 2 ) = σ −2(α+1) exp(−s20 /2σ 2 ). a. Compute the posterior distribution π(θ, σ 2 |x1 , . . . , xn ) and show that it Pn 2 only depends on x̄ and s = i=1 (xi − x̄)2 . b. Derive the posterior expectation Eπ [θ|x1 , . . . , xn ] and show that its behavior when α and s0 both converge to 0 depends on the limit of the ratio s20 /α − 1.

Warning: The second question is incorrect. a. Note that the prior on θ is flat. The posterior distribution is π(θ, σ 2 |x1 , ..., xn ) ∝

n Y

f (xi |θ, σ 2 )π(θ, σ 2 )

i=1

∝σ

−(n+2(α+1))

µ

s2 + s2 + n(x̄ − θ)2 exp − 0 2σ 2

¶

∝ IG((n + 2α − 1)/2, (s20 + s2 )/2)N (x̄, σ 2 /n), Pn Pn where x̄ = n1 i=1 xi and s2 = i=1 (xi − x̄)2 . This posterior only depends on x̄ and s2 . b. The expectation of σ 2 is Eπ [σ 2 |x1 , ..., xn ] =

Bayesian Choice

31

µ 2 ¶n/2+α−1 ¶! µ 2 s0 + s2 s0 + s2 1 exp − σ Γ (n/2 + α − 1) 2 2σ 2 µ µ ¶¶ n1/2 (x̄ − θ)2 exp − dθd(σ 2 ) 2n−1 σ 2 σ(2π)1/2 µ 2 ¶n/2+α−1 ¶! µ 2 Z à 1 s0 + s2 s0 + s2 −(n+2α−2) σ = d(σ 2 ) exp − Γ (n/2 + α − 1) 2 2σ 2 µ 2 ¶ µ 2 ¶!

Z Ã 2 n/2+α−1 2 + s s + s s 1 0 exp − 0 u u−2 du u(n+2α−2)/2 = Γ (n/2 + α − 1) 2 2 µ ¶n/2+α−1 µ 2 ¶−(n/2+α−2) Γ (n/2 + α − 2) s20 + s2 s0 + s2 = Γ (n/2 + α − 1) 2 2 2 2 s0 + s = (n + 2α) − 2 ZZ = σ2

Ã

−(n+2α)

Note that if α and s20 both converge to zero the prior on σ 2 is a diffuse s2 . Note also that the marginal one and the posterior mean reduces to n−2 posterior distribution of θ is the integral of a scale mixture of normal with Inverse Gamma scaling distribution. The marginal posterior is thus a Student’s t distribution. The expectation of θ is Eπ [θ|x1 , ..., xn ] = ZZ Ã

µ 2 ¶n/2+α−1 ¶! µ 2 1 s0 + s2 s0 + s2 = θ σ exp − Γ (n/2 + α − 1) 2 2σ 2 µ µ ¶¶ n1/2 (x̄ − θ)2 exp − dθd(σ 2 ) 2n−1 σ 2 σ(2π)1/2 µ 2 ¶n/2+α−1 µ 2 ¶−(n/2+α−1) s0 + s2 s0 + s2 + n(x̄ − θ)2 = 2 2 µ ¶(n/2+α−1) ¶! µ 2 Z Z à s20 + s2 + n(x̄ − θ)2 s0 + s2 σ −(n+2α) dθdσ 2 exp − θ Γ (n/2 + α − 1) 2 2σ 2 ¶(n/2+α−1) Z µ Γ ((n + 2α)/2) s20 + s2 √ = dθ θ s20 + s2 + n(x̄ − θ)2 Γ ((n + 2α − 1)/2) nπ Γ ((n + 2α)/2) √ = Γ ((n + 2α − 1)/2) nπ ¶−(n+2α−1)/2 Z µ 1 2 (x̄ − θ) dθ θ 1+ (n + 2α − 1)(s20 + s2 )/(n(n + 2α − 1)) = x̄ −(n+2α)

In this setting the posterior mean does not depend on the limit of s20 /α−1, contrary to the wording of the question!

32

Solution Manual

Exercise 1.55 Show that if the prior π(θ) is improper and the sample space X is finite, the posterior distribution π(θ|x) is not defined for some values of x.

If π(θ) is improper then

Z Θ

and

π(θ)dθ = +∞ Z

Z X ×Θ

π(θ)f (x|θ)dxdθ = +∞ ⇔

X ×Θ

m(x)π(θ|x)dxdθ = +∞

where m(x) is the normalizing constant of π(θ|x). If X is finite then Z X π(θ|x)dθ = +∞ m(x) x

Θ

which implies that m(x) = +∞ for some values of x.

Bayesian Choice

33

Chapter 2 - Decision-Theoretic Foundations Exercise 2.2 Consider four dice with respective numbers on their faces (4, 4, 4, 4, 0, 0), (3, 3, 3, 3, 3, 3), (6, 6, 2, 2, 2, 2), (1, 1, 1, 5, 5, 5). Two players roll one die each and compare their outcome. Show that the relation die [i] beats die [j] is intransitive, i.e., that for every choice of the first player the second player can choose a die so that the probability of winning is larger than 0.5. Relate this example to the Pitman closeness setting of Note 2.8.3.

Let di denote the i-th dice.

-

With the first dice d1 = (4, 4, 4, 4, 0, 0), the players have a probability of 2/3 to have 4 and a probability of 1/3 to have 0.

With the second dice d2 = (3, 3, 3, 3, 3, 3), the players are sure to have 3.

With the third dice d3 = (6, 6, 2, 2, 2, 2), the players have a probability of 1/3 to have 6 and a probability of 2/3 to have 2.

With the fourth dice d4 = (1, 1, 1, 5, 5, 5), the players have a probability of 1/2 to obtain 1 and a probability of 1/2 to obtain 5.

The first player rolls the dice d1 •

The second player rolls d2 . He wins if the other player obtains 0. The winning probability is 1/3.

• The second player rolls d3 . He wins if he obtains 6 and the other player obtains 4 or if the other player obtains 0. Thus the probability to win is 2 1 1 5 3 × 3 + 3 = 9 • The second player rolls d4 . He wins if he has 5 and the other player has 4 or if the other player obtains 0. The probability to win is 13 + 21 = 56 The first player rolls the dice d2 • The second player rolls d1 . He wins if he obtains 4. The probability is 2/3.

• The second player rolls d3 . He wins if he obtains 6. The probability is 1/3.

• The second player rolls d4 . He wins if he obtains 5. The probability is 1/2.

The first player rolls the dice d3 •

The second player rolls d1 . He wins if he obtains 4 and the other player obtains 2. The probability is 32 × 23 = 94 .

• The second player rolls d2 . He wins is he obtains 2. The probability is 2/3.

• The second player rolls d4 . He wins if he obtains 5 and the other player obtains 2. The probability is 32 × 12 = 31 .

The first player rolls the dice d4

34

Solution Manual

•

The second player rolls d1 . He wins if he obtains 4 and the other player obtains 1. The probability is 21 × 23 = 31 • The second player rolls d2 . He wins if he obtains 1. The probability is 1/2.

• The second player rolls d3 . He wins if he obtains 6 or if the other player obtains 1. The probability is 21 + 13 = 65 Thus for each choice of the first player the second player can chose a dice in such a way that the winning probability is larger than 0.5.

Exercise 2.4 Show Lemmas 2.4 and 2.5. (i) We prove that: If r1 , r2 , and r are rewards in R with r1 ≺ r2 and r1 ¹ r ¹ r2 , there exists a unique v (0 ≤ v ≤ 1) such that r ∼ vr1 + (1 − v)r2 .

Let f (v) = v r1 + (1 − v)r2 where 0 ≤ v ≤ 1. If v = 0 then f (0) = r2 and if v = 1 then f (1) = r1 . We have r1 ≺ r2 thus r1 ¹ r2 and r1 ¹ f (v)r2 .

The first derivative of f (v) is f ′ (v) = r1 − r2 < 0 thus the function f : [0, 1] → R is strictly decreasing and continuous on [0, 1]. r is in [r1 (= f (0)), r2 (= f (1))] thus ∃!v ∈]0, 1[ such that f (v) = r. (ii) We prove that: If r1 , r2 , and r3 are three rewards in R such that r2 ∼ αr1 + (1 − α)r3 U (r2 ) = αU (r1 ) + (1 − α)U (r3 ).

If r2 ∼ αr1 + (1 − α)r3 then U (r2 ) = U (αr1 + (1 − α)r3 ) = U (αr1 ) + U ((1 − α)r3 ) = αU (r1 ) + (1 − α)U (r3 ).

Exercise 2.6 In the setup of Example 2.8 on the Saint Petersburg paradox, give the average utility of a player for Saint Petersburg paradox, give the average utility of a player for δ = 1 and δ = 10. Compute the average number of games a player is ready to play in the modified game.

The gain is g = 3i with probability 0.5i . Consider X U (3i − c)P (g = 3i ) E(U (g − c)) = i∈N

X

3i − c 1 δ + 3i − c 2i i∈N X 1 X 1 1 = −δ i i 2 3 + δ − c 2i =

i∈N

i∈N

= 1 − δS(δ, c)

Bayesian Choice

35

where c is the entry cost and S(δ, c) is well defined because the general term of the series µ ¶i 1 1 1 ∼ i i i→∞ 3 +δ−c2 6 converges. The value of the average utility is approximately E(U (g − c)) ≈ 0.85572, E(U (g − c)) ≈ 0.4527,

for δ = 1 for δ = 10

To find the average number of games a player is ready to play in the modified game we are looking for arg maxE(U (gn )) = arg max ln E(U (gn )) n

where

n

½ n 3 with probability (1/2)n gn = 0 with probability 1 − (1/2)n

The expected utility of a player, who chooses to stop at the game n is 3n 1 +0 δ + 3n 2n ¶n µ 3n 1 = . δ + 3n 2

E(U (gn )) =

The first and second order derivatives are 3n−1 ∂ ln E(U (gn )) = ln 3 − ln 2 − n ∂n δ + 3n 2 n−2 ∂ ln E(U (gn )) 3 (n(n − 1) + 3)(δ + 3n ) − n2 32(n−1) = − ∂n2 (δ + 3n )2 n−2 3 = (3 − n)(δ + 3n ) ≤ 0 for n ≥ 3. (δ + 3n )2 The expected utility ln E(U (gn )) is concave for n ∈ [0, ∞[. Note that ∂ ln E(U (gn )) ¯¯ <0 ¯ ∂n n=3 33 <0 ln 3 − ln 2 − δ + 33 33 (ln 3 − ln 2 − 1) . δ>− ln 3 − ln 2 | {z } ∈[0.3,0.4]

Thus a sufficient condition for the previous inequality to hold is δ ≥ 1. Moreover for n ≥ 3

36

Solution Manual

∂ ln E(U (gn )) ¯¯ ∂ ln E(U (gn )) <0 ≤ ¯ ∂n ∂n n=3 thus the utility has a maximum for n ∈ {1, 2} arg maxE(U (gn )) = arg maxE(U (gn )). n∈N+

n∈{1,2}

Note that E(U (g1 )) ≥ E(U (g2 )) if and only if ln 3 − ln 2 − ln(δ + 3) ≥ 2(ln 3 − ln 2) − ln(δ + 9) µ ¶ 2δ+9 ⇔ ln ≥0 3δ+3 δ+9 3 ⇔ ≥ δ+3 2 ⇔ δ ≥ 9.

We conclude that the player stops the game at the first iteration if δ = 1 and at the second iteration if δ = 10.

Exercise 2.8 (Raiffa (1968)) In a first setup, a person has to choose between a sure gain of $10,000 (a1 ) and a gain of $50,000 with probability 0.89 (a2 ).

The second setup is such that a gain of $50,000 with probability 0.1 (a3 ) is opposed to a gain of $10,000 with probability 0.11 (a4 ). Show that, even if it seems natural to prefer a1 to a2 and a3 to a4 , there is no utility function preserving the order a1 ¹ a2 and a3 ¹ a4 .

Note that a1 ¹ a2 because a1 has sure gain, while a2 corresponds to a uncertain gain with a probability of 0.11 of null gain and a3 ¹ a2 because the probability of gaining $ 50,000 is higher in a3 . In terms of utility function: U ($10000) < U ($50000), which implies: a1 ≺ a2 iff a4 ≺ a3 , but this is not compatible with the preference structure. If we consider only the utility function we compare only the gain and not the risk level. In the firs setup following the utility function we prefer a2 and in the second setup we prefer a3 . Thus there does not exist a utility function preserving this preference structure: a1 ¹ a2 and a3 ¹ a4 .

Exercise 2.10 (Casella (1990)) Show that, if the function r, from R+ in R+ , is concave, then r(t) is nondecreasing and r(t)/t is nonincreasing.

Let r : R+ → R+ be concave. We show that r(t) is increasing and that r(t)/t is decreasing. If r(t) is non increasing: ∃t1 < t2 and r(t1 ) > r(t2 ) But r is concave: ∀α ∈ [0, 1], r(αt1 + (1 − α)t2 ) ≥ αr(t1 ) + (1 − α)r(t2 ) > αr(t2 ) + (1 − α)r(t2 )

Bayesian Choice

thus

37

r(αt1 + (1 − α)t2 ) > r(t2 ), ∀α ∈ [0, 1]

Consider α = 0, then r(t2 ) > r(t2 ) (false). It follows that r is increasing. Let t1 ≤ t2 . r(0) ≥ 0 and r is concave thus: ¶ · µ· ¸ · ¸ ¸ · ¸ t1 t1 t1 t1 r(t1 ) = r 1 − ×0+ × t2 ≥ 1 − r(0) + r(t2 ) t2 t2 t2 t2 thus

· r(t1 ) ≥

¸ t1 r(t2 ). t2

We conclude that r(t)/t is decreasing.

Exercise 2.11 Considering the loss proposed in Example 2.12, show that a perfect expert for N = 2 dominates a perfect expert for N = 1. Does the same phenomenon occur for N = 3?

The loss function is L(θ, p) =

N X

qi (pi − θi )2 +

N X

qi log(qi ).

i=1

i=1

A perfect expert estimates exactly θi , thus pi = θi and the resulting loss function is N X qi log(qi ).

L(θ, p) = i=1

For N = 1 the number of days there pi is forecasted, i.e. qi , is equal to one because pi is always predicted and the entropy term is null. Let N be the number of forecasts of the first perfect expert and N ′ the number of forecasts of the second perfect expert. The difference between entropy terms of two experts is N′ N X X qi′ log qi′ , qi log qi − i=1

i=1

which is equal to (q2 log q2 +q1 log q1 ) ≤ 0 when N = 2 and the second N ′ = 1.

Thus the perfect expert for N = 2 dominates the perfect expert for N ′ = 1.

When N = 3 and N ′ = 2 the dominance phenomenon does not occur as the sign of the difference between the entropy terms is unknown.

Exercise 2.19 (Raiffa and Schlaifer (1961)) Given a loss function L(θ, d) and a prior distribution π, the optimal prior decision is dπ which minimizes Eπ [L(θ, d)]. a. Consider D = {d1 , d2 } and L(θ, d1 ) = 0.5 + θ, L(θ, d2 ) = 2 − θ. Give the optimal prior decisions when π is Be(1, 1) and Be(2, 2).

38

Solution Manual

b. The value of sample information x is defined as ν(x) = Eπ [L(θ, dπ )|x] − Eπ [L(θ, δ π (x))|x], where δ π (x) is the regular Bayesian estimator of θ. Indicate why ν(x) ≥ 0 and give the value of sample information when x ∼ B(n, θ) for the above loss function and priors. c. When Θ = D = R, x ∼ N (θ, 1), and θ ∼ N (θ0 , 102 ), show that the optimal prior decision under squared error loss is dπ = θ0 and that the value of sample information is (θ0 − x)2 . Conclude by discussing the coherence of this notion. a. We have π

E (L(θ, d1 )) =

Z 1 0

Z 1

(0.5 + θ)θ1−1 (1 − θ)1−1 dθ B(1, 1)

(0.5 + θ) dθ B(1, 1) 0 · ¸1 θ2 −1 θ = B(1, 1) + = B(1, 1)−1 = 1 2 2 0 =

and π

E (L(θ, d2 )) =

Z 1 0

Z 1

(2 − θ)θ1−1 (1 − θ)1−1 dθ B(1, 1)

(2 − θ) dθ 0 B(1, 1) · ¸1 θ2 3 = B(1, 1)−1 2θ − = B(1, 1)−1 2 0 2 =

thus Eπ (L(θ, d2 )) > Eπ (L(θ, d1 )). It follows that for θ ∼ Be(1, 1), dπ = d1 is an optimal prior decision. For θ ∼ Be(2, 2) we obtain Z 1 (0.5 + θ)θ2−1 (1 − θ)2−1 dθ Eπ (L(θ, d1 )) = B(2, 2) 0 Z 1 (0.5 + θ)(θ − θ2 ) = dθ B(2, 2) 0 · 4 ¸1 θ θ3 θ2 = B(2, 2)−1 − + + 4 6 4 0 1 = 6B(2, 2) and

Bayesian Choice

Eπ (L(θ, d2 )) =

Z 1 0

Z 1

39

(2 − θ)θ2−1 (1 − θ)2−1 dθ B(2, 2)

(2 − θ)(θ − θ2 ) dθ B(2, 2) 0 ¸1 · 4 3 2 −1 θ −θ +θ = B(2, 2) 4 0 1 = 4B(2, 2) =

thus Eπ (L(θ, d2 )) > Eπ (L(θ, d1 )). We conclude that for θ ∼ Be(2, 2), dπ = d1 is a optimal prior decision. b. We have δ π = arg min ϕ(π, d|x) d

π

with ϕ(π, d|x) = E (L(θ, d)|x) (mean posterior loss) thus ∀d ∈ D, ϕ(π, d|x) = Eπ (L(θ, d)|x) ≥ ϕ(π, δ π |x) = Eπ (L(θ, δ π )|x) By setting d = dπ we obtain ν(x) = Eπ (L(θ, dπ )|x) − Eπ (L(θ, δ π )|x) ≥ 0 c. We find the value of the information when θ ∼ Be(1, 1). Following the result obtained in a., dπ = d1 . Due to the fact that θ|x ∼ Be(x + 1, n − x + 1)2 and ϕ(π, d1 |x) =

Z 1 0

L(θ, d1 )π(θ|x)dθ

Z 1

(0.5 + θ)θx (1 − θ)n−x dθ B(x + 1, n − x + 1) 0 ¸ · Z 1 Z 1 θx+1 (1 − θ)n−x dθ θx (1 − θ)n−x dθ + = B(x + 1, n − x + 1)−1 0.5 =

0

0

= B(x + 1, n − x + 1)−1 [0.5B(x + 1, n − x + 1) + B(x + 2, n − x + 1)] B(x + 2, n − x + 1) = 0.5 + B(x + 1, n − x + 1) 1 x+1 = + <1 2 n+2 while ϕ(π, d2 |x) = 2

Z 1 0

See Exercise 2.28

L(θ, d1 )π(θ|x)dθ

40

Solution Manual

=

Z 1 0

Z 1

(2 − θ)θx (1 − θ)n−x dθ B(x + 1, n − x + 1)

(2 − θ)θx (1 − θ)n−x dθ 0 B(x + 1, n − x + 1) ¸ · Z 1 Z 1 θx+1 (1 − θ)n−x dθ θx (1 − θ)n−x dθ − = B(x + 1, n − x + 1)−1 2 =

0

0

= B(x + 1, n − x + 1)−1 [2B(x + 1, n − x + 1) − B(x + 2, n − x + 1)] B(x + 2, n − x + 1) = 2− B(x + 1, n − x + 1) x+1 = 2− >1 n+2 and considering θ ∼ Be(1, 1) δ π = arg min ϕ(π, d|x) = d1 d

it follows that ν(x) = Eπ (L(θ, δ π )|x) − Eπ (L(θ, dπ )|x) = ϕ(π, d1 |x) − ϕ(π, d1 |x) = 0 We find the value of the information when θ ∼ Be(2, 2). Using the results in a., dπ = d1 . Due to θ|x ∼ Be(x + 2, n − x + 2) we find Z 1 L(θ, d1 )π(θ|x)dθ ϕ(π, d1 |x) = 0

Z 1

(0.5 + θ)θx+1 (1 − θ)n−x+1 dθ = B(x + 2, n − x + 2) 0 ¸ · Z 1 Z 1 θx+2 (1 − θ)n−x+1 dθ θx+1 (1 − θ)n−x+1 dθ + = B(x + 2, n − x + 2)−1 0.5 −1

0

0

= B(x + 2, n − x + 2) [0.5B(x + 2, n − x + 2) + B(x + 3, n − x + 2)] B(x + 3, n − x + 2) = 0.5 + B(x + 2, n − x + 2) 1 x+1 = + <1 2 n+4 and ϕ(π, d2 |x) = = =

Z 1

Z 1

L(θ, d1 )π(θ|x)dθ

0 x+1

0

(2 − θ)θ (1 − θ)n−x+1 dθ B(x + 2, n − x + 2)

0

(2 − θ)θx+1 (1 − θ)n−x+1 dθ B(x + 2, n − x + 2)

Z 1

Bayesian Choice

41

¸ · Z 1 Z 1 x+2 n−x+1 x+1 n−x+1 −1 θ (1 − θ) dθ θ (1 − θ) dθ − 2 = B(x + 2, n − x + 2) 0

0

= B(x + 2, n − x + 2)−1 [2B(x + 2, n − x + 2) − B(x + 3, n − x + 2)] B(x + 3, n − x + 2) =2− B(x + 2, n − x + 2) x+1 >1 =2− n+4 thus for θ ∼ Be(1, 1) δ π = arg min ϕ(π, d|x) = d1 d

It follows that ν(x) = Eπ (L(θ, δ π )|x) − Eπ (L(θ, dπ )|x) = ϕ(π, d1 |x) − ϕ(π, d1 |x) = 0.

Exercise 2.20 An investment plan can be implemented according to two different strategies, d1 and d2 . The benefit (or utility) of the investment depends on a rentability parameter θ ∈ R and is U (θ, di ) = ki + Ki θ. a. Given a prior distribution π on θ, what is the optimal prior decision? b. Let x ∼ N (θ, 1) and let θ ∼ N (0, 10). Give the optimal prior and posterior strategies. Give the improvement brought about by the observation of x in terms of posterior utility and expected utility. c. If there is a cost cs for the observation of x, determine the maximum cost cs when the advantage of observing x disappears. a. By definition dπ = arg maxE(U (θ, di )) d1 ,d2

= arg max(ki + Ki Eπ (θ)) d1 ,d2

b. If θ ∼ N (0, 10) then Eπ (θ) = 0. Thus the optimal prior decision is dπ = d1 if k1 > k2 and dπ = d2 otherwise.

The posterior optimal decision is arg max E(U (θ, di )|x) = arg max (ki + Ki Eπ (θ|x)). d1 ,d2

d1 ,d2

The posterior of θ given x is µ ½ ¾¶ µ ½ ¾¶ 1 1 π(θ|x) ∝ exp − (x − θ)2 exp − θ2 2 20 ½ ¾ 1 ∝ exp − (11θ2 − 20xθ) 20

42

Solution Manual

(

1 11 ∝ exp − 2 10

µ

¶2 ) 10 1 q θ− x 11 2π 10

11

10 10 x, 11 ). If K1 ≥ K2 then the optimal decision is thus π(θ|x) ∼ N ( 11

δ π = arg max(ki + Ki Eπ (θ|x)) d1 ,d2

= arg max(ki + Ki ½ =

d1 ,d2

10 x) 11

−k1 11 d1 if x ≥ Kk12 −K 2 10 d2 otherwise

The posterior gain is g(x) = E(U (θ, δ π )|x) − E(U (θ, dπ )) that is

 if δ π = d(x) 0 10 g(x) = k1 − 11 xK1 − k2 if δ π = d1 and dπ = d2  10 k2 − 11 xK2 − k1 if δ π = d2 and dπ = d1

Conclude that

¶ µ 10 g(x) = k1 + xK1 − k2 Ix≥ k2 −k1 11 Ik1 <k2 + K1 −K2 10 11 ¶ µ 10 k2 + xK2 − k1 Ix< k2 −k1 11 Ik1 <k2 K1 −K2 10 11

c. The gain in terms of expected utility is Z Z G= g(x)f (x|θ)π(θ)dxdθ ¶ Z µZ = g(x + θ)φ(x)dx π(θ)dθ R

R

where φ is the p.d.f. of a standard normal. Let k1 ≤ k2 and A= then G =

Z R

g(y + θ)φ(y)dy

µ =

k1 − k2 11 K1 − K2 10

k1 − k2 +

10 K1 θ 11

¶ Z +∞ A−θ

φ(x)dx +

10 K1 11

Z +∞ A−θ

xφ(x)dx

Bayesian Choice

43

¾ ½ (A − θ)2 10 1 10 = k1 − k2 + k1 θ (1 − Φ(A − θ)) + K1 √ exp − 11 11 2 2π 10 = (k1 − k2 ) + K1 θ − (k1 − k2 )φ(A − θ) + 11 ½ ¾ 10 1 10 1 2 √ K1 exp − (A − θ) K1 θΦ(A − θ) + 11 11 2π 2 ¶

µ

thus

Z 10 π G = (k1 − k2 ) + K1 E (θ) − (k1 − k2 ) φ(A − θ)π(θ)dθ + 11 ¾ ½ ZR Z 10 1 10 1 2 √ K1 exp − (A − θ) π(θ)dθ K1 θΦ(A − θ)π(θ)dθ + 11 11 2π 2 R R

Let us consider k1 ≥ k2 , then ½ ¾ Z 1 10 10 K2 g(y+θ)φ(y)dy = (k2 −k1 )φ(A−θ)+ K2 θφ(A−θ)− √ exp − (A − θ)2 11 11 2π 2 We conclude that µ Z Z 10 G = (k2 − k1 ) φ(A − θ)π(θ)dθ + K2 θφ(A − θ)π(θ)dθ 11 R R √ µ ¶ ¾! ½ 2 10 10 10 11 1 + 2 A2 Ik1 <k2 + √ K1 exp − 11 11 20 11 µ Z Z 10 + k1 − k2 − (k1 − k2 ) φ(A − θ)π(θ)dθ + K1 θφ(A − θ)π(θ)dθ 11 R R √ µ ¶ ¾! ½ 2 10 11 10 10 1 + 2 A2 Ik2 ≤k1 + √ K2 exp − 11 11 20 11 Exercise 2.25 Consider the case when Θ = {θ1 , θ2 } and D = {d1 , d2 , d3 }, for the following loss structure θ1 θ2

d1 2 0

d2 0 2

d3 0.5 1

a. Determine the minimax procedures. b. Identify the least favorable prior distribution. (Hint: Represent the risk space associated with the three actions as in Example 2.24.) a. The risk space associated with the three actions is given in Fig. C.1. The minimax risk is R̄ = inf supR(θ, δ) where R(θ, δ) is the loss function. Note that

δǫD θ

44

Solution Manual

(0,2)

δ2

δ3

1

(0.5,1) δ1 1

(2,0)

Fig. C.1. Risk set for the estimation of θ ∈ Θ.

sup R(d1 , θ) = 2 θ

sup R(d2 , θ) = 2 θ

sup R(d3 , θ) = 1 θ

We conclude that R̄ = 1 for δ = d3 b. Let π the prior probability on θ1 then r(π, d1 ) = Eπ (L(θ, d1 )) = 2π ∈ [0, 2] r(π, d2 ) = 2(1 − π) ∈ [1, 2] 1 1 r(π, d3 ) = π + (1 − π) = 1 − π ∈ [0.5, 1] 2 2 For all values of π, r(π, d3 ) ≤ r(π, d2 ). Thus inf r(π, di ) ∈ {r(π, δ1 ), r(π, δ3 )} i=1,2,3

and is r(π, δ1 ) r(π, δ3 )

2 3 2 if π > 3

if π ≤

then the maximum w.r.t. π is sup r(π) = sup inf r(π, d) d π π ( = sup

)

sup 2π, sup (1 − π/2)

π≤2/3

π>2/3

Bayesian Choice

45

= 4/5 for π = 25 . Thus the less favorable prior is π ∗ = 2/5.

Exercise 2.26 Consider the following risk function for Θ = {θ1 , θ2 } and D = {d1 , d2 , d3 } θ1 θ2

d1 1 2

d2 2 1

d3 1.75 1.75

a. Draw the risk diagram as in Example 2.24 and deduce the minimax estimators. b. Deduce from this example that minimaxity is not coherent in the following sense: d1 , d2 , d3 may be such that maxθ R(θ, d1 ) ≥ maxθ R(θ, d3 ) and maxθ R(θ, d2 ) ≥ maxθ R(θ, d3 ), while the minimax estimator is of the form αd1 + (1 − α)d2 . a. The minimax risk is R̄ = inf⋆ sup R(θ, δ) where R(θ, δ) is the loss function. δǫD

Note that

θ

sup R(d1 , θ) = 2 θ

sup R(d2 , θ) = 2 θ

sup R(d3 , θ) = 1.75 θ

We conclude that R̄ ≤ 1.75. Drawing the risk diagram shows that the minimax estimator is actually the randomised .5d1 + .5d2 . b. If max R(θ, d1 ) ≥ max R(θ, d3 ) and max R(θ, d2 ) ≥ max R(θ, d3 ) then d3 is θ

θ

θ

θ

a better candidate for being the minimax estimator than the two others.

But, if the minimax estimator is of the form αd1 + (1 − α)d2 then it does not involve d3 .

Exercise 2.27 Prove Lemma 2.22.

We prove that : R = sup r(π) = sup inf r(π, δ) ≤ R = inf ∗ sup R(θ, δ). π δ∈D

π

δ∈D

θ

Let δ be an estimator Z Z sup R(θ, δ) ≥ R(θ, δ)π(θ)dθ ≥ R(θ, δ π )π(θ)dθ = r(π), ∀π θ

θ

θ

46

Solution Manual

thus sup R(θ, δ) ≥ sup r(π) π

θ

⇒ inf sup R(θ, δ) = inf ∗ sup R(θ, δ) ≥ sup r(π) δ∈D

δ∈D

θ

π

θ

i.e. R ≥ R. Exercise 2.28 Consider x ∼ B(n, θ), with n known. √ √ a. If π(θ) is the beta distribution Be( n/2, n/2), give the associated posterior distribution π(θ|x) and the posterior expectation, δ π (x). b. Show that, when L(δ, θ) = (θ − δ)2 , the risk of δ π is constant. Conclude that δ π is minimax. c. Compare the risk for δ π with the risk function of δ0 (x) = x/n for n = 10, 50, and 100. Conclude about the appeal of δ π . ∗ a. We find the posterior distribution √ π(θ|x) and its posterior mean δ (x) √ when the prior on θ is Be( n/2, n/2). We have

f (x|θ)π(θ) f (x|θ)π(θ)dθ θ √ √ ¡n¢ x n−x θ n/2−1 (1 − θ) n/2−1 x θ (1 − θ) √ √ = B( n/2, n/2) √ √ B( n/2, n/2) × R 1 ¡n¢ √ √ θx (1 − θ)n−x θ n/2−1 (1 − θ) n/2−1 dθ 0 x

π(x|θ) = R

= R1 0

thus

R(θ, δ π ) = Xµ

√

√

√

θx+ n/2−1 (1 − θ)n−x+ n/2−1 dθ √

√ n/2, n − x + n/2) √ √ x + n/2 x + n/2 ∗ √ √ √ . = ⇒ δ (x) = x + n/2 + n − x + n/2 n+ n

θ|x ∼ Be(x +

b. Consider

√

θx+ n/2−1 (1 − θ)n−x+ n/2−1

Z X

L(θ, δ π )f (x|θ)dx

¶2 √ x + n/2 √ θ− = P(x) n+ n x∈ X ( ) X ¢X £ √ 2 √ ¤−2 ¡ 2 2 θ (n + n) + n/4 P(x) + x P(x) = n+ n x∈X

x∈X

√ ¤−2 + n+ n £

£ √ ¤−2 = n+ n

(Ã −2θ(n + (

³

√

!

√ X n) + n xP(x)

Bayesian Choice

47

) √ X √ − θ n(n + n) P(x)

x∈X

x∈X

√ 2 n n) + + E(x2 ) 4 ) µ √ ¶¶ √ √ √ n E(x) − θ n(n + n) , −2θ n + n − 2θ θ2 (n +

because E(x) = nθ and E(x2 ) = V(x) + (E(x))2 = nθ(1 − θ) + (nθ)2 . It follows ª £ √ √ √ ¤−2 © 2 θ (n + n)2 + n/4 − 2θ2 n n − nθ2 − n2 θ2 R(θ, δ π ) = n + n £ √ ¤−2 (n/4) = n+ n 1 √ = .

4(1 + n)2 Moreover

Z r(π) =

θ π

R(θ, δ π )π(θ)dθ = R(θ, δ π )

and, due to Lemma 2.25, δ is minimax. c. We have R(θ, δ π ) = 4(1+1√n)2 and R(θ, δ0 ) = E(L(θ, δ0 )) ³ x ´2 =E θ− n 2 −2 = n E (nθ − x) = n−2 nθ(1 − θ) thus R(θ, δ0 ) = n−1 nθ(1−θ). The graph of the risk function for the desired values is given in Fig. C.2. The figure has been generated by using the following R code cs =c(10,50,100) l=length(cs) p=100 xm=1 m=array(,dim=c(p,l+1)) m0=array(,dim=c(p,l+1)) for (i in 1:p){ m[i,1]=xm/p*i m0[i,1]=xm/p*i } for (i in 0:p-1){ for (j in 1:l){

48

Solution Manual

m[i,1+j]=1/(4*(1+sqrt(cs[j]))^2) m0[i,1+j]=m0[i,1]*(1-m0[i,1])/cs[j]

} } plot(m[,1],m[,2],type="l",lty=1,xlab="x",ylab="L(x)", main="Comparison between the risk functions",xlim=c(0,1),ylim=c(0,0.03)) lines(m[,1],m[,3],lty=2) lines(m[,1],m[,4],lty=3) lines(m[,1],m0[,2],lty=4) lines(m[,1],m0[,3],lty=5) lines(m[,1],m0[,4],lty=6) leg.txt=c("delta_{0}, n=10","delta_{0}, n=50","delta_{0}, n=100","delta, n=10","delta, n=50","delta, n=100") legend("topright",legend=paste(leg.txt),lty=c(1,2,3,4,5,6))

0.030

Comparison between the risk functions

0.015 0.000

0.005

0.010

L(x)

0.020

0.025

delta_{0}, n=10 delta_{0}, n=50 delta_{0}, n=100 delta, n=10 delta, n=50 delta, n=100

0.0

0.2

0.4

0.6

0.8

1.0

x

Fig. C.2. Representation of the Bayes and MLE estimators for different values of n

For n large the risk functions are peaked around the median θ0 = 0.5. This kind of behaviour is not present for small values of n.

Exercise 2.29 Prove Lemmas 2.25 and 2.27. (i) Let δ be an estimator different than δ0 , then:

Z sup R(θ, δ) ≥ θ

Bayesian Choice

Z

θ

R(θ, δ)π0 (θ)dθ ≥

θ

49

R(θ, δ0 )π0 (θ)dθ = r(π0 ) ≥ R(θ, δ0 )

thus δ0 is minimax. Let π0′ a prior different than π0 , then Z Z ′ ′ ′ r(π0 ) = R(θ, δ0 )π0 (θ)dθ ≤ R(θ, δ0 )π0′ (θ)dθ = r(π0 ). θ

θ

We conclude that π0 is the least favorable distribution. (ii) Let δ be an estimator different than δ0 , then ∀n ∈ N: Z Z sup R(θ, δ) ≥ R(θ, δ)πn (θ)dθ ≥ R(θ, δ0 )πn (θ)dθ = r(πn ) θ

θ

θ

thus:

sup R(θ, δ) ≥ lim r(πn ) ≥ R(θ, δ0 ). n→∞

θ

It follows that δ0 is minimax.

Exercise 2.30 Consider x ∼ N (θ, 1) and θ ∼ N (0, n). Show that the Bayes risk is equal to n/(n + 1). Conclude about the minimaxity of δ0 (x) = x.

Consider δ π (x) = Eπ (θ|x). π(θ|x) = R

f (x|θ)π(θ) f (x|θ)π(θ)dθ θ

(Z )−1 2 2 θ2 θ2 } exp{− 2n } } exp{− 2n } exp{− (x−θ) exp{− (x−θ) 2 2 √ √ √ √ = dθ 2π 2πn 2π 2πn R ) ( ½ ¾−1 2 (θ − θ2xn/(n + 1)) n = exp − n+1 2 (Z )−1 ½ ¾−1 2 n (θ − θ2xn/(n + 1)) exp{− }dθ n+1 2 R  ³ ´2    n   θ − x 2 2 n+1 x n 1 − } exp = exp{ n n   2 n+1 (n + 1)2 2 n+1     Z   R

³ 1 n

e 2 n+1

x2 n2 (n+1)2

exp{−

n θ − x n+1 n 2 n+1

´2 }dθ

−1    

 ³  ³ ´2   ´2  −1     n n  θ − x n+1  θ − x n+1 Z     − exp = exp − dθ n n     2 n+1 2 n+1    R    

50

thus

Solution Manual

θ|x ∼ N (nx/(n + 1), n/(n + 1)) ⇒ δ π (x) = nx/(n + 1).

It follows that

Z r(π) =

Θ

L(θ, δ)π(θ|x)dθ

Z µ =

R

Z =

R

n θ−x n+1 ′2

−2θn

θ′2 e

n+1

¶2

−

e

n ) (θ−x n+1 2

n n+1

2

dθ

dθ′

= E(θ′2 ) = V(θ′ ) n . where θ′ ∼ N (0, n/(n + 1)) and thus r(π) = n+1 Consider now δ0 (x) = x associated to a quadratic loss:

R(δ0 , θ) = E(x − θ)2 = 1 = lim r(πn ) = lim n→∞

n

n→∞ n + 1

with πn following N (0, 1), it follows from Lemma 2.27 that δ0 is minimax.

Exercise 2.36 Show that, under squared error loss, if two real estimators δ1 and δ2 are distinct and satisfy R(θ, δ1 ) = (θ − δ1 (x))2 = R(θ, δ2 ) = (θ − δ2 (x))2 , the estimator δ1 is not admissible. (Hint: Consider δ3 = (δ1 + δ2 )/2 or δ4 = δ1α δ21−α .) Extend this result to all strictly convex losses and construct a counter-example when the loss function is not convex.

Remember that δ1 is not admissible if there exists an estimator δ0 such 2 that R(δ1 , θ) ≥ R(δ0 , θ). Let δ0 = δ1 +δ 2 . We have R(δ0 , θ) = Ex (L(θ, δ0 )) µ ¶2 δ1 (x) + δ2 (x) = Ex θ − 2 2

= 4−1 Ex {(θ − δ1 (x)) + (θ − δ2 (x))} © ª = 4−1 Ex (θ − δ1 (x))2 + Ex (θ − δ2 (x))2 + 2Ex (θ − δ1 (x))(θ − δ2 (x)) © ª = 4−1 Ex (θ − δ1 (x))2 + Ex (θ − δ2 (x))2 absence of linear dep. = 4−1 (R(δ1 , θ) + R(δ2 , θ)) = 2−1 R(δ1 , θ), thus R(δ0 , θ) ≤ R(δ1 , θ).

Let us consider a strictly convex loss function L(θ, d), then:

Bayesian Choice

51

R(θ, δ1 ) = EL(θ, δ1 ) ≥ L(θ, E(δ1 )).

Let δ ′ = E(δ1 ) then by Jensen’s inequality, R(θ, δ1 ) ≥ R(θ, δ ′ ) thus it follows that δ1 is not admissible.

Exercise 2.42 (Zellner (1986a)) Consider the LINEX loss in R, defined by L(θ, d) = ec(θ−d) − c(θ − d) − 1. a. Show that L(θ, d) > 0 and plot this loss as a function of (θ − d) when c = 0.1, 0.5, 1, 2. b. Give the expression of a Bayes estimator under this loss. c. If x1 , . . . , xn ∼ N (θ, 1) and π(θ) = 1, give the associated Bayes estimator.

Let us define the LINEX loss function on R: L(θ, d) = ec(θ−d) − c(θ − d) − 1 a. Let g(x) = ecx − cx − 1 with x = θ − d. g is differentiable and g ′ (x) = c(ecx − 1) is null for x = 0. Now lim g(x) = lim g(x) = +∞

x→−∞

x→+∞

and g(0) = 0 then it follows that g(x) > 0 ∀x 6= 0 i.e. ∀θ 6= d, L(θ, d) > 0.

The graphical representation of the loss function is given in Fig. C.3.

The figure has been generated by using the following R code cs =c(0.1,0.5,1,2) l=length(cs) p=100 xm=140 m=array(,dim=c(p,l+1)) for (i in 1:p){ m[i,1]=xm/p*i } for (i in 1:p){ for (j in 1:l){ m[i,1+j]=exp(cs[j]*m[i,1])-cs[j]*m[i,1]-1 } } ym=max(m[,2]) plot.new() split.screen(c(2,2))

52

Solution Manual

0

40

80

120

0

40

80

120

Loss Function for c=1

Loss Function for c=2

0

L(x)

600000

x

600000

x

0

L(x)

600000 0

L(x)

600000

Loss Function for c=0.5

0

L(x)

Loss Function for c=0.1

0

40

80

120

0

x

40

80

120

x

Fig. C.3. Representation of the loss function for different values of c

screen(1) plot(m[,1],m[,2],type="l",xlab="x",ylab="L(x)", main="Loss Function for c=0.1",ylim=c(0,ym)) screen(2) plot(m[,1],m[,3],type="l",xlab="x",ylab="L(x)", main="Loss Function for c=0.5",ylim=c(0,ym)) screen(3) plot(m[,1],m[,4],type="l",xlab="x",ylab="L(x)", main="Loss Function for c=1",ylim=c(0,ym)) screen(4) plot(m[,1],m[,5],type="l",xlab="x",ylab="L(x)", main="Loss Function for c=2",ylim=c(0,ym)) b. We have

δ π = arg min ϕ(π, d|x) d

π

with ϕ(π, d|x) = E [L(θ, d)|x] . c. The likelihood is

) ( n X (xi − θ)2 1 f (xi |θ) = √ f (x|θ) = , exp − 2 ( 2π)n i i=1 n Y

and the posterior of θ|x is:

Bayesian Choice

π(θ|x) = R

53

f (x|θ)π(θ) f (x|θ)π(θ)dθ θ

∝ f (x|θ)π(θ) ( n ) X (xi − θ)2 ∝ exp − 2 i=1 ( n ) X θ2 − 2θxi ∝ exp − 2 i=1 ¾ ½ 1 = exp − (nθ2 − 2nθx) 2 n n o ∝ exp − (θ − x)2 2 thus θ|x ∼ N (x, 1/n). Moreover ϕ(π, d|x) = Eπ [L(θ, d)|x] h³ ´ i = Eπ ec(θ−d) − c(θ − d) − 1 |x £ ¤ = e−cd Eπ ecθ |x − c(Eπ [θ|x] − d) − 1

£ ¤ under suitable conditions for the existence and finiteness of Eπ ecθ |x , and taking the derivative w.r.t. d £ ¤ ϕ(π, d|x)′ = −ce−cd Eπ ecθ |x + c ¡ £ ¤¢ 1 ϕ(π, d|x)′ = 0 ⇒ d = log Eπ ecθ |x . c £ ¤ 2 Consider that θ|x ∼ N (x, 1/n), thus Eπ ecθ |x = ecx+c /2n . We conclude δ π (x) = x + c/2n.

Exercise 2.43 (Berger (1985a)) Consider x ∼ N (θ, 1), θ ∼ N (0, 1) and the loss 2 L(θ, δ) = e3θ /4 (θ − δ)2 . a. Show that δ π (x) = 2x. b. Show that δ π is uniformly dominated by δ0 (x) = x and that r(π) = +∞. a. We show that δ π (x) = 2x. Due to Corollary 2.42, δ π (x) =

Eπ (θω(θ)|x) Eπ (ω(θ)|x)

L(θ, δ) = ω(θ)(θ − δ)2 .

with

Consider that θ|x ∼ N (x/2, 1/2) (set n = 1 in the exercise 2.30) thus Z

2

(θ−x/2)2

θe3θ /4 e−1/2 2−1 √ E (θω(θ)|x) = 2π2−1 R π

dθ

54

Solution Manual

Z

2

(θ−2x)2

(θ−2x)2 2−1

 Z

3θ 2 /4 −1/2

θe3θ /4 e−1/2 2−1 √ dθ = 2π2−1 R thus Z

3θ 2 /4 −1/2

θe

e √ 2π2−1

e √ 2π2−1 R  −1 2 (θ−2x)2 Z Z e−1/2 (θ−2x)  2−1 θe−1/2 2−1 √ √ dθ dθ =  R  2π2−1 2π2−1 R

δ π (x) =

= E(θ′ )

dθ

e

(θ−2x)2 2−1

 R

dθ

−1  

θ′ ∼ N (2x, 1/2)

with

= 2x. b. We have

Z

2

1

2

e3θ /4 (θ − x)2 e− 2 (x−θ) √ dx 2π R Z 2 1 2 (θ − x)2 e− 2 (x−θ) √ dx = e3θ /4 2π R

R(θ, δ0 ) =

2

2

thus R(θ, δ0 ) = e3θ /4 Ex (θ − x)2 = e3θ /4 . Moreover Z 3θ2 /4 2 1 e (θ − 2x)2 e− 2 (x−θ) √ R(θ, δ π ) = dx 2π R 2 Z 1 (2x−2θ) 4 (2x − θ)2 e− 2 3θ 2 /4 √ dx =e 2π R © ª 2 2 = e3θ /4 Ex (θ − 2x)2 = e3θ /4 Ex (θ − x)2 + Ex (x2 ) − 2E(x(θ − x)) © ª 2 = e3θ /4 1 + Ex (x2 ) + 2Ex (x2 − xθ) thus

© ª 2 R(θ, δ π ) = e3θ /4 1 + Ex (x2 ) + 2V(x) ≥ R(θ, δ0 ).

Consider that Z r(π) = R(θ, δ π )π(θ)dθ R

Z

2

2

e3θ /4 e−θ /2 √ dθ with K = 1 + Ex (x2 ) + 2V(x) 2π R Z θ2 /4 Z +∞ θ2 /4 e e √ √ dθ =K dθ = 2K 2π 2π R 0

=K

then it follows that r(π) = ∞.

Bayesian Choice

55

Chapter 3 - From Prior Information to Prior Distributions Exercise 3.1 (Dupuis (1995)) Recall that the beta Be(α, β) distribution has a density given by π(θ) =

Γ (α + β) α−1 θ (1 − θ)β , Γ (α)Γ (β)

0 ≤ θ ≤ 1.

a. Give the mean of the Be(α, β) distribution. b. Show that there a one-to-one correspondence between (α, β) and the triplet (µ, θ0 , θ1 ), where π(θ ∈ [θ0 , θ1 ]) = p and µ is the mean of the distribution. c. What are the conditions on (µ, θ0 , θ1 ) for (α, β) to exist?

Let θ ∼ Be(α, β). a. Then E(θ) =

Z 1 0

Z 1

θπ(θ)dθ

Γ (α + β) α−1 θ (1 − θ)β−1 dθ Γ (θ)Γ (β) 0 Z Γ (α + β) 1 α = θ (1 − θ)β−1 dθ Γ (α)Γ (β) 0 Γ (α + β) = B(α + 1, β) Γ (α)Γ (β) Γ (α + β) Γ (α + 1)Γ (β) = Γ (α)Γ (β) Γ (α + β + 1) Γ (α + β) αΓ (α)Γ (β) = Γ (α)Γ (β) (α + β)Γ (α + β) α = . α+β =

θ

b. We have V(θ) = E(θ2 ) − (E(θ))2 Γ (α + β) Γ (θ + 2)Γ (β) − = Γ (θ)Γ (β) Γ (α + β + 2) αβ . = (α + β + 1)(α + β)2 It follows that

µ

α α+β

¶2

56

Solution Manual

E(θ) {(1 − E(θ)) − V(θ)} V(θ) © ª µ (1 − µ) − E(µ − θ)2 = E(θ − µ)2 (1 − E(θ)) {E(θ)(1 − E(θ)) − V(θ)} β= V(θ) © ª (1 − µ) µ(1 − µ) − E(θ − µ)2 = .

E(θ − µ)2

α=

Given a confidence interval at the significance level of p = π(θ ∈ [θ0 , θ1 ]) for θ, we define the bijection ¶ µ αβ α . , g : (α, β) 7−→ α + β (α + β + 1)(α + β)2 Warning: this question is not completed! c. θ ∼ Be(α, β) thus • supp θ = [0, 1] ⇒ [θ0 , θ1 ] ⊆ [0, 1] • α, β > 0 thus α, β are given above.

Exercise 3.2 (Seidenfeld (1987)) Consider a six-sided die, with θ the random variable corresponding to the upper face of the die. a. If π is the distribution of θ, give the maximum entropy prior associated with the information E[θ] = 3.5. b. Show that, if A is the event “θ is odd”, the updated distribution π(·|A) is (1/3, 0, 1/3, 0, 1/3, 0). c. Show that the maximum entropy prior associated with the constraints E[θ] = 3.5 and E[IA ] = 1 is (.22, 0, .32, 0, .47). a. Θ = {θ1 , θ2 , ..., θ6 } where θi corresponds to the event ”the face i-th appears” with a probability π(θi ) = 1/6 (equiprobability of all events). It follows that the maximum entropy prior is given by (1/6, 1/6, 1/6, 1/6, 1/6, 1/6). b. The event A is ”odds”={1, 3, 5} and P (A) = 36 = 12 thus ∀i ∈ {1, . . . , 6}: ( P ({ωi }) P ({ωi } ∩ A) if ωi ∈ A 1/2 = π(ωi |A) = P (A) 0 otherwise 2 It follows that π(·|A) = (1/3, 0, 1/3, 0, 1/3, 0).

Bayesian Choice

57

Exercise 3.3 Show that, if the constraints (3.1) are all associated with functions gk of the form gk (θ) = I(−∞,ak ] (θ), there is no maximum entropy prior distribution for Θ = R and π0 the Lebesgue measure on R. The maximum entropy density is given by ÃK !

X ∗ −1 π (θ) = Z exp λk gk (θ) π0 (θ) 1

R

with Z =

exp

Z Z= =

R

exp

Z a0 −∞

= +∞.

³P

´ K λ g (η) π0 (dη). Take gk (θ) = I(−∞,ak ] (θ), k k 1

ÃK X

! λk I(−∞,ak ] (η) dη

1

exp

ÃK X 1

! λk

dη +

Z +∞ a0

dη with

K \

(−∞, ak ] = (−∞, a0 ]

k=1

It follows that π ∗ (θ) does not exist.

Exercise 3.5 Let π0 be a reference measure for the maximum entropy method and π0′ a measure which is absolutely continuous with respect to π0 . a. Give examples where the maximum entropy priors associated with π0 and π0′ coincide. b. Apply this to the case when π0 is the Lebesgue measure on R, π0′ is the N (0, 1) distribution, and the constraints (3.1) are Eπ [θ] = 0, varπ (θ) = σ 2 , depending on the value of σ. ′

a. When π0 is the Lebesgue measure, if π0 is a normal or exponential distribution, and if the constraints on the prior are expressed through both first moments, then the prior with the maximum entropy associated to π0 and ′ π0 coincides. ′ b. Consider the case π0 is the Lebesgue measure on R and π0 ∼ N (0, 1). The maximum entropy prior associated to π0 is π ∗ (θ) ∝ exp{λ1 g1 (θ) + λ2 g2 (θ)}π0 (θ) with g1 (θ) = θ and g2 (θ) = θ2 because Vπ (θ) = σ 2 iff Eπ (θ2 ) = σ 2 + (Eπ (θ)) = σ 2 . Thus π ∗ (θ) ∝ exp{θλ1 + θ2 λ2 } that is π ∗ is the normal distribution N (0, σ 2 ). The maximum entropy ′ prior associated to π is

58

Solution Manual

π ∗ (θ) ∝ exp{θλ1 + θ2 λ2 } exp{−0.5θ2 } ∝ exp{θλ1 + θ2 (λ2 − 0.5)} that is the normal distribution N (0, σ 2 ).

Exercise 3.6 Consider θ ∈ R+ . Determine whether there exists a maximum entropy prior under the constraint Eπ [θ] = µ for π0 (θ) = 1 and π0 (θ) = 1/θ.

For π0 (θ) = 1 we have π ∗ (θ) ∝ eλθ which cannot be normalized to a probability function. For π0 (θ) = 1/θ, 1 λθ e θ

π ∗ (θ) ∝ which cannot be normalized because Z

1 λθ e dθ R+ θ

diverges.

Exercise 3.7 Let x ∼ P(λ).

√ a. Find the maximum entropy prior associated with π0 (θ) = 1/ θ and Eπ [θ] = 2. b. Determine the hyperparameters of the prior distribution π if π is (i) E xp(µ); (ii) G (2, ̺). c. Derive the three corresponding posterior distributions when x = 3 and compare the Bayes estimators of θ under the loss L(θ, δ) = θ(θ − δ)2 . a. There exists α such that π ∗ (θ) = exp{αθ}θ−1/2 and Eπ [θ] = 2. Thus ∗ ∗ 1 . We conclude that Eπ (θ) = 2 θ ∼ Ga( 21 , −α) under π ∗ and Eπ (θ) = − 2α 1 1 1 for α = − 4 and that θ ∼ Ga( 2 , 4 ). ′

′

b. If λ ∼ P(N ), then Eπ (λ) = N . Eπ (λ) = 2 for N = 2. If λ ∼ Ga(2, ̺), ′′ ′′ then Eπ (λ) = ̺/2. Eπ (λ) = 2 for ̺ = 4. c. We have π(λ|x) ∝ π(λ)f (x|λ) ¡ ¢ ∝ λα−1 exp{−βλ}I[0,+∞[ (λ) (exp{−λ}λx ) ∝ Ga(α + x, β + 1) ∝ Ga(α + 3, β + 1)

for x = 3. We conclude that the maximum entropy priors are given by the following table:

Bayesian Choice Prior Posterior π ∗ Ga( 27 , 54 ) π ′ Ga(4, 3) π ′′ Ga(5, 5)

d. The second and third order moments of a Ga(α, β) are E(λ2 ) = V(λ) + E(λ)2 α(α + 1) = β2 and 3

Z +∞

β α α−1+3 λ exp{−βλ}dλ Γ (α) 0 Z +∞ α(α + 1)(α + 2) β α+3 = λα+2 exp{−βλ}dλ β3 Γ (α + 3) 0 α(α + 1)(α + 2) = β3

E(λ ) =

respectively. Thus the posterior loss is ̺(π, d|x) = Eπ (L(λ, d)|x) Z = L(λ, d)π(λ|x)dλ ZR = (λ3 − 2dλ2 + d2 )π(λ|x)dλ R

α(α + 1)(α + 2) α(α + 1) = − 2d + d2 . β3 β2 The Bayes estimator is δ π (x) = arg min̺(π, d|x) δ

β α(α − 1) = α β2 α−1 = . β It follows that for x = 3 1

δ π∗ (3) = 21 = 2 4

3 δ (3) = = 1 3 π′

59

60

Solution Manual ′′

δ π (3) =

4 .

5

The prior associated to the maximum entropy gives the smallest Bayes estimator.

Exercise 3.8 Determine the prior distributions in Example 3.6, when the first and third quartiles are 2 and −2 and the median is 0.

Let x ∼ N (θ, 1) and the median of θ for the prior is 0 and the first quartile is -2 and the third one is +2. Moreover the distribution of θ is N (µ, τ ).

For the Gaussian distribution the mean and the median coincide. Thus the mean µ is null. The variance calculation is straightforward. τ is such that P (Z > −2/(τ )1/2 ) = 1/4, with Z ∼ N (0, 1) ⇔ τ = (2/0, 675)2 = 8, 78. Thus θ ∼ N (0, 8, 78).

Exercise 3.9 Let x ∼ B(n, θ) and θ ∼ Be(α, β). Determine whether there exist values of α, β such that π(θ|x) is the uniform prior on [0, 1], even for a single value of x.

π(θ|x) ∝ f (x|θ)π(θ) ¡n¢ x θ (1 − θ)n−x θα−1 (1 − θ)β−1 = x B(α, β) ∝ θx (1 − θ)n−x θα−1 (1 − θ)β−1 = θα+x−1 (1 − θ)n+β−x−1 thus θ|x ∼ Be(α + x, n + β − x). ½ θ|x ∼ U([0, 1]) ⇔

α+x=1 n+β−x=1

and, since α, β > 0, this is only possible for x = n − x = 0, thus there is no proper prior distribution producing a uniform posterior. α + β + n = 2.

Exercise 3.10 Let x ∼ Pa(α, θ), a Pareto distribution, and θ ∼ Be(µ, ν).

Show that, if α < 1 and x > 1, a particular choice of µ and ν gives π(θ|x) as the uniform prior on [0, 1].

We have π(θ|x) ∝ π(θ)f (x|θ) ¡ ¢¡ ¢ ∝ θµ−1 (1 − θ)ν−1 I[0,1] (θ) θα I[θ,+∞) (x) ∝ θα+µ−1 (1 − θ)ν−1 I(θ,+∞) (x)I[0,1] (θ).

Bayesian Choice

61

Due to the fact that x > 1 it follows that I[0,x) (θ)I[0,1] (θ) = I[0,1] (θ) and π(θ|x) = I[0,1] (θ) if and only if α + µ − 1 = 0 and ν = 1.

Exercise 3.16 Show that, if the support of f (x|θ) does not depend on θ and if there exists a parametrized conjugate prior family F = {π(θ|λ), λ ∈ Λ} with dim(Λ) < +∞, f (x|λ) is necessarily from an exponential family. (Hint: This is a consequence of the Pitman–Koopman lemma.)

We have a family f (x|θ) with a support that does not depend on θ. Assume there exists a parametric family with conjugate prior F = {π(θ|λ), λ ∈ Λ}, with dim(Λ) < +∞. Let us take π(θ|λ) as prior. The posterior is π(θ|x) ∝ π(θ|λ)f (x|θ). It follows that ∃ λ(x), π(θ|x) ∝ π(θ|λ(x))f (x|θ), because the prior is conjugate. Let us consider the normalizing constant h(x) and find: π(θ|λ(x)) = π(θ|λ)f (x|θ)/h(x), which can be written as: f (x|θ) = h(x)π(θ|λ(x))/π(θ). et g(θ, x) = π(θ|x)/π(θ). Thus f (x|θ) = h(x)g(θ, λ(x)). Due to the factorisation rule, λ(x) is a sufficient statistic. It is of constant dimension because it takes value in Λ. Due to the Pitman-Koopman Lemma and due to the fact that f (x|θ) has a constant support set and has a sufficient statistic it is from an exponential family.

Exercise 3.17 Give a sufficient statistic associated with a sample x1 , . . . , xn from a Pareto Pa(α, θ) distribution. α

The density is f (x|α, θ) = α xθα+1 I[θ,∞[ (xi ). The likelihood factorises as follows: Ã n !−α−1 Ã n !

Y Y n αn L(α, θ|x1 , . . . , xn ) = α θ xi I[θ,∞[ (xi ) i=1

= αn θαn

à n Y

! (xi )−α−1

i=1

I[θ,∞[ (wi )

i=1

where wi = min{x1 , . . . , xn }. Thus Qnthe sufficient statistic for (α, θ) is made of the product of the observations: i=1 xi and of wi .

62

Solution Manual

Exercise 3.18 Give a sufficient statistic associated with a sample x1 , . . . , xn from a truncated normal distribution 2

f (x|θ) ∝ e−(x−θ) /2 I[θ−c,θ+c] (x), when c is known.

Let x1 , . . . , xn be a iid sample from the truncated normal with density f (x|θ) ∝ I[θ−c,θ+c] (x) exp{−0.5(x − θ)2 } where c is known. The joint density of this sample is f (x1 , . . . , xn ) = n Y f (xi |θ) = i=1

(

= (2π)−n/2 exp −0.5

n X

(xi − θ)2

i=1

) n Y i=1

I[θ−c,θ+c] (xi ) (

∝ I(−∞,θ+c] (max{xi })I[θ−c,+∞) (min{xi }) exp −0.5 i

i

(

∝ I(−∞,θ+c] (max{xi })I[θ−c,+∞) (min{xi }) exp −0.5 i

We can write

i

n X i=1 n X

) 2

(xi − θ)

) 2

(−2θxi + θ ) .

i=1

f (x1 , . . . , xn |θ) ∝ g(T (x), θ)h(x)c(θ)

with g(T (x)|θ) = I(−∞,θ+c] (max{xi })I[θ−c,+∞) (min{xi }) exp {nθT (x)} i

i

Pn 1

and T (x) = n statistic for θ.

i=1 xi . Due to the factorization theorem T (x) is a sufficient

Exercise 3.19 (Brown (1986)) Show that, for every exponential family, there exists a reparameterization which gives a natural exponential family. Show also that the dimension of a natural reparameterization does not depend on the choice of the reparameterization.

Let f (x|θ) = φ(θ1 , . . . , θm )ψ(x) exp{r(θ)T (x)} be an exponential family and T (x) a sufficient statistics. Let dN = ψ(x)dλ and V (A) = N T −1 (A) for A ∈ B(Rp ). Thus the density of x for V exists and θ′ = r(θ), φ(θ) = − log(C(r−1 (θ))). If r(ω1 ) = r(ω2 ) then fω1 = fω2 and Cω1 = Cω2 . Moreover the dimension does not change after the reparameterization. Let {P0 } be a natural exponential family with dimension p, M an invertible matrix. Let z = µx + z0 and φ = (M ′ )−1 θ + φ0 where X ∼ P0 , then the distribution of Z

Bayesian Choice

63

is in the natural exponential family of dimension k. In fact the density of Z w.r.t. the measure V2 defined by V2 (A) = V (M −1 (A − z0 )) is exp{θ′ x(z) − ψ(θ)} = exp{(θ − θ0 )′ M M −1 (z − z0 ) − ψ(M ′ (φ − φ0 ))} = exp{d′ z − ψM ′ (φ − φ0 ) − φ′ z0 − d′0 z + φ0 z0 } where (A − z0 ) = {x|∃z ∈ A; x = z − z0 }. Let V1 (dz) = exp{−φ′0 z}V2 dz and ψ1 (θ) = ψ(M (φ − φ0 )) − φ′ z0 + φ0 z0 . The density of z w.r.t. V1 is exp{φ′ z − ψ1 (φ)}.

Exercise 3.23 *(Brown (1986)) A natural exponential family f (x|θ) = exp(θ· ◦

x − ψ(θ)) is said to be steep if, for every θ0 ∈N , the interior set of N , θ1 ∈ ◦

N/ N , and θ̺ = ̺θ1 + (1 − ̺)θ0 , it satisfies ∂ψ (θ̺ ) = +∞. ̺→1 ∂̺ lim

a. Show that the family is steep if and only if Eθ [||x||] = +∞ ◦

for every θ ∈ N/ N . b. Show that the Inverse Gaussian distribution, with density n o (π)−1/2 z −3/2 exp θ1 z + θ2 (1/z) − (2θ1 θ2 )1/2 + (1/2) log(−2θ2 ) where z ∈ R+ and θ1 , θ2 ∈ R− , is exponential and steep but not regular. c. Show that a minimal and steep exponential family can be re parametrized by ξ(θ) = Eθ [x] = ∇ψ(θ) and that this function defines a one-to-one ◦

◦

transformation from N to K . d. Show that, for minimal and steep exponential families, the maximum likelihood estimator of θ, θ̂(x), satisfies ξ(θ̂(x)) = x. ◦

a. If f is steep then θ ∈N

Z

Eθ (1) =

exp{Ψ (θ)} exp{θ · x}dx ◦

and due to the fact that θ ∈N ½ ¾ Z Z ∂Eθ (1) = exp{Ψ (θ)} Ψ ′ (θ) exp{θẋ}dx + x exp{θ · x}dx ∂θ

64

Solution Manual

= −Ψ ′ (θ) exp{−Ψ (θ)}

Z

Z exp{θ · x}dx + exp{Ψ (θ)}

x exp{θ · x}dx

= −Ψ ′ (θ) · 1 + Eθ (x) ⇒ ⇒ Eθ (x) = Ψ ′ (θ) ◦

In particular for f close enough to 1, then for θ̺ ∈N Eθ̺ (x) = Ψ ′ (θ̺ ) and hence Eθ̺ (x) = lim

∂Ψ

̺→1 ∂θ

= lim

∂Ψ

̺→1 ∂̺

(θ̺ ) · (θ1 − θ0 ) (θ̺ ) = +∞

◦

for all θ1 ∈ N/ N .

◦

Conversely, if Eθ (||x||) = +∞ for each θ1 ∈ N/ N then Eθ (x) has at least one infinite component. Writing Eθ (x) as a limit gives that lim

∂Ψ

̺→1 ∂̺

= +∞

as θ1 − θ0 is fixed and finite. b. For an inverse Gaussian distribution we have ¾ ½ z log(−2θ2 ) θ2 p f (z|θ1 , θ2 ) = x− 2 exp θ1 z + − 2θ1 θ2 + z 2 = h(z) exp(θ · x − Ψ (θ)) z

where h(z) = z − 2 , θ = (θ1 , θ2 ), x = (z, z1 ) and Ψ (θ) = Note that

◦

∂Ψ = ∂θ

p

2θ1 θ2 −

log(−2θ2 ) .

2

√ µ√ ¶ θ2 θ1 1 √ ,√ − .

2θ2 2θ1 2θ2 ◦

◦

Here N = R∗− × R∗− and N/ N = {0, −∞} × R− ∪ R− {0, +∞}. If θ ∈ N/ N such that ∂Ψ ∂θ → +∞ thus Eθ (||x||) = +∞ and the family is steep. On the other hand, the natural parameter space is ) ( Z θ2 − z2 N = θ : (θ1 , θ2 ) : z exp{θ1 z + }dz < ∞ = R∗− × R∗− z R+ which is not an open set and thus the family is not regular.

Bayesian Choice

65

Exercise 3.24 *(Morris (1982)) A restricted natural exponential family on R is defined by Z Pθ (x ∈ A) = exp {θx − ψ(θ)} dF (x), θ ∈ Θ.

3.19 A

a. Show that, if 0 ∈ Θ, F is necessarily a cumulative distribution function.

Otherwise, show that the transformation of F into dF0 (x) = exp {θ0 x − ψ(θ)} dF (x), for an arbitrary θ0 ∈ Θ and the replacement of θ by θ − θ0 , provides this case. b. Show that, in this restricted sense, Be(mµ, m(1 − µ)) and the lognormal distribution L ogN (α, σ 2 ) do not belong to an exponential family. c. If µ = ψ ′ (θ) is the mean of the distribution (3.19), the variance function of the distribution is defined by V (µ) = ψ ′′ (θ) = varθ (x). Show that V is indeed a function of µ and, moreover, that if the variation space of µ, Ω, is known, the couple (V, Ω) completely characterizes the family (3.19) by ¶ Z µ µZ µ m dm dm = . ψ V (m) V (m) µ0 µ0 Rµ (Notice that θ = µ0 dm/V (m).) Show that V (µ) = µ2 defines two families, depending on whether Ω = R− or Ω = R+ . d. Show that V (µ) = µ(1 − µ)/(m + 1) corresponds simultaneously to the binomial distribution B(m, µ) and to Be(mµ, m(1 − µ)). Deduce that the characterization by V is only valid for natural exponential families. e. Show that exponential families with quadratic variance functions, i.e., V (µ) = v0 + v1 µ + v2 µ2 ,

3.20

include the following distributions: normal, N (µ, σ 2 ), Poisson, P(µ), gamma, G (r, µ/r), binomial, B(m, mµ), and negative binomial, N eg(r, p), defined in terms of the number of successes before the rth failure, with µ = rp/(1 − p). f. Show that the normal distribution (respectively, the Poisson distribution) is the unique natural exponential distribution with a constant (respectively, of degree one) variance function. g. Assume v2 6= 0 in (3.20)√and define d = v12 − 4v0 v2 , discriminant of (3.20), a = 1 if d = 0 and a = dv2 otherwise. Show that x∗ = aV ′ (x) is a linear transformation of x with the variance function V ∗ (µ∗ ) = s + v2 (µ∗ )2 ,

3.21

where µ∗ = aV ′ (µ) and s = −sign(dv2 ). Show that it is sufficient to consider V ∗ to characterize natural exponential families with a quadratic

66

Solution Manual

variance function, in the sense that other families are obtained by inverting the linear transform. h. Show that (3.21) corresponds to six possible cases depending on the sign of v2 and the value of s (−1, 0, 1). Eliminate the two impossible cases and identify the families given in e, above. Show that the remaining case is v2 > 0, s = 1. For v2 = 1, show that this case corresponds to the distribution of x = log{y/(1 − y)}/π, where µ ¶ 1 π θ 1 θ y ∼ Be , |θ| < , + , − 2 π 2 π 2 and f (x|θ) =

exp[θx + log(cos(θ))] .

2 cosh(πx/2)

3.22

(The reflection formula B(0.5 + t, 0.5 − t) = π/ cos(πt) can be of use.) The distributions spanned by the linear transformations of (3.22) are called GHS(r, λ) (meaning generalized hyperbolic secant), with λ = tan(θ), r = 1/v2 , and µ = rλ. Show that the density of GHS(r, λ) can be written ¡ ¢−r/2 exp {x arctan(λ)} fr,0 (x) fr,λ (x) = 1 + λ2 (do not try to derive an explicit expression for fr,0 ).

[Note: Exercise 10.33 exhibits additional properties of the quadratic variance exponential families in terms of conjugate families and Bayes estimators.

Exercise 6.11 shows how orthogonal polynomials can be related to each distribution in the quadratic variance exponential families.] a. If 0 ∈ Θ then

Z Pθ (x ∈ A) =

exp {θx − ψ(θ)} dF (x) Z = exp {ψ(0)} dF (x) A

A

and hence F is a cumulative distribution function (up to a constant exp{ψ(0)}). If 0 ∈ / Θ, let θ0 ∈ Θ. One has Z Pθ (x ∈ A) = exp {θx − ψ(θ)} dF (x) A Z = exp {(θ − θ0 )x − ψ(θ) + ψ(θ0 )} exp {θ0 x − ψ(θ0 )} dF (x) A

Let dF0 = exp {θx − ψ(θ0 )} dF (x), then for the special case θ = θ0 one gets

Bayesian Choice

Z Pθ (x ∈ A) =

A

67

dF0

and F0 is a cumulative distribution function. b. If x ∼ Be(mµ, m(1 − µ)) then θ = {mµ, m(1 − µ)} and f (x|θ) = exp {(mµ − 1) log(x) + (m(1 − µ) − 1) log(1 − x) − log B(mµ, m(1 − µ) − 1)} cannot be written as exp{θ · x − ψ(θ)}. The same conclusion holds for x ∼ L ogN (α, σ 2 ). c. We have Z exp{θ · x − ψ(θ)}dF (x) = 1 R

and deriving on θ gives Z (x − ψ ′ (θ)) exp{θ · x − ψ(θ)}dF (x) = 0 ⇔ R Z µ= x exp{θ · x − ψ(θ)}dF (x) = ψ ′ (θ).

R

Then taking the second derivative gives Z Z ′′ ψ (θ) exp{θ · x − ψ(θ)}dF (x) + (x − ψ ′ (θ))2 exp{θ · x − ψ(θ)}dF (x) = 0 ⇔ R R Z ′′ 2 −ψ (θ) = (x − µ) exp{θ · x − ψ(θ)}dF (x) R

thus

Z V (µ)

R

(x − µ)2 exp{θ · x − ψ(θ)}dF (x)

which is a function µ because θ = ψ ′−1 (µ) and ψ(θ) = ψ(ψ ′−1 (θ)). Moreover if Ω the space of variation of µ is given, one has µ

¶ µZ µ ¶′ Z µ 1 dx dx ′ =ψ ) ψ( V (x) V (x) V (µ) µ0 µ0 µ

and

µ0 such that θ =

µZ µ

dx V (x) µ0

Rµ

¶′ = µ

µ V (µ)

dx and µ0 V (x)

ψ

µZ µ

dx V (x) µ0

¶ =

Z µ

xdx V (x) µ0

68

Solution Manual

Rµ When V (µ) = µ2 , the integral µ0 Vdx (x) is well defined on Ω if Ω ⊂ R+ or Ω ⊂ R− . In facts, it is well defined if Ω ⊂ R+ or Ω ⊂ R− . If Ω = R+ : θ = µ and dF (x|θ) = exp{µx + log µ}I]−∞,0] (x)dx. If Ω = R− : θ = −µ and dF (x|θ) = exp{−µx + log(−µ)}dx d. For a multiplicative model, x ∼ N (θ, σ 2 ) the likelihood is ¾ ½ 1 (x − θ)2 f (x|θ) ∝ exp − σ 2σ 2 if θ ∼ IN (α, µ, τ ) is the prior for θ, the posterior is ¾ ½ 1 1 1 1 2 2 −α exp{−( − )2 /(2x2 )} π(θ|x) ∝ |θ| exp ( − µ) /2τ σ σ σ x · ¸¾ ½ 1 2µ 2x2 1 1 1 2 −α−1 ( +x )− ( 2 + ) ∝ |θ| exp − 2 σ2 τ 2 σ 2τ 2xσ · ¸ ¾ ½ µ 1 1 1 − ( 2 + x) ( 2 + x2 )/2 ∝ |θ|−α−1 exp − 2 2 σ τ τ Exercise 3.26 Show that, for every exponential family, the natural space N is convex. ª © R Let N = θ : X eθ.x h(x)dµ(x) < ∞ , α ∈ [0, 1] and θ1 , θ2 ∈ N. The function θ 7→ eθ is convex, thus Z eαθ1 ·x+(1−α)θ2 ·x h(x)dµ(x) ≤ X Z Z ≤ eαθ1 ·x h(x)dµ(x) + e(1−α)θ2 ·x h(x)dµ(x) < ∞.

X

X

We conclude that N is convex.

Exercise 3.28 An alternative to the logistic regression introduced in Example 3.22 is the probit model, where Pα (yi = 1) = 1 − Pα (yi = 0) = Φ(αt xi ),

i = 1, . . . , n,

and Φ is the c.d.f. of the standard normal distribution. a. Show that this alternative does not belong to an exponential family, even conditional upon the xi ’s. b. The observation yi can be considered as the indicator function Izi ≤αt xi where zi is an unobserved N (0, 1) random variable. Show that, if the zi ’s are known, the Lebesgue measure provides an explicit posterior distribution. [Note: The interesting aspect of this remark will be made clearer in Chapter 6, since the missing data z1 , . . . , zn can be simulated.]

Bayesian Choice

69

a. We have L(yi |α) = = = = =

n Y i=1 n Y i=1 n Y i=1 n Y

n Y

P ({yi = 1})yi P ({yi = 0})1−yi

i=1

Φ(αt xi )yi (1 − Φ(αt xi ))1−yi (1 − Φ(αt xi )) (1 − Φ(αt xi )) (1 − Φ(αt xi ))

n Y i=1 n Y i=1 n Y i=1

i=1

© ª exp yi ln(Φ(αt xi )) − yi ln((1 − Φ(αt xi ))) © ª exp yi [ln(Φ(αt xi )) − ln(1 − Φ(αt xi ))] exp

( n X

) yi [ln(Φ(αt xi )) − ln(1 − Φ(αt xi ))] .

i=1

The sum in the exponential has n terms, where n is the sample size. We conclude that the model is not exponential and the use of conjugate priors is impossible. b. Let yi = Izi ≤αt xi where zi ∼ N (0, 1). The zi are known and we consider π(α) = 1. π(α|(zi ), (yi ), (xi)) ∝ π(α, (zi )|(yi ), (xi )) ∝ π(α|(xi ), (yi ))π((zi )|α, (xi ), (yi )) ∝ π(α)π((zi )) n Y φ(zi ) ∝ i=1

where φ is the p.d.f. of a standard normal. Thus π(α|(zi ), (yi ), (xi )) ∝ 1.

We can apply Gibbs sampling to simulate from the full conditional of α, (zi )|(yi ) and (xi).

Exercise 3.31 Consider x ∼ N (θ, θ) with θ > 0. a. Determine the Jeffreys prior π J (θ). b. Say whether the distribution of x belongs to an exponential family and derive the conjugate priors on θ. c. Use Proposition 3.21 to relate the hyperparameters of the conjugate priors with the mean of θ. a. The Jeffreys prior is π J (θ) = I 1/2 (θ)

70

Solution Manual

= If x ∼ N (θ, θ), then

¶¶1/2 µ µ 2 ∂ log f (x|θ) .

E ∂θ2

¶¶ µ −(x − θ)2 1 √ exp log (f (x|θ)) = 2θ2 2πθ ³√ ´ (x − θ)2 = − log 2πθ − 2θ2 2 1 3x2 2x ∂ log f (x|θ) = + − 3 2 2 4 ∂θ θ θ θ 1 3 2 2 ⇒ I(θ) = 2 + 2 − 2 = 2 θ √ θ θ θ 2 ⇒ I 1/2 (θ) = θ 1 ⇒ π J (θ) ∝ . θ

b. We have

µ

µ ¶ −(x − θ)2 1 exp f (x|θ) = √ 2θ2 2πθ µ ¶ −(x2 − 2xθ + θ2 1 exp = √ 2θ2 2πθ ¶ µ 1 (x2 − x) 1 1 exp − − = √ θ 2 2 2πθ ¶ µ 2 1 (x − x) exp(−1/2) 1 √ . exp − = θ θ 2 2π

Thus the distribution of x is in the exponential family because it can be written as: f (x|θ) = C(θ)h(x) exp(R(θ)T (x)), with C(θ) = θ1 , h(x) = exp(−1/2)(2π)−0.5 , R(θ) = − θ1 , T (x) = (x2 − x)/2. The conjugate prior on θ can be found by maximizing the likelihood function ℓ(x, θ) = f (x|θ) with ¶ µ n Y 1 −(−xi − θ)2 √ f (x|θ) = exp 2θ2 2πθ i=1 ! Ã ¶n µ n X −(xi − θ)2 1 exp = √ 2θ2 2πθ i=1 P ¶n µ ¶ µ n − i=1 (xi − θ)2 1 = √ exp 2θ2 2πθ ¶n µ Pn µ ¶ − i=1 x2i + 2θnx̄ + nθ 1 θ−n exp = √ .

2θ2 2π

Bayesian Choice

71

c. If Θ is open in Rk and θ follows πλ,X0 (θ) ∝ exp(θ)x0 − λω(θ), with X0 ∈ X , then Eπ [ξ(θ)] = Eπ [∇ψ(θ)] = Xλ0 . Consequently is x1 , ..., xn are i.i.d. f (x|θ) and Eπ [ξ(θ)|X1 , ..., Xn ] = (X0 + nx̄)(λ + n)−1 .

Exercise 3.33 *(Robert (1991)) The generalized inverse normal distribution I N (α, µ, τ ) has the density ¾ ½ 1 K(α, µ, τ )|θ|−α exp −( − µ)2 /2τ 2 , θ with α > 0, µ ∈ R, and τ > 0. a. Show that this density is well defined and that the normalizing factor is µ ¶ 2 2 α−1 α−1 µ2 K(α, µ, τ )−1 = τ α−1 e−µ /2τ 2(α−1)/2 Γ ( ) 1 F1 ; 1/2; 2 , 2 2 2τ where 1 F1 is the confluent hypergeometric function (see Abramowitz and Stegun (1964)). b. Show that this distribution generalizes the distribution of y = 1/x when x ∼ N (µ, τ 2 ). Check that the above normalizing constant is correct in this particular case. c. Deduce that the mean of I N (α, µ, τ ) is defined for α > 2 and is Eα,µ,τ [θ] =

2 2 µ 1 F1 ( α−1 2 ; 3/2; µ /2τ ) . α−1 2 τ 1 F1 ( 2 ; 1/2; µ2 /2τ 2 )

d. Show that these distributions IN (α, µ, τ ) constitute a conjugate family for the multiplicative model N (θ, θ2 ). a. Let

½ ¾ 1 f (θ) = K(α, µ, τ )|θ|−α exp −( − µ)2 /2τ 2 . θ

f is integrable around 0 because f (θ) ∼ |θ|−α exp{− θ→0

and near +∞ because

f (θ)

∼

θ→+∞

1 } −→ 0 2θ2 θ→0

|θ|−α

which is integrable for α > 1. The normalizing constant is µZ K(α, µ, τ ) =

R

−α

|θ|

¾ ¶−1 ½ 1 2 2 exp −( − µ) /2τ dθ θ

72

Solution Manual

=

¶−1 µ Z +∞ © ª |x|α−2 exp −(x − µ)2 /2τ 2 dx 2 µ

=

0

2 exp{µ2 /(2τ 2 )}

Z +∞ 0

© ª |x|α−2 exp −(x2 − 2xµ)/2τ 2 dx

¶−1

!−1 ∞ ªX 1 ³ µx ´k dx |x| exp −x /2τ = 2 exp{µ /(2τ )} k! τ 2 0 k=0 !−1 Ã Z +∞ ∞ X © 2 ª 1 µk k+α−2 2 2 2 |x| exp −x /2τ dx = 2 exp{µ /(2τ )} k! τ 2k 0 k=0 !−1 Ã Z +∞ ∞ X 1 µk (k+α−3)/2 k+α−1 2 2 (2z) exp {−z} τ dz = 2 exp{µ /(2τ )} k! τ 2k 0 k=0 Ã !−1 ∞ k X 1 µ k + α − 3 = 2 exp{µ2 /(2τ 2 )} 2(k+α−3)/2 Γ ( + 1) k! τ k−α+1 2 k=0 !−1 Ã µ 2 ¶k/2 ∞ X µ k + α − 3 1 + 1) Γ( = τ α−1 exp{µ2 /(2τ 2 )}2(α−1)/2 k! 2τ 2 2 k=0 ¶−1 µ α − 1 1 µ2 α−1 )1 F1 ( ; ; 2) = τ α−1 exp{µ2 /(2τ 2 )}2(α−1)/2 Γ ( 2 2 2 2τ Ã

2

2

Z +∞

©

α−2

2

2

b. If x ∼ N (µ, τ 2 ) and y = 1/x we have ¾ ½ 1 1 1 2 2 exp −( − µ) /2τ y 2πτ y 2 √ and thus y ∼ IG(2, µ, τ ). The normalizing constant is 1/( 2πτ ). Let us check that the normalizing constant is a particular case of the result of point a). f (y|µ, τ 2 ) = √

K(2, µ, τ )−1 = τ exp{−

1 1 µ2 µ2 1/2 1 }2 Γ ( )1 F1 ( , , 2 ) 2 2τ 2 2 2 2τ

with ∞

X 1 1 1 µ2 F1 ( , , 2 ) = 2 2 2τ k!

µ

k=0

and

µ2 2τ 2

¶k

Γ (k + 12 ) µ2 1 1 = exp{ 2τ 2 } Γ (k + 2 )Γ ( 2 )

√ K(2, µ, τ )−1 = τ 2π

c. We have Eα,µ,τ (θ) = K(α, µ, τ )

Z R

|θ|1−α exp{−(1/σ − µ)2 /2τ 2 }dθ

Bayesian Choice

73

K(α, µ, τ ) K(α + 1, µ, τ ) 2 2 √ Γ ( α2 ) 1 F1 (α/2, 1/2mµ /2τ ) = τ 2 α−1 Γ ( 2 ) 1 F1 ((α − 1)/2, 1/2mµ2 /2τ 2 ) =

Exercise 3.44 *(Dawid et al. (1973)) Consider n random variables x1 , . . . , xn , such that the first ξ of these variables has an E xp(η) distribution and the n − ξ other have a E xp(cη) distribution, where c is known and ξ takes its values in {1, 2, . . . , n − 1}. a. Give the shape of the posterior distribution of ξ when π(ξ, η) = π(ξ) and show that it only depends on z = (z2 , . . . , zn ), with zi = xi /x1 . b. Show that the distribution of z, f (z|ξ), only depends on ξ. c. Show that the posterior distribution π(ξ|x) cannot be written as a posterior distribution for z ∼ f (z|ξ), whatever π(ξ), although it only depends on z.

How do you explain this? d. Show that the paradox does not occur when π(ξ, η) = π(ξ)η −1 .

R +∞ a. Let In+1 = 0 xn+1 e−xK dx, integrating by part we get In+1 = n+1 K In = n! .

Consider K n+1 f (x|ξ, η) =

n Y

f (xi |ξ, η)

i=1

=

ξ Y

f (xi |ξ, η)

i=1

=

ξ Y i=1

n Y

f (xi |ξ, η)

i=ξ+1

ηe−ηxi

n Y

cηe−cηxi

i=ξ+1

   ξ n   X X xi  = cn−ξ η n exp −η  xi + c   i=1

i=ξ+1

then the joint posterior is π(ξ, η|x) ∝ f (x|ξ, η)π(dξ, dη) = f (x|ξ, η)π(ξ)dη.

Integrating out η we find Z ∞ f (x|ξ, η)π(ξ)dη π(ξ|x) ∝ 0

= π(ξ)

Z ∞ 0

   ξ n   X X xi  dη cn−ξ η n exp −η  xi + c   i=1

i=ξ+1

74

Solution Manual

By applying the result on the integration by part we find  π(ξ|x) ∝ π(ξ)cn−ξ n! 

ξ X

xi + c

i=1



∝ π(ξ)cn−ξ 

ξ X i=1

ξ X

zi + c

i=1

xi 

i=ξ+1

= π(ξ)cn−ξ n!(x1 )−(n+1)  

−(n+1)

n X

n X

n X

zi + c

−(n+1) zi 

i=ξ+1

−(n+1)

zi 

i=ξ+1

with z1 = 1, ξ depends only on z.

Exercise 3.53 Find a transform of θ, η = g(θ), such that the Fisher information I(η) is constant for: (i) the Poisson distribution, P(θ); (ii) the gamma distribution, G (α, θ), with α = 1, 2, 3; and (iii)the binomial distribution, B(n, θ). (i) The Fisher information is: I(θ) = Eθ

³ 2

∂ log f (x|θ) ∂θ 2

´

. If x ∼ P(θ), we find

θx x! ⇒ log f (x|θ) = −θ + x log θ − log w! ∂ 2 log f (x|θ) x ⇒ =− 2 ∂θ2 θ 1 ⇒ I(θ) = − . θ f (x|θ) = e−θ

(ii) If x ∼ G (α, θ), then: θα α−1 −θx e x Γ (α) ⇒ log f (x|θ) = α log(θ) − log(Γ (α)) + (α − 1) log x − θx ∂ 2 log f (x|θ) α ⇒ =− 2 ∂θ2 θ α ⇒ I(θ) = − 2 . θ

f (x|θ) =

If α = 1 ⇒ I(θ) = − θ12 . If α = 2 ⇒ I(θ) = − θ22 . If α = 3 ⇒ I(θ) = − θ32 . (iii) If x ∼ B(n, θ), then:

Bayesian Choice

µ ¶ n x f (x|θ) = θ (1 − θ)n−x x µ ¶ n ⇒ log f (x|θ) = log( ) + x log θ + (n − x) log(1 − θ) x x n−x ∂ 2 log f (x|θ) =− 2 + ⇒ ∂θ2 θ (1 − θ)2 ¶ µ n n − nθ n n(1 − θ) 1 1 ⇒ I(θ) = − + . =− + =n − θ (1 − θ)2 θ (1 − θ)2 1−θ θ

75

Exercise 3.56 The Fisher information is not defined when the support of f (x|θ) depends on θ. Consider the following cases: (i) x ∼ U[−θ,θ] ;

(ii) x ∼ Pa(α, θ);

(i) We have f (x|θ) = If x ∈ [−θ, θ],

½ 1

2θ

0

µ I(θ) = Eθ

2

(iii) f (x|θ) ∝ e−(x−θ) /2 I[0,θ] (x).

if x ∈ [−θ, θ] otherwise

∂ 2 log f (x|θ) ∂θ2

¶ =

1 . θ2

Thus the Fisher information is 1/θ2 if x ∈ [−θ, θ] and is not defined if x ∈] − ∞, −θ[∪]θ, +∞[. (ii) We have ½ θα α xα+1 ifx ∈ [θ, +∞[ f (x|θ) = 0 otherwise If x ∈ [θ, +∞[,

µ I(θ) = Eθ

∂ 2 log f (x|θ) ∂θ2

¶

α = − 2. θ

Thus the Fisher information is −α/θ2 if x ∈ [θ + ∞[ and is not defined if x ∈] − ∞, θ[.

2

(iii) We have f (x|θ) ∝ e−(x−θ) /2 I[0,θ] (x) ½ f (x|θ) ∝ If x ∈ [0, θ[,

µ I(θ) = Eθ

2

e−(x−θ) /2 if x ∈ [0, θ[ 0 otherwise ∂ 2 log f (x|θ) ∂θ2

¶ = −1.

Thus the Fisher information is -1 if x ∈ [0, θ[ and is not defined if x ∈ ] − ∞, 0[∪]θ, +∞[.

76

Solution Manual

Exercise 3.58 Consider x ∼ P(θ). a. Determine the Jeffreys prior π J and discuss whether the scale-invariant prior π0 (θ) = 1/θ is preferable. b. Find the maximum entropy prior for the reference measure π0J and the constraints Eπ [θ] = 1, varπ (θ) = 1. What about using π0 instead? c. Actually, x is the number of cars crossing a railroad in a period T . Show that x is distributed according to a Poisson distribution P(θ) if the interval between two arrivals is distributed according to E xp(λ). Note that θ = λT . d. Use the above derivation of the Poisson distribution to justify the use of π0 . ´´1/2 ³ ³ 2 . If x ∼ P(θ), a. The Jeffrey prior is π J (θ) = I 1/2 (θ) = Eθ ∂ log∂θf2(x|θ) then θx x! ⇒ log f (x|θ) = −θ + x log θ − log x! ∂ 2 log f (x|θ) x ⇒ =− 2 ∂θ2 θ 1 ⇒ I(θ) = − . θ ³ 2 ´ Consider that the information is also defined as: −Eθ ∂ log∂θf2(x|θ) . Thus f (x|θ) = e−θ

I(θ) = θ1 and π J (θ) = (θ)−1/2 . π J is not known, thus the prior π0 (θ) = 1/θ which is invariante by scale transform is a better choice. b. The information for θ is Eπ [gk (θ)] = ωk , k = 1, 2 with g1 (θ) = θ, ω1 = 1, g2 (θ) = θ2 , ω2 = 1 and π0J = 1/θ1/2 . The maximal entropy prior associated to that restriction is: π⋆ = R

1/θ1/2 exp(λ1 θ + λ2 θ2 ) .

1/θ1/2 exp(λ1 θ + λ2 θ2 )dθ

If we replace π0J = 1/θ1/2 by π0 = 1/θ, we have: π ⋆ (θ) = R

1/θ exp(λ1 θ + λ2 θ2 .

1/θ exp(λ1 θ + λ2 θ2 )dθ

c. Let t be the duration between two consecutive arrivals and Tt = x. If t ∼ exp(λ) and θ = λT then π(t|λ) = λe−λt .

Bayesian Choice

77

Exercise 3.60 Consider θ ∈ [a, b] and π(θ) ∝ 1/θ. a. Determine the normalizing factor of π. b. Compute pi = π(i ≤ θ < i + 1) for a ≤ i ≤ b − 1. c. Deduce the limit of pi as a goes to 0 or b goes to ∞. [Note: This exercise is related to the table entry problem, namely, that in many numerical tables the frequency of the first significant digit is log10 (1 + i−1 ) (1 ≤ i ≤ 9). See Berger (1985a, p. 86) for a detailed account.] a. We have

Z b a

b. Then pi = c. Therefore

Z π(θ)dθ = 1 log(b/a)

1 dθ = log(b) − log(a) θ

Z i+1 i

1 log((i + 1)/i) dθ = θ log(b/a)

lim pi = lim pi = 0

a→0

b→+∞

78

Solution Manual

Chapter 4 - Bayesian Point Estimation Exercise 4.1 (Smith (1988)) Consider x, a random variable with mean µ, c.d.f. F , and density f . The two functions f and f ′ are supposed to be bounded. Define a sequence of random variables yn with c.d.f.’s ¶ µ 1 1 F (y) + Hn (y), Gn (y) = 1 − n n satisfying (i) EHn [y] = n2 ; and (ii) Hn′ = hn and h′n are bounded.

Show that Gn → F , G′n = gn → f , and gn′ → f ′ , but that |µ − E[yn ]| → ∞.

As F and Hn are twice differentiable, Gn is too and ¶ µ ′ 1 ′ 1 F ′ + Hn Gn = gn = 1 − n n ¶ µ 1 1 f + hn = 1− n n and ′′

′

Gn = gn =

µ

1 1− n

¶

′

f +

1 ′ h n n

′

As f ′ and h′ are bounded, gn → f as n → +∞. The same holds for gn . Indeed f and hn are bounded so gn → f as n → +∞. The same holds for Gn because |F | ≤ 1 and |Hn | ≤ 1. But ¶ ¶ µ µ 1 1 2 µ − n = | µ − n2 | → +∞ |µ − E(µn )| = µ − 1 − n n as n → +∞.

Exercise 4.2 If ψ(θ|x) is a posterior distribution associated with f (x|θ) and a (possibly improper) prior distribution π, show that ψ(θ|x) = k(x)π(θ). f (x|θ) a. Deduce that, if f belongs to an exponential family, the posterior distribution also belongs to an exponential family, whatever π is. b. Show that if ψ belongs to an exponential family, the same holds for f .

Bayesian Choice

a. We show that

79

ψ(θ|x) = k(x)π(θ) f (x|θ)

Consider π(θ)f (x|θ) ψ(θ|x) = m(x) Z m(x) = π(θ)f (x|θ)dθ Θ

1 then taking k(x) = m(x) , it follows that:

ψ(θ|x) = k(x)π(θ) f (x|θ) Now, let f (x|θ) = C(θ)h(x)eR(θ)T (x) , and consider that ψ(θ|x) = k(x)π(θ) f (x|θ) =

C(θ)h(x)eR(θ)T (x) π(θ) m(x)

= C ′ (θ)h′ (x)eR(θ)T (x) h(x) thus ψ is within an exponential with C ′ (θ) = C(θ)π(θ) and h′ (x) = m(x) family. b. Assume that ψ is in a exponential family and that f does not belong to the exponential family. For all functions of θ C and for all measurable and positive functions h, of x we have that

f (x|θ) 6= C(θ)h(x)eR(θ)T (x) and considering the identity functions in the spaces Θ and X it follows that: f (x|θ) 6= eR(θ)T (x) (∗). Moreover due to the fact that ψ is in the exponential family (ψ(θ|x) = π(θ)h(x)eR(θ)T (x) ), by identification we have C(θ) = π(θ) h(x) = k(x) f (x|θ) = eR(θ)T (x) which contradicts (∗). We conclude that f is in the exponential family.

Exercise 4.3 *(Berger and Wolpert (1988)) In the following setting, Stein (1962b) points out some limitations of the Likelihood Principle. Assume that a value θ > 0 can be assessed either by x ∼ N (θ, σ 2 ) (with known σ 2 ) or by

80

Solution Manual

(

y ∼ f (y|θ) = cy

−1

d2 exp − 2

µ

θ 1− y

¶2 ) I[0,bθ] (y),

where b is huge and d large (50, say). a. Show that the two maximum likelihood estimators of θ are δ1 (x) = x and δ2 (y) = y. b. Consider the special case x = y = σd. Explain why the inference on θ should be the same in both cases. c. Explain why [x − 1.96 σ, x + 1.96 σ] could be proposed as a confidence interval on θ at the level 95%. d. Deduce that [y − (1.96)(y/d), y + (1.96)(y/d)] can be used as a confidence interval if y is observed. e. Show that P (y − (1.96)(y/d) < θ < y + (1.96)(y/d)) can be made as small as wished for an adequate choice of b. f. Conclude that the above confidence interval is not appropriate for large values of x = y and σ and discuss the relevance of a confidence interval for the Likelihood Principle. g. Study the problem with the prior distribution π(θ) = 1/θ. a. Let l be the likelihood and L the log-likelihood then (

µ ¶2 ) d2 θ l(θ|y) = f (y|θ) = cy exp − 1− 2 y µ ¶ 2 d2 θ L(θ|y) = log(x) − log(y) − − 2 y ¶ µ θ θ ∂L =0⇔1− = −d2 1 − =0⇔θ=y ∂θ y y −1

and

½ ¾ (x − θ)2 1 exp − l(θ|x) = f (x|θ) = √ 2σ 2 2πσ ´ ³√ 2 (x − θ) 2πσ − L(θ|x = − log 2σ 2 ∂L (x − θ) =− = 0 ⇔ θ = x. ∂θ σ2

Bayesian Choice

81

Both maximum likelihood estimators of θ are δ1 (x) = x and δ2 (y) = y. b. The inference on θ should be the same in both cases because the maximum likelihood estimators of θ for x and y are δ1 (x) = x and δ2 (y) = y respectively. c. The 95 % confidence interval for θ is: θǫ[θ − (1.96)σ, x + (1.96)σ]. x is and estimator for θǫ[θ − (1.96)σ, x + (1.96)σ]. d. y is an estimator of θ thus the confidence interval can be written as follows: θǫ[y−(1.96)σ, y+(1.96)σ]. Moreover y = σd ⇒ σ = y/d thus the confidence interval if we observe y is: θǫ[y − (1.96)(y/d), y + (1.96)(y/d)]. e. P (y − (1.96)(y/d) < θ < y + (1.96)(y/d)) = P (y(1 − (1.96/d)) < θ < y(1+1.96/d)). The thinner the interval for θ is the weaker is the probability that the true value of θ is in the confidence interval. f. For large values of x = y and σ, the confidence interval about θ is large and this represents a limit of the maximum likelihood principle.

Exercise 4.4 Show that, if p ∈ [0, 1], θ = p/(1 − p) and if π(θ) = 1/θ, the prior distribution π(p) is Haldane distribution.

We know that π(θ) = 1/θ and θ = p/(1 − p), thus: π(θ|p) = (1 − p)/p. The Jeffreys prior is µ

µ

∂ 2 log f (θ|p) π(p) = I (p) = Eθ ∂p2 ⇒ log f (θ|p) = − log(1 − p) − log p 1/2

⇒

¶¶1/2

∂ 2 log f (θ|p) 1 1 = 2− ∂p2 p (1 − p)2 (1 − p)2 − p2 = (1 − p)2 p2 1 − 2p + p2 − p2 = (1 − p)2 p2 1 − 2p = (1 − p)2 p2 1 − 2p (1 − p)2 p2 √ 1 − 2p ⇒ I 1/2 (p) = (1 − p)p

⇒ I(p) =

⇒ π(p) ∝ [(1 − p)p]

−1

Thus the prior π(p) is the Haldane distribution.

82

Solution Manual

Exercise 4.15 Consider x ∼ P(λ) and π(λ) = e−λ . The purpose of the exercise is to compare the estimators δc (x) = cx under the quadratic loss L(λ, δ) = (δ − λ)2 . a. Compute R(δc , λ) and show that δc is not admissible for c > 1. b. Compute r(π, δc ) and deduce the optimal cπ . c. Derive the best estimator δc for the minimax criterion. d. Solve the above questions for the loss ′

µ

L (λ, θ) =

¶2 δ −1 . λ

a. We have R(δc , λ) = EX [L(λ, δ)] = EX [(cx − λ)2 ] = EX [(cx − cλ + cλ + λ)2 ] = c2 V (X) + (cλ − λ)2 = c2 λ + λ2 (c − 1)2 For the ML estimator δ = x we have EX [(x−λ)2 ] = R(δ, λ) = λ < R(δc , λ) is c > 1. thus the estimator δc is not admissible in that case. b. Then r(π, δc ) = Eλ [c2 λ + λ2 (c − 1)2 ] = c2 E(λ) + (c2 − 2c + 1)E(λ2 ) and E(λ) = 1 because π(λ) is the exponential distribution with parameter 1 and E(λ2 ) = V(λ) + E(λ)2 . Then r(π, δc ) = 3c2 − 4c + 2 is minimized for c = 3/4. c. We have R(3/4x, λ) = R̄ = inf jǫD+ supλ R(δ, λ), thus supλ R(δ, λ) = c4 − 4(c2 −2c+1) because arg sup R(δ, λ) = − λ

and

Z arg

δǫD +

=

c2 2(c2 − 2c + 1)

c4 6= c = 1/4.

4(c2 − 2c + 1)

Thus there does not exist an optimal estimator δc in the minimax sense.

d. R(δc , λ) = E x

h¡

Bayesian Choice

83

¢2 i 2 cx = cλ + (c2 − 2c + 1) that is not admissible for − 1 λ

c > 1 because for δ = x R(δ, λ) = λ1 .

Exercise 4.17 Consider x ∼ B(n, p) and p ∼ Be(α, β). a. Derive the posterior and marginal distributions. Deduce the Bayes estimator under quadratic loss. b. If the prior distribution is π(p) = [p(1 − p)]−1 I(0,1) (p), give the generalized Bayes estimator of p (when it is defined). c. Under what condition on (α, β) is δ π unbiased? Is there a contradiction with Exercise 4.16? d. Give the Bayes estimator of p under the loss L(p, δ) =

(δ − p)2 . p(1 − p)

a. The posterior and the marginal are 1 π(p|x) = f (x|p)π(p) m(x) ¡n¢ x p (1 − p)n−x pα−1 (1 − p)β−1 B(α, β)I[0,1] (p) × x R1 B(α, β) 0 px (1 − p)n−x pα−1 (1 − p)β−1 dp ×

px+α−1 (1 − p)β+n−x−1 B(α + x, β + n − x)

because

Z 1 0

px+α−1 (1 − p)β+n−x−1 dp = 1 B(α + x, β + n − x)

Thus π(p|x) ∼ Be(α + x, β + n − x) and m(x) = B(α + x, β + n − x)

¡n¢ x

B(α, β)

.

The Bayes estimator associated to the quadratic loss function is the posterior mean, that is α+x δπ = α+β+n b. The posterior is π(p|x) ∝ f (x|p)π(p) px (1 − p)n−x p−1 (1 − p)−1 px−1 (1 − p)n−x−1

84

Solution Manual

that is a Beta distribution Be(x, n − x) under the condition x > 0. The Bayes estimator associated to the quadratic loss is δ π (x) =

x . n

Exercise 4.19 Consider x ∼ N (θ, 1), θ ∼ N (0, 1), and L(θ, δ) = I{δ<θ} .

Show that, in this case, there is no Bayes estimator.

Since π(θ|x) ∝ π(θ)f (x|θ) ½ 2¾ ½ ¾ θ −(x − θ)2 exp ∝ exp − 2 2 ½ ¾ 2 −2(θ − x/2) ∝ exp 2 thus θ|x ∼ N (x/2, 1/2). The mean posterior loss is ϕ(θ, δ|x) = E(L(θ, δ)|x) Z = Iδ<θ π(θ|x)dθ ZR∞ π(θ|x)dθ = 1 − F (δ) = δ

with F the c.d.f. of θ|x. We conclude that the Bayes estimator minimizing the mean posterior loss is: δ π = arg min ϕ(θ, δ|x) δ

= arg min(1 − F (δ)) = +∞. δ

Exercise 4.20 Consider x ∼ P(θ), with Θ = {θ1 , θ2 } and D = {d1 , d2 , d3 }.

The loss function is defined by the matrix ¶ µ 0 20 10 L= 50 0 20 (where Lij = L(θi , dj ), i = 1, 2, j = 1, 2, 3). Show that the Bayes estimators are of the form   d1 if x < k − log2 3, δ π (x) = d2 if x > k − 1,   d3 otherwise, and define k in terms of the prior distribution π.

Bayesian Choice

85

Let us consider the following prior π(θ1 ) = p and π(θ2 ) = 1 − p, the posterior risk is ̺(π, d|x) = L(θ1 , d)π(θ1 |x) + L(θ2 , d)π(θ2 |x) and thus exp{−θ2 }θ2x (1 − p) x! exp{−θ1 }θ1x ̺(π, d2 |x) = 20 p x! exp{−θ1 }θ1x exp{−θ2 }θ2x ̺(π, d3 |x) = 10 p + 20 (1 − p). x! x! ̺(π, d1 |x) = 50

Let us compare the posterior losses ̺(π, d1 |x) ≥ ̺(π, d2 |x) ⇔ µ ¶x p 2 θ2 ⇔ ≥ exp{θ2 − θ1 } θ1 5 1−p ln(2) − ln(5) + θ2 − θ1 + ln p − ln(1 − p) x≥ log θ2 − log θ1 (with θ1 ≤ θ2 and θ1 , θ2 > 0). It follows that ̺(π, d2 |x) ≥ ̺(π, d3 |x) iff x≥ and

ln 2 + θ1 − θ2 + ln(1 − p) − ln p ln θ1 − ln θ2 ̺(π, d1 |x) ≥ ̺(π, d3 |x)

iff x≥

ln 2 − ln 3 + θ2 − θ1 + ln p − ln(1 − p) . ln θ1 − ln θ2

Let k=− then

θ2 − θ1 + ln p − ln(1 − p) ln θ2 − ln θ1

2 (a) ̺(π, d1 |x) ≥ ̺(π, d2 |x) iff x ≤ −k + lnlnθ5−ln 2 −ln θ1 ln 2 (b) ̺(π, d2 |x) ≥ ̺(π, d3 |x) iff x ≤ k − ln θ2 −ln θ1 2 (c) ̺(π, d1 |x) ≥ ̺(π, d3 |x) iff x ≤ k + lnlnθ3−ln 2 −ln θ1

and the Bayesian optimal decision is δ π (x) = d1 if (a) and (c) are verified δ π (x) = d2 if (a) is not verified and (b) is verified δ π (x) = d3 otherwise.

86

Solution Manual

Exercise 4.21 (Ferguson (1967)) Consider x from the renormalized negative binomial distribution, µ ¶ r+x−1 x f (x|θ) = θ (1 + θ)−(r+x) , x = 0, 1, . . . , θ ∈ R∗+ , x so that Eθ [x] = rθ (i.e., θ = p/(1 − p)). The loss function is the weighted squared error (θ − δ)2 .

L(θ, δ) = θ(1 + θ) a. Give the maximum likelihood estimator of θ. b. Show that δ0 (x) = x/r has a constant risk and is a generalized Bayes estimator for π(θ) = 1 if r > 1. What happens for r = 1? c. Show that α+x−1 δα,β (x) = β+r+1 is a Bayes estimator for π(θ|α, β) ∝ θα−1 (1 + θ)−(α+β) and that this distribution is conjugate for f (x|θ). d. Deduce that δ1 (x) = x/(r + 1) is a minimax estimator. µ

r+x-1 x

¶

θx (1 + θ)−(r+x) a. We have ℓ(θ|x) = f (x|θ) = ¶¶ µµ r+x+1 + x log(θ) − (r + x) log(1 + θ) L(θ|x) = log x ∂ log L x r+x = − =0 ∂θ θ 1+θ r+x 1+θ = ⇔ θ x 1 r ⇔ +1= +1 θ x x ⇔θ= r b.

The ML estimator of θ is θ̂ = xr . ·

(θ − δ0 (x))2 θ(1 + θ) ¸ · (θ − x/r)2 =E θ(1 + θ)

R(θ, δ0 ) = E

¸

Bayesian Choice

87

£ ¤ 1 E (θ − x/r)2 θ(1 + θ) · ¸ · ¸ 1 2θ 1 2θx x2 1 = θ2 E(x) + 2 E(x2 ) E θ2 − + 2 = θ(1 + θ) r r θ(1 + θ) r r · ¸ £ ¤ 2θ 1 1 θ2 − = rθ + 2 (E(x))2 + V(x) θ(1 + θ) r r · ¸¸ · 1 rp2 1 θ2 − 2θ2 + 2 r2 θ2 + = θ(1 + θ) r 1−p · ¸ 2 p 1 −θ2 + θ2 + = θ(1 + θ) (1 − p)r p2 1 = θ(1 + θ) (1 − p)r θ p p = = θ(1 + θ) r r(1 + θ) 1 p(1 − p) p ´= . = ³ r 1+ p r =

1−p

Thus δ0 (x) = x/r has a constant risk function.

Exercise 4.23 (Casella and Strawderman (1981)) Consider x ∼ N (θ, 1) with |θ| ≤ m (m < 1). a. Show that δ m (x) = m tanh(mx) is a Bayes estimator associated with π m (θ) =

1 I{−m,m} (θ).

2

b. Show that, for the quadratic loss, r(π m , δ m ) = R(δ m , ±m) and deduce that δ m is minimax. [Note: This is actually the unique minimax estimator in this case.] c. Compare with the estimator δ U associated with the uniform prior π(θ) =

1 I[−m,m] (θ), 2m

in terms of m. [Note: Gatsonis et al. (1987) give a detailed study of the performance of δ U in term of minimaxity.] a. Under the quadratic loss the Bayes estimator is R θf (x|θ)π(θ)dθ π E (θ|x) = RΘ f (x|θ)π(θ)dθ Θ =

m(f (x|m)0.5 − f (x| − m)0.5) 0.5(f (x|m) + f (x| − m))

88

Solution Manual

1.0

Comparison between Bayesian Estimators

0.6 0.4 0.0

0.2

L(delta(x),theta)

0.8

delta^{m}, m=1 delta^{m}, m=0.99 delta^{u},m=1 delta^{u},m=0.99

0.0

0.2

0.4

0.6

0.8

1.0

x

Fig. C.4. Comparison between quadratic losses associated to δ u and δ m

exp{−0.5(x − m)2 } − exp{−0.5(x + m)2 } exp{−0.5(x − m)2 } + exp{−0.5(x + m)2 } = m tanh(mx) =m

b. We have r(π m , δ m ) =

Z Θ

R(δ m , m)dπ(m)

1 1 = R(δ m , m) − R(δ −m , −m) 2 2 = R(δ m , m) = R(δ m , −m) and δ m is minimax because R(δ m , m) ≤ r(π m ) ∀m c. δ u is the estimator defined by

  m if x ≥ m δ u = x if |x| < m  −m if x ≤ −m.

Bayesian Choice

89

Thus if |θ| < m then δ u dominates x and if m < 1 then R(θ; δ m ) < R(θ; δ u ) ∀θ ∈ [−m, m]. The following code and the graph in Fig. C.4 show a comparison between estimators p=100; xm=1; m=array(,dim=c(p,5)); mm=1; mm1=0.99;theta=0.9; for(i in 1:p) { m[i,1]=0+xm/p*i; m[i,2]=(mm*tanh(m[i,1]*mm)-theta)^2; m[i,3]=(mm1*tanh(m[i,1]*mm1)-theta)^2; m[i,4]=(mm*(m[i,1]>=mm)+m[i,1]*(abs(m[i,1])<mm)-mm*(m[i,1]<=-mm)-theta)^2; m[i,5]=(mm1*(m[i,1]>=mm1)+m[i,1]*(abs(m[i,1])<mm1)-mm1*(m[i,1]<=-mm1)-theta)^2; } plot(m[,1],m[,2],type="l",lty=1,xlab="x",ylab="L(delta(x),theta)", main="Comparison between Bayesian Estimators",xlim=c(0,1),ylim=c(0,mm)); lines(m[,1],m[,3],lty=2); lines(m[,1],m[,4],lty=3); lines(m[,1],m[,5],lty=4); leg.txt=c("delta^{m}, m=1","delta^{m}, m=0.99", "delta^{u},m=1", "delta^{u},m=0.99"); legend("topright",legend=paste(leg.txt),lty=c(1,2,3,4)) Exercise 4.25 Consider x1 , x2 i.i.d. with distribution f (x|θ) = (1/2) exp(−|x − θ|) and π(θ) = 1. Determine the Bayes estimators associated with the quadratic and absolute error losses. Same question for an additional observation.

Due to the independence of x1 and x2 f (x1 , x2 |θ) = f (x1 |θ)f (x2 |θ) =

1 exp{−|x1 − θ| − |x2 − θ|} 4

and

1 exp{−|x1 − θ| − |x2 − θ|} 4 Let x1 ≥ x2 , the Bayes estimator is the posterior mean π(θ|x1 , x2 ) =

δ π,q = E (θ|x1 , x2 ) Z = θπ(θ|x1 , x2 )dθ Z 1 θ exp{−|x1 − θ| − |x2 − θ|}dθ = 4 R Z x1 Z 1 1 x2 = θ exp{−(x1 − x2 )}dθ θ exp{−(x1 + x2 − 2θ)}dθ + 4 −∞ 4 x1

90

Solution Manual

1 + 4

Z +∞

θ exp{x1 + x2 − 2θ}dθ Z 1 x1 1 = exp{−x1 − x2 } θ exp{−2θ}dθ + exp{−(x1 − x2 )}(x2 − x1 ) 4 −∞ 4 Z +∞ 1 θ exp{−2θ}dθ + exp{x1 + x2 } 4 x2 1 1 = exp{−x1 − x2 } (x1 − 1) + exp{−(x1 − x2 )}(x2 − x1 ) 8 4 1 + exp{x1 + x2 }(x2 + 1) 4 1 = exp{x1 − x2 }(3x2 − 1) 8 x2

Note that f (θ|x1 , x2 ) is symmetric w.r.t. (x1 + x2 )/2, which is the median.

Thus x1 + x2 δ π,a = .

2 If another observation x1 , x2 , x3 ∼ f (x|θ) becomes available then π(θ|x1 , x2 , x3 ) =

1 exp{−|x1 − θ| − |x2 − θ| − |x3 − θ|} 8

Under the quadratic risk, the Bayes estimator is δ π,q = E (θ|x1 , x2 , x3 ) Z Z 1 x2 1 x1 θ exp{−x1 − x2 − x3 + 3θ)}dθ + = θ exp{−x2 − x3 + x1 − θ}dθ 8 −∞ 8 x1 Z Z 1 x3 1 +∞ θ exp{x1 + x2 + x3 − 3θ)}dθ + θ exp{−x3 + x1 + x2 − 3θ}dθ + 8 x2 8 x3 and integrating by part we find

¶¶ µ µ 1 1 1 exp{−x1 − x2 − x3 } exp{3x1 } x1 − 8 3 3 1 + exp{x1 − x2 − x3 } (exp{x2 } (x2 − 1) + exp{x1 } (1 − x1 )) 8 1 + exp{x1 + x2 − x3 } (exp{−x2 } (x2 + 1) + exp{−x3 } (x3 + 1)) 8 ¶ µ 1 1 1 + exp{x1 + x2 + x3 } exp{−x3 } x3 + 8 3 3 ¶ µ 1 1 = exp{2x1 − x2 − x3 } x1 − 24 3

δ π,q =

Bayesian Choice

91

1 + exp{x1 − x3 } (x2 − 1) 8 1 + exp{2x1 − x2 − x3 } (1 − x1 ) 8 1 1 + exp{x1 − x3 } (x2 + 1) + exp{x1 + x2 − 2x3 } (x3 + 1) 8 8 ¶ µ 1 1 + exp{x1 + x2 } x3 + 24 3 Exercise 4.41 Show that, if x ∼ G (α, β), 1/x ∼ J G (α, β) as defined in (4.12).

Let Z = 1/x and Fx , fx be the distribution and density of x. Then P(Z ≤ z) = P(1/x ≤ z) = P(x ≥ z −1 ) = 1 − P(x ≤ z−1) = 1 − Fx (z −1 ) thus the p.d.f. of Z is ¡ ¢′ f (z|α, β) = 1 − Fx (z −1 ) = z −2 fx (z|α, β) β α ¡ −1 ¢α−1 −βz−1 z e = z −2 γ(α) β α 1 −β/z e = γ(α) z α+1 it follows that 1/x ∼ IG (α, β).

Exercise 4.48 For the AR(1) model (4.18), give the covariance matrix of (x1 , . . . , xT ).

Let xt ∼ AR(1) i.e. xt = µ + ̺(xt−1 − µ) + ǫt ., and zt = xt − µ. We have γh = cov(xt , xt+h ) = cov(zt , zt+h )

∀h > 0.

By multiplying zt = ̺zt−1 + ǫt by zt+h and taking the expectation we have: γh = ̺γh−1 = ̺h γ0 .

Exercise 4.49 (Exercise 4.48 cont.) a. Show that the variance of xt is given by (4.19). b. What happens in the case ̺ = 1, since (4.19) has no meaning? c. Extend to the case when x0 is an arbitrary value.

92

Solution Manual

a. We show that the variance of xt is V(xt ) =

1 − ϕ2t 2 σ .

1 − ϕ2

Let xt = µ + ̺(xt−1 − µ) + ǫt and zt = (xt − µ), then V(zt ) = V(xt ) and V(zt ) = V (̺zt−1 + ǫt ) ! Ã t−1 X i t ̺ ǫt−i = V ̺ z0 + t=0

=

t−1 X

̺2i σ 2 =

t=0

1 − ϕ2t 2 σ .

1 − ϕ2

b. For ̺ = 1, it follows that xt − xt−1 = ǫt , thus (xt ) is a process integrated of the first order. A process xt is integrated of the first order (I(1) process) if it is not stationary and its first difference (xt − xt−1 ) is stationary.

Exercise 4.51 (Bernardo and Smith (1994)) Show that, for a two dimensional vector, ¸¶ µ · 2 σ1 ̺σ1 σ2 t t , (x1 x2 ) ∼ N2 (µ1 µ2 ) , ̺σ1 σ2 σ22 the Jeffreys prior is π(θ) ∝ (1 − ̺2 )−1 /σ1 σ2 .

Let θ = (µ1 µ2 )t and

· Σ=

σ12 ̺σ1 σ2 ̺σ1 σ2 σ22

¸

then the density and log-density of (x1 x2 )t are f (x1 , x2 |θ, Σ) =

1 1 exp{−0.5(x − θ)t Σ −1 (x − θ)} |Σ|0.5 2π

log f (x1 , x2 |θ, Σ) = −0.5(x − θ)t Σ −1 (x − θ) − 0.5 log((1 − ̺2 )σ12 σ22 ) Note that (1 − ̺2 )−1 L(x1 , x2 , µ1 , µ2 , σ1 , σ2 ) = = 0.5(x − θ)t Σ −1 (x − θ) + 0.5(1 − ̺2 )−1 log((1 − ̺2 )σ12 σ22 ) µ ¶ (x1 − µ1 )2 (x1 − µ1 )(x2 − µ2 ) (x2 − µ2 )2 = 0.5 − 2̺ + σ12 σ1 σ2 σ22 +0.5(1 − ̺2 )−1 log((1 − ̺2 )σ12 σ22 )

thus the elements of ∇2θ L are 1 ∂2L = 2 ∂µ21 σ1

Bayesian Choice

93

1 ∂2L = 2 ∂µ22 σ2 1 ∂2L = ∂µ1 ∂µ2 σ1 σ2 and finally the Jeffrey’s prior is 1 1 .

2 (1 − ̺ ) σ1 σ2

π J ∝ det((Eθ (∇2θ L))1/2 ) =

Exercise 4.52 (Bauwens et al. (1999)) For the AR(1) model (4.18), a. Show that µ is a location parameter and, therefore, that it does not appear in the Jeffreys prior. b. Show that "T −1 # ¸ ¸ · 2 · 2 X −T −1 ∂ log L(θ|x1:T ) ∂ log L(θ|x1:T ) = = 2E , E x2t .

E 2 4 2 ∂σ 2σ ∂̺ σ t=0 c. Using the stationary distribution of the yt ’s,pdeduce from E[yt2 ] = σ 2 /(1 − ̺2 ) that the Jeffreys prior π1J (σ 2 , ̺) = 1/σ 2 1 − ̺2 . b. We show that

Ã !2  t−1 X ̺i ǫt−i  E(x2t ) = E  ̺t x0 + i=0

=

2t

1−̺ 2 σ withx0 = 0 1 − ̺2

Assume xt is a centered, then xt = ̺xt−1 + ǫt = ̺ [̺xt−2 + ǫt−1 ] + ǫt = ̺2 xt−2 + ̺ǫt−1 + ǫt = ̺2 [̺xt−3 + ǫt−2 ] + ̺ǫt−1 + ǫt = ̺3 xt−3 + ̺2 ǫt−2 + ̺ǫt−1 + ǫt = ... = ̺t x0 +

t−1 X

̺i ǫt−i

i=0

by taking the expectation of the square Ã !2  t−1 X ̺i ǫt−i  E(x2t ) = E  ̺t x0 + i=0

94

Solution Manual



= E ̺2t x20 +  = ̺2t x20 + E 

t−1 X

̺i ǫt−i

t−1 X

i=0

̺j ǫt−j + 2̺t x0

j=0

t−1 X

t−1 X

 ̺i ǫt−i 

i=0



̺i ̺j ǫt−i ǫt−j 

i,j=0

= ̺2t x20 +

t−1 X

̺2i E(ǫ2t−i ) | {z } i=0 =σ 2

=

2t

1−̺ 2 σ 1 − ̺2

(x0 = 0).

c. The Jeffreys prior is

"µ

1/2 π2J (θ) ∝ [det (I(θ))]

with I(θ) = Eθ

∂logf (x|θ) ∂θ

¶2 # .

Moreover let x0 = 0, by using the elements of the main diagonal of the Fisher information matrix ¶ µ 2 ¶ µ 2 ∂ logf (x|̺, σ) ∂ logf (x|̺, σ) and − E −E ∂σ 2 ∂̺2 with

Ã

f (x|̺, σ) ∝ σ

−T

T 1 X (xt − ̺xt−1 )2 exp − 2 2σ i=1

we obtain π2J (̺) ∝ p

s

1 1 − ̺2

1−

!

1 − ̺2T .

T (1 − ̺2 )

Exercise 4.55 Give the Wold decomposition for the stationary AR(p) model.

(Hint: Use the lag polynomial representation of the AR(p) model, that is, P(B)xt = ǫt , where B d xt = xt−d .)

For a second order stationary process xt ∼ AR(p) we have: xt +

p X

φj xt−j = ǫt

j=1

and using the lag operator xt + with Φ(B) =

Pp

p X

φj xt−j = xt +

p X

j=1

j=0 φj B

j=1 j

and φ0 = 1.

φj B j xt = Φ(B)xt = ǫt

Bayesian Choice

95

Exercise 4.56 Show that the autocorrelations γs for the MA(q) model are given by (4.27).

We show that

q−|s|

X

γs =

ϑi ϑi+|s| .

i=0

Note that γs is a pair function of s. Moreover Pqwe can assume that the process xt ∼ M A(q) is centered (µ = 0 and xt = i=0 ϑi ǫt−i ). For a given lag s xt xt+s = =

q X

ϑi ǫt−i

i=0 q X q X

q X

ϑj ǫt+|s|−j

j=0

ϑi ϑj ǫt−i ǫt+|s|−i

i=0 j=0 q−|s|

X

=

ϑi ϑi+|s| ǫ2t−i +

i=0

| with constraints

½

{z

t−i=t+|s|−j

}

q X

ϑi ϑj ǫt−i ǫt+|s|−i

i,j

{z

|

t−i6=t+|s|−j

}

i ∈ [0, q] j ∈ [0, q] =⇒ i + |s| ∈ [0, q]

it follows that γs = E(xt xt+s ) q−|s|

=

X

ϑi ϑi+|s| E(ǫ2t−i )

i=0 q−|s|

= σ2

X

ϑi ϑi+|s| .

i=0

Exercise 4.58 An ARMA(p, q) model (t ≥ 1) xt − µ =

p X

̺i (xt−i − µ) +

i=1

q X

ϑj ǫt−j + ǫt ,

j=1

is invertible (Brockwell and Davis (1996, Section 3.1)) if there exists a sequence (̟j )j such that X j

|̟j | < ∞

and ǫt =

∞ X j=0

̟j xt−j .

96

Solution Manual

Show that invertibility is equivalent to the condition that Q(x) must have its roots outside the unit circle. (Hint: Use the lag polynomial representation of the ARMA(p, q) model, that is, P(B)xt = Q(B)ǫt , where B d xt = xt−d .)

Pp Pq Let µ = 0, P(B) = i=0 ̺i B i and Q(B) = j=1 where ̺0 = 1 then P(B)xt = Q(B)ǫt can be rewritten as follows ǫt = Φ(B)xt

with

Φ(x) =

P(x) Q(x)

if the following conditions hold i. P(x) is a polynomial and converges in x ; 1 ii. Q(x) < ∞, that is if Q(x) diverges or equivalently the equation Q(x) = 0 has a root outside the unitary circles The series expansion of Φ(x) converges for |x| ≤ 1 and thus for absolutely integrable coefficients.

Exercise 4.59 An ARMA(p, q) model is causal (Brockwell and Davis (1996, Section 3.1)) if there exists a sequence (ϕj )j such that X j

|ϕj | < ∞

and xt =

∞ X

ϕj ǫt−j .

j=0

Show that causality is equivalent to the condition that P(x) must have its roots outside the unit circle. (Hint: Use the lag polynomial representation of Exercise 4.58.)

If the polynomial Φ(x) in Exercise 4.58 is invertible, then ǫt = Φ(B)xt ⇔ xt = Φ−1 (B)ǫt and the model P is causal because one can take Ψ (B) = Φ−1 (B) which can be ∞ represented as j=0 ψj ǫt−j . Note that the polynomial Φ(x) is invertible if and 1 only if P(x) < ∞, that is if P(x) has roots outside of the unit circle.

Bayesian Choice

97

Chapter 5 - Tests and Confidence Regions Exercise 5.1 In the setting of Example 5.4, study the modification of the posterior probability of H0 when x = 0 and τ /σ goes to +∞. Compare with the noninformative answer associated with π(θ) = 1.

In Example 5.4 x ∼ N (θ, σ 2 ) θ ∼ N (µ, τ 2 ).

Then π(θ|x) ∼ N (µ(x), ω 2 ) with σ2 µ + τ 2 x σ2 + τ 2 σ2 τ 2 . ω2 = 2 σ + τ2

µ(x) =

In order to test if θ < 0 consider: ¶ µ ¶ µ −µ(x) −µ(x) θ − µ(x) =Φ . < P π (θ < 0|x) = pπ ω ω ω If Za0 ,a1 is the fractional quantile of order a1 /(a0 + a1 ), i.e. Φ(Za0 ,a1 ) = a1 /(a0 + a1 ), we will accept H0 if −µ(x) > Za0 ,a1 ω, i.e. if ¶ µ σ2 σ2 x < − 2 µ − 1 + 2 ωZa0 ,a1 . τ τ When x = 0 and τ /σ → +∞: µ(0) = 2

σ2 µ σ2 µ µ = 2 = 2 2 2 +τ σ (+1(τ /σ )) 1 + (τ /σ)2

and ω2 =

σ2 τ 2 σ2 τ 2 τ2 = 2 . = + 2 2 2 σ τ σ (1 + (τ /σ )) 1 + (τ /σ)2

The posterior probability of H0 is P π (θ < 0|x = 0) = Φ considering − 1+(τµ/σ)2 −µ(0) = q τ2 ω 1+(τ /σ)2

=−

µ 1 + (τ /σ)2

p

1 + (τ /σ)2 τ

³

−µ(0) ω

´

and

98

Solution Manual

µ −→ 0 =− p τ 1 + (τ /σ)2 τ /σ→+∞ it follows that P π (θ < 0|x = 0) = Φ

µ

−µ(0) ω

¶ −→

Φ(0).

τ /σ→+∞

If we consider the noninformative answer associated to π(θ) = 1, we obtain the posterior: π(θ|x) ∝ f (x|θ)π(θ) µ ¶ −(x − θ)2 ∝ exp 2σ 2 ¶ µ 2 −x + 2xθ − θ2 ∝ exp 2σ 2 ¶ µ 2xθ − θ2 ∝ exp 2σ 2 µ ¶ −(θ − x)2 ∝ exp 2σ 2 thus π(θ|x) is N (x, σ 2 ).

To test if θ < 0, we consider π

P (θ < 0 − x) = p

π

µ

−x θ−x < σ σ

¶

µ =Φ

−x σ

¶

If Za0 ,a1 is the fractional quantile of order a1 /(a0 + a1 ), i.e. Φ(Za0 ,a1 ) = a1 /(a0 + a1 ), we will accept H0 if x < −σZa0 ,a1 .

Exercise 5.2 Consider x ∼ N (θ, 1). The hypothesis to test is H0 : |θ| ≤ c versus H1 : |θ| > c when π(θ) = 1. a. Give the graph of the maximal probability of H0 as a function of c. b. Determine the values of c for which this maximum is 0.95 and the Bayes factor is 1. Are these values actually appealing? a. Consider that π(θ|x) ∝ f (x|θ)π(θ) µ ¶ −(x − θ)2 ∝ exp 2 ¶ µ 2 −x + 2xθ − θ2 ∝ exp 2

99

0.6 0.4 0.0

0.2

maximum

0.8

1.0

Bayesian Choice

0

2

4

6

8

10

c

Fig. C.5. Maximum value of P (|θ| ≤ c|x).

¶ 2xθ − θ2 2 ¶ µ −(θ − x)2 ∝ exp 2 µ

∝ exp

thus the posterior of θ is a N (x, 1). Let φ the p.d.f. of a standard normal, then note that −φ(c − x) + φ(−c − x) = 0 ⇔ φ(c − x) = φ(−c − x) 2

2

⇔ e−(c−x) /2 = e−(−c−x) /2 ⇔ x = 0.

¯ Thus the maximum of P (|θ| ≤ c¯x) is in x = 0 and its value is Φ(c)−Φ(−c).

The maximum as a function of c is given in Figure C.5 The graph has been generated in R using the following code p=10 q=100 n=p*q m=array(,dim=c(n,2))

100

Solution Manual

for (i in 1:n){ m[i,1]=i/q m[i,2]=pnorm(i/q)-pnorm(-i/q) } plot(m,type="l",xlab="c",ylab="maximum") b. The probability of H0 is P (|θ| ≤ c) = (−c ≤ θ ≤ c). This probability is P (−c − x ≤ θ − x ≤ c − x) = 0.95 thus P (θ − x ≤ c − x) = 1.95/2 and c − x = Φ−1 (0.95) ≈ 1.96.

The Bayes factor is equal to 1: Á P π (θ ∈ Θ0 |x) π(θ ∈ Θ0 ) π ¯ = 1.

B01 (x) = π P (θ ∈ Θ1 ¯x) π(θ ∈ Θ1 ) It follows that π(θ ∈ Θ0 ) = π(θ ∈ Θ1 ) = 1 and ¯ ¢ ¡ P π (θ ∈ Θ0 |x) = P π |θ| ≤ c¯x ¯ ¯ ¢ ¡ ⇔ P π (θ ∈ Θ1 ¯x) = P π |θ| > c¯x ¯ ¯ ⇔ P π (θ ∈ Θ0 ¯x) = −P π (θ ∈ Θ1 ¯x).

If the Bayes factor is equal to 1 then P π (θ ∈ Θ0 |x) = P π (θ ∈ Θ1 |x), π B01 (x) =

2Φ(c) − 1 2 − 2Φ(c)

and c − x is such that P (θ − x ≤ c − x) = 0.75. This implies c − x = 0 ⇔ c = x. The value of c such that the maximum probability of H0 is 0.95 and the Bayes factor is equal to 1 are such that: c = x + 1.96 and c = x ⇒ x = 0 and c = 1.96.

Exercise 5.6 When x ∼ N (θ, 1) and θ ∼ N (0, σ 2 ), compare the Bayesian answers for the two testing problems H01 : θ = 0 versus H11 : θ 6= 0, H02 : |θ| ≤ ǫ versus H12 : |θ| > ǫ, when ǫ and σ vary.

Note that

¶ µ ¶ µ 1 (x − θ)2 1 θ2 √ f (x|θ)π(θ) = √ exp − exp − 2 2 2σ 2π 2πσ 2 ¶ µ 2 2 2 x θ θ 1 exp − + xθ − − 2 = 2πσ 2 2 2σ ¶ µ 2 2 x σ 2 + 1 2σ 2 1 2σ + 1 exp − + θx − θ = 2πσ 2 2σ 2 σ 2 + 1 2σ 2

Bayesian Choice

101

¶¶ µ µ 2 σ2 x σ2 + 1 1 2 θ − 2xθ 2 exp − − = 2πσ 2 2σ 2 σ +1 à ¶2 ! ¶ µ µ 1 σ2 x2 σ 2 x2 σ2 + 1 2 = exp θ − x exp − − 2πσ 2 2σ 2 σ2 + 1 2(σ 2 + 1) à ¶2 ! µ 1 σ2 x2 σ2 + 1 2 = θ −x 2 exp − − 2πσ 2(σ 2 + 1) 2σ 2 σ +1 and that π(θ|x) = (f (x|θ)π(θ))/m1 (x), where µ ¶ x2 1 exp − . m1 (x) = p 2(σ 2 + 1) 2π(1 + σ 2 ) ´ ³ 2 2 x , σ2σ+1 .

Thus the posterior distribution of θ is N σσ2 +1 -

H01

It is a point-null hypothesis. The Bayes factor is ¡ ¢ (2π)−0.5 exp −x2 /2 f (x|0) π B01 = = m1 (x) (2π(1 + σ 2 ))−0.5 exp (−x2 /(2(1 + σ 2 )))

and the posterior with ̺1 = 1/2 is ¸−1 · 1 − ̺0 m1 (x) π(0|x) = 1 + ̺0 f (x|1/2) " r ¶#−1 µ σ 2 x2 1 exp = 1+ 1 + σ2 2(1 + σ 2 ) -

H02 P (|θ| ≤ ǫ|x) = P (−ǫ ≤ θ ≤ ǫ) µ ¶ µ ¶ ǫ(1 + σ 2 ) − σ 2 x −ǫ(1 + σ 2 ) − σ 2 x √ √ =Ψ −Ψ . σ 1 + σ2 σ 1 + σ2 Tab. C.1 shows some results for x = 0.2 and σ = 1. Note that P (H01 ) = 0.5799578. ǫ 0.1 0.2 0.5 1 P (H02 ) 0.1113 0.2205 0.5161 0.8386 Table C.1. Value of P (H02 ) for different values of ǫ.

The R code used in this solution is:

102

Solution Manual

proba1=function(x,sigma){ ((1+sqrt(1/(1+sigma^2)))*exp((sigma^2*x^2)/ (2*(1+sigma^2))))^(-1) } proba2=function(x,sigma,epsilon){ pnorm((epsilon*(1+sigma^{2})-sigma^{2}*x)/ (sqrt(1+sigma^{2})*sigma))pnorm((-epsilon*(1+sigma^{2})-sigma^{2}*x)/ (sqrt(1+sigma^{2})*sigma))} x=0.2 sigma=1 proba1(x,sigma) proba2(x,sigma,0.1) proba2(x,sigma,0.2) proba2(x,sigma,0.5) proba2(x,sigma,1) Exercise 5.7 In the setting of Example 5.3, if x ∼ B(n, p) and H0 : p = 1/2 is to be tested, study the variation of the Bayesian answers as a function of n for x = 0 and x = n/2 if the prior distribution is the Jeffreys distribution.

Let f (x|p) =

µ ¶ n x p (1 − p)n−x x

be the likelihood, the posterior of p can be determined using the Jeffreys approach. ¶¶1/2 µ µ 2 ∂ log f (x|p) π(p) = I (p) = Eθ ∂p2 2 x n−x ∂ log f (θ|p) = 2 − ∂p2 p (1 − p)2 · ¸ 1 n 1 ⇒ I(p) = n = + p 1−P p(1 − p) r n ⇒ I 1/2 (p) = p(1 − p) 1/2

−1/2

⇒ π(p) ∝ [p(1 − p)]

.

Thus the Jeffreys prior π J (p) is the density of a Be(1/2, 1/2). The posterior π(p|x) follows from: π(p|x) ∝ f (x|p)π(p) µ ¶ p−1/2 (1 − p)−1/2 n x ∝ p (1 − p)n−x x B(1/2, 1/2)

Bayesian Choice

103

∝ px−1/2 (1 − p)n−x−1/2 .

Thus the posterior is a Be(x + 1/2, n − x + 1/2).

In order to test H0 : p = 1/2, let π0 be the prior on p = 1/2 and g1 the prior on Θ1 = {p 6= 1/2}. the posterior of p = 1/2 is: ¸−1 · 1 − π 0 m1 (x) π(Θ0 |x) = 1 + π 0 f (x|θ0 ) thus f (x|θ0 ) = f (x|p = 1/2) = π(p) = with marginal

µ ¶ n 1 , x 2n

p−1/2 (1 − p)−1/2 ⇒ π0 = 2π B(1/2, 1/2)

µ ¶ n B(x + 1/2, n − x + 1/2) f (x|θ)g1 (θ)dθ = m(x) = . x B(1/2, 1/2) Θ1 Z

Thus

 π(Θ0 |x) = 1 + =

1 − 2π 2π

¡n¢ B(x+1/2,n−x+1/2) −1 x

¡b(1/2,1/2) ¢ n 1



x 22

¸−1 · n 2 B(x + 1/2, n − x + 1/2) .

2πB(1/2, 1/2)

The Bayes factor is:

¡n¢ 1 f (x|θ0 ) B(1/2, 1/2) x 2n B (x) = = n = ¡n¢ B(x+1/2,n−x+1/2) . m1 (x) 2 B(x + 1/2, n − x + 1/2) π

x

If x = 0

B(1/2,1/2)

¸−1 · n 2 B(1/2, n + 1/2) 2πB(1/2, 1/2) £ n−1 −2 ¤−1 = 2 π B(1/2, n + 1/2) B(1/2, 1/2) B π (x) = π 2 B(1/2, n + 1/2) £ ¤−1 = π −1 2π B(1/2, n + 1/2))

π(Θ0 |x) =

Some numerical results are in Tab. C.2.

If x = n/2,

104

Solution Manual Table C.2. Numerical Results for x = 0 n 2 4 6 10 100 π B01 0.1501 0.03638 0.0090 0.0006 4.46e-31 P (H01 ) 0.1305 0.0351 0.0089 0.0006 4.46e-31 Table C.3. Numerical Results for x = n/2 n 2 4 6 10 100 π B01 0.4502 0.4244 0.4158 0.4090 0.3999 P (H01 ) 0.3104 0.2980 0.2937 0.2903 0.2857

· n ¸−1 2 B(n/2 + 1/2, n − n/2 + 1/2) 2πB(1/2, 1/2) £ n−1 −2 ¤−1 = 2 π B((n + 1)/2, (n + 1)/2)

π(Θ0 |x) =

B(1/2, 1/2) 2π B(n/2 + 1/2, n − n/2 + 1/2) £ ¤−1 = π −1 2n B((n + 1)/2, (n + 1)/2)

B π (x) =

Some numerical results are provided in Tab. C.3.

We use the following R code to obtain the numerical results. factor=function(x,n) { ((1/2)^{n}*gamma(n+1))/(sqrt(n)*gamma(x+1/2)*gamma(n-x+1/2)) } factor(0,2) factor(0,4) factor(0,6) factor(0,10) factor(0,100) factor(1,2) factor(2,4) factor(3,6) factor(5,10) factor(50,100) posterior=function(x,n) { (1+1/factor(x,n))^{-1} } posterior(0,2) posterior(0,4) posterior(0,6)

Bayesian Choice

105

posterior(0,10) posterior(0,100) posterior(1,2) posterior(2,4) posterior(3,6) posterior(5,10) posterior(50,100) Exercise 5.8 (Berger and Delampady (1987)) Consider x ∼ N (θ, 1). The purpose of the exercise is to compare H0 : |θ −θ0 | ≤ ǫ with the approximation H0∗ : θ = θ0 . Denote by g0 and g1 the prior densities on {|θ − θ0 | ≤ ǫ} and {|θ − θ0 | > ǫ}. Let g be a density on R such that g(θ) ∝ g1 (θ) and

if |θ − θ0 | > ǫ,

Z λ=

is small enough. We denote R f (x|θ)g0 (θ) dθ |θ−θ0 |≤ǫ B=R f (x|θ)g1 (θ) dθ |θ−θ0 |>ǫ

|θ−θ0 |≤ǫ

and

g(θ) dθ,

B̂ =

f (x|θ) f (x|θ0 ) =R , mg (x) f (x|θ)g(θ) dθ

t = (x − θ0 ) and γ=

1 [Φ(t + ǫ) − Φ(t − ǫ)] − 1.

2ǫϕ(t)

Show that, if |t| ≥ 1, ǫ < |t| − 1, and B̂ ≤ (1 + γ)−1 , then B = B̂(1 + ̺) with −λ ≤

λ(B̂ − 1) 1 − λB̂

≤̺≤

γ + λ(1 + γ)(B̂ − 1) 1 − λB̂(1 + γ)

≤ γ.

Due to the properties of g0 we can write Z t 1 g0 (θ) = I(θ0 −a,θ0 +a) F (da) 2a 0 thus

Z Ω

f (x|θ)g0 (θ)dθ =

Z t ÃZ θ0 +a 0

θ0 −a

! f (x|θ)dθ F (da)

For |t| ≥ 1 and a < ǫ, we say that the integral between parentheses is increasing in a. Thus

106

Solution Manual

Z

f (x|θ0 ) ≤

Ω

1 2ǫ

f (x|θ)g0 (θ)dθ ≤

Z Ω

f (x|θ)dθ

that can be written as Z Ω

f (x|θ)g0 (θ)dθ = f (x|θ0 )(1 + ψ1 )

where 0 ≤ ψ1 ≤ γ. Under H0 g1 (θ) = Thus

g(θ) .

(1 − λ)

Z

1 f (x|θ)g1 (θ)dθ = 1 − λ H0

Z H0

f (x|θ)q(θ)dθ

Z

= mg (x) −

H0

f (x|θ)g(θ)dθ

= mg (x)(1 + ψ2 ) where

1 ψ2 = 1−λ

Ã

1 1− mg

! g(θ) dθ . f (x|θ) λ H0c

Z

Due to the properties g(θ)/λ we have Z g(θ) f (x|θ) dθ = f (x|θ0 )(1 + ψ1∗ ) c λ H0 where 0 ≤ ψ1∗ ≤ γ. Combining the previous results with λ λ (1 − B̂(1 + γ)) ≤ ψ2 ≤ (1 − B̂) λ−1 1−λ we find B=

f (x|θ0 )(1 + ψ1 ) = B(1 + ̺) mg (1 + ψ2 )

where 1 1 + λ(1 − λ)−1 (1 − B̂)

≤ (1 + ̺) ≤

1+γ 1 + λ(1 − λ)−1 (1 − B̂(1 + γ))

The result follows by considering 0 ≥ B̂ ≥ (1 + γ)−1 .

Bayesian Choice

107

Exercise 5.9 Consider x ∼ P(λ). The hypothesis to test is H0 : λ ≤ 1 versus H1 : λ > 1. Give the posterior probability of H0 for x = 1 and λ ∼ G (α, β). a. How does this probability get modified when α and β go to 0? Does this answer depend on the rates of convergence of α and β to 0? b. Compare with the probability associated with the noninformative distribution π(λ) = 1/λ. Is it always possible to use this improper prior?

We find the posterior distribution of λ as follows. π(λ|x) ∝ f (x|λ)π(λ) λx β α α−1 −βλ λ e ∝ e−λ x! Γ (α) ∝ e−λ−βλ λx+α−1 ∝ e−λ(1+β) λx+α−1 .

The posterior π(λ|x) is a G (x + α, 1 + β). If x = 1, the posterior π(λ|x) is a G (1 + α, 1 + β). a. For α, β going to infinity, the posterior distribution converge to a G (1, 1) distribution. b. We find the posterior distribution, by assuming the noninformative prior: π(λ) = 1/λ. π(λ|x) ∝ f (x|λ)π(λ) λx 1 ∝ e−λ x! λ x−1 −λ λ ∝e x! ∝ e−λ λx−1 .

The posterior π(λ|x) is a G (x, 1). If x = 1, the posterior is a G (1, 1) distribution.

Exercise 5.10 Consider x ∼ B(n, p), H0 : p = 1/2, and H1 : p 6= 1/2. The prior distribution π(p) is a beta distribution Be(α, α). Determine the limiting posterior probability of H0 when n = 10, x = 5 and n = 15, x = 7 as α goes to +∞. Are these values intuitively logical? Give the posterior probabilities for Laplace, Jeffreys, and Haldane noninformative priors.

The posterior π(p|x) can be found as follows π(p|x) ∝ f (x|p)π(p) µ ¶ pα−1 (1 − p)α−1 n x ∝ p (1 − p)n−x B(α, α) x

108

Solution Manual

µ ¶ n x+α−1 ∝ p (1 − p)n−x+α−1 . x

The posterior π(p|x) is a Be(x + α, n − x + α). The posterior of H0 is: ¢n−x+α−1 ¡ 1 ¢x+α−1 ¡ 1 − 21 2 π(p|x) = B(x + α, n − x + α) ¡ 1 ¢x+α−1+n−x+α−1 =

=

2

B(x + α, n − x + α) ¡ 1 ¢2α+n−2 2

B(x + α, n − x + α)

If n = 10 and x = 5 then

.

¡ 1 ¢2α+8 2

π(p|x) =

B(5 + α, 5 + α)

If n = 15 and x = 7 then

¡ 1 ¢2α+13 2

π(p|x) =

B(7 + α, 8 + α)

.

.

Exercise 5.13 Establish the decomposition (5.6) from the original definition (ℓ) (5.5) of B10 . (Hint: Use Bayes formula to develop π1 (θ1 |x(ℓ) ) and π0 (θ1 |x(ℓ) ).)

We have

R

f1 (x(−l) |θ1 )π1 (θ1 |x(l) )dθ1 (l) B10 = RΘ1 f (x |θ )π (θ |x )dθ0 Θ0 0 (−l) 0 1 0 (l)

Using the Bayes theorem we get f1 (x(l) |θ1 )π1 (θ1 ) f (x |θ )π (θ )dθ1 Θ1 1 (l) 1 1 1

π1 (θ1 |x(l) ) = R

and hence R Z f1 (x(−l) |θ1 )f1 (x(l) |θ1 )π1 (θ1 |x(l) )dθ1 f1 (x(−l) |θ1 )π1 (θ1 |x(l) )dθ1 = Θ1 R f (x |θ )π (θ |x )dθ1 Θ1 Θ1 1 (l) 1 1 1 (l) R f1 (x|θ1 )π1 (θ1 |x(l) )dθ1 = R Θ1 f (x |θ )π (θ |x )dθ1 Θ1 1 (l) 1 1 1 (l) We can write the same for πθ (θ0 |x(l) ) and πθ (θ0 |x(l) ) = R

f0 (x(l) |θ0 )π0 (θ0 ) f (x |θ )π (θ )dθ0 Θ0 0 (l) 0 0 0

Bayesian Choice

109

and therefore R Z f0 (x(l) |θ0 )f0 (x(−l) |θ0 )π0 (θ0 )dθ0 f0 (x(l) |θ0 )π0 (θ0 |x(l) )dθ0 = Θ0 R f (x(l) |θ0 )π0 (θ0 )dθ0 Θ0 Θ0 (l)

and finally B10 = B10 (x)B01 (x(l) ).

Exercise 5.14 In the setting of Example 5.13, show how B10 (2) depends on the choice of (x1 , x2 ) by computing the normalizing constants of both π0 (σ 2 |x1 , x2 ) and π1 (µ, σ 2 |x1 , x2 ) and by completing the integration in B10 (2).

Let xi ∼ N (µ, σ 2 ) ½ H0 : µ = 0 with H0 : µ 6= 0 with

π0 (σ 2 ) = 1/σ 2 π1 (µ, σ 2 ) = 1/σ 2

then π1 (µ, σ 2 |x1 , x2 ) =

1 exp{−2(µ − x̄1 )2 /2σ 2 }s51 σ −3 exp{−s21 /2σ 2 } σ

and

s2 s60 exp{− 02 } 4 σ 2σ 2 2 with x̄1 = (x1 + x2 )/2, s1 = (x1 − x2 ) /2 and s20 = x21 + x22 . In this setting 2 B10 is given by R f1 (x(−2) |θ1 )π1 (θ1 |x2 )dθ1 2 R B10 = Θ1 f (x |θ )π (θ |x )dθ0 Θ0 0 (−2) 0 0 0 (2) π0 (σ 2 |x3 , x2 ) =

with θ1 = (µ, σ 2 ), Θ1 = R4 × R+ , θ0 = σ 2 , Θ0 = R+ , µ = θ, x(2) = (x1 , x2 ) and x(−2) = (x3 , . . . , xn ). From previous assumptions, we get ½

½ 2 ¾−1 ¾ s0 + s23 −n−2 2 exp − dσ σ 2σ 2 Z ¢ ª © ¡ ×s51 exp − −(n − 2)(x̄2 − µ)2 − 2(x − x̄1 )2 − s22 − s21 /2σ 2 σ −n−2 dµdσ 2

2 B10 =

s60

Z

ª−1 s51 © Γ (n + 1)/((s23 + s20 )/2)n+1 6 s Z0 n h¡ ¢2 × exp − n (µ − (n − 2)x̄2 /n + 2x̄)2 − (n − 2)(x̄2 + x̄1 )/n io −(n − 2)x̄2 − 2x21 /2σ 2 − s22 − s21

=

s21 ((s23 + s20 )/2)n+1 s6 Γ (n + 1) ¾ ¸ ½ · Z0 σ 2(n − 2) 2(n − 1) 2 2n − 1 × √ exp − x̄2 + x̄1 − x̄1 x̄2 /2σ 2 σ −n−2 dσ 2 n n n n =

110

Solution Manual

s2 (s2 + s20 )n+1 Γ (n) √ = 61 3 s0 Γ (n + 1) n

µ

2(n − 2) 2(n − 1) 2 2n − 1 x̄2 + x̄1 − x̄1 x̄2 n n n

¶−n

which depends on (x̄1 , x̄2 , s1 , s0 ).

Exercise 5.20 For the intrinsic prior considered in Example 5.17, a. Show that

Z ∞

¡

θ0

¢−1 dθ = log(2).

2eθ−θ0 − 1

b. Deduce the expression (5.12). a. Using the change of variables ω = exp(θ − θ0 ) it follows Z ∞ Z ∞ ¢−1 ¡ θ−θ0 1 −1 (2ω − 1) − dω. −1 dθ = 2e ω 1 θ0 Now, decompose the fraction 1/((2ω − 1)ω) in 1 2 1 = − .

(2ω − 1)ω 2ω − 1 ω and find

Z ∞ 1

−1

(2ω − 1)

1 − dω = ω

Z ∞µ 1

2 1 − 2ω − 1 ω

¶ dω +∞

= |ln(2ω − 1) − ln(ω)|1 = ln(2) iid

b. Let x1 , . . . , xn ∼ Ix≥θ then H0 : θ = θ0 versus H1 : θ > θ0 with I(θ) = 1 and µ l B10 = med B01 = B10 (x)med B01 (x(l) ) ³ ´−1 = B10 (x) emed (xi ) − 1

if we consider the prior π3 (θ) ∝ (2eθ−θ0 − 1)−1 , we have B̃10 = B̃10 (x)B̃10 (x(l) ) using the results in Lemma 5.12. Moreover Z +∞ (2eθ−θ0 − 1)−1 dθ = ln(2) B̃10 (x(l) ) = θ0

so that B̃10 = 0.69B̃01 (x).

Bayesian Choice

111

Exercise 5.21 In the setting of Example 5.19, a. Show that Z ÃY n ½ p

−(xt −µ1 )2 /2σ12

e σ1 Z ÃY n p

t=1

≥

t=1

1 − p −(xt −µ2 )2 /2σ22 + e σ2 !b 2

σ1

2

e−(xt −µ1 ) /2σ1

¾!b dπ(µ, σ)

dπ(µ, σ) .

b. Deduce that the fractional Bayes factor does not exist for this model. a. We have Z I=

b

f (x|θ) π(θ)dθ =

à Z Y n i=1

2

−µ1 ) −µ2 ) p − (xi2σ 1 − p − (xi2σ 2 2 1 2 + e e σ1 σ2

2

!b dπ(µ, σ)

since each term in this product is positive and b is positive we get à !b Z Y n −µ1 )2 p − (xi2σ 2 1 I≥ dπ(µ, σ) e σ1 i=1 b. From a. we have Z Z nb n X p b f (x|θ) π(θ)dθ ≥ (xi − µ1 )2 /2σ12 }dπ(θ) ≥ ∞ exp{−b σ1nb i=1 Exercise 5.27 For x ∼ N (θ, 1) and H0 : θ = 0, determine when the p-value crosses the lower bounds P (x, GA ) and P (x, GS ).

The critical region associated with the hypothesis is of the type |x| > k and the p-value is p(x) = inf α such that |x| > kα = pX (|x| > |x|) = Φ(x) − Φ(−x) = ℓ(1 − Φ(x)) with Φ c.d.f. of a standard normal.

The lower bound of the posterior probability of H0 is called the Bayesian answer and is: #−1 " f (x|θ̂(x)) P (x, GA ) = 1 + f (x|θ0 ) · ½ ¾¸−1 1 = 1 + exp − x .

2 On the set of prior distributions which are symmetric about θ0 , the answer is P (x, GS ) such that

112

Solution Manual

µ

1/2(f (x| − ε) + f (x|ε)) f (x|θ) ε∈E −1  supf (x|ε) ε∈E  = P (x, Gn ) = 1 + f (x|0)

P (x, GS ) =

¶−1

1 + sup

because supε f (x|ε) corresponds to the maximum likelihood estimator.

For x < 0.8, note that the p-value is larger than the lower bound, that means that the lower bound of the posterior probability often rejects the null hypothesis.

Exercise 5.29 *(Casella and Berger (1987)) Establish the following lemma, used in Lemma 5.35 and Theorem 5.38: if the family G is constituted of the mixtures Z g(θ) = gξ (θ)h(ξ) dξ, Ξ

for every density h on Ξ, with gξ ∈ G0 and G0 = {gξ ; ξ ∈ Ξ}, then, for any f ,

Z sup

g∈G

Z f (x|θ)g(θ) dθ = sup ξ∈Ξ

f (x|θ)gξ (θ) dθ.

For π ∈ G, we denote with πξ the density of (θ|A), where the density of A is h. Let π ∈ G and B ∈ B(Rd ) then Pπ (B|X = x) = EA (P(θ ∈ B|A = α, X = x)|X = x) = EA (Pπξ (θ ∈ B|X = x)|X = x) ≤ EA (sup Pπξ (θ ∈ B|X = x)|X = x) ξ

= sup Pπξ (θ ∈ B|X = x) ξ

but G0 ⊂ G thus we find the opposite inequality, thus equality holds.

Exercise 5.30 In the case when x ∼ N (θ, 1) and H0 : θ ≤ 0, determine the lower bound P (x, GSU ) = inf P g (θ ≤ 0|x) g∈GSU R0 f (x − θ)g(θ) dθ = inf R −∞ +∞ g∈GSU f (x − θ)g(θ) dθ −∞

Bayesian Choice

113

for x < 0. Does the conclusion of Casella and Berger (1987) still hold? Can you explain why?

In the Gaussian case, p(x) = 1 − φ(x) and π

inf Pr(θ ≤ 0|x) = 0 and

Z 0 −∞

f (x − θ)dθ

. Z +∞ −∞

f (x − θ)dθ = 1/2.

Note that the conclusion in Casella and Berger (1987) holds if the conditions on the functions f and g are satisfied.

Exercise 5.31 (Casella and Berger (1987)) Consider a bounded symmetric unimodal function, g. The family of the scale mixtures of g is defined by Gg = {πσ ; πσ (θ) = (1/σ)g(θ/σ), σ > 0}.

If the sample density is f (x − θ), with f symmetric in 0, if it satisfies the monotone likelihood ratio property, and if x > 0, show that P (x, Gg ) = p(x) for the test of H0 : θ ≤ 0.

Let GUs denote the set of all uniform symmetric distributions, we have inf Pr(H0 |x) = inf Pr(H0 |x) = p(x).

g∈GSU

g∈GUs

The first equality is given in the previous exercise. The second inequality is showed in the following. Let g(θ) be uniform on [−k, k], then R −k Pr(H0 |x) = R0k −k

and d Pr(H0 |x) = dk

Ã

f (x − k) + f (x + k) Rk f (x − θ)dθ −k

!µ

f (x − θ)dθ f (x − θ)dθ

¶ f (x + k) − Pr(H0 |x) . f (x − k) + f (x + k)dθ

Let us assume that there exists a value k0 such that the first order derivative is null, then the sign of second order derivative ¯ d2 ¯ ¯ ¯d f (x + k) ¯ ¯ ¯ ¯ sgn ¯ 2 Pr(H0 |x)¯ , = sgn ¯ ¯ dk dk f (x − k) + f (x + k) k=k0 k=k0

114

Solution Manual

in k = k0 is always negative because the function f is symmetric and satisfies the monotone likelihood property and the function f (x + k)/f (x − k) is decreasing w.r.t. k for all x > 0. The minimum of Pr(H0 |x) is attained at the bounds and ¶ µZ 0 Z 0 f (x − θ)dθ = p(x). f (x − θ)dθ/ inf = lim g∈GUs

k→∞

Now we show that

−∞

−k

inf

gσ ∈Gg

Pr(H0 |x) = p(x).

Note that Gg ⊂ GSU thus due to the previous results inf

gσ ∈Gg

Pr(H0 |x) ≥ p(x)

The equality follows considering reverse inequality, that is inf

gσ ∈Gg

Pr(H0 |x) ≤ lim Pr(H0 |x) σ→∞ gσ

R0

= lim R −∞ +∞ σ→∞

−∞

f (x − θ)g(θ/σ)dθ f (x − θ)g(x/θ)dθ

.

By assumption g is bounded and the Lebesgue’s theorem applies. Moreover g is symmetric and unimodal thus the limit lim g(θ/σ) = g0

σ→∞

exists and is positive. We conclude that R0 f (x − θ)g0 dθ = p(x). lim Pr(H0 |x) = R −∞ +∞ σ→∞ gσ f (x − θ)g0 dθ −∞ Exercise 5.32 (Casella and Berger (1987)) Consider the test of H0 : θ ≤ 0 versus H1 : θ > 0 when x ∼ f (x − θ). Let h and g be densities on (−∞, 0] and (0, +∞). a. Show that, if π(θ) = ̺0 h(θ) + (1 − ̺0 )g(θ), sup P π (θ ≤ 0|x) = h

̺0 f (x) R +∞ ̺0 f (x) + (1 − ̺0 ) 0 f (x − θ)g(θ) dθ

and deduce that the supremum actually favors H0 by putting all the mass at the boundary θ = 0. b. If 1 1 π(θ) = ̺0 h(θ/σ1 ) + (1 − ̺0 )g(θ/σ2 ) , σ1 σ2

Bayesian Choice

115

show that, when σ1 is fixed lim P π (θ ≤ 0|x) = 1

σ2 →∞

and that, when σ2 is fixed, lim P π (θ ≤ 0|x) = 0.

σ1 →∞

a. We have R0 ̺0 −∞ f (x − θ)h(θ)dθ sup P (θ ≤ 0|x) = R0 R +∞ h ̺0 −∞ f (x − θ)h(θ)dθ + (1 − ̺0 ) 0 f (x − θ)g(θ)dθ π

=

̺0 f (x) .

R +∞ ̺0 f (x) + (1 − ̺0 ) 0 f (x − θ)g(θ)dθ

The last expression is equal to Pr(H0 |x), for H0 : θ = 0 against the alternative: H1 : θ > 0 and with the prior ̺0 δθ=0 + (1 − ̺0 )Iθ>0 . b. By virtue of the dominated convergence theorem π

lim Pr(θ ≤ 0|x) =

σ2 →∞

R0 ̺0 −∞ h(θ/σ1 ) σ11 f (x − θ)dθ = lim R0 R +∞ 1 σ2 →∞ ̺ g(θ/σ2 ) σ12 f (x − θ)dθ 0 −∞ h(θ/σ1 ) σ1 f (x − θ)dθ + (1 − ̺0 ) 0 R0 ̺0 −∞ h(θ/σ1 ) σ11 f (x − θ)dθ = R0 R +∞ ̺0 −∞ h(θ/σ1 ) σ11 f (x − θ)dθ + lim (1 − ̺0 ) 0 g(θ/σ2 ) σ12 f (x − θ)dθ σ2 →∞

=1

116

Solution Manual

Chapter 6 - Bayesian Calculations Exercise 6.5 (Smith and Makov (1978)) Consider x ∼ f (x|p) =

k X

pi fi (x),

i=1

P where pi > 0, i pi = 1, and the densities fi are known. The prior π(p) is a Dirichlet distribution D(α1 , . . . , αk ). a. Show that the computing time becomes prohibitive as the sample size increases.

A sequential alternative which approximates the Bayes estimator is to replace (n) (n) π(p|x1 , . . . , xn ) by D(α1 , . . . , αk ), with (n)

α1

(n−1)

= α1

(n)

+ P (zn1 = 1|xn ), . . . , αk

(n−1)

= αk

+ P (znk = 1|xn ),

and zni (1 ≤ i ≤ k) is the component indicator vector of xn as defined in Section 6.4. b. Justify this approximation and compare it with the updating of π(θ|x1 , . . . , xn−1 ) when xn is observed. c. Examine the performances of this approximation for a mixture of two normal distributions N (0, 1) and N (2, 1) when p = 0.1, 0.25, 0.5. d. If πin = P (zni = 1|xn ), show that (n)

(n−1)

p̂i (xn ) = p̂i (n)

where p̂i

(n−1)

(xn−1 ) − an−1 {p̂i

− πin },

is the quasi-Bayesian approximation of Eπ (pi |x1 , . . . , xn ).

a. The coherence condition is that for every integer k, the prior πk is invariant under index permutations. This means that for every permutation on {1, ..., k}, σ ∈ Sk , πk (p1 , ..., pk ) = πk (pσ(1) , ..., pσ(k) ). b. When πk is a Dirichlet distribution Dk (α1 , ..., αk ), that is, πk (p1 , ..., pk ) =

Γ (p1 + ... + pk ) α1 −1 αk −1 p ...pk , Γ (p1 )...Γ (pk ) 1

the coherence condition becomes α1 = ... = αk .

Q −1/k is invariant under permutation of the indices d. The prior πk (pk ) = i pi of the pi ’s. Therefore, this distribution satisfies the coherence property.

Bayesian Choice

117

Exercise 6.9 The regular accept–reject simulation algorithm is defined as follows: If f and g are densities such that there exists M with f (x) ≤ M g(x), 1. Generate y ∼ g(y) and u ∼ U[0,1] ; 2. If u > f (y)/M g(y), go back to 1.

3. Take x = y.

The c.d.f. of x is

µ

P (X ≤ y) = P

f (y) M g(Y ) ´

Y ≤ x, U ≤

¶

³ ) P y ≤ x, U ≤ Mf (Y g(y) ³ ´ = (Y ) P U ≤ Mf g(Y ) =

Z x Z f (y)M g(y) −∞

1 = M =

1 M

0

Z x

−∞ Z x −∞

dug(y)dy

µ f (y)dy µ f (y)dy

ÃZ

1 M 1 M

Z ∞ −∞ ¶−1

∞ Z f (y)/M g(y)

−∞

0

¶−1

f (y)dy

=

Z x −∞

f dy.

!−1 du g(y)dy

118

Solution Manual

Chapter 8 - Admissibility and Complete Classes Exercise 8.1 (Lehmann (1986)) Consider a random variable x with mean µ and variance σ 2 . a. Show that δ(x) = ax + b is an inadmissible estimator of µ under quadratic loss if (a) a > 1; (b) a < 0; and (c) a = 1 and b 6= 0. b. Generalize to the case where δ(x) = (1 + h(x))x with h(x) > 0. a. The classical risk for this estimator is R(µ, δa,b ) = E[(ax + bx − µ)2 ] = E((ax + b)2 − 2µ(ax + b) + µ2 ) = µ2 − ℓµb + b2 + 2aµ(b − µ) + a2 (σ 2 + µ2 ) = (a − 1)2 µ2 + 2b(a − 1)µ + b2 + a2 σ 2 = ((a−1 )µ + b)2 + a2 σ 2 and the risk of the estimator δ(x) = x is R(µ, δa,b = E[(x − µ)2 ] = E[x2 − µx + µ2 ] = σ 2 + µ2 − 2µ2 + µ2 = σ2 (a) For a > 1, R(µ, δa,b ) > R(µ, x) δa,b is not admissible. (b) For a < 0, ((a−1)µ+b)2 +a2 σ 2 ≥ a2 σ 2 ≥ σ 2 thus R(µ, δa,b ) > R(µ, x). δa,b is not admissible. (c) For a = 1, R(µ, δa,b ) = b2 + σ 2 ) thus if b 6= 0, δa,b is not admissible. b. The classical risk is R(µ, δ(x)) = E(((1 + h(x))x − µ)2 ) = E((1 + h(x))2 x2 − 2µx(1 + h(x)) + µ2 ) = E(x2 + 2h(x)x2 + h2 (x)x2 − µx − 2µxh(c) + µ2 ) = µ2 + E(x2 ) + ℓE(h(x)x2 ) + E(H 2 (x)x2 ) − 2µE(x) − ℓµE(xh(x)) = µ2 + σ 2 + µ2 − 2µ2 + 2E(h(x)x2 ) + E(h2 (x)x2 ) − 2µE(xh(x)) σ 2 + 2E(h(x)x2 ) + E(h2 (x)x2 ) − 2µE(xh(x)).

Bayesian Choice

119

Exercise 8.4 Consider x ∼ B(n, p) and determine whether δ0 ≡ 0 is an admissible estimator of p under quadratic loss.

The Bayes estimator δ π of p, associated to the prior π and to the quadratic loss, is the posterior mean: δ π (x) = Eπ [p|x].

Let x ∼ B(n, p) and consider a noninformative prior π(p) = p1 . The posterior follows from π(p|x) ∝ f (x|p)π(p) µ ¶ 1 n X ∝ p (1 − p)n−x p x ∝ px−1 (1 − p)n−x .

The posterior is Be(x, n − x + 1). Thus the Bayes estimator δ π of p is δ π (x) = Eπ [p|x] =

x x = . x+n−x+1 n+1

The estimator δ0 = 0 is not admissible under the quadratic loss because there x , which dominates it. exists another estimator, δ π (x) = n+1 Exercise 8.5 (Johnson (1971)) Consider x ∼ B(n, θ). a. Show that δ0 (x) = x is the maximum likelihood estimator of θ and also a Bayes estimator under quadratic loss for π(θ) = 1/θ(1 − θ). b. Show that (δ0 , 1 − δ0 ) is admissible under the loss L(θ, δ) = (θ − δ1 )2 + (1 − θ − δ2 )2 .

8.15

(Hint: Use the Bayes representation of δ0 to show that Z dθ [R(θ, δ) − R(θ, (δ0 , 1 − δ0 ))] ≥0 θ(1 − θ) and to deduce that equality only occurs for δ1 = δ0 , δ2 = 1 − δ0 .) c. Show that a complete class for the loss (8.15) is made of the estimators such that δ1 = 1 − δ2 . d. Generalize the result of b. to the multinomial setting x ∼ Mk (n, p1 , . . . , pk ). (Hint: Use induction.) a. Let x ∼ B(n, θ), then µ ¶ n x f (x|θ) = θ (1 − θ)n−x x µ ¶ n ⇒ log f (x|θ) = log( ) + x log(θ) + (n − x) log(1 − θ) x

120

Solution Manual

x (n − x) ∂ log f (x|θ) = − ∂θ θ 1−θ and ∂ log f (x|θ) =0 ∂θ x (n − x) ⇔ = θ 1−θ (n − x) 1−θ = ⇔ θ x n 1 ⇔ −1= −1 θ x x ⇔θ= . n Thus the ML estimator of θ is θ̂ = nx . The Bayes estimator δ π of θ, associated to the prior π and to the quadratic loss is the posterior mean: δ π (x) = Eπ [θ|x]. By assumption x ∼ B(n, θ) and θ ∼ π(θ) = 1/θ(1 − θ).

Thus the posterior follows from π(θ|x) ∝ f (x|θ)π(θ) µ ¶ 1 n x ∝ θ (1 − θ)n−x θ(1 − θ) x ∝ θx−1 (1 − θ)n−x−1 .

The posterior is Be(x, n − x) and the Bayes estimator δ π of θ is δ π (x) = x Eπ [θ|x] = x+n−x = nx . Thus δ0 (x) = x is the ML estimator of θ and the Bayes estimator under the quadratic loss and for π(θ) = 1/θ(1 − θ). b. The classical risk under the loss function L(θ, δ) = (θ − δ1 )2 + (1 − θ − δ2 )2 is £ ¤ R(θ, δ) = Eθ [L(θ, δ(X))] = Eθ (θ − δ1 )2 + (1 − θ − δ2 )2 and R(θ, (δ0 , 1 − δ0 )) = Eθ [L(θ, (δ0 , 1 − δ0 ))] £ ¤ = Eθ (θ − δ0 )2 + (1 − θ − (1 − δ0 ))2 Z ⇒

[R(θ, δ) − R(θ, (δ0 , 1 − δ0 ))]

Z [R(θ, δ) − R(θ, (δ0 , 1 − δ0 ))]

dθ ≥0 θ(1 − θ)

dθ =0 θ(1 − θ)

Bayesian Choice

121

⇔ R(θ, δ) = R(θ, (δ0 , 1 − δ)) ⇔ δ1 = δ0 and δ2 = 1 − δ0 .

Thus (δ0 , δ0 −1) is admissible under the cost L(θ, δ) = (θ, δ)2 +(1−θ−δ2 )2 . c. We already showed that (δ0 , 1 − δ0 ) is admissible under the loss function L(θ, δ) = (θ − δ1 )2 + (1 − θ − δ2 )2 . If we let δ1 = δ0 and δ2 = 1 − δ0 , then δ2 = 1 − δ1 thus δ1 = 1 − δ2 . We conclude that a complete class for the loss function in is given by the estimators such that δ1 = 1 − δ2 .

Exercise 8.9 Consider the bounded loss L(θ, δ) = 1 − e−a(θ−δ)

2

(a > 0),

for the estimation of θ when x ∼ N (θ, 1). a. Determine the Bayes estimators associated with the conjugate priors θ ∼ N (µ, τ 2 ). b. Determine the Bayes estimators associated with the prior distributions π(θ) ∝ exp(−λ|θ − µ|). c. Examine the admissibility of the generalized Bayes estimator associated with the Jeffreys prior π(θ) = 1 when a varies. (Hint: Determine whether the Bayes risk is finite and apply Blyth’s method if necessary.) a. The Bayes estimator, δ π , associated to the prior π and to a loss function L, minimizes r(π, δ). For x ∈ X, the Bayes estimator is δ(x) = arg min ̺(π, d|x)dθ d

= arg min ̺(π, δ|x) δ

= arg min Eπ [L(θ, δ)|x] δ Z = arg min L(θ, δ)π(θ|x)dθ. δ

Θ

Let us find the posterior π(θ|x) ∝ f (x|θ)π(θ) ¶ µ ¶ µ −θ − µ)2 −(x − θ)2 exp ∝ exp 2 2r2 µ 2 ¶ 2 2 x θµ µ2 θ θ ∝ − + θx − − 2+ 2 − 2 2 2τ τ 2τ 2 ¶ ¶ µ 2 µ ³ ´ µ τ +1 2 +θ x+ 2 ∝ exp −θ 2τ 2 τ Ã ¶2 ! µ 2 2 τ +1 xτ + µ ∝ exp θ− 2 .

2τ 2 τ +1

122

Solution Manual

The posterior is N (x − λ, 1). The Bayes estimator is Z δB = arg min L(θ, δ)π(θ|x)dθ δ Θ ¶ µ Z 1 = arg min (1 − exp(−a(δ − θ)2 )) exp − (θ − (x − λ))2 dθ. δ 2 Θ c. The posterior is π(θ − x) ∝ f (x|θ)π(θ) ¶ µ −(x − θ)2 ∝ exp 2 µ 2 ¶ −x + 2θx − θ2 ∝ exp 2 ¶ µ θ2 + θx ∝ exp − 2 ¶ µ 1 ∝ exp − (θ − x)2 .

2 The posterior is N (x, 1). The Bayes estimator is Z L(θ, δ)π(θ|x)dθ δB = arg min δ Θ µ ¶ Z 1 2 2 = arg min (1 − exp(−a(δ − θ) )) exp − (θ − x) dθ. δ 2 Θ The risk is bounded because

£ ¤ R(θ, δ)2 = E [L(θ, δ π (x))] = E 1 − exp(−a(δ − θ)2 ) .

© ª Exercise 8.14 Consider x ∼ N (θ, 1) and π(θ) ∝ exp −bθ2 /2 + abθ . a. Give necessary and sufficient conditions on (a, b) for the posterior distribution to be defined. Show that, in this case, the posterior distribution is normal with mean (x + ab)/(1 + b) and variance 1/(b + 1). b. Show that the transition kernel (8.7) is then given by ¶ µ θ + ab b + 2 . η|θ ∼ N , 1 + b (1 + b)2 c. Deduce that the Markov chain is an AR(1) model (Section 4.5.2) √ b+2 1 (t) ab (t+1) θ = θ + + ǫt .

1+b 1+b 1+b Conclude that it is transient when b < 0 and recurrent when b = 0.

a. We know that f (x|θ) ∝ exp the posterior follows from

³

−(x−θ) 2

´ 2

Bayesian Choice

and π(θ) ∝ exp

³

−bθ 2 2 + abθ

´

123

, thus

π(θ|x) ∝ f (x|θ)π(θ) ¶ µ −(x − θ)2 − bθ2 + abθ ∝ exp 2 ¶ µ 2 −x + 2xθ − θ2 − bθ2 + abθ ∝ exp 2 ¶ µ 2 −θ (1 + b) + θ(x + ab) ∝ exp 2 Ã µ ¶2 ! −(1 + b) x + ab ∝ exp θ− .

2 1+b The posterior is a normal distribution with mean (x + ab)/(1 + b) and variance 1/(b + 1). The necessary and sufficient condition on (a, b) for the posterior to be defined is that the variance is positive. 1/(b + 1) > 0 ⇔ (b + 1) > 0 ⇔ b³ > −1. The ´ posterior is well defined and is the normal distribution: N

x+ab 1 1+b , 1+b

if b > −1.

R b. The transition kernel is: K(θ|η) = X π(θ − x)f (x|η)dx.

Exercise 8.39 ∗ (Hobert and Robert (1999)) Consider x ∼ P(θ) with π(θ) ∝ θa−1 exp{−bθ}. a. Give necessary and sufficient conditions on (a, b) for the posterior distribution to be defined and the prior distribution to be improper. b. Give the transition kernel (8.7) in this case. In the special case b = −1/2 and a = k/2, show that the transition is a noncentral chi-squared distribution. c. Show that the transition kernel (8.16) is K ∗ (x, y) =

Γ (y + x + a) x+a p (1 − p)y , y!Γ (x + a)

where p = (b + 1)/(b + 2). d. Show that this distribution corresponds to the usual negative binomial distribution when a is a positive integer. e. In the general case, the distribution with mass function P (Z = z) =

Γ (z + c) c p (1 − p)z z!Γ (c)

is called a generalized negative binomial distribution N eg(c, p). Deduce from the probability generating function that, if z1 , . . . , zn are independent with zi ∼ N B(ci , p), z1 + . . . + zn ∼ N eg(c1 + . . . + cn , p).

124

Solution Manual

f. Deduce that (xt ) associated with the kernel K ∗ is a branching process, xt+1 =

xt X

ηi,t + ωt+1

i=1

where ηi,t ∼ N eg(1, p) and ωt+1 ∼ N eg(a, p). g. Conclude that the chain (xt ) is recurrent when b = 0 and 0 < a < 1, and transient otherwise. x

a. We know that f (x|θ) = e−θ θx! and π(θ) ∝ θn−1 exp(−bθ), thus the a posterior distribution follows from π(θ|x) ∝ f (x|θ)π(θ) θx ∝ e−θ θa−1 exp(−bθ) x! ∝ exp(−θ(1 + b))θx+a−1 .

The posterior is a G (x + a, 1 + b) distribution. Necessary and sufficient conditions on (a,b) for the posterior to be defined is that the parameters are positive x + a > 0 ⇔ a > −x and 1 + b > 0 ⇔ b > −1 thus the posterior is well defined and is a G (x + a, 1 + b) distribution if a > −x and b > −1. b. If b = −1/2 et a = k/2, the posterior is G (x + k/2, 1/2).

R The transition kernel is K(θ|η) = X π(θ|x)f (xη)dx.

(y+x+a) x+a (1 − p)y , with p = (b + 1)/(b + 2). c. K ⋆ (x, y) = Γz!Γ (x+a) p Px1 f. xt+1 = i=1 ηi,t + ωt+1 with ηi,t N eg(1, p) and ωt+1 ∼ N eg(a, p).

Exercise 8.40 ∗ (Hobert and Robert (1999)) Consider x ∼ N eg(k, θ) with π(θ) ∝ θa−1 (1 − θ)b−1 . a. Give necessary and sufficient conditions on (a, b) for the posterior distribution to be defined and the prior distribution to be improper. b. Give the transition kernel (8.7) when b = k. c. Show that the corresponding Markov chain (θ(t) ) can be written as θ(t+1) = θ(t) /(ωt + θ(t) ), where the ωt ’s are i.i.d. d. Show that E[log ωt ] = 0 if, and only if, a = 0 and deduce that the chain (θ(t) ) is recurrent when a = 0 and transient otherwise; a. Note that f (x|θ) = is

¡k−x+1¢ k θ (1−θ)x et π(θ)αθa−1 (1−θ)b−1 , the posterior x

π(θ|x) ∝ f (x|θ)π(θ)

µ ∝

¶

Bayesian Choice

125

k−x+1 k θ (1 − θ)x θa−1 (1 − θ)b−1 x

∝ θk+a−1 (1 − θ)x+b−1 that is a Be(k + a, x + b) distribution, with positive parameters: k + a > 0 ⇔ a > −k et x + b > 0 ⇔ b > −x. b. If b = k, Rthe posterior is Be(k + a, x + k). and the transition kernel is K(θ|η) = X π(θ|x)f (x|η)dx.

126

Solution Manual

Chapter 9 - Invariance, Haar Measures, and Equivariant Estimators Exercise 9.2 Consider x an observation from a C (θ, 1) distribution. Under quadratic loss, show that all equivariant estimators have infinite risk. Propose a finite risk estimator other than the constant estimator.

The invariant estimators for θ have the following form δc (x) = x + c. The risk associated to the estimator is: ¤ £ R(θ; δc ) = Ex (θ − (x + c))2 = θ2 − ℓθ(E(x) + c) + E(x2 ) + 2CE(θ) + C 2 =∞ because the second order moment of a Cauchy does not exist.

Exercise 9.7 (Lehmann (1983)) Consider the estimation of a scale parameter σ, under the loss µ ¶2 δ L(σ, δ) = −1 , 9.5 σ for n observations x1 , . . . , xn ∼

xn ´ 1 ³ x1 . f , . . . , σn σ σ

a. If z = (x1 /xn , . . . , xn−1 /xn , xn /|xn |), show that every estimator of σ equivariant under scale transformations can be written δ(x) = δ0 (x)/ω(z), where δ0 is a particular equivariant estimator, and that z is a maximal invariant statistic. b. Determine the function ω ∗ that minimizes E[L(σ, δ(x))|z] under (9.5), and deduce the best equivariant estimator. c. Write this estimator in an appropriate form in order to find the result of Section 9.4 with the corresponding Haar measure. d. Consider the previous questions for the estimation of σ r (r ∈ R∗+ ) under the loss µ ¶2 δ L(σ, δ) = −1 . σr a. If w(Z) is a function such that ω(az) = ω(Z) for all z and a > 0 then:

Bayesian Choice

127

δ(ax) = δ0 (ax)/ω(ax) = aδ0 (x)/ω(x) = aδ(x).

If δ is invariant, let ω(x) = δ(x)/δ = (x) w(ax) = δ(ax)/δ0 (ax) = aδ(x)/δ0 (x) = ω(x).

Thus all equivariant estimators of σ can be written as δ(x) = δ0 (x)w(z). b. The risk of the estimator does not depend on σ, thus we find the minimum of R0 (δ(x)) "µ ¶2 # δ0 (x) R0 (δ(x)) = E0 /z ω(z) "µ ¶2 # δ0 (x) /z . = E0 ω(z)

128

Solution Manual

Chapter 10 - Hierarchical and Empirical Bayes Extensions Exercise 10.4 Represent the hierarchical model of Example 10.4 as a directed acyclic graph, as in Figure 10.1.

Let xi ∼ B(βi , 10)(i = 1, ..., 7) be independent yearly measures of IQ for an individual observed for 7 consecutive years. The IQ test accounts for the age thus βi have the same mean θ, corresponding to the true value of IQ. Thus βi |θ ∼ N (θ, σπ2 )(i = 1, ..., 7). The individual belongs to a population with these features: θ ∼ N (123, 98). The DAG is provided in Fig. C.6

#Ã θ

σπ2

"! ? #Ã = β

"! ? xi

Fig. C.6. Directed Acyclic Graph.

Exercise 10.6 A fly lays N eggs according to a Poisson distribution P(λ) and each egg survives with probability p. a. Show that the distribution of the number of surviving eggs x is then hierarchical x|N ∼ B(N, p), N ∼ P(λ). b. Compute the marginal distribution of x and the posterior distribution of N.

Bayesian Choice

129

a. A hierarchical Bayesian model is a statistical model, (f (x|θ), π(θ)), with the prior π(θ) can be decomposed in the conditional distributions πt (θ|θ1 ), π2 (θ1 |θ2 ), ..., πn (θn−1 |θn ). Our model is (f (x|N ), π(N )), with a prior π(N ) depending on λ. The distribution of x is a hierarchical model. ¡ ¢ x N N −s et π(N ) = e−λ λN ! . The marginal is b. f (x|N ) = N x p (1 − p) Z m(x) = f (x|N )π(N )dN Z µ ¶ λN N c p (1 − p)N −x e−λ ! dN = N x x −λ Z N −x N p e (1 − p) λ = dN. x!

(N − x)!

The posterior of N is f (x|N )π(N ) m(x) ¾ ½ x −λ Z ¾−1 ½µ ¶ N (1 − p)N −s λN p e N x N −x −λ λ dN = p (1 − p) e N! x!

(N − x)! x ¾ ½Z ½ ¾−1 (1 − p)N −x λN (1 − p)N −x = dN .

(N − x)!

(N − x)!

π(N |x) =

Exercise 10.7 In the setting of Example 10.6, when p is known, give the posterior distribution of N if π2 (λ) = 1/λ. Examine the generalization to the case when p is unknown and π1 (p) = 1.

The posterior of N if π2 (λ) = 1/λ, is π(N |x) =

R Θ1

π(N |λ, x)π(λ|x)dλ where

f (x|N )π1 (N |λ) m1 (x|λ) Z m(x|λ) = f (x|N )π1 (N |λ)dN

π(N |λ, x) =

Θ

m1 (x|λ)π2 (λ) π(λ|x) = m(x) Z m1 (x|λ)π2 (λ)dλ. m(x) = Θ1

It follows

Z

f (x|N )π1 (N |λ) m1 (x|λ)π2 (λ) dλ m1 (x|λ) m(x) Θ1 Z f (x|N )π1 (N |λ)π2 (λ) dλ = m(x) Θ1

π(N |x) =

130

Solution Manual

Z = and considering that

R

R

Θ1

Θ1

f (x|N )π1 (N |λ)π2 (λ) dλ (x|N )π1 (N |λ)dN π2 (λ)dλ Θ1

µ ¶ N x f (x|N ) = p (1 − p)N −x x π1 (N |λ) = e−λ π2 (λ) =

it follows

1 λ

λN N!

¡N ¢ x N −x −λ λN 1 e N! λ x p (1 − p) π(N |x) = dλ R R ¡N ¢ N x N −x e−λ λN ! dN λ1 dλ Θ1 Θ x p (1 − p) 1 Θ ¡N ¢ x Z N −x −λ λN 1 e N! λ x p (1 − p) dλ = R R −λ (1−p)N −x λN px Θ1 dN e λ dλ x! Θ1 Θ (N −x)!

Z

Z =

(1−p)N −x −λ N −1 λ (N −x)! e dλ.

R R (1−p)N −x λN Θ1 Γ (0) dN dλ (N −x)! Θ1 Θ

When p is unknown and π1 (p) = 1, the posterior of N is ¡N ¢ x Z N −x −λ λN 1 e N! λ x p (1 − p) dλ π(N |x) = R R R ¡N ¢ N x N −x e−λ λN ! dpdN λ1 dλ Θ1 Θ x p (1 − p) 2 Θ1 Θ Z 1 x N −x −λ N −1 e λ x!(N −x)! p (1 − p) = dλ R R R N −λ 1 px (1 − p)N −x dp (Nλ−x)! DN e λ dλ Θ1 x! Θ 2 Θ1 Θ Z 1 x N −x −λ N −1 e λ (N −x)! p (1 − p) = dλ.

R R R x (1 − p)N −x dp λN dN e−µ dλ p Θ1 Θ (N −x)! λ Θ Θ 1 2 Exercise 10.11 Establish Lemmas 10.10 and 10.12.

Let f (x|θ)π1 (θ|θ1 ) , m1 (x|θ1 ) Z m1 (x|θ1 ) = f (x|θ)π1 (θ|θ1 )dθ

π(θ|θ1 , x) =

Θ

m1 (x|θ1 )π2 (θ1 ) π(θ1 |x) = m(x) Z m(x) = m1 (x|θ1 )π2 (θ1 )dθ1 . Θ1

Bayesian Choice

131

If x ∼ f (x|θ), θ ∼ π1 (θ|θ1 ), θ1 ∼ π2 (θ R 1 ), we find the usual Bayesian model x ∼ f (x|θ), θ ∼ π(θ), with π(θ) = Θ1 π1 (θ|θ1 )π2 (θ1 )dθ1 . By definition of posterior f (x|θ)π(θ) f (x|θ)π(θ)dθ R f (x|θ) Θ1 π1 (θ|θ1 )π2 (θ1 )dθ1 R . = R f (x|θ) Θ1 π1 (θ|θ1 )π2 (θ1 )dθ1 dθ Θ

π(θ|x) = R

R Following the notation in the lemma, we find: π(θ|x) = Θ1 π(θ|θ1 , x)π(θ1 |x)dθ1 .

R Due to Lemma 10.12, if the marginal m(x) = Θ f (x|θ)π(θ)dθ is finite for all x ∈ Rk , the posterior π(θ|x) has finite mean and variance.

The posterior f (x|θ)π(θ) π(θ|x) = m(x) depends on the marginal. Thus if the marginal is finite for all x ∈ Rk , the the posterior is well definite and has finite mean and variance.

Exercise 10.13 Spiegelhalter et al. (1990) consider the following model: the weights yij of 60 rats are measured weekly, the first 30 observations being associated with the control group (1 ≤ i ≤ 60, 1 ≤ j ≤ 5). The corresponding model is yij ∼ N (αi + βi j, σi ) , with σi = σc for i ≤ 30 and σi = σt for 31 ≤ i ≤ 60 and (αi , βi ) ∼ N2 ((αc , βc ), Σc )

(i = 1, . . . , 30) ,

(αi , βi ) ∼ N2 ((αt , βt ), Σt )

(i = 31, . . . , 60) .

Complete the model with noninformative priors on the hyperparameters and examine whether the posterior distribution is well defined.

For 1 ≤ i ≤ 30, let us consider the following model: yij |αi , βi ∼ N (αi + βi j, σC ) π1 (αi , βi |αc , βc ) ∼ N2 ((αc , βc ), Σc ) π2 (αc , βx ) = 1/(αc βc ) The posterior of (αi , βi ) is Z π(αi , βi |αc , βc , x)π(αc , βc |x)d(αc , βc ). π(αi , βi |x) = Θi

thus

132

Solution Manual

f (x|αi , βi )πi (αi , βi |αc , βc ) m(x|αc , βc ) Z m1 (x|αc , βc ) = f (c|αi , βi )πi (αi , βi |αc , βc )d(αi , βi )

π(αi , βi |αc , βc , x) =

Θ

mi (x|αc , βc )π2 (αc , βc ) π(αc , βc |x) = m(x) Z m(x) = mi (x|αc , βc )π2 (αc , βc )d(αc , βc ). Θi

The posterior density is well defined if m(x) is finite. We have ¶ µ −(x − (αi + βi j))2 f (x|αi , βi ) = (2π)−1/2 σx−1 exp 2σc2 © π1 (αi , βi |αc , βi ) ∝ (det Σc )−1/2 (2π)−1 exp −((αi , βi ) − (αc , βc ))t ª Σc−1 ((αo , βi ) − αc , βc )/2 π2 (αi , βi ) = 1/(αc βc ).

It follows that µ ¶ Z Z −(x − (αi + βi j))2 −1/2 −1 m(x) = (2π) σc exp (det Σc )−1/2 (2π)−1 · 0σc2 Θ1 Θ 1 Σc−1 ((αi , βi ) − (αc − βc )))d(αi , βi ) d(αc , βc ) 2 (αc , βc ) µ Z Z −(x − (αi + βi j))2 −3/2 −1 −1/2 (2π) σc (det Σc ) exp m(x) = ((αi , βi ) − (αc , βc ))′ 2σc2 Θ1 Θ ¶ Σc−1 1 ((αi , βi ) − (αc , βc )) d(αi , βi ) d(αc , βc ) 2 (αc , βc ) · exp(−((αi , βi ) − (αc , βc ))2

For 31 ≤ i ≤ 60 the model is: yij |αi , βi ∼ N (αi + βi j, σt ) π1 (αi , βi |αt , βt ) ∼ N2 ((αt , βt ), Σt ) π2 (αt , βt ) = 1/(αt , βt ).

The posterior of (αi , βi ) is Z π(αi , βi |x) =

Θ1

π(αi , βi |x)π(αt , βt |x)d(αt , βt )

where π(αi , βi |αt , βt , x) =

f (x|αi , βi )π1 (αi , βi |at , βt ) m(x|αt , βt )

Bayesian Choice

Z m1 (x|αt , βt ) =

Θ

133

f (x|αi , βi |αt , βt )d(αi , βi )

m1 (x|αt , βt )π2 (αt , βt ) m(x) Z m(x) = m1 (x|αt , βt )π2 (αt , βt )d(αt , βt )

π(αt , βt |x) =

Θ1

The posterior is well defined if m(x) is finite. We have ¶ µ −(x − (αi + βi ))2 −1/2 −1 f (x|αi , βi ) = (2π) σt exp 2σt2 π1 (αi , βi |αt , βt ) = (det Σt )−1/2 (2π)−1 exp (−((αi , βi ) − (αt , βt ))′ Σt−1 ((αi , nβi ) − (αt , βt ))/2) π2 (αi , βi ) = 1/(αt βt ).

Thus m(x) =

Z Θ1 ×Θ

(2π)−1/2 σt−1 exp

µ

−(x − (αi + βi j))2 2σt2

¶

(det Σt )−1/2 (2π)−1

µ ¶ Σ −1 1 exp −((αi , βi ) − (αc , βc ))′ c ((αi , βi ) − (αc , βc )) d(αi , βi ) d(αc , βc ) 2 (αc βc ) µ Z −(x − (αi + βi j))2 (2π)−3/2 σc−1 (det Σc )−1/2 exp m(x) = 2σc2 Θ1 ×Θ ´ Σ −1 1 ((αi , βi ) − (αc , βc ))′ c ((αi , βi ) − (αc , βc )) d(αi , βi ) d(αc , βv ).

2 (αc βc ) Exercise 10.29 (Casella (1985a)) In a survey of car-buying intentions, 447 households provide their evaluation of the probability of buying a new car in the coming year. The result of the survey is given in Table 10.28.

The answers xi (1 ≤ i ≤ 447) are modeled as issued from a renormalized binomial B(10, pi ) distribution, that is, 10xi ∼ B(10, pi ), and the pi are distributed according to Be(α, β). a. Use the marginal distribution to provide estimators of α and β by the method of moments. b. Derive an empirical Bayes estimator of the pi ’s under quadratic loss. c. The true intentions pi have actually been observed at the end of the year, and Table 10.29 gives the difference with the declared intentions. Compare the quadratic losses of the classical estimator (that is, p̂i = xi ), the empirical Bayes estimator, and a Bayes estimator of your choice.

R a. The marginal is m(xi |α, β) = Θ (xi |pi )π(pi |α, β)dpi with f (xi |pi ) = ¡10¢ xi P α−1 (1−p )β−1 10−xi . It follows that and π(pi|α, β) = i B(α,β)i xi pi (1 − pi )

134

Solution Manual

m(xi |α, β)

= =

R ¡10¢ xi α−1 (1−pi )β−1 p (1 − pi )10−xi p B(α,β) dpi Θ xi i

¡10¢

1 xi B(α,β)

R Θ

pxi i +α−1 (1 − pi )10−xi +β−1 dpi .

b. Under the quadratic loss the Bayesian estimator δ π of pi associated to the prior π, is δ π (x) = Eπ [pi |xi ]. Eπ [pi |xi ] can be estimated from data in Tab. 10.2.

1 (0 × 293 + 0.1 × 26 + 0.2 × 21 + 0.3 × 21 + 0.4 × 10 447 +0.5 × 9 + 0.6 × 12 + 0.7 × 13 + 0.8 × 11 + 0.9 × 10 + 1.0 × 21) 76.7 = = 0.17.

447

Eπ (pi |xi ) =

The empirical counterpart of the Bayes estimator of pi is 0.17.

http://www.springer.com/978-0-387-95231-4