SOLUTIONS MANUAL for A First Course in Linear Model Theory, 2nd Edition by Ravishanker, Chi, Dey

Solutions Manual for A First Course in Linear Model Theory, 2e by Nalini Ravishanker, Zhiyi Chi, Dipak Dey (All Chapters) Solutions to Chapter 1

1.1 |a • b| = | − 9| = 9, while kak kbk =

√ √ 6 22 ∼ = 11.489 > 9.

1.2 To verify the Cauchy–Schwarz inequality, first see that the inequality holds trivially if a and b are zero vectors. We therefore assume that both a and b are nonzero. Let c be the vector c = xa − yb, where x = b0 b, and y = a0 b. Clearly, c0 c ≥ 0. We express c0 c in terms of x and y: c0 c = (xa − yb)0 (xa − yb) = x2 a0 a − 2xya0 b + y 2 b0 b.

Since c0 c ≥ 0, and using the definitions of x and y, we see that (b0 b)2 (a0 a) − 2(a0 b)2 (b0 b) + (a0 b)2 (b0 b) ≥ 0 and dividing by b0 b in the last inequality, we see that (b0 b)(a0 a) − (a0 b)2 ≥ 0, which verifies the Cauchy–Schwarz inequality.

We use the Cauchy–Schwarz inequality to deduce the triangle inequality, which can be written in an equivalent form ka + bk2 ≤ (kak + kbk)2 .

The expression on the left is ka + bk2 = (a + b) • (a + b) = a • a + 2a • b + b • b = kak2 + 2a • b + kbk2 while the expression on the right is (kak + kbk)2 = kak2 + 2kakkbk + kbk2 .

Comparing these two formulas, we see that the triangle inequality holds if and only if a • b ≤ kak kbk. By Cauchy–Schwarz inequality, |a • b| ≤ kakkbk, so the triangle inequality follows as a consequence of the Cauchy–Schwarz inequality. The converse is also true; i.e., if the triangle inequality holds, then a • b ≤ kakkbk holds for a and for −a, from which Cauchy–Schwarz inequality follows. If equality holds, i.e., if a • b = kakkbk, then b = ca, for some scalar c. Hence, a • b = ckak2 , and kakkbk = |c|kak2 . For nonnull a, this implies that c = |c|, so that c ≥ 0. If b 6= 0, then b = ca, with c > 0.

1.3 Since x0 y = 0 = x0 z = y0 z, it follows that we must a2 + b2 = 1, and √ solve the equations √ 2 2 a − b = 0, for which the solutions are a = ±1/ 2 and b = ±1/ 2.

1.4 Let 

1  1 V=  0 −1

 2 0 0 −2 .

1 1 −1 1 1

2

Solutions to Chapter 1 Using elementary column transformations C2 − 2C1 , and then, C3 − C2 , the matrix V becomes   1 0 0  1 −2 0 , V=  0 1 0 −1 −1 0 so that the column rank of V (or the dimension of its column space) is 2 < 3. Therefore, v1 , v2 and v3 are linearly dependent. It is easily seen that v1 and v2 are LIN, and that v3 = v2 − 2v1 .

1.5 It is easy to see that c1 v1 + c2 v2 + c3 v3 = 0 results in the following three equations, and the only solution is c1 = 0, c2 = 0, and c3 = 0: 2c1 + 8c2 − 4c3 = 0 3c1 − 6c2 + 3c3 = 0 2c1 + 5c2 + c3 = 0.

1.6 Using elementary transformations, we can show that A is equivalent to the matrix   −3 3 3  0 4 4 , 0 0 −1 so that the columns of A are LIN.

2 1 3 1.7 We can solve the system = c1 + c2 to get c1 = −1 and c2 = 1. Hence, u 3 2 5 is in Span{v1 , v2 }.

1.8 If v1 ,P · · · , vm are linearly dependent, then there are scalars c1 , · · · , cm not Pall zero, such m that c v = 0.

For any k with c = 6 0, we then have v = − i i k k i=1 i6=k (ci /ck )vi , showing property 1 in Result 1.2.2.

Suppose without loss of generality that Pvs 1 , · · · , vs are linearly dependent and c1 , · · · , cs are constants, not all zero, such that P i=1 ci vi = 0. Let cj = 0 for all j = s + 1, · · · , n. n Then c1 , · · · , cn are not all zero and i=1 ci vi = 0. Hence v1 , · · · , vn are linearly dependent, showing property 2 in Result 1.2.2.

1.9 1. Let S = {v1 , · · · , vn } denote a set of nonzero orthogonal vectors, and let u belong to the span of S with u = c1 v1 + · · · + cn vn .

For a fixed i = 1, · · · , n, take the inner product of each side with vi . Since vi • vj = 0, i 6= j, u • vi = c1 (v1 • vi ) + · · · + cn (vn • vi ) = ci (vi • vi ).

Hence, ci = (u • vi ) (vi • vi ). To verify linear independence, set u = c1 v1 +· · ·+cn vn = 0.

This implies that ci = 0, i = 1, · · · , n, which in turn implies LIN of {v1 , · · · , vn }.

Solutions to Chapter 1

3

2. By definition, every v ∈ V1 + · · · + Vm can be written as v1 + · · · + vm , where vi ∈ Vi , i = 1, . . . , m. Let wi ∈ Vi , i = 1, . . . , m, such that we also have v = w1 + · · · + wm . Then (w1 − v1 ) + · · · + (wm − vm ) = 0. For each i, pre-multiply both sides by (wi − vi )0 . For j 6= i, since Vi ⊥ Vj , then (wi − vi )0 (wj − vj ) = 0. As a result, (wi − vi )0 (wi − vi ) = kwi − vi k2 = 0, giving wi = vi . Hence by definition, the sum of the Vi ’s is a direct sum.

1.10 Since {v1 , · · · , vm } is a basis of V, the vectors are LIN. So from y1 = v1 ,

yk = vk −

k−1 X

yi0 vk yi , k = 2, · · · , m, kyi k2 i=1

yk 6= 0, implying that zk = kyykk k are well-defined and each has length 1. On the other hand, for 1 ≤ j < k ≤ m, yj0 yk = yj0 vk −

j−1 k−1 X X y 0 vk yi0 vk 0 yi0 vk 0 i y y = − y y − y0 y . i j 2 j 2 i 2 j i ky k ky k ky k i i i i=1 i=1 i=j+1

k−1 X

If k = 2, then j = 1, and it is straightforward to see that y10 y2 = 0. Suppose we have shown that for all i < j < k, yi0 yj = 0. Then the above identity shows that for all j < k, yj0 yk = 0. By induction, the yi ’s are orthogonal to each other. Then z1 , · · · , zm are orthonormal. Since they are LIN from Exercise 1.9, and there are m of them, they form an orthonormal basis of V. 1.11 We have W∩(W ⊥ ∩V) ⊂ W∩W ⊥ = {0}. From W ⊂ V and W ⊥ ∩V ⊂ V, W⊕(W ⊥ ∩V) ⊂ V. On the other hand, for any v ∈ V, there are unique w ∈ W and u ∈ W ⊥ , such that v = w + u. Since w ∈ V, then u = v − w ∈ V, and so u ∈ W ⊥ ∩ V. As a result V ⊂ W ⊕ (W ⊥ ∩ V). Then V = W ⊕ (W ⊥ ∩ V). From the paragraph below Definition 1.2.8, dim V = dim W + dim(W ⊥ ∩ V), completing the proof of property 1. Next, by W ⊥ ∩V ⊂ W ⊥ , (W ⊥ ∩V)⊥ ∩V ⊃ (W ⊥ )⊥ ∩V = W ∩V = W. On the other hand, if v ∈ V and v ⊥ (W ⊥ ∩ V), then by property 1, there are unique w ∈ W and u ∈ W ⊥ ∩ V such that v = w+u. From assumption, v ⊥ u. Meanwhile w ⊥ u. Then u0 u = u0 (v−w) = 0, so u = 0. Then v = w ∈ W. As a result, (W ⊥ ∩V)⊥ ∩V ⊂ W. Then (W ⊥ ∩V)⊥ ∩V = W, showing property 2.

1.12 In order for a matrix A = {aij } i, j = 1, 2, 3, to commute with the matrix B, we require that AB = BA. Computing the product on both sides, and equating them, we see that the conditions are a11 = a22 = a33 , a12 = a23 , and a21 = a31 = a32 = 0.

1.13 By repeated multiplication, we see that k a A = 0 k

β , 1

where, β = b(1 + a + · · · + ak−1 ).

1.14 It is easily verified that the product (A − B)C is equal to Ak − Bk , where, C = Ak−1 + Ak−2 B + · · · + ABk−2 + Bk−1 .

1.15 Clearly, (A0 A)0 = A0 (A0 )0 = A0 A, and (AA0 )0 = (A0 )0 A0 = AA0 .

1.16 If A = O, then clearly A0 A = O. To show the converse, let the column vectors of A be a1 , · · · , an . Since A0 A = {a0i aj }, if A0 A = O, then for all i = 1, · · · , n, a0i ai = 0, giving ai = 0, and so A = O.

4

Solutions to Chapter 1

1.17 (a) From Result 1.3.5, tr(A

k X

xi x0i ) = tr(

i=1

k X

Axi x0i ) =

i=1

=

k X

k X

tr(Axi x0i )

i=1

k X tr(x0i Axi ) = (xi Axi ).

i=1

i=1

(b) From property 4 of Result 1.3.5, it follows that tr(B−1 AB) = tr(BB−1 A) = tr(A).

1.18 Suppose A = {aij } and B = {bij } are both lower triangular matrices and P aii = ai and n bii = bi . Then aij = bij = 0 for all i < j. Now C = AB = {cij } with cij = k=1 aik bkj .

If i > j, then for every k = 1, · · · , n, either i > k or k > j, so aik bkj = 0, giving cij = 0. Then C is lower triangular. If i = j, then aik bkj 6= 0 if and only if k = i, so cii = aii bii = an bn . This completes the proof for the lower triangular case. The proof for the upper triangular case is similar.

Pm Pn 1.19 Properties 1, 3, 5, and 6, of Result 1.3.5 are obvious. From tr(AB) = i=1 j=1 aij bji = Pn Pm Pn Pm j=1 i=1 aij bji = j=1 i=1 bji aij = tr(BA), Property 2 follows. Property 4 is a direct consequence, since we can regard ABC as the product of AB and C, or of A and BC. Both tr(AA0 ) and tr(A0 A) equal the sum of squares of the elements of A, which is nonnegative, and shows property 7, of which property 8 is an immediate consequence.

1.20 If A = {akl } is lower triangular, then akl = 0 for all k < l. Now Mij = {mkl }, where  akl if k < i, l < j    a if k ≥ i, l < j k+1,l mkl =  ak,l+1 if k < i, l ≥ j    ak+1,l+1 if k ≥ i, l ≥ j.

If i > j, then for k < l, the second case in the display is not possible, and all the other cases are possible, each one giving value 0. Therefore, mkl = 0, so that Mij is lower triangular. Furthermore, for any k = j, · · · , i − 1, mkk = ak,k+1 = 0, so Mij has at least one diagonal element equal to zero. This verifies the first fact.

From (1.3.1), the ith diagonal element of A−1 is |Mii |/|A|. If the diagonal elements of A are d1 , · · · , dn , then the diagonal elements of Mii are d1 , · · · , di−1 , di+1 , · · · , dn . Therefore |Mii | = d1 · · · di−1 di+1 · · · dn and |A| = d1 · · · dn . Then the ith diagonal element of A−1 is 1/di .

1.21 Using Definition 1.3.7, |A| = −8.

1.22 It is easy to verify the given resursive expression, from which we obtain ∆n = (1 + a2 + a4 + · · · + a2n ) = [1 − a2(n+1) ]/[1 − a4 ].

1.23 This follows directly from property 7 of Result 1.3.6.

1.24 Using Definition 1.2.7, we can show that the value of this determinant is 1 +

Pn

i=1 ai .

1.25 The determinant to be calculated is |(1 − ρ)In + ρJn |. If ρ 6= 1, then from 12 of Result 1.3.6, the determinant is (1 − ρ)n |In + ρ(1 − ρ)−1 1n 10n | = (1 − ρ)n [1 + ρ(1 − ρ)−1 10n 1n ] = (1 − ρ)n [1 + nρ(1 − ρ)−1 ] = (1 − ρ)n−1 [1 + (n − 1)ρ].

Solutions to Chapter 1

5

1.26 We find the RREF(A : I) = (B : C), say. We see that the matrix   1 2 −1 0 B = 0 −4 0 0 0 has a row of zeroes, so that A is singular.

1.27 If A is n × n and invertible, then there is an n × n matrix B such that AB = I. Then |A||B| = |AB| = |I| = 1, so |A| = 6 0.

1.28 We verify Result 1.3.8. The first four properties are straightforward. See Theorem 18.2.8 in Harville (1997) for a proof of the Sherman–Morrison–Woodbury formula. To prove property 6,

(A−1 a)(b0 A−1 ) 0 −1 (A ± ab ) A ∓ 1 ± b0 A−1 a = I ± ab0 A−1 ∓

ab0 A−1 ab0 A−1 ab0 A−1 − .

1 ± b0 A−1 a 1 ± b0 A−1 a

Let us denote the scalar b0 A−1 a by c. Hence, ab0 A−1 ab0 A−1 = acb0 A−1 = cab0 A−1 = (b0 A−1 a)ab0 A−1 so that

(A−1 a)(b0 A−1 ) (A ± ab0 ) A−1 ∓ 1 ± b0 A−1 a = I ± ab0 A−1 ∓

ab0 A−1 ab0 A−1 − (b0 A−1 a) = I. 0 −1 1±bA a 1 ± b0 A−1 a

1.29 (a) Since (I − A(I + BA)−1 B)(I + AB) = (I + AB) − A(I + BA)−1 (I + BA)B = I + AB − AB = I. Similarly, we can show that (I + AB)(I − A(I + BA)−1 B) = I, and it follows that (I + AB)−1 = I − A(I + BA)−1 B. (b) Note that we can write aIk + bJk = a(Ik + ab Jk ) = a(Ik + ab 1k 10k ). Set ab 1k = A and 10k = B in (a) and use that result to get

−1 1 b −1 0 (aIk + bJk ) = Ik + 1k 1k a a "

−1 # 1 b kb = Ik − 1 k 1 + 10k a a a

ab 1 b 1 Ik − Jk = I k − Jk . = a a(a + kb) a a(a + kb) 1.30 (a) Use Result 1.3.6. Since A is orthogonal, |A|2 = |AA0 | = I| = 1, which implies |A| = ±1. (b) This follows from Definition 1.2.9 and Definition 1.3.11. (c) AB(AB)0 = ABB0 A0 = AA0 = In .

1.31 Since r(A) = 1, A can be written as ab0 for some a 6= 0 and b 6= 0. Then by property 12 of Result 1.3.6, |In +A| = 1+b0 a = 1+tr(b0 a), which by Result 1.3.5 is 1+tr(ab0 ) = 1 + tr(A).

6

Solutions to Chapter 1

1.32 It is easy to see that the first two columns of A are LIN, while the third column is the first column times 2 minus the second one. Then the dimension of the column space is 2. We can see that   1 0 2 0 1 −1  RREF(A) =  0 0 0 0 0 0 so the dimension of the row space of A is also 2. Then dim(N (A)) = 3 − 2 = 1.

1.33 Using row transformations R2 − 3R1 , R3 − 5R1 , and then R3 − 2R2 , A '   1 2 4 3 0 −7 −10 −11, so that r(A) = 2.

0 0 0 0 1.34 Let c be an eigenvalue of A ∈ Rn×n with geometric multiplicity g ≥ 1. Then there are LIN vectors v1 , · · · , vg such that Avi = cvi . Expand {v1 , · · · , vg } into a basis cIg B {v1 , · · · , vn } of Rn and let V = (v1 , · · · , vn ). Then V−1 AV = . Then O A2 (c − λ)Ig B = (c−λ)g |A2 −λIn−g |, showing |A−λIn | = |V−1 AV −λIn | = O A2 − λIn−g the algebraic multiplicity of c is at least g.

1.35 Suppose vi , wi ∈ Vi , i = 1, . . . , k, such that v1 + · · · + vk = w1 + · · · + wk . Then Pk i=1 (vi − wi ) = 0. Let ui = vi − wi . If not all ui are 0, then let ui1 , · · · , uim be all those that are not 0. Each uij is an eigenvector corresponding to λij . On the other hand, ui1 + · · · + uim = 0, so ui1 , · · · , uim are linearly dependent. This contradicts 1.3.17. As a result, for all i, vi = wi . From Definition 1.2.8, the sum of V1 , · · · , Vk is a direct sum.

1.36 Let v 6= 0 be an eigenvector corresponding to λ, i.e., Av = λv. Then for k ≥ 2, Ak v = Ak−1 (Av) = Ak−1 (λv) = λAk−1 v. By induction, Ak v = λk v, so λk is an eigenvalue of Pp 0 k Ak . Note that with A = I, this is also true for k = 0.

As a result, if f (x) = a x , k=0 k Pp Pp Pp k k k then f (A)v = ( k=0 ak A )v = k=0 ak A v = k=0 ak λ v = f (λ)v. Therefore f (λ) is an eigenvalue of f (A), proving (a). If A is invertible, then Av = λv yields v = λA−1 v, and hence A−1 v = λ−1 v, proving (b).

Pn 1.37 Let |λIn − A| P = i=0 an λi . By Vieta’s formulas, the eigenvalues λ1 , · · · , λn have the Qn n property that i=1 λi = −an−1 /an , and i=1 λi = (−1)n a0 /an . It is easy to see that an = 1, an−1 = −tr(A), and a0 = | − A| = (−1)n |A|.

QnThen properties 1 and 2 of Result 1.3.19 follow. Property 3 follows from |λIn ± A| = i=1 (λ ± λi ) with λ = 1.

P∞ (k) 1.38 Let A = {aij } be n×n. Denote Ak = {aij }. It is clear that the convergence of k=0 Ak implies Ak → O. To show that the converse is true, denote kAk = max1≤i,j≤n |aij |.

If Ak → O, we can fix k0 , such that kAk0 k = b < 1/(2n). Then for m ≥ 2, Pn (mk −k ) (k ) kAmk0 k = maxi,j | k=1 aik 0 0 akj0 | ≤ kAmk0 −k0 k · (nb) < kAmk0 −k0 k/2. By induction, kAmk0 k ≤ (1/2)m−1 b. Let C = max0≤j<k kAj k. Then for k ≥ k0 , writing k = mk0 + j, m ≥ 1, 0 ≤ j < k0 , P kAk k = kAmk0 Aj k ≤ kAmk0 k(nC) ≤ ∞ m−1 k/k0 k (nCb)(1/2) ≤ (4nCb)(1/2) . Hence k=0 A (absolutely) converges. Let the P∞ k sum be M. Then M = In + k=1 A = In + AM, giving (In − A)M = I. Then In − A is nonsingular with inverse M.   1 1 1 1.39 Let B = A − aI3 = 1 1 1 = J3 , which clearly has rank 1. It is easy to verify 1 1 1

Solutions to Chapter 1

7

that solving |B − λI| = (1 − λ)3 − 2(1 − λ) + (1 + λ) = 0, we get λ = 3, and λ = 0 (with multiplicity two) as the eigenvalues of b. The eigenvector corresponding to λ = 3 is obtained as v = t(1, 1, 1)0 , for arbitrary t 6= 0.

1.40 Let λ denote a nonzero eigenvalue of AB. Then, there exists a nonnull vector u such that ABu = λu. Hence, BABu = λBu. Clearly v = Bu is a nonzero vector and BAv = λv, so that λ is an eigenvalue of BA. To show the converse, we just reverse the roles of A and B. 1.41 Since BB−1 = I, and |B−1 | = |B|−1 , |A − λI| = |B||A − λB−1 B||B−1 | = |BAB−1 − λI|.

Since A and BAB−1 have the same characteristic polynomial, they have the same eigenvalues. Let Avj = λj vj , where vj 6= 0. Then BAB−1 (Bvj ) = BAvj = λj (Bvj ), Note that Bvj 6= 0. Then Bvj is an eigenvector of BAB−1 corresponding to the eigenvalue λj .

1.42 (a) Given any x and y, (tx − y)0 A(tx − y) = (x0 Ax)t2 − 2(x0 Ay)t + y0 Ay = f (t) is a nonnegative function in t. If x0 Ax > 0, then f (t) is a quadratic function and its minimum is −(x0 Ay)2 /(y0 Ay) + x0 Ax ≥ 0, giving (x0 Ay)2 ≤ (x0 Ax)(y0 Ay). If x0 Ax = 0, then from the assumption, x = 0, and in order for f (t) to be nonnegative, there must be x0 Ay = 0, again showing the inequality.

To complete the proof of (a), it only remains to show that if (x0 Ay)2 = (x0 Ax)(y0 Ay) and x 6= 0, then y = cx for some scalar c. However, from the above argument, if the equality holds, then there is c such that f (c) = (cx−y)0 A(cx−y) = 0, so by the assumption, y = cx, as desired. (b) Let z = A−1 y. Then from (a), (x0 y)2 = (x0 Az)2 ≤ (x0 Ax)2 (z0 Az)2 = (xAx)2 (y0 A−1 y)2 . The necessary and sufficient condition of equality follows that from (a).

Solutions to Chapter 2

2.1 We see that A0 A is an m × m matrix and 0 0

A AA A B P0 P = I = = B0 B0 A

A0 B I = m B0 B O

O In−m

,

so that A0 A = Im . Then (AA0 )2 = A(A0 A)A0 = AIm A0 = AA0 , so AA0 is idempotent.

2.2 Since r(A) = r(M) = r and A is r × r, A is nonsingular. Consider Result 2.1.3. By using a similar argument, r(M) = r(A) + r(D − CA−1 B). Then r(A − CA−1 B) = 0.

Since a matrix has rank 0 if and only if it is O, then D = CA−1 B. 2.3 By Result 2.1.3, |A| = |P−xx0 | = |P||1−x0 P−1 x|, so that (1−x0 P−1 x)|P| = |P−xx0 |, from which the result follows.

I −B 2.4 The solution to (a) follows directly from Result 2.1.1. By Result 2.1.3, is O I nonsingular. Then by property 5 of Result 1.3.11 followed by Result 2.1.1, the solution to (b) follows:

A AB A AB I −B r =r CA D CA D O I

A 0 =r = r(A) + r(E).

CA E 2.5 This is immediately obvious from Example 2.1.3 by seeing that when A2 is an n × 1 matrix, the expression A02 (I − P1 )A2 is a scalar.

2.6 By Result 2.1.4, we can write −λIn B

−A = |Ik ||(−λIn ) − (−A)Ik B| Ik = | − λIn + AB| = |AB − λIn |.

Applying the same result again, we have −λIn B

−A = | − λIn ||Ik − B(λIn )−1 (−A)| = (−λ)n |Ik − λ−1 BA| Ik = (−λ)n | − λ−1 (−λIk + BA)| = (−λ)n (−λ)−k |BA − λIk | = (−λ)n−k |BA − λIk |.

Hence, (−λ)n−k |BA − λIk | = |AB − λIn |, proving (a). To prove (b), let λ be a nonzero eigenvalue of the matrix AB, so that we have P (λ) = |AB − λIn | = 0, where P (λ) is the characteristic polynomial of AB. Using (a), this implies |AB − λIn | = (−λ)n−k |BA − λIk | = 0, 9

10

Solutions to Chapter 2 i.e., λ is a non-zero eigenvalue of the matrix BA. Hence, each of the k non-zero eigenvalues of AB is an eigenvalue of BA. Further, an inspection of the characteristic polynomial shows that the n × n matrix AB has (n − k) zero eigenvalues. We now prove (c). Let v denote a nonzero eigenvector of AB corresponding to λ 6= 0. Then, ABv = λv, or BA(Bv) = λ(Bv), which implies that Bv is an eigenvector of BA corresponding to λ.

2.7 Clearly, r(A) = r(aa0 ) = 1, so that A has one nonzero eigenvalue. Since Aa = aa0 a = (a0 a)a, it follows that the nonzero eigenvalue of A is a0 a with corresponding eigenvector a (see Definition 1.3.16).

2.8 Note that we can write (A + BC) = A(Ik + A−1 that |A + BC| = |A||Ik + BC), so −1 I −A B k A−1 BC|. Applying Result 2.1.4 to the matrix , we see that its deC In −1 terminant is equal to |Ik ||In + CA B| = |In ||Ik + A−1 BC|, so that |A + BC| = |A||In + CA−1 B|.

2.9 Let J be the Jordan canonical form of A so that A = UJU−1 . Then for k ≥ 2, −1 −1 −1 Ak = (UJU−1 )(UJU ) = UJk UP . This also holds 0. As Pp ) · · · (UJU Pp for k k= −1 p k k a result, if f (x) = a x , then f (A) = a A = a UJ U = k=0 k k=0 k k=0 k Pp U( k=0 ak Jk )U−1 = Uf (J)U−1 , so f (A) and f (J) have the same eigenvalues, counting multiplicity. Now J is an upper triangular matrix with diagonal entries equal to the eigenvalues λ1 , · · · , λn counting multiplicity. From Result 1.3.3 and induction, Jk is an upper triangular matrix with diagonal entries λk1 , · · · , λkn , and so f (J) is an upper triangular matrix with diagonal entries f (λ1 ), · · · , f (λn ). As a result, Qn |f (J) − λIn | = (f k=1 (λk ) − λ), yielding that the eigenvalues of f (J), and hence those of f (A) are f (λ1 ), · · · , f (λn ) counting multiplicity. If A is nonsingular, then A−1 = (UJU−1 )−1 = UJ−1 U−1 . From Result 1.3.9, J is upper triangular with diagonal entries 1/λ1 , · · · , 1/λn . Then the eigenvalues of A−1 are 1/λ1 , · · · , 1/λn , showing (b).

2.10 We can write (A + aa0 ) = A(Ik + A−1 aa0 ), so that |A + aa0 | = |A||Ik + A−1 aa0 |.

Using Result 2.1.4, and substituting Ik , −A−1 a, a0 , and 1 for A11 , A12 , A21 , and A22 respectively, we see that |A + aa0 | = (1 + a0 A−1 a)|A|.

2.11 From Result 2.3.4, A = PDP0 , where P is an n × n orthogonal matrix, and D = diag(λ1 , · · · , λn ) with λi being the eigenvalues of A. Denote the orthonormal column vector of P by p1 , · · · , pn . Then for any x,  0  p1 x n   X x0 Ax = x0 PDPx = (x0 p1 , · · · , x0 pn )D  ...  = λi (p0i x)2 . i=1 p0n x Pn Then x0 Ax ≥ λmin i=1 (p0i x)2 = λmin x0 x, with “=” if and only if p0i x = 0 for all p0i with λi > λmin . That is, the equality holds if and only if x is an eigenvector corresponding to λmin . Let x1 be an eigenvector corresponding to λmin . Then x01 Ax1 /x01 x1 = λmin ≤ x0 Ax/x0 x with equality if and only if Ax = λmin x. The other half the question can be similarly proved.

2.12 (a) By Result 2.3.4, r(A) = r(D), which is equal to the number of nonzero elements of the diagonal matrix D. 0 0 2 0 2 0 0 2 (b) kAk Pn =2 tr(A A) = tr(A ) = tr(APP APP ) = tr(P APP AP) = tr(D ) = i=1 λi .

Solutions to Chapter 2

11 Pk

2.13 For any matrix A, since gj ≤ aj for each λj , j = 1, · · · , k, it follows that j=1 gj ≤ Pk Pk j=1 aj . Since the characteristic polynomial is of degree n, we must have j=1 aj ≤ n.

Pk Pk Pk

By Result 2.3.4, j=1 aj = n, from which it follows that j=1 gj = j=1 aj = n. Then, Pk j=1 (aj − gj ) = 0, and aj − gj ≥ 0, j = 1, · · · , k. We must have aj = gj , j = 1, · · · , k.

2.14 Let Q = (q1 , · · · , qn ). Since Q is nonsingular, the q’s are LIN. Then Q−1 AQ = diag(λ1 , · · · , λn ) if and only if AQ = Q diag(λ1 , · · · , λn ), which, by comparing the column vectors on both sides, is equivalent to Aqj = λj qj , j = 1, . . . , n, i.e., the q’s are the eigenvectors of A. This shows property 1. To show 2, since P is orthogonal, it is nonsingular. Then the proof immediately follows from property 1.

2.15 (a) Since x0 x is a scalar, it is easy to verify that

xx0 x0 x

0 =

xx0 , x0 x

which shows symmetry, while idempotency holds since

xx0 x0 x

2 =

xx0 xx0 (x0 x)xx0 xx0 = = 0 .

0 2 0 2 (x x) (x x) xx

(b) xx0 is clearly symmetric; however, it is not idempotent unless x0 x = 1.

2.16 Since the column vectors of U are orthonormal, we can construct an n × (n − k) matrix V such that the columns of (U V) form an orthogonal basis for Rn . Then 0

0

U AU U0 AV U V A U V = .

V0 AU V0 AV

By AU = UB, U0 AU = U0 UB = B, V0 AU = V0 UB = O, and since A is symmetric, U0 AV = (V0 AU)0 = (V0 UB)0 = O. 2.17 Since A is symmetric and idempotent, we have A = AA0 . For any matrix A, we know that AA0 is n.n.d.

2.18 Suppose A is an m × n matrix. For λ 6= 0, let p(λ) = |AA0 − λIm | = (−λ)m |Im − λ−1 AA0 |, and q(λ) = |A0 A − λIn | = (−λ)n |In − λ−1 A0 A| = (−λ)n |Im − λ−1 AA0 | (using Exercise 2.8). When m = n, p(λ) = q(λ). Hence, the eigenvalues coincide.

2.19 For λ 6= 0, let p(λ) = |AB − λIm | = (−λ)m |Im − λ−1 AB|, and q(λ) = |BA − λIn | = (−λ)n |In − λ−1 BA| = (−λ)n |Im − λ−1 AB|, using Exercise 2.8. If m > n, then AB is singular, and p(0) = 0. If m < n, then BA is singular and q(0) = 0. When m = n, p(0) = |AB| = |A||B| = |BA| = q(0). This proves (a). To show (b), since λ is an eigenvalue of AB with eigenvector x, we have ABx = λx. Now, BAy = BABx = Bλx = λBx = λy, so that y is the eigenvector of BA corresponding to λ.

2.20 Let p1 , · · · , pn ∈ Rn be the orthonormal eigenvectors of A corresponding to its eigenvalues λ1 , · · · , λn . For any v with kvk = 1, let yi = v0 pi , i = 1, . . . , n. It follows that ! n n n X X X 2 0 0 0 0 yi = v pi pi v = v pi pi v = kvk2 = 1 i=1

i=1

i=1

12

Solutions to Chapter 2 and that 0

v Av = v

0

n X

! λi pi p0i

v=

i=1

n X

λi yi2 .

i=1

For k = 1, . . . , n, define the vector spaces Pk = Span{p1 , · · · , pk }.

Let V be any subspace of dimension n − k + 1, i.e., dim(V) = n − k + 1. Since dim(Pk ) + dim(V) = n + 1, there exists one nonzero vector v ∈ Pk ∩ V. For such v, we have yi = v0 pi = 0 for i = k + 1, . . . , n, and therefore, 0

v Av =

n X

λi yi2 =

i=1

k X

λi yi2 ≥ λk

k X

i=1

yi2 = λk ,

i=1

which implies that max v∈V:kvk=1

v0 Av ≥ λk .

Since V is an arbitrary subspace of dimension n − k + 1, we have inf

max

dim(V)=n−k+1 v∈V:kvk=1

v0 Av ≥ λk .

To show equality, take V = Span{pk , . . . , pn } (Note dim(V) = n − k + 1), then for any v ∈ V s.t. kvk = 1, we have yi = v0 pi = 0,

i = 1, . . . , k − 1,

and therefore, v0 Av =

n X i=k

λi yi2 ≤ λk

n X

yi2 = λk .

i=k

This proves the min-max characterization. The max-min characterization can be similarly proved.

2.21 Given that QAQ−1 = D, where D is a diagonal matrix, the result follows by inverting both sides, so that (QAQ−1 )−1 = D−1 , i.e., QA−1 Q−1 = D−1 .

Pr 2.22 (a) Let λ1 , · · · , λr denote the non-zero eigenvalues of P A, so that P tr(A) = i=1 λi .

P r r r Then, tr(A2 ) = i=1 λ2i ≥ 0, which implies that i=1 λi = i=1 λ2i = p ≥ 0.

Pr Pr 2 2 Now,0 ≤ i=1 (λi − λ)2 = i=1 λ2i − rλ = p − pr = p(1 − pr ). Since p ≥ 0, this p implies that 1 − r ≥ 0, i.e., 0 ≤ p ≤ r, i.e., 0 ≤ tr(A) ≤ r.

Pr (b) Use Result 2.3.7. We are given that tr(A)P= r = r(A). Now, i=1 (λi − λ)2 = r p λ p(1 − r ) = p(1 − rr ) = 0, i.e., λi = λ = i=1 ri = rr = 1, i = 1, · · · , r. That is, r eigenvalues of A are equal to 1, while the remaining n − r are zero, so A is idempotent of rank r.

2.23 Suppose that on the contrary, C is nonsingular, i.e., r(C) < q. If the columns of C are linearly dependent, there exists a nonzero q-dimensional vector d such that Cd = 0.

Since B is symmetric and idempotent, (BA2 d)0 (BA2 d) = d0 A02 BA2 d = d0 Cd = 0,

Solutions to Chapter 2

13

which implies that (see Exercise 2.10) BA2 d = 0. It follows that A2 d = (I − B)A2 d = A1 (A01 A1 )−1 A01 A2 d = A1 e, say. Since the columns of A1 are LIN of the columns of A2 , d must be 0. This implies that the columns of C must be LIN. S ince C is symmetric, its rows must be LIN too. Thus, r(C) = q, and C is nonsingular.

2.24 Since P is orthogonal, we have PP0 = In = P0 P; also since A is symmetric and idempotent, we have A0 = A, and A2 = A. Now, (P0 AP)0 = P0 A0 (P0 )0 = P0 AP, and (P0 AP)2 = P0 APP0 AP = P0 AP, by the orthogonality of P and the symmetry and idempotency of A. 2.25 To show that A is p.d., we can show that all the principal minors of A are positive, 2 −1 = 1 > 0; and |A| = 3 > 0. To find a matrix B such that i.e., a11 = 2 > 0; −1 1 0 A = BB , we see by the sequence of elementary row and column transformations R1 /2; R2 + R1 , R3 + R1 ; R3 − R2 that A is reduced to     1 −1/2 −1/2 1/2 0 0 1/2  , where R = 1/2 1 0 .

RA = 0 1/2 0 0 3 0 −1 1  1/2 0 0 Verify that RAR0 = D =  0 1/2 0, with |D| = 3/4. Let Q = D−1/2 = 0 0 3 √  2 √0 0  0 2 0√  and let P = QR. Then, PAP0 = I3 and A = P−1 (P0 )−1 , so 0 0 1/ 3  √  2√ 0√ 0 that B = P−1 = −1/√2 1/√2 √0 . Alternately, we set P (λ) = |A − λI| = 0, −1/ 2 1/ 2 3 and solve for the eigenvalues as λ1 = 0.3446, λ2 = 1.7892, and λ3 = 4.8662; since all the eigenvalues are positive, A is p.d. The Cholesky decomposition of A, which may be implemented using standard software gives the matrix B. 

2.26 The eigenvalues of A are [1 + (n − 1)a] and (1 − a) with multiplicity (n − 1). Since the eigenvalues of a p.d. matrix must be positive, A is p.d. for −1/(n − 1) < a < 1.

2.27 Let the smallest eigenvalue of B be β and the largest eigenvalue of A be α. Then β > 0 because B is p.d. Then, there is λ > 0 such that β > λα. By Exercise 2.11, for any nonzero x, x0 (B − λA)x = x0 Bx − λx0 Ax ≥ β 0 x − λαx0 x > 0. Therefore, B − λA is p.d.

2.28 (a) Setting x = ei gives e0i Aei = aii = 0. Then setting x = ei + ej , i 6= j, gives (ei + ej )0 A(ei + ej ) = aii + ajj + 2aij = 0, and so aij = 0. Then A = O. (b) If A is p.s.d., there exists an n × n matrix R such that A = RR0 . Let b = R0 x.

Then 0 = x0 Ax = x0 R0 Rx = b0 b, so that b = 0. Then Ax = BR0 x = Bb = 0.

14

Solutions to Chapter 2

2.29 Substituting for c, the first term on the right side can be written as [x − (A + B)−1 (Aa + Bb)]0 (A + B)[x − (A + B)−1 (Aa + Bb)] = x0 (A + B)x − x0 (A + B)(A + B)−1 (Aa + Bb) − (Aa + Bb)0 (A + B)−1 (A + B)x + (Aa + Bb)0 (A + B)−1 (A + B)(A + B)−1 (Aa + Bb) = x0 (A + B)x − x0 (Aa + Bb) − (Aa + Bb)0 x + a0 A0 (A + B)−1 Aa + a0 A0 (A + B)−1 Bb + b0 B0 (A + B)−1 Aa + b0 B0 (A + B)−1 Bb. The second term on the right side is (a − b)0 A(A + B)−1 B(a − b) = a0 A(A + B)−1 Ba − a0 A(A + B)−1 Bb − b0 A(A + B)−1 Ba + b0 A(A + B)−1 Bb. Since B(A + B)−1 A is symmetric and equal to A(A + B)−1 B, adding the two terms gives the right side x0 (A + B)x − x0 (Aa + Bb) − (Aa + Bb)0 x + a0 Aa + b0 Bb, which is equal to the left side.

2.30 (a) Use property 5 of Result 1.3.8 and simplify. (b) Substitute for (A−1 + C0 B−1 C)−1 from (a) into the LHS of (b) and simplify, i.e., LHS = {AC0 − AC0 (B + CAC0 )−1 CAC0 }B−1 = AC0 {In − (B + CAC0 )−1 CAC0 }B−1 = AC0 (B + CAC0 )−1 {(B + CAC0 ) − CAC0 }B−1 = AC0 (B + CAC0 )−1 = RHS. 2.31 Suppose there exists an n × n nonsingular matrix P such that P−1 Ai P = Di , i = 1, · · · , k, for some diagonal matrices D1 , · · · , Dk . For j 6= i = 1, · · · , k, P−1 Aj Ai P = P−1 Aj PP−1 Ai P = Dj Di = Di Dj = P−1 Ai PP−1 Aj P = P−1 Aj Ai P, so that Aj Ai = P(P−1 Aj Ai P)P−1 = P(P−1 Ai Aj P)P−1 = Ai Aj .

2.32 Suppose r(A) = a, and r(B) = b. Since A is a symmetric, n.n.d. matrix, there exists a nonsingular matrix T such that

I O T0 AT = D = a .

O O

E F Let T BT = , where E, F and G are respectively a × a, a × (n − a) and F0 G (n − a) × (n − a) matrices. Since B is a n.n.d. matrix with r(B) = b, it follows that T0 BT is a n.n.d. matrix with r(T0 BT) = b. Suppose r(E) = r, where r ≤ a, and r ≤ b.

0

Solutions to Chapter 2

15

We perform elementary row and column operations on the last r rows and columns of T0 BT; suppose we denote them by S0 and S respectively. We reduce T0 BT to   Ca O O S0 T0 BTS =  O Ic O O O Id with r(Ca ) = r, c = b − r, and d = r − c. We also have

I O S0 T0 ATS = a .

O O Since Ca is a symmetric matrix with r(Ca ) = r, there exists an orthogonal matrix Qa such that Q0a Ca Qa = Da = diag(λ1 , · · · , λr , 0, · · · , 0). Define  

Qa O O Qa O   R = O Ic O = .

O Ir O O Id From the above results, we see that 

Da R0 S0 T0 BTSR =  O O

 O O Ic O = ∆, O O

which is a diagonal matrix. Then, R0 S0 T0 ATSR =

Q0a Qa O

O I = a O O

O O

= D,

which is also a diagonal matrix. We have constructed a matrix P = TSR, such that P0 AP = D, and P0 BP = ∆.

2.33 The result follows directly by using the properties of Kronecker product and trace.

2.34 If possible, let P1 and P2 be two such matrices. Since u is unique, it follows that (P1 − P2 )y = O for all y ∈ Rn , so that P1 must be equal to P2 .

2.35 Since Ω is symmetric, (y − Xβ)0 Ω(y − Xβ) = y0 Ωy − 2β 0 (y0 ΩX)0 + β 0 (X0 ΩX)β.

Then by Results 2.7.2 and 2.7.3, ∂(y − Xβ)0 Ω(y − Xβ)/∂β = −2(y0 ΩX)0 + 2(X0 ΩX)β = −2X0 Ω(y − Xβ) and also ∂ 2 (y − Xβ)0 Ω(y − Xβ)/∂β∂β 0 = 2X0 ΩX. 2.36 First, from property 6, property 11 follows from (A ⊗ B)(A−1 ⊗ B−1 ) = (AA−1 ) ⊗ (BB−1 ) = In ⊗ Im = Imn . Next, from Result 2.2.6, A = UJU−1 and B = VKV−1 , where U and V are nonsingular, and J and K are Jordan canonical forms of A and B, respectively. Then by properties 6 and 11, A ⊗ B = (U ⊗ V)(J ⊗ K)(U ⊗ V)−1 . It is straightforward to check that the diagonal elements of J ⊗ K are exactly all possible pairwise product λi γj of the eigenvalues of A and B, allowing for multiplicity. Then property 9 follows. Finally, since the determinate of a square matrix is the product of all its eigenvalues, counting for multiplicity, property 10 follows from property 9.

Solutions to Chapter 3

−1 2 0 3.1 (a) r(A) = 2, and A = .

1 −1 0   1/7 2/7 0 3/7 −1/7 0 . (b) r(A) = 2, and A− =   0 0 0 0 0 0 −

3.2 G is a g-inverse of A if and only if it has the form

u0

In−1

a , where u, v are any v

vectors in Rn−1 and a ∈ R. 3.3 Given that KAKB− KAK = KAK, and KK = K, it follows that KAK(KB− K)KAK = KAKB− KAK = KAK = B. 3.4 Since GA = AG, for any k ≥ 1, Ak Gk Ak = (AGA)k = Ak .

3.5 We use Result 3.1.9. (a) By property 1 of Result 3.1.9, G0 is also a g-inverse of A0 A. The result follows by applying this to property 2 of Result 3.1.9. (b) Transpose both sides of (a); AG is a g-inverse of A0 . (c) Transpose (3.1.7). (d) Since AGA0 is invariant to G (by property 3 of Result 3.1.9), AG0 A0 = AGA0 . (e) (AG0 A0 )0 = AGA0 = AG0 A0 by (d).

3.6 If B− A− is a g-inverse of AB, then by (3.1.1), ABB− A− AB = AB. Premultiply both sides by A− and post-multiply both sides by B− ; we see that A− ABB− is idempotent.

3.7 A(A− C)(A− C)0 A = AA− CC0 (A− )0 A = AA− A(A− )0 A = A(A− )0 A = (AA− A)0 = (A)0 = A. 3.8 From Result 3.1.6, since G is a g-inverse of A, then r(A) ≤ r(G). Now A is a ginverse of G, then r(G) ≤ r(A), so r(A) = r(G). Conversely, from Result 3.1.3, if A = BDC, where B and C are nonsingular and D = diag(Ir , 0) with r = r(A), then Ir K G = C−1 EB−1 , where E has the form . From Result 2.1.3, r(G) = r(E) = L M r + r(M − KL), so if r(G) = r, then M = LK. It is direct to check that EDE = E. As a result, A is a g-inverse of G. Finally, observe that the g-inverse obtained by the algorithm in Result 3.1.2 has the same rank as A. 3.9 If AG is symmetric, denote it by P. We have that PA = A and for any u ⊥ C(A), by P = G0 A0 , Pu = G0 A0 u = 0. Then, P is the orthogonal projection matrix onto C(A).

The converse is clear. This proves (a). To show (b), note that the projection matrix is P = A(A0 A)− A0 . Therefore, AG = P if and only if A[G − (A0 A)− A0 ] = 0. Then, (b) follows from Result 3.2.4.

17

18

Solutions to Chapter 3

3.10 We will show that A having a commuting g-inverse implies (a), which in turn implies (b), which in turns implies that A has a commuting g-inverse.

If AGA = A and AG = GA, then GA2 = A, so r(A) ≤ r(A2 ). Since it always holds that r(A2 ) ≤ r(A), then (a) follows. Suppose r(A) = r(A2 ) = r. We first show that C(A) ∩ N (A) = {0}. Indeed, since R(A) ⊂ R(A2 ) and now they have the same dimension, the two spaces are equal, so by property 5 of Result 1.3.10, A = CA2 for some matrix C. If v = Au ∈ C(A) ∩ N (A), then v = CA2 u = CAv = 0. By Result 1.3.10, dim[C(A)] + dim[N (A)] = n. Then Rn = C(A) ⊕ N (A). It follows that, if U = (u1 , · · · , ur ) is a matrix of basis vectors of C(A), and V = (v1 , · · · , vn−r ) a matrix of basis vectors of N (A), then P = (U, V) is nonsingular. Now each column of AU is in C(A) so it has a unique linear combination of u1 , · · · , ur . As a result, AU = UA1 A1 O for some r × r matrix A1 . Then, AP = (AU, AV) = (UA1 O) = P . Let O O −1 B = P and note that r(A1 ) = r so A1 is nonsingular. Then (b) follows. Finally, if (b) holds, then B−1 diag(A−1 1 O)B is a commuting g-inverse of A. A shorter proof based on Jordan canonical form that (a) implies (b) is as follows. A = B−1 JB, with J = diag(J1 , · · · , Ja ) and each Jk = {cij } such that cii = ak , ci,i+1 = 1, and all other cij = 0. If ak 6= 0, then Jk is nonsingular so r(Jk ) = r(J2k ). P If ak = 0, P 2 then unless Jk is 1 × 1, r(Jk ) = r(Jk ) − 1. Since r(Jk ) = r(A) = r(A2 ) = r(J2k ), it follows that all the Jk with diagonal entry 0 are 1 × 1, which leads to (b).

3.11 By Result 2.3.4, A = P diag(D, O)P0 where P is an orthogonal matrix and D = diag(d dr ) with di 6= 0. For an arbitrary symmetric submatrix C, verify that 1, · · · , −1 D O P P0 is a symmetric commuting g-inverse of A. O C 3.12 AG2 x = AG2 AG1 x = AG1 x = x.

0 0 0 3.13 A = CB = (P diag(D−1 1 , O)P )(P diag(D1 , O)P ) = P diag(Ir , O)P , where D1 = 0 0 diag(λ1 , · · · , λr ). Since pi P = ei , where ei is the ith standard basis vector, we have ( e0i P0 = p0i if i ≤ r . p0i A = e0i diag(Ir , 0)P0 = 0 else

3.14 Let Ω = N (A0 ). Then Ω ⊂ Rm . By Result 2.6.1, PΩ = Im − PΩ⊥ , while by property 2 of Result 1.3.10, Ω⊥ = C(A). The proof then follows from Result 3.1.13.

3.15 This follows from property 2 of Result 3.1.9.

3.16 Let K = (A, B). Then M = K0 K. Let G = (A0 A)− . From Result 1.3.13, in order to prove that

−GA0 B G O + S− −B0 AG I , I O O {z } | {z } | C

D

is a g-inverse of M, it suffices to show that KCK0 K + KDK0 K = K. Now

0 G O A 0 KCK = (A, B) =P O O B0

Solutions to Chapter 3

19

where by Result 3.1.9, P = A(A0 A)− A0 is the orthogonal projection matrix onto the column space of C(A). Then, KCK0 K = P(A, B) = (A, PB) Next,

K

−GA0 B −GA0 B = (A B) = −PB + B = (I − P)B := E, I I

while S = B0 B − B0 PB = B0 (I − P)B = E0 E. Then, KDK0 = [(I − P)B][B0 (I − P)B]− [(I − P)B]0 = E(E0 E)− E0 .

Note that E0 K = B0 (I − P)(AB) = (O, E0 E). Then KDK0 K = E(E0 E)− E0 K = (O, E(E0 E)− E0 E) = (O, E).

Then, KCK0 K + KDK0 K = (A, B) = K. 3.17 (a) Since A is nonsingular, G is a g-inverse of AB ⇐⇒ ABGAB = AB ⇐⇒ B(GA)B = B ⇐⇒ GA = B− ⇐⇒ G = B− A−1 . (b) The solution for (b) is similar.

3.18 We just need to verify that (3.1.1) holds. We have

A AB Ir O = A Ir B + CA D C O

O E

= F + G.

Denote

L=

A− O

O , O

−B K= E− −C, In−r Ir

Note that FK, KF, GL, and LG are all null. Then, (F + G)(L + K)(F + G) = (FL + GK)(F + G) = FLF + GKG. It is easy to verify that FLF = F and GKG = G, completing the proof.

3.19 Using Result 3.2.2, the system is consistent since r(A) = r(A b) = 3.

3.20 The unique solution is (−1, −4, 2, −1)0 .

3.21 From Result 3.2.8, c0 x is invariant to the solution to Ax = b if and only if c0 = t0 A for some t. The latter is equivalent to c0 A− A = c0 .

3.22 C(c) ⊂ C(A) and R(b) ⊂ R(A); use Result 3.2.8 3.23 The partial derivatives of the Lagrangian function L(x, λ) with respect to x and λ are ∂L(x, λ)/∂x = 2Vx − 2c + Aλ ∂L(x, λ)/∂λ = A0 x − b and lead to the set of equations given in (3.3.6). It is easy to verify that these linear equations in x and λ are consistent. Solving these, we get the constrained minimizer x∗ , while λ∗ is the corresponding value of the vector of Lagrangian multipliers.

Solutions to Chapter 4

4.1 By property 8 of Result 1.3.11, r(X0 X, X0 y) ≥ r(X0 X). Also, r(X0 X, X0 y) = r[X0 (X, y)] ≤ X0 = r(X0 X), using properties 4 and 6 of Result 1.3.11. Hence, r(X0 X, X0 y) = r(X0 X).

4.2 (a) This is a direct consequence of property 2 of Result 3.1.9. (b) (XGX0 )0 = XG0 X0 = XGX0 by property 3 of Result 3.1.9, so that XGX0 and hence I − XGX0 are symmetric. Again by property 2 of Result 3.1.9, we show that XGX0 is idempotent, and it follows that I − XGX0 is idempotent.

PN PN b = 4σ 2 / PN t4 .

4.3 θb = 2 i=1 t2i Yi / i=1 t4i , and Var(θ) i=1 i PN PN 2 P N 4.4 vb = i=1 Si ti / i=1 ti , and Var(b v ) = σ 2 / i=1 t2i . b = (0, 3, 1)0 , and σ 4.5 β b2 = 4. The least squares estimates of β1 , β2 , β3 , β1 − β2 and β1 + β3 are respectively 0, 3, 1, −3 and 1.

2n1 + n2 n1 − 2n2 4.6 Since X0 X = , βb1 and βb2 will be uncorrelated if n1 = 2n2 . n1 − 2n2 n1 + 4n2 4.7 (a) Compute X0 X =

W0 W x0k W

W 0 xk . x0k xk

By Result 2.1.3, |X0 X| = |W0 W||x0k xk − x0k W(W0 W)−1 W0 xk |; the result follows since the second term is a scalar. (b) Since x0k xk ≥ [x0k xk − x0k W(W0 W)−1 W0 xk ], it follows from (a) that |W0 W|/X0 X| = [x0k xk − x0k W(W0 W)−1 W0 xk ]−1 ≥ 1/x0k xk . In (4.1.1), let β = (β 0W , βk )0 , with β W = (β0 , · · · , βk−1 )0 , and ! b ) b , βbk ) Var( β Cov( β W W b = Var(β) = σ 2 (X0 X)−1 . b , βbk ) bk ) Cov(β V ar( β W

By Result 2.1.3, Var(βbk ) = σ 2 [x0k xk − x0k W(W0 W)−1 W0 xk ]−1 = σ 2 |W0 W|/|X0 X|, which implies that Var(βbk ) ≥ σ 2 (x0k xk )−1 with equality when x0k W(W0 W)−1 W0 xk = 0, i.e., columns of W are orthogonal to xk .

P 2 4.8 (a) Var(βb0 ) = σ 2 /{ Xi2 −N X }. Setting the first derivative equal to zero and solving, we can see that X = 0 is a stationary point, while second derivative evaluated at P X = 0 is nonnegative. The minimum value of Var(βb0 ) is σ 2 / Xi2 .

21

22

Solutions to Chapter 4 P P 2 (b) Var(βb0 ) = {σ 2 /N }{ Xi2 /( Xi2 − N X )} = {σ 2 /N }Q, say. Clearly, Q is infinite when Xi = a, i = 1, · · · , N , or Xi = b, i = 1, · · · , N . Since Q has the form {Var(X) + (E(X))2 }/ Var(X), Pwhich is minimum when EX is at the center of the distribution, it follows that Xi = N (a + b)/2, i.e., N/2 of Xi ’s are equal to a and N/2 of Xi ’s are equal to b.

4.9

(i) βb1 = (Y2 + Y4 + Y6 − Y1 − Y3 − Y5 )/6, with variance σ 2 /6. (ii) βb1 = {5(Y2 − Y5 ) + 8(Y4 − Y3 ) + 11(Y6 − Y1 )}/48, with variance 420σ 2 /(48)2 .

The ratio of variances is then 32/35.

4.10 (a) That the εbi ’s are uncorrelated with the Ybi ’s is a direct consequence of property 3 of Result 4.2.4. To directly verify it without invoking the result, denote Xi∗ = Xi − X ∗ 0 and x∗ = (X1∗ , · · · , XN ) . Then Yi − Y = β0 + β1 Xi + εi − (β0 + β1 X − ε) = β1 Xi∗ + εi − ε.

By

P

∗ i Xi = 0 and (4.2.18),

P βb1 =

P ∗ Xi εi Xi∗ (β1 Xi∗ + εi − ε) = β1 + . kx∗ k2 kx∗ k2

So δ1 := βb1 − β1 = ε0 x∗ /kx∗ k2 .

Since βb0 = Y − βb1 X = β0 + β1 X + ε − βb1 X, then δ0 := βb0 − β0 = −δ1 X + ε.

For each i, Ybi = (β0 + β1 Xi ) + (δ0 + δ1 Xi ) = (β0 + β1 Xi ) + δ1 Xi∗ + ε, with β0 + β1 Xi begin a constant. Then Cov(Ybi , εbj ) = Cov(ε + δ1 Xi∗ , εj − ε − δ1 Xj∗ ) Now Var(δ1 ) = Var(ε0 x∗ )/kx∗ k4 = σ 2 /kx∗ k2 , Cov(δ1 , ε) = 0 as 10 x∗ = 0, and Cov(ε, εj − ε) = 0. Then Cov(Ybi , εbj ) = Cov(δ1 Xi∗ , εj − δ1 Xj∗ ) = Xi∗ Cov(ε0 x∗ , εj ) − Xi∗ Xj∗ Var(δ1 ) = Xi∗ Xj∗ /kx∗ k2 − Xi∗ Xj∗ /kxk2 = 0.

Then Ybi and εbj are uncorrelated. Finally, ∗ Cov(Yi , Ybj ) = Cov(εi , δ1 X j + ε) = Xi∗ Xj∗ /kxk2 + 1/n, ∗

Cov(Yi , εbj ) = Cov(εi , εi − δ1 X j − ε) = 1 − 1/n − Xi∗ Xj∗ /kxk2 , which are nonzero. So the Ybi ’s and εbi ’s are correlated with the Yi ’s. (b) From the solution to (a), Ybi = (β0 + β1 Xi ) + δ1 Xi∗ + ε and εbi = εi − δ1 Xi∗ − ε. Note PN b that εb = 0. Then the slope of the regression line of εbi on Ybi is Yi εbi /kb y−Yb 1k2 = i=1

b0b y ε/kb y − Yb 1k2 = 0. On the other hand, let Yi∗ = Yi − Y . Then the slope of PN PN b the regression line of εbi on Yi is bi /ky∗ k2 = bi )b εi /ky∗ k2 = i=1 Yi ε i=1 (Yi + ε P P 2 ∗ 2 2 2 ∗ ∗ 2 b b b kb εk /ky k . Now kb εk = (Yi − β0 − β1 Xi ) = (Yi − β1 Xi ) = ky∗ − βb1 x∗ k2 ∗0 and by βb1 = x y∗ /kx∗ k2 , kεk2 = ky∗ k2 − (x∗0 y∗ )2 /kx∗ k2 . Then the slope is 1 − (x∗0 y∗ )2 /kx∗ k2 ky∗ k2 .

Solutions to Chapter 4

23

4.11 Since the regression model is Yi = β1 Xi + εi , we need to minimize S(β1 ) =

N X (Yi − β1 Xi )2 . i=1

PN

PN

Since S 0 (β1 ) = −2[ i=1 Xi Yi − β1 i=1 Xi2 ], setting S 0 (βb1 ) = 0 yields βb1 = PN PN Xi Yi / i=1 Xi2 . Since εi are uncorrelated and Var(εi ) = σ 2 , then Var(βb1 ) = Pi=1 PN PN N 2 2 2 2 i=1 Var(Xi εi )/( i=1 Xi ) = σ / i=1 Xi .

P P 0 2 4.12 Minimizing i j (Yij −µ0 −τi0 −βj0 −γij ) with respect to µ, τi , βj , and γij , i = 1, · · · , a, j = 1, · · · , b, we obtain the normal equations X X XX 0 Y·· = abµ0 + b τi0 + a βj0 + γij , i 0

Yi· = bµ

+ bτi0 +

j

X

βj0 +

j

Y·j = aµ0 + 0

Yij = µ

X

τi0 + bβj0 +

i 0 0 + τi + βj0 + γij ,

i

X

0 γij ,

j

i = 1, · · · , a,

j

X

0 γij , j = 1, · · · , b,

i

i = 1, · · · , a, j = 1, · · · , b.

The only LIN set of ab equations are the last normal equations, since we can obtain the first three by summing the last over both i and j, over j alone, and over i alone. The rank of the estimation space is ab, which is equal to the number of observations, leaving no degrees of freedom for the error space. Hence, although we may obtain b.l.u.e.’s of estimable functions, we cannot carry out further inference such as confidence interval estimation or hypothesis testing.

4.13 We see that (Xβ)0 H = β 0 X0 GX0 X = (Xβ)0 using (3.1.7). Again, by (3.1.7), (X0 Xβ)0 H = β 0 X0 XGX0 X = (X0 Xβ)0 .

4.14 Let X = (X1 , X2 ) and β = (β 01 , β 02 )0 . Then y = Xβ + ε and X1 β 1 = (X1 , O)β. If X1 β 1 is estimable, then there exists T such that (X1 , O) = T0 X = (T0 X1 , T0 X2 ), i.e., X1 = T0 X1 and T0 X2 = O. Let x ∈ C(X1 ) ∩ C(X2 ). Then x can be written as x = X1 a = X2 b. Then x = X1 a = T0 X1 a = T0 X2 b = 0. Thus C(X1 ) ∩ C(X2 ) = {0}.

Conversely, suppose C(X1 ) ∩ C(X2 ) = {0}. Let β 0 be any LS solution to the model, and partition it into β 01 and β 02 . From property 5 of Result 4.3.1, to show that X1 β 1 is b = Xβ 0 estimable, it suffices to show that X1 β 01 is invariant to the choice of β 0 . First, y b is the sum of two unique is the projection of y onto C(X1 , X2 ), so it is unique. Next, y b = X1 β 01 +X2 β 02 and Xi β 0i ∈ C(Xi ), vectors respectively in C(Xi ), i = 1, 2. Then, since y i = 1, 2, Xi β 0i are unique vectors, and so are invariant to the choice of β 0 .

4.15 Suppose c0 β is estimable for all c. Then, c0 H = c0 , where H = GX0 X. In particular, H = Ik if and only if columns of X are LIN. Pa Pa 4.16 We have seen that i=1 ci τi is estimable if and only if i=1 ci = 0. Hence, τi is not estimable for any i.

Pa Pb 4.17 A basis set of a + b − 1 estimable functions is µ + a1 i=1 τi + 1b j=1 βj , τ1 − τi , i = Pa Pa 2, = 2, · · · , b. If i=1 ciP τi is estimable, we mustP have i=1 ci τi = P·a· · , a, and β1 − βj , j P P a a a a di (τ1 − τi ). Since i=1 di (τ1 − Pi=1 Pτai ) = τ1 i=1 di − i=1 di τi = ( i=1 di − d1 )τ1 − a i=2 Pdai τi , which implies that c1 = i=1 di −d1 = d2 +· · ·+da , c2 = −d2 , · · · , ca = −da , i.e., i=1 ci = 0.

24

Solutions to Chapter 4

1N 0N 4.18 (a) Let y = Xβ + ε, where y = (Y11 , · · · , YN,1 , Y12 , · · · , YN,2 )0 , X = , and 0 1N N

IN ρIN 0 2 ε = (ε11 , · · · , εN,1 , ε12 , · · · , εN,2 ) with Var(ε) = σ V, where V = . ρIN IN Then, 0 −1 b β X)−1 X0 V−1 y GLS = (X V !

PN 1 τb1 Y i1 = Pi=1 = .

N N τb2 i=1 Yi2

2 0 −1 b (b) Since Var(β X)−1 σ 2 = σN GLS ) = (X V √ s.e.(b τ1 ) = σ/ N . Substituting

1 ρ

√ ρ , we have s.e.(b τ1 ) = σ/ N , and 1

0 −1 1/2 b b σ bGLS = {(y − Xβ (y − Xβ GLS ) V GLS )/(2N − 2)}

for σ, we obtain the estimated s.e.’s of τb1 and τb2 .

4.19 (a) Equating (4.5.5) with (4.2.3) and setting y = Xz, we must have X0 V−1 Xz = X0 V−1 X(X0 X)−1 X0 Xz, which is true. (b) Since V−1 = (1 − ρ)−1 I − ρ(1 − ρ)−1 [1 + (N − 1)ρ]−1 J, and since 10 y = 0, the result follows.

4.20 First assume that c0 β is estimable in the model y = Xβ + ε; then there exists t such that c0 β = E(t0 y) = E(t0 KK−1 y) = E(l0 z), which implies that c0 β is estimable in the model z = Bβ + η.

Next, assume that c0 β is estimable in the model z = Bβ + η. This implies that there exists a vector t such that c0 β = E(t0 z) = E(t0 K−1 y) = E(l0 y), and therefore, c0 β is estimable in the model y = Xβ + ε.

4.21 (a) Since V = KK0 , X0 VX = (K−1 X)0 (K−1 X) = B0 B where B is the regressor matrix in the transformed linear model (4.5.3). Hence, (X0 VX)β = B0 Bβ is estimable. (b) Since V = KK0 , B = K−1 X, z = K−1 y, and η = K−1 ε in the transformed OLS model, c0 β is estimable if and only if c0 = c0 H, where H = (B0 B)− B0 B, i.e., if and only if c0 = c0 (B0 B)− B0 B = c0 (X0 K0−1 K−1 X)− X0 K0−1 K−1 X = c0 (X0 V−1 X)− X0 V−1 X. 4.22 First, we show that for any unbiased estimator d0 y of c0 β, Var(d0 y) ≥ Var(c0 β 0GLS ) with equality holding if and only if d0 y = c0 β 0GLS . Let c0 = t0 X. Then c0 β 0GLS = t0 Xβ 0GLS = bGLS . From d0 Xβ = E(d0 y) = c0 β = t0 Xβ for all β, d0 y bGLS = d0 Xβ 0GLS = t0 y 0 0 0 0 0 0 0 0 0 bGLS = d0 b t Xβ GLS = c β GLS . Then d y − c β GLS = d (b yGLS + b εGLS ) − t y εGLS . From 0b 0 0 property 3 of Result 4.5.1, Cov(d εGLS , c β GLS ) = 0, so Var(d0 y) = Var(d0 b εGLS + c0 β 0GLS ) = Var(d0 b εGLS ) + Var(c0 β 0GLS ) ≥ Var(c0 β 0GLS ), with equality holding if and only if d0 b εGLS is constant. Since E(d0 b εGLS ) = 0, the latter is equivalent to d0 b εGLS = 0, 0 0 0 or d y ≡ c β GLS .

Solutions to Chapter 4

25

bGLS , so by property 2 of Result 4.5.1, Second, let c0i = t0 X. Then c0i β 0GLS = t0 y Cov(c01 β 0GLS , c02 β 0GLS ) = t01 Var(y)t2 = σ 2 t01 X(X0 V−1 X)− X0 t2 = σ 2 c01 (X0 V−1 X)− c2 .

4.23 From (4.5.7), WX = KPC(LX) LX = KLX = X as L = K−1 , showing the second identity in (4.5.8). It follows that C(X) = C(WX) ⊂ C(W). On the other hand, again from (4.5.7), C(W) = C(X(X0 V−1 X)− X0 V−1 ) ⊂ C(X). Then C(X) = C(W), showing the first identity in (4.5.8). Finally, W2 = KPC(LX) LKPC(LX) = KPC(LX) PC(LX) = W, showing the last identity in (4.5.8).

4.24 The least squares estimates subjectP to the constraint θ1 + θ2 + θ3 = 180◦ are θbi = 3 ◦ Yi − Y + 60 , i = 1, 2, 3, where Y = i=1 Yi /3.

4.25 Recall that Var(βb0 ) = {σ 2

X

Xi2 }/{N

X

Var(βb1 ) = σ 2 /

X

(Xi − X)2 .

(Xi − X)2 }, and

Consider X ∈ [a, b]. Assume that a proportion p of the X values assume the value a, so Pn 2 that a proportion 1−p assume the value b. Then, X = p(a−b)+b, and i=1 Xi2 −N X = n(a − b)2 p(p + 1). Minimizing this as a function of p yields p = 1/2. Therefore, 50% of the X values assume the value a and 50% assume the value b. b = y, and β b = (IN − VC(C0 VC)−1 C0 )y.

4.26 β r 4.27 r(X) = 2. We can verify that β1 + β3 is estimable, with b.l.u.e. (2Y1 + 3Y2 − Y3 + Y4 )/5, whose variance is 3σ 2 /5. It can be verified that β2 is estimable, with b.l.u.e. (3Y1 +2Y2 + Y3 − Y4 )/5 whose variance is 3σ 2 /5. br = (y − y b ) − (b b) = b b ), where y b and b 4.28 b εr = y − y yr − y ε − (b yr − y ε are the unconstrained br vectors of fitted values and residuals, respectively. Note that b ε ⊥ C(X) while both y b are in C(X). As a result, b b ). Then SSEr = kb b k2 . and y ε ⊥ (b yr − y εr k2 = kb εk2 + kb yr − y 4.29 In the proof of Result 4.6.4, we saw SSEr = SSE + (A0 β 0 − b)0 (A0 GA)−1 (Aβ 0 − b).

Then E(SSEr ) = E(SSE) + E[(A0 β 0 − b)0 (A0 GA)−1 (Aβ 0 − b)] = σ 2 (N − r) + E[(A0 β 0 − b)0 (A0 GA)−1 (Aβ 0 − b)] = σ 2 (N − r) + tr[(A0 GA)−1 Cov(A0 β 0 )] + [E(A0 β 0 − b)]0 (A0 GA)−1 E(Aβ 0 − b) where the second equality is due to property 4 of Result 4.2.4, and the third equality is due to Result 4.2.3. From Result 4.2.4 and A0 H = A0 (because A0 β is estimable), Cov(A0 β 0 ) = σ 2 A0 GX0 XG0 A = σ 2 A0 GA while E(A0 β 0 − b) = A0 β − b. Note that A has q columns. Then E(SSEr ) = σ 2 (N − r) + σ 2 tr(Iq ) + +(A0 β − b)0 (A0 GA)−1 (Aβ − b) = σ 2 (N − r + q) + (A0 β − b)0 (A0 GA)−1 (Aβ − b).

26

Solutions to Chapter 4

4.30 (a) Let β = (µ, τ1 , · · · , τa )0 . µ = a01 β and τ1 + τ2 = a02 β, where a1 = (1, 0, · · · , 0)0 and a2 = (0, 1, 1, 0, · · · , 0)0 . Directly check that ai 6∈ R(X) while b := 2a1 +a2 ∈ R(X). (b) The constraints µ = s and τ1 + τ2 = t can be written as A0 β = d, where A = (a1 , a2 ) and d = (s, t)0 . We have seen that C(A) 6⊂ R(X) and dim(C(A)) = r(A) = 2, so the dimension of C(A) ∩ R(X) is at most 1. On the other hand, b ∈ C(A) ∩ R(X). As a result, C(A) ∩ R(X) is spanned by b. Under the constraint A0 β = d, it is clear that b0 β = 2µ + τ1 + τ2 = 2s + t. Then by Result 4.6.5, the vector of fitted values under the constraint A0 β = d is identical to that under the constraint 2µ + τ1 + τ2 = 2s + t.

Solutions to Chapter 5

5.1 First, each e0i x = Xi has a normal distribution. In particular, E(Xi2 ) < ∞. As a result, x has a well-defined mean µ and a variance-covariance matrix Σ. Now for each c, E(c0 x) = c0 µ and Var(c0 x) = c0 Σc, so by assumption, c0 x ∼ N (c0 µ, c0 Σc). The proof is complete by using property 2 of Result 5.2.4.

5.2 (a) Using Result 5.1.1 with B = 12 Σ−1 , b = Σ−1 µ, b0 = 12 µ0 Σ−1 µ, A = O, a = 0, and a0 = 0, 1 1 I = π k/2 2k/2 |Σ|1/2 exp{ µ0 Σ−1 ΣΣ−1 µ) − µ0 Σ−1 µ 2 2 = (2π)k/2 |Σ|1/2 , from which the required result follows.

(b) Using Result 5.1.1 with A = O, a = 0, a0 = 1, b = 0, b0 = 0, and B =

1 1

1 , 4

we evaluate the integral to be √π3 .

5.3 By equating terms, it is easy to verify that

3 2 −1/2 µ= , Σ−1 = , 1 −1/2 4 √ and k = 2π det(Σ)1/2 = 4π/ 31. Using property 2 of Result 5.2.4 with c = (3, −1)0 , we see that 3Xp 1 − X2 ∼ N (8, 140/31), so that P(3X1 − X2 < a) = 0.01 p if and only if P(Z < (a−8)/ 140/31) = 0.01, where Z ∼ N (0, 1). Equating (a−8)/ 140/31 = −2.26 and solving, we get a = 3.056.

5.4 Let Y1 = X1 + X2 and Y2 = X1 − X2 . We are given that y = (Y1 , Y 2 )0 ∼ N2 (0, I2 ).

1/2 1/2 Then, X1 = (Y1 + Y2 )/2 and X2 = (Y1 − Y2 )/2, i.e., x = (X1 , X2 )0 = y.

1/2 −1/2 Hence, x ∼ N2 (0, 21 I2 ).

5.5 The m.g.f. of x has the form of the m.g.f. of a trivariate normal distribution; equating terms, we see that     4 5 −1 2 3 1 . µ = −2 and Σ = −1 1 2 1 6 5.6 (a) Directly follows from Result 5.2.4. (b) By Result 5.2.5, Var(Y1 − Y2 ) = 1/σ1

−1/σ2

σ12 ρσ1 σ2

ρσ1 σ2 σ22

1/σ1 −1/σ2

= 2(1 − ρ). ρ measures the linear dependence between X1 and X2 .

27

28

Solutions to Chapter 5

5.7 Use the hint to get that the answer is 1/3.

5.8 (a) log f (x) = − k2 log(2π) − 12 log |Σ| − 12 x0 Σ−1 x, so that Z k 1 1 f (x)x0 Σ−1 xdx h(x) = log(2π) + log |Σ| + 2 2 2 k 1 1 = log(2π) + log |Σ| + E(x0 Σ−1 x) 2 2 2 k 1 1 = log(2π) + log |Σ| + tr(Σ−1 Σ) 2 2 2 1 1 k = [log(2πe) + log |Σ|] = log[(2πe)k |Σ|].

2 2 (b) First show that for i = 2, · · · , k, h(X1 , · · · , Xk ) = h(X1 ) +

k X

h(Xi |Xi−1 , · · · , X1 ), and

i=2

h(Xi ) − h(Xi | Xi−1 , · · · , X1 ) ≥ 0, with equality implying that f (X1 , · · · , Xi ) = f (Xi )f (Xi−1 , · · · , X1 ) almost everywhere (since the logarithmic function is strictly concave); this implies that X1 , · · · , Xk are independently distributed. Hence, h(X1 , · · · , Xk ) ≤

k X

h(Xi ),

i.e.,

i=1 k X 1 1 log{(2πe)k |Σ|} ≤ log{2πe|Σii |}, 2 2 i=1

|Σ| ≤

k Y

i.e.,

|Σii |,

i=1

with equality if and only if X1 , · · · , Xk are independent, i.e., if and only if Σij = O, i 6= j.

5.9 From Result 5.2.5, y ∼ Nk (0, σ 2 PP0 ), and the result follows from the orthogonality of P. 5.10 By the singular value decomposition, there exists an orthogonal matrix P such that PΣP0 = D, where D = diag(λ1 , · · · , λr , 0). Let B = (EP, O), where E = −1/2 −1/2 diag(λ1 , · · · , λr ). Let b = −Bµ. It is easy to verify that z = Bx + b ∼ Nr (0, I).   Σ11 Σ12 Σ13 5.11 Let µ = (µ01 , µ02 , µ03 )0 , and Σ = Σ21 Σ22 Σ23 . Apply Result 5.2.12 to get the Σ31 Σ32 Σ33 conditional distribution as Nk3 (µ3.12 , Σ33.12 ) where

with B013

µ3.12 = µ3 + B013 (c1 − µ1 ) + B023 (c2 − µ2 ),

Σ11 Σ12 −1 B023 = Σ31 Σ32 and Σ21 Σ22

Σ11 Σ12 −1 Σ13 Σ33.12 = Σ33 − Σ31 Σ32 . Σ21 Σ22 Σ23

Solutions to Chapter 5

29

5.12 Since

Σ11 Σ= Σ21

Σ12 Σ22

is symmetric and p.d., Σ22 is symmetric and p.d., so is nonsingular. Then by Result 2.1.4,

∆11 ∆12 Σ−1 = ∆21 ∆22 −1 −1 = Σ−1 where ∆11 = (Σ11 − Σ12 Σ−1 11.2 . Then Σ11.2 = ∆11 . In order for the 22 Σ21 ) components of x1 to be mutually independent conditionally on x2 , it is necessary and sufficient that Σ11.2 is a diagonal matrix, which is equivalent to ∆11 being a diagonal matrix.

5.13 If x1 = µ1.2 (x2 ), then clearly x1 is a linear function of x2 . Conversely, if x1 = Ax2 +a = f (x2 ), then f (x2 ) is an unbiased linear predictor with MSE equal to 0, so it must be the b.l.u.p.

5.14 (a) X2 ∼ N (µ2 , σ 2 ) and X3 ∼ N (µ3 , σ 2 ). (b) (X1 | X2 , X3 ) is normal with conditional mean µ1 + ρ(x2 − µ2 )/(1 − ρ2 ) − ρ2 (x3 − µ3 )/(1−ρ2 ) and conditional variance σ 2 (1−2ρ2 )/(1−ρ2 ); it reduces to the marginal distribution of X1 when ρ = 0. (c) Since Cov(X1 + X2 + X3 , X1 − X2 − X3 ) = σ 2 (−1 − 2ρ), the two variables are independent if ρ = −1/2.

1 0 0 0 0 0 2 2 0 0 5.15 (a) Write Cov(X1 , X2 ) = Cov(x Ax, x Bx), where A = and B = .

0 0 0 0 1 0 0 0 0 0 Then (see Exercise 5.27) 2Cov(x Ax, x Bx) = E[x Ax − tr(AΣ)][x Bx − tr(BΣ)] = ρ ρ 2 tr(AΣBΣ) = 2 tr = 2ρ2 . Similarly, Var(x0 Ax) = 2 tr(AΣAΣ) = 0 0

1 ρ 0 0 = 2 and Var(x0 Bx) = 2 tr(BΣBΣ) = 2 tr = 2, from which 2 tr 0 0 ρ 1 the result follows. (b) follows from Definition 5.2.7. (c) follows from Definition 5.2.8.

5.16 By Result 5.2.14, X̃i = µi + σij (Xj − µj )/σjj with σjj = σj2 . The proof follows by direct verification.

5.17 From Result 5.2.21, |ρjl|(q+1,··· ,k) | = 1 if and only if |ρ(εj , εl )| = 1, where εj = Xj − X̃j with X̃j = µj + Cov(Xj , x2 )Σ− 22 Cov(x2 , Xj )(x2 − c2 ) and likewise for εl . From Result 5.2.17, |ρ(εj , εl )| = 1 if and only if εj = d1 εl + d2 for some constants d1 6= 0 and d2 .

If the latter holds, then Xj = X̃j + d1 (Xl − X̃l ) + d2 , implying Xj is linear function of Xl and x2 . Conversely, if Xj = a0 x2 + bXl + c, then replacing Xj with εj + X̃j and Xl with εl + X̃l , εj − bεl = f (x2 ), where f (x2 ) is a linear function of x2 . Since both εj and εl are uncorrelated with x2 , then Var(f (x2 )) = Cov(εj − bε1 , f (x2 )) = 0, so f (x2 ) = εj − bεl = 0 as the mean of every εi is 0. Then εj = bεl . Note that b 6= 0 because otherwise εj = 0 so that ρjl|(q+1,··· ,k) is not well-defined. But then |ρjl|(q+1,··· ,k) | = 1.

30

Solutions to Chapter 5

5.18 From E(X) = E[E(X | Y )] for any jointly distributed X and Y , Cov(X1 , X2 ) = E(X1 X2 ) − [E(X1 )][E(X2 )] = E[E(X1 X2 | X1 )] − E(X1 ) E[E(X2 | X1 )] = E[X1 (α + βX1 )] + E(X1 ) E(α + βX1 ) = β[E(X12 ) − {E(X1 )}2 ] = βσ12 .

Hence, Corr(X1 , X2 ) = Cov(X1 , X2 )/(σ1 σ2 ) = βσ1 /σ2 .

5.19 From Result 5.2.22, ρ0(1,··· ,k) = 1 if and only if Var(ε) = 0, where ε = X0 − W , where W is the b.l.u.p. of X0 based on x(1) . Then ρ0(1,··· ,k) = 1 if and only if X0 = W and the proof follows from Exercise 5.13.

5.20 The product of the m.g.f.’s of the two random variables on the right side gives the m.g.f. of the random variable on the left side.

Ru 5.21 P(U ≤ u) = 0 f (s)ds, where f (u) is given in (5.3.3). Interchange the infinite P∞ sum and the integral, and successively integrate by parts to obtain P(U ≤ u) = j=0 P(X2 = j) P(X1 ≥ k/2 + j), and the right side is an expression for P(X1 − X2 ≥ k/2).

5.22 T =

1 (Z 2 − 2ρZ1 Z2 + Z22 ) (1 − ρ2 ) 1

where Zi = (Xi − µi )/σi . Clearly, (Z1 , Z2 )0 has a N2 (0, 0, 1, 1, ρ) distribution.

0 We the quadratic can write form T as z Az, where z = (Z1 , Z2 ), and A = 1/(1 − ρ2 ) −ρ/(1 − ρ2 ) . Since AΣ = I, which is idempotent, T has a chi-square −ρ/(1 − ρ2 ) 1/(1 − ρ2 ) distribution with 2 d.f.

Pk 2 2 2 5.23 (a) By Result 5.4.5, x0 D−1 x = i=1 Xi /σi has a χ (k, λ) distribution with λ = P k µ0 D−1 µ = i=1 µ2i /2σi2 . By Result 5.3.4, E(U ) = k + 2λ and Var(U ) = 2(k + 4λ). (b) Let Σ = Γ0 Γ, with r(Γ) = k, and let z = Γ0−1 x. Then, z ∼ Nk (Γ0−1 µ, I), and U = z0 z. By Result 5.4.2, U ∼ χ2 (k, λ) with λ = µ0 Σ−1 µ. (c) We can show that x0 Ax ∼ χ2 (k − 1, µ0 aµ/2).

5.24 From Result 5.4.4, it suffices to show that property 1 =⇒ 2 =⇒ 3 =⇒ 1. Since Σ is nonsingular, all these implications can be checked directly.

5.25 The results follow from the required integrations.

5.26 (a) Use Result 5.4.3 with r = 1, so E(x0 Ax) = κ1 (x0 Ax). (b) Var(x0 Ax) = κ2 (x0 Ax). (c) Since odd moments of x − µ are zero, Cov(x, x0 Ax) = E[(x − µ)(x0 Ax − µ0 Aµ − tr(AΣ))] = E[(x − µ)(x − µ)0 A(x − µ)] + 2 E[(x − µ)(x − µ)0 Aµ] − E[(x − µ) tr(AΣ)] = 2ΣAµ 5.27 Use Result 5.4.4 with Σ = I.

Solutions to Chapter 5

31

5.28 From the hint, we can write x0 Ax = z0 (Γ0 AΓ)z. Then, from Result 5.4.4, x0 Ax has a chi-square distribution if and only if Γ0 AΓ is idempotent, in which case the d.f. of the chi-square distribution is p = r(Γ0 AΓ). Now from ΓΓ0 = Σ and Result 1.3.13, (Γ0 AΓ)2 = Γ0 AΣAΓ = Γ0 AΓ if and only if ΣAΣAΣ = ΣAΣ. Clearly p ≥ r(ΣAΣ).

On the other hand, from (Γ0 AΓ)3 = Γ0 AΣAΣAΓ = Γ0 AΓ, r(ΣAΣ) ≥ p, so p = r(ΣAΣ).

5.29 (a) Since Cov(x, Bx) = B0 and Cov(Bx, Bx) = BB0 , from Result 5.2.14, the b.l.u.p. b = B0 BB0 B0 x = Px. Then y = (I − P)x ∼ Nk (0, I − P). of x based on Bx is x (b) Direct expansion. b = Px and the latter is a function of Bx, from (b), for (c) Since y is independent of x any fixed x0 , the conditional distribution of x0 Ax + a0 x given Bx = Bx0 is the same as U (x0 ) = y0 Ay + (2APx0 + a)0 y + (x00 PAPx0 + a0 Px0 ). If x0 Ax + a0 x and Bx are independent, then the distribution of U (x0 ) is always the same as the unconditional distribution of x0 Ax + a0 x. (d) If E[U (x0 )] = E(y0 Ay) + x00 PAPx0 + a0 Px0 is invariant to x0 , then PAP = O and a0 P = 0. The latter yields a0 PB0 = a0 B0 = 0, i.e., Ba = 0. (e) Since Cov(y) = I − P, from Result 5.4.2, Var[U (x0 )] = Var[y0 Ay + (2APx0 + a)0 y] = 2[tr{[A(I − P)]2 } + 2(2APx0 + a)0 (I − P)(2APx0 + a)].

The invariance then implies PA(I − P)(AP) = O. Then from (d), PAAP = (AP)0 AP = O, so AP = O. (f) This part is identical to the reduction argument at the end of the proof of Result 5.4.7.

5.30 From Result 5.4.4, U ∼ χ2 (m, λ = µ0 A−1 µ/2).   1 −2 1 1 5.31 Writing Q = x0 Ax, with A = −2 4 −2, we see that Q ∼ χ21 .

6 1 −2 1 5.32 (a) Let B = Σ/σ 2 − λIk . By the elementary row transformations R1 − Rk , · · · , Rk−1 − Rk , followed by the column transformations C1 +Ck , · · · , Ck−1 +Ck , B is equivalent to L, which is lower triangular. Setting |B| = |L| = (1−λ−ρ)k−1 [1−λ+(k−1)ρ] = 0, and solving for λ, we see that the eigenvalues of σ 2 Σ are λ1 = σ 2 (1 − ρ) with multiplicity k − 1, and λ2 = σ 2 [1 + (k − 1)ρ] with multiplicity 1. (b) A22 = J2k /k 2 = kJk /k 2 = Jk /k = A2 . Since A1 = Ik − A2 , A1 is idempotent too.

A1 A2 = (Ik − Jk /k)(Jk /k) = O. It is easy to verify that λ1 A1 + λ2 A2 = Σ. (c) First, Q1 = x0 A1 x/λ1 and Q2 = x0 A2 x/λ2 are independent since, from (b), A1 ΣA2 = A1 [λ1 A1 + λ2 A2 ]A2 = 0. Since A1 Σ/λ1 = A1 , which is idempotent of rank r(A1 ) = tr(A1 ) = tr(Ik ) − [tr(Jk )]/k = k − 1, it follows from Result 5.4.5 that Q1 ∼ χ2k−1 , the noncentrality parameter being 0 since µ = 0. Similarly, since A2 is idempotent of rank 1, Q2 ∼ χ21 .

5.33 Verify that B is symmetric, and that BΣ is idempotent. Now r(BΣ) = tr(BΣ) = tr[I − (Σ−1 J)/(10 Σ−1 1)] = k − 1. Also, BΣB = B, so that r(B) = r(BΣB) ≤ r(BΣ) = k − 1.

But, r(B) ≥ r(BΣ) = k − 1, so that r(B) = k − 1. (a) By Result 5.2.5, y ∼ N (BΣµ, σ 2 B).

32

Solutions to Chapter 5 (b) y0 Σy/σ 2 = x0 B0 ΣBx/σ 2 = x0 Bx/σ 2 , which has a χ2 (k − 1, λ) with λ = µ0 ΣBΣµ)/2σ 2 . When µ = 0, y0 Σy ∼ σ 2 χ2k−1 , while when µ 6= 0, y0 Σy ∼ σ 2 χ2 (k − 1, λ).

5.34 (a) The proof is identical to the following proof for (b). (b) From (5.4.18), the necessary and sufficient conditions for the independence of x0 Ax + a0 x and x0 Bx + b0 x are (Γ0 AΓ)(Γ0 BΓ) = Γ0 AΣBΓ = O, (Γ0 AΓ)((2Γ0 Aµ + Γ0 a) = Γ0 AΣ(2Aµ + a) = 0, (Γ0 BΓ)((2Γ0 Bµ + Γ0 b) = Γ0 BΣ(2Bµ + b) = 0, (2Γ0 Aµ + Γ0 a)0 (2Γ0 Bµ + Γ0 b) = 4(Aµ + a/2)0 (Bµ + b/2)0 = 0 From Γ0 M = O if and only if ΓΓ0 M = ΣM = O for any matrix M, the above equalities are equivalent to those in (5.4.19).

5.35 We can verify that Q1 = x0 A1 x and Q2 = x0 A2 x, with

1 1 −1 1 , A2 = A1 = 1 1 2(1 − ρ) −1

1 .

1

Since A1 Σ is idempotent of rank 1 and noncentrality parameter 0, and since A1 ΣA2 = 0, the results follow.

5.36 The result follows directly from Result 5.4.7 and the positive definiteness of Σ.

5.37 The proof of the necessity part of Result 5.4.9 only uses the independence of x0 Aj x.

From Result 5.4.7, for i 6= j, A0i Aj = O, so C(Ai ) ⊥ C(Aj ). Since A1 + · · · + AL = I, PL then Rk = ⊕L j=1 C(Aj ), so j=1 r(Aj ) = k.

P PL 5.38 (a) k ≤ r(I − Aj ) + r(Aj ) = r( i6=j Ai ) + r(Aj ) ≤ i=1 r(Ai ) = k. (b) This follows from the spectral decomposition of a symmetric matrix. (c) From (a) and (b), (k − m1 ) + (k − m0 ) = k, so m0 + m1 = k. Therefore, Aj is idempotent.

PL P 2 3 (d) Since P Aj is idempotent, AP j = Aj = Aj ( i=1 Ai )Aj = Aj + i6=j Aj Ai Aj = Aj + i6=j Aj Ai Aj . Thus i6=j Aj Ai Aj = O. (e) Since Ai are n.n.d., from (d), Aj Ai Aj = O for i 6= j. Then Ai Aj = O. (f) Use Results 5.4.4 and 5.4.7.

5.39 Without loss of generality, let µ = 0. (a) The p.d.f. of x2 is −1 f (x2 ) = ck−q |V22 |−1/2 hk−q (x02 V22 x2 ),

and the conditional covariance is Z −1 −1 (x1 − V12 V22 x2 )(x1 − V12 V22 x2 )0 dFx1 |x2 (x1 ). −1 Suppose V11.2 = D0 D, and let y = (Y1 , · · · , Yq )0 = D−1 (x1 − V12 V22 x2 ). The covariance matrix is Z yy0 dFy,x2 (y, x2 )/f (x2 ) = C = {cij }, say.

Solutions to Chapter 5

33

We are given that C does not depend on x2 . Now, Z ∞ Y12 dFy,x2 (y, x2 )/f (x2 ), c11 = −∞

and we can write −1 c11 ck−q |V22 |−1/2 hk−q (x02 V22 x2 ) Z ∞ −1 Y12 hk−q+1 (Y12 + x02 V22 x2 )dy1 . = ck−q+1 |V22 |−1/2 −∞

−1 Set Z = x02 V22 x2 , and b = 2ck−q+1 /ck−q .

(b) Let V = diag(v11 , · · · , vkk ). Independence of the components of x implies that the Qk √ m.g.f. of x can be written as ψ(t0 Vt) = i=1 ψ(t2i vii ). Let ui = ti vii . Then, Pk Q k ψ( i=1 u2i ) = i=1 ψ(u2i ). This equation (called Hamel’s equation) has solution ψ(z) = exp(cz) for some constant c. Therefore, ψ(t0 Vt) is the m.g.f. of a normal random vector.

5.40 The m.g.f. of By is MBy (t) = My (B0 t) = ψ(t0 BVB0 t) exp{t0 Bµ}.

The result follows.

Solutions to Chapter 6

6.1 Let D = X − 1N x0 . Then D is an N × k matrix and SN = D0 D. Then r(SN ) = r(D), so SN has rank less than k if and only if the k columns of D are not LIN. Pm 0 0 6.2 W ∼ j=1 xj xj , where x1 , · · · , xm are i.i.d. ∼ Nk (0, Σ). Then B WB ∼ Pm 0 0 0 0 0 0 0 j=1 B xj (B xj ) . Since B x1 , · · · , B xm are i.i.d. ∼ Nq (0, B ΣB), then B WB ∼ 0 Wq (B ΣB, m).

PN 6.3 (a) E(W) = i=1 E(xi x0i ) = N (Σ + µµ0 ).

Pm (b) Suppose V = i=1 yi yi0 , where y1 ,P · · · , ym are i.i.d.

Pm ∼ N (ν, Σ). If ν = −µ, then m (y1 , · · · , ym ) ∼ −(x1 , · · · , xm ), so i=1 yi yi0 ∼ i=1 xi x0i , yielding V ∼ W. On the other hand, suppose V ∼ W. Then E(V) = E(W). From (a), νν 0 = µµ0 .

If either ν or µ is null, then the other one is null, and so ν = −µ. Otherwise, ν(ν 0 µ) = µ(µ0 µ), giving µ = cν for some scalar c. Then νν 0 = µµ0 = c2 νν 0 , giving c2 = 1, i.e., c = ±1. Then ν = ±µ.

b M L = N −1 SN , it follows that |Σ b M L |N/2 = N −1 kN/2 |SN |N/2 ; substituting 6.4 Since Σ N N b M L ) can be written as this into the expression (6.1.9), we see that L(b µM L , Σ b M L) = L(b µM L , Σ

(2π)N k/2

exp(− N2k )

N −1 kN/2 N

. |SN |N/2

6.5 The proof that xm+1 = (m + 1)−1 (mxm + xm+1 ) is immediate from the definition of the sample mean, and some algebra. To show the second identity, we write Sm+1 =

m+1 X

(xi − xm+1 )(xi − xm+1 )0

i=1

=

m X

(xi − xm + xm − xm+1 )(xi − xm + xm − xm+1 )0

i=1

+ (xm+1 − xm+1 )(xm+1 − xm+1 )0 = Sm + m(xm − xm+1 )(xm − xm+1 )0 + (xm+1 − xm+1 )(xm+1 − xm+1 )0 .

Using the first identity for xm+1 and simplifying, we get Sm+1 = Sm + m(m + 1)−1 (xm+1 − xm+1 )(xm+1 − xm+1 )0 , which is the required identity.

6.6 Let b = Σ1/2 a, and B = Σ−1/2 SN Σ−1/2 ; then 0

0

a SN a b Bb b0 b max = max = max B . a b b a0 Σa b0 b (b0 b)1/2 (b0 b)1/2 35

36

Solutions to Chapter 6 Let c = b/(b0 b)1/2 , B = PΛP0 be the spectral decomposition of the symmetric matrix B, and d = (d1 , · · · , dk )0 = P0 c. Let λ1 , · · · , λk be the eigenvalues of B, i.e., of Λ. Note that kck = 1, kP0 ck = 1 and max{c0 Bc} = max{c0 PΛP0 c} = max{d0 Dd} = max c

c

d

d

k X

d2i λi = max λi .

i=1

1≤i≤k

Now, |SN − cΣ| = 0 implies that |Σ−1/2 SN Σ−1/2 − cIk | = 0, i.e., that |B − cIk | = 0.

Hence, the solution to |SN − cΣ| = 0 gives the eigenvalues λi of B. We have shown that 0 Na is the largest root of |SN − cΣ| = 0. Replacing max by min, the other result maxa aa0SΣa follows.

6.7 We have SNi ∼ Wk (Σ, Ni − 1), i = 1, 2. In Exercise 6.2, setting B = a, we see that a0 SNi a ∼ a0 Σaχ2Ni −1 . Also, a0 SN1 a and a0 SN2 a are independent since SN1 is independent of SN2 . Hence a0 SN1 a/a0 SN2 a ∼ FN1 −1,N2 −1 .

6.8 The joint bivariate normal pdf of (Xij , Xil ), i = 1, · · · , N can be written as (see Example 5.2.6) N Y

1 1 exp − (xij − µj )2 /σj2 2 2πσj i=1

1 1 2 ×√ exp − [xil − µl − β(xij − µj )]2 /σl.j 2 2πσl.j √

PN ∗ 2 ∗ where σl.j = σl2 (1 − ρ2 ) and β = ρσl /σj . Let Sjj = i=1 (Xij − X j ) , Sll = PN P N 2 ∗ (X − X j )(Xil − X l ). Consider the transformai=1 (Xil − X l ) , and Sjl = p ∗i=1 ij ∗ ∗ 2 ∗ 2 ∗ tions U1 = {Sjl − βSjj }/σl.j Sjj , U2 = {Sll∗ − Sjl /Sjj }/σl.j , and U3 = Sjj /σj2 , with −2 −3 ∗ −1/2 Jacobian σj σl.j (Sjj ) . The joint p.d.f. of U1 , U2 and U3 is

1 2 (N −2)/2−1 (N −1)/2−1 u3 . const. × exp − (u1 + u2 + u3 ) u2 2

1 ∗ 2 ∗ ∗ 2 Since U12 + U2 + U3 = (1−ρ 2 ) {Sjj /σj − 2ρSjl /σj σl + Sll /σl }, we can write the joint ∗ ∗ ∗ density of Sjj , Sll , and Sjl as −(N −1) −(N −1) σl (1 − ρ2 )−(N −1)/2 ∗ ∗ (N −4)/2 (s s − s∗2 jl ) π2N −1 Γ(N − 2)/2Γ(N − 1)/2 jj ll

σj √

× exp −

1 ∗ 2 ∗ ∗ 2 {s /σ − 2ρs /σ σ + s /σ . j j jl ll l 2(1 − ρ2 ) jj j p ∗ Let Sjj = Sjj /(N − 1), Sll = Sll∗ /(N − 1), and Sjl = (N − 1)r Sjj Sll , with Jacobian 4(N − 1)3 Sjj Sll . The joint pdf of Sjj , Sll and r is then f (sjj , sll , r) =

(N − 1)N −1 (N −2)/2 (N −2)/2 s sll (σj σl )N −1 (1 − ρ2 )(N −1)/2 πΓ(N − 2) jj × (1 − r2 )(N −4)/2 exp{−(N − 1)/2(1 − ρ2 )}

× [sjj /σj2 − 2ρr{sjj sll }1/2 /σj σl + sll /σl2 ] √ which follows by using Γ(2p) = 22p−1 Γ(p)Γ(p + 1/2)/ π, and simplifying.

Solutions to Chapter 6

37

6.9 The proof of Result 6.2.3 shows that conditional on almost every value of Z, the distribution is S11.2 = U0 (A − PC(AZ) )U is the same Wq (Σ11.2 , N − 1 − k + q). Therefore, S11.2 is independent of Z, hence independent of S22 = Z0 AZ. Meanwhile, S11.2 is a the transpose of (A−PC(AZ) )U multiplied by itself, and S21 = Z0 A(ZB0 +U) only depends on Z0 AU. Let the columns of U be u(1) , · · · , u(q) . Then for any two i, j, Cov((A − PC(AZ) )u(i) , Z0 Au(j) ) = (A − PC(AZ) ) Cov(u(i) , u(j) )AZ = σ̃ij (A − PC(AZ) )AZ = O. This implies the conditional independence of S11.2 and S21 given Z. But the conditional distribution of S11.2 is the same as its unconditional one. Then the (unconditional) independence follows.

6.10 (a) ρb0(1,2,3) = 0.6378. (b) ρb0,3|1,2 = 0.6356.

6.11 ρb20(1,··· ,k) = 1 − |S|/{S00 |S(1) |}, which follows from (6.2.9) and (6.2.10).

6.12 Beta(k/2, (N − k − 1)/2) distribution, which follows from (6.2.11) and (B.13).

6.13 If xj ∼ Nk (µ, Σ), j = 1, · · · , N , then Dj2 has an approximate χ2k distribution, so that E(d2j ) ≈ k. The exact result is seen by using the fact that N d2j /(N − 1)2 has 2

−1) k a Beta(k/2, (N − k − 1)/2) distribution so that E(d2j ) = (N N (N −1) = (N − 1)k/N .

6.14 (a) Since Z(1) + · · · + Z(N ) = Z1 + · · · + ZN = N Z N , the first equality easily follows.

Next, from Cov(Zi − Z N , Z N ) = Cov(Zi , Z N ) − Var(Z N ) = 0, z − Z N 1N = (Z1 − Z N , · · · ZN − Z N )0 is independent from Z N , so that the order statistics of z − Z N 1N are also independent of Z N . However, the ith order statistic of z − Z N 1N is Z(i) − Z N .

As a result, Cov(Z(i) − Z N , N Z N ) = 0, yielding Cov(Z(i) , Z N ) = Cov(Z N , N Z N ) = 1. (b) follows directly from the definition of SW .

6.15 Since yi = Pxi + q, xi = P−1 (yi − q). Verify that yN = PxN + q, and that SN,y = PSN P0 . Using the form of rij defined below (6.3.2) on the yi ’s, we see that rij,y = (N − 1)(yi − yN )0 S−1 N,y (yj − yN ) −1 = (N − 1)(xi − xN )0 P0 (P0 )−1 S−1 P(xj − xN ) = rij ; N P

i.e., rij is invariant under the nonsingular linear transformation. So are b1,k and b2,k defined by (6.3.2) and (6.3.3). The coefficients do not depend on µ and Σ.

Solutions to Chapter 7

7.1 Property 1 of Corollary 7.1.1 is a direct consequence of Result 5.2.5, while property 2 follows from Result 5.2.4. Property 3 follows from the orthogonality of X and I − P. b = σ 2 (X0 X)−1 .

7.2 For β = (β0 , β1 , β2 )0 , Var(β) (a) The statistic t = βb0 /d s. e.(βb0 ) has a Student-t distribution with N − 3 d.f. under H0 , where sd . e.(βb0 ) denotes an estimate of the standard error of βb0 . (b) The statistic t = (βb1 − βb2 )/d s. e.(βb1 − βb2 ) has a Student-t distribution with N −3 d.f. under H0 . The sd . e. term on the denominator is obtained from (4.4.2), substituting σ b2 for σ 2 . (c) H0 : X1∗ = 0 implies H0 : β1 = 0. Clearly, βb0,H = βb0 , βb2,H = βb2 , while βb1,H = 0.

P The test statistic F0 = (N − 3)Q/SSE ∼ F1,9 under H0 where Q = βb1 Xi (2βb0 + P βb1 Xi + 2βb2 Xi2 ), and SSE = (Yi − βb0 − βb1 Xi − βb2 Xi2 ).   0 0 0 1 0 0 7.3 We can write H as C0 β = d with C0 = 0 0 0 0 1 0 and d = 0. Then, 0 0 0 0 0 1 F (H) = (N − 6)(SSEH − SSE)/3SSE ∼ F3,N −6 under H, where SSE = y0 (IN − P)y, SSEH = y0 (IN − PH )y, and PH = P − P1 = X(X0 X)−1 X0 − X(X0 X)−1 C[CX0 X)−1 C]−1 C0 (X0 X)−1 X0 . If we replace y by y−Y 1N , then, SSEnew = (y − Y 1N )0 (IN − P)(y − Y 1N ) = SSE, since (IN − P)1N = 0. A similar reasoning gives SSEnew,H = SSEH , so that F (H) is unaltered.

7.4 Let µ + τ1 = µ + τ2 = µ + τ3 = β, say. Under H, the model becomes Yij = β + εij , P3 P3 Pni 0 and βH = ( i=1 ni Y i· )/N , where N = n1 + n2 + n3 , and SSEH = i=1 j=1 Yij2 − P3 P P P 3 n 3 i 2 2 2 i=1 (ni Y i· ) /N . For this model, SSE = i=1 j=1 Yij − i=1 Yi· /ni , and F (H) = (N − 3)(SSEH − SSE)/(2SSE) ∼ F2,N −3 under H. 7.5 Least squares estimates are θbi = Yi· −Y0· , i = 1, · · · , m. The required distribution follows from Result 4.4.1 and Result 7.1.2.

7.6 The two lines intersect at (x0 , y0 ) if and only if they intersect at a point (0, y0∗ ). This is ∗ ∗ ∗ ∗ seen by writing a shifted model Yl,i = βl,0 + βl,1 Xl,i + εl,i , where βl,0 = βl,0 + βl,1 x0 , ∗ ∗ ∗ and Xl,i = (Xl,i − x0 ). To test β1,0 = β2,0 = β, the F -statistic is F (H) = (n1 + n2 − 4)(SSEH − SSE)/SSE ∼ F1,n1 +n2 −4

39

40

Solutions to Chapter 7 under H. Here, SSE is the full model sum of squares given in (7.2.10), and nl 2 X X

SSEH =

b 1, HX ∗ )2 , and (Yl,i − βbH − βl, l,i

l=1 i=1

P P l

βbH =

where Xl· =

P

i Xl,i , and βl,1,H =

b

P2

2 l=1 {Xl· i Yl,i Xl,i / i Xl,i } , P P2 2 } n1 + n2 − l=1 {Xl·2 / i Xl,i

i Yl,i −

P

P

l,i −βH )Xl,i i (YP 2 i Xl,i

b

P

, l = 1, 2.

7.7 The null distribution of R2 is Beta(k/2, (N − k − 1)/2), so E(R2 ) = k/(N − 1).

0 · · · 0 1 −c 0 0 · · · 0 7.8 The design matrix is X0 = , and β 0 = (λ1 , λ2 ).

0 · · · 0 0 1 −c 0 · · · 0 b1 = [(1 + c2 )Yk − c3 Yk+1 − c2 Yk+2 ]/(1 + c2 + c4 ), and λ b2 = [cYk + Yk+1 − (a) Then, λ 2 c(1 + c )Yk+2 ]/(1 + c2 + c4 ), with

σ2 1 + c2 c b = Var(β) . c 1 + c2 (1 + c2 + c4 ) (b) Under the restriction λ1 = −λ2 = λ, say, using results from section 4.6.1, we get b = [Yk − (1 + c)Yk+1 + cYk+2 ]/2(1 + c + c2 ), with Var(λ) b = σ 2 /2(1 + c + c2 ). λ (c) Q/b σ 2 ∼ χ21 , where Q=

(1 − c + c2 ) 2

(1 + c + c2 )Yk + (1 − c3 )Yk+1 − c(1 + c + c2 )Yk+2 (1 + c2 + c4 )

2 .

7.9 The restricted least squares estimate of β subject to the model constraint 10 β = 0 is P4 2 βbr = y − Y 14 , where Y = i=1 Yi /4. Also, SSEr = 4Y . The reduced and restricted 1 111 model consists of imposing H: β1 = β2 , i.e., β = 0. We can show that (see 1 −1 0 0 section 8.1) βbr,H = y − z, where z0 = (Y + 12 (Y1 − Y2 ), Y − 12 (Y1 − Y2 ), Y , Y ). Also, SSEr,H = 4(Y )2 + (Y1 − Y2 )2 /2. Then, 2(Y1 − Y2 )2 F0 = P4 . ( i=1 Yi )2 7.10 Under the full model (7.3.6), E(SSE) = σ 2 (N −r). On the other hand, under H: β 1 = 0, SSEH = y0 (I − P2 )y. Let µ = X1 β 1 + X2 β 2 . Then (I − P2 )µe = (I − P2 )X1 β 1 , so from Result 4.2.3, E(SSEH ) = tr((I − P2 )σ 2 Ip ) + µ0 (I − P2 )µ = σ 2 (N − r2 ) + β 01 X01 (I − P2 )X1 β 1 .

Then E(SSEH ) − E(SSE) = σ 2 (r − r2 ) + β 01 X01 (I − P2 )X1 β 1 . b = y − X1 β 01 − X2 β 02 = (I − P)y.

7.11 M SE = {y0 (I − P)y}/(N − r). Let b ε = y−y 1 1 2 2 E(M SE) = N −r σ tr(I − P) = N −r σ (N − r), which proves property 2 of Result 7.3.7.

Solutions to Chapter 7

41

7.12 (a) Let P be the projection matrix onto C(X1 , X2 ). Then Cov(b y) = σ 2 P and 2 2 bH ) = σ (P − P1 ). From property 3 of ReCov(b yH ) = σ P1 , so Cov(b y−y sult 2.6.7, C(X1 , X2 ) = C(X1 ) ⊕ (I − P1 )C(X2 ) = C(X1 ) ⊕ C((I − P1 )X2 ).

Since C(X1 ) ⊥ C((I − P1 )X2 ), then P = P1 + PC((I−P1 )X2 ) , i.e., P − P1 = (I − P1 )X2 )[X02 (I − P1 )X2 ]− X02 (I − P1 ).

2 (b) We have (N − r1 )b σH /σ 2 ∼ χ2N −r1 and (N − r)b σ 2 /σ 2 ∼ χ2N −r , where r1 = r(X1 ) 2 4 and r = r(X1 , X2 ). Then Var(b σH ) = σ /(N − r1 ) and Var(b σ 2 ) = σ 4 /(N − r), so 2 2 Var(b σ )/ Var(b σH ) = (N − r)/(N − r1 ) ≥ 1.

bH − µ = P1 y − µ = P1 ε − (I − P1 )µ and y b − µ = Py − µ = Pε, where 7.13 We have y P1 is the projection onto C(X1 ), and P the one onto C(X1 , X2 ). Then E(kb yH − µk2 ) = 2 2 0 2 E(kP1 εk ) + k(I − P1 )µk = E(ε P1 ε) + k(I − P1 )X2 β 2 k . By Result 4.2.3, E(ε0 P1 ε) = σ 2 r1 . Then E(kb yH − µk2 ) = r1 σ 2 + k(I − P1 )X2 β 2 k2 . Likewise, E(kb y − µk2 )(r1 + r2 )σ 2 .

2 2 2 2 Then E(kb yH − µk ) − E(kb y − µk ) = k(I − P1 )X2 β 2 k − r2 σ , so E(kb yH − µk2 ) < 2 2 2 E(kb y − µk ) if and only if k(I − P1 )X2 β 2 k < r2 σ .

7.14 H0 : X ∗ = 0 implies H0 : β1 = 0. Clearly, βb0,H = βb0 , βb2,H = βb2 , while βb1,H = 0. The test P statistic F0 = (N −3)Q/SSE ∼ F1,9 under H0 where Q = βb1 Xi (2βb0 + βb1 Xi +2βb2 Xi2 ), P and SSE = (Yi − βb0 − βb1 Xi − βb2 Xi2 ).

7.15 We describe the analysis for the balanced incomplete block design (BIBD). There are a levels of treatment, each replicated r times. There are b blocks, and the number of treatments per block is k. The total number of obsevations is N = ar = bk. The design is symmetric if a = b. We let λ = r(k − 1)/(a − 1) denote the number of times each pair of treatments appears in the same block. To test H0 : τ1 = · · · = τa , the ANOVA decomposition is SSTc = SST radj. + SSblock + SSE, where SSTc =

b a X X

2

(Yij − Y ·· ) =

a X b X

Yij2 − Y··2 /N, with N − 1d.f.

i=1 j=1

i=1 j=1

2 a b 1X k X Yi· − δij Y·j  with a − 1 d.f.

SST radj. = λa i=1 k j=1 

b

SSblock =

1X 2 1 Y·j − Y··2 with b − 1 d.f. k j=1 N

SSE = SSTm − SST radj. − SSblock with N − a − b + 1d.f. where δij = 1 is treatment i appears in block j and 0 otherwise. The test statistic is F = M ST radj. /M SE ∼ Fa−1,N −a−b−1 under H0 .

7.16 The estimates of β0 and β1 are not changed when β2 is set  is a consequence  to 0. This 1 −1 1 of the orthogonality of the columns of the matrix X = 1 0 −2, so that X0 X = 1 1 1 b diag(3, 2, 6). The least squares estimates of β0 and β1 are β0 = (Y1 + Y2 + Y3 )/3, and βb1 = (Y3 − Y1 )/2.

42

Solutions to Chapter 7

7.17 In general, suppose H tests equality of (µ+τi )/di , i = 1, · · · , a, with common value equal to γ, say, and di ’s are known constants. Under H, Yij =P di β + εij . The P design matrix 0 is X = (d1 1n1 , · · ·P , da 1na )0 . We canP verify that γH =P( Pi ni di Yi· )/( i ni d2i ). Hence, F (H) = (N − a)[ i Yi·2 /ni − (γ 0 )2 i ni d2i ]/[(a − 1) i j (Yij − Y i· )2 ] ∼ Fa−1,N −a under H. 7.18 In the model Yij = µi + εij , r(X) = a. The hypothesis H can be writen as C0 β = d, where C0 is an (a − 1) × a matrix   1 −2 0 0 ··· 0 0 0 2 −3 0 ··· 0 0   0 0 3 −4 · · · 0 0 ,    ..   .

0

0

0

0

···

a−1

−a

β = (µ1 , · · · , µa )0 , and d is an (a−1)-dimensional vector of zeroes. The test statistic has the form (7.4.5) and is distributed under H as an Fa−1,a(n−1) distribution. Alternately, we can write H as µ + τ1 = 2(µ + τ2 ) = · · · = a(µ + τa ) = δ, say. That is, i(µ + τi ) = δ, i =P 1, . . . , a =⇒ (µ + τi ) = δ/i, i = 1, . . . , a. Under H2 , the one-way ANOVA model a s.t. i=1 τi = 0 reduces to Yij =

1 δ + ij , i = 1, . . . , a, j = 1, . . . , n. i

We can show that the LS estimate is Pa Yi· /i δ = Pai=1 2 i=1 (1/i ) 0

Then we can find SSEH =

a X n X

(Yij − δ 0 /i)2

i=1 j=1

We know SSE for the one-way ANOVA model is SSE =

a X n X (Yij − Y i· )2 i=1 j=1

The F -statistic is obtained from (7.2.9).

7.19 The hypothesis can be written as C0 β = d, with C0 = (0, −1/2, 1, −1/2), β = (µ, τ1 , τ2 , τ3 )0 , and d = 0. From (7.4.5), F (H) = (3n − 3)Q/SSE ∼ F1,3n−3 under P3 Pn 1 2 2 H, where Q = 2n j=1 (Yij − Y i· ) . i=1 3 [Y 2· − 2 (Y 1· + Y 1· )] , and SSE = 7.20 Consider the nested sequence of subspaces SH0 ⊃ SH1 ⊃ SH2 ⊃ SH3 ⊃ SH4 ⊃ SH5 ⊃ SH6 = {0}. The least squares estimates are: µ0 = Y ··· , τi0 = Y i·· − Y ··· , θj0 = Y ·j· − Y ··· , γk0 = Y ··k − Y ··· , (τ θ)0ij = Y ij· − Y ··· , and (θη)0jk = Y ·jk − Y ··· . Hence, Q(H0 ) = P P P P P P 2 i j k (Yijk − Y ij· − i j k (Yij· − Y ·jk − Y ·j· ) with b(a + c − 1), SSE = P P 2 Y ·jk + Y ·j· ) with b(a − 1)(c − 1) d.f., Q(H0 | H1 ) = a j k (Y ·jk − Y ·j· − Y ··k − Y ··· )2 P with (b − 1)(c − 1) d.f., Q(H1 | H2 ) = ab k (Y ··k − Y ··· )2 with c − 1 d.f., P P PQ(H2 | H3 ) = c i j (Y ij· −Y i·· −Y ·j· −Y ··· )2 with (a−1)(b−1) d.f., Q(H3 | H4 ) = ac j (Y ·j· −Y ··· )2 P 2 with b − 1 d.f., Q(H4 | H5 ) = bc i (Y i·· − Y ··· )2 with a − 1 d.f., Q(H5 | H6 ) = abcY ··· with 1 d.f. The ANOVA table can be completed based on this information.

Solutions to Chapter 7

43

7.21 If all three factors have a levels each, and n = 1, the design corresponds to an a × a Latin Square Design (LSD): Yijk = µ + τi + θj + ηk + εijk , for i, j, k = 1, · · · , a. We assume that εijkl ∼ N (0, σ 2 ). This corresponds to a square with a rows (Factor A) and a columns (Factor B); each of the resulting a2 cells contains one of the a levels of Factor C occurring once and only once in each row and each column.

Note that in an LSD, N = a2 . The ANOVA decomposition is SSTc = SSA + SSB + SSC + SSE, where SSTc =

a X a X a X

(Yijk −

i=1 j=1 k=1

SSA = SSB =

SSC =

a X Y2

i··

i=1 a X

a

−

1 Y··· )2 with (a2 − 1) d.f.

N

Y···2 with (a − 1) d.f.

N

2 Y·j· Y2 − ··· with (a − 1) d.f. and a N j=1

a X Y2

··k

k=1

a

−

Y···2 with (a − 1) d.f.

N

To test the null hypothesis of no difference in effects due to Factor C, the test statistic is F = M SC /M SE which has an F(a−1),(a−1)(a−2) distribution under the null hypothesis.

PN PN 7.22 θbOLS = 2 i=1 Yi t2i / i=1 t4i , and follows a normal distribution with mean θ and variPN ance 4σ 2 / i=1 t4i . The 100(1 − α)% confidence interval for θ is 2b σ θbOLS ± qP

N 4 i=1 ti

tN −1,α/2 .

7.23 By Corollary 7.1.1, the least squares estimates of these parametric functions of β have normal distributions. Also, SSE/(N − p) ∼ χ2N −p . The resulting t-intervals have the p √ form in (7.4.3). The 95% C.I. for β1 is βb1 ± 4.73 11/4. For β2 , it is βb2 ± 4.73 3. For β3 , p p it is βb3 ±4.73. For β1 −β2 , it is βb1 − βb2 ±4.73 59/4. For β1 +β3 , it is βb1 + βb3 ±4.73 27/4.

7.24 Verify that P P l i (Xl,i − X l· )(Yl,i − Y l· ) , P P 2 l i (Xl,i − X l· ) βbl,0,H = Y l· − βbH X l· , l = 1, 2. βbH =

A point estimate of δ is then δb = (βb2,0,H − βb1,0,H )/βbH . Let U = (βb2,0,H − βb1,0,H )− δbβbH = b Then, U ∼ N (0, σ 2 ), with (Y 2· − Y 1· ) + βbH (X 1· − X 2· − δ). u " # b2 1 1 (X 1· − X 2· − δ) 2 2 σu = σ + +P P .

2 n1 n2 l i (Xl,i − X l· )

44

Solutions to Chapter 7 Then, SSEH =

P P l

i (Yl,i − Y l· )

T =

2

2 − βbH (Xl,i − X l· )2 , and

U/σu ∼ tn1 +n2 −3 [SSEH /σu2 (n1 + n2 − 3)]1/2

under H: δ = 0. We can obtain the limits of the C.I. by solving the quadratic equation obtained by setting T 2 ≤ tn1 +n2 −3,α/2 .

7.25 We use Scheffe’s projection approach to construct a simultaneous confidence set for c0 β for all c ∈ L in the GLS model. Let A = Σ−1/2 , so that AΣA0 = IN , and r(AX) = r(X) = r. Let z = Ay. Then, E(z) = AXβ, and Cov(z) = σ 2 IN , so that the linear model z = AXβ + ε1 , ε1 = Aε ∼ N (0, σ 2 IN ) corresponds to (4.1.1). The GLS 2 solution β 0GLS and error variance estimate σGLS are respectively given by (4.5.4) and property 3 of Result 4.5.1. The Scheffé simultaneous confidence set in this case has the form {β : |c0 β 0GLS − c0 β| ≤ σ bGLS (dFd,N −r,α )1/2 [c(X0 Σ−1 X)− c]1/2 ∀ c ∈ L}.

Solutions to Chapter 8

PN 8.1 L(θ, y) = (2π)−N/2 exp{− 12 t=1 (Yt − θYt−1 )2 }. The MLE of the θ is the minimizer of PN the function t=1 (Yt − θYt−1 )2 , and is PN θb =

t=1 Yt Yt−1 .

2 t=1 Yt

PN

8.2 (a) We can verify that the least squares solutions are µ0 = Y ··· , τi0 = Y i·· − Y ··· , Pa Pb Pn βj0 = Y ·j· − Y ··· , so that SSE = i=1 j=1 k=1 (Yijk − Y i·· − Y ·j· + Y ··· )2 = Pa Pb Pa Pb Pn n i=1 j=1 (Y ij· − Y i·· − Y ·j· + Y ··· )2 + i=1 j=1 k=1 (Yijk − Y ij· )2 with d.f. abn − (a + b − 1) = n∗ , say. We have σ b2 = SSE/n∗ . Let µij = µ + τi + βj , with 0 0 0 µ bij = µ + τi + βj . The joint p.d.f. of Yijk is     1 XXX 1 2 (Y − µ ) − exp ijk ij   2σ 2 (2πσ 2 )abn/2 i j k     1 X X 1 2 2 = (Y − µ b ) + n (b µ − µ ) , exp − ijk ij ij ij   2σ 2 (2πσ 2 )abn/2 i,j i,j,k because the product term vanishes by independence. From the properties of exponential families, it follows that σ b2 and µ bij are jointly complete, sufficient statistics 2 2 for σ and µij . Since E(b σ ) = σ2 , σ b2 is unbiased and a function of the complete, sufficient statistics, so it must be the MVUE of σ 2 . (b) Suppose X = (X1 , X2 , X3 ), where X1 corresponds to µ, X2 corresponds to τ1 , · · · , τa , and X3 corresponds to β1 , · · · , βb . We know that C(Xi ) ⊥ C(Xj ), i 6= j.

Pa Pb Now, Z1 = i=1 ci τi0 ∈ C(X2 ) and Z2 = j=1 ci βj0 ∈ C(X3 ), so that Z1 and Z2 are independent. Let c denote an N = 1 + a + b dimensional vector with c1 , · · · , ca in positions 2, · · · , a + 1, and zeroes elsewhere. Since Z1 = c0 (X0 X)− X0 y, and c0 (X0 X)− X0 [I − X(X0 X)− X0 ] = 0, it follows that Z1 and SSE are independent, i.e., Z1 and U are independent. A similar calculation shows that Z1 and U are independent.

8.3 Since N Y

1 Q 2 { N X X }1/2 2πσ i=1 i1 i2 i=1

1 × exp − 2 (Yi − β1 Xi1 − β2 Xi2 )2 2σ Xi1 Xi2 ( N ) N N X X X Yi2 Yi β1 Yi β2 = k1 exp − + + , 2σ 2 Xi1 Xi2 i=1 σ 2 Xi2 i=1 σ 2 Xi1 i=1

f (y; β1 , β2 , σ 2 ) =

√

45

46

Solutions to Chapter 8 PN PN PN Y2 i i it follows that { i=1 Xi1iXi2 , i=1 XYi1 , i=1 XYi2 } are complete-sufficient statistics for (β1 , β2 , σ 2 ). Let βe1 = Q1 /Q2 , where Q1 =

Q2 =

N X

PN −N Yi /Xi1 , and Yi /Xi2 + PN i=1 i=1 Xi2 /Xi1 i=1

N X i=1

Xi1 /Xi2 + PN

−N 2

i=1 Xi2 /Xi1

.

We can verify that E(Q1 ) =

(N X i=1

Xi1 /Xi2 + PN

)

−N 2

β1 ,

i=1 Xi2 /Xi1

so that E(βe1 ) = β1 . Similarly, let βe2 = Q3 /Q4 , where PN N X −N Yi /Xi2 , and Q3 = Yi /Xi1 + PN i=1 i=1 Xi1 /Xi2 i=1 Q4 =

N X i=1

Xi2 /Xi1 + PN

−N 2

i=1 Xi1 /Xi2

.

We can verify that E(Q3 ) =

(N X i=1

Xi2 /Xi1 + PN

−N 2

i=1 Xi1 /Xi2

) β2 ,

so that E(βe2 ) = β2 . βe1 and βe2 are respectively MVUE’s of β1 and β2 . It can be verified that (N )−1 2 X N , and Var(βe1 ) = σ 2 Xi1 /Xi2 − PN i=1 Xi2 /Xi1 i=1 (N )−1 X N2 2 e .

Var(β2 ) = σ Xi2 /Xi1 − PN i=1 Xi1 /Xi2 i=1 These may be compared with Var(βb1 ) and Var(βb2 ) respectively.

8.4 We can show that Λ=

a X

(b σi2 /b σ 2 )ni /2 ,

i=1

where σ bi2 denotes the M SE from the fitted regression in the ith group, and σ b2 is the

M SE from fitting the same model to all N observations. The statistic −2 log Λ has an approximate χ2a−1 distribution.

8.5 (a) From (8.2.6), PN

PN PN b2t + t=1 εb2t−1 bt εbt−1 t=1 ε t=2 ε − P DW = PN 2 N bt b2t t=1 ε t=1 ε PN 2 PN +1 2 εb + t=1 εbt−1 − (b ε21 + εb2N ) = t=1 t − 2rbε , PN 2 bt t=1 ε

Solutions to Chapter 8

47

Assuming that the effects of εb21 and εb2N are small compared to the overall sum, we get the result.

PN −p PN −p (b) We require moments of DW = U/V , where U = i=1 νi ξi2 and V = i=1 ξi2 (see the proof of Result 8.2.6). Since DW and V are independently distributed, it follows that E(DW s ) = E(U s )/ E(V s ), i.e., moments of DW are ratios of moments.

Since ξi2 has a χ21 distribution, the sth cumulant of U is κs (U ) = 2s−1 (s − 1)!

N −p X

νis .

i−1

PN −p PN −p In particular, κ1 (U ) = i−1 νi and κ2 (U ) = 2 i−1 νi2 . The moments of V are those of a χ2N −p variable, so that E(V ) = N − p, and E(V 2 ) = (N − p)(N − p + 2).

Therefore, N −p X 1 νi = ν, and E(DW ) = N − p i=1 "N −p # X 2 Var(DW ) = ν 2 − (N − p)ν 2 .

(N − p)(N − p + 2) i=1 i

Since N −p X

νi = tr{(I − P)A} = tr(A) − tr[X0 AX(X0 X)−1 ], and

i=1 N −p X

νi2 = tr{(I − P)A}2

i=1

= tr(A2 ) − 2 tr{X0 A2 X(X0 X)−1 } + tr{X0 AX(X0 X)−1 }2 , the result follows.

2 8.6 SSE(i) = SSE − εb2i /(1 − pii ) = SSE − ri∗2 σ b(i) (see Result 8.3.5), from which the result follows after simplification.

8.7 By Example 2.1.3, PZ = PX +

b (I − PX )yy0 (I − PX ) εb ε0 = P + X y0 (I − PX )y b ε0 b ε

from which the result follows. b and T (Fb(i) ) = β b ), 8.8 In (8.3.14), let (x0 , Y ) = (x0i , Yi ), F = Fb (so that T (Fb) = β, (i) ε = −1/(N − 1), and omit the limit. This gives b −β b }.

SICi = −(N − 1){β (i) Since b = [X0 X(i) ]−1 X0 Y(i) , and β (i) (i) (i) b = (X0 X)−1 X0 Y(i) + (X0 X)−1 xi Yi , β (i)

48

Solutions to Chapter 8 we see that b − (X0 X)−1 xi Yi .

(X0 X)−1 X0(i) Y(i) = β b and simplifying, we see that Substituting this into the form of β (i) b −β b = (X0 X)−1 xi β (i)

εbi , 1 − pii

from which the form of SICi in (8.3.18) follows. e = σ 2 X1 (X0 X1 )− X0 = σ 2 P1 and Cov(X1 β 0 ) = σ 2 X1 (X0 X)− X0 := 8.9 Since Cov(X1 β) 1 1 1 2 σ V, we only need to show P1 ≥ V, i.e., P1 − V is n.n.d., and equality holds if R(X1 ) ∩ R(X2 ) = {0}. For any vector v, v0 P1 v = kP1 vk2 , while v0 Vv = v0 X1 (X0 X)− X01 v = (P1 v)0 X1 (X0 X)− X01 (P1 v)

0 0 − 0 P1 v = (P1 v , 0)X(X X) X = kPzk2 0 where P is the projection onto C(X) and z = ((P1 v)0 , 0)0 . Clearly, kPzk2 ≤ kzk2 = kP1 vk2 . Then v0 P1 v ≥ v0 Vv, so P1 ≥ V. Furthermore, if R(X1 ) ∩ R(X2 ) = {0}, then there is T, such that X1 T = X1 and X2 T = O; see the proof of Exercise 4.14. Then since P1 v ∈ C(X1 ), it can be written as X1 Tt, and so z = XTt, giving Pz = z. Then it follows that v0 Vv = kzk2 = kP1 vk2 , so v0 P1 v = v0 Vv, yielding P1 = V. 8.10 For the simple linear regression model, Wi∗ = PN

βb12 (Xi − X)2

2 j=1 (Yj − Y ) /(N − 1)

(Xi − X)2 = (N − 1)ρ2X,Y PN .

2 j=1 (Xj − X) PN Since pii = 1/N + (Xi − X)2 / j=1 (Xj − X)2 in a simple regression model, the result follows.

8.11 (a) From Exercise 8.7, we see that PZ = PX +

b εb ε0 . b ε0 b ε ε b2

Let pZii be the ith diagonal element of PZ ; pZii ≤ 1 for all i, implying that pii + bε0ibε , and the result follows. (b) These follow directly from the relations in properties 2 and 3 of Result 8.3.4.

8.12 Ci =

1 b b 0 0 b b (β − β (i) ) X X(β − β (i) ). pb σ2

Since b −β b = (X0 X)−1 xi β (i)

εbi , 1 − pii

it follows that b −β b )0 X0 X(β b −β b ) = x0 (X0 X)−1 xi (β (i) (i) i

εb2i .

(1 − pii )2

Solutions to Chapter 8

49

8.13 Use the matrix result that if A is a k × k nonsingular matrix, while B and C are k × l matrices, then, |A − BC0 | = |A||I − C0 A−1 B| (see Exercise 2.8). Set A = X0 X, and B = C = xi , so that |X0(i) X(i) | = |X0 X − xi x0i | = |X0 X||1 − x0i (X0 X)−1 xi | = |X0 X|(1 − pii ).

2 Since σ(i) = σ 2 (N − p − ri2 )/(N − p − 1), the result follows.

8.14 We know that for i = 1, · · · , N , 0

APi = 1 −

0

|Z(i) Z(i) | SSE(i) |X(i) X(i) | =1− 0 SSE |X X| |Z0 Z|

where Z = (X, y), and Z(i) = (X(i) , y(i) ). Since |Z0(i) Z(i) | = |Z0 Z − zi z0i | = |Z0 Z|(1 − z0i (Z0 Z)−1 zi ) = |Z0 Z|(1 − pZii ), where pZii is the ith diagonal element of PZ (see Exercise 8.7), we see that APi = pZii , and the first result follows directly from Exercise 8.7. The second result follows from (8.3.8).

8.15 By definition of influence curve, ICT,F (x) = R

ψ(x − T (F )) . ψ 0 (x − T (F )) dF

For the median, ψ(z) = sign(z), and d sign(z) = 2δ(z = 0), dz where δ(z) = 1 if z = 0 and is 0 otherwise. Therefore Z Z Z ψ 0 (x − T (F )) dF = 2δ(z = 0) dF = 2δ(z = 0)f (z) dz = 2f (T (F )).

It follows that ICT,F (x) =

sign[x − T (F )] .

2f [T (F )]

Solutions to Chapter 9

9.1 Verify that X0 X is equal to   N 0 0 PN P N 2 0 (Xi1 − X 1 )(Xi2 − X 2 ) i=1 (Xi1 − X 1 ) i=1P PN N 2 0 i=1 (Xi1 − X 1 )(Xi2 − X 2 ) i=1 (Xi1 − X 1 ) since

PN

i=1 (Xij − X j ) = 0, j = 1, 2. Also,

Cov(Xi1 , Xi2 )}2 Var(Xi1 ) Var(Xi2 ) PN { i=1 (Xi1 − X 1 )(Xi2 − X 2 )}2 = PN PN 2 2 i=1 (Xi1 − X 1 ) i=1 (Xi2 − X 2 )

2 r12 =

Simplifying the (2, 2)th element of σ 2 (X0 X)−1 , we see that Var(βb1 ) = σ 2

X N

2 ).

(Xi1 − X 1 )2 (1 − r12

i=1

The width of the 95% confidence interval for β1 is 2tN −3,.025 est. s. e.(βb1 ).

9.2 Let y = (Y1 , · · · , YN )0 , X = (1N , X1 ), where X1 is an N × k matrix, β = (β0 , β1 , · · · , βk )0 , and ε = (ε1 , · · · , εN )0 with ε ∼ N (0, σ 2 IN ). Now, H: βj = d is equivalent to H: C0 β = d where C0 is a (k + 1)-dimensional vector with 1 in the jth position and zero elsewhere. Following the method in section 7.3.2, we obtain

0 −1 0 b = Y − x B (X1 y − N Y x) , β B−1 (X01 y − N Y x) b − d = βbj − d, and Q = (βbj − d)2 /ajj , ajj being where B = X01 X1 − N xx0 , so that C0 β 0 −1 the jth diagonal element of (X X) . Also, s = 1, and SSE = y0 (I − X(X0 X)−1 X0 )y, which has a χ2N −k−1 distribution. Then, by Result 4.3.5, F (H) =

Q ∼ F1,N −k−1 SSE/(N − k − 1)

under H. 9.3 Since σ b2 = y0 (I − P)y, by Exercise 5.27, E(b σ 2 ) = σ 2 , and Var(b σ 2 ) = 2σ 4 /(N − p) = 2 0 M SE(b σ ). Similar calculations show that E(y A1 y) = (N − p)σ 2 /(N − p + 2), the bias being 2σ 2 /(N − p + 2). Also, Var(y0 A1 y) = 2(N − p)σ 4 /(N − p + 2)2 . Then, M SE(y0 A1 y) = Var(y0 A1 y) + (Bias)2 = 2σ 4 /(N − p + 2) < M SE(b σ 2 ). b ∼ 9.4 Y0 = x00 β + ε0 , where ε0 ∼ N (0, σ 2 ). By the Gauss–Markov Theorem, Yb0 = x00 β N (x00 β, σ 2 x00 (X0 X)−1 x0 ).

51

52

Solutions to Chapter 9 (a) E(Y0 ) = E(x00 β + ε0 ) = x00 β, and since ε0 is independent of ε, Var(Y0 − Yb0 ) = Var(Y0 ) + Var(Yb0 ) = σ 2 + σ 2 x00 (X0 X)−1 x0 . Replacing σ 2 by its least squares estimate gives the required result. We know that (N − p)b σ 2 /σ 2 ∼ χ2N −p . The distribution of the given statistic is therefore tN −p . (b) The 95% symmetric two-sided prediction interval for Y0 is b ± tN −p,0.025 σ {x00 β b[1 + x00 (X0 X)−1 x0 ]1/2 }. (c) The 100(1 − α)% lower bound for γ(η) is d − zα σ γ(η) b[1 + x00 (X0 X)−1 x0 ]1/2 , d = x0 βb + zη σ, assuming σ is known. where γ(η) 0

9.5 The least squares estimates of the parameters are obtained by minimizing P2 unrestricted Pnl 2 bl,0 = Y l· −βbl,1 X l· and βbl,1 = [Pnl (Yl,i −Y l· )(Xl,i − (Y −β −β X ) as β l,i l,0 l,1 l,i i=1 l=1 Pi=1 nl X l· )]/ i=1 (Xl,i − X l· )2 , l = 1, 2. Also, the unrestricted error sum of squares is SSE = P2 Pnl 2 b2 Pnl (Xl,i − X l· )2 ]. l=1 [ i=1 (Yl,i − Y l· ) − βl,1 i=1 (a) For a test of parallelism, the restricted least squares estimates and SSEH under P2 Pnl H: β1,1 = β2,1 = β1 , say, are obtained by minimizing l=1 i=1 (Yl,i − βl,0 − β1 Xl,i )2 as βbl,0,H = Y l· − βb1,H X l· , l = 1, 2, βb1,H =

nl nl 2 X 2 X X X (Xl,i − X l· )2 .

(Yl,i − Y l· )(Xl,i − X l· )/ l=1 i=1

l=1 i=1

Also, SSEH =

nl 2 X X

2 (Yl,i − Y l· )2 − βb1,H

l=1 i=1

l=1 i=1

SSEH − SSE =

2 X

2 βb1,H

l=1

nl 2 X X (Xl,i − X l· )2 , and

nl X

nl 2 X X (Xl,i − X l· )2 .

i=1

l=1 i=1

2 (Xl,i − X l· )2 − βb1,H

The statistic F =

SSEH − SSE ∼ F1,N −4 SSE/(N − 4)

distribution under H. (b) We test the hypothesis that the two lines meet at a point on the y-axis, i.e., H : β1,0 = β2,0 = β0 , say. The restricted least squares estimates and SSEH under H are obtained by minimizing nl 2 X X

(Yl,i − β0 − βl,1 Xl,i )2

l=1 i=1

as ) ( ) Pnl 2 2 2 X X X Y X X 1· l,i l,i Pni=1 Pnl l· 2 , and βb0,H = Y·· − N− l 2 Xl,i i=1 i=1 Xl,i l=1 l=1 X nl nl X 2 βbl,1,H = (Yl,i − βb0,H )Xl,i Xl,i , l = 1, 2. (

i=1

i=1

Solutions to Chapter 9

53

Also, the restricted error sum of squares is SSEH =

nl 2 X X (Yl,i − βb0,H − βbl,1,H Xl,i )2 , l=1 i=1

and the test statistic F = (SSEH − SSE)/{SSE/(N − 4)} has an F1,N −4 distribution under H. (c) The restricted least squares estimates and SSEH under H: β1,0 = β2,0 = β0 ; β1,1 = β2,1 = β1 , say, are obtained by minimizing nl 2 X X

(Yl,i − β0 − β1 Xl,i )2

l=1 i=1

as βb0,H = Y ·· − βb1,H X ·· , and ( 2 n ) 2 n l l XX XX b β1,H = (Yl,i − Y ·· )(Xl,i − X ·· ) (Xl,i − X ·· )2 . l=1 i=1

l=1 i=1

The restricted error sum of squares is SSEH =

nl nl 2 X 2 X X X (Yl,i − Y ·· )2 − (βb1,H )2 (Xl,i − X ·· )2 . l=1 i=1

l=1 i=1

Then, F (H) = (N − 4)(SSEH − SSE)/(2SSE) ∼ F2,N −4 under H. Pnk Pnk Pnk (k) 2 9.6 (a) Let Y k = i=1 Yk,i /nk , X k = i=1 Xk,i /nk , SXX = i=1 (Xk,i − X k ) , and P (k) nk SXY = i=1 (Yk,i − Y k )(Xk,i − X k ), k = 1, 2. The least squares estimates are (1) (2) (1) (2) α bi = Y i , i = 1, 2, and βb = [SXY + SXY ]/[SXX + SXX ]. The vertical distance b 1 − X 2 ). b = Y 1 − Y 2 − β(X between the lines is D = (α1 − α2 ) + β(X 1 − X 2 ), with D b = β, we see that E(D) b = D. Since E(Y k ) = αk , for k = 1, 2, and E(β) 2 (b) We can verify that α bk ∼ N (αk , σ /nk ), k = 1, 2, and !

2 σ βb ∼ N β, (1) .

(2) SXX + SXX b − D ∼ N (0, σ 2 {1/n1 + 1/n2 + (X 1 − X 2 )2 /[S (1) + S (2) ]}). An Hence, d = D XX XX b − D. A 95% unbiased estimate of σ 2 is SSE/(N − 3), which is independent of D symmetric C.I. for D is b ± [F1,n +n −3,.05 ]1/2 sd b D . e.(D).

1 2 b ∗0 β b∗ − β b ∗0 (θ)β b ∗ (θ) > 0 for θ > 0. Since 9.7 We must show that β b ∗0 β b∗ − β b ∗0 (θ)β b ∗ (θ) = y∗0 X∗ {(X∗0 X∗ )−2 − (X∗0 X∗ + θIp )−2 }X∗0 y∗ , β it is sufficient to show that the matrix (X∗0 X∗ )−2 − (X∗0 X∗ + θI)−2 is p.d. Suppose X∗0 X∗ = PDP0 , where P is orthogonal and D = diag(d1 , · · · , dp ). Then (X∗0 X∗ )−2 − (X∗0 X∗ + θI)−2 = PD−2 P0 − (PDP0 + θIp )−2 = P[D−2 − (D + θIp )−2 ]P0 , −2 −2 which is p.d., since D−2 − (D + θIp )−2 = diag(d−2 , · · · , d−2 ), p − (dp + θ) 1 − (d1 + θ) −2 −2 and clearly, di − (di + θ) > 0 for all i.

54

Solutions to Chapter 9

9.8 (a) Let P1 be the projection matrix in the smaller model y = X1 β 1 + ε. Consider (I − P1 )y = (I − P1 )X1 β 1 + (I − P1 )X2 β 2 + (I − P1 )ε = (I − P1 )X2 β 2 + (I − P1 )ε, and let r((I − P1 )X2 ) = p2 . Then, b = [((I − P1 )X2 )0 (I − P1 )X2 ]−1 ((I − P1 )X2 )0 (I − P1 )y β 2 = (X02 (I − P1 )X2 )−1 X02 (I − P1 )y. b is independent of SSE1 = y0 ((I − P1 )y if and only if (b) From Result 5.4.8, β 2 (X02 (I − P1 )X2 )−1 X02 (I − P1 )(σ 2 IN )(I − P1 ) = O which can happen only if X02 (I−P1 ) = O, which contradicts the model assumption.

Hence, they are not independent.

9.9 (a) From Exercise 4.14, X1 β 1 is estimable under the expanded model. Since X1 has full column rank, β 1 = MX1 β 1 with M = (X01 X1 )−1 X01 . Then β 1 is estimable. (b) From (4.2.36), e = [X0 (I − P2 )X1 ]−1 X0 (I − P2 )y; β 1 1 1 note that X01 (I − P2 )X1 is invertible because (I − P2 )X1 has full column rank.

Then e ) = σ 2 [X0 (I − P2 )X1 ]−1 .

Cov(β 1 1 b ) = σ 2 (X0 X1 )−1 . Let A = X0 (I − P2 )X1 ). From On the other hand, Cov(β 1 1 1 Sherman–Morrison–Woodbury formula in Result 1.3.8, with B = X01 P2 , C = I, and D = P02 X1 therein, (X01 X1 )−1 = (A + BCD)−1 = A−1 − A−1 X01 P2 (I + P2 X1 A−1 X01 P2 )−1 P2 X1 A−1 , thus (X01 X1 )−1 ≤ A−1 with “=” if and only if A−1 X01 P2 (I + P2 X1 A−1 X01 P2 )−1 P2 X1 A−1 = O. Since A is nonsingular and (I + P02 X1 A−1 X01 P2 )−1 is p.d., the latter is equivalent to P2 X1 = O, i.e., all the columns of X1 are orthogonal to those of X2 .

Pk 9.10 Let M1 be the model Yi = β0 + j=1 βj Xij + εi , i = 1, · · · , N , i.e., y = Xβ + ε. We can Pk+1 write the bigger model M2 as Yi = β0 + j=1 βj Xij + εi , i = 1, · · · , N , or y = Wδ + ε,

where W = X xk+1 , and δ = (β 0 , βk+1 )0 . Then, b = (W0 W)−1 W0 y. δ Let (x00 , x0,k+1 )0 denote a vector of new predictors. Then, the predicted response under Model M2 is

b Yb0 = X xk+1 δ.

Solutions to Chapter 9

55

Let c = x0k+1 (I − P)xk+1 )−1 and d = (X0 X)−1 X0 xk+1 . Then, 0 −1 (X X) + cdd0 2 0 b Var(Y0 ) = σ (x0 , x0,k+1 ) −cd0

−cd x0 c x0,k+1

= σ 2 [x00 (X0 X)−1 x0 + c(d0 x0 − x0,k+1 )2 ] ≥ σ 2 x00 (X0 X)−1 x0 , the last term on the right being Var(Yb0 ) under M1 .

9.11 If t2j is the square of the t-statistic for tetsting H0 : βj = 0 in the model Yi = β0 + Pk j=1 βj Xij + εi , i = 1, · · · , N , it is the extra-SS F -statistic given by Fj = t2j =

SSEk−1 − SSEk , SSEk /(N − k)

SSEk−1 =

t2j SSEk + SSEk .

N −k

so that

Since Mallow’s Ck−1 is defined by Ck−1 =

SSEk−1 − (N − 2(k − 1)). σ b2

Substitute for SSEk−1 and simplify to give

t2j SSEk SSEk + − (N − 2k) −2 σ b2 (N − k) σ b2 t2j SSEk = 2 + Ck − 2. σ b (N − k)

Ck−1 =

Solutions to Chapter 10

10.1 Write the two-part hypothesis in the form C0 β = d, where

c1,1 c1,2 · · · c1,a 0 C = , β 0 = (µ1 , · · · , µa ), c2,1 c2,2 · · · c2,a and d = (d1 , d2 )0 . The test statistic follows directly from Result 7.2.1.

Pa Pa Pb Pa 10.2 U1a = j=1 Uij = Uii + j6=i Uij = Ni· − l=1 j=1 nij nlj /N·j = Ni· − Ni· = 0, i = 1, · · · , a. This implies that r(U) ≤ a − 1.

10.3 (a) From (10.1.9), we see that for i = 1, · · · , a, E(Wi ) = Ui1 τ1 + · · · + Uia τa = φi , say. Hence, for i = 1, · · · , a, Wi is the b.l.u.e. of φi , from which the required result follows, irrespective of the rank of U. (b) In (10.1.3), in order that     ! b a X b a b b a a X X X X X X X  cij Yij  = µ cij + cij  τi + cij θj E i=1 j=1

i=1 j=1

i=1

j=1

j=1

i=1

Pa Pb Pb is to be function only of θ’s, we must have i=1 j=1 cij = 0, and j=1 cij = 0, Pb Pa Pb Pb i = 1, · · · , a. Then, RHS = j=1 ( i=1 cij )θj = j=1 dj θj , where j=1 dj = Pb Pa j=1 i=1 cij = 0.

10.4 (a) Formulas for Var(ya ), Var(yb ), and Cov(ya , yb ) follow directly from the assumptions of the model (10.1.3).

Pb Pa (b) Since Wi = Yi·· − j=1 nij Y·j· /n·j , and Zj = Y·j· − i=1 nij Yi·· /ni· , the required results follow directly from (a). −1 (c) Cov(w, yb ) = Cov(ya − ND−1 b yb , yb ) = N − NDb Db = O. The other result is similar.

10.5 For any filled cell, the proof of property 1 of Result 10.1.6 is obvious since then µij = E(Yijk ). Property 2, property 3 and property 4 are immediate consequences of property 1 and property 2 of Result 4.3.1.

10.6 To show (a), for i = 1, · · · , n, we can write the function in (a) as ( ) b a X X nij nkj (µ + τk + θj + (τ θ)kj ) , nij (µ + τi + θj + (τ θ)ij ) − n·j j=1 k=1

which shows it is estimable. The proof of estimability of the function in (b) is similar.

57

58

Solutions to Chapter 10

10.7 (a) Yes (b) No 10.8 We see from Example 4.1.3 X0 X =

N n

n0 D

where N = n1 + · · · + na , n = (n1 , · · · , na )0 and D = diag(n1 , · · · , na ). Then G = 0 − 0 0 diag(0, n−1 , n−1 a ) is a (X X) . Let β = (µ, τ1 , · · · , τa ) and cj = (0, cj1 , · · · , cja ) , 1 , · · ·P a 0 j = 1, 2. Then i=1 cj,i τi = cj β. From Definition 4.3.3, the contrasts are orthogonal if Pa and only if c01 Gc2 = 0. But the left hand side exactly i=1 c1i c2i /ni , so thePfirst part of a Result 10.1.2 follows. From Gauss-Markov theorem, the b.l.u.e. of c0j β = i=1 cji τi is c0j β 0 and one choice of β 0 , according to Example 4.2.5, is (0, Y 1· , · · · , Y a· )0 , so c0j β 0 = P a i=1 cji Y i· . From Result 4.2.4, the covariance of the b.l.u.e.’s of the two contrasts is 2 0 σ c1 Gc2 and since the contrasts are orthogonal, we see from above derivation that covariance is 0.

10.9 Set cij = 1/b for i = 1, · · · , a and use property 3 of Result 10.1.6.

10.10 Define ui = ni· /N , i = 1, · · · , a and vj = n·j /N , j = 1, · · · , b. The least squares solutions are µ0 = Y ··· , τi0 = Y i·· − Y ··· , θj0 = Y ·j· − Y ··· , and (τ θ)0ij = Y ij· − Y i·· − Y ·j· + Y ··· , P P P for i = 1, · · · , a, and j = 1, · · · , b. Also, SSE = i j k (Yijk − Y ij· )2 ∼ σ 2 χ2N −ab .

Under H: (τ θ)ij = 0 for all i, j, the restricted solutions are µ0H = Y ··· , 0 τi,H = Y i·· − Y ··· , and 0 θj,H = Y ·j· − Y ··· ,

while SSEH =

XXX i

j

(Yijk − Y i·· − Y ·j· + Y ··· )2 ∼ σ 2 χ2N −(a+b−1) .

k

Then, F (H) =

(SSEH − SSE)/(a − 1)(b − 1) ∼ F(a−1)(b−1),N −ab SSE/(N − ab)

under H. 10.11 If we impose the restrictions τi0 = 0, i = 1, · · · , a and µ0 = 0, we can solve the normal 0 equations in the two-way nested model for θj(i) = Yij· /nij = Y ij· , j = 1, · · · , bi ;

0 0 i = 1, · · · , a. The g-inverse corresponding to this solution is verified to be G = , 0 D where D = diag{1/nij }.

Solutions to Chapter 10

59

10.12 Since the expected value of any observation is estimable, it follows that µ + τi + θj(i) is estimable, as is any linear combination. The form of the b.l.u.e. follows from Result 4.4.1 P as j wij Y ij· .

10.13 Minimizing S(µi , γ1 , γ2 ) =

a X b X

(Yij − µi − γ1 Zij − γ2 Wij )2

i=1 j=1

with respect to µi , i = 1, · · · , a, γ1 and γ1 , we obtain b1 Z i· − γ b2 W i· , µ bi = Y i· − γ Pa Pa Pa Pa S S − i=1 Swzi i=1 Sywi Pawwi i=1 Pa yzi Pa , and γ b1 = i=1 2 i=1 Szzi i=1 Swwi − ( i=1 Swzi ) Pa Pa Pa Pa S S − i=1 Szwi i=1 Syzi Pa zzi i=1Paywi Pa γ b2 = i=1 2 i=1 Swwi i=1 Szzi − ( i=1 Szwi ) where, Swwi =

b X

2

(Wij2 − W i· ), Szzi =

i=1

Syzi =

b X

b X 2 2 (Zij − Z i· ), i=1

(Yij Zij − Y i· Z i· ), Swzi =

i=1

Swyi =

b X

b X

(Wij Zij − W i· Z i· ), and

i=1

(Yij Wij − W i· Z i· ).

i=1

(a) It may be verified that E(Syzi ) = γ1 Szzi + γ2 Swzi , E(Sywi ) = γ2 Swwi + γ1 Swzi , so that E(b γ1 ) = γ1 , i.e., γ b1 is an unbiased estimator of γ1 . (b) Using the notation in the text, we see that Eww , Ezz Eww − (Ewz )2 Ezz Var(b γ2 ) = σ 2 , and Ezz Eww − (Ewz )2 Ewz Cov(b γ1 , γ b2 ) = −σ 2 .

Ezz Eww − (Ewz )2 Var(b γ1 ) = σ 2

(c) Clearly, γ b1 and γ b2 are independent if and only if Cov(b γ1 , γ b2 ) = 0 (under the normalPa Pb ity assumption). That is, they are independent if and only if i=1 j=1 Zij Wij = 0.

P2 Pb 10.14 Set Wij = Yij − γi Xij , and minimize i=1 j=1 (Wij − µi )2 with respect to µi to obtain P2 Pb µ0i = W i· ; then, SSEW = i=1 j=1 (Wij − W i· )2 . Substitute for Wij = Yij − γi Xij ,

60

Solutions to Chapter 10 and W i· = Y i· −γi X i· into SSEW . Minimize S ∗ = with respect to γi to obtain

P2

i=1

Pb

j=1 (Yij −γi Xij −Y i· +γi X i· )

2

Pb

j=1 (Yij − Y i· )(Xij − X i· ) , Pb 2 j=1 (Xij − X i· )

γ bi =

i = 1, 2.

Then, SSE =

2 X b X

(Yij − µ0i − γ bi Xij )2 ∼ χ22b−4 .

i=1 j=1

Under H, start with the model Yij = µi + γXij + εij , and obtain the solutions µ0i = W i· , i = 1, 2, and P2

i=1

γ b=

Pb

j=1 (Yij − Y i· )(Xij − X i· ) .

Pb 2 i=1 j=1 (Xij − X i· )

P2

Then, b 2 X X

SSEH =

(Yij − µ0i − γ bXij )2 ∼ χ22b−3 ,

i=1 j=1

and F (H) =

(SSEH − SSE)/1 ∼ F1,2b−4 SSE/(2b − 4)

under H. It can be verified that this is the same statistic we obtain when we test for parallelism between two lines.

10.15 (a) The normal equations are

XX i

where N =

Y·· = N βb0 X Yij (Zi − Z) = βb1 ni (Zi − Z)2 ,

j

i

Pa

i=1 ni . Then,

SS(β0 , β1 ) = [Y··2 /N ] + βb12

X

ni (Zi − Z)2 .

i

(b) SS(µ, τ1 , · · · , τa ) =

Pa

2 2 i=1 (Yi· /ni ) − Y·· /N . Hence,

Pa SS(β0 , β1 ) − SS(µ, τ1 , · · · , τa ) =

Pa Pa 2 2 i=1 di − ( i=1 ci di ) , Pa 2 i=1 ni (Zi − Z)

2 i=1 ci

√ √ where ci = ni (Y i· − Y ·· ) and di = ni (Zi − Z). The difference in SS is always nonnegative (by Cauchy–Schwarz inequality), with equality only when ci ∝ di , i = 1, · · · , a, i.e., only when Y i· − Y ·· = w(Zi − Z), where w is a constant.

Solutions to Chapter 10

61

10.16 Consider the model Wijk = µij + εijk , where Wijk = Yijk − γij Zijk . Here, µ0ij = W ij· , P P P 2 ∗ and SSEW = i j k (Wijk − W ij· ) . Let S denote the expression obtained by replacing Wijk and W ij· respectively by Yijk − γij Zijk and Y ij· − γij Z ij· in SSEW .

Minimizing S ∗ with respect to γij gives X X γ bij = (Yijk − Y ij· )(Zijk − Z ij· ) (Zijk − Z ij· )2 , and k

k

XXX SSE = [Yijk − Y ij· − γ bij (Zijk − Z ij· )]2 . i

j

k

Repeating the same steps with Wijk = Yijk − γZijk under H: γij = γ gives P P P i j k (Yijk − Y ij· )(Zijk − Z ij· ) , and γ bH = P P P 2 i j k (Zijk − Z ij· ) XXX SSEH = [Yijk − Y ij· − γ b(Zijk − Z ij· )]2 ∼ χ2abc−(ab+1) i

j

k

under H. Hence, F (H) = (abc−2ab)(SSEH −SSE)/(ab−1)SSE ∼ Fab−1,abc−2ab under H. 10.17 Let Ω1 = C ⊥ (X) and Ω2 = C(Z). Then Ω⊥ 1 + Ω2 = C(X, Z) and PΩ1 Ω2 = (I − P)C(Z) = C((I − P)Z). Then the proof follows from property 2 of Result 2.6.7.

10.18 Let SH0 = {y = Xτ + Zβ + ε}, with dim(SH0 ) = r+q < p+q. Let H1 : Zβ = 0, let τ 0H1 = (X0 X)− X0 y and YbH1 = Xτ 0H1 .

Then, YbH0 = Xτ 0 + Zβ 0 = X[(X0 X)− X0 y − (X0 X)− X0 Zβ 0 ] + Zβ 0 = Xτ 0H + (I − P)Zβ 0 = YbH + (I − P)Zβ 0 .

1

1

The sums of squares are R(H0 ) = YbH0 0 YbH0 = R(H1 ) + β 00 Z0 (I − P)Zβ 0 , so that R(H0 | H1 ) = R(H0 ) − R(H1 ) = β 00 Z0 (I − P)Zβ 0 .

Note that dim(SH1 ) = r, so that dim(SH0 )−dim(SH1 ) = q. Now, R(H1 )/σ 2 ∼ χ2 (r, λ1 ), where λ1 = 2σ1 2 {(Xτ +Zβ)0 [X(X0 X)− X0 ](Xτ +Zβ)}. Also, R(H0 | H1 )/σ 2 ∼ χ2 (q, λ2 ), where λ2 = 2σ1 2 (β 0 Z0 (I − P)Zβ). Since SSE = y0 y − R(H0 ) ∼ χ2N −r−q , R(H1 ) and R(H0 | H1 ) are distributed independently of each other, and of SSE, the usual F -tests follow.

10.19 We can write c0l = √ 1

l(l+1)

(0, 1, · · · , 1, −l, 0, · · · , 0). Since we can verify that c0l 1 = 0,

c0l β is a contrast in the τ ’s and must be estimable. By Result 4.4.1, " l # " l # X X 1 1 0 0 0 0 0 d τi − lτl+1 = p Y i· − lY l· , cl β = cl β = p l(l + 1) i=1 l(l + 1) i=1

62

Solutions to Chapter 10 since τi0 = Y i· , i = 1, · · · , a. Since Y i· ∼ N (µ + τi , σ 2 /n), 0 β) = l(l + 1)σ 2 /n, Var(cd l

and σ b2 = SSE/a(n − 1). The 95% marginal C.I. for c0l β is c0l β 0 ±

σ b ta(n−1),.025 n

To construct Scheffé simultaneous confidence intervals to control FBER at α, note that (C0 β 0 )0 (C0 GC)−1 (C0 β 0 )/(a − 1) ∼ F (a − 1, N − a, λ), M SE where λ is the non-centrality parameter and C0 ∈ R(a−1)×a given by  −1 √1 √ 0 0 ··· 0 2  √12 1 −2 √ √ 0 ··· 0  6 6 6  −3 √1 √1 √  √1 · · · 0 0 12 12 12 12 C =  . . . .. .. ..   1 1 1 1 √ √ √ √ · · · √ −a (a−1)a

(a−1)a

(a−1)a

The 100(1 − α)% Scheffé intervals are r c0` β 0 ±

M SE

(a−1)a

     .   

(a−1)a

(a − 1) Fa−1,N −a,α n

for l = 1, · · · , a − 1.

10.20 Let X be a univariate test statistic whose null distribution has a continuous c.d.f. F . By definition, its lower-sided p-value is F (X) and upper-sided p-value is 1 − F (X). Define the left-continuous inverse of F , F ∗ (u) = inf{x : F (x) ≥ u}. Then for u ∈ (0, 1), under the null distribution, P (F (X) < u) = P (X < F ∗ (u)) = P (X ≤ F ∗ (u)) = F (F ∗ (u)).

By continuity of F , F (F ∗ (u)) = u. Then P (F (X) < u) = u, so F (X) is uniformly distributed on (0, 1) and so is 1 − F (X).

10.21 (Hsu, 1984) derived constrained simultaneous C.I.s for multiple comparisons with the best. Assumptions are: 1. independent samples of size n for each of a treatment levels, 2. distributions of the responses only differ in their locations.

In the balanced one-way ANOVA model, Yij ∼ N (µ + τi , σ 2 ), and the assumptions are met. For each i = 1, · · · , a, Hsu constructed simultaneous C.I.s for τi − max`6=i τ` .

The idea is this: consider i = 1; if the upper bound for τ1 − max`6=1 τ` is < 0, then level 1 is not competitive and can be discarded. Suppose for i = 3, the lower bound for τ3 − max`6=3 τ` is ≥ −δ a st, where δ ∗ is close to zero, then level 3 is acceptable. Thus, this method is useful to see which treatment levels are useful or not useful. Hsu constructed simultaneous C.I.s for τi − max`6=i τ` , for i = 1, · · · , a as τi − max τ` ∈ (Y i· − max Y `· ± d)± `6=i

`6=i

Solutions to Chapter 10

63

or, equivalently as [−(Y i· − max Y `· − d)− , (Y i· − max Y `· − d)+ ], `6=i

`6=i

0

where c− = min(c, 0) and c+ = max(c, 0). Obtain d = d√Sn , where S = Z ∞Z ∞ 0

√

M SE and

(Φ(z + d0 S))a−1 dΦ(z) dQ(S) = α,

−∞

Q denoting the distribution of S/σ. For details, see (Hsu, 1984).

10.22 (a) The p.d.f. of y is f (y) = (2π)−N/2 exp{−kXβk2 /2} exp{−kyk2 /2} × exp{y0 Xβ}.

Noting that exp{y0 Xβ} = exp{(Py)0 (Xβ)}, the sufficiency of Py follows from Neyman’s factorization criterion (Lehmann and Casella, 1998, Section 1.6). Next, suppose h(Py) is an integrable function and Z h(Py)f (y) dy = 0 for all values of Xβ. Let Q = (q1 , · · · , qN ), with q1 , · · · , qr consisting of an orthonormal basis of C(X) and the rest consisting of an orthonormal basis of C(X)⊥ . Then P = Q Ir O Q0 , Q0 Xβ = (θ 0 , 00 )0 , where θ = (θ1 , · · · , θr )0 with 0 0 0 0 r θi = q0i Xβ. Make change of variable z = (z1 , z2 ) = Q y with z1 ∈ R . Then z1 h(Py) = h(Q 0 ) := k(z1 ) and from the factorization of f (y), the above condition is equivalent to Z 1 0= k(z1 ) exp{−kzk2 } exp{z0 Q0 Xβ} dz (2π)N/2 Z 1 = k(z1 ) exp{−kz1 k2 − kz2 k2 } exp{z01 θ} dz1 dz2 (2π)N/2 Z 1 = k(z1 ) exp{−kz1 k2 } exp{z01 θ} dz1 .

(2π)r/2 Note that θ can take any values in Rr . Therefore, the Laplace transform of k(z1 ) is constant 0, yielding k(z1 ) ≡ 0. Then h(y) ≡ 0, showing the completeness of Py. (b) If V is p.d., then V] = V−1 . By comparing (4.5.4) and (10.5.3), β 0 in Result 10.5.1 is the same as β 0GLS . Let L = V−1/2 . From Section 10.5.1, β 0GLS is the LS solution to fit the model z = Mβ + η, where z = Ly, M = LX, and η = Lε ∼ N (0, IN ).

By (a), Mβ 0GLS is complete and sufficient for Mβ, which combined with M = LX yields the proof.

(c) We have Cov(Lε) = LVL0 = In O so Lε = (η 0 , 0)0 with η ∼ N (0, In ). Then (10.5.5) follows easily. It is easy to check V] is a g-inverse of V. Let Q be the matrix of projection onto C(M02 ) = R(M2 ), so that M02 A0 = (AM2 )0 = (−M1 Q)0 = −QM01 . Let T = ((I − Q)M01 , M02 )0 . Then

I A M1 X0 V] X = (M01 , M02 ) n0 = T0 T (S.1) A In + A 0 A M2 and Tv = 0 ⇐⇒ M1 (I − Q)v = M1 v + AM2 v = 0 and M2 v = 0 ⇐⇒ Xv = 0, i.e., R(T)⊥ = R(X)⊥ , so r(X0 V] X) = r(T0 T) = r(T) = r(X). Next, put K = L−1

64

Solutions to Chapter 10 and let S1 consist of the first n rows of K0 N, and S2 consist of the rest of the rows of K0 N. Then

In A In O 0 ] 0 0 X V VN = (M1 , M2 ) K0 N = (I − Q)M01 S1 .

A 0 In + A 0 A O O On the other hand, M01 S1 + M02 S2 = (M01 , M02 )K0 N = X0 N = O, so premultiplying both sides by I − Q and combining with the above display yields X0 V] VN = O. Thus V] ∈ ΩV . (d) From Section 4.6.1, there is λ0 that together with β 0 solves M01 M01 β 0 + M02 λ0 = M01 z1 ,

M2 β 0 = z2 .

(S.2)

From (S.1), X0 V] Xβ 0 = (I − Q)M01 M1 (I − Q)β 0 + M02 M2 β 0 .

On the other hand, X0 V] y = (M01 , M02 )

In A0

A In + A 0 A

z1 z2

= (I − Q)M01 z1 + (I − Q)M01 Az2 + M02 z2 = (I − Q)(M01 M1 β 0 + M02 λ0 ) + (I − Q)M01 AM2 β 0 + M02 M2 β 0 = (I − Q)M01 M1 β 0 − (I − Q)M01 M1 Qβ 0 + M02 M2 β 0 .

Then β 0 solves (10.5.3). (e) The solutions to z2 = M2 β can be written as β = β ∗ + (I − M− 2 M2 )γ, where γ is any vector in RN −n . Plugging this into z1 = M1 β + η and kz1 − M1 βk2 yields the proof. (f) From Result 10.5.1, Xβ 0 is invariant to the choices of β 0 and V] . Hence we can let β 0 be as in (d). Then from (e), we have M1 β 0 = M1 β ∗ + Bγ 0 , with γ 0 minimizing kw − Bγk2 . Since Cov(η) = IN −n , from (a), Bγ 0 is a complete and sufficient statistic of Bγ. It follows that M1 β 0 = Bγ 0 + M1 β ∗ is a complete and sufficient statistic of M1 β = Bγ + M1 β ∗ under the restricted model (10.5.5), so

0 M1 β 1β LXβ 0 = M is a complete and sufficient statistic of M = LXβ under M2 β 0 2β 0 (10.5.5). Since L is nonsingular, Xβ is a complete and sufficient statistic of Xβ. (g) Since c0 β is estimable, it is equal to t0 Xβ for some t. From Result 10.5.1, c0 β 0 = t0 Xβ 0 is an unbiased estimator of t0 Xβ with finite variance and is a function that only depends on Xβ 0 . Since Xβ 0 is a complete and sufficient statistic of Xβ, then c0 β 0 = t0 Xβ 0 is the unique b.u.e. of c0 β (see Lehmann and Casella, 1998, Section 2.1).

Solutions to Chapter 11

11.1 (a) Let V1] and V2] denote any two members of ΩV . Since r(X0 V1] X) = r(X0 V2] X), there exists a matrix H such that H(X0 V1] X) = X0 V2] X. By Result 10.5.1, X0 V2] y b = X0 V] y. Again is essentially b.l.u.e. for X0 V2] Xβ. Let βb1 be such that X0 V1] Xβ 1

by Result 10.5.1, b = b.l.u.e. (HX0 V] Xβ) H(X0 V1] y) = H(X0 V1] Xβ) 1 = b.l.u.e. (X0 V2] Xβ) = X0 V2] y.

Hence H(X0 V1] X = X0 V1] y) is equivalent to (X0 V2] X = X0 V2] y). Since H is nonsingular, (X0 V1] X = X0 V1] y) is equivalent to (X0 V2] X = X0 V2] y), and the result follows. (b) Since C(N) = N (X0 ), we have X0 N = O and N0 X = O. Let u be an N -dimensional vector in C(X) ∩ C(VN); then u = Xr for some r and u = VNs for some s. Also, N0 VNs = N0 u = N0 Xr = 0, which implies that (Ns)0 VNs = s0 N0 VNs = 0, implying that VNs = 0, i.e., u = 0. This implies that C(X) and C(VN) are essentially disjoint, with only 0 in common. This, together with C(VN) ⊂ C(V) implies that C(X, VN) ⊂ C(X, V).

(S.3)

Since C(X) and C(VN) are essentially disjoint, r(X, VN) = r(X, V).

(S.4)

The result follows from S.3 and S.4.

11.2 The form of Var(b µGLS ) follows from (11.3.1) using Var(Y i· ) = στ2 + σε2 /ni , and simplifying.

11.3 Since y = (y10 , · · · , ya0 )0 , where yi = (Yi1 , · · · , Yin )0 . Then y0 ((Ia ⊗ Jn )/n − JN /N )y  Jn /n .. = (y10 , · · · , ya0 )  .

 y1   ..  − y0 (1N 10N /N )y . ya Jn /n a a X 1X 0 (10 y)2 = yi Jn yi /n − = n(Y i· )2 − N (Y ·· )2 = SSA . n i=1 N i=1 

11.4 From Exercise 11.3, SSA = y0 My, where M is a symmetric idempotent matrix of rank a − 1 given by M = (Ia ⊗ Jn )/n − JN /N = (Ia − Ja /a) ⊗ Jn /n. Now y = µ1N + η, where η = γ ⊗ 1n + ε ∼ N (0, V) with γ = (γ1 , · · · , γa )0 and V = Ia ⊗ (σγ2 Jn + σε2 In ).

Since M1N = 0, SSA = η 0 Mη. Then from MV = [(Ia − Ja /a) ⊗ Jn /n](Ia ⊗ (σγ2 Jn + σε2 In )) = (σε2 + nσγ2 )M 65

66

Solutions to Chapter 11 b = and Result 5.4.5, SSA / E(M SA ) ∼ χ2a−1 and E(M SA ) = σε2 + nσγ2 . Next, y − y (IN −Ia ⊗Jn /n)y = Bε, where B = Ia ⊗(In −Jn /n) is idempotent with rank N −a. Then SSE/ E(M SE) = ε0 Bε/σε2 ∼ χ2N −a and E(M SE) = σε2 . By Bη = B(γ ⊗ 1n + ε) = Bε, we also have SSE = η 0 Bη. Since MB = [(Ia −Ja /a)⊗Jn /n][Ia ⊗(In −Jn /n)] = O, from the above display, MVB = O, so by Result 5.4.7, SSA / E(M SA ) and SSE/ E(M SE) are independent.

2 0 11.5 Here, y ∼ N (Xβ, V), and L(β, V; y, X) is given in (11.2.8). Let σ 2 = (σε2 , σ12 , · · · , σm ).

Then,

`(β, V; y, X) = −

1 1 N log(2π) − log |V| − (y − Xβ)0 V−1 (y − Xβ).

2 2 2

Then, ∂` = X0 V−1 y − X0 V−1 Xβ, ∂β ∂2` = −X0 V−1 X. ∂β∂β 0 From (11.2.7), ∂V/∂σj2 = Zj Z0j . From (2.7.13), ∂ log |V|/∂V = V−1 . Then by the chain rule of differentiation ∂V−1 ∂V = −V−1 2 V−1 = −V−1 Zj Z0j V−1 , ∂σj2 ∂σj

∂ log |V| = tr[Z0j V−1 Zj ]. ∂σj2

Then 1 ∂` 1 = − tr(V−1 Zj Z0j ) + (y − Xβ)0 V−1 Zj Z0j V−1 (y − Xβ), ∂σj2 2 2 ∂2` 1 = tr(V−1 Zj Z0j V−1 Zh Z0h ) − (y − Xβ)0 V−1 Zj Z0j V−1 Zh Z0h V−1 (y − Xβ), 2 2 ∂σj ∂σh 2 and ∂` = −X0 V−1 Zj Z0j V−1 (y − Xβ). ∂β∂σj2 At the true parameter values, E(y − Xβ) = 0, and E(y − Xβ)0 A(y − Xβ) = tr(AV), so that

∂2` E − = E(X0 V−1 X) = X0 V−1 X; ∂β∂β 0 !

1 ∂2` E − 2 2 = tr(V−1 Zj Z0j V−1 Zh Z0h ) + 0; ∂σj ∂σh 2 ! ∂` E − = −X0 V−1 Zj Z0j V−1 (0) = 0. ∂β∂σj2 Let Sjh = tr(V−1 Zj Z0j V−1 Zh Z0h ) for j, h = 1, · · · , m + 1, and let S ∈ R(m+1)×(m+1) = {Sjh }. Then, the Fisher information matrix is 0 −1

XV X O I= O S/2

Solutions to Chapter 11

67

From the standard result on MLE, as N 1,

b β β −1 ML ∼ N , I σ2 b 2ML σ

approx.

Consequently, b ) ≈ (X0 V−1 X)−1 , Var(β ML Var(b σ 2ML ) ≈ 2S−1 , and b ,σ b 2 ) ≈ O. Cov(β ML

ML

11.6 The ANOVA estimators are σ bγ2 = {M SA − M SB(A) }/bn, σ bθ2 = {M SB(A) − M SE}/n, and σ bε2 = M SE. The ML solutions are σγ2 = {(1 − 1/a)M SA − M SB(A) }/bn, σθ2 = {M SB(A) − M SE}/n, and σε2 = M SE. Based on the range of values assumed by σγ2 and σθ2 , the ML estimators are as follows:  2 2  {M SA − M SB(A) }/bn if σγ ≥ 0, σθ ≥ 0, 2 2 σ bγ = {M SA − σ eε }/bn if σγ ≥ 0, σθ2 < 0,   0 otherwise;   if σγ2 ≥ 0, σθ2 ≥ 0, {M SB(A) − M SE}/n σ bθ2 = SSA /(nab) + SSB(A) /(nab) − M SE/n if σγ2 < 0, σθ2 ≥ 0,   0 otherwise;   if σθ2 ≥ 0, M SE 2 σ bε = {SSE + SSB(A) }/[a(bn − 1)] if σγ2 ≥ 0, σθ2 < 0,   SSTm /abn if σγ2 < 0, σθ2 < 0.

11.7 For the two-way cross-classified model without interaction, the ML equations are SSA a − 1 b − 1 abn − a − b + 1 SSB SSAB + SSE 1 + + + + 2 + = 2 δ11 δ12 δ02 δ4 δ11 δ12 δ0 SSA 1 a−1 = + 2 δ11 δ4 δ11 SSB 1 b−1 = + , 2 δ12 δ4 δ12 where δ0 = δ1 = σε2 , δ11 = σε2 + bnσγ2 , δ12 = σε2 + anσθ2 , and δ4 = δ11 + δ12 − δ0 . Again, the MLE of σε2 is M SE, while estimates of the remaining variance components must be obtained by numerical solution of these nonlinear equations.

68

Solutions to Chapter 11

11.8 The ith nonzero diagonal element of I − P is qii = (N − r)/N , i = 1, · · · , r. Then   N N X N X X 1 Var(y0 Ay) = Var(y0 (I − P)y) + 2σ 4 (µ4 − 3σ 4 ) r2 + r2  2 i=1 ii i=1 j=1 ij   N X X 1 2 2 . rii + rij = Var(y0 (I − P)y) + 2σ 4  (µ4 − 3σ 4 + 1) 2 i=1 i6=j

Now, µ4 − 3σ 4 > −2 =⇒ Var(y0 Ay) is minimum when rij = 0, for all i, j, i.e., if A = I − P. P P 11.9 We see from the proof of Result 11.2.4 that y0 Ay = ε0 Aε = i aii ε2i + 2 i<j aij εi εj .

Then X E(y0 Ay) = E(ε0 Aε) = σε2 aii .

(S.5) i

Next E[(y0 Ay)2 ] = E(ε0 Aεε0 Aε) = E[tr(εε0 Aεε0 A)]   X X = E εi εj ajk εk εl ali  = E[εi εj ajk εk εl ali ]. i,j,k,l

i,j,k,l

Since the εi are i.i.d. with mean 0, E[εi εj ajk εk εl ali ] = 0 if one of i, j, k, l is not equal to any of the others. Hence the expectation is nonzero only in the following four cases, i) i = j 6= k = l, with value σε4 a2ik , ii) i = k 6= j = l, with valueσε4 a2ij , iii) i = l 6= j = k, with value σε4 aii ajj , and iv) i = j = k = l, with value µ4 a2ii . Therefore, X X X X E[(y0 Ay)2 ] = σε4 a2ik + σε4 a2ij + σε4 aii ajj + µ4 a2ii i6=k

=2

X

i6=j

σε4 a2ik +

i,j

X

i

i6=j

σε4 aii ajj +

i,j

X

µ4 a2ii − 3

X

i

σ 2 a2ii

i

!2 = 2σε4 tr(A2 ) + σε4

X

aii

+ (µ4 − 3σε2 )a0 a.

i

This combined with (S.5) then yields (11.2.15).

11.10 From Result 11.2.3, the logarithm of the REML is, up to an additive constant, 1 1 1 bGLS 1N )0 V−1 (y − µ bGLS 1N ). `(σε2 , σγ2 ) = − log |V| − log[10N V−1 1N | − (y − µ 2 2 2 Let λ = σε2 + nσγ2 . From the proof of Result 11.3.6 and (11.3.3), |V| = σε2(n−1)a λa ,

V−1 = Ia ⊗ σε−2 [In − (σγ2 /λ)Jn ].

Then 10N V−1 1N = aσε−2 10n [In − (σγ2 /λ)Jn ]1n = an/λ.

From the proof of Result 11.3.6 and (11.3.15), we see that (y − µ1N )0 V−1 (y − µ1N ) =

Solutions to Chapter 11

69

SSE/(2σε2 ) + SSA /(2λ) + an(Y ·· − µ)2 /(2λ). Given σε2 and σγ2 , this is maximized by Y ·· , which by definition is µ bGLS . Thus (y − µ bGLS 1N )0 V−1 (y − µ bGLS 1N ) = Combining the displays then yields the proof.

SSA SSE + .

2σε2 2λ

Solutions to Chapter 12

12.1 Let Y | p ∼ Binom(m, p) and p ∼ Beta(α, β), 0 ≤ p ≤ 1, α > 0, and β > 0. The joint distribution of Y and p is

m k 1 P(Y = k, p) = p (1 − p)m−k pα−1 (1 − p)β−1 k B(α, β)

m 1 pα+k−1 (1 − p)m−k+β−1 = k B(α, β) R1 where B(α, β) = 0 pα−1 (1 − p)β−1 dp = Γ(α)Γ(β)/Γ(α + β) is the Beta function. The marginal distribution of Y is then Z 1 m 1 pα+k−1 (1 − p)m−k+β−1 dp P(Y = k) = B(α, β) k 0 m B(α + k, m − k + β − 1) = , k B(α, β) which is the Beta-Binomial distribution and does not belong to the exponential family.

It is straightforward to verify that E(Y ) = Ep {E(Y | p)} = m E(p) = mα/(α + β), and Var(Y ) = Ep {Var(Y | p)} + Varp {E(Y | p)} = E{mp(1 − p)} + Var(mp) m2 αβ mαβ + (α + β)(α + β + 1) (α + β)2 (α + β + 1) mαβ(α + β + m) = . (α + β)2 (α + β + 1) =

12.2 (a) For the GLM with Inverse link

−x0i β · Yi + log(x0i β) 1 Yi | xi , β ∼ exp + log(Yi /φi ) − log Yi − log Γ(1/φi ) . φi φi (b) For the GLM with Log link

− exp{−x0i β} · Yi − x0i β 1 + log(Yi /φi ) − log Yi − log Γ(1/φi ) .

Yi | xi , β ∼ exp φi φi (c) For the GLM with Identity link

−Yi /(x0i β) − log(x0i β) 1 Yi | xi , β ∼ exp + log(Yi /φi ) − log Yi − log Γ(1/φi ) , φi φi 71

72

Solutions to Chapter 12

12.3 Under the suggested model, Yt ∼ Poisson(β0 + β1 t). Suppose we are given N years of count data (here N = 20), Yt1 = y1 , · · · , Yt20 = yN , say, so that we can directly write down the likelihood function (N ) N X Y 1 exp (β0 + ti β1 ) (β0 + ti β1 )yi .

L(β0 , β1 ) = QN y ! i i=1 i=1 i=1 The log-likelihood is `(β0 , β1 ) = c − (N β0 + β1

N X

ti ) +

i=1

N X

ki log(β0 + β1 ti ),

i=1

where c denotes a constant term with respect to β0 and β1 . The first partial derivatives of `(β0 , β1 ) with respect to β0 and β1 are N X ∂`(β0 , β1 ) ki = −N + , ∂β0 (β0 + β1 ti ) i=1 N N X X ki ti ∂`(β0 , β1 ) . =− ti + ∂β1 (β 0 + β1 ti ) i=1 i=1

Estimates of β0 and β1 are obtained by setting these equal to zero, and solving the resulting equations. The likelihood can be viewed as L=

N Y

exp{θi yi − b(θi ) − ci (yi )}, so that

i=1

log L =

N X

{θi yi − b(θi )} −

i=1

N X

ci (yi ).

i=1

12.4 (a) Under Model 1, Yi are independent Poisson(λi ), i = 1, 2. Under Model 2, Y1 /(Y1 + Y2 ) is Binom(Y1 + Y2 , p), where Y1 + Y2 is Poisson(λ1 + λ2 ). The likelihood under Model 1 for y1 , y2 = 0, 1, · · · , λ1 > 0, and λ2 > 0 is L1 = exp{−(λ1 + λ2 )}λy11 λy22 /y1 !y2 !

The likelihood under Model 2 can be written as

y1 + y2 y1 exp{−(λ1 + λ2 )}(λ1 + λ2 )(y1 +y2 ) L2 = p (1 − p)y2 y1 (y1 + y2 )! y1 {p(λ1 + λ2 )} {(1 − p)(λ1 + λ2 )}y2 = exp{−(λ1 + λ2 )} , y1 !y2 ! which reduces to L1 since p = λ1 /(λ1 + λ2 ). (b) L1 can be written as (N ) X L1 = exp [yi log λi − λi − log yi !] . i=1

This has the exponential family form with w(λi ) = log λi , b(λi ) = λi and c(yi ) = − log yi !. The canonical link is thus the log link. Likewise, we may rewrite L2 as L2 =

e 1 )λ e y1 e2 )λ ey2 exp(−λ exp(−λ 1 2 × , y1 ! y2 !

Solutions to Chapter 12

73

e1 = p(λ1 + λ2 ) > 0 and λ e2 = (1 − p)(λ1 + λ2 ) > 0, so that the canonical link with λ is once again the log link. (c) In order to incorporate a covariate X, set g(λi ) = log λi = α + βXi , say.

12.5 Let Y ∼ Bernoulli(π). The probit link and its inverse are defined by g(π) = Φ−1 (π) = η and g −1 (η) = Φ(η) = π.

The complementary log-log (or Cloglog) link and its inverse are g(π) = log[− log(1 − π)] = η, and g

−1

(η) = 1 − exp(− exp(η)) = π.

All other expressions in the IRLS algorithm are the same except that the dπ dη in the two cases are φ(η) and exp(η) × exp(− exp(η)).

12.6 Consider the Poisson loglinear model defined in Example 12.1.2 and discussed in section 12.5. Write down in detail the steps in the IRLS algorithm for estimating β and show that the weight matrix W and the vector of adjusted dependent responses z are given by (12.5.1) and (12.5.2), respectively.

Iteration 0 consists of the following steps.

1. Specify an initial vector for β, say β (0) .

2. Evaluate η (0) = Xβ (0) , through the linear predictor function.

3. Evaluate λ(0) = g −1 (η (0) ) = exp(η (0) ). dηi 4. Compute dλ = 1/λi , i = 1, · · · , N and evaluate at it λ(0) . i (0)

(0)

5. Compute the adjusted dependent variate for i = 1, · · · , N : Zi = ηi + (Yi − (0) dηi (0) (0) λi ) dλ = ηi + (Yi /λi − 1) where the derivative is evaluated at λ(0) . i (0)

dηi −2 6. Compute weights Wi = b00w(θi i ) ( dλ ) = λi , all terms evaluated at the estimates i at the 0th iteration, and recalling φ = 1.

(0)

(0)

(0)

(0)

7. Let z(0) = (Z1 , · · · , ZN )0 and W(0) = diag(λ1 , · · · , λN ).

8. Fit the WLS regression z(0) = Xβ + ε(0) .

9. Obtain β (1) = (X0 W(0) X)−1 X0 W(0) z(0) .

Iterate Steps 1-9 starting from β (1) , β (2) , · · · until convergence. The converged estimate b is β . The above derivation also proves (12.5.1) and (12.5.2).

ML

12.7 (a) The Anscombe residual is defined to be the transformation that bring y closest to normality and then standardized to have mean 0 and variance 1. If Y ∼ Poisson, the function y 2/3 brings it closest to normality, which leads to the form of the Anscombe residual for a Poisson GLIM. The deviance residual in (b) follows from the definition and the form of di for the Poisson GLIM. 12.8 (a) For integer y, it can be seen that y−1 Y Γ(y + c) = (j + c). Γ(c) j=0

Apply this in the p.m.f. f (y; µ, α) and use the log-link which gives µi = exp(x0i β). This gives the required likelihood. Part(b) follows the IRLS steps shown in the text.

74

Solutions to Chapter 12

12.9 For each i, given Yi+ , suppose that the random vector yi = (Yi,1 , · · · , Yi,J )0 has a multinomial distribution, Multinom(Yi+ , πi,1 , · · · , πi,J ), with p.m.f.

J Y Yi+ ! Γ(Yi+ + 1) y y πi,ji,j , p(yi ; π i ) = QJ πi,ji,j = QJ y ! Γ(y + 1) i,j j=1 i,j j=1 j=1

PJ where π i = (πi,1 , · · · , πi,J )0 , with 0 ≤ πi,j ≤ 1 and j=1 πi,j = 1. For each i, assume that we have a k-dimensional vector of predictors zi,j , which may also vary by level j. Suppose that the proportions πi,j are proportional to the regression function gi,j (β, zi,j ), j = 1, . . . , J. For simplicity of notation, denote these by gi,j (β) and let PJ Gi (β) = j=1 gi,j (β). Then, πi,j =

gi,j , Gi

and the multinomial likelihood is proportional to LM (β; y1 , · · · , yN ) ∝

y J N Y Y gi,j (β) i,j i=1 j=1

Gt (β)

,

where the subscript M stands for multinomial.

We can use the well-known Multinomial-Poisson (M-P) transform to rewrite the multinomial likelihood as an equivalent Poisson likelihood, the Poisson-trick. Define an additional N -dimensional vector of parameters φ = (φ1 , · · · , φN )0 . The idea behind the Multinomial-Poisson (M-P) transformation is to assume that independently for all i, Yi+ ∼ Poisson(φi Gi (β)).

For each i = 1, . . . , N , we can write down the M-P mixture p.m.f. as P(yi and Yi+ = yi+ ) = P(Yi+ = yi+ ) × p(yi ), The marginal p.m.f. of yi can be obtained by summing the joint p.m.f. over all possible values of Yi+ to yield the Poisson likelihood LP (φ, β; y1 , · · · , yN ) ∝

N Y J Y

(gi,j (β)eφi )yi,j exp(−gi,j (β)eφi ).

i=1 j=1

Let φbi (β) = arg max LP (φ, β). φi

Substituting φbi (β) from this into the previous expression, we can write that as a function of β only and see that the resulting LP (β; y1 , · · · , yN ) is equivalent to LM (β; y1 , · · · , yN ). It follows that the maximum likelihood estimates of β and their asymptotic variances are identical for LP (φ, β) and LM (β; y1 , · · · , yN ).

Solutions to Chapter 13

13.1 Since B0 = GX0 Y, E(B0 ) = GX0 E(Y) = GX0 XB = HB, Cov(β 0(i) , β 0(j) ) = Cov(GX0 y(i) , GX0 y(j) ) = GX0 Cov(ε(i) , ε(j) )XG0 = σij GX0 XG0 by (13.1.5). This b = P E(Y) = PXB = XB, proves property 1 of Result 13.1.2. From (13.1.14), E(Y) b b(j) ) = Cov(Py(i) , Py(j) ) = P Cov(y(i) , y(j) )P = E(E) = (I − P) E(E) = O, Cov(b y(i) , y σij P, Cov(b ε(i) , b ε(j) ) = Cov((I − P)ε(i) , (I − P)ε(j) ) = (I − P) Cov(ε(i) , ε(j) )(I − P) = σij (I − P), proving property 2. Likewise, Cov(β 0(i) , ε(j) ) = Cov(GX0 y(i) , (I − P)ε(j) ) = σij GX0 (I − P) = O, proving property 3.

13.2 For any i, j, Cov(M01 ε(i) , M02 ε(j) ) = M01 Cov(ε(i) , ε(j) )M2 = σij M01 M2 = O, so M01 ε(i) and m02 ε(j) are uncorrelated, and hence by their joint normality, are independent.

13.3 There are p responses for each of the n replications at each crossed level of Factor A and Factor B. (a) The MANOVA decomposition is y`kr = y + (y`· − y) + (y·k − y) + (y`k − y`· − y·k + y) + (y`kr − y`k ).

Summing over ` = 1, · · · , g; j = 1, · · · , b; r = 1, · · · , n gives the breakup of the Corrected Total SSCP as Corrected Total SSCP = SSCP(Factor A) + SSCP(Factor B) + SSCP(Interaction) + SSCP(Error) from which the MANOVA table follows.

Pg Pb Pn • Corrected Total SSCP = `=1 k=1 r=1 (y`kr − y)(y`kr − y)0 with gbn − 1 d.f.

Pg • SSCP(Factor A) = `=1 bn(y`· − y)(y`· − y)0 with g − 1 d.f.

Pb • SSCP(Factor B) = k=1 gn(y·k − y)(y·k − y)0 with b − 1 d.f.

Pg Pb • SSCP(Interaction SSCP) = `=1 k=1 n(y`k −y`· −y·k +y)(y`k −y`· −y·k +y)0 with (g − 1)(b − 1) d.f.

Pg Pb Pn • Error SSCP = `=1 k=1 r=1 (y`kr − y`k )(y`kr − y`k )0 with gb(n − 1) d.f.

This matrix will be p.d. provided p ≤ gb(n − 1). (b) Note that if n = 1 (no replication, we cannot make inference on the interaction effect). The hypotheses are H0 : γ 11 = γ 12 = · · · = γ gb = 0 versus H1 : at least one γ `k 6= 0. The Wilk’s Lambda statistic is Λ∗ =

|Error SSCP| |Interaction SSCP + Error SSCP| 75

76

Solutions to Chapter 13 and we reject H0 at level α for small values of the statistic, i.e., (using Bartlett’s multiplier) if

p + 1 − (g − 1)(b − 1) log Λ∗ > χ2(g−1)(b−1)p (α) − gb(n − 1) − 2 If we reject H0 , interaction effects exist, and the factor effects do not then have a clear interpretation. (c) To test for Factor A main effects via H0 : τ 1 = · · · = τ g = 0 versus H1 : at least one τ ` 6= 0. Let Λ∗ =

|Error SSCP| |SSCP(Factor A) + Error SSCP|

Using Bartlett’s multiplier the decision rule is to reject H0 at level α if

p + 1 − (g − 1) log Λ∗ > χ2(g−1)p (α) − gb(n − 1) − 2 To test for Factor B main effects via H0 : β 1 = · · · = β g = 0 versus H1 : at least one β k 6= 0, Let Λ∗ =

|Error SSCP| | SSCP(Factor B) + Error SSCP|

Using Bartlett’s multiplier the decision rule is to reject H0 at level α if −Cbrgb(n − 1) −

p + 1 − (b − 1) log Λ∗ > χ2(b−1)p (α) 2

b 0 ∼ Wn (Σ0 , m − 1). b 0 ∼ Nn (µ0 , Σ0 /m), (b) mΣ 13.4 From Result 6.1.2, (a) µ 0 0 ) , X = (X01 , · · · , X0m )0 , 13.5 To prove property 1 of Result 13.2.2, write y = (y10 , · · · , ym 0 0 0 Z = diag(Z1 , · · · , Zm ), γ = (γ 1 , · · · , γ m ) , V = diag(V1 , · · · , Vm ), W = V−1 , and ε = (ε01 , · · · , ε0m )0 . Then y = Xτ + Zγ + ε, V = Cov(y) = diag(V1 , · · · , Vm ), W = diag(W1 , · · · , Wm ), and Σγ := Cov(γ) = diag(Dγ , · · · , Dγ ). By the assumption, V is nonsingular. Then

1 `(τ ; y, X) = −C − (y − Xτ )0 V−1 (y − Xτ ) 2 where C is independent of τ .P Therefore, the MLE of τ is the same as the GLS. Since by m the assumption, X0 V−1 X = i=1 X0i Wi Xi is nonsingular, then the MLE (and GLS) is τb = (X0 V−1 X)−1 X0 V−1 y, which is (13.2.9). From Result 11.2.1, the essentially-unique b.l.u.e. of c00 τ + b , where c = (c01 , · · · , c0m )0 , and c00 τ + c0 γ is c00 τb + c0 γ

(S.6) Pm

0 i=1 ci γ i =

b = Σγ Z0 V−1 (y − Xb b 0m )0 γ τ ) = (b γ 01 , · · · , γ b i = Dγ Z0i Wi (y − Xi τb ), giving (13.2.10). From (S.6), (13.2.11) follows. Put K = with γ b = Σγ Z0 W(I − K)y. By K being idempotent and WKV = X(X0 WX)−1 X0 W. Then γ K0 Cov(b γ ) = Σγ Z0 W(I − K)V(I − K0 )WZΣγ = Σγ Z0 (I − K0 )2 WZΣγ = Σγ Z0 (I − K0 )WZΣγ = Σγ Z0 [W − WX(X0 WX)−1 X0 W]ZΣγ .

Solutions to Chapter 13

77

b = (b b 0m )0 , the major submatrices on the right hand side corresponding to Since γ γ 01 , · · · , γ Cov(b γ i ) is exactly the right hand side of (13.2.12). This completes the proof of property 1. As a by-product, for i 6= j, b j ) = −Dγ Z0i Wi Xi (X0 WX)−1 X0j Wj Zj Dγ .

Cov(b γi, γ When θ is unknown, the log-likelihood now is 1 1 `(τ , θ; y, X) = − log[(2π)N/2 |V(θ)|] − (y − Xτ )0 [V(θ)]−1 (y − Xτ ) 2 2 b −1 (y − Xτ ). Then by It follows that the MLE of τ must minimize (y − Xτ )0 [V(θ)] the same argument for (13.2.9), (13.2.13) is proved. The MLE of Cov(b τ ) follows from (13.2.11) and the invariance principle. Then property 2 is proved.

13.6 In the formulas in section 13.2.2, set Vi = σ 2 I + Zi Dγ Z0i .

R 13.7 For any w > 0, P(kyk ≤ w) = ktk≤w ck h(t0 t) dt. By change to spherical coordinates, t = rω, where r > 0 and kωk = 1, dt = rk−1 dr dω. Then Z w Z w P(kyk ≤ w) ck h(r2 )rk−1 dr dω = Sk−1 ck h(r2 )rk−1 dr 0

0

where Sk−1 is the surface area of the unit sphere in Rk , so is equal 2π k/2 /Γ(k/2). Then differentiating both sides of the above equation yields the desired result.

13.8 From (13.3.4) and (13.3.5), σ b∗2 =

b )0 (y − Xβ b ) b ε0 b εLS (y − Xβ LS LS = LS .

N −p N −p

Then property 3 follows from property 4 of Result 4.2.4, which holds for all ε with E(ε) = 0 and Cov(ε) = cI, two properties that a spherical distribution has (see section 5.5.2.2).

13.9 The GLS estimator of β has the form in (4.5.5), i.e., b = (X0 V−1 X)−1 X0 V−1 y, β while the GLS estimator if σ 2 is c2 = σ

1 SSEGLS , N −p

where SSEGLS was shown in (4.5.9). It is easily verified that the maximum likelihood estimator of β coincides with its GLS estimator for any density generator h(.), and the fitted vector and residual vector are also the same as we showed in section 4.5. In fact, the distribution of the MLE of β is the same for all h(.) within the elliptical family.

However, the distribution of the MLE of σ 2 is affected by departures from normality within the family of elliptical distributions. If V = IN , the LS estimates of β and SSE have the forms in (4.2.3) and (4.2.16) respectively. Under the model (13.3.3), the GLS estimates of β and σ 2 have these properties: b ∼ Ep (β, σ 2 (X0 V−1 X)−1 , h) 1. 1 β b 0 V−1 (y − Xβ) b is an unbiased estimator of σ 2 .

2. 2 σ b2 = 1 (y − Xβ) N −p

78

Solutions to Chapter 13 To see this, let y ∼ EN (Xβ, σ 2 V, h), and suppose My (t) exists and has the form given in Definition 5.5.11. The m.g.f. of (X0 V−1 X)−1 X0 V−1 y is My (V−1 X(X0 V−1 X)−1 t) = ψ(t0 (X0 V−1 X)−1 X0 V−1 VV−1 X(X0 V−1 X)−1 t) × exp{t0 (X0 V−1 X)−1 X0 V−1 Xβ}, from which property 1 follows. The proof of property 2 is left to the reader.

13.10 Cook’s distance has the same form as the quantity under the normal linear model, and is hence invariant to h(.).

13.11 Under the GIG prior, β = σz + u, where σ and z ∼ Np (0, Ω) are independent. Since σ 2 ∼ IG(a, b), 1/σ 2 ∼ Gamma(a, 1/b) (see item 16 of Appendix B). Then ξ := 2b/σ 2 ∼ Gamma(a, 2) = χ22a . From p (b/a)z z +u= p +u β=p ξ/(2b) ξ/(2a) p and (b/a)z ∼ Np (0, (b/a)Ω), β follows a p-variate t-distribution with d.f. 2a, location u, and scale matrix (b/a)Ω (see Example 5.5.3).

13.12 Under the GIG prior in Result 13.4.3, y = Xβ + σz, where z ∼ N (0, I) is independent of (σ 2 , β). Then from β | σ 2 ∼ N (µ, σ 2 Ω), y | σ 2 ∼ N (Xµ, σ 2 (XΩX0 + I)). Applying Exercise 13.11, with y playing the role of β, y follows an N -variate t distribution with d.f. ν = 2a, location Xµ, and scale matrix (b/a)(XΩX0 + I).

13.13 For ease of notation, write v = σ 2 and vb = σ b2 . By Bayes rule,

where

f (y; β, v) × π(β, v) π(β, v | y) = , f (y) Z f (y) = f (y; β, v) × π(β, v) dv dβ.

Plug in the conditional p.d.f. of y given (β, v) and π(β, v) = 1/v. Then

1 1 1 1 2 exp − ky − Xβk × π(β, v | y) = f (y) (2πv)N/2 2v v

1 v −N/2−1 2 exp − ky − Xβk . = 2v f (y)(2π)N/2 b − β) + b b 2 + kb By ky − Xβk2 = kX(β εk2 = kX(β − β)k εk2 , π(β, v | y)

v −N/2−1 1 2 b 2 − 1 kb exp − kX(β − β)k ε k 2v 2v f (y)(2π)N/2

v −N/2−1 1 (N − p)b v 0 −1 0 b b = exp − (β − β) (v X X)(β − β) × exp − .

2 2v f (y)(2π)N/2 =

Then by |v −1 X0 X| = v −p |X0 X|, p

|v −1 X0 X| 1 b 0 (v −1 X0 X)(β − β) b π(β, v | y) = exp − (β − β) 2 (2π)p/2

−(N −p)/2−1 v (N − p)b v p × exp − .

2v f (y)(2π)(N −p)/2 |X0 X|

(S.7)

Solutions to Chapter 13

79

Given v, the second factor on the right hand side does not involve β. Then property 1 of Result 13.4.4 follows. Next, write m = (N − p)/2. Integrating out β on both sides, π(v | y) =

1 f (y)(2π)m

p

|X0 X|

× v −m−1 e−mbv/v .

(S.8)

Since the first factor on the right hand side only depends on y, property 2 of Result 13.4.4 follows. Thus, b v(X0 X)−1 ), β | v, y ∼ N (β,

v | y ∼ IG(m, mb v ).

By applying Exercise 13.11 to the posterior distribution of (β, v), property 3 follows, √ immediately yielding property 4. If X∗ is a new N × p matrix, then y∗ = X∗ β + vε with ε ∼ N (0, I) independent of all the other random variables. From the above display, b v[I + X∗ (X0 X)−1 X∗0 ]). Then applying Exercise 13.11, property 5 y∗ | v, y ∼ N (X∗ β, follows.

13.14 This follows directly from Lindley and Smith (1972). Note that the marginal distribution of Y is normal, i.e., Y ∼ N (A1 A2 θ 2 , C1 + A1 C2 A01 ) and θ 1 | θ 2 ∼ N (A2 θ 2 , C2 ). Then, −1 −1 0 −1 θ 1 | y ∼ N (Dd, D), where D−1 = A01 C−1 1 A1 + C2 , and d = A1 C1 y + C2 A2 θ 2 . We can repeat this step for the next hierarchy.

13.15 Let a = y − Xµ and recall Ω∗−1 = Ω−1 + X0 X. Then µ∗ = Ω∗ (Ω−1 µ + X0 y) = Ω∗ (Ω−1 µ + X0 Xµ + X0 a) = Ω∗ (Ω∗−1 µ + X0 a) = µ + Ω∗ X0 a, giving y0 y + µ0 Ω−1 µ − µ∗0 Ω∗−1 µ∗0 = (Xµ + a)0 (Xµ + a) + µ0 Ω−1 µ − (µ0 + a0 XΩ∗ )Ω∗−1 (µ + Ω∗ X0 a) = (µ0 X0 Xµ + 2µ0 X0 a + a0 a) + µ0 Ω−1 µ − (µ0 Ω∗−1 µ + 2µ0 X0 a + a0 XΩ∗ X0 a) = a0 (I − XΩ∗ X0 )a.

From Sherman–Morrison–Woodbury formula, I − XΩ∗ X0 = I − X(Ω−1 + X0 X)−1 X0 = I − X(I + ΩX0 X)−1 ΩX0 = (I + XΩX0 )−1 , completing the proof.

PNj 13.16 Suppose νj = Nj − 1, and Sj2 = i=1 (Yij − Y j )2 /νj , for j = 1, 2. Then, σj2 /{νj Sj2 } ∼ χ−2 νj , i.e., it has an inverse chi-square distribution with νj d.f. It follows that the posterior distribution of F =

σ22 /σ12 S12 /S22 σ22 /S22 = = σ12 /S12 S22 /S12 σ12 /σ22

has an Fν1 ,ν2 distribution.

13.17 (a) π(µ, σ 2 | Y ) ∝ π(Y | µ, σ 2 )π(S 2 | σ 2 )

1 −N 2 2 ∝σ exp − 2 [νS + N (µ − Y ) ] , 2σ where

√ π(Y | µ, σ 2 ) = √

N N exp − 2 (Y − µ)2 ; 2σ 2πσ

also, (N − 1)S 2 /σ 2 ∼ χ2N −1 . The joint posterior distribution is then π(µ, σ 2 | Y ) ∝ π(µ, σ 2 )π(Y | µ, σ 2 )π(S 2 | σ 2 ), which has the normal-χ2 form.

80

Solutions to Chapter 13 (b) 2

π(µ, σ | Y ) ∝ σ

−(N +1)

1 exp − 2 {νS 2 + N (µ − Y )2 2σ

The marginal poterior has a tN −1 distribution.

13.18 Yes, it is a conjugate class.

13.19 First consider an arbitrary y = Xβ + σε, ε ∼ Np (0, I), with (β, σ 2 ) having a prior of the form in Result 13.4.4. For ease of notation, write v = σ 2 and vb = σ b2 . Also, put m = (N − p)/2. From (S.8) and v | y ∼ IG(m, mb v ) (see Exercise 13.13), a comparison of the normalizing constants yields 1 f (y)(2π)m

p

|X0 X|

=

(mb v )m . Γ(m)

b k2 /(N − p), where y b = PC(X) y, Then with vb = ky − y f (y) =

Γ((N − p)/2) Γ(m) p p = . m 0 b k2 )(N −p)/2 |X0 X| (2πmb v) |X X| (πky − y

For two different models, since the parameters have the same structure of priors, they can only be distinguished by Xi and pi . Thus p b1 k2 )(N −p1 )/2 |X01 X1 | Γ((N − p0 )/2) (πky − y p B01 = × . Γ((N − p1 )/2) (πky − y b0 k2 )(N −p0 )/2 |X00 X0 | 13.20 Under the model, y = Xβ + ε and β are jointly normally distributed with

0 y Xµ XΩ 2 IN + XΩX ∼N ,σ . β µ ΩX0 Ω Then from Result 5.2.12, β | y ∼ N (µ∗ , σ 2 Ω∗ ), where µ∗ = µ − ΩX0 (IN + XΩX0 )−1 (Xµ − y), Ω∗ = Ω − ΩX0 (IN + XΩX0 )−1 XΩ = Ω[IN − X0 (IN + XΩX0 )−1 XΩ].

By Sherman–Morrison–Woodbury formula, Ω∗ = Ω(IN + X0 XΩ)−1 = (Ω−1 + X0 X)−1 .

Meanwhile, µ∗ = [Ip − ΩX0 (IN + XΩX0 )−1 X]µ + ΩX0 (IN + XΩX0 )−1 y b +b = (IN + ΩX0 X)−1 µ + ΩX0 [IN − XΩ(Ip + X0 XΩ)−1 X0 ](Xβ ε) 0 0 −1 0 b = Λµ + Ω[IN − X XΩ(Ip + X XΩ) ]X (Xβ + b ε).

Note that X0 b ε = 0 and Λ = Ip − ΩX0 (XΩX0 + IN )−1 X. Then b µ∗ = Λµ + Ω[IN − X0 XΩ(Ip + X0 XΩ)−1 ]X0 Xβ b = Λµ + ΩX0 [IN − XΩ(Ip + X0 XΩ)−1 X0 ]Xβ b = Λµ + (Ip − Λ)β, b = Λµ + ΩX0 (IN + XΩX0 )−1 Xβ giving the first formula in (13.4.26). Next, given y, y∗ = Xβ +ε, where β ∼ N (µ∗ , σ 2 Ω) and ε ∼ N (0, σ 2 IN ) is independent of β. Then y∗ ∼ N (Xµ∗ , σ 2 (Ω∗ + IN )), giving the second formula in (13.4.26).

Solutions to Chapter 13

81

13.21 The state equation is xt+1 = ρxt + wt , and the observation equation is Yt = xt .

13.22 The autoregressive signal-plus-noise model is defined by Zt = Yt + vt Yt = ρYt−1 + wt , 2 where vt and wt are independent error processes, vt ∼ N (0, σv2 ) and wt ∼ N (0, σw ). Express this in the form of (13.5.1) and (13.5.2); the Kalman filter recursions are obtained from (13.5.7)–(13.5.9).

Bibliography

Harville, D. A. (1997), Matrix Algebra from a Statistician’s Perspective, Springer-Verlag: New York.

Hsu, J. C. (1984), “Constrained simultaneous confidence intervals for multiple comparisons with the best,” The Annals of Statistics, 12, 1136–1144.

Lehmann, E. L. and Casella, G. (1998), Theory of Point Estimation, Springer-Verlag, New York, 2nd ed.

Lindley, D. V. and Smith, A. F. M. (1972), “Bayes estimates for the linear model,” Journal of the Royal Statistical Society Series B, 34, 1–18.

83