Matrix Analysis for Statistics, Third Edition James R. Schott Student_Solution Manual, Solutions

SOLUTION MANUAL STUDENT VERSION

Matrix Analysis For Statistics, 3rd edition Solutions to Selected Odd Numbered Problems James R. Schott

1

Chapter 1 1.1 (a)  A=

1

0

0

0





,

B=

0

0

0

1

 .

(b)  A=

1

0

0

0





,

B=

1

0

0

1





,

C=

1

0

0

2

 .

1.3 If AB is symmetric, then AB = (AB)0 = B 0 A0 = BA, and if AB = BA, then (AB)0 = B 0 A0 = BA = AB. 1.5 (a) We have tr(xy 0 ) =

m X

(xy 0 )ii =

i=1

m X

xi yi = x0 y.

i=1

(b) Using Theorem 1.3 (d) tr(BAB −1 ) = tr(B −1 BA) = tr(A).

1.7 Using Theorem 1.3 (a) and (d), we have tr(ABC) = tr{(ABC)0 } = tr(C 0 B 0 A0 ) = tr(CBA) = tr(ACB).

1.9 We need A = B + C, where B 0 = B and C 0 = −C. This would imply that A0 = B − C, and so A + A0 = 2B and A − A0 = 2C. Thus, we have the solutions B=

1 (A + A0 ), 2

C=

1 (A − A0 ).

2

1.11 Since A0 = −A, we have −A2 = (−A)A = A0 A, and so for any m × 1 vector x, x0 (−A2 )x = x0 A0 Ax = y 0 y =

m X i=1

where y = Ax. 2

yi2 ≥ 0,

1.13 (a) (αA)−1 = α−1 A−1 since α−1 A−1 αA = (α−1 α)A−1 A = Im . (b) (A0 )−1 = (A−1 )0 since 0 (A−1 )0 A0 = (AA−1 )0 = Im = Im .

(c) (A−1 )−1 = A since AA−1 = Im . (d) Note that 1 = |Im | = |A−1 A| = |A−1 ||A|, so |A−1 | = |A|−1 . −1 (e) A−1 = diag(a−1 11 , . . . , amm ) if A = diag(a11 , . . . , amm ) since −1 −1 −1 diag(a−1 11 , . . . , amm ) diag(a11 , . . . , amm ) = diag(a11 a11 , . . . , amm amm ) = diag(1, . . . , 1) = Im .

(f) It follows from (b) that (A−1 )0 = (A0 )−1 = A−1 if A = A0 . (g) (AB)−1 = B −1 A−1 since B −1 A−1 AB = B −1 Im B = B −1 B = Im .

1.15 Using the cofactor expansion formula on the first column of A, we get

|A| =

=

1×

1

2

0

2

2

1

−1

1

2

+1×

2

1

1

1

2

0

−1

1

2

{4 − 2 + 0 − (0 + 8 + 1)} + {8 + 1 + 0 − (−2 + 2 + 0)} = −7 + 9 = 2.

1.17 Let B be the matrix obtained from A by replacing the kth row of A by its ith row. Note that Akj = Bkj for j = 1, . . . , m, and so m X

aij Akj =

j=1

m X

bkj Bkj = |B| = 0,

j=1

due to Theorem 1.4 (h). Similarly, let C be the matrix obtained from A by replacing the kth column of A by its ith column. Note that Ajk = Bjk for j = 1, . . . , m, and so m X

aji Ajk =

j=1

m X j=1

by another application of Theorem 1.4 (h).

3

bjk Bjk = |B| = 0,

1.19 The adjoint matrix of A is 

−7

−5

   4 A# =    −2  3

−1

9



2

−4

0

2

1

−3

  0  ,  0   1

−5

9

−1

2

−4

0

2

1

−3

and since |A| = 2, the inverse of A is 

−7

   4 1 −1 A =  2  −2  3



  0  ,  0   1

1.21 Using Theorem 1.9 (a) with A = Im , B = 1, C = 1m , and D = 10m , we find that (Im + 1m 10m )−1 = Im −

1 1m 10m , 1+m

(b) and with A = Im , B = 1, C = e1 , and D = 10m , we find that 1 (Im + e1 10m )−1 = Im − e1 10m .

2 1.23 Applying Corollary 1.9.2 with A = D, c = αa, and d = b, we get (D + αab0 )−1 = D−1 − αD−1 ab0 D−1 /(1 + αb0 D−1 a).

1.25 Partition B = A−1 as A is partitioned and note that AB = Im yields the four equations A11 B11 = Im1 , A21 B11 + A22 B21 = (0),

A11 B12 = (0), A21 B12 + A22 B22 = Im2 .

−1 Premultiplying the first two equations by A−1 11 yield the solutions B11 = A11 and B12 = (0). Substi-

tuting these solutions into the third and fourth equations and then premultiplying by A−1 22 yield the −1 −1 solutions B22 = A−1 22 and B21 = −A22 A21 A11 . That is,  A−1 11 A−1 =  −1 −A−1 A 22 21 A11

4

(0) A−1 22

 .

1.27 Expanding along the last row of A, we find that

|A| = −2

0

1 −1

−1

1 −1

−1

2

−2

0

2

0

1

1

−1

1

1

−1

2

= −2(2) − 2(−2) = 0,

so rank(A) < 4. On the other hand, the determinant of the submatrix obtained by deleting the last row and column of A is 2

0

1

1

−1

1

1

−1

2

= −2 6= 0.

Thus, rank(A) = 3.

1.29 Theorem 1.12 (b) follows from part (a) and Theorem 1.7 since |P 0 AP | = |P 0 ||A||P | = |A||P |2 = |A|, while Theorem 1.12 (c) follows since (P Q)0 P Q = Q0 P 0 P Q = Q0 Q = Im .

1.31 (P P 0 )31 = (P P 0 )32 = 0 yield the equations p31 + p32 + p33 = 0 and p31 = p32 , which together imply that p3 = (p31 , p32 , p33 )0 has the form p3 = (a, a, −2a)0 for some a. But p03 p3 = 6a2 and so since (P P 0 )33 = 1, we must have a = ±1. Thus, there are two solutions: p31 = p32 = 1, p33 = −2 and p31 = p32 = −1, p33 = 2.

1.33 From the identity P 0 P = Im we get    i P10 h P 0P   P1 P2 =  1 1 P20 P20 P1

P10 P2 P20 P2





=

Im1

(0)

(0)

Im2

 ,

which immediately leads to P10 P1 = Im1 and P20 P2 = Im2 . From the identity P P 0 = Im we get   h i P0 1  = P1 P10 + P2 P20 = Im .

P1 P 2  P20

5

1.35 (a) 

2 −3

1



  2 4 .  4 −1

  A= 2  −3 (b) 

3

1

  A= 1  1

1



2

  2 .  2

1

1

5

(c) 

0

  A= 1  1

0 1



  1 .  0

1.37 Let y = Ax for an arbitrary m × 1 vector x. It follows that A = T 0 T is nonnegative definite since x0 Ax = x0 T 0 T x = y 0 y =

m X

yi2 ≥ 0.

i=1

1.39 Let y = B 0 x for an arbitrary n × 1 vector x. Since A is nonnegative definite, it follows that x0 BAB 0 x = y 0 Ay ≥ 0, and so BAB 0 is nonnegative definite.

1.41 We have 2

E(z)

=

m(1) (t)|t=0 = et /2 t|t=0 = 0,

E(z 2 )

=

m(2) (t)|t=0 = et /2 (1 + t2 )|t=0 = 1,

E(z 3 )

=

m(3) (t)|t=0 = et /2 t(1 + t2 ) + et /2 2t|t=0 = et /2 (3t + t3 )|t=0 = 0,

E(z 4 )

=

m(4) (t)|t=0 = et /2 t(3t + t3 ) + et /2 (3 + 3t2 )|t=0 = et /2 (3 + 6t2 + t4 )|t=0 = 3,

E(z 5 )

=

m(5) (t)|t=0 = et /2 t(3 + 6t2 + t4 ) + et /2 (12t + 4t3 )|t=0 = et /2 (15t + 10t3 + t5 )|t=0 = 0,

E(z 6 )

=

m(6) (t)|t=0 = et /2 t(15t + 10t3 + t5 ) + et /2 (15 + 30t2 + 5t4 )|t=0

=

et /2 (15 + 45t2 + 15t4 + t6 )|t=0 = 15.

2

2

2

2

2

2

2

2

2

2

2

2

2

6

1.43 Since XX 0 =

Pn

0 −1 X1n = x̄, we have i=1 xi xi and n

S

−1

n X

! xi x0i − nx̄x̄0

=

(n − 1)

=

(n − 1)−1 XAX 0 ,

= (n − 1)−1 (XX 0 − n−1 X1n 10n X 0 )

i=1

where A = In − n−1 1n 10n . −1/2

1.45 (a) With DΩ

√ √ √ = diag(1/ 2, 1/ 2, 1/ 3), the correlation is given by  √  1 1/2 −1/ 6  √    −1/2 −1/2 P = DΩ ΩDΩ = 1/2 1 1/ 6  .   √ √ −1/ 6 1/ 6 1

(b) It follows that u ∼ N (µu , σu2 ), where µu = 103 µ = 6,

σu2 = 103 Ω13 = 9.

(c) v ∼ N3 (µv , Ωv ), where µv = Aµ = (9, −2, −1)0 and     5 7 3 27 2 4         Ωv = AΩA0 =  3 0 −4  A0 =  2 7 4  .     2 1 −2 4 4 3 (d) w ∼ N5 (µw , Ωw ), where



9



     −2    A  µ =  µw =   −1  ,   B    6    1 



and

  Ωw = 

A B

 Ω

h

A0

27

2

4

15

2



     2 7 4 −1 −3   i    B0 =  4 4 3 1 −1  .      15 −1 1 9 2    2 −3 −1 2 2

7

(e) Since σu2 6= 0 and |Ωv | 6= 0, the distributions given in (b) and (c) are nonsingular. Clearly, |Ωw | = 0 since  rank 

A B

  = 3,

so the distribution given in (d) is singular.

1.47 (a) Since H1m = m1/2 e1 and HH 0 = Im , u ∼ Nm (m1/2 µe1 , σ 2 Im ). (b) Note that u1 = m1/2 x̄ and x0 x = x0 H 0 Hx = u0 u so m m m m X X X X 2 2 2 2 2 (xi − x̄) = xi − mx̄ = ui − u1 = u2i . i=1

i=1

i=1

i=2

It follows from part (a) that u1 is independent of u2 , . . . , um and, as a result, s2 = (m − Pm 1)−1 i=2 u2i is independent of x̄ = m−1/2 u1 .

1.49 Since E(z) = 0, and v and z are independent, we have E(x) = E(n1/2 z/v) = n1/2 E(v −1 )E(z) = 0.

In addition, var(x) = E(xx0 ) = E(nzz 0 /v 2 ) = nE(v −2 )E(zz 0 ) =

n Im , n−2

since E(v

−2

Z ∞ )

=

t 0

=

n 2 −1

Γ( n−2 e−t/2 2 ) dt = n n/2 2Γ( n2 ) 2 Γ( 2 )

−1 t

Γ( n2 − 1) 1 = , n 2( 2 − 1)Γ( n2 − 1) (n − 2)

as long as n > 2.

8

n−2

Z ∞

t 2 −1 e−t/2

0

2 2 Γ( n−2 2 )

n−2

dt

Chapter 2 2.1 (a) If α 6= 1, α(a, b, a + b, 1) is not in the set so it is not a vector space. (b) For any scalars α and β,     α   

a1 b1 c1 a1 + b1 − 2c1     =   





      +β      

a2 b2 c2 a2 + b2 − 2c2

        

αa1 + βa2 αb1 + βb2 αc1 + βc2

(αa1 + βa2 ) + (αb1 + βb2 ) − 2(αc1 + βc2 )

      

is in the set so it is a vector space. (c) If α + β 6= 1, then     α   

a1 b1 c1 1 − a1 − b1 − c1     =   





      +β      

a2 b2 c2 1 − a2 − b2 − c2

       

αa1 + βa2 αb1 + βb2 αc1 + βc2

(α + β) − (αa1 + βa2 ) − (αb1 + βb2 ) − (αc1 + βc2 )

       

is not in the set so it is not a vector space.

2.3 (a, a + b, a + b, −b)0 6= (1, 1, 1, 1)0 for any choices of a and b, so (1, 1, 1, 1)0 is not in S, whereas (4, 1, 1, 3)0 is in S since (a, a + b, a + b, −b)0 = (4, 1, 1, 3)0 when a = 4, b = −3.

2.5 Straightforward calculation reveals that  Ω−1 = 

9

4/3

2/3

2/3

4/3

 ,

√ dΩ (x1 , µ) = 2 and dΩ (x2 , µ) = 2/ 3, so x2 is closer to the mean.

2.7 (a) Since x0 x = 10, y 0 y = 11 and x0 y = 6, we have cos θ = p

6 10(11)

= .572,

and so θ = .9618 = π/3.2665. (b) We have x0 y cos(π/6) = .866 = p , 3(2) so x0 y = 2.121.

2.9 The identity holds since kx + yk2 + kx − yk2

= hx + y, x + yi + hx − y, x − yi = hx, xi + hx, yi + hy, xi + hy, yi + hx, xi − hx, yi − hy, xi + hy, yi =

2hx, xi + 2hy, yi = 2kxk2 + 2kyk2 .

2.11 (a) This is a linearly independent set since neither vector is a scalar multiple of the other. (b) This is a linearly dependent set since (4, −1, 2)0 − 2(3, 2, 3)0 + (2, 5, 4)0 = 00 . (c) This is a linearly independent set since the only solution to a(1, 2, 3)0 +b(2, 3, 1)0 +c(−1, 1, 1)0 = 00 is a = b = c = 0. (d) This is a linearly dependent set of vectors because the number of vectors in the set exceeds the dimension of the vectors.

2.13 (a) This is a linearly dependent set of vectors since 2(2, 1, 4, 3)0 = (4, 2, 8, 6)0 . (b) Any subset of two of the vectors is a linearly independent set with the exception of the subset consisting of the two vectors referred to in part (a).

2.15 (a) Any basis for R4 has four vectors, so this set is not a basis. (b) It is easily verified that the only solution to a(2, 2, 2, 1)0 +b(2, 1, 1, 1)0 +c(3, 2, 1, 1)0 +d(1, 1, 1, 1)0 = 00 is a = b = c = d = 0. Since it is a set of four linearly independent vectors it must be a basis for R4 .

10

(c) Since (2, 0, 1, 1)0 − 2(3, 1, 2, 2)0 + (2, 1, 1, 2)0 + (2, 1, 2, 1)0 = 00 , the set is linearly dependent and consequently not a basis for R4 .

2.17 (a) The dimension of S is less than 3 since 2x1 + 3x2 − x3 = 0. Any pair of the vectors form a linearly independent set so the dimension of S is 2, and we will use z 1 = x1 and z 2 = x2 . (b) We have x = α1 x1 + α2 x2 when α1 = −1 and α2 = 1. (c) Two solutions are α1 = −1, α2 = 1, α3 = 0 and α1 = 1, α2 = 4, α3 = −1.

2.19 (a) If A is m × n and B is m × p, then  A=

h

A

B

i 



In (0)

,

 B=

h

A

B

i 

(0) Ip

 ,

and so the result is a consequence of Theorem 2.8(a). (b) Clearly, any nonnull vector of the form (x0 , 00 )0 cannot be expressed as a linear combination of vectors of the form (00 , y 0 )0 . This implies that the number of linearly independent columns of   A (0)   (0) B is equal to the number of linearly independent columns of   A ,  (0) which equals rank(A), plus the number of linearly independent columns of   (0)  , B which equals rank(B). Thus, we have  A rank  (0)

(0) B

   = rank(A) + rank(B).

A similar argument can be used to prove the second identity. (c) We establish the inequality for the first matrix given in part (c) of the theorem. The proofs for the others are similar. Suppose A is m × n, B is p × q, and rank(A) = r. Then there exists an 11

n × n nonsingular matrix X such that AX = [A1

(0)], where A1 is m × r with rank(A1 ) = r.

C2 ], where C1 is p × r and define B∗ = [C2 B]. Then      A (0) A (0) X (0) A  = rank    = rank  1 rank  C B C B (0) Iq C1

Let CX = [C1 

(0) B∗

  .

Note that the first r columns of the last matrix given above are linearly independent of the remaining columns since A1 x 6= 0 for all nonnull x. Thus,       A1 (0) A1 (0)  = rank   + rank   rank  C1 B∗ C1 B∗ =

rank(A) + rank(B∗ ) ≥ rank(A) + rank(B).

2.21 Let X = (x1 , x2 ), Y = (y 1 , y 2 ), and

 A=

a

c

b

d

 ,

so that Y = XA. Note that y 1 and y 2 are linearly independent, that is, Y has full-column rank, if and only if A is nonsingular. Thus, we must have |A| = ad − bc 6= 0.

2.23 First note that the set {γ 1 , . . . , γ m } is linearly independent since the matrix Γ = (γ 1 , . . . , γ m ) is triangular with each diagonal element equal to one and, hence, |Γ| = 1. Also, for arbitrary x ∈ Rm , let αi = xi − xi+1 for i = 1, . . . , m − 1 and αm = xm . Then m X

αi γ i

=

i=1

m−1 X

(xi − xi+1 )γ i + xm γ m

i=1

=

m−1 X

m−1 X

xi γ i −

i=1

=

m−1 X

xi+1 γ i + xm γ m

i=1 m X

xi γ i −

i=1

= x1 γ 1 +

xi γ i−1 + xm γ m

i=2 m−1 X

xi (γ i − γ i−1 ) + xm (γ m − γ m−1 )

i=2

= x1 e 1 +

m−1 X

xi ei + xm em = x.

i=2

Thus, {γ 1 , . . . , γ m } is a basis for Rm .

12

2.25 If for some C, AC = B, then R(B) ⊆ R(A) follows from Problem 2.24(a). Now suppose that R(B) ⊆ R(A). Thus, any vector in R(B) is also in R(A), and so it can be written as a linear combination of the columns of A. In particular, each column of B can then be written as a linear combination of the columns of A and this immediately leads to the existence of a matrix C so that AC = B. 2.27 (a) According to Definition 2.8, the columns of X = (x1 , . . . , xn ) will be a basis if they span S. Suppose the columns of the m × n matrix Y form a basis for S so for each x ∈ S, there exists an n × 1 vector a such that x = Y a. In particular, there exists an n × n matrix A such that X = Y A since the columns of X are in S. But the columns of X are linearly independent so n = rank(X) = rank(Y A) ≤ rank(A), which implies rank(A) = n and Y = XA−1 . Thus, the columns of X span S since the columns of Y span S. (b) Let X = (x1 , . . . , xn ) and let the columns of the m × n matrix Y form a basis for S. Since the columns of Y must be in S and the columns of X span S, we must have Y = XB for some n × n matrix B. Note that rank(Y ) = rank(XB) ≤ rank(X).

But rank(Y ) = n since its columns form a basis, and so we must also have rank(X) = n, which confirms that the columns of X are linearly independent. (c) Let X1 = (x1 , . . . , xr ) and suppose the columns of the m × n matrix Y span S. Since the columns of X1 are in S, X1 = Y A1 for some n × r matrix A1 . Also r = rank(X1 ) = rank(Y A1 ) ≤ rank(A1 ), from which it follows that rank(A1 ) = r, and so we can find an n × (n − r) matrix A2 such that A = [A1

A2 ] is nonsingular. Then the n columns of X = [X1

X2 ] = [Y A1

Y A2 ] = Y A

are in S and are linearly independent since we have rank(X) = rank(Y ) = n. The result now follows from part (a) of this theorem.

2.29 Since

 D=

Im

−F

(0)

Ip





,

E=

13

In

(0)

−G

Iq

 

are both nonsingular, we have   C B  = rank  A (0)





C

rank D  

=

rank 

A

(0) A

 

B

 E (0)  B  = rank(A) + rank(B), (0)

where the final equality follows from Theorem 2.9(b).

2.31 If

 A=

1

0

0

0





,

B=

0

1

0

0

 ,

then rank(AB) = 1 whereas rank(BA) = 0.

2.33 (a) α1 x1 + α2 x2 + α3 x3 + α4 x4 = 0 yields solutions α2 = −2α1 , α3 = −α1 , α4 = 2α1 , where α1 is arbitrary, so {x1 , x2 , x3 , x4 } is a linearly dependent set. The only solution to α1 x1 +α2 x2 +α3 x3 = 0 has α1 = α2 = α3 = 0, so {x1 , x2 , x3 } is a linearly independent set and thus a basis for S. (b) z 1 = √110 (1, 2, 1, 2)0 . y 2 = (2, 3, 1, 2)0 −

1 13 (1, 2, 1, 2)0 = (7, 4, −3, −6)0 , 10 10

1 so z 2 = √110 (7, 4, −3, −6)0 . Finally

y 3 = (3, 4, −1, 0)0 −

4 1 1 10 (1, 2, 1, 2)0 − (7, 4, −3, −6)0 = (−6, 6, −10, 2)0 , 10 (11/10) 10 11

so z 3 = √144 (−3, 3, −5, 1)0 .

1 (c) We have z 01 x = √310 , z 02 x = √110 , and z 03 x = √−1 , so 11

3 1 1 1 u = √ z1 + √ z 2 − √ z 3 = (1, 1, 1, 1)0 .

2 10 110 11 (d) v = x − u = 21 (1, −1, −1, 1)0 .

2.35 We have



1

  X1 =  2  3

1



  1 ,  −1

 PS = X1 (X10 X1 )−1 X10 =

1 and so the point in S closest to x is PS x = 21 (16, 25, 20)0 .

14

17

1    20 42  −5

20 −5 26 4



  4 ,  41

2.37 (a) Letting xi denote the ith column of PS , we find that α1 x1 +α2 x2 +α3 x3 +α4 x4 = 0 has solutions α2 = α3 = .5α1 , α4 = α1 for any α1 , so PS is singular. On the other hand, α1 x1 +α2 x2 +α3 x3 = 0 has only the solution α1 = α2 = α3 = 0, so dim(S) = rank(PS ) = 3. (b) The first 3 columns of PS form a basis for S. 2.39 (a) Denoting the ith column of A by ai , note that a1 = a2 + a3 − a4 and a3 = 43 a4 − 23 a2 , while no column of A is a scalar multiple of another column. Thus, dim(S) = rank(A) = 2 and any two columns of A, for instance, {a2 , a4 }, form a basis for S. (b) We have dim{N (A)} = 4 − rank(A) = 4 − 2 = 2, and so a basis will be 2 linearly independent vectors satisfying Ax = 0. Note that (A)1· x = 0 yields x2 = − 21 (x1 + x4 ), while (A)3· x = 0 yields x3 = − 41 (x1 + 3x4 ). By choosing x1 = 4, x4 = 0, and x1 = 0, x4 = 4, we get the basis {x1 , x2 }, where x1 = (4, −2, −1, 0)0 and x2 = (0, −2, −3, 4)0 . (c) There is no α1 and α2 such that α1 a2 + α2 a4 = (3, 5, 2, 4)0 , so (3, 5, 2, 4)0 ∈ / S. (d) A14 6= 0, so 14 ∈ / N (A).

2.41 Let {x1 , . . . , xr } be a basis for T so that for any x ∈ T , there exist scalars α1 , . . . , αr such that Pr x = i=1 αi xi = Xα, where X = (x1 , . . . , xr ) and α = (α1 , . . . , αr )0 . Since u is linear, we have ! r r X X u(x) = u αi xi = αi u(xi ) = U α, i=1

i=1

where U = (u(x1 ), . . . , u(xr )). But u(x) = U α = U (X 0 X)−1 X 0 Xα = U (X 0 X)−1 X 0 x = Ax, where A = U (X 0 X)−1 X 0 .

2.43 Since x̄ = m−1 10m x, we have u(x) = x − 1m x̄ = Im x − m−1 1m 10m x = Ax, where A = Im − m−1 1m 10m . Since A1m = 1m − m−1 1m 10m 1m = 1m − 1m = 0, and for any vector x that is orthogonal to 1m , Ax = x − m−1 1m 10m x = x, 15

it follows that dim{R(A)} = m − 1 and dim{N (A)} = 1.

2.45 (a) The model y = Xβ + has   y11    ..   .       y1n1     .  ..  , y=       y  k1     ..   .    yknk



11 .. .



           1n1     .  ..  , =        k1     ..   .    knk

1n1

0

···

0



   0 X=  ..  .  0

1 n2 .. .

···

0 .. .

0

···

   .   

1 nk



−1 0 0 (b) Since X 0 X = diag(n1 , . . . , nk ), (X 0 X)−1 = diag(n−1 1 , . . . , nk ), and X y = (n1 ȳ1 , . . . , nk ȳk ) , the

least squares estimator of β is 

 ȳ1    .  β̂ = (X 0 X)−1 X 0 y =  ..  .   ȳk Since 

 ȳ1 1n1   ..   ŷ = X β̂ =  , .   ȳk 1nk it then follows that SSE1 = (y − ŷ)0 (y − ŷ) =

ni k X X (yij − ȳi )2 . i=1 j=1

(c) The reduced model has X = 1n . In this case, X 0 X = n, (X 0 X)−1 = n−1 , and X 0 y = nȳ, so the least squares estimator of µ in the reduced model is µ̂ = (X 0 X)−1 X 0 y = n−1 nȳ = ȳ.

This yields ŷ = X µ̂ = ȳ1n , and so the sum of squared errors for this reduced model is 0

SSE2 = (y − ȳ1n ) (y − ȳ1n ) =

ni k X X i=1 j=1

16

(yij − ȳ)2 .

The sum of squares for treatment is given by SST

= SSE2 − SSE1 =

ni k X X

(yij − ȳ)2 −

i=1 j=1

=

ni k X X

=

ni k X X

i=1 j=1

2 (yij − 2ȳyij + ȳ 2 ) −

i=1 j=1

ni k X X (yij − ȳi )2

ni k X X

2 (yij − 2ȳi yij + ȳi2 )

i=1 j=1

(−2ȳyij + ȳ 2 + 2ȳi yij − ȳi2 )

i=1 j=1

=

k X

(−2ni ȳ ȳi + ni ȳ 2 + 2ni ȳi2 − ni ȳi2 )

i=1

=

k X

ni (ȳi2 − 2ȳ ȳi + ȳ 2 ) =

i=1

k X

ni (ȳi − ȳ)2 .

i=1

(d) In Example 2.11, N is the number of observations, k + 1 is the number of parameters in the complete model, and k2 is the difference in the number of pararmeters in the complete and reduced models. For our models in this problem, these quantities are n, k, and k − 1, respectively.

Making these substitutions along with SSE2 − SSE1 = SST in (2.11), we get F =

SST /(k − 1) .

SSE1 /(n − k)

2.47 Suppose y i ∈ S1 + S2 for i = 1, 2. Then for i = 1, 2, there are vectors xi ∈ Si and ui ∈ Si such that y 1 = x1 + x2 and y 2 = u1 + u2 . Now α1 y 1 + α2 y 2 = α1 (x1 + x2 ) + α2 (u1 + u2 ) = v 1 + v 2 , where v 1 = α1 x1 + α2 u1 and v 2 = α1 x2 + α2 u2 . But since S1 and S2 are vector spaces, v i ∈ Si for i = 1, 2, and so α1 y 1 + α2 y 2 ∈ S1 + S2 and, hence, S1 + S2 is a vector space.

2.49 Suppose x ∈ S1 + S2 . This implies that there is an x1 ∈ S1 and an x2 ∈ S2 such that x = x1 + x2 .

Since x1 ∈ S1 , x1 ∈ S1 ∪ S2 , and since x2 ∈ S2 , x2 ∈ S1 ∪ S2 . Further, x1 ∈ T and x2 ∈ T since S1 ∪ S2 ⊂ T . But T is a vector space so x = x1 + x2 ∈ T . This confirms that S1 + S2 ⊆ T .

2.51 Each vector in {x1 , . . . , xr y 1 , . . . , y h } is in S1 + S2 since xi ∈ S1 implies that xi ∈ S1 + S2 and y i ∈ S2 implies that y i ∈ S1 + S2 . Now x ∈ S1 + S2 implies that x can be expressed as x = u1 + u2 , where Pr u1 ∈ S1 and u2 ∈ S2 . But since {x1 , . . . , xr } spans S1 and {y 1 , . . . , y h } spans S2 ,u1 = i=1 αi xi and 17

u2 =

Ph

j=1 βj y j for some scalars α1 , . . . , αr , β1 , . . . , βh . Thus,

x = u1 + u2 =

r X

α i xi +

i=1

h X

βj y j ,

j=1

and so {x1 , . . . , xr , y 1 , . . . , y h } spans S1 + S2 .

2.53 (a) max(r1 , r2 ) ≤ dim(S1 + S2 ) ≤ min(m, r1 + r2 ). (b) max(0, r1 + r2 − m) ≤ dim(S1 ∩ S2 ) ≤ min(r1 , r2 ).

2.55 Let S1 be the space spanned by the vector (1, 0, 0)0 and S2 be the space spanned by the vector (0, 0, 1)0 .

Then T ⊕ S1 = R3 , T ⊕ S2 = R3 , and S1 ∩ S2 = {0}.

2.57 This is an immediate consequence of Theorem 2.25 since dim(S1 + S2 ) = dim(S1 ) + dim(S2 ) if and only if dim(S1 ∩ S2 ) = 0, that is, S1 ∩ S2 = {0}.

2.59 (a) The rank of PS1 |S2 is 2 and so for a basis we can choose two linearly independent columns of PS1 |S2 , for instance, {(1, 0, 0)0 , (1, 3, 3)0 }. (b) Looking at the columns of I3 − PS1 |S2 , we see that S2 is spanned by the vector (1, 3, 2)0 .

2.61 (a) Note that (PS1 |S2 + PT1 |T2 )2

= PS21 |S2 + PS1 |S2 PT1 |T2 + PT1 |T2 PS1 |S2 + PT21 |T2 = PS1 |S2 + PS1 |S2 PT1 |T2 + PT1 |T2 PS1 |S2 + PT1 |T2 .

When PS1 |S2 PT1 |T2 = PT1 |T2 PS1 |S2 = (0), this reduces to PS1 |S2 + PT1 |T2 so it follows that PS1 |S2 + PT1 |T2 is a projection matrix. Conversely, if PS1 |S2 + PT1 |T2 is a projection matrix, we must have PS1 |S2 PT1 |T2 + PT1 |T2 PS1 |S2 = (0).

Premultiplying this equation by PS1 |S2 leads to PS1 |S2 PT1 |T2 = −PS1 |S2 PT1 |T2 PS1 |S2 , while postmultiplying by PS1 |S2 leads to PT1 |T2 PS1 |S2 = −PS1 |S2 PT1 |T2 PS1 |S2 , 18

The three preceding equations together imply that PS1 |S2 PT1 |T2 = PT1 |T2 PS1 |S2 = (0). (b) Since PS1 |S2 + PT1 |T2 is a projection matrix, it can be written as PU |V . We first show that U = S1 + T1 . Now for any x ∈ Rm , (PS1 |S2 + PT1 |T2 )x = PS1 |S2 x + PT1 |T2 x, where PS1 |S2 x ∈ S1 and PT1 |T2 x ∈ T1 , so U ⊂ S1 + T1 . Also, any w ∈ S1 + T1 can be written w = x + y, where x ∈ S1 , y ∈ T1 , and so using the conditions in (a), we have w = x + y = PS1 |S2 x + PT1 |T2 y = (PS1 |S2 + PT1 |T2 )z, where z = PS1 |S2 x + PT1 |T2 y is some vector in Rm . Thus, w ∈ U and so S1 + T1 ⊂ U . This establishes that U = S1 + T1 , and in a similar manner we will show that V = S2 ∩ T2 . For any x ∈ V , x = (Im − PS1 |S2 − PT1 |T2 )x and, using the conditions in (a), (Im − PS1 |S2 )x

=

(Im − PS1 |S2 )(Im − PS1 |S2 − PT1 |T2 )x

=

(Im − PS1 |S2 − PT1 |T2 )x = x,

=

(Im − PT1 |T2 )(Im − PS1 |S2 − PT1 |T2 )x

=

(Im − PS1 |S2 − PT1 |T2 )x = x.

and (Im − PT1 |T2 )x

That is, x ∈ S2 and x ∈ T2 so V ⊂ S1 ∩ S2 . Conversely, suppose x ∈ S2 and x ∈ T2 , so that (Im − PS1 |S2 )x = x and (Im − PT1 |T2 )x = x. Then x

=

(Im − PS1 |S2 )x = (Im − PS1 |S2 )(Im − PT1 |T2 )x

=

(Im − PS1 |S2 − PT1 |T2 )x,

so x ∈ V and S2 ∩ T2 ⊂ V . This confirms that V = S2 ∩ T2 .

2.63 (a) Suppose xi ∈ S1 ∩ S2 for i = 1, 2. Since this implies that xi ∈ S1 and xi ∈ S2 for i = 1, 2, and S1 and S2 are convex sets, we have cx1 + (1 − c)x2 ∈ Sj , for j = 1, 2, where 0 < c < 1. This confirms that cx1 + (1 − c)x2 ∈ S1 ∩ S2 , and so S1 ∩ S2 is a convex set.

19

(b) Suppose z i ∈ S1 + S2 for i = 1, 2, so that there exist xi ∈ S1 and y i ∈ S2 for i = 1, 2, such that z i = xi + y i . Then, for 0 < c < 1, z∗

= cz 1 + (1 − c)z 2 = c(x1 + y 1 ) + (1 − c)(x2 + y 2 ) = {cx1 + (1 − c)x2 } + {cy 1 + (1 − c)y 2 } = x∗ + y ∗ .

Since S1 and S2 are convex sets, it follows that x∗ = cx1 +(1−c)x2 ∈ S1 and y ∗ = cy 1 +(1−c)y 2 ∈ S2 , so z ∗ ∈ S1 + S2 . Thus, S1 + S2 is a convex set.

2.65 Suppose x0i xi ≤ n−1 for i = 1, 2. Then, using the Cauchy-Schwarz inequality, we have for any 0 < c < 1, {cx1 + (1 − c)x2 }0 {cx1 + (1 − c)x2 }

=

c2 x01 x1 + 2c(1 − c)x01 x2 + (1 − c)2 x02 x2

≤

c2 x01 x1 + 2c(1 − c)(x01 x1 x02 x2 )1/2 + (1 − c)2 x02 x2

≤

n−1 {c2 + 2c(1 − c) + (1 − c)2 }

=

n−1 {c + (1 − c)}2 = n−1 ,

so Bn is convex.

Pr 2.67 Let x ∈ C(S). Then according to Problem 2.66, for some r, x = i=1 αi xi , where each xi ∈ S and Pr α1 , . . . , αr are nonnegative scalars satisfying i=1 αi = 1. We will show that if r > m + 1, then x can be expressed as a convex combination of r − 1 vectors in S. This can be used repeatedly until we have x expressed as a convex combination of m+1 vectors. Since r −1 > m, the vectors x2 −x1 , . . . , xr −x1 are linearly dependent and so there are scalars β2 , . . . , βr , not all zero, such that 0=

r X

βi (xi − x1 ) = −

i=2

where β1 = −

Pr

i=2 βi .

r X

βi x 1 +

i=2

Note that

r X

β i xi =

i=2

r X

βi x i ,

i=1

Pr

i=1 βi = 0 and since not all of the βi are 0, at least one βi is

positive. Thus, for any scalar γ, x=

r X i=1

αi xi =

r X

αi xi − γ

i=1

r X i=1

20

βi xi =

r X (αi − γβi )xi . i=1

Suppose we choose γ = min1≤i≤r (αi /βi : βi > 0) so that γ = αj /βj for some j. Then γ > 0, αi − γβi ≥ 0 for all i, and αj − γβj = 0. Thus, x=

r r X X (αi − γβi )xi = (αi − γβi )xi i=1

and

i=1 i6=j

r r r r X X X X (αi − γβi ) = (αi − γβi ) = αi − γ βi = 1, i=1 i6=j

i=1

i=1

i=1

so we have expressed x as a convex combination of r − 1 vectors in S.

21

Chapter 3

3.1 (a) The characteristic equation is 9−λ

−3

−4

12

−4 − λ

−6

8

−3

−3 − λ

= −(λ + 1)(λ − 1)(λ − 2) = 0,

so the eigenvalues of A are −1, 1, 2. (b) An eigenvector corresponding to the −1 eigenvalue satisfies the equation Ax = −x, which leads to the constraints x2 = 2x1 and x3 = x1 . Thus, an eigenvector has the form (x, 2x, x)0 and a unit eigenvector is √16 (1, 2, 1)0 . An eigenvector corresponding to the 1 eigenvalue satisfies the equation Ax = x, which leads to the constraints x2 = 0 and x3 = 2x1 . Thus, an eigenvector has the form (x, 0, 2x)0 and a unit eigenvector is √15 (1, 0, 2)0 . An eigenvector corresponding to the 2 eigenvalue satisfies the equation Ax = 2x, which leads to the constraints x2 = x1 and x3 = x1 . Thus, an eigenvector has the form (x, x, x)0 and a unit eigenvector is √13 (1, 1, 1)0 . (c) It follows from Theorem 3.4(a) and Theorem 3.5(a) that tr(A10 ) = (−1)10 + 110 + 210 = 1026.

3.3 The eigenvalues of A0 are the same as those of A, so from Problem 3.1 we have the eigenvalues −1, 1, 2.

An eigenvector corresponding to the −1 eigenvalue satisfies the equation A0 x = −x, which leads to the constraints x1 = −2x2 and x3 = x2 . Thus, SA0 (−1) = {(−2x, x, x)0 : −∞ < x < ∞}. An eigenvector corresponding to the 1 eigenvalue satisfies the equation A0 x = x, which leads to the constraints x2 = 0 and x3 = −x1 . Thus, SA0 (1) = {(x, 0, −x)0 : −∞ < x < ∞}. An eigenvector corresponding to the 2 eigenvalue satisfies the equation A0 x = 2x, which leads to the constraints x1 = −4x2 and x3 = 2x2 .

Thus, SA0 (2) = {(−4x, x, 2x)0 : −∞ < x < ∞}. While the eigenvalues of A0 are the same as those of A, we see that the eigenspaces of A0 are not the same as those of A. 3.5 (a) The characteristic equation is |A − λI4 | = (λ + 2)(λ + 1)(λ − 1)2 = 0, so the eigenvalues are −2, −1, and 1 with multiplicity two.

22

(b) The equation Ax = −2x leads to the constraints x2 = x4 = 0 and x3 = −x1 so SA (−2) = {(x, 0, −x, 0)0 : −∞ < x < ∞}. The equation Ax = −x leads to the constraints x1 = −2x3 , x2 = x3 and x4 = 0, so SA (−1) = {(−2x, x, x, 0)0 : −∞ < x < ∞}. The equation Ax = x leads to the constraints x1 = 2x3 and x2 = 3x3 , so SA (1) = {(2x, 3x, x, y)0 : −∞ < x < ∞ −∞ < y < ∞}.

3.7 (a) Since N ≥ 2k + 1, we can find an N × k matrix W satisfying W 0 [Z1

y] = (0). We can choose

the columns of W so that they are orthogonal and scaled so that W 0 W = γIk . Under these conditions, the ordinary least squares estimate of δ 1 in the model y = δ0 1N + (Z1 + W )δ 1 + is {(Z1 + W )0 (Z1 + W )}−1 (Z1 + W )0 y

=

(Z10 Z1 + Z10 W + W 0 Z1 + W 0 W )−1 (Z10 y + W 0 y)

=

(Z10 Z1 + γIk )−1 Z10 y.

(b) Let U be any k × k matrix satisfying U 0 U = γIk . Then the ordinary least squares estimate of δ 1 in the model

 

y 0





=

δ0 1N 0





+

Z1 U





 δ1 + 

∗

 

is   0  −1  0   Z y Z1 Z1  1          U 0 U U 

=

(Z10 Z1 + U 0 U )−1 Z10 y

=

(Z10 Z1 + γIk )−1 Z10 y.

3.9 Suppose B is nonsingular. Then since |B||B −1 | = 1, we have |AB − λIm | = |B||AB − λIm ||B −1 | = |BABB −1 − λBB −1 | = |BA − λIm |.

Thus, the characteristic equations of AB and BA are the same so they have the same eigenvalues.

Similarly, if A is nonsingular, we have |AB − λIm | = |A−1 ||AB − λIm ||A| = |A−1 ABA − λA−1 A| = |BA − λIm |.

23

3.11 (a) P is an orthogonal matrix since  cos2 θ + sin2 θ P 0P =  cos θ sin θ − cos θ sin θ

cos θ sin θ − cos θ sin θ cos2 θ + sin2 θ

  = I2 .

(b) The characteristic equation of P is |P − λI2 | = (cos θ − λ)2 + sin2 θ = λ2 − 2λ cos θ + 1, which yields the two solutions λ = cos θ + i sin θ and λ = cos θ − i sin θ.

3.13 (a) Note that |A0 − λIm | = |(A − λIm )0 | = |A − λIm |.

Thus, A0 and A have the same characteristic equation, and so they have the same eigenvalues. (b) A is singular if and only if its columns are linearly dependent, that is, if and only if Ax = 0 for some nonnull x. However, this is equivalent to saying A has a zero eigenvalue. (c) It follows from Problem 1.22 that the characteristic equation of a triangular matrix A is |A − λIm | =

m Y

(aii − λ),

i=1

thereby confirming that the diagonal elements of A are its eigenvalues. (d) Note that |BAB −1 − λIm | = |BAB −1 − λBB −1 | = |B||A − λIm ||B −1 | = |A − λIm |.

Consequently, BAB −1 and A have identical characteristic equations, and so the two matrices have identical eigenvalues. (e) Although A is a real matrix, an eigenvalue λ and corresponding eigenvector x of A could be complex. Since A0 A = Im and Ax = λx, we have x∗ x = x∗ A0 Ax = (Ax)∗ Ax = (λx)∗ λx = λ̄λx∗ x.

Note that x∗ x 6= 0, and so λ̄λ =1 or |λ| = 1.

24

3.15 It is easily verified that the matrices  A=

0

0

0

0





,

C=

0

1

0

0

 ,

both have eigenvalue 0 with multiplicity 2. Clearly, no B exists for which C = BAB −1 since rank(A) = 0 and rank(C) = 1.

3.17 If rank(A−λIm ) = m−1, then there is one linearly independent vector x satisfying (A−λIm )x = 0 or, equivalently, Ax = λx. Thus, λ is an eigenvalue of A and dim{SA (λ)} = 1. It follows from Theorem 3.3 that the multiplicity of the eigenvalue λ is at least one.

3.19 Since A is a triangular matrix, it follows from Theorem 3.2(c) that it has eigenvalue 1 with multiplicity m. The eigenvalue-eigenvector equation, Ax = x, leads to the m equations xi + xi+1 = xi , for i = 1, . . . , m − 1, and xm = xm . It follows that xi+1 = 0 for i = 1, . . . , m − 1 and x1 is arbitrary. Thus, there is only one linearly independent eigenvector of the form (x, 0, . . . , 0)0 .

3.21 (a) Since (Im + A)xi = xi + Axi = xi + λi xi = (1 + λi )xi , 1 + λi is an eigenvalue of Im + A corresponding to the eigenvector xi . From Theorem 3.4(b), it then follows that (1 + λi )−1 is an eigenvalue of (Im + A)−1 corresponding to the eigenvector xi . (b) Since −1 (A + A−1 )xi = Axi + A−1 xi = λi xi + λ−1 i xi = (λi + λi )xi ,

λi + λ−1 is an eigenvalue of A + A−1 corresponding to the eigenvector xi . i (c) Since −1 (Im + A−1 )xi = xi + A−1 xi = xi + λ−1 i xi = (1 + λi )xi ,

1 + λ−1 is an eigenvalue of Im + A−1 corresponding to the eigenvector xi . i 3.23 (a) Using Theorem 3.4(b) (see also Problem 3.21), we have Bx

(Im + A)−1 x + (Im + A−1 )−1 x = (1 + λ)−1 x + (1 + λ−1 )−1 x 1 + λ−1 + 1 + λ = x = x, (1 + λ)(1 + λ−1 )

=

so 1 is an eigenvalue of B corresponding to the eigenvector x.

25

(b) An application of Theorem 1.9 reveals that (Im + A)−1 = Im − (Im + A−1 )−1 from which it immediately follows that B = Im .

3.25 (a) The characteristic equation of A is |A − λI2 | = λ2 − (a11 + a22 )λ + (a11 a22 − a12 a21 ). (b) Using the quadratic formula, we have p p a11 + a22 ± (a11 − a22 )2 + 4a12 a21 a11 + a22 ± (a11 + a22 )2 − 4(a11 a22 − a12 a21 ) = .

2 2 (c) The eigenvalues are real when (a11 − a22 )2 + 4a12 a21 ≥ 0.

3.27 Note that 1m 10m 1m = m1m , while 1m 10m x = 0x for any x orthogonal to 1m . Thus, m is an eigenvalue with corresponding eigenvector being any vector in the space S spanned by 1m , and 0 is an eigenvalue, having multiplicity m − 1, with corresponding eigenvector being any vector in S ⊥ .

3.29 (a) Note that (αIm + β1m 10m )1m = (α + mβ)1m , while (αIm + β1m 10m )x = αx for any x orthogonal to 1m . Thus, α + mβ is an eigenvalue with corresponding eigenvector being any vector in the space S spanned by 1m , and α is an eigenvalue, having multiplicity m − 1, with corresponding eigenvector being any vector in S ⊥ . (b) The eigenspaces are S and S ⊥ as indicated in part (a). The eigenprojection corresponding to the eigenvalue α + mβ is 1 1m 10m , m while the eigenprojection corresponding to the eigenvalue α is PS =

PS ⊥ = Im − PS = Im −

1 1m 10m . m

(c) None of the eigenvalues can be 0, so we must have α 6= 0 and β 6= −α/m. (d) We have A

−1

1 0 = (α + mβ) Im − 1m 1m m m

1 1 1 = α−1 Im + − 1m 10m α + mβ α m β = α−1 Im − 1m 10m . α(α + mβ) −1 1

26

1m 10m + α−1

(e) The determinant is the product of the eigenvalues so |A| = αm−1 (α + mβ).

3.31 (a) The characteristic equation of A is |A − λI3 | = −λ(λ − 2)(λ − 3) = 0, so A has eigenvalues 0, 2, 3. The equation Ax = 0 leads to the constraints x1 = x2 /2 and x3 = −x2 /2, so a normalized eigenvector corresponding to 0 is √16 (1, 2, −1)0 . The equation Ax = 2x leads to the constraints x1 = x3 and x2 = 0, so a normalized eigenvector corresponding to 2 is √12 (1, 0, 1)0 . The equation Ax = 3x leads to the constraints x1 = −x2 and x3 = x2 , so a normalized eigenvector corresponding to 3 is √13 (1, −1, −1)0 . (b) The rank of A is 2 since it is symmetric and has two nonzero eigenvalues. (c) The eigenspaces are SA (0) = {(x, 2x, −x)0 : −∞ < x < ∞}, SA (2) = {(x, 0, x)0 : −∞ < x < ∞}, SA (3) = {(x, −x, −x)0 : −∞ < x < ∞}, while the corresponding eigenprojections are    1 2 −1 1  1 1    PSA (0) =  2 4 −2  , PSA (2) =  0 6 2  −1 −2 1 1

0 0 0

1





1

 1   , P =   −1 0 SA (3) 3  1 −1

(d) tr(A4 ) = 04 + 24 + 34 = 97.

3.33 It follows from Theorem 3.4(a) that tr(A2 ) =

m X

λ2i .

i=1

But since A is symmetric, we also have tr(A2 ) = tr(A0 A) =

m X

(A0 A)jj =

j=1

m m X m X X (A0 )j· (A)·j = a2ij . j=1

Equating these two equations gives the result.

27

i=1 j=1

−1 1 1

−1



  1 .  1

3.35 Let λ1 , . . . , λm denote the eigenvalues of A. Using Problem 3.33, we have m X

λ2i

=

i=1

m X m X

a2ij =

i=1 j=1

=

m X

λ2i +

i=1

m X

a2ii +

i=1 m X m X

m X m X

a2ij

i=1 j=1 j6=i

a2ij .

i=1 j=1 j6=i

Thus, the final double sum must reduce to zero and this requires aij = 0 for all i 6= j.

3.37 Since A0 A is symmetric and rank(A0 A) = rank(A) = r, A0 A can be written as    0 Λ (0) U  , A0 A = [U X]  (0) (0) X0 where Λ is an r × r diagonal matrix, U and X are n × r and n × (n − r) matrices satisfying U 0 U = Ir , X 0 X = In−r , and X 0 U = (0). It follows that X 0 A0 AX = (0) which leads to AX = (0). Similarly, since AA0 is symmetric and rank(AA0 ) = rank(A) = r, AA0 can be written as    Λ (0) V  , AA0 = [V 0 Y 0 ]  (0) (0) Y where Λ is an r × r diagonal matrix, V 0 and Y 0 are m × r and m × (m − r) matrices satisfying V V 0 = Ir , Y Y 0 = Im−r , and V Y 0 = (0). It follows that Y AA0 Y 0 = (0) which leads to Y A = (0).

3.39 (a) Let λ1 , . . . , λm be the eigenvalues of A, so that by Theorem 3.4(a), λk1 , . . . , λkm are the eigenvalues of Ak . But Ak = (0), so every eigenvalue of Ak is 0. That is, λki = 0 for all i, so we must have λi = 0 for all i. (b) The matrix  A=

0

1

0

0

 

satisfies A2 = (0).

3.41 We will prove the max-min result; the proof of the min-max result is similar. Note that we may Pk 0 without loss of generality consider the max-min of the sum j=1 xj Axj if we restrict each xj to be a unit vector. Now suppose the columns of Cij are eigenvectors corresponding to the eigenvalues λij +1 , . . . , λm . Then Ci0j xj = 0 implies that xj is a linear combination of orthonormal eigenvectors 28

corresponding to the eigenvalues λ1 , . . . , λij , and so we must have x0j Axj ≥ λij . Thus, with this particular choice for Ci1 , . . . , Cik , we get k X

k X

x0j Axj ≥

j=1

λij .

j=1

Pk

That is, the max-min must be greater than or equal to

j=1 λij . As a result, the proof will be complete Pk if we can show that the max-min is less than or equal to j=1 λij . We prove this result by induction

on m. Note that the result is trivially true when k = m. In particular, the result is true when m = 1.

Now suppose m > 1, k < m, and the result holds for (m − 1) × (m − 1) matrices. Let Ci1 , . . . , Cik be given matrices satisfying the conditions of the theorem. We need to show that there are orthonormal Pk Pk vectors x1 , . . . , xk , with Ci0j xj = 0, such that j=1 x0j Axj ≤ j=1 λij .

First suppose that ik < m so that dim{N (Ci0k )} ≤ m − 1. Let the columns of Y = (y 1 , . . . , y m−1 ) be orthonormal vectors satisfying Ci0j y l = 0 for l = 1, . . . , ij and j = 1, . . . , k. Then if µ1 , . . . , µm−1 are the eigenvalues of B = Y 0 AY , it follows from Theorem 3.19 that µi ≤ λi ,

for i = 1, . . . , m − 1.

(1)

Let Dij = Y 0 Cij , so that if Di0j u = 0, then Ci0j w = 0 and u0 Bu = w0 Aw, where w = Y u. Thus, since B is (m − 1) × (m − 1), we know by our induction hypothesis that there are orthonormal vectors u1 , . . . , uk satisfying Di0j uj = 0 and k X

k X

u0j Buj ≤

j=1

Thus, if wj = Y uj , then Ci0j wj = 0,

µij .

(2)

j=1

Pk

0 j=1 uj Buj =

Pk

k X

k X

w0j Awj ≤

j=1

0 j=1 w j Aw j , and so from (1) and (2)

λ ij ,

j=1

as required.

Next suppose that ik = m and let l be the largest index for which il +1 < il+1 . We know such an l exists since k < m. Let Sm−1 be an (m − 1)-dimensional space that contains N (Ci0l ) and the eigenvectors of A corresponding to the eigenvalues λil+1 , . . . , λm ; there is such a space since m − il+1 + 1 + il < m. By the definition of l, it follows that il+1 , . . . , m are among the indices i1 , . . . , ik and 0 ) ∩ Sm−1 ⊂ Sm−1 .

N (Ci0l ) ⊂ N (Ci0l+1 ) ∩ Sm−1 ⊂ · · · ⊂ N (Cm−1

29

Since for j = il+1 , . . . , m − 1, dim{N (Cj0 ) ∩ Sm−1 ) ≥ j − 1, we can find subspaces Sil+1 −1 , . . . , Sm−2 such that dim(Sj ) = j, 0 Sil+1 −1 ⊂ N (Ci0l+1 ), . . . , Sm−2 ⊂ N (Cm−1 )

and N (Ci01 ) ⊂ · · · ⊂ N (Ci0l ) ⊂ Sil+1 −1 ⊂ · · · ⊂ Sm−2 ⊂ Sm−1 .

Consider orthonormal vectors x1 , . . . , xk that satisfy xj ∈ N (Ci0j ) for j = 1, . . . , l and xj ∈ Sij −1 for j = l + 1, . . . , k. Since each xj ∈ Sm−1 , there exist an m × (m − 1) semiorthogonal matrix Y and orthonormal (m − 1) × 1 vectors u1 , . . . , uk such that xj = Y uj . As a result, by using the induction hypothesis on the matrix B = Y 0 AY , we have k X

x0j Axj =

j=1

k X

u0j Buj ≤

j=1

l X

m−1 X

µij +

µj ≤

j=il+1 −1

j=1

l X

λij +

j=1

m−1 X

µj ,

(3)

j=il+1 −1

where the second inequality utilized Theorem 3.19. Since the eigenvectors of A corresponding to the eigenvalues λil+1 , . . . λm are in Sm−1 , it follows that λil+1 , . . . λm are also eigenvalues of B. Since µil+1 −1 , . . . , µm−1 are the smallest eigenvalues of B, this then guanantees that m−1 X

µj ≤

j=il+1 −1

m X

λj ,

j=il+1

and when this is substituted into (3), we get the desired result.

3.43 Consider the matrices

 A=

2

0

1

2





,

B=

2

1

0

2

 .

Both matrices have the eigenvalue 2 with multiplicity 2. Theorem 3.21 would hold only if A + B had the eigenvalue 4 with multiplicity 2. However, The eigenvalues of A + B are 3 and 5.

3.45 For any m × (m − h) matrix Ch satisfying Ch0 Ch = Im−h , we have 0

x Ax x0 Bx x0 (A + B)x = min + min 0 0 x0 x x0 x x0 x Ch x=0 Ch x=0 x6=0

x6=0

≥

x0 Ax x0 Bx + min 0 0 Ch x=0 x x Ch x=0 x0 x min 0 x6=0

x6=0 0

≥

x Ax x0 Bx + min 0 0 x6=0 x x Ch x=0 x x min 0 x6=0

30

=

x0 Ax + λm (B) Ch x=0 x0 x min 0 x6=0

≥

x0 Ax , Ch x=0 x0 x min 0 x6=0

where the last equality follows from Theorem 3.16. Now maximizing both sides of the equation above over all choices of Ch satisfying Ch0 Ch = Im−h and using (3.10) of Theorem 3.18, we get λh (A + B)

=

x0 (A + B)x x0 x Ch x=0

max min 0 Ch

x6=0

≥

x0 Ax = λh (A).

Ch x=0 x0 x

max min 0 Ch

x6=0

3.47 Define G, E, T and H as in the proof of Theorem 3.31 so that A = GΛG0 , B = GG0 , E = G0 F , E 0 E = T 2 , and H = ET −1 . (a) Using the lower bound in Theorem 3.19, we have λh−i+1 {(F 0 BF )−1 (F 0 AF )}

= λh−i+1 {(F 0 GG0 F )−1 F 0 GΛG0 F } = λh−i+1 {(T 2 )−1 E 0 ΛE} = λh−i+1 (T −1 E 0 ΛET −1 ) = λh−i+1 (H 0 ΛH) ≥ λm−h+(h−i+1) (Λ) = λm−i+1 (Λ) = λm−i+1 (B −1 A).

(b) We have min λ1 {(F 0 BF )−1 (F 0 AF )} F

=

min λ1 {(E 0 E)−1 E 0 ΛE}

=

min λ1 (T −1 E 0 ΛET −1 ) = min λ1 (H 0 ΛH)

E

E

H

= λm−h+1 (B

−1

A),

where the last equality follows from the lower bound in Theorem 3.19 and the fact that the bound is attained when H 0 = [(0) Ih ]. (c) We have min λh {(F 0 BF )−1 (F 0 AF )} F

=

min λh {(E 0 E)−1 E 0 ΛE}

=

min λh (T −1 E 0 ΛET −1 ) = min λh (H 0 ΛH)

E

E

= λm (B 31

H

−1

A),

where the last equality follows from the lower bound in Theorem 3.19 and the fact that the bound is attained when H 0 = [(0) Ih ].

3.49 (a) Now  TT0 = 

6

3

3

6

 

and |T T 0 − λI2 | = (λ − 3)(λ − 9), so the eigenvalues of T T 0 are 3 and 9. The equation T T 0 x = 3x yields the constraint x2 = −x1 , so an eigenvector corresponding to 3 has the form (x, −x)0 . The equation T T 0 x = 9x yields the constraint x2 = x1 , so an eigenvector corresponding to 9 has the form (x, x)0 . (b) We have 

5

  T 0T =  1  4

1 2 −1

4



  −1  .  5

The positive eigenvalues of T T 0 and T 0 T are the same, so T 0 T has eigenvalues 0, 3, and 9. The equation T 0 T x = 0x yields the constraints x3 = x2 = −x1 , so an eigenvector corresponding to 0 has the form (x, −x, −x)0 . The equation T 0 T x = 3x yields the constraints x2 = 2x1 and x3 = −x1 , so an eigenvector corresponding to 3 has the form (x, 2x, −x)0 . The equation T 0 T x = 9x yields the constraints x2 = 0 and x3 = x1 , so an eigenvector corresponding to 9 has the form (x, 0, x)0 .

3.51 Using the spectral decomposition of A, we have A = XΛX 0 = (XΛ1/2 )(Λ1/2 X 0 ) = Y 0 Y, where Y = (y 1 , . . . , y m ) = Λ1/2 X 0 . Note that aii = y 0i y i and so aii = 0 implies y i = 0. But this implies aij = aji = 0 for any j since aij = aji = y 0i y j .

3.53 (a) Since (A + B)xi = Axi + Bxi = λi xi + γi xi = (λi + γi )xi , it follows that λ1 + γ1 , . . . , λm + γm are the eigenvalues of A + B with corresponding eigenvectors x1 , . . . , xm .

32

(b) Since ABxi = Aγi xi = γi Axi = λi γi xi , it follows that λ1 γ1 , . . . , λm γm are the eigenvalues of AB with x1 , . . . , xm as the corresponding eigenvectors. (c) The spectral decompositions of A and B have a common orthogonal matrix, that is, they can be written as A = XΛX 0 and B = XΓX 0 . The matrices Λ and Γ are diagonal so ΛΓ = ΓΛ. As a result, AB = XΛX 0 XΓX 0 = XΛΓX 0 = XΓΛX 0 = XΓX 0 XΛX 0 = BA. 3.55 Let X = (X1 , X2 ) and Λ = diag(Λ1 , Λ2 ), where X1 = (x1 , . . . , xk−1 ), X2 = (xk , . . . , xm ), Λ1 = diag(λ1 , . . . , λk−1 ), and Λ2 = diag(λk , . . . , λm ). Note that A − λIm

= XΛX 0 − λXX 0 = X1 Λ1 X10 + X2 Λ2 X20 − λX1 X10 − λX2 X20 = X1 (Λ1 − λIk−1 )X10 + X2 (Λ2 − λIm−k+1 )X20 .

Since P = X2 X20 , we have P (A − λIm )P = X2 X20 (A − λIm )X2 X20 = X2 (Λ2 − λIm−k+1 )X20 ,

(1)

which clearly is (0) when λk = · · · = λm = λ. Conversely, if P (A − λIm )P = (0), then premultiplying (1) by X20 and postmultiplying by X2 confirms that λk = · · · = λm = λ.

3.57 (a) Since tr(B 0 AB) =

Pr

0 i=1 λi (B AB), it follows from Theorem 3.19 that r X

λm−i+1 ≤ tr(B 0 AB) ≤

r X

λi ,

i=1

i=1

if B 0 B = Ir . Note that the lower bound is attained when B contains as its columns, normalized eigenvectors of A corresponding to the eigenvalues λm−r+1 , . . . , λm , and so we must have min tr(B 0 AB) = 0

B B=Ir

r X

λm−i+1 .

i=1

Similarly, the upper bound is attained when B contains as its columns, normalized eigenvectors of A corresponding to the eigenvalues λ1 , . . . , λr , and so we must have max tr(B 0 AB) = 0

B B=Ir

r X i=1

33

λi .

(b) Using the bounds given in part (a) and the choice of B = (e1 , . . . , er ), the result follows 3.59 Using the spectral decomposition of C, we have C = XΛX 0 = XΛ1/2 Λ1/2 X 0 = T T 0 , where the m × m matrix T = XΛ1/2 is nonsingular since C is positive definite. (a) Using Theorem 3.2(d) and Theorem 3.32(a), we find λi (AB)

= λi (T −1 ABT ) = λi (T −1 AT −10 T 0 BT ) ≤ λj (T −1 AT −10 )λk (T 0 BT ) = λj (AT −10 T −1 )λk (T T 0 B) = λj (AC −1 )λk (CB).

(b) Using Theorem 3.2(d) and Theorem 3.32(b), we find λm−i+1 (AB)

=

λm−i+1 (T −1 ABT ) = λm−i+1 (T −1 AT −10 T 0 BT )

≥

λm−j+1 (T −1 AT −10 )λm−k+1 (T 0 BT ) = λm−j+1 (AT −10 T −1 )λm−k+1 (T T 0 B)

=

λm−j+1 (AC −1 )λm−k+1 (CB).

3.61 For a proof of this result, see Khattree, R., 2002, Generalized antieigenvalues and antieigenvectors, American Journal of Mathematical and Management Sciences 22, 89–98.

34

Chapter 4

4.1 Now

 AA0 = 

10

4

4

4

 ,

and |AA0 − λI2 | = (12 − λ)(2 − λ), so the eigenvalues of AA0 are 2 and 12. The equation AA0 x = 2x leads to the constraint x2 = −2x1 , so an eigenvector corresponding to 2 has the form (x, −2x)0 . The equation AA0 x = 12x leads to the constraint x1 = 2x2 , so an eigenvector corresponding to 12 has the form (2x, x)0 . The positive eigenvalues of 

2

   3 A0 A =    3  0

3

3

5

5

5

5

1

1

0



  1    1   2

are also 2 and 12. The equation A0 Ax = 2x leads to the constraints x2 = x3 = 0 and x4 = −3x1 , so an eigenvector corresponding to 2 has the form (x, 0, 0, −3x)0 . The equation A0 Ax = 12x leads to the constraints x1 = 3x4 and x2 = x3 = 5x4 , so an eigenvector corresponding to 12 has the form (3x, 5x, 5x, x)0 . As a result, the singular value decomposition of A is given by   √ √  √ √ 1/ 5 2/ 5 2 0 0 0 −1/ 10  A= √ √ √ √  √ √ −2/ 5 1/ 5 0 3/ 60 5/ 60 5/ 60 12

 √ 3/ 10 . √ 1/ 60

4.3 A has a zero eigenvalue if and only if rank(A) = r < m, and this condition is equivalent to rank(AA0 ) = r < m which hold if and only of A has r singular values.

4.5 We prove the result when m ≥ n. The proof for m < n is similar. Let ∆ be the n × n matrix ∆ = diag(µ1 , . . . , µr , 0, . . . , 0). Then the singular value decomposition of A can be written     ∆ ∆  Q0 = [P1 P2 ]   Q0 = P1 ∆Q0 , A=P (0) (0) where P1 is m × n and P2 is m × (m − n). Let  √ P1 / 2 U = √ Q/ 2 35

√ −P1 / 2 √ Q/ 2

P2 (0)

 .

Then U U 0 = U 0 U = Im+n and 

∆

  U  (0)  (0)

(0)

(0)



   0  (0) −∆ (0)  U =  A0 (0) (0)

A (0)

 ,

and so the result follows.

4.7 Let A = xy 0 . Note that AA0 x = (x0 x)(y 0 y)x, and A0 Ay = (x0 x)(y 0 y)y.

Thus, (x0 x)(y 0 y) is the one positive eigenvalue of AA0 and A0 A with corresponding unit eigenvectors p given by p = (x0 x)−1/2 x and q = (y 0 y)−1/2 y. The singular value is d = (x0 x)(y 0 y) and the singular value decomposition is then given by A = xy 0 = pdq 0 .

4.9 If A = P1 ∆Q01 and B = Q1 ∆−1 P10 , then since P10 P1 = Q01 Q1 = Ir , ABA = P1 ∆Q01 Q1 ∆−1 P10 P1 ∆Q01 = P1 ∆∆−1 ∆Q01 = P1 ∆Q01 = A, and BAB = Q1 ∆−1 P10 P1 ∆Q01 Q1 ∆−1 P10 = Q1 ∆−1 ∆∆−1 P10 = Q1 ∆−1 P10 = B. 4.11 (a) Note that ŷ can be expressed as ŷ = {N −1 10N + z 0 (Z10 Z1 )−1 Z10 }y.

Since E(ŷ) = δ0 + z 0 δ 1 = θ, Z10 Z1 = U D2 U 0 , and v = D−1 U 0 z, we have MSE(ŷ)

=

var(ŷ) = {N −1 10N + z 0 (Z10 Z1 )−1 Z10 }(σ 2 IN ){N −1 10N + z 0 (Z10 Z1 )−1 Z10 }0

= σ 2 {N −1 + z 0 (Z10 Z1 )−1 z} = σ 2 {N −1 + z 0 U D−2 U 0 z} ! k X 2 −1 0 2 −1 2 = σ {N + v v} = σ N + vi . i=1

36

(b) Note that since V10 1N = 0, ȳ and z 0 U1 D1−1 V10 y are independently distributed, and so var(ỹ)

=

var(ȳ) + var(z 0 U1 D1−1 V10 y)

=

N −1 σ 2 + (z 0 U1 D1−1 V10 )(σ 2 IN )(z 0 U1 D1−1 V10 )0

=

σ 2 (N −1 + z 0 U1 D1−2 U10 z) 2

= σ (N

−1

+ v 01 v 1 ) = σ 2

N

−1

+

k−r X

! vi2

,

i=1

where v = (v 01 , v 02 )0 and v 1 is (k − r) × 1. We also have (see the solution to Problem 3.8) E(ỹ) = E(ȳ) + z 0 U1 D1−1 V10 E(y) = δ0 + z 0 δ 1 − z 0 U2 D2−2 U20 Z10 E(y),

(1)

and so {E(ỹ) − θ}2

=

(z 0 U2 D2−2 U20 Z10 Z1 δ 1 )2 = (z 0 U2 D2−2 U20 U D2 U 0 δ 1 )2

=

([(0)

=

z 0 U2 D2−2 ]D2 U 0 δ 1 )2 = (z 0 U2 D2−1 D2 U20 δ 1 )2 !2 k X (v 02 D2 α12 )2 = di vi αi ,

(2)

i=k−r+1

where α1 = (α011 , α012 )0 = U 0 δ 1 . Combining (1) and (2) yields the result. (c) MSE(ỹ) <MSE(ŷ) when d2k vk2 αk2 < σ 2 vk2 , that is, when d2k αk2 < σ 2 .

4.13 (a) The characteristic equation of A is |A − λI3 | = −(λ − 4)2 (λ − 1) = 0, so the eigenvalues of A are 4 with multiplicity two and 1. The equation Ax = x leads to the constraints x2 = −x1 and x3 = x1 , so an eigenvector corresponding to 1 has the form (x, −x, x)0 .

The equation Ax = 4x leads to the constraint x3 = x2 − x1 . Consequently, two orthogonal eigenvectors corresponding to 4 are of the form (x, x, 0)0 and (x, −x, −2x)0 . As a result, the spectral decomposition of A can be expressed as A = XΛX 0 , where   √ √  √ 1 1/ 3 1/ 2 1/ 6   √ √ √     Λ= 0 X =  −1/ 3 1/ 2 −1/ 6  ,   √ √  0 −2/ 6 0 1/ 3

37

0 4 0

0



  0 .  4

(b) The symmetric square root of A is given by  XΛ1/2 X 0 =

5

1   1 3 −1

1 −1 5 1



  1 .  5

(c) A nonsymmetric square root matrix of A has the form A = XΛ1/2 Q0 , where Q is an orthogonal matrix other than X. For instance, when Q = I3 , we get  √ √ √  1/ 3 2 2/ 6  √ √ √    A1/2 =  −1/ 3 2 −2/ 6  .  √  √ 0 −4/ 6 1/ 3 4.15 This is an application of the previous problem. The characteristic equation of A is |A − λI3 | = −λ(λ − 5)(λ − 10) = 0, so the nonzero eigenvalues of A are 5 and 10. The equation Ax = 5x leads to the constraints x2 = 0 and 3x3 = −4x1 , so an eigenvector corresponding to 5 has the form (3x, 0, −4x)0 . The equation Ax = 10x leads to the constraints 4x2 = 5x1 and 4x3 = 3x1 , so an eigenvector corresponding to 10 has the form (4x, 5x, 3x)0 . Consequently, we can compute T as   √ 3/5 4/ 50  √   5 √   T = 0 5/ 50     0 √ −4/5 3/ 50

 

3

0 1    √ =  0 √  5 10 −4

4



  5 .  3

4.17 (a) An eigenanalysis of A reveals that it has eigenvalue 0 with multiplicity two and eigenvalue 3.

By inspection of the equation Ax = 0, we find any eigenvector corresponding to 0 has the form (−x, −2x, x)0 . Since there is only one linearly independent eigenvector corresponding to the 0 eigenvalue, A is not diagonalizable. It is easily shown that B has eigenvalues 0 and 1 with multiplicity two. The equation Bx = x yields the single constraint x2 = x1 , so an eigenvector corresponding to 1 has the form (x, x, y)0 . Since there are two linearly independent vectors of this form, B is diagonalizable. Finally, an eigenanalysis of C reveals that it has eigenvalue 4 with multiplicity two and eigenvalue 0. By inspection of the equation Cx = 4x, we find any eigenvector corresponding to 4 has the form (x, x, −x)0 . Since there is only one linearly independent eigenvector corresponding to the 4 eigenvalue, C is not diagonalizable.

38

(b) Each matrix has rank of 2 so B and C have their rank equal to the number of nonzero eigenvalues.

4.19 Suppose A is nonsingular. Since AB is diagonalizable, there exist an m × m nonsingular matrix X and an m × m diagonal matrix D such that AB = XDX −1 .

Thus, B = A−1 XDX −1 and BA = A−1 XDX −1 A = Y DY −1 , where Y = A−1 X. This confirms that BA is diagonalizable. A similar proof can be constructed for the case in which B is nonsingular. Next consider the 2 × 2 matrices     0 1 1 0 , .

A= B= 0 0 0 0 Then AB = (0) is a diagonal matrix, while BA = A is not diagonalizable.

4.21 Let x1 , . . . , xm be eigenvectors of A corresponding to the eigenvalues λ1 , . . . , λm , and let y 1 , . . . , y n be eigenvectors of B corresponding to the eigenvalues γ1 , . . . , γn . It follows from Problem 3.48 that the eigenvectors of C are (x01 , 00 )0 , . . . , (x0m , 00 ), (00 , y 1 )0 , . . . , (00 , y n )0 , and clearly these vectors are linearly independent if and only if the xi ’s are linearly independent and the y j ’s are linearly independent. This confirms that C is diagonalizable if and only if A and B are diagonalizable. The induction proof for the general case is immediate since we can write diag(A1 , . . . , Ak ) = diag(A, B), where A = diag(A1 , . . . , Ak−1 ) and B = Ak , and then again use the proof given for the 2 × 2 form.

4.23 (a) Note that y 0i xj = (Y 0 X)ij = (X −1 X)ij = (Im )ij , so y 0i xj = 0 if i 6= j and y 0i xj = 1 if i = j. Now PA (µi )PA (µj ) =

rj ri X X

xMi +k y 0Mi +k xMj +l y 0Mj +l .

(1)

k=1 l=1

Since the sets {Mi + 1, . . . , Mi + ri } and {Mj + 1, . . . , Mj + rj } have no elements in common when i 6= j, (1) reduces to (0) when i 6= j.

39

(b) We have h X

PA (µi )

=

i=1

ri h X X

xMi +j y 0Mi +j =

i=1 j=1

=

m X

xl y 0l

l=1

(X)·l (Y 0 )l· =

l=1

= XX

m X

m X

(X)·l (X −1 )l·

l=1 −1

= Im .

(c) Write Xi = (xMi +1 , . . . , xMi +ri ) and Yi = (y Mi +1 , . . . , y Mi +ri ), so that PA (µi ) = Xi Yi0 . Note that the null space of (A − µi Im ) is the eigenspace of µi so that N (A − µi Im ) = R(Xi ). Since Yi0 Xi = Iri follows from Y 0 X = X −1 X = Im , we have R{PA (µi )} = R(Xi Yi0 ) ⊆ R(Xi ) = R(Xi Yi0 Xi ) = R{PA (µi )Xi } ⊆ R{PA (µi )}.

Thus, R{PA (µi )} = N (A − µi Im ), that is, PA (µi ) is a projection matrix for the null space of (A − µi Im ). All that remains is to show that this projection is along the column space of (A − µi Im ) by showing N {PA (µi )} = R(A − µi Im ). Now using part (a), we have PA (µi )(A − µi Im )

=

h X

µj PA (µi )PA (µj ) − µi PA (µi )

j=1

= µi PA2 (µi ) − µi PA (µi ) = µi PA (µi ) − µi PA (µi ) = (0), so R(A − µi Im ) ⊆ N {PA (µi )}.

(2)

But since R{PA (µi )} = N (A − µi Im ), dim{R(A − µi Im )} = m − dim{N (A − µi Im )} = m − dim(R{PA (µi )}) = dim(N {PA (µi )}).

This along with (2) confirms that N {PA (µi )} = R(A − µi Im ) and so the proof is complete.

4.25 (a) 

0

   0    0  0

40

0

0

0

0

0

0

0

0

0



  0  .  0   1

(b) 

0

   0    0  0

0



0

0

0

1

0

0

0

0

  0  .  0   1

1

0

0

0

1

0

0

0

0

(c) 

0

   0    0  0



  0  .  0   1

4.27 (a) Inspection of the diagonal elements reveals that the eigenvalues are 2 and 3 with multiplicities four and two. (b) The two eigenspaces are SJ (2) = {(a, 0, b, 0, 0, 0)0 : −∞ < a < ∞, −∞ < b < ∞}, SJ (4) = {(0, 0, 0, 0, a, 0)0 : −∞ < a < ∞}.

4.29 Since (A − λI5 )2 = (0), the Jordan canonical form has no hi larger than 2. Thus, the forms from Problem 4.26 satisfying this constraint are diag(J1 (λ), J1 (λ), J1 (λ), J1 (λ), J1 (λ)), diag(J2 (λ), J1 (λ), J1 (λ), J1 (λ)), diag(J2 (λ), J2 (λ), J1 (λ)).

4.31 Let J = D+B be as in part (b) of Problem 4.28. According to Theorem 4.11, there exists a nonsingular matrix X such that A = XJX −1 . Since all of the eigenvalues of A are zero, D = (0), and so A = XBX −1 . As a result, Am = XB m X −1 . But from Problem 4.28 part (b), we know that B h = (0) if h ≥ maxi hi . Since maxi hi ≤ m, we must have B m = (0), and hence Am = (0).

4.33 Let A1 = A − λIm . Note that λ is an eigenvalue of A with multiplicity r if and only if A1 has m − r nonzero eigenvalues. But from Problem 4.32, rank(A1 ) = rank(A21 ) is equivalent to the condition rank(A1 ) = rank(A − λIm ) = m − r. The result now follows from Theorem 4.8.

41

4.35 Since V = T U is upper triangular, it follows that vij = 0 for 1 ≤ j < i ≤ r + 1. Now for 1 ≤ i ≤ j ≤ r, note that vij

=

(T U )ij = (T )i· (U )·j =

m X

til ulj

l=1 j X

=

til ulj = 0,

l=i

where the fourth equality follows from the fact that T and U are upper triangular and the final equality follows since tij = 0 for 1 ≤ i ≤ r, 1 ≤ j ≤ r. When 1 ≤ i ≤ j = r + 1, we have vij =

j X

til ulj = tir+1 ur+1r+1 = 0,

l=i

since ur+1r+1 = 0. √ √ √ 4.37 In Problem 4.17, we saw that (1/ 3, 1/ 3, −1/ 3)0 is a normalized eigenvector of C corresponding to the eigenvalue 4. Define

√ 1/ 3  √  Y =  1/ 3  √ −1/ 3

√ 1/ 2 √ −1/ 2



and note that

0

4

√ −1/ 6

  Y 0 CY =  0  0

2 √ −2/ 3



√  1/ 6 √   1/ 6  , √  2/ 6 √  5/ 2 √   −6/ 3  .  2

A normalized eigenvector of

√  2 −6/ 3  B= √ −2/ 3 2 √ corresponding to the eigenvalue 0 is ( 3/2, 1/2)0 and  √  0 −4/ 3  W 0 BW =  0 4 

when

 √ W =

3/2

1/2

42

 −1/2 . √ 3/2

A Schur decomposition of C is then given as C = XT X 0 , where  √ √   1/ 3 2/ 6  1 00 √ √ = X=Y   1/ 3 −1/ 6  0 W √ √ −1/ 3 1/ 6 and



√

4

  T = 0  0 4.39 Note that X

|tij |2 =

so that

2

i<j |tij |

|tii |2 +

i=1

i≤j

P

m X

2

0 0

X

√  8/ 6 √   −4/ 3  .  4

|tij |2 =

i<j

is uniquely defined if and only if

 0 √   1/ 2  , √  1/ 2

m X

|λi (A)|2 +

i=1

P

i≤j |tij |

X

|tij |2 ,

i<j 2

is uniquely defined. Let A = XT X ∗ be

a Schur decomposition of A so that T = X ∗ AX. The result follows since X

|tij |2

=

tr(T ∗ T ) = tr{(X ∗ AX)∗ X ∗ AX}

=

tr(X ∗ A∗ XX ∗ AX) = tr(A∗ AXX ∗ )

=

tr(A∗ A).

i≤j

4.41 The proof is very similar to the proof of Theorem 4.18. First suppose that such a nonsingular matrix X does exist; that is, there is a nonsingular matrix X such that X −1 AX = Λ1 and X −1 BX = Λ2 , where Λ1 and Λ2 are diagonal matrices. Then because Λ1 and Λ2 are diagonal matrices, clearly Λ1 Λ2 = Λ2 Λ1 , so we have AB

= XΛ1 X −1 XΛ2 X −1 = XΛ1 Λ2 X −1 = XΛ2 Λ1 X −1 = XΛ2 X −1 XΛ1 X −1 = BA,

and hence, A and B do commute. Conversely, now assuming that AB = BA, we need to show that such a nonsingular matrix X does exist. Let µ1 , . . . , µh be the distinct values of the eigenvalues of A having multiplicities r1 , . . . , rh , respectively. Since A is diagonalizable, a nonsingular matrix Y exists, satisfying Y −1 AY = Λ1 = diag(µ1 Ir1 , . . . , µh Irh ).

43

Performing this same transformation on B and partitioning the resulting matrix in the same way that Y −1 AY has been partitioned, we get 



C11

C12

···

C1h

   C21 −1 C = Y BY =   ..  .  Ch1

C22 .. .

···

Ch2

···

  C2h   ..  , .   Chh

where Cij is ri × rj . Note that because AB = BA, we must have Λ1 C

= Y −1 AY Y −1 BY = Y −1 ABY = Y −1 BAY = Y −1 BY Y −1 AY = CΛ1 .

Equating the (i, j)th submatrix of Λ1 C to the (i, j)th submatrix of CΛ1 yields the identity µi Cij = µj Cij .

Since µi 6= µj if i 6= j, we must have Cij = (0) if i 6= j; that is, the matrix C =

diag(C11 , . . . , Chh ) is block diagonal. Now clearly C is diagonalizable since B is, and so from Problem 4.21, we know that Cii is diagonalizable for each i. Thus, we can find an ri × ri nonsingular matrix Zi satisfying Zi−1 Cii Zi = ∆i , where ∆i is diagonal. Let X = Y Z, where Z is the block diagonal matrix Z = diag(Z1 , . . . , Zh ) so that X is a nonsingular matrix. Finally, the matrix ∆ = diag(∆1 , . . . , ∆h ) is diagonal, X −1 AX

=

Z −1 Y −1 AY Z = Z −1 Λ1 Z

=

diag(Z1−1 , . . . , Zh−1 ) diag(µ1 Ir1 , . . . , µh Irh ) diag(Z1 , . . . , Zh )

=

diag(µ1 Z1−1 Z1 , . . . , µh Zh−1 Zh )

=

diag(µ1 Ir1 , . . . , µh Irh ) = Λ1

and X −1 BX

=

Z −1 Y −1 BY Z = Z −1 CZ

=

diag(Z1−1 , . . . , Zh−1 ) diag(C11 , . . . , Chh ) diag(Z1 , . . . , Zh )

=

diag(Z1−1 C11 Z1 , . . . , Zh−1 Chh Zh )

=

diag(∆1 , . . . , ∆h ) = ∆,

and so the proof is complete.

44

4.43 From Problem 4.41, it follows that there exists a nonsingular matrix X such that X −1 AX = Λ = diag(λi1 , . . . , λim ) and X −1 BX = M = diag(µj1 , . . . , µjm ), where (i1 , . . . , im ) and (j1 , . . . , jm ) are permutations of (1, . . . , m), As a result, X −1 (A + B)X

= X −1 AX + X −1 BX =

diag(λi1 + µj1 , . . . , λim + µjm ).

Thus, A + B is diagonalizable with eigenvalues λi1 + µj1 , . . . , λim + µjm , so the result follows.

4.45 Consider

 A=

1

0

0

0





,

B=

−1

0

0

0

 ,

so that CAC 0 and CBC 0 are both diagonal when C = I2 . Any linear combination of A and B is singular so Theorem 4.15 does not apply, and B is not nonnegative definite so Theoreom 4.16 does not apply.

4.47 Let C be defined as in the solution to the previous problem. For any y ∈ Rm , let x = C 0 y so x0 (A + B)x = y 0 C(A + B)C 0 y = y 0 (Λ + Im )y.

Thus, A + B is positive definite if and only if Λ + Im is positive definite. The result follows since Λ + Im is positive definite if and only if λi > −1 for all i.

4.49 (a) Let B = XΛX 0 be the spectral decomposition of B and P DP 0 be the spectral decomposition of Λ1/2 X 0 AXΛ1/2 . Put Y = XΛ−1/2 P . Then Y −1 ABY

= P 0 Λ1/2 X 0 AXΛX 0 XΛ−1/2 P = P 0 Λ1/2 X 0 AXΛ1/2 P = P 0 P DP 0 P = D,

so AB is diagonalizable. (b) Let r = rank(A) and suppose A = P ΛP 0 is the spectral decomposition of A, where Λ = −1/2

diag(Λ1 , (0)) and Λ1 is r × r. Then C −1 = diag(Λ1

, Im−r )P 0 is a nonsingular matrix sat-

isfying C −1 AC −10 = diag(Ir , (0)). Consider the matrix   H H 11 12 , H = C 0 BC =  0 H22 H12 45

where H11 is r × r. Since B is nonnegative definite, so is H and as a result, it can be expressed as H = T T 0 for some m × m matrix T . Partitioning T appropriately, we have       0 0 H11 H12 T1 T T T T 1 1 1 2   =  [T10 T20 ] =  , 0 0 H12 H22 T2 T2 T1 T2 T20 which reveals that R(H12 ) ⊂ R(H11 ). Thus, H12 = H11 X for some r × (m − r) matrix X, and as a result

 Y 0 HY = 

where



H11

(0)

(0)

H22 − X 0 H11 X

 Y =

,



Ir

−X

(0)

Im−r

.

Since H11 and H22 − X 0 H11 X are symmetric, there are orthogonal matrices Q1 and Q2 , and diagonal matrices D1 and D2 , such that Q01 H11 Q1 = D1 and Q02 (H22 − X 0 H11 X)Q2 = D2 . Let F = CY Q, where Q = diag(Q1 , Q2 ), and note that F −1 AF −10 = diag(Ir , (0)) and F 0 BF = D = diag(D1 , D2 ). Consequently, F −1 AF −10 F 0 BF = F −1 ABF = diag(D1 , (0)), and so AB is diagonalizable.

4.51 Clearly, kAk∗ is nonnegative and reduces to zero if and only if aij = 0 for all i and j, so properties (a) and (b) of a matrix norm hold. Also, property (c) holds since |caij | = |c||aij |, while property (d) holds since

kA + Bk∗

= m max |aij + bij | ≤ m max {|aij | + |bij |} 1≤i,j≤m 1≤i,j≤m

≤ m max |aij | + m max |bij | = kAk∗ + kBk∗ .

1≤i,j≤m

1≤i,j≤m

Finally, property (e) holds since

kABk∗

= m

≤ m

≤ m

max |(AB)ij | = m

1≤i,j≤m

max

1≤i,j≤m

max

1≤i,j≤m

max

1≤i,j≤m

!

m X

|aik bkj |

k=1 m X

=m

m X

! aik bkj

k=1 m X

max

1≤i,j≤m

! |aik ||bkj |

k=1

!

(m

−1

k=1

46

kAk∗ )(m

−1

kBk∗ )

= kAk∗ kBk∗ .

4.53 (a) Using property (e) of the matrix norm k · k, we have 2 kIm k = kIm k = kIm Im k ≤ kIm kkIm k = kIm k2 ,

and this leads to kIm k ≥ 1. (b) For a nonsingular matrix A, we have kIm k = kAA−1 k ≤ kAkkA−1 k, which yields kA−1 k ≥

kIm k ≥ kAk−1 , kAk

where the second inequality follows from the result from part (a).

4.55 Let A = P DQ0 be a singular value decomposition of A so that the m × m matrices P and Q are orthogonal and the matrix D has the form given in (a) or (d) of Theorem 4.1 since n = m. Then we have kAkE

= {tr(A0 A)}1/2 = {tr[(P DQ0 )0 P DQ0 ]}1/2 = {tr(QDP 0 P DQ0 )}1/2 = {tr(D2 Q0 Q)}1/2 !1/2 r X 2 1/2 2 = {tr(D )} = δi . i=1

4.57 Let

 A=

0

1

0

0





,

B=

0

0

1

0

 .

Both A and B have eigenvalue 0 with multiplicity two, so ρ(A) = ρ(B) = 0. (a) Note that  A+B =

0

1

1

0

 .

Since A + B has eigenvalues of −1 and 1, we have ρ(A + B) = 1 > 0 = ρ(A) + ρ(B).

47

(b) Note that  AB = 

1

0

0

0

 .

Since AB has eigenvalues of 0 and 1, we have ρ(AB) = 1 > 0 = ρ(A)ρ(B).

4.59 (a) We prove the Ar = L∗ U∗ factorization by induction. The result holds for r = 2 since      a11 0 1 a12 /a11 a11 a12   = , a21 a22 − a21 a12 /a11 0 1 a21 a22 where clearly L∗ and U∗ are nonsingular since a11 6= 0 and |A2 | = a11 a22 − a21 a12 6= 0. Now suppose r > 2 and the factorization holds for (r − 1) × (r − 1) matrices. Partition Ar as   B c , Ar =  d0 e where B is (r − 1) × (r − 1). According to our induction hypothesis, B can be factored as B = LU , where L is a nonsingular lower triangular matrix and U is a nonsingular upper triangular matrix.

Our proof will be complete if we can find g, f , v, and w such that    L 0 U w  .

Ar =  f0 g 00 v Since L and U are nonsingular, the unique solutions to Lw = c and f 0 U = d0 are given by w = L−1 c and f 0 = d0 U −1 . Finally, we solve gv + f 0 w = e by setting v = 1 and g = e − f 0 w.

Note that our resulting lower and upper triangular matrices are nonsingular since L and U are, v 6= 0, and g = e − d0 U −1 L−1 c = e − d0 B −1 c 6= 0, a consequence of the fact that |Ar | = 6 0. This proves the Ar = L∗ U∗ factorization.

Partition A, L, and U as  A11 A= A21

A12 A22





,

L=

L11

(0)

L21

L22





,

U =

U11

U12

(0)

U22

where the (1, 1)th block matrices are r × r, and consider     A11 A12 L11 U11 L11 U12  = .

A21 A22 L21 U11 L21 U12 + L22 U22 48

 ,

(1)

Now L11 and U11 can be obtained via the process described in the first part of this problem, and −1 from (1) we immediately get U12 = L−1 11 A12 and L21 = A21 U11 . Since rank(A) = rank(A11 ) = r,

it follows that A21 = BA11 and A22 = BA12 for some (m − r) × r matrix B. Then from (1), we have A22

−1 −1 = L21 U12 + L22 U22 = A21 U11 L11 A12 + L22 U22

(1)

= BA11 A−1 11 A12 + L22 U22 = BA12 + L22 U22

(2)

= A22 + L22 U22 .

(3)

Thus, we must have L22 U22 = (0). For instance, one of the matrices can be chosen to be the null matrix, while the other matrix can be chosen so that L or U is nonsingular and triangular. (b) Consider the matrix  A=

0

1

1

1

 .

An LU factorization would have l11 u11 = a11 = 0, so that l11 = 0 or u11 = 0. But |A| = |LU | requires that both L and U be nonsingular since A is. Thus, an LU factorization is not possible for this matrix. (c) If A = LU , the equation Ax = c can be solved in two steps. First we find a solution y to the equation Ly = c,

(1)

U x = y.

(2)

and then find a solution x to the equation

As a result, we will then have Ax = LU x = Ly = c.

Since both L and U are triangular matrices, solutions to (1) and (2) are easily obtained. For instance, if both L and U are nonsingular matrices, the solution to (1) has Pi−1 ci − j=1 lij yj c1 , yi = , for i = 2, . . . , m, y1 = l11 lii while the solution to (2) has ym xm = , umm

xm−i =

ym−i −

Pi−1

j=0 um−i,m−j xm−j

um−i,m−i 49

, for i = 1, . . . , m − 1.

4.61 (a) It follows from Problem 4.59(a) that A can be expressed as A = L∗ U∗ , where L∗ and U∗0 are nonsingular lower triangular m × m matrices. Let D1 and D2 be the diagonal matrices that have the same diagonal elements as L∗ and U∗ , respectively. Define L = L∗ D1−1 , M 0 = D2−1 U∗ , and D = D1 D2 , and note that L and M are lower triangular matrices with all diagonal elements equal to one. Then the result follows since A = L∗ U∗ = L∗ D1−1 D1 D2 D2−1 U∗ = LDM 0 . (b) Note that if A has the LDM 0 decomposition from part (a), then (A)01· = d11 (M )·1 ,

(A)·1 = d11 (L)·1 ,

and so, since A is symmetric, we must have (M )·1 = (L)·1 . In addition, if the first k − 1 columns of M are the same as those of L, then (A)0k· =

k X

lki dii (M )·i =

k−1 X

i=1

lki dii (L)·i + dkk (M )·k ,

(1)

i=1

while (A)·k =

k X

mki dii (L)·i =

i=1

k−1 X

lki dii (L)·i + dkk (L)·k .

(2)

i=1

The symmetry of A implies (1) equals (2), and so (M )·k = (L)·k . Thus, we have established that that M = L.

50

Chapter 5

5.1 Each proof simply involves the verification of (5.1)–(5.4). (a) We have αAα−1 A+ αA = αAA+ A = αA, and α−1 A+ αAα−1 A+ = α−1 A+ AA+ = α−1 A+ .

In addition, αAα−1 A+ = AA+ and α−1 A+ αA = A+ A are both symmetric since AA+ and A+ A are symmetric. (b) We have A0 (A+ )0 A0 = (AA+ A)0 = A0 , and (A+ )0 A0 (A+ )0 = (A+ AA+ )0 = (A+ )0 .

In addition, A0 (A+ )0 = (A+ A)0 and (A+ )0 A0 = (AA+ )0 are both symmetric since AA+ and A+ A are symmetric. (c) (A+ )+ = A follows immediately since conditions (5.1), (5.2), (5.3) and (5.4) for the Moore–Penrose inverse A+ of A are conditions (5.2), (5.1), (5.4) and (5.3), respectively, for the Moore–Penrose inverse A of A+ . (d) Since A−1 A = Im , we have AA−1 A = A, and A−1 AA−1 = A−1 .

The symmetry conditions also follow since AA−1 = A−1 A = Im is symmetric.

1 (2, 1, 3, 2).

5.3 a0 a = 18, so a+ = (a0 a)−1 a0 = 18

51

5.5 Let Φ = diag(φ1 , . . . , φm ). Note that since λi φi λi = λi λ−1 i λi = λi if λi 6= 0 and λi φi λi = 0 = λi if λi = 0, ΛΦΛ = Λ. −1 Similarly since φi λi φi = λ−1 = λ−1 = φi if λi 6= 0 and φi λi φi = 0 = φi if λi = 0, i λi λi i

ΦΛΦ = Φ.

Further, ΛΦ and ΦΛ are diagonal matrices and, hence, are symmetric. This confirms that Φ = Λ+ .

5.7 (a) Now 

5

−4

5

  AA0 =  5  −4



  −4   14

5 −4

and |AA0 − λI3 | = −λ(λ − 6)(λ − 18), so AA0 has eigenvalues 0, 6, 18. The equation AA0 x = 6x yields the constraints x2 = x1 and x3 = x1 , so a normalized eigenvector corresponding to 6 is x1 = √13 (1, 1, 1)0 . The equation AA0 x = 18x yields the constraints x2 = x1 and x3 = −2x1 , so a normalized eigenvector corresponding to 18 is x2 = √16 (1, 1, −2)0 . Thus,   7 7 4  1 1 1    (AA0 )+ = x1 x01 + x2 x02 =  7 7 4 , 6 18 108   4 4 10 and

 A+ = A0 (AA0 )+ =

2

2

1    −1 18  4

5



4

  2 .  1

1

0

−1

(b) The projection for the range of A is  AA+ = AA0 (AA0 )+ =

1

1   1 2 0



0

  0 ,  2

2

1

1

while the projection for the row space of A is  A+ A = A0 (AA0 )+ A =

52

5

1   2 6 1

2 −2



  −2  .  5

5.9 Since rank(A) = 1, the singular value decomposition of A has the form A = pdq 0 , where d is a scalar, while p and q are m × 1 and n × 1 unit vectors. Now d is the positive square root of the one nonzero p eigenvalue of A0 A, and so d = tr(A0 A) = c1/2 and A = c1/2 pq 0 . Thus, A+ = c−1/2 qp0 = c−1 c1/2 qp0 = c−1 A0 .

5.11 Using the identity AA+ A = A, we have (AA+ )2 = (AA+ A)A+ = AA+ , and (A+ A)2 = A+ (AA+ A) = A+ A, so both AA+ and A+ A are idempotent. Since they are idempotent, we also find that (Im − AA+ )2 = Im − 2AA+ + (AA+ )2 = Im − 2AA+ + AA+ = Im − AA+ , and (Im − A+ A)2 = Im − 2A+ A + (A+ A)2 = Im − 2A+ A + A+ A = Im − A+ A, so (Im − AA+ ) and (Im − A+ A) are also idempotent.

5.13 Since B is positive definite, there exists an n × n nonsingular matrix T such that B = T T 0 . Letting A∗ = AT and using Problem 5.12(c), we have A∗ A0∗ (A∗ A0∗ )+ A∗ = A∗ , that is, AT T 0 A0 (AT T 0 A0 )+ AT = AT. Postmultiplying both sides by T −1 and using T T 0 = B yields ABA0 (ABA0 )+ A = A. 5.15 Let A = XΛX 0 be the spectral decomposition of A. If A has one nonzero eigenvalue λ with multiplicity r, then Λ and X can be chosen so that Λ = diag(λIr , (0)) and X = (X1 , X2 ), and the spectral decomposition will reduce to the form = X1 (λIr )X10 = λX1 X10 , where X1 is an m × r matrix satisfying X10 X1 = Ir . Thus, A+ = X1 (λIr )−1 X10 = λ−1 X1 X10 = λ−2 (λX1 X10 ) = λ−2 A. 53

5.17 (a) If A is nonnegative definite, then its eigenvalues are nonnegative. It then follows from Theorem 5.7 that the eigenvalues of A+ are also nonnnegative, and so it is a nonnegative definite matrix. (b) If Ax = 0, then A+ Ax = 0. Using the fact that A and A+ are symmetric, we have 0 = A+ Ax = (A+ A)0 x = A0 A+0 x = AA+ x.

Premultiplying by A+ , we get 0 = A+ AA+ x = A+ x.

5.19 Let rA = rank(A), rB = rank(B), while A = P DP 0 and B = XΛX 0 are the spectral decompositions of A and B, where D and Λ are rA × rA and rB × rB diagonal matrices. Note that if Ax = 0, we must have Bx = 0 since B and A − B are nonnegative definite, and so R(B) ⊆ R(A). First suppose that rA = rB in which case R(B) = R(A). As a result, there exists an orthogonal matrix Q such that X = P Q and A − B = P DP 0 − XΛX 0 = P D1/2 (IrA − D−1/2 QΛQ0 D−1/2 )D1/2 P 0 , while B + − A+ = XΛ−1 X 0 − P D−1 P 0 = P D−1/2 (D1/2 QΛ−1 Q0 D1/2 − IrA )D−1/2 P 0 .

Since A − B is nonnegative definite, so is IrA − D−1/2 QΛQ0 D−1/2 . But this implies that (see Problem 4.46) D1/2 QΛ−1 Q0 D1/2 − IrA is nonnegative definite and, hence, so is B + − A+ . Conversely, now suppose that B + − A+ is nonnegative definite. But this implies that R(A+ ) ⊆ R(B + ), that is, R(A) ⊆ R(B) in addition to the already established condition R(B) ⊆ R(A), and so rA = rB follows.

5.21 Since P 0 P = Im and QQ0 = In , we see that P AQQ0 A+ P 0 = P AA+ P 0 is symmetric, Q0 A+ P 0 P AQ = Q0 A+ AQ is symmetric, P AQQ0 A+ P 0 P AQ = P AA+ AQ = P AQ, Q0 A+ P 0 P AQQ0 A+ P 0 = Q0 A+ AA+ P 0 = Q0 A+ P 0 .

The four conditions of Definition 5.1 hold, so (P AQ)+ = Q0 A+ P 0 .

54

5.23 (a) Note that 

0

1

  A+ =  0  0

0



  1 ,  0

0 0

B + = diag(1, 0, 1/2), A0 A = diag(1, 1, 0), A+ A = diag(1, 1, 0) and BB 0 = diag(1, 0, 4). Consequently, 

0

  A+ ABB 0 A0 = BB 0 A0 =  0  0

1



0

  0 ,  0

0 0

and BB + A0 AB = A0 AB = diag(1, 0, 0), and so (AB)+ = B + A+ follows from Theorem 5.10(a). (b) Note that A+ A = PR(A+ ) = PR(A0 ) = diag(1, 1, 0) and 

0

0

  A+ ABB 0 = diag(1, 1, 0)  0  0



0



0

    1 = 0   1 0

2 1

0 2 0

0



  1 .  0

Since this is not symmetric, according to Theorem 5.10(b), (AB)+ 6= B + A+ . + 5.25 Let B = diag(A+ 11 , . . . , Arr ). Then B is the Moore-Penrose inverse of A since

ABA

=

+ diag(A11 A+ 11 A11 , . . . , Arr Arr Arr ) = diag(A11 , . . . , Arr ) = A,

BAB

=

+ + + + + diag(A+ 11 A11 A11 , . . . , Arr Arr Arr ) = diag(A11 , . . . , Arr ) = B,

AB

=

+ + diag(A11 A+ 11 , . . . , Arr Arr ) is symmetric since Aii Aii is for each i,

BA

=

+ + diag(A+ 11 A11 , . . . , Arr Arr ) is symmetric since Aii Aii is for each i.

5.27 Using the solution to Problem 5.10(a), we have  U+ =

while

1

1   1 9 1

1 1 1

1

  1 ,  1

 0 1 V + = (V 0 V )−1 V 0 =  3 −1 55



−3

3

−1

2

 .

Since U 0 V = (0), Corollary 5.13.1(d) applies and so  1/9      1/9 +  U = A+ =   1/9  V+   0  −1/3

1/9

1/9

1/9 1/9 −1 −1/3



  1/9    1/9  .   1   2/3

5.29 Let U = wx0 and V = yz 0 . Then U V 0 = (0) and U 0 V = (0), so Theorem 5.17 applies. Using this and the solution to Problem 5.10(d), we get A+

=

=

U + + V + = (w0 wx0 x)−1 xw0 + (y 0 yz 0 z)−1 zy 0   2 0 1 1     2 0 1 1  1   .  24   −1 −3 −2 −2    −3 3 0 0

5.31 We can write Ω = nD − npp0 , where p = (p1 , . . . , pm )0 , and so Ω takes the form A + cd0 in Theorem 5.18 with A = nD, c = −np, and d = p. Since 1 + d0 A−1 c = 1 − p0 D−1 p = 1 −

m X

pi = 0,

i=1

according to Theorem 5.18, Ω is singular. Note that for x = A−1 d = n−1 D−1 p = n−1 1m and y = A−1 c = −n−1 D−1 np = −1m , we have xx+ = yy + = m−1 1m 10m . Thus, from Theorem 5.18 Ω+

=

(Im − yy + )A−1 (Im − xx+ )

=

n−1 (Im − m−1 1m 10m )D−1 (Im − m−1 1m 10m ).

5.33 Since c and d are in the column space of A and A is symmetric, we have A+ Ac = AA+ c = c and A+ Ad = AA+ d = d. We will show that (A + cd0 )+ = A+ − α−1 A+ cd0 A+ , where α = 1 + d0 A+ c, by verifying the four conditions of the Moore–Penrose inverse. Now (A + cd0 )(A+ − α−1 A+ cd0 A+ )

= AA+ − α−1 AA+ cd0 A+ + cd0 A+ − α−1 cd0 A+ cd0 A+ = AA+ + cd0 A+ − α−1 (1 + d0 A+ c)cd0 A+ = AA+ + cd0 A+ − cd0 A+ = AA+ , 56

so condition 5.3 holds. Similarly, (A+ − α−1 A+ cd0 A+ )(A + cd0 )

= A+ A + A+ cd0 − α−1 A+ cd0 A+ A − α−1 A+ cd0 A+ cd0 = A+ A + A+ cd0 − α−1 (1 + d0 A+ c)A+ cd0 = A+ A + A+ cd0 − A+ cd0 = A+ A,

so (5.4) holds as well. Conditions (5.1) and (5.2) hold since (A + cd0 )(A+ − α−1 A+ cd0 A+ )(A + cd0 )

= AA+ (A + cd0 ) = AA+ A + AA+ cd0 = A + cd0 ,

and (A+ − α−1 A+ cd0 A+ )(A + cd0 )(A+ − α−1 A+ cd0 A+ )

= A+ A(A+ − α−1 A+ cd0 A+ ) = A+ AA+ − α−1 A+ AA+ cd0 A+ = A+ − α−1 A+ cd0 A+ .

5.35 (a) diag(0, 1/2, 1/3). (b) diag(1, 1/2, 1/3). (c) 

1

   1  1

1

1

1/2

0

0

1/3

   . 

5.37 Since rank(B) = n, it follows from Theorem 5.23(f) that BB − = In . Thus, ABB − A− AB = AA− AB = AB, and so B − A− is a generalized inverse of AB. 5.39 First suppose that B is a generalized inverse of A. Then ABA = A and so postmultiplying by B, we have ABAB = AB confirming that AB is idempotent. Further, from Theorem 2.8(a), rank(A) = rank(ABA) ≤ rank(AB) ≤ rank(A), 57

implying then that rank(A) = rank(AB). Now suppose AB is idempotent and rank(A) = rank(AB).

Thus, ABAB = AB.

(1)

But the column space of AB is a subspace of the column space of A, and when rank(A) = rank(AB) these two subspaces are the same. Thus, the columns of A can be written as a linear combination of those of AB meaning there is some matrix C such that A = ABC. Postmultiplying equation (1) by C yields ABA = A, so we have shown that B is a generalized inverse of A. 5.41 First let D = C −1 A− B −1 for some generalized inverse A− of A. Then BACDBAC = BACC −1 A− B −1 BAC = BAA− AC = BAC, so D is a generalized inverse of BAC. Conversely, suppose D is a generalized inverse of BAC so that BACDBAC = BAC. Premultiplying this equation by B −1 and postmultiplying by C −1 yields ACDBA = A. This implies that CDB is a generalized inverse of A, that is, CDB = A− for some A− , and this leads to D = C −1 A− B −1 .

5.43 (a) First suppose that B is a reflexive inverse of A. Then since ABA = A, we have rank(A) = rank(ABA) ≤ rank(AB) ≤ rank(B), and since BAB = B, we have rank(B) = rank(BAB) ≤ rank(BA) ≤ rank(A).

Together these two identities imply rank(B) = rank(A). Conversely, suppose ABA = A and rank(B) = rank(A). Then BABA = BA, that is, BA is idempotent, and from Theorem 5.23, rank(BA) = rank(A) = rank(B). It now follows from Problem 5.39 that A is a generalized inverse of B, so BAB = B.

58

(b) Note that  AB = P 

∆

(0)

(0)

(0)





 Q0 Q 

∆

−1

F

E F ∆E





P0 = P 

I

∆E

(0)

(0)

  P 0.

Thus,  ABA



I

∆E

(0)

(0)

∆

(0)

(0)

(0)

∆−1

E

F

F ∆E

∆−1

E

F

F ∆E

= P  = P



 P 0P 

∆

(0)

(0)

(0)

  Q0

  Q0 = A,

and  BAB

= Q  = Q





 P 0P 

I

∆E

(0)

(0)

 P0

  P 0 = B.

5.45 We prove the result by simply showing that the expression given for A− satisfies AA− A = A, and since   U W U + U XV , AA− A =  V W U + V XV this requires U W U +U XV = U and V W U +V XV = V . Now U XV = (0), while U W U = U U − U = U , and so the first of these two identities holds. To verify the second identity, note that V W U = V U − U − V (In − U − U ){V (In − U − U )}− V U − U and V XV = V (In − U − U ){V (In − U − U )}− V, so V W U + V XV

= V U − U + V (In − U − U ){V (In − U − U )}− V (In − U − U ) = V U − U + V (In − U − U ) = V.

5.47 We perform the following elementary row transformations on A: 59

1) Add row 1 to row 2.

2) Subtract from row 3 two times row 1.

3) Interchange rows 2 and 3.

4) Add row 2 to row 1.

This results in CA = H, where 

−1

0





1

  1 ,  1 0

  C =  −2  1

0

1

0

  H= 0  0

2



  3 .  0

1 0

Since H is in Hermite form, it follows from Theorem 5.31 that C is a generalized inverse of A. 5.49 First find the matrix



−8

6

   −8 0 AA=   −11  6

18 11 −8

−11

6



  11 −8  .  22 −11   −11 6

Following the approach used in the solution to Problem 5.47, we get CA0 A = H, where     9/22 4/22 0 0 1 0 −5/2 1          4/22 3/22 0 0   0 1 −1/2 0  , H =  .

C=      15  0 0 3 6 0  0 0      −1 0 0 1 0 0 0 0 C is a generalized inverse of A0 A and so we can use it to get  5 −2   1 4 1   AL = (A0 A)− A0 =  22  0 0  0 0

13



  7  .  0   0

5.51 We show that A+ = BA0 by verifying the four conditions of Definition 5.1. We know A0 ABA0 A = A0 A, so A+0 A0 ABA0 A = A+0 A0 A, which leads to the first condition, ABA0 A = A. We also know

60

(BA0 A)0 = BA0 A, which is the fourth condition. Postmultiplying this identity by A+ yields the identity A0 AB 0 A+ = BA0 AA+ = BA0 . Thus, BA0 − BA0 ABA0

=

(In − BA0 A)A0 AB 0 A+ = (In − A0 AB 0 )A0 AB 0 A+

=

(A0 − A0 AB 0 A0 )AB 0 A+ = (0),

where the second equality uses the fourth condition, while the last equality follows from the first condition. This gives us the second condition BA0 ABA0 = BA0 . Finally, we get the third condition by noting that ABA0 = AB(ABA0 A)0 = ABA0 AB 0 A0 is symmetrtic.

5.53 We show that A+ = A0 (AA0 )L by verifying the four conditions in Definition 5.1. Using Theorem 5.27(b) AA0 (AA0 )L A = AA0 (AA0 )+ A = AA0 A+0 A+ A = A(A+ A)0 A+ A = AA+ AA+ A = A, and A0 (AA0 )L AA0 (AA0 )L

=

A0 (AA0 )+ AA0 (AA0 )L = A0 A+0 A+ AA0 (AA0 )L

=

(A+ A)0 A+ AA0 (AA0 )L = A+ AA0 (AA0 )L

=

A0 A+0 A0 (AA0 )L = A0 (AA0 )L .

Note that the symmetry of AA0 (AA0 )L follows immediately from the definition of (AA0 )L . Finally, A0 (AA0 )L A is symmetric due to Theorem 5.27(c).

5.55 According to Theorem 5.27(b), A(A0 A)− A0 = A(A0 A)+ A0 and A0 (AA0 )− A = A0 (AA0 )+ A. Thus A0 (AA0 )− A(A0 A)− A0

=

A0 (AA0 )+ A(A0 A)+ A0

=

A0 A+0 A+ AA+ A+0 A0

=

(A+ A)0 A+ AA+ (AA+ )0

=

A+ AA+ AA+ AA+ = A+ .

61

5.57 Since H is upper triangular, we have (H 2 )ij

=

m X

hik hkj

k=1

=

0, if i > j,

=

h2ii , if i = j,

=

j X

hik hkj , if i < j.

k=i

Thus, (H 2 )ij = hij = 0 if i > j, and (H 2 )ii = h2ii = hii since hii = 0 or hii = 1, so all that remains is to show that (H 2 )ij = hij when i < j. Now if hii = 0, then hik = 0 for all k and so (H 2 )ij =

j X

hik hkj = 0 = hij .

k=i

When hii = 1, j X

j X

hik hkj = hij +

k=i

hik hkj = hij ,

k=i+1

since if for any k, hik 6= 0, then hkk = 0 and hkj = 0, and so hik hkj = 0 for k = i + 1, . . . , j.

5.59 We first compute 

3    1 A0 A =    4  7 This then leads to



1

4

3

4

4

8

5

12

  5  .  12   19

30

−1

−4

   −1 0 B2 = 33I4 − A A =    −4  −7

7



30 −4 −5

−7



  −4 −5  .  25 −12   −12 14

We next calculate 

0

   6 0 B2 A =    6  6

24

0



  −36 6  ,  −12 6   12 6



24

   −24 0 B2 A A =    0  24 62

−24 48 24 0

0

24



  24 0  .  24 24   24 48

Then since A has rank of 2, 

0

   1 1 + 0 −1 0  A = 2{tr(B2 A A)} B2 A = 12   1  1

4 −6 −2 2

5.61 Using the notation of Theorem 5.13 and its corollary, if A = [Aj−1

0



  1  .  1   1 aj ] = [U

V ], then C = aj −

Aj−1 dj = cj . If cj 6= 0, then C + C = c+ j cj = 1, and so Corollary 5.13.1(c) applies. This gives   A+ j =

+ + A+ j−1 − Aj−1 aj cj

c+ j





=

0 A+ j−1 − dj bj

b0j

 .

If cj = 0, then Corollary 5.13.1(b) applies and + −1 K = (1 + a0j A+0 = (1 + d0j dj )−1 , j−1 Aj−1 aj ) 0 + −1 0 + KV 0 U +0 U + = (1 + d0j dj )−1 a0j A+0 dj Aj−1 = b0j , j−1 Aj−1 = (1 + dj dj )

so  A+ j

= 

U + − U + V KV 0 U +0 U +

KV 0 U +0 U +   A+ − dj b0j j−1 . =  b0j

63





=

0 + A+ j−1 − Aj−1 aj bj



b0j



Chapter 6

6.1 (a) Using A+ which was obtained in Problem 5.2, we find  3 2 2   1  2 6 −1 AA+ =  7  2 −1 6  2 −1 −1 and then



1

2



  −1  ,  −1   6



      3 + .  AA c =    −1    0 Since this is c, the system is consistent. (b) One solution is 

2



    A+ c =  3  .   −4 (c) It is easily verified that rank(A) = 3, so from Theorem 6.7, it follows that there is n−rank(A)+1 = 1 linearly independent solution.

6.3 By augmenting A with a final column of zeros and then row reducing this matrix to one in Hermite form, we obtain a generalized inverse given by   0 2/5 0 −1/5     A− =  0 −1/5 0 3/5  .   0 −3 5 −1 (a) Since AA− c = (14/5, 2, 4/5, −2)0 6= c, the system is inconsistent. (b) Since AA− c = (3, 2, 1, −1)0 = c, the system is consistent. (c) Since AA− c = (36/5, 8, −4/5, −28)0 6= c, the system is inconsistent.

64

6.5 According to Theorem 6.3, since AXB = C is consistent, AA− CB − B = C. As a result AXY B = AA− CB − B + AY B − AA− AY BB − B = C + AY B − AY B = C, so XY is a solution. Next suppose X∗ is a solution. Then AX∗ B = C which leads to A− AX∗ BB − = A− CB − . The result now follows since XX∗ = A− CB − + X∗ − A− AX∗ BB − = A− CB − + X∗ − A− CB − = X∗ .

6.7 Suppose there exists a y such that A− c and (In − A− A)y 6= 0 are linearly dependent. After possibly making a scalar adjustment to this vector y, we would then have A− c = (In − A− A)y, and this leads to AA− c = A(In − A− A)y = (A − AA− A)y = (A − A)y = 0.

But this is a contradiction because if the system is consistent, we must have AA− c = c 6= 0. Thus, A− c and (In − A− A)y must be linearly independent.

6.9 From Theorem 5.24, for arbitrary n × m matrix C, we must have 0 = (C − A− ACAA− )c = Cc − A− ACAA− c = Cc − A− ACc = (In − A− A)Cc, since AA− c = c if Ax = c is consistent. But since C is arbitrary and c 6= 0, any b ∈ Rn can be written as b = Cc for some C, so we must have In − A− A = (0). The result now follows from Theorem 6.6.

6.11 The number of linearly independent solutions is r = n−rank(A) = 4−2 = 2. Using the Moore-Penrose inverse



9

17

  12 1   A+ = A0 (AA0 )−1 =  86  −34  −9 and



61

  24 1  +  I4 − A A = 86   18  25

24

18



  −6  ,  −26   −17 25



  32 24 −24  ,  24 18 −18   −24 −18 61

the general solution is given by xy = (In − A+ A)y. Choosing 86e1 and 86e2 for y, we get the two particular linearly independent solutions, (61, 24, 18, 25)0 and (24, 32, 24, −24)0 .

65

6.13 Since G is a generalized inverse of A, the arbitrary solution x∗ can be written as x∗ = Gc + (In − GA)y for some vector y. Then x0∗ x∗

=

c0 G0 Gc + y 0 (In − GA)0 (In − GA)y + 2c0 G0 (In − GA)y

=

c0 G0 Gc + y 0 (In − GA)0 (In − GA)y

since c0 G0 (In − GA)y

=

(Ax∗ )0 G0 (In − GA)y

=

x0∗ A0 G0 (In − GA)y

= x0∗ (GA)0 (In − GA)y = x0∗ GA(In − GA)y = x0∗ (GA − GAGA)y = x0∗ (GA − GA)y = 0.

Thus, x0∗ x∗ ≥ c0 G0 Gc

(1)

y 0 (In − GA)0 (In − GA)y

(2)

since

is nonnegative. We have equality in (1) only if the quantity in (2) is 0 and this requires (In −GA)y = 0.

This confirms that we have a strict inequality in (1) if x∗ 6= Gc. 6.15 Note that Im = XX −1 = X1 X1− + X2 X2− , so that X2 X2− = Im − X1 X1− . Then using Theorem 6.5, we can solve AX1 = X1 Λ1 for A to get A = X1 ΛX1− + Y (Im − X1 X1− ) = X1 ΛX1− + Y X2 X2− , where Y is an arbitrary m × m matrix. If we let W = −Y X2 , then A can be expressed as A = X1 ΛX1− − W X2− .

66

6.17 (a) Suppose there is a common solution. That is, there exists an X satisfying AX = C and XB = D. Postmultiplying the first of these two equations by B, while premultiplying the second by A yields two equations for AXB which when equated gives AD = CB. Now suppose AD = CB. We need to show that the general solution to AX = C, XY = A− C + Y − A− AY , is a solution to XB = D for some Y . Take Y = DB − so that XY = A− C + DB − − A− ADB − = A− C + DB − − A− CBB − .

Then since XB = D is consistent, we know that DB − B = D and so XY B = A− CB + DB − B − A− CBB − B = A− CB + D − A− CB = D. (b) The general solution to AX = C is X = A− C + W − A− AW for arbitrary W . If this is also to be a solution to XB = D, W must satisfy XB = A− CB + (In − A− A)W B = D, and since AD = CB, this leads to (In − A− A)W B = D − A− AD. The general solution for W in this last system of equations is W = (In − A− A)(D − A− AD)B − + Y − (In − A− A)Y BB − for arbitrary Y , since (In − A− A)− = (In − A− A). Substituting this back into the expression for X, we get X

=

A− C + (In − A− A)(D − A− AD)B − + Y − (In − A− A)Y BB − − A− AY

=

A− C + (In − A− A)DB − + (In − A− A)Y (Ip − BB − ),

as required.

6.19 (a) Using the least squares inverse obtained in the solution to Problem 5.49, we find that   32  1    AAL c =  18  6= c, 22   114 so the system is inconsistent.

67

(b) A least squares solution is given by 

73



      41 1 L .  A c=  22  0     0 (c) The sum of squared errors for a least squares solution to this system of equations is (AAL c − c)0 (AAL c − c) =

4 .

11

6.21 Suppose x∗ is a least squares solution. Then, according to Theorem 6.13, Ax∗ = AAL c, and this leads to A0 Ax∗ = A0 (AAL )0 c = A0 AL0 A0 = A0 c.

Conversely, now suppose A0 Ax∗ = A0 c. Then AL0 A0 Ax∗ = AL0 A0 c, or (AAL )0 Ax∗ = (AAL )0 c, and so AAL Ax∗ = AAL c or, finally Ax∗ = AAL c. Thus, it follows from Theorem 6.13 that x∗ is a least squares solution.

6.23 A solution to this problem can be found in Example 8.3.

6.25 The general solution for β̂ in B β̂ = b is given by β̂ u = B − b + (Im − B − B)u,

(1)

where u is an arbitrary m × 1 vector. Substituting this in the equation X β̂ = y yields XB − b + X(Im − B − B)u = y, or X(Im − B − B)u = y − XB − b.

(2)

Now using Theorem 6.14, the general least squares solution for u in (2) is given by uw = {X(Im − B − B)}L (y − XB − b) + [Im − {X(Im − B − B)}L X(Im − B − B)]w, where w is an arbitrary m × 1 vector. We then get the general restricted least squares solution for β by substituting uw for u in (1), and this leads to β̂ w

=

B − b + (Im − B − B)({X(Im − B − B)}L (y − XB − b) + [Im − {X(Im − B − B)}L X(Im − B − B)]w).

68

Chapter 7

7.1 (a) We have |A| = |A22 ||A11 − A12 A−1 22 A21 | = |dIm ||(a − bc/d)Im | = dm (a − bc/d)m = (ad − bc)m . (b) A is nonsingular for any values of a, b, c, and d for which |A| = 6 0, that is, for which ad 6= bc. (c) Writing  A−1 = 

B11

B12

B21

B22

 ,

and using Theorem 7.1, we get −1 B11 = (A11 − A12 A−1 = {(a − bc/d)Im }−1 = 22 A21 )

d Im , ad − bc

−1 B22 = (A22 − A21 A−1 = {(d − bc/a)Im }−1 = 11 A12 )

a Im , ad − bc

b Im , ad − bc c B21 = −A−1 Im , 22 A21 B11 = − ad − bc

B12 = −A−1 11 A12 B22 = −

That is,  A−1 =

dIm

1  ad − bc −cIm

−bIm aIm

 .

7.3 Using Theorem 7.4 and Problem 3.29, we have |A| = |bIm ||aIm − (cdm/b)1m 10m | = bm am−1 (a − cdm2 /b), so A is nonsingular if a 6= 0, b 6= 0, and ab 6= cdm2 . Letting g = ab − cdm2 and using Theorem 7.1 and Problem 3.29, we find that B11 = (aIm − (cdm/b)1m 10m )−1 = a−1 Im +

cdm 1m 10m , ag

B22 = (bIm − (cdm/a)1m 10m )−1 = b−1 Im +

cdm 1m 10m , bg

69

cdm c B12 = −(aIm )−1 (c1m 10m ) b−1 Im + 1m 10m = − 1m 10m , bg g

d cdm 1m 10m = − 1m 10m .

B21 = −(bIm )−1 (d1m 10m ) a−1 Im + ag g That is,  A−1 = 



0 a−1 Im + cdm ag 1m 1m

− gc 1m 10m

− dg 1m 10m

0 b−1 Im + cdm bg 1m 1m

7.5 Applying Theorem 7.4 with A11 = diag(4, 3, 2),    1 2   0   A12 =  1 2  , A21 =    1 2 3

0

1

1

0





,

A22 = 

.

2

3

1

2

we get |A| = |A11 ||A22 − A21 A−1 11 A12 | = 24

1

3/2

5/12

5/6

= 5.

Applying Theorem 7.1, we find  20 1 −1 B22 = (A22 − A21 A−1 =  11 A12 ) 5 −10  −1 −1 B11 = A−1 11 + A11 A12 B22 A21 A11 =

 1 B12 = −A−1  11 A12 B22 = 

5

7.7 (a) If rank(A) = rank(A11 ), it follows that    A11 A12 A11  = A21 A22 XA11

1 3 0 

 ,

0

12

−10

−8

5

A11 Y XA11 Y



  0 ,  5

  −4  ,  0

−5

1 9 B21 = −B22 A21 A−1 11 = 5 −6

70

−3

0



24

2

1   1 5 0

0

−36

 .

 

 ,

−1 for some matrices X and Y . This requires X = A21 A−1 11 and Y = A11 A12 , and so −1 −1 A22 = XA11 Y = A21 A−1 11 A11 A11 A12 = A21 A11 A12 .

(b) With B = diag(A−1 11 , (0)), we have   A11 A12 I   m1 A(BA) =  A21 A21 A−1 (0) 11 A12 (c) It follows from Theorem 5.9 that  A+

= 

A−1 11 A12 (0)

A21 A−1 11

 A11 Im1 

= Im1

A−1 11 A12

+

 A−1 11



=



A11

A12

A21

A21 A−1 11 A12

 = A.

+

 A−1 11 A12



Im1



Im1 A21 A−1 11

+ +  = G+ A−1 11 H .

Here  G+

= G0 (GG0 )−1 =   = 

A011 A012

Im1 A012 A−10 11

 −10 −1 0  (Im1 + A−1 11 A12 A12 A11 )

  (A11 A011 + A12 A012 )−1 A11 ,

and H+

=

−1 −1 0 (H 0 H)−1 H 0 = (Im1 + A−10 Im1 11 A21 A21 A11 )

= A11 (A011 A11 + A021 A21 )−1 [A011

0 A−10 11 A21

A021 ] ,

and so the result follows. −1 0 7.9 From Theorem 7.1, B11 = A11 − A12 A−1 22 A12 , and so       −1 0 −1 −1

A12 A A A11 − B11 A12 A A A 0  =  12 22 12  =  12 22  A22 A−1 C= 22 A12 0 0 A12 A22 Im2 A12 A22

Im2

is nonnegative definite since A22 is positive definite. Due to the identity above, rank(C) ≤ rank(A22 ) = m2 = m − m1 , while it follows that rank(C) ≥ rank(A22 ) since A22 is a submatrix of C. This confirms that C is positive semidefinite with rank of m − m1 .

71

7.11 (a) Applying Theorem 7.4, we have |A| = |A11 |(amm − a0 A−1 11 a) ≤ amm |A11 |, where the inequality follows from the fact that A−1 11 is positive definite. We have equality if and only if a0 A−1 11 a = 0, and this is equivalent to a = 0. (b) We can prove by induction. The result holds for m = 2 since in this case we can apply the result from part (a). Now assume the result holds for (m − 1) × (m − 1) positive definite matrices. That is, if A11 is (m − 1) × (m − 1) and positive definite, then |A11 | ≤ a11 · · · am−1,m−1 , with equality if and only if A11 is diagonal. Applying the result of part (a) to the matrix   A11 a12 , A= a012 amm we have |A| ≤ amm |A11 | with equality if and only if a12 = 0. Combining these two results yields the desired result.

7.13 The result follows by computing the determinant of both sides of the identity      A11 A12 Im1 (0) A11 A12  =  .

A21 + BA11 A22 + BA12 B Im2 A21 A22 7.15 If A11 A21 = A21 A11 , then    A11 A12 I   =  m1 (0) A21 A12 − A11 A22 A21

(0) −A11

 

A11

A12

A21

A22

 .

Thus, |A11 ||A21 A12 − A11 A22 | = | − A11 ||A|, which leads to |A| = |A11 A22 − A21 A12 |.

7.17 If B has rank 1, it can be expressed as B = −dc0 for some m × 1 vectors c and d. Using Theorem 7.4(a), we have A

d

0

1

c

= |A − dc0 | = |A + B|,

72

while Theorem 7.4(b) yields A

d

0

1

c

= |A|(1 − c0 A−1 d) = |A|{1 + tr(A−1 B)}.

The result follows by equating these two equations.

7.19 The proof utilizes the results given in Theorem 5.25. If R(A12 ) ⊂ R(A11 ), then A11 A− 11 A12 = A12 , and so

 A=

A11

(0)

A21

Im2

 



Im1

A− 11 A12

(0)

A22 − A21 A− 11 A12

If R(A021 ) ⊂ R(A011 ), then A21 A− 11 A11 = A21 , and so   Im1 (0) A   11 A= − A21 A− (0) 11 A22 − A21 A11 A12

.

A12 Im2

(1)

 .

(2)

Taking the determinant of both sides of (1) and (2) yields the desired result.

7.21 Consider



1

   0 A=   1  1

0

1

1

1

1

1

1

1

1



  1  ,  1   1

and partition as in (7.1) with m1 = 1 and m2 = 3. The conditions in Theorem 7.8(a) do not hold since the column spaces of both A21 and A012 are spanned by (0, 1, 1)0 , while the column space of A22 is spanned by (1, 1, 1)0 . However, the identity holds since both |A| and |A22 | are zero.

7.23 Using Theorem 5.25, if R(A21 ) ⊂ R(A22 ), then A21 = A22 A− 22 A21 . That is, A21 = A22 B for some matrix B. Similarly, if R(A012 ) ⊂ R(A022 ), then A12 A− 22 A22 = A12 , so that A12 = CA22 for some matrix C. The result then follows since − A12 A− 22 A21 = CA22 A22 A22 B = CA22 B.

Consider



1

  A= 1  1 73

0 1 1

1



  1 ,  1

and partition A as in (7.1) with m1 = 1 and m2 = 2. Note that R(A21 ) ⊂ R(A22 ) holds, but R(A012 ) ⊂ R(A022 ) does not. The matrices D = diag(1, 0) and E = diag(0, 1) are both generalized inverses of the matrix A22 = 12 102 , and A11 − A12 DA21 = 1, while A11 − A12 EA21 = 0.

7.25 Write the expression given for A− as A− = X + Y . Note that since A11 G = (0), we have   − A11 A− + CC F (0) 11 , AX =  − − − (Im2 − DD )(A21 A− 11 + BC F ) DD and then since F A11 = (0),  AXA =  Also

 AY = 



A11

A12

A21

A22 − (Im2 − DD− )B(Im2 − C − C)

.

(1)



−C −(Im2 − DD− )B

E

h

BC − F + A21 A− 11

−Im2

i

,

and since CE = (0), ED = (0), and (Im2 − DD− )BEB(Im2 − C − C) = (Im2 − DD− )B(Im2 − C − C)   (0) (0) .

AY A =  (2) (0) (Im2 − DD− )B(Im2 − C − C) Adding (1) and (2) confirms that AA− A = A. 7.27 With

 C=

B1−

−B1− A12 A− 22

− −A− 22 A21 B1

− − − A− 22 + A22 A21 B1 A12 A22

 ,

we need to verify that ACA = A if and only the stated conditions hold. Partitioning ACA and A into 2 × 2 forms and equating the (1, 1)th submatrices yields an identity equivalent to the identity A11 = A11 , which of course holds. Equating the (2, 1)th submatrices yields an identity equivalent to the identity given in (a), while equating the (1, 2)th submatrices yields the identity given in (b).

Finally, equating the (2, 2)th submatrices yields an identity equivalent to the final condition in (c).

7.29 We have

  A−1 11 =

74

1

−1

−1

2

 ,

from which we get B = 0. Consequently, −1 −1 −1 0 Z = A012 A−1 = 11 (I2 + A11 A12 A12 A11 )

and

 B ∼ = [Z

1] 

This then leads to

 A+ =

A−1 11

0

 

0

0

0

17

19

1    19 81  −4

Z

0

1 [2 9

− 2] ,



 = 20 .

81 1

−4



  −14  .  −14 20 26

7.31 Note that C is symmetric and idempotent and R(B) = R(B + ) ⊂ R(C). Thus, B + C = B + and CB + = B + , and so A012 B + = (0) and B + A12 = (0).

(1)

Using this, we find that  +

AA =  and

+ A12 A+ 12 + CA11 B

+0 + CA11 A+0 12 − CA11 B A11 A12



(0)

A012 A+0 12



 A+ A = 

(2)



0 + A+0 12 A12 + B A11 C

(0)

+ + A+ 12 A11 C − A12 A11 B A11 C

A+ 12 A12

.

(3)

Now CA11 B + = CA11 CB + = BB + and, since B = CA11 C = CT T 0 C 0 for some m1 × m1 matrix T , CA11 B + A11 = CA11 CB + CA11 = BB + CA11 = PR(CT ) CT T 0 = CT T 0 = CA11 .

When these are used in (2) and (3), we get  AA+ = A+ A = 

+ A12 A+ 12 + BB

(0)

(0)

A012 A+0 12

 

(4)

which is symmetric. Finally using (1), (4) and the fact that BB + A11 = BB + CA11 = CA11 , we get AA+ A = A and A+ AA+ = A+ . Thus, we have shown that the expression given for A+ satisfies the four conditions of the Moore–Penrose inverse.

75

7.33 In the proof of Theorem 7.18, it was shown that λm1 −j+1 (B1 ) ≥ λm−j+1 (A), so the lower bound clearly holds. Applying the inequality from Problem 7.32 to A−1 , we get ( ) k k X X λ−1 m2 (B2 ) −2 −2 {λm−j+1 (A) − λm1 −j+1 (B1 )} ≤ 2 1 + −1 λj (CC 0 ). −1 λ (B ) − λ (B ) m 1 2 2 m1 −k+1 j=1 j=1 Now combining this with the fact that k X

−2 {λ−2 m−j+1 (A) − λm1 −j+1 (B1 )}

=

j=1

k X λ2m −j+1 (B1 ) − λ2m−j+1 (A) 1

λ2m−j+1 (A)λ2m1 −j+1 (B1 )

j=1

≥

−2 λ−2 m−k+1 (A)λm1 −k+1 (B1 )

k X {λ2m1 −j+1 (B1 ) − λ2m−j+1 (A)}, j=1

leads to k X

{λ2m1 −j+1 (B1 ) − λ2m−j+1 (A)}

≤

2λ2m−k+1 (A)λ2m1 −k+1 (B1 )

j=1

(

λj (CC 0 ) −1 λ−1 m1 −k+1 (B1 ) − λm2 (B2 ) j=1 ) k ( −1 X λm2 (B2 ) 4 2λm1 −k+1 (B1 ) 1 + −1 λj (CC 0 ). λm1 −k+1 (B1 ) − λ−1 m2 (B2 ) j=1 ×

≤

) k X

λ−1 m2 (B2 )

1+

7.35 Note that the lower bound is an immediate consequence of the lower bound given for λh (B1 ) in Problem 7.34. Now let  C=

Q0

(0)

(0)

Im2





A

Q

(0)

(0)

Im2





=

Q0 A11 Q

Q0 A12

A012 Q

A22





=

C11

C12

0 C12

C22

 .

−1 0 Note that C is positive definite and C1 = C11 − C12 C22 C12 = ∆, so that the positive eigenvalues of

C1 and B1 are the same and, in particular, λ1 (C1 ) < λm2 (B2 ). Thus, we can apply Theorem 7.18 to −1 C. Note that −C1−1 C12 C22 = −∆−1 Q0 A12 A−1 22 = Ĉ, and so m1X −r+k

{λm1 −j+1 (B1 ) − λm−j+1 (A)}

=

m1X −r+k

{λm1 −j+1 (B1 ) − λm−j+1 (A)}

j=m1 −r+1

j=1

≤

k X

{λr−j+1 (C1 ) − λm2 +r−j+1 (C)}

j=1

≤

76

λ2r−k+1 (C1 )

k X

−1 {λ−1 r−k+1 (C1 ) − λm2 (B2 )} j=1

λj (Ĉ Ĉ 0 )

=

λ2r−k+1 (B1 )

k X

−1 {λ−1 r−k+1 (B1 ) − λm2 (B2 )} j=1

λj (Ĉ Ĉ 0 ).

The first equality follows from the fact that λh+m2 (A) = λh (B1 ) = 0 for h = r + 1, . . . , m1 , a consequence of rank(A) = rank(A22 ) + rank(B1 ) = m2 + r, while the first inequality follows from Theorem 3.19.

77

Chapter 8

8.1 (a) We have 

10

   6 A⊗B =   5  3

6

15

4

9

3

10

2

6

9





10

   5 B⊗A=   6  3

  6  ,  6   4

15

6

10

3

9

4

6

2

9



  6  .  6   4

(b) tr(A ⊗ B) = tr(A) tr(B) = 4(7) = 28. (c) |A ⊗ B| = |A|2 |B|2 = (12 )(12 ) = 1. (d) We find that |A − λI2 | = λ2 − 4λ + 1, and so the eigenvalues of A are 2 ±

√

|B − λI2 | = λ2 − 7λ + 1,

√ 3, while those of B are 72 ± 32 5. Consequently the four

eigenvalues of A ⊗ B are given by

√ √ √ √ 7 3√ 7 3√ 7 3√ 7 3√ (2 + 3) + 5 , (2 + 3) − 5 , (2 − 3) + 5 , (2 − 3) − 5 .

2 2 2 2 2 2 2 2 (e) Using Theorem 8.4, we have  (A ⊗ B)−1

= A−1 ⊗ B −1 =  

4    −6 =    −2  3

−6

−6

2

−3

−1

2 

9





⊗



2

−3

−3

5

···

a ⊗ Bk



  9 −15  .  3 4 −6   −5 −6 10 10

8.5 Using the definition of a Kronecker product, we have     a1 B a1 B1 · · · a1 Bk     h .. ..  .    a ⊗ B =  ..  =   = a ⊗ B1 . .     am B am B1 · · · am Bk

78

i

.

8.7 From Theorem 8.1(f), we have (A ⊗ B)0 = A0 ⊗ B 0 = A ⊗ B, so A ⊗ B is symmetric.

8.9 If cA and c−1 B are orthogonal, then A0 A = c−2 Im and B 0 B = c2 In , and so (A ⊗ B)0 (A ⊗ B) = A0 A ⊗ B 0 B = c−2 Im ⊗ c2 In = Imn , confirming that A⊗B is orthogonal. Conversely, if A⊗B is orthogonal, we must have A0 A⊗B 0 B = Imn .

This implies that (A0 A)ij B 0 B = (0) when i 6= j, so A0 A must be diagonal. Also, we must have (A0 A)ii B 0 B = In for i = 1, . . . , m, implying that B 0 B is diagonal with each diagonal element equal to −1 (A0 A)ii . Thus, we have shown that for some d > 0, A0 A = dIm and B 0 B = d−1 In , and this yields the

result with c = d−1/2 .

8.11 Each element of A ⊗ B is an element of A times an element of B so it immediately follows that A ⊗ B = (0) if A = (0) or B = (0). Now suppose that A ⊗ B = (0) and A 6= (0). Then for some i and j, aij 6= 0. But aij B is a submatrix of A ⊗ B, so we must have aij B = (0) implying B = (0). In a similar fashion, it can be shown that if B 6= (0), we must have A = (0).

8.13 Since rank(A) ≤ m < n, there exist n − m linearly independent n × 1 vectors x1 , . . . , xn−m such that Axi = 0 for each i. If y 1 , . . . , y m is a set of linearly independent m × 1 vectors, then xi ⊗ y j for i = 1, . . . , n − m and j = 1, . . . , m, form a linear independent set of vectors and (A ⊗ B)(xi ⊗ y j ) = (Axi ⊗ By j ) = (0 ⊗ By j ) = 0 for all i and j. This confirms that 0 is an eigenvalue with multiplicity at least (n − m)m.

8.15 Let λ1 , . . . , λm be the eigenvalues of A, and θ1 , . . . , θp be the eigenvalues of B. Since A and B are positive definite, λi > 0 and θj > 0 for all i and j. Thus, λi θj > 0 for all i and j, and so according to Theorem 8.5, all of the eigenvalues of A ⊗ B are positive. This confirms that A ⊗ B is positive definite.

8.17 The three matrices are the same since   x1  h  .  xy 0 =  ..  y1   xm



···

x1 y1 i   .. yn =  .  xm y1 79

···

···

x1 yn .. . xm yn

   , 



 x1 y1 · · · x1 yn  h i  ..  .  y 0 ⊗ x = y1 x · · · yn x =  .. , .   xm y1 · · · xm yn     x1 y1 · · · x1 yn x1 y 0     ..   .   . x ⊗ y 0 =  ..  =  .. . .     xm y1 · · · xm yn xm y 0 8.19 (a) For this model, we have X = (1a ⊗ 1b ⊗ 1n , Ia ⊗ 1b ⊗ 1n , 1a ⊗ Ib ⊗ 1n ), and



abn   X X =  bn1a  an1b

bn10a

0

bnIa n10a ⊗ 1b

an10b



  n1a ⊗ 10b  .  anIb

Using the generalized inverse (X 0 X)− = diag((abn)−1 , (bn)−1 (Ia − a−1 1a 10a ), (an)−1 (Ib − b−1 1b 10b )) we find that a least squares solution for β is given by 

ȳ··



   ȳ − ȳ  ··   1·   ..     .     0 − 0 β̂ = (X X) X y =  ȳa· − ȳ··  ,      ȳ·1 − ȳ··      ..   .   ȳ·b − ȳ·· where ȳ·· , ȳi· , and ȳ·j are as defined in Example 8.3. The fitted value for yijk is µ̂ + τ̂i + γ̂j = ȳi· + ȳ·j − ȳ·· , and so the sum of squared errors is given by SSE =

a X b X n X

(yijk − ȳi· − ȳ·j + ȳ·· )2 .

i=1 j=1 k=1

80

(b) The reduced model is a one-way classification model with b treatments and an observations for each treatment. Using Example 8.2, we find that the sum of squared errors for this reduced model is given by SSE1 =

a X b X n X

!2 (yijk − ȳ·j )2 =

i=1 j=1 k=1

X

2 yijk − (an)−1

X X j

ijk

yijk

.

ik

Noting that the SSE from part (a) can be expressed as  2  2 !2 X X X X X X 2  SSE = yijk − (bn)−1 yijk  − (an)−1 yijk + (abn)−1  yijk  , (1) i

ijk

j

jk

ik

ijk

we find that SSA

=

SSE1 − SSE = (bn)−1

X

  2 2 X X  yijk  − (abn)−1  yijk 

i

= nb

a X

jk

ijk

(ȳi· − ȳ·· )2 .

i=1

(c) The reduced model is a one-way classification model with a treatments and bn observations for each treatment. Again using Example 8.2, we find that the sum of squared errors for this reduced model is given by SSE2 =

a X b X n X

2

 (yijk − ȳi· )2 =

i=1 j=1 k=1

X

2 yijk − (bn)−1

X  i

ijk

X

yijk  .

jk

Thus, we have SSB

=

SSE2 − SSE = (an)−1

j

=

na

b X

yijk

2



!2 X X

− (abn)−1 

ik

X

yijk 

ijk

(ȳ·j − ȳ·· )2 .

j=1

(d) For each i, µ + τi is estimable and for each j, µ + γj is estimable with the corresponding estimates given by ȳi· and ȳ·j , respectively. (e) Note that the SSE given in Problem 8.18 can be expressed as !2 X X X 2 −1 yijk − n yijk . ij

ijk

81

k

Subtracting this from the SSE given in (1), we get !2 SSAB

= n−1

X X ij

− (bn)−1

yijk

X

 2 !2 X X X −1  yijk  − (an) yijk

i

k

jk

j

ik

 2 X + (abn)−1  yijk  ijk

= n

X

2 ȳij − nb

ij

= n

X

ȳi·2 − na

X

i

2 ȳ·j + nabȳ··2

j

a X b X

(ȳij − ȳi· − ȳ·j + ȳ·· )2 .

i=1 j=1

8.21 (a) We have (A1 ⊕ A2 ) + (A3 ⊕ A4 )

=

diag(A1 , A2 ) + diag(A3 , A4 )

=

diag(A1 + A3 , A2 + A4 ) = (A1 + A3 ) ⊕ (A2 + A4 ).

(b) Similarly (A1 ⊕ A2 )(A3 ⊕ A4 )

=

diag(A1 , A2 ) diag(A3 , A4 )

=

diag(A1 A3 , A2 A4 ) = A1 A3 ⊕ A2 A4 .

(c) Finally, (A1 ⊕ A2 ) ⊗ A3

=

diag(A1 , A2 ) ⊗ A3

=

diag(A1 ⊗ A3 , A2 ⊗ A3 ) = (A1 ⊗ A3 ) ⊕ (A2 ⊗ A3 ).

8.23 Solving the system of equations AXB = C in Theorem 6.5 is equivalent to solving the system of equations (B 0 ⊗ A) vec(X) = vec(C).

From Theorem 6.4, the general solution is vec(X)

=

(B 0 ⊗ A)− vec(C) + (Inp − (B 0 ⊗ A)− (B 0 ⊗ A)) vec(Y )

=

(B −0 ⊗ A− ) vec(C) + (Inp − (B −0 B 0 ⊗ A− A)) vec(Y )

=

vec(A− CB − ) + vec(Y ) − vec(A− AY BB − ).

The result follows since this identity is equivalent to X = A− CB − + Y − A− AY BB − .

82

8.25 (a) The result follows from Theorem 8.12 when p = n, q = m, and D = Im . (b) Applying Theorem 8.12, we have tr(AD0 BDC)

=

tr(D0 BDCA) = tr{D0 BD(CA)}

=

{vec{(D0 )0 }}0 {(CA)0 ⊗ B} vec(D)

=

{vec(D)}0 (A0 C 0 ⊗ B) vec(D).

8.27 If C is symmetric, then Kmm vec(C) = vec(C 0 ) = vec(C).

Using this and Theorem 8.26, we have {vec(C)}0 (A ⊗ B) vec(C)

=

{vec(C)}0 (A ⊗ B)Kmm vec(C)

=

{vec(C)}0 Kmm (B ⊗ A) vec(C)

= {Kmm vec(C)}0 (B ⊗ A) vec(C) = {vec(C)}0 (B ⊗ A) vec(C).

8.29 Using Theorem 8.11, we have vec(AB) = vec(ABIp ) = (Ip ⊗ A) vec(B), vec(AB) = vec(Im AB) = (B 0 ⊗ Im ) vec(A), and vec(AB) = vec(AIn B) = (B 0 ⊗ A) vec(In ).

8.31 Using Theorem 8.10 and the Cauchy-Schwarz inequality, {tr(A0 B)}2

=

[{vec(A)}0 vec(B)]2

≤

{vec(A)}0 vec(A){vec(B)}0 vec(B)

=

{tr(A0 A)}{tr(B 0 B)}.

We have equality if and only if one of the vectors, vec(A) or vec(B), is a scalar multiple of the other, or in other words, one of the matrices, A or B, is a scalar multiple of the other.

83

8.33 Since Im =

Pm

0 i=1 ei ei , we have

vec(Im ) = vec

m X

! ei e0i

=

i=1

m X

vec(ei e0i ) =

i=1

m X

ei ⊗ ei ,

i=1

where the last equality follows from Theorem 8.9(b).

8.35 (a) Straightforward computation yields  A

B=



4

2

2

12

.

(b) The characteristic equation of A is |A − λI2 | = λ2 − 5λ = 0. Thus, the eigenvalues of A are 0 and 5, so it is positive semidefinite. The characteristic equation of B is |B − λI2 | = λ2 − 7λ + 11 = 0. √ √ Thus, the eigenvalues of B are 21 (7 + 5) and 21 (7 − 5). Since these are both positive, B is positive definite. The characteristic equation of A B is |A B − λI2 | = λ2 − 16λ + 44 = 0. √ Thus, the eigenvalues of A B are 8 ± 20, so it is also positive definite. The fact that A B is positive definite follows from Theorem 8.17.

8.37 The identity holds since tr{(A0

B 0 )C}

=

n X {(A0

n X

i=1

i=1

=

n X (A

n X m X

B 0 )C}ii = B)0·i (C)·i =

i=1

(A0

B 0 )i· (C)·i aji bji cji ,

i=1 j=1

and 0

tr{A (B

C)}

= =

n X i=1 n X

0

{A (B

C)}ii =

(A)0·i (B

C)·i =

n X

(A0 )i· (B

i=1 n X m X

C)·i

aji bji cji .

i=1 j=1

i=1

8.39 (a) From Theorem 8.20, we have |A

B| ≥ |A|

m Y

! bii

.

i=1

Also, m Y

bii ≥ |B|,

i=1

when B is positive definite due to Theorem 8.18 and when B is positive semidefinte since |B| = 0 and the diagonal elements of B are nonnegative. Combining these two inequalitites yields (a).

84

(b) This follows directly from part (a) since A−1 | ≥ |A||A−1 | = |AA−1 | = |Im | = 1.

|A 8.41 (a) Since

 A clearly λ2 (A

B=

8

0

0

3

 ,

B) = 3. The lower bound from Theorem 8.21 is B) ≥ λ2 (A)(min bii ) = 1(2) = 2,

λ2 (A

i

while Theorem 8.23 yields the bound λ2 (A

B) ≥ λ2 (AB) = 3.

The bound from Theorem 8.23 is closer since it gives the actual value. (b) Since  A clearly λ2 (A

B=

2

0

0

3

 ,

B) = 2. The lower bound from Theorem 8.21 is B) ≥ λ2 (A)(min bii ) = 1(2) = 2.

λ2 (A

i

It is easily shown that AB = B has eigenvalues 1 and 4, so Theorem 8.23 yields the bound λ2 (A

B) ≥ λ2 (AB) = 1.

The bound from Theorem 8.21 is closer since it gives the actual value. −1/2

8.43 (a) Note that RA = DA −1/2

and DB |RA

−1/2

= diag(b11

−1/2

ADA

−1/2

and RB = DB

−1/2

BDB

−1/2

, where DA

−1/2

= diag(a11

−1/2

, . . . , amm )

−1/2

, . . . , bmm ). Then using Theorem 8.13(h), we find that

RB | + |RA ||RB | = =

−1/2

|DA

−1/2

ADA

|DA |−1 |DB |−1 {|A

85

−1/2

DB

−1/2

BDB

−1/2

| + |DA

B| + |A||B|},

−1/2

ADA

−1/2

||DB

−1/2

BDB

|

while |RA | + |RB | = = =

−1/2

|DA

−1/2

ADA

−1/2

| + |DB

−1/2

BDB

|

|DA |−1 |DB |−1 {|A||DB | + |B||DA |} ( ) m m Y Y −1 −1 |DA | |DB | |A| bii + |B| aii , i=1

i=1

so the two inequalities are equivalent. −1 (b) Let ρ = e01 RB e1 and

li+1 = |RAi

RBi | + |RAi ||RBi | − |RAi | − |RBi |,

i = 1, . . . , m − 1,

where RAi and RBi denote the matrices obtained by deleting the first i rows and columns of RA and RB . Note that we need to show that l1 ≥ 0. It follows from Theorem 8.19 that C is nonnegative definite and, hence, by Theorem 8.16, RA

C is nonnegative definite. An application

of Theorem 8.20 yields |RA |(1 − ρ−1 ) ≤

|RA

=

|RA

C| = |RA RB − ρ−1 e1 e01 |

|RA1 RB1 | .

RB | 1 − |RA RB |ρ

The last step was obtained by expanding along the first row. This last inequality can be equivalently expressed as |RA

RB | − |RA1

RB1 |/ρ ≥ |RA | − |RA |/ρ.

Using this and the fact that |RB | = |RB1 |/ρ, we have l1 − l2 /ρ

= |RA

RB | − |RA1

RB1 |/ρ + |RA ||RB | − |RA1 ||RB1 |/ρ

+ |RA1 |/ρ − |RA | + |RB1 |/ρ − |RB | ≥

|RA | − |RA |/ρ + |RA ||RB | − |RA1 ||RB1 |/ρ + |RA1 |/ρ − |RA | + |RB1 |/ρ − |RB |

=

(ρ−1 − |RB |)(|RA1 | − |RA |).

Note that ρ−1 − |RB | = ρ−1 (1 − |RB1 |) ≥ 0 since RB1 is a correlation matrix, and so we must have |RB1 | ≤ 1. In addition, |RA1 | − |RA | = |RA1 |(1 − |RA |/|RA1 |) ≥ 0 follows from Theorem 8.19. Thus, we have established that l1 ≥ l2 |RB |/|RB1 |. In a similar fashion, we 86

get li ≥ li+1 |RBi−1 |/|RBi | for i = 2, . . . , m − 1.

These inequalities lead to the inequality

l1 ≥ lm |RB |/|RBm−1 |. The result then follow since lm = 1 + 1 − 1 − 1 = 0.

8.45 It follows from Theorem 8.16 that A elements, so does A

B is nonnegative definite, and since B has r positive diagonal

B. If the ith diagonal element of A

B)ei = 0 (see Problem Q B| ≥ |A| bii of Theorem

B) ≤ r. Now apply the formula |A

3.51), and so we must have rank(A 8.20 on the submatrix of A

B is 0, then (A

B obtained by deleting the rows and columns corresponding to the 0 B has an r × r nonsingular submatrix and so rank(A

diagonal elements. This show that A

Combining these two inequalities for rank(A

B) yields rank(A

B) ≥ r.

B) = r.

8.47 If R 6= Im , then we must have λ < 1, so the matrix R − λIm has all of its diagonal elements positive.

It then follows from Theorem 8.17, that R normalized eigenvector of R 0

(R − λIm ) is positive definite. Consequently, if x is a

R corresponding to the eigenvalue τ , then

<

x0 {R

(R − λIm )}x = x0 (R

=

τ − λx0 Im x = τ − λ,

R)x − x0 (R

λIm )x

which yields the desired result.

8.49 With C = P (A ⊗ B)P 0 and D = C −1 partitioned into the 2 × 2 form of (7.1), we have from Problem −1 7.10, that D11 − C11 is nonnegative definite. Since C11 = A

B and

D = C −1 = {P (A ⊗ B)P 0 }−1 = P (A−1 ⊗ B −1 )P 0 so that D11 = A−1

B −1 , part (a) follows. Parts (b) and (c) are special cases of part (a) in which

B = A and B = A−1 , respectively.

8.51 



1

   0 K22 =    0  0

0 0 1 0

0

0



  1 0  ,  0 0   0 1

1

   0    0    0 K24 =    0    0    0  0

87

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

1

1

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0



  0    0    0  .  0    0    0   1

8.53 Using Theorem 8.1(g), we have Kmn

=

m X n X

eim e0jn ⊗ ejn e0im =

i=1 j=1

=

m X i=1

=

 eim ⊗

m X n X

eim ⊗ e0jn ⊗ ejn ⊗ e0im

i=1 j=1 n X

 (e0jn ⊗ ejn ) ⊗ e0im  =

j=1

m X

 eim ⊗

i=1

n X

 ejn e0jn ⊗ e0im 

j=1

m X (eim ⊗ In ⊗ e0im ). i=1

Consequently, 0 Kmn (x ⊗ A ⊗ y 0 )

=

m m X X (e0im x ⊗ A ⊗ eim y 0 ) = xi (A ⊗ eim y 0 ) i=1

=

A⊗

i=1 m X

! xi eim

y 0 = A ⊗ xy 0 .

i=1

8.55 (a) Using Theorem 8.26(a), we have 0 P 0 = {Kmn (A0 ⊗ A)}0 = (A0 ⊗ A)0 Kmn = (A ⊗ A0 )Knm = Kmn (A0 ⊗ A) = P.

(b) Since Kmn is nonsingular, it follows that rank(P ) = rank(A0 ⊗ A) = rank(A0 ) rank(A) = r2 . (c) Using Theorem 8.26(f), we have tr(P ) = tr{Kmn (A0 ⊗ A)} = tr{(A0 ⊗ A)Kmn } = tr(A0 A).

0 (d) Since P is symmetric and Kmn Kmn = Imn , we find that 0 P 2 = P 0 P = (A ⊗ A0 )Kmn Kmn (A0 ⊗ A) = (A ⊗ A0 )(A0 ⊗ A) = (AA0 ⊗ A0 A).

(e) It follows from part (d) and Theorem 8.5 that the r2 nonzero eigenvlaues of P 2 are λi λj , i = 1, . . . , r; j = 1, . . . , r. We then get the nonzero eigenvalues of P , except for sign, by taking the square roots of these eigenvalues. However, tr(P ) = tr(A0 A) =

r X

λi ,

i=1

and so the nonzero eigenvalues of P must be λi , i = 1, . . . , r, and ±(λi λj )1/2 for all i < j.

88

8.57 Using Problem 8.56, we have vec(In ⊗ Iq ) = (Knq,n ⊗ Iq ){vec(In ) ⊗ vec(Iq )}, and vec(A0 ⊗ B 0 ) = (Kmp,n ⊗ Iq ){vec(A) ⊗ vec(B 0 )}.

0 Now applying Theorem 8.10 and the fact that Knq,n Kmp,n = Imnp , we get

tr(A ⊗ B)

=

tr(A0 ⊗ B 0 ) = tr{(In ⊗ Iq )(A0 ⊗ B 0 )}

= {vec(In ⊗ Iq )}0 vec(A0 ⊗ B 0 ) 0 = {vec(In ) ⊗ vec(Iq )}0 (Knq,n ⊗ Iq )(Kmp,n ⊗ Iq ){vec(A) ⊗ vec(B 0 )}

= {vec(In ) ⊗ vec(Iq )}0 {vec(A) ⊗ vec(B 0 )}.

8.59 (a) Since (B ⊗ B)Kmm = Kmm (B ⊗ B), we have (B ⊗ B)Nm

= =

1 1 1 (B ⊗ B)(Im2 + Kmm ) = B ⊗ B + (B ⊗ B)Kmm 2 2 2 1 1 1 B ⊗ B + Kmm (B ⊗ B) = (Im2 + Kmm )(B ⊗ B) = Nm (B ⊗ B).

2 2 2

2 = Nm , Further, using this and the fact that Nm

Nm (B ⊗ B)Nm = Nm Nm (B ⊗ B) = Nm (B ⊗ B). (b) Using part (a), (B ⊗ B)Nm (B 0 ⊗ B 0 ) = Nm (B ⊗ B)(B 0 ⊗ B 0 ) = Nm (BB 0 ⊗ BB 0 ) = Nm (A ⊗ A).

8.61 (a) The result follows since m

m

1 XX (ei e0i ⊗ ej e0j + ei e0j ⊗ ej e0i ) 2 i=1 j=1

=

=

  m X m m m  X 1 X X (ei e0i ⊗ ej e0j ) + (ei e0j ⊗ ej e0i )  2  i=1 j=1 i=1 j=1    m m  X 1 X ei e0i ⊗ ej e0j  + Kmm  2 i=1

=

89

j=1

1 1 {(Im ⊗ Im ) + Kmm } = (Im2 + Kmm ) = Nm .

2 2

(b) For any m × 1 vectors a and b, Nm (a + b) =

1 1 1 (Im2 + Kmm )(a ⊗ b) = {a ⊗ b + Kmm (a ⊗ b)} = (a ⊗ b + b ⊗ a).

2 2 2

(c) The result follows since 1 (a ⊗ b ⊗ c + a ⊗ c ⊗ b + b ⊗ a ⊗ c + b ⊗ c ⊗ a + c ⊗ a ⊗ b + c ⊗ b ⊗ a) 6 m m m 1 XXX = (eh ah ⊗ ei bi ⊗ ej cj + eh ah ⊗ ei ci ⊗ ej bj 6 i=1 j=1 h=1

+ eh bh ⊗ ei ai ⊗ ej cj + eh bh ⊗ ei ci ⊗ ej aj + eh ch ⊗ ei ai ⊗ ej bj + eh ch ⊗ ei bi ⊗ ej aj m m m 1 XXX (eh e0h a ⊗ ei e0i b ⊗ ej e0j c + eh e0h a ⊗ ei e0j b ⊗ ej e0i c 6 i=1 j=1

=

h=1

+ eh e0i a ⊗ ei e0h b ⊗ ej e0j c + eh e0j a ⊗ ei e0h b ⊗ ej e0i c + eh e0i a ⊗ ei e0j b ⊗ ej e0h c + eh e0j a ⊗ ei e0i b ⊗ ej e0h c =

∆(a ⊗ b ⊗ c).

8.63 Note that A

=

X

aij Eij =

i,j

=

X

X

X

aii Eii +

i

aii Tii +

i

X

aij (Eij + Eji )

i>j

aij Tij =

i>j

X

aij Tij ,

i≥j

and so vec(A) =

X

vec(Tij )aij =

i≥j

X

vec(Tij )u0ij v(A)

i≥j

since aij = u0ij v(A).

8.65 (a) First suppose B is an m × m symmetric matrix and x0 Bx = 0 for all m × 1 vectors x. Then, in particular, e0i Bei = 0 for i = 1, . . . , m, and (ei + ej )0 B(ei + ej ) = 0

90

for all i 6= j. Together these two identities imply that bii = 0 for all i and bij = 0 for all i 6= j, and so we must have B = (0). Now for an arbitrary symmetric m × m matrix A, 0 {v(A)}0 Dm Dm v(A)

=

{vec(A)}0 vec(A) =

=

X

m X m X

a2ij

i=1 j=1

2

a2ij −

a2ii

i=1

i≥j

=

m X

2{v(A)}0 v(A) −

m X {v(A)}0 uii u0ii v(A) i=1

0

= {v(A)}

2Im(m+1)/2 −

m X

! uii u0ii

v(A).

i=1 0 Thus, we have shown that y 0 By = 0 for all m(m + 1)/2 × 1 vectors y when B = {Dm Dm − Pm (2Im(m+1)/2 − i=1 uii u0ii )} and, since B is symmetric, the result follows.

0 (b) From part (a), we see that Dm Dm is a diagonal matrix with m diagonal elements equal to 1

and m(m − 1)/2 diagonal elements equal to 2. Since the determinant of a diagonal matrix is the 0 Dm | = 2m(m−1)/2 . product of its diagonal elements, we have |Dm + = Nm , so 8.67 (a) From Theorem 8.33(c), Dm Dm + Dm Dm (A ⊗ A)Dm = Nm (A ⊗ A)Dm = (A ⊗ A)Nm Dm = (A ⊗ A)Dm .

(b) We prove the result by induction. The result holds for i = 2 since + {Dm (A ⊗ A)Dm }2

+ + = Dm (A ⊗ A){Dm Dm (A ⊗ A)Dm } + + = Dm (A ⊗ A)(A ⊗ A)Dm = Dm (A2 ⊗ A2 )Dm .

If the result holds for i, then it holds for i + 1 since + {Dm (A ⊗ A)Dm }i+1

=

+ + {Dm (A ⊗ A)Dm }i {Dm (A ⊗ A)Dm }

=

+ + Dm (Ai ⊗ Ai )Dm Dm (A ⊗ A)Dm

=

+ Dm (Ai ⊗ Ai )(A ⊗ A)Dm

=

+ Dm (Ai+1 ⊗ Ai+1 )Dm .

P 8.69 Since A is lower triangular, A = i≥j aij Eij . Thus, using the fact that aij = u0ij v(A), we get   X  X X vec(A) = vec aij Eij = vec(Eij )aij = {vec(Eij )}u0ij v(A).   i≥j

i≥j

91

i≥j

8.71 Note that if A is strictly lower triangular, then vec(A)0 vec(A) = ṽ(A)0 ṽ(A), and so 0 = vec(A)0 vec(A) − ṽ(A)0 ṽ(A) = ṽ(A)0 L̃m L̃0m ṽ(A) − ṽ(A)0 ṽ(A) = ṽ(A)0 (L̃m L̃0m − Im(m−1)/2 )ṽ(A).

Since any x ∈ Rm(m−1)/2 can be expressed as x = ṽ(A) for some strictly lower triangular matrix A, this implies that L̃m L̃0m − Im(m−1)/2 = (0) (see the solution to Problem 8.65(a)), which proves (b).

But (b) implies that L̃m is full row rank, which then yields (a) and (c) since 0 0 −1 L̃+ = L̃0m Im(m−1)/2 = L̃0m . m = L̃m (L̃m L̃m )

To prove (d), for arbitrary A, write A = AL + AU , where AL is strictly lower triangular and AU is upper triangular. Then 0 = vec(AL )0 vec(AU ) = ṽ(AL )0 L̃m vec(AU ), regardless of the choice of AL , so we must have L̃m vec(AU ) = 0. Thus, we have L̃m vec(A)

= L̃m vec(AL + AU ) = L̃m vec(AL ) = L̃m L̃0m ṽ(AL ) = ṽ(AL ) = ṽ(A).

8.73 Let A be an arbitrary strictly lower triangular matrix. Then Kmm L̃0m ṽ(A) = Kmm vec(A) = vec(A0 ), and since A0 is strictly upper triangular Lm vec(A0 ) = v(A0 ) = 0, and L̃m vec(A0 ) = ṽ(A0 ) = 0.

Thus, ṽ(A)0 L̃m Kmm L̃0m = vec(A0 )0 L̃0m = 00 , and ṽ(A)0 L̃m Kmm L0m = vec(A0 )0 L0m = 00 , which proves (a) and (b). We get (d) and (e) since for any strictly lower triangular matrix A, L0m Lm L̃0m ṽ(A) = L0m Lm vec(A) = L0m v(A) = vec(A) = L̃0m ṽ(A), 92

and Dm Lm L̃0m ṽ(A)

=

Dm Lm vec(A) = Dm v(A) = vec(A + A0 − DA )

=

vec(A) + vec(A0 ) = (Im2 + Kmm ) vec(A)

=

2Nm vec(A) = 2Nm L̃0m ṽ(A).

To get (c), postmultiply the transpose of (d) by Dm and then use Theorem 8.37(a). Finally, (f) follows from (d) and Theorem 8.38(b).

8.75 First suppose that A is an m × m nonsingular positive matrix and let B = A−1 . For i 6= j, (BA)ij =

m X

bik akj = 0.

(1)

k=1

Since akj > 0 for all k, (1) holds only if bik < 0 for at least one k or bik = 0 for all k. However, this second option is not possible since B is nonsingular, and so B is not nonnegative. Now suppose both A and B are nonnegative and assume that the jth column of A has more than one nonzero component. If ahj > 0 and alj > 0, then for (1) to hold, we must have bih = bil = 0 for all i 6= j. But this implies that the hth and lth columns of B are both scalar multiples of ej . This cannot be since B is nonsingular, and so each column of A has one nonzero component.

8.77 Consider the matrices

 A=

0

1

0

0





,

B=

1

0

0

2

 .

Both matrices are clearly reducible with A having both of its eigenvalues equal to 0, while B has eigenvalues of 1 and 2. (a) It follows that ρ(A) = 0. (b) An eigenvector of B corresponding to the eigenvalue ρ(B) = 2 is of the form (0, x2 ), which is not positive. (c) ρ(A) = 0 is a multiple eigenvalue of A. 8.79 (a) Let λ1 , . . . , λm be the eigenvalues of A, so that λ1 + 1, . . . , λm + 1 are the eigenvalues of Im + A. Then ρ(Im + A) = max |λi + 1| ≤ max |λi | + 1 = 1 + ρ(A).

1≤i≤m

1≤i≤m

93

However, since A is nonnegative, ρ(A) is an eigenvalue of A and, hence, 1 + ρ(A) is an eigenvalue of Im + A. This confirms that ρ(Im + A) = 1 + ρ(A). (b) If λ1 , . . . , λm are the eigenvalues of A, λk1 , . . . , λkm are the eigenvalues of Ak . Since A is nonnegative, ρ(A) is an eigenvalue of A and if it is a multiple eigenvalue of A, then ρ(A)k = ρ(Ak ) would be a multiple eigenvalue of Ak . However, from Theorem 8.45, we know this is not possible since Ak is positive. (c) It follows from part (a) that if ρ(A) is a multiple eigenvalue of A, then 1 + ρ(A) = ρ(Im + A) is a multiple eigenvalue of Im + A. But A is an irreducible nonnegative matrix and so by Theorem 8.46, (Im + A)m−1 is positive, and then according to part (b), 1 + ρ(A) is a simple eigenvalue of Im + A. Thus, ρ(A) must be a simple eigenvalue of A. 8.81 Since A is nonnegative and nonnegative definite, we know that aij ≥ 0 for all i and j, and a11 −a212 /a22 ≥ 0 if a22 > 0 (see Problem 7.8). We will find B of the form   a b , B= 0 c so that

 BB 0 = 

a2 + b2

bc

bc

c2





=

a11

a12

a12

a22

 .

If a22 = 0, a12 = 0 since A is nonnegative definite (Problem 3.51), and so BB 0 = A with   1/2 a11 0 .

B= 0 0 If a22 > 0, using (1), we solve for c, b and a to get 1/2

c = a22 ,

1/2

b = a12 /a22 ,

a = (a11 − a212 /a22 )1/2 .

Thus, since B is nonnegative, we have shown that A is completely positive.

8.83 Now θ = cos(2π/m) + i sin(2π/m) = e2πi/m and its complex conjugate is θ̄ = cos(2π/m) − i sin(2π/m) = cos(−2π/m) + i sin(−2π/m) = e−2πi/m = θ−1 .

Similarly, θh = cos(2πh/m) + i sin(2πh/m) = e2πhi/m and θ¯h = cos(2πh/m) − i sin(2πh/m) = cos(−2πh/m) + i sin(−2πh/m) = e−2πhi/m = θ−h .

94

(1)

This confirms that the conjugate transpose of F is as given in the problem. Now √ and m(F ∗ )kj = θ−(k−1)(j−1) , so m(F F ∗ )ij

= m(F )i· (F ∗ )·j =

m X

√

m(F )ik = θ(i−1)(k−1)

θ(i−1)(k−1) θ−(k−1)(j−1)

k=1 m X

=

θ(k−1)(i−j) =

k=1

m−1 X

{θi−j }k .

k=0

When i = j, this reduce to m. Otherwise m−1 X

{θi−j }k =

k=0

1 − (θm )i−j 1 − 1i−j 1 − (θi−j )m = = = 0, i−j i−j 1−θ 1−θ 1 − θi−j

and so we have shown that F F ∗ = Im . = (e2 , e3 , . . . , em , e1 ), we find that 8.85 Since Πm−1 m 

e02



   0   e3  m    ..  X m−1 Πm = Π Π = [e , e , . . . , e , e ] = ei e0i = Im .   m 2 3 m 1 m m  .   0  i=1  em    e01

This proves (b), while multiplying the identity in (b) by Π−1 m leads to (a). Finally, using (b), we have mn+r n r n r r Πm = (Πm m ) Πm = (Im ) Πm = Πm ,

which establishes (c).

8.87 The jth eigenvalue of circ(1, . . . , 1) is given by δj =

m−1 X

m−1 X

k=0

k=0

(θj−1 )k =

1 = m,

if j = 1, and δj =

m−1 X k=0

(θj−1 )k =

1 − (θm )j−1 1−1 1 − (θj−1 )m = = = 0, j−1 j−1 1−θ 1−θ 1 − θj−1

if j = 2, . . . , m. That is, it has the eigenvalue m and the eigenvalue 0 repeated m − 1 times.

95

8.89 The matrix



0

  A=B= 1  0

1 0 0

0



  0   1

is not a circulant matrix, while AB = I3 is a circulant matrix.

8.91 Now (AA−1 )11 = (AA−1 )mm = c − abc = c(1 − ab) = (1 − ab)−1 (1 − ab) = 1, while (AA−1 )jj = −abc + (ab + 1)c − abc = c − abc = 1, for j = 2, . . . , m − 1. Also (AA−1 )j1 = aj−1 c − aj−1 c = 0, for j = 2, . . . , m, and (AA−1 )jm = bm−j c − bm−j c = 0, for j = 1, . . . , m − 1. Finally, (AA−1 )ij

= −bcbj−i−1 + (ab + 1)cbj−i − acbj−i+1 = bj−i−1 (−bc + ab2 c + bc − ab2 c) = 0

for j = 2, . . . , m − 1, i = 1, . . . , j − 1, and (AA−1 )ij

=

−bcai−j+1 + (ab + 1)cai−j − acai−j−1

=

ai−j−1 (−a2 bc + a2 bc + ac − ac) = 0

for j = 2, . . . , m−1, i = j+1, . . . , m, so that the matrix given is the inverse of A. If ab = 1, then b = 1/a, (A)·1 = (1, a, a2 , . . . , am−1 )0 , (A)·2 = (1/a, 1, a, . . . , am−2 )0 , and (A)·1 − a(A)·2 = 0, so rank(A) < m.

8.93 We will prove the result by showing that |Ah | = (1 − ρ2 )h−1 ,

(1)

where Ah is the hth leading principal submatrix of A. Clearly, |A1 | = 1 and |A2 | = 1 − ρ2 , so (1) holds for h = 1 and h = 2. We show that it holds for general h by induction. Suppose (1) holds for h − 1, so

96

that |Ah−1 | = (1 − ρ2 )h−2 . The determinant of B = Ah can be computed by expanding along its first row to get |Ah | = |B| =

h X

ρi−1 B1i = B11 + ρB12 .

(2)

i=1

The last equality follows since when i > 2, the matrix obtained from Ah by deleting its first row and ith column has its first and second columns being scalar multiples of one another. In addition, note that B11 is identical to |Ah−1 | and, upon using Theorem 1.4(f), we find that B12 = (−1)1+2 ρ|Ah−1 | = −ρ|Ah−1 |.

Thus, (2) becomes |Ah | = |Ah−1 | − ρ2 |Ah−1 | = (1 − ρ2 )|Ah−1 | = (1 − ρ2 )(1 − ρ2 )h−2 = (1 − ρ2 )h−1 .

The result follows from Theorem 7.6 since the quantity in (1) is positive for all h.

8.95 

1

   1    1    1    1    1 H=   1    1    1    1    1  1

1 −1 −1

1

1

1

1

1

1

1

1

1 −1

1

1

1

−1

−1

−1

1 −1

1

1

1

−1

−1

1 −1

1

1

1

−1

1 −1

1

1

1

1 −1

1

1

1 −1

1

−1

1 −1 −1

−1

1 −1

−1

−1

−1

1 −1

−1

−1

−1

−1

1 −1

−1

−1

1

1 −1

−1

−1

1

1

1

−1

−1

−1

−1

1

1

1

−1

−1

−1

1 −1

1

1

1

−1

−1

1 −1

−1

1 −1

−1

1 −1

1 −1 −1

1

1 −1

−1

1 −1 −1

1

1

1



  1 −1    −1 1    −1 −1    −1 −1    1 −1  .  1 1    1 1    −1 1    1 −1    −1 1   −1 −1

8.97 Since each element of H is 1 or −1, we simply need to verify that HH 0 = 4mI4m . Partitioning HH 0 as H is partitioned, we find that each submatrix along the diagonal is AA0 + BB 0 + CC 0 + DD0 = 4mIm , as required. The (1, 2) submatrix of HH 0 is −AB 0 + BA0 − CD0 + DC 0 = −AB 0 + AB 0 − CD0 + CD0 = (0), 97

since BA0 = AB 0 and DC 0 = CD0 . In a similar fashion, using the condition XY 0 = Y X 0 , it can be shown that the remaining off-diagonal submatrices of HH 0 are null matrices.

8.99 The result follows immediately from Theorem 8.61 since A is nonsingular if and only if |A| 6= 0, and each term in the product given in (8.28) is nonzero if and only if ai 6= aj for all i 6= j.

8.101 Using (8.27), we find that

where bi =



b0

b1

b2

···

bm−1



     AA0 =     

b1

b2

b3

···

bm

b2 .. .

b3 .. .

b4 .. .

···

bm+1 .. .

bm−1

bm

bm+1

···

     ,    

b2m−2



bm−1

bm−2

bm−3

···

b0



   bm   AA0 P = AA0 [em , em−1 , . . . , e1 ] =  bm+1  ..   .  b2m−2

bm−1

bm−2

···

b1

bm .. .

bm−1 .. .

···

b2 .. .

b2m−3

b2m−4

···

         

bm−1

Pm

i k=1 ak . Then

is a Toeplitz matrix since it has the form given in (8.23). Similarly, we find that  



bm−1

 e0m     bm−2  0   e   m−1  AA0 =  P AA0 = P 0 AA0 =   bm−3 ..      .  ..    .  0 e1 b0 is also a Toeplitz matrix.

98



bm

bm+1

···

b2m−2

bm−1

bm

···

bm−2 .. .

bm−1 .. .

···

b1

b2

···

  b2m−3    b2m−4   ..   .  bm−1

Chapter 9 9.1 (a) With f (x) = log(x), we have f (i) (x) = (−1)i−1 (i − 1)!x−i . Consequently, the kth-order Taylor formula is given by log(1 + u)

=

log(1) +

k X ui (−1)i−1 (i − 1)!1−i

i!

i=1

=

k X ui (−1)i−1

i

i=1

+ rk (u, 1)

+ rk (u, 1).

(b) We get log(1.1) ≈ .1 − 9.3 Since

(.1)2 (.1)3 (.1)4 (.1)5 + − + = .0953103.

2 3 4 5

 −z2 ∂  z12 g = ∂z 0 z

2

and

 ∂ f = ∂x0

1 z1

z1

 ,

2x1

2x2

2x3

2

−1

−1

 ,

we have ∂ y ∂x0

=

∂ g ∂f 0

∂ f ∂x0

 =

2(x21 +x22 +x23 )−2x1 (2x1 −x2 −x3 ) (x21 +x22 +x23 )2

2(x21 + x22 + x23 ) + 2x1 (2x1 − x2 − x3 )

−(x21 +x22 +x23 )−2x2 (2x1 −x2 −x3 ) (x21 +x22 +x23 )2

−(x21 +x22 +x23 )−2x3 (2x1 −x2 −x3 ) (x21 +x22 +x23 )2

−(x21 + x22 + x23 ) + 2x2 (2x1 − x2 − x3 ) −(x21 + x22 + x23 ) + 2x3 (2x1 − x2 − x3 ) 9.5 We have df

= = =

x0 Bxd(x0 Ax) − x0 Axd(x0 Bx) (x0 Bx)2 0 x Bx{(dx)0 Ax + x0 Adx} − x0 Ax{(dx)0 Bx + x0 Bdx} (x0 Bx)2 0 0 0 2(x Bxx Adx − x Axx0 Bdx) , (x0 Bx)2

and so ∂ 2(x0 Bxx0 A − x0 Axx0 B) f= .

0 ∂x (x0 Bx)2 99

 .

9.7 (a) We have d tr(AX) = tr(AdX) = vec(A0 )0 vec(dX) = vec(A0 )0 d vec(X), so ∂ tr(AX) = vec(A0 )0 . ∂ vec(X)0 (b) Similarly, we get d tr(AXBX)

=

tr(A(dX)BX) + tr(AXBdX)

=

tr(BXAdX) + tr(AXBdX)

=

vec(A0 X 0 B 0 )0 vec(dX) + vec(B 0 X 0 A0 )0 vec(dX)

= {vec(A0 X 0 B 0 )0 + vec(B 0 X 0 A0 )0 }d vec(X), so ∂ tr(AXBX) = vec(A0 X 0 B 0 )0 + vec(B 0 X 0 A0 )0 . ∂ vec(X)0 9.9 Using Corollary 9.1.1, we have d log |X 0 AX| =

tr{(X 0 AX)−1 d(X 0 AX)}

=

tr[(X 0 AX)−1 {(dX)0 AX + X 0 AdX}]

=

2 tr{(X 0 AX)−1 X 0 AdX}

=

2 vec{AX(X 0 AX)−1 }0 d vec(X),

and so ∂ log |X 0 AX| = 2 vec{AX(X 0 AX)−1 }0 . d vec(X)0 9.11 Since dX n = (dX)X n−1 + X(dX)X n−2 + · · · + X n−1 dX =

n X

X j−1 (dX)X n−j ,

j=1

we find that d vec(X n )

=

vec

 n X 

=

n X

X j−1 (dX)X n−j

 

=



j=1

(X n−j0 ⊗ X j−1 )d vec(X),

j=1

100

n X j=1

vec{X j−1 (dX)X n−j }

and so

n X ∂ n vec(X ) = (X n−j0 ⊗ X j−1 ). ∂ vec(X)0 j=1

9.13 Since X# = |X|X −1 , we have dX#

=

(d|X|)X −1 + |X|dX −1 = |X| tr(X −1 dX)X −1 − |X|X −1 (dX)X −1

=

|X|{tr(X −1 dX)X −1 − X −1 (dX)X −1 }

and d vec(X# )

=

|X|{tr(X −1 dX) vec(X −1 ) − vec(X −1 (dX)X −1 )}

=

|X|{vec(X −1 ) vec(X −10 )0 d vec(X) − (X −10 ⊗ X −1 )d vec(X)},

and so ∂ vec(X# ) = |X|{vec(X −1 ) vec(X −10 )0 − (X −10 ⊗ X −1 )}. ∂ vec(X)0 9.15 (a) Since dF = d(AXA0 ) = A(dX)A0 and d vec(F ) = vec{A(dX)A0 } = (A ⊗ A)d vec(X) = (A ⊗ A)Dm d v(X), we get ∂ vec(F ) = (A ⊗ A)Dm . ∂ v(X)0 (b) Since dF = d(XBX) = (dX)BX + XB(dX) and d vec(F )

=

vec{(dX)BX} + vec(XBdX)

=

(XB ⊗ Im )d vec(X) + (Im ⊗ XB)d vec(X)

= {(XB ⊗ Im ) + (Im ⊗ XB)}Dm d v(X), we find that ∂ vec(F ) = {(XB ⊗ Im ) + (Im ⊗ XB)}Dm . ∂ v(X)0 9.17 (a) Using Theorem 8.27, we get d vec(F )

=

vec{(dX) ⊗ X + X ⊗ dX}

=

(In ⊗ Knm ⊗ Im ){vec(dX) ⊗ vec(X) + vec(X) ⊗ vec(dX)}

=

(In ⊗ Knm ⊗ Im ){Imn ⊗ vec(X) + vec(X) ⊗ Imn }d vec(X).

101

It then follows that ∂ vec(F ) = (In ⊗ Knm ⊗ Im ){Imn ⊗ vec(X) + vec(X) ⊗ Imn }. ∂ vec(X)0 (b) The result follows since d vec(X

X)

=

vec{(dX)

X +X

=

2Dvec(X) d vec(X).

dX} = 2 vec(X

dX)

9.19 We prove the result by induction. The result holds when n = 1 due to Theorem 9.2. Suppose the result holds for n − 1. The result then holds for n since dn X −1

= d(dn−1 X −1 ) = d{(−1)n−1 (n − 1)!(X −1 dX)n−1 X −1 (−1)n−1 (n − 1)![{d(X −1 dX)n−1 }X −1 + (X −1 dX)n−1 dX −1 ]  " n−1 X = (−1)n−1 (n − 1)!  (X −1 dX)j−1 {d(X −1 dX)}(X −1 dX)n−1−j  X −1

=

j=1

# − (X

=

−1

n−1

dX)

X

−1

dXX

−1

 " n−1 X (X −1 dX)j−1 (X −1 dX)2 (X −1 dX)n−1−j  X −1 (−1)n−1 (n − 1)!(−1)  j=1

# + (X −1 dX)n X −1 =

(−1)n (n − 1)!{(n − 1)(X −1 dX)n X −1 + (X −1 dX)n X −1 }

=

(−1)n n!(X −1 dX)n X −1 .

9.21 Using the perturbation formula for a matrix inverse given in Section 9.6, we have (Im + Y )−1 = Im − Y + 2 Y 2 − 3 Y 3 + 4 Y 4 − · · · .

Writing (Im + Y )−1/2 = Im + −1/2

(Im + Y )

(Im + Y )

−1/2

(1)

P∞

i i=1 Bi , we also have

=

Im +

∞ X

!2 i

Bi

i=1

=

Im + 2B1 + 2 (2B2 + B12 ) + 3 (2B3 + B1 B2 + B2 B1 ) + 4 (2B4 + B1 B3 + B3 B1 + B22 ) + · · · 102

(2)

Solutions for the Bi ’s can be obtained by equating (1) and (2). First we must have −Y = 2B1 , so B1 = − 21 Y . Next, we have

1 Y 2 = 2B2 + B12 = 2B2 + Y 2 , 4

and so

1 B2 = 2

3 2 Y 4

=

3 2 Y .

8

Then we have −Y 3 = 2B3 + B1 B2 + B2 B1 = 2B3 −

3 3 3 Y − Y 3, 16 16

and this leads to 1 B3 = 2

10 5 − Y 3 = − Y 3.

16 16

Finally, we have Y 4 = 2B4 + B1 B3 + B3 B1 + B22 = 2B4 +

5 4 5 9 Y + Y 4 + Y 4, 32 32 64

and this yields 1 B4 = 2

35 64

Y4 =

35 4 Y .

128

9.23 Note that DS can be written as DS = Im + DA . Then using the result from Problem 9.21, we have R

−1/2

−1/2

= DS SDS

1 3 2 5 3 1 3 2 5 3 = Im − DA + DA − DA + · · · (Ω + A) Im − DA + DA − DA + ··· 2 8 16 2 8 16 1 3 2 2 = Ω + A − (ΩDA + DA Ω) + (DA Ω + ΩDA ) 2 8 1 1 3 2 2 + DA ΩDA − (ADA + DA A) + (DA A + ADA ) 4 2 8 1 3 5 2 2 3 3 + DA ADA − (DA ΩDA + DA ΩDA ) − (DA Ω + ΩDA ) 4 16 16 = Ω + C1 + C2 + C3 .

9.25 (a) From the equation (X + U )(cel + b1 ) = (xl + a1 )(Im + V )(cel + b1 ) we get the equation Xb1 + cU el = xl b1 + cxl V el + ca1 el

103

(1)

involving first order terms. Premultiplying (1) by e0l yields xl bl1 + cull = xl bl1 + cxl vll + ca1 , and when solving for a1 , we get a1 = ull − xl vll . (b) Solving (1) for b1 , we get b1 = c(X − xl Im )+ (xl V − U )el + yel ,

(2)

where y is an arbitrary scalar. Note that γ 0l γ l = 1 implies that e0l b1 = 0, and so this confirms that bl1 = 0. Premultiplying (2) by ei when i 6= l yields bi1 = e0i (X − xl Im )+ (xl V − U )el = −

uil − xl vil xi − xl

since c = 1. (c) γ 0l (Im + V )γ l = 1 and c = 1 leads to the equation vll + 2bl1 = 0, and so bl1 = −vll /2. The formula for bil is obtained as in part (b).

1/2

(d) γ 0l γ l = λl and c = xl

1/2

leads to the equation 2xl bl1 =

a1 1/2 2xl

=

bl1 = a1 , and so

ull − xl vll 1/2

.

2xl

Premultiplying (2) by ei when i 6= l yields 1/2 x (uil − xl vil ) 1/2 0 ei (X − xl Im )+ (xl V − U )el = − l xi − xl

bi1 = xl 1/2

since c = xl

.

9.27 (a) Setting h i ∂ f = 6x21 − 6 3x22 − 27 0 ∂x 0 equal to 0 yields the equations x21 = 1 and x22 = 9, from which we get the stationary points (1, 3), (−1, 3), (1, −3), and (−1, −3). (b) The Hessian matrix is  Hf = 

12x1

0

0

6x2

 .

Hf is positive definite when (x1 , x2 ) = (1, 3) so this point is a minimum. Hf is negative definite when (x1 , x2 ) = (−1, −3) so this point is a maximum. The other two are saddle points.

104

9.29 (a) The differential of f is df = d(x0 Bx) + d(a0 x) = (dx)0 Bx + x0 Bdx + a0 dx = 2x0 Bdx + a0 dx, and ∂ f = 2x0 B + a0 = 00 ∂x0 yields the equation Bx = − 12 a. It follows from Theorem 6.4 that x = − 12 B + a + (Im − B + B)y is a solution for any choice of y. (b) If B is nonsingular, then B + = B −1 and the soluiton given in part (a) becomes 1 1 x = − B −1 a + (Im − B −1 B)y = − B −1 a, 2 2 which is unique. Since d2 f = d(df ) = 2(dx)0 B(dx), Hf = 2B. Thus, this unique solution is a minimum when B is positive definite and it is a maximum when B is negative definite.

9.31 Following the development in the one sample problem given in Example 9.6, we need to maximize the function g(µ1 , . . . , µk , Ω)

=

−

k k 1X 1X ni log |Ω| − tr(Ω−1 Ui ) 2 i=1 2 i=1

k 1 1X = − n log |Ω| − tr(Ω−1 Ui ), 2 2 i=1

where Ui =

ni X (xij − µi )(xij − µi )0 . j=1

The first differential of g is given by dg

=

=

k 1 1X − nd(log |Ω|) − [tr{(dΩ−1 )Ui } + tr(Ω−1 dUi )] 2 2 i=1 k 1X 1 tr{Ω−1 (dΩ)Ω−1 Ui } − n tr(Ω−1 dΩ) + 2 2 i=1    ni ni k   X X X 1 + tr Ω−1 (dµi ) (xij − µi )0 + (xij − µi )dµ0i    2 i=1

j=1

k

=

1X tr{(dΩ)Ω−1 (Ui − ni Ω)Ω−1 } 2 i=1 105

j=1

k

+

=

1X tr(Ω−1 {ni (dµi )(xi − µi )0 + ni (xi − µi )dµ0i }) 2 i=1

k k X 1X vec(Ui − ni Ω)0 (Ω−1 ⊗ Ω−1 ) vec(dΩ) + ni (xi − µi )0 Ω−1 dµi .

2 i=1 i=1

Setting the derivative with respect to µi equal to 0 yields ni Ω−1 (xi − µi ) = 0, and this leads to the solution µ̂i = xi .

Setting the derivative with respect to vec(Ω) equal to 0 yields k X (Ω−1 ⊗ Ω−1 ) vec(Ui − ni Ω) = 0, i=1

or, equivalently, k X

(Ω−1 Ui Ω−1 − ni Ω−1 ) = (0).

i=1

Premultiplying and postmultiplying this equation by Ω and then solving for Ω, we get Ω̂

=

k k ni 1X 1 XX Ui = (xij − µ̂i )(xij − µ̂i )0 n i=1 n i=1 j=1

=

i 1 XX (xij − xi )(xij − xi )0 . n i=1 j=1

k

n

Computing the second differential and then evaluating at µi = xi and Ω = Ω̂, we get d2 g = −

k X n vec(dΩ)0 (Ω̂−1 ⊗ Ω̂−1 ) vec(dΩ) − ni (dµi )0 Ω̂−1 dµi .

2 i=1

Consequently,     Hg =    

−n1 Ω̂−1

(0)

···

(0)



(0) .. .

−n2 Ω̂−1 .. .

···

(0) .. .

(0)

(0)

···

− n2 Ω̂−1 ⊗ Ω̂−1

   .   

Clearly, Hg is negative definite and so our solutions do maximize g.

106

9.33 Let z 1 = (x01 , y1 )0 ∈ T and z 2 = (x02 , y2 )0 ∈ T , so that xi ∈ S and yi ≥ f (xi ) for i = 1, 2. We need to show that if 0 ≤ c ≤ 1 and z = (x0 , y)0 = cz 1 + (1 − c)z 2 , then z ∈ T . Now since xi ∈ S and S is a convex set, it follows that x = cx1 + (1 − c)x2 ∈ S. Also, since f is a convex function, f (x)

= f (cx1 + (1 − c)x2 ) ≤ cf (x1 ) + (1 − c)f (x2 ) ≤ cy1 + (1 − c)y2 = y.

This confirms that T is a convex set.

9.35 Let x1 ∈ S, x2 ∈ S, and 0 ≤ c ≤ 1. Applying the given inequality to x = x1 and a = cx1 + (1 − c)x2 leads to

f (x1 ) ≥ f (a) +

∂ f (a) (1 − c)(x1 − x2 ), ∂a0

while applying it to x = x2 and the same a yields

∂ f (x2 ) ≥ f (a) + f (a) c(x2 − x1 ). ∂a0 Multiplying the first of these equations by c and the second by (1 − c), and then combining gives us cf (x1 ) + (1 − c)f (x2 ) ≥ f (a) = f (cx1 + (1 − c)x2 ), and so f is convex.

9.37 (a) The differential of f is df

=

(dxc1 )x1−c + xc1 dx1−c 2 2

=

1−c c −c cxc−1 1 x2 dx1 + (1 − c)x1 x2 dx2 ,

while the second differential is d2 f

1−c c−1 1−c = c(dxc−1 1 )x2 dx1 + cx1 (dx2 )dx1 −c c + (1 − c)(dxc1 )x−c 2 dx2 + (1 − c)x1 (dx2 )dx2 c−1 −c 2 = c(c − 1)x1c−2 x1−c 2 dx1 + c(1 − c)x1 x2 dx1 dx2 −c c −c−1 + c(1 − c)xc−1 dx22 .

1 x2 dx1 dx2 − c(1 − c)x1 x2

Thus,  −c−1  Hf = c(1 − c)xc−2 1 x2

107

−x22

x1 x2

x1 x2

−x21

 .

Since the leading principal minors of −Hf are nonnegative, −Hf is nonnegative definite. It follows from Problem 9.36 that −f (x) is a convex function or, equivalently, f (x) is a concave function. (b) Since −f (x) is a convex function, the inequality is an immediate consequence of Jensen’s inequality.

9.39 Setting the first derivative with respect to x of the Lagrange function L(x, λ) = x0 x − λ(x21 + x22 + x23 + 4x1 − 6x3 − 2) equal to 0, we obtain the equations 2x1 − 2λx1 − 4λ = 0, 2x2 − 2λx2 = 0, 2x3 − 2λx3 + 6λ = 0.

From these we get x1 = 2λ/(1 − λ), x2 = 0, and x3 = −3λ/(1 − λ). Since x21 + x22 + x23 + 4x1 − 6x3 = 2, p we then find that λ = 1 ± 13/15. Thus, we have the solutions, x0 = (−2(1 + x0 = (2(1 −

p

p

p p p p 13/15)/ 13/15, 0, 3(1 + 13/15)/ 13/15) when λ = 1 + 13/15,

p p p p 13/15)/ 13/15, 0, −3(1 − 13/15)/ 13/15) when λ = 1 − 13/15,

Using Theorem 9.16, we compute  0 2x1 + 4 2x2    2x1 + 4 2(1 − λ) 0 ∆3 =    2x2 0 2(1 − λ)  2x3 − 6 0 0

2x3 − 6 0 0 2(1 − λ)

     ,   

0

  ∆2 =  2x1 + 4  2x2

2x1 + 4

2x2

2(1 − λ)

0

0

2(1 − λ)

   . 

Consequently, |∆3 | = −16(1 − λ)2 {(x1 + 2)2 + x22 + (x3 − 3)2 } and |∆2 | = −8(1 − λ){(x1 + 2)2 + x22 }. p Here n = 3 and m = 1. Since |∆2 | < 0 and |∆3 | < 0 when λ = 1 − 13/15, we have (−1)m |∆2 | > 0,

(−1)m |∆3 | > 0,

p p p p 13/15)/ 13/15, 0, −3(1 − 13/15)/ 13/15) yields a minimum. Since |∆2 | > 0 p and |∆3 | < 0 when λ = 1 + 13/15, we have and so x0 = (2(1 −

(−1)2 |∆2 | > 0, and so x0 = (−2(1 +

p

(−1)3 |∆3 | > 0,

p p p 13/15)/ 13/15, 0, 3(1 + 13/15)/ 13/15) yields a maximum.

108

9.41 Setting the first derivative with respect to x of the Lagrange function L(x, λ) = x1 (x2 + x3 ) − λ1 (x21 + x22 − 1) − λ2 (x1 x3 + x2 − 2) equal to 0, we obtain the equation (x2 + x3 − 2λ1 x1 − λ2 x3 , x1 − 2λ1 x2 − λ2 , x1 − λ2 x1 )0 = 00 .

From the third component above, we see that λ2 = 1 since the two constraints ensure that x1 6= 0.

Multiplying the first component by 2λ1 and adding to the second component, we obtain an equation which yields the solution x1 = (1 − 4λ21 )−1 , and when this is substituted back into the first component, we also get x2 = 2λ1 (1 − 4λ21 )−1 . Substituting both of these into the first constraint, we find that p p λ1 = 0, 3/4, or − 3/4. Using these values, the solutions for x1 and x2 , and the solution for x3 obtained from the second constraint, we find that there are 3 stationary points: (x1 , x2 , x3 )0 = (1, 0, 2)0 , when λ1 = 0, p p p (x1 , x2 , x3 )0 = (−1/2, − 3/4, −4 − 2 3/4)0 , when λ1 = 3/4, p p p (x1 , x2 , x3 )0 = (−1/2, 3/4, −4 + 2 3/4)0 , when λ1 = − 3/4.

Straightforward calculation reveals that since λ2 = 1,  0 0 2x1    0 0 x3   ∆3 =  2x1 x3 −2λ1    2x2 1 1  0 x1 0

2x2 1 1 −2λ1 0

0



  x1    0 ,   0   0

p and |∆3 | = −8λ1 x41 − 8x31 x2 − 8λ1 x21 x22 . When λ1 = 3/4, we have |∆3 | < 0 and (−1)3 |∆3 | > 0, and so p this stationary point is a max. When λ1 = − 3/4, we have |∆3 | > 0 and (−1)m |∆3 | > 0, and so this stationary point is a min. When λ1 = 0, |∆3 | = 0. A closer inspection of this stationary point reveals that it is a saddle point. For instance, if we use the two constraints to obtain equations for x2 and x3 p in terms of x1 , and substitute these into f (x), we get f (x) = 2 ± (x1 − 1) 1 − x21 . There are values arbitrarily close to x1 = 1 for which f is larger than when x1 = 1 and there are values arbitrarily close to x1 = 1 for which f is smaller than when x1 = 1.

109

9.43 Since f (cx) = f (x) for any scalar c 6= 0, maximizing and minimizing f (x) is equivalent to maximizing and minimizing x0 Ax subject to the constraint x0 Bx = 1. Differentiating the Lagrange function L(x, λ) = x0 Ax − λ(x0 Bx − 1) with respect to x and setting equal to 00 , yields the equation Ax = λBx. Premultiplying by x0 and rearranging, we get λ=

x0 Ax = x0 Ax. x0 Bx

Thus, for any stationary point x, λ = f (x) and λ is an eigenvalue of B −1 A. Consequently, the minimum of f is λm (B −1 A), which is attained at any eigenvector of B −1 A corresponding to λm (B −1 A), and the maximum of f is λ1 (B −1 A), which is attained at any eigenvector of B −1 A, corresponding to λ1 (B −1 A).

9.45 (a) We have E(µ̂) = E

n X

! ai xi

=

i=1

n X

ai E(xi ) =

i=1

and so µ̂ is an unbiased estimator of µ only if

n X

ai µ = µ

i=1

n X

! ai

,

i=1

Pn

i=1 ai = 1.

(b) First note that since the xi ’s are independent, ! n n n X X X var(µ̂) = var ai xi = a2i var(xi ) = σ 2 a2i . i=1

We need to minimize var(µ̂) subject to

i=1

i=1

Pn

L(a, λ) = σ

i=1 ai = 1, so we consider the Lagrange function

2

n X

a2i − λ

i=1

n X

! ai − 1 .

i=1

Setting the first derivative with respect to a equal to 0, we obtain the equations 2σ 2 ai − λ = 0, for all i. Thus, a1 = · · · = an and the only solution satisfying

Pn

i=1 ai = 1 has ai = 1/n for all i.

This of course yields the estimator x̄. Now A and B in Theorem 9.15 are A = 2σ 2 In ,

B = 10n .

According to Theorem 9.15, our solution yields a minimum since x0 Ax > 0 for all x 6= 0 for which Bx = 0.

110

9.47 Since f (c1 a, c2 b) = f (a, b) for any c1 6= 0 and c2 6= 0, maximizing f is equivalent to maximizing (a0 Ω12 b)2 subject to the constraints a0 Ω11 a = 1 and b0 Ω22 b = 1. Thus, we consider the Lagrange function L(a, b, λ1 , λ2 ) = (a0 Ω12 b)2 − λ1 (a0 Ω11 a − 1) − λ2 (b0 Ω22 b − 1).

Differentiating L with respect to a and setting equal to 0 yields (a0 Ω12 b)Ω12 b − λ1 Ω11 a = 0,

(1)

while differentiation with respect to b leads to (a0 Ω12 b)Ω012 a − λ2 Ω22 b = 0.

(2)

Premultiplying (1) by a0 , we get (a0 Ω12 b)2 = λ1 a0 Ω11 a = λ1 , and premultiplying (2) by b0 , we get (a0 Ω12 b)2 = λ2 b0 Ω22 b = λ2 .

That is, λ1 = λ2 = (a0 Ω12 b)2 , which is the quantity we wish to maximize. Letting λ = λ1 = λ2 , it −1 0 follows from (1) that a = (a0 Ω12 b)−1 Ω−1 11 Ω12 b, and when substituting this in (2), we get Ω12 Ω11 Ω12 b = −1 −1 0 −1 0 λΩ22 b or, equivalently, Ω−1 22 Ω12 Ω11 Ω12 b = λb. Thus, λ is an eigenvalue of Ω22 Ω12 Ω11 Ω12 , and so the

maximum will be given by the largest eigenvalue. Similarly, if we use (2) to solve for b and substitute −1 0 this in (1), we get Ω−1 11 Ω12 Ω22 Ω12 a = λa. Thus, the maximum is attained when a is an eigenvector −1 0 −1 0 −1 of Ω−1 11 Ω12 Ω22 Ω12 corresponding to its largest eigenvalue, and b is an eigenvector of Ω22 Ω12 Ω11 Ω12

corresponding to its largest eigenvalue.

111

Chapter 10

10.1 If x ≺ y, then k X

k X

x[i] ≤

i=1

y[i] ,

i=1

for k = 1, . . . , m, and if y ≺ x, then k X

y[i] ≤

k X

i=1

x[i] ,

i=1

for k = 1, . . . , m. Combining these two inequalities, we get k X

x[i] =

i=1

k X

y[i] ,

i=1

for k = 1, . . . , m, which implies that x[i] = y[i] , for i = 1, . . . , m. That is, for some permutation matrix P , y = P x.

10.3 Let w = λx + (1 − λ)y. Since x ≺ z and y ≺ z, we have k X

w[i]

≤

λ

k X

x[i] + (1 − λ)

≤

λ

k X

z[i] + (1 − λ)

i=1

=

k X

y[i]

i=1

i=1

i=1

k X

k X

z[i]

i=1

z[i]

i=1

for k = 1, . . . , m − 1, and m X

w[i]

=

λ

i=1

= =

λ

m X i=1 m X

i=1 m X

x[i] + (1 − λ) z[i] + (1 − λ)

z[i] .

i=1

Thus, w = λx + (1 − λ)y ≺ z.

112

m X i=1 m X i=1

y[i] z[i]

10.5 If x ≺ y, then it follows from Theorem 10.9 that m X

|xi − a| ≤

i=1

m X

|yi − a|

(1)

i=1

since g(x) = |x − a| is a convex function. Conversely now suppose (1) holds for all a. Note that if xi > a and yi > a for all i, then m X

xi − ma =

i=1

m m m m m X X X X X (xi − a) = |xi − a| ≤ |yi − a| = (yi − a) = yi − ma, i=1

i=1

so we must have

i=1

m X

xi ≤

i=1

m X

i=1

i=1

yi .

(2)

i=1

On the other hand, if xi < a and yi < a for all i, then ma −

m X i=1

xi =

m m m m m X X X X X (a − xi ) = |xi − a| ≤ |yi − a| = (a − yi ) = ma − yi , i=1

i=1

so we must have

i=1

m X

xi ≥

i=1

m X

i=1

i=1

yi .

(3)

yi .

(4)

i=1

Combining (2) and (3) yields m X

xi =

i=1

m X i=1

Now define x+ = max(x, 0) and suppose that m m X X + (xi − a) ≤ (yi − a)+ i=1

i=1

holds for all a. Note that for fixed k, k k X X (y[i] − y[k] )+ = y[i] − ky[k] , i=1

whereas

i=1

m X

(y[i] − y[k] )+ = 0,

i=k+1

so that m k X X (y[i] − y[k] )+ = y[i] − ky[k] . i=1

i=1

113

(5)

Thus, due to (5) we have k X

y[i] − ky[k] ≥

i=1

m k k X X X (x[i] − y[k] )+ ≥ (x[i] − y[k] )+ ≥ x[i] − ky[k] , i=1

i=1

i=1

and so we must have k X

k X

x[i] ≤

i=1 +

y[i] .

(6)

i=1

Finally, note that since |xi − a| = 2(xi − a) − (xi − a), |yi − a| = 2(yi − a)+ − (yi − a), and (4) holds, it follows from (1) that (5) holds. We have shown this guarantees that (6) holds and this along with (4) confirms that x ≺ y.

10.7 Since P is doubly stochastic, P 1m = 1m and 10m P = 10m . Premultiplying the first of these equations

by P −1 and postmultiplying the second by P −1 yields P −1 1m = 1m and 10m P −1 = 10m . Also if P and P −1 are doubly stochastic, from Theorem 10.1, we know that P ej ≺ ej and P −1 (P ej ) = ej ≺ P ej .

By Problem 10.1, this means that the jth column of P can be obtained from ej by permutation of its components; that is, the jth column of P has one nonzero component equal to one. In a similar fashion, we can show that the jth row of P has one nonzero component equal to one. This establishes that P is a permutation matrix.

10.9 Write b = (b1 ,0 b02 )0 , where b1 = (b1 , . . . , bm1 )0 has the eigenvalues of A11 and b2 = (bm1 +1 , . . . , bm )0 has the eigenvalues of A22 . Let A11 = P1 Db1 P10 and A22 = P2 Db2 P20 be the spectral decompositions of A11 and A22 , so that P1 and P2 are m1 × m1 and m2 × m2 orthogonal matrices. It follows that P = diag(P1 , P2 ) is an m × m orthogonal matrix, and so the eigenvalues of A are the same as the eigenvalues of  P 0 AP = 

P10 A11 P1

P10 A12 P2

P20 A012 P1

P20 A22 P2





=

Db1

P10 A12 P2

P20 A012 P1

Db2

 .

(1)

Now applying Theorem 10.3 to the matrix given in (1), we find that b ≺ a.

10.11 Note that the function g(x) = x−1 is a convex function on (0, ∞) and x̄1m ≺ x. Thus, an application of Theorem 10.9 yields m X 1

xi

i=1

≥

m X 1 i=1

x̄

.

Multiplying both sides of this equation by mx̄ and then subtracting m leads to m X mx̄ − xi i=1

xi

114

≥ m(m − 1).

10.13 Since y is in the column space of A, it can be written as y = Az for some m × 1 vector z. Then using Problem 10.12, we get (x0 y)2

=

(x0 Az)2 ≤ (x0 Ax)(z 0 Az)

=

(x0 Ax)(z 0 AA− Az) = (x0 Ax)(y 0 A− y).

We have equality if and only if one of the vectors Ax and Az = y is a scalar multiple of the other.

10.15 If we let z = (x0 x)−1/2 x, the inequality can be equivalently expressed as (z 0 Az)(z 0 A−1 z) ≤

(λ1 + λm )2 , 4λ1 λm

for all m × 1 unit vectors z. Let A = P ΛP 0 be the spectral decomposition of A so that Λ = diag(λ1 , . . . , λm ). Note that for i = 1, . . . , m, λ2i − (λ1 + λm )λi + λ1 λm = (λi − λ1 )(λi − λm ) ≤ 0,

(1)

since λi − λ1 ≤ 0 and λi − λm ≥ 0. Dividing (1) by −λi , we get (λ1 + λm ) − λi −

λ1 λm ≥0 λi

for i = 1, . . . , m, or, in other words, (λ1 + λm )Im − Λ − λ1 λm Λ−1

(2)

is nonnegative definite. Premultiplying (2) by P and postmultiplying by P 0 , it then follows that (λ1 + λm )Im − A − λ1 λm A−1 is also nonnegative definite. Consequently, for any unit vector z, (λ1 + λm ) − z 0 Az − λ1 λm z 0 A−1 z ≥ 0.

Multiplying (3) by z 0 Az and rearranging, we get λ1 λm (z 0 Az)(z 0 A−1 z) ≤ (λ1 + λm )z 0 Az − (z 0 Az)2 2 2

λ1 + λm λ1 + λm + = − z 0 Az − 2 2 2 (λ1 + λm ) ≤ , 4 from which the required inequality follows.

115

(3)

10.17 Let λ = (λ1 , . . . , λm )0 , where λ1 ≥ · · · ≥ λm are the eigenvalues of A. Then using the Cauchy-Schwarz inequality, we have 2

{tr(A)}

m X

=

!2 = (10m λ)2 ≤ (10m 1m )(λ0 λ)

λi

i=1

=

m X

m

! λ2i

= m tr(A2 ),

i=1

with equality if and only if one of the vectors 1m and λ is a scalar multiple of the other, and this is equivalent to saying the eigenvalues are all equal.

10.19 Clearly tr{(A0 B)2 } = tr(A0 BA0 B) = tr(BA0 BA0 ) = tr{(BA0 )2 }.

Now applying the result in Problem 10.18 with A0 replaced by B and B replaced by A0 , we get tr{(A0 B)2 } = tr{(BA0 )2 } ≤ tr(BB 0 AA0 ) = tr(AA0 BB 0 ), with equality if and only if B 0 A is symmetric.

10.21 We prove the result by induction. It follows from Theorem 10.16 that the result holds for n = 2. Now Pn−1 assume the result holds for n − 1 and we will show that it must hold for n. Let A = i=1 βi Ai , where βi = αi /(1 − αn ) for i = 1, . . . , n − 1. Note that n−1 X

βi = (1 − αn )−1

i=1

n−1 X

αi = (1 − αn )−1 (1 − αn ) = 1,

i=1

and since A1 , . . . , An−1 are positive definite, we can apply our result to A, that is, |A| =

n−1 X

β i Ai ≥

i=1

n−1 Y

|Ai |βi ,

(1)

i=1

with equality if and only if A1 = · · · = An−1 . Also, an application of Theorem 10.16 yields n X

αi Ai = (1 − αn )

i=1

n−1 X

βi Ai + αn An = |(1 − αn )A + αn An | ≥ |A|1−αn |An |αn ,

i=1

with equality if and only if A = An . Combining (1) and (2), we get !(1−αn ) n n−1 X Y βi αi Ai ≥ |Ai | |An |αn i=1

i=1

=

n−1 Y

! |Ai |αi

i=1

116

|An |αi =

n Y i=1

|Ai |αi ,

(2)

with equality if and only if A1 = . . . = An .

10.23 Using Theorem 4.14, we can write A−1 = C 0 Λ−1 C and B −1 = C 0 C, where C is a nonsingular matrix and Λ is a diagonal matrix with positive diagonal elements. As a result W

= αA−1 + (1 − α)B −1 − {αA + (1 − α)B}−1 = C 0 {αΛ−1 + (1 − α)Im }C − [C −1 {αΛ + (1 − α)Im }C −10 ]−1 = C 0 DC,

where D = αΛ−1 + (1 − α)Im − {αΛ + (1 − α)Im }−1 . Note that since g(x) = x−1 is a convex function on (0, ∞), αg(x) + (1 − α)g(1)

=

α + (1 − α) x

≥

g(αx + (1 − α)) =

1 , αx + (1 − α)

with equality if and only if x = 1. This guarantees that D, and hence also W , is nonnegative definite.

Thus, we must have tr(W ) = α tr(A−1 ) + (1 − α) tr(B −1 ) − tr[{αA + (1 − α)B}−1 ] ≥ 0 which is the desired result. We have equality if and only A = B. 10.25 Using the result from the previous problem, we find that |A + B|1/m

= ≥

min m−1 tr{(A + B)X}

|X|=1

min m−1 tr(AX) + min m−1 tr(BX)

|X|=1

|X|=1

1/m

= |A|

+ |B|

1/m

.

We have equality if and only if the three minimums occur at the same X in which case from the previous problem, this requires that A−1 , B −1 , and (A + B)−1 be proportional to one another, and so A and B are proportional.

117

Chapter 11

11.1 Note that   I m rank   A

Im − A (0)

   

  I m = rank   (0)   (0) = rank   A

−Im



Im



Im − A

 

Im

(0)

   

A (0) −Im Im  Im − A   = rank(A) + rank(Im − A) = m,  (0) Im

and   I m rank   A

Im − A (0)

   = 

     I  (0) I I − A I −(I − A) m m  m m  m  rank   −A I  A (0) (0) Im m    I  (0) m  = rank(Im ) + rank{A(Im − A)} = rank   (0) −A(I − A)  m

= m + rank{A(Im − A)}, where we have used Theorem 1.10 and Theorem 2.9. Together these two identities imply rank{A(Im − A)} = 0, so that we must have A(Im − A) = (0) or or A2 = A. 11.3 The matrix A has r eigenvalues equal to one, with the remaining eigenvalues all being zero. Thus, A has a spectral decomposition A = XΛX 0 , where Λ = diag(Ir , (0)) for some orthogonal matrix X. Partition X as X = [P

Q], where P is m × r. Then X 0 X = Im implies that P 0 P = Ir and    0 I (0) P r   = P P 0.

A = XΛX 0 = [P Q]  0 (0) (0) Q

11.5 It follows from Problem 11.3 that a symmetric idempotent matrix is the projection matrix for the orthogonal projection onto its column space. Thus, since these projection matrices are unique and A and B have the same column space, we must have A = B. 11.7 (a) We must have (a1m 10m )2 = a2 1m 10m 1m 10m = ma2 1m 10m = a1m 10m , that is, ma2 = a. Thus, a = 0 or a = 1/m.

118

(b) We must have (bIm + c1m 10m )2 = b2 Im + 2bc1m 10m + mc2 1m 10m = bIm + c1m 10m .

That is, we must have b2 = b and 2bc + mc2 = c. Thus, we need b = 0 and c = 0 or c = 1/m, or b = 1 and c = 0 or c = −1/m.

11.9 If AB is idempotent, then ABAB = AB. Since B is nonsingular, we can postmultiply this equation by B −1 and premultiply by B to get BABA = BA, confirming that BA is idempotent.

11.11 Note that if c = 0, A2 = (0) and so all the eigenvalues of A equal zero. Thus, tr(A) = c rank(A) holds since both sides reduce to zero. If c 6= 0, then A2 = cA implies (c−1 A)2 = c−2 A2 = c−2 cA = c−1 A. That is, c−1 A is idempotent. Thus, rank(A) = rank(c−1 A) = tr(c−1 A) = c−1 tr(A), so tr(A) = c rank(A).

11.13 (a) Clearly A − BB + is symmetric since A and BB + are, and (A − BB + )2

=

A2 − ABB + − BB + A + BB + BB +

=

A − BB + − (BB + )0 A + BB + = A − B +0 B 0 A

=

A − B +0 (AB)0 = A − B +0 B 0

= A − (BB + )0 = A − BB + , so A − BB + is idempotent. Since A and BB + are idempotent as well, rank(A − BB + )

=

tr(A − BB + ) = tr(A) − tr(BB + )

=

rank(A) − rank(BB + ) = rank(A) − rank(B).

119

(b) Since Im − A is symmetric idempotent and (Im − A)B = B, we can apply the result from part (a). That is, (Im − A) − BB + is symmetric idempotent with rank{(Im − A) − BB + } = rank{(Im − A)} − rank(B) = m − rank(A) − rank(B) = 0, since rank(A) + rank(B) = m. This requires that (Im − A) − BB + = (0), so A = Im − BB + .

11.15 Since Ω is positive definite, it can be written Ω = T T 0 for some m × m nonsingular matrix T . Then z = T −1 x ∼ Nm (T −1 µ, Im ) and x0 Ax = x0 T −10 T 0 AT T −1 x = z 0 T 0 AT z.

Clearly, T 0 AT is symmetric and (T 0 AT )2 = T 0 AΩAT = T 0 AΩAΩΩ−1 T = T 0 (AΩ)2 Ω−1 T = T 0 AΩΩ−1 T = T 0 AT, so it is also idempotent. Consequently, it can be written T 0 AT = P ∆P 0 , where P is an orthogonal matrix and ∆ is the diagonal matrix diag(Ir , (0)) since rank(∆) = rank(T 0 AT ) = tr(T 0 AT ) = tr(AΩ) = rank(AΩ) = r.

Then z ∗ = P 0 z ∼ Nm (P 0 T −1 µ, Im ) and x0 Ax = z 0 T 0 AT z = zP ∆P 0 z = z 0∗ ∆z ∗ =

r X

2 zi∗ .

i=1

Thus, x0 Ax has a chi-squared distribution with r degrees of freedom. The noncentrality parameter is λ=

1 1 1 1 E(z ∗ )0 ∆E(z ∗ ) = µ0 T −10 P ∆P 0 T −1 µ = µ0 T −10 T 0 AT T −1 µ = µ0 Aµ.

2 2 2 2

11.17 Since A is symmetric of rank r, it can be expressed as A = P ΛP 0 , where P is an m × r matrix satisfying P 0 P = Ir , Λ = diag(λ1 , . . . , λr ), and λ1 , . . . , λr are the nonzero eigenvalues of A. Then z = P 0 x ∼ Nr (0, Ir ) and x0 Ax = x0 P ΛP 0 x = z 0 Λz =

r X

λi zi2 ,

i=1 0

confirming that x Ax is a linear combination of independent one-degree of freedom chi-squared random variables. When A is idempotent, the coefficients, that is, the λ0i s, are all equal to one.

120

11.19 Since x1 , . . . , xn are independent and identically distributed as N (µ, σ 2 ) random variables, we have x = (x1 , . . . , xn )0 ∼ Nn (µ1n , σ 2 In ), y = x − µ1n ∼ Nn (0, σ 2 In ), and n−1 (x − µ1n )0 1n = (x̄ − µ).

Thus, t=

n(x − µ1n )0 (n−1 1n )(n−1 10n )(x − µ1n ) n(x̄ − µ)2 = = y 0 (n−1 σ −2 1n 10n )y. σ2 σ2

Since (n−1 σ −2 1n 10n )(σ 2 In ) is idempotent and rank(n−1 1n 10n ) = 1, it follows from Theorem 11.10 that t ∼ χ21 .

11.21 (a) Now t1 = y 0 A1 y, where A1 = diag(Ω−1 11 , (0)) and y = x − µ ∼ Nn (0, Ω). Since   Ir Ω−1 Ω 12 11  A1 Ω =  (0) (0) is idempotent and rank(A1 Ω) = r, it follows from Theorem 11.10 that t1 ∼ χ2r . (b) Note that t2 = y 0 A2 y, where A2 = Ω−1 − A1 . Since A2 ΩA2 = Ω−1 − A1 − A1 + A1 ΩA1 = Ω−1 − A1 = A2 , it follows that A2 Ω is idempotent. Applying Theorem 7.1 to A = Ω, we find that   −1 −1 −1 0 Ω11 Ω12 B22 Ω12 Ω11 −Ω11 Ω12 B22  A2 =  −B22 Ω012 Ω−1 B 22 11   −1

Ω11 Ω12  B22 Ω012 Ω−1 =  − In−r , 11 −In−r −1 where B22 = (Ω22 −Ω012 Ω−1 . As a result, we see that rank(A2 Ω) = rank(A2 ) = rank(B22 ) = 11 Ω12 )

n − r, and so another application of Theorem 11.10 shows that t2 ∼ χ2n−r . (c) By Theorem 11.14, t1 and t2 are independently distributed since A1 ΩA2 = A1 Ω(Ω−1 − A1 ) = A1 − A1 ΩA1 = A1 − A1 = (0).

11.23 Since Ω is positive definite, we can write Ω = T T 0 for some nonsingular matrix T , and y = T −1 x ∼ Nm (T −1 µ, Im ). Let A∗ = T 0 AT and B∗ = BT , and note that B∗ A∗ = BT T 0 AT = BΩAT = (0)T = (0).

121

If the spectral decomposition of A∗ is given by  Λ A∗ = [P1 P2 ]  (0)

(0) (0)



P10



P20

  = P1 ΛP10 ,

then the condition B∗ A∗ = (0) implies that B∗ = CP20 for some matrix C. Now let z = P 0 y ∼ Nm (P 0 T −1 µ, Im ) and partition z = (z 01 , z 02 )0 similar to P = (P1 , P2 ). Then x0 Ax = y 0 T 0 AT y = y 0 A∗ y = z 0 P 0 A∗ P z = z 01 Λz 1 , and Bx = BT y = B∗ y = B∗ P z = CP20 P z = Cz 2 .

Since var(z) = Im , z 1 is independent of z 2 and so it follows that x0 Ax is independent of Bx. 11.25 (a) We have ti = x0 Ai x, where A1 =

1 1 14 104 + (e1 − e2 )(e1 − e2 )0 , 4 2

A2 =

1 (14 − 4e4 )(14 − 4e4 )0 , 12

A3 = (e1 + e2 − 2e3 )(e1 + e2 − 2e3 )0 + (e3 − e4 )(e3 − e4 )0 . (b) Since A21 = A1 and tr(A1 ) = 2, t1 ∼ χ22 . Since A22 = A2 and tr(A2 ) = 1, t2 ∼ χ21 . Finally, t3 does not have a chi-squared distribution because A23 6= A3 . (c) It is easily shown that A1 A2 = (0) and A1 A3 = (0), while A2 A3 =

1 (14 − 4e4 )(e3 − e4 )0 6= (0).

3

Thus, t1 and t2 are independently distributed, t1 and t3 are independently distributed, but t2 and t3 are not independently distributed.

11.27 Since Ω is positive semidefinite, there is an m × m matrix T satisfying Ω = T T 0 , and so condition (a) can be written as T T 0 AT T 0 BT T 0 = (0).

Premultiplying this by T + and postmultiplying by T +0 , we find that T 0 AT T 0 BT = (0), 122

which is the condition (11.12) established in the proof of Theorem 11.14. Writing x = µ + T P y, we find that x0 Ax = µ0 Aµ + y 0 Cy + 2µ0 AT P y,

(1)

x0 Bx = µ0 Bµ + y 0 Dy + 2µ0 BT P y,

(2)

and

where C, D, and P are as defined in the proof of Theorem 11.14, and y ∼ Nm (0, Im ). As demonstrated in the proof of Theorem 11.14, CD = (0) and this shows that y 0 Cy and y 0 Dy are independent. Next write y 0 Cy = y 0 CC + Cy and note that the covariance between Cy and µ0 BT P y is E(Cyy 0 P 0 T 0 Bµ) = CE(yy 0 )P 0 T 0 Bµ = CP 0 T 0 Bµ.

But condition (b) implies that T 0 AT T 0 Bµ = 0, that is, P CP 0 T 0 Bµ = 0, and this yields CP 0 T 0 Bµ = 0.

Thus, Cy and µ0 BT P y are independently distributed, and hence so are y 0 Cy and µ0 BT P y. In a similar fashion, it can be shown that the independence of y 0 Dy and µ0 AT P y follows from condition (c). Finally, due to condition (d), we have µ0 AT P E(yy 0 )P 0 T 0 Bµ = µ0 AT P P 0 T 0 Bµ = µ0 AΩBµ = 0, so that µ0 AT P y and µ0 BT P y are uncorrelated and, hence, independent. This then confirm that (1) and (2) are independently distributed.

11.29 Let F = [G0

H 0 ]0 , and note that F −1 = [G0 (GG0 )−1 y

H 0 (HH 0 )−1 ]. Then

=

Xβ + = XF −1 F β +

=

XG0 (GG0 )−1 Gβ + XH 0 (HH 0 )−1 Hβ +

=

X∗ β ∗ + XH 0 (HH 0 )−1 c + ,

which is equivalent to the model y ∗ = X∗ β ∗ + . The least squares estimate of β ∗ is β̂ ∗

=

(X∗0 X∗ )−1 X∗0 y ∗

= GG0 (GX 0 XG0 )−1 GX 0 (y − XH 0 (HH 0 )−1 c).

The sum of squared errors for this reduced model is then given by SSEr

=

(y ∗ − X∗ β̂ ∗ )0 (y ∗ − X∗ β̂ ∗ )

= {y − XH 0 (HH 0 )−1 c}0 {IN − XG0 (GX 0 XG0 )−1 GX 0 }{y − XH 0 (HH 0 )−1 c}, 123

while the sum of squared errors for the complete model is SSEc = y 0 {IN − X(X 0 X)−1 X 0 }y.

Consequently, the F statistic is F =

(SSEr − SSEc )/m2 .

SSEc /(N − m)

11.31 Using the moments of the standard normal distribution and the fact that the components of z are independent of one another, we have

0

E(zi zj zh zk zz ) =

          

0,

i 6= j 6= h 6= k,

Thk ,

i = j, i 6= h 6= k,

Im + Tii + Thh ,

i = j, h = k, i 6= h,

3Tik ,

i = j = h, i 6= k,

3Im + 6Tii ,

i = j = h = k.

          This can then be used to obtain 1 E{zi2 (zz 0 ⊗ zz 0 )} = 2

m X m X

(Thk ⊗ Thk ) + Im ⊗ Tii + Im ⊗ Im + Tii ⊗ Im + 2

m X

(Tik ⊗ Tik ),

k=1

h=1 k=1

and for i 6= j, E{zi zj (zz 0 ⊗ zz 0 )} = Im ⊗ Tij +

m X

(Tik ⊗ Tjk ) +

k=1

m X

(Tjk ⊗ Tik ) + Tij ⊗ Im .

k=1

This leads to m

0

0

E{zi zj (zz ⊗ zz )}

m

1 XX = δij {Im ⊗ Im + (Thk ⊗ Thk )} + Im ⊗ Tij + Tij ⊗ Im 2 +

m X

h=1 k=1 m X

(Tjk ⊗ Tik ),

(Tik ⊗ Tjk ) +

k=1

k=1

where δij denotes the (i, j)th component of Im . Thus, E(zz 0 ⊗ zz 0 ⊗ zz 0 )

=

=

m X m X (Eij ⊗ E{zi zj (zz 0 ⊗ zz 0 )}) i=1 j=1 m X

m

(Eii ⊗ {Im ⊗ Im +

i=1

+

m X m X

m

1 XX (Thk ⊗ Thk )}) 2 h=1 k=1 m X m X

(Eij ⊗ Im ⊗ Tij ) +

i=1 j=1

124

i=1 j=1

(Eij ⊗ Tij ⊗ Im )

+

=

m X m X m X

{Eij ⊗ (Tik ⊗ Tjk + Tjk ⊗ Tik )}

i=1 j=1 k=1 m X m X

Im3 +

1 (Im ⊗ Tij ⊗ Tij 2 i=1 j=1

+ Tij ⊗ Im ⊗ Tij + Tij ⊗ Tij ⊗ Im ) m X m X m X + (Tij ⊗ Tik ⊗ Tjk ). i=1 j=1 k=1

11.33 (a) We can write y = sx, where x ∼ Nm (0, Ω) and s is a nonnegative random variable that is distributed independently of x. Using Theorem 11.21, we then get E(yy 0 ⊗ yy 0 ) = c{2Nm (Ω ⊗ Ω) + vec(Ω) vec(Ω)0 }, where c = E(s4 ). (b) Premultiplying E(yy 0 ⊗ yy 0 ) in part (a) by e0i ⊗ e0i and postmultiplying by ei ⊗ ei yields the identity E(yi4 ) = 3c{E(yi2 )}2 , and solving for c leads to c=

E(yi4 ) .

3{E(yi2 )}2

(c) We can write S = (n − 1)−1 Y 0 AY , where Y 0 is the m × n matrix with ith column y i and A = In − n−1 1n 10n . Now var{vec(S)} = (n − 1)−2 [E{vec(Y 0 AY ) vec(Y 0 AY )0 } − E{vec(Y 0 AY )}E{vec(Y 0 AY )0 }],

(1)

and clearly E{vec(Y 0 AY )} =

n X

E{vec(y i y 0i )} − nE{vec(ȳ ȳ 0 )} = (n − 1) vec(Ω).

i=1

Since aii = 1 − n−1 and aij = −n−1 for i 6= j, and the y i ’s are independent, we have    !0    X X E{vec(Y 0 AY ) vec(Y 0 AY )0 } = E vec  aij y i y 0j  vec akl y k y 0l   ij kl X = aij akl E(y j y 0l ⊗ y i y 0k ) ijkl

125

(2)

=

X

aii ajj E(y i y 0j ⊗ y i y 0j ) +

i6=j

+

a2ii E(y i y 0i ⊗ y i y 0i ) +

i

=

a2ij E(y i y 0j ⊗ y j y 0i )

i6=j

X

X

X

X

a2ij E(y i y 0i ⊗ y j y 0j )

i6=j

aii ajj E{vec(y i y 0i ) vec(y j y 0j )0 }

i6=j

+

X

+

X

a2ij E(y i y 0i ⊗ y j y 0j )Kmm

i6=j

a2ii E(y i y 0i ⊗ y i y 0i ) +

i

=

X

a2ij E(y i y 0i ⊗ y j y 0j )

i6=j 3 −1

(n − 1) n

0

vec(Ω) vec(Ω) + (n − 1)n−1 Kmm (Ω ⊗ Ω)

+ (n − 1)2 n−1 c{2Nm (Ω ⊗ Ω) + vec(Ω) vec(Ω)0 } + (n − 1)n−1 (Ω ⊗ Ω) =

(n − 1)3 n−1 vec(Ω) vec(Ω)0 + (n − 1)n−1 (Im2 + Kmm )(Ω ⊗ Ω) + (n − 1)2 n−1 c{2Nm (Ω ⊗ Ω) + vec(Ω) vec(Ω)0 }

Thus, using this and (1) and (2), we get var{vec(Y 0 AY )}

=

(n − 1)3 n−1 vec(Ω) vec(Ω)0 + (n − 1)n−1 (Im2 + Kmm )(Ω ⊗ Ω) + (n − 1)2 n−1 c{2Nm (Ω ⊗ Ω) + vec(Ω) vec(Ω)0 } − (n − 1)2 vec(Ω) vec(Ω)0

= −(n − 1)2 n−1 vec(Ω) vec(Ω)0 + (n − 1)n−1 (Im2 + Kmm )(Ω ⊗ Ω) + (n − 1)2 n−1 c{2Nm (Ω ⊗ Ω) + vec(Ω) vec(Ω)0 }, and var{vec(S)}

=

(n − 1)−2 var{vec(Y 0 AY )}

= n−1 c{2Nm (Ω ⊗ Ω) + vec(Ω) vec(Ω)0 } − n−1 vec(Ω) vec(Ω)0 + (n − 1)−1 n−1 (Im2 + Kmm )(Ω ⊗ Ω) ≈ (n − 1)−1 {2cNm (Ω ⊗ Ω) + (c − 1) vec(Ω) vec(Ω)0 }. (d) Following the derivation given in Example 11.11 and using the formula from part (c), we find that var{vec(R)}

= =

1 H{2cNm (P ⊗ P ) + (c − 1) vec(P ) vec(P )0 }H 0 n−1 1 H{2cNm (P ⊗ P )}H 0 n−1 126

2c Nm ΘNm , n−1

=

where H and Θ are as defined in Example 11.11. The second equality above follows from the fact that

H vec(P )

1 Im2 − {(Im ⊗ P ) + (P ⊗ Im )}Λm vec(P ) 2 1 = vec(P ) − {(Im ⊗ P ) + (P ⊗ Im )} vec(Im ) 2 1 = vec(P ) − {vec(P ) + vec(P )} = 0, 2 =

whereas the last equality follows from the simplification given in Problem 11.53.

11.35 (a) We have E(x0 Axx0 Bxx0 Cx)

= E{(x0 ⊗ x0 ⊗ x0 )(A ⊗ B ⊗ C)(x ⊗ x ⊗ x)} = E[tr{(x0 ⊗ x0 ⊗ x0 )(A ⊗ B ⊗ C)(x ⊗ x ⊗ x)}] = E[tr{(A ⊗ B ⊗ C)(x ⊗ x ⊗ x)(x0 ⊗ x0 ⊗ x0 )}] =

tr{(A ⊗ B ⊗ C)E(xx0 ⊗ xx0 ⊗ xx0 )}.

=

tr(A ⊗ B ⊗ C) +

(b) Using Problem 11.31, we get m

E(x0 Axx0 Bxx0 Cx)

m

1 XX {tr(A ⊗ BTij ⊗ CTij ) 2 i=1 j=1

+ tr(ATij ⊗ B ⊗ CTij ) + tr(ATij ⊗ BTij ⊗ C)} m X m X m X tr(ATij ⊗ BTik ⊗ CTjk ). + i=1 j=1 k=1

Now tr(A ⊗ BTij ⊗ CTij )

=

tr(A) tr{B(ei e0j + ej e0i )} tr{C(ei e0j + ej e0i )}

=

tr(A)(2bij )(2cij ) = 4 tr(A)bij cij .

In a similar fashion, we get tr(ATij ⊗B⊗CTij ) = 4 tr(B)aij cij , tr(ATij ⊗BTij ⊗C) = 4 tr(C)aij bij , and tr(ATij ⊗ BTik ⊗ CTjk ) = 8aij bik cjk . Thus, E(x0 Axx0 Bxx0 Cx)

=

tr(A) tr(B) tr(C) + 2

m X m X {tr(A)bij cij + tr(B)aij cij i=1 j=1

127

+ tr(C)aij bij } + 8

m X m X m X

aij bik cjk

i=1 j=1 k=1

=

tr(A) tr(B) tr(C) + 2 tr(A) tr(BC) + 2 tr(B) tr(AC) + 2 tr(C) tr(AB) + 8 tr(ABC).

11.37 (a) We have E(xy 0 ⊗ xy 0 )

= E(x ⊗ x){E(y ⊗ y)}0 = E{vec(xx0 )}[E{vec(yy 0 )}]0 vec{E(xx0 )}[vec{E(yy 0 )}]0 = vec(V1 ){vec(V2 )}0 .

= (b) Similarly E(xy 0 ⊗ yx0 )

= E{(x ⊗ y)(y 0 ⊗ x0 )} = E{Kmn (y ⊗ x)(y 0 ⊗ x0 )} = Kmn E(yy 0 ⊗ xx0 ) = Kmn (V2 ⊗ V1 ) = (V1 ⊗ V2 )Kmn .

(c) Also E(x ⊗ x ⊗ y ⊗ y)

= E(x ⊗ x) ⊗ E(y ⊗ y) = E{vec(xx0 )} ⊗ E{vec(yy 0 )} =

vec(V1 ) ⊗ vec(V2 ).

(d) Using the result from (c), we get E(x ⊗ y ⊗ x ⊗ y)

=

E{x ⊗ Knm (x ⊗ y) ⊗ y}

=

(Im ⊗ Knm ⊗ In )E(x ⊗ x ⊗ y ⊗ y)

=

(Im ⊗ Knm ⊗ In ){vec(V1 ) ⊗ vec(V2 )}.

(e) Finally var(x ⊗ y)

= E{(x ⊗ y)(x ⊗ y)0 } − E(x ⊗ y){E(x ⊗ y)}0 = E(xx0 ) ⊗ E(yy 0 ) − {E(x) ⊗ E(y)}{E(x) ⊗ E(y)}0 = V1 ⊗ V2 − µ1 µ01 ⊗ µ2 µ02 .

11.39 We have t1 = x0 A1 x and t2 = x0 A2 x, where A1 = (e1 + e2 − 2e3 )(e1 + e2 − 2e3 )0 + (e3 − e4 )(e3 − e4 )0 = γ 1 γ 01 + γ 2 γ 02 , A2 = (e1 − e2 − e3 )(e1 − e2 − e3 )0 + (e1 + e2 − e4 )(e1 + e2 − e4 )0 = γ 3 γ 03 + γ 4 γ 04 , 128

(a) Using Theorem 11.23(c), we get var(t1 )

=

2 tr{(A1 Ω)2 } + 4µ0 A1 ΩA1 µ

=

2 tr[{(γ 1 γ 01 + γ 2 γ 02 )(4I4 + 14 104 )}2 ] + 4104 (γ 1 γ 01 + γ 2 γ 02 )(4I4 + 14 104 )(γ 1 γ 01 + γ 2 γ 02 )14

=

2 tr{(4γ 1 γ 01 + 4γ 2 γ 02 )2 }

=

1536

(b) Another application of Theorem 11.23(c) yields var(t2 )

=

2 tr{(A2 Ω)2 } + 4µ0 A2 ΩA2 µ

=

2 tr{(4γ 3 γ 03 + 4γ 4 γ 04 − γ 3 104 + γ 4 104 )2 } + 4104 (4γ 3 γ 03 + 4γ 4 γ 04 − γ 3 104 + γ 4 104 )(γ 3 γ 03 + γ 4 γ 04 )14

=

2(340) + 4(28) = 792.

(c) Using Theorem 11.23(b), we have cov(t1 , t2 )

=

2 tr(A1 ΩA2 Ω) + 4µ0 A1 ΩA2 µ

=

2 tr{(4γ 1 γ 01 + 4γ 2 γ 02 )(4γ 3 γ 03 + 4γ 4 γ 04 − γ 3 104 + γ 4 104 )} + 4104 (4γ 1 γ 01 + 4γ 2 γ 02 )(γ 3 γ 03 + γ 4 γ 04 )14

=

2(160) + 4(0) = 320.

11.41 Since Vi ∼ Wm (Ω, ni ), it can be written as Vi = Xi0 Xi , where the columns of the m × ni matrix Xi0 are independently distributed as Nm (0, Ω). Put X 0 = (X10 , X20 ), and note that since V1 and V2 are independent, the n1 + n2 columns of X 0 are independently distributed as Nm (0, Ω). It then follows from Theorem 11.25, that X 0 X = X10 X1 + X20 X2 = V1 + V2 ∼ Wm (Ω, n1 + n2 ). −1 0 11.43 (a) In finding the distribution of V1·2 = V11 − V12 V22 V12 in the proof of Theorem 11.27, it was

observed that the distribution of V1·2 given X2 does not depend on X2 . Since V22 = X20 X2 , this implies that V1·2 is independent of V22 . Also, since V1·2 = X10 {In − X2 (X20 X2 )−1 X20 }X1 ,

129

V12 = X10 X2 , and {In − X2 (X20 X2 )−1 X20 }X2 = (0), it follows that V1·2 and V12 are independent given X2 , that is, given V22 . Their unconditional independence also follows since Z fV1·2 ,V12 (U1 , U2 ) = fV1·2 ,V12 ,V22 (U1 , U2 , U3 )dU3 Z = fV1·2 ,V12 |V22 (U1 , U2 |U3 )fV22 (U3 )dU3 Z = fV1·2 |V22 (U1 |U3 )fV12 |V22 (U2 |U3 )fV22 (U3 )dU3 Z = fV1·2 (U1 )fV12 |V22 (U2 |U3 )fV22 (U3 )dU3 Z = fV1·2 (U1 )fV12 ,V22 (U2 , U3 )dU3 = fV1·2 (U1 )fV12 (U2 ). (b) Applying the result from Example 7.3, we have −1 0 0 vec(X10 )|X2 ∼ Nm1 n (vec(Ω12 Ω−1 22 X2 ), In ⊗ Ω11 − Ω12 Ω22 Ω12 ).

Then since vec(V12 ) = vec(X10 X2 ) = (X20 ⊗ Im1 ) vec(X10 ), we see that given V22 , −1 0 vec(V12 ) ∼ Nm1 m2 (vec(Ω12 Ω−1 22 V22 ), V22 ⊗ (Ω11 − Ω12 Ω22 Ω12 )).

11.45 Let U = Ω−1/2 V Ω−1/2 , where Ω1/2 is the symmetric square root of Ω. It follows from Theorem 11.26 that U ∼ Wm (Im , n). Note that if B = AΩ−1/2 , then (AV −1 A0 )−1 = (BΩ1/2 Ω−1/2 U −1 Ω−1/2 Ω1/2 B 0 )−1 = (BU −1 B 0 )−1 , and (AΩ−1 A0 )−1 = (BB 0 )−1 . Thus, the result will be proven if we can show that (BU −1 B 0 )−1 ∼ Wp ((BB 0 )−1 , n − m + p). Now it follows from Theorem 4.1 that B can be written as B = C[Ip where C is a p × p nonsingular matrix and Q is an m × m orthogonal matrix. Then   −1  I p  C 0 (BU −1 B 0 )−1 = C[Ip (0)]QU −1 Q0  (0)   −1 I p  C −1 = C −10 [Ip (0)](QU Q0 )−1  (0) 130

(0)]Q,





= C −10 [Ip

(0)]D−1 

Ip (0)

−1 C −1 ,



where D = QU Q0 ∼ Wm (Im , n) due to Theorem 11.26. Partition D and D−1 as     D11 D12 E E 11 12 , , D= E = D−1 =  0 0 D12 D22 E12 E22 −1 −1 −1 −1 0 where D11 and E11 are p × p. Now (BU −1 B 0 )−1 = C −10 E11 C and since E11 = D11 − D12 D22 D12 , −1 −1 −1 it follows from Theorem 11.27 that E11 ∼ Wp (Ip , n − m + p). Thus, C −10 E11 C ∼ Wp ((CC 0 )−1 , n −

m + p), and since (CC 0 )−1 = (BB 0 )−1 , the proof is complete.

11.47 Write A = P ΛP 0 , where P is an orthogonal matrix and Λ = diag(λ1 , . . . , λn ). Let Y 0 = (y 1 , . . . , y n ) = X 0 P , so that M∗0 = E(Y 0 ) = M 0 P . (a) Note that var{vec(Y 0 )}

=

var{vec(X 0 P )} = var{(P 0 ⊗ Im ) vec(X 0 )}

=

(P 0 ⊗ Im )(In ⊗ Ω)(P ⊗ Im ) = In ⊗ Ω.

Thus, 0

E(X AX)

=

0

0

n X

0

E(X P ΛP X) = E(Y ΛY ) = E

! λi y i y 0i

i=1

=

n X

λi E(y i y 0i ) =

i=1

=

n X

λi (Ω + µi∗ µ0i∗ ) = Ω

n X

i=1

! λi

+ M∗0 ΛM∗

i=1

Ω tr(A) + M 0 P ΛP 0 M = Ω tr(A) + M 0 AM.

(b) Since 0

0

vec(X AX) = vec(Y ΛY ) = vec

n X

! λi y i y 0i

i=1

=

n X

λi vec(y i y 0i ),

i=1

we have var{vec(X 0 AX)}

=

n X

λ2i var{vec(y i y 0i )} =

i=1

=

n X

n X

λ2i var(y i ⊗ y i )

i=1

λ2i (2Nm {Ω ⊗ Ω + Ω ⊗ µi∗ µ0i∗ + µi∗ µ0i∗ ⊗ Ω})

i=1

131

(

n X

!

) (Ω ⊗ Ω) + Ω ⊗ M∗0 Λ2 M∗ + M∗0 Λ2 M∗ ⊗ Ω

λ2i

=

2Nm

=

2Nm {tr(A2 )(Ω ⊗ Ω) + Ω ⊗ M 0 P Λ2 P 0 M + M 0 P Λ2 P 0 M ⊗ Ω}

=

2Nm {tr(A2 )(Ω ⊗ Ω) + Ω ⊗ M 0 A2 M + M 0 A2 M ⊗ Ω},

i=1

where we have used the result from Problem 11.36(a). −1 0 Y 1n . Note that B 11.49 Let Yi0 = (y i1 , . . . , y ini ) and Y 0 = (Y10 , . . . , Yk0 ), so that ȳ i = n−1 i Yi 1ni and ȳ = n

can be expressed as B

=

k X

ni ȳ i ȳ 0i − nȳ ȳ 0

i=1

=

k X

0 0 −1 0 n−1 Y 1n 10n Y i Yi 1ni 1ni Yi − n

i=1

= Y 0 A1 Y, where A1 = O(N −1 − n−1 1k 10k )O0 , N = diag(n1 , . . . , nk ), and 

1n1

0

···

0



   0 O=  ..  .  0

1 n2 .. .

···

0 .. .

0

···

   .   

1nk

Note that since O0 O = N and 10k N 1k = n, A21

=

O(N −1 − n−1 1k 10k )O0 O(N −1 − n−1 1k 10k )O0

=

O(N −1 N N −1 − n−1 N −1 N 1k 10k − n−1 1k 10k N N −1 + n−2 1k 10k N 1k 10k )O0

=

O(N −1 − n−1 1k 10k )O0 = A1 ,

and 1 1 0 M A1 M = 2 2

k X

! ni µi µ0i − nµ̄µ̄0

= Φ,

i=1

where M 0 = (µ1 10n1 , . . . , µk 10nk ). Since tr(A1 ) = tr{(N −1 − n−1 1k 10k )N } = tr(Ik ) − n−1 10k N 1k = k − 1,

132

it follows that B ∼ Wm (Ω, k − 1, Φ). Turning to W , we see that it can be expressed as   ni k X X  W = y ij y 0ij − ni ȳ i ȳ 0i  i=1

=

k X

j=1 0 Yi0 (Ini − n−1 i 1ni 1ni )Yi

i=1

= Y 0 A2 Y, −1 −1 0 0 0 where A2 = diag(In1 − n−1 1 1n1 1n1 , . . . , Ink − nk 1nk 1nk ). Clearly, since Ini − ni 1ni 1ni is idempotent,

A2 is idempotent and 1 0 M A2 M 2

k

=

1X 0 0 M (In − n−1 i 1ni 1ni )Mi 2 i=1 i i

=

1X 0 0 µ 10 (In − n−1 i 1ni 1ni )1ni µi = (0), 2 i=1 i ni i

k

where we have written M 0 = (M10 , . . . , Mk0 ) with Mi0 = µi 10ni . Thus, W ∼ Wm (Ω, n − k) since tr(A2 ) =

k X

0 tr{(Ini − n−1 i 1ni 1ni )} =

k X (ni − 1) = n − k. i=1

i=1

0 Finally, since (Ini − n−1 i 1ni 1ni )1ni = 0, we see that A2 A1 = (0). This guarantees that B and W are

independently distributed.

11.51 First we find var{vec(X 0 AX)}. Note that vec(X 0 AX)

=

(X 0 ⊗ X 0 ) vec(A) = vec{Im2 (X 0 ⊗ X 0 ) vec(A)}

=

{vec(A) ⊗ Im2 }0 vec(X 0 ⊗ X 0 ).

Since the columns of X 0 are independent, each with covariance matrix Ω, var{vec(X 0 )} = In ⊗ Ω, and so using Theorem 8.27, Theorem 11.21(d), and the fact that Knm,nm = (In ⊗ Kmn ⊗ Im )(Knn ⊗ Kmm )(In ⊗ Knm ⊗ Im ) due to Problem 8.58(b), we find that var{vec(X 0 AX)}

= {vec(A) ⊗ Im2 }0 {var(X 0 ⊗ X 0 )}{vec(A) ⊗ Im2 } = {vec(A) ⊗ Im2 }0 (In ⊗ Knm ⊗ Im )(var{vec(X 0 ) ⊗ vec(X 0 )}) 133

× (In ⊗ Kmn ⊗ Im ){vec(A) ⊗ Im2 } =

{vec(A) ⊗ Im2 }0 (In ⊗ Knm ⊗ Im )(Im2 n2 + Knm,nm ) × {(In ⊗ Ω) ⊗ (In ⊗ Ω)}(In ⊗ Kmn ⊗ Im ){vec(A) ⊗ Im2 }

=

{vec(A) ⊗ Im2 }0 (Im2 n2 + Knn ⊗ Kmm )(In ⊗ Knm ⊗ Im ) × {(In ⊗ Ω) ⊗ (In ⊗ Ω)}(In ⊗ Kmn ⊗ Im ){vec(A) ⊗ Im2 }

=

{vec(A) ⊗ Im2 )0 (Im2 n2 + Knn ⊗ Kmm )(In2 ⊗ Ω ⊗ Ω){vec(A) ⊗ Im2 }

=

{vec(A)0 vec(A)}(Ω ⊗ Ω) + {vec(A0 )0 vec(A)}Kmm (Ω ⊗ Ω)

=

tr(A0 A)(Ω ⊗ Ω) + tr(A2 )Kmm (Ω ⊗ Ω).

Now V = (X + M )0 A(X + M ) = X 0 AX + X 0 AM + M 0 AX + M 0 AM. Since the first and third-order moments of vec(X 0 ) are all zero, we have var{vec(V )}

=

var{vec(X 0 AX)} + var[{M 0 A0 ⊗ Im + Kmm (M 0 A ⊗ Im )} vec(X 0 )]

=

tr(A0 A)(Ω ⊗ Ω) + tr(A2 )Kmm (Ω ⊗ Ω) + {M 0 A0 ⊗ Im + Kmm (M 0 A ⊗ Im )} × (In ⊗ Ω){AM ⊗ Im + (A0 M ⊗ Im )Kmm }

=

tr(A0 A)(Ω ⊗ Ω) + tr(A2 )Kmm (Ω ⊗ Ω) + M 0 A0 AM ⊗ Ω + Kmm (M 0 AA0 M ⊗ Ω)Kmm + (M 0 A2 M ⊗ Ω)0 Kmm + Kmm (M 0 A2 M ⊗ Ω)

=

tr(A0 A)(Ω ⊗ Ω) + tr(A2 )Kmm (Ω ⊗ Ω) + M 0 A0 AM ⊗ Ω + Ω ⊗ M 0 AA0 M + Kmm (Ω ⊗ M 0 A2 M )0 + Kmm (M 0 A2 M ⊗ Ω).

11.53 Since Λm = Λm Nm = Nm Λm and, from Problem 8.60, {(Im ⊗ P ) + (P ⊗ Im )}Nm = 2Nm (Im ⊗ P )Nm , we have

1 Im2 − {(Im ⊗ P ) + (P ⊗ Im )}Λm Nm 2

= = =

134

1 Nm − {(Im ⊗ P ) + (P ⊗ Im )}Nm Λm 2 1 Nm − 2Nm (Im ⊗ P )Nm Λm 2 Nm {Im2 − (Im ⊗ P )Λm }.