Pattern Recognition 4th Edition By Sergios Theodoridis, Konstantinos Koutroumbas INSTRUCTORS SOLUT

INSTRUCTORS SOLUTIONS MANUAL

1 The solutions to the problems of the book are provided chapter by chapter.

The authors would welcome comments related to typos as well as to reworking some of the solutions to a more extended form, if this is necessary.

Please do not hesitate to contact the authors for suggestions.

Solutions to the problems of Chapter 2 2.1 Let Pc be the probability of correct classification. Then Pc =

M X i=1

P (x ∈ Ri , ωi ) =

M X i=1

P (ωi )P (x ∈ Ri |ωi )

or Pc =

M X i=1

P (ωi )

Z

Ri

P (x|ωi )dx =

M Z X

i=1 Ri

P (ωi )p(x|ωi )dx

or Pc =

M Z X

i=1 Ri

P (ωi |x)p(x)dx

For minimum classification error Pe , Pc must be maximum (Pc + Pe = 1).

Thus Pc is maximized if the regions Ri is chosen so that in each region the corresponding integrals, which are all positive, have the maximum possible value. That is Ri : P (ωi |x)p(x) > P (ωj |x)p(x)

∀ i 6= j

or Ri : P (ωi |x) > P (ωj |x)

∀ i 6= j

2.2 From the theory we have P (ω2 ) λ21 p(x|ω1 ) > (<) p(x|ω2 ) P (ω1 ) λ12 1

2 Taking the logarithm of both sides ln p(x|ω1 ) − ln p(x|ω2 ) > (<) ln

P (ω2 )λ21 P (ω1 )λ12

or −x2 (x − 1)2 P (ω2 )λ21 + > (<) ln 2σ 2 2σ 2 P (ω1 )λ12 where constants have been omitted. Hence P (ω2 )λ21 −2x + 1 > (<) 2σ 2 ln P (ω1 )λ12 or P (ω2 )λ21 1 − σ2 x0 = 2 P (ω1 )λ12 2.3 From the respective definitions we have r = λ11

Z

R1

λ12

P (ω1 )p(x|ω1 )dx + λ21 Z

R2

Z

R1

P (ω1 )p(x|ω1 )dx + λ22

P (ω2 )p(x|ω2 )dx + Z

R2

P (ω2 )p(x|ω2 )dx

which from the definitions of ε1 and ε2 become r = λ11 P (ω1 )(1 − ε1 ) + λ21 P (ω2 )ε2 + λ12 P (ω1 )ε1 + λ22 P (ω2 )(1 − ε2 ) and finally r = λ11 P (ω1 ) + λ22 P (ω2 ) + P (ω1 )(λ12 − λ11 )ε1 + P (ω2 )(λ21 − λ22 )ε2 2.4 By the definition of the probability M X i=1

P (ωi |x) = 1

since classes cover all space. Thus the maximum P (ωi |x) has to be larger than M1 , otherwise the sum will be smaller than one. Let us now consider the probability of correct classification Pc =

M X i=1

P (x ∈ Ri , ωi ) =

M X i=1

P (ωi )

Z

Ri

p(x|ωi )dx

3 or by the definition of Ri , M Z X

M Z 1 X Pc = P (ωi |x)p(x)dx ≥ p(x)dx M i=1 Ri i=1 Ri

or Pc ≥

1 M

Hence 1 M −1 = M M

Pe = 1 − Pc ≤ 1 −

2.5 The decision boundary point corresponds to −x20 x0 −x20 x0 exp( ) = exp( ) σ12 2σ12 σ22 2σ22 or by taking the logarithm −x20 σ12 x20 = ln − 2σ12 σ22 2σ22 and finally v u u 2σ 2 σ 2 σ2 x0 = t 2 1 2 2 ln 12

σ1 − σ2

σ2

2.6 From the problem requirements we have ε1 =

Z

P (ω1 )p(x|ω1 )dx = ε

R2

ε2 =

Z

R1

P (ω2 )p(x|ω2 )dx

Thus, minimizing ε2 subject to the first constraint equation is equivalent with minimizing Q =

Z

R1

Z

P (ω2 )p(x|ω2 )dx + θ(

= P (ω2 ) − θε +

Z

R2

R2

P (ω1 )p(x|ω1 )dx − ε)

(θP (ω1 )p(x|ω1 ) − P (ω2 )p(x|ω2 ))dx

4 Since the first two terms do not depend on R2 , Q is minimized if R2 is chosen so that R2 : P (ω2 )p(x|ω2 ) > θP (ω1 )p(x|ω1 ) or

p(ω2 |x) > θ p(ω1 |x) It remains to specify θ. In the general case, this is not easy to be computed, however its value must be chosen so that the integral in the constraint equation to be equal to ε.

R2

:

2.7 a) It suffices to compute the Mahalanobis distance of [1.6, 1.5]T from mean vectors of the classes. We have −1

Σ

=

"

0.9 0.2 −0.2 0.6

#

Thus d1 = 2.05 d2 = 0.64 d3 = 3.14 Hence [1.6, 1.5]T is assigned to ω2 . b) According to theory it suffices to compute the eigenvalues and eigenvectors of Σ. These are λ1 = 1, λ2 = 2 v 1 = [0.89 , −0.45]T v 2 = [0.45 , 0.89]T

Thus the ellipses, centered at µ2 and axis q

q

2 λ1 cv 1 and 2 λ2 cv 2 2.8 The inverse of Σ is 



5 −2.5 −2.5  5 2.5    −2.5 −2.5 2.5 5

Since the covariance matrix is the same in both classes the discriminant functions are linear given by gi (x) = xT Σ−1 µi −

1 T −1 µ Σ µi + ln P (ωi ) 2 i

5 where terms independent of the classes have been dropped. For our case we have g1 (x) = ln P (ω1 ) and g2 (x) = 2.5x2 + 2.5x3 − 1.25 + ln P ω2 )

The decision plane is

g2 (x) − g1 (x) = 0

Observe that this is basically a 2-dimensional problem due to the specific choice of µ1 and µ2 . This is not necessarily the case for other choices.

2.9 The Bayesian classifier relies on the test l12 =

p(x|ω1 ) > (<)1 p(x|ω2 )

Taking the logarithm, assignment to the class depends on the value of u = ln l12 = ln p(x|ω1 ) − ln p(x|ω2 ) whether it is positive or negative. For our specific case we have that u = xT Σ−1 (µ1 − µ2 ) −

1 (µ1 + µ2 )T Σ−1 (µ1 − µ2 ) 2

and the probability of error equals Pe =

1 1 P (u < 0, x ∈ ω1 ) + P (u > 0, x ∈ ω2 ) 2 2

The variable u is also normally distributed, since it is a linear combination of random variables x1 , . . . , xl , which are themselves jointly Gaussian, (Papoulis). Its mean value depends on whether x originates from ω1 or ω2 . The respective mean values are.

1 (µ + µ2 )T Σ−1 (µ1 − µ2 ) 2 1 1 1 = (µ1 − µ2 )T Σ−1 (µ1 − µ2 ) ≡ d2m 2 2

E1 [u] = µT1 Σ−1 (µ1 − µ2 ) −

and similarly

1 E2 [u] = − d2m 2

6 The corresponding variances are 2 σ1,u = E1 [(u − E1 [u])2 ] = E1 [(µ1 − µ2 )T Σ−1 (x − µ1 )(x − µ1 )T Σ−1 (µ1 − µ2 )]

= (µ1 − µ2 )T Σ−1 (µ1 − µ2 ) = d2m

Similarly 2 σ2,u = d2m

Thus, the probability of error is now given by Pe =

(u − 21 d2m )2 1Z 0 1 √ exp(− )du + 2 −∞ 2πdm 2d2m (u + 21 d2m )2 1Z ∞ 1 √ exp(− )du 2 0 2d2m 2πdm

which after changing the variables and taking into account the symmetry of Gaussian becomes Z ∞ z2 1 Pe = 1 √ exp(− )dz 2 d 2π 2 m 2.10 We have l12 =

p(x|ω1 ) P (ω2 ) λ21 − λ22 > (<) ≡θ p(x|ω2 ) P (ω1 ) λ12 − λ11

Taking the logarithm, for the Gaussian pdf’s the above becomes 1 1 − (x − µ1 )T Σ−1 (x − µ1 ) − ln |Σ1 | + 2 2 1 1 (x − µ2 )T Σ−1 (x − µ2 ) + ln |Σ2 | > (<) ln θ 2 2 or d2m (µ1 , x|Σ1 ) − d2m (µ2 , x|Σ2 ) + ln

|Σ1 | < (>) − 2 ln θ |Σ2 |

2.11 Rearrangement of the previous equation results in (µ1 − µ2 )T Σ−1 x > (<) Θ 1 Θ = ln θ + (||µ1 ||Σ−1 − ||µ2 ||Σ−1 ) 2

7

2.12 a) The Bayesian classifier that minimizes the error probability is given in this case by the minimizing Euclidean distance classifier. Thus, assign x ∈ ω1 if ||x − µ1 || < ||x − µ2 || b) In this case x is classified to ω1 if

P (ω2 ) λ21 p(x|ω1 ) > p(x|ω2 ) P (ω1 ) λ12 where λ12 = 1 and λ21 = 0.5. Thus following similar arguments as in theory, for Bayesian classification for normal distributions, we conclude that the decision hyperplane is g12 (x) = wT (x − x0 ) w = µ1 − µ2 and x0 =

1 P (ω1 )λ21 µ1 − µ2 (µ1 + µ2 ) − σ 2 ln 2 P (ω2 )λ12 ||µ1 − µ2 ||2

c) The following MATLAB function takes as input the variance (s), the mean (m) and the number of samples N. The output is a vector 1xN, whose elements are the N samples of the 1-D Gaussian. For 2-D independent variables, combine two samples generated above, in a single vector x = [x1 , x2 ]T Function x=gaussian(m,s,N); x=randn(1,N); x=x*sqrt(s)+m; 2.13 Generate, first, normally distributed random vectors with statistically independent components. That is, each component of x1 , x2 of the vector [x1 , x2 ]T follows a N (1, 1) for class ω1 and N (1.5, 1) for class ω2 . Then each vector is transformed as y = Ax,

A =

"

1 0.1 0.1 1

#

8 Then the covariance matrix of y is E[(y − µy )(y − µy )T ] = AE[(x − µx )(x − µx )T ]AT

= AAT = Σ

2.14 The constant Mahalanobis distance curves around, say, µ2 are 1 dm (x) = (x − µ2 )T Σ−1 (x − µ2 ) = c 2 The gradient with respect to x at x0 is ∂dm (x) = Σ−1 (x0 − µ2 ) ≡ y ∂x x=x0 Let, also , x − x0 be any vector on the hyperplane, which according to the theory is vertical to w = Σ−1 (µ1 − µ2 ) However

1 1 y = Σ−1 ( (µ1 + µ2 ) − µ2 ) = Σ−1 (µ1 − µ2 ) 2 2 is parallel to w, hence vertical to x−x0 , where for simplicity P (ω1 ) = P (ω2 ) has been assumed.

2.15 The only possible cases for overlap between the two pdf’s are shown in the figure 2.1. Other overlap possibilities are not possible, due to the constraint that the area under each pdf must be 1. The error probability is the area of the overlap shaded regions. For the (a) and (b) cases this area is bounded by (x − µ)2 1 exp(− )dx − Pe ≤ 2σ 2 −∞ 2πσ Z α (x − µ)2 1 √ exp(− )dx 2σ 2 −∞ 2πσ Z b

√

Pe ≤

Z b−µ

1 √ exp(−z)2 dz − −∞ 2π Z α−µ 1 σ √ exp(−z)2 dz = −∞ 2π b−µ a−µ G( ) − G( ) σ σ σ

9

Figure 2.1: Problem 2.15 ) − G( a−µ ) For the (c) and (d) cases the probability of error is equal to G( b−µ σ σ 2.16

∂p(x; θ) 1 ∂ ln p(x; θ) = ∂θ ∂θ p(x; θ) Thus, by definition of the mean value Z ∞

Z ∞ ∂p(x; θ) p(x; θ) ∂p(x; θ) dx = dx = ∂θ p(x; θ) ∂θ −∞ −∞ ∂ Z∞ ∂1 = = 0 p(x; θ)dx = ∂θ −∞ ∂θ 2.17 The likelihood function is

µ =

P (X; q) =

N Y

i=1

or P (X; q) = q

PN

i=1

q xi (1 − q)(1−xi )

xi

(1 − q)(N −

PN

i=1

xi )

N PN PN X ∂P (X; q) = ( xi )q ( i=1 xi −1) (1 − q)(N − i=1 xi ) ∂q i=1

−(N −

N X i=1

xi )(1 − q)(N −

PN

i=1

xi −1)

q

PN

i=1

xi

= 0

10 or q

PN

x i=1 i

(N −

(1 − q)

PN

x) i=1 i

(

PN

i=1 xi

q

P

N− N i=1 xi − ) = 0 1−q

The solutions q = 0, 1 result in a minimum of P (X; q). The maximum comes from PN

i=1 xi

q

P

N N− N 1 X i=1 xi xi − = 0⇒ q = 1−q N i=1

2.18 L(µ) =

N X i=1

ln p(xk ; µ) = −

Where constants have been omitted

N 1 X (xk − µ)2 2σ 2 i=1

N N 1 X 1 X N ∂L(µ) = 2 (xk − µ) = 2 xk − 2 µ ∂µ σ i=1 σ i=1 σ

N ∂ 2 L(µ) = − 2 2 ∂µ σ 2

Thus, the Cramer - Rao bound for the variance of the ML estimate is σN .

We know from theory, that the ML estimate of the mean is x̄ =

N 1 X xk N i=1

which is on unbiased estimate of the true mean and the variance of the 2 estimate (See chapter 5) is σN . Thus, for this case the ML estimate is efficient.

If the unknown is the variance we have N N 1 X (xk − µ)2 − ln σ 2 ln p(xk ; σ ) = − 2 L(σ ) = 2σ i=1 2 i=1 2

N X

2

N ∂L(σ 2 ) N 1 X = − 2+ 4 (xk − µ)2 2 ∂σ 2σ σ i=1

Equating the above to zero gives the estimator N 1 X (xk − µ)2 σˆ2 = N i=1

11 which is the ML estimate of the variance if the mean is known. This is also an unbiased estimator E[σ̂ 2 ] =

N 1 X E[(xk − µ)2 ] = σ 2 N i=1

For the Cramer-Rao bound we have N ∂ 2 L(σ 2 ) N 1 X −E[ 2 2 ] = −E[ 4 − 6 (xk − µ)2 ] ∂ σ 2σ σ i=1 N N N − 4+ 4 = 2σ σ 2σ 4 4

Thus, the variance of any estimator of σ 2 is lower bounded by 2σ . It turns N out that the ML estimate of the variance of a normal distribution is also efficient. Indeed, we have N 1 X (xk − µ)2 =⇒ σ̂ = N i=1 2

N X N σ̂ 2 xk − µ 2 ( = ) 2 σ σ i=1

However, each of the random variables xkσ−µ follows a N (0, 1) and they P xk −µ 2 are independent. Hence the random variable N i=1 ( σ ) follows a chi-square distribution with N degrees of freedom and its variance is equal to [Papoulis p117] 2N . That is N σ̂ 2 var( 2 ) = 2N σ or 2σ 4 var(σ̂ 2 ) = N Which is equal to the Cramer-Rao bound.

2.19 We shall focus on the simplest case where Σ = σ 2 I. The likelihood function is 2

L(θ) ≡ L(µ, σ ) = = −

N X

ln p(xk ; µ, σ 2 )

k=1

N X

1 N ln σ 2 − 2 (xk − µ)T (xk − µ) 2 2σ k=1

12 The unknown parameter is now θ T = [µT , σ 2 ] 







1 PN ∂L(θ ) 2 ∂L(θ) k=1 (xk − µ) σ ∂ µ PN  =  =0 = (xk −µ)T (xk −µ) N ∂L(θ ) k=1 ∂θ + − 2 4 ¯ 2σ 2σ ∂ σ2

Solving the above system w.r. to µ and σ 2 results in µ̂ =

N 1 X xk N k=1

N 1 X (xk − µ)T (xk − µ) σ = N k=1 2

2.20 Prove that the covariance estimate N 1 X (xk − µ̂)(xk − µ̂)T N − 1 k=1

Σ̂ = is an unbiased one, where

µ̂ = We have that

N 1 X xk N k=1

N h i 1 X E ((xk − µ) − (µ̂ − µ)) ((xk − µ) − (µ̂ − µ))T N − 1 k=1

E[Σ̂] =

N N h h i i 1 X 1 X E (xk − µ)(xk − µ)T + E (µ̂ − µ)(µ̂ − µ)T − N − 1 k=1 N − 1 k=1

=

N h i 1 X E (xk − µ)(µ̂ − µ)T − N − 1 k=1 N h i 1 X E (µ̂ − µ)(xk − µ)T N − 1 k=1

(2.1)

However h

T

E (µ̂ − µ)(µ̂ − µ)

i





N N 1 X 1 X  = E (xi − µ) (xj − µ)T  N i=1 N j=1 N h i 1 X T E (x − µ)(x − µ) i i N 2 i=1 1 1 = N Σ = Σ N2 N

=

(2.2)

13 i

h

where independence among the samples has been assumed, i.e., E (xi − µ)(xj − µ)T = δij Σ. Following a similar path we end up with h

i

h

i

E (xk − µ)(µ̂ − µ)T = E (µ̂ − µ)(xk − µ)T = Also

h

1 Σ N

i

E (xk − µ)(xk − µ)T = Σ Combining Eqs (2.1)-(2.4) we get E[Σ̂] =

(2.3)

(2.4)

1 1 1 N Σ+ Σ− Σ− Σ=Σ N −1 N −1 N −1 N −1

Hence the estimate is an unbiased one.

2.21 Prove that the ML estimates of the mean value and the covariance matrix (Problem 2.19) can be computed recursively, i.e., µ̂N +1 = µ̂N + and Σ̂N +1 =

1 (xN +1 − µ̂N ) N +1

N N (xN +1 − µ̂N )(xN +1 − µ̂N )T Σ̂N + N +1 (N + 1)2

where the subscript in the notation of the estimates, µ̂N , Σ̂N indicates the number of samples used for their computation.

From Problem 2.19 we know that +1 N 1 X 1 1 NX xk = xk + xN +1 N + 1 k=1 N + 1 k=1 N +1 N 1 1 = µ̂N + xN +1 = µ̂N + (xN +1 − µ̂N ) N +1 N +1 N +1

µ̂N +1 =

For the covariance matrix we get ΣN +1 = =

+1 1 NX (xk − µ̂N +1 )(xk − µ̂N +1 )T N + 1 k=1

N +1 +1 X 1 1 NX xTk − µ̂N +1 xk xTk − N + 1 k=1 N +1 k=1

14 1 N +1

N +1 X

!

xk µ̂TN +1 +

k=1 N X

+1 1 NX µ̂T µ̂ N + 1 k=1 N +1 N +1

!

1 xk xTk + xN +1 xTN +1 − µ̂N +1 µ̂TN +1 = N + 1 k=1

1 = N ΣN + N µ̂N µ̂TN + xN +1 xTN +1 − µ̂N +1 µ̂TN +1 N +1 Substituting in the last equation the recursion for the mean we get ΣN +1 =

1 N N µ̂N µ̂TN + ΣN + xN +1 xTN +1 − N +1 N +1 N +1 N2 1 2N µ̂N µ̂TN − µ̂N xTN +1 xN +1 xTN +1 − 2 2 2 (N + 1) (N + 1) (N + 1)

which results in ΣN +1 =

N N ΣN + (xN +1 − µ̂N )(xN +1 − µ̂N )T N +1 (N + 1)2

2.22 The likelihood function is L(θ) =

N X

ln θ2 +

k=1

k=1

= N ln θ2 +

N X

k=1

or

and finally

N X

ln xk −

ln xk − θ

N X

θxk

k=1

N X

xk

k=1

N ∂L(θ) 2N θ X xk = 0 = − ∂θ θ2 k=1

2N θ̂M L = PN

k=1 xk

2.23 From the theory we have concluded that N X ∂L(µ) Σ−1 (xk − µ) = ∂µ k=1

15 thus

∂ 2 L(µ) = −N Σ−1 ∂µ2 Since Σ is positive definite matrix the minimum corresponds to a maximum (− σN2 , in the scalar case).

2.24 We will show that provided the variables x, y are independent and Gaussian, their sum z = x + y is also Gaussian. For the sake of algebraic simplicity we will show the result for zero mean Gaussians.

From our statistics basics we know that the pdf of the sum of two random variables is given by Z p(z) = pxy (z − y, y)dy (2.5)

Since x and y are independent their joint pdf is given by the product of the respective marginal pdfs, i.e. (

1 1 pxy = exp − 2πσx σy 2

y2 x2 + σx2 σy2

!)

(2.6)

Substituting eq. (2.6) into Eq. (2.5) we obtain p(z) =

Z

(

(

1 1 exp − p(z) = 2πσx σy 2 Z

(z − y)2 y 2 + 2 σx2 σy

1 1 exp − 2πσx σy 2

  

(2.7)

!)

(2.8) 

z 2 σy4 z 2 σy4  2 2 2 1 y (σx + σy ) − 2σy zy + σx2 +σy2 − σx2 +σy2  2

exp − 

z 2 σy2 σx2 σy2

!)

σx2 σy2

2

 

dy (2.9) (2.10)

or p(z) =

  



1 exp −  2πσx σy  2

Z

    

1 exp −    2 

q

y

z2 σ4



y 2 2  1  z σy − σ2 +σ2  x

y



σx2 σy2

zσ 2 σx2 + σy2 − √ 2 y 2 σ +σ x

σx2 σy2

y

(2.11)

 

2         

dy

(2.12)

16 Finally we obtain (

1 1 z2 p(z) = exp − 2 2πσx σy 2 σx + σy2 Z

)

(2.13)

 

1 σ2 + σ2 zσ 2 exp − x 2 2 y y − 2 x 2 2 σx σy σx + σy

!2   

dy

(2.14)

and since the function under the integration is of a Gaussian form, performing the integration we get (

1 1 z2 p(z) = exp − 2 2πσx σy 2 σx + σy2

)

σx σy q

σx2 + σy2

√

2π

(2.15)

Hence, z is also a Gaussian one of zero mean and with variance equal to the sum of the variances σx2 + σy2 . If the Gaussians are not zero mean z has a mean value equal to the sum of the mean values of x and y. The result is also true if x and z are not independent but they are jointly Gaussian. If, however, they are neither independent nor jointly Gaussian, then their sum is no more Gaussian.

2.25 From the theory we have

or

p(µ|X) = R p(µ|X) = α

N Y

k=1

√

N Y p(X|µ)p(µ) p(xk |µ)p(µ) = α p(X|µ)p(µ)dµ k=1

1 (µ − µ0 )2 1 (xk − µ)2 1 1 √ exp(− exp(− ( )) )) ( 2 σ2 2 σ02 2πσ 2πσ0

N 1 X µ − xk 2 µ − µ0 2 = α1 exp(− { ( ) +( ) }) 2 k=1 σ σ0 N µ0 1 X 1 1 N xk + 2 )µ}) = α2 exp(− {( 2 + 2 )µ2 − 2( 2 2 σ σ0 σ k=1 σ0

where α, α1 , α2 are factors independent of µ. Further rearrangement leads to (µ − µN )2 p(µ|X) = α3 exp(− ) 2 2σN

17 where µN =

N σ02 x̄ + σ 2 µ0 N σ02 + σ 2

2 σN =

σ 2 σ02 N σ02 + σ 2

x̄ =

N 1 X xk N k=1

Since p(µ|X) is a pdf, then α3 is necessarily equal to α3 = √

1 2πσN

2 Hence for large N , σN → 0 and p(µ|X) behaves like a delta function centered around µN . Thus,

p(x|X) =

Z

p(x|µ)p(µ|X)dµ ≃ p(x|µN ) → N (µN , σ 2 )

Hence p(x|X) tends to a Gaussian pdf with mean µN and variance σ 2 . Furthermore, for large N, µN → x̄ For the more general case we have p(x|X) =

Z

p(x|µ)p(µ|X)dµ

(2.16)

Substituting the derived formula for p(µ|X) into the previous integral and recalling that !

1 (x − µ)2 p(x|µ) = exp − 2πσ 2σ 2 we obtain

(x − µN )2 exp − p(x|X) = q 2 2 2(σ 2 + σN ) ) 2π(σ 2 + σN 1

!

(2.17)

However, in order to get the above result we need not perform the integration. We can observe that x can be viewed as the sum of two independent 2 Gaussian distributions, one y ∼ N (0, σ 2 ) and the other µ ∼ N (µN , σN ). Indeed, if this were the case, and recalling from any statistics book the formula for the sum of random variables, the pdf of x is given by p(x) =

Z

pyµ (x − µ, µ)dµ

18 However, since y and µ are independent then the joint pdf is given by pyµ (y, µ) = py (y)pµ (µ), hence p(x) =

Z

py (x − µ)pµ (µ)dµ

Observe that this is the same as (2.16). However, recall from basic statistics that the sum of two independent Gaussians is also a Gaussian (see also problem 5.4), with mean the sum of the mean values ans variance the sum of the variances. Hence we end up in (2.17).

2.26 Show that the posterior pdf estimate in the Bayesian inference task can be computed recursively, i.e., p(θ|x1 , . . . , xN ) =

p(xN |θ)p(θ|x1 , . . . , xN −1 ) p(xN |x1 , . . . , xN −1 )

We will show the intrinsic recursive nature of the Bayesian inference task.

From the theory we have p(θ|x1 , . . . , xN ) =

p(x1 , . . . , xN |θ)p(θ) p(x1 , . . . , xN )

(2.18)

using the Bayes theorem we get p(θ|x1 , . . . , xN ) =

p(xN |x1 , . . . , xN −1 , θ)p(x1 , . . . , xN −1 |θ)p(θ) (2.19) p(x1 , . . . , xN ) p(xN |x1 , . . . , xN −1 , θ)p(θ|x1 , . . . , xN −1 )p(x1 , . . . , xN −1 ) p(xN |x1 , . . . , xN −1 )p(x1 , . . . , xN −1 )

Taking into account the mutual independence of x1 , . . . , xN , we finally obtain p(θ|x1 , . . . , xN ) =

p(xN |θ)p(θ|x1 , . . . , xN −1 ) p(xN |x1 , . . . , xN −1 )

2.27 Our starting point are the following equations

1 1 exp − (x − µ)T Σ−1 (x − µ) p(x|µ) = l/2 1/2 (2π) |Σ| 2

19 and

1 1 p(µ) = exp − (µ − µ0 )T Σ−1 0 (µ − µ0 ) l/2 1/2 (2π) |Σ| 2

From the theory, we have that the posterior pdf estimate of the mean value is p(x1 , . . . , xN |µ)p(µ) (2.20) p(µ|X) = p(X) Our goal is to show that the covariance matrix of p(µ|X) is given by ΣN = Σ0 [N Σ0 + Σ]−1 Σ

(2.21)

and also the respective mean value is given by µN = N Σ0 [N Σ0 + Σ]−1 x̄ + Σ[N Σ0 + Σ]−1 µ0

(2.22)

where x̄ is the sample mean. According to the problem 2.26 we can compute the pdf posterior estimate in a recursive manner. Thus, let us start from p(µ|x1 ) =

p(x1 |µ)p(µ) p(x1 )

(2.23)

One can easily check that the product of the two Gaussians in the numerator of the previous formula is also a Gaussian with mean value and covariance matrix equal to µ1 = Σ0 [Σ0 + Σ]−1 x1 + Σ[Σ0 + Σ]−1 µ0

(2.24)

Σ1 = Σ0 [Σ0 + Σ]−1 Σ

(2.25)

respectively. To verify the above, let us expand the exponent of the product term in the numerator of (2.23), which will be the sum of the respective exponents, i.e., A =

z

}|

{

xT1 Σ−1 x1 − xT1 Σ−1 µ − µT Σ−1 x1 T −1 + µT Σ−1 µ + µT Σ−1 0 µ +µ0 Σ0 µ0 | {z } | z

{z }|

} {

T −1 −µT0 Σ−1 0 µ − µ Σ0 µ0

(2.26)

20 Collecting the two under-braced terms we get B = = = ≡

µT [Σ−1 + Σ−1 0 ]µ T −1 µ Σ [I + ΣΣ−1 0 ]µ T −1 µ Σ [Σ0 + Σ]Σ−1 0 µ T −1 µ Σ1 µ

(2.27)

where we defined Σ1 ≡ Σ0 [Σ0 + Σ]−1 Σ

(2.28)

Thus, Eq. (2.25) has been shown.

Since we want to show that p(µ|x1 ) is Gaussian, the respective exponent must be of the form C = (µ − µ1 )T Σ−1 1 (µ − µ1 ) T −1 = µ Σ1 µ − µT1 Σ−1 1 µ T −1 −µ Σ1 µ1 + µ1 Σ−1 1 µ1

(2.29)

Comparing the two over-braced terms in (2.26) with the respective terms in (2.29), it turns out that µ1 must satisfy the following condition T −1 −1 µT Σ−1 1 µ1 = µ Σ [Σ0 + Σ]Σ0 µ1 −1 = µT (Σ−1 1 x1 + Σ0 µ0 )

(2.30)

Therefore, µ1 has to satisfy the following −1 −1 Σ−1 [Σ0 + Σ]Σ−1 0 µ1 = Σ x1 + Σ0 µ0

(2.31)

or µ1 = = = =

Σ0 [Σ0 + Σ]−1 Σ(Σ−1 x1 + Σ−1 0 µ0 ) Σ0 [Σ0 + Σ]−1 x1 + Σ0 [Σ0 + Σ]−1 ΣΣ−1 0 µ0 −1 −1 Σ0 [Σ0 + Σ] x1 + Σ[Σ0 + Σ] Σ0 Σ−1 0 µ0 −1 −1 Σ0 [Σ0 + Σ] x1 + Σ[Σ0 + Σ] µ0

(2.32)

where the identity for invertible matrices A(A + B)−1 B = B(A + B)−1 A has been used. This completes the proof of (2.24). One can easily now check out that the exponents A and C are obviously equal. Thus, we have shown that µ ∼ N (µ1 , Σ1 )

(2.33)

21 Let us now turn to the other quantity of interest, that is p(x|x1 ) =

Z

p(x|µ)p(µ|x1 )dµ Z

1 = ∝ exp − (x − µ)T Σ−1 (x − µ) 2

1 T −1 exp − (µ − µ1 ) Σ1 (µ − µ1 ) dµ 2

(2.34)

However, instead of performing the integrations (which can be done following arguments similar to those used for Problem 2.25) we can simplify the process by the following observation. Let us consider two independent normal vectors, namely µ ∼ N (µ1 , Σ1 ) and y ∼ N (0, Σ) and take their sum as x=µ+y Then, according to the relation concerning the pdf of the sum of itwo ndependent random vectors, x is distributed according to Z

py (x − µ)pµ (µ)dµ

(2.35)

It can be readily seen that (2.34) and (2.35) are exactly the same. Hence x can be considered as the sum of the previous two independent normal vectors, and, as we know form the statistics (see also Problem 2.25) it is also Gaussian (2.36) p(x|x1 ) ∼ N (µ1 , Σ + Σ1 ) This completes the first step of our recursion.

Using the recursive rule (Problem 2.26) p(µ|x1 , x2 ) =

p(x2 |µ)p(µ|x1 ) p(x1 , x2 )

(2.37)

and by introducing (2.33) into (2.37), and following exactly the same arguments as before, we can check that p(µ|x1 , x2 ) ∼ N (µ2 , Σ2 ) and also p(x|x1 , x2 ) ∼ N (µ2 , Σ + Σ2 )

22 where µ2 = Σ1 [Σ1 + Σ]−1 x2 + Σ[Σ1 + Σ]−1 µ1

(2.38)

Σ2 = Σ1 [Σ1 + Σ]−1 Σ

(2.39)

and In the sequel, we will use the easily checked matrix inversion lemma for invertible matrices (A−1 + B −1 )−1 = A(A + B)−1 B = B(A + B)−1 A to express (2.38) and (2.39) in a nice recursive form. By introducing (2.25) into (2.39) we get Σ2 = Σ1 [Σ1 + Σ]−1 Σ

(2.40) h

i−1

= Σ0 [Σ0 + Σ]−1 Σ Σ0 [Σ0 + Σ]−1 Σ + Σ

Σ

(2.41)

Identifying Σ0 [Σ0 +Σ]−1 Σ = A and Σ = B and applying the matrix inversion lemma we obtain −1 −1 −1 Σ2 = [Σ−1 [Σ0 + Σ]Σ−1 = [2Σ−1 + Σ−1 0 +Σ ] 0 ]

(2.42)

and involving the matrix inversion formula, once more, we get Σ2 = Σ0 [2Σ0 + Σ]−1 Σ

(2.43)

For the mean value we have µ2 = Σ1 [Σ1 + Σ]−1 x2 + n

Σ[Σ1 + Σ]−1 Σ0 [Σ0 + Σ]−1 x1 + Σ[Σ0 + Σ]−1 µ0 ≡ Σ1 [Σ1 + Σ]−1 x2 + C + D

(2.44) o

(2.45) (2.46)

Recalling the respective definitions we have C = = = =

Σ[Σ1 + Σ]−1 Σ0 [Σ0 + Σ]−1 x1 Σ[Σ1 + Σ]−1 Σ0 [Σ0 + Σ]−1 ΣΣ−1 x1 Σ[Σ1 + Σ]−1 Σ1 Σ−1 x1 Σ1 [Σ1 + Σ]−1 ΣΣ−1 x1

and finally C = Σ1 [Σ1 + Σ]−1 x1

(2.47)

23 Similarly, for the other term we have D = = = =

Σ[Σ1 + Σ]−1 Σ[Σ0 + Σ]−1 µ0 Σ[Σ1 + Σ]−1 Σ[Σ0 + Σ]−1 Σ0 Σ−1 0 µ0 −1 −1 −1 Σ[Σ1 + Σ] Σ0 [Σ0 + Σ] ΣΣ0 µ0 −1 Σ[Σ1 + Σ]−1 Σ1 Σ−1 0 µ0 = Σ2 Σ0 µ0

and recalling (2.40) and (2.43) we obtain 1 −1 −1 D = Σ0 [2Σ0 + Σ]−1 ΣΣ−1 0 µ0 = 2Σ0 [2Σ0 + Σ] ΣΣ0 µ0 2 = Σ[2Σ0 + Σ]−1 µ0

(2.48)

Hence, combining the values for C and D and (2.46) we have µ2 = Σ1 [Σ1 + Σ]−1 (x2 + x1 ) + Σ[2Σ0 + Σ]−1 µ0

(2.49)

which according to (2.40) and (2.43) finally results to µ2 = 2Σ0 [2Σ0 + Σ]−1 x̄ + Σ[2Σ0 + Σ]−1 µ0

(2.50)

where

1 x̄ = (x1 + x2 ) 2 Eqs. (2.50) and (2.43) are generalized by induction into Eqs. (2.21) and (2.22).

2.28 From the theory, we have that the MAP estimate corresponds to the maximum of Q(µ) ≡ = − or

or

N X (xk − µ)2

k=1

2σ 2

N X

ln p(xk ; µ) + ln p(µ)

k=1

µ2 − ln σµ2 + ln µ − 2 2σµ

N X µ − xk ∂Q(µ) µ 1 = − + − 2 = 0 2 ∂µ σ µ σµ k=1

1 N ( 2 + 2 )µ2 − Zµ − 1 = 0 σ σµ

24 and finally Z (1 + µ = 2R

s

1+

4R ) Z2

since µ > 0.

2.29 L(θ) = −

N X (ln xk − θ)2

2σ 2

k=1 N X

−

X

√ N ln(σ 2πxk )

k=1

N θ − ln xk 1 X ∂L(µ) = 0 = − ⇒ θ̂M L = ln xk ∂θ σ2 N k=1 k=1

2.30 The entropy function is H = −

Z +∞

p(x) ln p(x)dx

−∞

and the constraints are Z +∞

p(x)dx = 1,

−∞

Z +∞

xp(x)dx = µ,

−∞

Z +∞ −∞

(x − µ)2 p(x)dx = σ 2

Following the theory, if λ0 , λ1 , λ2 are the respective Lagrange multipliers, we obtain p(x) = exp(λ0 − 1 + λ1 x + λ2 x2 ) The Lagrange multipliers are obtained via the constraint equations Z +∞ −∞

exp(λ0 − 1 + λ1 x + λ2 x2 )dx = 1

Z +∞ −∞

x exp(λ0 − 1 + λ1 x + λ2 x2 )dx = µ

Z +∞

x2 exp(λ0 − 1 + λ1 x + λ2 x2 )dx = σ 2

−∞

Solving the above system of equations yields that p(x) = √

(x − µ)2 1 exp(− ) 2σ 2 2πσ

25

2.31 Taking the gradient of Q(Θ; Θ(t)) w.r. to µj we have N X µ − xk ∂Q(Θ; Θ(t)) P (j|xk ; Θ(t))(− j 2 ) = 0 ⇒ = ∂µj 2σj k=1

µj (t + 1) = The derivative w.r. to σj2 results in N X

P (j|xk ; Θ(t))(−

k=1

σj2 (t + 1) =

PN

k=1 P (j|xk ; Θ(t))xk

PN

k=1 P (j|xk ; Θ(t))

l 1 1 + 4 ||xk − µj ||2 ) = 0 ⇒ 2 2 σj 2σj

PN

2 k=1 P (j|xk ; Θ(t))||xk − µj || PN l k=1 P (j|xk ; Θ(t))

Maximizing w.r. to Pj is a constrained optimization subject to the constraint J X

Pi = 1

i=1

Forming the corresponding Lagrangian we have (∂Q(Θ; Θ(t)) + λ ∂Pj

PJ

i=1 Pi )

or Pj =

= 0 =

N X

P (j|xk ; Θ(t))

k=1

1 +λ Pj

N 1X P (j|xk ; Θ(t)) λ k=1

Substituting in the constraint equation we get 1 =

J N X N 1X P (j|xk ; Θ(t)) = λ k=1 j=1 λ

λ = N 2.32 This is basically a heads or tail problem. Assume that the random

26 variable is 1 if it falls inside h and zero otherwise. If N = 1 the average value of x is E[x] = 1P + (1 − P )0 = P ≡ µx

and the variance is

E[(x − µx )2 ] = E[x2 ] − µ2x = 1P − (1 − P )0 − P 2 = P − P 2 = P (1 − P ) Let us now consider the random variable y = x1 + x2 + . . . + xN Obviously E[y] = N P and σy2 = N P (1 − P )

However y = k, that is the number of times xj is 1,(inside h). Therefore E[ and E[(

k ] = P N

P (1 − P ) k − P )2 ] = σ 2y = N N N

2.33 We initialize the conditional probabilities P (j/xi ; Θ(t)) using the rand function of MATLAB using 0 as a seed. Of course the appropriate normalP ization takes place in order the constraint 3j=1 P (j/xi ; Θ(t)) = 1, to be satisfied for all xi ’s.

In this case, the EM algorithm 1 converges after 313 iterations. The estimates for the unknown parameters are: µ1 = 1.0055, µ2 = 3.0002, µ3 = 2.0277, σ12 = 0.0095, σ22 = 0.0104, σ32 = 0.0407, P1 = 0.2497, P2 = 0.4998, P3 = 0.2506.

2.34 The geometry of the problem is given in the figure 2.2. Since P (ω1 ) = P (ω2 ) = 12 the probability of having m points inside ω1 is given P1 = 1

1 N C 2N m

In problem 14.4 we give MATLAB code for the EM algorithm

27

Figure 2.2: Problem 2.34 N Cm =

N! m!(N − m)!

2N is the number of possible ways of partitioning N points in two classes, N and Cm the number of ways in which the N points can divided so that to have m of them in one class.

Assume now that we are given a point from ω1 . Using the N N classifier the probability of committing an error is equal to the probability of having as nearest neighbor to x from ω2 . However, due to the geometry this is equal to the probability of having all training patterns in ω2 which is equal to Pe′ =

1 2N

If the kN N classifier is used, k ≥ 3, then an error is committed if less or equal to k−1 points lie in ω1 and the rest in ω2 . Hence this probability is 2 k−1 2 1 X Pe = N C N > Pe 2 m=0 m

Observe that this is true for finite values of N . (The proof from Tou and Gonzalez).

2.35 Function [nnerr,nn3err]=nn % The output of the program is the percentage error of the NN and % the 3NN method. m1=[1,1]; m2=[1.5,1.5]; s=0.2;

28 % class1:Is a matrix (2X50) containing the training set of the first %class. % class2:Is a matrix (2X50) containing the training set of the second %class. class1=[gaussian(m1(1),s,50);gaussian(m1(2),s,50)]; class2=[gaussian(m2(1),s,50);gaussian(m2(2),s,50)]; %unknown:Is a matrix (2X200). The first 100 samples are distributed %from the first class and the other 100 samples are distributed %from the second one unknown=[gaussian(m1(1),s,100),gaussian(m2(1),s,100);... gaussian(m1(2),s,100),gaussian(m2(2),s,100)]; %%%%%%%%%% NN Method %%%%%%%%% for i=1:200 dist=[]; for j=1:50 dist1(j)=(unknown(:,i)-class1(:,j))’*(unknown(:,i)-class1(:,j)); dist2(j)=(unknown(:,i)-class2(:,j))’*(unknown(:,i)-class2(:,j)); end dist(1:100)=[dist1,dist2]; dist(2,:)=1:100; [a,b]=min(dist(1,:)); if dist(2,b)<=50 classifynn(i)=1; else classifynn(i)=2; end end %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%% 3NN Method %%%%%%%%%%%%%%%% for i=1:200 dist=[]; for j=1:50 dist1(j)=(unknown(:,i)-class1(:,j))’*(unknown(:,i)-class1(:,j)); dist2(j)=(unknown(:,i)-class2(:,j))’*(unknown(:,i)-class2(:,j)); end

29 dist(1:100)=[dist1,dist2]; dist(2,:)=1:100; cl1=0; cl2=0; for n=1:3 [a,b]=min(dist(1,:)); if dist(2,b)<=50 cl1=cl1+1; else cl2=cl2+1; end dist(:,b)=[]; end if cl1>cl2 classify3nn(i)=1; else classify3nn(i)=2; end end %classify3nn: %%%%%%%%%%%%%%%%%%%%%%%%%%%%% nnerr=(length(find(classifynn(1:100)==2))+... length(find(classifynn(101:200)==1)))*100/200; nn3err=(length(find(classify3nn(1:100)==2))+... length(find(classify3nn(101:200)==1)))*100/200; 2.36 function parzen(N,h) n=rand(1,N)*2; privi=0; p=[]; for m=-0.5:h:2.5 newp=0; for a=1:size(n,2) x=(n(a)-m)/h; newp=newp+(1/sqrt(2*pi))*exp(-(x)∧ 2/2);

30 end newp=(1/h)*(1/size(n,2))*newp; p=[p newp]; end x=[zeros(1,length(-0.5:0.01:-0.01)),ones(1,length(0:0.01:2))*0.5,... zeros(1,length(2.01:0.01:2.5))]; xax=-0.5:0.01:2.5; m=-0.5:h:2.5; plot(m,p,’r’,xax,x,’k’); 2.37 function p=knnest(N,k) % N: Number of samples % k: Number of points p=[]; samples=rand(1,N)*2; for i=-0.5:0.025:2.5 ok=0; xleft=i; xright=i; while ok==0 xleft=xleft-0.001; xright=xright+0.001; num=length(find(samples>xleft & samples<xright)); if num>=k num p=[p k/(N*(xright-xleft))]; ok=1; end end end x=[zeros(1,length(-0.5:0.01:-0.01)),ones(1,length(0:0.01:2))*...

0.5,zeros(1,length(2.01:0.01:2.5))]; xax=-0.5:0.01:2.5; i=-0.5:0.025:2.5; plot(i,p,’r’,xax,x,’k’);

31 2.38 We have already seen in the book that p̂(x) =

N X

1 xi − x φ l h i=1 N h

(2.51)

and also that the mean value of this estimate is given by E[p̂(x)] =

Z

y−x 1 p(y)dy φ l h h

Since each one of the terms in the summation in Eq. (2.51) has the same mean value, this is obviously equal to N1 E[p̂(x)] ≡ p̄. Assuming that the terms in the summation are statistically independent, the variance of their sum is equal to the sum of their variances, that is 2 σN (x) =

N X

E

i=1

or 2 σN (x) =

2 σN (x) =

#

2 1 xi − x − p̄ φ N hl h

1 2 xi − x xi − x E + p̄2 − φ2 p̄φ 2 2l l N h h Nh h i=1

N X i=1

and finally

N X

"

E

xi − x 1 φ2 2 2l N h h

2 σN (x) = N E

+ p̄2 − 2p̄E

1 xi − x φ2 2 2l N h h

−

1 xi − x φ l Nh h

1 (E[p̂(x)])2 N

The second term on the right hand side is constant for each x. Thus neglecting it we have 2 σN (x) ≤ N E

1 2 xi − x φ N 2 h2l h

and using the definition of the expectation, we obtain 2 σN (x)

1 Z 1 2 y−x p(y)dy φ ≤ N hl hl h

1 Z 1 y−x sup(φ)p(y)dy φ ≤ l h N hl h sup(φ)E[p̂(x)] ≤ N hl

Solutions to the problems of Chapter 3 3.1 The perceptron cost function is J(w) =

X

(δx wT x)

x∈Y As long as the set Y does not change, this is a linear function w.r. to w. As now the value of w changes, there will be a position for which a point x1 which does not belong to Y , will be misclassified (which belongs to Y will be correctly classified). When the hyperplane w crosses x1 its contribution is zero (wT x1 = 0). Once x1 is included (excluded) from Y , J(w) is not the same linear function as before.

3.2 The proof proceeds as in the theory with step varying ρk , up to the equation ||w(t + 1) − αw∗ ||2 ≤ ||w(t) − αw∗ ||2 + ρ2 β 2 − 2ρα|γ| 2

β If we choose α = |γ| then we get

||w(t + 1) − αw∗ ||2 ≤ ||w(0) − αwx ||2 + (t + 1)β 2 ρ(ρ − 2) Then as t keeps increasing, provided that 0 < ρ < 2 there will be a step t for which 0 ≤ ||w(t + 1) − αw∗ ||2 ≤ 0 1

2 ence the algorithm converges in a finite number of steps. t+1 =

||w(0) − αw∗ ||2 β 2 ρ(2 − ρ)

3.3 See S. Haykin’s book.

3.4 The vectors appear in the algorithm in the following sequence: [0, 0]T , [0, 1]T , [1, 0]T , [1, 1]T in their extended form. the iteration steps are: • w(0) = [0, 0, 0]T − wT (0)[0, 0, 1]T = 0, correction •

w(1) = w(0) + [0, 0, 1]T = [0, 0, 1]T − wT (1)[0, 1, 1]T = 1 > 0, no action

•

w(2) = w(1) − wT (2)[1, 0, 1]T = 1 > 0, correction

•

w(3) = w(2) − [1, 0, 1]T = [−1, 0, 0]T − wT (3)[1, 1, 1]T = −1, no action

•

w(4) = w(3) − wT (4)[0, 0, 1]T = 0, correction

•

w(5) = w(4) + [0, 0, 1]T = [−1, 0, 1]T − wT (5)[0, 1, 1]T = 1 > 0, no action

•

w(6) = w(5) − wT (6)[1, 0, 1]T = 0, correction

•

w(7) = w(6) − [1, 0, 1]T = [−2, 0, 0]T − wT (7)[0, 0, 1]T = 0, correction

•

w(8) = w(7) + [0, 0, 1]T = [−2, 0, 1]T

3 − wT (8)[0, 1, 1]T = 1 > 0, no action •

w(9) = w(8) − wT (9)[1, 0, 1]T = −1 < 0, no action

•

w(10) = w(9) − wT (10)[1, 1, 1]T = −1 < 0, no action

•

w(11) = w(10) − wT (11)[0, 0, 1]T = 1 > 0, no action Thus the hyperplane corresponding to w = [−2, 0, 1]T classifies correctly all the training vectors, i.e., −2x1 + 1 = 0 3.5 In order to produce 50 vectors for each class, disregarding vectors that violate linear seperability, we use the following Gener.m function function [x,y]=gener; %Produces 50 vectors for each class and disregarding %the improper vectors so that to guarantee linear % separability of the classes. y1=ones(50,1); y2=-ones(50,1); x1(:,1)=gaussian(1,0.2,50)’; x1(:,2)=gaussian(1,0.2,50)’; x2(:,1)=gaussian(0,0.2,50)’; x2(:,2)=gaussian(0,0.2,50)’; r=sum(x1’); k=find(r<1); x1(k,:)=[]; y1(k)=[]; r=sum(x2’); k=find(r>1); x2(k,:)=[];

4 y2(k)=[]; x=[x1;x2]’; y=[y1;y2]’; In the sequel we present these to the perceptron algorithm. function w=perc(x,y,ro); % The 2-D Perceptron algorithm. % x: 2xN Matrix of samples. % y: 1xN Vector with values +1 or -1 depending on the class % to where the respective sample belongs. % ro: Learning rate parameter. x(3,:)=ones(1,size(x,2)); classA=find(y==1); classB=find(y==-1); plot(x(1,classA),x(2,classA),’r*’,x(1,classB),x(2,classB),’b*’); hold on w=rand(3,1); s=[1 1 1]’ while s =[0 0 0]’ s=[0 0 0]’; for i=1:size(x,2); t=w’*x(:,i); if t*y(i)<0, s=s-ro*x(:,i)*y(i); end; end; w=w-s; end; d=-1.5:0.01:2.5; for i=1:size(d,2), y(i)=(-w(3)-w(1)*d(i))/w(2); end; plot(d,y); Thus, with the following commands we have the proper results.

5 [x,y]=gener; w=perc(x,y,0.7); 3.6 In order to produce 100 vectors for each class,we use the following function function [x,y]=gener2 x=[];y=[];x1=0;x2=0; while x1<100 | x2<100 rnd=rand; if rnd<0.5 & x1<100 newx=[gaussian(3,0.2,1);gaussian(3,0.2,1)]; x=[x newx]; y=[y 1]; x1=x1+1; elseif rnd>0.5 & x2<100 newx=[gaussian(1,0.2,1);gaussian(1,0.2,1)]; x=[x newx]; y=[y -1]; x2=x2+1; end end Generating the 100 samples we present these to the LMS algorithm function w=LMS(x,y,ro) %2-D LMS Algorithm %x: 2xN Matrix of samples. %y: 1xN Vector with values +1 or -1 depending on the class % to where the respective sample belongs %ro: Learning rate parameter classA=find(y==1); classB=find(y==-1); plot(x(1,classA),x(2,classA),’+’,x(1,classB),x(2,classB),’b*’); hold on

6 x(3,:)=ones(1,size(x,2)); w=[0.1 0.1 0.1]’; for i=1:size(x,2); w=w+ro*x(:,i)*(y(i)-x(:,i)’*w); end; d=-1:0.01:4; for i=1:size(d,2), k(i)=(-w(3)-w(1)*d(i))/w(2); end; plot(d,k); 3.7 The block weight vector is W T = [wT1 , wT2 , . . . , wTi , . . . , wTj , . . . wTM ] For an x(t) ∈ wi there are (M − 1) block vectors of the form xTij(t) = [0, 0, . . . , x(t) , . . . , −x(t) , 0, . . . , 0],

j = 1, 2, . . . , M

j 6= i

Correction takes place for those j positions of the vector −x(t) for which W (t)T xTij(t) ≤ 0 or wi (t)T x(t) ≤ wj (t)T x(t) and the correction is W (t + 1) = W (t) + ρxij(t) or equivalently wi (t + 1) = wi (t) + ρx(t) wj (t + 1) = wj (t) − ρx(t) 3.8 The cost function is equivalent to J(w) =

N 1 X e2 N i=1 i

7 which is the sample mean of the square errors. For large N and assuming ergodicity, this is nothing else but E[e2 ].

3.9 The following function realizes the sum of error squares classifier function w=Sum of error(x) % 2-D classifier using the sum of error squares criterion. % x: 2xN Matrix of samples. % y: 1xN Vector with values +1 or -1 depending on the % class to where the respective sample belongs. x(:,3)=ones(size(x,1),1); classA=find(y==1); classB=find(y==-1); plot(x(1,classA),x(2,classA),’r*’,x(1,classB),x(2,classB),’b*’); w=(inv(x’*x))*(x’*y); d=0:0.01:2; for i=1:size(d,2), k(i)=(-w(3)-w(1)*d(i))/w(2); end; plot(d,k); 3.10 Following similar notation as for the mean square case we have J(W ) =

N X i=1

=

T

2

||y i − W xi || =

N M X X

j=1 i=1

3.11 p(x, y) =

(yij − wTi xi )2

M N X X

i=1 j=1

=

M X

(yij − wTi xi )2

J(wj )

j=1

1

1 exp(− ([x, y]Σ−1 [x, y]T )) 2 2π|Σ| 1 2

or p(x, y) =

1 √

2πσx σy 1 − α

exp(− 2

(y − µy )2 (x − µx )2 1 + [ 2(1 − α2 ) σx2 σy2 (x − µx )(y − µy ) − 2α ]) σx σy

8 From the above, the marginal pdf’s can be obtained after integration, i.e., (x − µx )2 1 exp(− ) 2σx2 2πσx

p(x) =

Z

p(x, y)dy = √

p(y) =

Z

p(x, y)dx = √

1 (y − µy )2 ) exp(− 2σy2 2πσy

and p(x, y) = p(y|x)p(x) or

p(x, y) = p(y|x) p(x)

or p(y|x) =

1 q

σy 2π(1 − α2 )

exp(−

(y − µy − α σσxy (x − µx ))2 ) 2σy2 (1 − α2 )

where we have used the fact that (x − µx )2 α2 (x − µx )2 (x − µx )2 + = 2σx2 (1 − α2 ) 2σx2 2σx2 (1 − α2 ) Then E[y|x] = which is equal to

Z

yp(y|x)dy

E[y|x] = µy + α

σy (x − µx ) σx

3.12 The joint probability density of the available sample points is p(y, x; w),which depends on the weights, wT = [w1 , . . . , wM ]T , of the discriminant functions g(x; wk ), k = 1, 2, . . . , M . The vector y T = [y1 , . . . , yM ]T denotes the difference between the actual responses g(x; wk ), and the desired responses. Let also dik be the desired response of the k th class when xi appears at the input.

The loglikelihood function is L(w) =

N X

ln p(y i , xi ; w)

i=1

=

N X i=1

ln p(y i |xi ; w) +

N X i=1

ln p(xi )

9 The last term does not depend on w and for our purpose it can be omitted.

Thus L(w) =

N X i=1

ln p(y i |xi ; w)

Assuming independence of the outputs yk , k = 1, . . . , M we have L(w) =

M N X X

i=1 k=1

ln p(yki |xi ; w)

However, from the assumption of the problem we have that yki = g(xi ; wk ) − dik is normally distributed.

Thus, M N X 1 X L(w) = − 2 (g(xi ; wk ) − dik )2 + constants 2σ i=1 k=1

Hence maximizing L(w) is equivalent to minimizing the sum of squares error.

3.13 From the respective definitions we have J(w) = E[(d − f (x; w))2 ] with d = 1 (−1) for x ∈ ω1 (ω2 ) or Z

J(w) = P (ω1 ) (1 − f (x; w))2 p(x|ω1 )dx + P (ω2 )

Z

(1 + f (x; w))2 p(x|ω2 )dx

or J(w) = + −

Z

Z

Z

(P (ω1 )p(x|ω1 ) + P (ω2 )p(x|ω2 ))dx

(P (ω1 )p(x|ω1 ) + P (ω2 )p(x|ω2 ))f 2 (x; w)dx 2f (x; w)(P (ω1 )p(x|ω1 ) − P (ω2 )p(x|ω2 ))dx

But P (ω1 )p(x|ω1 ) + P (ω2 )p(x|ω2 ) = p(x)

10 and P (ω1 )p(x|ω1 ) − P (ω2 )p(x|ω2 ) =

= (P (ω1 |x) − P (ω2 |x))p(x) ≡ g(x)p(x)

Hence

J(w) = 1 + −2 If we add and subtract

R

Z

Z

f 2 (x; w)p(x)dx

f (x; w)(g(x)p(x))dx

g 2 (x)p(x)dx we obtain

J(w) = 1 +

Z

(f (x; w) − g(x))2 p(x)dx −

Z

g 2 (x)p(x)dx

Thus, minimizing J(w) is equivalent to minimizing E[(f (x; w) − g(x))2 ] 3.14 The MSE solution is provided by " "

x 1

#

x ] = P (ω1 )1 1

"

E1 [x] 1

E

[xT , 1]

#"

w w0

#

#

= E[d

"

x ] 1

"

#

=

"

1 (µ1 − µ2 ) 2

#

However, E[d

"

#

where Ei [x] =

R

#

− P (ω2 )1

E2 [x] 1

1 (µ1 − µ2 ) 2

xp(x|ωi )dx, i = 1, 2. Hence,

"

R E[x] E[xT ] 1

#"

w w0

#"

w w0

#

or

"

where µ =

R µ µT 1

#

=

"

=

"

1 (µ1 − µ2 ) 2

0

0

#

1 (µ + µ2 ) = P (ω1 )E1 [x] + P (ω2 )E2 [x] 2 1

0

#

11 From the definition of the covariance matrix we have Σ = E[(x − µ)(xT − µT )] = R − µµT or

"

Σ + µµT µ µT 1

#"

w w0

#

=

"

1 (µ1 − µ2 ) 2

0

#

µT w = −w0

and

Σw + µµT w + µw0 =

1 (µ − µ2 ) 2 1

and combining the above Σw = or w =

1 (µ − µ2 ) 2 1

1 −1 Σ (µ1 − µ2 ) 2

Hence the criterion now becomes (wT + w0 )x > (<)0 or

1 (µ1 − µ2 )T Σ−1 (x − (µ1 + µ2 )) > (<)0 2

3.15 Figure 3.1 demonstrates an example with three classes. Note that g1 (x) leaves ω1 on its positive side and the other two classes on its negative side.

Similarly, g2 (x), g3 (x) separate classes ω2 , ω3 from the others, respectively.

It is not difficult to see that there are four indeterminate regions, one that lies on the negative side of all decision hyperplanes, and three for which more than one of the discriminant functions result in positive values.

3.16 The problem can be stated as minimize subject to

w12 + w22 J(w) = 2 w1 + w2 − 1 ≥ 0 w1 − w2 − 1 ≥ 0

12

Figure 3.1: Problem 3.15 That is, only two constraints are available. The Lagrangian function and the resulting KKT conditions are J(w, λ1 , λ2 ) =

w12 + w22 − λ1 (w1 + w2 − 1) − λ2 (w1 − w2 − 1) 2 w1 = λ 1 + λ 2 w2 = λ 1 − λ 2 λ1 (w1 + w2 − 1) = 0 λ2 (w1 − w2 − 1) = 0 λ1 , λ2 ≥ 0

Assumption I: The second of the constrains is only active, thus λ1 = 0. Then w1 = λ 2 w2 = −λ2 w1 − w2 = 1 The solution of the above results in λ2 = 12 , w1 = 12 , w2 = − 12 , which is not a valid point since it lies outside the feasible region (the first constraint is

13

Figure 3.2: Problem 3.16 not satisfied), see fig. 3.2 Assumption II: The first of the two constraints is now assumed to be active, hence, λ2 = 0. Following similar arguments we obtain λ1 = w1 = w2 = 21 .

This is also a solution outside the feasible region (the second constrain is not satisfied).

Assumption III: Both constraints are active. Then w1 w2 w1 + w2 w1 − w2

= = = =

λ1 + λ2 λ1 − λ2 1 1

Solving the above system of equations results in λ1 = λ2 = 21 , w2 = 0, w1 = 1. In other words, the optimal separating line is g(x) = x1 = 0

(3.1)

Figure 3.2 illustrates the geometry. The optimal point in the (w1 , w2 ) plane is the corner point of the feasible region, where the increasing isovalue curves of the cost function meet the feasible region for the first time.

14 3.17 The Lagrangian is equal to N N X 1 1 X 2 L = ||w|| − νρ + µi ξi ξi − 2 N i=1 i=1

−

N X i=1

h

i

λi yi (wT xi + w0 ) − ρ + ξi − δρ

Substituting (3.130)-(3.133) in the above we obtain 



N N N N X X X 1 X T  λi yi ( λj yj xTj )xi  ( λi yi xi )( λj yj xj ) − L = 2 i=1 j=1 i=1 j=1

−νρ +

N X i=1

!

λi ρ − δρ

N N X X 1 X + µi ξi − ξi − λi ξi N i=1 i=1 i=1

−w0 or

N X i=1

λ i yi

!

!

N N X 1X λi λj yi yj xTi xj L=− 2 i=1 j=1

and since µ ≥ 0 and ν ≥ 0, (3.132) and (3.133) suggest that the constraints for λ, entering the optimization problem, are N X

λ i yi = 0

i=1

0 ≤ λi ≤ N X i=1

1 N

λi ≥ ν

3.18 Instead of starting from (3.85)-(3.87), a proof that is given in Keerthi et. al., we will provide a proof through the ν-SVM formulation. For the linearly separable problem, ξi = 0 in (3.149)-(3.151), the ν-SVM formulation primal problem becomes

15

||w||2 − 2ρ yi [wT xi + w0 ] ≥ ρ ρ≥0

minimize subject to

The Lagrangian is given by L(w, w0 , ρ) = ||w||2 − 2ρ −

X i

λi yi [wT xi + w0 ] − ρ − µρ

Taking derivatives and equating to zero we obtain w = X

1X λ i yi x i 2 i

λ i yi = 0

i

X

λi = µ + 2

i

Substituting the above in the Lagrangian results in L(w, w0 , ρ) = −

1 XX λi λj yi yj xTi xj 4 i j

and the KKT conditions are λi (yi [wT xi + w0 ] − ρ) = 0 µρ = 0 However, if ρ = 0 then it is easily seen from the primal formulation that the optimal solution is the trivial one, i.e., w = 0 and w0 = 0. Thus, µ = 0 and the dual problem becomes 1 XX yi yj λi λj xTi xj 4 i j

maximize

−

subject to

X

λi = 2

i

X

λ i yi = 0

i

λi ≥ 0

16 and the first equation is equivalent to 1 XX yi yj λi λj xTi xj 4 i j

minimize

The above is the same optimization problem as that given by (3.146)-(3.148), associated with the task of computing the nearest points between convex hulls. Once the nearest points have been found, i.e., x+ =

X

λi xi , x− =

i:yi =1

X

λi xi

i:yi =−1

for the optimal values of λi , i = 1, . . . , N then w = x+ − x− which is equal to the value obtained from the previous ν-SVM task. The difference in scaling is of no importance since both vectors point in the same direction.

Since in the nearest point geometric procedure w is the bisecting plane, the value of b is easily seen to be equal to

1 b = − wT x+ + x− 2

This is the same value resulting from the KKT conditions. Since the nearest points are support vectors we have wT x+ + b = ρ wT x− + b = −ρ Adding the above we end up with the same value of b as before.

3.19 This is trivially shown from (3.158). If the two reduced convex hulls intersect then there exist feasible points such that x=

X

i:yi =1

λi xi =

X

λi xi

i:yi =−1

which leads to a w = 0. If on the other hand, the two RCHs do not intersect, then, since there are no common points in the RCHs, (3.158) always results in a nontrivial solution. However, note that the set of Lagrange multiplieres

17 is not unique. This is obvious since there may be more than two pairs of points having the same (minimum) distance. Take for example two RCHs where the edges in which a pair of closest points lie are parallel. Then there is an infinite number of points with the same (minimum) distance.

3.20 Let the soft margin primal formulation have the form minimize subject to Define

N 1 CX ||w||2 + ξ2 2 2 i=1 i

wT xi + b ≥ 1 − ξi

√

1 Cξ T ]T , x̃i = [xTi , √ eTi ]T , b̃ = b C where ei is the N -dimensional vector with 0s everywhere except at the dimension i where it is equal to 1. It is now straightforward to see that the problem is equivalent to w̃ = [wT ,

minimize subject to

1 ||w̃||2 2 w̃T x̃i + b̃ ≥ 1

Solutions to the problems of Chapter 4 4.1 It is obvious from the figure 4.1 that the classes are not linearly separable.

Figure 4.1: Problem 4.1 Two lines that separate the classes are 2x1 + 2x2 − 1 = 0 2x1 + 2x2 − 3 = 0 Thus the two layer perceptron classifies the classes and it is shown in figure 4.2.

1

2

Figure 4.2: Problem 4.1

4.2 Function backprop2l(lr,fl,sl,epoch) % lr: Learning rate % fl: # of nodes of the first layer % sl: # of nodes of the second layer % epoch: # of epochs data1=[gaussian(0,0.01,100);gaussian(0,0.01,100)]; data2=[gaussian(1,0.01,100);gaussian(1,0.01,100)]; data3=[gaussian(0,0.01,100);gaussian(1,0.01,100)]; data4=[gaussian(1,0.01,100);gaussian(0,0.01,100)]; input=[data1,data2,data3,data4]; test=[test1,test2,test3,test4]; target=zeros(1,400); target(1:200)=0; target(201:400)=1; net=newff([-0.5,1.5;-0.5,1.5],[fl,sl,1],’logsig’,’logsig’,’logsig’,’traingd’); % For faster convergence you can use the traingda or the traingdx % method instead % of the traingd, eliminating the role of the learning rate constant. net=init(net);

3

net.trainParam.show=50; net.trainParam.lr=lr; net.trainParam.epochs=epoch; net.trainParam.goal=1e-3; net=train(net,input,target); test1=[gaussian(0,0.01,50);gaussian(0,0.01,50)]; test2=[gaussian(1,0.01,50);gaussian(1,0.01,50)]; test3=[gaussian(0,0.01,50);gaussian(1,0.01,50)]; test4=[gaussian(1,0.01,50);gaussian(0,0.01,50)]; a=sim(net,test); error=length(find(a(1:100)>0.5)); error=error+length(find(a(101:200)<=0.5)); disp(’the classification error is ’,num2str(error*100/200),’% ’); 4.3 The lines are g1 (x) = x1 + x2 = 0 1 g2 (x) = x2 − = 0 4 g3 (x) = x1 − x2 = 0 and are shown in the figure 4.3.

The (hyper)cube is shown in figure 4.4

For a two layer perceptron to be sufficient we combine (000) ∪ (100) ∪ (001) For a three layer perceptron to be required we can combine (000) ∪ (111)

4

Figure 4.3: Problem 4.3

Figure 4.4: Problem 4.3

5

4.4 The geometry is shown in the figure 4.5 we have

Figure 4.5: Problem 4.4

1 x − (x1 + x2 ) ⊥ x1 − x2 2 hence

or

1 (x1 − x2 )T (x − (x1 + x2 )) = 0 2 1 1 (x1 − x2 )T x − ||x1 ||2 + ||x2 ||2 = 0 2 2

For x = x1 we obtain 1 1 ||x1 ||2 + ||x2 ||2 − xT2 x1 > 0 2 2 4.5 J = −

kL N X X

yk (i) ln

i=1 k=1

= −

kL N X X

i=1 k=1

yk (i) ln ŷk (i) +

ŷk (i) yk (i)

kL N X X

i=1 k=1

yk (i) ln yk (i)

6 For binary valued desired responses the second term is zero. The first term takes its minimum value when yk (i) = ŷk (i) and it becomes J = −

kL N X X

yk (i) ln yk (i) = 0

i=1 k=1

For the second part of the problem we have that ŷk (i) = yk (i) + εk (i) where εk (i) is the error. Then kL N X X

yk (i) ln

kL N X X

yk (i) ln(1 +

J = −

i=1 k=1

= −

yk (i) + εk (i) yk (i)

i=1 k=1

εk (i) ) yk (i)

Thus J depends on the relative errors.

4.6 From the theory δjL (i) =

∂E(i) ∂vjL (i)

and ŷj (i) = f (vjL (i)) The cost function is J = −

N X

E(i)

i=1

with E(i) = =

kL X

k=1 k L X

yk (i) ln

yk (i) ln

k=1

ŷk (i) yk (i)

f (vkL (i) yk (i)

yj (i) ′ L ∂E(i) = + f (vj (i)) L ∂vj (i) f (vjL (i)) yj (i) ′ L = f (uj (i))(1 − f (uLj (i)) f (uLj (i))

δjL (i) ≡

7 or δjL (i) = α(1 − ŷj (i))yj (i) 4.7 E(i) =

kL X

yk (i) ln

k=1

ŷk (i) = P δjL (i) =

ŷk (i) yk (i)

exp(vkL (i)) L k′ exp(vk′ (i))

kL X ∂E(i) ∂ yk (i)vkL (i) = ( ∂vjL (i) ∂vjL (i) k=1

−

kL X

X

yk (i) ln(

exp(vkL′ (i))))

k′

k=1 kL X

exp(vjL (i)) yk (i) P = yj (i) − L k′ exp(vk′ (i)) k=1 = yj (i) − ŷj (i)

kL X

yk (i) = yj (i) − ŷj (i)

k=1

4.8 M X

J = −E[

yi ln ŷi + (1 − yi ) ln(1 − ŷi )]

i=1

Following similar arguments as in the relevant theory of chapter 3 we obtain J = −E[

M M X X

(

yi ln ŷi + (1 − yi ) ln(1 − ŷi ))P (ωj |x)]

j=1 i=1

M M X X

= −E[

(

(yi ln ŷi )P (ωj |x) +

i=1 j=1

M X

j=1

(1 − yi ) ln(1 − ŷi )p(ωj |x))]

8 Taking into account that ŷi ’s depend only on x and not on ωj we have M X

J = −E[

(E[yi |x] ln ŷi + (1 − E[yi |x]) ln(1 − ŷi ))]

i=1

the minimum of the above is achieved when ŷi = E[yi |x] To verify, take the derivative w.r.t. ŷi . Following the related results of chapter 3, the above proves the problem 4.9 Recall from combinatorics that

and also

M X

m=0

M m

!

m m+k

!

= 0

= 2M

k>0

4.10 This is available from the RR demo (example 3) in the book’s website. The MATLAB m.functions which are used are: bptr3pr.m, trbp3pr.m 4.11 The MATLAB functions which realise the pruning are prunning.m, sali.m and are available from the RR demo (example 3), choosing the pruning option in the propagation menu.

4.12

kL N X 1X J = (ŷm (i) − ym (i))2 2 i=1 m=1 kL N X ∂J X ∂ ŷm (i) (ŷm (i) − ym (i)) r r ∂wkj i=1 m=1 ∂wkj kL M X X ∂2J ∂ 2 ŷm (i) = (ŷm (i) − ym (i)) ′ r r r′ ∂wkr ′ j ′ ∂wkj i=1 m=1 ∂wk′ j ′ ∂wkj

+

kL N X X ∂ ŷm (i) ∂ ŷm (i) ′

i=1 m=1

r ∂wkj ∂wkr ′ j ′

9 If we are near the minimum ŷm (i) ≃ ym (i) thus, kL N X X ∂2J ∂ ŷm (i) ∂ ŷm (i) ≃ r r′ r r′ ∂wk′ j ′ ∂wkj i=1 m=1 ∂wkj ∂wk′ j ′

4.13

∂ ∂E ∂ ∂E ∂vjr ∂ 2E = [ ] = [ r ][ r ] = r r r r ∂ 2 wkj ∂wkj ∂wkj ∂wkj ∂vj ∂wkj

(4.1)

∂δjr r−1 ∂ r r−1 δ y y = r j k r k ∂wkj ∂wkj However

Hence

r ∂δjr ∂vjr ∂δjr r−1 ∂δj = = yk r r ∂wkj ∂vjr ∂wkj ∂vjr

(4.2)

∂ 2 E r−1 2 ∂ 2E = (y ) r ∂ 2 wkj ∂ 2 vjr k

(4.3)

L L where E = 21 km=1 (f (vm ) − ym )2 .

For the backpropagation of the second derivatives we have I r = L ∂E = f ′ (vjL )ej and L ∂vj

P

∂ej ∂ 2E = f ′′ (vjL )ej + (f ′ (vjL ) L r 2 ∂ vL ∂vj = f ′′ (vjL )ej + (f ′ (vjL ))2

(4.4)

r < L kr kr X X ∂vkr r ′ r−1 δkr r−1 δjr−1 = δkr wkj f (vj ) = ∂v j r=1 k=1

(4.5)

II

and

kl r X ∂δjr−1 ∂2E ′ r−1 r ∂δk = = ) w jk r−1 f (vj ∂ 2 vjr−1 ∂vjr−1 ∂v j k=1

+

kl X

k=1

r r wkj δk

∂f ′ (vjr−1 ) ∂vjr−1

10 = f ′′ (vjr−1 )

kl X

r r wkj δk +

r wkj

k=1

k=1

also

kl X

∂δkr ′ r−1 f (vj ) ∂vjr−1

X ∂δkr ∂vkr ′ X ∂ 2E ∂δkr wkr ′ j f ′ (vjr−1 ) = = r−1 r r r ∂v ∂v ∂v ∂vjr−1 ∂v ′ ′ k j k k k′ k′

(4.6)

(4.7)

∂ 2 E r ′ r−1 w f (vj ) ∂ 2 vkr kj

= from (4.6),(4.7)we obtain

kl X X ∂ 2E ∂ 2E r r r 2 ′′ r−1 ′ r−1 2 wkl δk (w ) + f (v ) = (f (v )) kj j j r ∂v ∂ 2 vkr−1 k k=1 k′

4.14 See Bishop 4.15 The only difference from theory lies in the derivative of f (x), and we have f (x) = c tanh(bx) ⇒ f ′ (x) = (cb) cosh−2 (bx) or f ′ (x) = cb(1 − tanh2 (bx)) or f ′ (x) = cb(1 − f 2 (x)) 4.16 µ = µ0

1 1 + tt0

a) For t ≪ t0 , µ = µ0 b) For t ≫ t0 , µ ≃ tt0 µ0 That is, for the early stages µ is constant and later it becomes smaller and smaller to smooth out the convergense 4.17 Function approx(lr,fl,sl,epoch) % lr: Learning rate % fl: # of nodes of the first layer

11 % sl: # of nodes of the second layer % epoch: # of epochs input=rand(1,1000); target=0.3+0.2*cos(2*pi*input); net=newff([-1,2],[fl,sl,1],’logsig’,’logsig’,’purelin’,’traingdx’); net=init(net); net.trainParam.show=50; net.trainParam.lr=lr; net.trainParam.mc=0.9; net.trainParam.epochs=epoch; net.trainParam.goal=1e-4; net=train(net,input,target); test=0:0.01:1; out=sim(net,testin); plot(test,0.3+0.2*cos(2*pi*test),’b’,test,out,’r’); legend(’real function’,’approximation’,4); 4.18 Let us assume that we start with N points in the l-dimencional space, and we add an extra point P . Then, the O(N, l) old dichotomies fall into either of the two categories.

• For those dichotomies where the dichotomizing plane can not pass through P , the new point P can only belong to the old dichotomies.

• For those dichotomies for which the plane could pass through P , then to each one of the old dichotomies correspond two new dichotomies. This is because by shifting infinitesimally the plane, P can change class ownership.

Thus, the total number of dichotomies with the N + 1 points will be O(N + 1, l) = O(N, l) + O(N, l − 1)

(4.8)

The second term is the number of dichotomies of N points in the l-dimensional space that are constrained to pass through a specific point P . This is equiv-

12 alent with drawing dichotomies of N points in the (l − 1)-dimensional space.

In the sequel, we will show by induction that O(N, l) =

!

!

O(N − k, l − k) for k = 1, 2, . . . , N − 1

k k

+

!

k 0

O(N − k, l) +

k 1

O(N − k, l − 1) + . . .

(4.9)

The above is true for k = 1. Indeed 1 0

!

1 1

= 1 =

!

Then assume it is true for some k. We will show that it is also true for k + 1.

We have that O(N − k, l) = O(N − k − 1, l) + O(N − k − 1, l − 1) Thus k 0

O(N, l) = k 1

+ k k

... +

!

!

{O(N − k − 1, l − k) + O(N − k − 1, l − k − 1)}

k r

k r

!

+

{O(N − k − 1, l) + O(N − k − 1, l − 1)}

{O(N − k − 1, l − 1) + O(N − k − 1, l − 2)} +

However

Indeed

!

!

!

!

k r

!

=

k! k! ,= r!(k − r)! r!(k − r − 1)!(k − r)

+

k r+1

!

!

=

k r+1

k r+1

=

=

k+1 r+1

k! r!(r + 1)(k − r − 1)!

(r + 1)k! + (k − r)k! = r!(r + 1)(k − r − 1)!(k − r) ! (k + 1)k! k+1 = r+1 (r + 1)!(k + 1 − r − 1)!

13 Also

!

=

O(N − k − 1, l) +

k+1 1

!

... +

k+1 k+1

k 0

!

k+1 0

!

k+1 0

=

!

k k

and

k+1 k+1

!

Hence O(N, l) =

!

O(N − k − 1, l − 1) +

O(N − k − 1, l − k − 1)

Thus (4.9) is true. By convention O(N, l) = 0 for l < 0. Then setting in (4.9) k = N − 2 and taking into account that O(1, l) = 2,

l ≥ 0

l X

N −1 i

we obtain O(N, l) = 2

i=0

!

Note that the convention O(1, 0) = 2 is in line with the recursion in (4.8).

Indeed for l = 1 and N = 1 we have O(2, 1) = O(1, 1) + O(1, 0) However O(2, 1) = 4, since 2 points in a line can be separated by a point in four possible groupings. Also O(1, 1) = 2, since a point can belong to either of two classes. Then O(1, 0) = 2 is in agreement with the recursion.

4.19 From the theory we have for N = 2l + 2 2l+1 X i=0

!

= 22l+1

l 1X = 2[ [ 2 i=0

2l + 1 i

2l + 1 i

Then O(2l + 2, l) = 2

l X i=0

2l + 1 i

!

!

+

2l+1 X

i=l+1

2l + 1 i

!

]]

1 = 2[ 22l+1 ] = 2N −1 2

14

4.20 Function rbf(num centres,stand dev,numtrain); % num centres: # of centres % stand dev: Standard deviation % numtrain: # of train samples c=0:1/(num centres-1):1; train source=rand(1,numtrain); Y=[0.3+0.2*cos(2*pi*train source)]’; X=[]; for k=1:length(train source) Xtemp=[]; for j=1:num centres Xtemp=[Xtemp exp(-norm(train source(k)-c(j))∧ 2)/(2*stand dev∧ 2)]; end X=[X;1 Xtemp]; end W=inv(X’*X)*X’*Y test=0:0.01:1; X=[]; for k=1:length(test) Xtemp=[]; for j=1:num centres Xtemp=[Xtemp exp(-norm(test(k)-c(j))∧ 2)/(2*stand dev∧ 2)]; end X=[X;Xtemp]; end out=W(1)+X*W(2:length(W)); f=figure plot(test,0.3+0.2*cos(2*pi*test),’b’,test,out,’r’) legend(’real function’,’approximation’,4); 4.21 Function rbf2dim(num centres,stand dev,numtrain);

15 % num centres:# of centres % stand dev: Standard deviation % numtrain:# of train samples n=0:1/floor(sqrt((num centres-1))):1; c=[]; for i=1:length(n) ctemp(1,:)=ones(1,length(n))*n(i); ctemp(2,:)=n; c=[c,ctemp]; end x1=rand(1,numtrain); x2=rand(1,numtrain); train source=[]; for i=1:numtrain trtemp(1,:)=ones(1,numtrain)*x1(i); trtemp(2,:)=x2; train source=[train source,trtemp]; end c1=0.2*cos(2*pi*x1); c2=cos(2*pi*x2); Ym=0.3+c1’*c2; Y=reshape(Ym’,1,numtrain∧ 2)’; X=[]; X(:,1)=ones(length(train source),1); for k=1:num centres diaf=ones(2,length(train source)); diaf(1,:)=diaf(1,:)*c(1,k); diaf(2,:)=diaf(2,:)*c(2,k); var1=train source-diaf; var1=exp(-sum(var1.∧ 2)); var1=var1/(2*stand dev∧ 2); X=[X,var1’];

16 end W=inv(X’*X)*X’*Y; test=[]; tst=0:0.05:1; for i=1:length(tst) ttemp(1,:)=ones(1,length(tst))*tst(i); ttemp(2,:)=tst; test=[test,ttemp]; end X=[]; for k=1:num centres diaf=ones(2,length(test)); diaf(1,:)=diaf(1,:)*c(1,k); diaf(2,:)=diaf(2,:)*c(2,k); var1=test-diaf; var1=exp(-sum(var1.∧ 2)); var1=var1/(2*stand dev∧ 2); X=[X,var1’]; end out=W(1)+X*W(2:length(W)); out=reshape(out,length(tst),length(tst)); f=figure c1=0.2*cos(2*pi*tst); c2=cos(2*pi*tst); Ytest=0.3+c1’*c2; subplot(1,2,1) mesh(tst,tst,Ytest) subplot(1,2,2) mesh(tst,tst,out) 4.22 We have that y Ti y j = φ(xi )T φ(xj ) =

k 1 X (cos(rxi ) cos(rxj ) + sin(rxi ) sin(rxj )) + 2 r=1

17 k k X 1 1 X cos r∆ = − + Real exp(jr∆) = − + 2 r=0 2 r=0

)

(

(

1 1 − exp(j(k + 1)∆) = − + Real 2 1 − exp(j∆)

=

)

sin (k + 21 )∆ 2 sin ∆2

where ∆ = xi − xj .

4.23 Let the output from each classifier be Pj (ωi |x) ≡ Pj (ωi ) and let P (ωi |x) ≡ P (ωi ) be the output of the combiner to be estimated. We will estimate P (ωi ) by minimizing the Kulback-Liebler average distance, i.e, M L X P (ωi ) 1X P (ωi ) log L j=1 i=1 Pj (ωi )

(4.10)

subject to the constraint M X

P (ωi ) = 1

(4.11)

i=1

Employing Lagrange multipliers, the above problem becomes equivalent with minimizing the Lagrangian M M L X X P (ωi ) 1X P (ωi ) log − λ( P (ωi ) − 1) L j=1 i=1 Pj (ωi ) i=1

(4.12)

Taking the derivative with respect to P (ωi ) we get L L 1X 1X log P (ωi ) + 1 − log Pj (ωi ) − λ = 0 L j=1 L j=1

(4.13)

and solving the above we have L 1X log Pj (ωi )} P (ωi ) = exp{λ − 1} exp{ L j=1

(4.14)

Substituting the above in the constraint equation it turns out that exp{λ − 1} = PM

1

1 PL j=1 log Pj (ωi )} i=1 exp{ L

(4.15)

18 Hence

exp{ L1 Lj=1 log Pj (ωi )} P (ωi ) = PM 1 PL j=1 log Pj (ωi )} i=1 exp{ L P

or

exp{ L1 log Lj=1 Pj (ωi )} P (ωi ) = PM QL 1 i=1 exp{ L log j=1 Pj (ωi )} Q

and finally

(4.16)

(4.17)

1 L

QL

(Pj (ωi )) P (ωi ) = PM j=1 QL 1 L i=1 j=1 (Pj (ωi ))

(4.18)

4.24 Following the rationale of the previous problem we will now minimize the following form of the Kulback-Liebler cost M M M L X L X X 1X 1X Pj (ωi ) log Pj (ωi ) − Pj (ωi ) log P (ωi ) − λ( P (ωi ) − 1) L j=1 i=1 L j=1 i=1 i=1 (4.19) Taking the derivative with respect to P (ωi ) and equating to zero, it turns out that L 1 X P (ωi ) = − Pj (ωi ) (4.20) λL j=1

Substituting into the constraint equation we obtain λ=−

L M X 1X Pj (ωi ) L i=1 j=1

(4.21)

L 1X Pj (ωi ) L j=1

(4.22)

and finally P (ωi ) =

4.25 The Lagrangian for the problem stated in (??)–(??) is given by L(w, ξ) = C||w||2 +

N X i=1

ξi2 +

N X i=1

λi (yi − hw, φ(xi )i − ξi )

(4.23)

19 Taking the gradient with respect to w, ξ and equating to zero we get w=

N 1 X λi φ(xi ) 2C i=1

(4.24)

and

λi 2 Substituting the previous two into the Lagrangian we obtain ξi =

L(λ) =

(4.25)

1 XX 1X 2 X λi λj hφ(xi ), φ(xj )i + λ + λi yi 4C i j 4 i i i −

= − +

1X 2 1 XX λi − λi λj hφ(xi ), φ(xj )i 2 i 2C i j

1 XX 1X 2 λi λj hφ(xi ), φ(xj )i − λ + 4C i j 4 i i X

λ i yi

i

= −

1 1 T λ Kλ − λT λ + λT y 4C 4

(4.26)

taking the gradient with respect to λ and equating to zero we obtain λ = 2C (K + CI)−1 y

(4.27)

4.26 We have already proved in problem 4.25 that minimizing the Lagrangian results in N 1 X w= λi φ(xi ) (4.28) 2C i=1

and

λi (4.29) 2 Substituting (4.29) in (4.28)and taking into account the definition of ξi from the constraint equations of the ridge regression, we obtain ξi =

w =

N 1X ξi xi C i=1

20 = =

N 1X (yi − xTi w)xi C i=1

N N 1X 1X yi xi − xi xTi w C i=1 C i=1

Recalling the definitions xT1 y1  T    x  2   y2     X =  ..  , y =  ..    .   .  





xTN



yN

we get Cw = X T y − X T Xw and rearranging we obtain the required result.

4.27 Taking into account the linear kernel the solution of the dual problem becomes w = X T (XX T + CI)−1 y (4.30) and for the primal w = (X T X + CI)−1 X T y

(4.31)

Hence we have to show that X T (XX T + CI)−1 = (X T X + CI)−1 X T Let A ≡ X T (XX T + CI)−1 We have A = (X T X + CI)−1 (X T X + CI)X T (XX T + CI)−1 = (X T X + CI)−1 (X T XX T + CX T )(XX T + CI)−1 = (X T X + CI)−1 X T (XX T + CI)(XX T + CI)−1 and finally X T (XX T + CI)−1 = (X T X + CI)−1 X T

(4.32)

21

4.28 The error rate on the training data set is given by N N 1 X 1 X I(1 − yi f (xi )) ≤ exp(−yi F (xi )) N i=1 N i=1

PeN =

which by the definition of the combined classifier is written as N N K X 1 X 1 X αk φ(xi ; θ k ) exp(−yi F (xi )) = exp −yi N i=1 N i=1 k=1

!

=

K N X 1 Y

i=1 N k=1

exp(−yi αk φ(xi ; θ k ))

(4.33)

However (K)

(K+1)

wi

exp(−yi αK φ(xi ; θ K )) ZK (K−1) wi exp(−yi αK−1 φ(xi ; θ K−1 )) exp(−yi αK φ(xi ; θ K )) = ZK−1 ZK QK k=1 exp(−yi αk φ(xi ; θ k )) = (4.34) Q N K k=1 Zk =

wi

However, we know that the weights sum up to one. Thus N X i=1

QK

k=1 exp(−yi αk φ(xi ; θ k )) =1 Q N K k=1 Zk

(4.35)

Combining (4.33) and (4.35) results in PeN ≤

K Y

Zk

(4.36)

k=1

By the respective definition we have

Zk =

N X

(k)

wi exp(−yi αk φ(xi ; θ k ))

i=1

=

X

yi φ(xi ;θ k )<0

(k)

wi exp(αk ) +

X

yi φ(xi ;θ k )>0

(k)

wi exp(−αk )

22 Using the respective formulas from the proof in the book, the above becomes Zk = Pk exp(αk ) + (1 − Pk ) exp(−αk ) Also we know that αk =

(4.37)

1 1 − Pk ln 2 Pk

(4.38)

Combining (4.37) and (4.38) we obtain q

Zk = 2 Pk (1 − Pk )

(4.39)

Hence (4.36) can now be written as PeN ≤

K q Y

2 Pk (1 − Pk ) =

k=1

K q Y

k=1

1 − 4γk2 ≤ exp(−2

K X

γk2 )

k=1

where by definition γk ≡ 1/2 − Pk . Since the base classifiers do better than a random guessing, we can write that γk ≥ γ > 0. Thus the error is bounded by exp(−2Kγ 2 ) and drops exponentially fast with K.

Solutions to the problems of Chapter 5 5.1 We have that z = xT µ =

l X

xi µ i

i=1

Thus z is the weighted sum of l Gaussian and independent variables and therefore, as we know from statistics, z is also a Gaussian variable with mean value equal to E[z] = E[xT ]µ = ||µ||2 =

l X 1 i=1 i

and variance σz2 = E[(xT µ − ||µ||2 )2 ] = E[(xT µ)2 ] − ||µ||4 The first term in the last formula is 

A ≡ E[(xT µ)2 ] = E  =

l X i=1

µ2i E[x2i ] +

l X i=1

XX i

µi xi

!2 

=

l X

µ2i E[x2i ] +

i=1

µ2i µ2j

XX i

µi µj E[xi ]E[xj ]

j6=i

(5.1)

j6=i

where the statistical independence between successive samples has been taken into account. However E[x2i ] = σx2i + (E[xi ])2 = 1 + µ2i Substituting (5.2) into (5.1) we obtain A = ||µ||2 + ||µ||4 1

(5.2)

2 hence σz2 = ||µ||2 =

l X 1 i=1 i

We are now ready to compute the error rate. The probability of an error for class ω1 is equal to the probability of z < 0 (z − E[z])2 q exp − dz Pe,ω1 = 2σz2 −∞ 2πσz2 !

1

Z 0

and normalizing the variable of integration we get Pe,ω1 =

Z ∞ bl

where

z2 1 √ exp − dz 2 2π !

v u

l X E[z] u 1 bl = =t σz i=1 i

and obviously bl −→ ∞ as l −→ ∞, since

Pl

1 i=1 i −→ ∞.

5.2 The ML estimate of the mean value is known to be equal to µ̂ =

N 1 X sk xk N k=1

where sk = 1 if xk ∈ ω1 and sk = −1 if xk ∈ ω2 . We know that this is an unbiased estimate, i.e., E[µ̂] = µ. Decisions are taken according to the sign of z = xT µ̂ Since the data are assumed to be iid, the random vector x is statistically independent from xk , k = 1, 2, . . . , N , hence we have T

T

2

E[z] = E[x] E[µ̂] = µ µ = ||µ|| =

l X 1 i=1 i

(5.3)

For the variance we have σz2 = E[(z − ||µ||2 )2 ] = E[(xT µ̂ − ||µ||2 )2 ] = E[(xT µ̂)2 ] − ||µ||4

(5.4)

3 For the first of the two terms in the right hand side of the previous formula we have T



2

A ≡ E[(x µ̂) ] = E 

l X i=1

  !2  XX xi µ̂i  = E  xi µ̂i xj µ̂j  i

j

Due to the statistical independence of the features, we have A = E

" l X i=1

=

l X

x2i µ̂2i

#

 l X +E

i=1 j=1, j6=i

E[x2i ]E[µ̂i 2 ] +

i=1

=

l X

l X

X

l X



xi xj µ̂i µ̂j 

E[xi ]E[xj ]E[µ̂i ]E[µ̂j ]

i j=1, j6=i

E[x2i ]E[µ̂i 2 ] +

i=1

X

l X

µ2i µ2j

i j=1, j6=i

We now have 2 E[x2i ] = σx,i + (E[xi ])2 = 1 + µ2i

and

1 + µ2i N In the previous equation we used the result that the variance of the sum of independent variables is equal to the sum of the respective variances.

Combining the previous results we obtain E[µ̂i 2 ] = σµ̂2 i + (E[µ̂i ])2 =

l l X XX l 1 X 2 A = µ2i µ2j µ4i + µi + + (1 + ) N N i=1 i=1 i j6=i l 1 X l µ2 + ||µ||4 + (1 + ) N N i=1 i

(5.5) (5.6)

Combining (5.4) and (5.6) we obtain σz2 =

l 1 1 X l + (1 + ) N N i=1 i

Having derived the mean and the variance of the approximately Gaussian variable z, we are ready to compute the error rate. For two equiprobable

4 classes, the probability of an error for a sample originating from class ω1 is equal to the probability of z < 0. This is given by (z − E[z])2 1Z 0 1 q exp − Pe,ω1 = dz 2 −∞ 2πσz2 2σz2 !

(5.7)

and after normalization of the integration variable z we have 1Z ∞ 1 z2 √ exp − dz Pe,ω1 = 2 bl 2 2π !

(5.8)

where bl = E[z] . Due to symmetry, the error for class ω2 is the same, thus σz the total error of the two equiprobable classes is given by 2Pe,ω1 .

For the convergence of bl as l −→ ∞ we have bl = q

Pl

1 i=1 i

P l + (1 + N1 ) li=1 1i N

and neglecting the second of the two positive terms under the square root in the denominator, we have s l NX 1 bl ≤ (5.9) l i=1 l From the Calculus it is known that lim

l→∞

l X 1 i=1 i

!

− ln(l) = Euler’s constant ≡ e

Using the property that the limit of the product of two sequences is equal to the product of the respective limits, we obtain lim

 l  N 12 X 1

l→∞ 

l

N − l i=1 i

21

 

ln(l) = lim

l→∞

N l

12 X l

N 1 = lim l→∞ l i=1 i

12

e=0 (5.10)

√ = 0, we obtain Combining (5.9) and (5.10), and recalling that liml−→∞ ln(l) l

bl −→ 0, as l −→ ∞ and obviously the error tends to 0.5.

5 5.3 From the theory of appendix A we have that the each of the random variables N N 1 X 1 X 2 (x − x̄) and (yi − ȳ)2 i σ 2 i=1 σ 2 i=1 is chi-square with N − 1 degrees of freedom. Hence, their sum is also chisquare with 2N − 2 degrees of freedom.

5.4 Following similar arguments as in theory, for independent Gaussian variables, xi , yi , the variable x̄ − ȳ =

N1 N2 1 X 1 X xi − yi N1 i=1 N2 i=1 2

2

is Gaussian with mean µ1 − µ2 and variance Nσ 1 + Nσ 2 (problem 5.6). Also the 2 −2 2 sz is chi square with N1 + N2 − 2 degrees of freedom. Thus, variable N1 +N σ2 the variable (x̄−ȳ)−(µ1 −µ2 )

q = q

q

σ2 σ2 +N N1 2

N1 +N2 −2 sz σ2

q

1 N1 +N2 −2

=

(x̄ − ȳ) − (µ1 − µ2 ) sz

q

1 + N12 N1

is t-distributed with N1 + N2 − 2 degrees of freedom.

5.5 It is easily verified by the matrix multiplication.

5.6 Let y = Ax Then from the theory of probability we have p(y) =

1 p(x)|x=A−1 y J

y |. where J is the Jacobian of the transformation, defined as | ∂∂x For a linear transformation |

∂y | = ∂x

∂y1 ∂y1 ∂x1 ∂x2 ∂yl ∂yl ∂x1 ∂x2

.. .

1 . . . ∂y ∂xl

∂yl . . . ∂x l

= |A|

6 Let p(x) =

1 exp (− xT Σ−1 x x) 2 (2π) |Σx | 1

l 2

1 2

where for simplicity we assumed E[x] = 0.

Then 1 1 T −T −1 −1 Σx A y) p(A−1 y) = l 1 exp (− y A 2 (2π) 2 |Σx | 2 However

Σy = E[yy T ] = E[AxxT AT ] = AΣx AT −1 Σ−1 = A−T Σ−1 y x A

Also |Σy | = |A||Σx ||AT | = |A|2 |Σx | ⇒ 1 |Σx | = |Σy | |A|2 Thus, p(y) =

1 T −1 1 1 1 exp (− y Σy y) l 2 |A| (2π) 2 |Σy | 2 |A|

and y is also Gaussian.

Obviously, if xi are independent, then Σx is diagonal, Σx = σ 2 I, and Σy = σ 2 AIAT = σ 2 AAT = σ 2 I 5.7 Assume first that the classes do not overlap. In the double sum take those terms with i = 1. Then X j

P (∆j )P (ω1 |∆j ) logM (P (ω1 |∆j ))

Since no overlap exists in the various intervals then P (ω1 |∆j ) is either 0 or 1.

Taking into account that 0 logM 0 = 0, the corresponding sum is zero. The same is true for the other classes. If the classes overlap completely, then P (ωi |∆j ) =

1 ∀i M

7 Thus A =

XX i

P (∆j , ωi ) = 1

j

5.8 The divergence never decreases for independent features. This is a direct consequence from the theory since dij ≥ 0 and dij (x1 , . . . , xl ) =

l X

dij (xr )

r=1

We will now show that it is also true for dependent variables which are jointly normally distributed. From the theory and for equal covariances for the classes we have dij = (µi − µj )T Σ−1 x (µi − µj ) We also know that linear transformation of jointly Gaussian variables results in jointly Gaussian variables. Let y = AT x where A is the unitary matrix diagonalizing the symmetric Σx matrix (Appendix B). Σx = A ∧ AT Then T dij (y) = (µi − µj )AΣ−1 y A (µi − µj )

But Σy = E[yy T ] = AT Σx A ⇒ Σ−1 = AT Σ−1 y x A Thus T dij (y) = (µi − µj )A(AT Σ−1 x A)A (µi − µj ) = dij (x)

Thus, this linear transformation does not change the divergence. However Σy = AT Σx A ⇒ Σy = ∧

8 Thus, the elements of y are uncorrelated and since they are Gaussian they are independent (Appendix A). Combining the above, we have proved our statement.

5.9 The two distributions are 1

pi (x) =

1 exp (− (x − µi )Σ−1 i (x − µi )) 2 (2π) |Σi |

pj (x) =

1 exp (− (x − µj )Σ−1 j (x − µj )) 2 (2π) |Σj |

l 2

1 2

1

l 2

1 2

From which are obtain ln

pi (x) 1 |Σj | 1 T = ln − trace{Σ−1 i (x − µi )(x − µi ) } pj (x) 2 |Σi | 2 T +trace{Σ−1 j (x − µj )(x − µj ) }

Then if Ei [·] is the mean w.r. to class ωi , we have, Ei [ln

or

1 |Σj | 1 pi (x) ] = ln − trace{Σ−1 i Σi } pj (x) 2 |Σi | 2

1 T + Ei [trace{Σ−1 j ((x − µi ) + (µi − µj ))((x − µi ) + (µi − µj )) } 2 Ei [ln

pi (x) 1 |Σj | 1 ] = ln − trace{Σ−1 i Σi } pj (x) 2 |Σi | 2

1 1 −1 T + trace{Σ−1 j Σi } + trace{Σj (µi − µj )(µi − µj ) } 2 2 Similarly, Ej [ln

pi (x) 1 |Σj | 1 ] = ln − trace{Σ−1 i Σj } pj (x) 2 |Σi | 2

1 1 −1 T + trace{Σ−1 j Σj } + trace{Σi (µi − µj )(µi − µj ) } 2 2 and finally dij (x) =

1 −1 trace{(Σi − Σj )(Σ−1 j − Σi )} 2

1 −1 T + trace{(Σ−1 i + Σj )(µi − µj )(µi − µj ) } 2

9 For equal covariance matrices 1 −1 T trace{(Σ−1 i + Σj )(µi − µj )(µi − µj ) } 2

dij (x) = or

dij (x) = (µi − µj )T Σ−1 (µi − µj ) which is equal to the Mahalanobis distance between the means.

Assume now that a new feature, which is jointly Gaussian distributed with the previous ones, is added. Then ′

Σ =

"

Σ r rT σ2

#

, µ1 =

"

µ1 m1

#

, µ2 =

"

µ2 m2

#

and according to the inversion matrix lemma ′−1

Σ

=

"

Σ−1 + dδdT −dδ −δdT δ

where d = Σ−1 r and δ =

1 σ2 − rT d

=

#

1 σ 2 − r T Σ−1 r

Substituting the new values of Σ′ , µ′1 , µ′2 in the divergence and after some algebraic manipulations we obtain d′ij (x) = dij (x) +

[(mi − mj ) − (µi − µj )T Σ−1 r]2 σ 2 − r T Σ−1 r

Obviously, for uncorrelated (independent) features r = 0 and d′ij (x) = dij (x) +

(mi − mj )2 σ2

5.10 We will show it for two variables and the generalization is obvious dij (x1 , x2 ) = =

Z

x1

Z

x2

Z

x1

Z

x2

(pi (x1 , x2 ) − pj (x1 , x2 )) ln

(pi (x1 )pi (x2 ) − pj (x1 )pj (x2 ))(ln

pi (x1 , x2 ) dx1 dx2 pj (x1 , x2 )

pi (x1 ) pi (x2 ) + ln )dx1 dx2 pj (x1 ) pj (x2 )

10

=

Z

x1

=

Z

x1

Z

x2

(pi (x1 )pi (x2 ) − pj (x1 )pj (x2 )) ln

pi (x1 ) dx1 dx2 pj (x1 )

+

Z

x1

Z

x2

(pi (x1 )pi (x2 ) − pj (x1 )pj (x2 )) ln

pi (x2 ) dx1 dx2 pj (x2 )

(pi (x1 ) − pj (x1 )) ln

Z pi (x1 ) pi (x2 ) dx1 + (pi (x2 ) − pj (x2 )) ln dx2 pj (x1 ) pj (x2 ) x2

= dij (x1 ) + dij (x2 ) 5.11 The Chernoff bound is defined by P (ω1 )s P (ω2 )1−s

Z +∞ −∞

p(x|ω1 )s p(x|ω2 )1−s dx

The integral for the Gaussian case becomes I≡

1 s exp(− (x − µ1 )T Σ−1 1 (x − µ1 ))) · 2 −∞ (2π) |Σ1 | 1 1 T −1 1−s dx ( l 1 exp(− (x − µ2 ) Σ2 (x − µ2 ))) 2 (2π) 2 |Σ2 | 2

Z +∞

=

(

1

l 2

1 2

1 k exp(− [s(x − µ1 )T Σ−1 1 (x − µ1 ) + 2 −∞ (1 − s)(x − µ2 )T Σ−1 2 (x − µ2 )])dx

Z +∞

and k = or

1 l 2

s

Z

e 2 J(x) dx

1−s

(2π) |Σ1 | 2 |Σ2 | 2

I≡k

1

We will expand J(x) in a Taylor series around the point for which the gradient becomes zero, i.e, ∇J(x) = 0 ⇒ −1 2sΣ−1 1 (x − µ1 ) + 2(1 − s)Σ2 (x − µ2 ) = 0 ⇒

−1 −1 −1 −1 x̂ = (sΣ−1 1 + (1 − s)Σ2 ) [sΣ1 µ1 + (1 − s)Σ2 µ2 ]

11 Hence, since ∇J(x) |x=x̂ = 0 1 J(x) = J(x̂) + (x − x̂)T ∇2 J(x̂)(x − x̂) 2 and the higher order terms are zero, since the function is quadratic. Now −1 ∇2 J(x) = 2sΣ−1 1 + 2(1 − s)Σ2

Thus,

Z +∞ 1 1 I = k exp(− J(x̂)) exp(− (x − x̂)T · 2 2 −∞ −1 −1 −1 [(sΣ−1 1 + 2(1 − s)Σ2 ) ] (x − x̂)dx

Hence l

I =

(2π) 2 l 2

s 2

(2π) |Σ1 | |Σ2 |

1−s 2

−1 1 |sΣ−1 1 + (1 − s)Σ2 | 2

1 exp(− J(x̂)) 2

However, −1 −1 sΣ−1 = Σ−1 1 + (1 − s)Σ2 1 (sΣ2 + (1 − s)Σ1 )Σ2

Thus, I = {

1 |Σ1 |1−s |Σ2 |s 1 } 2 exp(− J(x̂)) |sΣ2 + (1 − s)Σ1 | 2

(5.11)

For the other term J(x̂), observe that

s(x̂ − µ1 )T Σ−1 1 (x̂ − µ1 ) = −1 −1 −1 −1 T −1 s{(sΣ−1 1 + (1 − s)Σ2 ) [sΣ1 µ1 + (1 − s)Σ2 µ2 ] − µ1 } Σ1 {} = −1 −1 −1 T s{(sΣ−1 1 µ1 + (1 − s)Σ2 µ2 − (sΣ1 + (1 − s)Σ2 )µ1 ] −1 −T ·[sΣ−1 }Σ−1 1 + (1 − s)Σ2 ] 1 {} = −1 T −1 T s(1 − s)2 {(µ2 − µ1 )T Σ−1 2 [Σ1 (sΣ2 + (1 − s)Σ1 ) Σ2 ] }Σ1 {} = T s(1 − s)2 {(µ2 − µ1 )T (sΣ2 + (1 − s)Σ1 )−1 Σ1 }Σ−1 1 {} = −1 s(1 − s)2 (µ2 − µ1 )T (sΣ2 + (1 − s)Σ1 )−1 Σ1 Σ−1 1 Σ1 (sΣ2 + (1 − s)Σ1 ) (µ2 − µ1 )

12 where {·} on the right contains the same terms as {·} on the left.

Similarly the other term is s2 (1 − s)(µ1 − µ2 )T (sΣ2 + (1 − s)Σ1 )−1 Σ2 (sΣ2 + (1 − s)Σ1 )−1 (µ1 − µ2 ) Adding the two terms results in J(x̂) = s(1 − s)(µ1 − µ2 )T (sΣ2 + (1 − s)Σ1 )−1 (µ1 − µ2 ) Combining (5.1) and (5.2) the bound results.

For Σ = Σ1 = Σ2 the bound b(s) becomes b(s) = or

s(1 − s) (µ1 − µ2 )T Σ−1 (µ1 − µ2 ) 2

∂b(s) 1 − 2s = (µ1 − µ2 )T Σ−1 (µ1 − µ2 ) = 0 ⇒ ∂s 2 1 s = 2

5.12 From the respective definitions we have Sw =

M X i=1

Sb =

Pi Ei [(x − µi )(x − µi )T ]

M X i=1

Pi (µi − µ0 )(µi − µ0 )T

Sm = E[(x − µ0 )(x − µ0 )T ] Hence Sw + Sb =

M X i=1

Pi {(µi − µ0 )(µi − µ0 )T +

Ei [(x − µi )(x − µi )T ]} =

M X i=1

Pi {(µi − µ0 )(µi − µ0 )T +

Ei [((x − µ0 ) + (µ0 − µi ))((x − µ0 ) + (µ0 − µi ))T ]}

(5.12)

13 =

M X i=1

Pi {(µi − µ0 )(µi − µ0 )T + 2Ei [(x − µ0 )(µ0 − µi )T ]

+(µ0 − µi )(µ0 − µi )T } + =

M X i=1

M X i=1

Pi Ei [(x − µ0 )(x − µ0 )T ]

Pi Ei [(x − µ0 )(x − µ0 )T ] = E[(x − µ0 )(x − µ0 )T ] = Sw

5.13 It is a direct implication of Schwartz’s inequality.

5.14 For the equiprobable two class problem we have 1 1 1 (µ1 − (µ1 + µ2 ))(µ1 − (µ1 + µ2 ))T 2 2 2 1 1 1 + (µ2 − (µ1 + µ2 ))(µ2 − (µ1 + µ2 ))T 2 2 2

Sb =

=

1 1 (µ1 − µ2 )(µ1 − µ2 )T + (µ2 − µ1 )(µ2 − µ1 )T 4 4

Also Sw =

1 1 E1 [(x − µ1 )(x − µ1 )T ] + E2 [(x − µ2 )(x − µ2 )T ] 2 2 = E[(x − µ)(x − µ)T ] = Σ

Thus, trace{Sw−1 Sb } ∝ trace{Σ−1 (µ1 − µ2 )(µ1 − µ2 )T } = (µ1 − µ2 )T Σ−1 (µ1 − µ2 )

which is equal to the divergence for the specific case of Gaussians.

5.15 Initially we evaluate the feature vector [x1 , . . . , xm ] (1 evaluation).

Then we evaluate m feature vectors of dimension m − 1 (m evaluations).

Then we evaluate m−1 feature vectors of dimension m−2 (m-1 evaluations). .. .

14 Thus, the total number of evaluations is: 1 + (m + (m − 1) + · · · + (l + 1)) = 1 + [(1 + 2 + · · · + m) − (1 + 2 + · · · + l)] = = 1+

m(m + 1) l(l + 1) 1 − = 1 − [m(m + 1) − l(l + 1)] 2 2 2

5.16 From linear algebra we know that wT Sw max T w w w occurs if w is chosen to be an eigenvector of the largest eigenvalue of S (Rayleigh quotient), that is, if w satisfies the equation Sw = λw For the case of our problem (generalized Rayleigh quotient), let y ≡ Sw1/2 w ⇒ w = Sw−1/2 y Then the problem becomes equivalent with maximizing max y

y T Sw−1/2 Sb Sw−1/2 y yT y

where the symmetry of Sw has been taken into account. The above is maximized if y is the eigenvector corresponding to the largest eigenvalue λ, i.e., Sw−1/2 Sb Sw−1/2 y = λy and finally, by replacing y by w, if w is chosen to satisfy Sb w = λSw w or equivalently solving the eigenvalue task Sw−1 Sb w = λw The corresponding maximum value is w T Sb w w T Sw w = λ =λ w T Sw w w T Sw w

15 which justifies the choice of the maximum eigenvalue.

5.17 Recall from the matrix that ∂ ∂ trace{AB} = trace{BA} = B T ∂A ∂A ∂ trace{AT SA} = {S + S T }A ∂A ∂ trace{(AT SA)−1 B} = −SA(AT SA)−1 (B + B T )(AT SA)−1 ∂A Then

∂ trace{(AT S1 A)−1 (AT S1 A)} = ∂A ∂ trace{(AT1 S1 A1 )−1 (AT S2 A)}|A1 =A + ∂A1 ∂ trace{(AT S1 A)−1 (AT2 S2 A2 )}|A2 =A = ∂A2 −2S1 A(AT S1 A)−1 (AT S2 A)(AT S1 A)−1 + 2S2 A(AT S1 A)−1

5.18 We will show it for M = 2 and M = 3 and the result is easily generalized. For M = 2 2 X i=1

Pi (µi − µ0 )(µi − µ0 )T =

P1 ((1 − P1 )µ1 − P2 µ2 )( P2 ((1 − P2 )µ2 − P1 µ1 )( P1 (P2 µ1 − P2 µ2 )( P2 (P1 µ2 − P1 µ1 )(

)T + )T = )T + )T +

Thus, the vectors which enter into the sum are P2 (µ1 − µ2 ) and P1 (µ1 − µ2 ) which are linearly dependent. Also for M = 3, the vectors that enter into the summation are µ1 − µ0 , µ2 − µ0 , µ3 − µ0 or µ1 − µ0 = (1 − P1 )µ1 − P2 µ2 − P3 µ3 = P2 (µ1 − µ2 ) + P3 (µ1 − µ3 )

16 µ2 − µ0 = (1 − P2 )µ2 − P1 µ1 − P3 µ3 = P1 (µ2 − µ1 ) + P3 (µ2 − µ3 )

µ3 − µ0 = (1 − P3 )µ3 − P1 µ1 − P2 µ2 = P1 (µ3 − µ1 ) + P2 (µ3 − µ2 )

From the above it is obvious that

P1 (µ1 − µ0 ) + P2 (µ2 − µ0 ) = −P3 (µ3 − µ0 ) Thus, again two of them produce the third. This is easily generalized to M, that is P1 (µ1 − µ0 ) + . . . + PM −1 (µM −1 − µ0 ) = −PM (µM − µ0 ) Hence

P2

T i=1 Pi (µi − µ0 )(µi − µ0 ) is of rank M − 1

5.19 If fi (x) are the discriminant functions, then we decide ωi if fi (x) is maximum. Equivalently we can define the functions φ1 (x) = f1 (x) − f2 (x) φ2 (x) = f2 (x) − f3 (x)

φ3 (x) = f3 (x) − f4 (x) ......

φM −1 (x) = fM −1 (x) − fM (x)

Then, by adding the above appropriately, we can check the signs and decide on the class. For example if f3 (x) is maximum, then φ3 (x) > 0 φ3 (x) + φ4 (x) > 0 φ3 (x) + φ4 (x) + . . . + φM −1 (x) > 0 and φ2 (x) < 0 φ2 (x) + φ1 (x) < 0 Hence M − 1 functions are sufficient.

5.20 We have Sb =

2 X i=1

Pi (µi − µ0 )(µi − µ0 )T =

17 = P1 (P2 (µ1 − µ2 )P2 (µ1 − µ2 )T ) + P2 (P1 (µ2 − µ1 )P1 (µ2 − µ1 )T ) = P1 P2 (µ1 − µ2 )(µ1 − µ2 )T

Since Sb is of rank 1 Sw−1 Sb has one non zero eigenvalue. The sum of eigenvalues is equal to the trace of the matrix, that is λ = trace{P1 P2 Sw−1 (µ1 − µ2 )(µ1 − µ2 )T } = P1 P2 (µ1 − µ2 )T Sw−1 (µ1 − µ2 ) Then Sw−1 (µ1 − µ2 ) is an eigenvector. Indeed P1 P2 Sw−1 (µ1 − µ2 )(µ1 − µ2 )T (Sw−1 (µ1 − µ2 )) = P1 P2 (µ1 − µ2 )T Sw−1 (µ1 − µ2 )Sw−1 (µ1 − µ2 ) = λSw−1 (µ1 − µ2 )

5.21 Let Ψ be the matrix diagonalizing both matrices. Then ΨT R1 Ψ = I ⇒ R1 = Ψ−T Ψ−1 and ΨT R2 Ψ = D ⇒ R2 = Ψ−T DΨ−1 Thus R1−1 R2 = ΨΨT Ψ−T DΨ−1 = ΨDΨ−1 Let λ1 6= λ2

ΨDψ −1 v 1 = λ1 v 1 ⇒ DΨ−1 v 1 = λ1 Ψ−1 v 1

ΨDΨ−1 v 2 = λ2 v 2 ⇒ DΨ−1 v 2 = λ2 Ψ−1 v 2

Hence Ψ−1 v 1 and Ψ−1 v 2 are eigenvectors of the symmetric matrix D and thus they are orthogonal, i.e., v T2 Ψ−T Ψ−1 v 1 = 0 ⇒ v T2 R1 v 1 = 0

18

5.22 The cost function is given by J =

N X i=1

with

E(i),

(5.13)

kL 1 X (ym (i) − ŷm (i))2 , i = 1, 2, . . . , N E(i) = 2 m=1

(5.14)

ym (i) = f (uLm (i)) = uLm (i)

(5.15)

For linear outputs where

kL−1

uLm (i) =

X

L L wmk ykL−1 (i) + wm0

(5.16)

k=1 L wmk , k = 0, 1, . . . , kL−1 are the threshold and weights of the mth linear output neuron. Thus, k

kL X N X L−1 1X L wL y L−1 (i) + wm0 − ŷm (i))2 J = ( 2 i=1 m=1 k=1 mk k

(5.17)

The optimal J, with respect to the output weights, is achieved for ∂J = 0 L ∂wmk

m = 1, 2, . . . , kL , k = 0, 1, . . . , kL−1

L Let us first consider the derivative w.r. to wm0 , then

∂J = 0⇒ L ∂wm0 L N wm0 =

N X i=1

kL−1

ŷm (i) −

X

k=1

L wmk

N X i=1

ykL−1 (i) ⇒

k

L wm0 =

Setting

N N L−1 X 1 X 1 X L wmk ( ŷm (i) − ykL−1 (i)) N i=1 N i=1 k=1

ȳm =

N 1 X ŷm (i) N i=1

(5.18)

19 L−1 ȳm =

we have

N 1 X y L−1 (i) N i=1 m

(5.19)

kL−1 L wm0

= ȳm −

X

L wmk ȳkL−1

(5.20)

k=1

Observe that ȳm is the average, over time, desired response of the mth output mode and ȳkL−1 is the average, over time, output of the k th mode of the L − 1 layer. Substituting (5.10) in (5.7) we obtain k

k

kL X N X L−1 L−1 X 1X L−1 L L ( wmk yk (i) − wmk ȳkL−1 + ȳm − ŷm (i))2 J = 2 i=1 m=1 k=1 k=1

or

k

kL X N X L−1 1X L wmk ỹkL−1 (i) − ỹm (i))2 ( J = 2 i=1 m=1 k=1

where ỹm (i) = ŷm (i) − ȳm

ỹkL−1 (i) = ykL−1 (i) − ȳkL−1 This is an LS problem for N samples and kL outputs, with desired responses L ỹm , inputs ỹkL−1 and weights wmk . This can be rewritten, according to chapter 3 N 1X J = ||ỹ(i) − W T ỹ L−1 (i)||2 2 i=1 where

ỹ(i) = [ỹ1 (i), . . . , ỹkL (i)]T ỹ L−1 (i) = [ỹ1L−1 (i), . . . , ỹkL−1 (i)]T L−1 W = [wL1 , wL2 , . . . , wLkL ] and L L L wLm = [wm1 , wm2 , . . . , wmk ]T L−1

J = Define

N 1X (ỹ(i) − W T ỹ L−1 (i))T (ỹ(i) − W T ỹ L−1 (i)) 2 i=1

Ỹ T ≡ [ỹ(1), . . . , ỹ(N )]

20 Y T ≡ [ỹ L−1 (1), . . . , ỹ L−1 (N )] Then it is straightforward to see that J =

1 trace{(Ỹ − Y W )(Ỹ − Y W )T } 2

Furthermore, we know that the LS solution is given by W = (Y T Y )−1 Y T Ỹ = Y # Ỹ Thus, at the optimum, w.r. to the weights of the output nodes, we have J = or

1 trace{(Ỹ − Y Y # Ỹ )(Ỹ − Y Y # Ỹ )T } 2

1 trace{Ỹ Ỹ T } − trace{Y Y # Ỹ Ỹ T }+ 2 1 trace{(Y Y # Ỹ )(Y Y # Ỹ )T } 2 1 = trace{Ỹ Ỹ T } − trace{Y Y # Ỹ Ỹ T } 2

J =

1 + trace{(Y (Y T Y )−1 Y T Ỹ )(Y (Y T Y )−1 Y T Ỹ )T } 2 1 = trace{Ỹ Ỹ T } − trace{Y (Y T Y )−1 Y T Ỹ Ỹ T } 2 1 + trace{Y T Ỹ Ỹ T Y (Y T Y )−1 Y T Y (Y T Y )−1 } 2 1 1 = trace{Ỹ Ỹ T } − trace{Y T Ỹ Ỹ T Y (Y T Y )−1 } 2 2 The first term does not depend on the weights. Thus minimization affects only the second term. However T

Y Y = Sm =

N X i=1

(y L−1 (i) − ȳ L−1 )(y L−1 (i) − ȳ L−1 )T

Furthermore Y T Ỹ =

N X i=1

ỹ L−1 (i)ỹ T (i) =

21 N X

= [

ỹ L−1 (i)ỹ1 (i), . . . ,

N X

ỹ L−1 (i)ỹkL (i)]

i=1

i=1

Let us take first N X

ỹ

L−1

(i)ỹ1 (i) =

N X i=1

i=1

N X i=1

ỹ L−1 (i)(ŷ1 (i) − ȳ1 ) =

(y L−1 (i) − ȳ L−1 )(ŷ1 (i) − ȳ1 ) N X

=

i=1

(y L−1 (i) − ȳ L−1 )ŷ1 (i)

However, ŷ1 (i) is 1 if x(i) belongs to class ω1 or zero otherwise. Thus, the above becomes N X i=1

ỹ L−1 (i)ŷ1 (i) = N1 (ȳ L−1 − ȳ L−1 ) 1

where N1 is the number of patterns coming from class ω1 and ȳ L−1 ≡ 1

1 X L−1 y (i) N i∈ω1

Thus Y T Ỹ = [N1 (ȳ L−1 − ȳ L−1 ), . . . , NkL (ȳ L−1 − ȳ L−1 )] 1 1 T

T

Y Ỹ Ỹ Y =

KL X i=1

Ni2 (ȳ L−1 − ȳ L−1 )(ȳ L−1 − ȳ L−1 )T = Sb1 1 1

where Sb1 is a weighted version of Sb 5.23 Let r be the radius of the sphere and c the center. Then we must compute r, c so that minimize subject to

r2 ||xi − c||2 ≤ r2

This is a convex programming problem. The Lagrangian is L = r2 −

X i

λi (r2 − ||xi − c||2 )

22 or L = r2 −

X

λi r 2 +

i

X

+

i

X

λi xTi xi

i

T

λi c c − 2

X

λi xTi c

i

For the Wolfe dual representation we have ∂L =0 ∂c

∂L = 0, ∂r2 or X

λi = 1

c=

X

i

λi xi

i

Substituting the above into L we obtain the equivalent problem max L = λ≥0

X i

λi xTi xi −

subject to

X i

X

λi λj xTi xj

i,j

λi = 1

Solutions to the problems of Chapter 6 6.1 a) We have X = UY V H where U = [u1 , . . . , uN ] V = [v 1 , . . . , v N ] The (r, k) elements of (U Y )rk =

N −1 X

uir Y (i, k)

i=0

Thus, the (r, k) element of (X)rk =

−1 N −1 N X X

∗ Y (i, j)uir vjk

i=0 j=0

X =

−1 N −1 N X X

Y (i, j)ui v H j

i=0 j=0

b) uH 1   .

H  .

Y = U XV =  .   [Xv 1 , . . . , Xv N ] 



uH N

Thus,

Y (i, j) = uH i Xv j =

XX m

1

n

X(m, n)A∗ij (m, n)

2 where Aij = ui v H j 6.2 Let Y = U XV T Then XX

Y (k, l) =

m

=

X

U (k, m)X(m, n)V (l, n)

n

U (k, m)

m

X

V (l, n)X(m, n)

(6.1)

n

Let y Tk be the k th row of Y and xTm the mth row of X. Then from (6.1) we get for l = 1, 2, . . .

X yk = U (k, m)[V xm ] m

=

X m

[U ⊗ V ]km xm

Thus, y = (U ⊗ V )x 6.3 Let x =

N −1 X

xi ei ,

xi =< x, ei > = eTi x

i=0

The projection in the m-dimensional subspace spanned by, say, ei , i = 1, 2, . . . , m, is x̂ =

m−1 X

xi ei

i=0

x − x̂ = or

N −1 X

xi ei

i=m

N −1 X T x2i ] xi xj ei ej ] = E[ E[||x − x̂|| ] = E[ i=m i=m j=m 2

=

N −1 X i=m

−1 N −1 N X X

E[eTi xxT ei ] =

N −1 X i=m

eTi Rei

3 Thus, the problem is now to minimize the above, subject to the constraint eTi ei = 1,

i = m, . . . , N − 1

Hence, using Lagrange multipliers, this is equivalent with minimizing Q =

N −1 X

eTi Rei +

N −1 X j=m

i=m

λ(1 − eTj ej )

or

∂Q = 2(Rei − λi ei ) = 0 ⇒ Rei = λi ei , i = m, . . . , N − 1 ∂ei Hence, ei must be eigenvectors of R. Substituting the above in the cost function we obtain 2

E[||x − x̂|| ] =

N −1 X

λi

i=m

Since λi > 0, then the minimum value is achieved if λm , . . . λN −1 are the smalest eigenvalues.

The sum variance of its components, assuming zero mean values, is m−1 X

E[x2i ] =

i=0

m−1 X

ei E[xxT ]ei =

m−1 X

λi

i=0

i=0

which is obviously maximized for the above choice.

6.4 In this case the error x − x̂ =

N −1 X i=m

or E[||x − x̂||2 ] =

(xi − ci )ei

N −1 X i=m

E[(xi − ci )]2

The gradient of the above w.r. to ci is

2E[(xi − ci )] = 0 ⇒ ci = E[xi ] Hence, E[||x − x̂||2 ] =

N −1 X i=m

E[(xi − E[xi ])2 ] =

4 N −1 X i=m

eTi E(xi − E[xi ])(x − E[x])T ei = N −1 X

eTi Σei

i=m

Then the procedure is similar as in the previous problem.

6.5 Let (XX H )v = λv ⇒ (X H X)(X H v) = λ(X H v) Thus, both matrices share eigenvalues.

6.6 a) From the theory we have

X = U

"

1 2 Λr−1 0 0 0

#

VH

Also X̂ =

k−1 Xq

λ i ui v H i

= U

i=0

"

1

#

VH

...

...

0

...

... ···  

2 0 Λk−1 0 0

and 

0

... ... ...  ...   ··· ... ...  1

  0  X − X̂ = U   ···    0    ··· 

0

. . . λk2 . . . . . . . . . . ..

... ...

1

...

  

 0  

V ... ···    

2 ... 0  . . . . . . . . . λr−1   ... ···  ... ... ... ...  ... ... ... ... ... 0

H

5 or



0

... ... ...  ...   ··· ... ... 

0

. . . λk2 . . . . . . . . . . ..



...

...

... ···  

1

  0  H  ··· U (X − X̂)V =     0    ··· 

0

...

 

0  

...

...

...

... ···  

1

  

2 . . . . . . . . . λr−1 ... 0    ... ···  ... ... ... ...  ... ... ... ... ... 0

r−1 The Frobenious norm of the right hand side is i=k λi . Let us look at the Frobenious norm of the left hand side. Since U and V are unitary, this is equal to the Frobenious norm of X − X̂. Indeed, let in general a matrix A and let a the vector resulting from A by row ordering. Then, the row ordering vector b corresponding to U H AV will be, according to problem 6.2,

P

b = (U H ⊗ V H )a The Frobenious norm of the transformed matrix will be equal to bH b = aH (U H ⊗ V H )H (U H ⊗ V H )a = aH (U ⊗ V )(U H ⊗ V H )a = aH ((U U H ) ⊗ (V V H ))a = aH (I ⊗ I)a = aH a The inner product aH a is nothing else than the Frobenious norm of A. Hence ||X − X̂||F =

r−1 X

λi

i=k

where the properties (A ⊗ B)T = AT ⊗ B T and (A ⊗ B)(C ⊗ D) = (AC) ⊗ (BD) have been used.

6 (b) As we know from the theory, the l × N matrix X of rank r can be written as  √  λ0 v H 0 √   λ1 v H   1   X = [u0 , u1 , . . . , ur−1 ]  ..    . √ H λr−1 v r−1 = Ur [a0 , a1 , . . . , aN −1 ] (6.2) where Ur consists of the first r columns of U and the r-dimensional vectors 1 ai , i = 0, 1, . . . , N − 1, are the columns of the r × N product matrix Λr2 Vr , 1 where Vr consists of the first r columns of V and Λr2 is the diagonal matrix with elements the square roots of the respective r singular values. The above readily leads to xi = Ur ai =

r−1 X

n=0

un ai (n), i = 0, 1, . . . , N − 1

(6.3)

where ai (n), n = 0, 1, . . . , r − 1, denote the elements of the respective vector ai . Due to the orthonormality of the columns ui , i = 0, 1, . . . , r − 1, of Ur x̂i =

k−1 X

n=0

un ai (n), i = 0, 1, . . . , N − 1

(6.4)

is the Euclidean projection of xi on the subspace spanned by the first k columns of Ur .

6.7





1 2  X= 2 1  , 1 3

T

X X =

"

6 7 7 14

The corresponding eigenvalues are λ0 = 18.06,

λ1 = 1.94

and the corresponding eigenvectors are v0 =

"

0.50 0.86

#

,

v1 =

"

0.86 −0.50

#

#

7 The corresponding nonzero eigenvectors of XX T are √ √ u0 = 18.06Xv 0 , u1 = 1.94Xv 1 or

−0.14   u1 = 1.39  1.22  −0.64

2.22   u0 = 4.25  1.86  , 3.08

Then

















−0.14 2.22    X = 4.25  1.86  [0.5, 0.86] + 1.39  1.22   [0.86, −0.5] −0.64 3.08 and the best rank-1 approximation of X is 



2.22  X̂ = 4.25  1.86   [0.5, 0.86] 3.08 



4.72 8.11  =  3.95 6.80   6.54 11.26 6.8 a) Check that W W H = I b) −1 1 NX 2π exp(j (k − l)n) N n=0 N

The above is the sum of the N first terms is a geometric progression with α = exp(j We know that

N −1 X n=0

Now

exp(j

n

α =

(

2π (k − l)) N N

1−αN 1−α

α=1 α 6= 1

2π (k − l)) = 1 l = k + rn N

8 and it is different to 1 otherwise.

On the other hand 2π (exp(j (k − l)))N = 1 N Thus ( −1 1 NX 2π 1, l = k + rn exp(j (k − l)n) = 0, otherwise N n=0 N 6.9 For N = 3, we have 



1 1√ 1√  1  H 1 1 3 3  W3 = √   1 − 2 + √2 j − 2 − √2 j  3 1 − 21 − 23 j − 21 + 23 j

thus the 2-D DFT is given by





1 2 1 H H Y = W  0 1 1  W 2 1 2 6.10 Applying the following MATLAB steps, we calculate the DFT transform of the image4.bmp available at the web site. [x,map]=bmpread(’image4.bmp’); img=ind2gray(x,map); f=fft2(img); f=abs(f); f(1,1)=f(1,2); %In order to disregard the DC component. imshow(f,[]); The figure 6.1 shows the amplitude of the resulting Fourier Tranform.

6.11 We shall show that the columns of C are orthonormal. Let us look at two collumns, except the first one. Then s

s

2 3πk1 cos( ), . . . , N 2N

s

2 (2N − 1)πk1 cos( )] N 2N

s

s

2 3πk2 cos( ), . . . , N 2N

s

2 (2N − 1)πk2 cos( )] N 2N

2 πk1 x k1 = [ cos( ), N 2N 2 πk2 x k2 = [ cos( ), N 2N

9

Figure 6.1: Problem 6.10 Thus xTk1 xk1 =

−1 2 NX π(2n + 1)k2 π(2n + 1)k1 = ) cos( ) cos( N n=0 2N 2N

−1 1 NX π(k1 + k2 )(2n + 1) π(2n + 1)(k1 − k2 ) {cos( ) + cos( )} N n=0 2N 2N N −1 X π(k1 + k2 ) 1 (exp(j Re{ (2n + 1)) = N 2N n=0

+ exp(j =

π(k1 − k2 ) (2n + 1))} 2N

−1 1 π(k1 + k2 ) jπ(k1 + k2 ) NX exp(j Re{exp 2n)+ N 2N 2N n=0 −1 π(k1 − k2 ) jπ(k1 − k2 ) NX exp(j 2n)} exp 2N 2N n=0

Let m = k1 + k2 (or k1 − k2 6= 0). Then exp

−1 jπm NX πm 2n) = exp(j 2N n=0 2N

(6.5)

10 jπm 1 − ejπm exp( ) = 2N 1 − e jπm N

(

0, m even A, m odd πm

πm

πm

πm ej 2 (e−j 2 − ej 2 ) ) πm A = exp(j πm πm 2N ej 2N (e−j 2N − ej 2N ) sin πm j πm 2 2 = e sin πm 2N πm

However for m odd ej 2 = ±j. Thus, ReA = 0. Therefore for k1 6= k2 ( different columns) xTk1 xk2 = 0. If k1 = k2 then from (6.2) we get xTk1 xk1 = 1 Finally xTk1 xk1 = 1. Thus, the proof is complete.

6.12 For N = 3, we have 

CT =   

√1 3 √1 2 √1 6

√1 3

0 q

2 3

√1 3 − √12 √1 6

   

Thus, the 2 − D DCT is given by 



1 2 3   Y = CT  0 1 1  2 1 2 6.13 Applying the following MATLAB steps we calculate the DCT transform of the image4.bmp available at the web site. [x,map]=bmpread(’image4.bmp’); img=ind2gray(x,map); f=dct2(img); f=abs(f); f(1,1)=f(1,2); %In order to disregard the DC component. imshow(f,[]);

11 6.14 It is easy to see that H1T H1 = I. Assume that HnT Hn = I. Then T we will show that Hn+1 Hn+1 = I. Indeed 1 Hn+1 = Hn ⊗ Hn = √ 2 1 T Hn+1 Hn+1 = 2 1 = 2

"

"

"

HnT HnT HnT −HnT

Hn Hn Hn −Hn #"

2HnT Hn 0 0 2HnT Hn

#

"

#

# #

Hn Hn Hn −Hn = I

6.15 For N = 2, we have 1 Hn = √ 2

1 1 1 −1

Thus, for the first part of the image array of problem 6.9, the Hadamard transform is given by " # 1 2 Y = Hn Hn 0 1 6.16 It is easily checked from direct multiplication of the matrices for the various L’s.

6.17 Let y(n) be the subsampled by M version of ŷ(n). Then Y (z) =

+∞ X

y(n)z −n =

n=−∞

Define ŷ1 (n) =

(

+∞ X

ŷ(nM )z −n

n=−∞

ŷ(n) n = mult of M 0 otherwise

Thus, y(n) = ŷ(M n) = ŷ1 (M n) Then Y (z) =

+∞ X

n=−∞

ŷ1 (M n)z −n =

+∞ X

n=−∞

k

ŷ1 (k)z − M

12 Hence 1

Y (z) = Ŷ1 (z M ) However ŷ1 (n) = CM (n)ŷ(n) where CM (n) =

(

ŷ(n) n = mult. of M 0 otherwise

Thus, CM (n) = Hence Ŷ1 (z) =

−1 2π 1 MX e−j M kn M k=0

−1 +∞ X 2π 1 MX ŷ(n)e−j M kn z −n M k=0 n=−∞

−1 +∞ X 1 MX = ŷ(n)(zW k )−n M k=0 n=−∞

W = exp(−j or 1

Y (z) = Ŷ1 (z M ) =

2π ) M

−1 1 1 MX Ŷ (z M W k ) M k=0

If on the other hand y(n) is an upsampled version of ŷ(n) then it is easy to see that Y (z) = Ŷ (z M ) Hence, for the Noble identity of figure 6.5a, if ŷ(n) is the input and y(n) is the output, we have, for the left part, Y (z) =

M −1 X 1 1 Ŷ (z M W k ) H(z) M k=0

and for the right part Y (z) =

−1 1 1 1 MX (Ŷ (z M W k )H((z M W kM ) M ) M k=0

13 −1 1 1 MX Ŷ (z M W k )H(z) M k=0

Thus, the two parts are equivalent.

For the Noble identity of the figure 6.11, note that the output of the left part is Y (z) = Ŷ (z M )H(z M ).

For the right part Y (z) = Ŷ (z M )H(z M ) Hence the two parts equivalent.

6.18 It is a direct application of the Noble identities.

6.19 a) If each one of the two analysis filters is delayed by L, then the new filters will be z −L H0 (z) and z −L H1 (z). However the delay unit can be incorporated into the input signal, and assume that our input signal is x(n − L) (X(z)z −L ). Thus, the only thing that changes is that the output is also delayed. b) This is more interesting. We will show it for the N = 2 bands case. Consider the figure 6.2 From (a) and for a PR band the input to the upsampler of G1 , in the

Figure 6.2: Problem 6.19 lower branch, is delayed by L w.r. to the input of the lower analysis band, y0 (n). Thus, in order to safeguard the PR property we should shift by the same amount the input to the upsampler of G0 . Let us now compute the

14 overall delay. The output of G0 (z) will be (Problem 6.17) Y1 (z) =

Ŷ1 (z) =

1 1 1 1 1X X(z 2 wk )(z 2 wk )−L H0 (z 2 wk ) 2 k=0

1 1 1 1 1X z −L X(z 2 wk )(z 2 wk )−L H0 (z 2 wk ) 2 k=0

X1 (z) =

1 1X z −2L X(z)z −L H0 (z)G0 (z) 2 k=0

= z −3L X(z)H0 (z)G0 (z) That is, the signal is delayed by 3 = 2N − 1 samples w.r. to the delayless case. The same is true (it could not be different) for the lower branch.

The generalization is straightforward, by a recursive application of the above.

6.20 Xs (ω, m) = exp(−jωm)

+∞ X

n=−∞

x(n)θ(n − m) exp(jω(m − n))

Let v(n) = θ(−n) exp(jωn) This is a modulated version of θ(−n), which has the same spectral form as θ(−n) but shifted at the position ω in the frequency domain.

The summation is nothing else than the convolution of x(n) with u(n) computed at sample m. exp(−jωm) is another modulation to bring the spectral center back to zero. Thus, for different ω’s Xs (ω, m) is the output of x(n) with the corresponding v(n). The operation is demonstrated in the figure 6.3 6.21 We will demonstrate this property for the subsampling and upsam-

Figure 6.3: Problem 6.20

15 pling by 2 cases. a) Subsampling. We can think of process as in figure 6.4 S(2) is 1 for even samples and 0 for odd ones.

Figure 6.4: Problem 6.21 b) Consider the figure 6.5

Figure 6.5: Problem 6.21b It is easy to see that for even time instant of the output the even samples of h(n) are involved and for odd instants the odd ones.

6.22 From problem (6.23) we have 1 H(u, v) = exp(− σ 2 {(u′ − ωx′ )2 λ2 + (v ′ + ωy′ )2 }) 2 However for orientation angle θ equal to tan−1 ωωxy , we have ω =

q

ωx2 + ωy2

ωx = ω cos θ ωy = ω sin θ and ωx′ = ωx cos θ + ωy sin θ = ω ωy′ = −ωx sin θ + ωy cos θ = 0 Thus, 6.3 becomes 1 H(u, v) = exp(− σ 2 {(u′ − ω)2 λ2 + v ′2 }) 2

(6.6)

16

Figure 6.6: Problem 6.22 See figure 6.6 for the geometric interpretation of the above.

The frequency bandwidth in octave is computed by setting H(u, v) =

1 2

and v ′ = 0. Hence 1 1 exp(− σ 2 {(u′ − ω)2 λ2 }) = ⇒ 2 2 1 − σ 2 (u′ − ω)2 λ2 = − ln 2 2 or u′2 − 2u′ ω + (ω 2 − or u′1,2 = ω ± Thus,

2 ln 2 ) = 0 σ 2 λ2

1 √ 2 ln 2 σλ

√ ωλσ + 2 ln 2 √ Bf = log2 ωλσ − 2 ln 2

The orientation half peak bandwidth Bθ is computed as follows.

Set u′ = ω, and 1 2 ′2 1 e− 2 σ v = 2

17 or ′ v1,2 = ±

Hence

1√ 2 ln 2 σ

v1 Bθ = 2 ω √ 2 ln 2 −1 Bθ = 2 tan σω tan

or

6.23 From the basic Fourier transform pairs, we knoe that x2 σ2 ω2 1 √ e− 2σ2 ↔ e− 2 σ 2π

For the two dimensional case of g(x, y) =

1 − x2 +y2 2 e 2σ σ 2 2π

this is easily generalized to σ2 1 − x2 +y2 2 2 2 e 2σ ↔ e− 2 (u +v ) 2 σ 2π

The function which is of interest to us is 1 1 1 exp{− 2 ( 2 (x cos θ + y sin θ)2 + 2 2πλσ 2σ λ

g ′ (x, y) =

+ (−x sin θ + y cos θ)2 )} Let x = [x, y]T . Then g(x, y) ≡ g(x) =

1 xT x exp(− ) 2πσ 2 2σ 2

g ′ (x, y) ≡ g ′ (x) =

1 1 g(Ax) ≡ g(y) λ λ

and where A =

"

1 λ

0 0 1

#"

cos θ sin θ − sin θ cos θ

#

18 and y = Ax ⇒ x = A−1 y

From the Fourier integral we have

Z

G′ (u, v) =

1 T g(Ax)e−j x ω dx λ

where ω T = [u, v] or G′ (u, v) = where

1Z 1 T −T g(y)e−j y A ω dy J λ

J = |

∂y 1 | = |A| = ∂x λ

Therefore G′ (u, v) = G(A−T ω) = exp(− since

σ2

G(ω) ≡ G(u, v) = e− 2 (u +v ) = e− 2 ω ω σ

However

−σ 2 T −1 −T (ω A A ω)) 2

−1

A

A

#

cos θ sin θ − sin θ cos θ

#

=

cos θ − sin θ sin θ cos θ

=

"

λ 0 0 1

#"

or

2

λ 0 0 1

"

and −T

2

#"

T

ω T A−1 A−T ω = (λ2 u′2 + v ′2 ) where u′ = u cos θ + v sin θ v ′ = −u sin θ + v cos θ

Thus

G′ (u, v) = exp(−

σ 2 2 ′2 (λ u + v ′2 )) 2

from which we obtain H ′ (v, v) = exp(−

σ2 2 ′ (λ (u − ωx′ )2 + (v ′ − ωy′ )2 ) 2

Solutions to the problems of Chapter 7 7.1 The function cooccur(I) in matlab takes as input the image array I and gives as output the four directions P0, P45, P90, P135 for d = 1, as well as the statistical features ASM, CON, IDF, Hxy according to the corresponding formulas. function [P0, P90, P45, P135, ASM, CON, IDF, Hxy] = cooccur(I) % This function computes the % co-occurance matrix of I for % d=1 and the four directions. [ Ny , Nx ] = size(I); % max colour Ng = max(max(I)); % compute P0 for d=1 R = 2*(Nx-1)*Ny; P0 = zeros(Ng+1,Ng+1); for i=0:Ng, for j=0:Ng, s=0; for k=1:Ny, for l=1:Nx-1, if (I(k,l)==i & I(k,l+1)==j) s = s + 1; end; if (I(k,l)==j & I(k,l+1)==i) 1

2 s = s + 1; end; end; end; P0(i+1,j+1) = s; end; end; P0 = P0/R; % compute p90 for d=1; R = 2*(Ny-1)*Nx; P90 = zeros(Ng+1,Ng+1); for i=0:Ng, for j=0:Ng, s=0; for l=1:Nx, for k=1:Ny-1, if (I(k,l)==i & I(k+1,l)==j) s = s + 1; end; if (I(k,l)==j & I(k+1,l)==i) s = s + 1; end; end; end; P90(i+1,j+1) = s; end; end; P90 = P90/R; % compute p135 for d=1; R = 2*(Nx-1)*(Ny-1); P135 = zeros(Ng+1,Ng+1); for i=0:Ng, for j=0:Ng, s=0; for k=2:Ny, for l=2:Nx, if (I(k,l)==i & I(k-1,l-1)==j) s = s + 1; end; if (I(k,l)==j & I(k-1,l-1)==i) s = s + 1; end; end; end; P135(i+1,j+1) = s;

3 end; end; P135 = P135/R; % compute p45 for d=1; R = 2*(Nx-1)*(Ny-1); P45 = zeros(Ng+1,Ng+1); for i=0:Ng, for j=0:Ng, s=0; for k=2:Ny, for l=1:Nx-1, if (I(k,l)==i & I(k-1,l+1)==j) s = s + 1; end; if (I(k,l)==j & I(k-1,l+1)==i) s = s + 1; end; end; end; P45(i+1,j+1) = s; end; end; P45 = P45/R; % Compute ASM (Angular second moment) ASM = (sum(sum(P0.*P0))+ sum(sum(P45.*P45))+ sum(sum(P90.*P90))+ sum(sum(P135.*P135)))/4; % Compute CON (contrast) CON = (cont(P0)+cont(P90)+ cont(P45)+cont(P135))/4; % Compute IDF (Inverse difference moment) IDF =(idfm(P0)+idfm(P90)+idfm(P45)+ idfm(P135))/4; % Compute Hxy (Entropy) Hxy = (hxye(P0)+hxye(P90)+hxye(P45)+ hxye(P135))/4;

4 function c=cont(P) % computes the contrast CON for the % co-occurance matrix P [Ng, Ng]= size(P); s=0; for n=0:Ng-1, s1=0; for i=1:Ng, for j=1:Ng, if abs(i-j)==n s1=s1+P(i,j); end; end; end; s = s + (n∧ 2)*s1; end c=s; function c=idfm(P) % computes the IDF for the % co-occurance matrix P [Ng,Ng]=size(P); c=0; for i=1:Ng, for j=1:Ng, c=c+P(i,j)/(1+(i-j)∧ 2); end; end; function c=Hxye(P) [Ng,Ng]=size(P); c=0; for i=1:Ng, for j=1:Ng, a = P(i,j); if a==0 c = c - a; else

5 b=log(a)/log(2); c = c - a*b; end; end; end; The four arrays are: 2  2  A00 = (1/40) ∗   3 2

2 2 5 0

3 5 6 3

2 0    3  0

2  1  A450 = (1/32) ∗   4 0

1 2 2 1

4 2 4 4

3 0 3 3

4 3 6 2

0 1    4  0

0  3  A1350 = (1/32) ∗   4 0

3 0 3 0

4 3 4 4





0  3  A900 = (1/40) ∗   4 2 



ASM = 0.0971





2 3    2  0 

0 0    4  0 

COM = 2.0031 7.2 The next function is used to compute the run length for the four directions as well as the SRE, LRE function[RL0, RL90, RL45, RL135, SRE, LRE, GLNU, RLN] = qrl(I) % This function computes the Run-length % matrix of I for the four directions.

RL0 = qrl0(I);

6 RL90 = qrl0(I’); RL45 = qrl45(I); RL135 = qrl45(rot90(I)); function RL0 = qrl0(I) % This program computes the Run-length matrix of I for % the 0 direction.

[N,N] = size(I); % max colour Ng = max(max(I)); % I is into the center of I1 I1=-1*ones(N+2,N+2); I1(2:N+1,2:N+1)=I; % compute RL0 RL0 = zeros(Ng+1,N); for clr=0:Ng, % colours for rl=1:N, for i=2:N+1, for j=2:N-rl+2, if((I1(i,j:j+rl-1)==clr*ones(1,rl)) & (I1(i,j-1) =clr & I1(i,j+rl) =clr))n RL0(clr+1,rl)=RL0(clr+1,rl)+1; end end; end; end; end; function [RL45] = qrl45(I) % This program computes the Run-length matrix of I for % the four directions.

[N,N1] = size(I); % max colour Ng = max(max(I));

7 % I is into the center of I1 % rotate I to 45 degrees I1=-1*ones(2*N+1,2*N+1); for k=1:2*N-1, if k>N i=N; j=k-N+1; else i=k; j=1; end; jj=1; while (i>=1 & j<=N) I1(k+1,jj+1)=I(i,j); i=i-1; j=j+1; jj=jj+1; end end N=2*N-1; RL45 = zeros(Ng+1,N); for clr=0:Ng, % colours for rl=1:N, % run lengths % scan the image for i=2:N+1, for j=2:N-rl+2, if((I1(i,j:j+rl-1)==clr*ones(1,rl)) & (I1(i,j-1) =clr & I1(i,j+rl) =clr)) RL45(clr+1,rl)=RL45(clr+1,rl)+1; end end; end; end; end; RL45=RL45(1:Ng+1,1:N1); function c=sre(RL) % computes the Short Run Emphasis for a QRL [Ng,Nr]=size(RL); a=0; b=0; for i=1:Ng, for j=1:Nr, a = a + RL(i,j)/j∧ 2; b = b + RL(i,j); end; end; c=a/b;

8

function c=lre(RL) % computes the Long Run Emphasis for a QRL [Ng,Nr]=size(RL); a=0; b=0; for i=1:Ng, for j=1:Nr, a = a + RL(i,j)*j∧ 2; b = b + RL(i,j); end; end; c=a/b; function c=gnlu(RL) % computes the Gray Level nonuniformity [Ng,Nr]=size(RL); b=0; s=0; for i=1:Ng, a=0; for j=1:Nr, a = a + RL(i,j); b = b + RL(i,j); end; s = s + a∧ 2; end; c=s/b;

function c=rln(RL) % computes the Run Length Nonuniformity [Ng,Nr]=size(RL); s=0; for j=1:Nr, a=0;

9 for i=1:Ng, a = a + RL(i,j); end; s = s + a∧ 2; end; b=0; for i=1:Ng, for j=1:Nr, b = b + RL(i,j); end; end; c=s/b; SRE = (sre(RL0)+sre(RL90)+sre(RL45)+ sre(RL135))/4; LRE = (lre(RL0)+lre(RL90)+lre(RL45)+ lre(RL135))/4; GLNU = (glnu(RL0)+glnu(RL90)+glnu(RL45)+ glnu(RL135))/4; RLN = (rln(RL0)+rln(RL90)+rln(RL45)+ rln(RL135))/4; Using the above functions the following results are obtained for the matrix of problem 7.1: 4  4  RL0 =   4 4

1 1 1 0

0 0 1 0

0 0 0 0

0 0    0  0

6  6  RL90 =   5 4

0 0 0 0

0 0 0 0

0 0 1 0

0 0    0  0

4  4  RL45 =   5 4

1 1 2 0

0 0 0 0

0 0 0 0

0 0    0  0













10 6  6 RL135 =    5 4 

0 0 2 0

0 0 0 0

0 0 0 0

SRE = 0.8981

0 0    0  0 

LRE = 1.5910 GLN U = 5.5423 7.3 Such an array I can be of the form: a1 0 a2 0  0 a2 0 a3     I =   a2 0 a3 0  0 a3 0 a4 



where a1, a2, a3, a4 are real numbers. Then: CON (450 ) = 0 and CON (00 ) = value where value depends on the values of a1, a2, a3, a4. As we increase the values a1, a2, a3 and a4, we also increase the CON (00 ).

7.4 Having the functions from the above problems, we can solve this problem applying the following MATLAB steps.

A=Bmpread(’image1.bmp’); B=Bmpread(’image2.bmp’); [P0,P90,P45,P135,ASM1,CON1,IDF1,Hxy1]=coocur(A); [RL0,RL90,RL45,RL135,SRE1,LRE1,GLNU1,RLN1]=qrl(A); [P0,P90,P45,P135,ASM1,CON1,IDF1,Hxy1]=coocur(B); [RL0,RL90,RL45,RL135,SRE1,LRE1,GLNU1,RLN1]=qrl(B);

11 2 7.5 Minimize J = N i=1 Pi with the constraint lent with minimizing

P

Q =

N X

Pi2 − λ

N X

PN

i=1 Pi = 1.

This is equiva-

Pi ⇒

i=1

i=1

∂Q λ = 2Pi − λ = 0 ⇒ Pi = + ∂Pi 2 Hence

N X λ i=1 2

or

= 1⇒λ =

Pi =

2 N

1 N

7.6 a) By the definition µpq =

Z Z

I(x, y)(x − x̄)q (y − ȳ)q dxdy

with

xI(x, y)dxdy I(x, y)dxdy

x̄ =

RR

ȳ =

RR

yI(x, y)dxdy I(x, y)dxdy

For the translation we have x′ = x + a y′ = y + β Thus, µpq =

Z Z

However x̄ = and similarly

I(x′ − a, y ′ − β)(x′ − a − x̄)p (y ′ − β − ȳ)q dx′ dy ′ RR

(x′ − a)I(x′ − a, y ′ − β)dx′ dy ′ RR = x̄′ − a I(x′ − a, y ′ − β)dx′ dy ′ ȳ = ȳ ′ − β

12 Combining the above we obtain µpq =

Z Z

I(x′ − a, y ′ − β)(x′ − x̄′ )p (y ′ − ȳ ′ )q dx′ dy ′ ≡ µ′pq

b) For scaling x′ = ax y ′ = ay Thus

′ 1 Z Z x′ y ′ x′ p y I( , )( − x̄) ( − ȳ)q dx′ dy ′ a2 a a a a x′ y ′ 1 Z Z I( , )(x′ − ax̄)p (y ′ − aȳ)q dx′ dy ′ = p+q+2 a a a

µpq =

Also

x̄ =

′ ′ 1 R R x′ I( xa , ya )dx′ dy ′ a2 a ′ ′ 1 I( xa , ya )dx′ dy ′ a2

Similarly ȳ =

=

x̄′ a

ȳ ′ a

Combining, we obtain µpq =

1 ap+q+2

Z Z

µ′pq x′ y ′ ′ ′ p ′ ′ q ′ ′ I( , )(x − x̄ ) (y − ȳ ) dx dy = p+q+2 a a a

Therefore this is non invariant. However, we have that µ00 =

µ′00 a2

Thus, µ′

pq µpq p+q+2 ηpq ≡ γ = ′ a γ 1 γ = µ00 (µ00 ) ( a2 )

=

1 µ′pq ap+q+2 ′ = ηpq 1 (µ′00 )γ ap+q+2

thus, ηpq is invariant under scaling. Its invariance to translation is a consequence of invariance of the central moments.

13 7.7 The proof is mathematical and beyond the scope of this text. The interested reader may consult M.R. Teague: ’Image Analysis via the General Theory of Moments’,J. of official Society of America 70,no.8, pp. 920-930 7.8 From the definition of the Zernike moments we have p+1Z Z Apq = I(x, y)Rpq (p) exp(jqφ)dxdy π and in polar coordinates p+1Z Z I(r, φ)Rpq (p) exp(jqφ)rdrdφ Apq = π Rotating the coordinate system, i.e, φ′ = φ + θ we get A′pq = exp(−jqθ)Apq 7.9 The function for the computation of the Hu moments in Matlab is given below: function F = Hu(I) % computes the invariant Hu’s moments % for the I image array % F is the invariant moments from 1 to 7 [N1,N2]=size(I); mi=zeros(4,4); n =zeros(4,4); m00=0; m01=0; m10=0; for x=1:N1, for y=1;N2, m00=m00+I(x,y); m10=m10+x*I(x,y); m01=m01+y*I(x,y); end; end; xi=m10/m00;

14 yi=m01/m00; % compute mi, n invariant moments as=0.5; for p=1:4, for q=1:4, sum=0; for x=1:N1, for y=1:N2, sum=sum+((x-xi)∧ p)*((y-yi)∧ q)*I(x,y); end; end; g=1+(p+q)/2; mi(p,q)=sum; % mi moments n(p,q) =mi(p,q)/(m00∧ g); % n moments end; end; % moment invariant F(1) = n(3,1)+n(1,3); F(2) = (n(3,1)-n(1,3))∧ 2+4*(n(2,2)∧ 2); F(3) = (n(4,1)-3*n(2,3))∧ 2+(3*n(3,2)-n(1,4))∧ 2; F(4) = (n(4,1)+n(2,3))∧ 2+(n(3,2)+n(1,4))∧ 2; f5 1 = ((n(4,1)-3*n(2,3))*(n(4,1)+n(2,3))); f5 2 = ((n(4,1)+n(2,3)∧ 2)-3*(n(3,2)+n(1,4)∧ 2)); f5 3 = ((3*n(3,2)-n(1,4))*(n(3,2)+n(1,4))); f5 4 = (3*(n(4,1)+n(2,3)∧ 2)-(n(3,2)+n(1,4)∧ 2)); F(5) = f5 1*f5 2+f5 3*f5 4; f6 1=(n(3,1)-n(1,3))*((n(4,1)+n(2,3)∧ 2)-(n(3,2)+n(1,4)∧ 2)); f6 2 = 4*n(2,2)*(n(4,1)+n(2,3))*(n(3,2)+n(1,4)); F(6) = f6 1+f6 2; f7 1 = (3*n(3,2)-n(1,4))*(n(4,1)+n(2,3)); f7 2 = ((n(4,1)+n(2,3)∧ 2)-3*(n(3,2)+n(1,4)∧ 2)); f7 3 = (n(4,1)-3*n(2,3))*(n(3,2)+n(1,4)); f7 4 = (3*(n(4,1)+n(2,3)∧ 2)-(n(3,2)+n(1,4)∧ 2)); F(7) = f7 1*f7 2-f7 3*f7 4; Applying now the following Matlab commands, A=Bmpread(’image1.bmp’); B=Bmpread(’image2.bmp’);

15 F1=Hu(A); F2=Hu(B); we obtain the following.

F1 = F2 =

h

h

0.1313 9.9234 6.6179 1.5475 −3.5715 0.6321 0.8208

i

1.3951 42.7613 40.6064 10.3904 −64.2818 40.2243 124.2421

i

7.10 We use the following program to compute the Zernike moments of any order. function A=zernike(I , order, repet); % This program computes the Zernike moments of ”order”, for the % image I l=zeros(size(I)); N=length(I); n2=floor(N/2); A=0; n=order; m=repet; for x=1:N, for y=1:N, if (((x-n2)/n2)∧ 2+((y-n2)/n2)∧ 2<=1) & (I(x,y) =0) r=((x-n2)/n2)∧ 2+((y-n2)/n2)∧ 2; Rnm=0; for s=0:(n-m)/2 Rnm=Rnm+(-1)∧ s*fact(n-s)*r∧ ((n/2)s)/(fact(s)*fact(((n+m)/2)-s)*fact(((n-m)/2)-s)); end; Vnm=Rnm*exp(i*m*ancgle((x-n2)/n2,(y-n2)/n2)); A=A+I(x,y)*imag(Vnm); end; end; end; A=((n+1)/pi)*abs(A);

16 function u=angle(x,y); % This program computes in rand mode the angle between % the x-axis and the vector that passes through the point (x,y) if x==0, u=pi/2; if y<0, u=u+pi; end; elseif y==0, u=0; if x<0, u=pi; end; else if (x>0) & (y<0) u=atan(abs(x/y)); u=u+(3*pi)/2; end; if (x<0) & (y>0) u=atan(abs(x/y)); u=u+pi/2; end; if (x<0) & (y<0) u=atan(abs(y/x)); u=u+pi; end; if (x>0) & (y>0) u=atan(y/x); end; end; function f=fact(x); fa=1; for i=1:x fa=fa*i;

17 end if x==0 f=1; else f=fa; end The zernike moments are now obtained if we apply the following commands A=Bmpread(’image1.bmp’); B=Bmpread(’image2.bmp’); A11 = zernike(A, 1, 1); A20 = zernike(A, 2, 0); A02 = zernike(A, 2, 2); B11 = zernike(B, 1, 1); B20 = zernike(B, 2, 0); B02 = zernike(B, 2, 2); 7.11 The mean square error is given by σn2 = E[(I(m, n) −

XX k

a(k, l)I(m − k, n − l))2 ]

l

= E[I(m, n)η(m, n)] − E[η(m, n)

XX k

a(k, l)I(m − r, n − l)]

l

However the second term is zero for the optimal choise of weights, from (7.35).

Hence σn2 = E[(I(m, n))2 ] − E[I(m, n)

XX k

a(k, l)I(m − k, n − l)]

l

or σn2 = r(0, 0) −

XX k

7.12 function A = AR; % AR model for k=0:2, for l=0:2, r(k+1,l+1) = exp(-sqrt(k*k+l*l));

l

a(k, l)r(k, l)

18 end; end; R0 = [ r(1,1) r(1,2) r(1,3); r(1,2) r(1,1) r(1,2);r(1,3) r(1,2) r(1,1)] R1 = [ r(2,1) r(2,2) r(2,3); r(2,2) r(2,1) r(2,2);r(2,3) r(2,2) r(2,1)] R3 = [ r(3,1) r(3,2) r(3,3); r(3,2) r(3,1) r(3,2);r(3,3) r(3,2) r(3,1)] % Solution of the system s=-R(1:9,5); R(1:9,5)=[0 0 0 0 -1 0 0 0 0]’; A = inv(R) * s; The following results are obtained.

A= a(1,1) = -0.0358 a(1,0) = -0.2059 a(1,-1) = -0.0358 a(0,1) = -0.2059 σ 2 = 0.6622 a(0,-1) = -0.2059 a(-1,1) = -0.0358 a(-1,0) = -0.2059 a(-1,-1) = -0.0358 a(0,0) = 1 7.13 Rotation by θ results in (figure 7.1)

Figure 7.1: Problem 7.13

19 x′ = l cos (θ + θ1 ) = l cos θ cos θ1 − l sin θ sin θ1 x′ = x cos θ − y sin θ and similarly y ′ = x sin θ − y cos θ Hence z ′ = x′ + jy ′ = (x cos θ − y sin θ) + j(x sin θ + y cos θ) = (x + jy)(cos θ + j sin θ) = zejθ 7.14 The geometry is shown in the figure 7.2 From the Fourier series expansion theory we have

Figure 7.2: Problem 7.14

αn =

2πnt 2ZT dt x(t) cos T 0 T

m Z ti 2X 2πnt = x(t) cos dt T i=1 ti−1 T

m m Z ti X ∆xi 2πnt 2X [xi−1 + Ii dt ≡ (t − ti−1 )] cos = T i=1 ti−1 ∆ti T i

2 Z ti ∆xi 2πnt Ii = [xi−1 − dt ti−1 ] cos T ti−1 ∆ti T

20 +

2 Z ti ∆xi 2πnt dt t cos T ti−1 ∆ti T

The first integral is ∆xi 2πnti 2πnti−1 2 T [xi−1 − − sin ] ti−1 ][sin T 2πn ∆ti T T The second integral becomes 2πnti−1 2πnti 2 T ∆xi [ti−1 sin − ti sin ]+ T 2πn ∆ti T T 1 ∆xi Z ti 2πn sin tdt] [ πn ∆ti ti−1 T

or Ii =

1 2πnti−1 1 ∆xi ∆xi 2πnti (xi−1 ) sin − (xi−1 − ti−1 + ti ) sin πn T πn ∆ti ∆ti T

2πn 2πn T ∆xi ti − cos ti−1 ) (cos 2 2 2π n ∆ti T T T 2πnti−1 1 2πn 1 ∆xi = xi−1 sin − xi sin ti + 2 2 (cos φi − cos φi−1 ) πn T πn T 2π n ∆ti +

or

m X i=1

Ii =

m T X 2πnt0 T ∆xi x0 sin − [cos φi − cos φi−1 ] + 2 2 2π n i=1 ∆ti πn T

−

2πntm T xm sin mn T

However t0 = 0, tm = T , thus, αn =

m T X ∆xi [cos φi − cos φi−1 ] 2π 2 n2 i=1 ∆ti

The rest are shown similarly.

7.15 Rotation by θ, retains the lenghts, but the coordinates change as x′ (t) = x(t) cos θ + y(t) sin θ y ′ (t) = −x(t) sin θ + y(t) cos θ

21 Thus, αn′

2 ZT 2 ZT 2πnt 2πnt dt] cos θ + [ dt] sin θ = [ x(t) cos y(t) cos T 0 T T 0 T

b′n

2 ZT 2 ZT 2πnt 2πnt = [ dt] cos θ + [ dt] sin θ x(t) sin y(t) sin T 0 T T 0 T

2 ZT 2πnt 2πnt 2 ZT dt] sin θ + [ dt] cos θ x(t) cos y(t) cos c′n = − [ T 0 T T 0 T 2 ZT 2 ZT 2πnt 2πnt = − [ dt] sin θ + [ dt] cos θ x(t) sin y(t) sin T 0 T T 0 T which easily results in the invariance of d′n

′2 ′2 In = In′ = αn2 + b2n + c2n + d2n = αn′2 + b′2 n + cn + dn

The rest result in a similar way.

- Observe that we can write "

αn′ c′n

#

"

b′n d′n

#

=

"

cos θ sin θ − sin θ cos θ

#"

αn cn

#

=

"

cos θ sin θ − sin θ cos θ

#"

bn dn

#

"

d′n −b′n

#

Then Jn′ ≡ αn′ d′n − c′n d′n = [αn′ Jn′

= [an cn ]

"

= [αn cn ]

cos θ − sin θ sin θ cos θ "

1 0 0 −1

#"

#" #

dn bn

c′n ]

cos θ − sin θ − sin θ − cos θ

dn bn

= an dn − cn bn = Jn

The third invariance is similarly shown.

7.16 The result of a rotation by θ and a shift by r is ′ α1±n = e−jθ exp(j

#"

2π (i ± n)r)α1±n T

#

22 Hence

2π

2π

n m α1−n e−jθm ej T mr e−jθn ej T nr α1+m = 2π α1m+n e−jθ(m+n) ej T (m+n)r m αn α1−n = 1+m = dmn α1m+n

′n ′m α1+m α1−n ′ dmn = a′m+n 1

bn is a special case of the above result.

7.17 The solution follows the rationale of 7.14. Note that m Z ti 2π 2X z(t)exp(−j nt)dt αn = T i=1 ti−1 T

z(t) = x(t) + jy(t) = x(ti−1 ) + jy(ti−1 ) + = Vi−1 +

∆xi + ∆yi (t − ti−1 ) ∆ti

Vi − Vi−1 (t − ti−1 ) |Vi − Vi−1 |

= Vi−1 + bi−1 (t − ti−1 ) The final result comes from the integration by parts as in 7.14 7.18 XX ∂I(θ) = − 2[(i − x̄) cos θ − (j − ȳ) sin θ][(j − x̄) sin θ − (j − ȳ) cos θ] ∂θ i j

= −2

XX i

{(i − x̄)2 cos θ sin θ − (j − ȳ)2 sin θ cos θ+

j

+(i − x̄)(i − ȳ)(cos2 θ − sin2 θ)} = 0 XX 1 ∂I(θ) = ( {(i − x̄)2 − (j − ȳ)2 } sin 2θ ∂θ i j 2

+(i − x̄)(i − ȳ) cos 2θ) = 0 or tan 2θ = or θ=

2µ11 µ20 − µ02

2µ11 1 tan−1 2 µ20 − µ02

23

7.19 Recall that if x(t) denotes the graph of a FBm process with 0 < H < 1, then the function 1 y(t) = H x(mt) m for m > 0 has the same probability distribution as x(t). Thus, it has the same spectral density. If we assume that the processes are defined in time from 0 to some length T, then we may write Y (f ) =

Z T

y(t)e−2πif t dt

0

as the Fourier transform of y(t). Thus, Y (f ) =

f 1 Z mT 1 x(v)e−2πi m v dv H m 0 m

where we set v = mt. Then Y (f ) =

1 mH+1

X(

f ) m

We know that the spectral density is given by S(f ) = Thus, Sy (f ) =

1 |Y (f )|2 T

1

1

m2H+1 mT

|X(

f 2 )| m

As mT → ∞, we conclude that Sy (f ) =

1

m

S ( 2H+1 x

f ) m

Remember now that y is a properly rescaled version of x. Thus, the spectral densities of x and y must coinside. Therefore, Sy (f ) = Sx (f ) and Sx (f ) =

1 m2H+1

Sx (

f ) m

The above is valid for any m>0, thus, it is also true for m = f . Hence, Sx (f ) ∝

1 f 2H+1

24

7.20 We have

ln N

ln N

M = N l − ln m = N (Lm)− ln m or ln M = ln N − ln m or

ln N ln N − ln L ln m ln m

− ln N

M = L ln m = LD which is constant.

7.21 By definition E[(x(n + n0 ) − x(n)]2 ] ∝ n2H 0 or E[x2 (n + n0 ) + x2 (n) − 2x(n + n0 )x(n)] ∝ n2H 0 ⇒ 1 E[x(n)(x(n + n0 ) − x(n))] ∝ E[x2 (n + n0 )] − E[x2 (n)] − n2H 0 2 or

1 E[x(n)(x(n + n0 ) − x(n))] ∝ [(n + n0 )2H − n2H − n2H 0 ] 2

Solutions to the problems of Chapter 8 8.1 Verify that any other path from the one shown in the figure 8.1 results in higher cost The edit distance is D = 2.

Figure 8.1: Problem 8.1

8.2 We consider the four cases of Sakoe-Chiba constraints listed in chapter 8 of the book. It is assumed that the test utterance has a length of J nodes, whereas the reference utterance has a length of I nodes.

• For the first case, the vertical path starting at node (1,1) and ending at node (J,1) achieves a maximum expansion rate of inf, slope inf. For the same case the horizontal path starting at node (1,1) and ending at node (1,I) achieves a maximum compression rate of inf, slope 0. An itermediate rate of 1 is shown in the figure 8.2.

1

2

Figure 8.2: Problem 8.2 • The second case demonstrates that a diagonal transition must always precede a horizontal or vertical one. As a result, the maximum compression rate is 2 (slope 1/2) and can be achieved by the ”path” of the figure 8.3(a).

Figure 8.3: Problem 8.2 Similarly, the maximum expansion rate is 2 (slope 2) and can be achieved by the ”path” of the figure 8.3(b). A path corresponding to an intermediate expansion rate of 4/3 is the one of the figure 8.3(c).

• The third case demands that a diagonal transition should always precede two successive horizontal or vertical ones. This results to a maximum compression rate of 3, slope 1/3 and maximum expansion rate of 3, slope 3. This is illustrated in the figures 8.4(a) and 8.4(c). Figure 8.4(b) demonstrates a path achieving an intermediate rate compression rate of 6/5

3

Figure 8.4: Problem 8.2 • The fourth case imposes the constraint that a horizontal or vertical transition must always be preceded by two diagonal ones. This results in a maximum expansion factor of 3/2, slope 3/2 and maximum compression factor of 3/2, slope 2/3. This is illustrated in the figures 8.5(a) and 8.5(c). Figure 8.5(b) demonstrates a path achieving an intermediate expansion rate of 5/4.

Figure 8.5: Problem 8.2 8.3 The following is a computer program written in the Matlab programming language, that implements dynamic time warping, for a complete optimum path and the Itakura constraints. It is assumed that costs are assigned with the nodes of the grid and not with the transitions among them. Input matrix Cost Matrix holds the costs assigned to the nodes. Since we deal with the case of a complete path, the path should start at a fictitious (0,0) node.

4 Variables e reference and e test denote the range of allowable transitions originated from node (0,0), for the reference and test templates respectively. bf function [best path,cost]=itakura(Cost matrix,e reference,e test) %MATRIX Y HOLDS THE PREDECESSOR TO EACH NODE. IT IS COMPLEX BECAUSE %IT HAS TO STORE TWO INDEXES PER PREDECESSOR. THE REAL PART STORES %THE ROW INDEX J, WHILE THE IMAGINARY PART STORES THE COLUMN INDEX I. %INITIALIZATION f=sqrt(-1); [J,I]=size(Cost matrix); %ALLOWABLE STARTING NODES for j=1:e reference Dmin(j,1)=Cost matrix(j,1); Y(j,1)=0+f*0; end; for j=e reference+1:J Dmin(j,1)=10000; %Just a large constant Y(j,1)=0+f*0; end for i=1:e test Dmin(1,i)=Cost matrix(1,i); Y(1,i)=0+f*0; end; for i=e test+1:I Dmin(1,i)=10000; Y(1,i)=0+f*0; end; %RECURSION for i=2:I for j=2:J if j>2 if real(Y(j,i-1))<j,

5 [g,ind]=min(Dmin(j-2:j,i-1)+Cost matrix(j,i)); Dmin(j,i)=g; Y(j,i)=j-2+ind-1+f*(i-1); elseif real(Y(j,i-1))==j, [g,ind]=min(Dmin(j-2:j-1,i-1)+Cost matrix(j,i)); Dmin(j,i)=g; Y(j,i)=j-2+ind-1+f*(i-1); end %THE SECOND ROW HAS TO BE TREATED SEPERATELY. elseif j==2 if real(Y(j,i-1))<j, [g,ind]=min(Dmin(1:j,i-1)+Cost matrix(j,i)); Dmin(j,i)=g; Y(j,i)=ind+f*(i-1); elseif real(Y(j,i-1))==j, [g,ind]=min(Dmin(1:j-1,i-1)+Cost matrix(j,i)); Dmin(j,i)=g; Y(j,i)=1+f*(i-1); end end end end cost=Dmin(J,I); %BACK TRACKING TO SCAN BEST PATH best path(I+1)=J+f*I; for i=I:-1:2 if real(best path(i+1))==1 | imag(best path(i+1))==1 break end best path(i)=Y(real(best path(i+1)),imag(best path(i+1))); end; best path(1)=0; ind=find(abs(best path)==0); last zero=ind(length(ind)); %cost normalisation cost=cost/(I-last zero) ; %plot best path

6 plot(imag(best path(last zero:length(best path))),... real(best path(last zero:length(best path))),’r’); title(’Best path provided by DP’); xlabel(’Test Template’); ylabel(’Reference Template’); In order to test the above algorithm, download the file Speech seg.zip from the book’s Web site (URL http://www.di.uoa.gr/stpatrec/Speech seg.zip).

This is a compressed file. It includes utterances of the words love and kiss spoken by a male speaker. The utterances are named love1.wav-love5.wav and kiss1.wav-kiss5.wav. and have been recorded with a sampling frequency of 22050Hz using 8 bits/sample. In order to read each file, use the command wavread of Matlab, e.g., [love1, Fs]=wavread(love1); Consider love1.wav and kiss3.wav to be the reference words. Each one of the remaining files is treated as an unknown word that will be determined using the algorithm presented above. Prior to running the algorithm, a feature sequence has to be extracted from each reference and test pattern. This is achieved by means of a Short Time Fourier Transform. For a window length of 512 samples, an overlap of 100 samples between successive windows and a hamming multiplier, the Matlab command specgram performs the desired feature extraction as given by the following example: [B love1,F,T] =specgram(love1,512,22050,hamming(512),100) B love1 is subsequently normalised: B love1=(1/512)*(1/max(max(abs(B love1))));

We focus on Fourier coefficients fI, I=1:100, corresponding to frequencies 0:4263Hz. At a second step, a cost grid has to be calculated for each pair of reference and test patterns using the euclidean distance function. This is achieved by the M-file cost grid, e.g., Cost matrix=cost grid(abs(b test(1:100,:)),abs(b love1 The source code for cost grid.m is given below. function y=cost grid(b test,b ref) [b ref 1,b ref 2]=size(b ref); [b test 1,b test 2]=size(b test);

7 for i=1:b ref 2 for j=1:b test 2 temp=b test(:,j)-b ref(:,i); temp=temp .∧ 2; y(i,j)=sqrt(sum(temp)); end end At a last step the DP algorithm is invoked: [best path,cost]=itakura(Cost matrix,e reference,e test) 8.4 This is a correlation coefficient involving moments that are invariant to scale-rotation-translation, thus, the measure is invariant to the above actions.

8.5 ′

M (u, v) = =

Z ∞Z ∞ 0

Z ∞Z ∞ 0

0

= aj(u+j)

0

f (ax, ay)x−ju−1 y −jv−1 dxdy

f (x′ , y ′ )

Z ∞Z ∞ 0

0

x′ −ju−1 y ′ −jv−1 1 ′ ′ dx dy a a a2

f (x, y)x−ju−1 y −jv−1 dxdy

= aj(u+j) M (u, v) or |M ′ (u, v)| = |M (u, v)| (Recall that xju = exp(jup), where ln x = p) 8.6 The number of the resulting subblocks is MIJN . For each of these subblocks the number of operations is • Full search: There are (2p + 1)2 pixel positions to be searched. For each pixel position the required number of operations is 3M N , where 3 stands for one substraction, one square (absolute) value and one addition (eq. 8.4). Hence the total number of computations per second is IJ ⋆ f ⋆ (2p + 1)2 ⋆ M N ⋆ 3 = 29.89GOP S MN (GOPS: Giga Operations Per Second)

8 • Two dimencional logarithmic: There are 8k + 1 pixel positions, k = ⌈log2 p⌉ (4 for p = 15), or in total IJ ⋆ f ⋆ (8k + 1) ⋆ 3 ≃ 1GOP S • One dimensional logarithmic: There are 4k + 1 pixel positions, (8 for p = 15). Thus, IJ ⋆ f ⋆ (4k + 1) ⋆ 3 ≃ 530M OP S • We will assume full search at the lowest resolution level. ⋄ Level 2: – Image size 180 ⋆ 120. Subblock size 4 ⋆ 4. Hence the number of subblocks is 1350, or 1350 ⋆ 30 per second.

– Search parameter: ⌈ p4 ⌉ = 4 – number of operations: 1350 ⋆ 30 ⋆ (2 ⋆ 4 + 1)2 ⋆ 3 ⋆ 16 = 158M OP S ⋄ level 1 : – Image size 360 ⋆ 240. Subblock size 8 ⋆ 8. Hence the number of subblocks is 1350, or 1350 ⋆ 30 per second.

– Number of locations 9 – number of operations: 1350 ⋆ 30 ⋆ 9 ⋆ 3 ⋆ 64 ≃ 70M OP S ⋄ level 0 : – Image size 720 ⋆ 480. Subblock size 16 ⋆ 16.

– Number of locations 9 – number of operations: 1350 ⋆ 30 ⋆ 9 ⋆ 3 ⋆ 16 ⋆ 16 ≃ 280M OP S Hence the total is 508 MOPS. More general, it is easily checked that the number of operations is IJ p MN ⋆ f ⋆ ((2⌈ ⌉ + 1)2 + 180) ⋆ ⋆3 MN 4 16 where 180 = 9 ⋆ 4 + 9 ⋆ 16

Solutions to the problems of Chapter 9 9.1 The Complete data set is given by (X, Ωi ), where X = {x1 , . . . , xN } and Ωi = {i1 , . . . , ıN }, ik = {1, . . . , K}, k = 1, 2, . . . , N . Thus the log likelihood function of the unobserved data is ln p(X, Ωi ) = ln{p(X|Ωi )P (Ωi )} Let Θ be the unknown parameters vector, consisting of µs , s = 1, 2, . . . , K P (i|j), i, j = 1, 2, . . . , K and Pi i = 1, 2, . . . , K, for the initial state.

Then, according to the EM algorithm we have E step: Q(Θ|Θ(t)) = E[ln{p(X|Ωi , Θ)P (Ωi |Θ)}] =

X

P (Ωi |X, Θ(t)) ln{p(X|Ωi , Θ)P (Ωi |Θ)}

i

However

NY −1

P (Ωi |Θ) = Pi1

P (ik+1 |ik )

k=1

or Q(Θ|Θ(t)) =

X

P (Ωi |X, Θ(t))·

i

2

{− ln 2πσ −

N X

1 (xk − µik )T (xk − µik )) 2 k=1 2σ (

+ ln Pi1 +

N −1 X

ln P (ik+1 |ik )}

k=1

1

2 M step: Maximization takes place with respect to µs , s = 1, . . . K and Pi , P (ik+1 |ik ). ∂Q(Θ|Θ(t)) = 0⇒ ∂µs P

i P (Ωi |X, Θ(t))

µs = P

P

k:i =s xk

s = 1, 2, . . . , K

P k

i P (Ωi |X, Θ(t))

k:ik =s 1

(9.1)

Maximizing w.r. to the other two is a constrained maximization since k X

Pi1 = 1

i1 =1 N X

P (ik+1 |ik ) = 1

ik+1 =1

and it results in

i,Pi1 =s P (Ωi |X, Θ(t))

P

Ps = P (i|j) =

P

Furhermore

P

i P (Ωi |X, Θ(t))

i P (Ωi |X, Θ(t))

ik =j,ik+1 =i 1

P

i P (Ωi |X, Θ(t))

P

P (Ωi |X, Θ(t)) =

ik =j 1

P

P (X|Ωi , Θ(t))P (Ωi |Θ(t)) P (X; Θ(t))

(9.2)

(9.3)

Observe now that X

P (Ωi |X, Θ(t))

=

X X

X

xk =

k:ik =j

i

(

k

P (Ωi |X, Θ(t))xk

i,ik =s

However, the term in parenthesis is the probability of being at state s at stage k. This brings us into the interptation of (9.36) of the book.

The (9.36) formulation is just an efficient way for computing µ in (9.1) above. Similar arguments hold for the other two derive recursions.

3 9.2 If a model stays at state Si for d successive stages, then the state sequence should be O = {Si , Si , . . . Si , Sj 6= Si }. In addition, if the self transition probability is P (i|i) then the probability for leaving state Si is 1 − P (i|i). Thus, the probability P ri (d) of the model being at state i for d successive stages (d−1 transitions) is the product P (i|i) . . . P (i|i), (d − 1 times) multiplied by (1 − P (i|i)), due to the dth transition, upon which the model leaves state i, i.e., P ri (d) = (P (i|i))(d−1) (1 − P (i|i)) We can now readily calculate d̂i , the average duration for staying at state i, as d̂i =

∞ X

dP ri (d) =

∞ X

d(P (i|i))(d−1) (1 − P (i|i))

d=1

d=1

or d̂i = (1 − P (i|i))

∞ X ∂P (i|i)d

d=1

or d̂i = (1 − P (i|i))

∂

P∞

d d=1 P (i|i)

∂P (i|i)

and finally d̂i =

∂P (i|i)

∂ = (1 − P (i|i)) ∂P (i|i)

P (i|i) 1 − P (i|i)

!

1 1 − P (i|i)

9.3 The reestimation formulas, when using Q different versions of the spoken word, stem directly from the formulas for a single observation sequence ((9.31) and (9.32)). This is because both (9.31) and (9.32) are based on frequencies of occurence of various events (e.g., expected number of transitions from state i for the denominator of (9.31)) and therefore the reestimation for multiple observation sequences can be performed by adding together the individual frequencies of occurence for each observation sequence.

As an example, we will derive the denominator of P̂ (j|i) for the case of multiple observation sequences (all other quantities can be derived

4 with a similar procedure): (9.31) states that for a single observation sequence Xm the denominator is denm =

NX m −1

γkm (i)

k=1

Using (9.30), denm becomes denm =

1 (αm (ik = i)β m (ik = i)) P (Xm |S)

Summing together the individual denominators, we get the denominator of P̂ (j|i) for the case of multiple observation sequences, i.e., Q X

NX m −1 1 αm (ik = i)β m (ik = i) P (X |S) m m=1 k=1

9.4 We will prove that if α(ik ) and β(ik ) are replaced in (9.31) and (9.32) by their scaled versions, the reestimation formulas remain unchanged. This will be proved for 9.31 (the proof for (9.32) is similar).

Let α̂(ik = i) = c1k α(ik = i), β̂(ik+1 = i) = ck β(ik+1 = i) be the P s scaled versions of α and β respectively, where ck = K ik =1 α(ik ) First we use (9.29) and (9.30) in order to rewrite (9.31) in terms of the quantities α and β:

P̂ (j|i) =

PN −1

k=1 α(ik = i)P (j|i)P (xk+1 |j)β(ik+1 = j) PN −1 k=1 α(ik = i)β(ik = i)

(9.4)

At a second step we use (9.23) to replace β(ik = i) in the denominator. Thus, (9.4) becomes: PN −1

α(ik = i)P (j|i)P (xk+1 |j)β(ik+1 = j) P̂ (j|i) = PN −1k=1 PN j=1 α(ik = i)P (j|i)P (xk+1 |j)β(ik+1 = j) k=1

(9.5)

If we replace α and β with their scaled versions. Thus(9.5) becomes: PN −1

α̂(ik = i)P (j|i)P (xk+1 |j)β̂(ik+1 = j) P̂ (j|i) = PN −1k=1 PN j=1 α̂(ik = i)P (j|i)P (xk+1 |j)β̂(ik+1 = j) k=1

(9.6)

5 But α̂(ik = i) can be written as α̂(ik = i) = [

k Y 1

s=1 cs

]α(ik = i) = Ck α(ik = i)

(9.7)

and β̂(ik+1 = j) can be written as T Y

1 ]β(ik+1 = j) = Dk+1 β(ik+1 = j) s=k+1 cs

β̂(ik+1 = j) = [

(9.8)

where T is the length of the observation sequence.

Thus (9.6) can be written as PN −1

Ck α(ik = i)P (j|i)P (xk+1 |j)Dk+1 β(ik+1 = j) P̂ (j|i) = PN −1k=1 PN j=1 Ck α(ik = i)P (j|i)P (xk+1 |j)Dk+1 β(ik+1 = j) k=1 (9.9) But T Y 1 1 = = CT Ck Dk+1 = s=1 cs s=1 cs s=k+1 cs k Y 1

T Y

(9.10)

This means that the product Ck Dk+1 is independent of the stage k. Therefore, the terms Ck Dk+1 cancel out of both the numerator and the denominator and the reestimation formula for P̂ (j|i) remains unchanged.

9.5 According to the theory the best model, say S ∗ is the one resulting from S ∗ = max P (Si |X), i = 1, 2, . . . , M i

If the models are not equipropable this is NOT equivalent with maximizing P (X|Si ). In this case we have P (Si |X) =

p(X|Si )P (Si ) P (X)

However, all these quantities depend on the values of the parameters Λ of the models. Thus we can write p(X|Si , Λ)P (Si ) P (Si |X, Λ) = p(X, Λ)

6 Note that for fixed Λ p(x, Λ) depends only on X and it is independent of the models. This is not, however, true during training, where Λ is changing. Furthermore p(X, Λ) =

X

p(X|Sk , Λ)P (Sk )

k

or P (Si |X, Λ) =

p(X|Si , Λ)P (Si ) P p(X|Si , Λ)P (Si ) + k6=i p(X|Sk , Λ)P (Sk )

• If P (Sr |X, Λ) = P (Sr |X, Λr ), that is dependence of the conditional probabilities is only on the respective model parameters, maximizing P (Sr |X, Λ) w.r. to Λi is equivalent to maximizing P (Sr |X, Λ) = P (Sr |X, Λr ) =

1 1 + p(X|Sconst r ,Λr )P (Sr )

or equivalent to maximizing p(X|Sr , Λr )P (Sr ) or p(X|Sr , Λr ) as was the case with the theory in the book.

• If maximization takes place w.r. to model parameters Λ, then maximizing P (Sr |X, Λ) is equivalent with maximizing 1

P (Sr |X, Λ) = 1+

P

k6=r

p(X|Sk ,Λ)P (Sk )

p(X|Sr ,Λ)P (Sr )

or minimizing the ratio. k6=r p(X|Sk , Λ)P (Sk )

P

p(X|Sr , Λ)P (Sr )

Solutions to the problems of Chapter 10 10.1 Although the mean and variacne can be estimated using the propability function, we will follow another approach.

Assume N = 1 and let x to be the random variable that takes values 0 or 1. Then E[x] = E[k] = 1P + 0(1 − P ) = P Also σk2 = = P − P 2 = P (1 − P ) Assume now that we carry the experiment N times. We form the random variable k = x1 + · · · + xN Obviously k is the number of 1’s that occur, since each xi takes the value 0 or 1. Thus, we have E[k] = E[x1 ] + · · · + E[xN ] = N P Also σk2 = σx21 + · · · + σx2N = N P (1 − P ) 10.2 The solution follows Fukunaga’s book, page 221. For the minimum distance classifier, classification is achieved according to the rule (x − µ1 )T (x − µ1 ) > (<)(x − µ2 )T (x − µ2 ) 1

2 Assume Ni feature vectors from class ω1 and N2 from ω2 . Then in the resubstitution the place of the mean vectors is taken by their estimates µ̂1 =

N1 X

(1)

xk ,

µ̂2 =

N2 X

(2)

xk

k=1

k=1

where the superscript denotes the class of the corresponding training vector. (i) When the Leave-one-out method is used, then if a vector xk is used, for testing, this is not taken into account for the classifier design, hence in this case the mean vector estimate of the corresponding class ωi , will be Ni X 1 (i) (i) µ̂ik = [ xj − xk ] Ni − 1 j=1 = µ̂i −

1 (i) (x − µ̂i ) Ni − 1 k

or

Ni (i) (x − µ̂i ) Ni − 1 k Thus, for i = 1, i.e, the vecor from ω1 , then the test relies on the outcome of (i)

xk − µ̂ik =

(

N1 2 (1) (1) (1) (1) ) (xk − µ̂1 )T (xk − µ̂1 ) > (<)(xk − µ̂2 )T (xk − µ̂2 ) (10.1) N1 − 1

and if i = 2 on the outcome of (2)

(2)

(xk − µ̂1 )T (xk − µ̂1 ) > (<)(

N2 2 (2) (2) ) (xk − µ̂2 )T (xk − µ̂2 ) (10.2) N2 − 1

To prove the second part of the problem, observe that the error occurs (1) if for an xk the lefthand side of (10.1) is larger than the right hand (2) side, and if for an xk the right hand side of (10.2) is larger than the i > 1. Therefore the possibility that this is left hand side. However NNi −1 happening is larger for the leave-one-out method, than for the resubi stitution. In the latter method the factors NNi −1 do not enter into the test.

3 10.3 The classification error is a function of both the test set and the design set. Let PD be the underlying distribution of the design set and PT of the test set: Obviously ε(PT , PT ) ≤ ε(PD , PT )

(10.3)

The error of the Bayesian classifier, designed using the true statistical distribution P, and tested on a test set with the same distribution, is ε(P, P ) and it is the minimum. Obviously, in practice we only have an estimate P̂ of P and if we design a Bayesian classifier using P̂ , then, ε(P, P ) ≤ ε(P̂ , P )

(10.4)

ε(P̂ , P̂ ) ≤ ε(P, P̂ )

(10.5)

Also If now the error counting approach is used, it is known to be unbiased with respect to the test set, ε(P̂ , P ) = EP̂T [ε(P̂ , P̂T )] Also, by taking the expectation in (10.5) over all possible sets, we have E[ε(P̂ , P̂ )] ≤ E[ε(P, P̂ )] = ε(P, P ) Combining the above we obtain E[ε(P̂ , P̂ )] ≤ ε(P, P ) ≤ EP̂T [ε(P̂ , P̂T )] Obviously ε(P̂ , P̂ ) corresponds to the resubstitution method, where the same set is used for design and test. The right hand side corresponds to the hold out method where different sets are used for the design and test.

10.4 We will search for the minimum of Nl 1 X γI g T Lg (yi − (g(xi ))2 + γH ||g||2H + Nl i=1 (Nl + Nu )2

(10.6)

4 where g = [g(x1 ), g(x2 ), . . . , g(xNl +Nu )]T , among the set of functions g(x) =

NX l +Nu

aj K(x, xj )

(10.7)

j=1

and we will show that it occurs at [a1 , a2 , . . . , aNl +Nu ]T ≡ a = (JK + γH Nl I +

γI N l LK)−1 y (10.8) (Nl + Nu )2

where I is the identity matrix, y = [y1 , y2 , . . . , yNl , 0, . . . , 0]T , J the (Nl + Nu ) × (Nl + Nu ) diagonal matrix with Nl entries as 1 and the rest 0, i.e., J = diag(1, 1, . . . , 1, 0, . . . , 0) and K = [K(i, j)] is the (Nl + Nu ) × (Nl + Nu ) Gram matrix. Observe that J and K are symmetric matrices and that JJ = J. From (10.6) and (10.7) and for the first of the three in the cost function we have Nl Nl 1 X 1 X (yi − g(xi ))2 = (yi − aT K i )2 Nl i=1 Nl i=1

where



  Ki =   

K(xi , x1 ) K(xi , x2 ) .. .

K(xi , xNl +Nu )

(10.9)

     

Observe that aT K i is the ith component of the (row vector) vectormatrix product aT K, where K is the Gram matrix, K(i, j) = K(xi , xj ).

Let us consider now he square norm 2

T

||y − Ka|| = (y − Ka) (y − Ka) =

NX l +Nu

(yi − aT K i )2

(10.10)

i=1

Hence, in (10.9) only the first Nl elements of the sum in the square norm in (10.10) appear. Since the last Nu elements of y are zero, it is easy to see that we can write A≡

Nl 1 X 1 (yi − g(xi ))2 = (y − JKa)T (y − JKa) Nl i=1 Nl

(10.11)

5 Since

K T1 a T    K2 a    ..     . 



T  JKa =   K Nl a    T  0    ..     .





(10.12)

0T

Expanding (10.11) we get A=

o 1 n T y y + aT KJJKa − 2aT KJy Nl

where the symmetry of K, J has been taken into account. Taking the gradient w.r. to a, for the required optimization, we obtain ∂A 2 = {KJKa − KJy} ∂a Nl

(10.13)

where we used that JJ = J. Inserting (10.7) into (10.6), the second term in the cost becomes B ≡ γH ||g||2H = γH

NX l +Nu l +Nu NX i=1

ai aj K(·, xi )K(·, xj ) = γH aT Ka

j=1

where the reproducing property of an RKHS K(·, xi )K(·, xj ) = K(xi , xj ) = K(xj , xi ) has been used. Taking the derivative of B w.r. to a we obtain ∂B = 2γH Ka ∂a

(10.14)

Inserting, again, (10.7) into (10.6), and taking into account the definition of K i , i = 1, 2, . . . , Nl + Nu , the third term in the cost becomes γI g T Lg C ≡ (Nl + Nu )2

6 

= =

 γI T T  [a K , . . . , a K ]L 1 N +N u l  (Nl + Nu )2

γI aT KLKa (Nl + Nu )2

K T1 a .. .

K TNl +Nu a

   

where, once more, the symmetry of K has been taken into account.

Taking the gradient of C w.r. to a we have ∂C 2γI = KLKa ∂a (Nl + Nu )2

(10.15)

Combining (10.15), (10.14) and (10.13), equating to zero and multiplying both sides by the inverse of K, we finally obtain (JK + Nl γH I +

N l γI LK)a = Jy = y (Nl + Nu )2

from which we obtain a = (JK + Nl γH I +

N l γI LK)−1 y (Nl + Nu )2

where we used that Jy = y, due to the special structure of y.

Solutions to the problems of Chapter 11 11.1. Let s0 (< +∞) be the maximum value of s(x, y). This leads to a a ≤ s0 s(x, y) i.e., d0 = a/s0 .

It is easy to see that eqs. (11.5), (11.6) are satisfied. In addition, d(x, y) = d0 is equivalent to s(x, y) = s0 which implies that x ≡ y, since s is a metric.

Finally, let x, y, z ∈ X. It is d(x, y) + d(y, z) = a

s(x, y) + s(y, z) s(x, y)s(y, z)

Taking into account eq. (11.13) it easily follows the triangular inequality.

11.2. It is l X

d2 (x, y) + d2 (y, z) = (

l X

|xi − yi |2 )1/2 + (

i=1

i=1 l X

(

|yi − zi |2 )1/2 ≥

|xi − yi + yi − zi |2 )1/2 = d2 (x, y)

i=1

1

2 11.3. (a) Let s′ (x, y) = s(x, y) + a. It is easy to see that the maximum value of s′ (x, y), ∀x, y ∈ X is s′0 = s0 + a.

The proof of eqs. (11.10) and (11.11) is straightforward.

Let x, y ∈ X. It is clear that s′ (x, y) = s′0 is equivalent to s(x, y) = s0 , which in turn is equivalent to x ≡ y, since s is a metric.

Finally, let x, y, z ∈ X. Since s is a metric we have s(x, y)s(y, z) ≤ [s(x, y) + s(y, z)]s(x, z) This inequality leads to s(x, y)s(y, z) ≤ [s(x, y) + s(y, z)]s(x, z) + 2as(x, z) + a2 Adding to both sides of the above inequality the term a(s(x, y)+s(y, z))+ a (> 0), and after some manipulations, the last inequality leads to 2

(s(x, y) + a)(s(y, z) + a) ≤ [(s(x, y) + a) + (s(y, z) + a)](s(x, z) + a) i.e., eq. (11.13) holds for s′ . (b) Let d′ (x, y) = d(x, y) + a. It is easy to see that the minimum value of d′ (x, y), ∀x, y ∈ X is d′0 = d0 + a.

The conditions (11.5) (11.6) and (11.7) are easy to prove.

Let us consider now x, y, z ∈ X. Since d is a metric, d(x, y) + d(y, z) ≥ d(x, z) Moreover, since a > 0, the above inequality gives (d(x, y) + a) + (d(y, z) + a) ≥ (d(x, z) + a) which is the triangular inequality for d′ . .

11.4. Let d′ = f (d). Since f is a monotonically increasing function, it is one to one and onto. It is easy to show that the minimum value, d′0 , of d′ (x, y), ∀x, y ∈ X is d′0 = f (d0 ), where d0 is the minimum value of d(x, y).

Since f is a monotonically increasing function, the proof of eqs. (11.5), (11.6) and (11.7) for d′ is straightforward.

Finally, let x, y, z ∈ X. It is f (d(x, z) ≤ f (d(x, y) + d(y, z)) ≤ f (d(x, y)) + f (d(y, z))

3 i. e. the triangular inequality holds for f (d).

11.5. Let d′ = f (s). It is easy to prove that the conditions (11.4), (11.5), (11.6) and (11.7) hold for d′ .

Let us consider now x, y, z ∈ X. Then, condition (11.13) can be written for s as 1 s(x, z) ≥ 1 + s(y1,z ) s(x,y ) Since f is a monotonically decreasing function, the above inequality gives f (s(x, z)) ≤ f (

1

) 1 + s(y1,z ) s(x,y )

Taking into account the property of f , we easily conclude that the triangular inequality holds for d′ .

11.6. Before we proceed, let us derive the equation for d∞ . Let |xj − yj | = maxi=1,...,l |xi − yi |. It is X |xi − yi | dp (x, y) = (1 + ( )p )1/p |xj − yj | j6=i |xj − yj | |xi −yi | where |x < 1. Thus, j −yj |

lim

dp (x, y)

p→+∞ |xj − yj |

=1

and, therefore, d∞ (x, y) = max |xi − yi | i=1,...,l

The proof of d2 (x, y) ≥ d∞ (x, y) is obvious. The proof of d2 (x, y) ≤ d1 (x, y) stems from the inequality l X

(

|xi − yi |)2 ≥

l X

|xi − yi |2

i=1

i=1

11.7. (a) Clearly, the maximum value of s(xi , yi ), given by (11.39), with xi , yi ∈ [0, 1], is 1. Thus, eq. (11.40) gives l X

sqF (x, y) ≤ (

i=1

1)1/q = l1/q

4 Similarly, the minimum value of s(xi , yi ) = 0.5. Thus l X

sqF (x, y) ≥ (

0.5q )1/q = 0.5l1/q

i=1

(b) Let s(xj , yj ) = max1≤i≤l s(xi , yi ). It is X s(xi , yi ) sqF (x, y) lim ( = lim ( )q + 1)1/q = 1 q→∞ s(x , y ) q→∞ j j i6=l s(xj , yj ) s(xi ,yi ) since s(x < 1 and q → ∞. Thus, j ,yj )

lim sq (x, y) = max s(xi , yi ) q→∞ F 1≤i≤l 11.8. (a) Let s1 (x, y) =

Pk−1

i=1 aii

l where aii are the diagonal elements of the corresponding contingency table.

Clearly, 0 ≤ s1 (x, y) ≤ 1.

Let x be a vector that contains at least one 0. Then s1 (x, x) < 1. Thus s1 is not a similarity measure, since condition (11.10) is violated. (b) Let s2 (x, y) =

Pk−1

i=1 aii

l − a00 where aii are the diagonal elements of the corresponding contingency table.

Clearly, 0 ≤ s2 (x, y) ≤ 1.

Also, ∀x ∈ X, s2 (x, x) = 1.

Moreover, ∀x, y ∈ X, s2 (x, y) = s2 (y, x). Thus s2 is a similarity measure.

Pk−1 Let x, y ∈ X. The condition s2 (x, y) = 1 is equivalent to i=0 aii = l, i.e. all the off diagonal elements of the contingency table are 0. This implies that x = y.

However, the condition (11.13) is not always satisfied. To see this, let us consider three vectors x, y and z and let A(x, y) be the contingency table of x and y and B(y, z), C(x, z) be the contingecy tables for x, y and x, Pk−1 z, respectively. Moreover, let x, y, z be chosen such that i=1 aii > 0,

5 Pk−1

i=1 bii > 0 and

Pk−1

i=1 cii = 0.

Then, it is s2 (x, y) > 0, s2 (y, z) > 0 and

s2 (x, z) = 0. Thus,

s2 (x, y)s2 (y, z) > 0 = (s2 (x, y) + s2 (y, z))s2 (x, z) which contradicts condition (11.13). Thus, s2 is not a metric similarity measure.

As an example, we may consider k = 3 and x = [1, 2, 0, 1]T , y = [1, 1, 0, 0]T and z = [0, 1, 0, 0]T . (c) Let s3 (x, y) =

Pk−1

i=0 aii

l where aii are the diagonal elements of the corresponding contingency table.

Cleary the conditions (11.9), (11.10) and (11.11) are satisfied and s3,0 = 1.

Also, the condition (11.12) is easily verified. Finally, using similar arguments as above, we may prove that condition (11.13) is not satisfied. To see this, consider the vectors x = [1, 2, 0, 1]T , y = [1, 1, 0, 1]T , z = [0, 1, 1, 0]T .

It is s3 (x, y) > 0, s3 (y, z) > 0 and s3 (y, z) = 0. This implies that (11.13) is violated. Thus, s3 is a similarity measure but not a metric.

11.9. It is sps avg (x, C) =

1 X s(x, y) = nC y ∈C

1 X (dmax − d(x, y)) = dmax − dps avg (x, C) nC y ∈C 11.10. Squaring the left and the right hand side of eq. (11.27), the claim follows.

11.11. (a) It is dn,0 = 0 ≤ d(x, y), ∀x, y ∈ X. Obviously, dn (x, x) = 0, ∀x ∈ X and dn (x, y) = dn (y, x), ∀x, y ∈ X. Also, dn (x, y) = 0 implies that xi = yi , i = 1, . . . , l, i.e. x = y.

Finally, let x, y, z ∈ X. It is dn (x, y) + dn (y, z) = |xi − yi | + |yi − zi | + |xi − yi + yi − zi | +

X 1 (|xj − yj | + |yj − zj |) ≥ l − [ l−2 ] j6=i l

X 1 (|xj − yj + yj − zj |) = dn (x, z) l − [ l−2 ] j6=i l

6 where we make use of the Minkowski inequality. Thus, the claim has been proved. (b) The computation of d2 requires 2l − 1 additions and sutractions, l squared operations and one square root, while the computation of dn requires 2l − 1 additions and sutractions and 1 multiplication. That is dn is less demanding in computations than the Euclidean distance.

11.12. Let d0 (s0 ) be the minimum (maximum) value of d (s). ss ss ss • dss min : It is dmin,0 = d0 ≤ dmin (Di , Dj ), ∀Di , Dj . Obviously, dmin (Di , Di ) = ss ss d0 , ∀Di and dmin (Di , Dj ) = dmin (Dj , Di ), ∀Di , Dj . ss ss ss • dss mean : It is dmean,0 = d0 ≤ dmean (Di , Dj ), ∀Di , Dj . Obviously, dmean (Di , Di ) = ss d0 , ∀Di and dss mean (Di , Dj ) = dmean (Dj , Di ), ∀Di , Dj . ss • dss max : It is easy to see that dmax (Di , Di ) is not the same for all Di ’s.

Consider for example the set D1 that contains the two most distant vectors of X and the set D2 that contains a single vector. Clearly, ss dss max (D1 , D1 ) 6= dmax (D2 , D2 ). Thus, condition (11.5) is violated and, ss therefore, dmax is not a measure. ss • dss avg : As in the previous case, davg (Di , Di ) is not the same for all Di ’s.

For example, considering D1 and D2 from the previous case, we have ss that dss avg (D1 , D1 ) 6= davg (D2 , D2 ). Thus, once again condition (11.5) is violated and, therefore, dss avg is not a measure.

For the similarity functions we may proceed in an analogous way to prove the claim.

11.13. It is • ℘ps max (x, D) = maxy ∈D ℘(x, y) • ℘ps min (x, D) = miny ∈D ℘(x, y) • ℘ps mean (x, D) = ℘(x, mD ), where mD is the mean vector of D. 1 • ℘ps avg (x, D) = nD

P

y ∈D ℘(x, y), where nD is the cardinality of D.

7 These proximity functions are not measures since for each of them we have ℘ps : U1 × U → R where U1 contains subsets of X with cardinality 1. Thus, writing ℘ps (D, x) is meaningless.

Solutions to the problems of Chapter 12 12.1. For N = 1 we have S(1, 2) = 21−1 − 1 = 0 and for N = 2, S(2, 2) = 22−1 − 1 = 1. Suppose that the statement holds true for N − 1, i.e. S(N − 1, 2) = 2N −2 − 1. We will prove that S(N, 2) = 2N −1 − 1. From (12.1) we have that S(N, 2) = 2S(N − 1, 2) + S(N − 1, 1) = 2(2N −2 − 1) + 1 = 2N −1 − 1 12.2. Let mC be the mean vector of the cluster C = {x1 , x2 , . . . , xr }. We will prove that the mean vector of the cluster C ′ = C ∪ {xr+1 } may be computed as (nC ′ − 1)mC + xr+1 nC ′ Taking into account that nC ′ = nC + 1 and that mC ′ =

mC =

(12.1)

x1 + x2 + . . . + xr r

we can easily prove that eq. (12.1) is equivalent to mC =

x1 + x2 + . . . + xr + xr+1 r+1

12.3. (a) The clusters resulting from the BSAS algorithm are: C1 = {x1 , x2 , x3 }, C2 = {x4 , x5 , x6 }, C3 = {x7 , x8 , x9 }, C4 = {x10 } and C5 = {x11 , x12 , x13 , x14 }.

The clusters produced by the MBSAS algorithm are: C1 = {x1 , x2 }, C2 = {x3 , x4 , x5 }, C3 = {x6 , x7 , x8 }, C4 = {x9 , x10 } and C5 = {x11 , x12 , x13 , x14 }.

1

2 (b) The clusters resulting from the BSAS algorithm are: C1 = {x1 , x2 , x3 }, C2 = {x9 , x10 }, C3 = {x4 , x5 , x6 }, C4 = {x11 , x12 , x13 , x14 } and C5 = {x7 , x8 }.

The clusters resulting from the MBSAS algorithm are: C1 = {x1 , x2 }, C2 = {x9 , x10 }, C3 = {x3 , x4 , x5 }, C4 = {x11 , x12 , x13 , x14 } and C5 = {x6 , x7 , x8 }. (c) The clusters produced by the BSAS algorithm are: C1 = {x1 , x2 , x3 }, C2 = {x9 , x10 }, C3 = {x4 , x5 , x6 }, C4 = {x11 , x12 , x13 , x14 } and C5 = {x7 , x8 }.

The clusters produced by the MBSAS algorithm are: C1 = {x1 , x2 , x3 }, C2 = {x8 , x9 , x10 }, C3 = {x4 , x5 , x6 , x7 } and C4 = {x11 , x12 , x13 , x14 }.

Figure 12.1: Problem 12.3 (d) The data vectors are shown in fig. 12.1. We observe that for fixed Θ, different ordering of presentation of the data to BSAS and MBSAS may lead to different clusterings concerning the number of clusters as well as the clusters themselves. (e) Two. This number of clusters may be obtained by both BSAS and MBSAS if the value of Θ is chosen between 4 and 7.

12.4. Both algorithms assign the vectors produced from each distribution to separate clusters. Let us justify this result. Since both distributions are normal, all vectors lie from the corresponding means at distance less than

3 q

(3 × 0.5)2 + (3 × 0.5)2 = 2.12, at a very high level of confidence 1 . Thus, the maximum distance between two vectors of the same distribution cannot exceed the value 4.24, i.e. r1 = r2 ≤ 4.24. Taking into account the above, and q the fact that the distance between the means of the two distributions is (20 − 0)2 + (20 − 0)2 = 28.28, the minimum distance r between vectors of different clusters cannot be less than 28.28 − 2.12 − 2.12 = 24.04. This happens when the two nearest vectors x and y from both distributions and the means of the distributions are collinear and both x and y lie at the maximum distance from their corresponding means. Since Θ ∈ [r2 , r − r2 ] = [4.24, 24.04], both algorithms will lead to clusterings of two clusters.

12.5. (a) Two. (b) The plot of the number of clusters versus Θ for the BSAS algorithm is shown in figure 12.2. As we can see, the algorithm fails to identify the two clusters underlying the data set. This happens because these clusters are not compact.

It is worth noting that if we present all the vectors of one cluster first and then all the vectors of the other cluster, the algorithm would identify once more a single cluster. This is justified by the fact that the mean vectors for both clusters lie very close to each other.

The behavior of MBSAS algorithm is similar. (c) Plotting again the number of clusters produced by the BSAS algorithm versus Θ we obtain once again a similar behavior as above.

The behavior of the MBSAS algorithm is similar.

In the sequel we give MATLAB code for the BSAS and MBSAS algorithms. % WARNING: IN THE BSAS & MBSAS ALGORITHMS YOU MAY % USE EITHER EUCLIDEAN OR THE DISTANCE % BETWEEN A VECTOR AND A CLUSTER % The BSAS algorithm is given using the Euclidean distance % and the MBSAS algorithm is given using the distance between a vector % and a cluster % This is the code for the BSAS algorithm. % It applies it iteratively for different values of 1

Note that for a 1-dimensional normal distribution with mean µ and variance σ 2 , it is P (|x − µ| ≥ 3σ) ≤ 0.003.

4

Figure 12.2: Problem 12.5 % Theta % The files data* contain the data of the algorithm % mintheta = the minimum value between two vectors in the % data set % maxtheta = the maximum value between two vectors in the % data set clear; format compact; data125; q = 10000; m arr = []; theta arr = []; for theta = mintheta : 0.2 : maxtheta theta theta arr = [theta arrtheta]; [N, l] = size(pat); belclust = zeros(N, 1); % Initialization meanv = []; meanv = pat(1, :);

5 m = 1; belclust(1) = m; % Main phase for i = 2 : N sdist = d eucl(pat(i, :), meanv(1, :)); sind = 1; for j = 2 : m temp = d eucl(pat(i, :), meanv(j, :)); if(temp < sdist) sdist = temp; sind = j; end end if(sdist > theta)&(m < q) m = m + 1; belclust(i) = m; tempmean = pat(i, :); meanv = [meanv; tempmean]; else belclust(i) = sind; count = 0; for k = 1 : i if(belclust(k) == sind) count = count + 1; end end meanv(sind, :) = ((count − 1) ∗ meanv(sind, :) + pat(i, :))/count; end end m arr = [m arrm]; end % WARNING: The plottings work only for 2-dimensional data. plot(pat(:, 1), pat(:, 2),′ r∗′ ) pause hold off plot(theta arr, m arr,′ r′ )

6 ========================== % This is the code for the MBSAS algorithm. % It applies it iteratively for different values of % Theta % The files data* contain the data of the algorithm clear; format compact; data125; q=1000; theta arr = []; m arr = []; for theta = mintheta : 0.1 : maxtheta theta theta arr = [theta arrtheta]; [N, l] = size(pat); belclust = zeros(N, 1); % Initialization meanv = pat(1, :); m = 1; belclust(1) = m; % Cluster determination for i = 2 : N sdist = d ps min(pat(i, :), pat, belclust, 1); sind = 1; for j = 2 : m temp = d ps min(pat(i, :), pat, belclust, j); if(temp < sdist) sdist = temp; sind = j; end end if(sdist > theta)&(m < q) m = m + 1; belclust(i) = m; tempmean = pat(i, :);

7 meanv = [meanv; tempmean]; end end % Pattern classification phase for i = 1 : N if(belclust(i) == 0) sdist = d ps min(pat(i, :), pat, belclust, 1); sind = 1; for j = 2 : m temp = d ps min(pat(i, :), pat, belclust, j); if(temp < sdist) sdist = temp; sind = j; end end belclust(i) = sind; count = 0; for k = 1 : N if(belclust(k) == sind) count = count + 1; end end meanv(sind, :) = ((count − 1) ∗ meanv(sind, :) + pat(i, :))/count; end end belclust’; m arr = [m arrm]; end plot(pat(:, 1), pat(:, 2),′ r∗′ ) pause hold off plot(theta arr, m arr,′ r′ ) ========================== % This program generates the data for the problem 12.5 rand(’seed’,0)

8 n of points = 50; pat1 = []; pat2 = []; pat = []; icount1 = 0; icount2 = 0; while(icount1 < n of points), temp = 1.3 ∗ rand(1, 2) − [0.650.65]; if(temp(1) <= 0.15)&(temp(1) >= −0.15)&(temp(2) <= 0.15)&(temp(2) >= −0.15) icount1 = icount1 + 1; pat1 = [pat1; temp]; pat = [pat; temp]; end end while(icount2 < n of points), temp = 1.3 ∗ rand(1, 2) − [0.650.65]; if((temp(1) >= 0.50)|(temp(1) <= −0.50))|((temp(2) >= 0.50)|(temp(2) <= −0.50)) icount2 = icount2 + 1; pat2 = [pat2; temp]; pat = [pat; temp]; end end pat = []; for i = 1 : n of points pat = [pat; pat1(i, :); pat2(i, :)]; end % Determination of the minimum and the maximum distances % between two vectors in the data set.

[N, l] = size(pat); mintheta = 1000000; maxtheta = 0; for i = 1 : N for j = i + 1 : N

9 temp = sqrt((pat(i, :) − pat(j, :)) ∗ (pat(i, :) − pat(j, :))′ ); if(temp < mintheta) mintheta = temp; end if(temp > maxtheta) maxtheta = temp; end end end ========================== NAME OF THE FUNCTION: d eucl.m % This function computes the Euclidean distance function z = d eucl(x, y) z = sqrt((x − y) ∗ (x − y)′ ); end ========================== NAME OF THE FUNCTION: d ps min.m % This function computes the distance between a point % and a cluster as the minimum distance between the % and each point of the cluster. % The cluster is identified by the vector belclust. function z = d ps min(x, pat, bel, i) z = 10000000; [N, l] = size(bel); for j = 1 : l if(bel(j) == i) temp = d eucl(x, pat(j, :)); if(z > temp) z = temp; end end end end ==========================

10 12.6. The data set produced when the seed for the randn function of MATLAB is zero, gives the plot of fig. 12.3. From this plot we conclude that the number of clusters underlying the data set is 2, since the range of values of Θ for which the number of clusters is 2 is greater than the corresponding range of any other cluster. However, for different choices of the seed of the randn function, the ranges for m = 2 and 3 are of comparable size.

Figure 12.3: Problem 12.6 12.7. The algorithms are as stated in the text with the following modifications: • The statements of the form d(xi, Ck ) = min d(xi, Cj ) 1≤j≤m

are replaced by s(xi, Ck ) = max s(xi, Cj ) 1≤j≤m

and • The statements of the form d(xi, Ck ) > (<)Θ

11 are replaced by s(xi, Ck ) < (>)Θ 12.8. The inequality f (x, w1 ) > f (x, w2 ) is equivalent to xT w1 − 0.5||w1 ||2 − 0.5||x||2 > xT w2 − 0.5||w2 ||2 − 0.5||x||2 The last equation, after some manipulations, leads to d2 (x, w1 ) < d2 (x, w2 ) 12.9. The neural network implementation for the BSAS algorithm in this case is the same with that described in the text (pp. 398-399) except the f or − loop, which is ommitted.

12.10. The MBSAS algorithm involves two stages. During the first one, the determination of the clusters takes place and during the second one, the vectors that have not been assigned to a cluster at the first stage are assigned to their closest cluster. Let f lag be a N × 1 vector, whose ith value indicates whether the ith vector has been assigned to a cluster during the first (1) or during the second phase (0). Initially, all values of the vector flag are set to zero. The paramaters a, wi, ti , m are defined as in the case of BSAS. Then MBSAS may be stated as follows: • Initialization – a=0 – wj = 0, j = 1, . . . , q – tj = 0, j = 1, . . . , q – f lagi = 0, i = 1, . . . , N – m=1 – For the first feature vector x1 set ∗ w1 = x1 ∗ a1 = 1

12 ∗ t1 = 1 • Main Phase – Repeat ∗ Present the next feature vector x to the network. ∗ Compute the output vector s(x).

P ∗ GAT E(x) = AN D((1 − qj=1 (sj (x))), sgn(q − m)). ∗ f lag(x) = GAT E(x) ∗ m = m + GAT E(x) ∗ am = am + GAT E(x) ∗ wm = wm + GAT E(x)x ∗ Tm = Θ − 12 ||wm||2 ∗ tm = 1 – Until all the vectors have been presented once to the network.

– Repeat ∗ Present the next feature vector x to the network. ∗ Compute the output vector s(x). ∗ For j = 1 to m · aj = aj + (1 − f lag(x))sj (x) · wj = wj + (1 − f lag(x))sj (x)( a1j (wj − x)) · Tj = Θ − 12 ||wj ||2 ∗ Next j – Until all feature vectors have been presented once to the network.

Solutions to the problems of Chapter 13 13.1. (a) This is clear since the Euclidean distance between two vectors is unique. (b) Let P (X) be the proximity matrix for a given data set X and D(X) be the corresponding pattern matrix. Consider the set X ′ = {y : y = x + a, x ∈ X, a ∈ Rl } Clearly, P (X) = P (X ′ ) and D(X) 6= D(X ′ ). Thus, P (X) corresponds to more than one pattern matrices.

Finally, let l = 1 and let us consider the proximity matrix 0  2  P =  1 4 

2 0 3 5

1 3 0 6

4 5   .

6  0 

It is clear that there is no way to put four points on a line satisfying the restrictions implied by P . Thus, P does not correspond to any pattern matrix.

13.2. Taking into account eq. (13.41), eq. (13.8) may be written as dqs =

ni nj ||(mi−mq )+(mq −ms)||2 + ||(mj −mq )+(mq −ms)||2 − ni + nj ni + nj

nj ni ||mi − mq ||2 − ||mj − mq ||2 = ni + nj ni + nj 2 ||mq − ms||2 + [ni (mi − mq )T (mq − ms) + nj (mj − mq )T (mq − ms)] ni nj 1

2 After some manipulations and taking into account eq. (13.40), the last term of the above equation gives 2 (ni mi − ni mq + nj mj − nj mq )T (mq − ms) = 0 ni + nj Thus, the proof of eq. (13.10) is completed. Notice that this equation holds only when the Euclidean distance is in use.

13.3. One such case is described below. Consider the following dissimilarity matrix   0 1.8 2.4 2.3 4.5    1.8 0 2.5 2.7 4     P (X) =   2.4 2.5 0 1.2 3.5  .  3   2.3 2.7 1.2 0  4.5 4 3.5 3 0 From the dendrogram shown in figure 13.1, which has been produced by the WPGMC algorithm when applied on the above matrix, we observe that d({x1 , x2 }, {x3 , x4 }) = 1.725 < min(d({x1 }, {x3 , x4 }), d({x2 }, {x3 , x4 })) = min(2.05, 2.3) = 2.05.

Figure 13.1: Problem 13.3 The above situation is closely related to the crossover phenomenon.

13.4. Indeed, if we multiply both sides of the equation given in the hint by

3 (ni +nj )ns ni +nj +ns

and after some rearrangements we obtain

ns ni ns nj (ni + nj )ns dqs = ||mi − ms||2 + ||mj − ms||2 − ni + nj + ns ni + nj + ns ni + nj + ns ns ni nj ||mi − mj ||2 (ni + nj + ns )(ni + nj ) Eq. (13.42) follows from the above equation if we multiply the nominator and the denominator of the first two terms by ni + ns and nj + ns , respectively and take into account eq. (13.13).

13.5. (a) It is X

x∈Cr

||x − mr ||2 =

X

x∈Cr

||x||2 +

X

||mr ||2 − 2

x∈Cr

X

xT mr

x∈Cr

Taking into account the definition of mr , the last term of the above equation becomes 2nr ||mr ||2 and the claim is proved. (b) Taking the squares of both sides of eq. (13.18), we obtain n2i ||mi||2 + n2j ||mj ||2 + 2ni nj miT mj = n2q ||mq ||2 Substituting in the above equation the term 2miT mj with ||mi||2 + ||mj ||2 − ||mi − mj ||2 and after some manipulations, we obtain nq ||mq ||2 = ni ||mi||2 + nj ||mj ||2 −

ni nj ||mi − mj ||2 nq

(13.1)

Substituting nq ||mq ||2 from the last equation to eq. (13.17), we obtain eq. (13.19).

13.6. The two dendrograms are the same with the one shown in figure 13.11.

13.7. The proof is by induction on the level of the hierarchy, t. At the level t = 0, each one of Cr and Cs contain a single element. Thus, eq. (13.43) holds trivially.

Assume that for t = λ, eq. (13.43) holds. We will prove the claim for t = λ + 1. Let Cp and Cq be the clusters at the λ level that are going to be

4 merged. Let aslo Cr = Cp ∪ Cq . It is clear that all the clusters Cs at the λ level, different from Cp and Cq , are contained also in the clustering of the λ + 1 level. Then, from eq. (13.4) we have d(Cr , Cs ) = min{d(Cp , Cs ), d(Cq , Cs )} Using the induction hypothesis, the above equation gives d(Cr , Cs ) = min{ min d(x, y), min d(x, y)} = x∈Cp ,y ∈Cs x∈Cq ,y ∈Cs min d(x, y) x∈Cp ∪Cq ,y ∈Cs Thus the claim is proved.

13.8. The proof is by induction on the level of the hierarchy, t. At the level t = 0, each one of Cr and Cs contain a single element. Thus, eq. (13.44) holds trivially.

Assume that for t = λ, eq. (13.44) holds. We will prove the claim for t = λ + 1. Let Cp and Cq be the clusters at the λ level that are going to be merged. Let also Cr = Cp ∪ Cq . It is clear that all the clusters Cs at the λ level, different from Cp and Cq , are contained also in the clustering of the λ + 1 level. Then, from eq. (13.5) we have d(Cr , Cs ) = max{d(Cp , Cs ), d(Cq , Cs )} Using the induction hypothesis, the above equation gives d(Cr , Cs ) = max{ max d(x, y), max d(x, y)} = x∈Cq ,y ∈Cs x∈Cp ,y ∈Cs max d(x, y) x∈Cp ∪Cq ,y ∈Cs Thus the claim is proved.

13.9. (a) In the case where similarity measures are used, the single link algorithm is given by the following equation s(Cr , Cs ) = max{s(Cp , Cs ), s(Cq , Cs )} where Cr = Cp ∪ Cq .

(13.2)

5 Statement: The similarity between two clusters Cr and Cs , d(Cr , Cs ), which are at the same level of the hierarchy produced by the single link algorithm, given by eq. (13.2), may be written as s(Cr , Cs ) =

max s(x, y) x∈Cr ,y ∈Cs

(13.3)

The proof of eq (13.3) is by induction of the level of the hierarchy, t. At the level t = 0, Cr and Cs contain a single element each. Thus, eq. (13.3) holds trivially.

Assume that for t = λ, eq. (13.3) holds. We will prove the claim for t = λ + 1. Let Cp and Cq be the clsuters at the λ level that are going to be merged. Let also Cr = Cp ∪ Cq . It is clear that all the clusters Cs at the λ level, different from Cp and Cq , are contained also in hierarchy of the λ + 1 level. Then, using eq (13.2), we have s(Cr , Cs ) = max{s(Cp , Cs ), s(Cq , Cs )} Using the induction hypothesis, the above equation gives s(Cr , Cs ) = max{ max s(x, y), max s(x, y)} = x∈Cq ,y ∈Cs x∈Cp ,y ∈Cs max s(x, y) x∈Cp ∪Cq ,y ∈Cs (b) In the case where similarity measures are used, the complete link algorithm is given by the following equation s(Cr , Cs ) = min{s(Cp , Cs ), s(Cq , Cs )}

(13.4)

where Cr = Cp ∪ Cq .

Statement: The similarity between two clusters Cr and Cs , s(Cr , Cs ), which are at the same level of the hierarchy produced by the complete link algorithm, given by eq. (13.4), may be written as s(Cr , Cs ) =

min s(x, y) x∈Cr ,y ∈Cs

(13.5)

The proof is similar with that of the single link case.

13.10. The dendrograms produced by the two algorithms are shown in fig.

13.2. We observe that no matter how the ties are considered, each algorithm

6

Figure 13.2: Problem 13.10 leads to the same proximity dendrogram. Remember here that this is true in general only for the single link algorithm and not for the complete link algorithm.

Also, both algorithms lead to the same hierarchy of clusterings. However, the clusterings in the complete link hierarchy are formed at higher levels of dissimilarity, compared with their counterparts in the single link case.

13.11. No matter which of the two edges is considered first, the resulting dendrogram is the one given in fig. 13.3. Observe that three clusters are joined into a single cluster at level 9.

13.12. (a) We consider the single and complete link algorithms based on the matrix theory. Applying the single link algorithm on P results in the dendrogram shown in fig. 13.4(a). Note that the tie vanishes after merging x4 and x5 to the same cluster.

The complete link algorithm leads to two different hierarchies. At the first stage of the algorithm, x4 and x5 come into a single cluster. Then, according to which of the ties we consider first, we obtain the dendrograms shown in fig. 13.4(a). (b) Although no ties exist in the initial proximity matrix, ties arise at a later stage and, thus we are led to the two dendrograms depicted in fig.

13.4(b). We notice that this phenomenon is not met in the single and the complete link algorithms.

7

Figure 13.3: Problem 13.11

Figure 13.4: Problem 13.12

8 13.13. It is clear that at the t step of the algorithm t + 1 clusters are formed.

From the hypothesis, t of these clusters consist of a single vector each and one contains the remaining N − t vectors of the data set X. Clearly, the next clustering will be produced by the partition of the last set into two sets, one with a single vector and the other with N − t − 1 vectors. Thus, at step t, N − t pairs of clusters have to be examined. Therefore, during the whole clustering process, the algorithm has to examine (N − 1) + (N − 2) + . . . + 1 =

N (N − 1) 2

pairs of clusters.

The advantage of such a clustering scheme is that it requires the comparison of a reduced number of pairs of clusters, compared to the general scheme.

However, each clustering resulting from such an algorithm has a very restrictive form, since all of its clusters consist of a single vector except one which contains the remaining vectors of the data set X. As a consequence, there is no guarantee that the algorithm will detect the optimum clustering at each step.

13.14 The alternative divisive algorithm does not guarantee the determination of the optimum clustering 1 at each step, since not all possible clusterings are considered.

Moreover, since the determination of the best possible clustering at each step requires an exponential number of comparisons between pairs of clusters, it is reasonable to deduce that no divisive algorithm of practical interest guarantees the determination of the optimum clustering.

13.15 Consider a point x ∈ C. From the definition of f (θ), x has nf (θ) neighbors, say y 1 , y 2 , . . . , y nf (θ) . Each of the y i ’s has nf (θ) neighbors (actually, we should count nf (θ) − 1 neighbors for y i , since x is one of its neighbors).

Taking into account the definition of link, x contributes to nf (θ) × nf (θ) = n2f (θ) links, approximately. Since C contains n points, the total number of links in C is (approximately) n × n2f (θ) = n1+2f (θ) .

13.16 Instead of determining a proper value of the threshold θ by trial and error, we “connect” it with a quantity that can be estimated more easily. In 1

We remind that in order to characterize a clustering, a specific criterion has to be employed, according to which all clusterings are evaluated.

9 the case of eq. (13.38), θ equals to the average distance between any two vectors in X plus λ times its standard deviation. Expressing θ in this way, we need to experiment with a range of values of λ around 0, which is rather small, since large positive (negative) values of λ lead to clusterings with a very small (large) number of clusters that are not expected to capture the underlying clustering structure of X.

Solutions to the problems of Chapter 14 14.1. The GMDAS in this case is the same as the one given in the book (section 14.2) except that eq. (14.10) is replaced by eq. (14.15).

14.2. The GMDAS in this case is the same as the one given in the book (section 14.2) except that eq. (14.10) is replaced by eqs. (2.84) and (2.85), where the index j in the a posteriori probabilities is substituted by Cj (see also problem 2.26).

14.3. (a) It is not difficult to show that taking the gradient of the log likelihood function L(θ) =

N X i=1

ln p(xi|θ)

(14.1)

with respect to θj , setting it equal to 0 and assuming functional independence between different θj ’s we are led to m N X X

i=1 j=1

P (Cj |xi; θ(t))

∂ ln p(xi|Cj ; θj ) = 0 ∂θ

(14.2)

This equation resembles very much to eq. (14.3). The only difference is that in this case the a posteriori probabilities depend only on the parameter vector while in eq. (14.3) they depend both on θj ’s and the a priori probabilities Pj ’s. Using an approach similar to that discussed in pages 443-446 of the book, the updating equations for the unknown parameters are given by eqs. (14.15) and (14.16) with the difference that the a posteriori probabilities depend only on the parameter vectors. (b) The estimates for the means µj ’s are computed as in the previous case. Let us now proceed to the derivation of the ML estimate of Σ.

1

2 Eq. (14.1) can be written as: L(θ) =

N X i=1

ln[

m X

j=1

p(xi|Cj ; θ)P (Cj )]

(14.3)

Taking the gradient of L(θ) with respect to Σ−1 and setting it equal to O we obtain: N m ∂L(θ) X 1 ∂ X = ( p(xi|Cj ; θ)P (Cj )) = O −1 ∂Σ−1 i=1 p(xi; θ) ∂Σ j=1

or, taking into account eq. (2.87) and after some manipulations m N X X

i=1 j=1

P (Cj |xi; θ)

∂ ln p(xi|Cj ; θ) = O ∂Σ−1

(14.4)

Let frs be the (r, s) element of Σ−1 . Taking into account eq. (14.13) and the fact that |Σ| = |Σ−1 |−1 , the derivative of ln p(xi|Cj ; θ) with respect to frs , after some manipulations, becomes 1 1 ∂ ln p(xi|Cj ; θ) σ − (xir − µjr )(xis − µjs ) = rs ∂frs 2|Σ−1 | 2 where σrs is the cofactor of frs and xir , µjr are the rth coordinates of xi and µj , respectively.

Since the cofactor matrix σ of Σ−1 can be written as σ = |Σ−1 |Σ, the above equation, after some manipulations, gives ∂ ln p(xi|Cj ; θ) = Σ − (xi − µj )(xi − µj )T ∂Σ−1 Substituting the above equation in eq. (14.4) and taking into account Pm P that N j=1 P (Cj |xi; θ) = N , we obtain, after a bit of algebra i=1 NΣ =

m N X X

i=1 j=1

It is

m N X X

i=1 j=1

P (Cj |xi; θ)(xi − µj )(xi − µj )T

P (Cj |xi; θ)xixiT =

N X i=1

xixiT

(14.5)

3 Also, taking into account the ML estimate of µj (part (a) of the problem), and assuming that P (xi) = 1/N m N X X

i=1 j=1

P (Cj |xi; θ)xiµj T =

N m X X

N m X X

[

j=1 i=1

P (Cj |xi; θ)µj µj T = N

j=1 i=1

P (Cj |xi; θ)xi]µj T = m X

P (Cj )µj µj T

j=1

Similarly we can show that m N X X

P (Cj |xi; θ)µj xiT = N

m N X X

P (Cj |xi; θ)µj µj T = N

i=1 j=1

m X

P (Cj )µj µj T

j=1

Finally,

i=1 j=1

m X

P (Cj )µj µj T

j=1

Substituting the last four equations to eq. (14.5), we obtain Σ=

m N X 1 X P (Cj )µj µj T xixiT − N i=1 j=1

(14.6)

14.4. Clearly, the first (second) group of 8 points belong to the circle with center (0, 0) ((6, 0)) and radius 2. The two circles do not intersect. The results of the algorithm depend heavily on the initialization of p(xi/Cj )’s.

For example, if we use as seed of the random number generator function the number 400 1 , the algorithm assigns the points of each circle to the same cluster. The estimated means an covariance matrices for this case are: µ1 = [0.03, 0.00]T , µ2 = [5.98, 0.00]T and Σ1 =

"

2.15 0.00 0.00 2.00

#

Σ2 =

"

2.15 0.00 0.00 2.00

#

On the other hand, if we use as seed for the random number generator the number 0, the algorithm groups the “upper” three points of each circle to 1

For the implementation of the algorithm we used MATLAB, so the seed for the function rand is 400.

4 the same cluster and the rest 10 points of the data set to the second cluster.

The means and covariance matrices for this case are µ1 = [3.00, 1.61]T , µ2 = [3.00, −0.95]T and Σ1 =

"

10.33 0.00 0.00 0.08

#

Σ2 =

"

11.40 0.00 0.00 0.70

#

In addition, different initial conditions, may lead to totally different clusterings. The above behavior indicates that GMDAS algorithm is a bad canditate for this case, since the data set contains no compact clusters, while GMDAS is capable of dealing with the case of compact clusters.

In the sequel we give MATLAB code for the GMDAS algorithm % This program simulates the probabilistic algorithm % for the book % m = number of clusters % N = number of patterns % l = dimension of the pattern space % cp = conditional probability matrix, P(Ci/xm) % ap = a priori probablility vector P(Ci) % mv = mean vector matrix mu i % mc = mean covariance matrix % e = the minimum error that has to be caught during the % convergence phase clear format compact whitebg e = 0.01; maxiter = 300; %These are the data for problem 14.4 pat = [2 0; sqrt(2) sqrt(2); 0 2; −sqrt(2) sqrt(2); −2 0; −sqrt(2) −sqrt(2); 0 − 2; sqrt(2) − sqrt(2)]; pat1 = pat + [6 ∗ ones(8, 1)zeros(8, 1)]; pat = [pat; pat1]; m = 2; [N, l] = size(pat); % With seed=400, we obtain a ”good” clustering rand(′ seed′ , 0) rand(′ seed′ )

5 % Initialization of the conditional probabilities for k = 1 : N cp(k, :) = rand(1, m); sumcp = sum(cp(k, :)); cp(k, :) = cp(k, :)/sumcp; end % The iteration scheme iter = 0; dif f = 10000; mvold = 10000 ∗ ones(m, l); mcold = 10000 ∗ ones(l, m ∗ l); apold = 10000 ∗ ones(1, m); while (iter < maxiter)&(dif f > e) iter = iter + 1; dif f vec = [dif f vecdif f ]; % Computation of the a priori probabilities temp = sum(cp); ap = temp/N ; % Computation of the mean vectors for i = 1 : m aux = zeros(1 : l); for k = 1 : N aux = aux + cp(k, i) ∗ pat(k, :); end mv(i, :) = aux/temp(i); end % Computation of the covariance matrices for i = 1 : m aux = zeros(l); for k = 1 : N aux = aux + cp(k, i) ∗ (pat(k, :) − mv(i, :))′ ∗ (pat(k, :) − mv(i, :)); end mc(:, (i − 1) ∗ l + 1 : i ∗ l) = aux/temp(i); end dif f = sum(abs(ap−apold))+sum(sum(abs(mv−mvold)))+sum(sum(abs(mc− mcold))); apold = ap;

6 mvold = mv; mcold = mc; % Computation of the determinants of the covariance matrices for i = 1 : m dete(i) = det(mc(:, (i − 1) ∗ l + 1 : i ∗ l)); end % Computation of the conditional probabilities for k = 1 : N for i = 1 : m auxvec(k, i) = dete(i)( − .5) ∗ exp(−.5 ∗ (pat(k, :) − mv(i, :)) ∗ inv(mc(: , (i − 1) ∗ l + 1 : i ∗ l)) ∗ (pat(k, :) − mv(i, :))′ ) ∗ ap(i); end sumaux = sum(auxvec′ ); end for k = 1 : N for i = 1 : m cp(k, i) = auxvec(k, i)/sumaux(k); end end end ==========================.

14.5. The current data set is the same as the one of the previous problem except that now the two circles intersect. The behavior of the algorithm in this case features the characteristics of that of the previous problem, since the data set contains no compact clusters, while GMDAS is capable to deal with compact clusters.

For example, when the seed of the rand equals to 0, the algorithm groups the “upper” three of each circle to the same cluster and the rest 10 points of the data set to the second cluster. The means and the covariance matrices in this case are µ1 = [1.00, 1.61]T , µ2 = [1.00, −0.95]T and Σ1 =

"

2.32 0.00 0.00 0.08

#

Σ2 =

"

3.40 0.00 0.00 0.70

#

14.6. (a) After 200 iterations, we obtain the following estimates for the means µj : µ1 = [−0.2295, −0.4440]T , µ2 = [1.6917, 1.9082]T , µ3 =

7 [4.3688, 0.1376]T , µ4 = [7.1179, −0.1839]T .

The resulting estimates for the covariance matrices are: Σ1 =

"

Σ3 =

#

0.63 −0.27 −0.27 0.57 "

2.57 0.03 0.03 0.50

Σ2 =

#

Σ4 =

"

"

1.16 0.21 0.21 0.94

0.43 0.05 0.05 1.69

#

#

(b)-(c) By assigning each feature vector to the Cj according to the Bayes decision rule, the resulting confusion matrix is 73 0 7 0  19 61 0 0     A1 =   5 2 72 1  0 0 14 66 



(d) The algorithm for m=3 gives the following estimates for the means and the covariance matrices: µ1 = [1.00, 0.97]T , µ2 = [4.42, 0.13]T , µ3 = [7.11, −0.1542]T and Σ1 =

"

1.81 1.00 1.00 2.05

#

Σ2 =

"

1.75 0.05 0.05 0.51

#

,

Σ3 =

"

9.47 0.04 0.04 1.59

#

The confusion matrix in this case is 80 0 0  76 4 0     A2 =   11 66 3  0 13 67 



¿From the above result one can easily claim that the majority of the points of the first two distributions fall to the first cluster. Moreover, the majority of the points of the third (fourth) distribution fall into the second (third) cluster.

The results of the algorithm for m = 2 are: µ1 = [0.97, 0.93]T , µ2 = [5.55, 0.07]T and Σ1 =

"

1.72 0.92 0.92 2.06

#

Σ2 =

"

3.05 −0.30 −0.30 1.12

#

8 Finally, the confusion matrix in this case is 80 0  72 8     A2 =   11 69  0 80 



Clearly, in this case, the majority of the points of the first (last) two distributions fall into the first (second) cluster.

14.7. The cost function for this example becomes Jqf uzzy (θ, U ) =

4 √ √ X (ui1 + ui2 10) + (ui1 10 + ui2 )

2 X i=1

i=3

We remind that the value of the cost function for the best hard clustering is 4. (a) Let q = 2. In order to be J2f uzzy < 4, it is sufficient to hold √ u2i1 + u2i2 10 < 1, for i = 1, 2. √ u2i1 10 + u2i2 < 1, for i = 3, 4.

Taking the first of the above inequalities and substituting ui1 with 1 − ui2 we obtain √ ( 10 + 1)u2i2 − 2ui2 < 0 or ui2 < 0.4805.

Thus, for ui2 ∈ [0, 0.48], i = 1, 2, the first inequality holds. In a similar fashion, we may prove that for ui1 ∈ [0, 0.48], i = 3, 4, the second inequality also holds. Thus, for the above choices of uij ’s, the value of J2f uzzy is less than 4. (b) Let q = 3. Using similar arguments as in the previous case, it is sufficient to hold √ u3i1 + u3i2 10 < 1, for i = 1, 2. √ u3i1 10 + u3i2 < 1, for i = 3, 4.

Substituting ui1 with 1−ui2 in the first of the above inequalities and after some√rearrangements we obtain: ( 10 − 1)u2i2 + 3ui2 − 3 < 0 or

9 ui2 < 0.6733.

Thus, for ui2 ∈ [0, 0.67], i = 1, 2, the first inequality holds. In a similar fashion, we may prove that for ui1 ∈ [0, 0.67], i = 3, 4, the second inequality also holds. Thus, for the above choices of uij ’s, the value of J3f uzzy is less than 4.  aii , i = 1, . . . , l     2a , where p corresponds to the monomial x x , i = l + 1, . . . , r

14.8. pi = 

jk

bi−r ,    c,

i

j k

i = r + 1, . . . , r + l i=s

14.9. Let us first rotate the curve such that the matrix A becomes diagonal.

Note that rotation does not affect the distance between z and the curve. The 12 angle of this rotation is d = 12 tan−1 a112a−a . Equations (14.42) and (14.44) 22 may be written now as z ′T A′ z ′ + z ′T b′ + c′ = 0 and

1 z ′ = (I − λA′ )−1 (2x′ + λb′) 2

respectively, where x = Rx′, z = Rz ′, A′ = RT AR, b′ = Rb, c′ = c and R=

"

cos d sin d − sin d cos d

#

Since now A′ is diagonal, (I − λA′ )−1 is also diagonal and, thus, easily invertible. Therefore, z ′ may be written as: z′ = [

λb′1 + 2x′1 λb′2 + 2x′2 T , ] 2(1 − λa′1 ) 2(1 − λa′2 )

Substituting z ′ from the above equation to eq. (14.42), we obtain a′11 (

′ ′ ′ ′ λb′1 + 2x′1 2 ′ λb′2 + 2x′2 2 ′ λb1 + 2x1 ′ λb2 + 2x2 ) a ( ) + b ( ) + b ( ) + c′ = 0 22 1 2 ′ ′ ′ ′ 2(1 − λa1 ) 2(1 − λa2 ) 2(1 − λa1 ) 2(1 − λa2 )

Requiring that λ 6= 1/a′1 , 1/a′2 , and multiplying both sides of the above equation with 4(1 − λa′1 )(1 − λa′2 ), the claim is proved.

10 14.10. Substituting z − c from eq. (14.48) to eq. (14.47) and after some manipulations, we obtain a= q

1 (x − c)T A(x − c)

Solving eq. (14.48) with respect to z and substituting to eq. (14.46), we obtain d2r (x, Q) = (1 − a)2 ||x − c||2 or, substituting a from above, d2r (x, Q) =

q

( (x − c)T A(x − c) − 1)2 (x − c)T A(x − c)

||x − c||2

(14.7)

Taking into account eq. (14.49), the above equation becomes d2r (x, Q) =

d2nr (x, Q) ||x − c||2 (x − c)T A(x − c)

Using the value of a computed above, the last equation, after some manipulations, may be written as d2r (x, Q) = d2nr (x, Q)||a(x − c)||2 Substituting a(x − c) from (14.48) to the above equation, the claim is proved.

14.11. (a) Substituting d2r from eq. (14.7) to eq. (14.55) and taking the derivatives with respect to cj and Aj , it is a matter of simple algebra to verify (14.56) and (14.57). (b) The parameter determination part of the FCES algorithm, for the tth iteration, is the following • For j = 1 to m – Solve the system of eqs. (14.56) and (14.57) with respect to cj and Aj and set cj (t) and Aj (t) equal to the repected solutions of the above system.

• End For-j

11 14.12. Eq. (14.105) can be written as Ja (θ, U ) =

m N X X

i=1 j=1

uqij (aj T Ri aj + bj T Ti bj + 2aj T SiT bj ) −

k X i=1

λj (aj T aj − 1)

(14.8) Taking the gradient of Ja with respect to bj , the second term of eq. (14.8) gives zero, while from the first term we obtain N ∂Ja X uqij (2Ti bj + 2Si aj ) = ∂bj i=1 a Setting ∂∂J b equal to 0 and taking into account the definitions of Hj and

j

Gj we obtain Hj bj + Gj aj = 0 The last equation easily leads to eq. (14.106).

Taking the gradient of Ja with respect to aj , the first term of the right hand side of eq. (14.8) gives N X

uqij (2Ri aj + 2SiT bj )

i=1

while the second term gives 2λj aj Thus, N ∂Ja X uqij (2Ri aj + 2SiT bj ) − 2λj aj = ∂aj i=1 a Setting ∂∂J aj equal to 0 and using the definitions of Fj , Hj , Gj , we obtain

Fj aj + GTj bj − λj aj = 0 Substituting bj with its equal from eq. (14.106) it follows that Fj aj − GTj Hj−1 Gj aj − λj aj = 0 The last equation easily leads to eq. (14.107).

12 14.13. (a) The MFCQS follows the Generalized Fuzzy Algorithmic Scheme (GFAS), where its updating equation for uij ’s is uij (t) = P

m k=1 (

1 1 xi,θj (t)) q−1 ) xi,θk(t))

d2p ( d2p (

The updating of θj ’s is carried out by solving the equation N X i=1

uqij

∂d2a (xi, θj ) =0 ∂θj

with respect to θj and setting θj (t) equal to this solution. (b) The two distances are in general different. However, we may assume that the estimates of the parameter vectors, θj ’s, when a C-shells quadric algorithm employing the dp distance is used, are approximately equal to those obtained from FCQS algorithm (where da is used), under the assumption that all the data vectors lie close to hyperquadric shells (see also example 14.7).

It is this assumption that gives a reasonable background to the MFCQS algorithm.

14.14. The two distances coincide.

14.15. Since for a hypersphere the matrix Aj equals to r12 I, where I is j the identity matrix, the normalized radial distance between a point x and a hypersphere Qj may be written as d2nr (x, Qj ) = (

1 1 ((x − cj )T (x − cj )) 2 − 1)2 rj

Taking into account eq. (14.50) and the fact that for a hypersphere the distance ||z − c|| is always equal to the radius of the hypersphere rj we may write 1 d2r (x, Qj ) = (((x − cj )T (x − cj )) 2 − rj )2 (a) Working as in the general case of the AFCS algorithm, we take the derivative of d2nr (x, Qj ) with respect to cj , i.e. ∂d2nr (x, Qj ) 1 dnr (x, Qj ) (x − cj ) =− ∂cj r ((x − cj )T (x − cj )) 21

13 In a similar fashion, for rj we obtain 1 ∂d2nr (x, Qj ) 1 = −2 2 dnr (x, Qj )((x − cj )T (x − cj )) 2 ∂rj r

Thus, cj (t) is computed by solving the equation N X i=1

uqij (t − 1)

dnr (xi, Qj ) 1

((xi − cj )T (xi − cj )) 2

(xi − cj ) = 0

with respect to cj and setting cj (t) equal to the determined solution.

Also, rj (t) is computed via the following equation uq (t − 1)((xi − cj )T (xi − cj )) rj = PNi=1 qij 1 T 2 i=1 uij (t − 1)((xi − cj ) (xi − cj )) PN

(b) Working as in the general case of the FCES algorithm, we take the derivative of d2r (x, Qj ) with respect to cj , i.e. dr (x, Qj ) ∂d2r (x, Qj ) =− 1 (x − cj ) ∂cj ((x − cj )T (x − cj )) 2 In a similar fashion, for rj we obtain ∂d2r (x, Qj ) = −2dr (x, Qj ) ∂rj Thus, cj (t) is computed by solving the equation N X i=1

uqij (t − 1)

dr (xi, Qj ) 1

((xi − cj )T (xi − cj )) 2

(xi − cj ) = 0

with respect to cj and setting cj (t) equal to the determined solution.

Also, rj (t) is computed via the following equation rj =

1 q T 2 i=1 uij (t − 1)((xi − cj ) (xi − cj )) PN q i=1 uij (t − 1)

PN

14.16. Taking the derivative of J1 with respect to uij and setting it equal to zero we obtain ∂J1 (θ, U ) = d(xi, θj ) + ηj [ln uij + 1 − 1] = 0 ∂uij

14 or

d(xi, θj ) ) ηj Also, taking the derivative of J1 with respect to θj , and setting it equal to zero we obtain uij = exp(−

N ∂d(xi, θj ) ∂J1 (θ, U ) X uij = =0 ∂θj ∂θj i=1

The algorithmic scheme obtained for the case is the same GPAS given in pages 475-476, except the equations used for the updating of uij (t) and θj , which become: uij (t) = exp(− and

N X i=1

uij (t − 1)

d(xi, θj (t)) ) ηj

∂d(xi, θj ) =0 ∂θj

14.17. The plot of uij versus d(xi, θj ) using eq. (14.70), for q = 1.1, 1.2, 1.5, 1.8, 2, 2.5, 3, 5 and eq. (14.78) is shown in fig. 14.1. It is clear that in the second case, uij decreases more rapidly with respect to d(xi, θj ), than in the first case.

14.18. (i) In the Isodata algorithm, we have the aspect of convergence, while in the variant of BSAS algorithm no convergence aspects are encountered. (ii) The number of clusters, m, remains constant to a prespecified value in the isodata algorithm. However, in the variant of the BSAS discussed here, m is subject to change after the presentation of each new vector to the algorithm. (iii) In the isodata algorithm the vectors are reassigned (if necessary) to other clusters at the end of each iteration. On the contrary, in the variant of BSAS, the data vectors are reassigned (if necessary) to other clusters only once (after the presentation of all vectors to the algorithm).

14.19. Eq. (14.108) can be written as [

N N X 1X uqij (xi − cj )(xi − cj )T uqij (xi − cj )T Σ−1 (x − c )]Σ = i j j j l i=1 i=1

15

Figure 14.1: Problem 14.17 Substituting eq. (14.109) to the above and after a bit of algebra, we obtain PN q T −1 i=1 uij (xi − cj ) Σj (xi − cj ) =l (14.9) PN q i=1 uij

Since Σj is a positive definite matrix it can always be diagonalized (see Appendix B), i.e. we can write ΦTj Σj Φj = Λj

(14.10)

where Φj is a matrix consisting of the (orthogonal) eigenvectors of Σj and Λj is the diagonal matrix with the eigenvalues of Σj on its diagonal. The last equation gives −1 T Σ−1 (14.11) j = Φj Λj Φj Substituting eq. (14.11) to eq. (14.9), and after some manipulations, we obtain PN q ′ ′ T −1 ′ ′ i=1 uij (xi − cj ) Λj (xi − cj ) =l (14.12) PN q i=1 uij

where Λj is the covariance matrix in the transformed space x′ = ΦTj x.

16 The above equation may be written as q Pl −1 ′ ′ 2 i=1 uij k=1 λjk (xik − cjk ) =l PN q i=1 uij

PN

(14.13)

where λjk is the kth diagonal element of Λj .

Substituting eq. (14.109) to (14.10) and after some manipulations, we obtain PN q ′ ′ ′ ′ T i=1 uij (xi − cj )(xi − cj ) Λj = (14.14) PN q i=1 uij Since Λj is diagonal we have λjk =

q ′ ′ 2 i=1 uij (xik − cjk ) PN q i=1 uij

PN

(14.15)

Substituting eq. (14.15) into (14.13), and after some manipulations, we prove the claim.

Solutions to the problems of Chapter 15 15.1. Suppose first that k = 2. The weights of the edges of the MST, the corresponding means and standard deviations, computed by the MST algorithm are summarized in the following table: w(xi,xj ) m(xi,xj ) σ(xi,xj ) (x1 , x2 ) 1 1 0 q 4 5 (x1 , x3 ) 1 3 3 (x3 , x4 ) (x4 , x5 ) (x5 , x6 )

1 3 1

(x5 , x7 )

1

7 5

1 5 3 5 3

q

4 5

0

q

4

q3 4 3

An edge is not inconsistent if |w(xi,xj ) − m(xi,xj ) | < qσ(xi,xj ) Clearly, the first edge is not inconsistent since w(xi,xj ) = m(xi,xj ) . Applying the above condition to the rest √ except (x √ edges √ 4 , x5 ) and solving with respect to q we obtain q > max{ 3/3, 5/5} = 3/3. Note √ that for all values of q, (x4 , x5 ) is an inconsistent edge. Thus, for q > 3/3, we have the formation of two clusters.

The bound for q are the same for the case k = 3. Also, the edge (x4 , x5 ) is inconsistent for all values of q.

It is worth noting that there are no values of q for which a single cluster is formed. (b) If conditions (15.2) and (15.3) are used, we obtain the RNG and GG graphs that consist of a single component. On the other hand, the number of 1

2 disconnencted components that we be obtained by using condition (15.3) or (15.4) depend on the value of σ. Specifically, if σ < 1, we obtain 7 clusters, each one containing a single vector of the data set X. Also, if σ ≥ 3, we obtain a single cluster, and, finally, if 1 ≤ σ < 3, we obtain the following two clusters: C1 = {x1 , x2 , x3 , x4 } and C2 = {x5 , x6 , x7 }. (c) The directed tree based clustering algorithm, when applied in the √ given data set, gives two clusters for 2 ≤ θ < 3.

15.2. (a) We will prove by induction on the number of updating circles that w1 (w2 ) is always no greater (no less) than −1 (1). (i) Suppose we are at the first updating circle. When x1 is presented to the algorithm w1 wins, since it is closer to x1 than w2 . The updating equation of the basic competitive learning algorithm gives w1 = −2η − 1 = −1.4. When x2 is presented to the algorithm w1 wins again and the updating equation, applied on the new value of w1 gives w1 = 2η 2 − 3η − 1 = −1.52(< −1).

When x3 is presented to the algorithm w2 wins since it is closer to x3 than w1 1 . Thus, the updating equation of the algorithm gives w2 = 1 + η = 1.2.

Finally, when x4 is presented to the algorithm, w2 wins since it is closer to x4 than w1 2 . Thus, we obtain w2 = −η 2 + 3η + 1 = 1.56.

Thus, the induction basis has been proved. (ii) Suppose that for the updating circle k the claim holds, i.e. w1 ≤ −1 and w2 ≥ 1. We will prove it for the next, k + 1, updating circle. When x1 (= −3) is presented to the algorithm (in the framework of the k + 1 updating circle), w1 wins, since it is closer to x1 than w2 . Thus, from the updating equation of the algorithm we obtain w1 = (1 − η)w1prev + ηx1 where w1prev is the value of w1 before the presentation of x1 in the k + 1 updating circle. Taking into account that from the induction hypothesis we have w1prev < 1 we obtain w1 < −2η − 1 = −1.4 Similarly, when x2 is presented to the algorithm, w1 (< −1.4) wins. Taking into account the updating equation of the algorithm and the last inequality, 1 2

Remember that we consider the updated value of w1 , i.e. −1.52.

In this comparison, we have w1 = −1.52 and w2 = 1.2.

3 we easily conclude that w1 < 2η 2 − 3η − 1 = −1.52 When x3 is presented to the algorithm w2 wins since it is closer to x3 than w1 (< −1.52). Thus, the updating equation combined with the induction hypothesis gives w2 > 1 + η = 1.2 Finally, when x4 is presented to the algorithm w2 wins and we have w2 > −η 2 + 3η + 1 = 1.56 Thus, the claim has been proved.

Note that since 0 < η < 1, w1 lies always on the right of x1 (= −3). This happens since if for a given updating circle w1 lies on the left of x1 , then, all the values of the previous updating circles would lie on the left of x1 , which is a contradiction since the initial value of w1 lies on the right of x1 . Using similar arguments, we can show that w2 lies always on the left of x4 (= 3).

Thus, at any updating circle we have that |w1 − x1 | ≤ 2 < 4 ≤ |w2 − x1 | |w1 − x2 | ≤ 1 < 3 ≤ |w2 − x2 | |w1 − x3 | ≥ 3 > 1 ≥ |w2 − x3 | |w1 − x4 | ≥ 4 > 2 ≥ |w2 − x4 |.

That is w1 (w2 ) wins always on x1 , x2 (x3 , x4 ). (b) w1 will not converge to −2.5, i.e. the mean of x1 and x2 , since at each updating circle of the algorithm, x2 is presented after x1 and, therefore, there is a bias towards x2 . Let us see this issue more formally. Let w1 (k) denote the value of w1 after the completion of the kth updating circle. Taking into account that w1 always wins on x1 and x2 , we can easily show that w1 (1) = (1 − η)2 w1 (0) + (1 − η)ηx1 + ηx2 w1 (2) = (1 − η)4 w1 (0) + ((1 − η)2 + 1)η((1 − η)x1 + x2 ) and, in general k−1 X

w1 (k) = (1 − η)2k w1 (0) + (

i=0

(1 − η)2i )η((1 − η)x1 + x2 )

4 or

1 − (1 − η)2k η((1 − η)x1 + x2 ) 1 − (1 − η)2 Taking the limit of w1 as k → ∞, we easily conclude that w1 (k) = (1 − η)2k w1 (0) +

1 ((1 − η)x1 + x2 ) = −2.445 k→∞ 2−η Following similar arguments it can be shown that limk→∞ w2 (k) = 2.556, i.e., the limiting value of w2 as k → ∞ is biased towards x4 , since this appears after x3 at each updating circle. lim w1 (k) =

15.3. In this case all representatives converge to the same vector. To see this in a more rigorous fashion, let y1 , y2 , . . . , yk, . . . be the sequence of vectors presented to the algorithm. Clearly, yk ∈ X and each sample of X appears more than once in the above sequence. Let also ηw = ηl = η. Writing the updating equation for the wi representative as wi(t) = (1 − η)wi(t − 1) + ηyj it is not difficult to show by induction that t

wi(t) = (1 − η) wi(0) + η

t X

q=1

(1 − η)t−q yq

The second term of the right hand side of the above equation is the same for all representatives. Taking the limit t → ∞, the first term of the right hand side of the above equation becomes 0. Thus, in the limit t → ∞ all representatives converge to the same vector.

15.4. (a) Writing the updating equation (15.75) for the winner as xk new wjk = (1 − η)wjk + η ηx and taking into account that l X

ηwjk = η

k=1

we can say that all coordinates of wj give up a proportion equal to η which is equally distributed to the coordinates of wj that are equal to 1, due to the term xηxk .

5 (b) If wj loses on x, it remains unchanged. If wj wins on x, we have l X

new wjk =

wjk + η(

l X xk

k=1 ηx

k=1

k=1

since ηx .

l X

Pl

k=1 wjk = 1 from hypothesis, and

−

l X

wjk ) = 1

k=1

Pl

xk k=1 ηx = 1, from the definition of

15.5. Assume that xr+1 is assigned to the jth cluster. From the definition of J(Cr ), the term ∆J(Cr ) = J(Cr+1 ) − J(Cr ) can be written as B

A

z

}|

{

2

∆J(Cr ) = ||xr+1 − mj (Cr+1 )|| +

z r X i=1

}|

2

2

{

(||xi − mci (Cr+1 )|| − ||xi − mci (Cr )|| ) (15.1)

The term B in the above equation equals to B=

X

i=1,...,r, ci =j

(mj (Cr+1 ) − mj (Cr ))T (mj (Cr+1 ) + mj (Cr ) − 2xi)

or B = (mj (Cr+1 ) − mj (Cr ))T (nj (Cr )(mj (Cr+1 ) + mj (Cr ) − 2mj (Cr ))) and, finally, B = nj (Cr )||mj (Cr+1 ) − mj (Cr )||2

(15.2)

Formulating eq. (12.5) in the present context, we have mj (Cr+1 ) =

nj (Cr )mj (Cr ) + xr+1 nj (Cr ) + 1

(15.3)

Substituting mj (Cr+1 ) from the above equation to the term A of eq. (15.1) we obtain n2j (Cr ) A= ||xr+1 − mj (Cr )||2 (15.4) (nj (Cr ) + 1)2 Moreover, substituting mj (Cr+1 ) from eq. (15.3) to eq. (15.2), and after some manipulations, we obtain B=

nj (Cr ) ||xr+1 − mj (Cr )||2 2 (nj (Cr ) + 1)

(15.5)

6 Finally, combining eqs. (15.1), (15.4) and (15.5), the claim follows.

15.6. (a) We run the algorithm for r = 34, . . . , 76 and we see that for r between 42 and 68, the number of clusters remains constant and equals to 3. This may be visualized, if we plot the number of clusters, m versus r, we obtain a very flat region for m = 3. We adopt the value 50 for r and the results obtained by the algorithm are shown in the following confusion matrix   199 0 1 200 0  A=   0 0 1 199 As we can see, the algorithm separates the vectors from the three distribution almost perfectly. (b) The results obtained by GHAS when appied on the above data set for m = 3 are shown in the following confusion matrix 



0 1 199  ′ A =  0 200 0   199 1 0 Clearly, both algorithms work very satosfactory in this case. However, for this problem, where we deal with compact clusters, GHAS is more preferable due to the reduced computational requirements.

In the sequel we give MATLAB code for the BMCA algorithm. % This is the main program implementing the BMCA % algorithm. % We commence with the first tw stages of the algorithm. % S = the structuring element % R = the resolution parameter % R tot = array with all the values of R % nofcl= array with the number of clusters that are derived % for the corresponding value of R in R tot clear; format compact; whitebg hold off; S = [−1 − 1; −1 0; −1 1; 0 − 1; 0 0; 0 1; 1 − 1; 1 0; 1 1]; nof cl = [];

7 t0 = clock; randn(′ seed′ , 0); diaspora = sqrt(2); a = gaussgen(−1, −1, diaspora, 2, 200); a1 = gaussgen(6, 3, diaspora, 2, 200); a2 = gaussgen(−0.7, 7, diaspora, 2, 200); a = [a; a1; a2]; plot(a(:, 1), a(:, 2),′ r∗′ ) R tot = []; for R = 52 : 2 : 150 R R tot = [R tot; R]; b = normali(a, R); c = dbinset(b); f = opening(c, S, R); g = closing(f, S, R); %WARNING: The plots are valid only for two-dimensional % data subplot(221) hold off plot(a(:, 1), a(:, 2),′ r.′ ) subplot(222) hold off axis([0R − 10R − 1]); axis(′ square′ ) plot(c(:, 1), c(:, 2),′ r.′ ) subplot(223) hold off axis([0R − 10R − 1]); axis(′ square′ ) plot(f (:, 1), f (:, 2),′ r.′ ) subplot(224) hold off axis([0R − 10R − 1]); axis(′ square′ ) plot(g(:, 1), g(:, 2),′ r.′ ) % ’s3r25ne’ means: new symmetric structuring element % 3x3, R=25 % ’mys3re25’ means: old asymmetric structuring element

8 % 3x3, R=25 % ’gs3r20ne’ means: data stemed from a gaussian % distribution, new symmetric structuring element 3x3, % R=20 % This procedure assigns each pattern to the appropriate % cluster. This is the 3rd and last step of the % algorithm.

[N, l] = size(b); [m, l] = size(g); h = clusass(g); save tempor g h pause end for i = 1 : N if(belclust(i, :) == 0) z = mindist(b(i, :), cores); belclust(i) = belcores(z(2)); end end class1co = 0; class2co = 0; class3co = 0; for i = 1 : N if((i <= 200)&(belclust(i) == 1)) class1co = class1co + 1; end if((i > 200)&(i <= 400)&(belclust(i) == 2)) class2co = class2co + 1; end if((i > 400)&(belclust(i) == 3)) class3co = class3co + 1; end end class1co class2co class3co time = etime(clock, t0) % ’ns3r25cl’ means: new symmetric structuring element

9 % 3x3, R=25, results of clustering save ns3r25cl class1co class2co time; ========================== NAME OF THE FUNCTION: gaussgen.m % This code generates ”samples” 2-dimensional vectors % stemed from a gaussian distribution with center (a,b) % and the identity times d, as the covariance matrix function z = gaussgen(a, b, d, l, samples) c = [ab]; z = (c′ ∗ ones(1, samples))′ + d ∗ randn([samples, l]); end ========================== NAME OF THE FUNCTION: normali.m % This function normalizes a given data set such that % each coordinate of a data pattern lies in the range % [0,R-1] function z = normali(pat, R) [N, l] = size(pat); for i = 1 : l minim(i) = min(pat(:, i)); maxim(i) = max(pat(:, i)); end for i = 1 : l z(:, i) = ((pat(:, i) − minim(i))/(maxim(i) − minim(i))) ∗ (R − 1); end end ========================== NAME OF THE FUNCTION: dbinset.m % This function takes as input pattern matrix and % finds the corresponding discrete binary set function z = dbinset(pat) [N, l] = size(pat); for i = 1 : N

10 for j = 1 : l pat(i, j) = f loor(pat(i, j)); end end pati(1, :) = pat(1, :); count = 1; for i = 2 : N ktemp = 0; for j = 1 : count if(pat(i, :) == pati(j, :)) ktemp = 1; end end if(ktemp == 0) count = count + 1; pati(count, :) = pat(i, :); end end z = pati; end ========================== NAME OF THE FUNCTION: opening.m % This function performs the opening % operation of ”dbset” with the % structuring element ”S” function z = opening(dbset, S, R) a = dilation(dbset, S, R); z = erosion(a, S, R); end ========================== NAME OF THE FUNCTION: closing.m % This function performs the closing % operation of ”dbset” with the % structuring element ”S” function z = closing(dbset, S, R) a = erosion(dbset, S, R);

11 [N, l] = size(a); if(N ∗ l > 0) z = dilation(a, S, R); end end ========================== NAME OF THE FUNCTION: dilation.m % This function computes the dilation of a set dbset % using the structuring element S function z = dilation(dbset, S, R) [N, l] = size(dbset); [n, l] = size(S); count = 0; for i = 1 : N for j = 1 : n temp = dbset(i, :) + S(j, :); if((temp >= zeros(1, n))&(temp <= ones(1, n) ∗ (R − 1))) count = count + 1; tempi(count, :) = temp; end end end z = discard(tempi); end ========================== NAME OF THE FUNCTION: erosion.m % This function erodes a set ”dbset” with % a structuring element ”S” % WARNING: This procedure deals only with % two dimensional data function z = erosion(dbset, S, R) [N, l] = size(dbset); [n, l] = size(S); totmat = zeros(R, R); for i = 1 : N

12 for j = 1 : n temp = dbset(i, :) + S(j, :); if((temp >= zeros(1, n))&(temp <= ones(1, n) ∗ (R − 1))) totmat(temp(1) + 1, temp(2) + 1) = totmat(temp(1) + 1, temp(2) + 1) + 1; end end end count = 0; for i = 1 : R for j = 1 : R if(totmat(i, j) == l) count = count + 1; z(count, :) = [i − 1j − 1]; end end end end ========================== NAME OF THE FUNCTION: clusass.m % This is the function that determines the clusters % in the discrete binary set % NOTE: THIS CODE IS HAS NOT BEEN OPTIMIZED % immed = parameter that defines the immediate neighborhood % of a given vector % n of clus = the number of clusters determined in the % discrete binary set % proc = array whose i-th component indicates if the i-th % vector has been processed (1) or not (0). function z = clusass(g) n of clus = 0; [N, l] = size(g); proc = zeros(N, 1); belclust = zeros(N, 1); dq = sqrt(l); theta = 2; immed = sqrt(l); for i = 1 : N

13 if(proc(i) == 0) proc(i) = 1; count = 0; ind = []; for j = 1 : N if(d(g(i, :), g(j, :)) <= dq)&( (j == i)) count = count + 1; ind = [indj]; end end if(count >= theta) [i1, i2] = size(ind); max bel = belclust(ind(1)); max ind = ind(1); min bel = belclust(ind(1)); min ind = ind(1); for k = 2 : i2 if(belclust(ind(k)) > max bel) max bel = belclust(ind(k)); max ind = ind(k); end if(belclust(ind(k)) < min bel) min bel = belclust(ind(k)); min ind = ind(k); end end if(max bel == 0) n of clus = n of clus + 1; belclust(i) = n of clus; for k = 1 : i2 belclust(ind(k)) = n of clus; proc(ind(k)) = 1; end else if(min bel == 0) belclust(i) = max bel;

14 for k = 1 : i2 belclust(ind(k)) = max bel; proc(ind(k)) = 1; end else belclust(i) = min bel; for k = 1 : i2 belclust(ind(k)) = min bel; proc(ind(k)) = 1; end end end end end end iter = 0; f lag = 1; while( (f lag == 0)) iter = iter + 1 f lag = 0; for i = 1 : N for j = i + 1 : N if(d(g(i, :), g(j, :)) <= dq)&( (belclust(i) == belclust(j))) f lag = 1 [min b, min ind] = min([belclust(i)belclust(j)]); [max b, max ind] = max([belclust(i)belclust(j)]); end for k = 1 : N if(belclust(k) == max b) belclust(k) = min b; end end end end

15 end z = belclust′ ; end ========================== NAME OF THE FUNCTION: d.m % This function computes the Euclidean distance % between two vectors function z = d(x, y) z = sqrt((x − y) ∗ (x − y)′ ); end

15.7. The clustering produced by the isodata algorithm is shown in fig. 15.1.

Clearly, the algorithm fails to identify the two clusters underlying the data set. This happens since these clusters are non compact while the isodata algorithm is suitable when compact clusters are present.

Figure 15.1: Problem 15.7 In the sequel we give MATLAB code for the Isodata algorithm. % This program implements the isodata algorithm % pat = each row of this matrix corresponds to a data

16 % vector % belclust = a vector whose i-th component corresponds % to the cluster where the i-th vector belongs clear; format compact pat = myexamp(0, 0, −10, 3, 10, 500); k = 2; belclust = isodata(pat, k); ========================== NAME OF THE FUNCTION: myexamp.m % This is my example for constructing half moon % clusters with centers (a1,a2), (b1,b2) and % radius r. Each cluster contains q vectors. function z = myexamp(a1, a2, b1, b2, r, q) N = 2 ∗ q; x = (a1 − r) : (2 ∗ r)/(q − 1) : (a1 + r); pat(:, 1) = x′ ; for i = 1 : q pat(i, 2) = a2 + sqrt(r2 − (pat(i, 1) − a1)2 ) + randn; end x = (b1 − r) : (2 ∗ r)/(q − 1) : (b1 + r); pat1(:, 1) = x′ ; for i = 1 : q pat1(i, 2) = b2 − sqrt(r2 − (pat1(i, 1) − b1)2 ) + randn; end pat = [pat; pat1]; z = pat; end ========================== NAME OF THE FUNCTION: isodata.m % This function implements the isodata algorithm. % It takes as input the pattern matrix pat and the % desired number of clusters m. function z = isodata(pat, m) [N, l] = size(pat);

17 rand(′ seed′ , 0) % Initial assignment of patterns to clusters belclust = m ∗ rand(1, N ); belclust = f ix(belclust) + 1; for i = 0 : 5 ∗ m − 1 belclust(i + 1) = f ix(i/5) + 1; end % Estimate of the initial centers meank = zeros(m, l); index = zeros(1, m); for i = 1 : N index(belclust(i)) = index(belclust(i)) + 1; meank(belclust(i), :) = meank(belclust(i), :) + pat(i, :); end for j = 1 : m meank(j, :) = meank(j, :)/index(j); end e = 1; iter = 0; % Main iterative phase of the algorithm while (e == 0) iter = iter + 1; meankold = meank; %Assignment of the patterns to clusters for i = 1 : N tempbel = [belclust(1 : i − 1)belclust(i + 1 : p)]; qwe = abs(tempbel − belclust(i)); if(prod(qwe) == 0) u = d(pat(i, :), meank(1, :)); ind = 1; for j = 2 : m temp = d(pat(i, :), meank(j, :)); if(temp < u) u = temp; ind = j; end

18 end belclust(i) = ind; end end % Estimation of the cluster centers meank = zeros(m, l); index = zeros(1, m); for i = 1 : N index(belclust(i)) = index(belclust(i)) + 1; meank(belclust(i), :) = meank(belclust(i), :) + pat(i, :); end for j = 1 : m meank(j, :) = meank(j, :)/index(j); end e = sum(sum(abs(meank − meankold))); end z = belclust; % Plot the clusters. This works only for % 2-dimensional data for i = 1 : N if(belclust(i) == 1) plot(pat(i, 1), pat(i, 2),′ +r′ ) hold on elseif(belclust(i) == 2) plot(pat(i, 1), pat(i, 2),′ og ′ ) hold on elseif(belclust(i) == 3) plot(pat(i, 1), pat(i, 2),′ ∗b′ ) hold on elseif(belclust(i) == 4) plot(pat(i, 1), pat(i, 2),′ ∗y ′ ) hold on end end end

19 ========================== NAME OF THE FUNCTION: d.m % This function computes the Euclidean distance between % two patterns of the same size. function z = d(x, y) z = (x − y) ∗ (x − y)′ ; end ========================== .

15.8. (a) Since the exponent q in the second term of eq. (15.23) is a positive P 2 integer and the term ( N1 N i=1 f (g(xi; θ))) is less than 1 (from the definition of f (.)), we have that J(θ) ≥

N N 1 X 1 X f 2 (g(xi; θ)) − ( f (g(xi; θ)))2 N i=1 N i=1

Adding and subtracting the term ( N1 side of the above inequality we have

PN

2 i=1 f (g(xi; θ))) to the right hand

N N 1 X 1 X f 2 (g(xi; θ)) − ( f (g(xi; θ)))2 = N i=1 N i=1

N N N N 1 X 1 X 1 X 1 X f (g(xk; θ)))+( f 2 (g(xi; θ))−2( f (g(xi; θ)))( f (g(xi; θ)))2 = N i=1 N i=1 N k=1 N i=1

N N N N N 1 X 1 X 1 X 1 X 1 X 2 f (g(xk; θ))+ f (g(xk; θ)))2 = f (g(xi; θ))− 2f (g(xi; θ)) ( N i=1 N i=1 N k=1 N i=1 N k=1 N N 1 X 1 X f (g(xi; θ)))2 ≥ 0 (f (g(xi; θ)) − N i=1 N k=1

Thus, the claim is proved. (b) The gradient of J(θ) with respect to θj , is computed by applying the chain rule.

20 15.9. The general form of the updating equation (15.26) holds also for this case. However, in this case, the gradient of g(x, θ) with respect to the components of θ becomes ∂g(x,θ ) = 1, ∂w0 ∂g(x,θ ) = xi , i = 1, . . . , l ∂wi ∂g(x,θ ) = xs xr , s, r = 1, . . . , l. ∂wsr

15.10. (a) We run the BDA algorithm on X using as seed for the rand function of the MATLAB the value 0. Also, we set ε = 0.05, q = 1 and µ = 0.001. The algorithm converges after 6 iterations to the solution θ = [−0.03, −0.20, 0.58]T , for which J(θ) = 0.0837. This solution separates perfectly the vectors from the two distributions (see fig. 15.2).

Figure 15.2: Problem 15.10 (b) Running the BDA algorithm for the first of the above clusters, using the same settings as above, the value obtained for J(θ) is 0.0091. Repeating this procedure for the second cluster, we have J(θ) = 0.0056. Notice that both values are significantly less than the corresponding value computed at the first step. This may be taken as an indication that X + and X − do not contain clusters and, as a consequence, that X contains only two clusters.

21 However, we remind that the BDA algorithm is a steepest ascent method, thus, it converges to a local minimum of the function J(θ), which depends on the initial estimate of θ. In addition, it turns out that BDA is very sensitive to the initial conditions. Indeed, if we run BDA for different initial conditions, there are cases where BDA cannot separate the vectors of the two distributions satisfactorily.

In the sequel we give MATLAB code for the BDA algorithm. % This function returns the clustering produced by the % BDA algorithm when applied on the data set pat % It is assumed that the boundary used is a hyperplane % and the squashing function is the output function % This is the main program for the BDA algorithm % e = the parameter that is used in the termination % criterion of the BDA algorithm % r = the updating parameter % pat = each row of this matrix corresponds to the vector % of the data set % The function data1510 generates the data described in problem 15.10. clear; format compact; pat = data1510; e = 0.05; q = 1; r = 0.001; sed = −1; dif = 200; while (sed < 0) sed = sed + 1; [N, l] = size(pat); rand(′ seed′ , sed); w = rand(1, l); J new = 10000; iter = 0; rat = 1000; while rat > e, iter = iter + 1

22 J old = J new for j = 1 : l w new(j) = update(j, pat, w, r, q); end w = w new; J new = J(pat, w, q); rat = abs((J new − J old)/J old); end ppclust = zeros(1, 2); belclust = zeros(1 : 200); for i = 1 : N if(pat(i, :) ∗ w′ > 0) ppclust(1) = ppclust(1) + 1; belclust(i) = 1; else ppclust(2) = ppclust(2) + 1; belclust(i) = 0; end end pat1 = []; pat2 = []; for i = 1 : N if(belclust(i) == 1) pat1 = [pat1; pat(i, :)]; else pat2 = [pat2; pat(i, :)]; end end for i = 1 : N if(belclust(i) == 1) plot(pat(i, 1), pat(i, 2),′ r+′ ) hold on else plot(pat(i, 1), pat(i, 2),′ go′ ) hold on end

23 end pause hold off end save res1510 iter w ppclust sed pat1 pat2 ========================== NAME OF THE FUNCTION: data1510.m % This function creates the data set. Also, it adds to % all the vectors an additional coordinate which equals % to 1 function z = data1510 randn(′ seed′ , 0) a = gaussgen(0, 0, 1, 2, 100); a1 = gaussgen(5, 5, 1, 2, 100); a = [a; a1]; a = [aones(200, 1)]; z = a; end ========================== NAME OF THE FUNCTION: gaussgen.m % This code generates 2-dimensional vectors steming % from a gaussian distribution with center (a,b) % and the identity matrix times d, as the covariance % matrix function z = gaussgen(a, b, d, n, samples) c = [ab]; z = (c′ ∗ ones(1, samples))′ + sqrt(d) ∗ randn([samples, n]); end ========================== NAME OF THE FUNCTION: update.m % This function updates a single coordinate of the % parameter vector function z = update(j, pat, w, r, q) [N, l] = size(pat);

24 sum1 = 0; sum2 = 0; sum3 = 0; for i = 1 : N sum1 = sum1 + f (g(pat(i, :), w)) ∗ f der(g(pat(i, :), w)) ∗ pat(i, j); sum2 = sum2 + f (g(pat(i, :), w)); sum3 = sum3 + f der(g(pat(i, :), w)) ∗ pat(i, j); end z = w(j) + r ∗ ((2/p) ∗ sum1 − (2 ∗ q/p) ∗ sum2( 2 ∗ q − 1) ∗ sum3); end ========================== NAME OF THE FUNCTION: f der.m % This is the derivative of the hyperbolic tangent function z = f der(x) [N, l] = size(x); for i = 1 : l if(x(i) > 0) z(i) = exp(−x(i)/2)/(1 + exp(−x(i)/2))2 ; else z(i) = exp(x(i)/2)/(1 + exp(x(i)/2))2 ; end end end ========================== NAME OF THE FUNCTION: f.m %This is the hyperbolic tangent function function z = f (x) z = tanh(x/2); end ========================== NAME OF THE FUNCTION: g.m % This is the g(x,w) function, that corresponds to a % hyperplane

25 function z = g(x, w) z = w ∗ x′ ; end ========================== NAME OF THE FUNCTION: J.m % This function implements the function J(w). % It takes as input the data set as well as the % parameter vector w and the control parameter q. function z = J(pat, w, q) [N, l] = size(pat); suma = 0; sum tetr = 0; for i = 1 : N suma = suma + f (g(pat(i, :), w)); sum tetr = sum tetr + (f (g(pat(i, :), w)))2 ; end suma = suma/p; sum tetr = sum tetr/p; z = sum tetr − suma( 2 ∗ q); end ==========================

15.11. (a) The shape of V (x) in this case is hyperellipsoidal. (b) d(x, y) must be defined as the l1 distance. (c) The shape of the neighborhood affects the results of the valley seeking algorithm. Consider for example the data set X = {x1 , x2 , x3 , x4 , x5 , x6 }, with x1 = [1, 1]T , x2 = [2, 1]T , x3 = [3, 1]T , x4 = [1, 2]T , x5 = [2, 2]T , x6 = [3, 2]T . Initially, x1 , x2 , x3 are assigned to the first cluster and x4 , x5 , x6 are assigned to the second cluster. Let a = 1. Ties are broken first in favor of the current cluster and then in favor of the smallest index.

Using the hyperspherical version of V (x), the clustering remains unchanged.

However, the algorithm does not converge when the hypercubical version of V (x) is used. Specifically, the algorithm oscillates between the two solutions i.e. C1 = {x1 , x2 , x3 }, C2 = {x4 , x5 , x6 } and C1 = {x4 , x5 , x6 }, C2 = {x1 , x2 , x3 }.

26 15.12. We set a = 2. Also, referring to pp. 528-529, if for a given vector we have more than one kqi ’s with maximum value, we choose the one with the smallest index q. Applying now the algorithm on the given data set and for the given initial clustering, we obtain after two iterations a single cluster that contains all the data vectors (C2 becomes empty). This is reasonable since this algorithm seeks for modes in the data points space and the vectors of X form a single mode in this space. Note that although the algorithm is initialized with two clusters, it ends up with a single one.

Suppose now that if for a given vector we have more than one kqi ’s with maximum value, we choose the one with the largest index q. In this case, the algorithm gives also one cluster after 8 iterations. The only (minor) difference is that C1 is now empty.

15.13. Substituting the given values in eq. (15.33) we obtain the number of sweeps, which is approximately equal to 22027.

15.14. Initially, when β = 0, all the vectors of the data set are represented by a single representative, that is all vectors belong to a single cluster. As β increases and reaches a critical value a phase transition takes place. This indicates that the whole data set can no longer be represented satisfactorily by a single vector. Thus, we introduce the necessary number of new representatives. Clearly, continuing this procedure (as β increases), the number of representatives increases according to the current needs and, finally, it reaches the appropriate number of representatives that represent the data satisfactorily. Thus, there is no need to make any a priori guesses for the number of clusters underlying the data set.

Considering the above procedure from a different perspective, one can say that the above procedure leads to a hierarchy of clusterings, i.e., the above procedure may be viewed as a hierarchical clustering algorithm. However, in most of the cases, the produced hierarchy is not expanded through the trivial clustering where each vector belongs to a single cluster (see [Rose91]).

15.15. Let c(xi) = j if xi belongs to cluster Cj . Then a solution may be coded as a N -dimensional vector [c(x1 ), c(x2 ), . . . , c(xN )] Cutting with the crossover operator is allowed at any place in the above solution. Also, the mutation operator should be modified. One such modification

27 is the following: select randomly a coordinate of a solution for mutation and set it equal to one of the remaining clustering indices according to the prespecified mutation probability.

A merit of this coding is that it affects directly the clustering of the vectors of X. However, the number of clusters may be affected due to the crossover operator. More sophisticated operators, such as the strucutral crossover have also been proposed (see [Mich 94]).

Solutions to the problems of Chapter 16 16.1. Since q 6= m, the exists at least one pair of vectors that is either SD or DS, i.e., b ≥ 1 or c ≥ 1. Thus a+d <1 R = a+b+c+d a <1 J = a+b+c

FM =

q

a a < 1, since either a < a + b or a < a + c. a+b a+c

16.2. By the definition of X and Y it is not difficult to see that 1 , where m1 = a + b and M = a + b + c + d, µX = m M 2 , µY = m M √ m1 (M −m1 ) σX = , √ M m2 (M −m2 ) , σY = M

PN −1 PN i=1

j=i+1 X(i, j) = m1 ,

PN −1 PN i=1

j=i+1 Y (i, j) = m2 ,

PN −1 PN i=1

j=i+1 X(i, j)Y (i, j) = a.

Substituting the above into eq. (16.9) and after a bit of algebra we obtain (16.10).

16.3. (a) In order to estimate the distribution of the R, J, F M and Γ̂ statistics, we generate 11 groups Gj , j = 1, . . . , 11 each one consisting of 100 data sets. All these groups were generated from MATLAB using as seeds for the rand function, the values 0, 1, 11, 21, . . . , 91.

We repeat the example 16.2 for various data sets that comply with the given specifications. Specifically, we use as seed to the randn function of the 1

2 MATLAB the values 0, 1, 11, 21, . . . , 91. The external information in the case is: “The vectors of the data set X belong to four different groups P1 , P2 , P3 and P4 , such that P1 contains the first 25 vectors of X and P2 , P3 and P4 contain the second, the third and the fourth groups of 25 vectors of X, respectively”.

In all cases, all indices indicate that the null hypothesis is rejected at significance level ρ = 0.05, for all Gi ’s. (b) Again we used different data sets that comply with the given specifications, using the same seeds as above. In this case, the results are different.

Specifically, for the groups G1 , G7 and G11 , the null hypothesis is accepted by all indices, at significance level ρ = 0.05. On the contrary, for the groups G2 , G3 , G5 , G6 , G9 , G10 , the null hypothesis is rejected by all indices at the same significance level. Finally, it seems that no clear cut situations exist for the groups G4 and G8 . More specifically, for G4 the null hypothesis is not rejected according to Jaccard and Fowlkes-Mallows index, while it is rejected form the other two. Also, for G8 only the Rand index does not reject the null hypothesis is accepted only by the Rand index.

16.4. Clearly, we have N N −1 X −1 X 1 NX 1 NX 2 0≤ [a(dij − µp ) + (cij − µc )] = [ (dij − µp )2 ]a2 + M i=1 j=i+1 M i=1 j=i+1

2[

N N −1 X −1 X 1 NX 1 NX (dij − µp )(cij − µc )]a + (cij − µc )2 M i=1 j=i+1 M i=1 j=i+1

Since the above quantity is a non negative quadratic for any value of a, its discriminant is non positive, i.e., 4[

N N N −1 X −1 X −1 X 1 NX 1 NX 1 NX (dij −µp )(cij −µc )]2 −4( (dij −µp )2 )( (cij −µc )2 ) M i=1 j=i+1 M i=1 j=i+1 M i=1 j=i+1 (16.1) ≤0 (16.2)

Also, N N N −1 X −1 X −1 X 1 NX 1 NX 1 NX (dij − µc )(cij − µc ) = dij cij − ( dij )µc M i=1 j=i+1 M i=1 j=i+1 M i=1 j=i+1

3 −(

N N −1 X −1 X 1 NX 1 NX cij )µp + µc µp = dij cij − µp µc M i=1 j=i+1 M i=1 j=i+1

In a similar fashion, we can show that N N −1 X −1 X 1 NX 1 NX (dij − µp )2 = d2 − µ2p M i=1 j=i+1 M i=1 j=i+1 ij

and

N N −1 X −1 X 1 NX 1 NX (cij − µc )2 = c2 − µ2c M i=1 j=i+1 M i=1 j=i+1 ij

Taking into account the last two equations, inequality (16.2) gives [

N N N −1 X −1 X −1 X 1 NX 1 NX 1 NX dij cij − µp µc ]2 ≤ ( d2ij − µ2p )( c2 − µ2c ) M i=1 j=i+1 M i=1 j=i+1 M i=1 j=i+1 ij

A few manipulations in the last inequality prove the claim.

16.5. The cophenetic matrices that result from the application of the single link algorithm on P are 0   1   3 DCsingle =   6    6 6 

1 0 3 6 6 6

3 3 0 6 6 6

6 6 6 0 4 4

6 6 6 4 0 2

6 6    6  , 4    2  0 

and 0 1 5 14 14 14  0 5 14 14 14    1     5 5 0 14 14 14 complete ,  DC = 6 6    14 14 14 0    14 14 14 6 0 2  14 14 14 6 2 0 



respectively. Computing the value of the γ statistic for the above matrices, as indicated in the text, we have that γ = 1, that is, the results of the two

4 algorithms are in absolute agreement. This may be taken as an indication that the clustering structure underlying the data set is rather robust.

16.6. (a) Γ = 26.071, Γ̂ = 0.737 (b) Γ = 37.213, Γ̂ = 0.946 (c) Γ = 37.540, Γ̂ = 0.951 (d) The above results indicate the tendency of Γ (and Γ̂) to increase with m.

16.7. Let us consider the computation of the distance between two vectors as a single operation. The computation of the Dunn index when m clusters of equal size are present involves N • N single operations for the distance between two clusters and m(m−1) m2 2 m2 single operations for the computation of the nominator.

2

N

( N −1)

• m m2

2

single operations for the computation of the diameter of a clusN

( N −1)

ter and m m m2 inator.

single operations for the computation of the denom-

Thus, in total,

N N ( − 1) m(m − 1) N 2 +mm m 2 2 m 2 single operations are required for the computation of the Dunn index or O(N 2 ).

In the above we assumed that the distance between two vectors is computed in one step. However, if we want to express the number of computations in terms of additions and multiplications, the above estimate is not accurate. If for example the squared Euclidean distance is employed, the number of computations is O(N 2 l).

16.8. Let EiGG (EiRN G ) be the set of edges of the Gabriel Graph (Relative Neighborhood Graph) computed on Ci (see eq. (15.3) and (15.2), respectively). The weight we of an edge of that graph equals to the distance between its two end points x and y, that is we = d(x, y). Let G eGG ∈ EiGG (eRN ∈ EiRN G ) be the edge with the largest weight. Then i i G diamGG (diamRN G ) is defined as the weight of eGG (eRN ). i i The GG Dunn like index is defined as d(Ci , Cj ) GG Dm = min { min ( )} i=1,...,m j=i+1,...,m maxk=1,...,m diamGG k

5 where d(Ci , Cj ) is the distance between the mean vectors of Ci and Cj . The RN G GG RNG Dunn like index (Dm ) is defined as Dm , substituting the superscript GG with RN D. 16.9. We will prove first the hint. Let e(= (xj , xk)) ∈ EiM ST . Let xr ∈ Ci . Suppose that d(xj , xk) ≥ d(xj , xr ). Then, by removing the edge e = (xj , xk) from the MST and adding the edge (xj , xr ), the weight of the resulting spanning tree would be less than that of the MST, which, by the definition of the MST, is a contradiction. A similar conclusion is drawn if we consider the case where d(xj , xk) ≥ d(xk, xr ). Thus, we have that d(xj , xk) ≤ max{d(xj , xr ), d(xk, xr )}. Therefore, EiM ST ⊆ EiRN G .

We will prove next that EiRN G ⊆ EiGG . Let e(= (xj , xk)) ∈ EiRN G and let reg RN G (xj , xk) be the region defined by eq. (15.3). In a similar way we define reg GG (xj , xk) (based on eq. (15.2)). It is not difficult to see that reg GG (xj , xk) ⊆ reg RN G (xj , xk).

From the definition of e we have that reg RN G (xj , xk)∩(Ci −{xj , xk)}) = ∅. Combining the last two facts we have that reg GG (xj , xk)∩(Ci −{xj , xk)}) = ∅, that is, e ∈ EiGG . Thus, EiRN G ⊆ EiGG .

Having completed the proof of the hint we move now to the main part of the problem. From the hint it easily follows that ST G diamM ≤ diamRN ≤ diamGG i i i

(16.3)

M ST Taking into account that the numerators in the definitions of Dm (eq.

RN G GG (16.20)) and of Dm and Dm (problem 16.8) are the same and that the denominators are related as indicated by the last inequality, the claim easily follows.

16.10. (a) The conditions (C1)-(C3) hold trivially. For (C4) and (C5) we have ST ST ST sM + sM sM ST + sM i j k M ST M ST Rij = > i = Rik dij dij and

ST ST sM + sM i j M ST Rij = dij

ST ST sM + sM i j M ST > = Rik dik

respectively.

G G RN G (b) Defining sRN = diamRN and sGG = diamGG and i i i i , where diami GG RN G GG diami are defined in problem 16.8, the definitions of Rij and Rij easily

6 follow. To prove that the last two quantities satisfy conditions (C1) - (C5), we follow the steps of the first part of the problem.

RN G GG M ST 16.11. The indices DBm and DBm are defined as DBm (eq. (16.27)), by substituting the superscript M ST with RN G and GG, respectively. TakM ST ing into account eq. (16.3) and the fact that the denominators of Rij , RN G GG M ST RN G Rij and Rij are equal to each other, we conclude that Rij ≤ Rij ≤ GG Rij . This leads directly to the proof of the claim.

16.12. Let us consider a cluster Ci = Ci′ −{x}, where x is an outlier, that is, its distance from the closest point of Ci′ is “very large”. Clearly, it is desirable for diami to be approximated by diam′i , which is based on Ci′ . However, in the presence of outliers, diami is, in general, much higher than diam′i .

ST is not related to an edge with x as an On the other hand, if diamM i ST endpoint, it is a better approximation of diam′i . Also, even if diamM is i related to an edge with x as an endpoint, it is clearly less than diami and, ST thus it is also a better approximation of diam′i . Thus, using diamM , we i reduce the effect of the outliers in the DB index.

16.13. Eq. (14.26) gives the values of uij ’s, derived via the minimization of Jq (θ, U ) (eq. (14.19)), subject to the constraints (14.20) - (14.21). We repeat eq. (14.26) here for convinience 1 1 d(xi,θj ) m q−1 k=1 ( d(x ,θ ) ) i k

uij = P

(16.4)

(a) As q → 1+ , 1/(q − 1) → +∞. We consider the following two cases 1 (i) d(xi, θj ) < d(xi, θk), for k = 1, . . . , m, k 6= j. This implies that the terms 1 d(xi, θj ) q−1 ( → 0, k 6= j ) d(xi, θk) that is, all the terms in the denominator of eq. (16.4) tend to zero, except one, which equals to 1. This occurs for k = j. Therefore, in this case uij = 1. (ii) d(xi, θj ) > d(xi, θk), for some k = 1, . . . , m, k 6= j. In this case, 1 d(x ,θ ) there is at least one term of the form ( d(x i,θ j ) ) q−1 , that tends to +∞ and i k all the other terms tend to either 0 or 1. Thus, in this case, uij = 0.

1

Note that in the sequel, we adopt the reasonable assumption that mink=1,...,m d(xi , θk ) is unique.

7 Summarizing the above results we have uij =

(

1, if d(xi, θj ) = mink=1,...,m d(xi, θk) 0, otherwise

(16.5)

Taking into account this result, it is a simple matter of algebra to show that P C = 1 and P E = 0. For the latter case the relation 0 loga 0 = 0 has to be used. (b) As q → +∞, 1/(q−1) → 0, i.e., all the terms in the denominator of the right hand side of eq. (16.4), equal to 1. As a consequence, uij = 1/m. Again, it is a matter of simple algebra to show that P C = 1/m and P E = loga m. (c) Considering fig. 16.7 (b) and (c) for the cases where the fuzzifier equals to 5, we visually, see that the most significant knee occurs at m = 2.

However, if we want to give a more quantitative answer to this problem, we need to define an index that quantifies the “sharpness” of a knee. Such an index is the following Iq (m) =

|q(m) − q(m − 1)| |q(m + 1) − q(m)|

m = 2, 3, . . ., where q 2 stands for P C or P E. This index measures the relative change of the index q as we move for m − 1 to m + 1 through m.

The larger the value of Iq (m), the more sharp the knee at m.

For the case of P C we have from (b) that P C = 1/m. Thus, in this case, IP C (m) becomes IP C (m) = m+1 . The derivative of IP C (m) with respect to m m−1 ′ is easily shown to be IP C (m) = −2/(m − 1)2 < 0. Thus IP C (m) is decreasing and its largest value, which indicates the sharpest knee, occurs at m = 2.

For the case of P E we have from (b) that P E = loga m. In this case, IP E (m), after some manipulations, becomes IP E (m) =

m loga ( m−1 ) m+1 loga ( m )

Its derivative with respect to m, after a bit of algebra, becomes IP′ E (m) = 2

m m loga m+1 − m+1 loga m−1 1 m−1 m m log a log2a m+1 m

q is not the fuzzifier in this case.

8 where log a is the natural logarithm of a. It is easy to show that (m−1)/m < m/(m + 1) and loga ((m + 1)/m) < loga (m/(m − 1)). Combining the last two facts, we easily deduce that IP′ E (m) is negative and, thus, IP E (m) is a decreasing function and its largest value, which indicates the sharpest knee, occurs at m = 2.

16.14. We remind that the XB and XBq indices are used only when the Euclidean distance in the FCM algortihm is used. Rewritting eq. (14.32) in a form suitable for the current framework we have uqij xi q i=1 uij

PN

wi = Pi=1 N

(a) Taking into account the result of part (b) of the previous problem we have that as the fuzzifier q tends to +∞, all uij ’s tend to 1/m. Substituting in the last equation and after some manipulations, we finally obtain that limq→+∞ wi = w, where w is the mean of all vectors of the data set X. Taking into account the last two facts for uij ’s and wi, the numerator of XB (eq. (16.33)) J2 , which is given by eq. (16.32) for q = 2, becomes J2 =

m N X N 1 X 1 1 X ( )2 ||xi − w||2 = ||xi − w||2 N i=1 j=1 m m i=1

which is obviously a finite positive number.

The denominator of XB, which equals to N mini,j=1,...,m, i6=j ||wi − wj ||2 , clearly tends to zero, since limq→+∞ wi = w, for i = 1, . . . , m. Thus, limq→+∞ XB = +∞. (b) For the XBq index, as q → ∞, the numerator σq becomes σq =

N 1 X ||xi − w||2 mq−1 i=1

which clearly tends to zero. Taking into account that both the numerator and the denominator of XBq become zero as q → +∞, XBq becomes indeterminate.

16.15. (a) Substituting uij ’s from eq. (16.7) to eq. (16.35) and after some manipulations, we obtain lim+ F Sq =

q→1

m X X

j=1 xi∈Cj

||xi − wj ||2A −

m X

j=1

nj ||wj − w||2A

9 where nj is the number of points for which uij = 1, i = 1, . . . , N . Multiplying and dividing the right hand side of the above equation with N , and using the definitions for the within-class, between-class and mixture scatter matrices given in section 5.5.3, it is not difficult to show that lim F Sq = N trace(Sw ) − N trace(Sb ) =

q→1+

N trace(Sw ) − N (trace(Sm ) − trace(Sw )) = 2N trace(Sw ) − N trace(Sm ) (b) Taking into account that in this case uij = 1/m, i = 1, . . . , N , j = 1, . . . , m and wj → w, as q → +∞, we have for the first term of the right hand side of eq. (16.35) lim

q→+∞

m n X X

i=1 j=1

uqij ||xi − wj ||2A =

N X i=1

||xi − w||2A lim

1

q→+∞ mq−1

=0

For the second term we have lim

q→+∞

m n X X

i=1 j=1

uqij ||wj − w||2A = ||w − w||2A N lim

1

q→+∞ mq−1

= 0N 0 = 0

Combining the last two facts, we prove the claim.

16.16. Clearly, xij , belongs to the sphere with center cj and radius rj , as well as to the line connecting xi with cj . Thus, the following equations hold (xij − cj )T (xij − cj ) = rj2

(16.6)

(xij − cj ) = a(xi − cj )

(16.7)

and Substituting (xij − cj ) from eq. (16.7) to the (16.6) and after some manipulations we obtain rj a= ||xi − cj ||

Solving (16.7) with respect to xij and substituting into eq. (16.42) we obtain τij = (1 − a)(xi − cj )

Finally, substituting a from above to the last equation and after some manupulations, we prove the claim.

10 16.17. (a) The arc length Lφ is Lφ =

Z φ/2

−φ/2

dl = r

Z φ/2

−φ/2

dθ = rφ

The mean vector of the points of the arc is m=

1 Z φ/2 xdl = Lφ −φ/2

Z φ/2 2r sin φ/2 T 1 Z φ/2 , 0] [ r cos θdθ, r sin θdθ]T = [ Lφ −φ/2 φ −φ/2

The value of the first term in the right hand side of (16.60) becomes 

R φ/2

2 1 Z φ/2 r3  −φ/2 cos θdθ xxT dl = R φ/2 Lφ −φ/2 Lφ −φ/2 cos θ sin θdθ

or, after some manipulations,

r2 1 Z φ/2 xxT dl = Lφ −φ/2 2

"



R φ/2

−φ/2 cos θ sin θdθ  R φ/2 2 −φ/2 sin θdθ

1 − sinφ φ 0 0 1 + sinφ φ

#

Combining the last equation and the value of m computed above, we finally obtain   2 r2  1 − sinφ φ − 8 sinφ/2φ/2 0  Σ= 2 0 1 + sinφ φ

Since all the data points lie on a perfect arc, we have that S in eq. (16.48) equals to Lφ . Combining the above equations, and after some manipulations we easily prove the claim.

The value of δ for φ = 2π is equal to 1. (b) We set the (orthogonal) system of coordinate axes such that the first one contains the line segment and the second one bisects it. Then, clearly, all the points in the line segment are of the form x = [x, 0]T and m = [0, 0]T .

Thus, the covariance matrix Σ may be written as 1 Σ= L

" R L/2

2 −L/2 x dx

0

0 0

#

=

"

L2 12

0

0 0

#

11 As in the previous case, S = L. Substituting in eq. (16.48), we easily obtain that √ 3 δ= π 16.18. (a) Clearly, each point (vector) lies at distance 2r from any of its neighboring points. Moreover, the circle corresponding to each point tangents to the circles corresponding to its neighboring points. Clearly, the packing density ρ (see eq. (16.50)) in this case equals to ρ=

(a/2r)2 2 π πr = a2 4

Replacing r with r/2 and working similarly, we have ρ=

a )2 ( 2(r/2)

a2

r π( )2 = π/4 2

Thus, the packing density does not depend on r. (b) It is clear that any other arrangement of the (a/2r)2 circles, that lie entirely in the square should contain at least one pair of intersected circles.

Thus, the packing density computed above is maximum.

16.19. In the Hopkins test we need only the closest vector of a specific yj ∈ X ′ in X1 , i.e., intuitively speaking, we move “one step away” from a given yj . On the other hand, in the Cox-Lewis test we determine for a specific yj ∈ X ′ , its closest vector in X, say xj and then, the closest vector of xj in X. Thus, in this case, we move “two stpes away” from a given yj .

16.20. Assume that we are currently at the ith decomposition layer. When the data are clustered, b is significantly higher that the average length of the edges of the M ST (Ri (X)). Thus, only a small fraction of points of Ri (X) will move to S(Ri (X)). Thus, Li+1 (X) = Ri (X) − S(Ri (X)), will contain a “large” fraction of points of Li (X).

The situation is reversed when the data points are regularly distributed.

In this case b is of the same order of the average length of the edges of the M ST (Ri (X)). As a consequence, “many” more points of Ri (X) will move to S(Ri (X)), compared to the previous case. Thus, it is expected that the number of points in Li+1 (X) = Ri (X) − S(Ri (X)), will be significantly less than those in the previous case.

12 Finally, the case where the data points are randomly distributed lies in the middle of the previous two. In this case, it is expected that b is higher than the average length of the edges of M ST (Ri (X)), but not much higher.

As a consequence the number of points that will be moved to S(Ri (X)) in this case is likely to be greater than that of the clustering case and less than that of the case where the data are regularly spaced.