Solution Manual for Data Mining and Machine Learning Fundamental Concepts and Algorithms 2nd Edition

Solution Manual for Data Mining and Machine Learning Fundamental Concepts and Algorithms 2nd Edition by Mohammed J. Zaki, Wagner Meira Jr

Solution Manual for Data Mining and Machine Learning Fundamental Concepts and Algorithms 2nd Edition by Mohammed J. Zaki, Wagner Meira Jr

Contents

Contents

1

1

Data Mining and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . .

3

1.7

3

PART I

2

3

7 7

Exercises

20

Exercises

26

High-dimensional Data .............................................................................................. 29 Exercises

29

Dimensionality Reduction ....................................................................................... 39

PART II

Exercises FREQUENT PATTERN MINING

39 45

Itemset Mining ........................................................................................................... 47 8.5

9

16

Kernel Methods ......................................................................................................... 26

7.6

8

Exercises

Graph Data ................................................................................................................. 20

6.9

7

Exercises

Categorical Attributes .............................................................................................. 16

5.6

6

5

2.7

4.6

5

DATA ANALYSIS FOUNDATIONS

Numeric Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.7

4

Exercises

Exercises

47

Summarizing Itemsets .............................................................................................. 56 9.6

Exercises

56 1

Solution Manual for Data Mining and Machine Learning Fundamental Concepts and Algorithms 2nd Edition by Mohammed J. Zaki, Wagner Meira Jr 10

Sequence Mining....................................................................................................... 63 10.5

11

PART III

PART IV

Exercises

106

Exercises

111

Exercises CLASSIFICATION

118 123

Exercises

125

Exercises

129

Exercises

137

Support Vector Machines........................................................................................ 141 21.7

22

99

Linear Discriminant Analysis .................................................................................. 137 20.4

21

Exercises

Decision Tree Classifier........................................................................................... 129 19.4

20

91

Probabilistic Classification ..................................................................................... 125 18.5

19

Exercises

Clustering Validation .............................................................................................. 118 17.5

18

89

Spectral and Graph Clustering ............................................................................... 111 16.5

17

CLUSTERING

84

Density-based Clustering ........................................................................................ 106 15.5

16

Exercises

Hierarchical Clustering ............................................................................................ 99 14.4

15

75

Representative-based Clustering ............................................................................. 91 13.5

14

Exercises

Pattern and Rule Assessment................................................................................... 84 12.4

13

63

Graph Pattern Mining ................................................................................................ 75 11.5

12

Exercises

Exercises

141

Classification Assessment..................................................................................... 145 22.5

Exercises

145

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

CHAP T ER 1

Data Mining and Analysis

1.7 EXERCISES Q1. Show that the mean of the centered data matrix Z in Eq. (1.5) is 0.

1

1

(xi — µ)

1

— —

1

0

Q2. Prove that for the Lp-distance in Eq. (1.2), we have δ (x, y) = lim δ (x, y) = ∞

p→∞

p

d

max i=1

|x — y |} i

i

for x, y e Rd. Answer: We have to show that d

lim

d

x |x — y |

assume that |xi — yi | < m for all i /= a.

d

m

|x — y | m

i/=a

|x — y | m 3

Data Mining and Analysis

4

| m | i/=a

m 0

a

a

d

d

lim m

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

P ART ONE

DATA ANALYSIS FOUNDATIONS

CHAP T ER 2

Numeric Attributes

2.7 EXERCISES Q1. True or False: (a) Mean is robust against outliers.

Answer: False (b) Median is robust against outliers.

(c) Standard deviation is robust against outliers.

Answer: False Q2. Let X and Y be two random variables, denoting age and weight, respectively.

Consider a random sample of size n = 20 from these two variables X = (69, 74, 68, 70, 72, 67, 66, 70, 76, 68, 72, 79, 74, 67, 66, 71, 74, 75, 75, 76) Y = (153, 175, 155, 135, 172, 150, 115, 137, 200, 130, 140, 265, 185, 112, 140, 150, 165, 185, 210, 220)

(a) Find the mean, median, and mode for X.

1

2

(b) What is the variance for Y? 7

Numeric Attributes

8

2

1 20

2

0yi — µY = 27384.2/20 = 1369.21

(c) Plot the normal distribution for X. 2

0.10

0.05

0 60

65

70

75

80

(d) What is the probability of observing an age of 80 or higher?

2

∫ 0.010769 80

σ2

σ 2

13.8475 122.435

122.435 1369.21

(f) What is the correlation between age and weight?

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

2.7 Exercises

9

Answer: 122.435

(g) Draw a scatterplot to show the relationship between age and weight.

Answer: The scatterplot is shown in the figure below.

250

Y: Weight

230 210 190 170 150 130

110 65

67

69

71

75

73

77

79

Q3. Show that the identity in Eq. (2.15) holds, that is, n

n

i=1

i=1

Σ Σ (xi — µ)2 = n(µˆ — µ)2 + (xi — µˆ )2

Answer: Consider the RHS —

2

—

2

ˆ 2— 2µ

2

2

2

2

2

2

2

2

2

2

2

2

2

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Numeric Attributes

10

Q4. Prove that if xi are independent random variables, then ! n n Σx =Σ i var var(xi ) i=1

i=1

This fact was used in Eq. (2.12).

Answer: We assume for simplicity that all the variables are discrete. A similar approach can be used for continuous variables.

Consider the random variable x1 + x2. Its mean is given as Σ Σ µx1 +x2 = (a + b)f (a, b) x1=a x2=b

Since x1 and x2 are independent, their joint probability mass function is given as: f (x1, x2) = f (x1) · f (x2) Thus, the mean is given as µx1 +x2 = =

Σ Σ

(a + b)f (a, b)

x1 =a x2 =b

Σ Σ

(a + b)f (a)f (b)

x1 =a x2 =b

=

Σ

f (a)

x1 =a

=

Σ

f (a) ,

Σ

(a + b)f(b)

x2 =b

,

x1 =a

=

Σ

Σ

af (b) +

x2 =b

Σ

, bf ( b ) ,

x2 =b

f (a) a + µx2

x1 =a

= µx1 + µx2 In general, we can show that the expected value of the sum of the variables xi is the sum of their expected values, i.e., " n # n Σ Σ = E E[xi ] xi i=1

i=1

Now, let us consider the variance of the sum of the random variables: " n #!2, ! , n n Σ Σ Σ xi = E , xi , var xi — E i=1

i=1

i=1

, !2, n n Σ Σ = E, E[xi ] , xi — , = E,

i=1 n Σ

i=1

(xi — E[xi ])

!2, ,

i=1

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

2.7 Exercises

1 11

2

2

Q5. Define a measure of deviation called mean absolute deviation for a random variable X as follows: 1 Σn |x i — µ| n i=1

Is this measure robust? Why or why not?

Answer: No, it is not robust, since a single outlier can skew the mean absolute deviation.

Q6. Prove that the expected value of a vector random variable X = (X1, X2)T is simply the vector of the expected value of the individual random variables X1 and X2 as given in Eq. (2.18).

Answer: This follows directly from the definition of expectation of a vector random

1

1

2

x

∫ ∫

∫ ∫

1

2

1

2

1

1

2

2

1

2

x

1

1

1

2

1

1

2

2

1

1

2

1

2

1

1

2

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Numeric Attributes

12

Q7. Show that the correlation [Eq. (2.23)] between any two random variables X1 and X2 lies in the range [—1, 1].

Answer: The Cauchy-Schwartz inequality states that for any two vectors x and y in an inner product space, they satisfy: |(x, y⟩|2 ≤ (x, x⟩ · (y, y⟩ Define the inner product between two random variables X1 and X2 as follows: (X1, X2⟩ = E[X1X2] Expectation is a valid inner product since it satisfies the three conditions: i) symmetric: E[X1X2] = E[X2X2], ii) positive-semidefinite: E[X1X2] = E[X21] ≥ 0, and iii) linear: E[(aX1)X2] = aE[X1X2] and E[(X1 + Z)X2] = E[X1X2] + E[ZX2].

Then, we have |σ12 | = cov(X1 , X2 )

2

= E[(X1 — µ1)(X2 — µ2)] = ((X1 — µ1 )(X 2 — µ 2 )⟩

2

2

≤ (X1 — µ1, X1 — µ1⟩ · (X2 — µ2, X2 — µ2⟩ = E[X1 — µ1] · E[X2 — µ2] = σ1 · σ2 Since |σ12| ≤ σ1 · σ2, it follows that the correlation ρ12 = σ12/σ1σ2 lies in the range [—1, 1].

Q8. Given the dataset in Table 2.1, compute the covariance matrix and the generalized variance.

Table 2.1. Dataset for Q8

x1 x2 x3

X1

X2

X3

17 11 11

17 9 8

12 13 19

8.0 16.22

8.56 9.56

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

2.7 Exercises

13

Q9. Show that the outer-product in Eq. (2.31) for the sample covariance matrix is equivalent to Eq. (2.29).

1

1

—

—

Q10. Assume that we are given two univariate normal distributions, NA and NB, and let their mean and standard deviation be as follows: µA = 4, σA = 1 and µB = 8, σB = 2. (a) For each of the following values xi e {5, 6, 7} find out which is the more likely normal distribution to have produced it.

Answer: If we plug-in xi in the equation for the normal distribution, we obtain

—

—

zB(6) = (6 — 8)/2 = —1 zB(7) = (7 — 8)/2 = —0.5

(b) Derive an expression for the point for which the probability of having been produced by both the normals is the same.

1

—

4)

1

—

8)

2 2π This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Numeric Attributes

14 (x—4)2

(x—8)2

(x — 8)2 8 8 ln(2) + x2 + 64 — 16x 8

(x — 4)2 2 x2 + 16 — 8x 2

2

2

3 2 x — 4x — 2 ln(2) = 0 4 0.75x2 — 4x — 1.4 = 0 We can solve this equation using the general solution for a quadratic equation: —b±

b —4ac 2a

Q11. Consider Table 2.2. Assume that both the attributes X and Y are numeric, and the table represents the entire population. If we know that the correlation between X and Y is zero, what can you infer about the values of Y? Table 2.2. Dataset for Q11

X 1 0 1 0 0

Y a b c a c

Q12. Under what conditions will the covariance matrix Σ be identical to the correlation matrix, whose (i, j) entry gives the correlation between attributes Xi and Xj ? What can you conclude about the two variables?

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

2.7 Exercises

Answer:

15

If the covariance matrix equals the correlation matrix, this means that for

σσ

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

CHAP T ER 3

Categorical Attributes

3.7 EXERCISES Q1. Show that for categorical points, the cosine similarity between any two vectors in lies in the range cos θ e [0, 1], and consequently θ e [0◦, 90◦].

s

cos θ

d

Q2. Prove that E[(X1 — µ1)(X2 — µ2)T] = E[X1XT2] — E[X1]E[X2]T. Answer: E[(X1 — µ1)(X2 — µ2) ] = E[(X1 — µ1)(X2 — µ2 )]

Table 3.1. Contingency table for Q3

Z=f

16

Z=g

Y=d

Y=e

Y=d

Y=e

X=a

5

10

10

5

X=b

15

5

5

20

X=c

20

10

25

10

3.7 Exercises

17

Table 3.2. χ 2 Critical values for different p-values for different degrees of freedom (q): For example, for q = 5 degrees of freedom, the critical value of χ 2 = 11.070 has p-value = 0.05.

q

0.995

0.99

0.975

0.95

0.90

0.10

0.05

0.025

0.01

0.005

1 2 3 4 5 6

— 0.010 0.072 0.207 0.412 0.676

— 0.020 0.115 0.297 0.554 0.872

0.001 0.051 0.216 0.484 0.831 1.237

0.004 0.103 0.352 0.711 1.145 1.635

0.016 0.211 0.584 1.064 1.610 2.204

2.706 4.605 6.251 7.779 9.236 10.645

3.841 5.991 7.815 9.488 11.070 12.592

5.024 7.378 9.348 11.143 12.833 14.449

6.635 9.210 11.345 13.277 15.086 16.812

7.879 10.597 12.838 14.860 16.750 18.548

Q3. Consider the 3-way contingency table for attributes X, Y, Z shown in Table 3.1.

Compute the χ2 metric for the correlation between Y and Z. Are they dependent or independent at the 95% confidence level? See Table 3.2 for χ2 values.

Y=d Y=e

Y=d Y=e

Z=f 40 25

Z=g 40 35

80 60

65

75

140

Z=f (80 · 65)/140 = 37.14 (60 · 65)/140 = 27.86

Z=g (80 · 75)/140 = 42.86 (60 · 75)/140 = 32.14

Subtracting the expected and observed values, and squaring them, we get: Y=d Y=e

Z=f 2.862 = 8.16 —2.862 = 8.16

Z=g —2.862 = 8.16 2.862 = 8.16

Dividing by the expected counts, gives: Y=d Y=e

Z=f 0.22 0.29

Z=g 0.19 0.25

Q4. Consider the “mixed” data given in Table 3.3. Here X1 is a numeric attribute and X2 is a categorical one. Assume that the domain of X2 is given as dom(X2) = {a, b}.

Answer the following questions.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

18

Categorical Attributes

(a) What is the mean vector for this dataset? (b) What is the covariance matrix?

1

2

0.92

0.15 0.24

2

0.15

0.24

Q5. In Table 3.3, assuming that X1 is discretized into three bins, as follows: c1 = (—2, —0.5] c2 = (—0.5, 0.5] c3 = (0.5, 2] Answer the following questions: (a) Construct the contingency table between the discretized X1 and X2 attributes.

Include the marginal counts. (b) Compute the χ2 statistic between them. (c) Determine whether they are dependent or not at the 5% significance level. Use the χ2 critical values from Table 3.2.

Table 3.3. Dataset for Q4 and Q5

X1 0.3 —0.3 0.44 —0.60 0.40 1.20 —0.12 —1.60 1.60 —1.32

X2 a b a a a b a b b a

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

3.7 Exercises

19

c1 c2 c3

c1 c2 c3

a 2 4 0

b 1 1 2

3 5 2

6

4

10

a 1.8 3 1.2

b 1.2 2 0.8

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

CHAP T ER 4

Graph Data

4.6 EXERCISES Q1. Given the graph in Figure 4.1, find the fixed-point of the prestige vector.

a

b

Figure 4.1. Graph for Q1

0

1

7

1

1 0 1

1

9

0 0

1

2

12

2

16

3

21

1 0 0

1 0

4

5

0.57

28

0.57

normalization to p = (0.548, 0.726, 0.415)T .

Q2. Given the graph in Figure 4.2, find the fixed-point of the authority and hub vectors.

20

4.6 Exercises

21

a

b Figure 4.2. Graph for Q2.

1

1 0 1

1 1

0

1

2

6

10

28

48

132

1.0

228

1.0

624

1.0

1080

1.0

1 0 1

1 1

Q3. Consider the double star graph given in Figure 4.3 with n nodes, where only nodes 1 and 2 are connected to all other vertices, and there are no other links. Answer the following questions (treating n as a variable). (a) What is the degree distribution for this graph?

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Graph Data

22

3

4

···············

5

1

2 Figure 4.3. Graph for Q3.

2 2/n (b) What is the mean degree?

Answer: The average or mean degree is: 2(n — 2)/n + 2(n — 1)/n = (2n — 4 + 2n — 2)/n = (4n — 6)/n (c) What is the clustering coefficient for vertex 1 and vertex 3?

1

= 2(n — 2)/((n — 1)(n — 2)) = 2/(n — 1)

2

(d) What is the clustering coefficient C(G) for the entire graph? What happens to the clustering coefficient as n → ∞?

Answer: The average clustering coefficient for the graph is: C(G) =

(n — 2) · 1 + 2 · 2/(n — 1) = (n2 — 3n + 6)/(n2 — n) n

As n → ∞, C(G) → 1. (e) What is the transitivity T(G) for the graph? What happens to T(G) and n → ∞?

Answer: The transitivity is given as: T(G) =

3(n — 2) 2 (n—1) 2

+ (n — 2)

= 3/n

As n → ∞, T(G) → 0. (f) What is the average path length for the graph?

Answer: The average path length can be computed as follows: path from 1 to n — 1 other vertices path from 2 to n — 2 other vertices path from i to n — i other vertices

len = 1 len = 1 len = 2

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

4.6 Exercises

23

The sum of all the path lengths over pairs of nodes is therefore: n—3 Σ

(n—1)+(n—2)+2

2

2

i = 2n—3+(n—3·n—2) = n —5n+6+2n—3 = n —3n+3

i=1

But there are n2 = n(n — 1)/2 pairs. Thus, the average path length is 2(n2 — 3n + 3)/n2 — n. As n → ∞, average length tends to 2. (g) What is the betweenness value for node 1?

Answer: There are n — 2 shortest paths from 2 to i >= 3, which do not include 1. Furthermore for the other n — 2 pairs of nodes, only half go through 1. But there are 2 paths for each such pair. The betweenness is then given as: =

n—2 2

2 n—2 + (n — 1) 2 (n — 2)(n — 3)/2 (n — 2)(n — 3) + (n — 1)

= =

n2 —5n + 6 2(n2 — 4n + 5)

as n → ∞, betweenness goes to 1/2. (h) What is the degree variance for the graph?

E[X2] = 4(n — 2)/n + 2(n — 1)2/n = (2n2 + 2n — 6)/n

2

2

2

Q4. Consider the graph in Figure 4.4. Compute the hub and authority score vectors.

Which nodes are the hubs and which are the authorities?

1

3

2

4

5

Figure 4.4. Graph for Q4.

0 0 1 1 0 1 0 0 0 0 1 1 0 0 0

0 0 0 0 1

0 0 1 0 0

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Graph Data

24

For the hubs we can directly compute the AAT matrix and compute its dominant eigenvector ,

1 1 AAT = . 0 . ,1 0

1 2 0 1 0

0 0 1 0 0

1 1 0 2 0

, 0 0 0. .

0, 1

Starting with the initial vector h0 = (1, 1, 1, 1, 1)T, successive iterations give: h1 = (3, 4, 1, 4, 1)T h2 = (11, 15, 1, 15, 1)T h3 = (41, 56, 1, 56, 1)T h4 = (153, 209, 1, 209, 1)T After normalizing h4 the hub scores vector is h = (0.46, 0.63, 0.0, 0.63, 0.0)T . The eigenvalue is obtained as the ratio of the maximum element from the last two iterations, i.e., 209/56 = 3.73.

For the authorities we can directly compute the ATA matrix and compute its dominant eigenvector , , 1 0 1 0 0 0 1 1 0 0 ATA = . 1 1 3 0 0. . . ,0 0 0 1 0, 0 0 0 0 1 Starting with the initial vector a0 = (1, 1, 1, 1, 1)T, successive iterations give: a1 = (2, 2, 5, 1, 1)T a2 = (7, 7, 19, 1, 1)T a3 = (26, 26, 71, 1, 1)T a4 = (97, 97, 265, 1, 1)T After normalizing a4 the authority scores vector is a = (0.325, 0.325, 0.888, 0.003, 0.003)T . The eigenvalue is obtained as the ratio of the maximum element from the last two iterations, i.e.,265/71 = 3.73.

Vertices 2 and 4 are the good hubs; both point to the high authority node 3.

Q5. Prove that in the BA model at time-step t + 1, the probability πt (k) that some node with degree k in Gt is chosen for preferential attachment is given as k · nt (k) πt (k) = Σ i i · n t (i)

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

4.6 Exercises

25

d

d

t

k t

t

k· t t

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

CHAP T ER 5

Kernel Methods

5.6 EXERCISES Q1. Prove that the dimensionality of the feature space for the inhomogeneous polynomial kernel of degree q is m=

d +q q

partitioning: (∗∗||). This is mapped to the non-negative numbers n0 = 2, n1 = 0, n2 = 0, since there are two stars in the first partition, and none in the other two partitions.

q

Q2. Consider the data shown in Table 5.1. Assume the following kernel function: K(xi , xj ) = xi — xj 2. Compute the kernel matrix K. Answer: The kernel between x1 and x2 is given as: K(x1, x2) = x1 — x2 2 = 1.52 + 1.92 = 5.86 The complete kernel matrix is given as: , , 0 5.86 1.46 4.64 5.86 0 10 1.46 K= . . ,1.46 10 0 5.86, 4.64 1.46 5.86 0 26

5.6 Exercises

27 Table 5.1. Dataset for Q2

i

xi

x1 x2 x3 x4

(4, 2.9) (2.5, 1) (3.5, 4) (2, 2.1)

Q3. Show that eigenvectors of S and Sl are identical, and further that eigenvalues of Sl are given as (λi )l (for all i = 1,..., n), where λi is an eigenvalue of S, and S is some n × n symmetric similarity matrix.

is given as:

corresponding eigenvalue of Sl is λl .

2

2 l—2

1 Q4. The von Neumann diffusion kernel is a valid positive semidefinite kernel if |β| < ρ(S) , where ρ(S) is the spectral radius of S. Can you derive better bounds for cases when β > 0 and when β < 0?

Answer: For K to be a positive semi-definite kernel, we require that 1 — βλi ≥ 0 for

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Kernel Methods

28

When β < 0, then positive eigenvalues always satisfy 1 — βλi > 0, so the constraint

1

1

Q5. Given the three points x1 = (2.5, 1)T, x2 = (3.5, 4)T, and x3 = (2, 2.1)T. (a) Compute the kernel matrix for the Gaussian kernel assuming that σ 2 = 5.

—

10/10

1

10/10p 1

1.46/10

5.86/10

K

e 5.86/10 1

0.86

1 0.56

0.56 1

(b) Compute the distance of the point φ(x1) from the mean in feature space.

Answer: The squared distance in terms of kernel operations is given as 3Σ

||φ(x1) — µφ ||2 = K(x1, x1) — 2/3

K(x1, xj ) + 1/9

j =1

ΣΣ i

K(xi , xj )

j

= 1 — 2/3(2.23) + 0.73 = 0.24 The distance is therefore

, 0.24 = 0.49.

(c) Compute the dominant eigenvector and eigenvalue for the kernel matrix from (a).

Answer:

We use the power iteration method to find the dominant eigenvector,

scaling by the maximum value we have x1 = (0.92, 0.79, 1)T.

(0.6, 0.48, 0.64)T.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

CHAP T ER 6

High-dimensional Data

6.9 EXERCISES Q1. Given the gamma function in Eq. (6.6), show the following: (a) Ŵ(1) = 1 Answer: ∫

∫

0 —x

x

0

0

—1e—

0 e0

0

1

Ŵ 12 1 2

1/2

∫

∫ y

0

0

2 ∫

y

0

—

—

29

High-dimensional Data

30

(c) Ŵ(α) = (α — 1)Ŵ(α — 1) Answer: We use integration by parts. Let u = xα—1 so that du = (α — 1)xα—2dx. x

x

∫

∫ uv —

udv

∫

α—1 —x

e

x

— α—1 —x

vdu

∫ e x

0

2

0

0

Ŵ(α) = 0 + (α — 1)Ŵ(α — 1)

Q2. Show that the asymptotic volume of the hypersphere Sd (r) for any value of radius r eventually tends to zero as d increases.

Ŵ(d/2 + 1)

lim

(d/2)! ≃

1

lim

πd

2Ce d

(2n + 1)!! 2 n! = ((2n + 1)(2n — 1)...1)(2n · 2(n — 1) · 2(n — 2) · 2) = (2n + 1)! d!

1 π Ŵ(d/2 + 1)

d/2 ((d — 1)/2)!

d!

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

6.9 Exercises

3 11

X2 1.0

0.5

X1 1.0

0 0

0.5

Figure 6.1. Shape of L0.5 ball of radius 0.5 in 2D

Q3. The ball with center c e Rd and radius r is defined as Bd (c, r) = x e Rd | δ(x, c) ≤ r

}

where δ(x, c) is the distance between x and c, which can be specified using the Lp-norm: ! 1p d Σ |xi — ci|p Lp(x, c) = i=1

where p /= 0 is any real number. The distance can also be specified using the L∞-norm: } L∞(x, c) = max |xi — ci| i

Answer the following questions: (a) For d = 2, sketch the shape of the hyperball inscribed inside the unit square, using the Lp-distance with p = 0.5 and with center c = (0.5, 0.5)T.

|x 1— 0.5 | — 0.5|

|x 2— 0.5

2

— 0.5| —

0

(b) With d = 2 and c = (0.5, 0.5)T, using the L∞-norm, sketch the shape of the ball of radius r = 0.25 inside a unit square.

max{|x1 — 0.5|, |x2 — 0.5|} = 0.25

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

High-dimensional Data

32

X2 1.0

0.5

X1 1.0

0 0

0.5

Figure 6.2. Shape of L∞ ball of radius 0.25 in 2D

ǫ ǫ Figure 6.3. For Q4.

(c) Compute the formula for the maximum distance between any two points in the unit hypercube in d dimensions, when using the Lp-norm. What is the maximum distance for p = 0.5 when d = 2? What is the maximum distance for the L∞-norm?

d

|1

0

d

max{|1 — 0|} = 1

Q4. Consider the corner hypercubes of length ǫ ≤ 1 inside a unit hypercube. The 2-dimensional case is shown in Figure 6.3. Answer the following questions: This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

6.9 Exercises

33

(a) Let ǫ = 0.1. What is the fraction of the total volume occupied by the corner cubes in two dimensions?

(b) Derive an expression for the volume occupied by all of the corner hypercubes of length ǫ < 1 as a function of the dimension d. What happens to the fraction of the volume in the corners as d → ∞?

(c) What is the fraction of volume occupied by the thin hypercube shell of width ǫ < 1 as a fraction of the total volume of the outer (unit) hypercube, as d → ∞? For example, in two dimensions the thin shell is the space between the outer square (solid) and inner square (dashed).

d

volume of the shell approaches 1, i.e., it contains all of the volume of the outer

standard normal distribution we have

—2 ln(α) — d)/ 2d .

—

—2 ln(α) — d)/ 2d

—

—

—

0

Q6. Consider the conceptual view of high-dimensional space shown in Figure 6.4. Derive an expression for the radius of the inscribed circle, so that the area in the spokes This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

High-dimensional Data

34

accurately reflects the difference between the volume of the hypercube and the inscribed hypersphere in d dimensions. For instance, if the length of a half-diagonal 1 is fixed at 1, then the radius of the inscribed circle is , in Figure 6.4a.

2

b α

2

Figure 6.4. Computing the radius for d = 3

Answer: The difference between the volume of the hypercube and hypersphere is given as δhs = (2r)d — Kdrd = (2d — Kd )rd = 2d — Kd assuming r = 1, and where Kd is as given in (6.5).

In d dimensions, the inscribed circle is divided into 2 d sectors, with the area of 2 each sector being πr2d.

Now consider with the triangle defined by the two sides of the sector of length r, with the third sides length being b. The angle of the inner triangle at the center of the circle is 2π π α1 = d = d—1 2 2 Thus, the two other angles are both equal to 2α

π 1 = π — d—1 2 2

=

(2d—1 — 1)π 2d

By the law of sines, we have b r , or = sin(α ) sin(α ) 1

2

b=

sin(α1) r = cd r sin(α2)

1) where cd = sin(α . sin(α2) The height of the inner triangle is given as ,, , , 2 4 — cd2 1 2 ,r h= r — b =, 2 2

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

6.9 Exercises

35

1 2

2 d

d

2

4

2 d

d

d

2d

4

Finally, consider the length of the spokes or triangles with the same base b. Let x

1 2

d

1b

2 d

δ + π —2

2

2

cd

1c

d

2 d

1

Q7. Consider the unit hypersphere (with radius r = 1). Inside the hypersphere inscribe a hypercube (i.e., the largest hypercube you can fit inside the hypersphere). An example in two dimensions is shown in Figure 6.5. Answer the following questions:

Figure 6.5. For Q7.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

High-dimensional Data

36

(a) Derive an expression for the volume of the inscribed hypercube for any given dimensionality d. Derive the expression for one, two, and three dimensions, and then generalize to higher dimensions.

2

2

2

2 3

8

3

3

2

2d

d

d

(b) What happens to the ratio of the volume of the inscribed hypercube to the volume of the enclosing hypersphere as d → ∞? Again, give the ratio in one, two and three dimensions, and then generalize.

Answer: Let’s look at the 1d, 2d and 3d cases: With r = 1, for 1d, we have

1

2 2

2

2 3

3

Now for the general case we have: d

2d Ŵ(d/2 + 1)

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

6.9 Exercises

37

Let’s assume that d is even, in which case Ŵ(d/2 + 1) = (d/2)! So we get: 2d (d/2)!

2d

—1 dπ

dπ

2d 2d

dπ

dπ

dπ

dπ

—

d/2 2 d/2 — ··· dπ dπ

1 dπ dπ

d/2 2 — dπ dπ ···

···

dπ

1) —

1 dπ

dπ

1

2d 2

Q8. Assume that a unit hypercube is given as [0, 1]d , that is, the range is [0, 1] in each dimension. The main diagonal in the hypercube is defined as the vector from (0, 0) = d—1

d—1

z Ic ˛ z Ic ˛ (0, . . . , 0, 0) to (1, 1) = (1, . . . , 1, 1). For example, when d = 2, the main diagonal goes from (0, 0) to (1, 1). On the other hand, the main anti-diagonal is defined as the d—1

d—1

z Ic ˛ z Ic ˛ vector from (1, 0) = (1, . . . , 1, 0) to (0, 1) = (0, . . . , 0, 1) For example, for d = 2, the anti-diagonal is from (1, 0) to (0, 1). (a) Sketch the diagonal and anti-diagonal in d = 3 dimensions, and compute the angle between them.

(1, 1, 0)

(—1, —1, 1). The angle is therefore: cos θ 70.53◦.

1

1/3, which

(b) What happens to the angle between the main diagonal and anti-diagonal as d → ∞. First compute a general expression for the d dimensions, and then take the limit as d → ∞.

1

z Answer: The main diagonal is (1, 1, 1) and the anti-diagonal is ( —1, · · · , —1, 1).

Q9. Draw a sketch of a hypersphere in four dimensions.

Answer: The 1D hypersphere is simply an interval along dimension X1. The 2D hypersphere is a collection of closely spaced 1D intervals of decreasing radius along the new dimension X2, yielding a circle. The 3D hypersphere is a collection of This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

High-dimensional Data

38

Figure 6.6. 4D Hypersphere

closely spaced 2D circles of decreasing radius along the new dimension X3, to yield a sphere.

In a similar manner, the 4D hypersphere will be a collection of closely spaced 3D spheres along the new dimension X4. Of course we cannot adequately draw a 4D object, however Fig. 6.6 provides a conceptual sketch.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Dimensionality Reduction

CHAP T ER 7

7.6 EXERCISES Q1. Consider the following data matrix D: X1 8 0 10 10 2

X2 —20 —1 —19 —20 0

(a) Compute the mean µ and covariance matrix Σ for D.

2 -6 4 4 -4

-8 11 -7 -8 12

1 5 17.6 38

38 88.4

(b) Compute the eigenvalues of Σ.

Answer: The eigenvalues can be found by solving for the following determinant equation det (Σ — λI) = 0 39

Dimensionality Reduction

40

(17.6 — λ)(88.4 — λ) — 382 = λ 2 — 106λ + 1555.84 — 1444 = λ 2 — 106λ + 111.84 = 0

2

2

2

2

2

2 (c) What is the “intrinsic” dimensionality of this dataset (discounting some small amount of variance)?

(d) Compute the first principal component.

38 38 38 38

1 1.09

0.435 1

y

0

x y

0 0

0.399 0.917

(e) If the µ and Σ from above characterize the normal distribution from which the points were generated, sketch the orientation/extent of the 2-dimensional normal density function.

Answer: Figure 7.1 plots the normal distribution shape, along with the first PC and the mean.

5 4 , answer the following questions: 4 5 (a) Compute the eigenvalues of Σ by solving the equation det(Σ — λI) = 0.

Q2. Given the covariance matrix Σ =

Answer: We have to solve the following equation: (5 — λ)2 — 1 = 0 λ2 — 10λ

9

0

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

7.6 Exercises

4 11

X2

—10 —11 —12 —13 —14 —15

X1 4

5

6

7

Figure 7.1. Normal Distribution

(λ — 9)(λ — 1) = 0

(b) Find the corresponding eigenvectors by solving the equation Σui = λiui.

Answer: We can get the corresponding eigenvectors as follows. For λ1 = 9, we

y

4

1

1 2 1

1

0

0.707 0.707

Since the second eigenvector must be orthogonal, we immediately have 1

—1 1

0.707 0.707

Q3. Compute the singular values and the left and right singular vectors of the following matrix: 1 1 0 A= 0 0 1

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Dimensionality Reduction

42

Answer: The right singular vectors are the eigenvectors of ATA, and the left singular vectors are the eigenvectors of AAT. The eigenvalues of both matrices are the same and are the squares of the singular values. ,1

AT

A = ,1 0

1 1 0

0, 0, 1

AAT =

2 0

0 1

It is clear that t h,e eigenvalues of AAT are λ1 = 2 and λ2 = 1, and thus the singular values are δ1 = 2 and δ2 = 1.

For the left singular vectors, not that AAT is diagonal, which implies that the standard basis vectors constitute the eigenvectors, i.e., l1 = (1, 0)T, and l2 = (0, 1)T. We can also compute the dominant left singular vector via the power method.

Starting with x0 = (1, 1)T, we have AAT

1 2 = 1 1

AAT

2 4 = 1 1

AAT

4 8 = 1 1

After n iterations, the dominant eigenvector will be (2n, 1)T, or after normalization (2n/2n, 1/2n)T = (1, 0)T. The other left singular vector should be orthogonal to the first one, so we have L=

1 0 0 1

For the right singular vector note that ATA is rank deficient, having a rank of 2 instead of 3, e.g., the first two columns (and rows) are the same. It is clear that the r2 = (0, 0, 1)T is an eigenvector which corresponds to the eigenvalue 1. Now, we can compute the dominant right singular vector via the power method as follows.

Starting with x0 = (1, 1, 1)T, we have ATA 1, 1, 1

T

= 2, 2, 1

T

ATA 2, 2, 1

T

= 4, 4, 1

T

ATA 4, 4, 1

T

= 8, 8, 1

T

, , The dominant right singular vector is r1 = ,1 (1, 1, 0)T = (1/ 2, 1/ 2, 0)T, and the 2 , , third vector is orthogonal to this one, and is given as r3 = (—1/ 2, 1/ 2, 0)T. Thus, we have , , 0.707 0 —0.707 R = ,0.707 0 0.707 , 0 1 0 This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

7.6 Exercises

43

1 0

0 1

2 0

0 1

0.707 0 0.707

0 0

0.707 0 0.707

1 0

Q4. Consider2 the data in Table 7.1. Define the kernel function as follows: K(xi , xj ) = xi — xj . Answer the following questions: (a) Compute the kernel matrix K.

0 5.86 1.46 4.64

K

5.86 0 10 1.46

1.46 10 0 5.86

4.64 1.46 5.86 0

(b) Find the first kernel principal component.

, , 1 1 1 1

, , 11.96 17.32 17.32 11.96

, , 0.69 1 1 0.69

, , 10.52 15.05 15.05 10.52

, 0.7 1 1 0.7

In the above steps, we scale the intermediate vectors by dividing by the largest

1

0.58 0.58 0.41

Table 7.1. Dataset for Q4

i

xi

x1 x4 x7 x9

(4, 2.9) (2.5, 1) (3.5, 4) (2, 2.1)

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Dimensionality Reduction

44

Q5. Given the two points x1 = (1, 2)T, and x2 = (2, 1)T, use the kernel function K(xi , xj ) = (xiTxj )2 to find the kernel principal component, by solving the equation Kc = η1c.

K

5 4

4 5

dominant eigenvalue is c1 = (0.707, 0.707)T

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

P ART TWO

FREQUENT PATTERN MINING

Table 8.1. Transaction database for Q1

tid t1 t2 t3 t4 t5 t6 t7 t8

CHAP T ER 8

itemset ABCD ACDF ACDEG ABDF BCG DFG ABG CDFG

Itemset Mining

8.5 EXERCISES Q1. Given the database in Table 8.1. (a) Using minsup = 3/8, show how the Apriori algorithm enumerates all frequent patterns from this dataset.

A — 5, B — 4, C — 5, D — 6, F — 4, G — 5 AB — 3, AC — 3, AD — 4, AF — 2, AG — 2 BC — 2, BD — 2, BF — 0, BG — 2 CD — 4, CF — 3, CG — 3 — —

ABC — 1, ABD — 2, ACD — 3, CDF — 3, CDG — 2, CFG — 2, DFG — 3

(b) With minsup = 2/8, show how FPGrowth enumerates the frequent itemsets.

47

48

Itemset Mining

Answer: Count single items, remove infrequent and sort: D(6), A(5), C(5), G(5), B(4), F(4) We now construct the FP-tree for the whole DB in that order, as shown below:

D5

A4

C3

B1

G1

F 1

C1

A1

C1

G1

G1

G1

B1

G1

F1

B1

B1

F1

F1

In this tree the items D, A, C, G, B and F are frequent, so we output them and recursively project on each in turn and generate all frequent sets ending in those items as shown next.

D4

D2

D2

A2

C1

A1

Project: GF

Project: CF

D2 Project: AF

C1

C1

B1

G1

G1

Project on F

We first project on F to get the left most tree.

In this tree first process B, but it is not frequent.

Next we process G(2), and output GF(2). Within GF ’s tree C is not frequent but D(2) is, so we output GFD(2).

Next process C(2); output CF(2). Only D(2) is frequent; output DCF(2) Next process A(2), output AF(2); since D(2) is freq, output DAF(2); Finally D(4) is freq, so output DF(4).

D2

C1

G1 A2

C1 Project B

A1

D2

C1

A1

D1

G1

D2 Project AB

A2

C1

A1

Project CB

Project GB

We next process B. This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

8.5 Exercises

49 Table 8.2. Dataset for Q2

A

B

C

D

E

1 3 5 6

2 3 4 5 6

1 2 3 5 6

1 6

2 3 4 5

D4 D3

C1

D4

A1 Project A A3

A1

C1 Project C

C1 Project G

Q2. Consider the vertical database shown in Table 8.2. Assuming that minsup = 3, enumerate all the frequent itemsets using the Eclat method.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

50

Itemset Mining

Q3. Given two k-itemsets Xa = {x1, . . ., xk—1, xa} and Xb = {x1, . . ., xk—1, xb} that share the common (k — 1)-itemset X = {x1, x2, . . ., xk—1} as a prefix, prove that sup(Xab) = sup(Xa) — |d(Xab)| where Xab = Xa ∪ Xb, and d(Xab) is the diffset of Xab.

|t(Xa)| — |t(Xab)| sup(Xa) — |d(Xab)| = |t(Xa)| — |d(Xab)|

Q4. Given the database in Table 8.3. Show all rules that one can generate from the set ABE. Table 8.3. Dataset for Q4

tid

itemset

t1 t2 t3 t4 t5 t6

ACD BCE ABCE BDE ABCE ABCD

A — 4, B — 5, E — 4, AB — 3, AE — 2, BE — 4, ABE — 2

Q5. Consider the partition algorithm for itemset mining. It divides the database into k partitions, not necessarily equal, such that D = ∪k Di, where Di is partition i, and for i=1 any i /= j , we have Di ∩ Dj = ∅. Also let ni = |Di| denote the number of transactions in This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

8.5 Exercises

5 11

partition Di. The algorithm first mines only locally frequent itemsets, that is, itemsets whose relative support is above the minsup threshold specified as a fraction. In the second step, it takes the union of all locally frequent itemsets, and computes their support in the entire database D to determine which of them are globally frequent.

Prove that if a pattern is globally frequent in the database, then it must be locally frequent in at least one partition.

|Di|. Summing up the support over all partitions, we get

Q6. Consider Figure 8.1. It shows a simple taxonomy on some food items. Each leaf is a simple item and an internal node represents a higher-level category or item. Each item (single or high-level) has a unique integer label noted under it. Consider the database composed of the simple items shown in Table 8.4 Answer the following questions:

Table 8.4. Dataset for Q6

tid 1 2 3 4 5 6 7 8

itemset 2367 1 3 4 8 11 3 9 11 1567 1 3 8 10 11 3 5 7 9 11 4 6 8 10 11 1 3 5 8 11

(a) What is the size of the itemset search space if one restricts oneself to only itemsets composed of simple items?

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

52

Itemset Mining

1

6

14

yogurt 5

2

3

4

milk 13 11

7

8

9

10

Figure 8.1. Item taxonomy for Q6.

Answer: The search space size is 211, since there are 11 simple items. (b) Let X = {x1, x2, . . . , xk} be a frequent itemset. Let us replace some xi e X with its parent in the taxonomy (provided it exists) to obtain X′, then the support of the new itemset X′ is: i. more than support of X ii. less than support of X iii. not equal to support of X iv. more than or equal to support of X v. less than or equal to support of X

(c) Use minsup = 7/8. Find all frequent itemsets composed only of high-level items in the taxonomy. Keep in mind that if a simple item appears in a transaction, then its high-level ancestors are all assumed to occur in the transaction as well.

Answer: If we replace each low-level item by all of the high level ancestors, we obtain the following set of transactions: 1: 12, 14, 15 2: 12, 14, 13, 15 3: 12, 14, 13, 15 4: 14, 15 5: 12, 14, 13, 15 6: 12, 14, 13, 15 7: 12, 14, 13, 15 8: 12, 14, 13, 15

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

8.5 Exercises

53

The frequent high level itemsets are as follows: 12 — 8, 14 — 7, 15 — 8 {12, 14} — 7, {12, 15} — 7, {14, 15} — 7 {12, 14, 15} — 7 Q7. Let D be a database with n transactions. Consider a sampling approach for mining frequent itemsets, where we extract a random sample S ⊂ D, with say m transactions, and we mine all the frequent itemsets in the sample, denoted as FS. Next, we make one complete scan of D, and for each X e FS, we find its actual support in the whole database. Some of the itemsets in the sample may not be truly frequent in the database; these are the false positives. Also, some of the true frequent itemsets in the original database may never be present in the sample at all; these are the false negatives.

Prove that if X is a false negative, then this case can be detected by counting the support in D for every itemset belonging to the negative border of FS, denoted Bd—(FS), which is defined as the set of minimal infrequent itemsets in sample S. Formally, Bd—(FS) = inf Y | sup(Y) < minsup and ∀Z ⊂ Y, sup(Z) ≥ minsup } where inf returns the minimal elements of the set.

/e

Q8. Assume that we want to mine frequent patterns from relational tables. For example consider Table 8.5, with three attributes A, B, and C, and six records. Each attribute has a domain from which it draws its values, for example, the domain of A is dom(A) = {a1, a2, a3}. Note that no record can have more than one value of a given attribute.

Table 8.5. Data for Q8

tid

A

B

C

1 2 3 4 5 6

a1 a2 a2 a2 a2 a3

b1 b3 b3 b1 b3 b3

c1 c2 c3 c1 c3 c3

We define a relational pattern P over some k attributes X1, X2, . . ., Xk to be a subset of the Cartesian product of the domains of the attributes, i.e., P ⊆ dom(X1) × This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

54

Itemset Mining

dom(X2) × · · · × dom(Xk). That is, P = P1 × P2 × · · · × Pk, where each Pi ⊆ dom(Xi ).

For example, {a1, a2} × {c1} is a possible pattern over attributes A and C, whereas {a1} × {b1} × {c1} is another pattern over attributes A, B and C. The support of relational pattern P = P1 × P2 × · · · × Pk in dataset D is defined as the number of records in the dataset that belong to it; it is given as sup(P) = {r = (r1, r2, . . ., rn) e D : ri e Pi for all Pi in P } For example, sup({a1, a2} ×{c1}) = 2, as both records 1 and 4 contribute to its support.

Note, however that the pattern {a1} × {c1} has a support of 1, since only record 1 belongs to it. Thus, relational patterns do not satisfy the Apriori property that we used for frequent itemsets, that is, subsets of a frequent relational pattern can be infrequent.

We call a relational pattern P = P1 × P2 ×···× Pk over attributes X1, . . ., Xk as valid iff for all u e Pi and all v e Pj , the pair of values (Xi = u, Xj = v) occurs together in some record. For example, {a1, a2} ×{c1} is a valid pattern since both (A = a1, C = c1) and (A = a2, C = c1) occur in some records (namely, records 1 and 4, respectively), whereas {a1, a2} ×{c2} is not a valid pattern, since there is no record that has the values (A = a1, C = c2). Thus, for a pattern to be valid every pair of values in P from distinct attributes must belong to some record.

Given that minsup = 2, find all frequent, valid, relational patterns in the dataset in Table 8.5.

{a1, a2}, {b1} — 2 {a1, a2}, {c1} — 2 {a1, a2}, {b1}, {c1} — 2 {a2, a3}, {b3} — 4 {a2, a3}, {b3}, {c3} — 2 {a2, a3}, {c3} — 3 {a2}, {b1, b3} — 4 {a2}, {b3} — 3 {a2}, {b3}, {c3} — 2 {a2}, {b3}, {c2, c3} — 3 {a2}, {c1, c2} — 2 {a2}, {c1, c3} — 3 {a2}, {c1, c2, c3} — 5 {a2}, {c2, c3} — 3 {a2}, {c3} — 2 {b1}, {c1} — 2 {b3}, {c3} — 3 {b3}, {c2, c3} — 4 Q9. Given the following multiset dataset: tid 1 2 3

multiset ABCA ABABA CABBA

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

8.5 Exercises

55

Using minsup = 2, answer the following: (a) Find all frequent multisets. Recall that a multiset is still a set (i.e., order is not important), but it allows multiple occurrences of an item.

Level 1: A — 3, B — 3, C — 2 Level 2: AA — 3, AB — 3, AC — 2, BB — 2, BC — 2

(b) Find all minimal infrequent multisets, that is, those multisets that have no infrequent sub-multisets.

Answer: In the level-wise approach above we encounter the following minimal infrequent multisets: Level 2: CC — 0

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Table 9.1. Dataset for Q2

CHAP T ER 9

Tid

Itemset

t1 t2 t3 t4 t5 t6

ACD BCE ABCE BDE ABCE ABCD

Summarizing Itemsets

9.6 EXERCISES Q1. True or False: (a) Maximal frequent itemsets are sufficient to determine all frequent itemsets with their supports.

Answer: False (b) An itemset and its closure share the same set of transactions.

(c) The set of all maximal frequent sets is a subset of the set of all closed frequent itemsets.

Answer: False (d) The set of all maximal frequent sets is the set of longest possible frequent itemsets.

Answer: False Q2. Given the database in Table 9.1 (a) Show the application of the closure operator on AE, that is, compute c(AE). Is AE closed?

Answer: c(AE) = i(t(AE)) = i(3, 5) = ABCE. Thus AE is not closed. (b) Find all frequent, closed, and maximal itemsets using minsup = 2/6.

56

9.6 Exercises

57 Table 9.2. Dataset for Q3

Tid

Itemset

1 2 3 4 5 6

ACD BCD AC ABD ABCD BCD

Figure 9.1. Closed itemset lattice for Q4.

Answer: The set of all frequent itemsets F is given as: A4, B5, C5, D3, E4 AB3, AC4, AD2, AE2, BC4, BD2, BE4, CD2, CE3 ABC3, ABE2, ACD2, ACE2, BCE3 ABCE2 The set of closed frequent itemsets C is: B5, C5, D3 AC4, BC4, BD2, BE4 ABC3, ACD2, BCE3 ABCE2 The set of all maximal frequent itemsets M is: BD2, ACD2, ABCE2 Q3. Given the database in Table 9.2, find all minimal generators using minsup = 1.

Answer: The set of all minimal generators are: A — 4, B — 4, C — 5, D — 5 AB — 2, AC — 2, AD — 2, BC — 2, CD — 2 ABC — 1, ACD — 1

Q4. Consider the frequent closed itemset lattice shown in Figure 9.1. Assume that the item space is I = {A, B, C, D, E}. Answer the following questions: (a) What is the frequency of CD? Answer: The frequency of CD is 3.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

58

Summarizing Itemsets Table 9.3. Dataset for Q7

Tid

Itemset

1 2 3 4 5 6

ACD BCD ACD ABD ABCD BC

(b) Find all frequent itemsets and their frequency, for itemsets in the subset interval [B, ABD].

B — 8, AB — 6, AD — 6, BD — 6, ABD — 6. (c) Is ADE frequent? If yes, show its support. If not, why?

Answer: ADE is not frequent since it is not a subset of any closed itemset.

Q5. Let C be the set of all closed frequent itemsets and M the set of all maximal frequent itemsets for some database. Prove that M ⊆ C.

Q6. Prove that the closure operator c = i ◦ t satisfies the following properties (X and Y are some itemsets): (a) Extensive: X ⊆ c(X) Answer: Let x e X. Consider t(X) = {t |X ⊆ i(t)}. It follows that x e i(t) for all t e t(X).

T Now, c(X) = i(t(X)) = t et(X) i(t), which implies that x e c(X). Thus X ⊆ c(X). (b) Monotonic: If X ⊆ Y then c(X) ⊆ c(Y) Answer: By definition, c(X) = i(t(X)) and C(Y) = i(t(Y)). Also, X ⊆ Y implies that t(Y) ⊆ t(X), and furtherTt(X) = t(Y) ∪ (t(X) T — t(Y)).

T = Now, c(X) = i(t(X)) = i(t) ⊆ t et(X) t et(Y) i(t) ∩ t et(X)—t(Y) i(t) T t et(Y) i(t) = i(t(Y)) = c(Y) (c) Idempotent: c(X) = c(c(X))

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

9.6 Exercises

59

Q7. Let δ be an integer. An itemset X is called a δ-free itemset iff for all subsets Y ⊂ X, we have sup(Y) — sup(X) > δ. For any itemset X, we define the δ-closure of X as follows: } δ-closure(X) = Y | X ⊂ Y, sup(X) — sup(Y) ≤ δ, and Y is maximal Consider the database shown in Table 9.3. Answer the following questions: (a) Given δ = 1, compute all the δ-free itemsets.

Answer: The δ-free sets and their closures are as follows: 1-free sets A(4) B(4) C(5) D(5) AB(2) The figure below illustrates these sets:

their closures ACD(3) BC(3), BD(3) CD(4) AD(4), CD(4) ABCD(1)

Actually, the definition allows for the empty set to be counted as δ-free (though it is not very interesting). So, if you do count ∅ as δ-free, then C and D will not be δ-free, but ∅ will be. In that case your answer for the closure will also differ. δ-closure(∅) = {C, D}. The final answer should be: 1-free sets ∅(6) A(4) B(4) AB(2)

their closures C(5), D(5) ACD(3) BC(3), BD(3) ABCD(1)

(b) For each of the δ-free itemsets, compute its δ-closure for δ = 1.

Answer: The closures are given in part (a) above.

Q8. Given the lattice of frequent itemsets (along with their supports) shown in Figure 9.2, answer the following questions: (a) List all the closed itemsets.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

60

Summarizing Itemsets

A — 6, AB — 5, AC — 4, AD — 3, ABC — 3, ABD — 1, ACD — 2, ABCD — 1 (b) Is BCD derivable? What about ABCD? What are the bounds on their supports.

3 2 2—5—4—3 6

1

Figure 9.2. Frequent itemset lattice for Q8.

Q9. Prove that if an itemset X is derivable, then so is any superset Y ⊃ X. Using this observation describe an algorithm to mine all nonderivable itemsets.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

9.6 Exercises

61

Answer: Let X = YZ, then from Eqs.(9.4) and (9.3) we conclude that sup(YZ) = |sup(X) — IE(Y)| where Σ

IE(Y) =

—1(|X\W|+1) · sup(W)

Y⊆W⊂X

Also, let the upper and lower bounds on the support of X be given as n } U(X) = min IE(Y) Y ⊆ X, |X \ Y| is odd n } L(X) = max IE(Y) Y ⊆ X, |X \ Y| is even Now let Y′ ⊆ X be the subset that minimizes the bound |sup(X) — IE(Y)|, and let Z′ = X — Y′, then Y′ = argmin{|sup(X) — IE(Y)|} Y

= argmin{sup(YZ)} Y

Now it is clear that for L(X) and U(X), we have sup(X) — L(X) ≥ sup(Y′Z′) U(X) — sup(X) ≥ sup(Y′Z′) Adding the two bounds we have U(X) — sup(X) + sup(X) — L(X) ≥ 2 · sup(Y′Z′) ≥ 2sup(Y′Z′) Let Xa be an extension of X with a new item a /e X. If |X \ Y′| is odd, then |Xa \ Y′a| is also odd, and |X \ Y′a| is even. In this case, we have sup(Xa) — L(Xa) ≤ sup(Y′aZ′) U(Xa) — sup(Xa) ≤ sup(Y′aZ′) The above two bounds imply that U(Xa) — L(Xa) ≤ sup(Y′aZ′) + sup(Y′aZ′) On the other hand, if |X \ Y′| is even, then |Xa \ Y′a| is also even, but |X \ Y′a| is odd. However, in this case too we obtain the bound: U(Xa) — L(Xa) ≤ sup(Y′aZ′) + sup(Y′aZ′) Combining all of the above, we have U(Xa) — L(Xa) ≤ sup(Y′aZ′) + sup(Y′aZ′) ≤ sup(Y′Z′) ≤

1 2

(U(X) — L(X))

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

62

Summarizing Itemsets

Finally, this result can be used in an efficient algorithm to mine all non-derivable

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

CHAP T ER 10

Sequence Mining

10.5 EXERCISES Q1. Consider the database shown in Table 10.1. Answer the following questions: (a) Let minsup = 4. Find all frequent sequences.

Answer: The frequent sequences are: A — 4, G — 4, T — 4 AA — 4, AG — 4, AT — 4, GA — 4, TA — 4, TG — 4 AAT — 4, AGA — 4, ATA — 4, ATG — 4, GAA — 4, TAA — 4 (b) Given that the alphabet is Σ = {A, C, G, T}. How many possible sequences of length k can there be?

Answer: There can be 4k sequences of length k.

Table 10.1. Sequence database for Q1

Id

Sequence

s1 s2 s3 s4

AATACAAGAAC GTATGGTGAT AACATGGCCAA AAGCGTGGTCAA

Table 10.2. Sequence database for Q2

Id

Sequence

s1 s2 s3 s4 s5 s6 s7

ACGTCACG TCGA GACTGCA CAGTC AGCT TGCAGCTC AGTCAG

63

64

Sequence Mining

Q2. Given the DNA sequence database in Table 10.2, answer the following questions using minsup = 4 (a) Find the maximal frequent sequences.

A — 7, C — 7, G — 7, T — 7 — — — — — — — — — CG — 6, CT — 5, TA — 5, TC — 6, TG — 5 ACT — 4, AGC — 6, AGT — 5, ATC — 5, GAG — 4, GCA — 4, GCG — 4, GTC — 5, CAG — 4, CGC — 4, CTC — 4, TCA — 4, TCG — 4 AGTC — 4 The maximal frequent sequences are: AGTC—4, ACT—4, GAG—4, GCG—4, GCA—4, CAG—4, CGC—4, CTC—4, TCA — 4, TCG — 4 (b) Find all the closed frequent sequences.

Answer: The closed frequent sequences are: AGTC—4, ACT—4, AGC—6, AGT—5, GAG—4, GCG—4, GCA—4, GCT—5, CAG—4, CGC—4, CTC—4, TCA—4, TCG—4, AT—6, GA—5, GT—6, CA—6, CG — 6, TA — 5, TC — 6, TG — 5, A — 7, C — 7, G — 7, T — 7. (c) Find the maximal frequent substrings.

Answer: The frequent substrings are as follows: A — 7, C — 7, G — 7, T — 7 AG — 4, CA — 5, TC — 5 The maximal substrings are: AG — 5, CA — 5, TC — 5. (d) Show how Spade would work on this dataset.

Answer: We create the vertical format for each item as follows: t (A) = (1,1) (1,6) (2,4) (3,2) (3,7) (4,2) (5,1) (6,4) (7,1) (7,5) t (C) = (1,2) (1,5) (1,7) (2,2) (3,3) (3,6) (4,1) (4,5) (5,3) (6,3) (6,6) (6,8) (7,4) t (G) = (1,3) (1,8) (2,3) (3,1) (3,5) (4,3) (5,2) (6,2) (6,5) (7,2) (7,6) t (T) = (1,4) (2,1) (3,4) (4,4) (5,4) (6,1) (6,7) (7,3) We now intersect the vertical lists as follows (only frequent intersections for prefix A are shown): t (AC) = (1,2) (1,5) (1,7) (3,3) (3,6) (4,5) (5,3) (6,6) (6,8) (7,4) t (AG) = (1,3) (1,8) (3,5) (4,3) (5,2) (6,5) (7,2) (7,6) t (AT) = (1,4) (3,4) (4,4) (5,4) (6,7) (7,3) t (ACT) = (1,4) (3,4) (6,7) (7,3) t (AGC) = (1,5) (1,7) (3,6) (4,5) (5,3) (6,8) (7,4) t (AGT) = (1,4) (4,4) (5,4) (6,7) (7,3) t (ATC) = (1,5) (1,7) (3,6) (4,5) (6,8) (7,4) t (AGTC) = (1,5) (1,7) (4,5) (6,8) (7,4)

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

10.5 Exercises

65

The rest of the frequent sequences can be found in a similar manner. (e) Show the steps of the PrefixSpan algorithm.

Answer: We show the projections only for sequences with prefix A. The projected DB for A, DA is as follows. It consists of the remaining sequence, called the suffix, after the first occurrence of an A in each input sequence: s1 s3 s4 s5 s6 s7

CGTCACG CTGCA GTC GCT GCTC GTCAG

In this DB we count the single items: A(3), C(6), G(6), T(6). So AA is not freq, but AC, AG and AT all are.

We need to recursively project on each of these sequences.

Project database DA on AC to get DAC: s1 s3 s5 s6 s7

GTCACG TGCA T TC AG

only T is frequent, to get ACT(4).

Project DA on AG to get DAG: s1 s3 s4 s5 s6 s7

TCACG CA TC CT CTC TCAG

Counts: A(3), C(6), G(2), T(5). Only C and T are frequent, so we project on those.

Project DAG on AGC: s1 ACG s3 A s5 T s6 TC s7 AG We don’t find any other frequent item.

Project DAG on AGT: s1 CACG s4 C s6 C s7 CAG This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

66

Sequence Mining

Only C is frequent, to give AGTC(4).

Project DA on AT: s1 CACG s3 GCA s4 C s6 C s7 CAG only C is frequent, to give ATC(5).

Likewise we will project on other items and get the final answer.

Q3. Given s = AABBACBBAA, and Σ = {A, B, C}. Define support as the number of occurrence of a subsequence in s. Using minsup = 2, answer the following questions: (a) Show how the vertical Spade method can be extended to mine all frequent substrings (consecutive subsequences) in s.

(b) Construct the suffix tree for s using Ukkonen’s method. Show all intermediate steps, including all suffix links.

Answer: The different steps of the suffix tree construction method are shown below.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

10.5 Exercises

67

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

68

Sequence Mining

(c) Using the suffix tree from the previous step, find all the occurrences of the query q = ABBA allowing for at most two mismatches.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

10.5 Exercises

69

Answer: The occurrences with at most two mismatches are given as matches position AABB 1 ABBA 2 ACBB 5 BBAA 7 CBBA 6 (d) Show the suffix tree when we add another character A just before the $. That is, you must undo the effect of adding the $, add the new symbol A, and then add $ back again.

[6, e] 6 [3, 3]

[1, 1]

[6, e]

[3, 5]

[4, 5]

[10, e]

[6, e]

[10, e]

8

4

7

[3, e] 2

5

[2, e]

1

[6, e]

3

[6, e] 6 [3, 3]

[1, 1]

[6, e]

[2, 2]

[3, 5]

[4, 5]

[10, e]

[6, e]

[10, e]

8

4

7

[3, e] 2 [3, e]

1

5

[11, e]

9

[6, e]

3

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

70

Sequence Mining

(e) Describe an algorithm to extract all the maximal frequent substrings from a suffix tree. Show all maximal frequent substrings in s.

Answer: In a suffix tree, each node stands for a substring (the path from root to that node). So we can extend each node in the suffix tree with a new field “support”, which is the number of occurrences of the substring the node stands for. In each internal node, the support is the number of leaf nodes of that sub-tree. In each leaf node, the number is set to 1. Then we can traverse the suffix tree. If we reach to an internal node whose support is at least minsup but its children’s support are all less than minsup, then the path from root to this internal node is a potential maximal frequent substring. The only other check for maximality we have to do is that there is no character that can be added as a prefix, which still results in a frequent substring.

In the above example, by using this algorithm we can find the maximal frequent substrings: AA, BBA. Q4. Consider a bitvector based approach for mining frequent subsequences. For instance, in Table 10.1, for s1, the symbol C occurs at positions 5 and 11. Thus, the bitvector for C in s1 is given as 00001000001. Because C does not appear in s2 its bitvector can be omitted for s2. The complete set of bitvectors for symbol C is (s1, 00001000001) (s3, 00100001100) (s4, 000100000100) Given the set of bitvectors for each symbol show how we can mine all frequent subsequences by using bit operations on the bitvectors. Show the frequent subsequences and their bitvectors using minsup = 4.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

10.5 Exercises

7 11

(s4, 110000000011)

(s4, 001010110000)

5, 6, 8. When we do the bitvector operations for AG, we obtain the following results:

Q5. Consider the database shown in Table 10.3. Each sequence comprises itemset events that happen at the same time. For example, sequence s1 can be considered to be a sequence of itemsets (AB)10(B)20(AB)30(AC)40, where symbols within brackets are considered to co-occur at the same time, which is given in the subscripts. Describe an algorithm that can mine all the frequent subsequences over itemset events. The itemsets can be of any length as long as they are frequent. Find all frequent itemset sequences with minsup = 3.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

72

Sequence Mining Table 10.3. Sequences for Q5

Id

Time

Items

s1

10 20 30 40

A, B B A, B A, C

s2

20 30 50

A, C A, B, C B

s3

10 30 40 50 60

A B A C B

30 40 50 60

A, B A B C

s4

A — 4, B — 4, C — 4 AA — 4, {A, B} — 3, AB — 4, AC — 4, BA — 3, BB — 4 — — — — —

Q6. The suffix tree shown in Figure 10.5 contains all suffixes for the three sequences s1, s2, s3 in Table 10.1. Note that a pair (i, j) in a leaf denotes the j th suffix of sequence si. (a) Add a new sequence s4 = GAAGCAGAA to the existing suffix tree, using the Ukkonen algorithm. Show the last character position (e), along with the suffixes (l) as they become explicit in the tree for s4. Show the final suffix tree after all suffixes of s4 have become explicit.

Finally, when we consider the terminal character $, all the suffixes will become

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

10.5 Exercises

73

4

4

3

3

3

4

2

4

2

(2,5)

(1,1)

3

4

(4,5)

3

3

(b) Find all closed frequent substrings with minsup = 2 using the final suffix tree.

Q7. Given the following three sequences:

s1 : GAAGT s2 : CAGAT This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

74

Sequence Mining

s3 : ACGT Find all the frequent subsequences with minsup = 2, but allowing at most a gap of 1 position between successive sequence elements.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

CHAP T ER 11

Graph Pattern Mining

11.5 EXERCISES Q1. Find the canonical DFS code for the graph in Figure 11.1. Try to eliminate some codes without generating the complete search tree. For example, you can eliminate a code if you can show that it will have a larger code than some other code.

a

b

a

d

b

a

a

Figure 11.1. Graph for Q1.

a

3 b

c

5

4 a

d

b 7

a 8

6 a

75

Graph Pattern Mining

76

Q2. Given the graph in Figure 11.2. Mine all the frequent subgraphs with minsup = 1. For each frequent subgraph, also show its canonical code.

a

a

a

a Figure 11.2. Graph for Q2.

Answer: We show the DFS code for each pattern.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

11.5 Exercises

77 0 a | 1 a 0 1 a a

subgraphs with 2 edges: 2) extending 1) 0 a | 1 a | 2 a 0 1 a a 1 2 a a 3) extending 1) a | \ a a This subgraph is isomorphic to the previous one 2). subgraphs with 3 edges: 4) extending 2) 0 a | \ 1 a | | / 2 a 0 1 a a 1 2 a a 2 0 a a 5) extending 2) 0 a | 1 a | 2 a | 3 a

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Graph Pattern Mining

78 0 1 a a 1 2 a a 2 3 a a 6) extending 2) 0 a | 1 a |\ 2 a a 3 0 1 a a 1 2 a a 1 3 a a 7) extending 2) a |\ a a | a This subgraph is isomorphic to 5). subgraphs with 4 edges: 8) extending 4) 0 a | \ 1 a | | / 2 a | 3 a 0 1 a a 1 2 a a 2 0 a a 2 3 a a 9) extending 5) 0 a | \ 1 a | | | 2 a | | / 3 a/

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

11.5 Exercises

79

Q3. Consider the graph shown in Figure 11.3. Show all its isomorphic graphs and their DFS codes, and find the canonical representative (you may omit isomorphic graphs that can definitely not have canonical codes).

Answer: The figure below shows the potential isomorphic graphs that can have minimal DFS codes.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Graph Pattern Mining

80

a b a a

b

Figure 11.3. Graph for Q3.

G1 (0, 1, A, A, a) (1, 2, A, A, b) (2, 0, A, A, a) (1, 3, A, B, a) (0, 4, A, A, b)

G2 (0, 1, A, A, a) (1, 2, A, A, a) (2, 0, A, A, b) (1, 3, A, A, b) (0, 4, A, B, a)

G3 (0, 1, A, A, a) (1, 2, A, A, b) (2, 0, A, A, a) (2, 3, A, B, a) (0, 4, A, A, b)

G5 (0,1,A,A,a) (1,2,A,A,a) (2,0,A,A,b) (2,3,A,B,a) (1,4,A,A,b)

Q4. Given the graphs in Figure 11.4, separate them into isomorphic groups.

Q5. Given the graph in Figure 11.5. Find the maximum DFS code for the graph, subject to the constraint that all extensions (whether forward or backward) are done only from the right most path.

Answer: The maximum dfs code is for the graph shown below.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

11.5 Exercises

8 11 G1

G2

G3

G4

a

a

a

b

a

a

a

a

b

b

b

b

b

b

a G5

G6

G7

a

a

a

a

b

a

b

b

b

a

b

b Figure 11.4. Data for Q4.

a

b

a Figure 11.5. Graph for Q5.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Graph Pattern Mining

82

0,1,C,C 1,2,C,C 2,3,C,C 3,4,C,B 4,5,B,A 5,2,A,C 3,6,C,A 6,2,A,C 6,1,A,C 6,0,A,C 3,1,C,C

going depth-first or branching. In the maximum code, we still have to go depth-first,

Q6. For an edge labeled undirected graph G = (V, E), define its labeled adjacency matrix A as follows:

,

L(vi ) if i = j L(v , v ) if (vi , vj ) e E A(i, j) = i j ,0 Otherwise

where L(vi ) is the label for vertex vi and L(vi , vj ) is the label for edge (vi , vj ). In other words, the labeled adjacency matrix has the node labels on the main diagonal, and it has the label of the edge (vi , vj ) in cell A(i, j). Finally, a 0 in cell A(i, j) means that there is no edge between vi and vj .

Given a particular permutation of the vertices, a matrix code for the graph is obtained by concatenating the lower triangular submatrix of A row-by-row. For example, one possible matrix corresponding to the default vertex permutation v0v1v2v3v4v5 for the graph in Figure 11.6 is given as

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

11.5 Exercises

83

v0

v1

v2

a

b

b

b

b

a

v3

v4

v5

Figure 11.6. Graph for Q6.

a x 0

b y

b

0 0 0

y 0 0

y y 0

b y 0

b z

a

The code for the matrix above is axb0yb0yyb00yyb0000za. Given the total ordering on the labels 0<a<b<x <y <z find the maximum matrix code for the graph in Figure 11.6. That is, among all possible vertex permutations and the corresponding matrix codes, you have to choose the lexicographically largest code.

b y y y 0 0

b y y 0 0

b 0 z 0

b 0 x

a 0

a

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

CHAP T ER 12

Pattern and Rule Assessment

12.4 EXERCISES Q1. Show that if X and Y are independent, then conv(X —→ Y) = 1.

Q2. Show that if X and Y are independent then oddsratio(X —→ Y) = 1.

= 1 — P(X) — P(Y) — P(X)P(Y) = (1 — P(X)) · (1 — P(Y))

84

12.4 Exercises

85 Table 12.1. Data for Q5

Support

No. of samples

10,000 15,000 20,000 25,000 30,000 35,000 40,000 45,000

5 20 40 50 20 50 5 10

Q3. Show that for a frequent itemset X, the value of the relative lift statistic defined in Example 12.20 lies in the range h i 1 — |D|/minsup, 1

cannot exceed 1 since support cannot be negative. Now, the minimum support of X in D can be minsup, whereas the maximum support of X in Di can be |D|, therefore, |D|

Q4. Prove that all subsets of a minimal generator must themselves be minimal generators.

Q5. Let D be a binary database spanning one trillion (109) transactions. Because it is too time consuming to mine it directly, we use Monte Carlo sampling to find the bounds on the frequency of a given itemset X. We run 200 sampling trials Di (i = 1 . . . 200), with each sample of size 100, 000, and we obtain the support values for X in the various samples, as shown in Table 12.1. The table shows the number of samples where the support of the itemset was a given value. For instance, in 5 samples its support was 10,000. Answer the following questions: (a) Draw a histogram for the table, and calculate the mean and variance of the support across the different samples.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Pattern and Rule Assessment

86

Answer: The histogram is plotted in the figure below:

Let ni be the number of samples where X has frequency fi. The mean and variance for the itemset are given as: Σ nifi 5400000 µ = Σi = = 27000 200 i ni Σ 2 1395 × 107 i ni (fi — µ) 2 σ = Σ = = 69750000 200 i ni The standard deviation is therefore σ = 8351.647 Note that with respect to the whole dataset, the mean relative support is: µ = 27000/109 = 2.75 × 10—5, and its variance is σ 2 = 6.975 × 10—11, with standard deviation σ = 8.352 × 10—6. (b) Find the lower and upper bound on the support of X at the 95% confidence level.

The support values given should be for the entire database D.

Answer: We assume that support follows a normal distribution, for which the

(µ — 1.96 × σ, µ + 1.96 × σ) = (2.7 × 10—5 — 1.637 × 10—5, 2.7 × 10—5 + 1.637 × 10—5 = (1.0631, 4.3369) × 10—5 In terms of absolute support we have the bounds (10631, 43369). (c) Assume that minsup = 0.25, and let the observed support of X in a sample be sup(X) = 32500. Set up a hypothesis testing framework to check if the support of X is significantly higher than the minsup value. What is the p-value?

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

12.4 Exercises

87

Q6. Let A and B be two binary attributes. While mining association rules at 30% minimum support and 60% minimum confidence, the following rule was mined: A —→ B, with sup = 0.4, and conf = 0.66. Assume that there are a total of 10,000 customers, and that 4000 of them buy both A and B; 2000 buy A but not B, 3500 buy B but not A, and 500 buy neither A nor B. Compute the dependence between A and B via the χ 2 -statistic from the corresponding contingency table. Do you think the discovered association is truly a strong rule, that is, does A predict B strongly? Set up a hypothesis testing framework, writing down the null and alternate hypotheses, to answer the above question, at the 95% confidence level. Here are some values of chi-squared statistic for the 95% confidence level for various degrees of freedom (df):

marginal

df

χ2

1 2 3 4 5 6

3.84 5.99 7.82 9.49 11.07 12.59

4000 3500 7500 0.75

2000 500 2500 0.25

6000 4000 10000

0.60 0.40

1500 1000 —

166.67 250 This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Pattern and Rule Assessment

88

2

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

P ART THREE

CLUSTERING

CHAP T ER 13

Representative-based Clustering

13.5 EXERCISES Q1. Given the following points: 2, 4, 10, 12, 3, 20, 30, 11, 25. Assume k = 3, and that we randomly pick the initial means µ1 = 2, µ2 = 4 and µ3 = 6. Show the clusters obtained using K-means algorithm after one iteration, and show the new means for the next iteration.

C3 = {10, 11, 12, 20, 25, 30}

91

Representative-based Clustering

92 Table 13.1. Dataset for Q2

x

P(C1|x)

P(C2|x)

2 3 7 9 2 1

0.9 0.8 0.3 0.1 0.9 0.8

0.1 0.1 0.7 0.9 0.1 0.2

Q2. Given the data points in Table 13.1, and their probability of belonging to two clusters.

Assume that these points were produced by a mixture of two univariate normal distributions. Answer the following questions: (a) Find the maximum likelihood estimate of the means µ1 and µ2.

Answer: Based on the maximum likelihood equations, we know that: Σn x · P(Ci |xj ) µi = Σj n=1 j P(C |x ) j =1

i

j

We get the following counts: x 2 3 7 9 2 1 Sum

P(C1|x) 0.9 0.8 0.3 0.1 0.9 0.8 3.8

P(C2|x) 0.1 0.1 0.7 0.9 0.1 0.2 2.1

x · P(C1|x) 1.8 2.4 2.1 0.9 1.8 0.8 9.8

x · P(C2|x) 0.2 0.3 4.9 8.1 0.2 0.2 13.9

Therefore, µ1 = 9.8/3.8 = 2.58 and µ2 = 13.9/2.1 = 6.62. (b) Assume that µ1 = 2, µ2 = 7, and σ1 = σ2 = 1. Find the probability that the point x = 5 belongs to cluster C1 and to cluster C2. You may assume that the prior probability of each cluster is equal (i.e., P(C1) = P(C2) = 0.5), and the prior probability P(x = 5) = 0.029.

1

—(5—2) /2

1

—(5—7) /2

0.93.

Q3. Given the two-dimensional points in Table 13.2, assume that k = 2, and that initially the points are assigned to clusters as follows: C1 = {x1, x2, x4} and C2 = {x3, x5}.

Answer the following questions: (a) Apply the K-means algorithm until convergence, that is, the clusters do not change, assuming (1) the usual Euclidean distance or the L2-norm as the distance This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

13.5 Exercises

93 Table 13.2. Dataset for Q3

x1 x2 x3 x4 x5

X1

X2

0 0 1.5 5 5

2 0 0 0 2

between points, defined as ¨x i— xj ¨ =

Σd

1/2

(x ia — xja )2 , and (2) the Σd ¨¨ ¨ Manhattan distance or the L1-norm defined as xi — xj ¨ = a=1 |xia — xja|.

2

a=1

1

Answer: First, we consider the Euclidean distance. Initially, the two means are given as µ1 = (5/3, 2/3)T = (1.67, 0.67)T and µ2 = (6.5/2, 2/2)T = (3.25, 1)T. We compute the distance of each point to the cluster means, and assign it to the nearest mean, as follows: d(xi , µ1) d(xi , µ2) Cluster x1 2.1 3.4 c1 x2 1.8 3.4 c1 x3 0.7 2.0 c1 x4 3.4 2.0 c2 x5 3.6 2.0 c2 For the next iteration, we recompute the means, as follows: µ1 = (1.5/3, 2/3)T = (0.5, 0.67)T and µ2 = (10/2, 2/2)T = (5, 1)T. The new cluster assignments for the points are follows: d(xi , µ1) d(xi , µ2) Cluster x1 1.42 5.1 c1 x2 0.83 5.1 c1 x3 1.2 3.6 c1 x4 4.5 1.0 c2 x5 4.7 1.0 c2 Since there is no change in cluster assignments, so we stop.

Now we consider the Manhattan distance. From the two means, µ1 = (1.67, 0.67)T and µ2 = (3.25, 1)T, we compute the distance of each point to the cluster means, and assign it to the nearest mean, as follows: d(xi , µ1) d(xi , µ2) Cluster x1 3 4.25 c1 x2 2.34 4.25 c1 x3 0.84 2.75 c1 x4 4 2.75 c2 x5 4.66 2.75 c2 The assignments are the same as for Euclidean distance. For the next iteration, we recompute the means, as follows: µ1 = (1.5/3, 2/3)T = (0.5, 0.67) T and µ2 = (10/2, 2/2)T = (5, 1)T. The new cluster assignments for the points are follows: This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Representative-based Clustering

94

d(xi , µ1) 1.83 1.17 1.67 5.17 5.83

x1 x2 x3 x4 x5

d(xi , µ2) 6 6 4.5 1.0 1.0

Cluster c1 c1 c1 c2 c2

(b) Apply the EM algorithm with k = 2 assuming that the dimensions are independent.

Show one complete execution of the expectation and the maximization steps.

Start with the assumption that P(Ci|xja) = 0.5 for a = 1, 2 and j = 1,..., 5.

0 2

1

0 0

1.5 0

5 0

2

2.3 0.8

5 2

0.5 · 5 =

2.3 0.8

To compute the covariance matrix for C1, we first subtract the mean from each 2.3 2.3 —0.8 —0.8 2.7 0.8 2.7 1.2

0 1.44

5.29 0

1

7.29 0 25.8 0

0 0.64 0 4.8

5

5.29 0

0 0.64

7.29 0

0 1.44

5.16 0

0 0.96

0.64 0

0 0.64

0.5 · 5

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

13.5 Exercises

95

xi x1 x2 x3 x4 x5

f (xi |Σj , µj ) 0.0202 0.0307 0.4816 0.0253 0.0167

P(C1|xi) 0.5 0.5 0.5 0.5 0.5

P(C2|xi) 0.5 0.5 0.5 0.5 0.5

prior probability, mean and covariance matrix.

Q4. Given the categorical database in Table 13.3. Find k = 2 clusters in this data using the EM method. Assume that each attribute is independent, and that the domain of each attribute is {A, C, T}. Initially assume that the points are partitioned as follows: C1 = {x1, x4}, and C2 = {x2, x3}. Assume that P(C1) = P(C2) = 0.5.

Table 13.3. Dataset for Q4

x1 x2 x3 x4

X1 A A C A

X2 T A C C

The probability of an attribute value given a cluster is given as P(x |C ) = No. of times the symbol xja occurs in cluster Ci ja i No. of objects in cluster Ci for a = 1, 2. The probability of a point given a cluster is then given as .

2

P(xj |Ci) =

P(xja|Ci)

a=1

Instead of computing the mean for each cluster, generate a partition of the objects by doing a hard assignment. That is, in the expectation step compute P(Ci|xj ), and in the maximization step assign the point xj to the cluster with the largest P(Ci|xj ) value, which gives a new partitioning of the points. Show one full iteration of the EM algorithm and show the resulting clusters.

Expectation Step: First consider cluster C1. For attribute X1, we have:

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Representative-based Clustering

96

Assuming attribute independence, we can now compute the following values: P(x1|C1) = P(X1 = A|C1) · P(X2 = T|C1) = 1 · 1/2 = 1/2 P(x1|C2) = P(X1 = A|C2) · P(X2 = T|C2) = 1/2 · 0 = 0 P(x2|C1) = P(X1 = A|C1) · P(X2 = A|C1) = 1 · 0 = 0 P(x2|C2) = P(X1 = A|C2) · P(X2 = A|C2) = 1/2 · 1/2 = 1/4 P(x3|C1) = P(X1 = C|C1) · P(X2 = C|C1) = 0 · 1/2 = 0 P(x3|C2) = P(X1 = C|C2) · P(X2 = C|C2) = 1/2 · 1/2 = 1/4 P(x4|C1) = P(X1 = A|C1) · P(X2 = C|C1) = 1 · 1/2 = 1/2 P(x4|C2) = P(X1 = A|C2) · P(X2 = C|C2) = 1/2 · 1/2 = 1/4 We can then compute: P(x1) = P(x1|C1)P(C1) + P(x1|C2)P(C2) = 1/2 · 1/2 + 0 = 1/4 P(x2) = P(x2|C1)P(C1) + P(x2|C2)P(C2) = 0 + 1/4 · 1/2 = 1/8 P(x3) = P(x3|C1)P(C1) + P(x3|C2)P(C2) = 0 + 1/4 · 1/2 = 1/8 P(x4) = P(x4|C1)P(C1) + P(x4|C2)P(C2) = 1/2 · 1/2 + 1/4 · 1/2 = 1/4 + 1/8 = 3/8 Using Bayes theorem we get: P(C1|x1) = 1 P(C2|x1) = 0 P(C1|x2) = 0 P(C2|x2) = 1 P(C1|x3) = 0 P(C2|x3) = 1 P(C1|x4) = 2/3 P(C2|x4) = 1/3 Maximization Step: Based on the probabilities above, we obtain the partitions C1 = {x1, x4} and C2 = {x2, x3}.

Q5. Given the points in Table 13.4, assume that there are two clusters: C1 and C2, with µ1 = (0.5, 4.5, 2.5)T and µ2 = (2.5, 2, 1.5)T. Initially assign each point to the closest mean, and compute the covariance matrices Σi and the prior probabilities P(C i ) for i = 1, 2. Next, answer which cluster is more likely to have produced x8?

Answer: For x8 = (2.5, 3.5, 2.8)T, we first compute the distance from the two means

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

13.5 Exercises

97 Table 13.4. Dataset for Q5

X1

X2

X3

x1

0.5

4.5

2.5

x2

2.2

1.5

0.1

x3

3.9

3.5

1.1

x4

2.1

1.9

4.9

x5

0.5

3.2

1.2

x6

0.8

4.3

2.6

x7

2.7

1.1

3.1

x8

2.5

3.5

2.8

x9

2.8

3.9

1.5

x10

0.1

4.1

2.9

P(x8|C1) = c · exp—5.09/2 = 0.0785c P(x8|C2) = c · exp—3.94/2 = 0.1395c where c = 1/(2π)3/2. The probability of point x8 is P(x8) = (0.0785+0.1395)· 0.5· c = 0.218 · 0.5 · c.

P(C2|x8) = 1 — 0.36 = 0.64

Q6. Consider the data in Table 13.5. Answer the following questions: (a) Compute the kernel matrix K between the points assuming the following kernel: K(xi , xj ) = 1 + xiTxj

K

2.33 1.65 1.87 2.18

1.65 1.62 1.69 1.58

1.87 1.69 1.81 1.78

2.18 1.58 1.78 2.05

(b) Assume initial cluster assignments of C1 = {x1, x2} and C2 = {x3, x4}. Using kernel K-means, which cluster should x1 belong to in the next step?

Answer: Using φ 2

2

1 xa

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Representative-based Clustering

98 Table 13.5. Data for Q6

x1 x2 x3 x4

X1

X2

X3

0.4 0.5 0.6 0.4

0.9 0.1 0.3 0.8

0.6 0.6 0.6 0.5

The distance of x1 to C1 is given as: 2.33 — (2.33 + 1.65) + 1/4 · (2.33 + 1.65 + 1.65 + 1.64) = 0.165 and the distance to C2 as: 2.33 — (1.87 + 2.18) + 1/4 · (1.81 + 1.78 + 1.78 + 2.05) = 0.135 Thus, x1 should belong to C2 in the next step.

Q7.

Prove the following equivalence for the multivariate normal density function: ∂ f (x |µ , Σ ) = f(x |µ , Σ ) Σ—1 (x — µ ) j i i j i j i i i ∂µi 1

1 2

d

∂

—d

∂

1

∂

∂

1 2

1

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

CHAP T ER 14

Hierarchical Clustering

14.4 EXERCISES Q1. Consider the 5-dimensional categorical data shown in Table 14.1.

Table 14.1. Data for Q1

Point

X1

X2

X3

X4

X5

x1 x2 x3 x4 x5 x6

1 1 0 0 1 0

0 1 0 1 0 1

1 0 1 0 1 1

1 1 1 1 0 0

0 0 0 0 1 0

The similarity between categorical data points can be computed in terms of the number of matches and mismatches for the different attributes. Let n11 be the number of attributes on which two points xi and xj assume the value 1, and let n10 denote the number of attributes where xi takes value 1, but xj takes on the value of 0. Define n01 and n00 in a similar manner. The contingency table for measuring the similarity is then given as

xi

1 0

xj 1 0 n11 n10 n01 n00

Define the following similarity measures: • Simple matching coefficient: SMC(Xi , Xj ) = • •

11 Jaccard coefficient: JC(Xi , Xj ) = n11 +nn10 +n01 n11 Rao’s coefficient: RC(Xi , Xj ) = n11 +n10+n01+n00

n11+n00 n11 +n10+n01+n00

Find the cluster dendrograms produced by the hierarchical clustering algorithm under the following scenarios: 99

Hierarchical Clustering

100

(a) We use single link with RC. distance is simply 1 — RC(xi , xj ). The pair-wise RC distance matrix is given as:

x1 x2 x3 x4 x5

x2 3/5

x3 3/5 4/5

x4 4/5 3/5 4/5

x5 3/5 4/5 4/5 5/5

x6 4/5 4/5 4/5 4/5 4/5

smallest index. The first merge is therefore for x1 and x2. We get the new matrix:

x1, x2 x3 x4 x5

x3 3/5

x1, x2, x3 x4 x5

1 2 3 4 5 | | | | | 12 | | | | | | | 123 | | | | | 1234 | | | 12345 |

x4 3/5 4/5

x5 3/5 4/5 5/5

x6 4/5 4/5 4/5 4/5

x4 3/5

x5 3/5 5/5

x6 4/5 4/5 4/5

6 Distance | | 3/5 | | 3/5 | | 3/5 | | 3/5 |

(b) We use complete link with SMC. Answer: The distances based on SMC are given as: 1 — SMC(xi , xj ). The corresponding distance matrix is given as: x1 x2 x3 x4 x5

x2 2/5

x3 1/5 3/5

x4 3/5 1/5 2/5

x5 2/5 4/5 3/5 5/5

x6 3/5 3/5 2/5 2/5 3/5

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

14.4 Exercises

1011

x1, x3 x2, x4 x5

1 |

3 |

2 4 6 | | | 13 24 | | | | 1234 | | | 12346 |

x2, x4 3/5

x5 3/5 5/5

x6 3/5 3/5 3/5

5 Distance | | 1/5 | | 3/5 | | 3/5 |

(c) We use group average with JC. given as: x1 x2 x3 x4 x5

x2 2/4

x3 1/3 3/4

x4 3/4 1/3 2/3

x5 2/4 4/5 3/4 5/5

x6 3/4 3/4 2/3 2/3 3/4

is:

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Hierarchical Clustering

102 1 3 2 4 5 6 Distance | | | | | | 13 24 | | 1/3 | | | | | | 56 0.6 | | | 1234 | 0.625 | | 123456

Q2. Given the dataset in Figure 14.1, show the dendrogram resulting from the single -link hierarchical agglomerative clustering approach using the L1-norm as the distance between points Σ 2

δ(x, y) =

|xia — yia|

a=1

Whenever there is a choice, merge the cluster that has the lexicographically smallest labeled point. Show the cluster merge order in the tree, stopping when you have k = 4 clusters. Show the full distance matrix at each step.

Answer: We start with the distance matrix between the points.

a b c d e f g

b 2

c 4 2

d 7 7 5

e 6 6 4 1

f 4 4 2 3 2

g 6 4 2 5 4 2

h 8 6 4 7 6 4 2

i 7 7 5 2 1 3 5 7

j 9 7 5 8 7 5 3 3 6

j

a b c, g, k d, e, i f h

k 5 3 1 6 5 3 1 3 4

b 2

c, g, k 4 2

d, e, i 6 6 2

f 4 4 4 2

h 8 6 2 6 4

j 9 7 2 6 5 3

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

14.4 Exercises

103

9

a

8

b

7 6

k

5

d

4

g

e

3 2 1

1

2

3

4

5

6

7

8

9

Figure 14.1. Dataset for Q2.

| | cgk ab | | | abcgk |

| | | | | | | | | | | | | | |

dei | | | |

Table 14.2. Dataset for Q3

A B C D E

A

B

C

D

E

0

1

3

2

4

0

3

2

3

0

1

3

0

5 0

Q3. Using the distance matrix from Table 14.2, use the average link method to generate hierarchical clusters. Show the merging distance thresholds.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Hierarchical Clustering

104

Q4. Prove that in nthe Lance–Williams formula [Eq. (14.7)] (a) If αi = i , αj = nj , β = 0 and γ = 0, then we obtain the group average ni +nj

ni +nj

measure.

1

δ(x, y) 1

1

δ(x, y) +

δ(x, y)

δ(x, y) +

δ(x, y)

We can see that this matches the Lance-Williams formula. +n (b) If αi = ni +nr , αj = nj r , β = —nr and γ = 0, then we obtain Ward’s ni +nj +nr

ni +nj +nr

measure.

ni +nj +nr

2

— 2

2

2

2

2

— 2

2

—2

n n 2

2

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

14.4 Exercises

105

¨

¨

¨

µ ¨

¨µi — µj ¨

2

¨ ¨2 ¨

¨2 ¨ ¨2

¨

¨2

1

¨ ¨

¨

¨2

2

¨ ¨2

¨

¨

2

Q5. If we treat each point as a vertex, and add edges between two nodes with distance less than some threshold value, then the single-link method corresponds to a well known graph algorithm. Describe this graph-based algorithm to hierarchically cluster the nodes via single-link measure, using successively higher distance thresholds.

Answer:

Define the (complete) weighted graph over the points, where the weights

single-link are precisely the connected components of the distance restricted graph.

{d, e, i}. Next, when we raise the threshold to 2, we get two connected components,

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

CHAP T ER 15

Density-based Clustering

15.5 EXERCISES Q1. Consider Figure 15.1 and answer the following questions, assuming that we use the Euclidean distance between points, and that ǫ = 2 and minpts = 3 (a) List all the core points.

(b) Is a directly density reachable from d?

(c) Is o density reachable from i? Show the intermediate points on the chain or the point where the chain breaks.

(d) Is density reachable a symmetric relationship, that is, if x is density reachable from y, does it imply that y is density reachable from x? Why or why not?

(e) Is l density connected to x? Show the intermediate points that make them density connected or violate the property, respectively.

(f) Is density connected a symmetric relationship?

(g) Show the density-based clusters and the noise points.

106

15.5 Exercises

107

Answer: There are 2 density based clusters C1 :{a, d, h, k, p, q, r, s, t, l, v, w, x} C2 :{b, c, e, f, g, i, j, n, m, o, u} There are no noise points.

10 9

a

8

b e

d

7 6

k

5

m

o

s

4

w

3 2 1

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

Figure 15.1. Dataset for Q1.

Q2. Consider the points in Figure 15.2. Define the following distance measures: L (x, y) = ∞

|x — y |} i

i=1

L 1 (x, y) = 2

L

d

max d Σ i=1

(x, y) =

min

Lpow(x, y) =

d

1

|xi — yi | 2

2

} |x — y |

min

i

i=1 d Σ

i

i

2i—1(xi — yi)2

1/2

i=1

(a) Using ǫ = 2, minpts = 5, and L∞ distance, find all core, border, and noise points.

Answer: The core points are c, f, g, k. The border points are b, e, h. The noise points are a, d, i, j . (b) Show the shape of the ball of radius ǫ = 4 using the L 1 distance. Using minpts = 3 show all the clusters found by DBSCAN.

2

Answer: The shape is This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Density-based Clustering

108

The cluster is {a, b, c, d, e, f, g, h, i, k}. The outlier is j . (c) Using ǫ = 1, minpts = 6, and Lmin, list all core, border, and noise points.

(d) Using ǫ = 4, minpts = 3, and Lpow, show all clusters found by DBSCAN. given as:

a b c d e f g

b 1.73

c 4.36 2.83

d 6.4 5.83 4.24

e 6 5.20 3.32 1

f 5.66 4.36 1.73 3 2

g 6 4.36 1.73 5 4 2

h 6.93 5.2 3.32 7 6 4 2

i 7.35 6.4 4.12 1.73 1.41 2.45 4.24 6.16

j 9 7.35 4.69 6.63 5.74 4.12 3 3 5.20

k 4.69 3 1 5.20 4.24 2.45 1.41 2.45 4.9 4.36

Q3. Consider the points shown in Figure 15.2. Define the following two kernels: ( K1(z) =

1 If L∞(z, 0) ≤ 1

( K2(z) =

0 Otherwise Σ 1 If d |zj | ≤ 1 j =1 0 Otherwise

Using each of the two kernels K1 and K2, answer the following questions assuming that h = 2: (a) What is the probability density at e?

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

15.5 Exercises

109

9 a

8

b

7 6

k

5 d

4

g

e

3 2 1 1

2

3

4

5

6

7

8

9

Figure 15.2. Dataset for Q2 and Q3.

1 K 1 (e 11 · 2 2 1

1 K 2 (e 11 · 2 2 1

1

1

1

2

2

2

(b) What is the gradient at e?

1 1 0

2 0

0 1

1 176

1

(c) List all the density attractors for this dataset.

Q4. The Hessian matrix is defined as the set of partial derivatives of the gradient vector with respect to x. What is the Hessian matrix for the Gaussian kernel? Use the gradient in Eq. (15.6).

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Density-based Clustering

110

∂ ∂x

·

·

1

xi —

1 2

∂ ∂x

∂ 1 ∂x nh 1

1

· (xi — x) 1

I

2

Q5. Let us compute the probability density at a point x using the k-nearest neighbor approach, given as k fˆ(x) = nVx where k is the number of nearest neighbors, n is the total number of points, and Vx is the volume of the region encompassing the k nearest neighbors of x. In other words, we fix k and allow the volume to vary based on those k nearest neighbors of x. Given the following points 2, 2.5, 3, 4, 4.5, 5, 6.1 Find the peak density in this dataset, assuming k = 4. Keep in mind that this may happen at a point other than those given above. Also, a point is its own nearest neighbor.

6.1

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

CHAP T ER 16

Spectral and Graph Clustering

16.5 EXERCISES

1

A—

1

1

1

1

dd

1

1

Q2. Prove that both the normalized symmetric and asymmetric Laplacian matrices Ls [Eq. (16.6)] and La [Eq. (16.9)] are positive semidefinite. Also show that the smallest eigenvalue is λn = 0 for both.

Ls

1/2

1

1/2

—

2

111

Spectral and Graph Clustering

112

2

1

s

d1

s 1

2

s 2

s

3

s 3

s

the sense that it has the same set of non-negative eigenvalues λi ≥ 0 as the symmetric

a 1

a 2

a

the same reasoning as for the normalized symmetric Laplacian case above.

Q3. Prove that the largest eigenvalue of the normalized adjacency matrix M [Eq. (16.2)] is 1, and further that all eigenvalues satisfy the condition that |λi| ≤ 1.

Answer: The main property of a M is that it is a Markov or stochastic matrix, i.e., each row sums to 1 and each entry mij is non-negative and represents the probability of moving or transitioning from vertex i to vertex j . It is clear that M1 = 1, where 1 e Rn is the n-dimensional vector of ones. Note also that Mk is also a Markov matrix, and represents the k-step transition matrix (for a Markov chain).

We conclude that u = ,1 n1 is an eigenvector and λ = 1 is an eigenvalue of M. Now, let λ be any eigenvalue and u the corresponding eigenvector, so that Mu = λu. We assume that u 2 = 1, i.e, u is a unit vector. Consider the k-th power of M, we have Mku = λku Σ ′ Let M′ = Mk. Since the latter is also a Markov matrix, we have m′ij ≥ 0 and i m ij = ′ 1, which implies that m ij ≤ 1.

Now consider the i-th row of M′ and its dot product with u in the expression M′u = λku; we have Σ

m′ij uj = λkui

j

However, note that the 1-norm of u, namely u1 = Σ | ui|, can achieve a maximum i , value of n, since it is a fact that , u 2≤ u 1≤ n u 2 This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

16.5 Exercises

113

Σ Σ Σ Q4. Show that v r e Ci cirdrcir = nr= 1 ns=1 cir Δrs cis, where ci is the cluster indicator vector for cluster Ci and Δ is the degree matrix for the graph.

/=

Q5. For the normalized symmetric Laplacian Ls, show that for the normalized cut objective the real-valued cluster indicator vector corresponding to the smallest eigenvalue λn = 0 is given as cn =Σ,n 1 Δ1/21 i=1 di

Answer: Let us consider Ls · , 1

Δ1/21, we have

Σn

i=1 di

,Σ

a ,j /=1 1j d 1d 1

1

,Σ n

d i= 1 i

Ls Δ1/2 1 = ,

a

— , d12d Σ

···

1 2 a2j

. a 21 ,j d/=2d ··· 1 .— , d2 d1 2 2 . Σn d . . .. . . . i= 1 i . , an2 an1 — , dnd1 — , dnd2 · · · ,, Σ , , —a , a

,

,, , d , 1 . d2.

2 n.· . . . . . . , . , , Σj/=n anj , dn ,

— ,a1n

d1 d n a2n .

— ,d d .

dndn

, —a ,, 12 j,/= 1 1j , d1n 1 Σ,d1 .. . . . . ..

—a a 2n .. — ,ad211 . . j,/= 2 2j . . ,d ..

1 n .. . . .. d2 . d2 . =, . .+ . .+ · · · + . .. Σn . . . .. .. . . . . i=1 di ,, , , , ,, , Σ a j/=2 2j a n1 —a ,n2 , —, dn dn d ,

Σ

j/=1 a1j —a12—···—a1n

,

Σ

= ,Σ

1

·. n . i=1 di , Σ

,

2

d1 j/=2 a2j —a21—···—a2n , d2

. . anj —an1—···—an,n—1 , . .

j/=n

,, , d1 , . d2.

1 = 0· , . . . Σn . , d i=1 i , , dn

,

dn

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Spectral and Graph Clustering

114

1

1

2

4

3 Figure 16.1. Graph for Q6.

Q6. Given the graph in Figure 16.1, answer the following questions: (a) Cluster the graph into two clusters using ratio cut and normalized cut.

0 1 0 1

2 L

0

1 3

1 0 1 1

0 1 0 1

0 1 —1 —1 2 —1 —1 3

1 1 1 0

2 0 0 0

1 1/3 0 1/3

0 3 0 0

0 0 2 0

1/2 1 1/2 1/3

0 0 0 3

0 1/3 1 1/3

1/2 1/3 1/2 1

For the ratio cut, we have to find the two smallest eigenvalues and correspond-

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

16.5 Exercises

115

For the normalized cut, we have to find the two smallest eigenvalues and corresponding eigenvectors of La, which are as follows: λ4 = 0

λ3 = 1

, , u3 = (1/ 2, 0, —1/ 2, 0)T

u4 = (1/2, 1/2, 1/2, 1/2)T

For a clustering into k = 2 partitions, we can either get C1 = {1, 2, 3} and C2 = {4}, or we can get C1 = {1} and C2 = {2, 3, 4}, (b) Use the normalized adjacency matrix M for the graph and cluster it into two clusters using average weight and kernel K-means, using K = M + I. Answer: If we use the adjacency matrix, then the two largest eigenvalues and eigenvectors are: λ1 = 2.56

λ2 = 0

u1 = (0.44, 0.56, 0.44, 0.56)

T

, , u2 = (1/ 2, 0, —1/ 2, 0)T

For a clustering into k = 2 partitions, we can either get C1 = {1, 2, 3} and C2 = {4}, or we can get C1 = {1} and C2 = {2, 3, 4}.

If we use the normalized adjacency matrix, we have: , 0 1/3 1 M = D— A = . , 0 1/3

1/2 0 1/2 1/3

0 1/3 0 1/3

1/2, 1/3.

1/2, 0

The two largest eigenvalues and eigenvectors are: λ1 = 1

λ2 = —0.67

u1 = (1/2, 1/2, 1/2, 1/2)T

u2 = (—0.59, 0.39, —0.59, 0.39)T

However, since there is only one positive eigenvalue, we cannot cluster the data into two groups based only on u1, since all of its elements have the same value.

For kernel K-means we realize that we first have to modify the matrix K to have a value of K(x, x) = 1, otherwise, each point will be more similar to other points than to itself. So we first make sure that the diagonal in the kernel matrix K has all ones, so that , 1 1/2 0 1/2 , 1/3 1 1/3 1/3 K =M+I = . . , 0 1/2 1 1/2, 1/3 1/3 1/3 1 For Kernel K-means we start with a random split of the points. Let’s assume that C1 = {1, 2} and C2 = {3, 4}. The average of all kernel values within each 2 Σ cluster 1/ni · x,yeC iK(x, y) is 0.7075 for both clusters. Next, the average kernel Σ value of each point x with other points in a given cluster, 1/n1 · yeCi K(x, y) is as follows:

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Spectral and Graph Clustering

116

xi 1 2 3 4

C1 0.75 0.665 0.25 0.33

C2 0.25 0.33 0.75 0.665

Different answers are possible for different starting random clustering. (c) Cluster the graph using the MCL algorithm with inflation parameters r = 2 and r = 2.5.

Answer: For MCL, we first have to add self-loops to the adjacency matrix. The normalized adjacency matrix is then given as: ,1 1 A= . ,0 1

1 1 1 1

0 1 1 1

1, 1 .

1, 1

,1/3 1/4 M= . , 0 1/4

1/3 1/4 1/3 1/4

0 1/4 1/3 1/4

1/3 , 1/4 .

1/3, 1/4

Applying the expansion and inflation operations with r = 2, we have ,0.28 0.21 M·M = . ,0.17 0.21

0.28 0.29 0.28 0.29

0.17 0.21 0.28 0.21

0.28 , 0.29 .

0.28 , 0.29

,0.30 0.17 Ŵ(M, 2) = . ,0.11 0.17

0.30 0.33 0.30 0.33

0.11 0.17 0.30 0.17

0.30 , 0.33 .

0.30, 0.33

Continuing in this way the matrix converges to: ,0 0 M=. ,0 0

0.5 0.5 0.5 0.5

0 0 0 0

0.5 , 0.5 .

0.5 , 0.5

Here we can see that vertices 2 and 4 are the attractors and both 1 and 3 are attracted to it. Thus there is only one cluster comprising all the vertices in the graph.

The result of using r = 2.5 is the same, since the converge is even more rapid and we obtain the same final matrix as above, with the same clustering.

Table 16.1. Data for Q7

x1 x2 x3 x4

X1

X2

X3

0.4 0.5 0.6 0.4

0.9 0.1 0.3 0.8

0.6 0.6 0.6 0.5

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

16.5 Exercises

117

Q7. Consider Table 16.1. Assuming these are nodes in a graph, define the weighted adjacency matrix A using the linear kernel A(i, j) = 1 + xiTxj Cluster the data into two groups using the modularity objective.

Answer: We have A =

1

2.20 1.79 1.96 2.08

2.33 1.65 1.87 2.18 1.79 1.46 1.60 1.69

0.00444 0.00484 0.00303 0.00343

1.65 1.62 1.69 1.58 1.96 1.60 1.74 1.85

1.87 1.69 1.81 1.78

2.18 1.58 1.78 2.05

2.08 1.69 1.85 1.97

0.00484 0.00548 0.00323 0.00387

0.00303 0.00323 0.00224 0.00244

0.00343 0.00387 0.00244 0.00288

u1 = (0.55, —0.61, —0.38, 0.44)T. {x2, x3}.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

CHAP T ER 17

Clustering Validation

17.5 EXERCISES Q1. Prove that the maximum value of the entropy measure in Eq. (17.2) is log k.

Answer: First note that H(T |Ci) for a given cluster Ci has maximum entropy log k,

1 k

Q2. Show that if C and T are independent of each other then H(T |C) = H(T ), and further that H(C, T ) = H(C) + H(T ).

H(T |C)

—

118

17.5 Exercises

119

Q3. Show that H(T |C) = 0 if and only if T is completely determined by C. Answer: If H(T |C) = 0, then assuming that ni /= 0, it must be that H(T |Ci) = 0 for all

Q4. Show that I(C, T ) = H(C) + H(T ) — H(T , C).

I(C, T ) =

log p

—H(C, T )

= —H( C, T ) —

p log p

p

log p

—

p

log p

= H(C) + H(T ) — H(C, T )

Q5. Show that the variation of information is 0 only when C and T are identical.

Answer: When C and T are identical, we have VI(C, T ) = H(T ) + H(C) — 2I(C, T )

which follows from T

T

—

T

Q6. Prove that the maximum value of the normalized discretized Hubert statistic in Eq. (17.21) is obtained when FN = FP = 0, and the minimum value is obtained when TP = TN = 0.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Clustering Validation

120

µT(1 — µT) 2

2

= —1 2

2

Q7. Show that the Fowlkes–Mallows measure can be considered as the correlation between the pairwise indicator matrices for C and T , respectively. Define C(i, j) = 1 if xi and xj (with i /= j ) are in the same cluster, and 0 otherwise. Define T similarly Σ for the ground-truth partitions. Define (C, T⟩ = n i,j =1 Cij Tij . Show that FM = , (C,T⟩ (T,T⟩(C,C⟩

Now, (T, T⟩ is simply the number of pairs of points that are in the same partition.

of these are in the same partition, which comprise the true positives (TP), and some of these pairs are not in the same partition, which comprise the false negatives (FP).

(T,T⟩(C,C

Q8. Show that the silhouette coefficient of a point lies in the interval [—1, +1].

out

out

out

out

Q9. Show that the scatter matrix can be decomposed as S = SW + SB, where SW and SB are the within-cluster and between-cluster scatter matrices.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

17.5 Exercises

121

—

S

—

—

—

—

x

—

x x

—

x

—

—

—

x

—

—

—

—

—

x

The last step follows from the fact that —

—

Q10. Consider the dataset in Figure 17.1. Compute the silhouette coefficient for the point labeled c.

out

Thus, the silhouette coefficient for c is sc = (4.72 — 1.855)/4.72 = 5.375/7.5 = 0.61.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Clustering Validation

122 g

9 8 7

a

d

6 5

c

4 3

k

e

b

2 1

1

2

3

4

5

6

7

8

9

Figure 17.1. Data for Q10 .

Q11. Describe how one may apply the gap statistic methodology for determining the parameters of density-based clustering algorithms, such as DBSCAN and DENCLUE (see Chapter 15).

well-clustered dataset than we would expect under the null hypothesis that the data For estimating ǫ we can compute the number of points within the ǫ-neighborhood

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

P ART FOUR

CLASSIFICATION

CHAP T ER 18

Probabilistic Classification

18.5 EXERCISES Q1. Consider the dataset in Table 18.1. Classify the new point: (Age=23, Car=truck) via the full and naive Bayes approach. You may assume that the domain of Car is given as {sports, vintage, suv, truck}.

Table 18.1. Data for Q1

xi

Age

Car

Class

x1 x2 x3 x4 x5 x6

25 20 25 45 20 25

sports vintage sports suv sports suv

L H L H H H

110 4 425 4

2 0 2

from class L. 125

Probabilistic Classification

126

For Car, which is categorical, we immediately run into a problem, since the value truck does not appear in the training set. We could assume that P(truck|H) and P(truck|L) are both zero. However, we desire to have some small probability of observing each values in the domain of the attribute. We fix the problem using the pseudo-count approach of adding a count of one to the observed counts of each value for each class, as shown in the table below.

H P(sports|H) = 1(+1) = 2/8 4(+4) P(vintage|H) = 1(+1) = 2/8 4(+4)

P(suv|H) = 2(+1 ) = 3/8 4(+ 4) P(truck|H) = 0(+1) = 1/8 4(+4)

L P(sports|L) = 2(+1) = 3/6 2(+4) P(vintage|L) = 0(+1) = 1/6 2(+4)

P(suv|L) = 0(+ 1) = 1/6 2(+4) P(truck|L) = 0(+1) = 1/6 2(+4)

Using the above probabilities, we finally obtain P((23, truck)|H) = P(23|H) × P(truck|H) = 0.035 × 1/8 = 0.0044 P((23, truck)|L) = P(23|L) × P(truck|L) = 0 × 1/6 = 0 We next compute P(23, truck) = P((23, truck)|H) × P(H) + P((23, truck)|L) × P(L) 4 2 = 0.0044 × + 0 × = 0.003 6 6 We then obtain the posterior probabilities as follows 0.004× P((23, truck)|H) × P(H) 6 = 1 = 0.003 P(23, truck) 4

P(H |(23, truck)) =

0 × 26 P((23, truck)|L) × P(L) = =0 0.003 P(23, truck) Thus we classify (23, truck) as high risk (H) using Naive Bayes.

Full Bayes: We now turn out attention to the Full Bayes approach. Here we need the estimates for the joint probability of Age and Car. Now, there is obviously not enough data to estimate all the probabilities. Also, for Age, which is a numeric attribute we cannot get the probability of a specific value. Instead, we will use the cumulative distribution function for Age. To account for missing values for Car, we can assume a base occurrence of 1 for each value, but spread out uniformly among all the intervals for Age. So our estimates for the joint probabilities is based on the following cumulative occurrence counts: P(L|(23, truck)) =

class H

L

Car vintage sports suv truck vintage sports suv truck

Age < 20 1+1/5 1+1/5 1+1/5 1/5 1+1/5 1+1/5 1/5 1/5

Age ≤ 20 2+2/5 2+2/5 1+2/5 2/5 1+2/5 1+2/5 2/5 2/5

Age ≤ 25 2+3/5 2+3/5 2+3/5 3/5 1+3/5 3+3/5 3/5 3/5

Age ≤ 40 Age > 40 2+4/5 2+1 2+4/5 2+1 3+4/5 3+1 4/5 1 1+4/5 1+1 3+4/5 3+1 4/5 5/5 4/5 5/5

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

18.5 Exercises

127 Table 18 2. Data for Q2

xi

a1

a2

a3

Class

x1 x2 x3 x4 x5 x6 x7 x8 x9

T T T F F F F T F

T T F F T T F F T

5.0 7.0 8.0 3.0 7.0 4.0 5.0 6.0 1.0

Y Y N Y N N N Y N

likewise P(23, truck|L) = 1/5.

P(H|(23, truck)) ∝

Q2. Given the dataset in Table 18.2, use the naive Bayes classifier to classify the new point (T, F, 1.0).

Now, P(T, F, 1.0|Y) = 0.75 · 0.5 · 0.0106 = 0.003975 and P(T, F, 1.0|N) = 0.2 · 0.4 · 0.0502 = 0.004016 Next we have

Thus, we predict the class as N. Q3. Consider the class means and covariance matrices for classes c1 and c2: µ1 = (1, 3) Σ1 =

5 3

µ2 = (5, 5) 3 2

Σ2 =

2 0 0 1

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Probabilistic Classification

128

Classify the point (3, 4)T via the (full) Bayesian approach, assuming normally distributed classes, and P(c1) = P(c2) = 0.5. Show all steps. Recall that the inverse a b d —b of a 2 × 2 matrix A = is given as A—1 = 1 . det(A) c d —c a 1

2 3

3 5

1

1 2

0

0 1

.

Computing the mahalanobis distance we get for c1, 2 3

1/2 0 1

1

0 1

3 5

2 1

2 1

1 2

1/2

classify x as c1.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Table 19.1. Data for Q2: Age is numeric and Car is categorical. Risk gives the class label for each point: high (H) or low (L)

Point

Age

Car

Risk

x1 x2 x3 x4 x5 x6

25 20 25 45 20 25

Sports Vintage Sports SUV Sports SUV

L H L H H H

CHAP T ER 19

Decision Tree Classifier

19.4 EXERCISES Q1. True or False: (a) High entropy means that the partitions in classification are “pure.”

(b) Multiway split of a categorical attribute generally results in more pure partitions than a binary split.

Answer: True. In a binary split, there is more of a chance that the classes will be mixed. In multiway partitioning the partitions are smaller and there is more of a chance of being pure.

Q2. Given Table 19.1, construct a decision tree using a purity threshold of 100%. Use information gain as the split point evaluation measure. Next, classify the point (Age=27,Car=Vintage).

Answer: Let us consider how a complete decision tree is induced for the dataset in 4

2

129

Decision Tree Classifier

130

PL = 2 = 1 . Thus the entropy of D is 6

3

H(D) = —

2 2 1 1 log 2 + log 2 = —(—0.390 — 0.528) = 0.918 3 3 3 3

At the root of the decision tree, we consider all possible splits on Age and Car. For Age, the possible distinct splits to consider are Age ≤ 22.5 and Age ≤ 35, which were chosen to be the mid-points between the distinct values, namely, 20, 25, and 45, that we observe for Age. (a) For Age ≤ 22.5, DL includes only the points x2 and x5, whereas DR comprises the remaining points: x1, x3, x4, and x6. For DL, this yields PL = 0 and PH = 1, whereas for DR we have PL = 2 and PH = 2 . The weighted entropy is then 4

H(DL, DR)

4

4 H(DL) + H(DR) 6 6 2 4 1 1 1 1 = — (0) — log + log 2 2 6 6 2 2 2 2 2 1 = — log2 = 0.67 3 2 =

2

This yields an information gain of 0.918 — 0.67 = 0.248. (b) In a similar manner we can compute the weighted entropy for Age ≤ 35.

For DR = {x4} and DL has the remaining points, so that H(DL) = 2 5log2 25+ 3 3 5 log2 5 = 0.971 and H(DR) = 0. The split entropy is then H(DL, DR) = 5 6 (0.971) = 0.809, the information gain is: 0.915 — 0.809 = 0.106, which is not as high as for Age ≤ 22.5.

Next, we evaluate all possible splits for Car. Note that categorical data, in v general, yields 2 —1 possible splits, where v is the set of possible values for the 2 attribute. This can be reduced to O(v) by using a greedy split selection approach.

For Car the possible values are {Sports(S), Vintage(V), SUV(U)}, which yields the following three distinct splits: Car e Car /e {S} {V, U} {V} {S, U} {U} {S, V} Note that the split Car e {V, U} is essentially the same as the split Car e {S}, the only difference being that the decision has been “reversed”. It is therefore not a distinct split, and we do not consider such splits. Next we evaluate the three categorical splits as follows: (a) For the split Car e {S}, DL = {x1, x3, x5}, and DR = {x2, x4, x6}. For DL, this yields PL = 2 and PH = 1 , and for DR, PL = 0 and PH = 1. The weighted entropy 3

3

of the split is then H(DL, DR)

= = =

3

3 H(DL) + H(DR) 6 6 3 1 1 2 2 3 — log + log — (0) 2 2 6 3 3 3 3 6 0.459

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

19.4 Exercises

1311

This yields an information gain of 0.918 — 0.459 = 0.459. (b) For Car e {V}, we get the same information gain as for Age ≤ 35, i.e., 0.106 (c) For Car e {U}, the gain is the same as for Age ≤ 22.5, i.e., 0.248.

Among all the possible split points for both Age and Car, the one with the highest information gain is Car e {S}, which is chosen as the best split decision at the root of the decision tree, as shown in Figure 19.1. We therefore make this split and recursively call the decision tree algorithm on each new subset DL = {x1, x3, x5} and DR = {x2, x4, x6}.

Notice that for DR all points are already labeled as high risk (H). Since the partition is already pure, we make it a leaf node, labeled as H. On the other hand, DL is not completely pure, so we consider partitioning it further. Since all points in DL have Car e {S}, we cannot use Car to further distinguish the points. Further, for Age, Age ≤ 22.5 is the only possible split to consider. Note that the entropy of DL is given as 2 2 1 1 log2 + log 2 = 0.918 3 3 3 3 For Age ≤ 22.5, DLL = {x1, x3}, whereas DLR = {x5}. For DLL, we get PL = 1 and PH = 0, and for DLR, we get PL = 0 and PH = 1. The weighted entropy is then H(D L) = —

H(DLL = {1, 3}, DLR = {5}) =

2 1 2 1 H(DLL) + H(DLR) = — (0) — (0) = 0 3 3 3 3

Thus the information gain is 0.918 — 0 = 0.918. In this example, this is the only possible split decision. After DL is split, we obtain the two new leaves DLL, which is labeled as low-risk (L), and DLR, which is labeled as high-risk (H). The full decision tree is shown in Figure 19.1.

One of the advantages of decision trees, is that each path from the root to a leaf can be written as a rule. For our example tree above, we obtain the following three rules 1) R1: if Car e {S} and Age ≤ 22.5, then Risk = H 2) R2: if Car e {S} and Age > 22.5, then Risk = L 3) R3: if Car e/ {S}, then Risk = H This is one of the strengths of decision trees, namely the ability to aid understanding of the model via simple rules presented to the user.

Once a decision tree model has been built, it can be used to classify new points.

For example, for the test point Age = 27, and Car = Vintage, we can classify the point by applying the set of decisions starting at the root. First we check whether Car e {S}. Since this test will be false, we go to the right branch, and since it is a leaf, we predict the class to be H.

Q3. What is the maximum and minimum value of the CART measure [Eq. (19.7)] and under what conditions?

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Decision Tree Classifier

132

D

Figure 19.1. Induced Decision Tree

— The minimum value is obtained when the class distribution on both sides of the split — Therefore the minimum value is zero.

1 1

k

Q4. Given the dataset in Table 19.2. Answer the following questions: (a) Show which decision will be chosen at the root of the decision tree using information gain [Eq. (19.5)], Gini index [Eq. (19.6)], and CART [Eq. (19.7)] measures. Show all split points for all attributes.

Answer: For the information gain we use log10 as opposed to log2 since it does not qualitatively change the results.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

19.4 Exercises

133 Table 19.2. Data for Q4

Instance

a1

a2

a3

Class

1 2 3 4 5 6 7 8 9

T T T F F F F T F

T T F F T T F F T

5.0 7.0 8.0 3.0 7.0 4.0 5.0 6.0 1.0

Y Y N Y N N N Y N

The entropy for the whole dataset is given as 4 4 5 5 H(D) = — log10 — log10 = 0.2983 9 9 9 9 The Gini index for the whole dataset is: !

42 52 G(D) = 1 — = 0.4938 + 92 92 Consider the split for attribute a1, which has only one possible split, namely a1 e T. The split entropy is given as: H(DL , D R ) =

4

1 1 3 3 5 1 1 4 4 — log 10 4 — log 10 4 + — log 10 5 — log 10 5 4 5 9 4 9 5

= 0.2293

Thus the gain is 0.2983 — 0.2293 = 0.0690.

The gini for the split is ! !

4 12 32 5 12 42 G(DL, DR) = 1— 2 — 2 + 1 — 2 — 2 = 0.3444 9 9 4 4 5 5 The CART measure for the split is given as:

4 1 1 — + — 4 5 4 5

CART(D , D ) = 2 4 5 · · · L R 9 9

3

= 0.5432

Likewise for attribute a2, we have only one split, and the split entropy is: H(DL , D R ) =

5

2 2 3 3 4 1 1 1 1 — log 10 5 — log 10 5 + — log 10 2 — log 10 2 5 2 9 5 9 2

with gain 0.2983 — 0.2962 = 0.0021.

The gini for the split is G(DL, DR) =

4 9

12

12

1— 2 — 2 2 2

! +

5 9

The CART measure for the split is given as: CART(D , D ) = 2 4 5 · · · L R 9 9

1 2

—

22

32

= 0.2962

!

1— 2 — 2 5 5

= 0.4889

3 2 1 + — 5 2 5

= 0.0988

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Decision Tree Classifier

134

For attribute a3 there are several numeric split points, namely a3 < 3.0, a3 < 4.0, a3 < 5.0, a3 < 6.0, a3 < 7.0, and a3 < 8.0. The split entropy for each of these cases is as follows: (a) For a3 < 3.0 we have H(D , D ) = 0 + 8 — 1 log 1 —1 log 1 = 0.2676 L R 10 10 2 2 2 9 2 The gain is 0.2983 — 0.2676 = 0.0307.

The gini for the split is 1 2 12 8 G(D ,RD ) = 1 — — L 9 2 2

! = 0.4444

The CART measure for the split is given as: CART(D , D ) = 2 1 8 L R · · · 9 9

1—

1 2

+

0—

1

= 0.1975

2

(b) For a3 < 4.0, we have H(DL , D R) =

7 3 3 4 2 1 1 4 —2 · log 10 + — log 10 7 — log 10 7 2 9 2 7 9 7

The gain is 0.2983 — 0.2976 = 0.0007.

The gini for the split is G(DL, DR) =

2 9

12

12

!

1— 2 — 2 2 2

7

32

42

= 0.2976

!

1— 2 — 2 7 7

= 0.4921

4 3 1 — + — 2 7 2 7

= 0.0494

+

9

The CART measure for the split is given as: CART(D , D ) = 2 2 7 L R · · · 9 9

1

(c) For a3 < 5.0 we have H(DL , D R) =

1 2 2 2 1 1 1 1 — log 10 — log 10 — 2 · log 10 3 3 3 + 3 2 3 3 2

The gain is 0.2983 — 0.2928 = 0.0055.

The gini for the split is G(DL, DR) =

1 3

12

22

1— 2 — 2 3 3

!

2

12

12

= 0.2928

!

1— 2 — 2 2 2

= 0.4815

1 1 2 — + — 3 2 3 2

= 0.1481

+

3

The CART measure for the split is given as: CART(D , D ) = 2 1 2 L R · · · 3 3

1

(d) For a3 < 6.0 we have H(DL , D R) =

2 3 3 4 1 5 2 1 — log 10 — log 10 — 2 · log 10 5 5 5 + 9 9 5 2 2

= 0.2962

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

19.4 Exercises

135

The gain is 0.2983 — 0.2962 = 0.0021.

The gini for the split is 22

5

32

4

5 4 9 9

2 3

1 2

10

1 2

1 2

1 3

12

12

1 2

3 5

1 3

2 3

12

22

2 3

1 2

2 5

1 3

10

10

2 3

The gain is 0.2983 — 0.2928 = 0.0055.

The gini for the split is 12

2

12

1

2 1 3 3

8 9

1 2

1 3

10

1 2

1 2

1 2

10

1 2

The gain is 0.2983 — 0.2676 = 0.0307.

The gini for the split is 8

1 8 9 9

12

12

1 2

1 2

So the best split for all three measures is a1 e {T}. It has the highest gain (0.069), the lowest Gini value (0.3444), and the highest CART measure (0.5432). (b) What happens to the purity if we use Instance as another attribute? Do you think this attribute should be used for a decision in the tree?

Answer: It is not advisable to use the instance id as a split value. The reason is that in general the id is just some unique identifier for each instance and each row will thus have a different value. Further, there is no reason to assume that the value follows any numeric scale. So what might happen is that we might end

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Decision Tree Classifier

136

up with relatively pure splits, at least on one side, if we only allow binary splits, and a completely pure split if we allow multi-way splits. However such a split is of no “predictive” value.

Q5. Consider Table 19.3. Let us make a nonlinear split instead of an axis parallel split, given as follows: AB — B2 ≤ 0. Compute the information gain of this split based on entropy (use log2, i.e., log to the base 2).

Table 19.3. Data for Q5

x1 x2 x3 x4 x5 x6 x7 x8

A

B

Class

3.5 2 9.1 2 1.5 7 2.1 8

4 4 4.5 6 7 6.5 2.5 4

H H L H H H L L

H(D) = —(3/8 log2 3/8 + 5/8 log2 5/8) = 0.954

The split entropy is therefore:

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

CHAP T ER 20

Linear Discriminant Analysis

20.4 EXERCISES Q1. Consider the data shown in Table 20.1. Answer the following questions: (a) Compute µ+1 and µ—1, and B, the between-class scatter matrix.

2.25 2.85

2.85 3.61

Table 20.1. Dataset for Q1

i

xi

yi

x1 x2 x3 x4

(4,2.9) (3.5,4) (2.5,1) (2,2.1)

1 1 —1 —1

(b) Compute S+1 and S—1, and S, the within-class scatter matrix.

(—0.25, 0.55). Therefore, the scatter matrix for +1 is given as 1

0.0625 0.1375

0.1375 0.3025

0.25 1

1

0.125 0.275

0.275

0.55 1.21

137

Linear Discriminant Analysis

138

(c) Find the best direction w that discriminates between the classes. Use the fact that a b d b 1 1 . a c d

0.26 1.22

S—1 =

1

we get w =

2.875 1.319

1—

1.22 0.0147 —0.55

0.55 0.26

1.22 0.55

1.5 1.9

1

0.55 0.26

2.875 1.319

0.909 0.417

(d) Having found the direction w, find the point on w that best separates the two classes.

offset along w as x 1 w = 4.845. Likewise, we have x 2w = 4.849, x 3w = 2.689, and 4

Q2. Given the labeled points (from two classes) shown in Figure 20.1, and given that the inverse of the within-class scatter matrix is 0.056 —0.029

—0.029 0.052

Find the best linear discriminant line w, and sketch it.

µ1 — µ2 = (—3, 1) This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

20.4 Exercises

139

9 8 7 6 5 4 3 2 1 1

2

3

4

5

6

7

8

9

Figure 20.1. Dataset for Q2.

w

0.056 0.029

0.029

·

3 1

0.197 0.139

Q3. Maximize the objective in Eq. (20.7) by explicitly considering the constraint wTw = 1, that is, by using a Lagrange multiplier for that constraint.

w

w Sw

— α(w w — 1)

2Bw(wTSw) — 2Sw(wTBw)

— 2αw = 0

Bw(wTSw) — Sw(wTBw)

wTBw(wTSw) — wTSw(wTBw)

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Linear Discriminant Analysis

140

Q4. Prove the equality in Eq. (20.19). That is, show that N1 =

Σ

T T c1 KK i i —n m 1 1m 1 = (K ) (I n1—

xi eD1

1 1n ×n ) (Kc1 )T n1 1 1

Answer: — n1 m1m 1

KK

1

K K

··· ··· ··· ···

—

··· ···

. n1 .,

··· ··· 1

1 1

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Support Vector Machines

CHAP T ER 21

21.7 EXERCISES Q1. Consider the dataset in Figure 21.1, which has points from two classes c1 (triangles) and c2 (circles). Answer the questions below. (a) Find the equations for the two hyperplanes h1 and h2.

x —0

x —0

5 3

1

2

1

1

2

8

9

2

9 8

7 6 5 4 3 2 1 3

4

5

6

7

Figure 21.1. Dataset for Q1.

141

Support Vector Machines

142

(b) Show all the support vectors for h1 and h2.

Answer: The support vectors for h1 are (2, 6), (3, 4) and (6, 2).

The support vectors for h2 are (3, 4) and (7, 6). (c) Which of the two hyperplanes is better at separating the two classes based on the margin computation?

1

1

2

1 1

2

. Using

w 6+4—12 22 12+2—12

2 5 2 5

2 5 7 7

we get the distance: d(7, 6) =

14

(d) Find the equation of the best separating hyperplane for this dataset, and show the corresponding support vectors. You can do this witout having to solve the Lagrangian by considering the convex hull of each class and the possible hyperplanes at the boundary of the two classes.

10

h

9 8 7

X2

6 5 4 3 2 1 0 0

1

2

3

4

5 X1

6

7

8

9

10

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

21.7 Exercises

143 Table 21 1. Dataset for Q2

i

xi1

xi2

yi

αi

x1 x2 x3 x4 x5 x6 x7 x8 x9 x10

4 4 1 2.5 4.9 1.9 3.5 0.5 2 4.5

2.9 4 2.5 1 4.5 1.9 4 1.5 2.1 2.5

1 1 —1 —1 1 —1 1 —1 —1 1

0.414 0 0 0.018 0 0 0.018 0 0.414 0

4x1 — x2 — 8. The margin is 14 3.395, which is more than the axis-parallel hyperplane mid-way between (3, 4) and (6, 4), which has a margin of 3.

Q2. Given the 10 points in Table 21.1, along with their classes and their Lagranian multipliers (αi), answer the following questions: (a) What is the equation of the SVM hyperplane h(x)?

w

= (0.818 + .018,.3312 + .054)

b = 1 — (.836 · 4 + .385 · 2.9) = 1 — (3.344 + 1.1165) = 1 — 4.4605 = —3.46

(b) What is the distance of x6 from the hyperplane? Is it within the margin of the classifier?

Answer: The distance of x6 from the hyperplane is given as

(1.9, 1.9) · (.836,.385) — 3.46 = 1.5884 + 0.7315 — 3.46 = —1.14

(c) Classify the point z = (3, 3)T using h(x) from above.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Support Vector Machines

144

Answer: We have h(z) = (3, 3) · (.836,.385)T — 3.46 = 2.508 + 1.155 — 3.46 = 0.203 The class is therefore +1.

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

CHAP T ER 22

Classification Assessment

22.5 EXERCISES Q1. True or False: (a) A classification model must have 100% accuracy (overall) on the training dataset.

Answer: False (b) A classification model must have 100% coverage (overall) on the training dataset.

Answer: True Q2. Given the training database in Table 22.1a and the testing data in Table 22.1b, answer the following questions: (a) Build the complete decision tree using binary splits and Gini index as the evaluation measure (see Chapter 19).

Answer: The Gini for whole DB is Gini(D) = 1 — ((4/8)2 + (4/8)2) = 1 — 0.5 = 1

1

2

1

2

1 1 3 4

1 3 3 3

3 3 1 0

3 1 1 1

0.5 0.375 0.5 0.49

0.5 0.375 0.5 0

0.5 0.375 0.5 0.439

0 0.125 0 0.061

1 2 2

0 2 4

3 2 2

4 2 0

0 0.5 0.44

0.49 0.5 0

0.439 0.5 0.33

0.061 0 0.17

4

0

0

4

0

0

0

0.5

145

Classification Assessment

146

(b) Compute the accuracy of the classifier on the test data. Also show the per class accuracy and coverage.

and divide them by the True class and the Predicted class, respv.

Table 22.1. Data for Q2

X

Y

Z

Class

15 20 25 30 35 25 15 20

1 3 2 4 2 4 2 3

A B A A B A B B

1 2 1 1 2 1 2 2

X

Y

Z

Class

10 20 30 40 15

2 1 3 2 1

A B A B B

2 1 2 2 1

(b) Testing

(a) Training

Q3. Show that for binary classification the majority voting for the combined classifier decision in boosting can be expressed as !

K

M (x) = sign K

Σ

αt Mt (x)

t =1

be positive, and the class will be predicted as +1.

t

t

t

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

22.5 Exercises

147

h3

h2

h1

9 8 7 6 5

4 3 2 1 1

2

3

4

5

6

7

8

9

Figure 22.1. For Q4.

Table 22.2. Critical values for t-test

dof tα/2

1

2

3

4

5

6

12.7065

4.3026

3.1824

2.7764

2.5706

2.4469

Q4. Consider the 2-dimensional dataset shown in Figure 22.1, with the labeled points belonging to two classes: c1 (triangles) and c2 (circles). Assume that the six hyperplanes were learned from different bootstrap samples. Find the error rate for each of the six hyperplanes on the entire dataset. Then, compute the 95% confidence interval for the expected error rate, using the t -distribution critical values for different degrees of freedom (dof) given in Table 22.2.

— 0.074/2.236) = 0.1154 ± 0.0847 = (0.0307, 0.2001) This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.

Classification Assessment

148

Q5. Consider the probabilities P(+1|xi) for the positive class obtained for some classifier, and given the true class labels yi x10 x1 x2 x3 x4 x5 x6 x7 x8 x9 +1 —1 +1 +1 —1 +1 —1 +1 —1 —1 yi P(+1|xi ) 0.53 0.86 0.25 0.95 0.87 0.86 0.76 0.94 0.44 0.86 Plot the ROC curve for this classifier.

0.94

threshold FP TP

1 0 0

0.95 0 1

0.87

0.86

0.86

0.86

0.94 0 2

0.87 1 2

0.86 3 3

0.2

0.4

0.6

0.76

0.76 4 3

0.53 4 4

0.53

0.44 5 4

0.44

0.25

0.25 5 5

1

0.8

TPR

0.6

0.4

0.2

0 0

0.8

1

FPR

This book has been published by Cambridge University Press. No unauthorized distribution shall be allowed.