Discovering Knowledge in Data An Introduction to Data Mining, Second Edition [2nd Ed] Daniel T. Laro

SOLUTION MANUAL

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Solutions to Chapter 1 AN INTRODUCTION TO DATA MINING Prepared by James Cunningham, Graduate Assistant 1. Refer to the Bank of America example early in the chapter. Which data mining task or tasks are implied in identifying “the type of marketing approach for a particular customer, based on customer’s individual profile”? Which tasks are not explicitly relevant?

Relevant tasks include the following:  Description  Classification  Clustering  Associating Non-relevant tasks are:  Estimation  Prediction

2. For each of the following, identify the relevant data mining task(s): a. The Boston Celtics would like to approximate how many points their next opponent will score against them.

Estimation: estimating the number of points (numeric target).

b. A military intelligence officer is interested in learning about the respective proportions of Sunnis and Shias in a particular strategic region.

Description: exploratory data analysis finds similarities and differences between the Sunni and Shias proportions.

c. A NORAD defense computer must decide immediately whether a blip on the radar is a flock of geese or an incoming nuclear missile.

Classification: a trained model detects incoming missiles assigns the blip on the radar screen (unclassified record) as being either a “missile” or “not missile” (categorical target); Estimation: an estimated numeric value may indicate the blip as an incoming missile.

1 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

d. A political strategist is seeking the best groups to canvass for donations in a county.

Description: relevant patterns describe the characteristics of one or more groups are located in the county; Clustering: examine the profile of each homogeneous group derived from a particular county’s population; Association: discover interesting rules pertaining to a large proportion of the population. e. A Homeland Security official would like to determine whether a certain sequence of financial and residence moves implies a tendency to terrorist acts.

Description: the sequences of financial and residential moves (patterns) may suggest a tendency (explanation) for terrorist activities; Classification: build a model to classify behavior as “suspicious”; Estimation: the model generates a numeric score indicating a propensity for committing terrorist acts. f. A Wall Street analyst has been asked to find out the expected change in stock price for a set of companies with similar price/earnings ratios.

Estimation: the expected change in stock price (numeric target) using the price/earnings ratio for a similar set of companies (predictors); Prediction: applied when results expected to predict future price.

3. For each of the following meetings, explain which phase in the CRISP-DM process is represented: a. Managers want to know by next week whether deployment will take place.

Therefore, analysts meet to discuss how useful and accurate their model is.

The Evaluation Phase determines whether the data mining model achieves the objectives established in the first phase. b. The data mining project manager meets with the data warehousing manager to discuss how the data will be collected.

Although the data warehouse is identified as a resource during the Business Understanding Phase, the actual data collection takes place during the Data Understanding Phase.

2 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

c. The data mining consultant meets with the Vice President for Marketing, who says that he would like to move forward with customer relationship management.

The primary objectives of the business are stated as part of the Business Understanding Phase. d. The data mining project manager meets with the production line supervisor, to discuss implementation of changes and improvements.

The requirements of a data mining technique used during the Modeling Phase may cause the process to loop back to the Data Preparation Phase, with the goal of improving data quality. The Evaluation Phase determines whether specific improvements or process changes are required to ensure that all important aspects of the business are accounted for. e. The analysts meet to discuss whether the neural network or decision tree models should be applied.

During the Modeling Phase one or more modeling techniques are chosen.

4. Discuss the need for human direction of data mining. Describe the possible consequences of relying on completely automatic data analysis tools.

The case studies emphasize the need for human involvement during every phase of the data mining process. For example, data mining initiatives using legacy database systems should not underestimate the time or importance required from domain experts to interpret the data. Taking shortcuts during this initial phase leads to potentially costly, inaccurate results in subsequent phases.

5. CRISP-DM is not the only standard process for data mining. Research an alternative methodology. (Hint: SEMMA, from the SAS Institute.) Discuss the similarities and differences with CRISP-DM. SEMMA is an acronym representing the core data mining processes: sample, explore, modify, model, and assess. As compared to CRISP-DM, SEMMA places emphasis on the model development process of data mining and therefore does not contain a Business Understanding Phase or a Deployment Phase; however, it does describe the importance of having clear business objectives and using quality data sources for modeling.

3 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Both processes are iterative and may loop back to other process steps as new information is learned or data mining requirements change. Also, both methods emphasize the use of an adaptive process. The following table shows how the phases of the two processes correspond to one another:

SEMMA N/A Sample Explore Modify Model Assess N/A

CRISP-DM Business Understanding Phase N/A1 Data Understanding Phase Data Preparation Phase Modeling Phase Evaluation Phase Deployment Phase

Table 5.1. SEMMA vs CRISP-DM

1

Although Sample does not correspond to a specific CRISP-DM phase, it often occurs during the Data Understanding, Data Preparation, and Modeling Phases.

4 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Solutions to Chapter 2 DATA PREPROCESSING Prepared by James Cunningham, Graduate Assistant 1. Describe the possible negative effects of proceeding directly to mine data that has not been preprocessed.

Neglecting to preprocess the data adequately before data modeling begins will likely produce data models that are unreliable and whose results should be considered dubious as best.

Performing data cleaning and data transformation during the data preparation phase is absolutely necessary for successful data mining efforts.

For example, suppose you are analyzing a data set that includes a person’s Age and Date_of_Birth attributes, and you want to calculate the average Age. Now, if 5% of the records contain a value of 0 for Age, the mean value would be very misleading and inaccurate. One solution to this problem would be to derive Age for the zero-based records based on information contained in the Date_of_Birth variable. Now, the mean value for Age is more representative of those persons in the data set.

2. Refer to the income attribute of the five customers in Table 2.1, before preprocessing. a. Find the mean income before preprocessing.

The mean value for Income before preprocessing is 38,999.80 and is derived by the possible inclusion of Income values -40,000 (erroneous) and 100,000 (possible outlier). b. What does this number actually mean?

In this case the mean value has little meaning because we are combining real data values with erroneous values. c. Now, calculate the mean income for the three values left after preprocessing.

Does this value have a meaning?

However, the mean value for Income produced by values 75,000, 50,000, and 10,000 (9,999 rounded to nearest 5,000) is 45,000. The latter value is certainly more representative of the true mean for Income, now that the records containing questionable values have been excluded.

1 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 3. Explain why zip codes should be considered text variables rather than numeric.

Zip codes should be considered text variables because they cannot be quantified on any numeric scale. Even their order has no numerical significance.

4. What is an outlier? Why do we need to treat outliers carefully?

Consider a set of numerical observations and the center of this observation set. An outlier is an observation that lies much farther away from the center than the majority of the other observations in the set.

We must treat outliers carefully because they can cause us to misrepresent the true center of an observation set incorrectly if they lie significantly farther away from the other observations in the set.

5. Explain why a birthdate variable would be preferred to an age variable in a database.

A birthdate variable is preferable to an age variable in a database because (1) one can always derive age from birthdate by taking the difference from the current date, and (2) age is relative to the current date only and would need to be updated continuously over time in order to remain accurate.

6. True or false: All things being equal, more information is almost always better.

The answer is true. In general, more information is almost always better. The more information we have to work with, the more insight into the underlying relationships of a particular domain of discourse we can glean from it.

7. Explain why it is not recommended, as a strategy for dealing with missing data, to simply omit the records or fields with missing values from the analysis.

It is not recommended to omit records or fields from an analysis simply because they have missing values. The rationale for this recommendation is that omitting these fields and records may cause us to lose valuable insight into the underlying relationships that we may have gleaned from the partial information that we do have.

2 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 8. Which of the four methods for handling missing data would tend to lead to an underestimate of the spread (e.g., standard deviation) of the variable? What are some benefits to this method?

Replacing a missing value by the attribute value’s mean artificially reduces the measure of spread for that particular attribute. Although the mean value is not necessarily a typical value, for some data sets this form of substitution may work well. Specifically, the effectiveness of this technique depends on the size of the variation of the underlying population. In other words, the technique works well for populations having small variations, and works less effectively for populations having larger variations.

Several benefits to leveraging this method include (1) ease of implementation (i.e. only one value to impute), (2) preservation of the standard error (i.e. no additional residual error is introduced).

9. What are some of the benefits and drawbacks for the method for handling missing data that chooses values at random from the variable distribution?

By using the data values randomly generated from the variable distribution, the measures of center and spread are most likely to remain similar to the original; however, there is a chance that the resulting records may not make intuitive sense.

10. Of the four methods for handling missing data, which method is preferred?

Having the analyst choose a constant to replace missing values based on specific domain knowledge is overall, probably the most conservative choice. If missing values are replaced with a flag such as “missing” or “unknown”, in many situations those records would ultimately be excluded from the modeling process; that is, all remaining valid, potentially important, values contained in those records would not be included in the data model.

3 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 11. Make up a classification scheme which is inherently flawed, and would lead to misclassification, as we find in Table 2.2. For example, classes of items bought in a grocery store.

Breakfast Cold Cereals Sugar Smacks Cheerios Hot Cereals Cream of Wheat

Count 72 1 2 28 3

Table 11.1. Flawed classification scheme

Using the table above, the “Breakfast” categorical attribute contains 5 apparent classes.

However, upon further inspection the classes are discovered to be inconsistent. For example, both “Sugar Smacks” and “Cheerios” are cold cereals, and “Cream of Wheat” is a hot cereal. Below, the cereals are now classified according to one of two classes, “Cold Cereals” or “Hot Cereals.”

Breakfast Cold Cereals Hot Cereals

Count 75 31

Table 11.2. Valid classification scheme

12. Make up a data set, consisting of the heights and weights of six children, in which one of the children is an outlier with respect to one of the variables, but not the other. Then alter this data set so that the child is an outlier with respect to both variables.

In the table below, Child #1 is an outlier with respect to Weight only. All children in the table are close in Height differing at most by 9 inches. However, all children except for Child # 1 are close in Weight differing at most by 7 pounds. Child #1 is an outlier as the Weight differs by 18 pounds from the second-heaviest child (Child #6), making this righttailed difference in Weight greater than the entire Weight range for the other five children.

Child 1 2 3 4 5 6

Height (in) 49 50 52 55 57 58

Weight (lbs) 100 75 77 79 80 82

Table 12.1. Heights & Weights of Children – Weight-only outlier

4 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. In the table below, Child #1 is an outlier with respect to both Height and Weight. All children except for Child #1 in the table are close in Height differing at most by 8 inches and are close in Weight differing at most by 7 pounds. Child #1 is an outlier for both Height and Weight as the Height differs by 14 inches from the second-shortest child (Child#2) (which is greater than the entire Height range of the other five children), and the Weight differs by 18 pounds from the second-heaviest child (Child #6) (which is greater than the entire Weight range of the other five children).

Child 1 2 3 4 5 6

Height (in) 36 50 52 55 57 58

Weight (lbs) 100 75 77 79 80 82

Table 12.2. Heights & Weights of Children – Height and Weight outlier

10

7

Use the following stock price data (in dollars) for Exercises 13–18 20 12 75 15 9 18 4 12 8

14

Table A. Stock prices

13. Calculate the mean, median, and mode stock price.

The mean is calculated as the sum of the data points divided by the number of points as follows: Mean Stock Price = (10+7+20+12+75+15+9+18+4+12+8+14) / 12 = 204 / 12 = $17.

The median is calculated by placing the prices in order and (a) selecting the middle value if the number of points is odd, or (b) taking the average of the two middle values if the number of points is even. Since we have twelve points, median is calculated as follows: Median Stock Price = mean of center values {4,7,8,9,10,12,12,14,15,18,20,75} = 24/2 = $12.

The mode is calculated as the value that occurs the most often in the set and is calculated as follows: Mode Stock Price = highest frequency of {4,7,8,9,10,12,12,14,15,18,20,75} = $12.

5 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

14. Compute the standard deviation of the stock price. Interpret what this number means.

The standard deviation represents the expected distance of a point chosen at random from a data set to the center of that set and is calculated by taking the square root of the variance.

The variance is the average of the sum of squared distances of each point from the data-set mean. Given that the mean is $17 (see Exercise #13) for this set, the variance for the set of stock prices is calculated as follows: Stock Price Variance (Var) = (4-17)2+(7-17)2+(8-17)2+(9-17)2+(10-17)2+(12-17)2+(12-17)2+(14-17)2+(15-17)2+(18-17)2+(20-17)2+(75-17)2 =

(-13)2 + (-10)2 + (-9)2 + (-8)2 + (-7)2 + (-5)2 + (-5)2 + (-3)2 + (-2)2 + (1)2 + (3)2 + (58)2 = 169 + 100 + 81 + 64 + 49 + 25 + 25 + 9 + 4 + 1 + 9 + 3364 = 3900 / 12 = 325 $2.

Taking the square root of the Variance, the Standard Deviation (SD) is calculated as follows: Stock Price Standard Deviation (SD) of Stock Price = √(325) = ±$18.03.

Since the mean is $17 and the standard deviation is plus/minus $18.03, the expected price of a stock drawn at random from the set of twelve stocks is expected to lie mathematically between ($17–$18.03) = -$1.03 (i.e. $0.01 since we assume that a stock price can never be less than one penny USD) and ($17+$18.03) = $35.03.

As we can see, each stock with the exception of the one priced at $75 is priced within this range.

15. Find the min-max normalized stock price for the stock worth $20.

Min-Max normalization scales an observation relative to the data-set’s range resulting in a value between 0 and 1 (this value has no units) and is formulated as follows: MinMaxXi = [Xi – Min(X)] / [Max(X) – Min(X)] Therefore, the min-max normalized stock price of $20 is calculated as follows: MinMax($20) = ($20 - $4) / ($75 - $4) = ($16) / ($71) = 0.2254.

6 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

16. Calculate the midrange stock price.

The midrange stock price is the central price for the entire price range and is formulated as follows: MidRangeX = [Max(X) + Min(X)] / 2 For the problem at hand we have as follows: MidRangeX = ($75 + $4) / 2 = ($79) / 2 = $39.5

17. Compute the Z-score standardized stock price for the stock worth $20.

Z-Score standardization scales an observation where the mean value is zero, the SD is 1 and most values lie between -4 and 4 (this value has no units) and is formulated as follows: Z-Score(X) = [Xi – Mean(X)] / |SD(X)| Given the mean of $17 (see Exercise #13) and |SD| of 18.03 (see Exercise #14), The Z-Score for the stock price of $20 is calculated as follows: Z-Score($20) = ($20 - $17) / $18.03 = ($3) / $18.03 = 0.1664.

Please note that this value makes sense as it is slightly greater than zero just as $20 is slightly greater than $18.03.

18. Find the decimal scaling stock price for the stock worth $20.

Decimal standardization scales an observation to a value between -1 and 1 (this value has no units) and is formulated as follows: Decimal(Xi) = Xi / 10d where d is the number of digits in the observation in the data set having the largest absolute value. Since the largest stock price is $75, d = 2 as there are two digits in this price. The decimal standardization is then calculated as follows: Decimal($75) = $75 / $102 = $75 / $100 = 0.75

7 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

19. Calculate the skewness of the stock price data.

Skewness is the lack of normalization of a Z-Score-standardized distribution and is measured using the following formula: Skewness = 3 [Mean(X) – Median(X)] / SD(X) Given the mean of $17 and median of $12 (see Exercise #13), and an SD of $18.03 (see Exercise #14), the skewness for the stock price distribution is calculated as follows: Skewness = 3 [$17 - $12] / $18.03 = 3[$5] / $18.03 = $15 / $18.03 = 0.8319.

We observe that this distribution is right-skewed since a right-skewed distribution has a mean that is greater than its median yielding a positive skewness value. In contrast, a left-skewed distribution will have a mean that is less than its median and thus a negative skewness value.

20. Explain why data analysts need to normalize their numeric variables.

Data analysts need to normalize their numeric variables as it places all variables on the same scale. Normalizing all variables to the same scale is critical when performing operations that are sensitive to data variation or spread so that variables having larger variations do not adversely overpower variables having smaller variations. Most (if not all) analytic operations involving linearization (e.g. Regression, PCA, MANOVA, etc.) are sensitive to data spread.

21. Describe three characteristics of the standard normal distribution.

The three main characteristics of the Standard Normal Distribution are as follows:  The mean is zero  The SD is 1  It is symmetric (equal and opposite in shape and size) about the mean and normal (the mean has the highest frequency, and frequency decreases symmetrically as distance from the mean increases).

22. If a distribution is symmetric, does it follow that it is normal? Give a counterexample.

If a distribution is symmetric, it is not guaranteed to be normal. In order for a distribution to be normal it has to have a single expected value (i.e. the value with the highest frequency).

8 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. A classic counterexample is the Uniform Distribution, which is symmetric about the center of its interval, yet since it all values on the interval occur with equal frequency, it has an infinite number of expected values making it non-normal.

23. What do we look for in a normal probability plot to indicate non-normality?

A normal probability plot is simply a plot of the quantiles of a given distribution to the quantiles of the Standard Normal Distribution. If the quantiles are approximately equal, then the plot will approximate a straight line indicating that the given distribution is normal.

In contrast, if the quantiles of the distribution are not equal to the Standard Normal Distribution, then the plot will not approximate a straight line indicating non-normality.

Use the stock price data for Exercises 24–26.

24. Do the following: a. Identify the outlier.

The outlier is the stock price of $75. The difference from the next-closest stock price ($20) is $55, which is nearly 3.5X larger than the entire range of the other eleven stocks (i.e. $16). b. Verify that this value is an outlier, using the Z-score method.

We can also verify that $75 is in fact an outlier using the Z-score method. The Zscore for this stock is calculated using our mean of $17 (see Exercise #13) and our SD of $18.03 (see Exercise #14) as follows: Z-Score($75) = ($75 - $17) / $18.03 = ($58) / $18.03 = 3.2169.

Since a Z-score that is less than -3 or greater than 3 is considered an outlier, we conclude that stock price $75 is an outlier as its Z-score is 3.2169 which is greater than 3. c. Verify that this value is an outlier, using the IQR method.

We can also verify that $75 is in fact an outlier using the Inter-Quartile Range or IQR method. The quartiles are determined by placing the stock prices in ascending order and dividing them onto four parts as follows: The ordered stock prices are: {4,7,8,9,10,12,12,14,15,18,20,75}, and since there are an even number of values, we partition as {4,7,8,9,10,12} and {12,14,15,18,20,75} The quartiles are then determined as follows: Q1 = {4,7,8,9,10,12} = $8 Q3 = {12,14,15,18,20,75}= $15

9 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

We then calculate IQR = Q3 – Q1 as follows: IQR = $15 – $8 = $7 If an observation is an outlier, then it will have a value that is less than Q1 - 1.5IQR or a value greater than Q3 + 1.5IQR. We then calculate the upper and lower boundary values for the stock price set as follows: LowerBound = Q1 - 1.5IQR = 8 – 1.5(7) = 8 – 10.5 = -$2.50 UpperBound = Q3 + 1.5IQR = 15 + 1.5(7) = 15 + 10.5 = $25.50 Since $75 is greater than $25.5, we conclude that $75 is an outlier.

25. Identify all possible stock prices that would be outliers, using: a. The Z-score method.

The ordered stock prices are: {4,7,8,9,10,12,12,14,15,18,20,75} where the mean is $17 lying between the $15 and $18 stock indicated in bold text, and the SD is $18.03. Working from the left, we have as follows: Z-Score($4) = ($4 - $17) / $18.03 = (-$13) / $18.03 = -0.7210.

Z-Score($7) = ($7 - $17) / $18.03 = (-$10) / $18.03 = -0.5546.

Z-Score($8) = ($8 - $17) / $18.03 = (-$9) / $18.03 = -0.4992.

Z-Score($9) = ($9 - $17) / $18.03 = (-$8) / $18.03 = -0.4437.

Z-Score($10) = ($10 - $17) / $18.03 = (-$7) / $18.03 = -0.3882.

Z-Score($12) = ($12 - $17) / $18.03 = (-$5) / $18.03 = -0.2773.

Z-Score($14) = ($14 - $17) / $18.03 = (-$3) / $18.03 = -0.1664.

Z-Score($15) = ($15 - $17) / $18.03 = (-$2) / $18.03 = -0.1109.

Z-Score($18) = ($18 - $17) / $18.03 = ($1) / $18.03 = 0.0555.

Z-Score($20) = ($20 - $17) / $18.03 = ($3) / $18.03 = 0.1664.

We already know that $75 is an outlier having a Z-score of 3.2169 (see Exercise #24).

However, no other outliers were identified using Z-score standardization.

10 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

b. The IQR method.

The ordered stock prices are: {4,7,8,9,10,12,12,14,15,18,20,75} where we have an IQR of $7, a Lower Bound of $2.5, and an Upper Bound of $25.50.

Therefore, stock prices $75 is once again the only outlier as it is greater than the upper bound of $25.50.

26. Investigate how the outlier affects the mean and median by doing the following: a. Find the mean score and the median score, with and without the outlier.

The mean for the entire set of stock prices is $17 (see Exercise #13),, and the mean without the $75 outlier is calculated as follows: MeanNo_Outlier = (10+7+20+12+15+9+18+4+12+8+14) / 11 = 129 / 11 = $11.73.

The median is calculated by placing the prices in order and (a) selecting the middle value if the number of points is odd, or (b) taking the average of the two middle values if the number of points is even. Since we have twelve points, median is calculated as follows: Median Stock Price = mean of center values {4,7,8,9,10,12,12,14,15,18,20,75} = 24/2 = $12.

MedianNo_Outlier = mean of center values {4,7,8,9,10,12,12,14,15,18,20} = $12.

b. State which measure, the mean or the median, the presence of the outlier affects more, and why.

It is obvious that the presence of the outlier affects the mean more than the median. It increases the mean by $5, and has no effect on the median.

For this particular data set, the outlier affects the mean more than the median because the mean determines the numerical center of the data set through interpolation and this data is right-skewed having a large right-tailed outlier. In contrast, the median determines the distributive center of the dataset through physical partitioning and the largest value of the lower half of the data is equal to the smallest value of the upper half of this data set.

11 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

27. What are the four common methods for binning numerical predictors? Which of these are preferred?

The four common methods for binning numerical predictors are as follows: 1. Equal width binning – this method divides into k categories of equal width chosen by the client or analyst.

2. Equal frequency binning - this method divides the numerical predictor into k categories, each having k/n records, where n is the total number of records.

3. Binning by clustering – this method uses a clustering algorithm, such as k-means clustering.

4. Binning based on predictive value - this method partitions the numerical predictor based on the effect each partition has on the value of the target variable.

The preferred methods are Binning by Clustering (method #3) and Binning based on predictive value (method #4). Both methods determine the partitions by the nature of the data and its underlying relationships.

In contrast, the Equal Width Binning (method #1) and Equal Frequency Binning (method #2) determine the partitions simply by their individual numeric values. In general, Equal Width Binning should not be used for anything more than rough exploration as the use of equal width bins is very susceptible to outliers. The method of Equal Frequency Binning is inherently flawed in that it can produce bins having overlapping values.

Use the following data set for Exercises 28–30: 1 1 1 3 3 7 28. Bin the data into three bins of equal width (width = 3).

Using equal-width binning with width=3, the bin boundaries are calculated as follows: Bin1: 0 <= X < 3 containing {1,1,1} Bin2: 3 <= X < 6 containing {3,3} Bin3: 6 <= X < 9 containing {7}

12 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 29. Bin the data into three bins of two records each.

Binning this set into three bins of two records each is an application of equal-frequency binning with k=3, and since n=6, the bin size is k/n => 3/6 = 2. The bins are as follows: {1,1}, {1,3}, {3,7}

30. Clarify why each of the binning solutions above are not optimal.

Although this toy data set is relatively small and the clarification may not be obvious, the equal-width binning from Exercise #28 is suboptimal since Bin3 contains the outlier value 7, giving the illusion that the discretized class Bin3 is just as close to Bin2 as Bin1 when in fact it is much farther away.

The equal-frequency binning from Exercise #29 is also suboptimal since the three bins contain overlapping values. For example, the value 1 lies in both the first and second bins, and the value 3 lies in both the second and third bins. Therefore, models constructed from this new set of discretized classes are bound to produce unpredictable results.

31. Explain why we might not want to remove a variable that had 90% or more missing values.

In general, analysts should not remove variables that have a large number of missing values.

The small number of values that are present may in fact be representative of the underlying population, and it would therefore be worthwhile to attempt to impute the missing values.

However, if the small of values are not representative of the underlying population, the fact that the variable has missing values may actually be able to be correlated to other variables and produce predictive power that would have been lost if this data were discarded.

32. Explain why we might not want to remove a variable just because it is highly correlated with another variable.

In general, analysts should not remove variables, even when they are highly correlated. An analyst may be tempted to remove one of a pair of highly correlated variables in order to avoid over-emphasizing a particular informational characteristic. However, although removing a highly correlated variable avoids potential “double-counting”, doing so may also cause a loss of valuable predictive relationships to other variables that the target variable is not highly correlated with.

As an alternative to removing highly correlated variables, it is recommended that the analyst employ Principle Component Analysis to translate the highly correlated variables into a set of uncorrelated principal components.

13 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. HANDS-ON ANALYSIS Use the churn data set on the book series website for the following exercises.

The Churn data set is located at the following location: http://www.dataminingconsultant.com/DKD2e.htm

33. Explore whether there are missing values for any of the variables.

An analysis using SPSS Modeler v16 identifies that there are no missing values for any variables and is depicted below:

Figure 33.1. Variable Audit

34. Compare the area code and state fields. Discuss any apparent abnormalities.

Although each record has a value for Area Code and for State, there are only three distinct values for Area Code in the entire data set (408, 415, and 510), and each of the three values for Area Code is associated to each of the values for State. This presents an abnormality as each of these three Area Codes are for the state of California, so each record that does not have the State code CA has an invalid combination of State and Area Code values. The complete distribution of State and Area Code mappings are depicted below.

14 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 34.1. State & Zip Code comparison

15 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 35. Use a graph to visually determine whether there are any outliers among the number of calls to customer service.

Using Modeler, we set the type the Customer Service Calls to Ordinal and produce a Distribution graph as depicted below in order to identify potential outliers. The center of the distribution appears to lie around 1 service call making potential outliers appear to be at 7, 8, and 9 service calls.

Figure 35.1. Distribution of Customer Service calls

16 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 36. Identify the range of customer service calls that should be considered outliers, using: a. The Z-score method, and We treat the Customer Service Calls as Continuous, which gives us a mean of 1.563, and an SD of 1.315 as depicted in the Data Audit below.

Figure 36.1. Customer Service Calls mean and SD.

The Z-scores for the number of service calls in descending order are as follows: Z-Score(9) = (9 – 1.563) / 1.315 = 7.437 / 1.315 = 5.656 => Outlier Z-Score(8) = (8 – 1.563) / 1.315 = 6.437 / 1.315 = 4.885 => Outlier Z-Score(7) = (7 – 1.563) / 1.315 = 5.437 / 1.315 = 4.134 => Outlier Z-Score(6) = (6 – 1.563) / 1.315 = 4.437 / 1.315 = 3.374 => Outlier Z-Score(5) = (5 – 1.563) / 1.315 = 3.437 / 1.315 = 2.614 => Not an Outlier Z-Score(4) = (4 – 1.563) / 1.315 = 2.437 / 1.315 = 1.853 => Not an Outlier Z-Score(3) = (3 – 1.563) / 1.315 = 1.437 / 1.315 = 1.093 => Not an Outlier Z-Score(2) = (2 – 1.563) / 1.315 = 0.685 / 1.315 = 0.521 => Not an Outlier Z-Score(1) = (1 – 1.563) / 1.315 = -0.563 / 1.315 = -0.428 => Not an Outlier Z-Score(0) = (0 – 1.563) / 1.315 = -1.563 / 1.315 = -1.189 => Not an Outlier b. The IQR method.

The value set is {0,1,2,3,4,5,6,7,8,9}. Since there is an even number of values, we partition between values 4 and 5 as {0,1,2,3,4} and {5,6,7,8,9}. This yields the four quartiles as follows: Q1 = {0,1,2,3,4} = 2 Q3 = {5,6,7,8,9} = 7 We then calculate IQR = Q3 – Q1 as follows: IQR = 7 – 2 = 5.

17 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Using the IQR, we calculate the upper and lower boundaries as follows: LowerBound = Q1 - 1.5IQR = 1 – 1.5(5) = 2 – 7.5 = -5.5 UpperBound = Q3 + 1.5IQR = 7 + 1.5(5) = 7 + 7.5 = 14.5 The IQR method does not identify any outliers as all values lie within the upper and lower quartile boundaries.

37. Transform the day minutes attribute using Z-score standardization.

We treat the Day Minutes as Continuous, which gives us a mean of 179.775, and an SD of 54.467 as depicted in the Summary Statistics below.

Figure 37.1. Day Minutes mean & SD

18 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. We then derive a new field called Z_DayMinutes specifying the Z-score calculation as depicted below.

Figure 37.2. Z_DayMinutes formulation

38. Work with skewness as follows: a. Calculate the skewness of day minutes.

Recall that skewness is formulated as follows: Skewness = 3 [Mean(X) – Median(X)] / SD(X) From Exercise #37 we have mean 179.775, median 179.400, and SD 54.467, and calculate the skewness of Day Minutes as follows: SkewnessDayMinutes = 3 [179.775 – 179.400] / 54.467 = [1.125] / 54.467 = 0.021.

b. Then calculate the skewness of the Z-score standardized day minutes.

Comment.

We calculate the summary statistics of the derived variable Z_DayMinutes as depicted below.

19 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 38.1. Z_DayMinutes mean & SD

We calculate the skewness of Z_DayMinutes as follows: SkewnessZ_DayMinutes = 3 [0.000 – (-0.007)] / 1.000 = [0.007] / 1.000 = 0.007. c. Based on the skewness value, would you consider day minutes to be skewed or nearly perfectly symmetric?

We conclude that the Day Minutes values are nearly perfectly symmetric based on the calculated skewness value.

20 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 39. Construct a normal probability plot of day minutes. Comment on the normality of the data.

The normal probability plot of day minutes is calculated by first sorting the data set in ascending order by DayMins and then plotting Z_DayMinutes by DayMins as depicted below. We observe that there is an almost perfect linear relationship between the Z-score quantiles and the raw data quantiles indicating that the DayMins values are almost perfectly normally distributed. Please note that this aligns well with the near-zero skewness observed in Exercise #38.

Figure 39.1. Normal Probability Plot of Day Minutes

21 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

40. Work with international minutes as follows: a. Construct a normal probability plot of international minutes.

We treat the International Minutes as Continuous and calculate the mean 10.237 and the SD 2.792 as depicted in the summary statistics below.

Figure 40.1. International Minutes mean & SD

22 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. We then derive a new variable named Z_InternationalMinutes specifying the Z-score calculation as depicted below.

Figure 40.2. Z_InternationalMinutes formulation

Finally, we construct a Normal Probability Plot by first sorting the data set by International Minutes values in ascending order and plotting Z_InternationalMinutes by IntlMins as depicted below.

23 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 40.3. Normal Probability Plot of International Minutes

b. What is stopping this variable from being normally distributed?

There are 18 records in the data set that have zero International Minutes, and this is preventing this variable from being normally distributed as depicted below.

Figure 40.4. Distribution of International Minutes

24 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. c. Construct a flag variable to deal with the situation in (b).

In order to perform analyses requiring normally distributed data, we can construct a flag variable that we can use to partition the variable into the subset that has international minutes and the subset that as no international minutes. The HasIntlMins flag is set to true when the number of minutes is non-zero and false otherwise as depicted below.

Figure 40.5. HasIntlMins forumation

25 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

d. Construct a normal probability plot of the derived variable nonzero international minutes. Comment on the normality.

We sort the partition where HasIntlMins is true by the number of International Minutes in ascending order, calculate the Z-scores for each and create the Normal Probability Plot as depicted below. We have also included a histogram of the Z-scored non-zero international minutes in order to observe the near-perfect normality.

Figure 40.6. Side-by-Side view of Z_IntlMins Normal Probability Plot and Distribution

41. Transform the night minutes attribute using Z-score standardization. Using a graph, describe the range of the standardized values.

We treat night minutes as Continuous and calculate the mean 200.872 and SD 50.574 as depicted in the summary statistics below.

Figure 41.1. Night Minutes mean & SD

We then derive a new variable named Z_NightMins specifying the Z-score calculation as depicted below.

26 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 41.2. Z_NightMins formulation

We examine the range of Z_NightMins both graphically using a histogram and quantitatively using summary statistics as depicted below. Observe in the histogram how the range of the data appears to expand into what is typically considered as the outlier space. We confirm this 27 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. quantitatively noting that the minimum value is less than -3 and the maximum value is greater than 3.

Figure 41.3. Range of Z_NightMins

28 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Solutions to Chapter 3 EXPLORATORY DATA ANALYSIS Prepared by James Cunningham, Graduate Assistant 1. Explain the difference between EDA and hypothesis testing, and why analysts may prefer EDA when doing data mining.

The primary difference between hypothesis testing and EDA is that hypothesis testing requires that the analyst have some a priori insight into or assumptions about the relationships between predictors and target whereas EDA requires no preconceived notions about the data.

An analyst would prefer to use EDA over hypothesis testing when (1) the data set has a relatively large number of predictors, and / or (2) the analyst is generally unfamiliar with the data. The purpose of EDA is to explore the data distributions and compare proportions of the various elements in order to uncover interesting relationships among predictors and between the predictors and the target.

2. Why do we need to perform exploratory data analysis? Why should not we simply proceed directly to the modeling phase and start applying our high powered data mining software?

One benefit of EDA is to uncover correlations between pairs of variables in the data set.

Without first identifying and removing correlated variables, data mining models may produce unreliable results.

In most cases, a raw data set will require a certain level of preparation. EDA is employed to ensure that (1) the desired modeling approach will be effective, and (2) the modeling phase will have all of the required predictors and target. The analyst employs EDA to identify such scenarios as (1) highly correlated predictors, (2) missing values, and (3) dimension-reduction opportunities.

3. Why do we use contingency tables, instead of just presenting the graphical results?

We use contingency tables so that we can compare proportion percentages with absolute precision. Graphical techniques provide us with only a general or abstract insight into the related proportions.

1 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 4. How can we find the marginal distribution of each variable in a contingency table?

The marginal distribution of a predictor variable is on the bottom of the contingency table. Each column accounts for the proportion of the data having a particular predictor value. The marginal distribution of the target variable is on the right side of the contingency table. Each row accounts for the proportion of the data having a particular target value.

5. What is the difference between taking row percentages and taking column percentages in a contingency table?

As mentioned in Exercise #4, taking row percentages gives the percentages of the possible values for the target variable. Taking column percentages gives the percentages of the possible values for the predictor variable being explored.

6. What is the graphical counterpart of a contingency table?

The graphical counterpart to a contingency table is a comparison bar chart. This chart renders the proportions of a particular predictor variable’s possible values with an overlay of the proportions of the target variable’s values.

7. Describe what it would mean for interaction to take place between two categorical variables, using an example.

An interaction between two categorical variables exists when there is a significant difference in the proportions of one of the variables that corresponds to a significant difference in proportions of the other variable.

As an example, recall from the text the interaction between Churn and International Plan customers. There is a significant proportion of customers having the International Plan that churn. We observe that 137/(137 + 186) = 42.4% of the International Plan holders churned, and only 346/(346 + 2664) = 11.5% of those without the International Plan.

8. What type of histogram is useful for examining the relationship between a numerical predictor and the target?

The preferred type of histogram to employ for examining the relationship between a numerical predictor and the target is an overlay histogram. In addition to illustrating the frequency distribution of the predictor’s possible values, this histogram also contains an overlay of the proportions of the target variable’s possible values. The overlay enables the analyst to determine whether or not there is an interaction by comparing the proportions directly.

2 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 9. Explain one benefit and one drawback of using a normalized histogram. Should we ever present a normalized histogram without showing its non-normalized counterpart?

Normalized histograms help provide contrast when determining the proportions, using a specific overlay, across an attribute’s categories. However, high variability displayed for some values may be somewhat misleading when small record counts exist for that portion of the distribution. For these reasons it is appropriate to properly annotate the use of normalized results and display non-normalized histogram as well.

10. Explain whether we should omit a predictor from the modeling stage if it does not show any relationship with the target variable in the EDA stage, and why.

In general, we should never omit a predictor from the modeling stage if it does not show any relationship with the target variable in the EDA stage. The reason for this is that although the predictor under examination may not show a direct interaction with the target, it may provide predictive power in collaboration with other predictors that are just not readily observable from a univariate perspective.

11. Describe how scatter plots can uncover patterns in two dimensions that would be invisible from one-dimensional EDA. Performing two-dimensional EDA helps uncover relationships between the variables. For example, one might observe that as the value of x increases, the value of y also increases.

In addition, particularly interesting subsets of the two-dimensional space from the scatter plot may be chosen and further analyzed.

12. Make up a fictional data set (attributes with no records is fine) with a pair of anomalous attributes. Describe how EDA would help to uncover the anomaly.

Suppose there are two attributes named Level and Major to describe students at a university.

Appropriate values for Level include “undergraduate” or “graduate”, where their Major may be “Math”, “English”, or “Business”. Furthermore, suppose an anomaly exists in the data set, since no graduate students appear to be studying Business at the university?

EDA would uncover this anomaly by examining the distribution for the Major attribute using a Level overlay. These results could be verified with a domain expert (graduate office) to confirm the possible anomaly.

3 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 13. Explain the objective and the method of binning based on predictive value.

The primary objective of binning based on predictive value is to explore interactions using discretized predictor values with the expectation of observing a stronger interaction with the target than with the original predictor values.

Binning is executed by inferring the boundary values to discretize the predictor values on that will produce the greatest difference in proportions to the target values. Since binning is more of an art than a science, it may require a bit of trial-end-error in order to maximize the difference in interaction with the target value proportions.

14. Why is binning based on predictive value considered to be somewhat of an art?

As mentioned in Exercise # 13, binning is somewhat of an art as it requires a bit of educated guessing and some trial-and-error in order to maximize the differences in target proportions across the discretized predictor values.

15. What step should precede the deriving of a new numerical variable representing the mean of two other numerical variables?

When deriving a new numerical variable representing the mean of two other variables it is necessary to standardize both original variables first. This places both variables on the same scale and prevents one variable from outweighing the other.

16. What does it mean to say that two variables are correlated?

Two variables are considered correlated when an incremental change in one variable is proportional to an incremental change in the second variable.

17. Describe the possible consequences of allowing correlated variables to remain in the model.

Allowing correlated variables into the data model will overemphasize one or more of the data components, and may produce unreliable results.

18. A common practice among some analysts when they encounter two correlated predictors is to omit one of them from the analysis. Is this practice recommended?

Unless two variables are perfectly correlated, it is actually a bad practice to omit a predictor from an analysis when it is correlated to some other predictor. The omission can remove valuable information and thus reduce predictive power.

4 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 19. Describe the strategy for handing correlated predictor variables at the EDA stage.

The preferred approach to handling correlated variables at the EDA stage is to (1) identify any variables that are perfectly correlated, and (2) identify any groups of variables that are less than perfectly correlated. For perfectly correlated predictor pairs, the analyst should omit one or the other during the modeling phase. For less than perfectly correlated variables, the analyst should employ a dimension-reduction technique such as Principal Components Analysis (PCA) to generate lower-dimension estimates of the correlated variables.

20. For each of the following descriptive methods, state whether it may be applied to categorical data, continuous numerical data, or both. a. Bar charts Categorical b. Histograms Both: numeric with categorical overlay. c. Summary statistics Both d. Cross-tabulations Categorical e. Correlation analysis Numeric f. Scatter plots Both: numeric with categorical overlay. g. Web graphs Categorical h. Binning Both

5 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. HANDS-ON ANALYSIS 21. Using the churn data set, develop EDA which shows that the remaining numeric variables in the data set (apart from those covered in the text above) indicate no obvious association with the target variable.                    

State: Categorical, for the 50 states and the District of Columbia Account Length: Integer-valued, how long account has been active Area code: Categorical Phone Number: Essentially a surrogate for customer ID International Plan: Dichotomous categorical, yes or no Voice Mail Plan: Dichotomous categorical, yes or no Number of Voice Mail Messages: Integer-valued Total Day Minutes: Continuous, minutes customer used service during the day Total Day Calls: Integer-valued Total Day Charge: Continuous, perhaps based on above two variables Total Eve Minutes: Continuous, minutes during the evening Total Eve Calls: Integer-valued Total Eve Charge: Continuous, perhaps based on above two variables Total Night Minutes: Continuous, minutes during the night Total Night Calls: Integer-valued Total Night Charge: Continuous, perhaps based on above two variables Total International Minutes: Continuous, minutes to make international calls Total International Calls: Integer-valued Total International Charge: Continuous, perhaps based on above two variables Number of Calls to Customer Service: Integer-valued

The Account Length is the outstanding numeric variable that has not been addressed in the text above. Using a normalized histogram on Account Length with an overlay of Churn, we observe that there is no obvious interaction as depicted in the figure below.

Figure 21.1 Histogram of Account Length

6 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. Use the Adult data set from the book series website for the following exercises. The target variable is income, and the goal is to classify income based on the other variables.

22. Which variables are categorical and which are continuous?

There are fifteen variables in the Adult data set. 

The continuous variables in the Adult data set are age, demogweight, educationnum, capital-gain, capital-loss, and hours-per-week are continuous variables.



The categorical variables in the Adult data set are workclass, education, maritalstatus, occupation, relationship, race, sex, native-country, and income.

The complete listing of variables is typed via the SPSS Modeler as depicted in the figure below.

Figure 22.1 Variables and their respective measurement types

7 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 23. Using software, construct a table of the first 10 records of the data set, in order to get a feel for the data.

Using SPSS Modeler, we construct a table of the first 10 records by leveraging the Preview function on the Select node as depicted in the figure below.

Figure 23.1 Use of the Select Node Preview feature

The table of the first ten records retrieved via the Preview is depicted in the figure below.

Figure 23.2 Preview results

24. Investigate whether we have any correlated variables.

8 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

We calculate the correlations between each pair of continuous variables using SPSS Modeler’s Statistics node as depicted in figure below.

Figure 24.1 Configuration of the Statistics node for Pearson Correlations

The calculated correlations with their respective discretized strengths are depicted in the figure below.

9 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 24.2 Pearson Correlation results

25. For each of the categorical variables, construct a bar chart of the variable, with an overlay of the target variable. Normalize if necessary.

10 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. As stated in Exercise #22, the categorical variables in the Adult data set not including the target income, are workclass, education, marital-status, occupation, relationship, race, sex, native-country.

The standard and normalized bar chart for each categorical variable is depicted side-by side in the figures below.

Workclass

Figure 25.1 Workclass Histogram and Normalized Histogram

Education

Figure 25.2 Education Histogram and Normalized Histogram

11 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. Marital-Status

Figure 25.3 Marital-Status Histogram and Normalized Histogram

Occupation

Figure 25.4 Occupation Histogram and Normalized Histogram

12 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. Relationship

Figure 25.5 Relationship Histogram and Normalized Histogram

Race

Figure 25.6 Race Histogram and Normalized Histogram

13 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. Sex

Figure 25.7 Sex Histogram and Normalized Histogram

Native-Country

Figure 25.8 Native-Country Histogram and Normalized Histogram

a. Discuss the relationship, if any, each of these variables has with the target variables.

The relationship of each categorical variable with the target variable income is discussed below as follows: 

Workclass – This variable has its mode on private employees, but appears to be divided into two sub-groups with in its relation to the target. The first sub-group is comprised of unspecified work-class, federal government employees, and local government employees with the largest interaction on federal government employees.

14

Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. The second sub-group is comprised of private employees, the incorporated selfemployed, the non-incorporated self-employed, and state-government employees with the largest interaction on the incorporated self-employed. 

Education – This variable has its mode on high-school graduates but appears to be divided into three sub-groups in relation to the target. The first sub-group consists of the various high-school drop-out levels and seems to be uniformly distributed with respect to interaction with the target. The second sub-group consists of the degree holders and high-school graduates having the strongest interaction with doctorate holders. The third sub-group consists of preschool, professional school, and some college having the strongest interaction with professional school adults.



Marital-Status – This variable has its mode on Married-civ-spouse having a tie for the strongest interactions between Married-civ-spouse and Married-AF-spouse.



Occupation – This variable has its mode on Prof-specialty workers and appears to be divided into three sub-groups in relation to the target. The first sub-group consists of armed forces, unspecified occupations, and unskilled laborers. The second sub-group consists of semi-skilled and skilled laborers including craft-repair, farming-fishing, and transport-moving, the third sub-group consists of professionals including Execmanagerial, Prof-specialty, Protective-serv, Sales, and Tech-support having the strongest interaction on Exec-managerial.



Relationship – This variable has its mode on Husband and a tie for strongest interactions with the target on Husband and Wife.



Race – This variable has its mode on White adults and a tie for strongest interactions with the target on Asian-Pac-Islander and White adults.



Sex – This variable has its mode on Male adults, which is also the strongest interaction with the target.



Native Country – This variable is difficult to make inferences upon based on visual inspection alone. It appears to have its mode on adults from Trinidad&Tobago, but it is nearly impossible to glean any interaction patterns from the normalized bar chart alone. b. Which variables would you expect to make a significant appearance in any data mining classification model we work with?

Of the categorical variables, the ones that we would expect to make a significant appearance in any data mining classification model would be (1) Workclass, (2) Education, (3) Marital-status, (4) Occupation, (5) Relationship, (6) Race, and (7) Sex. Each one of these variables has one or more values with significant interactions with the target variable.

15 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 26. For each pair of categorical variables, construct a cross-tabulation. Discuss your salient results.

Figure 26.1 Workclass Cross-Tabulation

Figure 26.2 Education Cross-Tabulation

Figure 26.3 Marital-Status Cross-Tabulation

16 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 26.4 Occupation Cross-Tabulation

Figure 26.5 Relationship Cross-Tabulation

Figure 26.6 Race Cross-Tabulation

17 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 26.7 Sex Cross-Tabulation

Figure 26.8 Native-Country Cross-Tabulation

18 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 27. (If your software supports this.) Construct a web graph of the categorical variables.

Fine tune the graph so that interesting results emerge. Discuss your findings.

We use SPSS Modeler to construct a Directed Web Graph specifying the eight predictors as the from-fields and the target income as the to-field. Filtering on a minimum proportion of 60% of the target gives the results as depicted in the figure below.

Figure 27.1 Directed Web Graph at 60% of Target

Referring to the figure above, we observe that the values Marital-status = Married-civspouse, Relationship = Husband, and Sex = Male are excellent predictors of income >50K. The values Native-Country = United States and Race = White are good predictors of income >50K. In contrast, a value of Workclass = Private is a fair indicator of income <=50K.

19 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 28. Report on whether anomalous fields exist in this data set, based on your EDA, which fields these are, and what we should do about it.

The EDA of the Adult data set has not identified any anomalous fields. For example, none of the fields appear to have target values that are uniformly / randomly distributed. However, if there were anomalous fields, we would not include them in an analysis unless they have been cleared by a domain expert.

29. Report the mean, median, minimum, maximum, and standard deviation for each of the numerical variables.

We use SPSS Modeler to calculate the summary statistics for each of the numerical variables using the Statistics node as depicted in the figure below.

Figure 29.1 Summary Statistics configuration

20 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. The resulting summary statistics are depicted in the output as depicted in the figure below.

Figure 29.2 Summary Statistics for Numerical Variables

21 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 30. Construct a histogram of each numerical variable, with an overlay of the target variable income. Normalize if necessary.

The numerical variables in the Adult data set are Age, Demogweight, Education-num, Capital-gain, Capital-loss, and Hours-per-week. The Histograms for the numerical variables are depicted in the figures below.

Figure 30.1 Age Histogram and Normalized Histogram

Figure 30.2 Demogweight Histogram and Normalized Histogram

22 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 30.3 Education-num Histogram and Normalized Histogram

Figure 30.4 Capital-gain Histogram and Normalized Histogram

23 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 30.5 Capital-loss Histogram and Normalized Histogram

Figure 30.6 Hours-per-week Histogram and Normalized Histogram

a. Discuss the relationship, if any, each of these variables has with the target variables. 

Age - Adults having an income over 50K appear to be normalized around the age of 50.



Demogweight – This variable appears to have income values that are randomly / uniformly distributed. There is no obvious interaction with the target.



Education-num – This variable is directly proportional with the target Income. In other words, the proportion of adults having an income over 50K increases as the education-num value increases.

24

Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 

Capital-gain – This variable appears to indicate that nearly all adults that have a Capital Gain have an income greater than 50K.



Capital-loss – This variable appears indicate that the proportion of adults with an income greater than 50K have a capital loss that is approximately normally distributed around $3000.00.



Hours-per-week – This variable appears to indicate that the proportion of adults with an income greater than 50K have hours worked per week that are approximately normally distributed around 50 hours per week.

b. Which variables would you expect to make a significant appearance in any data mining classification model we work with?

The variables that are expected to make a significant appearance in any data mining classification model constructed would be (1) Age, (2) Education-num, and (3) Hours-per-week.

31. For each pair of numerical variables, construct a scatter plot of the variables.

Discuss your salient results.

Since there are six numerical variables, there are thirty pairings and thus 30 pairwise scatterplots. The scatterplots are depicted in the panels below.

25 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 31.1 Scatterplots for Age by the other Numerical Variables

26 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 31.2 Scatterplots for Demogweight by the other Numerical Variables

27 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 31.3 Scatterplots for Education-num by the other Numerical Variables

Figure 31.4 Scatterplots for Capital-gain by the other Numerical Variables

28 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 31.5 Scatterplots for Capital-loss by the other Numerical Variables

32. Based on your EDA so far, identify interesting sub-groups of records within the data set that would be worth further investigation.

Based on the EDA of the numerical variables, several interesting sub-groups of records that would be worth further investigations are as follows: 

Age v. Hours-per-week: This relationship appears to identify a high concentration of adults with an income greater than 50K between ages 35 and 65 that work between 25 and 60 hours per week.



Education-num v. Capital-loss: This relationship appears to identify a high concentration of adults with an income greater than 50K that have an educationnum between 9 and 16 that have a capital-loss between $1500.00 and $3000.00.



Capital-Gain v. Hours-per-week: This relationship appears to identify a high concentration of adults with an income greater than 50K with a capital gain between 0$ and 30K that work between 30 and 60 hours per week.



Capital-loss v. Hours-per-week: This relationship appears to identify a high concentration of adults with an income greater than 50K with a capital loss between $1500.00 and 3000.00 that work between 30 and 55 hours per week.

29 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 33. Apply binning to one of the numerical variables. Do it in such a way as to maximize the effect of the classes thus created (following the suggestions in the text). Now do it in such a way as to minimize the effect of the classes, so that the difference between the classes is diminished. Comment.

Binning by predictive value is performed to demonstrate how binning can be leveraged to elucidate the interaction between a predictor and its target. The chart on the left side of the figure below depicts a histogram of Capital-loss with an Income overlay showing a partitioning into three intervals as follows:   

Small Losses are less than or equal to $1480 Medium Losses range greater than $1480 and less than or equal to $3050 Large Losses are greater than $3050

The chart on the right of the figure below depicts a distribution of a variable called LossCategories that was derived from capital-loss using the three intervals as categorization conditions. The result elucidates the interaction between capital-loss and income quite well.

Figure 33.1 Binning Capital-Loss by Predictive Value

To demonstrate that binning by predictive value is an art that requires decisiveness, the chart on the left of the figure below depicts a histogram of the capital-loss with income 30 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. overlay where the boundaries are placed by dividing the range of losses $4356.00 into three even intervals as follows:   

Small Losses are less than or equal to $1452 Medium Losses range greater than $1452 and less than or equal to $2904 Large Losses are greater than $2904

The chart on the right of the figure below depicts a distribution of a variable called LossCategories2 that was derived from capital-loss using the three intervals as categorization conditions. The result is a loss in overall discriminatory power as the Low and High classes now have equal proportions. This is actually the equivalent of equalwidth binning using three bins.

Figure 33.2 Suboptimal Binning of Capital-loss by Predictive Value

31 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 34. Refer to the previous exercise. Apply the other two binning methods (equal width, and equal number of records) to this same variable. Compare the results and discuss the differences. Which method do you prefer?

The histogram on the top left of the figure below depicts the binning of capital-loss by number of bins. The binning algorithm takes the range of 4356 and divides it by the number of bins specified to determine the intervals.

The histogram on the top right of the figure below depicts the binning of capital-loss by the specified interval width. Since the width of 1452 specified is exactly one-third of the range, the resulting intervals are identical to the ones determined above.

The result is two identical histograms produced using two different methods. However, both of these methods are unsupervised approaches and therefore completely inferior to the supervised approach of binning by predictive value. The supervised binning approach maximizes the discriminatory power of the resulting bins.

Figure 34.1 Binning of Capital-loss by Bin Count and Bin Width

32 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 35. Summarize your salient EDA findings from the above exercises, just as if you were writing a report.

The categorical variables in the Adult data set not including the target Income, are Workclass, Education, Marital-status, Occupation, Relationship, Race, Sex, and Nativecountry. The numerical variables are Age, Demogweight, Education-num, Capital-gain, Capital-loss, and Hours-per-week. 

Workclass – The self-employed-inc adults are approximately 2X more likely than the other classes to have incomes greater than 50K. Federal-gov adults are approximately 2X more likely than the other classes (with the exception of selfemployed-inc) to have incomes greater than 50K. Local-gov, Self-emp-not-inc, and state-gov adults are about equally likely to have incomes greater than 50K.



Education – Adults with doctorates and prof-school degrees are the most likely to have incomes greater than 50K (adults with a Master’s degree are close behind). In fact, they are approximately 2X more likely than adults with a Bachelor’s degree, 4X more likely than adults with an Associate’s degree or Some-college, and nearly 10X more likely than adults who never graduated High School.



Marital-status – Married-AF-Spouse and Married-civ-spouse classes are about equally likely to have incomes greater than 50K and are approximately 4X more likely to than the other classes.



Occupation – Exec-managerial and Prof-specialty adults are about equally likely to have incomes greater than 50K. They are approximately 1.67X more likely than protective-serv, Sales, and Tech-support adults, about 2X more likely than Craftrepair and Transport-moving adults, and are at least 4X more likely than all of the remaining classes including the unspecified ‘?’, Admin-clerical, Armed-forces, Farming-fishing, Handlers-cleaners, Machine-op-inspect, Other-service, and Privhouse-serv.



Relationship - Husbands and Wives are both equally likely to have incomes greater than 50K, and both of these classes are approximately 4X to 5X more likely than any of the other classes.

33

Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 

Race – Asian-Pac-Islander and White adults are equally likely to have incomes greater than 50K. They are approximately 2X more likely than the other classes.



Sex – Male adults are almost 3X more likely than Female adults to have incomes greater than 50K



Native-country – Adults that are natives of France, Italy, Japan, and Taiwan are about equally likely to have incomes over 50K. These natives are about 1.1X to 1.4X more likely than natives of Canada, England, Germany, Greece, Hong Kong, India, Iran, and Yugoslavia. They are also approximately 2X more likely than the unspecified ‘?’, Cuba, Ireland, Philippines, Poland, South, Thailand, and the United States. They are at least 3X more likely than the remaining classes.



Age - The mean is 38.6 and the Median is 37, so the data appears to be normalized.

The histogram appears to identify a left skewness. The proportion of adults having an income over 50K appear to be normalized around the age of 50. Age and Hours-perweek co-identify a high concentration of adults with an income greater than 50K between ages 35 and 65 that work between 25 and 60 hours per week.



Demogweight – The mean is 189741.8 and the Median is 178353, and this data appears to be normalized around the mean with slight left skewness. However, the proportion of adults having an income over 50K appears to be uniformly distributed across much of the range showing no material pattern. There also does not appear to be any real discriminatory from a bi-variate perspective.



Education-num – The mean is 10.08 and the Median is 10, so the data appears to be normalized with slight left skewness. The proportion of adults having an income over 50K are directly proportional with education-num increasing as the education-num value increases. Education-num and Capital Loss co-identify a high concentration of adults with an income greater than 50K for an education-num between 9 and 16 that have a capital-loss between $1500.00 and $3000.00.



Capital-gain – The Mean is 1088.58 and the Median is 0, and the data does not appear normalized at all. This variable appears to indicate that nearly all adults that have a Capital Gain have an income greater than 50K so there is no obvious 34

Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. discriminatory power here. However, Capital-gain and Hours-per-week appear to coidentify a high concentration of adults with an income greater than 50K with a capital gain between 0$ and 30K that work between 30 and 60 hours per week.



Capital-loss – The mean is 86.5 and the Median is 0, and the data does not appear to be normalized at all. However, the proportion of adults with an income greater than 50K have a capital loss does appear to be approximately normally distributed around $3000.00 with right skewness. Per the binning by predictive value exercise, it does appear that adults with the Medium-range losses are about 2X more likely than those with Low-range losses to have incomes greater than 50K, and are nearly 3X more likely than those with High-range losses to have incomes greater than 50K. Finally, Capital-loss and Hours-per-week appear to co-identify a high concentration of adults with an income greater than 50K with a capital loss between $1500.00 and 3000.00 that work between 30 and 55 hours per week.



Hours-per-week – The mean is 40.4 and the Median is 40, and the data appears to be normalized around the mean. Also, the proportion of adults having an income greater than 50K is also somewhat normalized around 50 hours per week with left skewness.

Finally, Age and Hours-per-week, Capital-gain and Hours-per-week, and Capital-loss and Hour-per-week all co-identify interesting subsets of records having incomes greater than 50K.

35 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Solutions to Chapter 4 UNIVARIATE STATISTICAL ANALYSIS Prepared by James Cunningham, Graduate Assistant 1. Explain what is meant by statistical inference. Give an example of statistical inference from everyday life, say, a political poll.

Statistical inference is a generalization (i.e. an estimation or prediction) that is made on an entire population based on the nature of the data that can be collected on that population. As an example, consider a political leader such as a Governor who wants to determine how his/her constituents feel about his/her performance. Although it may be most accurate to survey each and every citizen to determine the public opinion, it is most often impractical and cost-prohibitive to do so. Therefore, a political analyst would poll a representative sample of the constituency and use statistical inference in order to estimate the public opinion of the entire constituency.

2. What is the difference between a population and a sample?

A population is the entire set of items or individuals both observed and unobserved that pertain to a particular analysis. Examples of populations would be (1) all fans of a particular sports team, (2) all Salmon in a particular fish hatchery, and (3) all Spanish-speaking Americans.

A sample is a subset of the population. Ideally, this sample is representative of the underlying population. Therefore, it is important to obtain the sample as randomly as possible in order to maximize the potential for the sample to be a representative one.

3. Describe the difference between a parameter and a statistic.

A parameter is a characteristic of a population. Since it is not feasible to determine the exact values of these characteristics in most situations, they are often represented by Greek letters.

In contrast, a statistic is a characteristic of a sample. These values are calculated from the sample data collected and used as estimates of the parameters of the underlying population.

4. When should statistical inference not be applied?

Statistical inference should not be applied if the sample is not representative of its underlying population. Estimation and prediction rely completely on the statistics calculated from the sample, and the statistics calculated will only be acceptable estimates of population parameters only when the sample is representative of its population.

1 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 5. What is the difference between point estimation and confidence interval estimation?

In point estimation, the population parameter is estimated by a single known statistical value.

In general, the point estimate is expected to entail a certain degree of error (i.e. distance from the point estimate to the population parameter). However, since the population parameter is not known definitively, the size of the error is also unknown. In summation, point estimation makes no statement about the degree of confidence that the analyst can have regarding the accuracy of a particular point estimate.

In contrast, confidence interval estimation produces an interval of values consisting of a point estimate that is bounded by a particular level of confidence. This level of confidence specifies a probability that the interval contains the population parameter.

6. Discuss the relationship between the width of a confidence interval and the confidence level associated with it.

Smaller margin of errors (narrower interval estimates) indicate greater precision in the interval estimate. The actual width of the interval estimate is directly affected by the sample variability, sample size, and confidence level. The confidence level tells us the measure of confidence we have in our estimate. For example, with 95% confidence, 95 times out of 100, the point estimate for the sample will be contained in the confidence interval. Because we cannot influence the sample variance and are usually hesitant to lower our confidence level, increasing the sample will yield a more precise confidence interval estimate.

7. Discuss the relationship between the sample size and the width of a confidence interval. Which is better, a wide interval or a tight interval? Why? Where obtaining a larger sample is feasible, increasing the sample size results in higher precision for the confidence interval estimate. This produces a narrower interval, which enables the analyst to more accurately estimate the true population parameter.

8. Explain what we mean by sampling error.

Sampling error is the absolute value of the difference between the point estimate of a statistic and its corresponding target parameter. As an example, the sampling error for the mean is formulated as follows: |𝒙 − 𝝁| Since this is a distance measure, it is always expressed as a positive value.

2 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

9. What is the meaning of the term margin of error?

The margin of error refers to the degree of precision for a point estimate. In other words, it is the absolute value of the maximum distance that a point estimate can lie away from the actual population parameter. Therefore, smaller margins of error imply greater precision.

In general, we calculate a confidence interval estimate in addition to the point estimate. The confidence interval estimate is the interval of numbers surrounding the point estimate to a desired confidence level. For a univariate point estimate, the size of its confidence interval is typically equal to twice its margin of error and is typically conceptualized as follows: Confidence Interval = Point Estimate ± Margin of Error As an example, Student’s t-distribution governs the margin of error (and therefore the confidence interval) for a sample drawn randomly from a normally distributed population such that the margin of error is formulated as follows: 𝐬 𝐭 𝛂/𝟐 ( ) 𝐧 Where α is the desired significance level (i.e. 1 – (confidence level/100)), tα/2 is the critical tscore at the desired significance level, s is the sample standard deviation, and n is the sample size.

10. What are the two ways to reduce margin of error, and what is the recommended way?

There are two ways to reduce the margin of error. The first way is simply to reduce the desired confidence level. Mathematically, doing so decreases the critical value on the target probability distribution (e.g. tα/2 on Student’s t) thus reducing the margin of error. While this may seem like an acceptable approach, it is actually damaging to the analysis (and therefore should be avoided) since we would be reducing the margin of error by relaxing the precision of the estimate.

If we want to reduce the margin of error while maintaining the desired level of confidence, we should increase the size of the sample. Doing so will reduce the standard error and thus the overall margin of error without having to relax the desired level of precision.

11. A political poll has a margin of error of 3%. How do we interpret this number?

The margin of error is typically expressed in the same units as the point estimate. However, in the case of political polling, the point estimate is expressed as a proportion (e.g. percentage) of the constituency polled. As an example, consider a favorability poll where some random sample of a public official’s constituency is asked to identify whether or not 3 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. the official’s performance is favorable. Let’s say that the point estimate is found to be 55% favorable, meaning that 55% of the constituency polled find the official’s performance to be favorable. Now if the margin of error (at the desired confidence level) is calculated to be 3%, we would interpret this as “for the desired confidence level, on average, the percentage of a random sampling of the constituency finding the official’s performance to be favorable would lie somewhere between 52% (i.e. 55% - 3%) and 58% (i.e. 55% + 3%)”. Please note that this assumes that all polls conducted employ a similar sampling method.

12. What is hypothesis testing?

Hypothesis testing is an operation that evaluates claims about the value of a particular population parameter using statistics calculated from sample data and an understanding of the parameter’s probability distribution. A hypothesis test is comprised of two competing statements. 

The null hypothesis H0 claims a particular truth about a particular population parameter.



The alternative hypothesis Ha claims that the null hypothesis is either entirely false or at least false under certain conditions (i.e. ‘not entirely true’).

Hypothesis testing is often used to make inferences about one or more underlying population means µ or variances Σ (as in the Analysis of Variance (ANOVA) procedure).

13. Describe the two ways a correct conclusion can be made, and the two ways an incorrect conclusion can be made.

There are four possible outcomes that a hypothesis test can take. The first two are correct conclusions, and the second two are incorrect conclusions defined as follows: 

True Positive: the test rejects the null hypothesis H0 based on the sample, and the null hypothesis would be rejected if the entire underlying population were leveraged (i.e. Ha is the reality).



True Negative: the test fails to reject the null hypothesis H0 based on the sample, and the null hypothesis would fail to reject if the entire underlying population were leveraged (i.e. H0 is the reality).



False Positive: the test rejects the null hypothesis H0 based on the sample, but and the null hypothesis would actually fail to reject if the entire underlying population were leveraged (i.e. H0 is actually the reality). This is also referred to as a Type I Error.

4 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 

False Negative: the test fails to reject the null hypothesis H0 based on the sample, but the null hypothesis would actually be rejected if the entire underlying population were leveraged (i.e. Ha is actually the reality). This is also referred to as a Type II Error.

14. Explain clearly why a small p-value leads to rejection of the null hypothesis.

In order to explain why a small p-value leads to rejection of the null hypothesis, we first define what the p-value is. A p-value or probability-value is the probability of observing a sample statistic that is at least as extreme as the point estimate of that sample at a desired significance level.

Therefore, in order for a null hypothesis to hold true (i.e. fail to reject) at a desired significance level, the p-value (i.e. the probability of observing a sample statistic at least as extreme as the point estimate being assessed) would have to be greater than the desired significance level. In other words, when the p-value is greater than or equal to the target significance level, the null hypothesis holds true because observing statistics that are more extreme than the point estimate is just not probable beyond that significance level.

Alternatively stated, we would reject a null hypothesis at a desired significance level when the p-value (i.e. the probability of observing a sample statistic at least as extreme as the point estimate being assessed) is less than the desired significance level. In other words, when the p-value is less than the target significance level, the null hypothesis no longer holds true at that significance level because observing test statistics that are at least as extreme as the point estimate are actually probable beyond that significance level.

15. Explain why it may not always be desirable to draw a black-and-white, up-or-down conclusion in a hypothesis test. What can we do instead?

The dichotomous nature of the hypothesis test is such that it identifies whether a null hypothesis is accepted or rejected without providing any insight into the level of statistical evidence supporting the outcome. In certain domains, especially those where multiple tests are conducted on different random draws of a data set, we may actually need to assess the evidence supporting a test outcome, so it may not be in our best interest simply to choose a significance level and mechanically accept or reject a null hypothesis.

When the evidence supporting a hypothesis test outcome needs to be taken into consideration, the analyst can leverage additional insights provided by the p-value from each test in order to draw conclusions about the validity of the results as well as the efficacy of the testing protocols employed.

5 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

16. How can we use a confidence interval to conduct hypothesis tests?

When the probability distribution and basic assumptions are the same, a confidence interval may be considered complimentary to a two-tailed hypothesis test. As an example, we can state specifically that a (1-α)*100% confidence interval for a population mean µ is equivalent to a two-tailed hypothesis test for µ at a significance level of α.

Therefore, if a hypothesized population mean µ0 lies outside the desired confidence interval (i.e. (1 -α)*100%), then the corresponding two-tailed hypothesis test at the significance level α will reject the null hypothesis H0 for that hypothesized population mean.

17. The duration of customer service calls to an insurance company is normally distributed, with mean 20 minutes, and standard deviation 5 minutes. For the following sample sizes, construct a 95% confidence interval for the population mean duration of customer service calls.

The confidence interval is equal to the sample mean plus / minus the margin of error, and on the t-distribution, it is formulated as follows:

𝐱 ± 𝐭 𝛂/𝟐 (

𝐬 𝐧

)

Since our confidence level is 95%, our significance level α is calculated as follows: α = (1-95%/100%) = 0.05 and our Cumulative Probability is calculated as follows: Cumulative probability = 1 – (0.05/2) = 1 – 0.025 = 0.975. a. n = 25 The Critical Value tα/2 on the t-distribution for our Cumulative Probability of 0.975 using 24 (i.e. n-1) degrees of freedom is 2.064, and our margin of error for this sample is calculated as follows: 5 𝑀𝑎𝑟𝑔𝑖𝑛 𝑜𝑓 𝐸𝑟𝑟𝑜𝑟 = 2.064 ∗ = 𝟐. 𝟎𝟔𝟒 25 Therefore, the confidence interval is (20) ± 2.064 = (17.936, 22.064) minutes.

6 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. b. n = 100 The Critical Value tα/2 on the t-distribution for our Cumulative Probability of 0.975 using 99 (i.e. n-1) degrees of freedom is 1.984, and our margin of error for this sample is calculated as follows: 5 𝑀𝑎𝑟𝑔𝑖𝑛 𝑜𝑓 𝐸𝑟𝑟𝑜𝑟 = 1.984 ∗ = (1.984) ∗ (0.5) = 𝟎. 𝟐 100 Therefore, the confidence interval is (20) ± 0.992 = (19.008, 20.992) minutes. c. n = 400 The Critical Value tα/2 on the t-distribution for our Cumulative Probability of 0.975 using 399 (i.e. n-1) degrees of freedom is 1.966, and our margin of error for this sample is calculated as follows: 5 𝑀𝑎𝑟𝑔𝑖𝑛 𝑜𝑓 𝐸𝑟𝑟𝑜𝑟 = 1.966 ∗ = (1.966) ∗ (0.25) = 𝟎. 𝟒 𝟏𝟓 400 Therefore, the confidence interval is (20) ± 0.4915 = (19.5085, 20.4915) minutes.

18. For each of the confidence intervals in the previous exercise, calculate and interpret the margin of error.

The margin of error using the t-distribution is formulated as follows:

Therefore, given a confidence level of 95%, a sample mean of 20 minutes, and a sample standard deviation of 5 minutes, the margins of error for each sample size are calculated as follows: 

For n = 25, the 𝑀𝑎𝑟𝑔𝑖𝑛 𝑜𝑓 𝐸𝑟𝑟𝑜𝑟 ∗ 𝟐 𝟎𝟔𝟒, which means that we expect that the true population mean will be located no more and no less than 2.064 minutes from the sample mean of 20 minutes.



For n = 100, the 𝑀𝑎𝑟𝑔𝑖𝑛 𝑜𝑓 𝐸𝑟𝑟𝑜𝑟 ∗ ∗ 𝟎 𝟐, which means that we expect that the true population mean will be located no more and no less than 0.992 minutes from the sample mean of 20 minutes.

7 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 

For n = 400, the 𝑀𝑎𝑟𝑔𝑖𝑛 𝑜𝑓 𝐸𝑟𝑟𝑜𝑟 ∗ ∗ 𝟎 𝟒 𝟏𝟓, which means that we expect that the true population mean will be located no more and no less than 0.4915 minutes from the sample mean of 20 minutes.

19. Refer to the previous exercise. Describe the relationship between margin of error and sample size.

In the previous exercise, we observe that for sample sizes n = 25, n = 100, and n = 400, the margins of error are 2.064, 0.992, and 0.4915 respectively. Therefore, the margin of error and the sample size are inversely proportional. In other words, the margin of error decreases as the sample size increases.

20. Of 1000 customers who received promotional materials for a marketing campaign, 100 responded to the promotion. For the following confidence levels, construct a confidence interval for the population proportion who would respond to the promotion.

In order to construct the confidence interval for the proportion of the customer population π who would respond to the promotion, we begin by calculating the estimated population proportion p as follows:

𝑝=

𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑒𝑛𝑡𝑠 100 = = 𝟎. 𝟏𝟎 (𝑠𝑎𝑚𝑝𝑙𝑒 𝑠𝑖𝑧𝑒) 1000

The Confidence Interval for the population proportion π is equal to the sample proportion p plus / minus the margin of error, and on the Standard Normal distribution, it is formulated as follows:

𝑝 ± 𝑍∝/2

𝑝∗ 1−𝑝 𝑛

a. 90% Since our confidence level is 90%, our significance level α is calculated as follows: α = (1-90%/100%) = 0.10, and our Cumulative Probability is calculated as follows: Cumulative probability = 1 – (0.10/2) = 1 – 0.05 = 0.95.

8 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. The Critical Value Zα/2 on the Standard Normal distribution for our Cumulative Probability of 0.95 is 1.645, and our margin of error for the proportion is calculated as follows:

𝑀𝑎𝑟𝑔𝑖𝑛 𝑜𝑓 𝐸𝑟𝑟𝑜𝑟 = 1.645 ∗

0.10 ∗ 0.90 = 1.645 ∗ 0.03 = 𝟎. 𝟎𝟒 𝟒 100

Therefore, the confidence interval for the proportion of customers who would respond is (0.10) ± 0.0494 = (0.0506, 0.1494).

b. 95% Since our confidence level is 95%, our significance level α is calculated as follows: α = (1-95%/100%) = 0.05, and our Cumulative Probability is calculated as follows: Cumulative probability = 1 – (0.05/2) = 1 – 0.025 = 0.975.

The Critical Value Zα/2 on the Standard Normal distribution for our Cumulative Probability of 0.975 is 1.960, and our margin of error for the proportion is calculated as follows:

𝑀𝑎𝑟𝑔𝑖𝑛 𝑜𝑓 𝐸𝑟𝑟𝑜𝑟 = 1.960 ∗

0.10 ∗ 0.90 = 1.960 ∗ 0.03 = 𝟎. 𝟎𝟓𝟖𝟖 100

Therefore, the confidence interval for the proportion of customers who would respond is (0.10) ± 0.0588 = (0.0412, 0.1588).

9 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. c. 99% Since our confidence level is 99%, our significance level α is calculated as follows: α = (1-99%/100%) = 0.01, and our Cumulative Probability is calculated as follows: Cumulative probability = 1 – (0.01/2) = 1 – 0.005 = 0.995.

The Critical Value Zα/2 on the Standard Normal distribution for our Cumulative Probability of 0.995 is 2.576, and our margin of error for the proportion is calculated as follows:

𝑀𝑎𝑟𝑔𝑖𝑛 𝑜𝑓 𝐸𝑟𝑟𝑜𝑟 = 2.576 ∗

0.10 ∗ 0.90 = 2.576 ∗ 0.03 = 𝟎. 𝟎𝟕𝟕𝟑 100

Therefore, the confidence interval for the proportion of customers who would respond is (0.10) ± 0.0773 = (0.0227, 0.1773).

21. For each of the confidence intervals in the previous exercise, calculate and interpret the margin of error.

Using the Standard Normal distribution, the margin of error for the proportion is formulated as follows:

𝑀𝑎𝑟𝑔𝑖𝑛 𝑜𝑓 𝐸𝑟𝑟𝑜𝑟 = 𝑍∝/2

𝑝∗ 1−𝑝 𝑛

Therefore, given a proportion estimate of 0.10, the margins of error for each confidence level are calculated as follows: 

For a confidence level of 90%

𝑀𝑎𝑟𝑔𝑖𝑛 𝑜𝑓 𝐸𝑟𝑟𝑜𝑟 = 1.645 ∗

0.10 ∗ 0.90 = 1.645 ∗ 0.03 = 𝟎. 𝟎𝟒 𝟒 100

, which means that we expect that the true population proportion of respondents will be located no more and no less than 0.0494 from the sample proportion of 0.10.

10 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 

For a confidence level of 95%

𝑀𝑎𝑟𝑔𝑖𝑛 𝑜𝑓 𝐸𝑟𝑟𝑜𝑟 = 1.960 ∗

0.10 ∗ 0.90 = 1.960 ∗ 0.03 = 𝟎. 𝟎𝟓𝟖𝟖 100

, which means that we expect that the true population proportion of respondents will be located no more and no less than 0.0588 from the sample proportion of 0.10. 

For a confidence level of 99%

𝑀𝑎𝑟𝑔𝑖𝑛 𝑜𝑓 𝐸𝑟𝑟𝑜𝑟 = 2.576 ∗

0.10 ∗ 0.90 = 2.576 ∗ 0.03 = 𝟎. 𝟎𝟕𝟕𝟑 100

which means that we expect that the true population proportion of respondents will be located no more and no less than 0.0773 from the sample proportion of 0.10.

22. Refer to the previous exercise. Describe the relationship between margin of error and confidence level.

In the previous exercise, we observe that for confidence levels 90%, 95%, and 99%, the margins of error are 0.0494, 0.0588, and 0.0773 respectively. Therefore, the margin of error and the confidence level are directly proportional. In other words, the margin of error increases as the confidence level increases.

23. A sample of 100 donors to a charity has a mean donation amount of $55 with a sample standard deviation of $25. Test using 𝛼 = 0.05 whether the population mean donation amount exceeds $50. a. Provide the hypotheses. State the meaning of 𝜇.

This is a right-tailed hypothesis test with hypotheses formed as follows: 

Null hypothesis H0: The population mean µ is less than or equal to the hypothesized mean µ0 = $50.



Alternative hypothesis Ha: The population mean µ is greater than the hypothesized mean µ0 = $50. b. What is the rejection rule?

We will reject the null hypothesis H0 if the p-value for the test statistic is less than the significance level α. For our problem, we would reject the null if the p-value < α = 0.05.

11 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

c. What is the meaning of the test statistic tdata?

The test statistic tdata represents the sample-mean distance above or below the hypothesized mean relative to the standard error (the standard error loosely indicates how much the test statistic can vary). For our problem, the test statistic would be expressed in dollars per standard error. Therefore, if the number of dollars per standard error were found to be too extreme, we would reject the null hypothesis and conclude that the population mean must be greater than $50. d. Is the value of the test statistic tdata extreme? How can we tell? What is the meaning of the p-value in this example?

The test statistic tdata is formulated using as follows:

𝐭 𝐝𝐚𝐭𝐚 =

𝐱 − 𝛍𝟎 𝐬 𝐧

The test statistic for our problem is calculated as follows:

t data =

x − μ0 $55 − $50 $5 = = =𝟐 s $25 $2.50 n 100

Where the sample size n = 100, the sample mean 𝒙 , the sample standard deviation s = $25, and the hypothesized mean µ0 = $50.

In order to determine the extremity of the test statistic, we must calculate its corresponding p-value on the t-distribution. Since this is a right-tailed test, the appropriate p-value expression is P(t > tdata) and represents the probability that there exists test-statistics more extreme than the one that was calculated. For the problem at hand, we need the p-value where P(t > 2) on the t-distribution, which at the 0.05 significance level in 99 degrees of freedom is 0.02412. Therefore, there is definitely solid evidence against the null hypothesis.

e. What is our conclusion?

Our conclusion is to reject the null hypothesis in favor of the alternative hypothesis and presume that the population mean must be greater than $50 based on the data observed.

12 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. f. Interpret our conclusion so that a non-specialist could understand it.

Our initial assumption that the amount donated on average would be less than or equal to $50 is contradicted by the data observed. Specifically, there is only about a 2% chance that this assumption is actually true given the fact that there were already 100 donors and the average donation amount among these 100 donors is $55 where most (approximately 65) of the donations range between $30 and $80 (i.e. $55 ± $25).

24. Refer to the hypothesis test in the previous exercise. Suppose we now set 𝛼 = 0.01. a. What would our conclusion now be? Interpret this conclusion.

If the significance level were reduced to 0.01, we would now fail to reject the null hypothesis concluding that there is not enough evidence to disprove it. Specifically, there is more than our required 1% chance (i.e. ~2%) that the null hypothesis is actually true based on the data observed. b. Note that the conclusion has been reversed simply because we have changed the value of 𝛼. But have the data changed? No, simply our level of what we consider to be significance. Instead go ahead and assess the strength of evidence against the null hypothesis.

Instead of selecting a significance level and mechanically rejecting or accepting (i.e. failing to reject) the null hypothesis, we can assess the strength of statistical evidence by evaluating the p-value of the statistic calculated using the table below. p-Value p-value ≤ 0.001 0.001 < p-value ≤ 0.01 0.01 < p-value ≤ 0.05 0.05 < p-value ≤ 0.10 0.10 < p-value ≤ 0.15 0.15 < p-value

Strength of evidence against H0 Extremely strong evidence Very strong evidence Solid evidence Mild evidence Slight evidence No evidence

Table 24.1 Strength of evidence against H0 for several key p-values

Referring to the table above, we observe that our p-value of ~0.024 lies between 0.01 and 0.05 indicating that there is solid evidence against H0 (i.e. that the average donation is ≤ $50). In other words, it is unlikely given the fact that we have already observed 100 donors and their average donation amount is $55 with a standard deviation of $25.

13 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 25. Refer to the first confidence interval you calculated for the population mean duration of customer service calls. Use this confidence interval to test whether this population mean differs from the following values, using level of significance 𝛼 = 0.05.

We can leverage a confidence interval to perform hypothesis testing as it is complimentary to a two-tailed hypothesis test. Our hypotheses for this type of test would be formed as H0: µ = µ0; Ha: µ ≠ µ0.

Since our confidence level is 95%, our significance level α for these tests is calculated as follows: α = (1-95%/100%) = 0.05 The first confidence interval we calculated for population mean duration of customer calls was in Exercise 17a and was based on a sample size n=25, sample mean of 20 minutes, and sample standard deviation of 5 minutes and was calculated as follows: Confidence Interval = (20) ± 2.064 = (17.936, 22.064) minutes.

a. 15 minutes Since 15 minutes lies outside of (i.e. below) the confidence interval, we would reject the null hypothesis for this test and conclude that the population mean-duration is not 15 minutes. b. 20 minutes Since 20 minutes lies within the confidence interval, we would fail to reject the null hypothesis for this test and conclude that there is not enough evidence to indicate that the population mean-duration is not 20 minutes.

c. 25 minutes Since 25 minutes lies outside of (i.e. above) the confidence interval, we would reject the null hypothesis for this test and conclude that the population mean-duration is not 15 minutes.

14 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 26. In a sample of 1000 customers (note that the original text says 100), 240 churned when the company raised rates. Test whether the population proportion of churners is less than 25%, using level of significance 𝛼 = 0.01.

We begin by calculating the sample proportion p as follows:

𝑝=

𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑒𝑛𝑡𝑠 240 = = 𝟎. 𝟐𝟒 (𝑠𝑎𝑚𝑝𝑙𝑒 𝑠𝑖𝑧𝑒) 1000

We conduct a left-tailed test forming the hypotheses as follows: H0: population proportion π ≥ 0.25; Ha: population proportion π < 0.25.

The test statistic using the Standard Normal distribution is calculated as follows: 𝑍𝑑𝑎𝑡𝑎 =

𝑝 − 𝜋0 𝜋0 (1 − 𝜋0 ) 𝑛

=

0.24 − 0.25 0.25 0.75 1000

=

−0.01 = −𝟎. 𝟕𝟑𝟎𝟓 0.01369

This is a left-tailed test, so we calculate the p-value on the Standard Normal distribution as P(Z < -0.7305) = 0.232542. Since the p-value is greater (much greater) than the significance level of 0.01, we fail to reject the null hypothesis concluding that there is no evidence to dispute it. In other words, there is no evidence to indicate that the population proportion of churners will remain less than 25% even though the sample proportion is only 24%.

15 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Solutions to Chapter 5 MULTIVARIATE STATISTICS Prepared by James Cunningham, Graduate Assistant 1. In Chapter 6, we will learn to split the data set into a training data set and a test data set. To test whether there exist unwanted differences between the training and test set, which hypothesis test do we perform, for the following types of variables: a. Flag variable For a flag variable, use the two-sample Z-test for the difference in proportions. b. Multinomial variable For a multinomial variable, use the test for the homogeneity of proportions c. Continuous variable For a continuous variable, use the two-sample t-test for the difference in means.

2. Table 5.10 contains information on the mean duration of customer service calls between a training and a test data set. Test whether the partition is valid for this variable, using 𝛼 = 0.10.

TABLE 5.10 Summary statistics for duration of customer service calls Data set Sample Mean Sample Standard Deviation Sample Size Training set s1 = 5.2 n1 = 2000 x1 = 20.5 Test set s2 = 4.9 n2 = 600 x2 = 20.4 This is a continuous variable, so we will use the two-sample t-test to determine whether or not the partition is valid. The partition will be considered valid if there is no evidence indicating that the population means for the two partitions are different. The hypotheses are H0: 𝜇1 = 𝜇2 vs. Ha: 𝜇1 ≠ 𝜇2, and the statistic is calculated as follows: 𝑡𝑑𝑎𝑡𝑎 =

𝑥1 − 𝑥2 𝑠12 𝑠22 + 𝑛1 𝑛2

=

20.5 − 20.4 5.22 4.92 2000 + 600

=

0.1 = 𝟎. 𝟎𝟒𝟑𝟐𝟐 0.2314

This is a two-tailed t-test with 2599 degrees of freedom where the statistic is greater than zero, and is calculated as follows: p-value = 2 *P (t > 0.04322) = 0.9655 Since the p-value is greater than 0.15, there is no evidence to presume that the population means for the training and test sets are different, so the partitioning is valid.

1 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 3. Our partition shows that 800 of the 2000 customers in our test set own a tablet, while 230 of the 600 customers in our training set own a tablet. Test whether the partition is valid for this variable, using 𝛼 = 0.10.

Ownership of a tablet is a flag variable, so we use the two-sample Z-test to determine whether or not the partition is valid. The partition will be considered valid if there is no evidence indicating a difference in proportions.

The hypotheses are H0: 𝜋1 = 𝜋2 vs. Ha: 𝜋1 ≠ 𝜋2 , and the statistic is formulated as follows:

𝑍𝑑𝑎𝑡𝑎 =

𝑝1 − 𝑝2 1 1 𝑝𝑝𝑜𝑜 𝑙𝑒𝑑 ∗ 1 − 𝑝𝑝𝑜𝑜𝑙𝑒𝑑 ∗ 𝑛 + 𝑛 1 2

The proportions p1, p2, and ppooled proportion is calculated as follows: 𝑝1 =

𝑥1 800 𝑥2 230 = = 𝟎. 𝟒𝟎𝟎; 𝑝2 = = = 𝟎. 𝟑𝟖𝟑𝟑 𝑛1 2000 𝑛2 600

𝑝𝑝𝑜𝑜𝑙𝑒𝑑 =

𝑥1 + 𝑥2 800 + 230 1030 = = = 𝟎. 𝟑𝟗𝟔𝟐 𝑛1 + 𝑛2 2000 + 600 2600

The statistic is calculated as follows: 𝑍𝑑𝑎𝑡𝑎 =

0.400 − 0.3833 1 1 0.3962 ∗ 0.6038 ∗ (2000 + 600)

=

0.0167 = 𝟎. 𝟕𝟐𝟑𝟓 0.0228

This is a two-tailed Z-test where the statistic is greater than zero, and the p-value is calculated as follows: p-value = 2 * P(Z > 0.7235) = 0.4694 Since the p-value is greater than 0.15, there is no evidence to presume that the population proportions for the training and test sets are different, so the partitioning is valid.

2 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 4. Table 5.11 contains the counts for the marital status variable for the training and test set data. Test whether the partition is valid for this variable, using 𝛼 = 0.10.

Data set Training set Test set Total

TABLE 5.11 Observed frequencies for marital status Married Single Other 800 750 450 240 250 110 1040 1000 560

Total 2000 600 2600

Since marital status is a multinomial variable, we will use the test for the homogeneity of proportions. The proportions will be deemed homogenous if there is no evidence indicating heterogeneity. The hypotheses are as follows: H0: pmarried,training = pmarried,test, psingle,training = psingle,test, pother,training = pother,test Ha: At least one of the claims in H0 is wrong We calculate the expected frequencies (i.e. the frequencies we would observe if H0 were true) using the following formulation: 𝑐𝑙𝑎𝑠𝑠 𝑜𝑏𝑠𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛𝑠 ∗ 𝑝𝑎𝑟𝑡𝑖𝑡𝑖𝑜𝑛 𝑠𝑖𝑧𝑒 𝐸𝑥𝑝𝑒𝑐𝑡𝑒𝑑 𝐹𝑟𝑒𝑞𝑢𝑒𝑛𝑐𝑦 = 𝑑𝑎𝑡𝑎 𝑠𝑒𝑡 𝑠𝑖𝑧𝑒 The expected frequencies are calculated as follows: Training Set

Test Set

Married

𝟖𝟎𝟎

𝟐𝟒𝟎

Single

𝟕𝟔𝟗

𝟐𝟑

Other

𝟒𝟑

𝟐𝟗

Table 4.1. Marital Status Expected Frequencies

The Chi-Square statistic is formulated as follows: 𝑑

𝑐

2 𝑋𝑑𝑎𝑡𝑎 = 𝑖=1 𝑗 =1

𝑂𝑖𝑗 − 𝐸𝑖𝑗 𝐸𝑖𝑗

2

3 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. Where d is the number of partitions (e.g. training, test), and c is the number of classes (e.g. married, single, etc.). The individual ratios and final chi-square statistic are calculated as follows: Training Set

Test Set

Married Single Other Χ2data

5.6682

Table 4.2. Marital Status Ratios and Chi-Square statistic

This is a chi-square statistic with 2 degrees of freedom (i.e. sets-1* classes-1), and is calculated as follows: p-value = P(Χ2 > 5.6682) = 0.05877 Not only is the p-value small in comparison to the significance level of 0.10, it is small in general, so there is solid evidence against the null hypothesis that the proportions are homogeneous. We conclude that the partitioning is not valid and thus should be reconstructed.

5. The multinomial variable payment preference takes the values credit card, debit card, and check. Now, suppose we know that 50% of the customers in our population prefer to pay by credit card, 20% prefer debit card, and 30% prefer to pay by check. We have taken a sample from our population, and would like to determine whether it is representative of the population. The sample of size 200 shows 125 customers preferring to pay by credit card, 25 by debit card, and 50 by check. Test whether the sample is representative of the population, using 𝛼 = 0.05.

The payment variable is another example of a multinomial variable, but this problem supplies the population proportions, so we would use the Chi-Square Goodness of Fit test. The proportions of the sample data will be deemed representative of the population proportions if there is no evidence indicating a difference in proportionality.

The hypotheses are as follows: H0: pcredit-card = 0.50, pdebit-card = 0.20, pcheck = 0.30 Ha: At least one of the proportions in H0 is wrong 4 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. The expected frequencies are calculated as follows:

𝐸𝑐𝑟𝑒𝑑𝑖𝑡 −𝑐𝑎𝑟𝑑 = 200 ∗ 0.50 = 𝟎𝟎; 𝐸𝑑𝑒𝑏𝑖𝑡 −𝑐𝑎𝑟𝑑 = 200 ∗ 0.20 = 𝟒𝟎; 𝐸𝑐ℎ𝑒𝑐𝑘 = 200 ∗ 0.30 = 𝟔𝟎 The Chi-Square statistic is formulated as follows: 𝑐 2 𝑋𝑑𝑎𝑡𝑎 = 𝑗 =1

𝑂𝑗 − 𝐸𝑗 𝐸𝑗

2

Where c is the number of classes (e.g. credit card, debit card, etc.).

The individual ratios and the chi-square statistic are calculated as follows: Calculation Credit Card Debit Card Check X2data

38.8263

Table 5.1. Payment Ratios and Chi-Square statistic

This is a chi-square statistic with 2 degrees of freedom (i.e. classes-1), and the p-value is calculated as follows: p-value = P(Χ2 > 38.8263) = 0.00001 Not only is the p-value small in comparison to the significance level of 0.10, it is small in general, so there is extremely strong evidence against the null hypothesis that the sample proportions are representative of the population proportions. We conclude that the sampling is not representative and thus should be resampled.

5 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 6. Suppose we wish to test for difference in population means among three groups. a. Explain why it is not sufficient to simply look at the differences among the sample means, without taking into account the variability within each group.

When testing for differences in population means, it is not sufficient to look at the differences among the sample means. In order to determine whether there is a difference in population means among several samples, we must look at the ratio of the betweensample variability (i.e. the sample-mean differences) and the within-sample variability (i.e. the spread within each sample). b. Describe what we mean by between-sample variability and within-sample variability.

As described in the previous question, the between-sample variability is the variation among the sample means. The within-sample variability is the variation of the data within each sample. c. Which statistics measure the concepts in (b).

There are three statistics used in order to analyze the variation. These are the (1) meansquare treatment (MSTR), (2) mean-square error (MSE), and (3) the F-statistic. The three statistics are formulated as follows: 𝑴𝑺𝑻𝑹 =

𝒌 𝒊=

𝒏𝒊 𝒙𝒊 − 𝑴 𝟐 𝒅𝒇

The MSTR measures the between-sample variation where ni is the size of sample i, k is the number of samples, M is the overall mean or Grand Mean, and df1 is the betweensample degrees of freedom calculated as df1 = k - 1. 𝑴𝑺𝑬 =

𝒌 𝒊=

𝒏𝒊 − 𝒅𝒇𝟐

𝒔𝟐𝒊

The MSE measures the within-sample variation where ni is the size of sample i, k is the number of samples, si is the standard deviation of sample I, and df2 is the within-sample degrees of freedom calculated as df2 = N – k where N is the Grand Sample Size or total number of observations across all samples. 𝑴𝑺𝑻𝑹 𝑭𝒅𝒂𝒕𝒂 = 𝑴𝑺𝑬 The F-statistic is just the ratio of the between-sample variation and the within-sample variation. When the F-statistic is large, the p-value will be small and so will be the likelihood that the null hypothesis is true (i.e. the population means for the underlying samples are equal).

6 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. d. Explain how analysis of variance would work in this situation.

When the MSTR is large relative to the MSE, the F-statistic will be large and the corresponding p-value will be small. Therefore, the small p-value indicates that there is evidence against the null hypothesis and we would conclude that the population means underlying the samples are not equal.

7. Table 5.12 contains the amount spent (in dollars) in a random sample of purchases where the payment was made by credit card, debit card, and check, respectively.

Test whether the population mean amount spent differs among the three groups, using 𝛼 = 0.05.

TABLE 5.12 Purchase amounts for three payment methods Credit Card 100 110 90 100

Debit Card 80 120 90 110

Check 50 70 80 80

Analysis of Variance (ANOVA) is used test whether the underlying population means for each of three payment methods differs. The hypotheses are as follows: H0: 𝜇Credit-Card = 𝜇Debit-Card = 𝜇Check Ha: Not all the population means are equal.

For this problem, it is helpful to note the following supporting statistics:

CreditCard

Sample Size

Sample Mean

4

(100+110+90+100)/4=100

DebitCard

4

(80+120+90+110)/4=100

Check

4

(50+70+80+80)/4=70

Sample Variance [(100-100)2+(110-100)2+(90-100)2+(100-100)2]/(4-1)

=(0+100+100+0)/3 = 66.6667 [(80-100)2+(120-100)2+(90-100)2+110-100)2]/(4-1)

=(400+400+100+100)/3 = 333.3333 [(50-70)2+(70-70)2+(80-70)2+(80-70)2]/(4-1)

=(400+0+100+100)/3 = 200

Table 6.1. Payment sample mean & sample variance

In addition to the statistics in the table above, the number of samples k = 3, the Grand Mean M = 90, and the Grand Sample Size N = 12 as there are four observations in each sample.

Also, the between-samples degrees of freedom df1 = k-1 = 2, and the within-sample degrees of freedom df2 = N – k = 9.

7 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. The testing statistics are calculated as follows:

𝑀𝑆𝑇𝑅 =

𝑀𝑆𝐸 =

𝑘 𝑖=1 𝑛𝑖

𝑥𝑖 − 𝑀 2 4 100 − 90 2 + 4 100 − 90 2 + 4 70 − 90 2 = = 𝟐𝟎𝟎 𝑑𝑓1 2

𝑘 2 𝑖=1 𝑛𝑖 − 1 𝑠𝑖

𝑑𝑓2

=

3 66.6667 + 3 333.3333 + 3 200 = 𝟐𝟎𝟎 9

𝐹𝑑𝑎𝑡𝑎 =

𝑀𝑆𝑇𝑅 1200 = =𝟔 𝑀𝑆𝐸 200

This is an F statistic with df1 = 2 and df2 = 9, and the p-value is calculated as follows: p-value = P(F > 6) = 0.0221 Since the p-value is less than 0.05, we reject the null hypothesis as there is solid evidence indicating that the underlying population means are not the same.

8. Refer to the previous exercise. Now test whether the population mean amount spent differs among the three groups, using 𝛼 = 0.01. Describe any conflict between your two conclusions. Suggest at least two courses of action to ameliorate the situation.

If we were to reduce the significance level to 0.01, we would fail to reject the null hypothesis as there would not be enough evidence to indicate otherwise. This leaves us with two courses of amelioration. First, we can simply fail to reject the null hypothesis at this smaller significance level and utilize this sample set. Second, we can dispense with selecting arbitrary significance levels and accept the fact that a p-value of ~0.02 indicates that there is strong evidence against the null hypothesis and reconstruct the sample sets.

9. Explain why we use regression analysis and for which type of variables it is appropriate.

Regression analysis is used to estimate or predict values of a target variable based on the values of one or more predictor variables. This type of analysis is appropriate for continuous variables only as this analysis is performed on an interval scale.

8 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 10. Suppose that we are interested in predicting weight of students based on height. We have run a regression analysis with the resulting estimated regression equation as follows: “The estimated weight equals (–180 pounds) plus (5 pounds times the height in inches).” a. Suppose that one student is 3 inches taller than another student. What is the estimated difference in weight?

In order to estimate the relative difference in weight, we note that weight is expressed in units of pounds, and height is expressed in units of inches. The regression equation is formulated as follows:

𝑾𝒆𝒊𝒈𝒉𝒕 = 𝟓𝒍𝒃𝒔 ∗ 𝒉𝒆𝒊𝒈𝒉𝒕 − 𝟖𝟎𝒍𝒃𝒔 𝒊𝒏

Let us define the heights for the two hypothetical students as (h) and (h+3). The difference in weight would then be calculated as follows:

𝑊𝑒𝑖𝑔ℎ𝑡𝑑𝑒𝑙𝑡𝑎 = 5𝑙𝑏𝑠 ℎ + 3 𝑖𝑛 − 180𝑙𝑏𝑠 − 5𝑙𝑏𝑠 ℎ𝑖𝑛 − 180𝑙𝑏𝑠 𝑖𝑛

𝑖𝑛

= 5ℎ𝑙𝑏𝑠 − 15𝑙𝑏𝑠 − 5ℎ𝑙𝑏𝑠 = 𝟓𝒍𝒃𝒔

b. Suppose that a given student is 65 inches tall. What is the estimated weight?

The estimated weight for a student that is 65 inches tall is calculated as follows: 𝑊𝑒𝑖𝑔ℎ𝑡 = 5𝑙𝑏𝑠 ∗ ℎ𝑒𝑖𝑔ℎ𝑡 − 180𝑙𝑏𝑠 = 5𝑙𝑏𝑠 65𝑖𝑛 − 180𝑙𝑏𝑠 𝑖𝑛

𝑖𝑛

= 325𝑙𝑏𝑠 − 180𝑙𝑏𝑠 = 𝟒𝟓𝒍𝒃𝒔 c. Suppose that the regression equation above was based on a sample of students ranging in height from 60 to 75 inches. Now estimate the height of a 48-inch-tall student. Comment.

The weight of the 48-inch student is calculated as follows: 𝑊𝑒𝑖𝑔ℎ𝑡 = 5𝑙𝑏𝑠 ∗ ℎ𝑒𝑖𝑔ℎ𝑡 − 180𝑙𝑏𝑠 = 5𝑙𝑏𝑠 48𝑖𝑛 − 180𝑙𝑏𝑠 𝑖𝑛

𝑖𝑛

= 240𝑙𝑏𝑠 − 180𝑙𝑏𝑠 = 𝟔𝟎𝒍𝒃𝒔

9 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. However, is 60 pounds a valid estimate? It is important to note that a regression equation is valid for estimates within the domain of its predictor values. Deriving an estimate outside of the domain of predictor values is referred to as extrapolation, and should be avoided if possible.

Therefore, if the equation was derived based on a sample of students ranging in height between 60 and 75 inches, attempting to estimate the weight of a 48-inch student would produce erroneous or even nonsensical results if the relationships between predictor and target are different outside of the domain of values used to build a predictive model. d. Explain clearly the meaning of the 5 in the equation above.

The 5 in the regression equation above is actually the ratio of a student’s weight in pounds to a student’s height in inches. If we were to graph the equation, this value would determine the slope of the regression line. e. Explain clearly the meaning of the –180 in the equation above.

The -180 is the number of pounds is the amount that must be added (i.e. subtracted) from the relative estimate (e.g. 5*height) in order to determine the absolute estimate. If we were to graph the equation, this value would determine the y-intercept of the regression line.

10 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. HANDS-ON ANALYSIS Use the cereals data set included, at the book series website, for the following exercises. Use Regression to estimate rating based on fiber alone.

11. What is the estimated regression equation?

Using SPSS Modeler, the Regression Node produces the output depicted in the figure below.

Figure 11.1. Regression equation

The information from the table is used to construct the regression equation as follows:

𝑹𝒂𝒕𝒊𝒏𝒈 = 𝟑. 𝟒𝟒𝟑 ∗ 𝑭𝒊𝒃𝒆𝒓 + 𝟑𝟓. 𝟐𝟓𝟕 12. Explain clearly the value of the slope coefficient you obtained in the regression.

The slope coefficient 3.443 is the ratio of Rating points per grams of Fiber. We may therefore define the units for this coefficient as pts/g.

13. What does the value of the y-intercept mean for the regression equation you obtained? Does it make sense in this example?

The value of the y-intercept is 35.257 and is the number of rating points that must be added to the relative estimate of Rating in order to determine the absolute estimate of Rating. This makes sense in this example as it will always yield a positive Rating for any cereal containing Fiber.

11 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 14. What would be a typical prediction error obtained from using this model to predict rating? Which statistic are you using to measure this? What could we do to lower this estimated prediction error?

A typical prediction error for this model is approximately 11.5 points and is given by the Standard Error of the Estimate as depicted in the figure below.

Figure 14.1 Regression Model quality statistics

The best way to lower the prediction error is to add predictor variables to the regression model.

15. How closely does our model fit the data? Which statistic are you using to measure this?

The model does not fit the data well at all as indicated by the R2 statistic value of 0.341as depicted above in Figure 14.1. Values close to 1 identify an excellent fit, but values close to zero identify no fit at all.

16. Find a point estimate for the rating for a cereal with a fiber content of 3 grams.

A point estimate rating for a cereal with 3 grams of fiber would be calculated using the regression equation as follows: 𝑅𝑎𝑡𝑖𝑛𝑔 = 3.443𝑝𝑡𝑠 3𝑔 + 35.257𝑝𝑡𝑠 = 𝟒𝟓. 𝟓𝟖𝟔𝒑𝒕𝒔 𝑔

17. Find a 95% confidence interval for the true mean rating for all cereals with a fiber content of 3 grams.

The confidence interval (CI) is defined as the point estimate ± margin of error, and for a regression equation is formulated as follows: 𝒙𝒑 − 𝒙 𝑪𝑰𝒚𝒑 = 𝒚𝒑 ± 𝒕𝜶 (𝒔) + 𝒏 𝑺𝑺𝑬 𝟐

𝟐

12 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

The margin of error (ME) is then calculated as follows:

𝑀𝐸 = 1.992 11.4771

1 3 − 2.152 2 + = 22.8624 0.0130 + 7.28𝑒 −5 77 9879.241

= 22.8624 0.1145 = 𝟐. 𝟔 𝟕𝟕 Therefore, the 95% confidence interval is 45.586𝑝𝑡𝑠 ± 2.6177𝑝𝑡𝑠 = 𝟒𝟐. 𝟗𝟔𝟖, 𝟒𝟖. 𝟐𝟎𝟒

18. Find a 95% prediction interval for a randomly chosen cereal with a fiber content of 3 grams.

The prediction interval (PI) is defined as the point estimate ± margin of error, and for a regression equation is formulated with an additional ‘+1’ in the radical to ensure that the prediction interval is always wider than its corresponding confidence interval as follows:

𝑪𝑰𝒚𝒑 = 𝒚𝒑 ± 𝒕𝜶 (𝒔) 𝟐

𝒙𝒑 − 𝒙 + + 𝒏 𝑺𝑺𝑬

𝟐

The margin of error (ME) is then calculated as follows:

𝑀𝐸 = 1.992 11.4771

1 3 − 2.152 2 1+ + = 22.8624 1 + 0.0130 + 7.28𝑒 −5 77 9879.241

= 22.8624 1.0065 = 𝟐𝟑. 𝟎

13 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. Based on the regression results, what would we expect a scatter plot of rating versus fiber to look like? Why? Based on the regression results, we would expect a scatter plot of rating vs. fiber to be spread fairly wide above and below the line defined by the regression equation. In other words, the data points would not be at all close to the line. The reason for this expectation is due to the fact that the R2 statistic is relatively close to zero (i.e. 0.341) indicating a poor fit of the data.

For the following exercises, use Multiple Regression to estimate rating based on fiber and sugars.

20. What is the estimated regression equation?

Adding the sugars to the predictors (without imputation) causes us to lose an observation, and SPSS Modeler produces the output depicted in the figure below.

Figure 20.1. Regression equation with Sugars added

The information from the table is used to construct the regression equation as follows:

𝑹𝒂𝒕𝒊𝒏𝒈 = 𝟐. 𝟖𝟔𝟖 ∗ 𝑭𝒊𝒃𝒆𝒓 − 𝟐. 𝟖𝟒 ∗ 𝑺𝒖𝒈𝒂𝒓𝒔 + 𝟓 . 𝟔 𝟎

21. Explain clearly and completely the value of the coefficient for fiber you obtained in the regression.

The slope coefficient 2.868 is still the ratio of Rating points per grams of Fiber. We may therefore define the units for this coefficient as pts/g. However, it now comprises only part of the relative estimate. The other part of the relative estimate is determined by the sugars coefficient.

14 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 22. Compare the r2 values from the multiple regression and the regression done earlier in the exercises. What is going on? Will this always happen?

The R2 value for the multiple regression equation is 0.809 as depicted in the figure below. This value indicates a good fit of the data.

Figure 22.1 Multiple-Regression quality statistics

In comparison with R2 from the fiber-only regression equation (0.341) we conclude that the multiple-regression model produces a fit far superior to that of the fiber-only model. In general, adding predictors (especially uncorrelated ones) to a model is always expected to produce a better model.

23. Compare the s values from the multiple regression and the regression done earlier in the exercises. Which value is preferable, and why?

The standard error of the estimate s for the multiple-regression equation is 6.219 as depicted in Figure 22.1 above. In comparison with the standard error of the estimate from the fiber-only regression equation, which is 11.477, we conclude that the multiple-regression model is preferred as it provides estimates that are approximately 5 points more accurate.

15 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal Larose, Ph.D.

Solutions to Chapter 6 PREPARING TO MODEL THE DATA Prepared by James Cunningham, Graduate Assistant 1. Explain the difference between supervised and unsupervised methods. Which data mining tasks are associated with unsupervised methods? Supervised? Both?

Unsupervised methods do not specify a target variable, and attempt to find patterns and structure among the variables. In contrast, supervised methods identify a target variable and attempt to learn which values in the predictor fields correspond with values of the target variable.

The most commonly used unsupervised method is clustering. Association rule mining can be unsupervised or supervised. Most data mining algorithms are supervised methods and include the use of decision trees, neural networks, and k-nearest neighbor algorithms.

2. Describe the differences between the training set, test set, and validation set.

The training set is used to build the provisional data model and consists of records containing the predictor variables and pre-classified values for the target variable. The test set is used to evaluate the performance of the provisional data model using unseen data, where the target values are temporarily hidden. The provisional data model is adjusted to minimize the error rate on the test set. Finally, the performance of the adjusted data model is evaluated against the validation test set. The adjusted data model is fine-tuned to minimize the error rate on the validation data set.

3. Should we strive for the highest possible accuracy with the training set? Why or why not? How about the validation set?

Maximizing the accuracy of the data model against the training set leads to an overly-complex model (overfitting). In this case, the model will tend to memorize the idiosyncrasies in the training set, and not generalize well to the test and validation data sets. The optimal level of model complexity is achieved when the error rate of the provisional data model, as measured against the validation set, is minimized.

4. How is the bias–variance trade-off related to the issue of overfitting and underfitting? Is high bias associated with overfitting and underfitting, and why?

High variance?

A low-complexity model has low variance, which results in a tradeoff of high bias (high error rate). In this case the model tends to under fit the data, although it has better generalizability to unseen data. In contrast, a highly-complex model that has high variance is combined with a low bias (low error rate). This describes a case of overfitting the data.

1 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal Larose, Ph.D.

5. Explain why we sometimes need to balance the data.

The training set is balanced so that a sufficient number of common, as well as less-common, classifications occur in the data set. This way, the algorithm will have less of a tendency to classifying only the most-commonly occurring classifications.

6. Suppose we are running a fraud classification model, with a training set of 10,000 records of which only 400 are fraudulent. How many fraudulent records need to be resampled if we would take the proportion of fraudulent records in the balanced data set to be 20%?

If we want the resampled data to comprise 20% of the training set, we would leverage the following equation:

𝟒𝟎𝟎 + 𝒙 = 𝟎. 𝟐𝟎(𝟏𝟎𝟎𝟎𝟎 + 𝒙) We then solve the above equation for x as follows:

400 + 𝑥 = 2000 + 0.20𝑥 0.80𝑥 = 1600 𝒙 = 𝟐𝟎𝟎𝟎 Therefore, 2000 additional fraudulent records need to be resampled for the fraudulent records to comprise 20% of the training set. As a cross-check, we note that the training set would have 12,000 records after resampling where 2,400 of them would be fraudulent, and since 2400 is 20% of 12,000 we know we have calculated at the correct answer.

7. When should the test data set be balanced?

The test data set should never be balanced. The test data set represents real-world data and should be evaluated as-is.

8. Explain why we should always report a baseline performance, rather than merely citing the uncalibrated results from our model.

The reason we should always report a baseline performance is so that the consumers of the analysis have a frame of reference to compare model performance results to. We should also identify when we are making an absolute comparison between baseline and 2 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal Larose, Ph.D. model performance and when we are making a relative comparison between baseline and model performance.

9. Explain the distinction between reporting an absolute difference versus a relative difference.

An absolute difference is the subtractive difference between two performance measures whereas a relative difference is a ratio of the absolute difference and the larger of the two performance measures. It is important to note that the smaller the measures are, the more extreme a relative difference will appear to be.

As an example, let’s say we have an existing model that identifies that 10% of a customer base will buy our new product, and we replace the existing model with a revised model that now identifies that 14% of the customer base will buy our new product. The absolute difference in customer identification is calculated as follows:

𝑆𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦𝐴𝑏𝑠𝑜𝑙𝑢𝑡𝑒 = 14% − 10% = 𝟒% However, the relative difference (which appears to be way more extreme) is calculated as follows: 4% 𝑆𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦𝑅𝑒𝑙𝑎𝑡𝑖𝑣𝑒 = = 𝟐𝟖. 𝟓𝟕% 14% 10. If we are using a regression model, what form may our baseline model take?

If we are using a regression model, our baseline model may be a mean-prediction model, where we would train the model on a data set where the target value for all observations is the target variable mean.

We would then use the quality statistics such as the standard error of the estimate and the R2 from this model in order to measure the effectiveness of the real predictive model.

3 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Solutions to Chapter 7 k-NEAREST NEIGHBOR ALGORITHM Prepared by James Cunningham, Graduate Assistant 1. Clearly describe what is meant by classification.

Classification is a type of predictive modeling where the target variable is categorical having a predefined finite set of possible outcomes or classes. The objective of the classification model is to classify each new observation determining which class each new observation belongs to.

2. What is meant by the term instance-based learning?

Instance-based learning is a type of learning where the training data is stored in memory and new observations are classified or estimated by comparing them to the training data in memory.

3. Make up a set of three records, each with two numeric predictor variables and one categorical target variable, so that the classification would not change regardless of the value of k.

The table below identifies three customers that have purchased cars at a particular dealership and the dealer wants to be able to classify new customers in order to assign the sales staff member who has the greatest expertise in the car type of interest. If we were to use KNN in order to classify a new potential buyer based on the nearest neighbors below, then our choice of k will have no effect on the classification as all three neighbors are in the same class (i.e. the SUV class).

Customer 1 2 3

Age 25 35 37

Height (Inches) 70 68 69

CarType SUV SUV SUV

Table 3.1. Set of 3 records – two numeric variables and one categorical variable

1 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 4. Refer to Exercise 3. Alter your data set so that the classification changes for different values of k.

If we were to alter the set of neighbors depicted in the below to classify a new potential buyer, then our choice of k will now affect the classification as these three neighbors are in different classes (i.e. the SUV and Sedan classes). When k=1, the new record will be classified as either an SUV-buyer or a Sedan-buyer. When k=3 (I am deliberately saving k=2 for last), the new record will be classified as an SUV-buyer with 66.67% confidence.

Finally, when k=2, the model would need to employ additional criteria for breaking the potential tie that could arise when the two nearest neighbors were either {1,3} or {2,3}.

Customer 1 2 3

Age 25 35 37

Height (Inches) 70 72 68

CarType SUV SUV Sedan

Table 4.1. Set of 3 records – altered classification

5. Refer to Exercise 4. Find the Euclidean distance between each pair of points. Using these points, verify that Euclidean distance is a true distance metric.

Using the records in Exercise #4, we calculate the Euclidean distance between each of the three points. The results are given the table below.

Distance Measurement D(1,2)

Euclidean Distance

D(1,3) D(2,3) Table 5.1. Euclidean distances

Referring to the table above, we observe that D(2,3) is the shortest distance. However, these two end points are classified differently (i.e. #2 is an SUV-buyer and #3 is a Sedan-buyer).

This is due to the fact that the Euclidean distance metric is more heavily influenced by the predictors having the greatest variation. Since the range of age is 37-25 = 12 and the range of Height is only 72-68 = 4, the Euclidean distances between records will be driven more by Age than by Height, which explains why D(2,3) is shortest.

If we were to standardize the predictors say using Min-Max normalization, the values would vary on similar scales. The results are given in the table below.

Customer 1 2 3

Age 0 0.83 1

Height (Inches) 0.05 1 0

CarType SUV SUV Sedan

Table 5.2. Set of three records – Min-Max normalized

2 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. We then recalculate the Euclidean distances using these scaled values. The results are given in the table below.

Distance Measurement D(1,2) D(1,3) D(2,3)

√( √( √(

Euclidean Distance ) ( ) ) ( ) ) ( )

Table 5.3. Euclidean distances for normalized variables

Now that we are using similar scales for both predictors, we observe that the closest points are 1 and 2, which is what we would expect. Therefore, we conclude that Euclidean distance is a true distance metric, but care must be taken to transform the predictors so that they vary on similar scales.

6. Compare the advantages and drawbacks of unweighted versus weighted voting.

Unweighted voting is the simplest to implement, new observations are classified simply by the majority class of its k nearest neighbors. While this works well for training sets that have good separation by class, it works poorly when the training set does not have good separation by class (i.e. the classes of the training points are thoroughly mixed from a geometric perspective). For example, let’s say we have classes A, B, and C and k=3. If the three neighbors happen to be {A,A,B}, then we classify the new observation as an ‘A’. However, if the three neighbors happened to be {A,B,C}, then we would have a three-way tie.

In contrast, weighted voting is more difficult to implement, but it gives precedence to the neighbors that are closest to the new observation. This approach works well even for training sets that do not have good geometric separation by class. However it is not completely rigorous. In the rare case that there is k neighbors lying at a distance of zero from the new observation, the inverse distance would be undefined, and the prescribed approach is to fall back to majority classification, which can result in multi-way ties as in the non-weighted voting approach.

7. Why does the database need to be balanced?

It is important to ensure that the database is balanced when constructing KNN models so that he algorithm does not become limited to predicting the common classifications. The data must contain enough rare occurrences so that the model can predict similarly rare observations as well.

3 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 8. The example in the text regarding using the k-nearest neighbor algorithm for estimation has the closest record, overwhelming the other records in influencing the estimation. Suggest two creative ways that we could dilute this strong influence of the closest record.

The example in the text has the closest record overwhelming the other records in influencing the estimation. However, sometimes classification based on the closest record may not be the best approach. When geometry does not align well with domain expertise, we can mitigate the overpowering influence of the closest record(s) by having the algorithm concentrate on the most relevant predictors. There are actually two creative ways to do this.

The first of these ways is to limit the distance calculation to using the relevant predictors only or having the algorithm ignore less relevant predictors. This approach will prevent the learning algorithm from basing the distance calculations on predictors that have large yet less relevance.

The second of these ways is to retain all predictors in the learning algorithm, but determine appropriate multipliers to the more relevant predictors so they outweigh the less relevant predictors. This approach is referred to as stretching the axes and these multipliers may be determined directly by domain experts, or they can be data-driven using cross-validation. In cross-validation, a small subset is drawn at random and multipliers are chosen until the classification error on the training set is minimized.

9. Discuss the advantages and drawbacks of using a small value versus a large value for k.

Choosing the optimal value for k will most likely not be obvious and may require several attempts. If we choose a small value for k, then the model may be unduly influenced by outliers or noise that may lead to overfitting.

In contrast, if we were to choose a large value for k, then the model will be less likely to be influenced by noise, but it may cause the model to completely overlook the rare conditions generalizing the new observations to the larger mainstream classes.

10. Why would one consider stretching the axes?

We may consider stretching the axes whenever a model is unduly influenced by the less relevant predictors. In other words, the less relevant predictors are causing the more appropriate training samples to reside farther away from observations than they should.

We stretch the axes by choosing multipliers either directly based on domain expertise, or using cross-validation. In cross-validation, a small subset is drawn at random and multipliers are chosen until the classification error on the training set is minimized. This results in distance calculations that are driven more by the most relevant predictors.

4 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

11. What is locally weighted averaging, and how does it help in estimation?

Locally weighted averaging is an approach to estimating / predicting continuous values via a KNN model. It helps in estimation by leveraging the target values of an observation’s nearest neighbors and calculating the target value of the new observation as a weighted average of the neighboring target values. A locally weighted average using KNN is formulated as follows: 𝐲𝐧𝐞𝐰 =

𝐢 𝐰𝐢 𝐲𝐢 𝐢 𝐰𝐢

HANDS-ON ANALYSIS 12. Using the data in Table 7.5, find the k-nearest neighbor for record #10, using k = 3.

Record 1 2 3 4 5 6 7 8 9 10

Age 22 33 28 51 25 39 54 55 50 66

Marital Single Married Other Other Single Single Single Married Married Married

Income $46,156.98 $24,188.10 $28,787.34 $23,886.72 $47,281.44 $33,994.90 $28,716.50 $49,186.75 $46,726.50 $36,120.34

Risk Bad Loss Bad Loss Bad Loss Bad Loss Bad Loss Good Risk Good Risk Good Risk Good Risk Good Risk

5 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. Setting k=3 and using the data above, we calculate the distances to record #10 in order to compare the predicted class to the actual class. However, before we can calculate the distances, we must first standardize the data. As mentioned in the text, we will apply Min-Max normalization to the numerical variables as it tends to work better when both numerical and categorical predictors are involved. 

For the Age predictor, the Min value is 22 and the Range is (66-22) = 44.



For the Income predictor, the Min value is $23,886.72 and the Range is ($49,186.75 - $23,886.72) = $25300.03.

The standardized values for Age and Income are depicted in the table below.

Record 1 2 3 4 5 6 7 8 9 10

Age 0 (33-22)/44 = 0.2500 (28-22)/44 = 0.1364 (51-22)/44 = 0.6591 (25-22)/44 = 0.0682 (39-22)/44 = 0.3864 (54-22)/44 = 0.7273 (55-22)/44 = 0.7500 (50-22)/44 = 0.6364 1

Income ($46,156.98-$23,886.72)/$25300.03 = 0.8802 ($24,188.10-$23,886.72)/$25300.03 = 0.0119 ($28,787.34-$23,886.72)/$25300.03 = 0.1937 0 ($47,281.44-$23,886.72)/$25300.03 = 0.9247 ($33,994.90-$23,886.72)/$25300.03 = 0.3995 ($28,716.50-$23,886.72)/$25300.03 = 0.1909 1 ($46,726.50-$23,886.72)/$25300.03 = 0.9028 ($36,120.34-$23,886.72)/$25300.03 = 0.4835

Table 12.1. Standardized values for Age and Income

Now that we have standardized the numerical predictors, we can calculate the distances.

In addition to using the standardized Age and Income values, we also define the Marital distance as 0 when it has the same value as Record #10 (i.e. “Good Risk”) and 1 otherwise. The distance calculations are depicted in the table below.

Measurement D(1,10) D(2,10) D(3,10) D(4,10) D(5,10) D(6,10) D(7,10) D(8,10) D(9,10)

Distance Calculation

Table 12.2. Distance calculations for each record

6 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. Now since k=3, we need the records having the smallest distance from Record #10.

These are Records #9 (0.2006), #8 (0.1542), and #6 (0.6088), which all happen to be classified as “Good Risk”. We therefore conclude that the predicted class for Record #10 using KNN on this training set would be “Good Risk”.

13. Using the ClassifyRisk data set with predictors age, marital status, and income, and target variable risk, find the k-nearest neighbor for record #1, using k = 2 and Euclidean distance.

Using the SPSS Modeler, we separate the first record of the ClassifyRisk data set from the remainder of the records and train a KNN Model on the remainder. The model structure is depicted in the figure below.

Figure 13.1. KNN Model structure in SPSS Modeler

In the KNN node, we set the predictors as age, marital, and income, and set the target as risk.

The configuration is depicted below.

Figure 13.2. KNN Model – field configuration in SPSS Modeler

7 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. We then set k = 2 and leave Euclidean as the distance measure as depicted as below.

Figure 13.3. KNN Model – neighbor configuration in SPSS Modeler

After reconnecting the stream containing Record 1 to the model nugget and scoring the record, we observe that the model has classified Record 1 as a “bad loss”. We also observe that the two nearest neighbors are Records #3 (with distance of 0.048), and #57 (with distance of 0.050). The scoring results are depicted in the figure below.

Figure 13.4. KNN Model scoring results of the holdout record

8 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

14. Using the ClassifyRisk data set with predictors age, marital status, and income, and target variable risk, find the k-nearest neighbor for record #1, using k = 2 and Minkowski (city-block) distance (see Chapter 10).

Using the SPSS Modeler, we leverage the same model used in Exercise #13, but we change the distance calculation to City-Block as depicted in the figure below.

Figure 14.1. KNN Model – neighbor configuration using City-Block metric instead of Euclidean Distance

After rebuilding the KNN model and reconnecting the stream containing Record 1 to the model nugget and scoring the record, we observe that the model has classified Record 1 as a “bad loss”. We also observe that the two nearest neighbors are now Records #59 (with distance of 0.062), and #3 (with distance of 0.065). The scoring results are depicted in the figure below.

Figure 14.2. KNN Model scoring results using the City-Block metric

9 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D.

Solutions to Chapter 8 DECISION TREES Prepared by James Cunningham, Graduate Assistant 1. Describe the possible situations when no further splits can be made at a decision node.

When all branches of the decision tree terminate at pure leaf nodes, no additional splits are required. In this case, all records contained at each leaf node have the same target class value.

One or more branches may terminate at diverse leaf nodes. The algorithm will attempt to split the records leading to a pure leaf node; however, if all records contain the same predictor values, no additional splits are possible.

2. Suppose that our target variable is continuous numeric. Can we apply decision trees directly to classify it? How can we work around this?

To use a continuous target variable for classification, you must first discretize it. This process takes the range of values and coverts them into discrete categories.

3. True or false: Decision trees seek to form leaf nodes to maximize heterogeneity in each node.

False. Decision trees algorithms attempt to create leaf nodes that are as pure as possible.

This occurs when the leaf node contains the maximum number of records with the same target class value. This results in higher confidence, and increased classification accuracy.

1 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. 4. Discuss the benefits and drawbacks of a binary tree versus a bushier tree.

Consider the data in Table 8.11. The target variable is salary. Start by discretizing salary as follows: a. b. c. d.

Less than $35,000 Level 1 $35,000 to less than $45,000 Level 2 $45,000 to less than $55,000 Level 3 Above $55,000 Level 4

A binary tree is limited in its ability to split on categorical attributes, and may result in sub-optimal classification accuracy; however, a decision tree containing binary splits is often more easily interpretable.

The table below includes the additional field called “Level” for the discretized values of Salary.

Occupation Service Service Service Management Management Management Management Sales Sales Staff Staff

Gender Female Male Male Male Female Male Female Female Male Female Male

Age 45 25 33 25 35 26 45 40 30 50 25

Salary $48,000 $25,000 $35,000 $45,000 $65,000 $45,000 $70,000 $50,000 $40,000 $40,000 $25,000

Level 3 1 2 3 4 3 4 3 2 2 1

Table 4.1. Records with variable Salary discretized to Levels 1-4

TABLE 8.11 Decision tree data Occupation Gender Service Female Male Male Management Male Female Male Female Sales Female Male Staff Female Male

Age 45 25 33 25 35 26 45 40 30 50 25

Salary $48,000 $25,000 $35,000 $45,000 $65,000 $45,000 $70,000 $50,000 $40,000 $40,000 $25,00 2

Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. 5. Construct a classification and regression tree to classify salary based on the other variables. Do as much as you can by hand, before turning to the software.

The table below shows all possible candidate splits that could occur at the root note using predictor variables Occupation, Gender, and Age. Candidate 1 2 3 4 5 6 7 8 9 10 11

L Node Occupation = Service Occupation = Management Occupation = Sales Occupation = Staff Occupation = Service or Management Occupation = Service or Sales Occupation = Service or Staff Gender = Male Age <= 25 Age <= 35 Age <= 45

R Node Occupation = Management or Sales or Staff Occupation = Service or Sales or Staff Occupation = Service or Management or Staff Occupation = Service or Management or Sales Occupation = Sales or Staff Occupation = Management or Staff Occupation = Management or Sales Gender = Female Age > 25 Age > 35 Age > 45

Table 5.2. Candidate splits for CART algorithm using Occupation, Gender, and Age as predictors

3 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. Next, using the candidate splits defined in the table above, the optimality measure for each split is calculated.

Candidate

PL

PR

1

0.273

0.727

2

0.364

0.636

3

0.182

0.818

4

0.182

0.818

5

0.636

0.364

6

0.455

0.545

7

0.455

0.545

8

0.545

0.455

9

0.273

0.727

10

0.636

0.364

11

0.909

0.091

P( j | t L )

P( j | t R )

1: 0.333 2: 0.333 3: 0.333 4: 0 1: 0 2: 0 3: 0.50 4: 0.50 1: 0 2: 0.50 3: 0.50 4: 0 1: 0.50 2: 0.50 3: 0 4: 0 1: 0.143 2: 0.143 3: 0.429 4: 0.286 1: 0.20 2: 0.40 3: 0.40 4: 0 1: 0.40 2: 0.40 3: 0.20 4: 0 1: 0.333 2: 0.333 3: 0.333 4: 0 1: 0.667 2: 0 3: 0.333 4: 0 1: 0.286 2: 0.286 3: 0.286 4: 0.143 1: 0.20 2: 0.20 3: 0.40 4: 0.20

1: 0.125 2: 0.25 3: 0.375 4: 0.25 1: 0.286 2: 0.429 3: 0.286 4: 0 1: 0.222 2: 0.222 3: 0.333 4: 0.222 1: 0.111 2: 0.222 3: 0.444 4: 0.222 1: 0.25 2: 0.50 3: 0.25 4: 0 1: 0.167 2: 0.167 3: 0.333 4: 0.333 1: 0 2: 0.167 3: 0.50 4: 0.333 1: 0 2: 0.20 3: 0.40 4: 0.40 1: 0 2: 0.375 3: 0.375 4: 0.25 1: 0 2: 0.25 3: 0.50 4: 0.25 1: 0 2: 1 3: 0 4: 0

2 PL PR

Q(s | t )

( s | t )

0.397

0.583

0.231

0.463

1.429

0.662

0.298

0.889

0.265

0.298

1.333

0.397

0.463

0.929

0.430

0.496

0.666

0.330

0.496

1.266

0.628

0.496

0.933

0.463

0.397

1.334

0.530

0.463

0.643

0.298

0.165

1.6

0.264

Table 5.2. Optimality measures for each candidate split

According to the optimality measure, candidate 2 achieves the best split with a value of 0.662. This occurs at the split Occupation = Management (L) and Occupation = Service or Sales or Staff (R). The second best split is achieved using candidate 7 with an optimality measure equal to 0.628.

4 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. The records in Table 4.1 are placed in a comma-separated file that is read by SPSS Modeler. The decision tree produced by CART is shown next:

Figure 5.3. CART model generated by SPSS Modeler

The results generated by CART agree with the calculations performed by hand. That is, the root-level split occurs where Occupation = Management, or Occupation = {Sales, Service, Staff}.

6. Construct a C4.5 decision tree to classify salary based on the other variables. Do as much as you can by hand, before turning to the software.

For C4.5, the list of candidate splits for the root node are shown below.

Candidate 1

Attribute = Value Occupation

2

Gender

3

Age <= 25

4

Age <= 35

5

Age <= 45

Branches Occupation = Service Occupation = Management Occupation = Sales Occupation = Staff Gender = Male Gender = Female Age <= 25 Age > 25 Age <= 35 Age > 35 Age <= 45 Age > 45

Table 6.4. Candidate splits for C4.5 algorithm using Occupation, Gender, and Age as predictors

5 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. Using the records defined in Table 4.1, the Information Gain for the 5 candidate splits is calculated. First, however, the Entropy is calculated for the data set before splitting occurs.

Entropy Calculation:

2 3 4 2 , PLevel2  , PLevel3  , PLevel4  11 11 11 11 2 2 3 3 4 4 2 2 H (T )   log 2    log 2    log 2    log 2    1.936 11  11  11  11  11  11  11  11  PLevel1 

Candidate 1:

3 4 2 2 , PManagem ent  , PSales  , PStaff  11 11 11 11 1 1 1 1 1 1 0 0 H Occupation( Service)   log 2    log 2    log 2    log 2    1.585 3  3 3 3 3 3 3 3 0 0 0 0 2 2 2 2 H Occupation( Management)   log 2    log 2    log 2    log 2    1.0 4 4 4 4 4 4 4 4 0 0 1 1 1 1 0 0 H Occupation( Sales)   log 2    log 2    log 2    log 2    1.0 2 2 2 2 2 2 2 2 1 1 1 1 0 0 0 0 H Occupation( Staff )   log 2    log 2    log 2    log 2    1.0 2 2 2 2 2 2 2 2 PService 

3 4 2 2 (1.585)  (1.0)  (1.0)  (1.0)  1.160 11 11 11 11 1.936  1.160  0.776 bits H Occupation(T ) 

6 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. Candidate 2:

6 5 , PFemale  11 11 2 2 2 2 2 2 0 0 H Gender ( Male)   log2    log2    log2    log2    1.585 6 6 6 6 6 6 6 6 PMale 

0 0 1 1 2 2 2 2 H Gender ( Female)   log2    log2    log2    log2    1.522 5 5 5 5 5 5 5 5 6 5 (1.585)  (1.522)  1.556 11 11 1.936  1.556  0.38 bits H Gender (T ) 

Candidate 3:

3 8 , PAge  25  11 11 2 2 0 0 1 1 0 0 H Age ( 25)   log2    log2    log2    log2    0.918 3 3 3 3 3  3 3 3 0 0 3  3 2 2 2 2 H Age ( 25)   log2    log2    log2    log2    1.531 8 8 8 8 8 8 8 8 PAge  25 

3 8 (0.918)  (1.531)  1.364 11 11 1.936  1.364  0.572 bits H Gender (T ) 

7 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. Candidate 4:

7 4 , PAge  35  11 11 2 2 2 2 2 2 1 1 H Age ( 35)   log 2    log 2    log 2    log 2    1.950 7 7 7 7 7 7 7 7 0 0 1 1 2 2 1 1 H Age ( 35)   log 2    log 2    log 2    log 2    1.5 4 4 4 4 4 4 4 4 PAge  35 

7 4 (1.950)  (1.5)  1.786 11 11 1.936  1.786  0.15 bits H Age (T ) 

Candidate 5:

10 1 , PAge 45  11 11 2 2 2 2 4 4 2 2 H Age ( 35)   log 2    log 2    log 2    log 2    1.922 10  10  10  10  10  10  10  10  0 0 1 1 0 0 0 0 H Age ( 35)   log 2    log 2    log 2    log 2    0 1 1 1 1 1 1 1 1 PAge 45 

10 1 (1.922)  (0)  1.747 11 11 1.936  1.747  0.189 bits H Age (T ) 

All five candidate splits are evaluated and Information Gain is maximized to 0.776 bits when splitting on the Occupation attribute at the root-level node, according to the calculations performed above.

Now, the records are classified using the C5.0 algorithm in Clementine. Results, using default options, are shown below:

8 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D.

Figure 6.5. C5.0 model generated by SPSS Modeler

C5.0 splits on the Gender attribute at the root level. Interestingly, these results do not agree with the calculations performed manually, above. Perhaps, one or more options such as setting the “Minimum records per child branch” must have a non-default value specified.

7. Compare the two decision trees and discuss the benefits and drawbacks of each.

C4.5 has the benefit that it is able to create branches for each categorical attribute value at a decision node split. Depending on the characteristics of the training set, this feature may likely improve classification accuracy. Conversely, this functionality may also suffer from the drawback of creating overly “bushy” trees. CART is limited in its ability to only perform binary splits, which may result in sub-optimal classification accuracy, as compared to results produced by C4.5.

Both decision trees have the benefit of being extremely efficient, while also producing output that is easily interpretable by an analyst. In this way, it is easier to learn what attributes are most important when the algorithm is applied to new domains.

9 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D.

8. Generate the full set of decision rules for the CART decision tree.

The decision rules, as generated by CART, are shown below: Rules for 1 - contains 1 rule(s) Rule 1 for 1 (2, 1.0) if Occupation in [ "Sales" "Service" "Staff" ] and Age < 27.5 then 1 Rules for 2 - contains 2 rule(s) Rule 1 for 2 (1, 1.0) if Occupation in [ "Sales" "Service" "Staff" ] and Age >= 27.5 and Gender in [ "Female" ] and Occupation in [ "Staff" ] then 2 Rule 2 for 2 (2, 1.0) if Occupation in [ "Sales" "Service" "Staff" ] and Age >= 27.5 and Gender in [ "Male" ] then 2 Rules for 3 - contains 2 rule(s) Rule 1 for 3 (2, 1.0) if Occupation in [ "Management" ] and Gender in [ "Male" ] then 3 Rule 2 for 3 (2, 1.0) if Occupation in [ "Sales" "Service" "Staff" ] and Age >= 27.5 and Gender in [ "Female" ] and Occupation in [ "Sales" "Service" ] then 3 Rules for 4 - contains 1 rule(s) Rule 1 for 4 (2, 1.0) if Occupation in [ "Management" ] and Gender in [ "Female" ] then 4 Default: 3

10 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. 9. Generate the full set of decision rules for the C4.5 decision tree.

The decision rules below are generated using C5.0, in Clementine: Rules for 1 - contains 1 rule(s) Rule 1 for 1 (4, 0.5) if Gender = Male and Age <= 26 then 1 Rules for 2 - contains 1 rule(s) Rule 1 for 2 (2, 1.0) if Gender = Male and Age > 26 then 2 Rules for 3 - contains 1 rule(s) Rule 1 for 3 (5, 0.4) if Gender = Female then 3 Default: 3

10. Compare the two sets of decision rules and discuss the benefits and drawbacks of each.

This time CART grows a slightly more complex tree, as compared to C5.0. All rules in CART result in 100% confidence, while C5.0 has two rules with 40% and 50% confidence. Most likely, if the pruning or records per child node options are further analyzed, a tree of greater depth and higher accuracy will be created.

HANDS-ON ANALYSIS 11 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. For the following exercises, use the churn data set available at the book series website.

Normalize the numerical data and deal with the correlated variables.

11. Generate a CART decision tree.

A CART Node specifies “Churn?” as the target variable, and includes the 11 numeric and 2 categorical fields are inputs fields. All default options are used by CART to generate the tree shown in the figure below.

Figure 11.1. CART decision tree

According to CART, day_charge_MM is the most important variable. The second most important variables include custserv_calls_MM and vmail_messages_MM that occur at the second level split.

12. Generate a C4.5-type decision tree.

12 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. SPSS Modeler is used to generate a C5.0 decision tree, using default options, predicting “Churn?” in Figure 12. Again, 11 numeric and 2 categorical variables are used as input.

Note that most, but not all, of the tree is shown in the figure below.

Figure 12.1. C5.0 decision tree

13. Compare the two decision trees and discuss the benefits and drawbacks of each.

13 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. Both algorithms split on day_charge_MM at the root node, and consider it the most important variable in the training set. In addition, each of the algorithms split on custserv_calls_MM and the second level, and then Intl Plan at the third level, in the branch where day_charge_MM <= 0.7.

In general, the C5.0 tree is more complex than the CART tree. For example, C5.0 reaches a depth equal to 8, whereas CART is a maximum of 5 levels deep.

14. Generate the full set of decision rules for the CART decision tree.

The rules below are generated from the CART Node by choosing “Generate > Rule Set...”.

Rules for False. - contains 7 rule(s) Rule 1 for False. (2,248, 0.97) if day_charge_MM < 0.739363465218596 and custserv_calls_MM < 0.436507936507937 and Intl Plan in [ "no" ] and day_charge_MM < 0.640885436073229 then False.

Rule 2 for False. (223, 0.924) if day_charge_MM < 0.739363465218596 and custserv_calls_MM < 0.436507936507937 and Intl Plan in [ "no" ] and day_charge_MM >= 0.640885436073229 and eve_charge_MM < 0.653965615072128 then False.

Rule 3 for False. (168, 0.952) if day_charge_MM < 0.739363465218596 and custserv_calls_MM < 0.436507936507937 and Intl Plan in [ "yes" ] and intl_calls_MM >= 0.127631578947368 and intl_charge_MM < 0.65462962962963 then False.

Rule 4 for False. (64, 0.547) if day_charge_MM < 0.739363465218596 and custserv_calls_MM >= 0.436507936507937 and day_charge_MM < 0.454628025553579 and custserv_calls_MM >= 0.585714285714286 then False.

Rule 5 for False. (165, 0.812) if day_charge_MM < 0.739363465218596 and custserv_calls_MM >= 0.436507936507937 and day_charge_MM >= 0.454628025553579 then False.

Rule 6 for False. (47, 0.681) if day_charge_MM >= 0.739363465218596 and vmail_message_MM < 0.127450980392157 and eve_charge_MM < 0.448220640569395 then False.

Rule 7 for False. (53, 0.887) if day_charge_MM >= 0.739363465218596 and vmail_message_MM >= 0.127450980392157 then False.

Rules for True. - contains 5 rule(s) Rule 1 for True. (66, 0.576) if day_charge_MM < 0.739363465218596 and custserv_calls_MM < 0.436507936507937 and Intl Plan in [ "no" ] and day_charge_MM >= 0.640885436073229 and eve_charge_MM >= 0.653965615072128 then True.

14 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. Rule 2 for True. (51, 1.0) if day_charge_MM < 0.739363465218596 and custserv_calls_MM < 0.436507936507937 and Intl Plan in [ "yes" ] and intl_calls_MM < 0.127631578947368 then True.

Rule 3 for True. (43, 0.977) if day_charge_MM < 0.739363465218596 and custserv_calls_MM < 0.436507936507937 and Intl Plan in [ "yes" ] and intl_calls_MM >= 0.127631578947368 and intl_charge_MM >= 0.65462962962963 then True.

Rule 4 for True. (79, 0.848) if day_charge_MM < 0.739363465218596 and custserv_calls_MM >= 0.436507936507937 and day_charge_MM < 0.454628025553579 and custserv_calls_MM < 0.585714285714286 then True.

Rule 5 for True. (126, 0.881) if day_charge_MM >= 0.739363465218596 and vmail_message_MM < 0.127450980392157 and eve_charge_MM >= 0.448220640569395 then True.

Default: False.

15. Generate the full set of decision rules for the C4.5 decision tree.

Rules for False. - contains 12 rule(s) Rule 1 for False. (155, 0.974) 15 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. if day_charge_MM <= 0.70022601 and custserv_calls_MM <= 0.42857099 and Intl Plan = yes and intl_calls_MM > 0.117647 and intl_charge_MM <= 0.65393299 then False.

Rule 2 for False. (2,147.61, 0.973) if day_charge_MM <= 0.70022601 and custserv_calls_MM <= 0.42857099 and Intl Plan = no and day_charge_MM <= 0.618545 then False.

Rule 3 for False. (251.083, 0.956) if day_charge_MM <= 0.70022601 and custserv_calls_MM <= 0.42857099 and Intl Plan = no and day_charge_MM > 0.618545 and eve_charge_MM <= 0.66152698 then False.

Rule 4 for False. (16.022, 1.0) if day_charge_MM <= 0.70022601 and custserv_calls_MM <= 0.42857099 and Intl Plan = no and day_charge_MM > 0.618545 and eve_charge_MM > 0.66152698 and VMail Plan = yes then False.

Rule 5 for False. (11, 0.909) if day_charge_MM <= 0.70022601 and custserv_calls_MM <= 0.42857099 and Intl Plan = no and day_charge_MM > 0.618545 and eve_charge_MM > 0.66152698 and VMail Plan = no and night_charge_MM <= 0.415546 then False.

Rule 6 for False. (25.025, 0.84) if day_charge_MM <= 0.70022601 and custserv_calls_MM > 0.42857099 and day_charge_MM <= 0.45453 and custserv_calls_MM > 0.555556 and custserv_calls_MM <= 0.60000002 then False.

Rule 7 for False. (144.043, 0.861) if day_charge_MM <= 0.70022601 and custserv_calls_MM > 0.42857099 and day_charge_MM > 0.45453 and eve_charge_MM > 0.31901601 then False.

Rule 8 for False. (3, 1.0) if day_charge_MM > 0.70022601 and VMail Plan = yes and Intl Plan = yes and vmail_message_MM > 0.77083302 then False.

Rule 9 for False. (69.098, 0.971) if day_charge_MM > 0.70022601 and VMail Plan = yes and Intl Plan = no then False.

Rule 10 for False. (69, 0.855) if day_charge_MM > 0.70022601 and VMail Plan = no and eve_charge_MM <= 0.493691 and day_charge_MM <= 0.79076999 then False.

Rule 11 for False. (7, 0.857) if day_charge_MM > 0.70022601 and VMail Plan = no and eve_charge_MM <= 0.493691 and day_charge_MM > 0.79076999 and eve_charge_MM <= 0.34163699 then False.

Rule 12 for False. (30, 0.633) if day_charge_MM > 0.70022601 and VMail Plan = no and eve_charge_MM > 0.493691 and night_charge_MM <= 0.39987999 and day_charge_MM <= 0.79076999 then False.

Rules for True. - contains 10 rule(s) Rule 1 for True. (46, 1.0) if day_charge_MM <= 0.70022601 and custserv_calls_MM <= 0.42857099 and Intl Plan = yes and intl_calls_MM <= 0.117647 then True.

Rule 2 for True. (40, 1.0) if day_charge_MM <= 0.70022601 and custserv_calls_MM <= 0.42857099 and Intl Plan = yes and intl_calls_MM > 0.117647 and intl_charge_MM > 0.65393299 then True.

Rule 3 for True. (40, 0.875) 16 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. if day_charge_MM <= 0.70022601 and custserv_calls_MM <= 0.42857099 and Intl Plan = no and day_charge_MM > 0.618545 and eve_charge_MM > 0.66152698 and VMail Plan = no and night_charge_MM > 0.415546 then True.

Rule 4 for True. (73.073, 0.862) if day_charge_MM <= 0.70022601 and custserv_calls_MM > 0.42857099 and day_charge_MM <= 0.45453 and custserv_calls_MM <= 0.555556 then True.

Rule 5 for True. (44.044, 0.658) if day_charge_MM <= 0.70022601 and custserv_calls_MM > 0.42857099 and day_charge_MM <= 0.45453 and custserv_calls_MM > 0.555556 and custserv_calls_MM > 0.60000002 then True.

Rule 6 for True. (12.004, 0.833) if day_charge_MM <= 0.70022601 and custserv_calls_MM > 0.42857099 and day_charge_MM > 0.45453 and eve_charge_MM <= 0.31901601 then True.

Rule 7 for True. (8, 0.75) if day_charge_MM > 0.70022601 and VMail Plan = yes and Intl Plan = yes and vmail_message_MM <= 0.77083302 then True.

Rule 8 for True. (23, 0.87) if day_charge_MM > 0.70022601 and VMail Plan = no and eve_charge_MM <= 0.493691 and day_charge_MM > 0.79076999 and eve_charge_MM > 0.34163699 then True.

Rule 9 for True. (13, 1.0) if day_charge_MM > 0.70022601 and VMail Plan = no and eve_charge_MM > 0.493691 and night_charge_MM <= 0.39987999 and day_charge_MM > 0.79076999 then True.

Rule 10 for True. (106, 0.934) if day_charge_MM > 0.70022601 and VMail Plan = no and eve_charge_MM > 0.493691 and night_charge_MM > 0.39987999 then True.

Default: False.

16. Compare the two sets of decision rules and discuss the benefits and drawbacks of each.

12 decision rules are generated from the CART tree, and 22 rules are produced from C5.0. Thus, the number of rules produced by C5.0 is almost twice that of CART. This is 17 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. not surprising, given the general complexity of the C5.0 tree observed in Problem 12.

In general, this suggests that the C5.0 may be overfitting the data somewhat. For example, for CART the minimum number of records applicable to a rule is 43. In contrast, C5.0 has rules that apply to only 8 (Rule 7 True), 3 (Rule 8 False), and 7 (Rule 11 False) records. This shows that the rules produced by CART more easily interpreted and understood, compared to those generated by C5.0.

Both algorithms found an almost identical rule that applies to about ~2,000 records, with 97% confidence, and a classification of “False.”

CART Rule 1 for False. (2,248, 0.97) if day_charge_MM < 0.739363465218596 and custserv_calls_MM < 0.436507936507937 and Intl Plan in [ "no" ] and day_charge_MM < 0.640885436073229 then False.

C5.0 Rule 2 for False. (2,147.61, 0.973) if day_charge_MM <= 0.70022601 and custserv_calls_MM <= 0.42857099 and Intl Plan = no and day_charge_MM <= 0.618545 then False.

18 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D.

Solutions to Chapter 9 NEURAL NETWORKS Prepared by James Cunningham, Graduate Assistant 1. Suppose that you need to prepare the data in figure 9.8 of section 9.10 (note that the text says Table 6.10) for a neural network algorithm. Define the indicator variables for the occupation attribute.

The occupation attribute has four classes as follows:    

Government Self-Employed Private Missing

Since neural networks take numerical inputs only and occupation is categorical, each class is encoded as a binary indicator variable where the values for each observation are mutually exclusive. In other words, for an Adult observation having an occupation of Private, the Private indicator variable would be set to 1 and the other three would be set to 0. The approach is depicted in the table below.

Observation A B C

Occupation Class Private Government Self-Employed

Government 0 1 0

SelfEmployed 0 0 1

Private

Missing

1 0 0

0 0 0

Table 1.1. Indicator variable expansion for the Occupation predictor

2. Clearly describe each of these characteristics of a neural network: a. Layered The neural network is composed to two or more layers of nodes and typically includes an input layer, one or more hidden layers, and an output layer. b. Feedforward This characteristic implies that information flows in a single direction and does not loop or cycle. c. Completely connected Each node in a given layer connects to all nodes in the next layer.

1 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. 3. What is the sole function of the nodes in the input layer?

The nodes in the input layer accept the data from the input variables. The data values are passed directly to nodes in the hidden layer.

4. Should we prefer a large hidden layer or a small one? Describe the benefits and drawbacks of each.

As the number of nodes in the hidden layer becomes overly large, the neural network becomes increasingly likely to overfit the data. In contrast, using too few hidden nodes may lead to poor accuracy. Therefore, a balance must be struck when configuring the hidden layer that strives to deliver the best accuracy while also generalizing to the data set.

5. Describe how neural networks function nonlinearly.

A combination function computes a scalar value from a linear combination of node inputs and connection weights. This scalar value is input to the activation function (sigmoid function), which produces an output that is not necessarily linearly-related to its input.

6. Explain why the updating term for the current weight includes the negative of the sign of the derivative (slope).

The gradient decent method is used to determine the direction (negative or positive) that each of the weights for the vector w should be adjusted, together, to decrease SSE. Because each partial derivative

SSE is the slope of the SSE curve at wi , we negate the wi *

sign of slope, such that adjusting each wi moves it towards the optimal value w i , the value that minimizes SSE.

2 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. 7. Adjust the weights W0B, W1B, W2B, and W3B from the example on backpropagation in the text.

We are given that  B  0.0011 and  0.1, therefore For W1B we get

W1B   B x1  (0.1)(0.0011)(0.4)  0.000044

W1B,NEW  W1B,CURRENT  W1B  0.9  0.000044  0.899956

For W2 B we get

W2 B   B x2  (0.1)(0.0011)(0.2)  0.000022

W2 B,NEW  W2 B,CURRENT  W2 B  0.8  0.000022  0.799978

For W3 B we get

W3 B   B x3  (0.1)(0.0011)(0.7)  0.000077

W3B,NEW  W3B,CURRENT  W3B  0.4  0.000077  0.399923

For W0 B we get

W0 B   B x0  (0.1)(0.0011)(1)  0.00011

W0 B,NEW  W0 B,CURRENT  W0 B  0.7  0.00011  0.69989

3 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. 8. Refer to Exercise 7. Show that the adjusted weights result in a smaller prediction error.

Now that Exercise 7 is complete we have all the adjusted weights after back propagation has occurred and so we compute a new predicted valued:

net A  W0 A (1)  W1 A x1  W2 A x2  W3 A x3

(0.499877 )(1)(0.5999508 )(0.4)  (0.7999754 )(0.2)  (0.5999139 )(0.7)  1.3198 1 f (net A )   0.7891 1  e 1.3198 net B  W0 B (1)  W1B x1  W2 B x2  W3 B x3

(0.69989 )(1)(0.899956 )(0.4)  (0.799978 )(0.2)  (0.399923)(0.7)  1.4998 1 f (net A )   0.8175 1  e 1.4998 net Z  W0 Z (1)  WAZ x AZ  WBZ xBZ

(0.49918)(1)(0.899353)(0.7891)  (0.89933)(0.8175)  1.9441 1 f (netZ )   0.8745 1  e 1.9441 Now, the prediction error = actual – output = 0.8 – 0.8745 = –0.0745. Recall that the initial prediction, before back propagation was 0.8 – 0.8750 = –0.075. Thus, the prediction error after back propagation has indeed decreased.

9. True or false: Neural networks are valuable because of their capacity for always finding the global minimum of the SSE. A neural network model is not guaranteed to find the global minimum of the SSE; however, the algorithm will find a local minimum that often represents a good solution to the problem at hand. Learning rate and momentum parameters are often adjusted to search local minima for near-optimal solutions.

4 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. 10. Describe the benefits and drawbacks of using large or small values for the learning rate.

Choosing very small values for the learning rate will tend to cause the algorithm to take an unacceptably long time to converge on a solution. Conversely, setting the learning rate to too large a value may cause the neural network oscillate and have trouble converging a solution. Ideally, the neural network should begin by using a larger value for the learning rate, and gradually decrease the value as the algorithm approaches a solution.

11. Describe the benefits and drawbacks of using large or small values for the momentum term.

Using small values for momentum may cause the algorithm to converge prematurely to a non-optimal local minimum from which it doesn’t have enough inertia to escape. In contrast, large values chosen for momentum may cause it to have so much inertia that it overshoots an optimal solution too easily.

5 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. HANDS-ON ANALYSIS For Exercises 12–14, use the Churn data set. Normalize the numerical data, recode the categorical variables, and deal with the correlated variables.

The Churn data set has twenty candidate predictor fields. The table below identifies the type of each field. Please note that the Phone field will be omitted from any modeling due to the fact that it is a unique identifier and therefore has no predictive power.

Field Account Length Area Code CustServ Calls Day Calls Day Charge Day Mins Eve Calls Eve Charge Eve Mins Intl Calls Intl Charge Intl Mins Intl Plan Night Calls Night Charge Night Mins Phone State VMail Message VMail Plan

Type Numeric Categorical Numeric Numeric Numeric Numeric Numeric Numeric Numeric Numeric Numeric Numeric Dichotomous Categorical Numeric Numeric Numeric Unique Identifier Categorical Numeric Dichotomous Categorical

Table D.1. Predictors in the Churn data set

There are 15 Numeric fields, and these were explored for their correlations. The table below depicts the correlation matrix. We observe that Day Mins and Day Charge are perfectly correlated, Eve Mins and Eve Charge are perfectly correlated, Intl Mins and Intl Charge are perflectly correlated, and Night Mins and Night Charge are perfectly correlated (as highlighted in yellow). Therefore, we will want to omit one field from each of these four pairs from the modeling phase. Since the Mins fields have a wider range than their corresponding Charge fields, we will retain the Mins fields and omit the Charge fields as the Mins fields should be less influenced by extreme values.

6 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. Intl Charge

Intl Mins

Night Charge

Night Mins

0.007

0.006

-0.013

0.007

1

Eve Calls

0.019

0.002

0.006

0.016

0.016

Eve Charge Eve Mins

-0.007

-0.013

-0.021

0.007

0.007

-0.011

-0.007

-0.013

-0.021

0.007

0.007

-0.011

1

Intl Calls

0.021

-0.018

0.005

0.008

0.008

0.017

0.003

0.003

Intl Charge Intl Mins Night Calls Night Charge Night Mins Vmail Message

0.01

-0.01

0.022

-0.01

-0.01

0.009

-0.011

-0.011

0.032

0.01

-0.01

0.022

-0.01

-0.01

0.009

-0.011

-0.011

0.032

1

-0.013

-0.013

-0.02

0.023

0.023

0.008

0.008

0.008

0

-0.014

-0.014

-0.009

-0.009

0.023

0.004

0.004

-0.002

-0.013

-0.013

-0.012

-0.015

-0.015

0.011

-0.009

-0.009

0.023

0.004

0.004

-0.002

-0.013

-0.013

-0.012

-0.015

-0.015

0.011

1

-0.005

-0.013

-0.01

0.001

0.001

-0.006

0.018

0.018

0.014

0.003

0.003

0.007

0.008

0.008

Night Calls

Eve Mins

-0.013

Intl Calls

Eve Charge

0.006

Eve Calls

Day Mins

-0.019

Day Calls

Day Charge

CustServ Calls

Account Length 0.038

Account Length CustServ Calls Day Calls Day Charge Day Mins

-0.004

Table D.2. Correlation matrix for Churn predictors

7 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. There are two Categorical fields. The Area Code field has three classes (408, 415, and 510), and the State field has 51 classes (i.e. all 50 states and the District of Columbia). However, exploratory analysis in the form of a Cross-tabulation identifies that Area Code is definitely anomalous as (1) there are only three area codes, which all belong to California yet they are distributed across all states and DC, and (2) the Chi-Square p-value is 0.608 indicating that the area codes are distributed approximately uniformly across the state classes (i.e. there is no evidence against the null hypothesis that the observed approximate uniform distribution is indicative of the underlying population, and we know for a fact that there should be!). The cross-tabulation showing a partial view of the state classes and the Chi-Square p-value is depicted in the figure below. We will therefore omit the Area Code field from the modeling phase. Also, since State and Area Code to not make sense when taken together, we will omit the State field as well.

Figure D.1. Cross-tabulation of State and Area Code predictors

8 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. 12. Generate a neural network model for classifying churn based on the other variables.

Describe the topology of the model.

We construct a neural network model for classifying churn based on the remaining predictor variables. The variables used for modeling are depicted in the table below along with the treatment applied in order to construct a neural network model to predict the target churn. Numeric fields will be Min-Max normalized, Categorical fields will be recoded into indicator variables per their respective class sets, and DichotomousCategorical fields are translated into Binary values (i.e. 1 or 0).

Field Account Length CustServ Calls Day Calls Day Mins Eve Calls Eve Mins Intl Calls Intl Mins Intl Plan Night Calls Night Mins VMail Message VMail Plan

Type Numeric Numeric Numeric Numeric Numeric Numeric Numeric Numeric Dichotomous Categorical Numeric Numeric Numeric Dichotomous Categorical

Treatment Min-Max Normalized Min-Max Normalized Min-Max Normalized Min-Max Normalized Min-Max Normalized Min-Max Normalized Min-Max Normalized Min-Max Normalized Translated to Binary (1/0) Min-Max Normalized Min-Max Normalized Min-Max Normalized Translated to Binary (1/0)

Table 12.1. Indicator variable expansion for the Occupation predictor

9 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. Using SPSS Modeler, we apply Min-Max normalization to the Numeric variables by deriving new variables using a Set Globals node to determine the global minimum and maximum values for each of these variables. A sample of the approach is depicted in the Derive Node for the Account Length variable depicted below.

Figure 12.1. Sample of Min-Max normalization formuations

The complete structure depicting employment of Set Globals and all transformed variables are depicted in the figure below.

Figure 12.2. Model structure for variable transformations and global variable calculations

10 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. The resulting neural network model topology is depicted in the figure below. The model is a multilayer perceptron with an (1) input layer containing 14 nodes (13 predictor nodes and one additional bias/constant node), (2) a single hidden layer containing 7 nodes (6 artificial neurons and one bias/constant node), and (3) an output layer containing a single output node for our target variable.

Figure 12.3. Neural Network generated by SPSS Modeler

13. Which variables, in order of importance, are identified as most important for classifying churn?

The model provides insight into predictor importance. The predictors are listed from top to bottom in order of their importance in the figure below. DayMins is the most important predictor of churn, and VMailPlan is the least important predictor.

Figure 13.1. Ordering by predictor importance

11 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. 14. Compare the neural network model with the CART and C4.5 models for this task in Chapter 6. Describe the benefits and drawbacks of the neural network model compared to the others. Is there convergence or divergence of results among the models?

Clearly, a drawback of using Neural Networks is that their solution lacks interpretability by an analyst. In addition, Neural Networks are much slower as compared to decision trees when computing a solution.

Recall that CART and C5.0 split on day_charge_MM at the root node, and considered it the most important variable in the training set. In addition, each of the algorithms split on custserv_calls_MM and the second level, and then Intl Plan at the third level, in the branch where day_charge_MM <= 0.7. In Problem 13 we see that CustServ Calls, Day Charge, and Intl Plan are similarly important predictors to the Neural Network classification model.

12 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. For Exercises 15–17, use the ClassifyRisk data set 15. Run an NN model predicting income based only on age. Use the default settings and make sure there is one hidden layer with one neuron.

We construct an NN predicting Income based on Age only using default settings. First we apply Min-Max normalization to the Age variable by using a Set Globals node and deriving new variables AgeMM and IncomeMM. The technique is depicted for Age in the figure below.

Figure 15.1. Sample Min-Max normalization formulation using the Age predictor

13 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. The complete structure depicting employment of Set Globals and the transformed AgeMM variable are depicted in the figure below.

Figure 15.2. Model structure for variable transformations and global variable calculations

The resulting neural network model topology is depicted in the figure below. We observe that the model has a single hidden layer containing one Neuron.

Figure 15.3. Neural Network generated by SPSS Modeler

14 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. 16. Consider the following quantity: (weight for Age-to-Neuron1) + (weight for Bias-toNeuron1) ∗ (weight for Neuron 1-to-Output node). Explain whether this makes sense, given the data, and why.

The NN model is depicted this time including the synaptic weights in the figure below.

Figure 16.1. Neural Network generated by SPSS Modeler with synaptic weights

The prescribed quantity is calculated as follows: Quantity = (-0.977) + (0.108) * (-0.703) = -1.0529.

15 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D.

17. Make sure the target variable takes the flag type. Compare the sign of (weight for Age- to-Neuron1) + (weight for Bias-to-Neuron1) ∗ (weight for Neuron 1-to-Output node) for the good risk output node, as compared to the bad loss output node.

Explain whether this makes sense, given the data, and why.

The target variable risk is Dichotomous Categorical, so we ensure that it takes the binary flag type by deriving a new field RiskBinary, which takes the value 1 when Risk is “good risk” and 0 otherwise. The revised structure is depicted in the figure below.

Figure 17.1. Revised model structure for variable transformations and global variable calculations

16 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. The resulting model with binary target is depicted below with synaptic weights.

Figure 16.1. Revised Neural Network generated by SPSS Modeler with synaptic weights

The prescribed quantities are calculated as follows: QuantityBadLoss0 = (0.460) + (0.177) * (-0.651) = 0.3448.

QuantityGoodRisk1 = (0.460) + (0.177) * (0.399) = 0.5306.

IBM/SPSS Modeler Analysis. For Exercises 18–19, use the nn1 data set.

17 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D.

18. Set your neural network build options as follows: Use a Multilayer Perceptron and customize number of units in Hidden Layer 1 to be 1 and Hidden Layer 2 to be 0.

For Stopping Rules, select ONLY Customize number of maximum training cycles.

Start at 1 and go to about 20. For Advanced, deselect Replicate Results.

Using SPSS Modeler, we construct the Neural Network as prescribed. The configuration instructions are depicted in the figure below.

Figure 18.1. Neural Network implementing prescribed configuration instructions

The structure used to construct the model is depicted in the figure below.

18 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D.

Figure 18.2. Model structure using prescribed instructions

19. Browse your model. In the Network window of the Model tab, select the Style: Coefficients. Record the Pred1-to-Neuron1 weight and the Pred2-to-Neuron1 weight for each run. Describe the behavior of these weights. Explain why this is happening.

We browse the model, display the synaptic weights, and record the weights for Pre11 to Neuron1 and Pred2 to Neuron 1 as directed. The model is depicted in the figure below.

Figure 19.1. Neural Network generated by SPSS Modeler with synaptic weights

Referring to the figure above, we observe that the weights for the two predictor input nodes are approximately equal in magnitude but opposite in direction. This is happening 19 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D.& Chantal D. Larose, Ph.D. because the predictor data appears to be inversely proportional. For example, when the value for Pred1 is small, the value for Pred2 is large, and when value for Pred1 is large, the value for Pred2 is small. Therefore, they have approximately equal yet opposite influence on the target response. The small toy data set is depicted in the figure below.

Figure 19.2. Toy data set

20 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Solutions to Chapter 10 HIERARCHICAL AND k-MEANS CLUSTERING Prepared by James Cunningham, Graduate Assistant 1. To which cluster for the 90210 zip code would you prefer to belong?

Most Data Miners would most likely want to identify with Cluster # 07: Money and Brains.

2. Describe the goal of all clustering methods.

The primary objective of Clustering is to divide the entire data set into relatively homogeneous subgroups or clusters such that similarity of observations within a cluster is maximized and at the same time similarity of observations between clusters is minimized.

3. Suppose that we have the following data (one variable). Use single linkage to identify the clusters. Data: 0 0 1 3 3 6 7 9 10 10 The agglomerative clustering method that starts by placing each observation into its own sub-cluster, and then progressively agglomerates each observation based on the desired linkage rule. The single-linkage approach will agglomerate the observation or sub-cluster in question to the sub-cluster having the shortest distance to any observation in another sub-cluster.

The steps are described below as follows:  Step 1 – All ten observations are placed in their own clusters. 

Steps 2, 3, and 4 – The algorithm searches the distance matrix for the closest two clusters to merge. There are actually three candidates {0, 0}, {3, 3}, and {10, 10} all having a distance of zero. The algorithm selects them in some arbitrary order.

At the end of Step 4, the clusters are then formed as {0, 0}, {1}, {3, 3}, {6}, {7}, {9}, and {10, 10}.



Steps 5, 6, and 7 – The algorithm searches the distance matrix for the closest two clusters to merge and there are actually three candidates {0, 0, 1}, {6, 7}, and {9, 10, 10} all having a distance of 1. The algorithm selects them in some arbitrary order. At the end of Step 4, the clusters are then formed as {0, 0, 1}, {3, 3}, {6, 7}, and {9, 10, 10}.



Steps 8 and 9 – The algorithm searches the distance matrix for the closest two clusters to merge. There are two candidates {0,0,1,3,3} and {6,7,9,10,10} each having a distance of 2. The algorithm selects them in some arbitrary order. At the end of Step 9, the clusters are then formed as {0,0,1,3,3} and {6,7,9,10,10}.

1 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 

Step 10 – The algorithm searches the distance matrix for the closest two clusters to merge. Since there are only two clusters, the algorithm merges them into a single cluster {0, 0, 1, 3, 3, 6, 7, 9, 10, 10} and then ends.

4. Suppose that we have the following data (one variable). Use complete linkage to identify the clusters. Data: 0 0 1 3 3 6 7 9 10 10 The agglomerative clustering method that starts by placing each observation into its own cluster, and then progressively agglomerates each observation based on the desired linkage rule. The complete-linkage approach will agglomerate the observation or subcluster in question to the sub-cluster having the shortest distance to the most distant observation in another sub-cluster.

The steps are described below as follows:  Step 1 - All ten observations are placed in their own clusters. 

Steps 2, 3, and 4 – The algorithm searches the distance matrix for the closest two clusters to merge. There are actually three candidates {0, 0}, {3, 3}, and {10, 10} all having a distance of zero. The algorithm selects them in some arbitrary order.

At the end of Step 4, the clusters are then formed as {0, 0}, {1}, {3, 3}, {6}, {7}, {9}, and {10, 10}.



Steps 5, 6, and 7 – The algorithm searches the distance matrix for the closest two clusters to merge based on the most distant points in each cluster and there are actually three candidates {0, 0, 1}, {6, 7}, and {9, 10, 10} all having a distance of 1. The algorithm selects them in some arbitrary order. At the end of Step 4, the clusters are then formed as {0, 0, 1}, {3, 3}, {6, 7}, and {9, 10, 10}.



Steps 8 and 9 – The algorithm searches the distance matrix for the closest two clusters to merge. There are two candidates {0,0,1,3,3} and {6,7,9,10,10} each having a distance of 3. The algorithm selects them in some arbitrary order. At the end of Step 9, the clusters are then formed as {0,0,1,3,3} and {6,7,9,10,10}.

Note that although the result is identical to the single-linkage approach, the distance measure calculated using single-linkage was 2 whereas it is 3 using complete linkage.



Step 10 - The algorithm searches the distance matrix for the closest two clusters to merge. Since there are only two clusters, the algorithm merges them into a single cluster {0, 0, 1, 3, 3, 6, 7, 9, 10, 10} and then ends.

2 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 5. What is an intuitive idea for the meaning of the centroid of a cluster?

The centroid of a cluster can be envisioned as its geometric center. In other words, we can envision a cluster of observations say in R3 as a point cloud, and the center of that cloud would be the cluster centroid.

6. Suppose that we have the following data: a (2,0)

b (1,2)

c (2,2)

d (3,2)

e (2,3)

f (3,3)

g (2,4)

h (3,4)

i (4,4)

j (3,5)

Identify the cluster by applying the k-means algorithm, with k = 2. Try using initial cluster centers as far apart as possible.

Using points a and j as the initial cluster centers, the figure below illustrates how the K-means algorithm clusters the data points in two passes.

K-Means K=2 a (2,0) b(1,2) c(2,2) d(3,2) e(2,3) f(3,3) g(2,4) h(3,4) i(4,4) j(3.5)

Pass 1 Distance (2,0)

Distance (3,5) Assignment 0 5.099 C1 2.2361 3.6056 C1 2 3 C1 2.3607 3 C1 3 2.23607 C2 3.1623 2 C2 4 1.4142 C2 4.1231 1 C2 4.4721 1.4142 C2 5.099 0 C2

K-Means K=2 a (2,0) b(1,2) c(2,2) d(3,2) e(2,3) f(3,3) g(2,4) h(3,4) i(4,4) j(3.5)

Pass 2 Distance (2,1.5) Distance (2.833,3.833) Assignment 1.5 3.9229 C1 1.118 2.5927 C1 0.5 2.0138 C1 1.118 1.8409 C1 1.5 1.1785 C2 1.8028 0.8498 C2 2.5 0.8498 C2 2.6926 0.2357 C2 3.2012 1.1785 C2 3.7401 1.1785 C2

Figure 6.1. K-means clustering in two passes

From the figure above, we observe that the algorithm completes in two passes. The algorithm for this exercise is initialized to form two clusters using points a at (2,0) and j at (3,5) as the initial cluster centers. For each point in the data set, the algorithm calculates the Euclidean distance to each initial cluster center, and then assigns that point to the cluster with the shortest distance. The cluster assignments for Pass 1 define Cluster 1 as containing points {a, b, c, d} and Cluster 2 as containing points {e, f, g, h, i, j}.

The centroids are adjusted using these new cluster assignments. In Pass 2, for each point in the data set, the algorithm calculates the Euclidean distance to each adjusted centroid and then reassigns that point to the cluster with the shortest distance. However, in Pass 2 no new cluster assignments are made. Therefore, the centroids do not change and execution of the algorithm completes.

3 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 7. Refer to Exercise 6. Show that the ratio of the between-cluster variation to the within-cluster variation decreases with each pass of the algorithm.

In general, the ratio of between-cluster variation to within-cluster variation or pseudo-Fstatistic will actually increase with each pass of the k-means algorithm. The algorithm optimizes the clustering by maximizing between-cluster variation or MSB while simultaneously minimizing within-cluster variation or MSE. The pseudo-F statistic is calculated for each pass of the algorithm using the performance measures described in the table below.

Figure 7.1. Performance measures used to calculate the pseudo-F statistic

4 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. The figures below summarize the calculations of the performance measures for each pass of the algorithm from the previous exercise. The algorithm completes in two passes and the Fstatistic calculated for each pass is highlighted in grey.

Pass 1 Pass 2 Calculation Value Caculation SSB (4) * (2.21) + (6) * (0.9822) 14.7333 (4)*(2.21) + (6)*(0.9822) MSB (14.7333) / (2-1) 14.7333 (14.7333) / (2-1) SSE 0+5+4+5+5+4+2+1+2+0 28 2.25+1.25+0.25+1.39z=0.72+0.72+0.06+1.39+1.39 MSE (28) / (10-2) 3.5 (10.6667) / (10-2) F (14.73333) / (3.5) 4.2095 (14.7333) / (1.3333) Figure 7.2. Pseudo-F calculated for two passes of the k-means algorithm

Value 14.7333 14.7333 10.6667 1.3333 11.05

Referring to the table above, we observe that the F-statistic increases from with each pass as expected. The MSB in Pass 1 is calculated using the Grand Mean and the centroids that were computed from the initial cluster assignments. Since there were no reassignments made in Pass 2, the cluster centroids remain the same and so does the MSB in Pass2. However, the MSE in Pass 1 is calculated from the squared distances from each point to the initial cluster centers, and the MSE in Pass 2 is calculated from the squared distances from each point to the centroids computed at the end of Pass 1. The MSE decreases from Pass 1 (3.5) to Pass 2 (1.3333) while the MSB does not change making the F-statistic increase as expected from Pass 1 (4.20952) to Pass 2 (11.05).

8. Once again identify the clusters in Exercise 6 data, this time by applying the kmeans algorithm, with k = 3. Try using initial cluster centers as far apart as possible.

The approach taken to determine the best initial cluster centers to use is distance calculation of eight candidate sets. The distance calculation entails determining the Euclidean distance between each pair of points in the three-point set and taking the summation. The table below identifies the eight candidate point sets and their respective total distances.

Now one might use the point set with the largest total distance as the initial cluster centers.

However, it is interesting to observe that the point set with the largest total distance does not yield the best performance measures. Observe that Set # 8 had the largest total Euclidean distance (highlighted in gray), but Set # 3 had the second-largest total Euclidean distance and ties with Sets #1 and #2 as having the largest F-statistic (highlighted in yellow). Therefore, Set #3 will be used for the three-cluster analysis.

5 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

1 2 3 4 5 6 7 8

Point Set (2,0),(3,2),(3,5) (2,0),(2,2),(3,5) (2,0),(1,2),(3,5) (2,0),(2,3),(3,5) (2,0),(3,3),(3,5) (2,0),(2,4),(3,5) (2,0),(3,4),(3,5) (2,0),(4,4),(3,5)

Distance 10.3351 10.2613 10.9406 10.3351 10.2613 10.5132 10.2221 10.9854

MSB 9.325 9.325 9.325 8.95 9.075 8.86667 7.95 8.2

MSE 0.9643 0.9643 0.9643 1.0714 1.0357 1.0952 1.3571 1.2857

F 9.6703 9.6703 9.6703 8.3533 8.7621 8.0957 5.8579 6.3778

Figure 8.1. Pseudo-F vs Euclidean distance for eight candidate initial centroid sets

Using points a, b, and j as the initial cluster centers, the figure below illustrates how the kmeans algorithm clusters the data points.

Pass 1 K-Means k=3 Distance (2, 0) Distance (3, 5) Distance (1, 2 ) Assignment a (2,0) 0 5.09002 2.23607 C1 b (1,2) 2.3607 3.60555 0 C3 c (2,2) 2 3.16228 1 C3 d(3,2) 2.3607 3 2 C3 e(2,3) 3 2.23607 1.41421 C3 f(3,3) 3.16228 2 2.23607 C2 g(2,4) 4 1.41421 2.23607 C2 h(3,4) 4.12311 1 2.82843 C2 i(4,4) 4.47214 1.41421 3.60555 C2 5.09902 0 3.60555 C2 j(3,5) Pass 2 K-Means Distance (2, k=3 Distance (2, 0) Distance (3, 4) 2.25) Assignment a (2,0) 0 4.12311 2.25 C1 b (1,2) 2.3607 2.82843 1.03078 C3 c (2,2) 2 2.23607 0.25 C3 d(3,2) 2.3607 2 1.03078 C3 e(2,3) 3 1.41421 0.75 C3 f(3,3) 3.16228 1 1.25 C2 g(2,4) 4 1 1.75 C2 h(3,4) 4.12311 0 2.01556 C2 i(4,4) 4.47214 1 2.65754 C2 5.09902 1 2.92617 C2 j(3,5) Figure 8.2. K-means using points a, b, and j as initial centroids

6 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. From the figure above, we observe that the algorithm completes in two passes. The algorithm for this exercise is initialized to form two clusters using points a at (2,0), b at (1,2), and j at (3,5) as the initial cluster centers. For each point in the data set, the algorithm calculates the Euclidean distance to each initial cluster center, and then assigns that point to the cluster with the shortest distance. The cluster assignments for Pass 1 define Cluster 1 as containing singular point {a}, Cluster 2 as containing points {f, g, h, i, j}, and Cluster 3 as containing points {b, c, d, e}.

The centroids are adjusted using these new cluster assignments. In Pass 2, for each point in the data set, the algorithm calculates the Euclidean distance to each adjusted centroid and then reassigns that point to the cluster with the shortest distance. However, in Pass 2 no new cluster assignments are made. Therefore, the centroids do not change and execution of the algorithm completes.

9. Refer to Exercise 8. Show that the ratio of the between-cluster variation to the within-cluster variation decreases with each pass of the algorithm.

The tables below summarize the calculations of the performance measures for each pass of the algorithm from the previous exercise. The algorithm completes in two passes and the Fstatistic calculated for each pass is highlighted in grey.

Pass 1 Calculation Value SSB (1)*(8.66)+(5)*(1.46)+(4)*(0.6725) 18.65 MSB (18.65) / (3 - 1) 9.325 SSE 0+0+1+4+2+4+2+1+2+0 16 MSE (16) / (10 - 3) 2.2857 F (9.325) / (2.28571) 4.0797 Pass 2 Calculation Value SSB (1)*(8.66)+(5)*(1.46)+(4)*(0.6725) 18.65 MSB (18.65) / (3 - 1) 9.325 SSE 0+1.062+0.062+1.062+0.562+1+1+0+1+1 6.75 MSE (6.75) / (10-3) 0.9643 F (9.325) / (0.096429) 9.6704 Figure 9.1. Pseudo-F calculations

Referring to the table above, we observe that the F-statistic actually increases from with each pass as expected. The MSB in Pass 1 is calculated using the Grand Mean and the centroids that were computed from the initial cluster assignments. Since there were no reassignments made in Pass2, the cluster centroids remain the same and so does the MSB in Pass2.

7 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

However, the MSE in Pass 1 is calculated from the squared distances from each point to the initial cluster centers, and the MSE in Pass 2 is calculated from the squared distances from each point to the centroids computed at the end of Pass 1. The MSE decreases from Pass 1 (2.2857) to Pass 2 (0.9643) while the MSB does not change making the F-statistic increase as expected from Pass 1 (4.0797) to Pass 2 (9.6704).

10. Which clustering solution do you think is preferable? Why? In general, the two-cluster solution is preferable over the three-cluster solution. From Exercise 7 we observe that the F-statistic of the two-cluster solution is 11.05, which is larger than the F-statistic of the three-cluster solution (9.7603).

We have also observed from Exercise 8 that for this relatively small number of data points, the three-cluster operations become unstable and require a significant effort to determine the best set of initial cluster centers to use.

11. Confirm the calculations for the second pass and third pass for MSB, MSE, and pseudo-F for Step Four of the example given in the chapter.

The K-means example in Section 10.6 of the text has Grand Mean M = (2.625 2.25).

The cluster assignments for the second pass are given in the table below.

Point a(1,3) b(3,3) c(4,3) d(5,3) e(1,2) f(4,2) g(1,1) h(2,1)

Assignment C1 C2 C2 C2 C1 C2 C1 C1

Table 11.1. Cluster assignments for Step 4 second pass

Cluster C1 has points {a, e, g, h} and its centroid is (1.25, 1.75), and cluster C2 has points {b, c, d, f} and its centroid is (4, 2.75). Therefore, the calculations for the quality metrics are given in the figure below.

8 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

SSB MSB SSE MSE F

Pass 2 Calculation (4) * (2.1406) + (4) * (2.1406) (12.9748) / (2-1) 1+0.7225+0.5184+2.3104+0+0.3249+1+1.9881 (7.8643) / (8-2) (17.1248) / (1.311)

Value 17.1248 17.1248 7.8643 1.311 13.0624

Figure 11.1. Pseudo-F calculations for Step 4 second pass

For the second pass the text gives: MSB = 17.125, MSE = 1.313333, F = 13.03934. The calculations above confirm these values (assuming small round-off differences are acceptable).

The cluster assignments for the third pass are given in the table below.

Point a(1,3) b(3,3) c(4,3) d(5,3) e(1,2) f(4,2) g(1,1) h(2,1)

Assignment C1 C2 C2 C2 C1 C2 C1 C1

Table 11.2. Cluster assignments for Step 4 third pass

Cluster C1 now has points {a, e, g} but its centroid remains as (1.25, 1.75), and cluster C2 now has points {b, c, d, f, h} but its centroid remains as (4, 2.75). Therefore, the MSB remains the same, but the MSE changes. The calculations for the quality metrics are given in the figure below.

SSB MSB SSE MSE F

Pass 3 Caculation (4) * (2.1406) + (4) * (2.1406) (12.9748) / (2-1) 1.6129+1.0609+0.0625+1.0609+0.1225+0.5625+0.6241+1.1236 (6.2299) / (8-2) (17.1248) / (1.0383)

Value 17.1248 17.1248 6.2299 1.0383 16.4931

Figure 11.2. Pseudo-F calculations for Step 4 third pass

For the third pass the text gives: MSB = 17.125, MSE = 1.041667, F = 16.44. The calculations above confirm these values (assuming small round-off differences are acceptable).

9 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

HANDS-ON ANALYSIS Use the cereals data set, included at the book series website, for the following exercises.

Make sure that the data are normalized.

The Cereals dataset obtained from the DKD companion website [1] contains information about 77 breakfast cereals and is described in the table below.

Field Name NAME

Measure Nominal

Description Brand name of cereal

MANUF

Nominal

TYPE

Flag

Manufacturer encoding as follows: A = American Home Food Products G = General Mills K = Kellogg’s N = Nabisco P = Post Q = Quaker Oats R = Ralston Purina Hot or Cold Indicator

CALORIES

Continuous

Calories per serving

PROTEIN

Continuous

Protein per serving in grams

FAT

Continuous

Fat per serving in grams

SODIUM

Continuous

Sodium per serving in milligrams

FIBER

Continuous

Fiber per serving in grams

CARBO

Continuous

Carbohydrates per serving in grams

SUGARS

Continuous

Sugars per serving in grams

POTASS

Continuous

Potassium per serving in milligrams

VITAMINS

Continuous

WEIGHT

Ordinal

Vitamins per serving as a percentage of RDA Weight per serving in ounces

CUPS

Ordinal

Cups per serving

SHELF

Ordinal

Display shelf - Range [1,3]

RATING

Continuous

Cereal rating

Disposition This field will not be used in the analysis Modeling Tool will use indicator coding and if needed Min-Max normalization.

Modeling Tool will treat as binary flag.

Modeling Tool will Min-Max normalize Modeling Tool will Min-Max normalize Modeling Tool will Min-Max normalize Modeling Tool will Min-Max normalize Modeling Tool will Min-Max normalize Modeling Tool will Min-Max normalize Modeling Tool will Min-Max normalize Modeling Tool will Min-Max normalize Modeling Tool will Min-Max normalize Modeling Tool will Min-Max normalize Modeling Tool will Min-Max normalize Modeling Tool will Min-Max normalize This field will not be used in the analysis

Table D.1. Cereals data set

10 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 12. Using all of the variables except name and rating, run the k-means algorithm with k = 5 to identify clusters within the data.

The Cereals data set is described in Table D1 above. It is important to note that a -1 indicates a missing observation. K-Means clustering was performed on the data using all fields except NAME and RATING to partition the data into five clusters. Since this is a clustering exercise, no imputation of missing values was employed. The Modeler performs Min-Max normalization of all continuous measures and indicator coding of all nominal measures as well, so no additional standardization was needed. The clustering results are given in the table below.

Figure 12.1 Cluster assignments for k=5

11 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. Referring to the table above, we observe that the five clusters vary in size where Cluster 2 and Cluster 5 are relatively large (and about the same size), Cluster 1 is mid-size, and Cluster 3 and Cluster 4 are relatively small. In general, the clustering seems to be driven primarily by Manufacturer and Type.

13. Develop clustering profiles that clearly describe the characteristics of the cereals within the cluster.

The table below summarizes the exploratory analysis to form concise cluster profiles.

Cluster Cluster 1 - “High Fiber Cereals”

Profile In general Cluster 1 contains the cold high fiber cereals. These are heavier per serving and therefore have a smaller serving volume. These also have moderate amounts of sodium and sugar. Most of them are made by Post Cluster 2 - “High Energy Cereals” In general, Cluster 2 contains cold cereals that are high in complex carbohydrates and moderately high in calories and sugars. They also have either very high protein or very low protein. These also have large serving sizes.

Cluster 3 - “Healthy Cereals” In general, Cluster 3 contains the hot cereals that are high-protein and low in sodium and sugars. These are also low in fiber and vitamins Cluster 4 - “Filler Cereals” Cluster 4 contains cereals that are moderately low in calories yet high in fat. They are also moderate in carbohydrates and protein. These are made by Quaker.

Cluster 5 - “Children Cereals” Cluster 5 appears to contain most of the children’s cereals. These are high in carbohydrates, and sugars, but also high in vitamins. Most of these are made by Kellogg’s and General Mills.

Table 13.1. Cereals data set cluster profiles for k=5

12 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. The figures below contain exploratory visualizations and commentary for each predictor attribute.

Figure 13.1. Cluster profiling by attributes Manufacturer, Type, and Calories

13 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 13.2. Cluster profiling by attributes Protein, Fat, and Sodium

14 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 13.3. Cluster profiling by attributes Fiber, Carbohydrates, and Sugars

15 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 13.4. Cluster profiling by attributes Potassium, Vitamins, and Display Shelf

16 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 13.5. Cluster profiling by attributes Weight, and Cups

17 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 14. Rerun the k-means algorithm with k = 3.

The Cereals data set is described in Table D1 above. It is important to note that a -1 indicates a missing observation. K-Means clustering was performed on the data using all fields except NAME and RATING to partition the data into five clusters. Since this is a clustering exercise, no imputation of missing values was employed. The Modeler performs Min-Max normalization of all continuous measures and indicator coding of all nominal measures as well, so no additional standardization was needed. The clustering results are given in the table below.

Figure 14.1 Cluster assignments for k=3

18 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. Referring to the table above, we observe that the three clusters vary in size where Cluster 2 is largest, Cluster 1 is nearly as large, and Cluster 3 is relatively small compared to the other two clusters. In general, the clustering seems to be driven primarily by Manufacturer and Type. At a glance it appears that hot cereals have been combined with Nabisco cereals to form Cluster 3.

15. Which clustering solution do you prefer, and why?

The calculations for the Cereals performance measures are given in the stacked tables below.

Although the MSE value for the three-cluster solution is slightly larger than the value for the five-cluster solution, the MSB is nearly twice as large for the three-cluster solution. We can observe that this situation results in an F-statistic that is much larger for the three-cluster solution (70.55) than for the five-cluster solution (43.16).

The within-cluster variations in both solutions are fairly comparable. However, the betweencluster variations are much cleaner in the three-cluster solution. Therefore, the three-cluster solution is the preferred solution.

Figure 15.1 Pseudo-F calculations for k=3 and k=5

19 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

16. Develop clustering profiles that clearly describe the characteristics of the cereals within the cluster.

The table below summarizes the exploratory analysis to form concise cluster profiles.

Cluster Profile Cluster 1 - “High Energy Cereals” In general, Cluster 1 contains the cold high calorie cereals. These are high in complex carbohydrates, high in fat, and moderately high in sodium and sugars. They are also high in potassium making them ideal for endurance athletes.

Cluster 2 - “Filler Cereals” In general, Cluster 2 contains the cold cereals that are high in complex carbohydrates, moderately high in sodium and sugars and either very low or very high in protein. They are low in most nutrients like potassium and vitamins, and they have large serving sizes, and are typically found on the lower shelves.

Cluster 3 - “Healthy Cereals” In general, Cluster 3 contains the hot and cold healthier cereals. These are high-protein, low calorie cereals, that are also low in sugar and sodium.

However, these are also low in fiber and vitamins.

These are made by Nabisco, Quaker, and American.

Table 16.1 Cluster profiles for k=3

The figures below contain the exploratory visualizations and commentary for each predictor attribute.

Figure 16.1 Cluster profiling by attribute Manufacturer

20 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 16.2 Cluster profiling by attributes Type, Calories, Protein, and Fat

21 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 16.3 Cluster profiling by attributes Sodium, Fiber, and Carbohydrates

22 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 16.4 Cluster profiling by attributes Sugars, Potassium, and Vitamins

23 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Figure 16.5 Cluster profiling by attributes Display Shelf, Weight, and Cups

24 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 17. Use cluster membership to predict rating. One way to do this would be to construct a histogram of rating based on cluster membership alone. Describe how the relationship you uncovered makes sense, based on your earlier profiles.

The table below summarizes the rating predictions based on clustering for the threecluster solution.

Cluster Cluster 1 – Moderately High

Cluster 2 – Moderately Low

Cluster 3 – Moderately High

Predictive Analysis In general, Cluster 1 rates moderately high and in some cases very high. This aligns well with the fact that they are high-energy cereals packed with calories, carbohydrates, and are sweetened to some degree.

In general, Cluster 2 rates moderately low. This aligns well with the fact that they are filler cereals, and most likely the low-end generic cereals that are high in carbohydrates, but low in most nutrients In general, Cluster 3 rates moderately high. This aligns well with the fact that these are the healthier cereals that are high in protein and low in carbohydrates and sugars. They rate higher than the Filler cereals but lower than the High-Energy cereals.

Table 17.1 Rating predictions by cluster profile for k=3

The figure below depicts the visualization of the distribution of ratings across each cluster.

Figure 17.1 Rating distributions across clusters for k=3

25 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

The table below summarizes the rating predictions based on clustering for the five-cluster solution.

Cluster Predictive Analysis Cluster 1 – Moderately High In general, Cluster 1 rates moderately high and in some cases very high. This aligns well with the fact that they are high-fiber cereals that have small serving sizes and are heavier and most likely combat hunger longer. They also have moderate amounts of sodium and sugar, so I surmise that they taste good.

Cluster 2 – Moderately Low In general, Cluster 2 rates moderately low. This is difficult to explain, but I surmise that this cluster is a combination of high carb-high protein cereals that would rate well and high-carb-low-protein cereals that would rate poorly. This mix seems to have been driven by the fact that they all have relatively high serving sizes.

Cluster 3 – Moderately High In general, Cluster 3 rates moderately high. This aligns well with the fact that these are the three hot cereals, and all are healthier cereals that are high in protein and low in carbohydrates and sugars.

Cluster 4 – Moderately High In general Cluster 4 rates moderately high. Although these seem to be the Filler cereals, they seem to have moderate amounts of carbohydrates, protein giving them satisfactory nutritional value.

Cluster 5 – Moderately Low In general, Cluster 5 rates moderately low. This aligns well with the fact that this cluster contains the Children cereals that are high in carbohydrates and sugars. They are also vitamin-fortified. Perhaps the lower protein content is what is driving these ratings down.

Table 17.2 Rating predictions by cluster profile for k=5

26 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. The figure below depicts the visualization of the distribution of ratings across each cluster.

Figure 17.2 Rating distributions across clusters for k=5

27 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Solutions to Chapter 11 KOHONEN NETWORKS Prepared by James Cunningham, Graduate Assistant

1. Describe some of the similarities between Kohonen networks and the neural networks of Chapter 7. Describe some of the differences.

Similarities between Neural Networks and SOMs include:      

Architectures contain one input node per variable Input nodes simply pass values to next layer Networks are feed-forward and completely connected Have weights associated with connected nodes that are initialized randomly to 0 – 1 Adjusting weights represents key learning mechanism Input attributes require normalization or standardization

Differences include:     

SOMs have no hidden layer Nodes compete to “win” record SOMs include a cooperative “neighborhood” learning property The output layer forms a lattice, usually in two dimensions Output nodes in close proximity are more similar to one another

2. Describe the three characteristic processes exhibited by self-organizing maps such as Kohonen networks. What differentiates Kohonen networks from other selforganizing map models?

The three characteristic processes found in SOMs include competition, cooperation, and adaptation.

As part of the cooperative process, Kohonen Networks include “Kohonen Learning” where all nodes in the neighborhood of the particular winning node have their weights adjusted. In this way, they share in the “excitement”. These weights are adjusted using a linear combination of the input and current weight vectors, combined with the learning rate parameter, to update their connection weights.

1 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

3. Using weights and distance, explain clearly why a certain output node will win the competition for the input of a certain record.

Each input variable has a 1-1 association with a specific input node, and every input node is connected to all output nodes in the network. In addition, a weight value is associated with each input-output node connection in the network.

The algorithm uses the Euclidean Distance formula to compute the distances between the input vector, and all possible output nodes in the network. Minimizing this distance determines the output node, which has connection weight values most similar to the values of the input vector.

4. For larger output layers, what would be the effect of increasing the value of R? As the size of R increases, it decreases the possible number of natural clusters that the network reports. As the value of R becomes too large with respect to the 2-dimensional output node lattice, a pathological case develops where only a single cluster is created by the network.

5. Describe what would happen if the learning rate 𝜂 did not decline?

If the learning rate does not decline during the learning process, the network would be unable to focus on the fine-level details in the data set leading to better differentiation between the clusters.

6. This chapter shows how cluster membership can be used for downstream modeling.

Does this apply to the cluster membership obtained by hierarchical and k-means clustering as well?

Yes, when the target variable is not included as input to the cluster models.

2 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. HANDS-ON ANALYSIS Use the adult data set at the book series website for the following exercises.

7. Apply the Kohonen clustering algorithm to the data set, being careful not to include the income field. Use a topology that is not too large, such as 3 × 3.

Six numeric (defined above) fields, seven categorical fields, and a single Boolean field are specified as input. Note income is not included. Default options are used for the Kohonen Node, except that network width and length are both set to 3 when generating the model.

8. Construct a scatter plot (with x/y agitation) of the cluster membership, with an overlay of income. Discuss your findings.

A scatter plot of the cluster membership variables $KX-Kohonen and $KY-Kohonen with and income overlay, and agitation values = 0.4, is shown below.

Figure 8.1 Scatterplot of cluster membership variables

Clusters 00 and 20 appear to have the highest proportion of those with income = “>50K.” Next, clusters 21, 22, and 02 appear to have the next highest proportion of those with income = “>50K.” In contrast, clusters 01, 11, and 12 have a larger proportion of records where income = “<=50K.” Cluster 11 is very small, and cluster 10 is extremely small containing about 5 records.

3 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 9. Construct a bar chart of the cluster membership, with an overlay of income. Discuss your findings. Compare to the scatter plot.

First, $KX-Kohonen and $KY-Kohonen cluster membership variables to mapped to a derived variable named “Kohonen”. The distribution of Kohonen is shown with an income overlay below. Proportion values for income are normalized and values sorted in decreasing order according to count.

Value

Proportion

%

Count

1

10

0.02

5

2

11

0.36

90

3

21

3.63

907

4

12

3.98

996

5

01

4.66

1164

6

00

16.18

4044

7

22

18.97

4743

8

20

26.06

6515

9

02

26.14

6536

Income

<=50K.

>50K.

Figure 9.1 Normalized Histogram of cluster membership with Income overlay

4 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. Cluster 02 is the largest cluster. Contrary to analysis reported in Exercise 8, this cluster has the largest proportion of those with income = “<=50K.” This difference in results is perhaps due to the fact that the scatter plot displays all records, and the density of points and/or size of the image is not sufficient to reveal the proportion of records clearly. For example, a larger, cropped image of the scatter plot below indicates cluster 02 has a larger proportion of records (shown in blue) where income = “<=50K.”

Figure 9.2 Scatterplot of cluster membership variables focusing on Cluster 02

Clusters 00 and 20 have the largest proportion of records where income = “>50K.” These results agree with observations made in Exercise 8. The number of records for small clusters 10 and 11 is determined to be 5 and 90, respectively.

5 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 10. Construct a bar chart of the cluster membership, with an overlay of marital status.

Discuss your findings.

Next, the distribution of Kohonen is shown with a marital-status overlay. The proportion of marital-status values are normalized for each cluster, and clusters sorted in decreasing order according to count.

Value

Proportion

%

Count

1

10

0.02

5

2

11

0.36

90

3

21

3.63

907

4

12

3.98

996

5

01

4.66

1164

6

00

16.18

4044

7

22

18.97

4743

8

20

26.06

6515

9

02

26.14

6536

marital-status

Divorced

Married-AF-spouse Married-civ-spouse Never-married Separated Widowed

Married-spouse-absent

Figure 10.1 Normalized Histogram of cluster membership with Marital Status overlay

In cluster 02, all records appear to have marital-status = Never-married. Clusters 12 and 01 also have very high proportions of those with marital-status = Never-married. Clusters 21, 00, and 20 have large proportions of records where marital-status = Never-civ-spouse.

Cluster 22 contains a large proportion of divorced individuals. Cluster 11 contains the largest proportions of those either separated or having an absent spouse.

6 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

11. If your software supports this, construct a web graph of income, marital status, and the other categorical variables. Fine-tune the web graph so that it conveys good information.

A Web Graph is created including workclass, marital-status, race, sex, income, and Kohonen as input. The graph is adjusted to show one or more links to the largest clusters 00, 22, 20, and 02.

Figure 11.1 Directed Web Graph of cluster membership

As observed earlier, the connection from the “02” node to the “<=50K.” node indicates the high proportion of those in cluster 03 having income = “<=50K.” Also note the link between “02” and “Never-married”, which represents the high proportion of those in the cluster who never married.

7 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 12. Generate numerical summaries for the clusters. For example, generate a cluster mean summary.

The cluster mean values for six numeric attributes are shown in the table below. capital- capitalgain loss 00 4,044 46.79 2121.74 139.99 01 1,164 32.85 629.69 72.78 02 6,536 27.09 314.14 50.56 10 5 54.20 1942.40 0 11 90 34.20 149.78 88.32 12 996 35.84 638.98 55.24 20 6,515 41.80 1617.20 113.76 21 907 38.88 1622.00 86.63 22 4,743 45.12 670.71 62.94 Table 12.1 Cluster mean values for numeric variables Cluster

Count

age

demogweight 182315.22 187254.21 196278.96 149639.20 204094.80 198820.30 190203.16 186941.21 185441.54

education -num 10.73 10.31 9.99 10.20 8.63 9.42 10.12 9.98 9.74

hoursper-week 43.30 39.56 36.48 32.00 41.83 38.61 44.04 37.79 39.43

13. Using the information above and any other information you can bring to bear, construct detailed and informative cluster profiles, complete with titles.

Cluster 00: Married and Making Money All members of this large are marital-status = Married-civ-spouse, and are the most educated, on average, of all clusters. The mean values for capital gains and losses are also the largest of all cluster groups.

Cluster 01: Young and Single This middle-sized cluster contains about 90% of those who have never married. On average, the members in this cluster are the almost the youngest among the cluster groups.

Cluster 02: Youngest and Single This is the largest cluster group. All of those in this group have never married and contain those who, on average, are the youngest of all cluster groups. With the exception of tiny cluster 10, this group works the fewest number of mean hours per week.

Cluster 10: Looking for Tax Write-Off This tiny cluster has the highest mean age of all clusters. This group has the secondhighest capital gains, on average. However, this group doesn’t have any capital losses.

Cluster 11: Seeking a Raise This small cluster contains the least educated group of individuals, on average. In addition, this group reported the least income from capital gains, on average.

Cluster 12: Single in School This is a mid-sized cluster, containing a high proportion of singles. This group has the next-to-least number of years of education, and may suggest this group includes students.

8 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. Cluster 20: Married and Working Hard This is a very large cluster where all members are marital-status = Married-civ-spouse.

This group works the highest number of mean hours per week. All other numeric values appear on average, compared to the other clusters.

Cluster 21: Mostly Married Most individuals in this mid-sized cluster are married.

Cluster 22: Divorced About 75% of those in this cluster are divorced.

14. Use cluster membership as a further input to a CART decision tree model for classifying income. How important is clustering membership in classifying income?

A CART model is configured to predict income using one Boolean attribute, 7 categorical attributes, and 6 numeric inputs. Default CART options are used and the first decision tree model is generated (not shown.)

Next, the two Kohonen cluster membership variables are added as input to CART. A second model is generated. A comparison of the decision tree hierarchy determines that the CART does not split on the cluster membership variables, $KX-Kohonen or $KYKohonen, within the first 5 levels of the tree. Therefore, the conclusion is drawn that the cluster membership variables are not important, according to CART, when classifying income.

Note that additional testing may actually determine a CART model where one or more cluster membership variables are important for classification. For example, a CART model using a subset of the 16 inputs, which include the cluster membership variables, may result in a good classification model.

9 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 15. Use cluster membership as a further input to a C4.5 decision tree model for classifying income. How important is clustering membership in classifying income?

Compare to the CART model.

C5.0 is used with default options to predict income. The same inputs are used as in Exercise 14. According to C5.0, the $KY-Kohonen cluster membership variable is important for classification. In fact, the figure below shows that C5.0 splits on $KYKohonen at the second level.

Figure 15.1 C5.0 decision tree for predicting Income using cluster membership as a predictor

Recall that when splitting on a categorical attribute, C5.0 creates a separate branch for each attribute value. This way, when used as input to the decision tree building process in C5.0, the cluster membership information in $KY-Kohonen appears to be important in classifying income.

10 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose

Solutions to Chapter 12 ASSOCIATION RULES Prepared by James Cunningham, Graduate Assistant 1. Describe the two main methods of representing market basket data. What are the benefits and drawbacks of each?

Market basket data is represented using either transactional or tabular data formats.

One benefit of the transactional format is that only items actually purchased appear in the log. For example, a grocery store may sell 100,000 items; however, only items purchased by the customer appear in the log, which saves disk space. One possible drawback of the transactional format is the case where a customer purchases many items of the same type.

For example, if a customer purchases 20 candy bars, the item is repeated in the log 20 identical times, thus wasting valuable disk space.

Again, consider a grocery store that sells 100,000 items. Because most, if not all, transactions are sparse, valuable disk space is used storing 0’s, which represent nonpurchased items. This is considered a drawback.

2. Describe support and confidence. Express the formula for confidence using support.

For rule A  B, support is proportion of records containing both A and B. Confidence is proportion of records containing A, which also contain B.

Confidence  P( B | A) 

P ( A  B) Support  P ( A) P ( A)

3. Restate the a priori property in your own words.

The A Priori Property states that if a collection of n items representing an itemset I = {i1, i2, ..., in} are not frequent, adding any other item iNEW to the set results in a new itemset INEW that cannot be frequent.

1 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose For the following several exercises, consider the following data set from Quinlan [4] shown as Table 12.8. The goal is to develop association rules using the a priori algorithm for trying to predict when a certain (evidently indoor) game may be played.

Therefore, unlike the vegetable stand example, we may restrict our itemset search to items that include the attribute play.

4. Let 𝜙 = 3. Generate the frequent 1-itemsets.

Combination Outlook = sunny Outlook = overcast Outlook = rain Temperature = hot Temperature = mild Temperature = cool Humidity = high Humidity = normal Windy = false Windy = true Play = no Play = yes

Count 5 4 5 4 6 4 7 7 8 6 5 9

Table 4.1 Frequent 1-item sets

All 1-itemsets listed are frequent, since their count >= 3.

2 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose 5. Let 𝜙 = 3. Generate the frequent 2-itemsets.

Combination Outlook = sunny, Temperature = hot Outlook = sunny, Temperature = mild Outlook = sunny, Temperature = cool Outlook = sunny, Humidity = high Outlook = sunny, Humidity = normal Outlook = sunny, Windy = false Outlook = sunny, Windy = true Outlook = sunny, Play = no Outlook = sunny, Play = yes

Count 2 2 1 3 2 3 2 3 2

Outlook = overcast, Temperature = hot Outlook = overcast, Temperature = mild Outlook = overcast, Temperature = cool Outlook = overcast, Humidity = high Outlook = overcast, Humidity = normal Outlook = overcast, Windy = false Outlook = overcast, Windy = true Outlook = overcast, Play = no Outlook = overcast, Play = yes

2 1 1 2 2 2 2 0 4

Outlook = rain, Temperature = hot Outlook = rain, Temperature = mild Outlook = rain, Temperature = cool Outlook = rain, Humidity = high Outlook = rain, Humidity = normal Outlook = rain, Windy = false Outlook = rain, Windy = true Outlook = rain, Play = no Outlook = rain, Play = yes

0 3 2 2 3 3 2 2 3

Temperature = hot, Humidity = high Temperature = hot, Humidity = normal Temperature = hot, Windy = false Temperature = hot, Windy = true Temperature = hot, Play = no Temperature = hot, Play = yes

3 1 3 1 2 2

3 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose Temperature = mild, Humidity = high Temperature = mild, Humidity = normal Temperature = mild, Windy = false Temperature = mild, Windy = true Temperature = mild, Play = no Temperature = mild, Play = yes

4 2 3 3 2 4

Temperature = cool, Humidity = high Temperature = cool, Humidity = normal Temperature = cool, Windy = false Temperature = cool, Windy = true Temperature = cool, Play = no Temperature = cool, Play = yes

0 4 2 2 1 3

Humidity = high, Windy = false Humidity = high, Windy = true Humidity = high, Play = no Humidity = high, Play = yes

4 3 4 3

Humidity = normal, Windy = false Humidity = normal, Windy = true Humidity = normal, Play = no Humidity = normal, Play = yes

4 3 1 6

Windy = false, Play = no Windy = false, Play = yes

2 6

Windy = true, Play = no Windy = true, Play = yes

3 3

Table 5.1 Frequent 2-item sets

All 2-itemsets are shown above, however frequent 2-itemsets only include those that have a count >= 3.

4 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose 6. Let 𝜙 = 3. Generate the frequent 3-itemsets.

Combination Outlook = sunny, Humidity = high Outlook = sunny, Windy = false Outlook = sunny, Play = no Outlook = overcast, Play = yes Outlook = rain, Temperature = mild Outlook = rain, Humidity = normal Outlook = rain, Windy = false Outlook = rain, Play = yes

Count 3 3 3 4 3 3 3 3

Temperature = hot, Humidity = high Temperature = hot, Windy = false Temperature = mild, Humidity = high Temperature = mild, Windy = false Temperature = mild, Windy = true Temperature = mild, Play = yes Temperature = cool, Humidity = normal Temperature = cool, Play = yes

3 3 4 3 3 4 4 3

Humidity = high, Windy = false Humidity = high, Windy = true Humidity = high, Play = no Humidity = high, Play = yes Humidity = normal, Windy = false Humidity = normal, Windy = true Humidity = normal, Play = yes

4 3 4 3 4 3 6

Windy = false, Play = yes Windy = true, Play = no Windy = true, Play = yes

6 3 3

Table 6.1 Step 1for generating frequent 3-item sets – Frequent 2-item sets

Next, 2-itemsets F2 are joined to form candidate 3-itemsets C3 below.

5 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose

Combination Outlook = sunny, Humidity = high, Windy = false Outlook = sunny, Humidity = high, Play = no Outlook = sunny, Windy = false, Play = no Outlook = rain, Temperature = mild, Humidity = normal Outlook = rain, Temperature = mild, Windy = false Outlook = rain, Temperature = mild, Play = yes Outlook = rain, Humidity = normal, Windy = false Outlook = rain, Humidity = normal, Play = yes Outlook = rain, Windy = false, Play = yes

Count 2 3 2 1 2 2 2 2 3

Temperature = hot, Humidity = high, Windy = false Temperature = mild, Humidity = high, Windy = false Temperature = mild, Humidity = high, Windy = true Temperature = mild, Humidity = high, Play = yes Temperature = mild, Windy = false, Play = yes Temperature = mild, Windy = true, Play = yes Temperature = cool, Humidity = normal, Play = yes

2 2 2 2 2 2 3

Humidity = high, Windy = false, Play = no Humidity = high, Windy = false, Play = yes Humidity = high, Windy = true, Play = no Humidity = high, Windy = true, Play = yes Humidity = normal, Windy = false, Play = yes Humidity = normal, Windy = true, Play = yes

2 2 2 1 4 2

Table 6.2 Step 2for generating frequent 3-item sets – Candidate 3-item sets

Now, A Priori Property is applied to all 3-itemsets, and those with count are deleted.

Therefore, F3 = {{Outlook = sunny, Humidity = high, Play = no}, {Outlook = rain, Windy = false, Play = yes}, {Temperature = cool, Humidity = normal, Play = yes}, {Humidity = normal, Windy = false, Play = yes}}

6 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose 7. Using 75% minimum confidence and 20% minimum support, generate oneantecedent association rules for predicting play.

Four one-antecedent rules are shown below, that meet the minimum support and confidence criteria.

Rules If Outlook = overcast, then Play = yes If Temperature = cool, then Play = yes If Humidity = normal, then Play = yes If Windy = false, then Play = yes

Support 4/14 = 28.6% 3/14 = 21.4% 6/14 = 42.9% 6/14 = 42.9%

Confidence 4/4 = 100% 3/4 = 75.0% 6/7 = 85.7% 6/8 = 75.0%

Table 7.1 One-antecedent rules

No. 1 2 3 4 5 6 7 8 9 10 11 12 13 14

TABLE 12.8 Weather data set for association rule mining Outlook Temperature Humidity Windy Sunny Hot High False Sunny Hot High True Overcast Hot High False Rain Mild High False Rain Cool Normal False Rain Cool Normal True Overcast Cool Normal True Sunny Mild High False Sunny Cool Normal False Rain Mild Normal False Sunny Mild Normal True Overcast Mild High True Overcast Hot Normal False Rain Mild High True

Play No No Yes Yes Yes No Yes No Yes Yes Yes Yes Yes No

8. Using 75% minimum confidence and 20% minimum support, generate twoantecedent association rules for predicting play.

Four two-antecedent rules are shown below, that meet the minimum support and confidence criteria.

Rules If Outlook = sunny, Humidity = high, then Play = no If Outlook = rain, Windy = false, then Play = yes If Temperature = cool, Humidity = normal, then Play = yes If Humidity = normal, Windy = false, then Play = yes

Support Confidence 3/14 = 21.4% 3/3 = 100.0% 3/14 = 21.4% 3/3 = 100.0% 3/14 = 21.4% 3/4 = 75.0% 4/14 = 28.6% 4/4 = 100.0%

Table 8.1 Two-antecedent rules

7 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose 9. Multiply the observed support times the confidence for each of the rules in Exercises 7 and 8, and rank them in a table.

Exercise 7 Rankings Rules Support x Confidence If Humidity = normal, then Play = yes 0.368 If Windy = false, then Play = yes 0.322 If Outlook = overcast, then Play = yes 0.286 If Temperature = cool, then Play = yes 0.161 Table 9.1 Rules from Exercise 7 ranked in descending order

Exercise 8 Rankings Rules If Humidity = normal, Windy = false, then Play = yes If Outlook = sunny, Humidity = high, then Play = no If Outlook = rain, Windy = false, then Play = yes If Temperature = cool, Humidity = normal, then Play = yes

Support x Confidence 0.286 0.214 0.214 0.161

Table 9.2 Rules from Exercise 8 ranked in descending order

8 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose 10. Verify your manually found results using association rule software.

This table shows the output from SPSS Modeler’s A Priori Node with one-antecedent rules sorted by Support x Confidence. Note that these rules correspond to those manually derived in Exercise 9. Remember, Modeler reports “Support” differently; that is, rule 1 “If humidity = normal, then play = yes” has support = (50.00)(85.70) = 0.4285, and therefore has Support x Confidence = 0.4285 x 0.857 = 0.367 = 36.7%. Note this value corresponds to the value of 36.8% derived manually above.

Instances

Support

Confidence

Lift

Consequent

Antecedent 1

1

7

50.00

85.70

1.33

play = yes

humidity = normal

2

8

57.10

75.00

1.17

play = yes

windy = false

3

4

28.60

100.00

1.56

play = yes

outlook = overcast

4

4

28.60

75.00

1.17

play = yes

temperature = cool

Figure 10.1 One-antecedent rules ranked in descending order

The next table shows SPSS Modeler’s A Priori Node results for two-antecedent rules.

Again, the rules shown correspond to those derived manually in Exercise 9.

Instances

Support

Confidence

Lift

Consequent

Antecedent 1

Antecedent 2

4

4

28.60

100.00

1.56

play = yes

humidity = normal

windy = false

7

3

21.40

100.00

2.80

play = no

outlook = sunny

humidity = high

8

3

21.40

100.00

1.56

play = yes

outlook = rain

windy = false

4

28.60

75.00

1.17

play = yes

temperature = cool

humidity = normal

Figure 10.2 Two-antecedent rules ranked in descending order

9 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose 11. For each of the association rules found above by the a priori algorithm, find the Jmeasure. Then order the rules by J-measure. Compare the ordering with that from the a priori support × confidence ordering.

Below, the J-measure is calculated for one-antecedent rules, where J = p(x) [ p(y | x) ln[ p(y | x) / p(y) ] + (1 – p(y | x)) ln[ (1 – p(y | x)) / (1 – p(y))] ] If Humidity = normal, then Play = yes p(x) = 7/14 = 0.5, p(y) = 9/14 = 0.643, p(y | x) = 6/7 = 0.857 J = 0.5 [0.857 ln[0.857 / 0.643 ] + (1 – 0.857) ln[ (1 – 0.857) / (1 – 0.643)] ] = 0.5 [0.857 ln[1.33] + (0.143) ln[0.401] ] = 0.057 If Windy = false, then Play = yes p(x) = 8/14 = 0.571, p(y) = 9/14 = 0.643, p(y | x) = 6/8 = 0.75 J = 0.571 [0.75 ln[0.75 / 0.643 ] + (1 – 0.75) ln[ (1 – 0.75) / (1 – 0.643)] ] = 0.571 [0.75 ln[1.166] + (0.25) ln[0.700] ] = 0.015 If Outlook = overcast, then Play = yes p(x) = 4/14 = 0.286, p(y) = 9/14 = 0.643, p(y | x) = 4/4 = 1 J = 0.286 [1 ln[1 / 0.643 ] + (1 – 1) ln[ (1 – 1) / (1 – 0.643)] ] = 0.126 If Temperature = cool, then Play = yes p(x) = 4/14 = 0.286, p(y) = 9/14 = 0.643, p(y | x) = 3/4 = 0.75 J = 0.286 [0.75 ln[0.75 / 0.643 ] + (1 – 0.75) ln[ (1 – 0.75) / (1 – 0.643)] ] = 0.286 [0.75 ln[1.166] + (0.25) ln[0.700] ] = 0.007 Therefore, according to the J-measure the rules as ranked as follows: Rules If Outlook = overcast, then Play = yes If Humidity = normal, then Play = yes If Windy = false, then Play = yes If Temperature = cool, then Play = yes

J-Measure 0.126 0.057 0.015 0.007

Table 11.1 Rules ranked in descending order using J-measure

10 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose Recall the rules, as ranked by Support x Confidence: Rules If Humidity = normal, then Play = yes If Windy = false, then Play = yes If Outlook = overcast, then Play = yes If Temperature = cool, then Play = yes

Support x Confidence 0.368 0.322 0.286 0.161

Table 11.2 Rules ranked in descending order using Support and Confidence

Both measures agree on the ranking of the last rule, but don’t agree with the ranking of the other three.

12. Find the value of the J-measure for the sixth rule from Figure 12.5.

Rule: “If age < 0.445 and education-num < 0.767, then Work_Class = Private” p(x) = 0.2948, p(y) = 0.6958, p(y | x) = 0.8 J = 0.2948 [0.8 ln[0.8 / 0.6958 ] + (1 – 0.8) ln[ (1 – 0.8) / (1 – 0.6958)] ] = 0.2948 [0.8 ln[1.150 ] + (0.2) ln[0.657] ] = 0.0082

11 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose HANDS-ON ANALYSIS Use the churn data set, given at the book series website, for the following exercises. Use the Churn_Training_File. Filter out all variables except the following: VMail Plan, Intl Plan, CustServ Calls, and Churn. Set CustServ Calls to be ordinal. Allow the three predictors to be in either antecedent or consequent, but do not allow Churn to be in the antecedent.

The text prescribes using the Churn_Training_File, but there is only one file named Churn.txt on the companion website. Using this file, we leverage the fields identified in the table below.

Field Name VMail Plan Intl Plan CustServ Calls Churn

Description Dichotomous - Indicates whether customer has Voice Mail Plan.

Dichotomous - Indicates whether customer has International Plan.

Integer – Identifies the number of calls placed to Customer Service.

Dichotomous Categorical – Target variable indicating whether customer will churn.

Disposition Used as Antecedent and Consequent.

Used as Antecedent and Consequent.

Setting to Ordinal. Used as Antecedent and Consequent.

Used as Consequent only.

Table D.1 Churn data set

13. Set the minimum antecedent support to 1%, the minimum rule confidence to 5%, and the maximum number of antecedents to 1. Use rule confidence as your evaluation measure.

Using SPSS Modeler, we employ the A Priori node to generate Association Rules per the prescribed criteria. The configuration is depicted in the figure below.

Figure 13.1 A Priori node configuration

12 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose The structure used to generate this model is depicted in the figure below.

Figure 13.1 A Priori model structure

a. Find the association rule with the greatest lift.

The Association Rules generated by the model are depicted in the figure below. We show all measures and sort in descending order by Lift, and observe that Rule #2: (CustServ Calls = 5) => (Churn?)

Has the greatest Lift (4.182).

Figure 13.a.1 Rules ranked by Lift

13 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose b. Report the following for the rule in (a).

The rule found in exercise (a) is as follows: Rule #2: If (CustServ Calls = 5) Then (Churn? = True) i.

Number of instances The number of instances found for Rule #2 is 66. However, this is the number of instances of the Antecedent occurrence only. The actual number of instances is found by querying for occurrences of both Antecedent and Consequent, which for Rule #2 are actually 40. ii.

Support % (as defined in this chapter) The Support% for Rule #2 as defined by the chapter is different than the value given by the model as the instance count given by the model pertains to the occurrences of the Antecedent only. The true value is found by multiplying the Support% reported by the model by the Confidence% reported by the model as follows: Support = (0.0198) * (0.60606) = 0.012 or 1.2% Alternatively, this is calculated using the true number instances as follows: Support = (40) / (3333) = 0.012 or 1.2% iii. Confidence % The Confidence is 0.60606 or 60.606%. iv.

Rule support % The Rule support% for Rule #2 is 1.2% v.

Lift The Lift for Rule #2 is 4.182. vi.

Deployability The Deployability for Rule #2 is 0.78.

14 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose c. Using hand calculations show how the measures were calculated.

The measures are calculated as follows: 

Number of Instances: The number of instances is determined for this model by finding the number of k-item sets that meet the minimum Confidence threshold.

For our model, k = 1 and minimum Confidence is 5%. Therefore, the model has found 66 transactions where CustServ Calls is 5 and Churn? = True.



Support: The Support measure as defined in the text is formulated as follows: 𝑺𝒖𝒑𝒑𝒐𝒓𝒕 = 𝑷 𝑨 ∩ 𝑩 Therefore, the Support for Rule #2 is calculated as follows: Support = (40) / (3333) = 0.012 or 1.2%



Confidence: The Confidence measure is formulated as follows: 𝑷 𝑨∩𝑩 𝑪𝒐𝒏𝒇𝒊𝒅𝒆𝒏𝒄𝒆 = 𝑷 𝑩 𝑨 = 𝑷 𝑨 Which is really Support (as defined in the text) divided by the proportion of Antecedent occurrences (Support as reported in the Modeler). Therefore, the Confidence for Rule #2 is calculated as follows: Confidence = (0.012) / (0.0198) = 0.60606 or 60.606% Alternatively, this can be calculated by dividing the number of rule occurrences (i.e. Antecedent and Consequent) by the total number of Antecedent occurrences as follows: Confidence = (40) / (66) = 0.60606 or 60.606%



Rule Support: The Rule Support in the Modeler is the Support as defined in the text (see above).



Lift: The Lift measure is formulated as follows: 𝑅𝑢𝑙𝑒 𝑐𝑜𝑛𝑓𝑖𝑑𝑒𝑛𝑐𝑒 𝐿𝑖𝑓𝑡 = 𝑃𝑟𝑖𝑜𝑟 𝑝𝑟𝑜𝑝𝑜𝑟𝑡𝑖𝑜𝑛 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑜𝑛𝑠𝑒𝑞𝑢𝑒𝑛𝑡

The prior proportion of the consequent just is the number of occurrences of the consequent in the data set. Therefore, the Lift for Rule #2 is calculated as follows: Lift = (0.60606) / (483/3333) = 4.18

15 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose 

Deployability: The Deployability is the proportion of the data set that meets the Antecedent condition but does not meet the Consequent condition and is formulated as follows: 𝐴𝑛𝑡𝑒𝑐𝑒𝑑𝑒𝑛𝑡 𝑂𝑐𝑐𝑢𝑟𝑒𝑛𝑐𝑒 − 𝑅𝑢𝑙𝑒 𝑂𝑐𝑐𝑢𝑟𝑒𝑛𝑐𝑒 𝐷𝑒𝑝𝑙𝑜𝑦𝑎𝑏𝑖𝑙𝑖𝑡𝑦 = ∗ 100 𝐷𝑎𝑡𝑎 𝑆𝑒𝑡 𝑆𝑖𝑧𝑒 Therefore, the Deployability for Rule #2 is calculated as follows: Deployability = (66 – 40) / 3333 * 100 = (26) / (3333)*100 = 0.7801.

d. Explain, in terms of this data, what each of the measures in (c) means (you can skip (i)).

The measures (with the exception of the Instance count) are interpreted as follows: 

Support: The Support measure identifies the proportion of all transactions that meet the Rule Condition (i.e. both the Antecedent and Consequent conditions).



Confidence: The Confidence measure identifies the proportion of transactions that meet the Rule Condition (i.e. both the Antecedent and Consequent) divided by the proportion of transactions that the meet the Antecedent condition minimally.



Rule Support: The Rule Support measure identifies the Support as defined in the text (see above).



Lift: The Lift measure identifies the ratio of the Confidence to the proportion of the transactions that meet the Consequent minimally.



Deployability: The Deployability is the proportion of the data set that meets the Antecedent condition but does not meet the Consequent condition.

16 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose 14. Set the minimum antecedent support to 1%, the minimum rule confidence to 5%, and the maximum number of antecedents to 1. a. Generate rules using confidence difference as your evaluation measure with evaluation measure lower bound = 40. Explain what this evaluation measure means.

Using SPSS Modeler, we modify the A Priori node’s configuration so that it uses the Confidence Difference as the evaluation measure setting the lower bound to 40. The modified configuration is depicted in the figure below.

Figure 14.a.1 A Priori model configuration modified

We run the model, and observe that it generates a single rule (the same as Rule #2 in Exercise #13) as depicted in the figure below.

Figure 14.a.2 A Priori model output – single rule

17 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose b. For the rules that are generated, use hand calculations to compute the reported evaluation measure, and show that the evaluation measure lower bound has been met.

The reported evaluation measure is 46.115 meeting (actually exceeding) the lower bound of 40. This measure is calculated as follows: Evaluation = Confidence – (Prior Proportion of Consequent) Therefore the Evaluation for the Rule above is calculated as follows: Evaluation = (0.60606) – (483/3333) = (0.60606) – (0.14491) = 46.115. c. Generate rules using confidence difference as your evaluation measure with evaluation measure lower bound = 30.

We modify the lower bound and rerun the model. This generates two Association Rules as depicted in the figure below.

Figure 14.c.1 A Priori model output using confidence differences using lower bound = 30

d. Select the rule with the highest deployability. Explain why the deployability of this rule is greater than the rule we found in Question 1a. 

The Deployability is the proportion of the data set that meets the Antecedent condition but does not meet the Consequent condition. The Deployability for each rule is calculated below.

DeployabilityRule1 = (66 – 40) / 3333 * 100 = (26) / (3333)*100 = 0.7801.

DeployabilityRule2 = (166 – 76) / 3333 * 100 = (90) / (3333)*100 = 2.7003.

We observe that Rule #2 has a higher Deployability. This is due to the fact that Rule #2 has a greater difference between the number of Antecedent-condition occurences and Rule-condition (i.e. both Antecedent and Consequent) occurences.

18 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose 15. Set the minimum antecedent support to 1%, the minimum rule confidence to 5%, and the maximum number of antecedents to 1. a. Generate rules using confidence ratio as your evaluation measure with evaluation measure lower bound = 40. Explain what this evaluation measure means.

We modify the A Priori node configuration to use Confidence Ratio with a lower bound of 40 as prescribed above. This generates three Association Rules, which are depicted in the figure below.

The Confidence Ratio is the ratio is the compliment to the ratio of the Confidence to the Proportion of the Consequent and is formulated as follows:

𝒑 𝒚𝒙 𝒑 𝒚 𝑪𝒐𝒏𝒇𝒊𝒅𝒆𝒏𝒄𝒆𝑹𝒂𝒕𝒊𝒐 = 𝟏 − 𝐦𝐢𝐧⁡ ( , ) 𝒑 𝒚 𝒑 𝒚𝒙 This formulation gives a small value when the proportion of the transactions that meet the Rule condition is close to the proportion of the transactions the meet the Consequent condition as the quotient of these two proportions would be close to 1. Therefore, an analyst would use this measure to identify the Association Rules that have a large difference in proportions of transactions meeting the Rule Condition and transactions meeting the Consequent only.

Figure 15.a.1 A Priori model output using confidence ratio with lower bound = 40

19 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose b. Select the rule involving Intl Plan. Use hand calculations to compute the reported evaluation measure, and show that the evaluation measure lower bound has been met.

The rule involving the Intl Plan from exercise 15a is Rule #3 with an evaluation measure of 65.834 using Confidence Ratio. The Confidence Ratio for this rule is calculated as follows: First, we find the minimal ratio as follows: Min ((0.42415) / (483/3333), (483/3333) / (0.42415)) = Min ((0.42415) / (0.14491), (0.14491) / (0.42415)) = Min ((2.93), (0.3416)) = 0.3416 Then we subtract this value from 1 to find the Confidence Ratio as follows: Confidence Ratio = 1 – 0.3416 = 0.65835 or 65.835%.

16. Compare the results from Exercise 13 with the results from the EDA and decision tree analysis in Chapters 3 and 6. Discuss similarities and differences. Which analysis format do you prefer? Do you find a confluence of results?

According to CART and C5.0, day_charge_MM is the most important variable for splitting at the root node. Because day_charge_MM is not used with A Priori, no similarities can be drawn.

For the second and third level splits, both CART and C5.0 split on Intl Plan and/or Voice Mail Plan. For example, C5.0 generated the following decision rule when classifying churn: Rule 9 for False. (69.098, 0.971) if day_charge_MM > 0.70022601 and VMail Plan = yes and Intl Plan = no then False.

A Priori produced the rule “If VMail Plan = yes and Intl Plan = no, then Churn? = False” with support = 24.90 and confidence = 94.70. The actual support equals (0.249) x (0.947), which means 23.58% (786/3,333) of the records contain both the antecedents and consequent. In this case, the rule applies to 786 records, whereas the C5.0 rule applies to only 69 records.

One additional note is that A Priori produced rules containing only “Churn? = False” for the consequent. Experimentation likely generates rules for “Churn? = True” as the minimum support and/or confidence thresholds are lowered.

20 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose 17. Apply the GRI algorithm to uncover association rules for predicting either churn or nonchurn behavior. Specify reasonable lower bounds for support and confidence.

The GRI Node is configured in SPSS Modeler with “Churn?” as the consequent, and “Intl Plan”, “Voice Mail Plan”, “day_charge_MM”, and “custserv_calls_MM” are the antecedents. Note that the addition of the numeric inputs are chosen based on results from CART and C5.0. That is, these two variables are important for classifying churn.

Initially, no rules are found using the other GRI options; therefore, experimentation determines the values minimum support = 30 and minimum confidence = 90 produce interesting rules. The results, ranked according to Support x Confidence, are shown below: Instances Support Confidence Lift

Consequent

Antecedent 1

Antecedent 2

1

2686

80.59

90.00

1.05 Churn? = False. day_charge_MM < 0.632

2

2451

73.54

93.00

1.08 Churn? = False. day_charge_MM < 0.632

3

2381

71.44

93.00

1.08 Churn? = False.

4

2089

62.68

94.00

1.09 Churn? = False. day_charge_MM < 0.632

custserv_calls_MM < 0.310

5

1885

56.56

90.00

1.05 Churn? = False.

day_charge_MM < 0.619

Intl Plan = no

VMail Plan = no

Intl Plan = no day_charge_MM < 0.619

Figure 17.1 GRI model rules ranked in descending order by Support and Confidence

These results, similar to those reported using A Priori, apply only to records where “Churn? = False”.

18. Compare the results from the a priori algorithm with those of the GRI algorithm.

Which algorithm yields a richer set of rules, and why? Which algorithm is probably preferable for this particular data set? Why? GRI provides a richer set of association rules for this data set, since it is able to form rules using both Boolean and numeric inputs. This enables the algorithm to find local patterns not detectable by A Priori, which may result in actionable or profitable results.

21 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Solutions to Chapter 13 IMPUTATION OF MISSING DATA Prepared by James Cunningham, Graduate Assistant 1. Why do we need to impute missing data?

How missing data is addressed has a significant impact on the outcome of any analysis.

There are several popular methods that replace missing values. For example, we can replace a missing value with a constant selected by the analyst. For continuous fields, we can replace a missing value with the arithmetic mean, and for categorical fields, we can replace a missing value with one of the attribute’s classes chosen at random.

However, none of these methods ensure alignment with the other values in an observation. Therefore, the most optimal way to replace missing values is to impute them. We impute missing data as the replacement values are driven by the values of the other attributes in an observation thus guaranteeing that these replacement values are sensible.

2. When imputing a continuous variable, explain what we use for the set of predictors, and for the target variable.

When imputing values for a continuous variable, we would employ multiple regression where the variable containing the missing values would be the model target and all other predictors in the data set would be predictors of the imputation model. Please note that we would never use the original response variable in an imputation model.

Since not all predictors would be significant with respect to the imputation target, we would employ a stepwise approach where the regression model is constructed from the most significant predictor only. The other predictors are added in significance order until no further improvement can be made on the model’s efficacy.

3. When imputing a missing value, do we include the original target variable as one of the predictor variables for the data imputation model? Why or why not?

As stated in exercise #2, we would never include the original response variable as one of the predictors for the imputation model. The rationale for not doing this is that the original response variable is expected to be highly correlated to all of the original predictors and would thus create a model that was highly multi-collinear and produces unstable results.

1 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 4. Describe what we should do if there are many variables with many missing values.

If there are multiple variables with missing variables, the analyst should employ a multipass imputation strategy. Each pass would in turn impute the variable having the fewest missing values. Once each variable is imputed in a prior pass, it would then be used as a predictor to impute the variable in the next pass.

5. On your own, think of a data set where a potential pattern in missingness would represent good information.

In most practical cases, data is often missing due to voluntary omission. Therefore, when this voluntary omission can be inferred based on the remaining values in a data set’s observations, the missingness actually represents a useful pattern.

As an example, consider a scenario where an electronics manufacturer surveys a random subset of its past and / or current customers for feedback on its product line and one of the questions happens to be household income. At the end of the campaign, the analyst observes that the household income variable has missing values. However, since the analyst is able to identify that the missing income values are associated with higher numbers of products purchased, and perhaps the market value of each customer’s address, the analyst infers that income is not disclosed by customers that have a relatively high income.

6. State two methods for handling patterns in missingness.

There are two popular methods for handling patterns in missingness. First, the analyst can simply derive a new field as a Boolean flag indicating whether the original field’s value is imputed or not.

Second, an analyst can derive a weight field that identifies the significance of each observation based on how many of its fields contain imputed values. As an example, consider a data set with ten predictors. For a record containing no imputed fields, the significance field would have a value of 1. A record containing one imputed field would have a significance value of 0.90, and a record containing two imputed fields would have a significance value of 0.80, and so on.

2 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. HANDS-ON ANALYSIS Use the cereals data set for Exercises 7–12. Report the standard error of each imputation.

The Cereals data set has three records with missing values (the missing values are represented by a -1). Almond Delight and Cream of Wheat have no potassium value, and Quaker Oatmeal has no Sugar or Carbohydrate value. The missing values are depicted in the figure below.

7. Impute the potassium content of Almond Delight using multiple regression.

In order to construct a regression model for this set of predictors, the flag variable TYPE and the categorical variable MANUF need to be converted to binary indicator variables.

A subset of the prepared data with the binary indicator variables is depicted below.

Figure 7.1 Binary indictor variables for use in Multiple Regression

We exclude the three observations with missing values (Almond_Delight, Cream_of_Wheat, and Quaker_Oatmeal) from the observation set and fit a Regession Model using MinTab 17 as depicted in the figure below.

Figure 7.2 Fitting a Regression model in Mintab

3 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Using stepwise regression, we fit the model with a Standard Error of the estimate minimized as 21.3976, and the final stepwise regression equation is depicted in the figure below.

Figure 7.3 Regression equation from Minitab output

We then impute the Potassium value for Almond Delight using the fitted model as entering the values for the Almond_Delight observation as depicted in the figure below.

Figure 7.4 Imputing Potassium value for Almond Delight in Minitab

4 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. The imputed Potassium value for Almond_Delight is 77.9666, and its Standard Error of Imputation (labeled as SE Fit in MiniTab 17) is 4.41197 as depicted in the figure below.

Figure 7.4 Imputed Potassium value for Almond Delight using Mintab

5 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 8. Impute the potassium content of Cream of Wheat.

We then impute the Potassium value for Cream_of_Wheat using the same fitted model constructed in exercise #7. The imputed Potassium value is 67.1076, and its Standard Error of Imputation (labeled as SE Fit in MiniTab 17) is 5.84367 as depicted in the figure below.

Figure 8.1 Imputed Potassium value for Cream of Wheat using Mintab

6 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 9. Impute the carbohydrates value of Quaker Oatmeal.

Now that we have imputed Potassium values for Almond_Delight and Cream_of_Wheat, we can include these observations and use them to fit a new Regression model for imputing Carbohydrates for Quaker_Oatmeal. Using stepwise regression, we fit the model with a Standard Error of the estimate minimized as 2.74847, and the final stepwise regression equation is depicted in the figure below.

Figure 9.1 Regression equation from Mintab output

7 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. We use the fitted model to impute the Carbohydrates value for Quaker_Oatmeal. This value is 11.6820, and its Standard Error of Imputation (labeled as SE Fit in MiniTab 17) is 1.19394 as depicted in the figure below.

Figure 9.2 Imputed Carbohydrates value for Quaker Oatmeal using Mintab

10. Impute the sugars value of Quaker Oatmeal.

Using stepwise regression, we fit the model with a Standard Error of the estimate minimized to 1.04078, and the final stepwise regression equation is depicted in the figure below.

Figure 10.1 Regression equation from Mintab output

8 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. We use the fitted model to impute the Sugars value for Quaker_Oatmeal using its newly imputed Carbohydrates value of 11.6820 (since Carbohydrates is a predictor in the fitted model). This value is 4.57216, and its Standard Error of Imputation (labeled as SE Fit in MiniTab 17) is 0.403276 as depicted in the figure below.

Figure 10.2 Imputed Sugars value for Quaker Oatmeal using Mintab

9 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 11. Insert the value obtained in Exercise 10 for the sugars value of Quaker Oatmeal, and impute the carbohydrates value of Quaker Oatmeal.

Using stepwise regression, we fit a new model similar to the one in the previous exercise, but we make Carbohydrates the response, and Sugars a predictor. This new model has a Standard Error of the estimate of 0.943340, and the regression equation is depicted in the figure below.

Figure 11.1 Regression equation from Mintab output

10 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. We then use the fitted model to impute the Carbohydrates value for Quaker_Oatmeal using its newly imputed Sugars value of 4.57216. This value is 11.9089, and its Standard Error of Imputation (labeled as SE Fit in MiniTab 17) is 0.364467 as depicted in the figure below.

Figure 11.2 Imputed Carbohydrates value for Quaker Oatmeal using imputed Sugars in Mintab

12. Compare the standard errors for the imputations obtained in Exercises 9 and 11.

Explain what you find.

The Standard Error of the Estimate from exercises #9 and #11 are 2.74847 and 0.943340 respectively. The Standard Error of the Fit from exercises #9 and #11 are 1.19394 and 0.364467 respectively.

We observe that the error in exercise #11 is less than one-third the size of the error in exercise #11. This is due to the fact that Sugars and Carbohydrates have a high degree of interaction, but since both of them were missing from the Quaker_Oatmeal, we had to choose one to impute without using the other as a predictor for exercise #9.

However, once we had imputed a value for Carbohydrates, we were free to inpute Sugars using Carbohydrates as a predictor, and then subsequently reverse the two variables in exercise #11 to impute Carbohydrates using Sugars as a predictor.

11 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 13. Open the ClassifyRisk_Missing data set. Impute the missing value for marital status.

Using SPSS Modeler we impute the missing value for marital status by (1) standardizing the income and age variables, (2) deriving loans as an integer and typing it to an Ordinal measure to prevent the algorithm from splitting by arithmetic mean, and (3) reserving 30.7% of the dataset for overfit prevention. Note that since all values for the three predictors that we are using to build the model are present, we can utilize all records to build the model. The resulting CART decision tree is depicted in the figure below.

Figure 13.1 CART model decision tree for imputing Marital Status in ClassifyRisk_Missing

The standardized values for the record with missing marital status are depicted in the figure below.

Figure 13.2 Missing Marital Status value in the ClassifyRisk_Missing data set

12 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. We impute the missing marrital status as ‘married’ with 96.7% confidence by walking the decision tree as follows: 1. Since the record has loans = 1, we traverse the left / top subtree (loans_integer[0 1]). 2. Since the record has z-score of income = 1.498, we traverse the right sub-tree (z_income > 0.812) 3. Since this is a leaf, we impute the missing value as ‘married’.

This node has a support value of 30 and a confidence level of 96.7%.

13 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. Use the ClassifyRisk_Missing2 data set for Exercises 14–15.

14. Impute all missing values in the data set. Explain the ordering that you are using.

This dataset has missing values as follows:  Mortgage – dichotomous, missing 1  Maritalstatus – categorical, missing 1  Loans – ordinal, missing 1  Age – continuous, missing 1 (record #11 is missing both age and income).  Income – continuous, missing 2 Since there are multiple variables that have missing values, the general approach is to impute the values of the variable having the fewest missing values. Therefore, the imputation is performed in the following order: 1. We impute the missing mortgage value using C&R with all other predictors in a stepwise fashion until confidence is maximized. Since it’s dichotomous, it has the lowest maximum probability at 50% of imputing incorrectly. The missing mortgage value inputes as “y” with support 5 and 80% confidence and the decision tree is depicted in the figure below.

Figure 14.1 CART model decision tree for imputing Mortgage value in ClassifyRisk_Missing2

14 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 2. We impute the missing maritalstatus value using C&R with all other predictors including and imputed mortgage value in a stepwise fashion until confidence is maximized. This has three possible values and therefore a maximum probability of 66% of imputing incorrectly. The missing marital status value inputes as “married” with support 27 and 100% confidence and the decision tree is depicted in the figure below.

Figure 14.2 CART model decision tree for imputing Marital Status value in ClassifyRisk_Missing2

15 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 3. We impute the missing loans value using C&R with all other predictors including the imputed marital status value in a stepwise fashion until confidence is maximized.

This has four possible values and therefore a maximum probability of 75% of imputing incorrectly. The missing loans value is imputed as 1 with support 52 and 71.2% confidence and the decision tree is depicted in the figure below.

Figure 14.3 CART model decision tree for imputing Loans value in ClassifyRisk_Missing2

16 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 4. We impute the missing age value using Regression with all other predictors including the imputed loan value and deriving enumerations for all categorical variables in a stepwise fashion until standared error is minmized. This variable has fewer missing values than income and it also has a smaller range of values than income. The regression model identifies that the Standard Error of the Estimate is 9.206953 and is depicted in the figure below.

Figure 14.4 Regression model for imputing Age value in ClassifyRisk_Missing2

The missing age value is imputed as 35.938 as depicted in the figure below.

Figure 14.5 Imputed Age value in ClassifyRisk_Missing2

17 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. Using Minitab, we determine the Standard Error of the Fit to be 1.546 as depicted in the figure below.

Figure 14.6 Standard Error of the Fit for Observation #11

18 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 5. Finally, we impute the two missing income values using Regression with all other predictors including the imputed the imputed age value and deriving enumerations for all categorical variables in a stepwise fashion until standard error is minimized. The regression model identifies that the Standard Error of the Estimate as 7489.6287 and is depicted in the figure below.

Figure 14.7 Imputing Income values in ClassifyRisk_Missing2

19 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. The missing income values are imputed as 34002.595 for age = 35.938 and 35949.110 for age = 42 and are depicted in the figure below.

Figure 14.8 Imputed Income values in ClassifyRisk_Missing2

Using Minitab, we determine the Standard Error of the Fit to be 1261 for age = 35.938 (Observation #11), and 1167 for age = 42 (Observation #15) as depicted in the figure below.

Figure 14.9 Standard Error of the Fit for observations #11 and #15

20 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D. 15. Report the standard errors (for continuous values) or confidence levels (for categorical values) for your imputations in Exercise 14.

The standard errors and confidence levels from exercise 14 are as follows: 

Mortgage is imputed as ‘y’ for Observation #8 with support of 5 and 80% confidence.



Marital Status is imputed as “married” for Observation #19 with support of 27 and 100% confidence.



Loans is inputed as 1 for Observation #7 with support 52 and 71.2% confidence.



Age is imputed as 35.938 for Observation #11 with Standard Error of the Estimate of 9.206953 and a Standard Error of the Fit value of 1.546.



Income values are imputed with a Standard Error of the Estimate of 7489.6287 as 34002.595 with a Standard Error of Fit of 1261 for Observation #11, and 35949.110 with a Standard Error of the Fit value of 1167 for Observation #15.

21 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Solutions to Chapter 14 MODEL EVALUATION TECHNIQUES Prepared by James Cunningham, Graduate Assistant HANDS-ON ANALYSIS Use the churn data set at the book series website for the following exercises. Make sure that the correlated variables have been accounted for.

The Churn data set has twenty candidate predictor fields. The table below identifies the type and disposition of each field.

Field Account Length Area Code CustServ Calls Day Calls Day Charge Day Mins Eve Calls Eve Charge Eve Mins Intl Calls Intl Charge Intl Mins Intl Plan Night Calls Night Charge Night Mins Phone State VMail Message VMail Plan

Type

Disposition

Numeric Categorical Numeric Numeric Numeric Numeric Numeric Numeric Numeric Numeric Numeric Numeric Dichotomous Categorical Numeric Numeric Numeric Unique Identifier Categorical Numeric Dichotomous Categorical

Used as is Anomalous with State and omitted from analysis Used as is Used as is Perfectly correlated with Day Mins and omitted Used as is Used as is Perfectly correlated with Eve Mins and omitted Used as is Used as is Perfectly correlated with Intl Mins and omitted Used as is Used as is Used as is Perfectly correlated with Night Mins and omitted Used as is Unique identifier and omitted from analysis Anomalous with Area Code and omitted Used as is Used as is

Table D.1 Churn data set

1 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

There are 15 Numeric fields, and these were explored for their correlations. The table below depicts the correlation matrix. We observe that Day Mins and Day Charge are perfectly correlated, Eve Mins and Eve Charge are perfectly correlated, Intl Mins and Intl Charge are perflectly correlated, and Night Mins and Night Charge are perfectly correlated (as highlighted in yellow). Therefore, we will want to omit one field from each of these four pairs from the modeling phase. Since the Mins fields have a wider range than their corresponding Charge fields, we will retain the Mins fields and omit the Charge fields as the Mins fields should be less influenced by extreme values.

Intl Charge

Intl Mins

Night Charge

Night Mins

0.007

0.006

-0.013

0.007

1

Eve Calls

0.019

0.002

0.006

0.016

0.016

Eve Charge Eve Mins

-0.007

-0.013

-0.021

0.007

0.007

-0.011

-0.007

-0.013

-0.021

0.007

0.007

-0.011

1

Intl Calls

0.021

-0.018

0.005

0.008

0.008

0.017

0.003

0.003

Intl Charge Intl Mins Night Calls Night Charge Night Mins Vmail Message

0.01

-0.01

0.022

-0.01

-0.01

0.009

-0.011

-0.011

0.032

0.01

-0.01

0.022

-0.01

-0.01

0.009

-0.011

-0.011

0.032

1

-0.013

-0.013

-0.02

0.023

0.023

0.008

0.008

0.008

0

-0.014

-0.014

-0.009

-0.009

0.023

0.004

0.004

-0.002

-0.013

-0.013

-0.012

-0.015

-0.015

0.011

-0.009

-0.009

0.023

0.004

0.004

-0.002

-0.013

-0.013

-0.012

-0.015

-0.015

0.011

1

-0.005

-0.013

-0.01

0.001

0.001

-0.006

0.018

0.018

0.014

0.003

0.003

0.007

0.008

0.008

Night Calls

Eve Mins

-0.013

Intl Calls

Eve Charge

0.006

Eve Calls

Day Mins

-0.019

Day Calls

Day Charge

CustServ Calls

Account Length 0.038

Account Length CustServ Calls Day Calls Day Charge Day Mins

-0.004

Table D.2 Correlation Matrix for Churn data set numeric fields

2 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

There are two Categorical fields. The Area Code field has three classes (408, 415, and 510), and the State field has 51 classes (i.e. all 50 states and the District of Columbia). However, exploratory analysis in the form of a Cross-tabulation identifies that Area Code is definitely anomalous as (1) there are only three area codes, which all belong to California yet they are distributed across all states and DC, and (2) the Chi-Square p-value is 0.608 indicating that the area codes are distributed approximately uniformly across the state classes (i.e. there is no evidence against the null hypothesis that the observed approximate uniform distribution is indicative of the underlying population, and we know for a fact that there should be!). The cross-tabulation showing a partial view of the state classes and the Chi-Square p-value is depicted in the figure below. We will therefore omit the Area Code field from the modeling phase. Also, since State and Area Code to not make sense when taken together, we will omit the State field as well.

Figure D.1 Cross Tabulation of State and Area Code for Churn data set

Once the data is prepared, we partition the original 3333 records into a training partition containing 70% of the data (2275 records) and a testing partition containing 30% of the data (1058 records). The Churn_Training.xlsx will be for training all models in this unit, and the Churn_Testing.xlsx will be used to score all models in this unit.

3 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

1. Apply a CART model for predicting churn. Use default misclassification costs.

Determine the following measures. a. Proportion of false positives. b. Proportion of false negatives. c. Overall error rate. d. Overall model accuracy (1 – overall error rate). e. Sensitivity f. Specificity We use SPSS Modeler to construct a CART model to predict churn using our Churn_Training.xlsx file. The structure used is depicted in the figure below.

Figure 1.1 Model structure CART model predicting Churn

We then score the model using the Churn_Testing.xlsx file. The resulting Confusion Matrix is depicted below.

Figure 1.2 Confusion Matrix with no misclassification costs

4 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

Finally, we use the values from the confusion matrix to calculate the various performance measures. These measures are depicted in the table below.

Performance Measure Proportion of false positives Proportion of false negatives Overall error rate Overall model accuracy Sensitivity Specificity

Value (9)/(103) = 0.0874 (67)/(955) = 0.0702 (9+67) / (1058) = 0.0718 1 – 0.0708 = 0.9292 (94)/(161) = 0.5839 (888) / (897) = 0.9900

Table 1.1 Performance measures for no-cost model

2. In a typical churn model, in which interceding with a potential churner is relatively cheap but losing a customer is expensive, which error is more costly, a false negative or a false positive (where positive = customer predicted to churn)? Explain.

In a typical churn model, a positive prediction means that the model predicts a customer as one who would churn, and a negative prediction means that the model predicts a customer as one who would not churn.

Given these two definitions, it is straightforward to conclude that the False Negative prediction is the most costly. In other words, a False Negative prediction means that the model has predicted a customer as one who would not churn, but in fact the customer does actually churn. In other words, the business this model supports would lose that customer, and losing a customer is expensive.

5 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

3. Based on your answer to Exercise 2, adjust the misclassification costs for your CART model to reduce the prevalence of the more costly type of error. Rerun the CART algorithm. Compare the false positive, false negative, sensitivity, specificity, and overall error rate with the previous model. Discuss the trade-off between the various rates in terms of cost for the company.

Let us assume that the cost of losing a customer is 10x that of customer intercession. We reconstruct the CART model adjusting the misclassification cost for false negative predictions by a factor of 10 as depicted in the figure below.

Figure 3.1 CART model configuration using 10x misclassification cost

We then score the model using the Churn_Testing.xlsx file. The resulting Confusion Matrix is depicted below.

Figure 3.2 Confusion Matrix with 10x misclassification costs

6 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

We then use the values from the confusion matrix to calculate the various model performance measures. We also bring down the performance measure values from the No-Cost model for comparison. These measures (superior measures are highlighted in yellow) are depicted in the table below.

Performance Measure Proportion of false positives Proportion of false negatives Overall error rate Overall model accuracy Sensitivity Specificity

10X Cost Value (163)/(297) = 0.5488 (27)/(761) = 0.0355 (163+27) / (1058) = 0.1796 1 – (0.1796) = 0.8204 (134)/(161) = 0.8323 (734)/(897) = 0.8183

No Cost Value (9)/(103) = 0.0874 (67)/(955) = 0.0702 (9+67) / (1058) = 0.0718 1 – 0.0708 = 0.9292 (94)/(161) = 0.5839 (888) / (897) = 0.9900

Table 3.1 Performance measures for 10x cost model

Referring to the table above, we observe the No-Cost model has superior Proportion of false positives, but the 10X model has superior Proportion of false negatives. This is a direct result of assigning a misclassification cost to the false negative predictions. In other words, assigning the misclassification cost causes the algorithm to be more judicious when making a negative prediction thus resulting in a lower proportion of false negatives (since they are 10x more expensive than false positives). This also explains why the 10X model has superior Sensitivity as this model will predict positives much more accurately.

However, we also observe that assigning misclassification costs will also negatively impact the overall accuracy of the model. We observe that the No Cost model has superior model accuracy / overall error rate. Finally, assigning misclassification will also negatively impact model Specificity. We observe that although the 10X model has superior Sensitivity, it gains this at the price of inferior Specificity.

7 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

4. Perform a cost/benefit analysis for the default CART model from exercise 1 as follows. Assign a cost or benefit in dollar terms for each combination of false and true positives and negatives, similar to Table 14.4. Then, using the contingency table, find the overall anticipated cost.

We perform a cost-benefit analysis by leveraging the fact that the cost of a false negative is assumed to be (from the previous exercise) 10X more expensive than the cost of a false positive. We also assume that all customers we identify as churners will stay if we intercede. Therefore, all positive predictions will have a cost of say $1, true negative predictions will have a cost of 0 as no action is needed, and false negative predictions will have a cost of $10. We then calculate the cost of the No-Cost model as follows: No-Costs = ($1)(103) + (0)(888) + ($10)(67) = $103 + $670 = $773

5. Perform a cost/benefit analysis for the CART model with the adjusted misclassification costs. Use the same cost/benefits assignments as for the default model. Find the overall anticipated cost. Compare with the default model, and formulate a recommendation as to which model is preferable.

We perform a comparable cost-benefit analysis by leveraging the fact that the cost of a false negative is assumed to be (from the previous exercise) 10X more expensive than the cost of a false positive. We also assume that all customers we identify as churners will stay if we intercede. Therefore, all positive predictions will have a cost of say $1, true negative predictions will have a cost of 0 as no action is needed, and false negative predictions will have a cost of $10. We then calculate the cost of the 10X model as follows: 10X Cost = ($1)(297) + (0)(734) + ($10)(27) = $297 + $270 = $549 We conclude that the 10X model is preferable since it is approximately one-third less expensive than the No-Costs model. We also note that it is critical to account for misclassification costs when they exist and to not rely completely on model accuracy as the primary means of model evaluation.

8 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

6. Construct a lift chart for the default CART model. What is the estimated lift at 20%? 33%? 40%? 50%?

We construct a Lift Chart for the No-Cost model as depicted in the figure below.

Figure 6.1 Lift Chart for no-cost CART model

9 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

7. Construct a gains chart for the default CART model. Explain the relationship between this chart and the lift chart.

We construct a Gains chart for the No-Cost model as depicted below.

Figure 7.1 Gains Chart for no-cost CART model

The concept of lift and gains are directly related. Lift is defined as follows: 𝑻𝑷 𝑻𝑷 + 𝑭𝑷 𝑳𝒊𝒇𝒕 = 𝑻𝑷 𝑻𝑷 + 𝑭𝑷 + 𝑻𝑵 + 𝑭𝑵

The ratio in the numerator is referred to as the proportion of positive hits (i.e. true positives) in the set of positive classifications (i.e. positive predictions). The ratio in the numerator is referred to as the proportion of positive hits (i.e. true positives) in the entire data set.

Gain is defined as cumulative lift. In other words, the gain at a particular percentile p is the summation of the Lift at each percentile up to and including p.

10 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

8. Construct a lift chart for the CART model with the adjusted misclassification costs.

What is the estimated lift at 20%? 33%? 40%? 50%?

We construct a Lift Chart for the 10X-Cost model as depicted in the figure below.

Figure 8.1 Lift Chart for 10x cost CART model

11 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

9. Construct a single lift chart for both of the CART models. Which model is preferable over which regions?

We construct a single Lift Chart (also referred to as a Combined Lift Chart) for both the No-Cost and 10X-Cost models using SPSS Modeler by routing the scoring stream through both models in an inline fashion as depicted in the figure below.

Figure 9.1 Model structure for Combined Lift Chart for both no-cost and 10x cost CART models

The Combined Lift chart itself is depicted in the figure below.

Figure 9.2 Combined Lift Chart for both no-cost and 10x cost CART models

Referring to the figure above, the 10X-Cost model has superior lift from 0 to 4th percentile, but the No-Cost Model has superior lift from the 5th to the 21st percentile.

Both models have comparable beyond the 21st percentile.

12 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

It makes sense that the No-Cost model has superior lift in general to the 10X-Cost model.

This is due to the fact that the 10X cost model attempts to reduce False Negative predictions, which in turn increases the number of False Positive predictions thus reducing hit accuracy (i.e. the proportion True Positives in the total set of positive predictions).

10. Now turn to a C4.5 decision tree model, and redo Exercises 1–9. Compare the results. Which model is preferable?

In lieu of the C4.5 algorithm availability, we use SPSS Modeler to construct a C5.0 model to predict churn using our Churn_Training.xlsx file. The structure used is depicted in the figure below.

Figure 10.1 Model structure for Combined Lift Chart for both no-cost and 10x cost C5.0 models

13 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

We then score the model using the Churn_Testing.xlsx file. The resulting Confusion Matrix is depicted below.

Figure 10.2 Confusion Matrix for no-cost C5.0 model

We then construct a C5.0 model applying a 10X cost for predicting False Negatives to predict churn using our Churn_Training.xlsx file, and score it using the Churn_Testing.xlsx file. The resulting Confusion Matrix is depicted below.

Figure 10.3 Confusion Matrix for 10x cost C5.0 model

14 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

We then use the values from the confusion matrices to calculate the various model performance measures. These measures (superior measures are highlighted in yellow) are depicted in the table below.

Performance Measure Proportion of false positives Proportion of false negatives Overall error rate Overall model accuracy Sensitivity Specificity

10X Cost Value (98)/(227) = 0.4317 (32)/(831) = 0.0385 (98+32)/1058 = 0.1229 1 – 0.1229 = 0.8771 (129)/(161) = 0.8012 (799)/(897) = 0.8907

No Cost Value (16)/(128) = 0.1250 (49)/930) = 0.0527 (16+49)/1058 = 0.0614 1 – 0.0614 = 0.9386 (112)/(161) = 0.6957 (881)/(897) = 0.9822

Table 10.1 Performance measures for both no-cost and 10x cost C5.0 models

Referring to the table above, we observe the No-Cost model has superior Proportion of false positives, but the 10X model has superior Proportion of false negatives. This is a direct result of assigning a misclassification cost to the false negative predictions. In other words, assigning the misclassification cost causes the algorithm to be more judicious when making a negative prediction thus resulting in a lower proportion of false negatives (since they are 10X more expensive than false positives). This also explains why the 10X model has superior Sensitivity as this model will predict positives much more accurately.

However, we also observe that assigning misclassification costs will also negatively impact the overall accuracy of the model. We observe that the No Cost model has superior model accuracy / overall error rate. Finally, assigning misclassification will also negatively impact model Specificity. We observe that although the 10X model has superior Sensitivity, it gains this at the price of inferior Specificity.

We perform a cost-benefit analysis by leveraging the fact that the cost of a false negative is assumed to be (from the previous exercise) 10X more expensive than the cost of a false positive. We also assume that all customers we identify as churners will stay if we intercede.

Therefore, all positive predictions will have a cost of say $1, true negative predictions will have a cost of 0 as no action is needed, and false negative predictions will have a cost of $10.

We then calculate the cost of the No-Cost model as follows: No-Costs = ($1)(128) + (0)(881) + ($10)(49) = $128 + $490 = $618 We then calculate the cost of the 10X model as follows: 10X Cost = ($1)(227) + (0)(799) + ($10)(32) = $227 + $320 = $547 We conclude that the 10X model is preferable since it is approximately one-tenth less expensive than the No-Costs model. We also note that it is critical to account for 15 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

misclassification costs when they exist and to not rely completely on model accuracy as the primary means of model evaluation.

We construct a Lift Chart for the No-Cost model as depicted in the figure below.

Figure 10.4 Lift Chart for no-cost C5.0 model

We construct a Gains chart for the No-Cost model as depicted below.

Figure 10.5 Gains Chart for no-cost C5.0 model

16 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

We construct a Lift Chart for the 10X-Cost model as depicted in the figure below.

Figure 10.6 Lift Chart for 10x cost C5.0 model

We construct a single Lift Chart (also referred to as a Combined Lift Chart) for both the No-Cost and 10X-Cost models using SPSS Modeler by routing the scoring stream through both models in an inline fashion as depicted in the figure below.

Figure 10.7 Model structure for Combined Lift Chart for both no-cost and 10x cost C5.0 models

17 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

The Combined Lift chart itself is depicted in the figure below.

Figure 10.8 Combined Lift Chart for both no-cost and 10x cost C5.0 models

Referring to the figure above, the 10X-Cost model has superior lift from 0 to 4th percentile, but the No-Cost Model has superior lift from the 5th to the 12th percentile.

Both models have comparable beyond the 12th percentile.

18 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

11. Next, apply a neural network model to predict churn. Calculate the following measures. a. Proportion of false positives. b. Proportion of false negatives. c. Overall error rate. d. Overall model accuracy (1 – overall error rate). e. Sensitivity f. Specificity In order to apply a neural network model to the churn data, the data must be prepared such that all continuous predictors are standardized to the same scale, and the two dichotomous categorical variables must be converted to binary variables (i.e. 1 and 0). The model structure used is depicted in the figure below.

Figure 11.1 Model structure for Min-Max normalized predictors 10x cost C5.0 models

19 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

We train the neural network model using Churn_Training.xlsx file and then score it using Churn_Testing.xlsx. The resulting Confusion Matrix is depicted in the figure below.

Figure 11.2 Confusion Matrix for neural network model

We then use the values from the confusion matrix to calculate the various model performance measures. These measures (superior measures are highlighted in yellow) are depicted in the table below.

Performance Measure Proportion of false positives Proportion of false negatives Overall error rate Overall model accuracy Sensitivity Specificity

Neural Network Value (29)/(122) = 0.2377 (68)/(936) = 0.0726 (29+68)/1058 = 0.0917 1 – 0.0917 = 0.9083 (93)/(161) = 0.5776 (868)/(897) = 0.9677

Table 11.1 Performance measures for neural network model

20 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

12. Construct a lift chart for the neural network model. What is the estimated lift at 20%? 33%? 40%? 50%?

We construct a Lift Chart for the Neural Network model as depicted in the figure below.

Figure 12.1 Lift Chart for Neural Network model

13. Construct a single lift chart which includes the better of the two CART models, the better of the two C4.5 models, and the neural network model. Which model is preferable over which regions?

We construct a single Lift Chart (also referred to as a Combined Lift Chart) for the CART No-Cost, the C5.0 No-Cost, and the Neural Network models using SPSS Modeler by routing the scoring stream through all three models in an inline fashion as depicted in the figure below.

Figure 13.1 Model structure for Combined Lift Chart

21 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.

Discovering Knowledge in Data 2nd Edition Daniel T. Larose, Ph.D. & Chantal D. Larose, Ph.D.

The Combined Lift chart itself is depicted in the figure below.

Figure 13.1 Combined Lift Chart for Neural Network, C5.0, and CART models

Referring to the figure above, the Neural Network model has superior lift from 0 to 3rd percentile, the C5.0 Model has superior lift from the 4th to the 22nd percentile. Both the CART and C5.0 models have comparable lift (superior to the Neural Network model) beyond the 22nd percentile.

14. In view of the results obtained above, discuss the overall quality and adequacy of our churn classification models.

In terms of lift, the models that do not apply misclassification costs actually have higher lift. However, since the models that do apply misclassification costs have a lower overall cost (i.e. the loss is much less with these models), the models with the highest overall adequacy and quality would be those with the lower overall lift. This is due to the fact that predictive models that minimize loss will typically have lower lift because emphasis is on reducing the number of False Negative predictions.

22 Discovering Knowledge In Data: An Introduction to Data Mining, By Daniel T. Larose & Chantal D. Larose Copyright © 2014 John Wiley & Sons, Inc.