ITEC 4230 Data Analyst Assignment 1a - York University

downloaded from

Scholarfriends

https://scholarfriends.com Downloaded by: writerforacademic@gmail.com | https://scholarfriends.com/singlePaper/305950/itec-4230-data-analyst-assignment-1a-yorkuniversity Distribution of this document is illegal.

1

ITEC 4230 Data Analyst Assignment 1a Haque Hussain

Student Number 216886673 Q (1) What is data mining? In your answer, address the following: (a) Is it another hype?

Data mining is the process of discovering interesting patterns from massive amounts of data. As a knowledge discovery process, it typically involves data cleaning, data integration, data selection, data transformation, pattern discovery, pattern evaluation, and knowledge presentation.

Data mining is not a hype. Loads of data which was up until now not used and utilized can be made efficient with the aid of data mining. Data mining give value and meaning to otherwise useless data.

(b) Is it a simple transformation or application of technology developed from databases, statistics, machine learning, and pattern recognition?

No. Data mining is more than a simple transformation of technology developed from databases, statistics, and machine learning. Instead, data mining involves an integration, rather than a simple transformation, of techniques from multiple disciplines such as database technology, statistics, machine learning, high-performance computing, pattern recognition, neural networks, data visualization, information retrieval, image and signal processing, and spatial data analysis.

(c) We have presented a view that data mining is the result of the evolution of database technology. Do you think that data mining is also the result of the evolution of machine learning research? Can you present such views based on the historical progress of this discipline? Address the same for the fields of statistics and pattern recognition.

Machine learning investigates how computers can learn (or improve their performance) based on data. A main research area is for computer programs to automatically learn to recognize complex patterns and make intelligent

2

decisions based on data. For example, a typical machine learning problem is to program a computer so that it can automatically recognize handwritten postal codes on mail after learning from a set of examples. Machine learning is a fast-growing discipline. I believe data mining is the result of the evolution of machine learning in the following ways1.Supervised learning 2. Unsupervised learning 3. Semi-supervised learning 4. Active learning Statistics studies the collection, analysis, interpretation or explanation, and presentation of data. Data mining has an inherent connection with statistics.

A statistical model is a set of mathematical functions that describe the behavior of the objects in a target class in terms of random variables and their associated probability distributions. Statistical models are widely used to model data and data classes.

For example, in data mining tasks like data characterization and classification, statistical models of target classes can be built. In other words, such statistical models can be the outcome of a data mining task. Alternatively, data mining tasks can be built on top of statistical models. For example, we can use statistics to model noise and missing data values. Then, when mining patterns in a large data set, the data mining process can use the model to help identify and handle noisy or missing values in the data. Statistics research develops tools for prediction and forecasting using data and statistical models. Statistical methods can be used to summarize or describe a collection of data. Statistics is useful for mining various patterns from data as well as for understanding the underlying mechanisms generating and affecting the patterns. Inferential statistics (or predictive statistics) models’ data in a way that accounts for randomness and uncertainty in the observations and is used to draw inferences about the process or population under investigation. Statistical methods can also be used to verify data mining results. For example, after a classification or prediction model is mined, the model should be verified by statistical

3

hypothesis testing. A statistical hypothesis test (sometimes called confirmatory data analysis) makes statistical decisions using experimental data. A result is called statistically significant if it is unlikely to have occurred by chance. If the classification or prediction model holds true, then the descriptive statistics of the model increases the soundness of the model.

Pattern analysis is useful in the analysis of spatiotemporal data, time-series data, image data, video data, and multimedia data. An area of spatiotemporal data analysis is the discovery of colocation patterns. These, for example, can help determine if a certain disease is geographically collocated with certain objects like a well, a hospital, or a river.

In time-series data analysis, researchers have discretized time-series values into multiple intervals (or levels) so that tiny fluctuations and value differences can be ignored. The data can then be summarized into sequential patterns, which can be indexed to facilitate similarity search or comparative analysis. In image analysis and pattern recognition, researchers have also identified frequently occurring visual fragments as “visual words,” which can be used for effective clustering, classification, and comparative analysis. Pattern mining has also been used for the analysis of sequence or structural data such as trees, graphs, subsequences, and networks. In software engineering, researchers have identified consecutive or gapped subsequences in program execution as sequential patterns that help identify software bugs. Copy-and-paste bugs in large software programs can be identified by extended sequential pattern analysis of source programs. Plagiarized software programs can be identified based on their essentially identical program flow/loop structures. Authors’ commonly used sentence substructures can be identified and used to distinguish articles written by different authors. Frequent and discriminative patterns can be used as primitive indexing structures (known as graph indices) to help search large, complex, structured data sets and networks. These support a similarity search in graph-structured data such as chemical compound databases or XML-structured databases. Such patterns can also be used for data compression and summarization.

Furthermore, frequent patterns have been used in recommender systems, where people can find correlations, clusters of customer behaviors, and classification models based on commonly occurring or discriminative patterns.

4

Finally, studies on efficient computation methods in pattern mining mutually enhance many other studies on scalable computation. (d) Describe the steps involved in data mining when viewed as a process of knowledge discovery.

The knowledge discovery process is as an iterative sequence of the following steps: 1. Data cleaning (to remove noise and inconsistent data) 2. Data integration (where multiple data sources may be combined) 3. Data selection (where data relevant to the analysis task are retrieved from the database) 4. Data transformation (where data are transformed and consolidated into forms appropriate for mining by performing summary or aggregation operations) 5. Data mining (an essential process where intelligent methods are applied to extract data patterns) 6. Pattern evaluation (to identify the truly interesting patterns representing knowledge based on interestingness measures 7. Knowledge presentation (where visualization and knowledge representation techniques are used to present mined knowledge to users)

Q (2) Suppose that a hospital tested the age and body fat data for 18 randomly selected adults with the following results: age %fat age %fat

23 23 9.5 26.5 52 54 34.6 42.5

27 7.8 54 28.8

27 17.8 56 33.4

39 31.4 57 30.2

41 25.9 58 34.1

47 27.4 58 32.9

49 27.2 60 41.2

(a) Calculate the mean, median, and standard deviation of age and %fat.

For Age, X

(X - X̄) ²

23

549.64

23

549.64

50 31.2 61 35.7

5

27

378.09

27

378.09

39

55.42

41

29.64

47

0.31

49

6.53

50

12.64

52

30.86

54

57.09

54

57.09

56

91.31

57

111.42

58

133.53

58

133.53

60

183.75

61

211.86

X

(X - X̄) ²

total sum

836

2970.444

N

18

18

mean = ΣX/n =

836.000 / 18 = 46.4444

sample variance =

Σ(X - X̄)²/(n-1)= 2970.4444 / 17 = 174.7320

sample std dev = √ [ Σ(X - X̄)²/(n-1)] = √ 174.7320 =

13.2186

6

Median=0.5(n+1)th value =

9.5 th value of sorted data

= 51.000 For %Fat, X

(X - X̄) ²

9.5

371.85

26.5

5.21

7.8

440.30

17.8

120.63

31.4

6.85

25.9

8.31

27.4

1.91

27.2

2.51

31.2

5.84

34.6

33.83

42.5

188.15

28.8

0.00

33.4

21.31

30.2

2.01

34.1

28.27

32.9

16.95

41.2

154.17

35.7

47.84

X

(X - X̄) ²

total sum

518.1

1455.945

N

18

18

7

mean = ΣX/n =

518.100 / 18 = 28.7833

sample variance =

Σ (X - X̄) ²/(n-1) = 1455.9450 / 17 = 85.6438

sample std dev = √ [ Σ (X - X̄) ²/(n-1)] = √ 85.6438 = Median=0.5(n+1) th value =

9.2544

9.5 th value of sorted data

= 30.700 (b)Draw the boxplots for age and %fat.

For Age, we can see that the minimum value is 23 and the maximum value is 61. Q1 is 36 and Q3 is 57.25. The median is 51. The box plot for AGE indicates that it is positively skewed.

8

For Fat, the minimum is 17.8 and the maximum is 42.5. Q1 is 26.35 and Q3 is 34.225. The median is 30.7. There are two outliers at 7.8 and 9.5. The box plot for FAT is almost normally distributed.

(c) Draw a scatter plot and a q-q plot based on these two variables.

Scatter Plot

9

Scatter plot indicates that age and fat are positively correlated.

Q-Q Plot

10

(b) Mean deviation=

197.3/18=10.96

Measure of skewness= X̄-MODE/S = (46.44-23)/13.21 =1.773

Coefficient of Variation=s/ X̄ *100

=13.21/46.44*100 =0.65

Q (3) It is important to define or select similarity measures in data analysis. However, there is no commonly accepted subjective similarity measure. Results can vary depending on the similarity measures used. Nonetheless, seemingly different similarity measures may be equivalent after some transformation.

Suppose we have the following 2-D data set: A1 A2 x1 1.5 1.7 x2 2 1.9 x3 1.6 1.8

11

x4 1.2 1.5 x5 1.5 1.0 (a) Consider the data as 2-D data points. Given a new data point, x = .1.4, 1.6/ as a query, rank the database points based on similarity with the query using Euclidean distance, Manhattan distance, supremum distance, and cosine similarity. (b) Normalize the data set to make the normof each data point equal to 1. Use Euclidean distance on the transformed data to rank the data points.

12

13

14

15

16

17

Q (5) Using the data for age given in Exercise 3.3, answer the following: (a) Use min-max normalization to transform the value 35 for age onto the range [0.0, 1.0].

18

Using the corresponding equation with minA = 13, maxA = 70, new minA = 0, new maxA = 1.0, then v = 35 is transformed to v ′ = 0.39.

(b) Use z-score normalization to transform the value 35 for age, where the standard deviation of age is 12.94 years.

Using the corresponding equation where A = 809/27 = 29.96 and σA = 12.94, then v = 35 is transformed to v ′ = 0.39.

(c) Use normalization by decimal scaling to transform the value 35 for age.

Using the corresponding equation where j = 2, v = 35 is transformed to v ′ = 0.35.

(d) Comment on which method you would prefer to use for the given data, giving reasons as to why.

Given the data, one may prefer decimal scaling for normalization as such a transformation would maintain the data distribution and be intuitive to interpret, while still allowing mining on specific age groups. Min-max normalization has the undesired effect of not permitting any future values to fall outside the current minimum and maximum values without encountering an “out of bounds error”. As it is probable that such values may be present in future data, this method is less appropriate. Also, z-score normalization transforms values into measures that represent their distance from the mean, in terms of standard deviations. It is probable that this type of transformation would not increase the information value of the attribute in terms of intuitiveness to users or in usefulness of mining results.

19

Q (6) Using the data for age and body fat given in Exercise 2.4, answer the following: (a) Normalize the two attributes based on z-score normalization. (b) Calculate the correlation coefficient (Pearson’s product moment coefficient). Are these two attributes positively or negatively correlated? Compute their covariance. (a)

(b) The correlation coefficient is 0.82. The variables are positively correlated.

Q (7) The following table consists of training data from an employee database. The data have been generalized. For example, “31: : : 35” for age represents the age range of 31 to 35.

For a given row entry, count represents the number of data tuples having the values for department, status, age, and salary given in that row. department status age salary count sales senior 31: : :35 46K: : :50K 30 sales junior 26: : :30 26K: : :30K 40 sales junior 31: : :35 31K: : :35K 40 systems junior 21: : :25 46K: : :50K 20

20

systems senior 31: : :35 66K: : :70K 5 systems junior 26: : :30 46K: : :50K 3 systems senior 41: : :45 66K: : :70K 3 marketing senior 36: : :40 46K: : :50K 10 marketing junior 31: : :35 41K: : :45K 4 secretary senior 46: : :50 36K: : :40K 4 secretary junior 26: : :30 26K: : :30K 6 Let status be the class label attribute. (a) How would you modify the basic decision tree algorithm to take into consideration the count of each generalized data tuple (i.e., of each row entry)? (b) Use your algorithm to construct a decision tree from the given data. (c) Given a data tuple having the values “systems,” “26 . . . 30,” and “46–50K” for the attributes department, age, and salary, respectively, what would a naive Bayesian classification of the status for the tuple be?

(a) The basic decision tree algorithm should be modified as follows to take into consideration the count of each generalized data tuple.

• The count of each tuple must be integrated into the calculation of the attribute selection measure (such as information gain).

• Take the count into consideration to determine the most common class among the tuples. (b) The resulting tree is as follows-

21

(salary = 26K...30K: junior = 31K...35K: junior = 36K...40K: senior = 41K...45K: junior = 46K...50K (department = secretary: junior = sales: senior = systems: junior = marketing: senior) = 66K...70K: senior) (c) P(X|senior) = 0; P(X|junior) = 31/113 × 46/113 × 20/113 = 0.018. Thus, a naive Bayesian classification predicts “junior”.