HYBRID DIMENSIONALITY REDUCTION METHOD USING KAISER COMPONENT ANALYSIS AND INDEPENDENT COMPONENT ANA

GRD Journals | Global Research and Development Journal for Engineering | International Conference on Innovations in Engineering and Technology (ICIET) - 2016 | July 2016

e-ISSN: 2455-5703

Hybrid Dimensionality Reduction Method Using Kaiser Component Analysis and Independent Component Analysis 1K.

Arunasakthi 2J.Sulthan Alikhan 1,2 Assistant Professor 1,2 K.L.N. College of Engineering, Pottapalayam, Sivagangai 630612, India Abstract Dimensionality Reduction is the process of extracting the more relevant information. Conventional dimensionality reduction is categorized into two methods like Stand alone and Hybrid method. Standalone dimensionality reduction reduces the dimensions based on a single criterion whereas Hybrid method combines two or more criterion. In this paper, we proposed new hybrid method for dimensionality reduction Using Kaiser Component Analysis (KCA) and Independent Component Analysis (ICA). Kaiser component Analysis extracts the uncorrelated information and Independent Component Analysis maximizing the Independency among the data. The hybrid method using these Kaiser Component Analysis and Independent Component Analysis achieves both correlations and Independency among the Information and it is applied on SVM classification. The result improves the accuracy of the classification.

Keyword- Dimensionality reduction, Kaiser Component Analysis, Independent Component Analysis, Stand alone and Hybrid methods, SVM classification. __________________________________________________________________________________________________

I. INTRODUCTION A. Dimensionality Reduction Dimensionality reduction is a process of extracting the essential information from the data. The high-dimensional data can be represented in a more condensed form with much lower dimensionality to both improve classification accuracy and reduce computational complexity.

Due to the increasing demand for high-dimensional data analysis from various applications such as electrocardiogram signal analysis and content-based image retrieval, dimensionality reduction becomes a viable process to provide robust data representation in relatively low-dimensional space [2].

Dimensionality reduction is an important pre-processing step in many applications of data mining, machine learning, and pattern recognition, due to the so-called curse of dimensionality.

B. Need for Dimensionality Reduction High-dimensional dataset presented many mathematical challenges. One of the problems with high-dimensional datasets is that, in many cases, not all the measured variables were important to understand the problem. The main purpose of feature selection is to reduce the number of features used in classification while maintaining acceptable classification accuracy. Less discriminatory features were eliminated, leaving a subset of the original features which retained the sufficient information to discriminate well among classes. Feature extraction is a more general method in which the original set of features is transformed to provide a new set of features.

In mathematical terms, the problem we investigate can be stated as follows: given the p-dimensional random variable X = (x1 . . . . xp) , and a lower dimensional representation of it, S = (s1. . . . . sk), that captures the content in the original data, according to some criteria.

C. Dimensionality Reduction methods Dimensionality reduction reduces the number of variables to improve the performance of the classification. High dimensional data is the major problem in many applications which increase the complexity by taking the more execution time.

Conventionally dimensionality reduction is categorized into two methods: Stand Alone method and Hybrid method. In standalone method, dimensionality reduction was done by using single criteria but in the case of hybrid approach dimensionality reduction is achieved based on two or more criteria.

There are number of techniques available for dimensionality reduction. Each and every technique reduces the dimensions of the data based on particular criteria. In recent years, Principal Component Analysis (PCA) and Linear Discriminant Analysis

All rights reserved by www.grdjournals.com

415

Hybrid Dimensionality Reduction Method Using Kaiser Component Analysis and Independent Component Analysis (GRDJE / CONFERENCE / ICIET - 2016 / 069)

(LDA), Independent Component Analysis (ICA) are regarded as the most fundamental and powerful tools of dimensionality reduction for extracting effective features from high-dimensional vectors of input data.

1) Principal Component Analysis (PCA) Principal Component analysis is one of the mostly used feature extraction technique works based on assumption that all the observations are real valued and Euclian vector. Principal component analysis (PCA) is a relatively old and well developed linear independent feature extraction technique which has been applied to several pattern recognition tasks. Pattern recognition tasks are divided into two phases; feature analysis and classification. Feature extraction is a part of the feature analysis phase where we attempt to reduce redundancy in the feature vectors.

2) Independent Component Analysis (ICA) Independent Component Analysis (ICA) is the feature extraction technique recently used in many applications like audio age classification, speaker recognition, and hyper spectral data classification. In [14], the author developed an unsupervised algorithm for Independent Component Analysis.

Since data independence represents a stronger optimization criterion than correlations, the maximum independence in ICA provides more intrinsic information resulting generally in contributing more to improve the performance with robustness than PCA [2].

Recently, many authors tried to improve the performance by combining the results of more than techniques [16-21]. They achieved good results on hybrid methods rather than the stand alone methods. Chuxiong Miao, et.al [4] introduced Principal Component Analysis(PCA) as as feature extraction technique to reduce the computational complexity of classification, and it is evaluated on SVM classification. Amiri-Simkooei,et.al[7] used a new technique of Principal Component Analysis(PCA) for reducing the dimensions of the data to improve the accuracy of the classification. Xu Chunming,et.al[6] used Principal Component Analysis(PCA) for handling feature extraction on two dimensional data vectors. Senthilnath,J,et.al[11] used the Principal Component Analysis (PCA) to find the physical structure by reducing the dimensions of the hyper spectral images.

Kamath,et.al[13] addressed high dimensional data is the major problem in classification. To improve the accuracy of the classification, they reduce the dimensions of the data by using Independent component Analysis (ICA). Van De Ville, et.al [12] applied dimensionality reduction on Nonlocal Means (NLM) denoising method. To improve the accuracy, Independent Component Analysis (ICA) is used to reduce the dimensionality of the data. Taiping Zhang,et.al[16] addressed that, Linear Discriminant Analysis(LDA) is not suitable for small size data. To overcome this problem, they introduced a hybrid method using Principal Component Analysis (PCA) and LDA, which improved the classification accuracy. S. Moon,et.al[2] combined the supervised and unsupervised technique to form a new hybrid method using Support Vector Machine(SVM) and Independent Component Analysis(ICA). From above papers, it comes to know that, there are many techniques available for dimensionality reduction. Most of the techniques are used as standalone method to achieve good performance. Recently, many authors proposed hybrid method to further improve the performance. Principal Component Analysis(PCA) is the most widely used feature extraction technique, has been applied on different applications and Independent Component Analysis(ICA) is recently used in most of the applications. In this paper, we proposed a new hybrid method for dimensionality reduction using Kaiser Component Analysis and Independent Component Analysis.

II. DESIGN AND METHODOLOGY A. Architecture for Hybrid Dimensionality Reduction Using Kaiser Component Analysis (KCA) and Independent Component Analysis (Ica)

Fig. 1: Architecture for hybrid dimensionality reduction method using Kaiser Component Analysis and Independent Component Analysis

All rights reserved by www.grdjournals.com

416

Hybrid Dimensionality Reduction Method Using Kaiser Component Analysis and Independent Component Analysis (GRDJE / CONFERENCE / ICIET - 2016 / 069)

Figure 1 shows the architecture for Hybrid dimensionality method using Kaiser Component Analysis (KCA) and Independent Component Analysis (ICA). In this paper, the process is divided into five modules. They are,  SVM Classification  Standalone method  Hybrid Method (KCA + ICA)  Performance Analysis This method has been implemented on three high dimensional dataset, which has large number of attributes. They are, Insurance bench mark dataset consisting of 85 attributes and 750 records, Spam dataset contains 57 attributes and 4600 instances and Cancer dataset consisting of 57 attributes and 26 instances. These datasets are effectively processed by the SVM classification and Hybrid reduction method using Kaiser Component Analysis (KCA) and Independent Component Analysis (ICA).

1) SVM Classification The Support Vector Machine is the classification technique which classifies the high dimensional data directly. SVM is a powerful classifier which maps the input onto a high dimensional space and then finds an optimal hyper plane to separate the data in that space. The optimal hyper plane is found by maximizing the distance of the closest patterns.

Suppose we have a binary classification problem, where each example belongs to either class +1 or -1. SVM seeks to maximize the margin between the two classes by finding the separating hyper plane which lies halfway between the data classes.

In the case of non- linear data, the data are transformed by some non-linear transformation onto a higher dimensional space.

In general the optimal hyper plane is selected by using eqn (4.1), Y= WX + b (4.1) Where, W = weight for each vector x= data vector b= bias There are number of hyper planes among the data. Equation 4.1 selects the optimal hyper plane by maximizing the margin between the plane and the support vectors. Thus the hyper plane separates the data into different classes by maximizing the distance between the two different classes.

2) Standalone method for Dimensionality reduction The second module of this project is Standalone method. It involves reducing the dimensions of the data using single criteria to improve the performance of the Support Vector Machine classifier. Here, we choose two nonlinear techniques, Kaiser Component Analysis (KCA) and Independent Component Analysis (ICA).

Let us consider the sample data with two variables and ten instances.

B. Kaiser Component Analysis (KCA) There are many techniques available for dimensionality reduction. Among the techniques, Principal Component Analysis (PCA) is the most feasible and transformation technique and Kaiser Method of component selection improves the performance of PCA. The steps involved in Kaiser Component Analysis (KCA) are as follows: 1) Calculate the mean of the each attribute.

2) Subtract the mean from each data. so that, all data will be centralized.

3) Find the covariance data using the original data and the centralized data.

4) Find the eigen values and corresponding eigen vectors.

5) Arrange the vectors in decreasing order.

6) Select the components based on eigen values which are greater than 1.0 Using the above steps, the dataset will be transformed from high dimensional data into low dimensional data which represents the more important data.

C. Independent Component Analysis (ICA) Independent Component Analysis (ICA) is the unsupervised feature extraction technique, which has been applied on many applications. It transforms hte original data by using a transformation function. The model of the ICA is defined as, Y=s X Where, Y – Transformed data. s - Scalar matrix.

X – Original data.

Here, the original data is transformed into transformation data by using tanh transformation function as a scalar function.

The non-linearity among the data will be maximized and orthogonally for each data vector is achieved using this tanh transformation function. Selecting the number of Independent components is one of the important problem in ICA. The components which having greater than the 0.1 of average in the newly transformed data set.

All rights reserved by www.grdjournals.com

417

Hybrid Dimensionality Reduction Method Using Kaiser Component Analysis and Independent Component Analysis (GRDJE / CONFERENCE / ICIET - 2016 / 069)

3) Hybrid method In this section, we introduce hybrid method for dimensionality reduction which combines two or more standalone criteria.

Compared with the stand alone methods, hybrid method provides good performance in terms of accuracy and the execution time in SVM Classification. Here, PCA maximizes the decor relation among the features and ICA maximizes the independency among the data. In this project, we used the hybrid method of PCA + ICA. The principal components form the PCA is forwarded to ICA, which combines the features of both PCA and ICA. The performance of SVM shows the better result of PCA + ICA rather than the stand alone methods PCA and ICA. 4) Performance Analysis For this project, we select three datasets from UCI Repository named Insurance Benchmark, Spam and cancer dataset. The performance of SVM is measured by the Accuracy and elapsed time. PCA and ICA is applied on the original data and SVM is applied on the newly obtained low dimensional data.

The result of this standalone method improves the performance of the SVM. Then the result of PCA + ICA improves the performance of the SVM classification by reducing the dimensions from 85 to 16, 57 to 11 on Insurance and spam data sets respectively.

III. IMPLEMENTATION AND RESULT ANALYSIS In this project, the dimensionality reduction is done by hybrid method using Kaiser Component Analysis and Independent Component Analysis and it is applied on SVM classification in Matlab environment. The accuracy of the classifier is compared with the accuracy with both high and low dimensional data.

This project is done by and using three datasets from UCI Repository. One is Insurance BenchMark dataset, that contains 85 attributes and 750 records to analyze whether the person is eligible to get Insurance or not. Second, we used Spam dataset, which contains 57 attributes and 4600 records to analyse whether the data is spam or not. Third, cancer dataset contains 57 attributes and 26 instances for the analysis. Analyzing and computing all these high dimensional data is very difficult and not all these variables affect the result of the classification. So, we tried to remove the irrelevant data for this analysis. Though removing of attributes from the data takes less execution time, there may be a loss of some data which may affect the accuracy of the classification. Using these methods, the dimensionality of the data has been reduced and the performance is measured in terms of both Accuracy and Elapsed time. The code for SVM classification, PCA and ICA was done in matlab.

IV. RESULT ANALYSIS A. Result of SVM Classification When executing support vector machine on the original high dimensional data set, the performance of the SVM Classification is shown on table 1 DATASET

ACCURACY

ELAPSED TIME (in sec)

Insurance dataset

39.4667

1.6659

Spam

35.5217

0.2643

Cancer dataset

76.9231

1.2634

Table 1: Performance of SVM Classification

B. Results of Standalone methods 1) Kaiser Component Analysis To show the effectiveness of the dimensionality reduction, the high dimensional data is processed by the Kaiser Component Analysis. From this analysis 85,58, and 57 variables are transformed into 24,11 and 7 on Insurance, spam and cancer datasets respectively. The Kaiser components obtained from this analysis are shown in table 2 2) Independent Component Analysis Stand Alone DATA SET

No.of variables in HDD

Hybrid

Kaiser components

Independent Components

Kaiser PCA + ICA

Insurance dataset

85

24

15

15

Spam

58

11

2

2

All rights reserved by www.grdjournals.com

418

Hybrid Dimensionality Reduction Method Using Kaiser Component Analysis and Independent Component Analysis (GRDJE / CONFERENCE / ICIET - 2016 / 069)

Cancer dataset

57

7

15

7

Table 2: Dimensions of data after transformation

C. Result of Hybrid method KCA and ICA The hybrid approach is designed by combining the results of both Kaiser component Analysis and Independent Component Analysis. By using this Hybrid method, the high dimensional data is reduced into low dimensional data having 15,2 and 7 variables on Insurance, Spam and cancer datasets respectively.

D. Performance Analysis STAND ALONE METHOD KCA

HYBRID ICA

KCA + ICA

DATASETS Dimension

Accuracy

Time

Dimension

Accuracy

Time

Dimension

Accuracy

Time

Insurance (85)

24

52

1.54s

15

49.86

1.21s

15

58.93

1.65

Spam (57)

11

72.82

5.05s

2

70.08

1.14s

2

74.43

1.21

Cancer (58)

7

38.46

0.01s

15

61.53

0.09s

7

69.23

0.009

Table 3: Performance Analysis

From the table 3, it comes to know that the accuracy of hybrid method is better than that of the standalone methods and elapsed time may vary time to time based on the system performance. The graphical representation of Dimensionality reduction and its performance is shown in the fig 5.1 and 5.2.

The second technique in this project is Independent Component Analysis. First, ICA is directly applied on the original High dimensional data vector. By doing this analysis, we got 15,2 and 15 independent components for Insurance, spam and Cancer datasets respectively. The details are shown in table 5.2 The SVM classification is again processed with these low dimensional data obtained from both stand alone and hybrid methods. When we feed Principal components into SVM, we got better performance than that of the SVM with high dimensional data. The performance of the SVM is measured by using two metrics like Accuracy and Elapsed time. Accuracy is found using confusion matrix in Matlab. The Performance details are shown in table 5.3 From Figure 2, its clearly comes to understand that the high dimensional data is transformed into low dimensional data using standalone methods and its further reduced by the hybrid approach.

Insurance

85 57

Cancer

58

24 11

HDD

Spam

KCA

15

7 15 2 ICA

7 15 2

KCA+ICA

Fig. 2: Comparison on dimension of KCA, ICA and KCA + ICA

Insurance

Spam

Cancer

85.5217 76.923172.82 74.43 70.08 61.5358.93 69.23 52 49.86 38.46 39.4667

HDD

KCA

ICA

KCA+ICA

Fig. 3: Comparison on Accuracy of KCA, ICA and KCA + ICA

All rights reserved by www.grdjournals.com

419

Hybrid Dimensionality Reduction Method Using Kaiser Component Analysis and Independent Component Analysis (GRDJE / CONFERENCE / ICIET - 2016 / 069)

Figure 3 shows that, improvement on accuracy with the transformed low dimensional data. For example, In insurance dataset, the accuracy of high dimensional data is 39.4647. But we can get the better accuracy of 52, 49.86 and 58.93 with the low dimensional data from KCA, ICA and KCA+ ICA respectively. The Screen shots of this projects is shown in Appendix:

V. CONCLUSION & FUTURE ENHANCEMENT In this paper, we used Hybrid approach using Kaiser Component Analysis(KCA) and Independent Component Analysis(ICA) for dimensionality reduction and is successfully applied on SVM classification with three datasets like Insurance Benchmark, Spam and cancer datasets. It can be further enhanced by implementing hybrid techniques using various Machine Learning techniques like Principal Component Analysis, Linear Discriminant Analysis, Support Vector Machine, Independent Component Analysis, etc., to get better results. The dimensionality reduction can also be applied to the regression problem similar to classification problems.

REFERENCES [1] Jun Li ; Dacheng Tao ,”Simple Exponential Family PCA”, IEEE Transactions on Neural Networks and Learning Systems, Volume: 24 , Issue:3 Publication Year: 2013 , Page(s): 485 - 497 [2] S. Moon and H. Qi ,“ Hybrid dimensionality reduction method based on Support vector machine and independent component analysis", IEEE Transaction on Neural Networks, volume. 23, pages.749 -761 year 2012 [3] Qi Ding ; Kolaczyk, E.D. , “ A Compressed PCA Subspace Method for Anomaly Detection in High-Dimensional Data “,IEEE Transactions on InformationTheory,Volume:59, Publication Year: 2013 ,Page(s):7419 – 7433 [4] Chuxiong Miao ; Yu Wang ; Yonghong Zhang ,” A SVM classifier combined with PCA for ultrasonic crack size classification“ , International conference on Computer Engineering, 2010, Page(s): 001627 – 001630 [5] N. Kwak , “Principal component analysis based on L1-norm maximization", IEEE Transaction on Pattern Analysis and Machine Intelligence, volume. 30, no. 9, pages.1672 -1680 Year : 2008 [6] Xu Chunming ; Jiang Haibo ; Yu Jianjiang ,” Robust two-dimensional principle component analysis “, IEEE transaction on signals and control Publication Year: 2010 , Page(s): 452 - 455 [7] Amiri-Simkooei, A.R.; Snellen,M. ; Simons, D.G., “ Principal Component Analysis of Single-Beam Echo-Sounder Signal Features for Seafloor Classification” ,IEEE Transaction on Oceanic Engineering, Volume: 36, Issue: 2 Publication Year: 2011 , Page(s): 259 – 272 [8] Ran He; Bao-Gang Hu; Xiang-Wei Kong ,"Robust Principal Component Analysis Based on Maximum Correntropy Criterion", IEEE Transactions on Image page(s): 1485 - 1494 , Issue: 6, June 2011 [9] J. Zhao , P. Yu and J.Kwok “ Bilinear probabilistic principal component analysis", IEEE Transaction on Neural Networks and Learning system., volume. 23, no. 3, pages.492 -503, publication year: 2012 [10] Dobry, G. ; Hecht, R.M. ; Avigal, M. ; Zigel, Y., “ Supervector Dimension Reduction for Efficient Speaker Age Estimation Based on the Acoustic Speech Signal”, IEEE Transactions on Audio, Speech, and Language Processing, Issue: 7 Publication Year: 2011 , Page(s): 1975 - 1985 [11] Senthilnath, J. ;Omkar, S.N. ; Mani,”Crop Stage Classification of Hyperspectral Data Using Unsupervised Techniques”, IEEE Transcation on Applied Earth Observations and Remote Sensing, Volume: 6, Issue: 2 , Part:3 Publication Year: 2013 , Page(s): 861 - 866 [12] Van De Ville, D. ; Kocher, M. , “Nonlocal Means With Dimensionality Reduction and SURE-Based Parameter Selection“, IEEE Transactions on Image Processing, Volume: 20 , Year: 2011 , Page(s): 2683 - 2690 [13] Kamath, Sunil ; Ravindran, S. ; Anderson, D.V., ”Independent Component Analysis for audio classification“ IEEE Transactions on Digital Signal Processing Year: 2009, Page(s): 352 355 [14] S. Dhir and S.-Y. Lee, “ Discriminant independent component analysis “, IEEE Transaction on Neural Networks, volume. 22, pages : 845 -857 2011

All rights reserved by www.grdjournals.com

420