MATH MA 5790 Predictive Modeling - Michigan Technological University. Assignment-4. Q&A

downloaded from

Scholarfriends

https://scholarfriends.com Downloaded by: writerforacademic@gmail.com | https://scholarfriends.com/singlePaper/313372/math-ma-5790-predictive-modeling-michigantechnological-university-assignment-4-qa Distribution of this document is illegal.

Assignment-4 7.1. Simulate a single predictor and a nonlinear relationship, such as a sinwave shown in Fig. 7.7, and investigate the relationship between the cost, and kernel parameters for a support vector machine model. (a) Fit different models using a radial basis function and different values of the cost (the C parameter) and epsilon. Plot the fitted curve.

Cost of 1,2,4,8 and 16 is selected along with epsilon values 0.1,0.01,0.001,0.0001,0.00001

(b) The sigma parameter can be adjusted using the kpar argument, such as kpar = list(sigma = 1). Try different values of sigma to understand how this parameter changes the model fit. How do the cost, epsilon, and sigma values affect the model?

The Sigma values are also changed as 1,10,100,1000,10000.

The effect of cost is that when we increase the cost of the model, the error is high. Alternatively, if the cost is low, the error is low.

The effect of epsilon is that it represent the smoothness of the curve, the loss function is represented by epsilon. It considers the number of support vectors to make the decision boundary.

The effect of sigma represents the Bias-Variance tradeoff. As the sigma is high, the variance of the model is high and vice-versa.

7.2. Friedman (1991) introduced several benchmark data sets create by simulation. One of these simulations used the following nonlinear equation to create data: y = 10 sin(πx1x2) + 20(x3 − 0.5)2 + 10x4 + 5x5 + N(0, σ2) where the x values are random variables uniformly distributed between [0, 1] (there are also 5 other non-informative variables also created in the simulation). The package mlbench contains a function called mlbench.friedman1 that simulates these data Which models appear to give the best performance?Does MARS select the informative predictors (those named X1–X5)?

1. K Nearest Neighbors: Train Data:

Plot:

2. MARS Model: Train Data:

Plot:

3. Support Vector Machines: TrainData:

Plot:

4. Neural Networks Train Data:

Test Data Output: MODELS KNN

RMSE 3.1750657

R-SQUARED 0.6785946

MARS

1.6543157

0.8890979

SVM

4.8618512

0.2038278

NNET

2.6580454

0.7348028

The variable Importance for MARS is given as: earth variable importance Overall X2 100.00 X4 86.27

X1 68.87 X5 46.32 X3 35.88 X7 0.00 X6 0.00 X8 0.00 X10 0.00 X9 0.00 For X1-X5, the overall importance value is high and from X6 it becomes 0.

7.3. For the Tecator data described in the last chapter, build SVM, neural network, MARS, and KNN models. Since neural networks are especially sensitive to highly correlated predictors, does preprocessing using PCA help the model?

1. MARS Model: Train Data:

Plot:

2. Support Vector Machines Train Data:

Plot:

3. K Nearest Neighbors Train Data:

Plot:

4. Neural Networks (Without PCA): TrainData:

5. Neural Networks (AfterPCA): Train Data:

Test Data Output: MODELS MARS

RMSE 6.7668477

R-SQUARED 0.7497979

SVM

6.6704310

0.7041211

KNN

7.218743

0.656922

NNET (Without PCA)

0.7268305

0.9977028

NNET (After PCA)

11.2804439

0.1798604

After PCA, highly correlated variables are removed which decreases the overfit of the Neural Network model. The r-squared value reduces from 0.99 to 0.179.

7.4. Return to the permeability problem outlined in Exercise 6.2. Train several nonlinear regression models and evaluate the resampling and test set performance. (a) Which nonlinear regression model gives the optimal resampling and test set performance? (b) Do any of the nonlinear models outperform the optimal linear model you previously developed in Exercise 6.2? If so, what might this tell you about the underlying relationship between the predictors and the response? (c) Would you recommend any of the models you have developed to replace the permeability laboratory experiment?

1. MARS Model: Train Data:

Plot:

2. Support Vector Machines: Model:

Plot:

3. K Nearest Neighbors: Train Data:

Plot:

4. Neural Network Train Data:

(a) Which nonlinear regression model gives the optimal resampling and test set performance?

Train Data Output: MODELS MARS

RMSE 13.52928

R-SQUARED 0.3240359

SVM (sigma = 0.0017 C 4)

16.09471

0.05161469

KNN (K = 2)

11.63586

0.4788460

NNET (Without PCA)

4.2508863

0.9272996

NNET(After PCA)

14.9628748

0.1209436

Test Data Output: MODELS MARS SVM KNN NNET (Without PCA)

RMSE 8.7826831 11.764225 11.7923896 14.1935179

R-SQUARED 0.6361668 0.400227 0.4514292 0.4278676

NNET (After PCA)

13.5401775

0.2291902

When considering test set, it is seen that MARS model can be considered better good test set performance.

(b) Do any of the nonlinear models outperform the optimal linear model you previously developed in Exercise 6.2? If so, what might this tell you about the underlying relationship between the predictors and the response?

The PLS model has better R-squared value among linear models whereas the non-linear models such as MARS outperform better than PLS model. The underlying relationship between the response and the predictors is non-linear.

(c) Would you recommend any of the models you have developed to replace the permeability laboratory experiment?

MARS model can be suggested for the permeability dataset.