STAT 306 - Finding Relationships in Data 1-12 questions and answers University of British Columbia

STAT 306 - Finding Relationships in Data 1-12 questions and answers University of British Columbia Links: Index, Third Year, Fall 2025 Term 1, STAT 306 - Finding Relationships in Data

#classnotes

Subject: MATH307 MATH340

#STAT306

PHYS119

Topic::

Slide 18 - Principal Component Analysis (PCA) We want to transform a data frame with many columns into one with fewer columns.

With high dimensional spaces, visualization, model fitting and distance metrics (SD, Var) become problematic To get around this, there are strategies such as regularization and dimensionality reduction.

PCA

In this example, we drop magnesium and we project the points onto a horizontal line, we can instead project it into other lines, planes, hyperplanes -> We project onto a line where the variability is the highest so that we don't lose out on information when we move to lower dimensions, and retain the most information as possible.

Ide a This s t u d y s o u r c e was downloaded by 100000900134793 from CourseHero.com on 01-10-2026 12:10:17 GMT -06:00

I want to “compress” my data in lower dimension in such a way that if I were to try to recover https://wm ww m/a fi tlea/25f3r5o6m 3930t /h ST y.coourrisgehienroa.clod eAcT-o3 0m6 - p1 - r1 e2 psd fs/ ed data, the reconstructed points would be as close as

possible to the original points. <=> I want to bring my data to a lower dimension, but I want to keep the most amount of information (variance) as possible. Let’s find directions of maximum variance

When going from the compression point to the original point, the reconstructed point loses information due to the loss of the 2 variables.

Formalizing the problem

We want to maximize the variance of the projection of X onto the direction of highest variance.

u is the direction of the line that we want to project onto

This study source was downloaded by 100000900134793 from CourseHero.com on 01-10-2026 12:10:17 GMT -06:00 https://wewgw..coursehero.com/file/253563930/STAT-306-1-12pdf/

Since this vector is scalable, we want to constrain it so the problem is bounded.

-> This problem is unbounded, so we have to constraint this problem - constrain u to be unit length

-> The Var of the transformed variable is the eigenvalue of Σ The first principal component has higher variance <=> it captures the most information

This study source was downloaded by 100000900134793 from CourseHero.com on 01-10-2026 12:10:17 GMT -06:00 https://www.coursehero.com/file/253563930/STAT-306-1-12pdf/

-> Since we want to maximize the var ie. the total information, it will always be higher than the variance of a single variable. In lower dimensional representation (PCA), we select the first "few" components that has the highest variable and discard the rest. Therefore Z is basically the maximum variance sum of the first "few" component.

The transformation Z is not a linear regression, because we are taking the orthogonal distance from the point to the projected line (projection). Whereas linear regression, it is the vertical distance from the point to the projected line.

(1) (1) Z (1) = u X1 + ⋯ + u X 1

p

p

(1) here indicates the first principal component. u is 1 × p (2) would be the second principal component ie. the next principal component which captures the most information but less than (1).

This study source was downloaded by 100000900134793 from CourseHero.com on 01-10-2026 12:10:17 GMT -06:00

Overall Idea:

https: // ww w.c ou rseh ero.c om/file/253563930/STAT-306-1-12pdf/

-> In PCA, we want to find the unit vector in a unit circle that centers around the origin (0, 0) that captures the most information ie. the unit vector that maximizes the variance of the transformation that you get when you project the data points onto the unit vector.

Thus, we also have to center all the data points around the origin ie. the data must be meancentered

Lasso When we are fitting a model, we want to minimize the RSS. We want to minimize this at all costs, so we don't care how wild β's get.

-> LASSO introduces a penalty to keep the β's are in check

-> the λ can be interpreted as the budget, it determines how much you want to penalize the β's ->

C is the "budget" eg.

This study source was downloaded by 100000900134793 from CourseHero.com on 01-10-2026 12:10:17 GMT -06:00

-> The solution of LASSO will be on the vertices, thus some coefficients will be 0 but this isn't always the case, usually happens for higher dimensions.

-> LASSO selects variables for you to use -> Since LASSO penalizes the coefficients, we should ALWAYS scale/standardize the covariates before fitting LASSO.

-> When using LASSO on categorical variables, it could set one of the dummy variables ieasndtow(β vearor.icaombloen 0w nTe-0a6:n00d This stuc dyose oufrfciecw nl)oatdo ed 0 b y. 1T 00h 00u0s 9 00m 13e 47r9g3 ifn rog m tCh ouarsteH 1-i1t0h-20t2h6e12b :1 a 0:1s7eGliM

makes it hard to interpret

the model. The baseline is now the union of the original baseline and the dummy variable that was set to 0.

-> When we reduce λ ie the "budget", the absolute value of EACH coefficient doesn't always reduce, there are many ways to satisfy the "budget" so one coefficient could izncrease but the other may reduce while still satisfying the reduced budget. But the SUM of the absolute values will always reduce as the "budget" decreases.

-> LASSO restricts the solution to be within the "square" so it isn't always better than OLS where there isn't any restriction.

Post-selection Inference Double-dipping problem -> Using the same data for model selection and inference causing to be "overconfident"

eg.

We should not reject H0 -> We do not reject the null hypothesis in this case because since Y depends on the covariates and since the covariates are independent there are no true relationship between Y and the covariates so the coefficients β's are 0 -> H0 : βj = 0 is true in this case The probability of rejecting H0 should be 5% because the significance level is 5% -> When H0 is true, a correctly specified hypothesis test rejects 5% of the time.

Data splitting Splits the data into 2 -> "Search split": the data used to select your variables and fit the model -> "Inference split": the data used to compute the inference We sacrifice a large chunk of our data for search, which will increase the std. error in our inference step -> wider confidence intervals -> less powerful hypothesis tests