2026-2027 STAT 306 - Finding Relationships in Data 03-09 QUESTIONS AND ANSWERS University of British

Links: Index, Third Year, Fall 2025 Term 1, STAT 306 - Finding Relationships in Data 2026-2027 STAT 306 - Finding Relationships in Data 03-09 QUESTIONS AND ANSWERS University of British Columbia

#classnotes

Subject: MATH307 MATH340

#STAT306

PHYS119

Topic :: Simple linear regression (SLR)

Slide 1 - Relationships between two variables Deterministic relations: value of one variable is entirely determined by the value of another variable - there is no uncertainty.

Stochastic relations: situations in which the outcome of a given input cannot be precisely predicted - there is uncertainty.

This is due to some variables not being considered, or noise/error. eg. Relationship between height and weight - other factors such as age, lifestyle isn't considered and impacts the relationship.

Typically includes an error term (ϵ) to account for variability in the response/outcome.

Types of relations: Linear relation Quadratic relation Exponential relation

Y : response variable, output variable, NOT dependent variable X: predictors, feature, input variable, covariate, regressor, NOT independent variable

iClicker Proportion Z test - for proportions of 2 populations Paired t-test - determines the mean difference between 2 sets Two-sample t-test - test whether the unknown population means of two groups are equal or not.

Chi-square test - test whether the unknown population means of two groups are equal or not.

ANOVA - compare the means of two or more groups by analyzing variance.

A straight line model could be used as a reasonable approximation for a relation, this is called simple linear regression.

Slide 2 - Simple linear regression Explanatory variable (X) : the variable used to predict the response.

Response variable (Y ): what we are trying to predict/explain There is a probability distribution of Y for each value of X The means of these distributions are linearly relation to x x are constants/variables X is a random variable The explanatory variable x is assumed to be fixed for each individual/sample The model relating X and Y is given by:

Yi = β0 + β1xi + ϵi xi is the value of the explanatory variable for the ith sample (a scalar/constant) β0, β1 are parameters (constants) ϵi is a random error term (random variable) Yi the response variable for the ith sample (random variable) the distribution is based on the error term - which assumes to be normal β0 = intercept, value of Y when X = 0 β1 = slope, how much change in Y to expect for a unit increase in X ϵi = captures the variability of the response not explained by the model. has a distribution (assume to be normal), mean E[ϵ] = 0, variance (all errors ei have the same variance) The output from a model when given an input -> mean of the distribution based on the input and the standard the deviation of the input. It is not a singular number, it is the mean of the distribution of what the output could be.

Given that the E[ϵ] = 0, when trying to predict the value of the output given an input using the model, the output would follow the form "output + ϵ" but since the output is the mean of

the distribution this would result in "mean + 0".

The regression line is the conditional average of Y for a given value of X = x E[Y |X = x] = y = β0 + β1x

Yi = β0 + β1x + ϵi is a given point, which is off the line (because of the error term, each error term is different for each point)

We can estimate the values of X in a population using β0 and β1 aka β^ 0and β^1 with a sample.

Residual (ei)

ei = Yi − Y^i difference between the i-th observed point and the point on the regression line.

Residual sum of squares (RSS): sum of squared difference between the observed point and the expected point.

We want to find the values of β0 and β1 to minimize the RSS. Residuals are sensitive to outliers.

-> Outliers will "drag" the regression line towards the outlier -> increasing/decrease the slope and intercept -> the residuals are also greatly impacted.

Low-x, high-y outlier → higher intercept, smaller slope.

High-x, high-y outlier → lower intercept, larger slope.

Center-x, high-y outlier → mainly increases intercept.

We can find the values of β0 and β1 that minimizes the value of RSS using algebra. eg.

Interpretation of β0 "an increase of 1 unit in x is associated with an expected increase of β1 in y."

Y^ = β0^ + β1 ^ x: predicted value

We can't predict outside the range of data -> the relationship may not be linear outside the range, the value may be invalid (eg. negative heigh etc).

Correlation: the strength of linear association between two variables Linear regression: estimating the conditional average of the response given the value of the covariate/predictor variable We can encode categorical variables into multiple variables (dummy variables) eg.

thought we use them as estimators

Slide 3 - Least squares estimators

iClicker Given Yi = β0 + β1x1 + ϵi -> E[Yi] = β0 + β1xi -> V ar[Yi] = V ar[ϵi] = σ2 -> The distribution of Yi ~ N(β0 + β1x1, σ2) because of above -> Cov(Yi, Yj) for i ≠ j = Cov(β0 + β1xi + ϵi, β0 + β1xj + ϵj) = Cov(ϵi, ϵj) = 0 -> The distribution of Y¯ ~ N(β0 + β1 x̄ , nσ ) 2

-> Cov(Y , Y¯ ) = Cov(Y , 1 ∑ Y ) = 1 ∑ Cov(Y , Y ) = 1 Cov(Y , Y ) = 1 V ar(Y ) = σ

2

i

i n

j

n

i

j

n

i

i

n

i

n

E[β^1 ] = β1

1 is an unbiased estimator β1 We say that β^

V ar[β^1]

The more spread our X is in our sample data, less β^1 will vary from to sample -> smaller

V ar[β^1] will be.

E[β^0 ] = β0 β^0 is an unbiased estimator β0

V ar[β^0]

Cov[Ȳ , β^1 ] = 0

Slide 4 - Inference for Least Squares Estimators

iClicker The confidence interval for parameter θ is

θ^ ± c∗S^E(θ) As the sample size n increases, the width of the confidence interval decreases The null model is the sampling distribution of the test statistic when the null hypothesis is true.

An estimator for σ2 is given by the residual mean square

where RSS is the residual sum squares If we replace σ with σ^ , we obtain the estimate standard error of the least squares estimators

Chi-squared distribution:

t-distribution:

RSS 2 and is independent of β^0 and β^ 1 σ2 ≈ x n−2

of 1

Calculating the confidence interval for β1

of 1

When using σ^2, SE(β^ ) becomes 1

ˆ^ ) SE(β1

Interpretation: A 95% confidence interval for β1 means that if we were to take many samples and construct a confidence interval from each sample, approximately 95% of those intervals would contain the true value of β1 eg. We are 95% confident that 1mm increase in flipper length is associated with an expected increase in body mass between 47g and 53g.

-> We cannot interpret the slope far away from the range of data Hypothesis testing for β1

eg.

Hypothesis testing for β0

Slide 5 - Confidence Intervals Confidence interval of expected response:

ˆ Y new is an Y new is an estimator for the mean value of the response when X = x new ie. ˆ estimator for E[Y ∣ X = xnew] This is called the expected response

The width of the confidence interval for E[Y ∣ X = x]increasesas$xnew moves away from

x̄ .

Prediction: is a guess of the value of a new observation Ynew at a new given value xnew a prediction for a point xi, in our data is called a fitted value The regression line is an estimate of the conditional mean of Y for a given value X = x.

^ is both an estimator for the conditional mean and a predictor for Y new Y new Expected response != prediction

Prediction interval: is an interval that likely contains the value of a new observation Ynew at a given value X = x.

-> a statistical range that estimates where a single, future data point is likely to fall, based on past data and a given probability level This is different from the confidence interval for the expected response -> A confidence interval for the expected response is an interval of plausible values for the mean value of Y when X = xnew.

When interpreting the prediction interval -> We can say that there's a (1 − α) ∗ 100% probability that the interval will contain the value of a new observation Ynew at X = xnew.

Prediction intervals are wider than confidence intervals for the expected response because the uncertainty in estimating the mean response the variability of individual observations around that mean.

Var(Ynew) = Var(ϵ)

Slide 6 - Model Diagnostics and Final Remarks Model diagnostics -> residuals used to to model diagnostics, to assess how well the model fits the data Using Y to check the assumptions is hard because Y contains both the signal (the linear part) and the noise (the errors) If the model is appropriate -> errors ϵi should behave like i.i.d. observations from N(0, σ2). the errors are not observable, so we use the residuals ei as a proxy for the errors εi.

first constraint means that the residuals always have a mean of 0.

-> the sample mean of the residuals is not useful for checking the assumption E[εi] = 0. when the sample size is large in comparison to the number of parameters, the residuals behave similarly to the errors.

Model diagnostics using residual plots

Q-Q plot/normal quantile plot used to assess if a set of data follows a normal distribution or not.

When the model fits well, the residuals should be (approximately) independent of the fitted value/covariate.

-> This means that the plots of 1. ei against the fitted value y^i , or 2. ei against the covariate xi should not show any obvious pattern in the points, which are centered around zero If the relationship between Y and x is not linear, then the residual-vs-fitted value plot may show a curved pattern.

The scale of Y may obscure the non-linearity -> it may not be linear outside the range given Heteroscedastic: If the variance of the errors is not constant -> then the residual-vs-fitted value plot may show a “fanning” pattern.

eg.

Another useful plot for detecting heteroscedasticity is the absolute residuals against the fitted value

-> in this case, it shows a linear pattern instead of a "fanning" patter Q-Q plot: -> The normality assumption for the errors can be checked using a normal quantile plot of the residuals.

That small deviations from normality are not a big concern because the inference procedures (e.g., hypothesis tests and confidence intervals) based on the linear model are fairly robust to mild departures from normality, especially when the sample size is large.

-> Prediction intervals are more sensitive to non-normality.

Note that it is common to see some small deviations from normality in the tails of the normal quantile plot, even when the errors are normally distributed. ie. not as large sample size -> if the sample size is large and there are still small deviations, then it is not normally distributed.

Serial correlation (autocorrelation): refers to the correlations in the responses across time points, for when the covariate represents (most commonly) discrete time points -> the values of a variable are correlated with their own past values over time -> If there is no serial correlation, then plotting sequential residuals ei+1 against ei should show no obvious patterns.

eg.

-> there is a pattern -> there is serial correlation

SSE (RSS) = error sum of squares -> measures how well a model fits data SSR (MSS) = how much of the variation in the dependent variable Y is explained by the regression model.

SSTO (TSS) = measures the total variation in the dependent variable Y around its mean.

R2 = MSS/TSS = 1 - RSS/TSS: coefficient of determination -> measures the proportion of variability in the response Y that is explained by the linear model -> 0 ≤ R ≤ 1 -> reliant on the least squares method -> value close to 1 indicate a large proportion of the variability in Y is explained by the model, close to 0 model explains very little of the variability.

-> High value of R2 doesn't mean that the model is appropriate for the data Simpson's paradox: statistical phenomenon in which a trend or characteristic observed within individual data groups undergoes a reversal or disappearance eg. By controlling for species, the correlation between bill depth and body mass becomes positive

Everything in the course so far holds for a function of x.