NEW WELL DETAILED COMPLETE COURSE NOTES FOR DATA1001 Foundations of Data Science at University of Sy

NEW WELL DETAILED COMPLETE COURSE NOTES FOR DATA1001: Foundations of Data Science at University of Sydney

1 Controlled Experiments

Method of comparison

RCT

Controlled experiment

Randomized double blind placebo controlled trial

Controlled experiment we control the effects of the other variables on the treatment

Data story | does anti-acne drug cause

e.g. age, gender, diet, etc.

depression Domain knowledge is required for accurately assessing the data. This can include chemistry, side effects, alternative drugs, etc...

Domain knowledge: ● Roaccutane is part of the retinoid family of drugs (vitamin-A) and known to affect the nervous system. ● Usually only prescribed to teenagers with severe cystic acne ● Highly effective and used by 13 million patients ● 1588 cases of suspected adverse events, including 25 suicides

There are 2 types of control

1. Historical control An individual treated in the past and used in a comparison group when researchers analyze the results of a clinical study that had no control group. The use of a control, or comparison, group helps researchers determine the effects of a new treatment more accurately.

2. Contemporaneous control groups Control groups occur at the same time as treatment group NB: Allocation of subjects should be randomised.

Important considerations ● Does depression cause acne -or- other way around? ● interrogate the 'cases' of suspected adverse events ● suicide truly caused by the drug

Confounding & bias Confounding

effects of two distinct variables

● compare suicide rate to baseline

mix, causing confusion

● Why does Roche have a warning about depression? (exclude liability?)

Bias

Something which affects the ability of data to accurately

Types of evidence Evidence must be accurately assessed for its reliability:

measure the treatment effect Confounding is caused by biases:

●

personal testimony is limited to generalised

●

selection

●

every stage of statistical study should be

●

observer

documented and checked in the review process

●

consent

●

reproducible research so that others can come to similar/same conclusions

Confounding occurs when the

Selection bias Assignment based on investigator's judgement leads to

Research Mice were tested with R. They were found to be less mobile. Researchers interpreted it as depression.

Is immobility a measure of depression?

Was the change significant?

Does depression in mice equate to human depression?

an non-randomised controlled experiment, which carries selection bias.

Observer bias

Observational Studies

Placebo

an effect from the subject (patient)

Observational

the investigator cannot control which

effect

thinking they have the treatment

Studies

subjects are assigned into treatment or control groups. (randomly or otherwise)

Double-blind

The experiment must ensure the subjects and investigators are unaware of any group's identity. The placebo

Many research questions cannot be done in a

must closely mirror the treatment and be

Randomised Controlled Trial because of ethics, logistics,

administered by a third party.

or otherwise. An observational study is useful.

Double blinding controls for the patient's

e.g. Unethical to make people smokers and non-smokers

expectations and the investigator's observations.

When either subject or investigator are aware of the

●

most research uses observational studies

●

conclusions must be taken more cautiously ○

people have self-selected into treatment & control groups

identity of the 2 groups there will be bias in the responses

■

OR evaluations.

e.g. vaccines and anti-vax

Subjects / investigators may deliberately or

Four Precautions

subconsciously report more / less favourable results

1 Establishing Causation

Consent bias

Observational studies cannot establish causation.

Occurs when subjects choose (or not) to take part in study.

To avoid consent bias, investigators can allocate subjects into groups. This removes the choice of subjects and allows for us to control their 'consent bias', essentially.

e.g. other words can be used such as: However: OS can establish association—which can suggest causation, but not prove.

Smoking can increase risk of cancer, but it doesn't necessarily cause it. Perhaps smokers tend to drink more alcohol, and alcohol consumption leads to

BUT: this raises ethical questions on withholding

cancer. The effect of smoking, here, is confounded

treatment for those in control or enforcing treatment for

with the effect of alcohol.

those in treatment.

2 Hidden confounders Confounding occurs when Treatment and Control groups are different due to a third variable (i.e. NOT independent or dependent).

Alcohol can be the confounding variable.

We think that smoking is the independent variable and cancer is the dependent variable, but alcohol consumption can be a confounding variable that affects the effects of the Independent.

Selection bias

How to deal with Confounding variables

The uneven allocation of subjects for the treatment vs

make groups more comparable by dividing into subgroups

control group, or even the study itself.

with respect to confounders.

e.g. for a certain surgery method, unsuitable

Alcohol on smoking We can use heavy, medium, and

candidates are screened out of the sample. This may

light drinkers to categorize subjects, allowing us to

give treatment groups an unusually high success rate

control for alcohol consumption.

as the surgery candidates were already 'healthy'.

Since confounding variables are sometimes difficult to

Survivor bias When certain types of subjects finish the study. e.g. the worst subjects drop out of a weight-watch program, thus making the 'treatment group' more

identify, the dividing strategy is limited. It also helps explain the lengthy time it took to establish smoking is behind the cause of lung cancer, as researchers had to control for: ●

health

●

fitness

●

diet

Adherer bias

●

lifestyle

Known also as compliance bias, it is when differences in

●

environment

successful than it should actually be.

adherence to treatment/intervention affects the study outcome.

People who are more compliant are more likely to be healthy, active, etc. Thus, the compliance level should be taken into account.

3 Simpson’s paradox1 Simpson’s

When a clear trend in an individual

paradox

group of data disappears or reverses when the groups are pooled together.

This can be both good and bad !!

High compliance levels in the treatment group can produce more accurate assessment of the success

Observational studies with a confounding variable can

of the treatment.

lead to Simpson's paradox.

However, the baseline compliance level of the general / target population should also be considered, as the treatment often works in non-ideal environments.

On the other hand, non-adherers can make the treatment group seem less successful than in actuality.

Adherer bias can make seemingly controlled studies become observational studies as investigators cannot control for adherence.

Whether or not we should aggregate the data depends on the process generating the data — that is, the causal model of the data.

Wisconsin vs Texas education Individually, the confounding variable of race comes into play. The trend shows that Texas > Wisconsin when race is taken into account.

When pooled together, the trend reverses and shows Wisconsin > Texas. This is because the numbers are not being weighted such that Wisconsin has disproportionately more white children.

Simpson’s paradox: How to Prove Op

1

posite Arguments with the Same

Smokers' mortality rate When simply comparing Smoking vs Mortality, the non-smokers seem to have it worse. However, when compared in age groups, the effect reverses because: There are substantially more older, non-smoking subjects. We have to weigh each age group separately to control for age to see the clear effects.

4 Historical control Control

subject who did not get treatment

Controlled

study where investigators allocate

experiment

subjects to two groups

Controlling for

trying to reduce the influence of

confounders

confounding variables

Studies which have a historical control, where time is a confounding variable, are not controlled experiments.

They might compare Drug 2.0 (new) on current patients vs Drug 1.0 (old) on past patients. ie: The Treatment vs Control groups differ in aspects more than simply treatment. graphs generated through RStudio, with data sourced from (https:// pubmed.ncbi.nlm.nih.gov/598014/)

NB: Controlled experiments must be performed contemporaneously.

2 Data & Qualitative Data Initial data analysis A first general look at data, without formally answering research questions. ●

determine whether data is appropriate for

Structure of data Variables measures or describes some attribute of the subjects: ➢ Data with p variables has dimensions p ➢ Types of data

answering questions ●

might pose other questions

Initial data analysis involves: ●

data background — quality and integrity?

●

structure—what information been collected

●

wrangling—scraping, cleaning, tidying, reshaping, splitting, combining

●

summary—graphical & numerical

Categories can include dice rolls, colours, etc…

Ordinal (ordered) ○

Socioeconomic status

○

Education level

○

Income bracket

Nominal (non-ordered) ○

Blood Type

○

Genotype, Blood type, Zip code, Gender, Race, Eye color, Political party

➢ Discrete Number of people Correct answers

➢ Continuous Time, Wavelength, Height

Double bar plot

Graphical summaries Graphical summary aims to best highlight features of this data. ●

Sometimes we use trial and error

●

Don't use pie charts

●

Use 'Shiny Apps' in third year

Qualitative data # Select the DayWeek variable from the whole data frame Dayweek = data$Dayweek # Produce a frequency table of fatalities per day of the week table(Dayweek) ## Dayweek ## Friday Thursday ## 58

Monday

Saturday

Tuesday Wednesday 68 50 67 58

85

Simple bar plot

Sunday

# Select DayWeek and Gender variables Dayweek = data$Dayweek Gender = data$Gender # 'data' is the dataset # '$' is the function # 'Gender' or 'Dayweek' is the variable you want to select # Produce a double frequency table (contingency table) data1 = table(Gender, Dayweek) data1

## Dayweek ## Gender Friday Monday Saturday Sunday Thursday Tuesday Wednesday ## Female 15 17 25 13 16 17 13 ## Male 53 33 60 43 42 50 45

56 barplot(data1, main = "Fatalities by Day of the Week and Gender", xlab = "Day of the week", col = c("lightblue", "lightgreen"), legend = rownames(data1))

barplot(table(Dayweek))

Line plot # A simpler version plot(table(Dayweek))

side-by-side bar plot barplot(data1, main = "Fatalities by Day of the Week and Gender", xlab = "Day of the week", col = c("lightblue", "lightgreen"), legend = rownames(data1), beside = TRUE

Graphical summaries of big data

Quantitative Data

Big data

massive amounts of data collected

Data story

from fields such as genomics,

Age of Road fatalities. Sometimes data is weird, such as

astrophysics, marketing &

negative age. We must refer to the data dictionary.

sociology.

Primitive cleaning ●

high dimensional ○

Instead of manually cleaning the data by altering the

more variables p than subjects

source data, import it into R studio and then clean it inside

genomics data has 3 billion variables

it. A data wrangling report will log how the data was

(base pairs)

cleaned.

Four Vs of Big Data Big data is described by the V ●

volume

●

velocity

●

variety

●

veracity

●

variability

●

vulnerability

●

volatility

●

value

Histogram Histogram is used for quantitative data. ●

There is only one class (e.g. age, height, grade)

●

Shows the frequency of data in each class (i.e. the distribution)

Big data requires complex visualisations

The data block for ~20 year olds is much more crowded than the middle aged group. # C is the cleaned version of data data <read.csv("data/2016FatalitiesC.csv",header=T) How to read histogram: ●

Each height represents crowding

●

Horizontal scale is divided into class intervals

●

The whole area adds up to 100%

●

Area of each block represents the percentage of subjects in each class.

Bar graph v Histogram ●

Bar graphs are used for qualitative categories (e.g. colours, car types)

●

Histograms display quantitative classes.

Choices with histogram 1. No need for vertical scale Density scale can be used when modelling is required later

2. Density scale calculation:

Common mistakes with histograms 3. Decide endpoint convention for data points ○

1. Making block heights equal to percentage

do you use ≥ or >

○

For example [0,18) and [18, 21)

○

Making raw percentage the height is incorrect ⇒ density must be used class intervals are usually unequal, thus balancing must occur

An 18yo would fit into the second interval because ) right-open

Histograms in RStudio #Read in data data <read.csv("data/2016FatalitiesC.csv",header=T) # Choose a variable Age <- data$Age # Choose the class intervals breaks=c(0,18,25,70,100)

2. Too many class intervals ○

# Produce a distribution table table(cut(Age,breaks,right=F)) # Produce a histogram hist(Age,br=breaks,freq=F,right=F, # freq=F produces histogram on density scale xlab="Age (in years)", ylab="% per year", # right=F makes intervals right-open main="Histogram for Age of Road Fatality in Australia: Jan-June 2016")

Controlling for a variable

AgeF <- data$Age[data$Gender=="Female"] This selects just the female ages.

AgeM <- data$Age[data$Gender=="Male"] par(mfrow=c(1,2)) # This puts the graphic output in 1 row with 2 columns hist(AgeF,freq = F) hist(AgeM,freq = F)

#

10–15 is fair

Box plot

Scatter plot

Box plot is used for comparing multipleAge <- data$Age

Examines the relationship between 2 quantitative values (e.g. age v speed limit)

summary(Age) ## Min. 1st Qu. Median Max. NA's ## 0.00 25.00 43.00 96.00 2 data sets.

Mean 3rd Qu. 44.85

61.00

SpeedLimit <- data$SpeedLimit plot(Age, SpeedLimit)

Creating box plot par(mfrow=c(1,2)) boxplot(Age) boxplot(Age, horizontal=T)

Comparative box plots A comparative box plot splits up a quantitative variable by a qualitative variable.

speedlimit should be a Qualitative variable here

Divide Age into 2 genders

Histogram by factor variable

Gender = data$Gender summary(Age[Gender == "Female"]) ## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's ## 0.00 25.50 44.00 47.19 69.00 93.00 1 summary(Age[Gender == "Male"]) ## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's ## 1.00 25.00 42.00 44.02 59.00 96.00 1

hist(USmelanoma$mortality[USmelanoma$ocean == "yes"]

Plotting box plot boxplot(Age~Gender, horizontal = T)

# displays of frequency against "mortality" for US states which have oceans

Data visualisation Data visualisation is the advanced version of graphical summaries.

E.g.: infographics, heat maps and networks.

Good data visualisation Good visualisations tell an interesting story in a visually appealing way. Aesthetically pleasing.

Data visualisation ggplot tutorial

ggplot2 | grammar of graphics ●

part of tidyverse package

●

looks more complicated than Base R graphics

●

more complex plots with layers.

What question are we trying to answer? What variables

aes

are we highlighting? What does the eye focus on? What is

Aesthetic is what we see

the effect of different colours?

Poor and bad data ●

●

chartjunk (poor)

●

Position

●

Outside colour

●

Fill colour

●

Shape of points

○

unnecessary information

●

Linetype

○

bad gradients

●

Size

○

visually boring/distracting

misleading, but appealing (bad)

ggplot2 First you must install ggplot2 install.packages("ggplot2") Then to load ggplot2 library(ggplot2) Checking the structure str(diamonds)

geom_< > Geometric objects are the actual marks on the plot ●

geom_point for points

●

geom_line for lines

●

geom_boxplot for boxplots

Steps to use ggplot 1. Install package 2. Check data is tidy head(data, 2) 3. Check classification of variables str(data) 4. Sketch by hand 5. Write the code and run ggplot

Barplots | qualitative 1 qualitative

p = ggplot(diamonds, aes(x = cut)) # Defines the x axis (1 variable).

Histogram | quantitative p1 = ggplot(diamonds, aes(x=price)) # Defines the x axis (1 variable). p1 + geom_histogram() # Represents the data by histogram. # Compare in Base R: hist(diamonds$price)

p + geom_bar() # Represents the data by bar chart.

Separating by qualitative p1 + geom_histogram(aes(fill=cut)) # Adds colour by a 2nd variable (cut).

2 qualitative p + geom_bar(aes(fill=clarity)) # Adds colour by a 2nd variable (clarity)

Separating by qualitative | side-by-side p1 + geom_histogram(aes(fill=cut),position = "dodge",binwidth=1000)

2 qualitative | side by side

p + geom_bar(aes(fill=clarity),position="dodge")

Scatterplot | 2 quantitative p2 = ggplot(diamonds, aes(x=carat, y=price)) # Defines the x and y axis (2 variables). p2 + geom_point() # Represents the data by points.

Box plot | quantitative Separating by qualitative p3 = ggplot(diamonds, aes(x=color, y=price)) # Defines the x and y axis (2 variables). p3 + geom_boxplot() # Represents the data by box plot.

Separating by qualitative p2 + geom_point(aes(colour = carat)) # Adds colour by a 3rd variable (carat).

Separating by qualitative | log scale p3 + geom_boxplot() + scale_y_log10() # Changes scale of y axis

Separating by 2 qualitative p2 + geom_point(aes(color=clarity, shape=cut)) # Adds colour by a 3rd variable (clarity), and shape by a 4th variable (cut). ## Warning: Using shapes for an ordinal variable is not advised

Separating by qualitative | facet qualitative 4 Normal Curve A normal curve is found

3 Centre Advantages of numerical summaries include:

hist(data$Sold[data$Type=="House" & data$Bedrooms=="4"], main="Newtown 4 Bedrooms", xlab="Price (in 1000s)")

➢ Reduces all of data to one simple number ➢ Major features

abline(v=mean(data$Sold[data$Type=="House" & data$Bedrooms=="4"]),col="green")

Notation for data Dataset of size n can be represented by:

Ranked dataset:

Sum of data:

Median Median is less affected by outliers. It’s the middle data

Mean & median Balancing point

unique point at which data is balanced.

• higher reading and lower readings cancel each other out

mean(data$Sold)

Filtered mean | 4 bedrooms mean(data$Sold[data$Type=="House" & data$Bedrooms=="4"]) ## [1] 2198.857

Mean Balancing point where the data values cancel each other out. hist(data$Sold, main="Newtown properties", xlab="Price (in 1000s)") abline(v=mean(data$Sold),col="green")

point. ●

Odd data sets ○

middle value

● Even data sets ○

average of 2 middles

median(data$Sold) ## [1] 1387.5

Filtered median | 4 bedrooms median(data$Sold[data$Type=="House" & data$Bedrooms=="4"]) ## [1] 1975

Histogram hist(data$Sold) abline(v=mean(data$Sold),col="green") abline(v=median(data$Sold),col="purple")

Histogram | 4 bedrooms

Robust

hist(data$Sold[data$Type=="House" & data$Bedrooms=="4"], main="Newtown 4 Bedrooms", xlab="Price (in 1000s)") abline(v=mean(data$Sold[data$Type=="House" & data$Bedrooms=="4"]),col="green") abline(v=median(data$Sold[data$Type=="House" & data$Bedrooms=="4"]),col="purple")

The median is said to be robust and is a good summary for skewed data as it is not affected by outliers. ●

symmetric data mean = median

●

left skewed ○

●

mean < median

right skewed ○

mean > median

Optimal centre Boxplots | 4 bedrooms

For the housing market, median is used because the

boxplot(data$Sold[data$Type=="House" & data$Bedrooms=="4"], main = "Newtown 4B Properties") abline(h=mean(data$Sold[data$Type=="House" & data$Bedrooms=="4"]),col="green") abline(h=median(data$Sold[data$Type=="House" & data$Bedrooms=="4"]),col="purple")

median would be a better indicator of the centre.

Boxplots | Newtown

boxplot(data$Sold, main = "Newtown properties") abline(h=mean(data$Sold),col="green") abline(h=median(data$Sold),col="purple")

expensive and cheap houses would skew the mean. The

●

median ⇒ good for skewed and outlier-rich data

●

mean ⇒ good for theoretical analysis and symmetric data

●

neither median nor mean is suitable for bimodal data

Spread

Standard units

Standard deviation

Standard Unit

Measures distance of data point from mean, in terms of SDs.

The root mean square measures the average of a set of numbers, regardless of sign.

Standard deviation is the RMS of ⇒

Standard units makes it easier to compare the distribution of one data point from another.

sqrt(mean(gaps^2)) ●

SD compares distribution between datasets

●

SU compares dist. between data points

popsd() Used when data represents the population. Such as a perfect census ●

requires multicon package

sd()

IQR

Used when data is a sample (subset) of the population, thus helping estimate the population sd.

Quantile

Divides data into q number of sets. q = 4 means quartiles q = 6 means six quantiles, etc.

Threshold

A threshold is 1.5 * IQR above or below Q1 or Q3.

Measure of spread

Outliers

Outliers are points lying outside the thresholds

Many data sets are normally distributed, thus we find 68 % of data within 1 sd from the mean (2 sd interval about mean).

Threshold

●

1sd : 0.68

Q1 - 1.5 * IQR

●

2sd : 0.95

Q3 + 1.5 * IQR

●

3sd : 0.997

Reporting spread Spread is reported in either: ●

mean and SD or

●

median and IQR

Data Wrangling Data wrangling

preparing data for analysis

Cleaning and Tidying Data Data wrangling with R and RStudio

Reshaping Data

Wrangling involves:

Grouping data helps us analyse

●

sourcing data

●

scraping data

●

cleaning and tidying data

●

variables form columns

●

reshaping data

●

subjects form row

●

splitting data

●

combining data

Tidy data

●

summarising data

Data tidy

Sourcing data Questions to consider when interrogating the source of data: ●

Reliability

●

Type of data

●

Primary source?

Scraping data Data scraping

extracting data

Web scraping

the act of automatically extracting data from websites and transforming it into a structured dataset for ease of analysis.

API Data is extracted from an Application Programming Interface.

screen scraping Data is extracted from the source code directly. e.g: HTML ●

Read html table into R, termed parsing

●

Convert to nice format using pander

Neat datasets

4 Normal Curve

Standard units

A normal curve is found in many natural phenomena. The

Standard unit Z

standard curve has µ = 0 and SD = 1.

How far a point is from the mean in standard normal curve SD units.

For a general curve, we want to find the z-value (standard unit) which is the SD from the mean relative to the standard curve.

Histogram When we plot the histogram of our data, if the shape appears symmetrical and a bell-curve, we may use the normal curve to approximate the area under the histogram.

Approximation For us to approximate a data set normally, there are some requirements.

Histogram The histogram must appear symmetrical and bell-shaped.

And the proportions must follow the 68.5 / 95 / 99.7 rule.

QQ plot

R code For P( Z ≤ x ) use the general term.

The quantile-quantile plot must be a straight line.

pnorm(x, lower.tail = F) Curve pnorm(x) For example: pnorm(0.8) ## 0.7881446

Shapiro test A large value indicates normality.

pnorm(x, mean, sd) For example: pnorm(171, 161.9, 7.7) ## 0.8813611

Measurement Error

Reproducibility

Measurement Error

Reproducible research

Unavoidable error in measuring the sample, due to chance error and bias.

Individual measurement = exact value + chance error + bias

Chance error Chance error is random and unavoidable, but can be reduced by resampling at large amounts. From this we calculate the standard deviation.

Outliers

research that can be replicated to verify the original conclusion

Reproducible research is not favoured due to: ➢ avoid selectively picking results ➢ pressure to publish hypothesis proofs ➢ generate grants and personal advancement

R Markdown Markdown is a robust file format that allows for embedded coding, text formatting and encouraging transparent data analysis. So that each action is logged and easily reproducible.

Outliers are considered data points that are extreme and generally 3 SD from the mean—assuming normality.

Tips ● Annotate code as you go

➢ Sometimes outliers may not be true outliers, countering our normality assumption.

● Customise code chunks ● Inline code require r

R code

● Hyperlinks

To identify outliers, we can convert our data into standard units and filter larger than 3.

● Adding tables

standardunits = ( x - mean(x) ) / sd(x)

Bias | systematic error Bias is a constant amount that contributes to our measurement error. It can’t be reduced by repeated sampling.

● Embed images

5 Scatter Plot Scatter plots can indicate correlation if there is crowding

Manual r-coefficient Population r

around a line.

Mean of the product of the variables in standard units.

Bivariate data Data with an independent and dependent variable allows us to plot a scatter plot.

Sample r

Linear association

cor(x, y)

How strongly the points cluster around the line

Adjusted mean of the product of the variables in standard units

Describing association Association can be described numerically by: ➢ Mean

NB: We usually assume sample correlation coefficient

How does ‘r’ work?

➢ SD Correlation coefficient

By separating the scatter plot into quadrants, and checking if two opposite quadrants have most of the

Mean | x, y

points, then we know there might be a relationship.

We can take separate mean/medians of the x and y variables, which gives us the point of averages.

SD | x, y

SD line | basic • NO r

The standard deviation of the x and y can also be used to

The SD lines connect the point of averages to the point

describe the scatter.

SDx and SDy away, respectively.

Correlation coefficient r Correlation coefficient r

➢ Does not account for clustering ➢ Inaccurate estimation with extreme r values ➢ Does NOT use r-value

Summarises cluster strength around the line.

For positive correlation:

It is possible for two datasets to have the same SDs and point of averages, but cluster differently around the line. r helps distinguish between positive and negative linear association, as well as strength of clustering.

R code

For negative correlation:

cor(x, y) ➢ r does not indicate the percentage of clustered points ➢ r does not scale linearly (e.g. 2 is twice as much as 1)

More on Correlation

Regression Line

Properties of r

A better version of the SD line is the regression line, which

➢ Unitless ➢ Perfect correlation if |r| = 1 ➢ Symmetry ○

cor(x, y) = cor(y, x)

➢ Shift invariant ○

a=x+1

○

cor(a, y) = cor(x, y)

uses the regression coefficient as a scalar for the gradient.

Slope The slope of the SD line does not take into account r: SDy / SDx The slope of the regression line multiples the slope by r:

➢ Scale invariant ○

b = 34 * x

○

cor(b, y) = cor(x, y)

Caution on r ➢ Small sd values inflate r ➢ Outliers overly influence r ➢ Nonlinear association cannot be detected ○

May require taking the log of variables

➢ Same r can arise from different data ○

Anscombe quartet (all have same r)

R code lm(x ~ y)

➢ Averages / rates can falsely strengthen r ○

Mean/rate correlation is called ecological correlation / spatial correlation

This generates a linear model object, which we can view to see the intercept and coefficient (adjusted slope).

Caution on r-line Non-linear Always check the residual plot for random distribution to confirm linearity.

➢ Association is not causation

* Regression fallacy

6 Residual Plot

Regression fallacy

Residual

When the regression effect is incorrectly assumed to have

Difference between r-line and data point

been Regression effect

In virtually all test-retest situations, the bottom group

Formally, a residual is equal to the actual point – predicted value

will improve and the top group will decline, on average

The fallacy is being worried about the regression to mediocrity / effect.

-

Scoring above likely means positive chance error

-

Scoring below likely means negative chance error

R code Generate linear model: l = lm(x ~ y) Generate residues: residuals = l$residuals

RMS Error Root mean square error (RMS)

The mean of the squared residuals.

R code error = sqrt( mean(model$residuals^2) )

Regression RMS The population regression RMS

R code | sample RMS error rms_error_S = sqrt(1-cor(x~y)) * sd(y)

Baseline prediction Baseline prediction is the flat line of y = ŷ, thus the RMS population error = population SDy.

Residual plot

Normal approximation

To plot the residuals, we need our linear model object and the original data.

For a scatter plot with a regression line, we can approximate the points in a vertical strip as normally distributed.

R code model = lm(x~y) plot(x, model$residuals) abline(h = 0) # line of perfect fit

Example | simple normal approximation -

Find the percentage of days in which CE AQI was above 90.

Vertical strips Homoscedastic

The data shows equal spread in the y direction

Heteroscedastic

Central East (CE) Air quality index (AQI)

The data shows unequal spread in y direction

We assume CE’s AQI is normally distributed, and use AQI ~ N(50.77, 21.882) P(AQI ≥ 90) = P( Z ≥ (90 – 50.77) / 21.88 ) = P( Z ≥ 1.79 ) = 0.04

Example | vertical strip normal approximation Find the percentage of days in which NW AQI was above 95 of the days when CE was above 90. i.e. P( NW ≥ 95 | CE ≥ 90 ) We find the vertical strip using the formula. zx = 90 – 50.77 / 21.88 = 1.79 For homoscedastic data, the RMS error can be used to estimate spread / SD for each strip.

ŷ = 56.12903 + (0.758 * 1.79 * 20.6) = 84.08 SDy* ≈ 13.44

Then, by calculating the mean of the strip, we can approximate the data points inside it as normally distributed.

c(mean(NW), sd(NW), cor(CE,NW)) ## [1] 56.129032 20.605407 0.757917 Calculating p-value from N distribution: P(X ≥ 95) = P( Z ≥ (95 – 84.08) / 13.44) = P( Z ≥ 0.813 ) = 0.21

Summary: linear regression

Example | linear model

1. Produce a Scatter Plot (cont. if linear) 2. Produce a Regression Line ○

abline(lm(Conc ~ Time), col = "darkgreen", lwd = 3)

Using linear model

3. Calculate r — regression coefficient 4. Produce Residual Plot ○

Random residuals ⟹ homoscedastic

5. Check our assumptions ○

Homoscedastic

○

Use regression line

○

Use vertical strip

6. Predict:

Example | quadratic model

Multiple regression We aim to find the connection between y and two or more x variables.

Multicollinearity

When 2 variables are highly correlated to each other

lm(Sold~Bedrooms+Bathrooms+Carspots) ## Call: ## lm(formula = Sold ~ Bedrooms + Bathrooms + Carspots) ## ## Coefficients: ## (Intercept) Bedrooms Bathrooms Carspots ## 103.33 416.23 211.47 81.72

# Fit Model Time2 = Time^2 # Quadratic variable quad.model = lm(Conc ~ Time + Time2) summary(quad.model) # Predicted Values timevalues = seq(0, 7, 0.1) predictedvalues = predict(quad.model,list(Time=timevalues, Time2=timevalues^2)) # Plot of data with predicted values as curve plot(Time, Conc, pch=16, xlab = "Time (s)", ylab = "Conc", cex.lab = 1.3, col = "indianred") lines(timevalues, predictedvalues, col = "darkgreen", lwd = 3)

Non-linear Models Exponential growth

Exponential decay

Logistic growth

Gompetz

Linear models Linear models are easy to use and simple to build. They can involve quadratics, cubics, and more when a simple linear relationship doesn’t work.

However, they are difficult to link to mechanistic processes which do not follow linear relationships.

Such as: population carrying capacity.

Example | cubic model # Fit Model Time3<-Time^3 # Cubic variable cubic.model <-lm(Conc ~ Time + Time2 + Time3) summary(cubic.model) # Predicted Values timevalues <- seq(0, 7, 0.1) predictedvalues <predict(cubic.model,list(Time=timevalues, Time2=timevalues^2, Time3=timevalues^3)) # Plot of data with predicted values as curve plot(Time, Conc, pch=16, xlab = "Time (s)", ylab = "Conc", cex.lab = 1.3, col =

"indianred") lines(timevalues, predictedvalues, col = "darkgreen", lwd = 3)

## Summarise model predictedvalues <predict(fit.mod,list(Time=timevalues)) predictedinitial <exp(-2.7243)*exp(timevalues*-0.4178)

Non-linear models Non-linear models are closer to mechanistic processes, resulting in interpretable parameters. Constraints, such as asymptotes are easily built-in.

Numerical solutions are required, which needs

## Plot fitted model (darkgreen) and initial model(blue) plot(Time, Conc, pch=16, xlab = "Time (s)", ylab = "Conc", cex.lab = 1.3, col = "indianred") lines(timevalues, predictedvalues, col = "darkgreen", lwd = 3) lines(timevalues, predictedinitial, col = "blue", lwd = 3)

computational power to calculate.

Example | exponential logConc<-log(Conc) log.linear.model<-lm(logConc~Time) summary(log.linear.model) plot(Time, logConc, pch=16, xlab = "Time(s)",ylab = "log Counts ", cex.lab = 1.3, col = "indianred" ) abline(lm(logConc ~ Time), col = "darkgreen", lwd = 3)

Example | Von Richter ## Initial parameters non.mod<-c(a=4,b=0.2) ## Fit Von Richter model nl.mod<-nls(Conc~a*Time*exp(-b*Time),start=no n.mod,trace=T) summary(nl.mod) timevalues<- seq(0, 7, 0.1) predictedvalues <predict(nl.mod,list(Time=timevalues)) plot(Time, Conc, pch=16, xlab = "Time (s)", ylab = "Conc", cex.lab = 1.3, col = "indianred") lines(timevalues, predictedvalues, col = "darkgreen", lwd = 3)

## Initial parameters initial.mod<-c(Y0=exp(-2.7243),k=-0.4178) ## Fit exponential model fit.mod<-nls(Conc~Y0 * exp(k*Time),start=initial.mod,trace=T)

7 Probability

Binomial formula2

Making Lists

The binomial coefficient tells us the ways of arranging n

For small problems, it is good to make lists and count.

binary objects in a row.

Such as, given 5 people, how many ways can we

We can use R to simulate using sample() Do the lab!!!

choose a team of 2?

The Addition Rule

Binary trial

where only two results can occur

Mutually exclusive

when two events cannot occur at the same time.

Binomial model Given that A and B are mutually exclusive, the chance of A

The key features of a binomial distribution are:

or B occurring is equal to their sum.

1. Only 2 possible outcomes 2. The probability of success is the same for each trial (p) 3. Events are independent and do not occur simultaneously 4. There is a set number of trials (n) For success to occur x times with probability p out of n

example: A die is rolled 6 times.

trials, we use the above formula.

1. P( ) that 1st roll is 1 or the 6th roll is 1.

This is a test of independence rather than mutual exclusivity, as you can have both events occurring at the same time. Thus:

2. P( ) of a randomly shuffled deck that the top card

R code

is the Ace of Spades or the bottom is?

dbinom(x, n, p)

Since the two events are mutually exclusive we use the addition rule:

## returns the binomial probability of event A occurring x times out of n with probability p.

Chevalier de Méré Game A: tossing a die 4 times, what is the chance of rolling at least 1 ‘ace’?

It’s easier to solve in terms of

Game B: tossing two dice 24 times, what is the chance of rolling at least 1 double-ace?

Again, it’s easier to solve using the complement, which is:

2

L21 Week 7

8 Chance Variability Law of large numbers Law of Large Numbers / Averages

as more observations are

Modelling sum/mean Sum The sum of random draws from a box model with replacement is

made, the proportion of an event converges to the probability p of that event.

EV = number of draws × mean of the box The law can be justified by the decrease in the percentage

SE = √number of draws × SD of the box

size of chance error, i.e. relative to the number of observations. Note that the absolute chance error will Standard Deviation of a binary box where the ticket values

increase.

are ‘big’ and ‘small’:

Chance Error = up to 3 Standard Errors, meaning that the OV can be up to 3 SE from EV.

Mean The mean is defined similarly:

The box model

EV = mean of the box

The box model can be used to simplify problems down to

SE =

a uniform distribution. ➢ There are only 2 outcomes (types of tickets) — success and failure

SD of the box

√number of draws

Approximation to the normal model

➢ Any number of tickets — representing proportion

Given a large number of draws we can approximate the

➢ Number of draws from the box, representing

box model to the normal model — separately for the mean

occurrences

Example: box model A dice game costs $1 to play. Rolling a ‘6’ returns $2. After playing 25 times, what is the net gain / loss?

To model this game, we setup the box to contain: ➢ Ticket A = $1 — net gain from success ➢ Ticket B = -$1 — net loss ➢ 1 x A and 5 x B, representing the P(success) Then, we draw 25 times with replacement from the box, which generates a sample.

and the sum.

Normal distribution

Empirical histogram | simulated or observed

The normal distribution is unimodal, symmetric and

When the simulated histogram is built from a large

bell-shaped. It is the most common distribution, with

number of simulations, the shape of the observed values

many variables being near-perfect fits.

will converge to the probability histogram.

Z-score | standard unit Z-score

The Z-score of an observation is the number of standard

➢ Expected value = centre ➢ Standard error = spread

Central limit theorem

deviations it falls above or below the mean.

Central limit theorem When repeatedly sampling at random with

The z-score helps us compare the difference between a data point x and the mean in terms of standard deviation.

replacement from a box, given a large enough sample size and repeats, the probability histogram for the sum / average will approximate the normal curve.

➢ Note the difference between repeat number (N) and Thus, we can compare how close two data points are to their respective distributions.

68-95-99.7 rule A rule of thumb to help gauge the probability of falling within 1, 2, and 3 standard deviations from the mean.

Probability histogram A probability histogram represents chance by area. The y-axis usually represents probability density.

●

The graph represents theoretical chance

sample size (n) — the number of data points per sample.

9 Sample Surveys

Sensitive questions These can result in false answers being given.

Population

Full amount of information

➢ Do you use drugs?

being studied and collected

➢ Do you pay tax?

through a census.

➢ Have you physically disciplined your child?

Sample

A portion of the population.

Parameter

True numerical value

Questions can be interpreted differently, thus not as

describing population. (µ)

objective as believed. Certain words can be ambiguous.

Sample numerical value which

Sample size

Estimate

describes the parameter. (x̄)

Lack of clarity in question

While a larger sample size can reduce chance error, it can

Common sample biases Selection bias

Systematic bias to sample certain types of subjects (not human-caused).

amplify systematic error or bias.

For example: When The Literary Digest sent out 10 million questionnaires about the US election, 2.5 million answered. However, because it inherently favoured property-owning middle-class white Americans, the results were inherently biased.

Non-response bias

Caused by non-responding participants.

Good sampling

Interviewer bias

When the interviewer affects

Use a probability method to pick the sample.

the participant response.

Measurement bias

➢ Selection does not involve interviewers. ○

When the form of question

E.g. interviewer on the sidewalk choosing who they want to interview.

affects the response.

➢ Interviewer should be able to compute the chance of any individual being selected. ○

➢ Bias = non-sampling error ➢ Chance error = sampling error 3

Measurement bias3 The ordering or wording of questions can affect how subjects respond: ➢ Should doctors be allowed to murder unborn children? ➢ Should children be protected from disease? ➢ Should vaccines be used?

Recall bias A form of measurement bias, is when the retrospective recall of memory results in inaccuracies.

Measurement bias

E.g. every fifth person.

1. Multi-stage cluster sampling Using simple random sampling to group the population into smaller and smaller clusters before taking a sample.

2. Quota sampling | non probability Using known proportions of the population the sample is constructed by filling up quotas. Such as, 60% Caucasion, 55% male, 80% on mortgage.

Can result in hidden selection bias.

3. Convenience sampling Subjects are selected because of their convenient accessibility.

To sample the SAT of highschool students we can sample the 50 states. From the chosen states we then sample counties. Within the counties we sample schools, and finally of those chosen we sample seniors. At each stage probability sampling must be used.

Box model for Sample surveys We can use the Box Model to calculate the chance error for sample surveys. The Box contains the population and

Bootstrapping & Confidence interval Bootstrapping

properties using a sample’s

the SE measures the variability across the different

properties.

samples.

Population

Estimating population

The Box Model formulae so far require knowledge of the true composition of the box (population), which is not useful when the population parameters are not known.

Thus, we create a box1 using sample proportions.

Sample

The SEMEAN decreases as the sample size increases, because =

SD

YouGov conducted a survey about same-sex marriage, which gave support of 59%, oppose 33% and undecided 8%, with sample size n = 1012.

Create a box1 with proportion equal to the sample: support

√sample size

Correction factor With replacement Suppose we sample the population of NSW and Australia for their political party. If we sample 1 million voters, with replacement, SE would be the same (assuming equal σ).

if the SE can be made smaller (since as n approaches population size, SE decreases).

However, the Correction Factor is: √16655856 – 1012 / √16655856 − 1 ≈ 1

➢ SE depends only on the sample size. ➢ Accuracy is the same, but not necessarily %

Without replacement When sampled without replacement, then variability decreases as sample size approaches the population size. ●

Here, the box model does not apply.

To calculate the Standard Error for this model, we multiply the SE (without replacement) by the Correction Factor.

This can be done as bootstrap in R.

R code N=16655856 box=c(rep(1,0.64*N),rep(0,0.36*N)) mean = mean(sample(box, 1012, rep = F)) means = replicate(10000, mean) cumsum(table(round(means, 3))

Confidence interval A confidence interval is NOT saying there is 95% chance the parameter falls in such interval.

Instead, suppose we sample the population and generate CIs repeatedly using the z-score for x%. Then, the proportion of CIs containing the parameter is 95%.4

➢ IF: sample size (n) << population size, the factor ≈ 1

The confidence interval follows the 68-95-99.7 rule.

Using the previous example: ●

68% CI is the range 0.64 ± 0.015

●

95% CI is the range 0.64 ± 0.030

●

99.7% CI is the range 0.64 ± 0.045

4 Visualisation (Brown) and Visual (Kris

10 Hypothesis Testing We say the data is not consistent with our hypothesis if our observed value is not consistent with the expected

Test statistic | z Test statistic

value, with a gap of more than 2 or 3 SEs.

Measure the difference between the expected value of the null hypothesis and the observed value.

Framework There are 3 parts to the framework:

1. Set up research question [H] declare Null and Alternative hypothesis 2. Weigh evidence A—assumptions

NB: The formula is similar to the standard unit formula.

And given H0 is true, then it is the Standard Unit.

T—test statistic P—p-value 3. Explain conclusion C—conclusion

Hypothesis

p-value The probability of observing the test statistic or more extreme—which depends whether we’re doing one tail

Null hypothesis

assumes the difference between Observed and Expected values to be due to random chance alone.

(and which side) or two tails—when assuming the null hypothesis.

i.e.: the effect of 'change' (e.g. new drug) is more likely due to luck than actual improvement.

Alternative hypothesis

assumes the difference between Observed and Expected to be NOT due to chance alone. i.e.: the effect of 'change' did lead to a change in results, at least partially.

We can set up 2 box models to represent H0 and H1

(not necessarily ≥) ➢ Small value: we reject H0 Meaning our observed results were unlikely to have occurred under the null hypothesis. ➢ Large value: we reject H1 / retain H0 Very likely that our observations could have occurred under the null hypothesis, suggesting the change to our system likely did not lead to significant system changes.

Assumptions | general For our testing metric, (e.g. the number of traffic incidents) we must make assumptions about how the subjects will behave. For example: ●

Each event is independent of another

●

Data is normally (symmetrically) distributed

●

Data have linear relationship

Z-test assumptions

Example A factory makes cereal boxes with mean = 1.04 kg and σ = 0.055 kg. After an earthquake, the machine may have been damaged. A sample of 30 boxes gives a sample mean = 1.02 kg. Test at 1% significance whether there is a change.

Null — weight has not changed, machine is same Alternative — weight has changed.

P( x ≤ 1.02 ) = P( Z ≤ (1.02 – 1.04) / 0.055 ) = 0.358 = 35.8 % >> 1 — we retain H0

●

Sample is random

●

Sample is normal OR large enough for CLT

●

Population SD is known

Alternative method We can directly use the test statistic (z) and compare it to the standard unit of the significance level.

Proportion testing | z-test • one-tailed Using the Box Model We assume the box model as our Null hypothesis. We find the EV and SE from a sample of the box, and then identify the OV from our H1.

Example A new treatment for desensitisation to peanuts can help cure people’s peanut allergies. In a trial of 29 participants, 26 showed desensitisation. The research team claims their desensitisation rate is higher than 80 %, test this claim.

Hypothesis ➢ Null hypothesis—the rate is 80 % ➢ Alternative hypothesis—the rate is higher than 80 %

●

Set up the test-statistic using the above formula

Assumption

●

Calculate the probability of getting the z or more extreme, and compare to the sig level.

➢ We assume the probability of desensitisation is equal for all participants

Only if P(z) is smaller, then we can reject the H0

➢ We each patient’s desensitisation is independent of

●

each other’s

Binomial testing Using the z-test for binomial models (Box Model), the sample size must be sufficiently large to allow for Central

statistic The observed value is 26 / 29.

0.80

Limit Theorem (where we take the mean and convert it to a proportion).

When testing with an actual sample (bicyclist example below) and not a Box Model generated sample, we do not need to consider the sample size as much.

NB: For both mean and proportion testing, this applies.

Conclusion

At a 5% significance level, we would say the H0 is retained.

NB: since the sample size was small, CLT may not have been a valid assumption.

We could have used the binomial model to test the previous test. Which gives us the direct accurate probability of the null hypothesis yielding more than 80% successful trials = 0.14. This is much higher than the 0.9 we calculated. It’s also more robust, as the box model is skewed, so the normal approximation is less valid.

11 One Sample z-test and t-test

One-tailed Test

One sample test finds differences between the sample

The one tailed test is the two-tailed test, but with the

and the population.

probability not being doubled.

To test if a sample has a certain mean, the alternative

t- Test

hypothesis would be x ≠ value. We would need to find the

The Z-test requires our population SD, while the t-test uses

test-statistic of the mean (not proportion) and use the

sample SD and the T-distribution, which adjusts for extra

two-tailed test.

variability by having degrees of freedom.

Mean testing | z-test • two-tailed

●

Use degree of freedom n – 1

For a normal distribution (or just symmetric) we find one tail and double the probability to get the p-value.

public domain image

Example

T-test assumptions minutes.

H0 — mean exhaustion time = 45 mins

Assumptions: samples are random assume population is Normally distributed assume population sd = 12 mins

●

Sample is random

●

Sample is normal OR large enough for CLT

●

Population SD is known

Example | using t-test instead of z-test Assumptions: samples are random assume population is Normally distributed P-value:

statistic:

P-value:

= 2 * pt(teststat1,8,lower.tail=F)

t.test(<data>, mu = <x>)

Paired t-test

SD vs SE

A paired t-test aims to find whether two samples of the

➢

sample mean.

same subjects are different from each other. ➢ For example: comparing

SD indicates variability of each data point from the SE indicates variability of sample mean from the true mean.

employee performance to themselves one year ago each patient’s health on new vs old drug ➢ We must pair each participant.

* Confidence interval To find the confidence interval, we use

➢ Perform two-tailed test on the difference between each pair

Paired T-test assumptions ●

Sample is random

x = mean of sample

●

Sample difference is normal OR large

z = standard unit of half of sig. level

enough for CLT

(95% ⟹ find 2.5% standard unit) σ = SD (whether Pop or Sample depends on test)

Example

n = sample size

caf13 = c(37.55,59.3,79.12,58.33,70.54,69.47,46.48 ,66.35,36.2)

cafdiff = caf13 - caf0 Performing the two-tailed test on cafdiff: t.test(cafdiff, mu = 0) t.test(caf13, caf0, paired = T)

method statistic:

P-value:

= 2 * pt(cafdiff,8,lower.tail=F)

t.test(<data>, mu = <x>)

level. There is a difference in performance when using caffeine.

Two Sample t-test The aim of a two sample t-test is to analyse the differences between two separate samples, in contrast to a paired t-test—which involves resampling the same subjects. ➢ Sample size is not necessarily equal

Two Sample T-test assumptions ●

Each sample is independent.

●

The population of each sample have equal SD

●

Samples are normal OR large enough for CLT

Test statistic

Here, we will use the two-tailed test. Using t.test t.test(No_RB, RB, var.equal = T) We need to let the variance be equal, as stated in our assumptions. ## ## Two Sample t-test ## ## data: No_RB and RB ## t = -1.5418, df = 24, p-value = 0.1362 ## alternative hypothesis: true difference in means is not equal to 0 ## 95 percent confidence interval: ## -13.892538 2.011586 ## sample estimates: ## mean of x mean of y ## 73.64286 79.58333

retained.

Conclusion: The evidence suggests that drinking Red bull does not lead to a significant change in the heart rate of university students.

Pooled SD

Example No_redbull = 84,76,68,80,64,62,74,84,68,96,80,64,65,66 Redbull = 72,88,72,88,76,75,84,80,60,96,80,84,NA,NA Two independent groups of students had their heart rates measured. One group had drunk a can of Red Bull, while the other did not.

Hypothesis: µ1 = control group mean µ2 = treatment group mean H0 = No difference in mean (µ1 – µ2 = 0) H1 = Yes difference in mean (µ1 – µ2 ≠ 0)

Choosing 1-tail or 2-tail If existing research indicates that the treatment group increases or decreases, then choose the 1-tail test for that side. Otherwise, choose the 2-tailed test to see if there is any difference at all first.

Justifying assumptions Box plot | normality & variance A simple boxplot shows that the variance appears similar, because there are no extreme outliers. Furthermore, boxplots with symmetry can be used to justify normality.

Q-Q Plot | normality

Levene’s Test F-Test | variance

QQ plots plot the sample quantiles against their

The aim is to test the population for equal variance.

corresponding normal quantiles. A linear relationship

➢ H0 — both groups have equal variance

indicates normality.

➢ H1 — they are not equal

R code var.test(No_RB,RB)

R code

require(ggplot2) p3 = ggplot(RB_data, aes(sample = rate, colour = group)) + stat_qq() + stat_qq_line() + ggtitle("QQplot") p3

Shapiro-Wilk test | normality

## ## F test to compare two variances ## ## data: No_RB and RB ## F = 1.1357, num df = 13, denom df = 11, p-value = 0.8428 ## alternative hypothesis: true ratio of variances is not equal to 1 ## 95 percent confidence interval: ## 0.334832 3.631266 ## sample estimates: ## ratio of variances ## 1.135659 Conclusion: P-value = 0.8428 >> 0.05 We can retain the H0

The aim is to hypothesis test for normality. ➢ H0 is data is normal ➢ H1 is non-normal data. ➢ Small sample sizes are overestimated to be normal

R code shapiro.test(No_RB) ## ## Shapiro-Wilk normality test ## ## data: No_RB ## W = 0.90604, p-value = 0.138 Conclusion: 0.138 >> 0.05, we retain H0

Welch two sample T-test | variance Two samples with different variances cannot be used in the basic two sample T-test. Instead, we use the Welch version.

R code t.test(No_RB, RB, var.equal = F) ## ## Welch Two Sample t-test ## ## data: No_RB and RB ## t = -1.6111, df = 19.747, p-value = 0.123 ## alternative hypothesis: true difference in means is not equal to 0 ## 95 percent confidence interval: ## -21.180686 2.728887 ## sample estimates: ## mean of x mean of y ## 70.35743 79.58333

12 Chi Square Test Chi square tests find relationships between qualitative data.

The three main ways of using the chi-square test: 1. Goodness of Fit Test hypothesis about distribution of a qualitative variable in a population.

2. Homogeneity Test hypothesis about the distribution of qualitative

statistic:

variables in several populations.

3. Independence Test a hypothesis about the relationship between two qualitative variables in a population.

Test-statistic |

2

P-value: We need to find P( 2 ≥ 4.4 ) on the 52 distribution. pchisq(4.4,5,lower.tail=F) # 0.4933735

innocent.

2

The test-statistic is given by chi-squared 𝝌 value.

Chi-square assumptions Expected categories: ➢ None are empty ➢ No more than 20% are < 5 (Cochran’s Rule) (i.e. at least 80% must be > 5)

Goodness of fit We can investigate the distribution of qualitative variables such as the roll of a dice, or colour of cars.

Example ## throws1 ## 1 2 3 4 5 6 ## 7 7 9 14 10 13 A gambler rolled a dice 60 times. Test if her rolls are due to an unfair die.

Hypothesis: H0 = The distribution is fair H1 = The distribution is different

R code chisq.test(throws1a, p = c(1/6,1/6,1/6,1/6,1/6,1/6)) ## Chi-squared test for given probabilities ## ## data: throws1a ## X-squared = 4.4, df = 5, p-value = 0.4934

Independence |

2

For two variables of qualitative category, we can test if they are independent.

Such as: smoking and domestic student

Mosaic plot Using our data value Smoke we can create a mosaicplot() which highlights large differences between expected and observed.

Example In a class of 25 students, can we say that smoking and domestic status are dependent?

H0 — smoking is independent of background H1 — smoking is not independent of background

Here, most of the differences are small, suggesting independence.

Smoke = matrix(c(9,4,6,6), nrow = 2, ncol = 2, byrow = TRUE, dimnames = list(c("Smoke", "No smoke"), c("Domestic", "International"))) Test-statistic: chisq.test(Smoke)

Summary Begin with a contingency table, where we have the variables with their observed values.

Degrees of Freedom

## Pearson's Chi-squared test with Yates'

Our degrees of freedoms are multiplied to find which

continuity correction

distribution to use.

## ## data:

Smoke

## X-squared = 0.32719, df = 1, p-value = 0.5673 Conclusion: Since p-value = 0.5673 >> 0.05 we retain H0 and say that the student status and smoking are independent.

Expected values The expected values are calculated by taking the product of the column and row and dividing by the total n.

For example, smoke22 = 10 * 12 / 25 = 4.8 Since more than 20% (25%) of the expected values is under 5, it breaks one of our assumptions.

Yates Continuity Correction The test automatically adjusts for this. But we can remove it by using the parameter correct = F when using chisq.test()

P-value The p-value is calculated by taking the upper tail of the test-statistic

2

Regression Test The aim of regression testing is to test whether our linear model’s slope is robust — whether a linear relationship exists or not.

Example Beer consumption is claimed to be decreasing in the past few years in Australia. Test this claim with the yearly per capita consumption of beer.

➢ H0 — there is no linear relationship ➢ H1 — there is a linear relationship

H0 — there is no linear trend H1 — linear trend present

Residuals are: ➢ Independent

➢ Normal ➢ Constant variance (homoscedastic) statistic: The relationship between y ~ x should appear linear

summary( lm(Value~Time, PCBeer) )

Population equation The population equation gives the line of our real data.

➢ Yi — dependent variable ➢ Xi — independent variable ➢ β0 — y-intercept ➢ β1 — slope (gradient) ➢ εi — residual / ‘gap’

Test statistic | one sample t-test

NB: focus on ‘time’ and the SE and t-value and p-value.

P-value:

The expected value of the slope is 0 (non-linear) while the observed value is the slope of our linear model. ➢ SE is given by R ➢ Degrees of Freedom = n – p – 1 ○

p = number of independent variables

hypothesis.

Conclusion:

There is strong evidence that there is a linear trend in beer consumption which this model predicts.

Manual We can take the t-value from summary and perform a one sample t-test on it, finding P( t ≥ t-value ).