USYD DATA1001 Foundations of Data Science Summary of the lecture slides and additional notes taken 2

USYD DATA1001 Foundations of Data Science Summary of the lecture slides and additional notes taken 2026 JUNE EXAM PREP (University of Sydney)

1. Articulate the importance of statistics in a data-rich world, including current challenges such as ethics, privacy and big data.

2. Identify the study design behind a dataset and how the study design affects context specific outcomes.

3. Produce, interpret and compare graphical and numerical summaries, , using base R and ggplot (extension).

4. Apply the Normal approximation to data, with consideration of measurement error.

5. Model and explain the relationship between 2 variables using linear regression.

6. Use the box model to describe chance and chance variability, including sample surveys and the central limit theorem.

7. Given real multivariate data and a problem, formulate an appropriate hypothesis and perform a range of hypothesis tests.

8. Interpret the p-value, conscious of the various pitfalls associated with testing.

9. Critique the use of statistics in media and research papers in a wide variety of data contexts, with attention to confounding and bias.

EXPLORING DATA ➢ Controlled Experiments Domain Knowledge ➢ Background context that helps you understand data (need curiosity and become specialist in area investigated) ➢ Eg. What is Roaccutane prescribed? how does it work? What are known side effects?

Types of Evidence ➢ Personal testimony/observation → more generalised finding ➢ source(s) behind media article often poorly cited ➢ In reputable research journal → every study stage should be documented and reviewed ○ journals require reproducible research → data sets available for verification & analysis Design of the Study ➢ Scientists gave Roaccutane to young adult mice for 6 weeks → tested response to stress ➢ Mice on Roaccutane were less mobile → assumed sign of depression.

The Method of Comparison ➢ Scientists use controlled experiment to determine effect of treatment on a response variable (thing trying to model/predict eg. depression) ○ Treatment Group given new drug

○ Control Group is not ➢ Types of control groups ○ Contemporaneous = occur at the same time as treatment groups ○ Historical = earlier than treatment groups (comparing past experiment) ■ Used if currently an ethical issue ■ BUT were conditions exactly the same? ➢ Must control all other variables on treatment → same for both groups eg. psychological factors (susceptibility to depression) ○ If groups not comparable → differences can confound (mix up) effect of the treatment.

3 Potential Confounders (method of allocation) ➢ SELECTION BIAS → Calls for random allocation ○ Bias affects accuracy if based on investigator's judgment (nonrandomized) eg. doctor’s choosing healthy people to undergo operation due to risk of death.

They lived longer, but operation or health? ➢ OBSERVER BIAS → calls for double-blind design (not aware of the identity of the 2 groups) ○ Placebo effect = when subject responds to idea of treatment. ○ If the subjects/investigators aware of identity of groups → bias in responses or evaluations ➢ CONSENT BIAS ○ When subjects choose if they take part in experiment eg. polio vaccine → richer people said yes, poorer more likely to say no ■ Should say they may or may not get vaccine ∴ BEST METHOD OF COMPARISON ➢ Random allocation → no selection bias ➢ Double blind → no observer bias

➢ Observational Studies In observational studies ➢ Investigator cannot use randomisation for allocation of subjects into treatment and control groups ➢ Used in most educational research Precautions

1. Observational studies can't establish causation ➢ Can only establish association (link) ➢ Points to but does not prove causation (may not cause, but increase risk of) ➢ Eg. smoker more likely to get liver cancer but this does not imply it causes it ○ Smokers drink more alcohol ∴ effect of smoking confounded with alcohol consumption 2. Can have misleading hidden confounders ➢ Confounding occurs when effect of treatment caused by some other variable/s in Treatment and Control Groups ➢ Confounding variables can introduced due to: ○ selection bias → some subjects more likely to be chosen eg. investigators select healthier subjects for surgery ○ survivor bias → dropout of some subjects eg. "improvement" due to dropout of worst/unresponsive subjects ○ adherers and non-adherers → some subjects more compliant and healthier already (stick to it) eg. not take drug Strategy for dealing with confounders (controlling for confounders) ➢ Make groups more comparable by dividing into subgroups with respect to confounder ○ Eg. Controlling for alcohol consumption → split up smokers according to alcohol consumption ○ Limitations ■ Need to find confounder (often hard to find) 3. Observational studies with confounding variable can lead to Simpson's Paradox ➢ Simpson’s paradox (reversing paradox) → trend in individual groups of data that disappears when the groups are pooled together (trend/percentages reversed) due to confounding variable

➢ ➢ Eg. More young women smoked than older women, and since younger expected to live longer, adding all groups makes smoking appear beneficial (age = confounding variable) 4. Observational studies result from using historical control ➢ Time is a confounding variable eg. comparing new medication on current patients vs. old medication on past patients (treatment & control groups may differ due to societal change)

➢ ∴ Controlled experiments need to be in same time period (contemporaneously).

Diff uses of the word ‘control’ 1. A control = subject who did not get the treatment.

2. A controlled experiment = study where investigators allocate subjects to 2 groups 3. Controlling for confounders = trying to reduce the influence of confounding variables

➢ Data & Qualitative Data Data = info about the set of subjects being studied, often in reference to sample

Many types of data eg. survey data, spreadsheet type data, MRI image data Initial Data Analysis (IDA) ➢ First general look at the data (without formally answering research questions) → ensures later analysis more efficient & minimizes risk of incorrect/misleading results ○ helps see whether data can answer research questions ○ poses other research questions ○ identify data's main qualities ○ suggests population that sample derives from ➢ Commonly involves ○ data background → checking quality & integrity of the data ○ data structure → what info has been collected? ○ data wrangling → scraping, cleaning, tidying, reshaping, splitting, combining ○ data summaries → graphical and numerical Structure of the Data ➢ Variable = measures/describes attribute of the subjects (columns of a data set) ○ p variables → dimension p Number of variables

Types of variables

* Age (also income) technically is a quantitative, continuous variable but may be recorded as a qualitative variable in a survey as rounded integer → However, optimal to record quantitative data if possible.

Graphical Summaries ➢ Aim = best highlight features of this data ➢ May use use trial and error ➢ Pie chart may be popular but not usually informative Graphical summaries of QUALITATIVE DATA ➢ The bar plot → simple summary of qualitative data.

Graphical summaries of big data ➢ Massive amounts of data collected ➢ Highly dimensional → more variables p than subjects n eg. genomics data can have 3 billion variables (person's DNA sequence is 3 billion base pairs long) ➢ Big data requires more complex visualisations.

➢ Quantitative data Histogram (used for quantitative data)

*18-20 has most crowding even though there are more people from 20-70 ➢ ➢ ➢ ➢ ➢

Highlights percentage of data in one class interval compared to another Blocks represent the percentages of by area in class interval → all adds to 100% Horizontal scale is divided into class intervals Height of each block represents crowding Don’t need vertical scale

Cleaning data ➢ Primitive cleaning → replace 2 cells with ‘-9’ with a blank or N/A ○ Commentate that you’ved removed it Will mostly use density scale

Endpoint convention For continuous data, we need an endpoint convention for data points on border of two class intervals Eg. If interval contains left endpoint but excludes the right endpoint → 18 year old counted in [18,21) not [0,18) → left-closed and right-open Tips ➢ 10-15 MAX class variables (don’t over condense data)

*Just need class intervals and height for histogram

How to produce in R # Read in data data <- read.csv("data/2016FatalitiesC.csv", header = T) # Choose a variable

Age <- data$Age # Choose the class intervals breaks = c(0, 18, 25, 70, 100) # Produce a distribution table table(cut(Age, breaks, right = F)) # Produce a histogram hist(Age, br = breaks, freq = F, right = F, xlab = "Age (in years)", ylab = "% per year", main = "Histogram for Age of Road Fatality in Australia: Jan-June 2016")

Controlling for a variable AgeF <- data$Age[data$Gender == "Female"] # This selects just the female ages.

AgeM <- data$Age[data$Gender == "Male"] par(mfrow = c(1, 2)) # This puts the graphic output in 1 row with 2 columns hist(AgeF, freq = F) hist(AgeM, freq = F)

Simple boxplot ➢ For comparing multiple data sets. ➢ Plots median data point, the middle 50% of the data in a box, and determines any outliers Age <- data$Age summary(Age) par(mfrow = c(1, 2)) boxplot(Age) boxplot(Age, horizontal = T)

Comparative boxplot # Divide Age into 2 genders Gender = data$Gender summary(Age[Gender == "Female"])

## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's ## 0.00 25.50 44.00 47.19 69.00 93.00

1

summary(Age[Gender == "Male"]) ## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's ## 1.00 25.00 42.00 44.02 59.00 96.00

1

boxplot(Age ~ Gender, horizontal = T)

Scatter plot (for 2 quantitative variables) SpeedLimit <- data$SpeedLimit plot(Age, SpeedLimit)

➢ Data Visualisation Data visualisation = graphical summaries on steroids eg. infographics, heat maps and networks Good Data Viz ➢ Tells interesting story in a visually appealing way (understanding of data and design) → be informative and communicate insights. ○ What question are we trying to answer? ○ What variables are we highlighting? ○ What does the eye focus on? ○ What is the effect of different colours)?

Poor and Bad data biz ➢ Poor Data Viz → tells story in a visually boring or distracting way, ➢ Bad Data Viz → tells misleading story (especially in an appealing way!) ggplot2 ➢ Intuitive way to produce data visualisations → plotting system ➢ ggplot(dataset, aes(v1,v2)) + geom_point() + facet_grid(v3~.)

○ Data, which variables to focus on, type of plot ➢ Building blocks of plot → create any visualisation ○ data ○ aesthetic mapping (aes) ■ what you can/features you’re looking at see eg. position, outside color, inside colour (fill), shape of points, linetype, size ○ geometric object ■ geom_xxx → marks we put on a plot eg. points (geom_point), lines (geom_line), boxplot (geom_boxplot) ○ statistical transformations ○ scales ○ coordinate system ○ position adjustments ○ faceting (divide by subsets) ■ Slicing up by diff controlled variables Simple bar plot (1 qual) p = ggplot(diamonds, aes(x = cut)) # Defines the x axis (1 variable). p + geom_bar() # Represents the data by bar chart.

OR easier in # Compare in Base R: barplot(table(diamonds$cut)) Bar plot [1 qual] + aesthetic [1 qual] p + geom_bar(aes(fill = clarity)) # Adds colour by a 2nd variable (clarity) OR SIDE BY SIDE p + geom_bar(aes(fill = clarity), position = "dodge")

Sophisticated scatter plots and box plots http://www.maths.usyd.edu.au/u/UG/JM/DATA1001/r/current/lectures/ 6.DataVisualisation.html#36

➢ Centre - Numerical Summaries Data on Newtown Property Sales - Sold → how much it was sold for Numerical summaries ➢ Reduces all the data to 1 simple number ("statistic"). ○ loses a lot of information yet allows easy communication and comparisons ➢ Major features that we can summarise numerically are: ○ Maximum of data ○ Minimum of data ➢ Centre [mean, median] ➢ Spread [standard deviation, range, IQR] → must know for centre Useful notation for data

Mean Mean = average of the data ➢ unique point at which the data is balanced → higher readings and lower readings cancel each other out ➢ balancing point on histogram → expensive and "cheap" properties cancel each other out

➢ Mean of all properties → mean(data$Sold) ➢ Mean of houses with 4 bedrooms (large) → mean(data$Sold[data$Type == "House" & data$Bedrooms == "4"]) hist(data$Sold, main = "Newtown properties", xlab = "Price (in 1000s)") abline(v = mean(data$Sold), col = "green")

4 bedroom → hist(data$Sold[data$Type == "House" & data$Bedrooms == "4"], main = "Newtown 4 Bedrooms", xlab = "Price (in 1000s)") abline(v = mean(data$Sold[data$Type == "House" & data$Bedrooms == "4"]), col = "green")

Median Median (x̃ ) = middle data point, when the data is ordered from smallest to largest

➢ Median of all properties → median(data$Sold) ➢ Median of houses with 4 bedrooms (large) → median(data$Sold[data$Type == "House" & data$Bedrooms == "4"]) On histogram

➢ Median = half way point on the histogram - ie 50% of the houses sold are below and above $1.4 million. hist(data$Sold) abline(v = median(data$Sold), col = "purple") 4 bedroom → hist(data$Sold[data$Type == "House" & data$Bedrooms == "4"], main = "Newtown 4 Bedrooms", xlab = "Price (in 1000s)") abline(v = median(data$Sold[data$Type == "House" & data$Bedrooms == "4"]), col = "purple") Mean vs median ➢ Median gets rid of outliers ➢ Usefulness depends on context → median for buyers, mean for agent to work out commission Mean and median on boxplot ➢ Median = line in centre on boxplot ➢ 50% of houses sold → Inside box ○ Reaming 50% outside box ➢ If mean and medium close together → symmetry of data boxplot(data$Sold, main = "Newtown properties") abline(h = mean(data$Sold), col = "green") abline(h = median(data$Sold), col = "purple")

boxplot(data$Sold[data$Type == "House" & data$Bedrooms == "4"], main = "Newtown 4B Properties") abline(h = mean(data$Sold[data$Type == "House" & data$Bedrooms == "4"]), col = "green") abline(h = median(data$Sold[data$Type == "House" & data$Bedrooms == "4"]), col = "purple")

Robustness and comparisons

➢ Median is ROBUST and a good summary for skewed data as it is not affected by outliers.

➢ Eg. Recently the heritage Properts Building sold for 13 million in Newtown. ○ The mean would be a lot higher. ○ The median would be a bit higher: it moves from the average of the 28th and 29th points to the 29th point data2 = c(data$Sold, 13000) sort(data2) mean(data2) median(data2) → robust

Which is optimal for describing centre? ➢ Sometimes neither gives a sensible sense of location, for example if the data is bimodal. ➢ median is robust ∴ it is preferable for data which is skewed or has many outliers, eg. Sydney house prices ➢ mean is helpful for data which is basically symmetric, which not too many outliers, and for theoretical analysis ➢ Limitations of both ○ need to be paired with a measure of spread.

➢ Spread ➢ calculate gap between the house and the mean (how much above or below mean?) ○ gaps = data$Sold - mean(data$Sold) ➢ Calculate the average of the gaps → Mean gap = Mean of (data-mean) ○ round(mean(gaps)) ○ Mean gap will always be zero as the mean is the balancing point of the gaps

Standard deviation = measures spread of data (shows average spread, how much above or below) → however problem; averaging gives 0 Root mean square (of gaps) ➢ ∴ RMS measures the average of a set of numbers, regardless of the signs. ○ Square the numbers (gaps) ○ Mean the result ○ Root the result

➢ In Rstudio ○ sd(data$Sold) → use sample (usually more diff if small population, negligible if large) ○ Slightly different → works out sample version ○ To use population version → install package ■ library(multicon) ■ popsd(data$Sold) ○ Narrowness of research question determines if data is a population or sample

Standard deviation on histogram hist(data$Sold, main = "Newtown properties", xlab = "Price (in 1000s)") abline(v = mean(data$Sold), col = "green") abline(v = mean(data$Sold) - sd, col = "blue") → below by 1 unit of standard deviation abline(v = mean(data$Sold) + sd, col = "blue")→ above by 1 unit of standard deviation abline(v = mean(data$Sold) - 2 * sd, col = "purple") → less common abline(v = mean(data$Sold) + 2 * sd, col = "purple") abline(v = mean(data$Sold) - 3 * sd, col = "brown") abline(v = mean(data$Sold) + 3 * sd, col = "brown") → much rarer

Standard units Standard units = how many standard deviations below/above mean

Comparing 2 data points Compare standard units

Interquartile Range (IQR)

➢ Use quantiles → quantile(data$Sold) (divides data into 0%, 25%, 50%, 75%) ➢ IQR is robust → summary of spread for skewed data IQR on the boxplot

➢ IQR = length of the box in the boxplot (middle 50% of the houses sold) ➢ The lower and upper thresholds are a distance of 1.5 from the quartiles (by convention). ➢ Data outside these thresholds is considered an outlier ("extreme reading")

We report in pairs → (mean,SD) or (median,IQR) Coefficient variation ➢ Measured volatility by combining mean and standard deviation (higher number = more volatile)

m = mean(data$Sold) sd = sd(data$Sold) sd/m

➢ Data Wrangling

DATA WRANGLING = Time consuming process of doing whatever is needed to get the data ready for analysis Can involve ➢ sourcing data ➢ scraping data ➢ cleaning and tidying data ➢ reshaping data ➢ splitting data ➢ combining data ➢ summarising data Sourcing data ➢ Integrity of data ○ Is it a reliable source or is "fake data"? Which year is the data from? ○ Is it a primary data source or has it been manipulated? ○ Sometimes in media articles, the original source of the data is not clear ("requoted" many times) → conclusions are unverifiable or misleading Scraping data ➢ Web scraping → extracting data from a web site(s) and transforming it into a structured dataset that can be analysed ○ Methods ■ Screen scraping → extract data directly from the source code (html) of a website ● Parsing → directly read the html table into R ● Convert table into a "nice" format readable by R eg. using pander ■ API → interact with an Application Programming Interface which shares data ➢ Can also be scraped from non-websites eg. pdf ○ Scraping from a pdf ■ Download file, read into R and convert to text ■ Remove unwanted lines of text by pattern matching ■ Remove unwanted lines of text by line numbers ■ Split each line of text into table components → create bar chart Cleaning and tidying data

➢ Slicing up a string ○ A string → set of characters, including integers, numbers and spaces ○ Pattern matching and replacement → series of functions used to identify patterns in a string ○ Extract the position in the string where the value is found.

Reshaping data ➢ diplyr ○ Select the Value column and the first 8 rows ○ Remove the white space at the front and back of all column values. ○ Filter by the 2 bedroom units that have been sold. ○ Change characters to numeric values ○ Find mean and sd ➢ Tidyr ○ Not pretty data →

MODELLING DATA ➢ Normal curve ➢ curve(dnorm(x, mean = m2, sd = sd2), col = "darkblue", lwd = 2, add = TRUE)

Fairly symmetric & bell-shaped ➢ How many over 189 ○ data2[data2 > 189] ➢ Chance of finding someone over 189 ○ length(data2[data2 > 189])/length(data2) Normal curve ➢ approximates many natural phenomenon ➢ can model data caused by combining a large number of independent variables General & standard normal curves Standard normal curve (Z) ➢ Mean = 0 and SD = 1 ➢ Domain is can go negative

General normal curve (X) ➢ More versatile and flexible → any mean and SD ➢ Can shift, move mean, stretch it ➢ Exponential of x^2 ➢ Need to know x value, μ and σ (defines the normal curve)

Approximating histogram by a Normal curve ➢ If the Normal curve seems to fit the histogram → can use the area under the Normal curve to approximate area under the histogram ○ Useful as we often only know mean and sd

Area under a Standard Normal Curve ➢ Always sketch the Normal curve and the relevant area … and then use R! ➢ Lower tail area → P(Z < x) ○ pnorm(x) # x = value of x ➢ Right tail area → P(Z > x) (1-pnorm) ○ pnorm (x,lower.tail=F) ➢ Area of interval ○ Big area - small area ○ pnorm(0.8) - pnorm(0.3)

Area under a General Normal Curve ➢ pnorm(x,mean,sd) → lower tail ➢ pnorm(189, 161.9, 7.7, lower.tail = F) → upper tail Special Properties of the Normal

1. All Normal curves satisfy the "68%-95%-99.7% Rule. ○ area 1 SD out from the mean in both directions is 0.68 (68%) ○ area 2 SDs out from the mean in both directions is 0.95 (95%) ○ area 3 SDs out from the mean in both directions is 0.997 (99.7%)

*1 = SD^2 2. Any General Normal can be rescaled into the Standard Normal ○ For any point on a Normal curve → standard units (z score) is how many standard deviations that point is above (+) or below (-) the mean.

How do we know when to use the Normal curve? ➢ Does the histogram look Normal? ○ Is the histogram basically balanced without long tails or many outliers? ➢ Do the proportions look right? ○ Go out by 1 sd (is it 68% etc.) ➢ Does the quantile-quantile (QQ) plot look like a straight line? ➢ Shapiro test

➢ Measurement Error Simulation = one version of your data, using parameters of mean and sd ➢ chopweight = rnorm(20, 550, 5) → 20 trays, 550g, 5sd Measurement error ➢ Individual measurement = exact value + chance error Chance error ➢ Why chance errors occur → way you place it, the environment/diff conditions ➢ To estimate the chance error → replicate the measurement under the same conditions, and calculate the standard deviation (sd of replicated measurements) ➢ This is why there may be different weight when weighed again Outliers ➢ Outliers = small percentage of extreme measurements → measurements more than 3 standard deviations away from the mean (beyond 99.7%) if data is normally distributed (not skewed) ➢ The mean and SD are strongly influenced by outliers (not robust to outliers) ○ The histogram will not follow the Normal curve well.

Standard units (distance of data from mean) ➢ standardunits = (chopweight - mean(chopweight))/sd(chopweight) ➢ hist(standardunits, prob = T) x = seq(min(standardunits), max(standardunits), 100) curve(dnorm(x, mean = mean(standardunits), sd = sd(standardunits)), add = TRUE)

➢ How much is between -1 and 1? ○ length(standardunits[(standardunits <= 1) & (standardunits >= -1)])/20 Distance from the advertised tray weight ➢ Which customer would have been most undercharged? By how much? ➢ max(chopweight - 550) * 0.001 * 23 # convert to kilograms, and multiply by $23/kg Bias (Systematic error) ➢ Bias = constant amount added to or subtracted from each measurement ➢ Can be deliberate or accidental. ➢ Cannot be estimated by replicating the measurements. ➢ ∴ Individual measurement = exact value + chance error + bias Australian Standard ➢ To determine whether measuring instrument is calibrated properly → compare to a working standard (3rd level standard), the compare to national (1st & 2nd level) and international standards

➢ Reproducible Reports REPRODUCIBILITY = data sets and software to be made available for verifying published findings and conducting alternative analyses ➢ can run code in html and verify/change it, ensuring it is transparent ➢ Reproducible research is about being responsible with possible human errors, or worse, detecting intentionally changed results Reproducibility crisis ➢ Selective reporting → failure to show all the data ➢ Not double blind ➢ Failure to repeat experiments ➢ Lack of positive and negative controls ➢ inappropriate use of statistical tests ➢ Use of reagents that were not appropriately validated To make reproducibility better ➢ Better mentorship ➢ Proper use of statistics ➢ More robust experimental design R markdown R Script (.R) ➢ An R script file is a text file which saves your R code and comments ➢ A collaborator runs the .R script file, and then the code and results appear in the console.

R Markdown (.Rmd) ➢ R Markdown → produces dynamic, interactive documents with R. ➢ Saves and executes code AND produces a high quality report for the collaborator to view and validate the results. ➢ An R Markdown file combines (knit = combine all) ○ chunks of text (written in markdown) ○ embedded code (eg R, Python, SQL) ○ Latex Main steps 1) Open file 2) Write report in plain text

3) Choose output in YAML header

4) Embed code 5) Render → knit (combines everything together) Handy tricks ➢ Annotate code as you go ○ ‘’’’ r ’’’’ → code chunk should have commentary ➢ Customise code chunk output ○ ```r``` → code chunk ○ To run code chunks → can run result inline (one line) or in console

➢ Put R code in text

➢ Add hyperlink → [Name of Link](url) ➢ Add images ○ Make a folder called figure, and store 2pencils.jpg in it. ○  ➢ Add videos

➢ Add table

Dangers of Non-reproducible Research (distortion of truth & filter out unfavourable results) ➢ Data versions can change (eg. people edit an Excel file without documenting what has changed and why) ➢ Graphical summaries can change (eg. people can photoshop images without keeping record of what changed and why)

➢ Scatter plots (linear model)

➢ Plotting the pairs of heights creates a cloud of points → Densely clustered ➢ Generally, taller fathers tend to have taller sons. ➢ Dotted line ○ joins points where the son has exactly the same height as the father. ○ If a son’s height is close to his father’s height → father-son point close to the line ○ As there is a lot of spread around this line → weak relationship between father’s height and son’s height.

Bivariate data

➢ involves a pair of quntitative variables → relationship between the 2 variables. Can one variable be used to predict the other? ○ X = independent variable (or explanatory variable, predictor or regressor) ○ Y = dependent variable (or response variable) Scatter plot A graphical summary of 2 variables on the same 2D plane, resulting in a cloud of points.

Linear association ➢ Describes how tightly the points cluster around a line. ○ Tightly clustered around line → strong association ○ Allows for good predictions The scatter plot can be summarised by the following 5 numerical summaries: ➢ mean and SD of X (x¯, SDx) ➢ mean and SD of Y (y¯, SDy) ➢ correlation coefficient (r).

Centre and spread of the cloud ➢ centre = represented by the point of averages (x¯, y¯). ➢ horizontal spread of cloud → SDx. We expect most of the points to fall with 2 SDs from x¯. ➢ vertical spread of cloud is → SDy. We expect most of the points to fall with 2 SDs from y¯.

Correlation Coefficient (r) ➢ numerical summary which measures the clustering around the straight line ➢ indicates sign and strength (tight/loose) of the linear association. ➢ The correlation coefficient is between -1 and 1. ○ If r is positive: the cloud slopes up.

○ ○ ○ ○

If r is negative: the cloud slopes down.

As r gets closer to ±1: the points cluster more tightly around the line When r=±1, all the points lie on a line (no cloud; perfect correlation) When r=0, points don’t fit around the line

Calculating the correlation coefficient The population correlation coefficient (rpop) = mean of the product of the variables in standard units.

For sample correlation coefficient → cor(data$fheight, data$sheight) For population correlation coefficient → n = length(data$fheight) cor(data$fheight, data$sheight) * (n - 1)/n # to match up with hand calculation (Ext)

Why does r measure association? ➢ It divides the scatter plot into 4 quadrants, at the point of averages (centre) ➢ Majority of points in the upper right (+) and lower left quadrants (+, because multiplying both negs) → overall + value of r]

SD Line ➢ connects the point of averages (x¯, y¯) to (x¯+SDx, y¯+SDy) (for r>0) or (x¯, y¯) to (x¯+SDx, y¯−SDy) (for r<0) . ➢ 1 sd in x direction, 1 sd in y direction

Features of the SD line ➢ goes through point of averages. ➢ A father-son pair where both are 0.5 SDs above the mean would lie on the SD line. ➢ But a father who is 1 SD taller than the mean and a son who is 0.8 SDs shorter than the mean, would not be on the line.

Limitations of the SD Line ➢ As the SD line does not use r → insensitive to clustering (does not take into account how tightly the points are clustered in the cloud) ➢ At the extremes with positive [or negative] correlation, the SD Line will over-estimate in RHS [LHS] and under-estimate in LHS [RHS] ➢ We need something better for predictions!

➢ More on correlation

➢ Weak → cor = 0 ➢ Strong → cor = 1 Properties of the Correlation Coefficient ➢ The correlation coefficient is not affected by interchanging the variables. ○ cor(CE, NW) same as cor(NW, CE) ➢ The correlation coefficient is shift and scale invariant → calculated on scaled standardised variables ○ cor(2 * CE + 2, 3 * NW - 3) ■ Shift → add/subtract ■ Scale → multiply divide Misleading correlations ➢ Outliers can overly influence the correlation coefficient ○ Want observations between pos quadrants to maintain strong correlation ○ Outlier → Standard score for CE very large and +ve, for NW very large and -ve ○ Mean product will be extra big and -ve → it is reduced (0.75 to 0.55) ➢ Correlation coefficient can’t detect nonolinear association ○ If want to test strength of relationship (up or down) in general → use method = spearman in correlation function ➢ Nonlinear association can't be detected by the correlation coefficient eg. curved parabola ➢ The same correlation coefficient can arise from very different data ○ Same summary sets eg. mean of x and y, even though very distinct

○ Summary stats alone cannot define data ○ Set 1 → linear model (suitable), set 2 → curved line, set 3 → perfect relationship but one outlier, Set 4→ tend to fit mean of each group, shows that a cluster og high and low points can give a correlation that is good but has no predictability

➢ Rates or averages tend to inflate the correlation coefficient (overestimates strength) ○ Removes site to site variation in each group

➢ Association is not causation ○ Both variables may be simultaneously influenced by a 3rd variable (confounder). ➢ Small SDs can make the correlation look bigger ○ The correlation coefficient measures clustering, not in absolute terms, but relative to the SDs. Summary ➢ Make a scatter plot, yet correlation is only for linear relationships

➢ Regression line Scatter plot CE = data$SydneyCEAQI NW = data$SydneyNWAQI plot(CE, NW, xlab = "CE AQI", ylab = "NW AQI") #plot(x variable, y variable) points(mean(CE), mean(NW), col = "indianred", pch = 19, cex = 2) # indicates point of averages (centre) legend("topright", c("point of averages (centre)"), col = "indianred", pch = 19)

Regression line Option 1 - SD line ➢ visually looked like a good candidate as it connects the point of averages to point of averages + 1 SD ➢ However, it does not use the correlation coefficient → insensitive to clustering ○ underestimates (LHS) and overestimates (RHS) at the extremes.

Best option - Regression line ➢ Uses 5 summaries to describe scatter plot → x¯, y¯, SDx, SDy, r. ➢ Connects point of averages (x¯,y¯) to point of averages + adjusted SD (x¯+SDx,y¯+rSDy) * scaled ➢ We can predict Y from X or X from Y, depending on what fits the context (what is predicted should go on y axis) ➢ Better at capturing extremes

➢ ➢ ➢ ➢ ➢

lm(NW ~ CE) # lm(y variable → x variable) model = lm(NW ~ CE) model$coeff plot(mtcars$mpg, mtcars$wt) abline(model)

SD line vs. Regression line

Graph of averages ➢ plots the average y for each x. ➢ The regression line is a smoothed version of the graph of averages. ○ If the graph of averages is a straight line, that line is the regression line.

Prediction ➢ plots the average y for each x.

Method 1 - Baseline prediction ➢ Given a certain value x, a basic prediction of y would be the average of y over all the x values in the data ➢ mean(NW) Method 2 - Prediction in a strip

➢ Graph of avergaes → Given a certain value x, average of all the y values in the data corresponding to that x value. ➢ data[CE == 99] → for x value 99 ➢ mean(NW[CE == 99]) Method 3 - Regression line ➢ For AQI, we have y=19.8874+0.7138x.

Method 4 - Predicting percentile ranks ➢ If x is in a certain percentile of all the the x's, what percentile would we predict the corresponding y to be in?

1) Find the z score in the x direction : zx.

2) Find the predicted z score in the y direction : zy=r∗zx. (percentile scaled by correlation) 3) Translate zy back to the percentile in the y direction. z_x = qnorm(0.9) #90th percentile for x z_y = cor(CE, NW) * z_x pnorm(z_y) Mistakes in prediction ➢ Extrapolating ○ If we predict from x value that is not within the range of the data → can be completely unreliable

➢ Not checking the scatter plot

○ We can have a high correlation coefficient and then fit a regression line, but the data may not even be linear → always check the scatter plot first and draw residual plot (plot of the gaps between the actual values and the line) ■ plot(mtcars$mpg, model$residuals, xlab = "mpg", ylab = "Residuals") ■ abline(h = 0, col = "blue")

➢ Residual plot When predicting ➢ Should start at x axis to predict y → can’t go backwards ○ X is the independent variable ○ Will have to refit model to suit Residual (prediction error) = vertical distance (or 'gap') of a point above and below the regression line ➢ represents the error between the actual value and the prediction ○ ei=yi−ŷ i → given the actual value (yi) and the prediction (ŷ i) ○ l = lm(NW ~ CE) ○ l$residuals[10] ➢ Under the line → neg residual ➢ Above line → pos residual

The RMS Error ➢ Represents the average gap between the points and the regression line. ○ It is like a "standard deviation for the line" ○ l$residuals

○ sqrt(mean(res^2)) # RMS error (Pop)

The RMS error for Baseline Prediction ➢ uses y¯ for every value of x. ➢ The RMS error for the baseline method is the standard deviation for y

➢ Must scale baseline error ○ sqrt(1 - (cor(CE, NW))^2) * sd(NW) # RMS Error (Sample) ○ sqrt(1 - (cor(CE, NW))^2) * sd(NW) * sqrt((length(CE) - 1)/(length(CE))) # RMS Error (Pop)

Special Cases ➢ Perfect correlation (line): r=±1 ○ RMS error = 0, as the points lie on a line ➢ r=0 ○ RMS error = SDy, as the regression line is no help in predicting y. ➢ Smallest RMS error ○ The smallest possible RMS error is for the regression line.

Residual plot ➢ A residual plot graphs the residuals vs x ○ (kinda like making the regression line the x axis_ ○ plot(CE, l$residuals, ylab = "residuals") ○ abline(h = 0) ➢ If the linear fit is appropriate for the data → it should show no pattern (random about 0) ○ General spread above and below

Vertical strips

➢ Homoscedastic data → If vertical strips on scatter plot show equal spread in the y direction, then the data is homoscedastic ○ The RMS Error can be used as a measure of spread for individual strips ○ Can model by the normal distribution ➢

○ The RMS Error can't be used as a mesaure of spread for individual strips.

The Normal distribution within strips ➢ If the data is homoscedastic → can use the Normal approximation within the vertical strips. ➢ Consider the y values within the strip as a new data set y∗ with ○ New mean in strip = mean of y + correlation x standard unit in strip x SD of y

1) 2)

3) 4)

Standard example

➢ Linear Regression Summary Steps to use given bivariate (x,y), quantitative data 1) Produce a scatter plot → Does it look linear?

2) Produce a Regression line

3) Calculate the correlation coefficient (r) → How strong is the linear association?

4) Produce a residual plot → Does it look random? Is the linear model appropriate or would another model be better?

5) Check assumptions → Does the data look homoscedastic? a) If fanning → not homoscedastic and assumptions not appropriate b) If yes, you can perform predictions 6) Perform predictions a) Predict y for given x b) Predict y within a vertical strip More on regression Transformation ➢ If the data is very spread out → try transforming 1 or both of the original variables. ➢ Eg. take ln(y) or ln(x) as the new variables Multiple regression ➢ look at the connection between multiple variables (y and 2+ x variables)

➢ Multicollinearity occurs when 2 variables are highly correlated with each other ➢ lm(Sold ~ Bedrooms + Bathrooms + Carspots)

➢ Non Linear Models Why is it non-linear?

-

From 0-1 hours → appears linear But not overall linear trend

Linear model vs. non-linear model Linear model A (statistical) linear model takes the form ➢ y=a+bx (a and b are constants) Advantages ➢ Easy to fit ➢ Results in a simple equation (analytical solution) ○ Can prove ➢ If we increase the power or degree (multiple regression) we can fit many shapes

Disadvantages ➢ Hard to link parameters (slope and intercept) to mechanistic processes, eg. biological or environmental context.

Non-linear model cannot be written as a linear model Advantages ➢ Most are linked to mechanistic processes → interpretable parameters ➢ parameters may be directly related to processes under study eg. carrying

Disadvantages ➢ Numerical solutions are often needed, but this is now straight-forward on a computer

capacity or maximum rate of growth. ➢ The constraints (eg. asymptotes) can easily be built in Linear model vs. non-linear model Quadratic model

➢ Plot of data with predicted values as curve Cubic model

➢ Better → r is closer to 1 ➢ Not bad up to 6 hours ➢ Can’t be used after 6 hours → will go up Cubic model Fit linear model to log of readings

➢ Bad but play around with parameters to fit exponential model

Von Richter model

➢ Nice decline with data ➢ Best model for BAC Other models Exponential model ➢ Uses: biology (population), physics (nuclear chain reaction), econcomics/finance, IT (processing power)

Exponential decay ➢ fluid dynamics, luminescence, geophysics, vibrations

Logistics growth

➢ Uses: ecology (population), linguistics (language), agriculture (crops), chemistry (reactions), machine learning (neural networks)

Gompetz ➢ Uses: cancer tumours, bacterial growth

SAMPLING DATA ➢ Chance The prosecutor’s fallacy The prosecutor's fallacy = mistake in statistical thinking where it is assumed that the probability of a random match is equal to the probability that the defendant is innocent. ➢ Probability that A given B = Probability of B given A ➢ It has used by the prosecution to argue for the guilt of a defendant during famous criminal trials ➢ It can also be used by defense lawyers to argue for the innocence of their client.

Example

Sally clark case ➢ Ignored association between events (can’t be treated as separate events) → one child dying increases chance of second child ➢ Prosecutor’s fallacy

People vs. collins case ➢ Association between events ignored → maybe lots of black men have a beard ➢ Prosecutor’s fallacy Properties of chance

CHANCE (frequentist definition) = Chance (or probability) is the percentage of time a certain event is expected to happen, if the same process is repeated long-term (repeat event many times)

Conditional probability CONDITIONAL PROBABILITY = Conditional probability is the chance that a certain event occurs, given another event has occurred

Multiplication Principle The probability that 2 events occur is the chance of the 1st event multiplied by the chance of 2nd event, given first occurs)

Eg. Drawing without replacement ➢ Often results in dependent events as the probability of each event depends on the outcome of the previous event ➢ Not simply a product of each unconditional probability.

Independence & dependence Independent events INDEPENDENCE = if the chance of the 2nd given the 1st is the same as the 2nd eg. tossing a coin ➢ Conditional probability = unconditional probability

➢ If 2 events are independent → chance of both occurring is the product of their unconditional probabilities (special case of multiplication principle)

Eg. Sally Clark Case (not independent)

Eg. Tossing a coin (independent)

➢ More on chance Making lists ➢ Lists can only be used for really small problems Simulating in R ➢ Allows close estimate to theoretical probability → closer with more replicates Eg. Two dice are thrown. What is the chance of getting a total of 6 spots? set.seed(1) totals = sample(1:6, 1000, rep = T) + sample(1:6, 1000, rep = T) ← always ⅙ , shows the two dice table(totals) barplot(table(totals), main = "1000 rolls: sum of 2 dice")

Addition rule MUTUALLY EXCLUSIVE = when the occurrence of one event prevents the other

(Difference between mutually exclusive and independence)

ADDITION RULE = If 2 events are mutually exclusive then the chance of at least 1 occuring is the sum of the individual chances

How the Chevalier de Mere stopped losing Game A: Toss a die 4 times. Win = at least 1 "Ace".

Game B: Toss a pair of dice 24 times: Win = at least 1 double-Ace.

P(A and B) = red P(A or B) = green red and blue P(A|B) = red/red+blue = ⅖

➢ Binomial formula Simulating coin tosses set.seed(5) table(sample(c("H", "T"), 4040, replace = T))/4040 → 4040 tosses

Binomial coefficients ➢ Binomial model allows us to work out the exact chance of P(Head) ○ N factorial (n!) = the number of ways of rearranging n distinct objects in a row ○ Product of falling numbers eg. 3! = 3 x 2 x 1 = 6 ○ Factorial n R → factorial(6)

BINOMIAL COEFFICIENT = gives number of ways of rearranging the n objects ➢ Suppose we have n objects in a row, made up of 2 types: x (type1) and n−x (type2). ➢ Eg. person wearing shorts in position 1 is same as person wearing shorts in position 2

Binomial coefficient in R Eg. P(getting 2 heads in 6 tosses of a coin) choose(6, 2) Binary trial BINARY TRIAL = where only 2 things can occur, so P(event) = p and P(not event) = 1-p. ➢ Coin tosses can be a model for any binary trial Eg. Head and not head, even and odd dice, 6 or not 6 (using biased coin), medical trials Binomial theorem ➢ If we have n independent, binary trials, with P(event)=p at every trial (same chance at every trial), and n is fixed ➢ The chance that exactly x events occur is

Binomial theorem in R dbinom(x, n, p) Eg. A fair coin is tossed 500 times. What is the probability of getting 300 heads? dbinom(300, 500, 0.5) = 1.544255e-06 Must test the assumptions Eg. Suppose 100 babies are born at RPA hospital today with P(boy) = 0.51. ➢ Can be used however p may not be fixed eg. having twins/triplets

➢ Law of Averages

Chance processes ➢ Every time you toss a fair coin, there is chance variability

➢ Chance error = difference between what we expect and observe ➢ As the number of tosses increases ○ the absolute size of chance error increases ○ Yet the absolute percentage size of chance error decreases Law of Large Numbers (or Law of Averages) LAW OF AVERAGES = the proportion of heads becomes more stable as the length of the simulation increases and approaches a fixed number (the relative frequency) ➢ As the number of trials increases → the observed proportion will converge to the expected proportion ➢ Chance error in the number of heads is likely to be large in absolute size, but small relative to the number of tosses. ➢ OR more people sampled → sample average closer to population average

Gambler’s fallacy ➢ After a sequence of 4 heads in a row, the chance of getting a tail increases. ○ Independent event, unaffected by other events The box model ➢ A simple way to describe the chance of generating a number ○ Box = summary of the population (what's in there, and in what proportions). ○ Sample = draws from the box → what is the expected value (EV) of the sum or mean of the sample? ➢ Must know ○ the distinct numbers that go in the box/population ("tickets").

○ the number of each kind of tickets in the box (proportion of each type of ticket) ○ the number of draws from the box (how big sample is) The box model for gambling ➢ tickets represent the amount won (+) and lost (-) in each play ➢ chance of drawing a particular value = chance of winning that amount in 1 play ➢ number of draws = number of plays The net gain is the sum of the draws from the box (sum of how much you win/lose) Eg. Throw a fair dice 25 times set.seed(1) dietosses = sample(c(1:6), 25, repl = T) dietosses

Eg. Model game ➢ it costs $1 to play a game. ➢ If you roll a "6" → you get back your $1, plus win another dollar. ➢ If you get any other number → you lose your $1. set.seed(1) dietosses = sample(c(1, -1, -1, -1, -1, -1), 25, repl = T) # Or: dietosses= sample(c(1,-1), 25, repl = T,prob=c(1/6,5/6)) sum(dietosses) ➢ The distinct tickets are $1 and -$1, representing the "win" and "loss". ➢ Net gain of getting 6 → 1 ➢ Net loss of getting 2 → -1 ➢ If played 25 times, net loss → -21 Eg. Multiple choice quiz ➢ A quiz has 25 multiple choice questions, with 5 answers each ➢ You get 4 points for a correct answer. ➢ You lose 1 point for an incorrect answer.

set.seed(1) answer = sample(c(4, -1), 25, repl = T, prob = c(1/5, 4/5)) head(answer) cumsum(answer)[25]

➢ Expected Value & Standard Error Sum of draws from a box model

➢ Result for the standard error (SE) → square root law ➢ As box represents a population → SD of the box is the population SD How to work out the SD of the box

How does chance error relate to standard error? ➢ An observed value is likely to be around expected value with a chance error similar to the SE ➢ Observed values are rarely more than 2 or 3 SEs aways from the expected value. ○ Assuming a normal distribution Example → Keno Classic Step 1. Draw box model

*chance of winning individual draw is ¼

Step 2. Calculate the mean, then EV

box = c(2, -1, -1, -1) n = 100 n * mean(box)

Step 3: Calculate the SD, then SE

library(multicon) sqrt(n) * popsd(box)

Step 4: Conclusion ➢ In 100 plays of Classic Keno we expect to lose $25 (EV) with a SE of $13. ➢ Hence, it would be very common to lose between $12 and $38.

Simulation set.seed(1) totals = replicate(20, sum(sample(box, 100, rep = T))) table(totals) Sample from box 100 times without replacement Sum up results → net winnings of 100 plays Replicating 20 times → 20 students play keno classic Looking at a region length(totals[totals >= -38 & totals <= -12])/20 70% of people losing between $12 and $-38 hist(totals)

The Normal Curve in the box model ➢ For large amounts of draws from the box, the observed value of the sum often follows the Normal curve.

In 100 plays of Keno Classic, how often would you lose more than $20? (Assume a Normal curve).

Step 5. Draw the Normal curve using EV and SE.

Red axis → (data point - EV)/SE Step 6. Work out the chance ➢ The Normal curve has EV -25 and SE 12.99 (from previous working). ➢ The x value is -20. ➢ The z score is (−20−(−25))/12.99= 0.3849 → threshold in standard units ➢ So we expect the loss to be $20 or more around 0.65 of the time. ev = 100 * mean(box) se = sqrt(100) * popsd(box) pnorm(-20, ev, se) pnorm(0.3849) Using the box model for classifying and counting ➢ We can use the box model for the sum of draws from a population with many types of tickets. ➢ Yet often interested in 1 particular "ticket" in the box (binary box), like in Keno Classic.

(6 or not 6)

➢ Normal approximation The probability histogram

3 types of histograms

Probability histogram Represents chance by area

sum = c(2:12) chance = c(1, 2, 3, 4, 5, 6, 5, 4, 3, 2, 1)/36 t = data.frame(sum, chance) barplot(t$chance, names.arg = t$sum, main = "Probability histogram of sums")

➢ On horizontal axis ○ EV measures the centre ○ SE measures the spread

Simulation histogram Represents chance by area, for a simulation of a chance process ➢ For repeated simulations of a chance process resulting in a sum → simulation histogram of the observed values converges to the probability histogram

par(mfrow = c(1, 4)) # breaks=c(2:12) breaks = c(0.5:12.5) hist(sim1, br = breaks, freq = F) hist(sim2, br = breaks, freq = F) hist(sim3, br = breaks, freq = F) hist(sim4, br = breaks, freq = F)

Central Limit Theorem ➢ When drawing at random with replacement from a box → if the sample size is sufficiently large, the probability histogram for the sum (or average) will closely follow the normal curve ○ Even if box does not eg. box is exponential ➢ Starting distribution determines how large sample must be ○ The crazier the initial distribution → need bigger sample size

Sample size vs replicates

➢ Sample size (n) → how many things dropping down that you average ○ Increasing sample size approaches normal curve ➢ Replicates (N) → repeat process ○ Increasing replicates will approach box distribution (not normal curve)

The Continuity Correction (cc) ➢ To approximate a discrete distribution by the Normal distribution (continuous) → adjust endpoints by 0.5 ➢ To work out whether to add or minus 0.5, draw a sketch of the histogram.

Eg. What is the probability of getting a sum between 6 and 8? data histogram length(sim4[sim4 >= 6 & sim4 <= 8])/length(sim4)

Normal curve pnorm(0.41) - pnorm(-0.41) → standard units between +-0.41

➢ ○ Lower threshold = 6 -> 5.5 ■ standard unit = -0.6 (approx) ○ Upper threshold = 8 -> 8.5 ■ standard unit = 0.6 (approx) ○ ∴ pnorm(0.6) - pnorm(-0.6)

➢ Sample Surveys Population vs. sample ➢ Population = full amount of information being studied → through a census ○ Limitations of a census ■ Hard ■ Time consuming ■ Expensive ■ Requires lots of resources. ➢ Sample = part of the population eg. polls ○ Why? ■ Can’t access all info ■ Too expensive ■ Too time consuming Parameter & estimate ➢ Parameter = a numerical fact about the population (no replacement) eg. population mean μ, percentage of US population voting for Trump ○ What you want to know ➢ Estimate/statistic = calculation of sample values which best predicts the parameter eg. sample mean x¯ ○ Evidence that you have Finding the best estimate of the parameter ➢ How was the sample chosen? Is it representative of the population?

➢ What estimate is closest to the parameter?

Selecting a sample Types of bias ➢ Selection bias → tendency to exclude/include one type of person from the sample ➢ Non-response bias → participants fail to complete surveys ○ Must include response rate ○ Non-respondents can be very different to respondents ➢ Interviewer's bias → ○ when interviewer has to make a choice of participants in the survey ○ Or characteristics of the interviewer have an effect on the answer given by participants ➢ Measurement bias → when the form of the question in the survey affects the response to the question ○ Bias in question wording and order eg. should be objective not subjective ○ Recall bias → people forget details eg. Drugs, age, tax ○ Sensitive questions → people may not tell the truth ○ Lack of clarity in question → may be misinterpreted, certain words may have different meanings ○ Attributes of the interview process may cause bias Bias and sample size ➢ When a section process is biased → larger sample does not reduce bias, but can amplify How to pick a good sample ➢ Fair sampling procedure → representative cross section of the population. ➢ Probability method ○ interviewer is not involved in the selection (impartial) ○ Defined procedure for selecting sample by chance (objective) eg. talk to every 5th person ○ Eg. Simple random sampling involves drawing at random without replacement (yet often not practical) 1. Multi-stage cluster sampling a. Probability sampling technique → takes samples in stages, and individuals or clusters are chosen at random at each stage

2. Quota sampling a. Non-probability sampling technique → assembled sample has the same proportions of individuals as the entire population (respect to traits etc.) b. Yet results in unintentional bias from the interviewers when they choose subjects to survey

3. Convenience sampling a. Non-probability sampling technique → subjects selected because of convenient accessibility b. Not recommended, except possibly to test a survey (pilot) Unavoidable bias

Why the US election failed ➢ Random digit dialing ○ Non-response bias ○ Some groups (eg. less educated voters) consistently hard for pollsters to reach ○ Anti-institutional feelings may also have aligned with unwillingness to respond to polls ➢ Some people lie to pollsters ○ Social undesirability → ‘secret Trumpers’ ➢ Hard to capture real voter → can’t know in advance who is actually going to vote

➢ Chance Errors in Sampling We can model a simple survey by a box model with 0 and 1 in the relevant proportions ➢ 1 is the person you’re concentrating on Summary of box model

Mean of the box

box = c(rep(1, 3091), rep(0, 3581)) mean(box) Standard Deviation of the Box Big/small refers to the ticket number g. 1, 0

EV and SE in sum and proportion

Effect of sample size on SE

Bigger sample size → bigger SE sum

Bigger sample size → smaller SE sum Modelling by the Normal Curve ➢ Only requires centre and standard units ○ Whenever you have mean and SD or EV and SE → can model by a normal curve

Sum of the sample What is the chance that the number of men in the sample (sum) is between 40 and 50?

➢ Sum → must standardise by EV and SE

➢ pnorm(0.7380666) - pnorm(-1.267549) OR

Proportion in the sample What is the chance that the number of men in the sample (sum) is between 40 and 50?

➢ pnorm(0.736061) - pnorm(-1.269555) The correction factor ➢ Sampling with replacement (box model) ○ SE is determined by sample size ○ Proportion in box stays the same eg. Drawing with replacement, the box (0,1) is equivalent to (0,0,1,1) ■ Both boxes have the same mean 0.5 and SD 0.5. ➢ Without replacement ○ increasing the ratio of sample size to population size (higher proportion sampled) → variability decreases → SE decreases ○ If small part of population → almost no effect on SE of the estimate (correction factor almost 1) ○ We need to slightly adjust the SE from the box model, to get the exact SE.

➢ Accuracy of Proportions When you have a 0 and 1 (binary box) → mean = proportion Bootstrapping ➢ Bootstrapping = estimating the properties of the population, by using the properties of the sample

Bootstrapping estimates

Bootstrapping distribution Using bootstrapping, find the chance that the proportion of support in the population is less than 0.625. set.seed(123) N = 16655856 box = c(rep(1, 0.64 * N), rep(0, 0.36 * N)) means = round(replicate(10000, mean(sample(box, 1012, rep = F))), 3) cumsum(table(means))

Confidence Intervals Confidence interval → estimates how likely our estimate is ➢ Different samples of the same population give different results → sampling error ➢ We have often taken the estimate of the chance error to be 1 unit of the SE ➢ ∴ the proportion of Australians supporting plebiscite is anywhere between 0.64 ± 0.015 (SE) ➢ Chance error could be out by as much as 2 or even 3 SEs → generalise, using Confidence Intervals.

Interpreting Confidence Intervals For a 95% Confidence Interval: ➢ Cannot say "the probability that the interval contains the unknown parameter is 0.95". ➢ Rather, we say 'if we worked out a series of CIs for a series of samples, then 95% of the CIs would contains the unknown parameter."

Factors affecting width of confidence intervals ➢ Variation in population ○ More varied population → more varied sample → wider confidence interval ➢ Sample size ○ Small samples → more variation between samples → wider confidence interval ○ Larger samples → narrow confidence interval

DECISIONS WITH DATA ➢ Hypothesis HYPOTHESIS TESTING = scientific method for weighing up the evidence given in the data against a given hypothesis (model)

➢ If gap between observed value (data) and the expected value (assuming the hypothesis) is bigger than 2 or 3 standard errors → data not consistent with hypothesis ➢ Data = evidence Framework for hypothesis testing

Hypothesis ➢ Null hypothesis H0 → assumes that the difference between the observed value (data) and expected value (EV) is due to chance alone ○ Gap between expected and data is small ○ If sample doesn’t match H0 → evidence for H1 ➢ Alternative hypothesis H1 → assumes that the difference between the observed value (data) and expected value (EV) is NOT due to chance alone

Example: Often start with 2 sided alternative (80 or not 80)

Example: Move to 1 sided alternative (if research suggests)

Risk of using 1 tailed test → miss out on info eg. maybe far less than 80% saw improvement Assumptions ➢ A conclusion is not transparent → if the assumptions are not stated ➢ A conclusion is potentially invalid → if the assumptions are not justified Example

Test Statistic ➢ Measures the difference between what is observed in the data and what is expected from the null hypothesis ➢ Standardised the gap to tell how unusual observed value is relative to H0

P-value (observed significance level) ➢ Way of weighing up whether the sample is consistent with H0 ➢ Shows chance/likelihood of observing the test statistic (or something more extreme) if H0 really is true → less than 5% is statistically significant ➢ Low P value → suggestion to reproduce experiment to help validate results

Common mistakes with p value

Conclusion To reject null hypothesis → P value < 0.05 ➢ “If the P value is low, the null hypothesis must go”

➢ Proportion Test (null hypothesis about proportion) *Applying proportion test to only treatment group

Hypothesis 1 sided

Assumptions ➢ The participants in the treatment group are independent of each other ➢ The chance of becoming desensitised is the same for all participants Test statistic ➢ Assume H0 is true → then we can model the Treatment participants by a simple box model where a 1 ("shows desensitisation") and 0 ("doesn't show desensitisation") ➢ 26 1s got better → summing of the 1s ∴ sum of the sample

P value

➢ Or lower.tail = F ➢ Look at upper tail bc alternative hypothesis is >80 ➢ P-value is just larger than 0.05 Conclusion ➢ Not enough evidence against H0 → Retain H0 ➢ Data is consistent with the null hypothesis ➢ Therefore, the new treatment does not have an effectiveness rate higher than 0.8. ➢ We have not proved that the effectiveness rate is 0.8. Rather, we have just failed to find sufficient evidence to claim an effectiveness rate of higher than 0.8.

Effect of H1 on P values ➢ 1 sided ○ Specifies the direction of the altenative hypothesis. Eg H1:p>0.8. ➢ 2 sided ○ Does not specify the direction of the alternative hypothesis. Eg H1:p≠0.8. ○ In this case we (usually) double the p-value.

Simulating P-values

P value = chance of getting what you got or something more extreme ➢ 26 or more → 8+3+1/100 Overall ➢ Box model represents null hypothesis

➢ Proportion Test Hypothesis

Assumptions The pack of card is well shuffled, and has exactly 52 cards.

Test statistic

*0.1

P value

Conclusion ➢ The data gives evidence against the null hypothesis → didn’t fit with H0 (5 is extreme) ➢ The result is highly statistically significant ➢ Therefore, the pack of card does not seem to be balanced.

Finding the exact p value ➢ What is the exact distribution of the binary box? → Binomial theorem

x = 5 # this is the test statistic dbinom(x, 5, 0.5) # this is the p-value

➢ Accuracy of means (null hypothesis about mean) Mean of draws from a known box ➢ Know the population mean and SD ➢ We take a simple random sample of size n, and consider the chance error of the sample mean. ie How close is the sample mean to the population mean?

eg.

Known Mean and SD of the box ➢ mean = 4 ➢ SD = 2 Calculate & average the 3 sample means ➢ 4.2, 4.24, 3.8 → average = 4.08 ○ The gaps between the population mean and the sample means are {0.2,0.24,0.2} ○ The variation between sample means is around 0.2.

BOX MODEL FOR MEAN OF DRAWS

(our simulation results fit)

➢ To make sample mean closer to the population mean (a better estimate) we can increase the sample size.

The Central Limit Theorem for the Sample Mean ➢ When drawing at random from a population (box), the distribution of the sample means is Normally distributed, even if the box does not follow a Normal distribution

Mean of draws from an unknown box ➢ Using mean and SD of sample to estimate mean/SD mean(caf 0) = 46.4 sd(caf 0) = 12.4 mean(caf13) = 58.1 sd(caf13) = 15.1 Differences between 0 and 13 mg caffiene cafdiff <- caf13 - caf0 1.50 6.83 22.57 13.13 35.29 3.09 5.91 9.20 7.86 Mean of differences mean(cafdiff) → 11.7 mins SD of differences ➢ Tells us how far each individual cyclist varies from the mean in this sample of 9 riders. sd(cafdiff) → 10.8 mins SE of differences (scaled) ➢ Tells us how far the sample means vary from the true population mean value, for all elite cyclists sd(cafdiff)/sqrt(length(cafdiff)) → 3.6 mins Conclusion

Confidence interval represents sensible null hypothesis

➢ If H0 = 0 mins ○ H0 does not fit in confidence interval ∴ reject H0 (not sensible value) ➢ If H0 = 6 mins ○ Accept H0

➢ Model for measurement error MEASUREMENT ERROR = chance error in measurement

The Normal (Gauss) model for measurement error ➢ Repeated measurements are made of the same quantity ➢ When we make a series of n repeated measurement → each measurement differs from the exact value by a chance error ➢ Model chance errors as n draws from the Error Box with zero mean (meanErrorBox=0). ○ Assumes mean error is zero ○ Each draw from box → chance error for 1 measurement ○ We add that draw to the exact value to give the actual measurement eg. 1st measurement = exact measurement + 1st draw from error box = 3 + Error ➢ Applies to continuous data (not binary)

SD of error box ➢ SDErrorBox Usually unknown

➢ For a large number of measurements → can use the SD of the draws to estimate the SD of the error box (inferred from SD of data) Bias in the Gauss Model for Error Box? ➢ Bias → measurements we make may be consistently under-predicted or over-predicted ○ We want bias to equal 0 ➢ SD tells us how the far the mean is from the exact value + bias

Estimating the accuracy of a mean Error of 1 measurement ➢ Estimate by calculating the SD of repeated measurements ➢ The chance error of 1 tray of chops would be around the SD Given sample set.seed(1) chopweight = rnorm(20, 550, 5) → draw from normal distribution, 20 trays, mean = 550, sd = 5 mean(chopweight) → 550.9526 for each chop sd(chopweight) → 4.566269 for each chop

Error in a series of measurements ➢ every individual measurement has error → so will the average of these measurements ○ This average is more accurate than any individual measurement

EV → mean(chopweight) SE → sd(chopweight)/sqrt(20) ➢ A 95% CI would be 550.95±2×1.02 giving [548.91,552.99] ○ 95% of time, CI contains the true mean

Example

Caution with time series data ➢ Box models only apply if tickets are drawn independently with replacement ➢ However when looking at observations (eg. stock market indices and weather) over time, there are often trends or patterns which violate the independence assumption. ○ One point in time impacts the next ➢ Hence the box model does not apply to time series data

➢ Z and Tests Mean of difference = 11.7 SD of difference = 10.7 (lots of variation) The Z test (larger samples) We have to assume

➢ We know the population SD (for the box) ➢ OR the sample SD can estimate the population SD → If the sample size is small (<30), then not reasonable

Hypothesis (2 sided) ➢ Null hypothesis → difference is by chance alone ○ Underlying population has specified mean ○ Caffeine → assume treatment effect is zero (mean = 0)

➢ Alternative hypothesis → difference is not due to chance ○ The treatment effect is not zero Assumptions ➢ Random sample of cyclists → all independent, none related to each other ➢ The box is Normal, or we have a bigger enough sample to enable use of the CLT. ➢ population SD = sample SD as we don't know the population SD here.

Test statistic

➢ SE for the MEAN of the sample ➢ (mean(cafdiff) - 0)/(sd(cafdiff)/sqrt(length(cafdiff))) → 3.252508 standard errors away ○ Not popsd as it is a sample ➢ From normal curve → 99.7% observations lie with 3 SE of the mean. ➢ Hence it is highly unlikely to have observed data that lies that far away from the mean P-value ➢ 2 * pnorm((mean(cafdiff) - 0)/(sd(cafdiff)/sqrt(length(cafdiff))), lower.tail = F) → 0.001143915

➢ about 1 in 1,000 chance of getting a test statistics → highly unlikely The Z test for counting and classifying non-continuous data ➢ DIFFERENCE from other Z test → use the SE for sum of sample, SD given by null hypothesis and not estimated ➢ considering a 0-1 box model → Proportion Test ➢ test the proportion of people who have a certain → 1's representing the preference of interest

P-value = 0.0002326291 Conclusion ➢ Hence, it would appear that the subjects are clairvoyants ➢ However other possibilities ○ Was the Aquarius random number generator working properly? Was there an equal number of targets chosen or a pattern? ○ Did Aquarius give the subjects any indication as to which target was chosen?

T-test (smaller samples) ➢ Uses sample SD and T distribution ➢ The T distribution varies in shape according to the sample size → specified by the "degrees of freedom" (sample size - 1)

Test statistic ➢ sample SD to calculate the SE. ➢ T-distribution to calculate a p-value rather than the Z-distribution ➢ Sample size dictates which T-distribution we use ○ Stretches distribution according to n ○ As sample size increases → closer to normal distribution, P-value gets smaller ■ Less area under the curve on the sides

t.test(mu = 0, cafdiff) → mu is the hypothesis Or t.test(a, b, mu = 0, alt = "two.sided", paired = T)

Now the P-value is larger, due to the sample size of n=9 Comparing Z and T test

➢ 2-Sample T-Test Inference =making a decision about population parameter(s) based on a sample.

2 sample T test

Assumptions ➢ The 2 samples are independent ○ different people ○ The paired differences can eliminate personal effect → yet harder to find the same person to undergo both treatment and control ➢ The 2 populations have equal spread (SD/variance) ○ eg. check → Boxplots (suggest), Histograms, Variance Test (confirms)

○ Levene's Test (F-Test) → confirms equal spread ■ tests the null hypothesis that the spread is equal between the two populations.

■ var.test(No_RB, RB) ■ Want to retain H0 → p > 0.05 ○ If not → Welch test ➢ The 2 populations are normally distributed ○ eg. check → Boxplots (expect no or few outliers), Histograms, QQ-Plots, Normality Test ○ Comparative boxplot → ■ Normality: the 2 samples look symmetrical and so are consistent with Normality, although the control one looks a bit asymmetric ■ Equality of variance: 2 samples have similar spread and so are consistent with the equality of variance assumption.

○ Q-Q plot → ■ graphs theoretical quantiles based on the normal curve against the actual quantiles.

■ Slice data into diff percentages and see if proportions match up ■ Normality: line reasonably straight, we can assume the data is normally distributed

○ Shapiro Wilk test → confirms normality

■ Tests null hypothesis that the data is Normal

■ shapiro.test(RB) ■ Want to obtain H0 → p > 0.05 Test statistic

➢ t.test(No_RB, RB, var.equal = T) Conclusion

What if the assumptions are not met? → other tests

Unequal variance ➢ Welch 2-sample T-test ○ Leave off var.equal = T → t.test(No_RB, RB, var.equal = FALSE) ○ P > 0.05 Non-normal data

➢ Can transform non-Normal data (eg. log or square root) before performing a T-test. ➢ use non-parametric tests such as the Mann-Whitney-Wilcoxon test → wilcox.test (particularly for smaller datasets) Non-independent data ➢ T-test ➢ May design an experiment with dependent to control variation between experimental groups.

➢ Chi Square Tests (for qualitative variables) Chi squared tests (χ2) Used in 3 ways 1. Goodness of Fit → Test a hypothesis about the distribution (model) of a qualitative variable in a population.

2. Homogeneity → Test a hypothesis about the distribution of a qualitative variable in several populations.

3. Independence → Test a hypothesis about the relationship between two qualitative variables in a population.

Test statistic (*sum over all categories)

Test for Goodness of Fit of Model (1 variable)

How significant are the gaps?

Hypothesis H0 = the gambler is innocent ➢ drawing 60 times from a box, where the box has equal numbers of 1,2,3,4,5,6 (model) ➢ proportions are equal → ⅙ each H1 = the gambler is guilty ➢ drawing 60 times from a box, where the box does not have equal numbers of 1,2,3,4,5,6 ➢ at least one proportion is different the stated proportions Assumptions ➢ Expected categories ○ None are empty (get rid of that category) ○ No less than 20% are < 5 ■ If 2 categories are small (eg. 2 and 3) → pool them together Test statistic We combine all this data by considering the weighted sum of the gaps ➢ Must square because if you just add them up → will be 0

sum(((table(throws1) - 10)^2)/10) test statistic = 4.4 - As the test statistic is based on the 'gaps', large values of the test statistic will argue against H0 P-value ➢ Chance of observing 4.4 or more extreme on a chi-squared distribution (with k−1 degrees of freedom) → where k=number of categories ○ Here: degrees of freedom = 6-1 = 5. ➢ pchisq(4.4, 5, lower.tail = F)

➢ High p-value → retain H0 Quicker in R Use Chi squared to test against equal chances

If unfair dice

Test for Independence (2 variables)

*double degrees of freedom Summarise data in contingency table

➢ 9,4, 6, 6 are our 4 observed values ➢ Our expected values → row x column / total eg. male smokers → 13 x 15 / 25 ➢ R looks at gap between O and E Hypothesis ➢ H0 = Smoking is independent of gender (no association between smoking and gender) ➢ H1 = Smoking is NOT independent of gender (an association between smoking and gender) Test statistic and p-value chisq.test(Smoke)

Conclusion ➢ Retain H0 (no gender bias) Mosaic plot with standardised residuals ➢ Indicate the "gaps" (size and direction) for each individual combination in the 2x2 matrix ➢ lack of shading indicates that these "gaps" are not large.

➢ Regression Tests 2 quantitative variables Fit a line to the data ➢ To find the "best" line → we summed the square of the residuals ("gaps") between our line and our observations → looked for line with minimum sum.

Is it a good line? ➢ May be the best line but not a good line ➢ Equation of the sample line → y = a + bx ➢ As slope (b) approaches 0 → no trend

1- Sample T-Test to Test the Slope

Hypothesis

Assumptions ➢ The residuals should be independent, normal, with constant variance (homoscedasticity) ○ Check: Residual Plot ➢ The relationship between the dependent and independent variable should look linear ○ Check: Scatter Plot Test statistic ➢ Under H0 → EV of slope should be 0 ➢ OV = -0.08 ➢ To find SE → summary(lm(Value ~ Time, PCBeer))

➢ N - 2 degrees of freedom → 2 parameters (slope and intercept) P-value & conclusion ➢ p-value=7.18×10−10 → highly significant ➢ Reject H0 ∴ a linear trend