COGS 108 - Assignment 4: Data Analysis Important Reminders You must submit this file ( A4_DataAnalys

downloaded from

Scholarfriends

https://scholarfriends.com Downloaded by: browsegrades@gmail.com | https://scholarfriends.com/singlePaper/186923/cogs-108-assignment-4-data-analysis-importantreminders-you-must-submit-this-file-a4-dataanalys Distribution of this document is illegal.

5/20/2019

A4_DataAnalysis

COGS 108 - Assignment 4: Data Analysis Important Reminders You must submit this file ( A4_DataAnalysis.ipynb ) to TritonED to finish the homework.

This assignment has hidden tests: tests that are not visible here, but that will be run on your submitted assignment for grading.

This means passing all the tests you can see in the notebook here does not guarantee you have the right answer!

In particular many of the tests you can see simply check that the right variable names exist. Hidden tests check the actual values.

It is up to you to check the values, and make sure they seem reasonable.

A reminder to restart the kernel and re-run the code as a first line check if things seem to go weird.

For example, note that some cells can only be run once, because they re-write a variable (for example, your dataframe), and change it in a way that means a second execution will fail.

Also, running some cells out of order might change the dataframe in ways that may cause an error, which can be fixed by re-running.

In [1]:

1 2 3 4 5

# Run this cell to ensure you have the correct version of patsy # You only need to do the installation once # Once you have run it you can comment these two lines so that the cell doesn't execute everytime. import sys !conda install --yes --prefix {sys.prefix} patsy=0.5.1

Collecting package metadata: done Solving environment: done ## Package Plan ## environment location: /Users/brianbarry/anaconda3 added / updated specs: - patsy=0.5.1 The following packages will be downloaded: package | build ---------------------------|----------------conda-4.6.7 | py36_0 1.6 MB openssl-1.1.1b | h1de35cc_0 3.4 MB patsy-0.5.1 | py36_0 376 KB -----------------------------------------------------------Total: 5.5 MB The following packages will be UPDATED: conda openssl patsy

4.6.4-py36_0 --> 4.6.7-py36_0 1.1.1a-h1de35cc_0 --> 1.1.1b-h1de35cc_0 0.5.0-py37_0 --> 0.5.1-py36_0

Downloading and Extracting Packages conda-4.6.7 | 1.6 MB | ##################################### | 100% patsy-0.5.1 | 376 KB | ##################################### | 100% openssl-1.1.1b | 3.4 MB | ##################################### | 100% Preparing transaction: done Verifying transaction: done Executing transaction: done In [3]:

1 2 3 4 5 6 7 8 9 10 11 12 13

# Imports - These are all you need for the assignment: do not import additional packages %matplotlib inline import numpy as np import pandas as pd import matplotlib.pyplot as plt import patsy import statsmodels.api as sm import scipy.stats as stats from scipy.stats import ttest_ind, chisquare, normaltest # Note: the statsmodels import may print out a 'FutureWarning'. Thats fine.

Notes - Assignment Outline Parts 1-6 of this assignment are modeled on being a minimal example of a project notebook.

This mimics, and gets you working with, something like what you will need for your final project.

Parts 7 & 8 break from the project narrative, and are OPTIONAL (UNGRADED).

They serve instead as a couple of quick one-offs to get you working with some other methods that might be useful to incorporate into your project.

Setup Data: the responses collected from a survery of the COGS 108 class.

There are 417 observations in the data, covering 10 different 'features'.

Research Question: Do students in different majors have different heights?

Background: Physical height has previously shown to correlate with career choice, and career success. More recently it has been demonstrated that these correlations can actually be explained by height in high school, as opposed to height in adulthood (1). It is currently unclear whether height correlates with choice of major in university.

Reference: 1) http://economics.sas.upenn.edu/~apostlew/paper/pdf/short.pdf (http://economics.sas.upenn.edu/~apostlew/paper/pdf/short.pdf) Hypothesis: We hypothesize that there will be a relation between height and chosen major.

Part 1: Load & Clean the Data Fixing messy data makes up a large amount of the work of being a Data Scientist.

localhost:8888/notebooks/Downloads/A4_DataAnalysis.ipynb

1/20

5/20/2019

A4_DataAnalysis The real world produces messy measurements and it is your job to find ways to standardize your data such that you can make useful analyses out of it.

In this section, you will learn, and practice, how to successfully deal with unclean data.

1a) Load the data Import datafile 'COGS108_IntroQuestionnaireData.csv' into a DataFrame called 'df'.

In [83]:

1 2

# YOUR CODE HERE df = pd.read_csv('COGS108_IntroQuestionnaireData.csv')

In [84]:

1 2

assert isinstance(df, pd.DataFrame)

 In [85]:

1 2

# Check out the data df.head(5)

Out[85]:

Timestamp

What year (in school) are you?

What is your major?

How old are you?

What is your gender?

What is your height?

What is your weight?

What is your eye color?

Were you born in California?

What is your favorite flavor of ice cream?

0

1/9/2018 14:49:40

4

Cognitive Science

21

Male

5'8"

147

Brown

Yes

Vanilla

1

1/9/2018 14:49:45

3

Cognitive Science

20

Male

5'8

150

Brown

Yes

Cookies and Cream

2

1/9/2018 14:49:45

Third

Computer Science

21

Male

178cm

74kg

Black

Yes

Matcha

3

1/9/2018 14:49:45

2

Cogs HCI

20

Male

5’8

133

Brown

Yes

Cookies and Cream

4

1/9/2018 14:49:47

3

Computer Science

20

Male

5'8"

160

Brown

Yes

Cookies n' Cream

Those column names are a bit excessive, so first let's rename them - code provided below to do so.

In [86]:

1 2 3

# Renaming the columns of the dataframe df.columns = ["timestamp", "year", "major", "age", "gender", "height", "weight", "eye_color", "born_in_CA", "favorite_icecream"]

Pandas has a very useful function for detecting missing data. This function is called 'isnull()'.

If you have a dataframe called 'df', then calling 'df.isnull()' will return another dataframe of the same size as 'df' where every cell is either True of False.

Each True or False is the answer to the question 'is the data in this cell null?'. So, False, means the cell is not null (and therefore, does have data). True means the cell is null (does not have data).

This function is very useful because it allows us to find missing data very quickly in our dataframe. As an example, consider the code below.

In [87]:

1 2

Out[87]:

# Check the first few rows of the 'isnull' dataframe df.isnull().head(5)

timestamp

year

major

age

gender

height

weight

eye_color

born_in_CA

favorite_icecream

0

False

False

False

False

False

False

False

False

False

False

1

False

False

False

False

False

False

False

False

False

False

2

False

False

False

False

False

False

False

False

False

False

3

False

False

False

False

False

False

False

False

False

False

4

False

False

False

False

False

False

False

False

False

False

If you print out more, and scroll down, you'll see some rows with missing data. For example: In [88]:

1

Out[88]:

df.isnull().iloc[48:50, :]

timestamp

year

major

age

gender

height

weight

eye_color

born_in_CA

favorite_icecream

48

False

False

False

False

False

False

False

False

False

False

49

False

True

False

False

False

True

True

False

False

False

Check an example, row 49, in which an entry has missing data In [89]:

1

df.iloc[49, :]

Out[89]: timestamp 1/9/2018 14:50:12 year NaN major Cog Sci age 21 gender Female height NaN weight NaN eye_color Brown born_in_CA Yes favorite_icecream Chocolate Name: 49, dtype: object Granted, the example above is not very informative. As you can see, the output of 'isnull()' is a dataframe where the values at each cell is either True or False. Most cells have the value of 'False'. We expect this to be the case since most people gave out answers to each question in our survey.

However, some rows such as row 49 show that some people chose not to answer certain questions. In the case of row 49, it seems that someone did not give out an answer for 'What year (in school) are you?'

However, what if wanted to use 'isnull()' to see all rows where our dataframe 'df' has missing values? In other words, what if we want to see the ACTUAL rows with missing values instead of this dataframe with True or False cells. For that, we need to write the following line of code:

df[df.isnull().any(axis=1)]

localhost:8888/notebooks/Downloads/A4_DataAnalysis.ipynb

2/20

5/20/2019

A4_DataAnalysis 1b) Find missing data Find all rows that have missing data in them. Save the ouput, as a dataframe, into a variable called 'rows_to_drop'. In other words, copy over and use the line of code that we gave out in the cell above.

In [90]:

1 2

# YOUR CODE HERE rows_to_drop = df[df.isnull().any(axis=1)]

In [91]:

1 2

assert isinstance(rows_to_drop, pd.DataFrame) assert rows_to_drop.shape == (29, 10)

You need to run & read the following couple of cells - but don't have to add any code: Real world data is messy. As an example of it, we consider the data shown in rows_to_drop (below). If you've done everything correctly so far, you should see an unexpected response with emojis at index 357. These types of responses, although funny, are hard to parse when dealing with big datasets. We'll learn about solutions to these types of problems in the upcoming cells In [92]:

1

rows_to_drop

timestamp

year

major

age

gender

height

weight

eye_color

born_in_CA

favorite_icecream

29

1/9/2018 14:50:00

senior

math&econ

20

female

173

130

black

No

NaN

38

1/9/2018 14:50:05

3rd

Computer Engineering

21

Female

NaN

NaN

Brown

No

NaN

49

1/9/2018 14:50:12

NaN

Cog Sci

21

Female

NaN

NaN

Brown

Yes

Chocolate

58

1/9/2018 14:50:14

3rd

Computer Science

20

Female

5ft 2in

NaN

brown

No

cookies and cream

66

1/9/2018 14:50:16

Sophomore

Cognitive Science - Design and Interaction

19

Female

5'5

NaN

Brown

No

Chubby hubby ben and jerrys

67

1/9/2018 14:50:16

2

Computer Science

19

Male

NaN

170

Brown

Yes

Chocolate

83

1/9/2018 14:50:23

Senior

Computer Science

22

Male

5'11

155 lb

dark brown

No

NaN

91

1/9/2018 14:50:25

Third

Computer Science

21

Female

NaN

NaN

Black

No

Green tea Mint chip

Out[92]:

92

1/9/2018 14:50:26

4

Computer Science

21

Female

5'2"

NaN

Brown

Yes

133

1/9/2018 14:50:36

2018

computer science

22

male

NaN

NaN

brown

No

NaN

140

1/9/2018 14:50:39

Senior

Chemical Engineering

25

Female

5 ft 3

NaN

Black

No

Green tea

150

1/9/2018 14:50:41

Second

Cognitive Science

19

Female

NaN

NaN

NaN

Yes

Cookies and Cream

159

1/9/2018 14:50:43

Fourth

Cognitive Science

21

Female

NaN

NaN

Black

Yes

Sea salt cookies and cream

168

1/9/2018 14:50:45

Second

Cogs sci

20

female

178

NaN

brown

No

green tea

181

1/9/2018 14:50:50

Second

Computer science

18

Male

NaN

NaN

Brown

No

Cookies and cream

232

1/9/2018 14:51:05

Second

Bioengineering

20

Male

1.7 m

70kg

Black

No

NaN

241

1/9/2018 14:51:10

4th

Cogs Sci HCI

21

Female

5 feet 4 in

NaN

brown

Yes

Mint chocolate

249

1/9/2018 14:51:13

2

Computer Science

20

Male

175cm

NaN

Brown

No

Mint

264

1/9/2018 14:51:22

4

Computer Science

21

Female

168cm

NaN

Dark Brown

Yes

Vanilla

285

1/9/2018 14:51:33

4

Molecular Biology

21

NaN

NaN

NaN

NaN

NaN

Green Tea

293

1/9/2018 14:51:36

2nd

Computer Engineering

NaN

Male

5-10

160 pounds

Brown

Yes

NaN

304

1/9/2018 14:51:40

3

Computer Science

20

Male

NaN

NaN

Brown

Yes

Strawberry

352

1/9/2018 14:52:06

4th

Cognitive Science

21

Female

5’7’’

NaN

Brown

No

NaN

354

1/9/2018 14:52:07

4

Cognitive Science - Machine Learning and Neura...

21

Nonbinary

5'5"

120 lbs.

Brown

Yes

NaN

357

1/9/2018 14:52:10

2

EECS

19

NaN

6.21

150

chocolate pudding

Yes

👌👀👌👀👌👀👌👀👌👀 good shit go౦ԁ sHit👌 thats ✔ some g...

376

1/9/2018 14:52:30

1

MathCS

18

M

5 10

NaN

Brown

No

Matcha

381

1/9/2018 14:52:37

second year

computer science

19

male

NaN

NaN

NaN

No

strawberry

402

1/9/2018 14:53:13

4

Computer Science

21

Female

5'8"

NaN

Brown

Yes

Strawberry

413

1/9/2018 14:55:00

3

Computer Science

20

M

68in

NaN

Brown

Yes

NaN

In the cell below, briefly explain below how 'df[df.isnull().any(axis=1)]' works, in a couple sentences.

Include an explanation of what 'any(axis=1)' means and how it affects the code.

The df.isnull() creates the dataframe showing whether there is NaN value (True) or not (False). any(axis=1) looks for any row that has NaN as a value. The outer df[] just puts it in a dataframe with the actual responses instead of True/False.

1c) Drop the rows with NaN values Drop any rows with missing data, but only for the columns 'major', 'height', 'gender' and 'age'. These will be the data of primary interest for our analyses, so we drop missing data here. Note that there are other missing data (in other rows) but this is fine for our analyses, so we keep them. To do this, ese the pandas 'dropna' method, inplace, using the 'subset' arguments to specify columns.

In [93]:

1 2

# YOUR CODE HERE df.dropna(subset=['major', 'height', 'gender', 'age'], inplace = True)

In [94]:

1

assert df.shape == (404, 10)

Now we have to standardize the data!

Check all different values given for majors. It's a lot!

localhost:8888/notebooks/Downloads/A4_DataAnalysis.ipynb

3/20

5/20/2019 In [95]:

A4_DataAnalysis 1

df["major"].unique()

Out[95]: array(['Cognitive Science', 'Computer Science', 'Cogs HCI', 'Cognitive Science w/ a specialization in Computation', 'International Studies', 'Computer Engineering', 'computer science', 'Chemical Engineering / Literature Writing', 'Cognitive Science w/ Specialization in Human Computer Interaction', 'Cognitive Science-- Machine Learning and Neural Computation ', 'Cse', 'Cognitive Science - Design and Interaction', 'cognitive science', 'CSE', 'Cog sci design interaction', 'Math CS', 'Electrical Engineering', 'math&econ', 'Cognitive Science spec/ HCI', 'Cognitive science', 'Math-CS', 'Data Science', 'Cognitive science ', 'Cognitive Science ', 'Math - CS ', 'Cognitive Science Human Computer Interaction', 'Cognitive Science with specialization in HCI ', 'Cognitive Science (HCI)', 'Probability & Statistics', 'Cog Sci', 'Cognitive Science - Machine Learning', 'Mathematics-computer science', 'Computer science ', 'Computer Science-Bioinformatics', 'Computer Scientist', 'Cognitive science (HCI)', 'Cogsci specializing in machine learning and neural computation ', 'cognitive science with specialization in machine learning and neural computation', 'Cogntive Science with emphasis on Clinical Aspects', 'Computer Science ', 'Cognitive Science ML&NC major, CSE minor', 'math-cs', 'Cognitive Science: Human-Computer Interaction', 'Political Science', 'Cognitive Science with a specialization in Neuroscience', 'Molecular Biology', 'Mathematics-Computer Science', 'Cognitive Science with a Specialization in Neuroscience ', 'Economics', 'computer engineering ', 'Cognitive Science with Specialization in Machine Learning and Neural Computation', 'Computer Engineering ', 'Economics ', 'Cognitive Science (Design & Interaction)', 'Cognitive Science Specialization in Design and Interaction', 'Cognitive Science w/ Focus on Human Computer Interaction', 'Math', 'Math-Computer Science', 'Cognitive Science with an emphasis in machine learning and neural computation', 'Math-Cs', 'Cogs Sci design specilization', 'Cognitive Science / HCI', 'Applied Mathematics', 'Applied math', 'Cognitive Science (Specialization in Design & Interaction)', 'MathCS', 'Chemical Engineering', 'Cogsci HCI', 'Cognitive Science-Design', 'cognitive science - hci', 'Computer science', 'Cognitive Science- Machine Learning & Neural Computation ', 'Bioinformatics', 'CogSci', 'Cognitive Science HCI', 'Cognitive Science in HCI', 'Cogs sci', 'Cogs machine learning', 'cogs hci', 'Bioinformatics ', 'Psychology', 'Math & Computer Science', 'Cog Sci HCI', 'MATH-CS', 'CogSci w/ Spec in HCI', 'Cogs Spec ML and NC', 'Probability and Statistics', 'Cognitive Neuroscience', 'Computer Science Engineer', 'Cogsci/HCI', 'CS', 'Physiology and neuroscience', 'Computer engineering ', 'Cognitive Science - HCI', 'Management Science', 'Cognit ive Science', 'Bioengineering ', 'ICAM & COGS', 'Cognitive Science with a Specialization in Human Cognition', 'cog sci', 'Cogs Sci HCI', 'Mathematics Computer Science ', 'Psychology and Cognitive Science ', 'Computer Engineer', 'Biochemistry and Cell Biology', 'Cognitive Science ', 'Cognitive Science w/ HCI ', 'Cog Sci ', 'Cognitive science spec HCI', 'Cog sci', 'Cognitive Science (Machine Learning and Neural Computation)', 'Cognitive science neuroscience', 'CogSci specialization ML', 'Psychology Cognitive Science', 'Probability and statistics', 'Mathematics - Computer Science', 'Interdisciplinary Computing & the Arts', 'Cognitive Science - Neuroscience', 'Management Science ', 'Cognitive Science with a Specialization in Design and Interaction', 'MathCs', 'Cogs with specialization in HCI', 'Cognitive Science (with specialization in computation)', 'Cogs', 'COGS（ML）', 'Cognitive Science, Human-Computer Interaction', 'Cognitive Science specializing in Machine Learning and Computer Science double major', 'Cognitive Science - Specialization in Neuroscience.', 'Cognitive science machine learning and neural computation', 'Cognitive Science (Spec. Design and Interaction)', 'Machine Learning', 'Cognitive and Behavioral Neuroscience', 'Cog Sci, B.S and Poli Sci, B.A.', 'Human Computer Interaction', 'Electrical engineering', 'Linguistics', 'MathCS & CogsSci ML Double Major', 'bioinformatics', 'Cognitive science HCI', 'Cognitive Science- Human Computer Interaction', 'Cognitive Science - Machine Learning and Neural Computation', 'Cognitive science- design and interaction ', 'Cognitive Science- Neuroscience', 'Cognitive Science: Machine Learning & Neural Computation', 'Cognitive science specialized in machine learning and neural computetion', 'Cognitive Science with a Specialization in Machine Learning & Neural Computation ', 'Computer engineering', 'Neuroscience', 'General Linguistics and Cognitive Science (with Spec. in Clinical Aspects)', 'Cog Sci w/ specialization in Neuroscience', 'cs', 'COGS spc Machine Learning', 'COGS HCI ', 'Electrical engineer', 'Cognitive Science with specialization in Neuroscience ', 'Cognitive Science specializing in HCI', 'Cognitive Science: Neuroscience', 'Under-water Basket Weaving', 'computer science ', 'Cognitive Science specializing in Neuroscience ', 'CogSci Machine Learning', 'Bioengineering'], dtype=object)

We'll write a function performing some simple substring checking in order to group many responses together

localhost:8888/notebooks/Downloads/A4_DataAnalysis.ipynb

4/20

5/20/2019 In [96]:

A4_DataAnalysis 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

def standardize_major(string): string = string.lower() string = string.strip() if "cog" in string: output = "COGSCI" elif "computer" in string: output = "COMPSCI" elif "cs" in string: output = "COMPSCI" elif "math" in string: output = "MATH" elif "electrical" in string: output = "ECE" elif "bio" in string: output = "BIO" elif "chem" in string: output = "CHEM" # Otherwise, if uncaught - keep as is else: output = string return output

Applying the transformation In [97]:

1

df["major"] = df["major"].apply(standardize_major)

Previewing the results of the previous transformation. It looks a lot better, though it's not perfect, but we'll run with this In [98]:

1

df["major"].unique()

Out[98]: array(['COGSCI', 'COMPSCI', 'international studies', 'CHEM', 'ECE', 'MATH', 'data science', 'political science', 'BIO', 'psychology', 'physiology and neuroscience', 'management science', 'interdisciplinary computing & the arts', 'machine learning', 'neuroscience', 'under-water basket weaving'], dtype=object) Next let's check the 'gender' column.

Check the different responses received for gender, including how many of each response we have In [99]:

1

df["gender"].value_counts()

Out[99]: Male 229 Female 95 female 26 male 18 F 10 M 6 Male 6 Female 6 Apache Helicopter 1 Transgender 1 Woman 1 men 1 Rainbow 1 MALE 1 Famale 1 Nonbinary 1 Name: gender, dtype: int64 standardize_gender : Function to standardize the gender responses Note: for the purposes of the following analyses, we will keep self-reported gender for categories in which we have a sizable number of responses, in this case, those which correspond to 'female' and 'male' In [100]:

1 2 3 4 5 6 7 8 9 10 11 12 13

def standardize_gender(gender): gender = gender.lower() gender = gender.strip() if gender in ['female', 'f', 'woman', 'women']: output = 'female' elif gender in ['male', 'm', 'man', 'men']: output = 'male' else: output = np.nan return output

Apply the transformation, and drop any rows with missing gender information In [101]:

1 2

df["gender"] = df["gender"].apply(standardize_gender) df.dropna(subset=['gender'], inplace=True)

In [102]:

1 2

# Check the results df["gender"].unique()

Out[102]: array(['male', 'female'], dtype=object) Now you will write some code to standardize some of the other data columns.

1d) Standardize other columns

localhost:8888/notebooks/Downloads/A4_DataAnalysis.ipynb

5/20

5/20/2019

A4_DataAnalysis Find, programatically, the number of unique responses to in the 'year' column. Save the result in a variable named 'num_unique_responses'. Hint: you can answer this question using the 'unique' method, used above.

In [103]:

1 2

# YOUR CODE HERE num_unique_responses = len(df["year"].unique())

In [104]:

1 2 3

assert num_unique_responses assert isinstance(num_unique_responses, int)

Print out all the different answers in 'year' In [105]:

1

df['year'].unique()

Out[105]: array(['4', '3', 'Third', '2', '3rd', '3rd year', '5th', 'Second', '4th Year', '5', 'Senior', 'Junior', '2nd', '1', 'senior', 'third', 'first year', '4th', 'Junior ', 'Freshman', 'Sophomore', '4th year', '6', 'second year', 'Third Year', '2nd year', 'First', '5th (2nd year transfer)', '2018', 'last year', 'UCSD', 'Fifth Year', '2021', 'Fifth year', '3rd Year', '3rd year ', 'Second year', 'fourth year', 'First year', 'Sophmore', 'Second Year', 'Junior year', '2020', 'Fifth ', 'Sophomore ', '2nd year ', '4th year ', '2nd Year', '4+', 'second', 'Fourth', 'Senior Year', '2nd (Sophomore)', 'Senior ', ' 3rd Year', 'Fourth Year', 'Junior standing', 'Fourth year', 'Fifth', 'Third year', 'Sineor '], dtype=object) The line of code above shows us the different values we got, to the question 'What year (in school) are you?'.

As you can tell, it is a mess!. For example, if you are a junior student, then you might have answered: 3, three, third, 3rd year, junior, junior year, Junior, etc.

That is an issue. We want to be able to analyze this data and, in order to do this successfully, we need to all answers with the same meaning to be written in the same way. Therefore, we're gonna have to transform answers such as '3, third, 3rd, junior, etc' into a single possible value. We'll do this for all values that mean the same.

In the rest of Part 1, we will work on writing code, organized into functions that will allow us to transform similar respones into the same value. We will call this process: standardizing the data.

The cell below provides an example for the kind of code you will need to write to answer this question. This example is separate from our actual data, and is a potential function we might use to standardize messy data in this case, hypothetical data to the question 'What is your favourite major python version?'.

Note some things used in this example that you need to use to standardize data: string methods, such as 'lower' and 'strip' to transform strings the 'replace' string method, to replace a set of characters with something else if/else statements that check what's in our string (number, letters, etc) type casting, for example using 'int()' to turn a variable into an integer using 'np.nan' (which stands for 'not a number') to denote missing or unknown data

In [106]:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

def example_standardize_function(str_in): """Standardize data to the question 'what is your favourite major python version?'

Parameters ---------str_in : string A provided answer.

Returns ------int_out : int or np.nan A standardized integer response. """ # Make the input all lowercase str_in = str_in.lower() # Drop all whitespace str_in = str_in.strip() # Replace things (and then strip again afterwords) # Note that the 'replace' replaces the first argument, with the second # The first argument does not need to be present in the string, # if it's not there 'replace' does nothing (but does not error), so the code moves on. str_in = str_in.replace('version', '') str_in = str_in.replace('python', '') str_in = str_in.strip() # Cast to integer, if what's left seems appropriate if str_in.isnumeric() and len(str_in) == 1: out = int(str_in) # Otherwise, consider input was probably ill-formed, return nan else: out = np.nan return out # Check how this function help standardize data: # Example possible answers to the question 'What is your favourite major version of Python': print('INPUT', '\t\t-\t', 'OUTPUT') for inp in ['version 3', '42', '2', 'python 3', 'nonsense-lolz']: print('{:10s} \t-\t {:1.0f}'.format(inp, example_standardize_function(inp)))

INPUT version 3 42 2 python 3 nonsense-lolz

-

OUTPUT 3 nan 2 3 nan

1e) Standardize 'year' column Write a function named 'standardize_year' that takes in as input a string and returns an integer.

The function will do the following (in the order specified): Note that for these detailed instructions, each line corresponds to one line of code you need to write.

localhost:8888/notebooks/Downloads/A4_DataAnalysis.ipynb

6/20

5/20/2019

A4_DataAnalysis 1) convert all characters of the string into lowercase 2) strip the string of all leading and trailing whitespace 3) replace any occurences of 'first' with '1' 4) replace any occurences of 'second' with '2' 5) replace any occurences of 'third' with '3' 6) replace any occurences of 'fourth' with '4' 7) replace any occurences of 'fifth' with '5' 8) replace any occurences of 'sixth' with '6' 9) replace any occurences of 'freshman' with '1' 10) replace any occurences of 'sophomore' with '2' 11) replace any occurences of 'junior' with '3' 12) replace any occurences of 'senior' with 4' 13) replace any occurences of 'year' with '' (remove it from the string) 14) replace any occurences of 'th' with '' (remove it from the string) 15) replace any occurences of 'rd' with '' (remove it from the string) 16) replace any occurences of 'nd' with '' (remove it from the string) 17) strip the string of all leading and trailing whitespace (again) 18) If the resulting string is a number and it is less than 10, then cast it into an integer and return that value 19) Else return np.nan to symbolize that the student's response was not a valid entry HINTS: you will need to use the functions 'lower()', 'strip()', isnumeric() and 'replace()'

In [107]:

In [108]:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

# YOUR CODE HERE def standardize_year(str_in): str_in = str_in.lower() str_in = str_in.strip()

1 2 3 4 5 6

assert standardize_year('2nd') == 2 assert standardize_year('sophomore') == 2 assert standardize_year('3rd year') == 3 assert standardize_year('5th') == 5 assert standardize_year('7 ') == 7 assert standardize_year('randomText') is np.nan

str_in = str_in.replace('first', '1') str_in = str_in.replace('second', '2') str_in = str_in.replace('third', '3') str_in = str_in.replace('fourth', '4') str_in = str_in.replace('fifth', '5') str_in = str_in.replace('sixth', '6') str_in = str_in.replace('freshman', '1') str_in = str_in.replace('sophomore', '2') str_in = str_in.replace('junior', '3') str_in = str_in.replace('senior', '4') str_in = str_in.replace('year', '') str_in = str_in.replace('th', '') str_in = str_in.replace('rd', '') str_in = str_in.replace('nd', '') str_in = str_in.strip() if str_in.isnumeric() and int(str_in) < 10: out = int(str_in) else: out = np.nan return out

1f) Transform 'year' column Use 'standardize_year' to transform the data in column 'What year (in school) are you?'. Hint: use the apply function AND remember to save your output inside the dataframe In [109]:

1 2 3 4 5

# YOUR CODE HERE df["year"] = df["year"].apply(standardize_year)

In [110]:

1

assert len(df["year"].unique()) == 7

Assuming that all is correct up to this point, the line below should show all values now found in df.

It should look a lot better. With this data, we can now make insightful analyses.

You should see an array with elements 1,2,3,4,5,6 and nan (not necessarily in that order).

Note that if you check the data type of this column, you'll see that pandas converts these numbers to 'float', even though the applied function returns 'int', because 'np.nan' is considered a float. This is fine.

In [111]:

1

df["year"].unique()

Out[111]: array([ 4.,

3.,

2.,

5.,

1.,

6., nan])

Let's do it again. Let's take a look at the responses in the 'weight' column, and then standardize them.

In [112]:

1 2

# First, ensure that all types are consistent, use strings df["weight"] = df["weight"].astype(str)

localhost:8888/notebooks/Downloads/A4_DataAnalysis.ipynb

7/20

5/20/2019 In [113]:

A4_DataAnalysis 1 2

# Check all the different answers we received df["weight"].unique()

Out[113]: array(['147', '150', '74kg', '133', '160', '106 lb', '155', '170 pounds', '170', '135', '115', '110', '167 lbs', '60kg', '136', '140', '120', '78kg', '128 lbs', '220', '145lb', '180', '95', '160lbs', '130', '177g', '125 pounds', '125', '210', '69kg', '165', '160 lbs', '123', '135lbs', '112 pounds', '190', '145lbs', '150 lbs', 'nan', '173lbs', '73kg', '133 lb', '107 pounds', '200', '240', '47kg', '175', '120 lbs', '164', '110 lbs', '128lbs', '155 lb', '185', '167', '84 kg', '113', '235', '128lb', '119 lbs', '110 lb', '72kg', '122', '140 lbs', '70 kg', '155 pounds ', '128', '125 lb', '108', '15', '215', '48kg', '52kg', '145', '185lbs. ', '230lb', '125 lbs', '140 lb', '168lb', '100', '130 pounds', '124', '84kg', '119', "I don't know dude but heavy probably", '103', '140lbs', '170lbs', '155 lbs', '50kg', '190 lbs', '52 kg', '168lbs', '189', '168', '76KG', '77kg', '181', '146', '156', '140 pounds', '128 lb', '46kg', '140 lb ', '120 lb', '146lb', '169lb', '250', '130lb', '146 Pounds', '58 kg', '173lb', '70kg', '56 kg', '190lb', 'Secret', '66 kg', '134', '125lb', '145 lbs', '102 lbs', '67kg', '51kg', '105', '320', '142', '130 lbs.', '145 pounds', '119lb', '280', '95kg', '175 lbs', '58lbs', '134 pounds', '129', '85kg', '152 lbs', '138 pounds', '112', '141', '100lbs', '102 kbs', '98 lb', '91 kg', '107 lb', '150lb', '145 lbs ', '205', '98lbs', '145 lb.', '112 lbs', '106', '124pounds', '110 pounds', '82kg', "6'1", '100 lbs', '45kg', '154', '122 pounds', '150 lb', '155lb', '179lb', '130 lbs', '158', '116', '107 lbs', '105 lbs', '91 lbs', '168 lb', '140lb', '76kg', '75', '145 lb', '85 kg', '114', '105 pounds', '115 lbs', '85', '102', '85 lbs', '63kg', '160 pounds', '185 lbs'], dtype=object)

1g) Standardize 'weight' column Write a function named 'standardize_weight' that takes in as input a string and returns an integer. The function will do the following (in the order specified): 1) convert all characters of the string into lowercase 2) strip the string of all leading and trailing whitespace 3) replace any occurences of 'lbs' with '' (remove it from the string) 4) replace any occurences of 'lb' with '' (remove it from the string) 5) replace any occurences of 'pounds' with '' (remove it from the string) 6) If the string contains the substring 'kg', then: 6.1) replace 'kg' with '' 6.2) cast the string into an integer type using the function 'int()' 6.3) multiply the resulting integer by 2 (an approximate conversion of kilograms to pounds) 6.4) cast the resulting number back into a string with the function 'str()' 7) Strip the string of its whitespaces (again) 8) If the resulting string is numeric: cast it into an integer and return the resulting value 9) Else: return np.nan

In [115]:

In [116]:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

# YOUR CODE HERE def standardize_weight(str_in): str_in = str_in.lower() str_in = str_in.strip()

1 2 3

assert standardize_weight('34 lbs') == 34 assert standardize_weight('101 kg') == 202

str_in = str_in.replace('lbs', '') str_in = str_in.replace('lb', '') str_in = str_in.replace('pounds', '') if "kg" in str_in: str_in = str_in.replace('kg', '') str_in = int(str_in) str_in = 2*str_in str_in = str(str_in) str_in = str_in.strip() if str_in.isnumeric(): out = int(str_in) else: out = np.nan return out

1h) Transform 'weight' column Use 'standardize_weight' to transform the data in the 'weight' column. Hint: use the apply function AND remember to save your output inside the dataframe In [117]:

1 2

# YOUR CODE HERE df["weight"] = df["weight"].apply(standardize_weight)

In [118]:

1

assert df["weight"].unique().shape == (83,)

Now, let's see the result of our hard work . The code below should output all numbers (or nan).

In [119]:

1

df["weight"].unique()

Out[119]: array([147., 150., 148., 133., 160., 106., 155., 170., 135., 115., 110., 167., 120., 136., 140., 156., 128., 220., 145., 180., 95., 130., nan, 125., 210., 138., 165., 123., 112., 190., 173., 146., 107., 200., 240., 94., 175., 164., 185., 168., 113., 235., 119., 144., 122., 108., 15., 215., 96., 104., 230., 100., 124., 103., 189., 152., 154., 181., 92., 169., 250., 116., 132., 134., 102., 105., 320., 142., 280., 58., 129., 141., 98., 182., 205., 90., 179., 158., 91., 75., 114., 85., 126.])

localhost:8888/notebooks/Downloads/A4_DataAnalysis.ipynb

8/20

5/20/2019

A4_DataAnalysis So far, you've gotten a taste of what it is like to deal with messy data. It's not easy, as you can tell.

The last variable we need to standardize for the purposes of our analysis is 'height'. We will standardize that one for you.

Do read the code below and try to understand what it is doing.

In [120]:

1 2

# First, we'll look at the possible values for height df["height"].unique()

Out[120]: array(['5\'8"', "5'8", '178cm', '5’8', '5\'11"', "5'9", '5 ft, 3 in', "5'10", '180 cm', '5\'3"', '5\'9"', '5\'2"', '5ft 5inches', '158cm', '5\'7"', '5 feet 8 inches', "5'7''", "5' 8'' ", '184cm', '5\' 7"', '6’2”', "6'1''", "4'11", '5\'10"', '173', '177cm', '5\'6"', "5'4", '5’5', "5'5", "6'2", '171cm', "5' 5''", "5'2", "5'7", '5\' 8"', "5'3", '5 feet 9 inches', '6’0', '5’3', "5'11", '64 in', "5' 7''", '5ft 2in', '5’3”', '66', '173cm', '5’8”', '5 feet 2 in', '5 7', '6’', '5.4', '6 feet', '5’0”', '1.6m', '5 ft 2 in', '5\'1"', '5’5”', "5' 8''", '6\'2"', '6\'4"', '5\'8.5"', '178 cm', '5\'5"', "6'1", '5”7’', '174cm', '71', "5' 1", '6 ft', '5” 4’', '177 cm', '69 inches ', "5'4''", '5\' 4"', '5 foot 11', "5''2'", '6 feet 0 inches', '67', '167cm', '6\'1"', "5'11''", '168cm', '5"8', '5\' 10"', '6ft', '163 cm', '5 Feet 6 inches', '6.1', "5' 3''", '5 foot 7', '5 ft 3', '5 7"', '172 cm', '6-5', '5’ 6’’', '6’0”', "5'0", '6 foot', '6’ 1”', '5-10', "5'", '5 ft 8 in', '5 foot 11 inches', '6', '5’6”', '1.7 m', '178', '5\' 6"', '5 11', "5'1", '5’4', '5 ft. 9 in', '170 cm ', '6 ft 1 in', '5’9', '5’ 10”', "6'", '190cm', "5'8 feet", '6’q', '5 ft 7 in', '5 foot 10', '5 foot 8 inches', '5’9”', '161cm', '5 ft', '5’ 5”', '5’8’’', '6\'3"', '5’10', '5’6', "5' 10''", '5 Feet 11 Inches', '5 foot 6', "6'3''", '162 cm', '65in', '5’8.5”', "5' 1''", '185 cm', '6 feet 1 in', '176cm', 'Secret', '5’11', '170 cm', '5 feet 4 in', "5'3''", '175cm', '5ft 7in', '67in', '5’1', '179cm', '160cm', '5\'0"', "5'6", '5 ft 10 in', '5\' 9"', '6 feet 0 inch', '5”6’', '4\'11"', '65 inches', '172cm', '189', "165 cm/5'4'' feet", '6\'0"', '5ft 6in', '5\' 6.5"', '6\' 1.5"', '5’7”', '5’2”', '5 ft 8in', '5 feet and 9 1/2 inches', '5.8', '6’4”', '5\'4"', '5”4', '5 ft 2 in.', '5’ 2”', '72 inches', '5 ft 3 inches', '5 feet 6 inches', '6\' 1"', '5”6', "6'0", '186cm', '167', '5’10”', '5 8', '6`1``', '1.6meter', '5 foot 3 inches', '5.9', '70', '5 ft 7', '5’7’’', '5’ 2', '5 foot ', '4\'20"', '4’11', "5'0''", '5foot 7inch', '5’7', '5\' 11"', '5 10', '4 ft 11 inches', '176', '5', '5\' 02"', '5’1”', '4’ 11”', '5’ 3.75”', '5 ft 4 in', '4’ 10”', '5 5', '68in', '6’1”'], dtype=object) It seems like we'll have to handle different measurement systems. Ugh, ok... Let's write a function that converts all those values to inches

localhost:8888/notebooks/Downloads/A4_DataAnalysis.ipynb

9/20

5/20/2019 In [121]:

A4_DataAnalysis 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113

def standardize_height(string): orig = string output = None # Basic string pre-processing string = string.lower() string = string.strip() string = string.replace("foot", "ft") string = string.replace("feet", "ft") string = string.replace("inches", "in") string = string.replace("inch", "in") string = string.replace("meters", "m") string = string.replace("meter", "m") string = string.replace("centimeters", "cm") string = string.replace("centimeter", "cm") string = string.replace(",", "") string = string.strip() # CASE 1: string is written in the format FEET <DIVIDER> INCHES dividers = ["'", "ft", "’", '”', '"','`', "-", "''"] for divider in dividers: # Split it into its elements elements = string.split(divider) # If the divider creates two elements if (len(elements) >= 2) and ((len(string) -1) != string.find(divider)): feet = elements[0] inch = elements[1] if elements[1] is not '' else '0' # Cleaning extranious symbols for symbol in dividers: feet = feet.replace(symbol, "") inch = inch.replace(symbol, "") inch = inch.replace("in","") # Removing whitespace feet = feet.strip() inch = inch.strip() # By this point, we expect 'feet' and 'inch' to be numeric # If not...we ignore this case if feet.replace('.', '').isnumeric() and inch.replace('.', '').isnumeric(): # Converting feet to inches and adding it to the curent inches output = (float(feet) * 12) + float(inch) break # CASE 2: string is written in the format FEET ft INCHES in if ("ft" in string) and ("in" in string): # Split it into its elements elements = string.split("ft") feet = elements[0] inch = elements[1] # Removing extroneous symbols and stripping whitespace inch = inch.replace("inch", "") inch = inch.replace("in", "") feet = feet.strip() inch = inch.strip() # By this point, we expect 'feet' and 'inch' to be numeric # If not...we ignore this case if feet.replace('.', '').isnumeric() and inch.replace('.', '').isnumeric(): # Converting feet to inches and adding it to the curent inches output = (float(feet) * 12) + float(inch) # CASE 3: answer was given ONLY in cm # Convert to inches: approximately 0.39 inches in a meter elif "cm" in string: centimeters = string.replace("cm", "") centimeters = centimeters.strip() if centimeters.replace('.', '').isnumeric(): output = float(centimeters) * 0.39 # CASE 4: answer was given ONLY in meters # Convert to inches: approximately 39 inches in a meter elif "m" in string: meters = string.replace("m", "") meters = meters.strip() if meters.replace('.', '').isnumeric(): output = float(meters)*39 # CASE 5: answer was given ONLY in feet elif "ft" in string: feet = string.replace("ft", "") feet = feet.strip() if feet.replace('.', '').isnumeric(): output = float(feet)*12 # CASE 6: answer was given ONLY in inches elif "in" in string: inches = string.replace("in", "") inches = inches.strip() if inches.replace('.', '').isnumeric(): output = float(inches) # CASE 7: answer not covered by existing scenarios / was invalid. # Return NaN if not output: output = np.nan return output

localhost:8888/notebooks/Downloads/A4_DataAnalysis.ipynb

10/20

5/20/2019

A4_DataAnalysis Applying the transformation and dropping invalid rows

In [122]:

1 2

df["height"] = df["height"].apply(standardize_height) df.dropna(subset=['height'], inplace=True)

Check the height data, after applying our standardization In [123]:

1

df["height"].unique()

Out[123]: array([68. , 69.42, 71. , 69. , 63. , 70. , 70.2 , 62. , 65. , 61.62, 67. , 71.76, 74. , 73. , 59. , 69.03, 66. , 64. , 66.69, 72. , 67.47, 60. , 62.4 , 61. , 76. , 68.5 , 67.86, 65.13, 65.52, 63.57, 67.08, 77. , 66.3 , 74.1 , 62.79, 75. , 63.18, 72.15, 68.64, 68.25, 69.81, 66.5 , 73.5 , 72.54, 63.75, 58. ]) Ensuring that the data types are correct - type cast age to int.

In [124]:

1 2 3 4

Out[124]:

df["age"] = df["age"].astype(np.int64) # Check out the data, after we've cleaned it! df.head()

timestamp

year

major

age

gender

height

weight

eye_color

born_in_CA

0

1/9/2018 14:49:40

4.0

COGSCI

21

male

68.00

147.0

Brown

Yes

favorite_icecream Vanilla

1

1/9/2018 14:49:45

3.0

COGSCI

20

male

68.00

150.0

Brown

Yes

Cookies and Cream

2

1/9/2018 14:49:45

3.0

COMPSCI

21

male

69.42

148.0

Black

Yes

Matcha

3

1/9/2018 14:49:45

2.0

COGSCI

20

male

68.00

133.0

Brown

Yes

Cookies and Cream

4

1/9/2018 14:49:47

3.0

COMPSCI

20

male

68.00

160.0

Brown

Yes

Cookies n' Cream

Check that the dataframe has the right number of columns. If this doesn't pass - check your code in the section above.

In [125]:

1

assert len(df) == 367

Part 2: Exploratory Data Vizualization First, we need to do some exploratory data visualization, to get a feel for the data.

For plotting questions, do not change or move the 'plt.gcf()' lines.

2a) Scatter Matrix Plot the data, using scatter_matrix, from Pandas. Assign it to a variable called 'fig'.

In [126]:

1 2

# YOUR CODE HERE fig = pd.plotting.scatter_matrix(df)

In [127]:

1

assert np.all(fig)

2b) Bar Chart Plot a bar chart showing the number of students in each majors. Hints: you can use 'value_counts' to get the counts for each major. You can then use the 'plot' method from pandas for plotting - you don't need matplotlib.

localhost:8888/notebooks/Downloads/A4_DataAnalysis.ipynb

11/20

5/20/2019 In [128]:

In [129]:

A4_DataAnalysis 1 2 3 4 5

# YOUR CODE HERE

1

assert f1.gca().has_data()

df['major'].value_counts().plot.bar() f1 = plt.gcf()

2c) Histogram for COGSCI Plot a histogram of the height data for all students who wrote 'COGSCI' as their major.

In [130]:

In [131]:

1 2 3 4

# YOUR CODE HERE df['height'][(df['major'] == 'COGSCI')].hist()

1

assert f2.gca().has_data()

f2 = plt.gcf()

2d) Histogram for COMPSCI Plot a histogram of the height data for all students who wrote 'COMPSCI' as their major.

In [132]:

In [133]:

1 2 3 4

# YOUR CODE HERE df['height'][(df['major'] == 'COMPSCI')].hist()

1

assert f3.gca().has_data()

f3 = plt.gcf()

Part 3: Exploring The Data Beyond just plotting the data, we should check some other basic properties of the data. This serves both as a way to get a 'feel' for the data, and to look for any quirks or oddities about the data, that may indicate issues that need resolving. To do this, let's explore that data a bit (not limiting ourselves to only features that we plan to use - exploring the dataset as a whole can help us find any issues).

Notes: You answers should NOT be pandas objects (Series or DataFrames), extract answers so the variables are ints, floats or strings (as appropriate).

You must answer these questions programmatically: do not count / check and hard code particular values.

3a) Number of majors

localhost:8888/notebooks/Downloads/A4_DataAnalysis.ipynb

12/20

5/20/2019

A4_DataAnalysis How many different majors are in the dataset? Save this number to a variable 'n_majors'.

In [137]:

1 2 3

# YOUR CODE HERE n_majors = len(df['major'].unique())

In [138]:

1 2

assert n_majors

3b) Range of 'age' What is the range (max value - min value) of ages in the dataset? Save this number to a variable 'r_age' In [139]:

1 2

# YOUR CODE HERE r_age = int(max(df['age'])) - int(min(df['age']))

In [140]:

1 2

assert r_age

3c) Most popular ice-cream flavour What is the most popular ice cream flavour? Save the ice cream name to the variable 'f_ice', and the number of people who like it to a variable 'n_ice'.

Hint: you can get these values using the 'value_counts' method.

In [141]:

1 2 3

# YOUR CODE HERE f_ice = df['favorite_icecream'].value_counts().idxmax() n_ice = max(df['favorite_icecream'].value_counts())

In [142]:

1 2 3

assert n_ice assert f_ice

3d) Unique favourite ice cream How many people have a unique favourite ice cream? (How many ice cream flavours are only 1 persons favourite?) Save this number to a variable 'u_ice' In [143]:

1 2

# YOUR CODE HERE u_ice = len(df['favorite_icecream'].value_counts() == 1)

In [144]:

1 2

assert u_ice

Part 4: Testing Distributions Soon, in the data analysis, we will want to run some statistical tests on our data. First, we should check the distributions!

When using methods / statistical tests that make certain assumptions, it's always best to explicitly check if your data meet those assumptions (otherwise the results may be invalid). Let's test if our data are in fact normally distributed.

See an example of how to test the disributions of data in the 'TestingDistributions' notebook in Tutorials.

In [145]:

1 2 3

# For convenience, and consistency, we're providing this code to pull out the required data h_co = df[df['major'] == 'COGSCI']['height'].values h_cs = df[df['major'] == 'COMPSCI']['height'].values

4a) Normal Test For each of 'h_co', and 'h_cs', use the 'normaltest' function to test for normality of the distribution. 'normaltest' returns two values, a test statistic and a p-value. Save these values as 'st_co', 'p_co', 'st_cs', and 'p_cs' respectively.

In [148]:

1 2 3

# YOUR CODE HERE st_co, p_co = stats.normaltest(h_co) st_cs, p_cs = stats.normaltest(h_cs)

In [149]:

1 2 3 4 5

assert st_co assert p_co assert st_cs assert p_cs

4b) Are they normal?

Have a look at the values returned.

Based on these results, and using an alpha significance value of 0.01: Set boolean values (True, False) of whether each distribution can be considered to be normally distributed (set as True if the test supports it is normally distributed (or, more formally, we have not rejected the null hypothesis), and False if the test suggests the data is not normally distributed (we should reject the null hypothesis).

Set boolean values, as specified above: For the 'h_co' data, set a boolean value to the var 'is_n_co'

localhost:8888/notebooks/Downloads/A4_DataAnalysis.ipynb

13/20

5/20/2019

A4_DataAnalysis For the 'h_cs' data, set a boolean value to the var 'is_n_cs'

In [150]:

1 2 3 4 5 6 7 8 9 10

# YOUR CODE HERE if p_co > 0.01: is_n_co = True elif p_co <= 0.01: is_n_co = False if p_cs > 0.01: is_n_cs = True elif p_cs <= 0.01: is_n_cs = False

In [151]:

1 2 3

assert isinstance(is_n_co, bool) assert isinstance(is_n_cs, bool)

CO data: plot the comparison of the data and a normal distribution (this code provided) In [152]:

1 2 3 4 5 6 7

# This plots a histogram, with the hypothetical normal distribution (with same mean and variance) xs = np.arange(h_co.min(), h_co.max(), 0.1) fit = stats.norm.pdf(xs, np.mean(h_co), np.std(h_co)) plt.plot(xs, fit, label='Normal Dist.', lw=4) plt.hist(h_co, normed=True, label='Actual Data'); plt.title('Cognitive Science - Height Data') plt.legend();

/Users/brianbarry/anaconda3/lib/python3.6/site-packages/matplotlib/axes/_axes.py:6521: MatplotlibDeprecationWarning: The 'normed' kwarg was deprecated in Matplotlib 2.1 and will be removed in 3.1. Use 'density' instead. alternative="'density'", removal="3.1")

CS data: plot the comparison of the data and a normal distribution (this code provided) In [153]:

1 2 3 4 5 6 7

# This plots a histogram, with the hypothetical normal distribution (with same mean and variance) xs = np.arange(h_cs.min(), h_cs.max(), 0.1) fit = stats.norm.pdf(xs, np.mean(h_cs), np.std(h_cs)) plt.plot(xs, fit, label='Normal Dist.', lw=4) plt.hist(h_cs, normed=True, label='Actual Data'); plt.title('Computer Science - Height Data') plt.legend();

/Users/brianbarry/anaconda3/lib/python3.6/site-packages/matplotlib/axes/_axes.py:6521: MatplotlibDeprecationWarning: The 'normed' kwarg was deprecated in Matplotlib 2.1 and will be removed in 3.1. Use 'density' instead. alternative="'density'", removal="3.1")

Part 5: Data Analysis Now let's analyze the data, to address our research question.

For the purposes of this analysis, let's assume we need at least 75 students per major to analyze the height data.

This means we are only going to use data from people who wrote 'COGSCI' or 'COMPSCI' as their major.

5a) Select the data Pull out the data we are going to use: Save the height data for all 'COGSCI' majors to a variable called 'h_co' Save the height data for all 'COMPSCI' majors to a variable called 'h_cs'

In [154]:

1 2 3

# YOUR CODE HERE h_co = df['height'][(df['major'] == 'COGSCI')] h_cs = df['height'][(df['major'] == 'COMPSCI')]

localhost:8888/notebooks/Downloads/A4_DataAnalysis.ipynb

14/20

5/20/2019 In [155]:

A4_DataAnalysis 1 2 3 4 5

assert np.all(h_co) assert np.all(h_cs) assert len(h_co) == 179 assert len(h_cs) == 165

5b) Mean height What is the average (mean) height for students from each major?

Save these values to 'avg_h_co' for cogs students, and 'avg_h_cs' for cs students.

In [156]:

1 2 3

# YOUR CODE HERE avg_h_co = h_co.mean() avg_h_cs = h_cs.mean()

In [157]:

1 2 3

assert avg_h_co assert avg_h_cs

Print out the average heights - this code provided In [158]:

1 2

print('Average height of cogs majors is \t {:2.2f} inches'.format(avg_h_co)) print('Average height of cs majors is \t\t {:2.2f} inches'.format(avg_h_cs))

Average height of cogs majors is Average height of cs majors is

66.75 inches 67.91 inches

Based on the cell above, it looks like there might indeed be a difference in the average height for students in cogs vs cs majors.

Now we want to statistically test this difference. To do so, we will use a t-test.

5c) T-test Use a t-test ('ttest_ind' function) to compare the two height distributions ('h_co' vs 'h_cs') 'ttest_ind' returns a t-statistic, and a p-value. Save these outputs to 't_val' and 'p_val' respectively.

In [160]:

1 2

# YOUR CODE HERE t_val, p_val = ttest_ind(h_co, h_cs)

In [161]:

1 2 3

assert t_val assert p_val

Check if statistical test passes significance, using an alpha value of 0.01. This code provided.

In [162]:

1 2 3 4 5 6 7

if p_val < 0.01: print('Data Science accomplished, there is a significant difference!') else: print('There is NOT a significant difference!') # Editorial note: # Chasing signitificant p-values as the goal itself is not actually a good way to do data (or any) science :)

Data Science accomplished, there is a significant difference!

Note: this test should pass significance. If it doesn't, double check your code up until this point.

So - we've reached a conclusion! We're done right!?

Nope. We have a first pass analysis, and an interim conclusion that happens to follow our hypothesis.

Now let's try to break it.

Let's explore some more You should always interrogate your findings, however they come out. What could be some alternate explanations, that would change our interpretations of the current analyses?

In this case, we should be worried about confounding variables. We want to be able to say whether height relates to major specifically, but it could be the case that some other variable, that happens to differ between majors, better explains the differences in height.

In this case, we also have data on gender. Let's check if differences in the gender ratio of the two majors can explain the difference in height.

5d) Separate the genders Using 'value_counts' from pandas, extract the number of 'male' and 'female', separately for cogs and cs students.

To do so, select from the df each major, separately, extract the gender column, and use the 'value_counts' method.

Save the counts for each gender for 'COGSCI' majors to a variable called 'g_co' Save the counts for each gender for 'COMPSCI' majors to a variable called 'g_cs'

In [163]:

1 2 3

# YOUR CODE HERE g_co = df['gender'][df['major'] == 'COGSCI'].value_counts() g_cs = df['gender'][df['major'] == 'COMPSCI'].value_counts()

localhost:8888/notebooks/Downloads/A4_DataAnalysis.ipynb

15/20

5/20/2019 In [164]:

A4_DataAnalysis 1 2 3 4 5

assert np.all(g_co) assert np.all(g_cs) assert g_co[0] == 93 assert g_cs[1] == 38

5e) Ratio of women What is the ratio of women in each major?

By ratio, we mean the proportion of students that are female, as a ratio. This will be value between 0.0 and 1.0 , calculated as #F / (#M + #F) - done separately for each major. You can use the g_co and g_cs variables to calculate these.

Save the ratio of women in COGSCI to a variable 'r_co' Save the ratio of women in COMPSCI to a variable 'r_cs' Note: keep these numbers as ratios (they should be decimal numbers, less than 1) In [168]:

1 2 3 4 5

# YOUR CODE HERE r_co = g_co[1]/(g_co[1] + g_co[0]) r_cs = g_cs[1]/(g_cs[1] + g_cs[0]) print(r_co) print(r_cs)

0.48044692737430167 0.23030303030303031 In [169]:

1 2 3

assert r_co assert r_cs

Make sure you print out and check the values of these ratios. They seem pretty different.

We can actually ask, using a chi-squared test, whether this difference in gender-ratio between the majors is signficantly different.

Code to do this is provided below.

In [170]:

1 2 3 4 5

# Run a chisquared test of the difference of ratios of categorical data between groups chisq, p_val_chi = stats.chisquare(np.array([g_co.values, g_cs.values]), axis=None) if p_val_chi < 0.01: print('There is a significant difference in ratios!')

There is a significant difference in ratios!

5f) Separate the majors Create a new dataframe, called df2 , which only includes data from COGSCI and COMPSCI majors.

Hint: you can do this using the or operater | , with loc.

In [172]:

1 2

# YOUR CODE HERE df2 = df.loc[(df['major'] == 'COGSCI') | (df['major'] == 'COMPSCI')]

In [173]:

1 2 3

assert isinstance(df2, pd.DataFrame) assert df2.shape == (344, 10) assert set(df2['major']) == set(['COGSCI', 'COMPSCI'])

5g) Pivot Tables Another way to look at these kinds of comparisons is pivot tables.

Use the pandas pivot_table method to create pivot table, assign it to a variable pv . Set the values as height , and the indices as gender and major in the pivot table. Make sure you do this using df2 .

In [174]:

1 2

# YOUR CODE HERE pv = pd.pivot_table(df2, index=['gender','major'], values = ['height'])

In [175]:

1 2 3

assert np.all(pv) assert isinstance(pv.index, pd.MultiIndex)

Print out the pivot table you just created.Compare the average height values, split up by major and gender.

Does it look like there are differences in heights by major, when spit up by major?

In [176]:

1

pv

Out[176]:

height gender female

male

major COGSCI

63.708140

COMPSCI

63.760789

COGSCI

69.557419

COMPSCI

69.150709

Let's recap where we are: Our initial hypothesis suggested there is a significant difference between heights of people in different majors.

localhost:8888/notebooks/Downloads/A4_DataAnalysis.ipynb

16/20

5/20/2019

A4_DataAnalysis However, further analyses suggested there may be a confounding variable, as there is also a significantly different gender balance between majors.

Checking the average height, per major, split up by gender, suggests there may not be a difference between major, other than what is explained by gender.

Now we want to statistically ask this question: is there still a different in height between majors, when controlling for differences in gender.

Linear Models For the following question you will need to make some linear models, using Ordinary Least Squares (OLS).

There is more than one way to do this in Python. For the purposes of this assignment, you must use the method that is outlined in the 'LinearModels' Tutorial, using patsy, and statsmodels.

That is: Create design matrices with 'patsy.dmatrices' Iniliaize an OLS model with 'sm.OLS' Fit the OLS model Check the summary for results.

5h) Predict height from major Create a linear model to predict height from major (using df2 as data).

Use patsy.dmatrices to create the design matrices, calling the outputs outcome_1 , predictors_1 . Create an OLS model (sm.OLS) using outcome_1 and predictors_1 . Call it mod_1 . Fit the model, assigning it to res_1 .

In [177]:

1 2 3 4

# YOUR CODE HERE outcome_1, predictors_1 = patsy.dmatrices('height ~ major', df2) mod_1 = sm.OLS(outcome_1, predictors_1) res_1 = mod_1.fit()

In [178]:

1 2 3 4 5

assert isinstance(outcome_1, patsy.design_info.DesignMatrix) assert isinstance(predictors_1, patsy.design_info.DesignMatrix) assert isinstance(mod_1, sm.regression.linear_model.OLS) assert isinstance(res_1, sm.regression.linear_model.RegressionResultsWrapper)

Print out the summary results of the model fitting In [179]:

1

print(res_1.summary())

OLS Regression Results ============================================================================== Dep. Variable: height R-squared: 0.023 Model: OLS Adj. R-squared: 0.020 Method: Least Squares F-statistic: 8.144 Date: Sun, 03 Mar 2019 Prob (F-statistic): 0.00458 Time: 23:11:41 Log-Likelihood: -943.96 No. Observations: 344 AIC: 1892.

Df Residuals: 342 BIC: 1900.

Df Model: 1 Covariance Type: nonrobust ==================================================================================== coef std err t P>|t| [0.025 0.975] -----------------------------------------------------------------------------------Intercept 66.7472 0.282 236.645 0.000 66.192 67.302 major[T.COMPSCI] 1.1622 0.407 2.854 0.005 0.361 1.963 ============================================================================== Omnibus: 5.810 Durbin-Watson: 1.893 Prob(Omnibus): 0.055 Jarque-Bera (JB): 3.629 Skew: -0.004 Prob(JB): 0.163 Kurtosis: 2.497 Cond. No. 2.57 ============================================================================== Warnings: [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

5i) Analysis Based on the model you ran above (using alpha value of 0.01), does major significantly predict height?

Set your answer as a boolean (True / False) to a variable called lm_1 .

In [180]:

1 2

# YOUR CODE HERE lm_1 = True

In [181]:

1 2

assert isinstance(lm_1, bool)

5j) Predict height from both major & gender Create a linear model to predict height from both major and gender (using df2 as data).

Use patsy.dmatrices to create the design matrices, calling the outputs outcome_2 , predictors_2 . Create an OLS model (sm.OLS) using outcome_2 and predictors_2 . Call it mod_2 . Fit the model, assigning it to res_2 .

In [182]:

1 2 3 4

# YOUR CODE HERE outcome_2, predictors_2 = patsy.dmatrices('height ~ major + gender', df2) mod_2 = sm.OLS(outcome_2, predictors_2) res_2 = mod_2.fit()

localhost:8888/notebooks/Downloads/A4_DataAnalysis.ipynb

17/20

5/20/2019

A4_DataAnalysis

In [183]:

1 2 3 4 5

assert isinstance(outcome_2, patsy.design_info.DesignMatrix) assert isinstance(predictors_2, patsy.design_info.DesignMatrix) assert isinstance(mod_2, sm.regression.linear_model.OLS) assert isinstance(res_2, sm.regression.linear_model.RegressionResultsWrapper)

In [184]:

1 2

# Print out the results print(res_2.summary())

OLS Regression Results ============================================================================== Dep. Variable: height R-squared: 0.500 Model: OLS Adj. R-squared: 0.497 Method: Least Squares F-statistic: 170.2 Date: Sun, 03 Mar 2019 Prob (F-statistic): 5.57e-52 Time: 23:13:49 Log-Likelihood: -828.95 No. Observations: 344 AIC: 1664.

Df Residuals: 341 BIC: 1675.

Df Model: 2 Covariance Type: nonrobust ==================================================================================== coef std err t P>|t| [0.025 0.975] -----------------------------------------------------------------------------------Intercept 63.8026 0.260 245.387 0.000 63.291 64.314 major[T.COMPSCI] -0.2555 0.302 -0.845 0.399 -0.850 0.339 gender[T.male] 5.6675 0.315 18.014 0.000 5.049 6.286 ============================================================================== Omnibus: 8.283 Durbin-Watson: 2.001 Prob(Omnibus): 0.016 Jarque-Bera (JB): 5.965 Skew: 0.199 Prob(JB): 0.0507 Kurtosis: 2.492 Cond. No. 3.38 ============================================================================== Warnings: [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

5k) Analysis Based on the model you ran above (using alpha value of 0.01), does major significantly predict height?

Set your answer as a boolean (True / False) to a variable called lm_2 In [185]:

1 2

# YOUR CODE HERE lm_2 = True

In [186]:

1 2

assert isinstance(lm_2, bool)

Part 6: Discussion & Conclusions 6a) Conclusion Set a boolean variables, called 'ans', as True or False as the answer to the following question: We have evidence supporting our research question: People in different majors have systematically different heights (and this difference can be tied to their major).

In [187]:

1 2

# YOUR CODE HERE ans = False

In [188]:

1 2

assert isinstance(ans, bool)

6b) Summary Write a short response (1-2 sentence) summarizing the results.

Did we support our hypothesis? Why or why not? What turned out to be the finding(s).

We found out that the data in fact did not support the hypothesis. By looking at how gender and major worked independtly it was shown that major did not have a connection to height at all.

Parts 7 & 8: Other Stuff - OPTIONAL! These parts of the assignment (the rest of it) are completely OPTIONAL, and are ungraded.

The next couple sections break from the project narrative we've using above, and briefly explores some other topics, approaches and techniques that have (or will) be addressed in the class, and may be useful for you projects.

This section is still set up like an assignment, asking you to answer questions to practice working with these methods, with the solutions posted later, but all tests for this section are public.

There are example of using these methods in the Tutorials.

In [ ]:

1 2 3 4 5

# You will need the following functions for the last section of the assignment # You should not need to import any other functions from sklearn.cluster import KMeans from scipy.cluster.vq import whiten from sklearn.decomposition import PCA

Part 7: Clustering Suppose we have reason to believe there are distinct clusters of students in the class. However, we do not know which students are in which group, nor the boundaries that define the groups.

localhost:8888/notebooks/Downloads/A4_DataAnalysis.ipynb

18/20

5/20/2019

A4_DataAnalysis This is a clustering problem - here you will use KMeans to try and find the clusters from the data.

See an example of how to use sklearn, and specifically KMeans, for clustering in the 'Clustering' notebook in the Tutorials.

Make a copy of the dataframe, to work with here, and drop any empty data in columns we're using

In [ ]:

1 2

df3 = df.copy(deep=True) df3.dropna(subset=['age', 'weight', 'year'], inplace=True)

In [ ]:

1 2 3

# For convenience, and consistency, we're providing this code to pull out the required data d1 = df3[df3['major'] == 'COGSCI']['weight'].values d2 = df3[df3['major'] == 'COGSCI']['age'].values

7a) Whiten the data For both d1 , and d2 (separately) use the whiten function to whiten the data. Save the outputs to d1w , and d2w repectively.

Note: look into what this function does, and why you might want or need to do it.

In [ ]:

1 2

# YOUR CODE HERE raise NotImplementedError()

In [ ]:

1 2 3 4 5

assert np.all(d1w) assert np.all(d2w) assert round(np.var(d1w)) == 1 assert round(np.var(d2w)) == 1

7b) K-Means Initialize a Kmeans model object. Call it km . Initialize it with n_clusters of 2, and random_state of 42.

In [ ]:

1 2

# YOUR CODE HERE raise NotImplementedError()

In [ ]:

1 2 3 4

assert isinstance(km, KMeans)

1 2

# For convenience, and consistency, we're providing this code to combine the data into required format. cl_dat = np.array([d1w, d2w]).T

In [ ]:

assert km.n_clusters == 2 assert km.random_state == 42

7c) Fit the data Use the fit method of the KMeans object km to fit the data cl_dat .

In [ ]:

1 2

# YOUR CODE HERE raise NotImplementedError()

In [ ]:

1 2

assert np.any(km.labels_) assert sum(km.labels_) == 14

Let's check out our clusters (this code provided).

In [ ]:

1 2 3

plt.scatter(d1, d2, c=km.labels_); plt.xlabel('Weight'); plt.ylabel('Age');

Editorial note: It does not appear, for this data, that there are particular clusters (or at least not that our current KMeans approach is picking up), but that's fine, we can consider this to have been an exploratory analysis.

Part 8: Dimensionality Reduction Sometimes we have data that have many dimensions, perhaps too many to deal with, and so we might need to try and use dimensionality reduction: that is to try and find a lower dimensional space that captures as much of the information as we can from the original, high-dimensional, dataset.

As an example, in our data, we have height and weight that are quite correlated, and we could use Principal Components Analysis (PCA), in order to try and capture the most information about a persons' height and weight in a single number - projecting this 2D data down into 1 dimension.

See an example of how to use sklearn, and specifcally PCA, for dimensionality reduction in the 'Dimensionality Reduction' notebook in the Tutorials.

In [ ]:

1 2 3 4

# Check out the height vs. weight data plt.plot(df3['height'].values, df3['weight'].values, '.') plt.xlabel('Height'); plt.ylabel('Weight');

In [ ]:

1 2 3

# For convenience, and consistency, we're providing this code to pull out the required data d1_h = df3['height'].values d2_w = df3['weight'].values

localhost:8888/notebooks/Downloads/A4_DataAnalysis.ipynb

19/20

5/20/2019

A4_DataAnalysis 8a) PCA Initialize a PCA object. Call it dr . Initialize it with n_components of 1, and whiten as True.

In [ ]:

1 2

# YOUR CODE HERE raise NotImplementedError()

In [ ]:

1 2 3 4

assert isinstance(dr, PCA)

1 2

# For convenience, and consistency, we're providing this code to combine the data into required format dr_dat = np.array([d1_h, d2_w]).T

In [ ]:

assert dr.n_components == 1 assert dr.whiten

8b) Fit and tranform the data Use the fit_transform method of the PCA object dr on the data dr_dat . This method fits the dimensionality reduction model, and then transforms the data with it. Save the output (the transformed data) to out .

In [ ]:

1 2

# YOUR CODE HERE raise NotImplementedError()

In [ ]:

1

assert np.all(out)

Check the transformed data (this code provided).

Since this data is now 1D, the x-axis is just the index number of each data point. Now, for each student, we have a (whitened) single number representing combined height & weight.

In [ ]:

1 2 3

plt.plot(out, '.'); plt.xlabel('Subject Number'); plt.ylabel('PC1');

Well done, Note: before you submit to TED make sure you click 'Kernel'>'Restart & Run All' and check that none of the assert statements fail. If any asserts fail in this process they will fail during autograding.

localhost:8888/notebooks/Downloads/A4_DataAnalysis.ipynb

20/20