COGS 108 - Assignment 4: Natural Language Processing

downloaded from

Scholarfriends

https://scholarfriends.com

Downloaded by: browsegrades@gmail.com | https://scholarfriends.com/singlePaper/186930/natural-language-processing Distribution of this document is illegal.

A4_NLP March 10, 2021

1

COGS 108 - Assignment 4: Natural Language Processing

This assignment covers working with text data, machine learning, and NLP. This assignment is out of 9 points, worth 9% of your grade.

PLEASE DO NOT CHANGE THE NAME OF THIS FILE. PLEASE DO NOT COPY & PASTE OR DELETE CELLS INLCUDED IN THE ASSIGNMENT.

2

Important • This assignment has hidden tests: tests that are not visible here, but that will be run on your submitted assignment for grading.

– This means passing all the tests you can see in the notebook here does not guarantee you have the right answer!

– In particular many of the tests you can see simply check that the right variable names exist. Hidden tests check the actual values. ∗ It is up to you to check the values, and make sure they seem reasonable.

• A reminder to restart the kernel and re-run the code as a first line check if things seem to go weird.

– For example, note that some cells can only be run once, because they re-write a variable (for example, your dataframe), and change it in a way that means a second execution will fail.

– Also, running some cells out of order might change the dataframe in ways that may cause an error, which can be fixed by re-running.

3

Background & Work Flow

In this homework assignment, we will be analyzing text data.

Part 1 will look at readability index, which identifies the level of difficulty one might face reading a piece of text. While many methods exist to calculate readability index, we will use Dale Chall criterion for this assignment.

In Parts 2 and 3, we will build models to perform classification tasks. A common approach to analyzing text data is to use methods that allow us to convert text data into some kind of numerical representation - since we can then use all of our mathematical tools on such data. In this assignment, we will explore 2 feature engineering methods that convert raw text data into numerical vectors: 1

- Bag of Words (BoW) - BoW encodes an input sentence as the frequency of each word in the sentence. - In this approach, all words contribute equally to the feature vectors. - Term Frequency - Inverse Document Frequency (TF-IDF) - TF-IDF is a measure of how important each term is to a specific document, as compared to an overall corpus. - TF-IDF encodes each word as its frequency in the document of interest, divided by a measure of how common the word is across all documents (the corpus). - Using this approach, each word contributes differently to the feature vectors. - The assumption behind using TF-IDF is that words that appear commonly everywhere are not that informative about what is specifically interesting about a document of interest, so it is tuned to representing a document in terms of the words it uses that are different from other documents.

• We will apply both the above methods to quantify textual data and perform classification tasks. We explore 2 tasks here, one being multiclass classification of newsgroup emails into various categories like sports, politics, religion etc using a Decision Tree (Part 2). Then, we will explore sarcasm detection using a SVM (support vector machine) (Part 3).

3.1

Setup

Run the following cell to import all of the packages/modules/functions we’ll need for this assignment.

Note: these are all the imports you need for this assignment! Do not import any other functions / packages [1]: # Imports - these are all the imports needed for the assignment %matplotlib inline import numpy as np import pandas as pd import matplotlib.pyplot as plt import seaborn as sns # Import nltk package # PennTreeBank word tokenizer # English language stopwords import nltk from nltk.tokenize import word_tokenize from nltk.corpus import stopwords import warnings warnings.filterwarnings('ignore') # scikit-learn imports # SVM (Support Vector Machine) classifer # Vectorizer, which transforms text data into bag-of-words feature # TF-IDF Vectorizer that first removes widely used words in the dataset and␣ ,→then transforms test data # Metrics functions to evaluate performance from sklearn.svm import SVC

2

from sklearn.tree import DecisionTreeClassifier from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer from sklearn.metrics import classification_report,␣ ,→precision_recall_fscore_support from sklearn.model_selection import train_test_split Run the following cell to download the NLTK English tokenizer and the stopwords of all languages.

[2]: nltk.download('punkt') nltk.download('stopwords') [nltk_data] Downloading package punkt to /home/v1lu/nltk_data… [nltk_data] Package punkt is already up-to-date! [nltk_data] Downloading package stopwords to /home/v1lu/nltk_data… [nltk_data] Package stopwords is already up-to-date!

[2]: True Finally, run the following cell to define a fixed random state. This will ensure that the results you generate are the same same each time you run the code/do sampling.

[3]: # define random state for use random_state = 10

3.2

# Part 1: Data, EDA, and Readability index of Newsgroups (3 pts)

In this first set of questions you will: - read the dataset in and do minimal cleaning - use the data provided to calculate metrics needed - carry out some basic EDA 3.2.1

1a) Import Data

Read the CSV file 'data/20news_data.csv', storing its contents into a dataframe news_df. Set the column names as ‘category’, ‘subject’ and ‘message’.

The dataset we’ll be using here are email discussions from News groups. The category being discussed is in the first column. The subject of the email thread is in the second. And its contents is in the third. Note that these are from actual discussions. Some sensitive topics and discussions are included; the statements within these datasets do not necessarily reflect the sentiments of the course staff.

[4]: news_filepath = "data/20news_data.csv" # YOUR CODE HERE news_df = pd.read_csv(news_filepath, names=['category','subject','message']) [5]: # take a look at 5 random rows from the dataset # note that b/c random_state is set, you will get same 5 rows each time news_df.sample(5, random_state=random_state)

3

[5]: 16150 256 8355 9609 8288

category politics politics recreational religion recreational

subject National Crime Survey Re: Europe vs. Muslim Bosnians Re: wife wants convertible Re: After 2000 years, can we say that Christia…

Re Aftermarket A/C units

16150 256 8355 9609 8288

message Well, I dropped by the library yesterday, and …

I like what Mr. Joseph Biden had to say yester… : : : HELP!!! : my wife has informed me tha…

I'll take a wild guess and say Freedom is obje… | I looked into getting a/c installed on my 19…

\

[6]: assert isinstance(news_df, pd.DataFrame) assert len(news_df) == 18773 assert list(news_df.columns) == ['category', 'subject', 'message'] 3.2.2

1b) Remove null observations

Given that we are carrying out a classification/prediction task, we’re going to drop null values from the dataframe. Drop any rows containing null values in any of the three columns of news_df. Store this back into news_df.

[7]: # YOUR CODE HERE news_df = news_df.dropna() [8]: assert news_df.isna().sum().sum() == 0 assert news_df.shape == (18731, 3)

3.3

Readability Index

Now, it’s time to calculate the readability index of each text entry. This is the measure of how easy or hard it might be to read each message. We will use the Dale-Chall method to calculate it in the next few steps, using the following: Readability Score of a text = 0.1579 (Percentage Difficult Words not on the Dale– Chall word list) + 0.0496 * (Average Sentence Length) + 3.6365* 3.3.1

1c) Average sentence length

We will create a function (avg_sentence_len) to calculate the average sentence length across a piece of text.

This function should take text as an input parameter.

Within this function: 1. sentences: Use the split() string method to split the input text at every '.'. This will split the text into a list of sentences. Store this in the variable sentences. To keep things simple, we will consider every “.” as a sentence separator. (This decision could lead to

4

misleading answers. For example, “Hello Dr. Jacob.” is actually a single sentence, but our function will consider this 2 separate sentences). 2. words: Use the split() method to split the input text into a list of separate words, storing this in words. Again, to limit complexity, we will assume that all words are separated by a single space (" "). (So, while “I am going.to see you later” actually has 7 words, since there is no space after the “.”, so we will assume the this to contain 6 separate words in our function.) 3. Calculate the average sentence length, returning this from your function: - if the last value in sentences is an empty string: the average sentence length should be the number of words divided by the len(sentences) - 1. - otherwise, the average sentence length should be the number of words divided by the number of sentences.

For the “I am going.to see you later” example, your function should return 3.0.

[9]: # YOUR CODE HERE def avg_sentence_len(text): sentences = text.split('.') words = text.split(" ") avg = len(words)/len(sentences) if sentences[-1] == '': avg = len(words)/(len(sentences)-1) else: avg = avg return avg [10]: assert avg_sentence_len("each sentence. has two. words right?.") == 2 assert avg_sentence_len("a. a. a") == avg_sentence_len("a. a. a.") assert avg_sentence_len("a. a. a") == 1 assert avg_sentence_len("hello Dr. Jacob.") == 1.5 assert avg_sentence_len(news_df['message'][0])//1 == 16 assert avg_sentence_len("one. two.") != avg_sentence_len("one.two.")

3.3.2

1d) Calculate Average Sentence Length

Apply the above method to 'message' column of news_df, creating a new column in news_df called 'ASL'. This column should store the average sentence length of each message.

[11]: # YOUR CODE HERE news_df['ASL'] = news_df['message'].apply(avg_sentence_len) [12]: assert "ASL" in news_df.columns assert np.isclose(news_df.loc[0,'ASL'], 16.07, 0.01) [13]: # look at output news_df.head()

5

[13]: 0 1 2 3 4

category politics politics politics politics politics

0 1 2 3 4

message ASL Srinivas Suder writes: If the Haitian people's… 16.066667 From: Center for Policy Research <cpr Subject:… 13.307692 My question is, given so many separations in t… 15.469697 Come again? The image-conscious Armenians sore… 17.461538 I have never seen such immaturity among semito… 19.000000

3.3.3

subject Re: The Stage is Being Set Re: Zionism - racism The Armenians did not form a distinct race.

Re: As Muslim women and children were being ma…

Re: Israel's Expansion II

\

1e) Difficult Words

The file 'data/dale_chall.txt' contains a list of words which are considered easy in general.

Read this file in, storing the file in a DataFrame called dale_chall_file. (Note that this is a .txt file, so be sure to consider what the sep parameter should be when reading this in and what the value for header should be.)

Set the column name to ‘easy_words’ We will use this list of words to calculate the percentage of difficult words for each message.

[14]: # YOUR CODE HERE dale_chall_file = pd.read_csv('data/dale_chall.txt', sep='\n',␣ ,→names=['easy_words']) [15]: dale_chall_file [15]: 0 1 2 3 4 …

8485 8486 8487 8488 8489

easy_words a able abler ablest abling … yourself yourselves yous youth youths

[8490 rows x 1 columns] [16]: assert dale_chall_file.shape == (8490, 1) assert list(dale_chall_file.columns) == ['easy_words']

6

3.3.4

1f) Percent Difficult Words

Using dale_chall_words, define a function percent_difficult_words that given a text input will calculate the percent difficult words, using the following approach: Return the percent of words in a given text input that are not in dale_chall_words['easy_words'].

Note that a “word” is defined by being separated from other text by a space.

[17]: # YOUR CODE HERE def percent_difficult_words(text): words = text.split(' ') difficult_words = 0 dale = set(dale_chall_file['easy_words']) for word in words: if word not in dale: difficult_words +=1 else: continue return difficult_words/len(words) [18]: assert callable(percent_difficult_words) assert percent_difficult_words("there are only easy words here") == 0 assert percent_difficult_words("let us put one difficult word here") == 1/7 assert percent_difficult_words("thare aree noo unintentional misstakes") == 1 3.3.5

1g) Calculate Percent Difficult words

Apply the above method to 'message' column of news_df, creating a new column in news_df called 'PDW'. This column should store the percent difficult words of each message.

[19]: # YOUR CODE HERE news_df['PDW'] = news_df['message'].apply(percent_difficult_words) [20]: assert news_df['PDW'].max() == 1.0 assert news_df['PDW'].min() == 0 assert round(news_df['PDW'].mean(), 2) == 0.48 [21]: # look at output news_df.head() [21]: 0 1 2

category politics politics politics

subject Re: The Stage is Being Set Re: Zionism - racism The Armenians did not form a distinct race.

7

\

3 4

politics politics

0 1 2 3 4

message ASL PDW Srinivas Suder writes: If the Haitian people's… 16.066667 0.381743 From: Center for Policy Research <cpr Subject:… 13.307692 0.624277 My question is, given so many separations in t… 15.469697 0.403526 Come again? The image-conscious Armenians sore… 17.461538 0.515419 I have never seen such immaturity among semito… 19.000000 0.445344

3.3.6

Re: As Muslim women and children were being ma…

Re: Israel's Expansion II

1h) Readability Score

Now we’re ready to calculate each message’s readability score, using the following approach: ReadabilityScore = 0.1579 ∗ (P DW ) + 0.0496 ∗ (ASL) + 3.6365 Here: - P DW = Percentage of difficult words not on the Dale–Chall word list. - ASL = Average sentence length The larger the readability score the higher the necessary reading level of the reader would need to be to comprehend its contents. For example, a score of 4.9 or below would be able to be understood by someone at the 4th grade reading level or below, where a score of 9 and above would require a college level reading level.

Store the readability score in the column 'readability' in news_df.

[22]: news_df [22]: 0 1 2 3 4 …

18768 18769 18770 18771 18772

category politics politics politics politics politics … religion religion religion religion religion

subject Re: The Stage is Being Set Re: Zionism - racism The Armenians did not form a distinct race.

Re: As Muslim women and children were being ma…

Re: Israel's Expansion II …

Wanted: Moltmann's God in Creation Re: 666 - MARK OF THE BEAST - NEED INFO Re: 14 Apr 93 God's Promise in 1 John 1: 7 Re: Spreading Christianity (Re: Christian Extr…

Re: 'Imaginary' Friends - Info and Experiences

0 1 2 3 4 …

18768

message ASL PDW Srinivas Suder writes: If the Haitian people's… 16.066667 0.381743 From: Center for Policy Research <cpr Subject:… 13.307692 0.624277 My question is, given so many separations in t… 15.469697 0.403526 Come again? The image-conscious Armenians sore… 17.461538 0.515419 I have never seen such immaturity among semito… 19.000000 0.445344 … … …

I'm greatly in need of Jurgen Moltmann's book …

9.800000 0.530612

8

\

18769 18770 18771 18772

: Marian CATHOLIC high school, outside of chic… 10.800000 0.796296 Human blood sacrifice! Martyrdom of an innocen… 16.666667 0.400000 Not to be too snide about it, but I think this… 15.500000 0.516129 Concerniong this thread… Has anyone ever se…

9.363636 0.601942

[18731 rows x 5 columns] [23]: # YOUR CODE HERE r_scores = [] for asl, pdw in zip(news_df.ASL, news_df.PDW): score = 0.1579 * pdw + 0.0496 * asl + 3.6365 r_scores.append(score) news_df['readability'] = r_scores [24]: news_df.shape [24]: (18731, 6) [25]: assert news_df.shape == (18731, 6) assert np.isclose(news_df.loc[0,'readability'], 4.49, 0.1) assert round(news_df['readability'].mean(), 2) == 4.46 [26]: # look at output news_df.head() [26]: 0 1 2 3 4

category politics politics politics politics politics

subject Re: The Stage is Being Set Re: Zionism - racism The Armenians did not form a distinct race.

Re: As Muslim women and children were being ma…

Re: Israel's Expansion II

0 1 2 3 4

message ASL PDW Srinivas Suder writes: If the Haitian people's… 16.066667 0.381743 From: Center for Policy Research <cpr Subject:… 13.307692 0.624277 My question is, given so many separations in t… 15.469697 0.403526 Come again? The image-conscious Armenians sore… 17.461538 0.515419 I have never seen such immaturity among semito… 19.000000 0.445344

0 1 2 3 4

readability 4.493684 4.395135 4.467514 4.583977 4.649220

9

\

\

3.3.7

1i) Readability by category

Generate a boxplot that will summarize the readability scores within each category.

[27]: # YOUR CODE HERE politics = news_df.drop(news_df.index[news_df['category'] !=␣ ,→'politics'])['readability'] recreational = news_df.drop(news_df.index[news_df['category'] !=␣ ,→'recreational'])['readability'] computer = news_df.drop(news_df.index[news_df['category'] !=␣ ,→'computer'])['readability'] religion = news_df.drop(news_df.index[news_df['category'] !=␣ ,→'religion'])['readability'] science = news_df.drop(news_df.index[news_df['category'] !=␣ ,→'science'])['readability'] misc = news_df.drop(news_df.index[news_df['category'] != 'misc'])['readability'] fig = plt.figure(figsize =(10, 10)) ax = fig.add_subplot() ax.set_xticklabels(['politics', 'recreational', 'computer', 'religion',␣ ,→'science', 'misc']) data = [politics,recreational,computer,religion,science,misc] bp = plt.boxplot(data) f1 = plt.gcf()

10

[28]: assert f1.gca().has_data() In the above plot, you should notice that one message from the recreational dataset has a way higher readability score than the rest of the dataset. Run the following two cells to see why that is the case…

[29]: # take a look at extreme readability value news_df[news_df['readability'] > 80] [29]: 2640

category recreational

subject PLAYOFFS STATS through Sunday 4/25

2640

message ASL PDW Stanley Cup Playoff stats through games played… 1560.0 0.996154

11

\

\

2640

readability 81.169793

[30]: # see full message list(news_df.loc[news_df['readability'] > 80, 'message']) [30]: ['Stanley Cup Playoff stats through games played Sunday, April 25 1993 -------------------------------------------------------------------- Top 20 Point Scorers Rnk Pts G A PG SG GW EN PIM Cty Tm Ps # Player --- --- -- --- --- -- -- --- --- --- -- -- ------ 1 9 0 9 0 0 0 0 4 CAN BOS C 12 Adam Oates 2 8 5 3 4 0 2 0 0 USA STL RW 16 Brett Hull 3 7 6 1 2 0 0 0 6 RUS BUF RW 89 Alexander Mogilny 3 7 4 3 1 1 0 0 4 CAN PIT C 66 Mario Lemieux 3 7 3 4 2 1 0 0 19 CAN CAL RW 14 Theoren Fleury 3 7 2 5 1 0 0 0 12 CAN TOR C 93 Doug Gilmour 3 7 1 6 0 0 0 0 0 CAN DET D 77 Paul Coffey 3 7 1 6 0 0 0 0 0 USA BUF C 16 Pat LaFontaine 9 6 4 2 0 0 2 0 4 CAN NYI C 20 Ray Ferraro 9 6 3 3 0 0 1 0 2 RUS VAN RW 10 Pavel Bure 9 6 3 3 1 0 1 0 0 RUS BUF LW 13 Yuri Khmylev 9 6 3 3 1 0 1 0 6 CAN VAN RW 16 Trevor Linden 9 6 2 4 1 0 0 0 2 CAN BUF C 10 Dale Hawerchuk 9 6 2 4 1 0 2 0 0 USA STL C 15 Craig Janney 9 6 2 4 2 0 0 0 6 CAN BOS C 49 *Joe Juneau 9 6 2 4 0 0 0 1 6 CAN CAL C 25 Joe Nieuwendyk 9 6 1 5 0 0 0 0 4 CAN STL D 21 Jeff Brown 9 6 1 5 0 0 1 0 4 CAN PIT C 10 Ron Francis Top 10 Goal Scorers Rnk Pts G A PG SG GW EN PIM Cty Tm Ps # Player --- --- -- --- -- -- -- -- --- --- --- -- -- ------ 1 7 6 1 2 0 0 0 6 RUS BUF RW 89 Alexander Mogilny 2 8 5 3 4 0 2 0 0 USA STL RW 16 Brett Hull 2 5 5 0 3 0 1 0 14 CAN WAS C 32 Dale Hunter 4 7 4 3 1 1 0 0 4 CAN PIT C 66 Mario Lemieux 4 6 4 2 0 0 2 0 4 CAN NYI C 20 Ray Ferraro 4 5 4 1 0 0 1 0 6 CAN TOR LW 14 Dave Andreychuk 4 5 4 1 1 0 0 0 4 CAN BOS RW 8 Cam Neely 4 4 4 0 2 0 0 0 2 USA LA C 12 Jimmy Carson 4 4 4 0 0 0 2 0 0 USA QUE RW 48 Scott Young Top 10 Assist Scorers Rnk Pts G A PG SG GW EN PIM Cty Tm Ps # Player --- --- -- ---- -- -- -- --- --- --- -- -- ------ 1 9 0 9 0 0 0 0 4 CAN BOS C 12 Adam Oates 2 7 1 6 0 0 0 0 0 CAN DET D 77 Paul Coffey 2 7 1 6 0 0 0 0 0 USA BUF C 16 Pat LaFontaine 4 7 2 5 1 0 0 0 12 CAN TOR C 93 Doug Gilmour 4 6 1 5 0 0 0 0 4 CAN STL D 21 Jeff Brown 4 6 1 5 0 0 1 0 4 CAN PIT C 10 Ron Francis 4 5 0 5 0 0 0 0 0 USA WIN D 6 Phil Housley 4 5 0 5 0 0 0 0 4 SWE WAS D 6 Calle Johansson 4 5 0 5 0 0 0 0 8 CAN CAL D 2 Al MacInnis Top 10 Power-play Goal Scorers Rnk Pts G A PG SG GW EN PIM Cty Tm Ps # Player --- --- -- --- -- -- -- -- --- --- --- -- -- -----1 8 5 3 4 0 2 0 0 USA STL RW 16 Brett Hull 2 5 5 0 3 0 1 0 14 CAN WAS C 32 Dale Hunter 3 7 3 4 2 1 0 0 19 CAN CAL RW 14 Theoren Fleury 3 7 6 1 2 0 0 0 6 RUS BUF RW 89 Alexander Mogilny 3 6 2 4 2 0 0 0 6 CAN BOS C 49 *Joe Juneau 3 5 2 3 2 0 0 0 6 CAN STL LW 19 Brendan Shanahan 3 4 4 0 2 0 0 0 2 USA LA C 12 Jimmy Carson 3 4 2 2 2 0 0 0 2 TCH CAL C 26 Robert Reichel 3 4 3 1 2 0 1 0 2 FIN WIN RW 13 *Teemu Selanne 3 3 2 1 2 0 1 0 6 CAN MON LW 25 Vincent Damphousse All the Shorthanded Goal Scorers Rnk Pts G A PG SG GW EN PIM Cty Tm Ps # Player --- --- ---- -- -- -- -- --- --- --- -- -- ------ 1 7 3 4 2 1 0 0 19 CAN CAL RW 14 Theoren Fleury 1 7 4 3 1 1 0 0 4 CAN PIT C 66 Mario Lemieux 1 4 3 1 0 1 1 0 4 USA CAL C 29 Joel Otto 1 4 2 2 0 1 0 0 6 USA CAL D 20 Gary Suter 1 3 2 1 1 1 0 0 23 RUS DET C 91 Sergei Fedorov 1 2 1 1 0 1 0 0 10 CAN BOS C 19 Dave Poulin 1 1 1 0 0 1 0 0 4 CAN WIN C 38 Luciano Borsato 1 1 1 0 0 1 0 0 2 CAN DET LW 11 Shawn

12

Burr 1 1 1 0 0 1 0 0 2 CAN STL LW 10 Dave Lowry 1 1 1 0 0 1 1 0 0 USA BUF RW 18 Wayne Presley Top 10 Game-winning Goal Scorers Rnk Pts G A PG SG GW EN PIM Cty Tm Ps # Player --- --- -- --- -- -- -- -- --- --- --- -- -- ------ 1 8 5 3 4 0 2 0 0 USA STL RW 16 Brett Hull 1 6 4 2 0 0 2 0 4 CAN NYI C 20 Ray Ferraro 1 6 2 4 1 0 2 0 0 USA STL C 15 Craig Janney 1 4 4 0 0 0 2 0 0 USA QUE RW 48 Scott Young 5 6 3 3 0 0 1 0 2 RUS VAN RW 10 Pavel Bure 5 6 1 5 0 0 1 0 4 CAN PIT C 10 Ron Francis 5 6 3 3 1 0 1 0 0 RUS BUF LW 13 Yuri Khmylev 5 6 3 3 1 0 1 0 6 CAN VAN RW 16 Trevor Linden 5 5 4 1 0 0 1 0 6 CAN TOR LW 14 Dave Andreychuk 5 5 5 0 3 0 1 0 14 CAN WAS C 32 Dale Hunter Top 5 Point Scoring Defensemen Rnk Pts G A PG SG GW EN PIM Cty Tm Ps # Player --- --- -- --- -- -- -- -- --- --- --- -- -- -----3 7 1 6 0 0 0 0 0 CAN DET D 77 Paul Coffey 9 6 1 5 0 0 0 0 4 CAN STL D 21 Jeff Brown 19 5 0 5 0 0 0 0 0 USA WIN D 6 Phil Housley 19 5 0 5 0 0 0 0 4 SWE WAS D 6 Calle Johansson 19 5 0 5 0 0 0 0 8 CAN CAL D 2 Al MacInnis Top 4 Rookie Point Scorers Rnk Pts G A PG SG GW EN PIM Cty Tm Ps # Player --- --- -- --- -- -- --- --- --- --- -- -- ------ 9 6 2 4 2 0 0 0 6 CAN BOS C 49 *Joe Juneau 33 4 3 1 0 0 1 0 8 USA PIT C 15 *Shawn McEachern 33 4 3 1 2 0 1 0 2 FIN WIN RW 13 *Teemu Selanne 33 4 0 4 0 0 0 0 2 TCH BUF D 42 *Richard Smehlik Top 5 Penalty Minute Leaders Rnk Pts G A PG SG GW EN PIM Cty Tm Ps # Player --- --- -- --- -- -- --- --- --- --- -- -- ------ 1 1 0 1 0 0 0 0 26 CAN DET LW 55 Keith Primeau 2 3 2 1 1 1 0 0 23 RUS DET C 91 Sergei Fedorov 3 0 0 0 0 0 0 0 22 CAN NYI D 47 Richard Pilon 4 2 1 1 0 0 0 0 21 CAN TOR RW 12 Rob Pearson 4 1 1 0 0 0 0 0 21 CAN WIN RW 20 Tie Domi %*%*%*%**%*%%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%* * __ ______________ ____________________________________ % % _)____________/ A L L E Z L E S B L U E S ! ! ! * * __________/ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ % % ________/ * * _______/ Joe Ashkar % % Contact for the Blues * % (___) BLUES * *%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%'] 3.3.8

1j) Remove outlier & boxplot

As you can see above, the outlier is not quite text, but rather sports stats. Remove that observation from your dataset, and store the output back into news_df.

Then, regenerate your boxplot looking at readability by category [31]: # YOUR CODE HERE news_df = news_df.drop([2640]) politics = news_df.drop(news_df.index[news_df['category'] !=␣ ,→'politics'])['readability'] recreational = news_df.drop(news_df.index[news_df['category'] !=␣ ,→'recreational'])['readability'] computer = news_df.drop(news_df.index[news_df['category'] !=␣ ,→'computer'])['readability'] religion = news_df.drop(news_df.index[news_df['category'] !=␣ ,→'religion'])['readability']

13

science = news_df.drop(news_df.index[news_df['category'] !=␣ ,→'science'])['readability'] misc = news_df.drop(news_df.index[news_df['category'] != 'misc'])['readability'] fig = plt.figure(figsize =(10, 10)) ax = fig.add_subplot() ax.set_xticklabels(['politics', 'recreational', 'computer', 'religion',␣ ,→'science', 'misc']) data = [politics,recreational,computer,religion,science,misc] bp = ax.boxplot(data) f2 = plt.gcf()

14

[32]: assert f2.gca().has_data() Using the plot you just generated, answer the following questions, storing the answer in the variable at the front of each question. (For example, if you thought the answer to the first question was A, you would include longest_tail = 'A' in the cell below.) Remember that higher the score, more difficult it is to read the text. longest_tail : Which category has the most messages that are difficult to comprehend using the Dale-Chall criteria?

•

A) politics

•

B) recreation

•

C) computer

•

D) religion

•

E) science

•

F) misc

true_false : The typical readability score across this dataset is between 4 and 5. - A) True - B) False [33]: # YOUR CODE HERE longest_tail = 'B' true_false = 'A' [34]: assert longest_tail in ['A','B','C','D','E', 'F'] [35]: assert true_false in ['A','B']

3.4

# Part 2: category Classification (4.1 points)

In this set of questions, you’ll prepare the data and then generate three different models for classification. All of these models will use a Decision Tree classifier to predict category as the task’s outcome; however the predictors used in each will differ: - Task #1: Subject Only - Task #2: Message Only - Task #3: Subject + Message combined 3.4.1

2a) Shuffle Data

An important part of the data science process is to make sure that your dataset is not skewed or biased. Let us look at the various category values in news_df for example.

Run the two cells below to look at the category breakdown.

[36]: # breakdwon in firsst 15000 values news_df['category'][:15000].value_counts() [36]: computer recreational science

4390 3956 2754 15

politics 2079 religion 1439 misc 382 Name: category, dtype: int64 [37]: # breakdown from 15000 to the end of df news_df['category'][15000:].value_counts() [37]: science 1180 religion 973 misc 578 politics 546 computer 453 Name: category, dtype: int64 As you can see in the above results, there is no data point for recreational category (numerical category 2) after 15000 datapoints in the current data. If we were to split the data into training and test sets naively, we might end up with 0 datapoints for recreational category in the test set and thus our analysis would remain biased.

Randomly shuffle the dataframe using .sample() method in pandas such that the resulting dataset has the same number of rows as news_df but randomly shuffled. Store this back into news_df.

Also, provide random_state=random_state to use the value defined in random_state at the start of the assignment as an argument to the method.

And, be sure to reset the indices in the resulting news_df; be sure to do this such that there is no index column in the resulting dataframe (see the drop parameter in the reset_index() method.

[38]: # YOUR CODE HERE news_df = news_df.sample(frac=1,random_state=random_state). ,→reset_index(drop=True) news_df.head() [38]: 0 1 2 3 4

category computer computer recreational recreational recreational

subject Re: Is 980-1MB/sec. HD transfer slow for 486DX… Re: Computer Engr vs. Computer Science Re: '93 Wing Ding? stats Re: ESPN

0 1 2 3 4

message ASL PDW Hi. I recently switched my old Amiga 500 with … 10.826087 0.578313 A professor of mine once said 'The difference … 15.145833 0.352132 Wing ding XV will take place in Louisville, Ke… 6.500000 0.750000 Im trying to find a site that has UPDATED(dail… 10.333333 0.451613 I live in Pittsburgh (yes i'm a pens fan) wher… 11.666667 0.557143 readability

16

\

\

0 1 2 3 4

4.264790 4.443335 4.077325 4.220343 4.303140

[39]: assert news_df.shape == (18730, 6) # if you fail the following assert # go back and be sure you followed instructions assert news_df.loc[0, 'subject'] == 'Re: Is 980-1MB/sec. HD transfer slow for␣ ,→486DX-50 EISA with UltraStor24F' 3.4.2

2b) Numerical Labels

We need to convert the category labels to numerical labels. Use the following mapping to convert the values in category into numerical labels. Store the numeric values in a new column called 'category_num' • politics -> 1 • recreational -> 2 • computer -> 3 • religion -> 4 • science -> 5 • misc -> 6 Hint: you can use the .replace() method from pandas.

[40]: # YOUR CODE HERE category_num = [1 if item=='politics' else 2 if item=='recreational' else 3 if␣ ,→item=='computer' else 4 if item=='religion' else 5 if item=='science' else 6␣ ,→for item in news_df['category']] news_df['category_num'] = category_num [41]: assert set(np.unique(news_df['category_num'])) == {1,2,3,4,5,6} assert sum(news_df['category_num'] == 2) == 3956 3.4.3

2c) Convert Text data into vector

We will now create a CountVectorizer object to transform the text data into vectors with numerical values.

To do so, we will initialize a CountVectorizer object, and store this object in vectorizer.

We need to pass 4 arguments to initialize a CountVectorizer: 1. analyzer: 'word' Specify to analyze data at the word-level. 2. max_features: 2000 Set a max number of unique words. 3. tokenizer: word_tokenize Set to tokenize the text data by using the word_tokenizer from NLTK . 4. stop_words: stopwords.words('english') Set to remove all stopwords in English. We do this since they generally don’t provide useful discriminative information.

17

[42]: # YOUR CODE HERE vectorizer = CountVectorizer(tokenizer=word_tokenize, stop_words=stopwords.words('english'), analyzer='word', max_features=2000) [43]: assert vectorizer.analyzer == 'word' assert vectorizer.max_features == 2000 assert vectorizer.tokenizer == word_tokenize assert vectorizer.stop_words == stopwords.words('english') assert hasattr(vectorizer, "fit_transform") 3.4.4

2d) Vectorize Subject and Reviews

Transform the subject and message columns of news_df into vectors using the vectorizer we created above.

Save the transformed data into a variable called subject_X and message_X respectively.

Note: X in a variable name will indicate that we’re referring to a predictor variable.

Hint: You might need to cast the datatype of subject_X and message_X to numpy.ndarray by using .toarray() Note: This may take a minute or two to run, given the size of the dataset.

[44]: # YOUR CODE HERE subject_X = vectorizer.fit_transform(news_df['subject']).toarray() message_X = vectorizer.fit_transform(news_df['message']).toarray() [45]: assert type(subject_X) == np.ndarray assert type(message_X) == np.ndarray 3.4.5

2e) Store outcome as array

Store the outcome variable (category_num) as an np.array() object in its own variable category_Y. Note: Y in a variable name indicates that we’re referring to an outcome variable.

[46]: # YOUR CODE HERE category_Y = np.array(category_num) [47]: assert category_Y.shape == (18730,) assert set(np.unique(category_Y)) == {1,2,3,4,5,6} 3.4.6

2f) Split Data

Define the size of training data and test data using a 80:20 split, such that num_training determines how many samples are 80% of the data (should be an integer) and num_testing the remaining 20% of samples.

18

[48]: # YOUR CODE HERE num_training = int(len(news_df)*0.8) num_testing = len(news_df) - num_training [49]: assert isinstance(num_training, int) assert isinstance(num_testing, int) Having defined the number of observations in our training and testing datasets, now let’s actually carry out the split on the data!

Use the num_training value that you just derived to: 1. Split subject_X into 2 subsets as follows: • subject_train_X -> first 80% of data points • subject_test_X -> last 20% of data points 2. Split message_X into 2 subsets as follows: • message_train_X -> first 80% of data points • message_test_X -> last 20% of data points 3. Split category_Y into 2 subsets: • category_train_Y -> first 80% of data points • category_test_Y.-> last 20% of data points [50]: # YOUR CODE HERE subject_train_X = subject_X[:num_training] subject_test_X = subject_X[num_training:] message_train_X = message_X[:num_training] message_test_X = message_X[num_training:] category_train_Y = category_Y[:num_training] category_test_Y = category_Y[num_training:] [51]: assert len(subject_train_X)==num_training assert len(subject_test_X)==num_testing [52]: assert len(message_train_X)==num_training assert len(message_test_X)==num_testing [53]: assert len(category_train_Y)==num_training assert len(category_test_Y)==num_testing 3.4.7

2g) Decision Tree Classification

Define a function train_DT() which takes as input X and Y and returns a Decision Tree model trained on the given X,Y.

19

Inputs: - X: np.ndarray, training samples (predictors), - Y: np.ndarray, training labels (outcomes), Output: a trained classifier clf Hint: There are 2 steps involved in this function: - Initializing an Decision Tree classifier: clf = DecisionTreeClassifier(...) - Training the classifier: clf.fit(X, Y) [54]: # YOUR CODE HERE def train_DT(X, Y): clf = DecisionTreeClassifier() clf.fit(X, Y) return clf [55]: assert callable(train_DT) 3.4.8

2h) Classification #1: Using Only Subject

Let’s try to classify the email conversation using only the subject field of the dataframe only.

Using the function train_DT(), train a decision tree classifier on subject_train_X (as your predictor) and category_train_Y (as your outcome) and save the model as subject_clf.

[56]: # YOUR CODE HERE subject_clf = train_DT(subject_train_X, category_train_Y) [57]: assert isinstance(subject_clf, DecisionTreeClassifier) assert hasattr(subject_clf, "predict") Now we will use the function classification_report to print out the performance of the classifier on the training set: [58]: # Your classifier should observe an accuracy of around 96%. subject_predicted_train_Y = subject_clf.predict(subject_train_X) print(classification_report(category_train_Y,subject_predicted_train_Y)) precision

recall

f1-score

support

1 2 3 4 5 6

0.99 0.97 0.97 0.98 0.85 0.98

0.95 0.93 0.95 0.93 0.98 0.93

0.97 0.95 0.96 0.95 0.91 0.96

2103 3163 3877 1951 3118 772

accuracy macro avg weighted avg

0.96 0.95

0.94 0.95

0.95 0.95 0.95

14984 14984 14984

And now, let’s check the performance of the trained classifier on the test set:

20

[59]: # You should observe an accuracy of around 79%. subject_predicted_test_Y = subject_clf.predict(subject_test_X) print(classification_report(category_test_Y, subject_predicted_test_Y)) precision

recall

f1-score

support

1 2 3 4 5 6

0.82 0.80 0.83 0.88 0.67 0.72

0.80 0.78 0.72 0.85 0.82 0.63

0.81 0.79 0.77 0.86 0.74 0.67

522 793 966 461 816 188

accuracy macro avg weighted avg

0.79 0.79

0.77 0.78

0.78 0.77 0.78

3746 3746 3746

[60]: assert subject_predicted_train_Y.shape == (14984,) assert subject_predicted_test_Y.shape == (3746,) precision, recall, _, _ =␣ ,→precision_recall_fscore_support(category_train_Y,subject_predicted_train_Y) assert np.isclose(precision[0], 0.99, 0.02) assert np.isclose(precision[1], 0.97, 0.02) 3.4.9

2i) Classification #2: Using Only Message

Now, rather than using only the subject line, let’s try to classify the email conversation using the message field.

Using the function train_DT, train a decision tree on message_train_X and category_train_Y and save the model as message_clf.

[61]: # YOUR CODE HERE message_clf = train_DT(message_train_X, category_train_Y) [62]: assert isinstance(message_clf, DecisionTreeClassifier) assert hasattr(message_clf, "predict") We’ll now check performance in our training data for this classification model: [63]: # You should observe an accuracy of around 100%. message_predicted_train_Y = message_clf.predict(message_train_X) print(classification_report(category_train_Y, message_predicted_train_Y)) precision

recall

f1-score

21

support

1 2 3 4 5 6

1.00 1.00 1.00 1.00 1.00 1.00

1.00 1.00 1.00 1.00 1.00 0.99

1.00 1.00 1.00 1.00 1.00 1.00

2103 3163 3877 1951 3118 772

accuracy macro avg weighted avg

1.00 1.00

1.00 1.00

1.00 1.00 1.00

14984 14984 14984

And again, we’ll check performance in our test data, for this model [64]: # You should observe an accuracy of around 67%. message_predicted_test_Y = message_clf.predict(message_test_X) print(classification_report(category_test_Y, message_predicted_test_Y)) precision

recall

f1-score

support

1 2 3 4 5 6

0.64 0.74 0.70 0.68 0.57 0.65

0.61 0.74 0.71 0.68 0.60 0.58

0.63 0.74 0.71 0.68 0.58 0.61

522 793 966 461 816 188

accuracy macro avg weighted avg

0.66 0.67

0.65 0.67

0.67 0.66 0.67

3746 3746 3746

[65]: assert message_predicted_train_Y.shape == (14984,) assert message_predicted_test_Y.shape == (3746,) precision, recall, _, _ = precision_recall_fscore_support(category_train_Y,␣ ,→message_predicted_train_Y) assert np.isclose(precision[0], 1.00, 0.02) assert np.isclose(precision[1], 1.00, 0.02) 3.4.10

2j) Classification #3: Combine subject and message

Create a new column called combined in news_df which stores a combination of the subject and message fields.

For example, let’s say that the subject for a data point is “Fire in California” and the message is “Fire started because of a gender reveal party”, then your combined column should contain “Fire in California Fire started because of a gender reveal party”.

22

Note: The subject and the message should be split by a space.

[66]: # YOUR CODE HERE combined = [] for sub, mess in zip(news_df['subject'], news_df['message']): combined.append(sub + ' ' + mess) news_df['combined'] = combined [67]: assert news_df.shape == (18730, 8) assert list(news_df.columns)[-1] == 'combined' [68]: # look at updated dataframe news_df.head() [68]: 0 1 2 3 4

category computer computer recreational recreational recreational

0 1 2 3 4

message ASL PDW Hi. I recently switched my old Amiga 500 with … 10.826087 0.578313 A professor of mine once said 'The difference … 15.145833 0.352132 Wing ding XV will take place in Louisville, Ke… 6.500000 0.750000 Im trying to find a site that has UPDATED(dail… 10.333333 0.451613 I live in Pittsburgh (yes i'm a pens fan) wher… 11.666667 0.557143

0 1 2 3 4

readability 4.264790 4.443335 4.077325 4.220343 4.303140

0 1 2 3 4

combined Re: Is 980-1MB/sec. HD transfer slow for 486DX… Re: Computer Engr vs. Computer Science A profe…

Re: '93 Wing Ding? Wing ding XV will take plac… stats Im trying to find a site that has UPDATE… Re: ESPN I live in Pittsburgh (yes i'm a pens …

3.4.11

subject Re: Is 980-1MB/sec. HD transfer slow for 486DX… Re: Computer Engr vs. Computer Science Re: '93 Wing Ding? stats Re: ESPN

category_num 3 3 2 2 2

\

\

\

2k) Vectorize combined data

Note: the rest of the steps for this classification task involve the same process as the previous two; however, our predictor data now changes.

23

Vectorize the combined column using the vectorizer we previously defined and store it in the variable combined_X. Note: This may take a minute or two to run, given the size of the dataset.

[69]: # YOUR CODE HERE combined_X = vectorizer.fit_transform(news_df['combined']).toarray() [70]: assert type(combined_X) == np.ndarray assert combined_X.shape == (18730, 2000) 3.4.12

2l) 80:20 split

Now let’s split the data into train and test subsets.

Split combined_X into 2 subsets as follows: - combined_train_X -> first 80% of data points combined_test_X -> last 20% of data points Again, use the num_training value that you derived previously.

[71]: # YOUR CODE HERE combined_train_X = combined_X[:num_training] combined_test_X = combined_X[num_training:] [72]: assert len(combined_train_X)==num_training assert len(combined_test_X)==num_testing 3.4.13

2m) Model Training

Using the function train_DT, train a decision tree on combined_train_X and category_train_Y, and save the model as combined_clf.

[73]: # YOUR CODE HERE combined_clf = train_DT(combined_train_X, category_train_Y) [74]: assert isinstance(combined_clf, DecisionTreeClassifier) assert hasattr(combined_clf, "predict") For this third classification model, we’ll again check performance in our training dataset…

[75]: # You should observe an accuracy of around 100%. combined_predicted_train_Y = combined_clf.predict(combined_train_X) print(classification_report(category_train_Y, combined_predicted_train_Y))

1 2 3 4

precision

recall

f1-score

support

1.00 1.00 1.00 1.00

1.00 1.00 1.00 1.00

1.00 1.00 1.00 1.00

2103 3163 3877 1951

24

5 6 accuracy macro avg weighted avg

1.00 1.00

1.00 1.00

1.00 1.00

1.00 1.00

3118 772

1.00 1.00

1.00 1.00 1.00

14984 14984 14984

…and in our test dataset: [76]: # You should observe an accuracy of around 71%. combined_predicted_test_Y = combined_clf.predict(combined_test_X) print(classification_report(category_test_Y, combined_predicted_test_Y)) precision

recall

f1-score

support

1 2 3 4 5 6

0.72 0.75 0.76 0.75 0.63 0.62

0.67 0.75 0.74 0.74 0.67 0.63

0.69 0.75 0.75 0.74 0.65 0.62

522 793 966 461 816 188

accuracy macro avg weighted avg

0.70 0.71

0.70 0.71

0.71 0.70 0.71

3746 3746 3746

[77]: assert combined_predicted_train_Y.shape == (14984,) assert combined_predicted_test_Y.shape == (3746,) precision, recall, _, _ = precision_recall_fscore_support(category_train_Y,␣ ,→combined_predicted_train_Y) assert np.isclose(precision[0], 1.00, 0.02) assert np.isclose(precision[1], 1.00, 0.02) 3.4.14

2n) Model Choice

After having run these three models, your boss asks you which model you should use for category classification of emails at your company. How would you advise them?

•

A) #1: Subject Only

•

B) #2: Message Only

•

C) #3: Subject + Message Combined

Store your choice in the variable model_choice. (For example, if you think the Subject Only model is best, your answer below would be model_choice = 'A')

25

[78]: # YOUR CODE HERE model_choice = 'B' [79]: assert model_choice in ['A', 'B', 'C'] Briefly (1-3 sentences) explain how you came to your decision above in the cell below.

I would advise to use “Message Only” because according to our model, “Message Only” has 100% accuracy.

3.5

# Part 3: Sarcasm Detection (1.9 points)

Now that you’ve got the general process for working with text data down, in this final set of questions, let’s approach a somewhat more difficult and convoluted problem: detecting whether a text contains sarcasm or not.

Here, the approach will be similar to what you did in Part 2, but will differ in the following ways: 1. We will use a new set of data 2. Our outcome will be binary: sarcasm (1) or not (0) 3. We will use TF-IDF to vectorize our text data 4. We will use an SVM model for classification 3.5.1

3a) Import data

Import "data/political_subreddit_sarcasm.csv" into the sarcasm_df dataframe using pd.read_csv function. Set the column names as ‘label’ and ‘comment’, in that order.

[80]: # YOUR CODE HERE sarcasm_df = pd.read_csv("data/political_subreddit_sarcasm.csv",␣ ,→names=['label','comment']) sarcasm_df.head() [80]: 0 1 2 3 4

label 0 0 0 0 0

comment Ive always said youre delusional if you think …

I think they are well aware, which is why they…

Time to warm up a new phrase: "Thanks Hillary" Who are you mad at?

Hawaii wasn't American territory yet because i…

[81]: assert sarcasm_df.shape == (8000, 2) assert list(sarcasm_df.columns) == ['label', 'comment'] 3.5.2

3b) TFIDF Vectorizer

Rather than using a CountVectorizer, here we will create a TfidfVectorizer object to transform the text data into vectors with TF-IDF. To do so, we will initialize a TfidfVectorizer object, tfidf.

We need to pass 4 arguments into TfidfVectorizer to initialize tfidf: 1. sublinear_tf: True Set to apply TF scaling. 2. analyzer: 'word' Set to analyze the data at the word-level 3.

26

max_features: 2000 Set the max number of unique words 4. tokenizer: word_tokenize Set to tokenize the text data by using the word_tokenizer from NLTK [82]: # YOUR CODE HERE tfidf = TfidfVectorizer(sublinear_tf=True, analyzer='word', max_features=2000, tokenizer=word_tokenize) [83]: assert tfidf.analyzer == 'word' assert tfidf.max_features == 2000 assert tfidf.tokenizer == word_tokenize assert tfidf.stop_words == None assert hasattr(vectorizer, "fit_transform") 3.5.3

3c) Transform

As above, transform the comment column of sarcasm_df and store it in a variable named sarcasm_X. Store the label column into a variable named sarcasm_Y. Make sure that both the variables are of data type numpy.ndarray.

Hint: you cna use the .toarray() and .to_numpy() methods.

[84]: # YOUR CODE HERE sarcasm_X = tfidf.fit_transform(sarcasm_df['comment']).toarray() sarcasm_Y = sarcasm_df['label'].to_numpy() [85]: assert sarcasm_X.shape == (8000, 2000) assert sarcasm_Y.shape == (8000,) 3.5.4

3d) Split into train and test

We’ll do this in a different way compared to previous one. In the last section we first shuffled the data then divided it manually into separate train and test sets. Now, we’ll instead use sklearn’s train_test_split function instead. Store train data into X_train and train labels into Y_train.

Similarly, store test data into X_test and test labels into Y_test.

We will use the following arguments for this task: - test_size: 0.2 - random_state: random_state (variable defined at the start) [86]: # YOUR CODE HERE X_train, X_test, Y_train, Y_test = train_test_split(sarcasm_X, sarcasm_Y,␣ ,→test_size=0.2, random_state=random_state) [87]: assert X_train.shape == (6400, 2000) assert X_test.shape == (1600, 2000) assert Y_train.shape == (6400,)

27

assert Y_test.shape == (1600,) 3.5.5

3e) SVM

Now, rather than using a Decision Tree Classifier, we’ll use a Support Vector Machine model for our classification task.

Define a function called train_SVM that initializes an SVM classifier and trains it.

Inputs: - X: np.ndarray, training samples, - Y: np.ndarray, training labels, - kernel: string, set the default value of kernel as “linear” Output: a trained classifier clf Hint: There are 2 steps involved in this function: - Initializing an SVM classifier: clf = SVC(...)

- Training the classifier: clf.fit(X, Y) [89]: # YOUR CODE HERE def train_SVM (X, Y, kernel = 'linear'): clf = SVC(kernel = 'linear') clf.fit(X, Y) return clf [90]: assert callable(train_SVM) 3.5.6

3f) Train Model

Using your train_SVM function, train an SVM classifier with the default linear kernel on the predictors X_train and outcome labels Y_train. Store this model in the variable model_sarcasm Note: It may take a minute or two to train your model.

[93]: # YOUR CODE HERE model_sarcasm = train_SVM(X_train, Y_train) [94]: assert isinstance(model_sarcasm, SVC) assert hasattr(model_sarcasm, "predict") Assess accuracy in your training dataset: Note: Again, this may take a minute [95]: # accuracy should be ~80% pred_train = model_sarcasm.predict(X_train) print(classification_report(Y_train, pred_train))

0 1

precision

recall

f1-score

support

0.78 0.83

0.84 0.77

0.81 0.80

3213 3187

28

accuracy macro avg weighted avg

0.81 0.80

0.80 0.80

0.80 0.80 0.80

6400 6400 6400

Now, see how your model is performing in your testing dataset: [96]: # accuracy should be ~70% pred_test = model_sarcasm.predict(X_test) print(classification_report(Y_test, pred_test)) precision

recall

f1-score

support

0 1

0.68 0.72

0.73 0.66

0.70 0.69

787 813

accuracy macro avg weighted avg

0.70 0.70

0.70 0.70

0.70 0.70 0.70

1600 1600 1600

[97]: assert Y_train.shape == (6400,) assert Y_test.shape == (1600,) precision, recall, _, _ = precision_recall_fscore_support(Y_train, pred_train) assert np.isclose(precision[0], 0.78, 0.02) assert np.isclose(precision[1], 0.83, 0.02) 3.5.7

3g) Decision Time

In a few sentences, explain whether you think this model is performing the task of sarcasm detection well, and defend your position. If you think it is performing well, explain how you came to that conclusion. If you think it is not performing well, explain how you could improve the model.

YOUR ANSWER HERE

4

Complete!

Good work! Have a look back over your answers, and also make sure to Restart & Run All from the kernel menu to double check that everything is working properly. While you can typically use the ‘Validate’ button above, which runs your notebook from top to bottom and checks to ensure all assert statements pass silently, this may fail on this assignment as the code takes too long to run. Use Restart & Run All instead. When you are ready, submit on Datahub!

Note that the final validation is for your reassurance and is not a required step. You can submit without validating. You can also submit without passing all asserts (for partial credit on the assignment). We grade whatever is submitted on Datahub. We will grade your most recent submission.

29