EECS 445 ML Introduction to Machine Learning-Homework 3 Plus Solutions | University of Michigan

EECS 445, Winter 2020 – Homework 3, Due: Tues. 4/7 at 11:59pm

1

UNIVERSITY OF MICHIGAN Department of Electrical Engineering and Computer Science EECS 445 — Introduction to Machine Learning Winter 2020 Homework 3, Due: Tues. 4/7 at 11:59pm Submission: Please upload your completed assignment by 11:59pm Tuesday, April 7th to Gradescope.

1

Neural Networks [8 pts]

Sahas was inspired by Noa’s findings in Project 2. He is now interested in determining the price of a dish is based on two inputs Size and Tastiness. Sahas eats ten different dishes and records their mass in kilograms and tastiness on a continuous scale of 1-10. Then he records the price of the dish.

The following is the neural network he sets up to perform this regression task. xi ’s x1

(2)

(1) wji

hj ’s (2)

(3)

(2) wkj

h1

hk ’s (3)

h1

x2

h2

(2)

h2

+1

+1

+1

(3) w1j

zk ’s z1

(3)

We will use the Sigmoid activation function on each hidden neuron and use squared loss to evaluate our model. All of the current weights are given below. The tables are organized such that the row designates which input the weight is from and the column indicates which output the weight is to. For example weight (1) (2) w21 is found in the x1 row and the h2 column.

(2)

x1 x2 +1

h1 0.5 0.1 0.3

(2)

h2 0.1 0.2 0.1

Table 1: Layer 1 Weights

EECS 445, Winter 2020 – Homework 3, Due: Tues. 4/7 at 11:59pm (3)

(2)

h1 (2) h2 +1

h1 0.3 0.2 0.2

2

(3)

h2 0.1 0.1 -0.1

Table 2: Layer 2 Weights a (l)

Note: wji is the weight for lth fully connected layer, connecting unit j with input i.

We are currently in the process of training our neural network and randomly sample the data point {x̄ = [1, 5]t , y = 4} (a) [2 pt] What is the output of our neural network given this datapoint?

(1)

(2)

Solution: h1 = act(0.5(1) + 0.1(5) + 0.3) = 0.785 h2 = act(0.1(1) + 0.2(5) + 0.1) = 0.768 (2) (2) h1 = act(0.3(0.785) + 0.2(0.768) + 0.2) = 0.6431 h2 = act(0.1(0.785) + 0.1(0.768) 0.1) = 0.514 z1 = 1(0.6431) + 3(0.514) + 1 = 3.1851 (b) [1 pt] Next, compute the loss between the expected output and our prediction.

Solution: 0.5(4

3.1851)2 = 0.332

(c) [5 pts] Since we are still training this network, we now need update. Compute

@L @L @L (1) , (2) , (3) @w12 @w12 @w12

@L (l) for all i, j and l to perform an SGD @wij

EECS 445, Winter 2020 – Homework 3, Due: Tues. 4/7 at 11:59pm

3

Solution: We will roll out the gradients with chain rule and substitute in the corresponding expressions

Z1

y

(3) (3)

(3)

(3)

(2)

(2)

(2) (2)

EECS 445, Winter 2020 – Homework 3, Due: Tues. 4/7 at 11:59pm

2

4

Food Clusters [15 pts]

Thanks to all of your hard work in Project 2, Noa has a way to easily sort through all of her food options.

Unfortunately, in her excitement to finally choose, Noa misplaced the labels of her images so you have promised to create her an unsupervised algorithm for categorizing the food.

Like the autoencoder in Project 2, we will attempt to identify structure in the data without training on labels.

You will be implementing the k-means clustering algorithm as well as a variant of the algorithm, k-means++, which has the additional property of encouraging good cluster centroid initializations. These are simple yet powerful unsupervised methods for grouping similar examples within a dataset. You will write your code in clustering classes.py and clustering.py. (a) [1 pt] Why are poor centroid initializations often a problem?

Solution: Poor centroid initialization could cause the model to converge to an arbitrarily bad local minimum. Specifically for k-means, each iteration makes a greedy update with no regard for global optimality. (b) [1 pt] Implement the function Point.distance() in the file clustering classes.py. This function takes as arguments two Point objects and returns the Euclidean distance between the points.

The feature data for each point is stored in the features member variable.

Solution: See solution code Point.distance(). (c) [1 pts] Implement the function Cluster.get centroid() in the file clustering classes.py.

This function returns the centroid of the current Cluster object as a Point object. As in lecture, this is calculated by finding the mean point of the cluster. You may denote a point as having no label by excluding the label argument. The default value specified in the Point constructor will initialize this value to None.

Solution: See solution code Cluster.get centroid(). (d) [1 pt] Implement the function ClusterSet.get centriods() in the file clustering classes.py.

This function returns the centroids of a ClusterSet object as a list of Point objects.

Solution: See solution code ClusterSet.get centroids(). (e) [1 pt] Implement the function random init() in the file clustering.py according to the function specification. This function returns the cluster centroid initialization for the regular k-means algorithm.

EECS 445, Winter 2020 – Homework 3, Due: Tues. 4/7 at 11:59pm

5

Solution: See solution code random init(). (f) [2 pts] Implement the function k means pp init() in the file clustering.py according to the function specification. This function returns the cluster centroid initialization for the k-means++ algorithm. Pseudocode for this initialization can be found below.

1) Randomly select one point as the first centroid.

2) For each point x, calculate the distance from x to the nearest centroid that has already been selected. Denote this distance Dx .

3) Select the next centroid from the list of points with the probability of selecting point x being proportional to Dx2 . Note that for the sake of implementing this in Python it may be necessary to normalize your calculated distances.

4) Continue the previous 2 steps until k points have been selected.

Solution: See solution code k means pp init(). (g) [4 pts] Implement the function k means() in the file clustering.py according to the function specification. Note that you must construct your own implementation; the use of library functions that trivialize this problem (e.g., sklearn implementations) is prohibited.

Solution: See solution code k means(). (h) [2 pts] You will now apply these algorithms to the food dataset. For the sake of reducing computation time, we will consider only the first 500 images from the dataset. We have provided a preprocessed version of this dataset and the code for loading it in. Within the main function in clustering.py, use your implementation of the k means function to determine the best k-value in the range [1, 10] for each k-means initialization method. Note that there is not a single correct answer for these values. You should evaluate clustering performance using cluster purity, which is implemented for you in ClusterSet.get score(). The purity metric is the ratio of points within a cluster with the most common label to all points in the cluster. Both of these algorithms require a random initialization, so performance may vary from run to run. To mitigate this effect, run your algorithm 10 times for each candidate k value and average over the purity scores. A correct implementation may take up to ten minutes to run.

Plot the average performance vs. k for each clustering algorithm on one graph by implementing the function plot performance(). Include this graph in your write-up. Don’t forget to label your axes and include a legend. Your plot should look similar to Figure 1:

EECS 445, Winter 2020 – Homework 3, Due: Tues. 4/7 at 11:59pm

6

Figure 1: Purity of each clustering method as a function of k

Solution: See solution code plot performance(). See Figure 1 for plot.

We look for an ‘elbow’ in the graph. A reasonable answer for the graph above is k = 3 — although one could claim that any k = 3, 4, 5, 6 is the best for k. Your specific answer must reflect the elbow in your plot for points to be awarded. Note that purity will approach 1 as k approaches the number of points since we can simply learn 1 cluster per point. In general the k-means++ should be above the k-means line. The performance of the two algorithms, however, is very close, and the lines may cross. (i) [2 pt] Using the best k-values you found for each algorithm, compare the performance of both initialization techniques. Report the average, minimum, and maximum cluster purity of the resulting clusters.

Briefly discuss which method works best and why. Compare your purity score to the score we would expect from a random assignment of points to the k clusters (assume that the class distribution is uniform).

Solution: See table below for best k value choices and the corresponding purity values. Your best k could be different, but it should in general be around 3 to 6.

Algorithm k-means k-means++

best k 3 3

Average 0.294 0.2935

Max 0.3125 0.3175

Min 0.28 0.2675

Since we utilize 5 classes of food images, the purity score of random cluster assignments is 0.2.

The above clustering results are better than random.

EECS 445, Winter 2020 – Homework 3, Due: Tues. 4/7 at 11:59pm

3

7

Spectral Clustering [5 pts]

Here you will run through the spectral clustering algorithm by hand. Figure 2 shows the graph that we will be working with.

1 2

2

1 4

3 4

1

1 3

1 3

4 4 5

3 1 5

5

Figure 2: Spectral Clustering graph (a) [1 pt] Compute our similarity matrix W , degree matrix D, and the corresponding graph Laplacian L based on the graph in Figure 2.

Solution: We first construct our similarity matrix W and the corresponding degree matrix D. 2 3 2 29 3 1 34 13 0 13 0 0 0 12 0 6 3 1 1 1 07 60 5 0 0 07 2 6 41 1 4 24 1 7 6 7 7 6 0 0 31 0 0 7 1 W =6 D = 12 6 3 41 4 5 5 7 6 7 40 5 4 0 0 0 23 0 5 2 5 1 0 10 1 1 0 0 0 0 23 3 0 5 0 1 15 From these matrices, we compute our graph Laplacian.

2 17

12 6 3 6 41 L=6 6 3

4 0

1 3

3 2

3 4 1 4 1 2

0

1 3 1 4 19 12 4 5 1 5

0 1 2 4 5 13 10

0

13 3

0 7 7 17 57 0 5

8 15

(b) [1 pt] Using the scipy.linalg.eigh method to solve for the eigenvectors corresponding to the k = 2 lowest eigenvalues. List out the eigenvalues and the corresponding eigenvectors (round to four decimal places). Refer to the documentation for more details: https://docs.scipy.org/doc/scipy/ reference/generated/scipy.linalg.eigh.html.

Solution:

EECS 445, Winter 2020 – Homework 3, Due: Tues. 4/7 at 11:59pm 1. Eigenvalue: 5.6976 ⇥ 10 17 ⇡ 0 Eigenvector: [ 0.4472, 0.4472,

0.4472,

2. Eigenvalue: 5.731110 1 ⇡ 0.5731 Eigenvector: [0.0246, 0.2568, 0.2101,

0.4472,

8

0.4472]T

0.4079, 0.8502]T

(c) [1 pt] Based on the eigenvalues and the eigenvectors computed in part b, what is our cluster assignment?

(Hint: as in lecture 16, plot the rows of the eigenvectors.)

Solution: Following the hint, we can plot the rows of the eigenvectors and obtain the plot in Figure 3.

Figure 3: Plotting rows of the eigenvectors computed in part b From this plot, it is obvious that the cluster assignments should be points 1, 2, 3, 4 in one cluster and point 5 in the other cluster. (d) [1 pt] Complete problem b again but with k = 3. List out the eigenvalues and the corresponding eigenvectors (round to four decimal places).

Solution: 1. Eigenvalue: 5.6976 ⇥ 10 17 ⇡ 0 Eigenvector: [ 0.4472, 0.4472,

0.4472,

2. Eigenvalue: 5.731110 1 ⇡ 0.5731 Eigenvector: [0.0246, 0.2568, 0.2101, 3. Eigenvalue: 1.1935 Eigenvector: [0.6281, 0.4326,

0.3380,

0.4472,

0.4472]T

0.4079, 0.8502]T 0.5079,

0.2147]T

EECS 445, Winter 2020 – Homework 3, Due: Tues. 4/7 at 11:59pm

9

(e) [1 pt] What are our cluster assignments now?

Solution: Following the hint in part c, we can plot the rows of the eigenvectors and obtain the plot in Figure 4 (we plot only the coordinates from the second and third eigenvectors since the coordinates from the first eigenvector are all the same).

Figure 4: Plotting rows of the eigenvectors computed in part d From this plot, it is obvious that the cluster assignments should be points 1 and 2 in one cluster, points 3 and 4 in one cluster, and point 5 in the other cluster.

3.1

Code Prerequisites • If you are currently using scipy version 1.2.1 (or later), you should be sufficiently up-to-date. You can verify this by running conda list scipy • If not, please run conda update scipy to update

EECS 445, Winter 2020 – Homework 3, Due: Tues. 4/7 at 11:59pm

4

10

Collaborative Filtering [8 points]

In this question, we will approach collaborative filtering via the k-Nearest Neighbors algorithm and matrix factorization using UV decomposition. Specifically, we will be predicting some of the Winter 2020 EECS445 IAs’ ratings (preferences) for different sports. The following table shows each IAs’ ratings on each of the chosen sports.

Sahas Karl Noa Alice John

Badminton 5 2 5 None 1

Golf 2 None 1 4 3

Soccer 5 5 1 2 2

Bowling None 4 2 4 1

Football 3 5 5 3 None

A rating of 1 means the IA does not enjoy the sport at all; a rating of 5 means the IA enjoys the sport very much. None means the rating is unknown.

4.1 k-Nearest Neighbors [4 pts] We store all ratings in a matrix Y , such that Yai denotes IA a’s rating of sport i. In order to measure the similarity between two IAs a and b, we use Pearson’s Correlation Coefficient as below. Here Ỹb:a represents the average rating of IA b over all of the commonly rated sports with IA a: sim(a, b) = qP

P

j2R(a,b) (Yaj

j2R(a,b) (Yaj

Ỹa:b )

Ỹa:b )(Ybj 2P

Ỹb:a )

j2R(a,b) (Ybj

Ỹb:a )

2

(1)

Recall that in order to make a prediction on IA a’s rating on sport i, we will turn to the k other most similar IAs. Here Ȳb represents the average rating of IA b over all of the sports this IA rated: Ŷai = Ȳa + P

1

X

b2kN N (a,i) |sim(a, b)| b2kN N (a,i)

sim(a, b)(Ybi

Ȳb )

(2)

(a) [2 pts] We will use a 2-NN algorithm to predict Sahas’s rating on Bowling. Find the two IAs who are the most similar to Sahas using the absolute value of Pearson’s Correlation Coefficient as the similarity metric.

Solution: We will plug and chug the Pearson’s Correlation Coefficient as the similarity metric for Sahas and every IA

EECS 445, Winter 2020 – Homework 3, Due: Tues. 4/7 at 11:59pm

11

|-0.5| = 0.5

0.1924

|-0.982| = 0.982

(b) [2 pts] Use the IAs from the answer of part (a) to make a prediction on Sahas’s rating on Bowling. (If necessary, round down the predicted rating to the nearest integer).

Solution: Using equation 2 from above, we can plug in Alice/Sahas and John/Sahas similarity scores to follow the 2-NN Algorithm

EECS 445, Winter 2020 – Homework 3, Due: Tues. 4/7 at 11:59pm

12

0.982

3.797

3

4.2

Matrix Factorization via UV Decomposition [4 pts]

Now we will approach collaborative filtering as matrix factorization using UV decomposition. Any n ⇥ m matrix X can be approximated by a lower rank matrix U V T where U is n ⇥ d and V is m ⇥ d and d  min{m, n}. In essence, we now have turned matrix factorization into a regression problem as below:

min U,V

X

i,j:Yij 6=None

(Ŷi,j

Yi,j )2 +

n X 1 i=1

2

kū(i) k2 +

m X 1 j=1

2

kv̄ (j) k2

Here we approximate Y with Ŷ = U V T such that both of their ranks are limited to one (d = 1). (a) [1 pts] After a few steps of coordinate descent, we have obtained U = [2, 1, 2, 1, 2]T and V = [1, 2, 1, 2, 1.5]T . For these values of U and V , what is the prediction for Sahas’s rating on Bowling?

Solution: u[1]v[4] = (2)(2) = 4

(b) [3 pts] Now, let’s fix V , and update u(1) . Show the updated U and the new prediction of Sahas’s rating on Bowling. Use the value of V from part (a).

EECS 445, Winter 2020 – Homework 3, Due: Tues. 4/7 at 11:59pm

13

Solution: r(1) u J = ū

(1)

2 P

X

1,i2D

(Y1,i

ū(1) · v̄ (i) )v̄ (i) + ū(i) = 0

(i) 5(1) + 2(2) + 5(1) + 3(1.5) 1,i2D Y1,i · v̄ = 1 P = 1.451 2 = 1 (i) 1,i2D v̄ 2 + 2 + (1 + 4 + 1 + 4 + 2.25)

u[1]v[4] = (1.451)(2) = 2.901

REMEMBER Submit your completed assignment to Gradescope by 11:59pm on Tuesday, April 7th.

Attach any code as an appendix.