CS 361 Computational Probability - University of Illinois_ CS 361 Probability and Statistics for Com

downloaded from

Scholarfriends

https://scholarfriends.com Downloaded by: writerforacademic@gmail.com | https://scholarfriends.com/singlePaper/314798/cs-361-computational-probability-universityof-illinois-cs-361-probability-and-statistics-for-com Distribution of this document is illegal.

CS 361 Probability and Statistics for Computer Science

CS 361 Project (December 5th, 2019) Professor: Liu

Student:

Exercise 1.

1. Algorithm: Bisection Method • Needs Value: Yes • Needs Derivative: No • 6 word description: Binary search between interval to find root 2. Algorithm: False Position Method • Needs Value: Yes • Needs Derivative: No • 6 word description: Secant and Bisection Method combined together 3. Algorithm: Secant Method • Needs Value: Needs two initial points to draw a secant line • Needs Derivative: No • 6 word description: X-intercept of secant line estimates root 4. Algorithm: Newton-Raphson Method • Needs Value: Yes • Needs Derivative: Yes • 6 word description: Root in direction of tangent line Exercise 2. No it does not make sense to use the secant method since it requires that the function can be evaluated precisely. Meaning that the function must always give the same output for the same input.

Exercise 3. The family of learning rates that I have come up with is the sequence: an = n1α where 21 < α ≤ 1. This set of possible sequences will diverge given the p-value test since the exponent is less than or equal to 1, but will diverge when each element is squared since the exponent will be greater than 1.

Exercise 8. Yes, gradient descent is essentially a root finding algorithm. The procedure of gradient descent and root finding are identical, both procedures use the same input (vector) and outputs (vector) and require the same amount of information (gradient).

Exercise 9.

Yes, SGD is a stochastic approximation algorithm. Stochastic approximation algorithms iteratively find roots or optimize functions where the data is corrupted by noise, the goal is to recover properties of the function without needing to evaluate it directly. SGD meets all these properties.

Exercise 10. We want to show that under the following definitions: 1

• f (x) = Q(x) = k1

Pk

i=1 Qi (x)

• g(x) = Qi (x) and ∇g(x) = ∇Qi (x) • z − Qi (x) − Q(x) and g(x) = f (x) + z these expressions hold true: • E[g(x)] = f (x) • E[∇g(x)] = ∇f (x) • E[z] = 0 We will begin by proving the first expression: E[g(x)] = E[Qi (x)] =

k X

Qi (x)Pi (x)

i=1

The function Pi (x) is the probability of picking the ith data point with input x. Since the sampling of data points is done uniformly meaning each data point has an equal probability of being selected, this means out of k data points each point has a probability of k1 of being picked.

This our summation becomes: k X i=1

Qi (x)

k 1 1X Qi (x) = f (x) = k k i=1

from the definitions listed above.

The next expression can be proved in the following manner: E[∇g(x)] = E[∇Qi (x)] =

k X

∇Qi (x)Pi (x)

i=1

Just as in the last proof we showed that Pi (x) = k1 , we will use the same definition because it follows the same logic as above. Our summation becomes k X i=1

∇Qi (x)

k k 1 1X 1X = ∇Qi (x) = ∇( Qi (x)) = ∇Q(x) = ∇f (x) k k i=1 k i=1

Lastly, to prove E[z] = 0 we can do the following: E[z] = E[Qi (x) − Q(x)] = E[Qi (x)] − E[Q(x)] Pk We’ve shown above that E[Qi (x)] = k1 i=1 Qi (x) so we can substitute this function as Qi (x) to get the following, and using the definition of Q(x) we can substitute that function in for E[Q(x)]: k

1X E[Qi (x)] = E[ Qi (x)] k i=1 k

E[Q(x)] = E[

1X Qi (x)] k i=1

Thus: E[z] = E[Qi (x)] − E[Q(x)] = E[

k k 1X 1X Qi (x)] − E[ Qi (x)] = 0 k i=1 k i=1

2

Exercise 11. d 1. dx g(x) = (x + z)

2. Yes, f does have a unique root. We can prove as such by taking the second derivative of f (x). So, d2 1 2 ( x )=1 dx2 2 . Since the second derivative is always positive we know that the function is always concave up, or more importantly never switches concavity (never change direction after the global optimum). When we solve for the function we know that the root is 0, which is also the global minimum, and since the function is always concave up it is the only minimum.

3.

1 1 1 1 1 1 E[g(x)] = E[ x2 + z 2 + xz − ] = E[x2 ] + E[z 2 ] − E[xz] − E[ ] 2 2 2 2 2 2 . From the above expression, since x and z are independent and expected value of z is 0 we know that E[xz] = 0. Furthermore, to find E[z 2 ] we can use the following property V ar[z] = E[z 2 ] − E[z]2 Since we know the variance is 1, and E[z] = 0, we know that E[z 2 ] = 1. Lastly, the expected value of a constant is that constant Now our expression is 1 1 1 1 1 E[x2 ] + E[z 2 ] − E[xz] − E[ ] = E[x2 ] = x2 2 2 2 2 2 . Since the variable x is not a random variable the expected value of x is just x2 .

4.

d d 1 1 g(x)] = E[ ( (x + z)2 − ] = E[(x + z)] = E[x] + E[z] dx dx 2 2 We already know that the expected value of z is 0, and that the expected value of a nond random variable is just the function the variable is used in. Thus: E[ dx g(x)] = x Which d gives the same value when taking dx f (x) = x E[

5. Using theorem 2.1 we find that, O( 1 ) updates are required. We can use this theorem because both the problem and the theorem have the same requirements (strongly convex functions) 1 6. The balance equation is n−2α = 2cn−α−2β → − nα−2β = 2c −2 −1−2β Since the problem states α = 1 we get: n = 2cn → − −2 = −1 − 2β → − β = 12 1 1 1 Now with beta = 2 and α = 1 we get 1 = c , solving for c gives 2 , and now e2n = 2 → − e2n = nc → − e2n = O( n1 ). Since, the order is a constant we can replace n with to get O( 1 ).

Thus, we have shown that n = 1

3

Exercise 12.

SGD: • Computational Cost per Update: O(1) Given.

• Number of updates to reach ρ: O( ρ1 ), using theorem 2.1 • Total Cost: O( ρ1 ), total cost is the product of the cost per update with how many times we need to update GD: • Computational Cost per Update: O(k), since we have to compute the cost for each data point • Number of updates to reach ρ: O(ln( ρ1 ), using theorem 2.3 • Total Cost: O(k × ln( ρ1 )), total cost is the product of the cost per update with how many times we need to update Exercise 13.

SGD: • Computational Cost per Update: O(1) • Number of updates to reach : O( 1 ), using theorem 2.1 • Total Cost: O( 1 ), total cost is the product of the cost per update with how many times we need to update GD: 1

• Computational Cost per Update: O(( 1 ) γ ). Since we know that = O(k −γ ) we can solve 1

1

1

for k, such that γ = k1γ γ => γ = k1 => k =

1 1

γ

• Number of updates to reach : O(ln( 1 )), using theorem 2.3 1

• Total Cost: O(( 1 ) γ × ln( 1 )) Total cost is the product of cost per update and number of updates required.

If we use 0.2 for gamma, then the total cost for training using Gradient Descent will be incredibly large as opposed to SGD which more or less stays constant and is independent of gamma. For 1 1 0.2 1 example if we use = 0.1 for GD we get a total cost of ( 0.1 ) ∗ ln( 0.1 ) = 230259. Whereas the total cost of SGD = 10. Thus for smaller errors, the training cost will grow rapidly.

4

Exercise 14.

1. The least squares solution is: β̂ = (X T X)−1 X T y Using the least squares formula gives the following values:

1.00890

1.00576

1.00656

0.99632

0.99770

0.99996

0.99254

0.99597

1.00206

Which is close to the truth value of a vector of size 10, with 1’s in every component.

2. Finding the gradient of the cost function will yield the following : −γsign(y (j) − θT x(j) ) × |y (j) − θT x(j) |γ−1 x(j) If we compare my calculations for the equation of the gradient, with degree two, with the calculations for the already implemented mean square error, we should see the values for θ to be quite similar: My gradient function:

0.88444

0.51884

0.76475

0.97001

0.99770

0.74257

0.74163

0.79375

0.78314

0.8657

Already implemented MSE function:

0.84367

0.50978

0.7434

0.97011

0.7217

0.74241

0.79693

0.79854

0.88037

Thus, I can be confident that my gradient equation is similar, since the values between the two test runs are very similar.

3.

With one iteration we see that SGD converges to a lower loss than ADAM with a gamma value of 2 and only one training session. This finding, however, could be due to random noise, so in the next section we will run 10 repetitions of the experiment and see which optimizer performs better.

5

0.76023

4.

With 10 iterations of the experiment we still see that SGD converges quicker than ADAM. This leads me to believe that the learning rate and the gamma parameter will show the true difference between these two algorithms. In the graph we compute the mean of the loss and the standard error every 50 iterations, and they never intersect so we know that with the learning rate set at 0.001 and an cost function degree of 2, SGD performs better than ADAM. In the next section we will experiment with different values of gamma to see which optimizer works better. This test was run with 20 replicates to ensure that randomness was not a factor in the outcomes.

6

5.

7

8

The figures show us the following: higher degree of objectivity allows SGD to perform better in terms of loss than ADAM. More specifically when the objective degree is less than or equal to 1, ADAM outperforms SGD, but when it’s larger than 1 SGD outperforms ADAM. The reason for this observation can be due to the magnitudes of errors. When gamma is high i.e. 2, 3 or 5, the magnitude of the error will be larger. Since, ADAM scales the learning rate based on the results of the cost function, a higher cost will cause the step size in each iteration to be smaller (and vice-versa), thus it takes longer to converge. SGD does not scale the learning rate in any way, allowing it to find the local optima quicker when gamma is high. On the other hand when gamma is less than one there is a lower penalty associated with the cost function which means that ADAM takes larger step sizes and converges a lot faster than SGD. We also notice that when the degree is 0.4, we get large bars of error. This is due to the derivative of x0.4 is very difficult to approximate at 0 which results the large standard error.

6.

The graph above compares the three algorithms with objective degree of 2 and learning rate of 0.001. We see that there is not much change in the loss for ADAGRAD. In fact, compared to SGD and ADAM it performs a lot worse and does not appear to converge to a local optima. The reason for this is due to the update rule. The update rule is as follows: θt = θt−1 − √

α Gt +

Where Gt is the running squared sum of the gradient.

We notice from the update rule that the running squared sum is a quadratic function which will bring this term to 0 as training increases. This means that with a small learning rate the impact and the magnitude that we update θ with will be less.

9

In this instance ADAGRAD converges a lot quicker than ADAM and SGD do, but has a larger variance in the early stages of training. This does give us more confidence in our belief, but since this is not a fair experiment we can’t say larger learning rates is where ADAGRAD excels for certainty. To make this a fair experiment we must compare each algorithm with the same hyperparamters to see which one performs better under each circumstance. Let’s examine under which circumstances ADAGRAD performs worse than SGD, but better than ADAM.

In this case these are the changes I made to the hyper parameter: changed the initial values of theta and modifying the degree of the error function. I changed the initial values of theta, such that the initial values were between 4 and 5. I wanted to give a bad starting place to all of the algorithms. Since ADAGRAD is dependent on having good initial values early on (since in later parts of the training stage it learns less and less), I wanted to throw the algorithm off significantly. This would allow SGD to get ahead of ADAGRAD. Then to make sure that ADAM did not out perform SGD I changed the degree of the function to 3, so that it was greater than 1.

Exercise 16.

1.

• The only difference between SGD and GD is essentially the batch size. GD requires the entire training set in each batch where as SGD takes a subset. Therefore, using a 10

batch size of 60,000, we are essentially using gradient descent.

• Model had an accuracy of 11% which is basically a random guesser, the training time was just shy of 3 minutes. If we look at the losses for each epoch of training we notice the loss isn’t getting better between epochs. Furthermore if we look at the class activation map we notice that the model is producing roughly equal probabilities for each class.

• One way we can improve the accuracy is by increasing the amount of training time we allow the algorithm to run for. 13 Epochs is clearly not enough epochs to allow the algorithm to converge to the minimum. We can also increase the learning rate to allow bigger steps between each iteration of the algorithm to be taken.

2.

• This model had a training time of 4 and a half minutes, and ended up getting an accuracy of 92%. And taking a look at the class activation graph when trying to predict a digit the model is showing a strong consensus for what it believes the digit is.

• One way to improve the training is by using a higher learning rate so that the model can converge quicker than it is with the default learning rate.

3.

• This model took 5 and a half minutes to train and ended up getting an accuracy of 97%. Similar to above the model is able to make predictions with more confidence.

This observation is based on the prediction graph made at the bottom of the notebook.

• Using the adam optimizer converges faster than SGD because it uses the momentum of past iterations to scale the learning rate. This allows the learning rate to increase when loss is being decreased, but the learning rate can also decrease if the loss isn’t decreasing as fast as it should.

11