COMPUTER S 123 F29AI – Artificial Intelligence and Intelligent Agents Lab 6 – MDPs Q&A

downloaded from

Scholarfriends

https://scholarfriends.com Downloaded by: browsegrades@gmail.com | https://scholarfriends.com/singlePaper/304285/computer-s-123-f29ai-artificial-intelligence-andintelligent-agents-lab-6-mdps-qa Distribution of this document is illegal.

F29AI – Artificial Intelligence and Intelligent Agents Lab 6 – MDPs QUESTION 1: MDP Notation Consider a gridworld MDP as shown below. The available actions in each state are to move to the neighboring grid squares. Left and Right actions are successful 80% of the time. When not successful, the agent stays in place.

From state a, there is also an exit action available, which results in going to the terminal state and collecting a reward of 10. Similarly, in state e, the reward for the exit action is 1. Exit actions are successful 100% of the time.

Fill in all of the following quantities.

T(c,Right,d) = 0.8 T(c,Right,e) = 0 T(c,Right,c) = 0.2 T(c,Right,b) = 0 T(c,Right,a) = 0 T(c,Right,terminal state) = 0 T(a,Exit,terminal state) = 1.0 T(a,Right,b) = 0.8 T(a,Right,a) = 0.2

R(a,Right,a) = 0 R(a,Exit,terminal state) = 10 R(c,Right,d) = 0

QUESTION 2: Discounting Consider the same gridworld MDP as in the previous exercise. The available actions in each state are to move to the neighboring grid squares. Assume all actions are ALWAYS (100%) successful. From state a, there is also an exit

This study source was downloaded by 100000859607764 from CourseHero.com on 03-22-2023 22:30:26 GMT -05:00

https://www.coursehero.com/file/161116716/F29AI-ActivitySheet-W7-8-mdp-solpdf/

action available, which results in going to the terminal state and collecting a reward of 10. Similarly, in state e, the reward for the exit action is 1. Exit actions are also successful 100% of the time. Note that the agent is now in state d, as shown below.

Part 1 Let the discount factor γ=0.1. What is the optimal action in the current state (d)? (Hint: Write down the computation of the discounted reward) • Left (0*1 + 0*0.1^1 + 0*0.1^2 + 10*0.1^3 = 0.01) • Right (0*1 + 1*0.1^1 = 0.1) Part 2 Now let the discount factor γ=0.9999. What is the optimal action in the current state (d)? (Hint: Write down the computation of the discounted reward) • •

Left (0*1 + 0*0.9999^1 + 0*0.9999^2 + 10*0.9999^3 = 9.9970003) Right (0*1 + 1*0.9999^1 = 0.9999)

QUESTION 4: Solving MDPs Consider the same gridworld MDP again, except that now Left and Right actions are 100% successful. Specifically, the available actions in each state are to move to the neighboring grid squares. From state a, there is also an exit action available, which results in going to the terminal state and collecting a reward of 10. Similarly, in state e, the reward for the exit action is 1. Exit actions are successful 100% of the time.

Let the discount factor γ=1. Fill in the following quantities.

Answer: The MDP is completely deterministic. And there is no discount. This means, intuitively, that the optimal action from all of the states b, c, d & e is to move left to get to the 10, and exit from a. Let’s see how this comes out in the numbers.

Note that in all the caculations in the table below, we are considering both the left and the right actions, and picking the maximum. This is shown in bold. We don’t here who the calculation for the suboptimal action.

V0 V1 V2

a 0 Exit:10 10

V3

10

b 0 0 Left:1.0*(0+1*10) =10 10

V4

10

10

c 0 0 Left:1.0*(0+1*0) =0 Left:1.0*(0+1*10) =10 10

V5

10

10

10

d 0 0 Right:1.0*(0+1*1) =1 Right:1.0*(0+1*1) =1 Left:1.0*(0+1*10) = 10 10

V6

10

10

10

10

This study source was downloaded by 100000859607764 from CourseHero.com on 03-22-2023 22:30:26 GMT -05:00

https://www.coursehero.com/file/161116716/F29AI-ActivitySheet-W7-8-mdp-solpdf/

e 0 Exit:1 1 1 Exit:1 Left:1.0*(0+1*10) = 10 10

QUESTION 5: Consider the following Markov Decision Process (MDP) of a robot running with an ice-cream:

Consider the following Markov Decision Process (MDP) of a robot race: The robot can either go fast or go slow. The states represent battery life remaining: fully charged (F), decreasing (D), or empty battery (E). Going fast brings double the reward than going slow, but also decreases battery life. This might eventually lead to an empty battery. Assume no discount of future actions ( = 1.0) and a living reward of zero. (i)

V0 V1

V2

V3

(ii)

Compute the time limited value for 4 time steps using the Value Iteration algorithm, and fill the table below.

F 0 Fast: 0.7*(+2 + 1.0 *0) + 0.3*(+2 + 1.0*0)=2 Slow: 1 * (+1 + 1.0*0)=1 +2 Fast: 0.7*(+2 + 1.0 *1) + 0.3*(+2 + 1.0*2)=3.3 Slow: 1 * (+1 + 1.0*2)=3 +3.3 Fast: 0.7*(+2 + 1.0 *2.3) + 0.3*(+2 + 1.0*3.3)= 4.6 Slow: 1 * (+1 + 1.0*3.3)=4.3 4.6

D 0 Fast: 1*(-10+1.0*0)=-10 Slow: 0.7*(+1 + 1.0 *0) + 0.3*(+1+1.0*0) = 1 +1 Fast: 1*(-10+1.0*0)=-10 Slow: 0.7*(+1 + 1.0 *1) + 0.3*(+1+1.0*2) = 2.3 +2.3 Fast: 1*(-10+1.0*0)=-10 Slow: 0.7*(+1 + 1.0 *2.3) + 0.3*(+1+1.0*3.3) = 3.6 3.6

What is the optimal policy in your own words? Fill the values below:

This study source was downloaded by 100000859607764 from CourseHero.com on 03-22-2023 22:30:26 GMT -05:00

https://www.coursehero.com/file/161116716/F29AI-ActivitySheet-W7-8-mdp-solpdf/

*(F) = Fast *(D) = Slow In plain language, the optimal policy is: when your battery is full, go fast. When it’s decreasing, go slow.

This study source was downloaded by 100000859607764 from CourseHero.com on 03-22-2023 22:30:26 GMT -05:00

https://www.coursehero.com/file/161116716/F29AI-ActivitySheet-W7-8-mdp-solpdf/ Powered by TCPDF (www.tcpdf.org)