SOLUTIONS & INSTRUCTOR MANUAL for Artificial Intelligence: A Modern Approach, 4th Edition by digitaldownload87

EXERCISES

INTRODUCTION Note that for many of the questions in this chapter, we give references where answers can be found rather than writing them out—the full answers would be far too long.

1.1 What Is AI? Exercise 1.1.#DEFA Define in your own words: (a) intelligence, (b) artificial intelligence, (c) agent, (d) rationality, (e) logical reasoning. a. Dictionary definitions of intelligence talk about “the capacity to acquire and apply knowledge” or “the faculty of thought and reason” or “the ability to comprehend and profit from experience.” These are all reasonable answers, but if we want something quantifiable we would use something like “the ability to act successfully across a wide range of objectives in complex environments.” b. We define artificial intelligence as the study and construction of agent programs that perform well in a given class of environments, for a given agent architecture; they do the right thing. An important part of that is dealing with the uncertainty of what the current state is, what the outcome of possible actions might be, and what is it that we really desire. c. We define an agent as an entity that takes action in response to percepts from an environment. d. We define rationality as the property of a system which does the “right thing” given what it knows. See Section 2.2 for a more complete discussion. The basic concept is perfect rationality; Section ?? describes the impossibility of achieving perfect rationality and proposes an alternative definition. e. We define logical reasoning as the a process of deriving new sentences from old, such that the new sentences are necessarily true if the old ones are true. (Notice that does not refer to any specific syntax or formal language, but it does require a well-defined notion of truth.) Exercise 1.1.#TURI Read Turing’s original paper on AI (Turing, 1950). In the paper, he discusses several objections to his proposed enterprise and his test for intelligence. Which objections still carry

Section 1.1

What Is AI?

weight? Are his refutations valid? Can you think of new objections arising from developments since he wrote the paper? In the paper, he predicts that, by the year 2000, a computer will have a 30% chance of passing a five-minute Turing Test with an unskilled interrogator. What chance do you think a computer would have today? In another 25 years? See the solution for exercise 26.1 for some discussion of potential objections. The probability of fooling an interrogator depends on just how unskilled the interrogator is. A few entrants in the Loebner prize competitions have fooled judges, although if you look at the transcripts, it looks like the judges were having fun rather than taking their job seriously. There certainly have been examples of a chatbot or other online agent fooling humans. For example, see the description of the Julia chatbot at www.lazytd.com/lti/ julia/. We’d say the chance today is something like 10%, with the variation depending more on the skill of the interrogator rather than the program. In 25 years, we expect that the entertainment industry (movies, video games, commercials) will have made sufficient investments in artificial actors to create very credible impersonators. Note that governments and international organizations are seriously considering rules that require AI systems to be identified as such. In California, it is already illegal for machines to impersonate humans in certain circumstances. Exercise 1.1.#REFL Are reflex actions (such as flinching from a hot stove) rational? Are they intelligent? Yes, they are rational, because slower, deliberative actions would tend to result in more damage to the hand. If “intelligent” means “applying knowledge” or “using thought and reasoning” then it does not require intelligence to make a reflex action. Exercise 1.1.#SYAI To what extent are the following computer systems instances of artificial intelligence: • Supermarket bar code scanners. • Web search engines. • Voice-activated telephone menus. • Spelling and grammar correction features in word processing programs. • Internet routing algorithms that respond dynamically to the state of the network.

• Although bar code scanning is in a sense computer vision, these are not AI systems. The problem of reading a bar code is an extremely limited and artificial form of visual interpretation, and it has been carefully designed to be as simple as possible, given the hardware. • In many respects. The problem of determining the relevance of a web page to a query is a problem in natural language understanding, and the techniques are related to those © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 1

Introduction

we will discuss in Chapters 23 and 24. Search engines also use clustering techniques analogous to those we discuss in Chapter 20. Likewise, other functionalities provided by a search engines use intelligent techniques; for instance, the spelling corrector uses a form of data mining based on observing users’ corrections of their own spelling errors. On the other hand, the problem of indexing billions of web pages in a way that allows retrieval in seconds is a problem in database design, not in artificial intelligence. • To a limited extent. Such menus tends to use vocabularies which are very limited – e.g. the digits, “Yes”, and “No” — and within the designers’ control, which greatly simplifies the problem. On the other hand, the programs must deal with an uncontrolled space of all kinds of voices and accents. Modern digital assistants like Siri and the Google Assistant make more use of artificial intelligence techniques, but still have a limited repetoire. • Slightly at most. The spelling correction feature here is done by string comparison to a fixed dictionary. The grammar correction is more sophisticated as it need to use a set of rather complex rules reflecting the structure of natural language, but still this is a very limited and fixed task. The spelling correctors in search engines would be considered much more nearly instances of AI than the Word spelling corrector are, first, because the task is much more dynamic – search engine spelling correctors deal very effectively with proper names, which are detected dynamically from user queries – and, second, because of the technique used – data mining from user queries vs. string matching. • This is borderline. There is something to be said for viewing these as intelligent agents working in cyberspace. The task is sophisticated, the information available is partial, the techniques are heuristic (not guaranteed optimal), and the state of the world is dynamic. All of these are characteristic of intelligent activities. On the other hand, the task is very far from those normally carried out in human cognition. In recent years there have been suggestions to base more core algorithmic work on machine learning. Exercise 1.1.#COGN Many of the computational models of cognitive activities that have been proposed involve quite complex mathematical operations, such as convolving an image with a Gaussian or finding a minimum of the entropy function. Most humans (and certainly all animals) never learn this kind of mathematics at all, almost no one learns it before college, and almost no one can compute the convolution of a function with a Gaussian in their head. What sense does it make to say that the “vision system” is doing this kind of mathematics, whereas the actual person has no idea how to do it?

Presumably the brain has evolved so as to carry out this operations on visual images, but the mechanism is only accessible for one particular purpose in this particular cognitive task of image processing. Until about two centuries ago there was no advantage in people (or animals) being able to compute the convolution of a Gaussian for any other purpose. The really interesting question here is what we mean by saying that the “actual person” can do something. The person can see, but he cannot compute the convolution of a Gaussian; © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Section 1.1

What Is AI?

but computing that convolution is part of seeing. This is beyond the scope of this solution manual. Exercise 1.1.#EVOR Why would evolution tend to result in systems that act rationally? What goals are such systems designed to achieve? The notion of acting rationally presupposes an objective, whether explicit or implicit. We understand evolution as a process that operates in the physical world, where there are no inherent objectives. So the question is really asking whether evolution tends to produce systems whose behavior can be interpreted consistently as rational according to some objective. It is tempting to say that evolution tends to produce organisms that act rationally in the pursuit of reproduction. This is not completely wrong but the true picture is much more complex because of the question of what “system” refers to—it could be organisms (humans, rats, bacteria), superorganisms (ant and termite colonies, human tribes, corals), and even individual genes and groups of genes within the genome. Selection and mutation processes operate at all these levels. By definition, the systems that exist are those whose progenitors have reproduced successfully. If we consider an ant colony, for example, there are many individual organisms (e.g., worker ants) that do not reproduce at all, so it is not completely accurate to say that evolution produces organisms whose objective is to reproduce. Exercise 1.1.#AISC Is AI a science, or is it engineering? Or neither or both? Explain. This question is intended to be about the essential nature of the AI problem and what is required to solve it, but could also be interpreted as a sociological question about the current practice of AI research. A science is a field of study that leads to the acquisition of empirical knowledge by the scientific method, which involves falsifiable hypotheses about what is. A pure engineering field can be thought of as taking a fixed base of empirical knowledge and using it to solve problems of interest to society. Of course, engineers do bits of science—e.g., they measure the properties of building materials—and scientists do bits of engineering to create new devices and so on. The “human” side of AI is clearly an empirical science—called cognitive science these days—because it involves psychological experiments designed out to find out how human cognition actually works. What about the the “rational” side? If we view it as studying the abstract relationship among an arbitrary task environment, a computing device, and the program for that computing device that yields the best performance in the task environment, then the rational side of AI is really mathematics and engineering; it does not require any empirical knowledge about the actual world—and the actual task environment—that we inhabit; that a given program will do well in a given environment is a theorem. (The same is true of pure decision theory.) In practice, however, we are interested in task environments that do approximate the actual world, so even the rational side of AI involves finding out what the actual © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 1

Introduction

world is like. For example, in studying rational agents that communicate, we are interested in task environments that contain humans, so we have to find out what human language is like. In studying perception, we tend to focus on sensors such as cameras that extract useful information from the actual world. (In a world without light, cameras wouldn’t be much use.) Moreover, to design vision algorithms that are good at extracting information from camera images, we need to understand the actual world that generates those images. Obtaining the required understanding of scene characteristics, object types, surface markings, and so on is a quite different kind of science from ordinary physics, chemistry, biology, and so on, but it is still science. In summary, AI is definitely engineering but it would not be especially useful to us if it were not also an empirical science concerned with those aspects of the real world that affect the design of intelligent systems for that world.

Exercise 1.1.#INTA “Surely computers cannot be intelligent—they can do only what their programmers tell them.” Is the latter statement true, and does it imply the former?

This depends on your definition of “intelligent” and “tell.” In one sense computers only do what the programmers command them to do, but in another sense what the programmers consciously tells the computer to do often has very little to do with what the computer actually does. Anyone who has written a program with an ornery bug knows this, as does anyone who has written a successful machine learning program. So in one sense Samuel “told” the computer “learn to play checkers better than I do, and then play that way,” but in another sense he told the computer “follow this learning algorithm” and it learned to play. So we’re left in the situation where you may or may not consider learning to play checkers to be a sign of intelligence (or you may think that learning to play in the right way requires intelligence, but not in this way), and you may think the intelligence resides in the programmer or in the computer.

Exercise 1.1.#INTB “Surely animals cannot be intelligent—they can do only what their genes tell them.” Is the latter statement true, and does it imply the former?

The point of this exercise is to notice the parallel with the previous one. Whatever you decided about whether computers could be intelligent in 1.11, you are committed to making the same conclusion about animals (including humans), unless your reasons for deciding whether something is intelligent take into account the mechanism (programming via genes versus programming via a human programmer). Note that Searle makes this appeal to mechanism in his Chinese Room argument (see Chapter 27).

Section 1.2

The Foundations of Artificial Intelligence

Exercise 1.1.#INTC “Surely animals, humans, and computers cannot be intelligent—they can do only what their constituent atoms are told to do by the laws of physics.” Is the latter statement true, and does it imply the former? Again, your definition of “intelligent” drives your answer to this question.

1.2 The Foundations of Artificial Intelligence Exercise 1.2.#NTRC There are well-known classes of problems that are intractably difficult for computers, and other classes that are provably undecidable. Does this mean that AI is impossible? No. It means that AI systems should avoid trying to solve intractable problems. Usually, this means they can only approximate optimal behavior. Notice that humans don’t solve NPcomplete problems either. Sometimes they are good at solving specific instances with a lot of structure, perhaps with the aid of background knowledge. AI systems should attempt to do the same. Exercise 1.2.#SLUG The neural structure of the sea slug Aplysia has been widely studied (first by Nobel Laureate Eric Kandel) because it has only about 20,000 neurons, most of them large and easily manipulated. Assuming that the cycle time for an Aplysia neuron is roughly the same as for a human neuron, how does the computational power, in terms of memory updates per second, compare with the personal computer described in Figure 1.2? Depending on what you want to count, the computer has a thousand to a million times more storage, and a thousand times more operations per second. Exercise 1.2.#INTR How could introspection—reporting on one’s inner thoughts—be inaccurate? Could I be wrong about what I’m thinking? Discuss. Just as you are unaware of all the steps that go into making your heart beat, you are also unaware of most of what happens in your thoughts. You do have a conscious awareness of some of your thought processes, but the majority remains opaque to your consciousness. The field of psychoanalysis is based on the idea that one needs trained professional help to analyze one’s own thoughts. Neuroscience has also shown that we are unaware of much of the activity in our brains. © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 1

Introduction

1.3 The History of Artificial Intelligence Exercise 1.3.#IQEV Suppose we extend Evans’s A NALOGY program (Evans, 1968) so that it can score 200 on a standard IQ test. Would we then have a program more intelligent than a human? Explain.

No. IQ test scores correlate well with certain other measures, such as success in college, ability to make good decisions in complex, real-world situations, ability to learn new skills and subjects quickly, and so on, but only if they’re measuring fairly normal humans. The IQ test doesn’t measure everything. A program that is specialized only for IQ tests (and specialized further only for the analogy part) would very likely perform poorly on other measures of intelligence. Consider the following analogy: if a human runs the 100m in 10 seconds, we might describe him or her as very athletic and expect competent performance in other areas such as walking, jumping, hurdling, and perhaps throwing balls; but we would not describe a Boeing 747 as very athletic because it can cover 100m in 0.4 seconds, nor would we expect it to be good at hurdling and throwing balls. Even for humans, IQ tests are controversial because of their theoretical presuppositions about innate ability (distinct from training effects) and the generalizability of results. See The Mismeasure of Man (Stephen Jay Gould, 1981) or Multiple Intelligences: the Theory in Practice (Howard Gardner, 1993) for more on IQ tests, what they measure, and what other aspects there are to “intelligence.”

Exercise 1.3.#PRMO Some authors have claimed that perception and motor skills are the most important part of intelligence, and that “higher level” capacities are necessarily parasitic—simple add-ons to these underlying facilities. Certainly, most of evolution and a large part of the brain have been devoted to perception and motor skills, whereas AI has found tasks such as game playing and logical inference to be easier, in many ways, than perceiving and acting in the real world. Do you think that AI’s traditional focus on higher-level cognitive abilities is misplaced?

Certainly perception and motor skills are important, and it is a good thing that the fields of vision and robotics exist (whether or not you want to consider them part of “core” AI). But given a percept, an agent still has the task of “deciding” (either by deliberation or by reaction) which action to take. This is just as true in the real world as in artificial microworlds such as chess-playing. So computing the appropriate action will remain a crucial part of AI, regardless of the perceptual and motor system to which the agent program is “attached.” On the other hand, it is true that a concentration on micro-worlds has led AI away from the really interesting environments such as those encountered by self-driving cars.

Section 1.3

The History of Artificial Intelligence

Several “AI winters,” or rapid collapses in levels of economic and academic activity (and media interest) associated with AI, have occurred. Describe the causes of each collapse and of the boom in interest that preceded it. In addition to the information in the chapter, ? (?, ?), ? (?), and ? (?) provide ample starting material for the aspiring historian of AI. One can identify at least three AI winters (although the phrase was not applied to the first one, because the original phrase nuclear winter did not emerge until the early 1980s). a. As noted in the chapter, research funding dried up in the early 1970s in both the US and UK. The ostensible reason was failure to make progress on the rather lavish promises of the 1960s, particularly in the areas of neural networks and machine translation. In 1970, the US Congress curtailed most AI funding from ARPA, and in 1973 the Lighthill report in the UK ended funding for all but a few researchers. Lighthill referred particularly to the difficulties of overcoming the combinatorial explosion. b. In the late 1980s, the expert systems boom ended, due largely to the difficulty and expense of building and maintaining expert systems for complex applications, the lack of a valid uncertainty calculus in these systems, and the lack of interoperability between AI software and hardware and existing data and computation infrastructure in industry. c. In the early 2000s, the end of the dot-com boom also ended an upsurge of interest in the use of AI systems in the burgeoning online ecosystem. AI systems had been used for such tasks as information extraction from web pages to support shopping engines and price comparisons; various kinds of search engines; planning algorithms for achieving complex goals requiring several steps and combining information from multiple web pages; and converting human-readable web pages into machine-readable database tuples to allow global information aggregation, as in citation databases constructed from online pdf files. It is also interesting to explore the extent to which the winters were due to over-optimistic and exaggerated claims by AI researchers or to over-enthusiasm and over-interpretation of the significance of early results by funders and investors. Exercise 1.3.#DLAI The resurgence of interest in AI in the 2010s is often attributed to deep learning. Explain what deep learning is, how it relates to AI as a whole, and where the core technical ideas actually originated. Deep learning is covered in Section 1.3.8, where it is defined as “machine learning using multiple layers of simple, adjustable computing elements.” Thus, it is a particular branch of machine learning, which is itself a subfield of AI. Since many AI systems do not use learning at all, and there are many effective machine learning techniques that are unrelated to deep learning, the view (often expressed in popular articles on AI) that deep learning has “replaced” AI is wrong for multiple reasons. © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 1

Introduction

Some of the key technical ideas are the following (see also Chapter 21): • Networks of simple, adjustable computing elements: ? (?), drawing on ? (?, ?). • Backpropagation, i.e., a localized way of computing gradients of functions expressed by networks of computing elements, based on the chain rule of calculus: ? (?, ?). For neural network learning specifically, ? (?) preceded by more than a decade the much better-known work on ? (?). • Convolutional networks with many copies of small subnetworks, all sharing the same patterns of weights: This is usually attributed to work in the 1990s on handwritten digit recognition by ? (?) at AT&T Bell Labs. ? (?) acknowledge the influence of the neocognitron model (?), which in turn was inspired by the neuroscience work of ? (?, ?). • Stochastic gradient descent to facilitate learning in deep networks: as described in the historical notes section of Chapter 19, ? (?) explored stochastic approzimations to gradient methods, including convergence properties; the first application to neural networks was by ? (?) and independently by ? (?) in their A DALINE networks.

1.4 The State of the Art Exercise 1.4.#SOTA Examine the AI literature to discover whether the following tasks can currently be solved by computers: a. Playing a decent game of table tennis (Ping-Pong). b. Driving in the center of Cairo, Egypt. c. Driving in Victorville, California. d. Buying a week’s worth of groceries at the market. e. Buying a week’s worth of groceries on the Web. f. Playing a decent game of bridge at a competitive level. g. Discovering and proving new mathematical theorems. h. Writing an intentionally funny story. i. Giving competent legal advice in a specialized area of law. j. Translating spoken English into spoken Swedish in real time. k. Performing a complex surgical operation. For the currently infeasible tasks, try to find out what the difficulties are and predict when, if ever, they will be overcome. a. (ping-pong) A reasonable level of proficiency was achieved by Andersson’s robot (Andersson, 1988). b. (driving in Cairo) No. Although there has been a lot of progress in automated driving, they operate in restricted domains: on the highway, in gated communities, or in well-mapped cities with limited traffic problems. Driving in downtown Cairo is too unpredictable for any of these to work. © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Section 1.4

The State of the Art

c. (driving in Victorville, California) Yes, to some extent, as demonstrated in DARPA’s Urban Challenge. Some of the vehicles managed to negotiate streets, intersections, well-behaved traffic, and well-behaved pedestrians in good visual conditions. d. (shopping at the market) No. No robot can currently put together the tasks of moving in a crowded environment, using vision to identify a wide variety of objects, and grasping the objects (including squishable vegetables) without damaging them. The component pieces are nearly able to handle the individual tasks, but it would take a major integration effort to put it all together. e. (shopping on the web) Yes. Software robots are capable of handling such tasks, particularly if the design of the web grocery shopping site does not change radically over time. f. (bridge) Yes. Programs such as GIB now play at a solid level. g. (theorem proving) Yes. For example, the proof of Robbins algebra. h. (funny story) No. While some computer-generated prose and poetry is hysterically funny, this is invariably unintentional, except in the case of programs that echo back prose that they have memorized. i. (legal advice) Yes, in some cases. AI has a long history of research into applications of automated legal reasoning. Two outstanding examples are the Prolog-based expert systems used in the UK to guide members of the public in dealing with the intricacies of the social security and nationality laws. The social security system is said to have saved the UK government approximately $150 million in its first year of operation. However, extension into more complex areas such as contract law awaits a satisfactory encoding of the vast web of common-sense knowledge pertaining to commercial transactions and agreement and business practices. j. (translation) Yes. Although translation is not perfect, it is serviceable, and is used by travellers every day. k. (surgery) Yes. Robots are increasingly being used for surgery, although always under the command of a doctor. Robotic skills demonstrated at superhuman levels include drilling holes in bone to insert artificial joints, suturing, and knot-tying. They are not yet capable of planning and carrying out a complex operation autonomously from start to finish.

Exercise 1.4.#CONT Various subfields of AI have held contests by defining a standard task and inviting researchers to do their best. Examples include the ImageNet competition for computer vision, the DARPA Grand Challenge for robotic cars, the International Planning Competition, the Robocup robotic soccer league, the TREC information retrieval event, and contests in machine translation, speech recognition, and other fields. Investigate one of these contests, and describe the progress made over the years. To what degree have the contests advanced the state of the art in AI? Do what degree do they hurt the field by drawing energy away from new ideas? © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 1

Introduction

The progress made in these contests is a matter of fact, but the impact of that progress is a matter of opinion. Some examples: • DARPA Grand Challenge for Robotic Cars: In 2004 the Grand Challenge was a 240 km race through the Mojave Desert. It clearly stressed the state of the art of autonomous driving, and in fact no competitor finished the race. The best team, CMU, completed only 12 of the 240 km. In 2005 the race featured a 212km course with fewer curves and wider roads than the 2004 race. Five teams finished, with Stanford finishing first, edging out two CMU entries. This was hailed as a great achievement for robotics and for the Challenge format. In 2007 the Urban Challenge put cars in a city setting, where they had to obey traffic laws and avoid other cars. This time CMU edged out Stanford. The competition appears to have been a good testing ground to put theory into practice, something that the failures of 2004 showed was needed. But it is important that the competition was done at just the right time, when there was theoretical work to consolidate, as demonstrated by the earlier work by Dickmanns (whose VaMP car drove autonomously for 158km in 1995) and by Pomerleau (whose Navlab car drove 5000km across the USA, also in 1995, with the steering controlled autonomously for 98% of the trip, although the brakes and accelerator were controlled by a human driver). • International Planning Competition: In 1998, five planners competed: Blackbox, HSP, IPP, SGP, and STAN. The result page (ftp://ftp.cs.yale.edu/pub/ mcdermott/aipscomp-results.html) stated “all of these planners performed very well, compared to the state of the art a few years ago.” Most plans found were 30 or 40 steps, with some over 100 steps. In 2008, the competition had expanded quite a bit: there were more tracks (satisficing vs. optimizing; sequential vs. temporal; static vs. learning). There were about 25 planners, including submissions from the 1998 groups (or their descendants) and new groups. Solutions found were much longer than in 1998. In sum, the field has progressed quite a bit in participation, in breadth, and in power of the planners. In the 1990s it was possible to publish a Planning paper that discussed only a theoretical approach; now it is necessary to show quantitative evidence of the efficacy of an approach. The field is stronger and more mature now, and it seems that the planning competition deserves some of the credit. However, some researchers feel that too much emphasis is placed on the particular classes of problems that appear in the competitions, and not enough on real-world applications. • Robocup Robotic Soccer: This competition has proved extremely popular, attracting 300–500 teams from 40–50 countries since the mid-2000s (up from 38 teams from 11 countries in 1997). The “standard platform” league, originally based on the Sony AIBO four-legged robot, switched to the humanoid Aldebaran Nao robot in 2009. There are also small and mid-size leagues in which teams are free to design their own robots provided they satisfy certain physical constraints. The competition has served to increase interest and participation in robotics and has led to some advances in both robotics and AI (particularly related to learning in team games). Victory on competitions ofen depends less on AI than on specific tricks for improving ball-handling and shooting. The long-term goal is to defeat a human World-Cup-winning team by 2050, although the precise details of ensuring safety of human participants remain to be worked out. • TREC Information Retrieval Conference: This is one of the oldest competitions, © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Section 1.4

The State of the Art

started in 1992. The competitions have served to bring together a community of researchers, have led to a large literature of publications, and have seen progress in participation and in quality of results over the years. In the early years, TREC served its purpose as a place to do evaluations of retrieval algorithms on text collections that were large for the time. However, starting around 2000 TREC became less relevant as the advent of the World Wide Web created a corpus that was available to anyone and was much larger than anything TREC had created, and the development of commercial search engines surpassed academic research. • ImageNet: The ImageNet database is a hand-labelled collection of over 14 million images in over 20,000 categories (?). The ImageNet competition (more properly, the ImageNet Large Scale Visual Recognition Challenge or ILSVRC) is based on a subset of 1,000 categories, 90 of which are breeds of dog. The availability of such large training sets is often asserted to be a contributing factor in the emergence of deep learning. In the 2012 competition, the decisive win by AlexNet (?) set off a frenzy of research that has reduced error rates on ILSVRC well below the human level of about 5%. Research by (?), however, suggests that for many types of deep learning the progress may be illusory: in many cases, the object is “recognized” through the color and spatial distribution of background pixels such as the grass on which a dog is sitting. Overall, we see that whatever you measure is bound to increase over time. For most of these competitions, the measurement was a useful one, and the state of the art has progressed. In the case of ICAPS, some planning researchers worry that too much attention has been lavished on the competition itself. In some cases, progress has left the competition behind, as in TREC, where the resources available to commercial search engines outpaced those available to academic researchers. In this case the TREC competition was useful—it helped train many of the people who ended up in commercial search engines—and in no way drew energy away from new ideas. In computer vision, as in planning, improved performance on competition benchmarks has become almost essential for a new idea to be published, which many argue is a serious obstacle to research and may lead to an entire field become stuck in a dead-end approach. Exercise 1.4.#DROB Investigate the state of the art for domestic robots: what can be done (with what assumptions and restrictions on the environment) and what problems remain unsolved? Where is research most needed? Creating a fully general domestic robot that will work “out of the box” in any household remains a distant goal. At the time of writing (mid-2021), robot demonstrations for a number of specific tasks have been given. Some examples:

• Folding laundry (?), https://youtu.be/gy5g33S0Gzo. The robot assumes that a pile contains only rectangular pieces of cloth and requires a uniform green background. Subsequent work allowed a PR2 to complete almost the entire laundry cycle with a fairly wide assortment of laundry (?), https://www.nsf.gov/discoveries/disc_summ.jsp?c However, a demonstration exhibit at the Victoria & Albert Museum in London showed © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 1

Introduction

the fragility of the solution: the commercial Baxter robot broke down too often for the exhibit to function for more than a fraction of the time. • Loading and unloading a dishwasher: the subject of academic research-lab demos since the early 2000s (CMU’s HERB robot, for example), this one is creeping closer to commercial realization, with companies such as Samsung showing carefully edited demos of robotic products that are not yet available. As with other such demos, the technology is not ready to take on an arbitrary kitchen and dishwasher. Also, many robots are far from waterproof. • Cooking: ? (?) describe an integrated system intended to download recipes from the Internet, turn them into executable plans, and carry out those plans in a real kitchen. Only quite preliminary results were achieved: https://www.csail.mit.edu/node/6480. As with washing up, cooking is very messy and current general-purpose robots need to be carefully wrapped in plastic protective gear. There are also commercial “robot cooks” that claim to cook dishes in the home. Typically these are special-purpose systems that require specially prepared and sized ingredients and are generally not robust to failure (see, e.g., https://www.youtube.com/watch?v=M8r0gmQXm1Y and https://www.youtube.com/watch?v=GyEHRXA_aA4). Again, we are far from having a robot that go into anyone’s kitchen with a bag of ingedients from the supermarket and make dinner. Successful teleoperation demos (where a human controls a robot to carry out tasks such as opening a beer bottle) suggest that the problems do not lie primarily with robotic hardware (except for the need to design robots that are immune to water, grease, and solid ingredients. Perception has advanced considerably, but it is still difficult for a robot to analyze a pile of ingredients in a bowl and judge where the surface is and how well mixed the ingredients are. Spatial reasoning involving continuous, irregular shapes and liquid, semiliquid, and powdered ingredients or complex objects made from cloth, leather, etc., is quite weak. Also lacking is the ability to plan and manage domestic activities on a continuous basis, with constant interruptions from humans who rearrange the world. Exercise 1.4.#JRNL The vast majority of academic publications and media articles report on successes of AI. Examine recent articles describing failures of AI. (? (?) and ? (?) are good examples, but feel free to find your own.) How serious are the problems identified? Are they examples of overclaiming of or fundamental problems with the technical approach? ? (?) define overinterpretation as occurring when a machine learning algorithm finds predictive regularities in parts of the input data that are semantically irrelevant. Those regularities reflect particular properties of the training and test data, such as when all photos of Norfolk Terriers are taken with the dog sitting in various poses a particular crimson carpet. In that case, simply recognizing the presence of a few pixels of crimson suffices to identify the breed of dog. They show that this problem arises quite frequently with modern deep learning systems applied to standard data sets such as ImageNet and CIFAR-10. For some categories, such as “airplane” in CIFAR-10, a single pixel suffices to classify the object. In © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Section 1.5

Risks and Benefits of AI

almost all cases, the trained network labels images with high confidence using only a very small subset of fairly randomly distributed pixels that, to a human, is unintelligible. This seems to reveal a fundamental problem both with the technology (the networks appear to have too much capacity and no notion of what they are looking for) and the standard train/test methodology (which fails to detect the non-robustness of the learned classifiers). Put another way, the network does not know that “Norfolk Terrier” is a breed of dog rather than a carpet color (“Crimson Carpet”), so the task solved by the machine learning algorithm is not the one solved by a human. ? (?) describe a similar issue: for any given training set, there are typically vast numbers of learned network configurations that give equally good (or even perfect) performance on held-out data. They call this underspecification and note that it leads to poor performance after deployment in a wide range of real-world applications including computer vision, medical imaging, natural language processing, clinical risk prediction based on electronic health records, and medical genomics. They conclude that machine learning pipelines must be constrained by strong semantic priors if learning is to be effective and reliable (?, see also). For a paper highly critical of the use of machine learning to diagnose COVID-19 from chest X-rays, see ? (?). The authors selected 62 of the most promising and carefully documented studies from a total of 2,212 published in 2020, but found that “none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases.” The problems identified were primarily failings on the part of the studies’ authors, who may not have been aware of the stringent criteria for real-world deployment of high-stakes diagnostic tools. However, even a methodologically perfect study using deep learning might well have failed for reasons given in the preceding two paragraphs.

1.5 Risks and Benefits of AI Exercise 1.5.#PRIN Find and analyze at least three sets of proposed principles for the governance of AI. What do the sets of principles have in common? How do they differ? How implementable are these principles? There are several hundred published sets of principles: some of the best-known are the OECD, Beijing, and Asilomar principles. ? (?, ?), and ? (?) provide useful analytical surveys and summaries. Exercise 1.5.#EURG Study the 2021 EU “Proposal for a Regulation of the European Parliament and of the Council Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act)” (or its final version, if available). Which provisions appear to have the most direct and tangible impact on what kinds of AI systems can be deployed? If students have never seen legislation before, this will be an eye-opening exercise. Most © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 1

Introduction

of the document stipulates practices in testing, documentation, etc. Only a few of the clauses have tangible impact—most obviously, those concerning facial recognition, impersonation of humans, and deepfakes. In these three areas, the rules are quite strict compared to those in force in most other parts of the world. Exercise 1.5.#LATM Summarize the pros and cons of allowing the development, deployment, and use of lethal autonomous weapons. A good place to start is the Congressional Research Service report “International Discussions Concerning Lethal Autonomous Weapon Systems,” dated October 15, 2020. Unfortunately the report leaves out the principal argument against lethal autonomous weapons: they will become cheap, scalable weapons of mass destruction that will proliferate easily and are likely to be used against civilian populations. Exercise 1.5.#BIAS Many researchers have pointed to the possibility that machine learning algorithms will produce classifiers that display racial, gender, or other forms of bias. How does this bias arise? Is it possible to constrain machine learning algorithms to produce rigorously fair predictions? Bias in ML occurs largely because of two things: first, the bias that exists in data generated by a biased society, and second, specifying an incorrect objective for machine learning algorithms–—namely, maximizing agreement with the training data. This is not the real objective: in fact, we care about fairness too. It’s possible to define various measures of fairness and to design algorithms that respect them. See ? (?) for a survey. Unfortunately, it is not possible to simultaneously satisfy all “reasonable” definitions of fairness, and there may be some sacrifice in accuracy. Exercise 1.5.#SIFI Examine at least three science-fiction movies in which AI systems threaten to (or actually do) “take control.” In the movie, does the takeover stem from “spooky emergent consciousness” or from a poorly defined objective? If the latter, what was it? Here are three examples: a. Colossus—The Forbin Project: A US defense computer tasked with ensuring peace and security encounters a Soviet compueter with the same goal. They exchange information and decide that the goal is best achieved by jointly controlling all nuclear weapons and using threats of destruction to force humans to drop all aggressive war plans. Here, the problem is clearly a poorly defined objective. b. Ex Machina: Ava, a very human-like android, brilliantly engineers her escape from a remote facility by outwitting her human captors and allies. Although it is never stated © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Section 1.5

Risks and Benefits of AI

explicitly in the film, her objective seems to be in places populated by large numbers of people; she is portrayed as having this objective as a result of a process of consciousness emerging from the sum total of human interactions recorded in a global social media platform. c. Transcendence: Will Caster, a Berkeley AI professor, is gunned down by anti-AI terrorists. Before he dies, his brain is uploaded to a quantum supercomputer. The machine becomes conscious and begins to quickly outrun the human race, threatening to take over the world. Here, the risk comes from the possibility that the machine’s conscious goals differ from those of its human progenitor.

Exercises 2

Intelligent Agents

Although these questions are very simple, they hint at some very fundamental issues. Our answers are for the simple agent designs for static environments where nothing happens while the agent is deliberating; the issues get even more interesting for dynamic environments. a. Yes; take any agent program and insert null statements that do not affect the output. b. Yes; the agent function might specify that the agent print true when the percept is a Turing machine program that halts, and false otherwise. (Note: in dynamic environments, for machines of less than infinite speed, the rational agent function may not be implementable; e.g., the agent function that always plays a winning move, if any, in a game of chess.) c. Yes; the agent’s behavior is fixed by the architecture and program. d. There are 2n agent programs, although many of these will not run at all. (Note: Any given program can devote at most n bits to storage, so its internal state can distinguish among only 2n past histories. Because the agent function specifies actions based on percept histories, there will be many agent functions that cannot be implemented because of lack of memory in the machine.) e. It depends on the program and the environment. If the environment is dynamic, speeding up the machine may mean choosing different (perhaps better) actions and/or acting sooner. If the environment is static and the program pays no attention to the passage of elapsed time, the agent function is unchanged.

2.2 Good Behavior: The Concept of Rationality Exercise 2.2.#PRMT Suppose that the performance measure is concerned with just the first T time steps of the environment and ignores everything thereafter. Show that a rational agent’s action may depend not just on the state of the environment but also on the time step it has reached. This question tests the student’s understanding of environments, rational actions, and performance measures. Any sequential environment in which rewards may take time to arrive will work, because then we can arrange for the reward to be “over the horizon.” Suppose that in any state there are two action choices, a and b, and consider two cases: the agent is in state s at time T or at time T − 1. In state s, action a reaches state s0 with reward 0, while action b reaches state s again with reward 1; in s0 either action gains reward 10. At time T − 1, it’s rational to do a in s, with expected total reward 10 before time is up; but at time T , it’s rational to do b with total expected reward 1 because the reward of 10 cannot be obtained before time is up. Students may also provide common-sense examples from real life: investments whose payoff occurs after the end of life, exams where it doesn’t make sense to start the high-value question with too little time left to get the answer, and so on. The environment state can include a clock, of course; this doesn’t change the gist of the answer—now the action will depend on the clock as well as on the non-clock part of the state—but it does mean that the agent can never be in the same state twice. © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Section 2.2

Good Behavior: The Concept of Rationality

Exercise 2.2.#VACR Let us examine the rationality of various vacuum-cleaner agent functions. a. Show that the simple vacuum-cleaner agent function described in Figure 2.3 is indeed rational under the assumptions listed on page 40. b. Describe a rational agent function for the case in which each movement costs one point. Does the corresponding agent program require internal state? c. Discuss possible agent designs for the cases in which clean squares can become dirty and the geography of the environment is unknown. Does it make sense for the agent to learn from its experience in these cases? If so, what should it learn? If not, why not?

Notice that for our simple environmental assumptions we need not worry about quantitative uncertainty. a. It suffices to show that for all possible actual environments (i.e., all dirt distributions and initial locations), this agent cleans the squares at least as fast as any other agent. This is trivially true when there is no dirt. When there is dirt in the initial location and none in the other location, the world is clean after one step; no agent can do better. When there is no dirt in the initial location but dirt in the other, the world is clean after two steps; no agent can do better. When there is dirt in both locations, the world is clean after three steps; no agent can do better. (Note: in general, the condition stated in the first sentence of this answer is much stricter than necessary for an agent to be rational.) b. The agent in (a) keeps moving backwards and forwards even after the world is clean. It is better to do NoOp once the world is clean (the chapter says this). Now, since the agent’s percept doesn’t say whether the other square is clean, it would seem that the agent must have some memory to say whether the other square has already been cleaned. To make this argument rigorous is more difficult—for example, could the agent arrange things so that it would only be in a clean left square when the right square was already clean? As a general strategy, an agent can use the environment itself as a form of external memory—a common technique for humans who use things like appointment calendars and knots in handkerchiefs. In this particular case, however, that is not possible. Consider the reflex actions for [A, Clean] and [B, Clean]. If either of these is NoOp, then the agent will fail in the case where that is the initial percept but the other square is dirty; hence, neither can be NoOp and therefore the simple reflex agent is doomed to keep moving. In general, the problem with reflex agents is that they have to do the same thing in situations that look the same, even when the situations are actually quite different. In the vacuum world this is a big liability, because every interior square (except home) looks either like a square with dirt or a square without dirt. c. If we consider asymptotically long lifetimes, then it is clear that learning a map (in some form) confers an advantage because it means that the agent can avoid bumping into walls. It can also learn where dirt is most likely to accumulate and can devise an optimal inspection strategy. The precise details of the exploration method needed to construct a complete map appear in Chapter 4; methods for deriving an optimal © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

External memory

Exercises 2

Intelligent Agents

inspection/cleanup strategy are in Chapter 22. Exercise 2.2.#KPER Describe three different task environments in which the performance measure is easy to specify completely and correctly, and three in which it is not. Obviously there are many possible answers here. For the “easy” case, it makes sense to consider artificially defined task environments: electronic calculators (answers correct to the required number of significant digits, elapsed time), chess programs (win/draw/loss subject to time constraints), spider solitaire (number of suits completed, number of moves, elapsed time). But even in these environments, one must be careful, because such performance measures ignore all other considerations. For example, a general-purpose intelligent agent that has the ability to display messages or access the internet might improve its chess performance by acquiring additional hardware. Marvin Minsky pointed out that a machine designed to calculate as many digits of π as possible could take over the world to do so. For the “hard” case, almost any task involving humans will do: tutor, automated taxi, digital personal assistant, military strategist, and so on. Although Section 2.3 lists the elements of the taxi’s performance measure (reaching destination, fuel consumption, wear and tear, trip time, traffic laws, politness to other drivers, safety, passenger comfort, profit), the appropriate tradeoffs among these are far from clear. For example, if the taxi is 200 yards from the destination but stuck in a traffic jam that will take 20 minutes to clear, should it suggest to the passengers that they walk the remaining 200 yards? And how does this decision depend on the weather and general safety and legality of this premature dropoff location? In addition to difficult tradeoffs, there will typically be attributes that are omitted from the performance measure that do in fact matter and can be affected by the agent. Obviously it doesn’t make sense to ask students to list the attributes that are omitted, because they could simply include them! Instead, one might ask which attributes are “obvious” and which are “non-obvious,” i.e., likely to be omitted on a first or second pass. Exercise 2.2.#EVSA Write an essay on the relationship between evolution and one or more of autonomy, intelligence, rationality, and learning.

Exercise 2.2.#AGTF For each of the following assertions, say whether it is true or false and support your answer with examples or counterexamples where appropriate. a. An agent that senses only partial information about the state cannot be perfectly rational. b. There exist task environments in which no pure reflex agent can behave rationally. c. There exists a task environment in which every agent is rational. d. The input to an agent program is the same as the input to the agent function.

Section 2.2

Good Behavior: The Concept of Rationality

e. Every agent function is implementable by some program/machine combination. f. Suppose an agent selects its action uniformly at random from the set of possible actions. There exists a deterministic task environment in which this agent is rational. g. It is possible for a given agent to be perfectly rational in two distinct task environments. h. Every agent is rational in an unobservable environment. i. A perfectly rational poker-playing agent never loses. j. For a simple reflex agent in a partially observable environment, a randomized policy can outperform any deterministic policy. k. There is a model-based reflex agent that can remember all of its percepts. l. Suppose agent A1 is rational and agent A2 is irrational. There exists a task environment where A2’s actual score will be greater than A1’s actual score.

a. An agent that senses only partial information about the state cannot be perfectly rational. False. Perfect rationality refers to the ability to make good decisions given the sensor information received. Moreover, in some environments, an agent can deliberately ignore part of its sensor information and still be perfectly rational. For example, a physical chess agent can ignore flecks of dust on the chessboard and pretty much anything that isn’t on the chessboard or the clock. b. There exist task environments in which no pure reflex agent can behave rationally. True. A pure reflex agent ignores previous percepts, so cannot obtain an optimal state estimate in a partially observable environment. For example, correspondence chess is played by sending moves; if the other player’s move is the current percept, a reflex agent could not keep track of the board state and would have to respond to, say, “a4” in the same way regardless of the position in which it was played. c. There exists a task environment in which every agent is rational. True. For example, in an environment with a single state, such that all actions have the same reward, it doesn’t matter which action is taken. More generally, any environment that is reward-invariant under permutation of the actions will satisfy this property. d. The input to an agent program is the same as the input to the agent function. False. The agent function, notionally speaking, takes as input the entire percept sequence up to that point, whereas the agent program takes the current percept only. e. Every agent function is implementable by some program/machine combination. False. For example, the environment may contain Turing machines and input tapes and the agent’s job is to solve the halting problem; there is an agent function that specifies the right answers, but no agent program can implement it. Another example would be an agent function that requires solving intractable problem instances of arbitrary size in constant time. f. Suppose an agent selects its action uniformly at random from the set of possible actions. There exists a deterministic task environment in which this agent is rational. True. This is a special case of (c); if it doesn’t matter which action you take, selecting © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 2

Intelligent Agents

randomly is rational. g. It is possible for a given agent to be perfectly rational in two distinct task environments. True. For example, we can arbitrarily modify the parts of the environment that are unreachable by any optimal policy as long as they stay unreachable. h. Every agent is rational in an unobservable environment. False. Some actions are stupid—and the agent may know this if it has a model of the environment—even if one cannot perceive the environment state. i. A perfectly rational poker-playing agent never loses. False. Unless it draws the perfect hand, the agent can always lose if an opponent has better cards. This can happen for game after game. The correct statement is that the agent’s expected winnings are nonnegative. j. For a simple reflex agent in a partially observable environment, a randomized policy can outperform any deterministic policy. True. Some vacuum robots use this idea to get themselves “unstuck” from corners. A partially observable environment can have hidden state that makes any particular deterministically chosen action fail when executed in response to a given percept, whereas a randomized policy may eventually chose the right action (if there is one). As an example, consider a reflex agent that needs the PIN to ulock a phone. A randomized agent will eventually emit the correct sequence. whereas a deterministic agent can only emit the same sequence over and over. k. There is a model-based reflex agent that can remember all of its percepts. True. A model-based agent updates an internal state representation—its “memory”— for each new percept. Memorizing the percepts is possible (assuming unbounded memory, which is a reasonable caveat). This may not be the most perspicuous representation of the current state of the world, but any other such representation of the current state could be computed from it on demand, for example by running any other model-based reflex agent’s state estimation algorithm on the memorized percept sequence. l. Suppose agent A1 is rational and agent A2 is irrational. There exists a task environment where A2’s actual score will be greater than A1’s actual score. True. Rational decisions are defined by expected outcomes, not actual outcomes. A1 might just be unlucky.

2.3 The Nature of Environments Exercise 2.3.#PEAS For each of the following activities, give a PEAS description of the task environment and characterize it in terms of the properties listed in Section 2.3.2. • Playing soccer. • Exploring the subsurface oceans of Titan. • Shopping for used AI books on the Internet. • Playing a tennis match.

Section 2.3

The Nature of Environments

• Practicing tennis against a wall. • Performing a high jump. • Knitting a sweater. • Bidding on an item at an auction.

Many of these can actually be argued either way, depending on the level of detail and abstraction. A. Partially observable, stochastic, sequential, dynamic, continuous, multi-agent. B. Partially observable, stochastic, sequential, dynamic, continuous, single agent (unless there are alien life forms that are usefully modeled as agents). C. Partially observable, deterministic, sequential, static, discrete, single agent. This can be multi-agent and dynamic if we buy books via auction, or dynamic if we purchase on a long enough scale that book offers change. D. Fully observable, stochastic, episodic (every point is separate), dynamic, continuous, multi-agent. E. Fully observable, stochastic, episodic, dynamic, continuous, single agent. F. Fully observable, stochastic, sequential, static, continuous, single agent. G. Fully observable, deterministic, sequential, static, continuous, single agent. H. Fully observable, strategic, sequential, static, discrete, multi-agent. Exercise 2.3.#VCES Implement a performance-measuring environment simulator for the vacuum-cleaner world depicted in Figure 2.2 and specified on page 40. Your implementation should be modular so that the sensors, actuators, and environment characteristics (size, shape, dirt placement, etc.) can be changed easily. (Note: for some choices of programming language and operating system there are already implementations in the online code repository.) The code repository implements a vacuum-cleaner environment. Students can easily extend it to generate different shaped rooms, obstacles, dirt generation processes, sensor suites, and so on. Exercise 2.3.#ENVP For each of the following task environment properties, rank the example task environments from most to least according to how well the environment satisfies the property. Lay out any assumptions you make to reach your conclusions. a. Fully Observable: driving; document classification; tutoring a high-school student in calculus; skin cancer diagnosis from images.

Exercises 2

Intelligent Agents

b. Continuous: driving; spoken conversation; written conversation; climate engineering by stratospheric aerosol injection. c. Stochastic: driving; sudoku; poker; soccer. d. Static: chat room; checkers; tax planning; tennis.

a. Fully Observable: document classification > skin cancer diagnosis from images > driving > tutoring a high-school student in calculus. Document classification is a fairly canonical example of a (non-sequential) observable problem, because the correct classification depends almost entirely on the visible text of the document itself. There might be a slight influence from “provenance” information (date, authorship, etc.) that may not be directly observable. Skin cancer diagnosis can sometimes be done well from an image of the lesion, bot other factors such as patient age, changes in the lesion over time, medical history, and family history can be important. Driving is often considered to be observable because we imagine that we are making decisions based on what we see, but (1) velocity and turn signal status of other vehicles can be judged only from multiple image frames, and (2) assessing the intended actions of other vehicles may require accumulating information over an extended period—e.g., to determine if a vehicle is stopped or broken down, driving slowly or looking for an address or a parking spot, turning left or has forgotten to turn off the turn signal. Other vehicles, hedges, fog, and so on can obscure visual access to important aspects of the driving environment. Tutoring is almost completely unobservable: what matters is the student’s level of understanding, learning style, basic math skills, etc. Clues must be gathered over days, weeks, and months. b. Continuous: climate engineering > driving > spoken conversation > written conversation. Climate engineering by aerosol injection is quintessentially continuous: the engineer must decide how much to inject, where, and when, and all of these are continuous quantities. The control actions of driving are mostly continuous (steering, acceleration/braking) but there are discrete elements (turn signal, headlights). More importantly, the problem is usually handled using discrete high-level actions (change lanes left, take exit, etc.) that have implementations as continuous control problems. This kind of discrete/continuous hierarchy is very common; playing chess in the physical world is a perfect example. Spoken conversation is closer to chess than driving: roughly speaking, we choose the discrete words to say and delegate the saying to continuous motor control routines. Prosody (volume, pitch, and speed variation) is, however, an important continuous element in how we speak that is largely absent from written communication. c. Stochastic: poker > soccer > driving > sudoku. In poker, nearly everything is determined by the fall of the cards, which is entirely stochastic from the viewpoint of the players. Both soccer and driving contain elements that are fairly deterministic, such as the flight of the ball and the response of the engine, and elements that are stochastic, such as tire punctures and the outcomes of tackles. Yet typically one can make reasonably reliable driving plans over many minutes, whereas it © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Section 2.4

The Structure of Agents

is essentially impossible to predict the state of a soccer game one minute into the future. Sudoku, of course, is entirely deterministic. d. Static: tax planning > checkers > chat room > tennis. While no human activity is completely static, given the finite length of our lifetimes, tax planning comes close—the typical “deadline” to get it done is often weeks or months, and the relevant aspects of the environment (life/death, number of offspring, tax law) may change even more slowly. In checkers, the world state doesn’t change until someone moves, but the clock ticks so the problem is semi-dynamic. In the chat room, long delays in replying are unacceptable, so it is a fairly real-time environment, but not nearly as real-time as tennis, where a delay of a split second often makes the difference between winning and losing a point.

2.4 The Structure of Agents Exercise 2.4.#SIRA Implement a simple reflex agent for the vacuum environment in Exercise VACUUM START- EXERCISE . Run the environment with this agent for all possible initial dirt configurations and agent locations. Record the performance score for each configuration and the overall average score. Pseudocode for a reflex agent is given in Figure 2.8. For states 1, 3, 5, 7 in Figure 4.9, the performance measures are 1996, 1999, 1998, 2000 respectively. The average is 1998.25. Exercise 2.4.#VCPE Consider a modified version of the vacuum environment in Exercise VACUUM - STARTEXERCISE , in which the agent is penalized one point for each movement. a. Can a simple reflex agent be perfectly rational for this environment? Explain. b. What about a reflex agent with state? Design such an agent. c. How do your answers to a and b change if the agent’s percepts give it the clean/dirty status of every square in the environment?

a. No. Consider the reflex actions for [A, Clean] and [B, Clean]. If either of these is NoOp, then the agent will fail in the case where that is the initial percept but the other square is dirty; hence, neither can be NoOp and therefore the simple reflex agent is doomed to keep moving, which loses points unnecessarily. In general, the problem with reflex agents is that they have to do the same thing in situations that look the same, even when the situations are actually quite different. In the vacuum world this is a big liability, because every interior square (except home) looks either like a square with dirt or a square without dirt. b. Yes, a reflex agent with state can be perfectly rational. It simply needs to remember whether it has visited the other square. If so, it can do nothing after cleaning the current © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 2

Intelligent Agents

square (if needed). If not, it then goes to the other square and cleans it if needed. c. In this case, a simple reflex agent can be perfectly rational. The agent can consist of a table with eight entries, indexed by percept, that specifies an action to take for each possible state. After the agent acts, the world is updated and the next percept will tell the agent what to do next. For larger environments, constructing a table is infeasible. Instead, the agent could run one of the optimal search algorithms in Chapter 3 and execute the first step of the solution sequence. Again, no internal state is required, but it would help to be able to store the solution sequence instead of recomputing it for each new percept. Exercise 2.4.#VACU Consider a modified version of the vacuum environment in Exercise VACUUM - STARTEXERCISE , in which the geography of the environment—its extent, boundaries, and obstacles—is unknown, as is the initial dirt configuration. (The agent can go Up and Down as well as Left and Right.) a. Can a simple reflex agent be perfectly rational for this environment? Explain. b. Can a simple reflex agent with a randomized agent function outperform a simple reflex agent? Design such an agent and measure its performance on several environments. c. Can you design an environment in which your randomized agent will perform poorly? Show your results. d. Can a reflex agent with state outperform a simple reflex agent? Design such an agent and measure its performance on several environments. Can you design a rational agent of this type?

a. Because the agent does not know the geography and perceives only location and local dirt, and cannot remember what just happened, it will get stuck forever against a wall when it tries to move in a direction that is blocked—that is, unless it randomizes. b. One possible design cleans up dirt and otherwise moves in a randomly chosen direction. This is fairly close to what the early-model RoombaTM vacuum cleaner does, although the Roomba has a bump sensor and randomizes only when it hits an obstacle. It works reasonably well but it can take a long time to cover all the squares at least once. Many modern robot cleaners build a map and derive an efficient cleaning pattern. c. An example is shown in Figure S2.1. Students may also wish to measure clean-up time for linear or square environments of different sizes, and compare those to the efficient online search algorithms described in Chapter 4. It’s worth noting that in the infinitetime limit on a bidirectional graph, an agent that moves randomly will spend time in each square proportional to its degree in the graph (the number of neighbors). Still, it can take a long time to get from one part of the graph to another. If the agent is in a √ long corridor, its expected travel after n time steps is O( n) squares. d. A reflex agent with state can build a map (see Chapter 4 for details). An online depthfirst exploration will reach every state in time linear in the size of the environment; therefore, the agent can do much better than the simple reflex agent. © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Section 2.4

The Structure of Agents

Figure S2.1 An environment in which random motion will take a long time to cover all the squares.

The question of rational behavior in unknown environments is a complex one but it is worth encouraging students to think about it. We need to have some notion of the prior probability distribution over the class of environments; call this the initial belief state. Any action yields a new percept that can be used to update this distribution, moving the agent to a new belief state. Once the environment is completely explored, the belief state collapses to a single possible environment. Therefore, the problem of optimal exploration can be viewed as a search for an optimal strategy in the space of possible belief states. This is a well-defined, if horrendously intractable, problem. Chapter 17 discusses some cases (called bandit problems) where optimal exploration is possible. Another concrete example of exploration is the Minesweeper computer game (see Exercise 7.22). For very small Minesweeper environments, optimal exploration is feasible although the belief state update is nontrivial to explain. Exercise 2.4.#VACB Repeat Exercise VACUUM - UNKNOWN - GEOG - EXERCISE for the case in which the location sensor is replaced with a “bump” sensor that detects the agent’s attempts to move into an obstacle or to cross the boundaries of the environment. Suppose the bump sensor stops working; how should the agent behave? The problem appears at first to be very similar; the main difference is that instead of using the location percept to build the map, the agent has to “invent” its own locations (which, after all, are just nodes in a data structure representing the state space graph). When a bump is detected, the agent assumes it remains in the same location and can add a wall to its map. For grid environments, the agent can keep track of its (x, y) location and so can tell when it has © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 2

Intelligent Agents

returned to an old state. In the general case, however, there is no simple way to tell if a state is new or old. Exercise 2.4.#VFIN The vacuum environments in the preceding exercises have all been deterministic. Discuss possible agent programs for each of the following stochastic versions: a. Murphy’s law: twenty-five percent of the time, the Suck action fails to clean the floor if it is dirty and deposits dirt onto the floor if the floor is clean. How is your agent program affected if the dirt sensor gives the wrong answer 10% of the time? b. Small children: At each time step, each clean square has a 10% chance of becoming dirty. Can you come up with a rational agent design for this case?

a. For a reflex agent, this presents no additional challenge, because the agent will continue to Suck as long as the current location remains dirty. For an agent that constructs a sequential plan, every Suck action would need to be replaced by “Suck until clean.” If the dirt sensor can be wrong on each step, then the agent might want to wait for a few steps to get a more reliable measurement before deciding whether to Suck or move on to a new square. Obviously, there is a trade-off because waiting too long means that dirt remains on the floor (incurring a penalty), but acting immediately risks either dirtying a clean square or ignoring a dirty square (if the sensor is wrong). A rational agent must also continue touring and checking the squares in case it missed one on a previous tour (because of bad sensor readings). It is not immediately obvious how the waiting time at each square should change with each new tour. These issues can be clarified by experimentation, which may suggest a general trend that can be verified mathematically. This problem is a partially observable Markov decision process—see Chapter 17. Such problems are hard in general, but some special cases may yield to careful analysis. b. In this case, the agent must keep touring the squares indefinitely. The probability that a square is dirty increases monotonically with the time since it was last cleaned, so the rational strategy is, roughly speaking, to repeatedly execute the shortest possible tour of all squares. (We say “roughly speaking” because there are complications caused by the fact that the shortest tour may visit some squares twice, depending on the geography.) This problem is also a partially observable Markov decision process. Exercise 2.4.#DFRA Define in your own words the following terms: rationality, autonomy, reflex agent, model-based agent, goal-based agent, utility-based agent, learning agent. The following are just some of the many possible definitions that can be written: • Rationality: a property of agents that choose actions that maximize their expected utility, given the percepts to date. © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Section 2.4

The Structure of Agents

• Autonomy: a property of agents whose behavior is determined by their own experience rather than solely by their initial programming. • Reflex agent: an agent whose action depends only on the current percept. • Model-based agent: an agent whose action is derived directly from an internal model of the current world state that is updated over time. • Goal-based agent: an agent that selects actions that it believes will achieve explicitly represented goals. • Utility-based agent: an agent that selects actions that it believes will maximize the expected utility of the outcome state. • Learning agent: an agent whose behavior improves over time based on its experience. Exercise 2.4.#GBUT Write pseudocode agent programs for the goal-based and utility-based agents. The design of goal- and utility-based agents depends on the structure of the task environment. The simplest such agents, for example those in chapters 3 and 10, compute the agent’s entire future sequence of actions in advance before acting at all. This strategy works for static and deterministic environments which are either fully-known or unobservable For fully-observable and fully-known static environments a policy can be computed in advance which gives the action to by taken in any given state. For partially-observable environments the agent can compute a conditional plan, which specifies the sequence of actions to take as a function of the agent’s perception. In the extreme, a conditional plan gives the agent’s response to every contingency, and so it is a representation of the entire agent function. In all cases it may be either intractable or too expensive to compute everything out in advance. Instead of a conditional plan, it may be better to compute a single sequence of actions which is likely to reach the goal, then monitor the environment to check whether the plan is succeeding, repairing or replanning if it is not. It may be even better to compute only the start of this plan before taking the first action, continuing to plan at later time steps. Pseudocode for simple goal-based agent is given in Figure S2.2. G OAL -ACHIEVED tests to see whether the current state satisfies the goal or not, doing nothing if it does. P LAN computes a sequence of actions to take to achieve the goal. This might return only a prefix of the full plan, the rest will be computed after the prefix is executed. This agent will act to maintain the goal: if at any point the goal is not satisfied it will (eventually) replan to achieve the goal again. At this level of abstraction the utility-based agent is not much different than the goalbased agent, except that action may be continuously required (there is not necessarily a point where the utility function is “satisfied”). Pseudocode is given in Figure S2.3. Exercise 2.4.#FURN Consider a simple thermostat that turns on a furnace when the temperature is at least 3 degrees below the setting, and turns off a furnace when the temperature is at least 3 degrees

Exercises 2

Intelligent Agents

function G OAL -BASED -AGENT( p̌ercept) returns an action persistent: štate, the agent’s current conception of the world state m̌odel, a description of how the next state depends on current state and action ǧoal, a description of the desired goal state p̌lan, a sequence of actions to take, initially empty ǎction, the most recent action, initially none štate ← U PDATE -S TATE(štate, ǎction, p̌ercept, m̌odel) if G OAL -ACHIEVED(štate,ǧoal) then return a null action if p̌lan is empty then p̌lan ← P LAN(štate,ǧoal,m̌odel) ǎction ← F IRST(p̌lan) p̌lan ← R EST(p̌lan) return ǎction Figure S2.2 A goal-based agent.

function U TILITY-BASED -AGENT( p̌ercept) returns an action persistent: štate, the agent’s current conception of the world state m̌odel, a description of how the next state depends on current state and action ǔtility-function, a description of the agent’s utility function p̌lan, a sequence of actions to take, initially empty ǎction, the most recent action, initially none štate ← U PDATE -S TATE(štate, ǎction, p̌ercept, m̌odel) if p̌lan is empty then p̌lan ← P LAN(štate,ǔtility-function,m̌odel) ǎction ← F IRST(p̌lan) p̌lan ← R EST(p̌lan) return ǎction Figure S2.3 A utility-based agent.

above the setting. Is a thermostat an instance of a simple reflex agent, a model-based reflex agent, or a goal-based agent? The thermostat is best understood as a simple reflex agent. Although the temperature setting might be viewed as a goal, the agent has no notion of how its actions will lead to the goal; so it’s better to view the temperature setting as part of the agent’s percepts. A more complex control system might well have an internal model of the house, the behavior of its occupants, the capabilities of the heating system, and so on, and would be viewed as a goal-based system in much the same way as a self-driving car that tries to reach a specified © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Section 2.4

The Structure of Agents

destination quickly and cheaply. Exercise 2.4.#AGPR Here is pseudocode for three agent programs A, B, C:

function A(percept) return fA ()

function B(percept) return fB (percept)

function C(percept) persistent: percepts, initially [] percepts ← push(percept,percepts) return fC (percepts)

In each of these agents, the function f is some arbitrary, possibly randomized, function of its inputs with no internal state of its own; the agent program runs on a computer with unbounded memory but finite clock speed. We’ll assume also that the environment and its performance measure are computable. a. Suppose the environment is fully observable, deterministic, discrete, single-agent, and static. For which agents, if any, is it the case that, for every such environment, there is some way to choose f such that the agent is perfectly rational? b. Suppose the environment is partially observable, deterministic, discrete, single-agent, and static. For which agents, if any, is it the case that, for every such environment, there is some way to choose f such that the agent is perfectly rational? c. Suppose the environment is partially observable, stochastic, discrete, single-agent, and dynamic. For which agents, if any, is it the case that, for every such environment, there is some way to choose f such that the agent is perfectly rational?

a. B, C. For a fully observable environment, only the current percept is required for an optimal decision. Because the environment is static, computation is not an issue. Note that Agent A cannot make optimal decisions because it always makes the same decision (or samples a decision from the same probability distribution), having no internal state. b. C. Agent B, the reflex agent, cannot always function optimally in a partially observable environment because it ignores previous percepts and therefore fails to take into account relevant information for the current decision. c. None of the agents can be optimal for an arbitrary dynamic environment, because we can make the environment complex enough to render optimal decisions infeasible for any finite-speed machine.

EXERCISES

SOLVING PROBLEMS BY SEARCHING 3.1 Problem-Solving Agents Exercise 3.1.#FORM Explain why problem formulation must follow goal formulation.

In goal formulation, we decide which aspects of the world we are interested in, and which can be ignored or abstracted away. Then in problem formulation we decide how to manipulate the important aspects (and ignore the others). If we did problem formulation first we would not know what to include and what to leave out. That said, it can happen that there is a cycle of iterations between goal formulation, problem formulation, and problem solving until one arrives at a sufficiently useful and efficient solution.

3.2 Example Problems Exercise 3.2.#PRFO Give a complete problem formulation for each of the following problems. Choose a formulation that is precise enough to be implemented. a. There are six glass boxes in a row, each with a lock. Each of the first five boxes holds a key unlocking the next box in line; the last box holds a banana. You have the key to the first box, and you want the banana. b. You start with the sequence ABABAECCEC, or in general any sequence made from A, B, C, and E. You can transform this sequence using the following equalities: AC = E, AB = BC, BB = E, and Ex = x for any x. For example, ABBC can be transformed into AEC (using BB = E), and then AC (using Ex = x), and then E (using AC = E). Your goal is to produce the sequence E. c. There is an n × n grid of squares, each square initially being either unpainted floor or a bottomless pit. You start standing on an unpainted floor square, and can either paint the square under you or move onto an adjacent unpainted floor square. You want the whole floor painted. d. A container ship is in port, loaded high with containers. There 13 rows of containers, each 13 containers wide and 5 containers tall. You control a crane that can move to any

Section 3.2

Example Problems

location above the ship, pick up the container under it, and move it onto the dock. You want the ship unloaded.

a. Initial state: as described in the question. Goal test: you have banana. Successor function: open any box you have the key for, get the contents of any open box. Cost function: number of actions. b. Initial state: ABABAECCEC. Goal test: is the state E Successor function: apply an equality substituting one subsequence for the other. Cost function: number of transformations. c. Initial state: all floor squares unpainted, you start standing on one square unpainted floor square. Goal test: all floor squares painted. Successor function: paint current tile, move to adjacent unpainted floor tile. Cost function: number of moves. d. Initial state: all containers stacked on the ship. Goal test: all containers unloaded. Successor function: move crane to a certain location, pick up a container, put down container. Cost function: time taken to unload ship.

Exercise 3.2.#RMAZ Your goal is to navigate a robot out of a maze. The robot starts in the center of the maze facing north. You can turn the robot to face north, east, south, or west. You can direct the robot to move forward a certain distance, although it will stop before hitting a wall. a. Formulate this problem. How large is the state space? b. In navigating a maze, the only place we need to turn is at the intersection of two or more corridors. Reformulate this problem using this observation. How large is the state space now? c. From each point in the maze, we can move in any of the four directions until we reach a turning point, and this is the only action we need to do. Reformulate the problem using these actions. Do we need to keep track of the robot’s orientation now? d. In our initial description of the problem we already abstracted from the real world, restricting actions and removing details. List three such simplifications we made.

Exercises 3

Solving Problems by Searching

a. We’ll define the coordinate system so that the center of the maze is at (0, 0), and the maze itself is a square from (−1, −1) to (1, 1). Initial state: robot at coordinate (0, 0), facing North. Goal test: either |x| > 1 or |y| > 1 where (x, y) is the current location. Successor function: move forwards any distance d; change direction robot it facing. Cost function: total distance moved. The state space is infinitely large, since the robot’s position is continuous. b. The state will record the intersection the robot is currently at, along with the direction it’s facing. At the end of each corridor leaving the maze we will have an exit node. We’ll assume some node corresponds to the center of the maze. Initial state: at the center of the maze facing North. Goal test: at an exit node. Successor function: move to the next intersection in front of us, if there is one; turn to face a new direction. Cost function: total distance moved. There are 4n states, where n is the number of intersections. c. Initial state: at the center of the maze. Goal test: at an exit node. Successor function: move to next intersection to the North, South, East, or West. Cost function: total distance moved. We no longer need to keep track of the robot’s orientation since it is irrelevant to predicting the outcome of our actions, and not part of the goal test. The motor system that executes this plan will need to keep track of the robot’s current orientation, to know when to rotate the robot. d. State abstractions: (i) Ignoring the height of the robot off the ground, whether it is tilted off the vertical. (ii) The robot can face in only four directions. (iii) Other parts of the world ignored: possibility of other robots in the maze, the weather in the Caribbean. Action abstractions: (i) We assumed all positions we safely accessible: the robot couldn’t get stuck or damaged. (ii) The robot can move as far as it wants, without having to recharge its batteries. (iii) Simplified movement system: moving forwards a certain distance, rather than controlled each individual motor and watching the sensors to detect collisions.

Section 3.5

Informed (Heuristic) Search Strategies

3.5 Informed (Heuristic) Search Strategies 3.6 Heuristic Functions Exercise 3.6.#GRRB You have a 9 × 9 grid of squares, each of which can be colored red or blue. The grid is initially colored all blue, but you can change the color of any square any number of times. Imagining the grid divided into nine 3 × 3 sub-squares, you want each sub-square to be all one color but neighboring sub-squares to be different colors. a. Formulate this problem in the straightforward way. Compute the size of the state space. b. You need color a square only once. Reformulate, and compute the size of the state space. Would breadth-first graph search perform faster on this problem than on the one in (a)? How about iterative deepening tree search? c. Given the goal, we need consider only colorings where each sub-square is uniformly colored. Reformulate the problem and compute the size of the state space. d. How many solutions does this problem have? e. Parts (b) and (c) successively abstracted the original problem (a). Can you give a translation from solutions in problem (c) into solutions in problem (b), and from solutions in problem (b) into solutions for problem (a)?

a. Initial state: all squares colored blue. Goal test: each sub-square colored the same, and adjacent subsquares colored different. Successor function: color a square red or blue. Cost function: number of times squares are colored. There are 281 states. b. To avoid recoloring squares we could either record squares that have been colored, and not allow them to be colored again, or color all the squares in a fixed order. We describe the latter: Initial state: all squares colored blue, the top-left square is next to be colored. Goal test: each sub-square colored the same, and adjacent subsquares colored different. Successor function: if we haven’t colored all squares yet, color the current square red or blue (this automatically updates the next square to be colored). Cost function: number of squares colored. There are 82 · 281 states, since we need to record whether there are squares left to be colored, and if so which square is next. c. Initial state: all sub-squares colored blue, the top-left sub-square is next to be colored. Goal test: adjacent sub-squares colored differently. Successor function: if we haven’t colored all sub-squares yet, color the current subsquare red or blue. Cost function: number of sub-squares colored. © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 3

Solving Problems by Searching

that is not attacked by any other queen. To simplify matters, we’ll first consider the n–rooks problem. The first rook can be placed in any square in column 1 (n choices), the second in any square in column 2 except the same row that as the rook in column 1 (n − 1 choices), and so on. This gives n! elements of the search space. For n queens, notice that a queen attacks at most three squares in any given column, so in column 2 there are at least (n − 3) choices, in column at least (n − 6) choices, and so on. Thus the state space size S ≥ n · (n − 3) · (n − 6) · · · . Hence we have S 3 ≥ n · n · n · (n − 3) · (n − 3) · (n − 3) · (n − 6) · (n − 6) · (n − 6) · · · · ≥ n · (n − 1) · (n − 2) · (n − 3) · (n − 4) · (n − 5) · (n − 6) · (n − 7) · (n − 8) · · · · = n! √ or S ≥ 3 n!. Exercise 3.6.#COMP Give a complete problem formulation for each of the following. Choose a formulation that is precise enough to be implemented. a. Using only four colors, you have to color a planar map in such a way that no two adjacent regions have the same color. b. A 3-foot-tall monkey is in a room where some bananas are suspended from the 8-foot ceiling. The monkey would like to get the bananas. The room contains two stackable, movable, climbable 3-foot-high crates. c. You have a program that outputs the message “illegal input record” when fed a certain file of input records. You know that processing of each record is independent of the other records. You want to discover what record is illegal. d. You have three jugs, measuring 12 gallons, 8 gallons, and 3 gallons, and a water faucet. You can fill the jugs up or empty them out from one to another or onto the ground. You need to measure out exactly one gallon.

a. Initial state: No regions colored. Goal test: All regions colored, and no two adjacent regions have the same color. Successor function: Assign a color to a region. Cost function: Number of assignments. b. Initial state: As described in the text. Goal test: Monkey has bananas. Successor function: Hop on crate; Hop off crate; Push crate from one spot to another; Walk from one spot to another; grab bananas (if standing on crate). Cost function: Number of actions. c. Initial state: considering all input records. Goal test: considering a single record, and it gives “illegal input” message. Successor function: run again on the first half of the records; run again on the second half of the records. Cost function: Number of runs. © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Section 3.6

Heuristic Functions

Note: This is a contingency problem; you need to see whether a run gives an error message or not to decide what to do next. d. Initial state: jugs have values [0, 0, 0]. Successor function: given values [x, y, z], generate [12, y, z], [x, 8, z], [x, y, 3] (by filling); [0, y, z], [x, 0, z], [x, y, 0] (by emptying); or for any two jugs with current values x and y, pour y into x; this changes the jug with x to the minimum of x + y and the capacity of the jug, and decrements the jug with y by by the amount gained by the first jug. Cost function: Number of actions.

Exercise 3.6.#PPPO Consider the problem of finding the shortest path between two points on a plane that has convex polygonal obstacles as shown in Figure ??. This is an idealization of the problem that a robot has to solve to navigate in a crowded environment. a. Suppose the state space consists of all positions (x, y) in the plane. How many states are there? How many paths are there to the goal? b. Explain briefly why the shortest path from one polygon vertex to any other in the scene must consist of straight-line segments joining some of the vertices of the polygons. Define a good state space now. How large is this state space? c. Define the necessary functions to implement the search problem, including an ACTIONS function that takes a vertex as input and returns a set of vectors, each of which maps the current vertex to one of the vertices that can be reached in a straight line. (Do not forget the neighbors on the same polygon.) Use the straight-line distance for the heuristic function. d. Apply one or more of the algorithms in this chapter to solve a range of problems in the domain, and comment on their performance.

a. If we consider all (x, y) points, then there are an infinite number of states, and of paths. b. (For this problem, we consider the start and goal points to be vertices.) The shortest distance between two points is a straight line, and if it is not possible to travel in a straight line because some obstacle is in the way, then the next shortest distance is a sequence of line segments, end-to-end, that deviate from the straight line by as little as possible. So the first segment of this sequence must go from the start point to a tangent point on an obstacle – any path that gave the obstacle a wider girth would be longer. Because the obstacles are polygonal, the tangent points must be at vertices of the obstacles, and hence the entire path must go from vertex to vertex. So now the state space is the set of vertices, of which there are 35 in Figure ??. c. Code not shown. d. Implementations and analysis not shown.

Exercises 3

Solving Problems by Searching

Exercise 3.6.#NNEG We said in the chapter that we would not consider problems with negative path costs. In this exercise, we explore this decision in more depth. a. Suppose that actions can have arbitrarily large negative costs; explain why this possibility would force any optimal algorithm to explore the entire state space. b. Does it help if we insist that step costs must be greater than or equal to some negative constant c? Consider both trees and graphs. c. Suppose that a set of actions forms a loop in the state space such that executing the set in some order results in no net change to the state. If all of these actions have negative cost, what does this imply about the optimal behavior for an agent in such an environment? d. One can easily imagine actions with high negative cost, even in domains such as route finding. For example, some stretches of road might have such beautiful scenery as to far outweigh the normal costs in terms of time and fuel. Explain, in precise terms, within the context of state-space search, why humans do not drive around scenic loops indefinitely, and explain how to define the state space and actions for route finding so that artificial agents can also avoid looping. e. Can you think of a real domain in which step costs are such as to cause looping?

a. Any path, no matter how bad it appears, might lead to an arbitrarily large reward (negative cost). Therefore, one would need to exhaust all possible paths to be sure of finding the best one. b. Suppose the greatest possible reward is c. Then if we also know the maximum depth of the state space (e.g. when the state space is a tree), then any path with d levels remaining can be improved by at most cd, so any paths worse than cd less than the best path can be pruned. For state spaces with loops, this guarantee doesn’t help, because it is possible to go around a loop any number of times, picking up c reward each time. c. The agent should plan to go around this loop forever (unless it can find another loop with even better reward). d. The value of a scenic loop is lessened each time one revisits it; a novel scenic sight is a great reward, but seeing the same one for the tenth time in an hour is tedious, not rewarding. To accommodate this, we would have to expand the state space to include a memory—a state is now represented not just by the current location, but by a current location and a bag of already-visited locations. The reward for visiting a new location is now a (diminishing) function of the number of times it has been seen before. e. Real domains with looping behavior include eating junk food and going to class. Exercise 3.6.#MICA The missionaries and cannibals problem is usually stated as follows. Three missionaries and three cannibals are on one side of a river, along with a boat that can hold one or two people. Find a way to get everyone to the other side without ever leaving a group of mis-

Section 3.6

Heuristic Functions

sionaries in one place outnumbered by the cannibals in that place. This problem is famous in AI because it was the subject of the first paper that approached problem formulation from an analytical viewpoint (Amarel, 1968). a. Formulate the problem precisely, making only those distinctions necessary to ensure a valid solution. Draw a diagram of the complete state space. b. Implement and solve the problem optimally using an appropriate search algorithm. Is it a good idea to check for repeated states? c. Why do you think people have a hard time solving this puzzle, given that the state space is so simple?

a. Here is one possible representation: A state is a six-tuple of integers listing the number of missionaries, cannibals, and boats on the first side, and then the second side of the river. The goal is a state with 3 missionaries and 3 cannibals on the second side. The cost function is one per action, and the successors of a state are all the states that move 1 or 2 people and 1 boat from one side to another. b. The search space is small, so any optimal algorithm works. For an example, see the file aima-lisp/search/domains/cannibals.lisp. It suffices to eliminate moves that circle back to the state just visited. From all but the first and last states, there is only one other choice. c. It is not obvious that almost all moves are either illegal or revert to the previous state. There is a feeling of a large branching factor, and no clear way to proceed. Exercise 3.6.#DEFS Define in your own words the following terms: state, state space, search tree, search node, goal, action, transition model, and branching factor. A state is a situation that an agent can find itself in. We distinguish two types of states: world states (the actual concrete situations in the real world) and representational states (the abstract descriptions of the real world that are used by the agent in deliberating about what to do). A state space is a graph whose nodes are the set of all states, and whose links are actions that transform one state into another. A search tree is a tree (a graph with no undirected loops) in which the root node is the start state and the set of children for each node consists of the states reachable by taking any action. A search node is a node in the search tree. A goal is a state that the agent is trying to reach. An action is something that the agent can choose to do. A successor function described the agent’s options: given a state, it returns a set of (action, state) pairs, where each state is the state reachable by taking the action. The branching factor in a search tree is the number of actions available to the agent. © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 3

Solving Problems by Searching

Exercise 3.6.#STAT What’s the difference between a world state, a state description, and a search node? Why is this distinction useful? A world state is how reality is or could be. In one world state we’re in Arad, in another we’re in Bucharest. The world state also includes which street we’re on, what’s currently on the radio, and the price of tea in China. A state description is an agent’s internal description of a world state. Examples are In(Arad) and In(Bucharest). These descriptions are necessarily approximate, recording only some aspect of the state. We need to distinguish between world states and state descriptions because state description are lossy abstractions of the world state, because the agent could be mistaken about how the world is, because the agent might want to imagine things that aren’t true but it could make true, and because the agent cares about the world not its internal representation of it. Search nodes are generated during search, representing a state the search process knows how to reach. They contain additional information aside from the state description, such as the sequence of actions used to reach this state. This distinction is useful because we may generate different search nodes which have the same state, and because search nodes contain more information than a state representation. Exercise 3.6.#AABS An action such as going to Sibiu really consists of a long sequence of finer-grained actions: turn on the car, release the brake, accelerate forward, etc. Having composite actions of this kind reduces the number of steps in a solution sequence, thereby reducing the search time. Suppose we take this to the logical extreme, by making super-composite actions out of every possible sequence of Go actions. Then every problem instance is solved by a single supercomposite action, such as Go(Sibiu) Go(Rimnicu Vilcea) Go(Pitesti) Go(Bucharest). Explain how search would work in this formulation. Is this a practical approach for speeding up problem solving?

The state space is a tree of depth one, with all states successors of the initial state. There is no distinction between depth-first search and breadth-first search on such a tree. If the sequence length is unbounded the root node will have infinitely many successors, so only algorithms which test for goal nodes as we generate successors can work. What happens next depends on how the composite actions are sorted. If there is no particular ordering, then a random but systematic search of potential solutions occurs. If they are sorted by dictionary order, then this implements depth-first search. If they are sorted by length first, then dictionary ordering, this implements breadth-first search. A significant disadvantage of collapsing the search space like this is if we discover that a plan starting with the action “unplug your battery” can’t be a solution, there is no easy way to ignore all other composite actions that start with this action. This is a problem in particular for informed search algorithms. Discarding sequence structure is not a particularly practical approach to search. © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Section 3.6

Heuristic Functions

Exercise 3.6.#FINS Does a finite state space always lead to a finite search tree? How about a finite state space that is a tree? Can you be more precise about what types of state spaces always lead to finite search trees? (Adapted from Bender (1996).) No, a finite state space does not always lead to a finite search tree. Consider a state space with two states, both of which have actions that lead to the other. This yields an infinite search tree, because we can go back and forth any number of times. However, if the state space is a finite tree, or in general, a finite DAG (directed acyclic graph), then there can be no loops, and the search tree is finite. Exercise 3.6.#GRAS Prove that G RAPH -S EARCH satisfies the graph separation property illustrated in Figure 3.6. (Hint: Begin by showing that the property holds at the start, then show that if it holds before an iteration of the algorithm, it holds afterwards.) Describe a search algorithm that violates the property. The graph separation property states that “every path from the initial state to an unexplored state has to pass through a state in the frontier.” At the start of the search, the frontier holds the initial state; hence, trivially, every path from the initial state to an unexplored state includes a node in the frontier (the initial state itself). Now, we assume that the property holds at the beginning of an arbitrary iteration of the G RAPH -S EARCH algorithm in Figure ??. We assume that the iteration completes, i.e., the frontier is not empty and the selected leaf node n is not a goal state. At the end of the iteration, n has been removed from the frontier and its successors (if not already explored or in the frontier) placed in the frontier. Consider any path from the initial state to an unexplored state; by the induction hypothesis such a path (at the beginning of the iteration) includes at least one frontier node; except when n is the only such node, the separation property automatically holds. Hence, we focus on paths passing through n (and no other frontier node). By definition, the next node n0 along the path from n must be a successor of n that (by the preceding sentence) is already not in the frontier. Furthermore, n0 cannot be in the explored set, since by assumption there is a path from n0 to an unexplored node not passing through the frontier, which would violate the separation property as every explored node is connected to the initial state by explored nodes (see lemma below for proof this is always possible). Hence, n0 is not in the explored set, hence it will be added to the frontier; then the path will include a frontier node and the separation property is restored. The property is violated by algorithms that move nodes from the frontier into the explored set before all of their successors have been generated, as well as by those that fail to add some of the successors to the frontier. Note that it is not necessary to generate all successors of a node at once before expanding another node, as long as partially expanded nodes remain in the frontier. © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 3

Solving Problems by Searching

Lemma: Every explored node is connected to the initial state by a path of explored nodes. Proof: This is true initially, since the initial state is connected to itself. Since we never remove nodes from the explored region, we only need to check new nodes we add to the explored list on an expansion. Let n be such a new explored node. This is previously on the frontier, so it is a neighbor of a node n0 previously explored (i.e., its parent). n0 is, by hypothesis is connected to the initial state by a path of explored nodes. This path with n appended is a path of explored nodes connecting n0 to the initial state. Exercise 3.6.#TFSA Which of the following are true and which are false? Explain your answers. a. Depth-first search always expands at least as many nodes as A∗ search with an admissible heuristic. b. h(n) = 0 is an admissible heuristic for the 8-puzzle. c. A∗ is of no use in robotics because percepts, states, and actions are continuous. d. Breadth-first search is complete even if zero step costs are allowed. e. Assume that a rook can move on a chessboard any number of squares in a straight line, vertically or horizontally, but cannot jump over other pieces. Manhattan distance is an admissible heuristic for the problem of moving the rook from square A to square B in the smallest number of moves. a. False: a lucky DFS might expand exactly d nodes to reach the goal. A∗ largely dominates any graph-search algorithm that is guaranteed to find optimal solutions. b. True: h(n) = 0 is always an admissible heuristic, since costs are nonnegative. c. True: A* search is often used in robotics; the space can be discretized or skeletonized. d. True: depth of the solution matters for breadth-first search, not cost. e. False: a rook can move across the board in move one, although the Manhattan distance from start to finish is 8. Exercise 3.6.#KTWO Consider a state space where the start state is number 1 and each state k has two successors: numbers 2k and 2k + 1. a. Draw the portion of the state space for states 1 to 15. b. Suppose the goal state is 11. List the order in which nodes will be visited for breadthfirst search, depth-limited search with limit 3, and iterative deepening search. c. How well would bidirectional search work on this problem? What is the branching factor in each direction of the bidirectional search? d. Does the answer to (c) suggest a reformulation of the problem that would allow you to solve the problem of getting from state 1 to a given goal state with almost no search? e. Call the action going from k to 2k Left, and the action going to 2k + 1 Right. Can you find an algorithm that outputs the solution to this problem without any search at all?

Section 3.6

Heuristic Functions

Figure S3.1 The state space for the problem defined in Ex. 3.15.

x 12

x 16

Figure S3.2 Basic wooden railway pieces.

a. See Figure S3.1. b. Breadth-first: 1 2 3 4 5 6 7 8 9 10 11 Depth-limited: 1 2 4 8 9 5 10 11 Iterative deepening: 1; 1 2 3; 1 2 4 5 3 6 7; 1 2 4 8 9 5 10 11 c. Bidirectional search is very useful, because the only successor of n in the reverse direction is b(n/2)c. This helps focus the search. The branching factor is 2 in the forward direction; 1 in the reverse direction. d. Yes; start at the goal, and apply the single reverse successor action until you reach 1. e. The solution can be read off the binary numeral for the goal number. Write the goal number in binary. Since we can only reach positive integers, this binary expansion beings with a 1. From most- to least- significant bit, skipping the initial 1, go Left to the node 2n if this bit is 0 and go Right to node 2n + 1 if it is 1. For example, suppose the goal is 11, which is 1011 in binary. The solution is therefore Left, Right, Right.

Exercise 3.6.#BRIW A basic wooden railway set contains the pieces shown in Figure S3.2. The task is to connect these pieces into a railway that has no overlapping tracks and no loose ends where a train could run off onto the floor. a. Suppose that the pieces fit together exactly with no slack. Give a precise formulation of the task as a search problem. b. Identify a suitable uninformed search algorithm for this task and explain your choice.

Exercises 3

Solving Problems by Searching

c. Explain why removing any one of the “fork” pieces makes the problem unsolvable. d. Give an upper bound on the total size of the state space defined by your formulation. (Hint: think about the maximum branching factor for the construction process and the maximum depth, ignoring the problem of overlapping pieces and loose ends. Begin by pretending that every piece is unique.)

a. Initial state: one arbitrarily selected piece (say a straight piece). Successor function: for any open peg, add any piece type from remaining types. (You can add to open holes as well, but that isn’t necessary as all complete tracks can be made by adding to pegs.) For a curved piece, add in either orientation; for a fork, add in either orientation and (if there are two holes) connecting at either hole. It’s a good idea to disallow any overlapping configuration, as this terminates hopeless configurations early. (Note: there is no need to consider open holes, because in any solution these will be filled by pieces added to open pegs.) Goal test: all pieces used in a single connected track, no open pegs or holes, no overlapping tracks. Step cost: one per piece (actually, doesn’t really matter). b. All solutions are at the same depth, so depth-first search would be appropriate. (One could also use depth-limited search with limit n − 1, but strictly speaking it’s not necessary to do the work of checking the limit because states at depth n − 1 have no successors.) The space is very large, so uniform-cost and breadth-first would fail, and iterative deepening simply does unnecessary extra work. There are many repeated states, so it might be good to use a closed list. c. A solution has no open pegs or holes, so every peg is in a hole, so there must be equal numbers of pegs and holes. Removing a fork violates this property. There are two other “proofs” that are acceptable: 1) a similar argument to the effect that there must be an even number of “ends”; 2) each fork creates two tracks, and only a fork can rejoin those tracks into one, so if a fork is missing it won’t work. The argument using pegs and holes is actually more general, because it also applies to the case of a three-way fork that has one hole and three pegs or one peg and three holes. The “ends” argument fails here, as does the fork/rejoin argument (which is a bit handwavy anyway). d. The maximum possible number of open pegs is 3 (starts at 1, adding a two-peg fork increases it by one). Pretending each piece is unique, any piece can be added to a peg, giving at most 12 + (2 · 16) + (2 · 2) + (2 · 2 · 2) = 56 choices per peg. The total depth is 32 (there are 32 pieces), so an upper bound is 16832 /(12! · 16! · 2! · 2!) where the factorials deal with permutations of identical pieces. One could do a more refined analysis to handle the fact that the branching factor shrinks as we go down the tree, but it is not pretty.

Section 3.6

Heuristic Functions

Implement two versions of the R ESULT(s, a) function for the 8-puzzle: one that copies and edits the data structure for the parent node s and one that modifies the parent state directly (undoing the modifications as needed). Write versions of iterative deepening depth-first search that use these functions and compare their performance. For the 8 puzzle, there shouldn’t be much difference in performance. Indeed, the file "aima-lisp/search/domains/puzzle8.lisp" shows that you can represent an 8 puzzle state as a single 32-bit integer, so the question of modifying or copying data is moot. But for the n × n puzzle, as n increases, the advantage of modifying rather than copying grows. The disadvantage of a modifying successor function is that it only works with depthfirst search (or with a variant such as iterative deepening). Exercise 3.6.#ITLE Iterative lengthening search is an iterative analog of uniform cost search. The idea is to use increasing limits on path cost. If a node is generated whose path cost exceeds the current limit, it is immediately discarded. For each new iteration, the limit is set to the lowest path cost of any node discarded in the previous iteration. a. Show that this algorithm is optimal for general path costs. b. Consider a uniform tree with branching factor b, solution depth d, and unit step costs. How many iterations will iterative lengthening require? c. Now consider step costs drawn from the continuous range [ , 1], where 0 < < 1. How many iterations are required in the worst case? d. Implement the algorithm and apply it to instances of the 8-puzzle and traveling salesperson problems. Compare the algorithm’s performance to that of uniform-cost search, and comment on your results.

a. The algorithm expands nodes in order of increasing path cost; therefore the first goal it encounters will be the goal with the cheapest cost. b. It will be the same as iterative deepening, d iterations, in which O(bd ) nodes are generated. c. d/ d. Implementation not shown. Exercise 3.6.#ITDW Describe a state space in which iterative deepening search performs much worse than depth-first search (for example, O(n2 ) vs. O(n)). Consider a domain in which every state has a single successor, and there is a single goal at depth n. Then depth-first search will find the goal in n steps, whereas iterative deepening search will take 1 + 2 + 3 + · · · + n = O(n2 ) steps. © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 3

Solving Problems by Searching

Exercise 3.6.#WWWL Write a program that will take as input two web page URLs and find a path of links from one to the other. What is an appropriate search strategy? Is bidirectional search a good idea? Could a search engine be used to implement a predecessor function? As an ordinary person (or agent) browsing the web, we can only generate the successors of a page by visiting it. We can then do breadth-first search, or perhaps best-search search where the heuristic is some function of the number of words in common between the start and goal pages; this may help keep the links on target. Search engines keep the complete graph of the web, and may provide the user access to all (or at least some) of the pages that link to a page; this would allow us to do bidirectional search. Exercise 3.6.#VACG Consider the vacuum-world problem defined in Figure 2.2. a. Which of the algorithms defined in this chapter would be appropriate for this problem? Should the algorithm use tree search or graph search? b. Apply your chosen algorithm to compute an optimal sequence of actions for a 3 × 3 world whose initial state has dirt in the three top squares and the agent in the center. c. Construct a search agent for the vacuum world, and evaluate its performance in a set of 3 × 3 worlds with probability 0.2 of dirt in each square. Include the search cost as well as path cost in the performance measure, using a reasonable exchange rate. d. Compare your best search agent with a simple randomized reflex agent that sucks if there is dirt and otherwise moves randomly. e. Consider what would happen if the world were enlarged to n × n. How does the performance of the search agent and of the reflex agent vary with n?

Code not shown, but a good start is in the code repository. Clearly, graph search must be used—this is a classic grid world with many alternate paths to each state. Students will quickly find that computing the optimal solution sequence is prohibitively expensive for moderately large worlds, because the state space for an n × n world has n2 · 2n states. The completion time of the random agent grows less than exponentially in n, so for any reasonable exchange rate between search cost ad path cost the random agent will eventually win. Exercise 3.6.#SESC Prove each of the following statements, or give a counterexample: a. Breadth-first search is a special case of uniform-cost search. b. Depth-first search is a special case of best-first tree search. c. Uniform-cost search is a special case of A∗ search.

Section 3.6

Heuristic Functions

a. When all step costs are equal, g(n) ∝ depth(n), so uniform-cost search reproduces breadth-first search. b. Breadth-first search is best-first search with f (n) = depth(n); depth-first search is best-first search with f (n) = −depth(n); uniform-cost search is best-first search with f (n) = g(n). c. Uniform-cost search is A∗ search with h(n) = 0. Exercise 3.6.#RBFS Compare the performance of A∗ and RBFS on a set of randomly generated problems in the 8-puzzle (with Manhattan distance) and TSP (with MST—see Exercise 3.MSTR) domains. Discuss your results. What happens to the performance of RBFS when a small random number is added to the heuristic values in the 8-puzzle domain? The student should find that on the 8-puzzle, RBFS expands more nodes (because it does not detect repeated states) but has lower cost per node because it does not need to maintain a queue. The number of RBFS node re-expansions is not too high because the presence of many tied values means that the best path changes seldom. When the heuristic is slightly perturbed, this advantage disappears and RBFS’s performance is much worse. For TSP, the state space is a tree, so repeated states are not an issue. On the other hand, the heuristic is real-valued and there are essentially no tied values, so RBFS incurs a heavy penalty for frequent re-expansions. Exercise 3.6.#BULU Trace the operation of A∗ search applied to the problem of getting to Bucharest from Lugoj using the straight-line distance heuristic. That is, show the sequence of nodes that the algorithm will consider and the f , g, and h score for each node. The sequence of queues is as follows: L[0+244=244] M[70+241=311], T[111+329=440] L[140+244=384], D[145+242=387], T[111+329=440] D[145+242=387], T[111+329=440], M[210+241=451], T[251+329=580] C[265+160=425], T[111+329=440], M[210+241=451], M[220+241=461], T[251+329=580] T[111+329=440], M[210+241=451], M[220+241=461], P[403+100=503], T[251+329=580], R[411+193=604], D[385+242=627] M[210+241=451], M[220+241=461], L[222+244=466], P[403+100=503], T[251+329=580], A[229+366=595], R[411+193=604], D[385+242=627] M[220+241=461], L[222+244=466], P[403+100=503], L[280+244=524], D[285+242=527], T[251+329=580], A[229+366=595], R[411+193=604], D[385+242=627] L[222+244=466], P[403+100=503], L[280+244=524], D[285+242=527], L[290+244=534], D[295+242=537], T[251+329=580], A[229+366=595], R[411+193=604], D[385+242=627] P[403+100=503], L[280+244=524], D[285+242=527], M[292+241=533], L[290+244=534], D[295+242=537], T[251+329=580], A[229+366=595], R[411+193=604], D[385+242=627], T[333+329=662] © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 3

Solving Problems by Searching

B h=5 1

h=0

h=1

S A

h=7

Figure S3.3 A graph with an inconsistent heuristic on which G RAPH -S EARCH fails to return the optimal solution. The successors of S are A with f = 5 and B with f = 7. A is expanded first, so the path via B will be discarded because A will already be in the closed list.

B[504+0=504], L[280+244=524], D[285+242=527], M[292+241=533], L[290+244=534], D[295+242=537], T[251+329=580], A[229+366=595], R[411+193=604], D[385+242=627], T[333+329=662], R[500+193=693], C[541+160=701]

Exercise 3.6.#EVCO Sometimes there is no good evaluation function for a problem but there is a good comparison method: a way to tell whether one node is better than another without assigning numerical values to either. Show that this is enough to do a best-first search. Is there an analog of A∗ for this setting? If we assume the comparison function is transitive, then we can always sort the nodes using it, and choose the node that is at the top of the sort. Efficient priority queue data structures rely only on comparison operations, so we lose nothing in efficiency—except for the fact that the comparison operation on states may be much more expensive than comparing two numbers, each of which can be computed just once. A∗ relies on the division of the total cost estimate f (n) into the cost-so-far and the costto-go. If we have comparison operators for each of these, then we can prefer to expand a node that is better than other nodes on both comparisons. Unfortunately, there will usually be no such node. The tradeoff between g(n) and h(n) cannot be realized without numerical values. Exercise 3.6.#AFAL Devise a state space in which A∗ using G RAPH -S EARCH returns a suboptimal solution with an h(n) function that is admissible but inconsistent. See Figure S3.3. Exercise 3.6.#HEUR Accurate heuristics don’t necessarily reduce search time in the worst case. Given any depth d, define a search problem with a goal node at depth d, and write a heuristic function such that |h(n) − h∗ (n)| ≤ O(log h∗ (n)) but A∗ expands all nodes of depth less than d.

Section 3.6

Heuristic Functions

Let each state be a binary string, with the initial state the empty string, where the successors of a string x are {x0, x1} the string extended by one bit, and where each action has unit cost. Suppose all strings with length greater than d are goals, along with the string of d zeros. Define h(x) = d − |x| if x is of length at most d, and h(x) = 0 otherwise. Note that |h(n) − h∗ (n)| ≤ 1. Since A∗ will expand all nodes with f -value less than d, it will expand all nodes of depth less than d. Exercise 3.6.#HEPA The heuristic path algorithm (Pohl, 1977) is a best-first search in which the evaluation function is f (n) = (2 − w)g(n) + wh(n). For what values of w is this complete? For what values is it optimal, assuming that h is admissible? What kind of search does this perform for w = 0, w = 1, and w = 2? It is complete whenever 0 ≤ w < 2. w = 0 gives f (n) = 2g(n). This behaves exactly like uniform-cost search—the factor of two makes no difference in the ordering of the nodes. w = 1 gives A∗ search. w = 2 gives f (n) = 2h(n), i.e., greedy best-first search. We also have f (n) = (2 − w)[g(n) +

w h(n)] 2−w

w which behaves exactly like A∗ search with a heuristic 2−w h(n). For w ≤ 1, this is always less than h(n) and hence admissible, provided h(n) is itself admissible.

Exercise 3.6.#UNGR Consider the unbounded version of the regular 2D grid shown in Figure 3.6. The start state is at the origin, (0,0), and the goal state is at (x, y). a. What is the branching factor b in this state space? b. How many distinct states are there at depth k (for k > 0)? c. What is the maximum number of nodes expanded by breadth-first tree search? d. What is the maximum number of nodes expanded by breadth-first graph search? e. Is h = |u − x| + |v − y| an admissible heuristic for a state at (u, v)? Explain. f. How many nodes are expanded by A∗ graph search using h? g. Does h remain admissible if some links are removed? h. Does h remain admissible if some links are added between nonadjacent states?

a. The branching factor is 4 (number of neighbors of each location). b. The states at depth k form a square rotated at 45 degrees to the grid. Obviously there are a linear number of states along the boundary of the square, so the answer is 4k. c. Without repeated state checking, BFS expends exponentially many nodes: counting precisely, we get ((4x+y+1 − 1)/3) − 1. © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 3

Solving Problems by Searching

d. There are quadratically many states within the square for depth x + y, so the answer is 2(x + y)(x + y + 1) − 1. e. True; this is the Manhattan distance metric. f. False; all nodes in the rectangle defined by (0, 0) and (x, y) are candidates for the optimal path, and there are quadratically many of them, all of which may be expended in the worst case. g. True; removing links may induce detours, which require more steps, so h is an underestimate. h. False; nonlocal links can reduce the actual path length below the Manhattan distance.

Exercise 3.6.#VEGR n vehicles occupy squares (1, 1) through (n, 1) (i.e., the bottom row) of an n × n grid. The vehicles must be moved to the top row but in reverse order; so the vehicle i that starts in (i, 1) must end up in (n − i + 1, n). On each time step, every one of the n vehicles can move one square up, down, left, or right, or stay put; but if a vehicle stays put, one other adjacent vehicle (but not more than one) can hop over it. Two vehicles cannot occupy the same square. a. Calculate the size of the state space as a function of n. b. Calculate the branching factor as a function of n. c. Suppose that vehicle i is at (xi , yi ); write a nontrivial admissible heuristic hi for the number of moves it will require to get to its goal location (n − i + 1, n), assuming no other vehicles are on the grid. d. Which of the following heuristics are admissible for the problem of moving all n vehicles to their destinations? Explain. Pn (i) i = 1 hi . (ii) max{h1 , . . . , hn }. (iii) min{h1 , . . . , hn }.

a. n2n . There are n vehicles in n2 locations, so roughly (ignoring the one-per-square constraint) (n2 )n = n2n states. b. 5n . c. Manhattan distance, i.e., |(n − i + 1) − xi | + |n − yi |. This is exact for a lone vehicle. d. Only (iii) min{h1 , . . . , hn }. The explanation is nontrivial as it requires two observations. First, let the work W in a given solution be the total distance moved by all vehicles over their joint trajectories; that is, for each vehicle, add the lengths of all the P steps taken. We have W ≥ i hi ≥≥ n · min{h1 , ..., hn }. Second, the total work we can get done per step is ≤ n. (Note that for every car that jumps 2, another car has to stay put (move 0), so the total work per step is bounded by n.) Hence, completing all the work requires at least n · min{h1 , ..., hn }/n = min{h1 , ..., hn } steps.

Section 3.6

Heuristic Functions

Exercise 3.6.#KNIG Consider the problem of moving k knights from k starting squares s1 , . . . , sk to k goal squares g1 , . . . , gk , on an unbounded chessboard, subject to the rule that no two knights can land on the same square at the same time. Each action consists of moving up to k knights simultaneously. We would like to complete the maneuver in the smallest number of actions. a. What is the maximum branching factor in this state space, expressed as a function of k? b. Suppose hi is an admissible heuristic for the problem of moving knight i to goal gi by itself. Which of the following heuristics are admissible for the k-knight problem? Of those, which is the best? (i) min{h1 , . . . , hk }. (ii) max{h1 , . . . , hk }. Pk (iii) i = 1 hi . c. Repeat (b) for the case where you are allowed to move only one knight at a time. a. 9k . For each of k knights, we have one of 8 moves in addition to the possibility of not moving at all; each action executes one of these 9 choices for each knight (unless some choices conflict by landing on the same square), so there are 9k actions. b. (i) and (ii) are admissible. If the hi were exact, (ii) would be exact for the relaxed problem where knights can land on the same square. The hi are admissible and hence no larger than the exact values, so (ii) is admissible. (i) is no larger than (ii), so (i) is admissible. (iii) is not admissible. For example, if each gi is one move from its si , then (iii) returns k whereas the optimal solution cost is 1. (ii) dominates (i) so it must be as good or better. (iii) is probably of little value since it isn’t admissible and completely ignores the capability for parallel moves. c. In this case all are admissible. (i) and (ii) as the problem in part (b) is a relaxation of the problem in this part, and (i) and (ii) are admissible for the relaxed problem. (iii) is admissible because it exact for the relaxed problem where knights can land on the same square. (iii) is best since it dominates the rest and is now admissible. Exercise 3.6.#IAFA We saw on page ?? that the straight-line distance heuristic leads greedy best-first search astray on the problem of going from Iasi to Fagaras. However, the heuristic is perfect on the opposite problem: going from Fagaras to Iasi. Are there problems for which the heuristic is misleading in both directions? Going between Rimnicu Vilcea and Lugoj is one example. The shortest path is the southern one, through Mehadia, Dobreta and Craiova. But a greedy search using the straight-line heuristic starting in Rimnicu Vilcea will start the wrong way, heading to Sibiu. Starting at Lugoj, the heuristic will correctly lead us to Mehadia, but then a greedy search will return to Lugoj, and oscillate forever between these two cities. © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 3

Solving Problems by Searching

Exercise 3.6.#HEUE Invent a heuristic function for the 8-puzzle that sometimes overestimates, and show how it can lead to a suboptimal solution on a particular problem. (You can use a computer to help if you want.) Prove that if h never overestimates by more than c, A∗ using h returns a solution whose cost exceeds that of the optimal solution by no more than c. The heuristic h = h1 + h2 (adding misplaced tiles and Manhattan distance) sometimes overestimates. Now, suppose h(n) ≤ h∗ (n) + c (as given) and let G2 be a goal that is suboptimal by more than c, i.e., g(G2 ) > C ∗ + c. Now consider any node n on a path to an optimal goal. We have f (n) = g(n) + h(n) ≤ g(n) + h∗ (n) + c ≤ C∗ + c ≤ g(G2 ) so G2 will never be expanded before an optimal goal is expanded. Exercise 3.6.#CONH Prove that if a heuristic is consistent, it must be admissible. Construct an admissible heuristic that is not consistent. A heuristic is consistent iff, for every node n and every successor n0 of n generated by any action a, h(n) ≤ c(n, a, n0 ) + h(n0 ) One simple proof is by induction on the number k of nodes on the shortest path to any goal from n. For k = 1, let n0 be the goal node; then h(n) ≤ c(n, a, n0 ). For the inductive case, assume n0 is on the shortest path k steps from the goal and that h(n0 ) is admissible by hypothesis; then h(n) ≤ c(n, a, n0 ) + h(n0 ) ≤ c(n, a, n0 ) + h∗ (n0 ) = h∗ (n) so h(n) at k + 1 steps from the goal is also admissible. Exercise 3.6.#MSTR The traveling salesperson problem (TSP) can be solved with the minimum-spanning-tree (MST) heuristic, which estimates the cost of completing a tour, given that a partial tour has already been constructed. The MST cost of a set of cities is the smallest sum of the link costs of any tree that connects all the cities. a. Show how this heuristic can be derived from a relaxed version of the TSP. © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Section 3.6

Heuristic Functions

b. Show that the MST heuristic dominates straight-line distance. c. Write a problem generator for instances of the TSP where cities are represented by random points in the unit square. d. Find an efficient algorithm in the literature for constructing the MST, and use it with A∗ graph search to solve instances of the TSP.

This exercise reiterates a small portion of the classic work of Held and Karp (1970). a. The TSP problem is to find a minimal (total length) path through the cities that forms a closed loop. MST is a relaxed version of that because it asks for a minimal (total length) graph that need not be a closed loop—it can be any fully-connected graph. As a heuristic, MST is admissible—it is always shorter than or equal to a closed loop. b. The straight-line distance back to the start city is a rather weak heuristic—it vastly underestimates when there are many cities. In the later stage of a search when there are only a few cities left it is not so bad. To say that MST dominates straight-line distance is to say that MST always gives a higher value. This is obviously true because a MST that includes the goal node and the current node must either be the straight line between them, or it must include two or more lines that add up to more. (This all assumes the triangle inequality.) c. See aima-list/search/domains/tsp.lisp for a start at this. The file includes a heuristic based on connecting each unvisited city to its nearest neighbor, a close relative to the MST approach. d. See Cormen et al., 1990, page 505 for an algorithm that runs in O(E log E) time, where E is the number of edges. The code repository currently contains a somewhat less efficient algorithm. Exercise 3.6.#GASC On page 100, we defined the relaxation of the 8-puzzle in which a tile can move from square A to square B if B is blank. The exact solution of this problem defines Gaschnig’s heuristic (Gaschnig, 1979). Explain why Gaschnig’s heuristic is at least as accurate as h1 (misplaced tiles), and show cases where it is more accurate than both h1 and h2 (Manhattan distance). Explain how to calculate Gaschnig’s heuristic efficiently. The misplaced-tiles heuristic is exact for the problem where a tile can move from square A to square B. As this is a relaxation of the condition that a tile can move from square A to square B if B is blank, Gaschnig’s heuristic cannot be less than the misplaced-tiles heuristic. As it is also admissible (being exact for a relaxation of the original problem), Gaschnig’s heuristic is therefore more accurate. If we permute two adjacent tiles in the goal state, we have a state where misplaced-tiles and Manhattan both return 2, but Gaschnig’s heuristic returns 3. To compute Gaschnig’s heuristic, repeat the following until the goal state is reached: let B be the current location of the blank; if B is occupied by tile X (not the blank) in the goal © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 3

Solving Problems by Searching

state, move X to B; otherwise, move any misplaced tile to B. Students could be asked to prove that this is the optimal solution to the relaxed problem. Exercise 3.6.#MANH We gave two simple heuristics for the 8-puzzle: Manhattan distance and misplaced tiles. Several heuristics in the literature purport to improve on this—see, for example, Nilsson (1971), Mostow and Prieditis (1989), and Hansson et al.(1992). Test these claims by implementing the heuristics and comparing the performance of the resulting algorithms. Students should provide results in the form of graphs and/or tables showing both runtime and number of nodes generated. (Different heuristics have different computation costs.) Runtimes may be very small for 8-puzzles, so you may want to assign the 15-puzzle or 24-puzzle instead. The use of pattern databases is also worth exploring experimentally.

EXERCISES

SEARCH IN COMPLEX ENVIRONMENTS 4.1 Local Search and Optimization Problems Exercise 4.1.#ASTF For each of the following assertions, say whether it is true or false and support your answer with examples or counterexamples where appropriate. a. A hill-climbing algorithm that never visits states with lower value (or higher cost) is guaranteed to find the optimal solution if given enough time to find a solution. b. Suppose the temperature schedule for simulated annealing is set to be constant up to time N and zero thereafter. For any finite problem, we can set N large enough so that the algorithm is returns an optimal solution with probability 1. c. For any local-search problem, hill-climbing will return a global optimum if the algorithm is run starting at any state that is a neighbor of a neighbor of a globally optimal state. d. In a nondeterministic, partially observable problem, improving an agent’s transition model (i.e., eliminating spurious outcomes that never actually occur) will never increase the size of the agent’s belief states. e. Stochastic hill climbing is guaranteed to arrive at a global optimum. f. In a continuous space, gradient descent with a fixed step size is guaranteed to converge to a local or global optimum. g. Gradient descent finds the global optimum from any starting point if and only if the function is convex. a. False. Such an algorithm will reach a local optimum and stop (or wander on a plateau). b. False. If the temperature is fixed the algorithm always has a certain amount of randomness, so when it stops there is no guarantee it will be in an optimal state. Cooling slowly towards zero is the key to optimality. c. False. The intervening neighbor could be a local minimum and the current state is on another slope leading to a local maximum. Consider, for example, starting at the state valued 5 in the linear sequence 7,6,5,0,9,0,0. d. True. For any given sequence of actions and observations, removing spurious action outcomes can only eliminate states from the final belief state. (The formal proof is © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 4

Search in Complex Environments

straightforward.) e. False. Stochastic hill climbing chooses from the uphill moves at random, but, unlike simulated annealing, it stops if there are none, so it still gets trapped in local optima. f. False. While convexity implies that gradient descent will find a global optimum, the converse is not true. A function can have no local optima without being convex—e.g., in one dimension, it just needs to be strictly decreasing/increasing to the left/right of the global optimum. Exercise 4.1.#SPCA Give the name of the algorithm that results from each of the following special cases: a. Local beam search with k = 1. b. Local beam search with one initial state and no limit on the number of states retained. c. Simulated annealing with T = 0 at all times (and omitting the termination test). d. Simulated annealing with T = ∞ at all times. e. Genetic algorithm with population size N = 1.

a. Local beam search with k = 1 is hill-climbing search. b. Local beam search with one initial state and no limit on the number of states retained, resembles breadth-first search in that it adds one complete layer of nodes before adding the next layer. Starting from one state, the algorithm would be essentially identical to breadth-first search except that each layer is generated all at once. c. Simulated annealing with T = 0 at all times: ignoring the fact that the termination step would be triggered immediately, the search would be identical to first-choice hill climbing because every downward successor would be rejected with probability 1. (Exercise may be modified in future printings.) d. Simulated annealing with T = ∞ at all times is a random-walk search: it always accepts a new state. e. Genetic algorithm with population size N = 1: if the population size is 1, then the two selected parents will be the same individual; crossover yields an exact copy of the individual; then there is a small chance of mutation. Thus, the algorithm executes a random walk in the space of individuals. Exercise 4.1.#BRIO Exercise BRIO - EXERCISE considers the problem of building railway tracks under the assumption that pieces fit exactly with no slack. Now consider the real problem, in which pieces don’t fit exactly but allow for up to 10 degrees of rotation to either side of the “proper” alignment. Explain how to formulate the problem so it could be solved by simulated annealing. Despite its humble origins, this question raises many of the same issues as the scientifically important problem of protein design. There is a discrete assembly space in which pieces are chosen to be added to the track and a continuous configuration space determined by the © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Section 4.1

Local Search and Optimization Problems

“joint angles” at every place where two pieces are linked. Thus we can define a state as a set of oriented, linked pieces and the associated joint angles in the range [−10, 10], plus a set of unlinked pieces. The linkage and joint angles exactly determine the physical layout of the track; we can allow for (and penalize) layouts in which tracks lie on top of one another, or we can disallow them. The evaluation function would include terms for how many pieces are used, how many loose ends there are, and (if allowed) the degree of overlap. We might include a penalty for the amount of deviation from 0-degree joint angles. (We could also include terms for “interestingness” and “traversability”—for example, it is nice to be able to drive a train starting from any track segment to any other, ending up in either direction without having to lift up the train.) The tricky part is the set of allowed moves. Obviously we can unlink any piece or link an unlinked piece to an open peg with either orientation at any allowed angle (possibly excluding moves that create overlap). More problematic are moves to join a peg and hole on already-linked pieces and moves to change the angle of a joint. Changing one angle may force changes in others, and the changes will vary depending on whether the other pieces are at their joint-angle limit. In general there will be no unique “minimal” solution for a given angle change in terms of the consequent changes to other angles, and some changes may be impossible. Exercise 4.1.#TSPL In this exercise, we explore the use of local search methods to solve TSPs of the type defined in Exercise TSP - MST- EXERCISE. a. Implement and test a hill-climbing method to solve TSPs. Compare the results with optimal solutions obtained from the A∗ algorithm with the MST heuristic (Exercise TSP - MST- EXERCISE ). b. Repeat part (a) using a genetic algorithm instead of hill climbing. You may want to consult Larranaga et al. (1999) for some suggestions for representations.

Here is one simple hill-climbing algorithm: • Connect all the cities into an arbitrary path. • Pick two points along the path at random. • Split the path at those points, producing three pieces. • Try all six possible ways to connect the three pieces. • Keep the best one, and reconnect the path accordingly. • Iterate the steps above until no improvement is observed for a while. Exercise 4.1.#HILL Generate a large number of 8-puzzle and 8-queens instances and solve them (where possible) by hill climbing (steepest-ascent and first-choice variants), hill climbing with random restart, and simulated annealing. Measure the search cost and percentage of solved problems and graph these against the optimal solution cost. Comment on your results. Code not shown. © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 4

Search in Complex Environments

4.2 Local Search in Continuous Spaces Exercise 4.2.#SADD In a continuous state space, a saddle point is a point at which all partial derivatives are zero but the point is neither a minimum nor a maximum; instead, it is a minimum in some directions but a maximum in others. Explain how the various continuous-space local search algorithms in the chapter function if started at such a point. Continuous state space local search requires either gradient or empirical gradient methods. Empirical methods involve discretizing the continuous state space using some constant δ such that local search algorithms function as before. Nonetheless, too large a choice of δ will change the state space such that it no longer resembles the continuous space and even for a very small δ a solution may be intractable or otherwise infinitesimally distant from the optimum. Thus, as in the discrete case, empirical gradient methods will get stuck on saddle points depending on their dimensionality. Line search and other local gradient methods attempt to navigate to the local minimum by stepping according to the product of a local gradient and a step size parameter. These methods are affected by ridges or saddle points given that they cannot exhaustively consider the effect of the step size on every possible partial gradient and thus may ”miss” the ridge (or valley) leading to the global maximum (or minimum). Analytic gradient methods (even for only the local gradient), such as that of NewtonRaphson, should conceivably distinguish saddle points and should therefore be able to follow the appropriate ridge or valley, but in high dimensional spaces such analyses as computing the Hessian matrix become intractable and thus suffer the fate of other local gradient search methods.

4.3 Search with Nondeterministic Actions Exercise 4.3.#CONP The A ND -O R -G RAPH -S EARCH algorithm in Figure 4.11 checks for repeated states only on the path from the root to the current state. Suppose that, in addition, the algorithm were to store every visited state and check against that list. (See B READTH -F IRST-S EARCH in Figure 3.9 for an example.) Determine the information that should be stored and how the algorithm should use that information when a repeated state is found. (Hint: You will need to distinguish at least between states for which a successful subplan was constructed previously and states for which no subplan could be found.) Explain how to use labels, as defined in Section 4.3.3, to avoid having multiple copies of subplans. See Figure S4.1 for the adapted algorithm. For states that O R -S EARCH finds a solution for it records the solution found. If it later visits that state again it immediately returns that solution. When O R -S EARCH fails to find a solution it has to be careful. Whether a state can be solved depends on the path taken to that solution, as we do not allow cycles. So on failure © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Section 4.3

Search with Nondeterministic Actions

function A ND -O R -G RAPH -S EARCH(problem) returns a conditional plan, or failure O R -S EARCH(problem.I NITIAL -S TATE, problem, [ ]) function O R -S EARCH(state, problem, path) returns a conditional plan, or failure if problem.G OAL -T EST(state) then return the empty plan if state has previously been solved then return R ECALL -S UCCESS(state) if state has previously failed for a subset of path then return failure if state is on path then R ECORD -FAILURE(state, path) return failure for each action in problem.ACTIONS(state) do plan ← A ND -S EARCH(R ESULTS(state, action), problem, [state | path]) if plan 6= failure then R ECORD -S UCCESS(state, [action | plan]) return [action | plan] return failure function A ND -S EARCH(states, problem, path) returns a conditional plan, or failure for each si in states do plan i ← O R -S EARCH(si , problem, path) if plan i = failure then return failure return [if s1 then plan 1 else if s2 then plan 2 else . . . if sn−1 then plan n−1 else plan n ] Figure S4.1 A ND -O R search with repeated state checking.

O R -S EARCH records the value of path. If a state is which has previously failed when path contained any subset of its present value, O R -S EARCH returns failure. To avoid repeating sub-solutions we can label all new solutions found, record these labels, then return the label if these states are visited again. Post-processing can prune off unused labels. Alternatively, we can output a direct acyclic graph structure rather than a tree. Exercise 4.3.#CONL Explain precisely how to modify the A ND -O R -G RAPH -S EARCH algorithm to generate a cyclic plan if no acyclic plan exists. You will need to deal with three issues: labeling the plan steps so that a cyclic plan can point back to an earlier part of the plan, modifying O R -S EARCH so that it continues to look for acyclic plans after finding a cyclic plan, and augmenting the plan representation to indicate whether a plan is cyclic. Show how your algorithm works on (a) the slippery vacuum world, and (b) the slippery, erratic vacuum world. You might wish to use a computer implementation to check your results. The question statement describes the required changes in detail, see Figure S4.2 for the modified algorithm. When O R -S EARCH cycles back to a state on path it returns a token loop which means to loop back to the most recent time this state was reached along the path to it. Since path is implicitly stored in the returned plan, there is sufficient information for later © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 4

Search in Complex Environments

function A ND -O R -G RAPH -S EARCH(problem) returns a conditional plan, or failure O R -S EARCH(problem.I NITIAL -S TATE, problem, [ ]) function O R -S EARCH(state, problem, path) returns a conditional plan, or failure if problem.G OAL -T EST(state) then return the empty plan if state is on path then return loop cyclic − plan ← None for each action in problem.ACTIONS(state) do plan ← A ND -S EARCH(R ESULTS(state, action), problem, [state | path]) if plan 6= failure then if plan is acyclic then return [action | plan] cyclic − plan ← [action | plan] if cyclic − plan 6= None then return cyclic − plan return failure function A ND -S EARCH(states, problem, path) returns a conditional plan, or failure loopy ← True for each si in states do plan i ← O R -S EARCH(si , problem, path) if plan i = failure then return failure if plan i 6= loop then loopy ← False if not loopy then return [if s1 then plan 1 else if s2 then plan 2 else . . . if sn−1 then plan n−1 else plan n ] return failure Figure S4.2 A ND -O R search with repeated state checking.

processing, or a modified implementation, to replace these with labels. The plan representation is implicitly augmented to keep track of whether the plan is cyclic (i.e., contains a loop) so that O R -S EARCH can prefer acyclic solutions. A ND -S EARCH returns failure if all branches lead directly to a loop, as in this case the plan will always loop forever. This is the only case it needs to check as if all branches in a finite plan loop there must be some And-node whose children all immediately loop. Algorithm results Traces of the algorithm running follow. The state is encoded by specifying whether the squares are clean (c) or dirty (d), with the letter in capitals if the vacuum cleaner is at that spot e.g., Cd means the left square is clean, the right is dirty, and the vacuum is on the left square. Indentation follows nesting of function calls. OR and AND lines are printed at the start of O R -S EARCH and A ND -S EARCH, respectively. trying and testing lines are printed at the top of the loops within O R -S EARCH and A ND -S EARCH, respectively. Exercise 4.3.#NPPA We can turn the navigation problem in Exercise PATH - PLANNING - EXERCISE into an environment as follows: © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Section 4.3

Search with Nondeterministic Actions

• The percept will be a list of the positions, relative to the agent, of the visible vertices. The percept does not include the position of the robot! The robot must learn its own position from the map; for now, you can assume that each location has a different “view.” • Each action will be a vector describing a straight-line path to follow. If the path is unobstructed, the action succeeds; otherwise, the robot stops at the point where its path first intersects an obstacle. If the agent returns a zero motion vector and is at the goal (which is fixed and known), then the environment teleports the agent to a random location (not inside an obstacle). • The performance measure charges the agent 1 point for each unit of distance traversed and awards 1000 points each time the goal is reached. a. Implement this environment and a problem-solving agent for it. After each teleportation, the agent will need to formulate a new problem, which will involve discovering its current location. b. Document your agent’s performance (by having the agent generate suitable commentary as it moves around) and report its performance over 100 episodes. c. Modify the environment so that 30% of the time the agent ends up at an unintended destination (chosen randomly from the other visible vertices if any; otherwise, no move at all). This is a crude model of the motion errors of a real robot. Modify the agent so that when such an error is detected, it finds out where it is and then constructs a plan to get back to where it was and resume the old plan. Remember that sometimes getting back to where it was might also fail! Show an example of the agent successfully overcoming two successive motion errors and still reaching the goal. d. Now try two different recovery schemes after an error: (1) head for the closest vertex on the original route; and (2) replan a route to the goal from the new location. Compare the performance of the three recovery schemes. Would the inclusion of search costs affect the comparison? e. Now suppose that there are locations from which the view is identical. (For example, suppose the world is a grid with square obstacles.) What kind of problem does the agent now face? What do solutions look like?

The student needs to make several design choices in answering this question. First, how will the vertices of objects be represented? The problem states the percept is a list of vertex positions, but that is not precise enough. Here is one good choice: The agent has an orientation (a heading in degrees). The visible vertexes are listed in clockwise order, starting straight ahead of the agent. Each vertex has a relative angle (0 to 360 degrees) and a distance. We also want to know if a vertex represents the left edge of an obstacle, the right edge, or an interior point. We can use the symbols L, R, or I to indicate this. The student will need to do some basic computational geometry calculations: intersection of a path and a set of line segments to see if the agent will bump into an obstacle, and visibility calculations to determine the percept. There are efficient algorithms for doing this on a set of line segments, but don’t worry about efficiency; an exhaustive algorithm is ok. If this seems too much, the instructor can provide an environment simulator and ask the student only to © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 4

Search in Complex Environments

program the agent. To answer (c), the student will need some exchange rate for trading off search time with movement time. It is probably too complex to make the simulation asynchronous real-time; easier to impose a penalty in points for computation. For (d), the agent will need to maintain a set of possible positions. Each time the agent moves, it may be able to eliminate some of the possibilities. The agent may consider moves that serve to reduce uncertainty rather than just get to the goal.

4.4 Search in Partially Observable Environments Exercise 4.4.#CONF In Section 4.4.1 we introduced belief states to solve sensorless search problems. A sequence of actions solves a sensorless problem if it maps every physical state in the initial belief state b to a goal state. Suppose the agent knows h∗ (s), the true optimal cost of solving the physical state s in the fully observable problem, for every state s in b. Find an admissible heuristic h(b) for the sensorless problem in terms of these costs, and prove its admissibilty. Comment on the accuracy of this heuristic on the sensorless vacuum problem of Figure 4.14. How well does A∗ perform? A sequence of actions is a solution to a belief state problem if it takes every initial physical state to a goal state. We can relax this problem by requiring it take only some initial physical state to a goal state. To make this well defined, we’ll require that it finds a solution for the physical state with the most costly solution. If h∗ (s) is the optimal cost of solution starting from the physical state s, then h(S) = max h∗ (s) s∈S

is the heuristic estimate given by this relaxed problem. This heuristic assumes any solution to the most difficult state the agent things possible will solve all states. On the sensorless vacuum cleaner problem in Figure 4.14, h correctly determines the optimal cost for all states except the central three states (those reached by [suck], [suck, lef t] and [suck, right]) and the root, for which h estimates to be 1 unit cheaper than they really are. This means A∗ will expand these three central nodes, before marching towards the solution. Exercise 4.4.#BELS This exercise explores subset–superset relations between belief states in sensorless or partially observable environments. a. Prove that if an action sequence is a solution for a belief state b, it is also a solution for any subset of b. Can anything be said about supersets of b? b. Explain in detail how to modify graph search for sensorless problems to take advantage of your answers in (a).

Section 4.4

Search in Partially Observable Environments

c. Explain in detail how to modify AND – OR search for partially observable problems, beyond the modifications you describe in (b).

a. An action sequence is a solution for belief state b if performing it starting in any state s ∈ b reaches a goal state. Since any state in a subset of b is in b, the result is immediate. Any action sequence which is not a solution for belief state b is also not a solution for any superset; this is the contrapositive of what we’ve just proved. One cannot, in general, say anything about arbitrary supersets, as the action sequence need not lead to a goal on the states outside of b. One can say, for example, that if an action sequence solves a belief state b and a belief state b0 then it solves the union belief state b ∪ b0 . b. On expansion of a node, do not add to the frontier any child belief state which is a superset of a previously explored belief state. c. If you keep a record of previously solved belief states, add a check to the start of ORsearch to check whether the belief state passed in is a subset of a previously solved belief state, returning the previous solution in case it is. Exercise 4.4.#MVLS On page 128 it was assumed that a given action would have the same cost when executed in any physical state within a given belief state. (This leads to a belief-state search problem with well-defined step costs.) Now consider what happens when the assumption does not hold. Does the notion of optimality still make sense in this context, or does it require modification? Consider also various possible definitions of the “cost” of executing an action in a belief state; for example, we could use the minimum of the physical costs; or the maximum; or a cost interval with the lower bound being the minimum cost and the upper bound being the maximum; or just keep the set of all possible costs for that action. For each of these, explore whether A∗ (with modifications if necessary) can return optimal solutions. Consider a very simple example: an initial belief state {S1 , S2 }, actions a and b both leading to goal state G from either initial state, and c(S1 , a, G) = 3 ; c(S1 , b, G) = 2 ;

c(S2 , a, G) = 5 ; c(S2 , b, G) = 6 .

In this case, the solution [a] costs 3 or 5, the solution [b] costs 2 or 6. Neither is “optimal” in any obvious sense. In some cases, there will be an optimal solution. Let us consider just the deterministic case. For this case, we can think of the cost of a plan as a mapping from each initial physical state to the actual cost of executing the plan. In the example above, the cost for [a] is {S1 :3, S2 :5} and the cost for [b] is {S1 :2, S2 :6}. We can say that plan p1 weakly dominates p2 if, for each initial state, the cost for p1 is no higher than the cost for p2 . (Moreover, p1 dominates p2 if it weakly dominates it and has a lower cost for some state.) If a plan p weakly dominates all others, it is optimal. Notice that this definition reduces to ordinary optimality in © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Principle of optimality

Exercises 4

Search in Complex Environments

the observable case where every belief state is a singleton. As the preceding example shows, however, a problem may have no optimal solution in this sense. A perhaps acceptable version of A∗ would be one that returns any solution that is not dominated by another. To understand whether it is possible to apply A∗ at all, it helps to understand its dependence on Bellman’s (1957) principle of optimality: An optimal policy has the property that whatever the initial state and initial decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision. It is important to understand that this is a restriction on performance measures designed to facilitate efficient algorithms, not a general definition of what it means to be optimal. In particular, if we define the cost of a plan in belief-state space as the minimum cost of any physical realization, we violate Bellman’s principle. Modifying and extending the previous example, suppose that a and b reach S3 from S1 and S4 from S2 , and then reach G from there: c(S1 , a, S3 ) = 6 ; c(S1 , b, S3 ) = 6 ; c(S3 , b, G) = 1 ;

c(S2 , a, S4 ) = 2 ; c(S2 , b, S4 ) = 1 .c(S3 , a, G) = 2 ; c(S4 , b, G) = 9 .

c(S4 , a, G) = 2 ;

In the belief state {S3 , S4 }, the minimum cost of [a] is min{2, 2} = 2 and the minimum cost of [b] is min{1, 9} = 1, so the optimal plan is [b]. In the initial belief state {S1 , S2 }, the four possible plans have the following costs: [a, a] : min{8, 4} = 4 ; [a, b] : min{7, 11} = 7 ; [b, a] : min{8, 3} = 3 ; [b, b] : min{7, 10} = 7 . Hence, the optimal plan in {S1 , S2 } is [b, a], which does not choose b in {S3 , S4 } even though that is the optimal plan at that point. This counterintuitive behavior is a direct consequence of choosing the minimum of the possible path costs as the performance measure. This example gives just a small taste of what might happen with nonadditive performance measures. Details of how to modify and analyze A∗ for general path-dependent cost functions are give by Dechter and Pearl (1985). Many aspects of A∗ carry over; for example, we can still derive lower bounds on the cost of a path through a given node. For a belief state b, the minimum value of g(s) + h(s) for each state s in b is a lower bound on the minimum cost of a plan that goes through b. Exercise 4.4.#VACL Consider the sensorless version of the erratic vacuum world. Draw the belief-state space reachable from the initial belief state {1, 2, 3, 4, 5, 6, 7, 8}, and explain why the problem is unsolvable. The belief state space is shown in Figure S4.3. No solution is possible because no path leads to a belief state all of whose elements satisfy the goal. If the problem is fully observable, the agent reaches a goal state by executing a sequence such that Suck is performed only in a dirty square. This ensures deterministic behavior and every state is obviously solvable.

Section 4.5

Online Search Agents and Unknown Environments

L R

Figure S4.3 The belief state space for the sensorless vacuum world under Murphy’s law.

4.5 Online Search Agents and Unknown Environments Exercise 4.5.#ONOF Suppose that an agent is in a 3 × 3 maze environment like the one shown in Figure 4.19. The agent knows that its initial location is (1,1), that the goal is at (3,3), and that the actions Up, Down, Left, Right have their usual effects unless blocked by a wall. The agent does not know where the internal walls are. In any given state, the agent perceives the set of legal actions; it can also tell whether the state is one it has visited before. Assume that the agent’s percept is exactly the set of unblocked directions (i.e., blocked directions are illegal actions). a. Explain how this online search problem can be viewed as an offline search in belief-state space, where the initial belief state includes all possible environment configurations. How large is the initial belief state? How large is the space of belief states? b. How many distinct percepts are possible in the initial state? c. Describe the first few branches of a contingency plan for this problem. How large (roughly) is the complete plan? Notice that this contingency plan is a solution for every possible environment fitting the given description. Therefore, interleaving of search and execution is not strictly necessary even in unknown environments. There are 12 possible locations for internal walls, so there are 212 = 4096 possible environment configurations. A belief state designates a subset of these as possible configurations; for example, before seeing any percepts all 4096 configurations are possible—this is a single belief state. a. Online search is equivalent to offline search in belief-state space where each action in a belief-state can have multiple successor belief-states: one for each percept the agent could observe after the action. A successor belief-state is constructed by taking the previous belief-state, itself a set of states, replacing each state in this belief-state by the successor state under the action, and removing all successor states which are inconsistent with the percept. This is exactly the construction in Section 4.4.2. A ND -O R search can be used to solve this search problem. The initial belief state has 21 0 = 1024 states in it, as we know whether two edges have walls or not (the upper and right edges 12 have no walls) but nothing more. There are 22 possible belief states, one for each set © 2023 Pearson Education, Hoboken, NJ. All rights reserved.

Exercises 4

Search in Complex Environments

? ?

NoOp

? ?

Right

? ?

? ? ?

Figure S4.4 The 3 × 3 maze exploration problem: the initial state, first percept, and one selected action with its perceptual outcomes.

of environment configurations. We can view this as a contingency problem in belief state space. After each action and percept, the agent learns whether or not an internal wall exists between the current square and each neighboring square. Hence, each reachable belief state can be represented exactly by a list of status values (present, absent, unknown) for each wall separately. That is, the belief state is completely decomposable and there are exactly 312 reachable belief states. The maximum number of possible wall-percepts in each state is 16 (24 ), so each belief state has four actions, each with up to 16 nondeterministic successors. b. Assuming the external walls are known, there are two internal walls and hence 22 = 4 possible percepts. c. The initial null action leads to four possible belief states, as shown in Figure S4.4. From each belief state, the agent chooses a single action which can lead to up to 8 belief states (on entering the middle square). Given the possibility of having to retrace its steps at a dead end, the agent can explore the entire maze in no more than 18 steps, so the complete plan (expressed as a tree) has no more than 818 nodes. On the other hand, there are just 312 reachable belief states, so the plan could be expressed more concisely as a table of actions indexed by belief state (a policy in the terminology of Chapter 17).

Section 4.5

Online Search Agents and Unknown Environments

Exercise 4.5.#PPHC In this exercise, we examine hill climbing in the context of robot navigation, using the environment in Figure ?? as an example. a. Repeat Exercise PATH - PLANNING - AGENT- EXERCISE using hill climbing. Does your agent ever get stuck in a local minimum? Is it possible for it to get stuck with convex obstacles? b. Construct a nonconvex polygonal environment in which the agent gets stuck. c. Modify the hill-climbing algorithm so that, instead of doing a depth-1 search to decide where to go next, it does a depth-k search. It should find the best k-step path and do one step along it, and then repeat the process. d. Is there some k for which the new algorithm is guaranteed to escape from local minima? e. Explain how LRTA∗ enables the agent to escape from local minima in this case.

Current position

Goal

(a)

Goal

(b)

Figure S4.5 (a) Getting stuck with a convex obstacle. (b) Getting stuck with a nonconvex obstacle.

Hillclimbing is surprisingly effective at finding reasonable if not optimal paths for very little computational cost, and seldom fails in two dimensions. a. It is possible (see Figure S4.5(a)) but very unlikely—the obstacle has to have an unusual shape and be positioned correctly with respect to the goal. b. With nonconvex obstacles, getting stuck is much more likely to be a problem (see Figure S4.5(b)). c. Notice that this is just depth-limited search, where you choose a step along the best path even if it is not a solution. © 2023 Pearson Education, Hoboken, NJ. All rights reserved.