An Artificial Intelligence for Seeing

A NASA scientist is working to develop an A-Eye.

Oceanographer John Moisan said artificial intelligence will direct his A-Eye, a movable sensor. After analyzing images his AI will not just find known patterns in new data, but also steer the sensor to observe and discover new features or biological processes.

Moisan said existing AI and machine learning technologies don’t come close to replicating the kind of human visual processing or intelligence needed to make real-time decisions about unfamiliar data.

“A truly intelligent machine needs to be able to recognize when it is faced with something truly new and worthy of further observation,” Moisan said. “Most AI applications are mapping applications trained with familiar data to recognize patterns in new data. How do you teach a machine to recognize something it doesn’t understand, stop and say ‘What was that? Let’s take a closer look.’ That’s discovery.”

Finding and identifying new patterns in complex data is still the domain of human scientists, and how humans see plays a large part. Goddard AI expert James MacKinnon said scientists analyze large data sets using visualizations to reveal relationships between variables.

A complex data visualization tool called an embedding space allows scientists to manipulate different aspects or dimensions of the data in a multi-dimensional visualization. By carefully manipulating the scale of specific dimensions, the trained human eye finds relationships between different aspects of the data, which can be investigated to identify key variables.

“You need some way to take a perception of a scene and turn that into a decision and that’s really hard,” he said. “The scary thing, to a scientist, is to throw away data that could be valuable. An AI might prioritize what data to send first or have an algorithm that can call attention to anomalies, but at the end of the day, it’s going to be a scientist looking at that data that results in discoveries.”

An image of a coastal marshland combines aerial and satellite views in a technique similar to hyperspectral imaging. Combining data from multiple sources gives scientists information that can support environmental management.

NASA/John Moisan

Other investigators like Goddard’s Bethany Theiling also hope to train an AI to look for new patterns or absences of various chemicals or organic compounds. Theiling is preparing for ocean worlds missions by creating artificial oceans that are different from Earth’s bodies of water, then analyzing them for data sets to train algorithms.

Moisan is developing his AI to interpret hyperspectral mapping images from complex aquatic and coastal regions.

“How do you pick out things that matter in a scan,” Moisan asked. “I want to be able to quickly point that hyperspectral imager at something swept up in the scan, so that from a remote area we can get whatever we need to understand the environmental scene.”

Using real-time estimates of surface topology and spatial patterns from a high-resolution camera, thermal emission from an infrared camera, and course, reflected light from a wide-angle “pushbroom” optical camera in flight, Moisan’s on-board AI would scan the analyzed data in real-time to search for significant features, then steer an optical pointing instrument (the eye in A-Eye) to collect subnanometer hyperspectral reflectance data on those identified areas of interest.

This year, Moisan is training the AI using observations from prior flights over the Delmarva Peninsula. Follow-up funding would help him complete the optical pointing goal.

Thinking machines are set to play a larger role in future exploration of our universe. Sophisticated computers taught to recognize chemical signatures that could indicate life processes, or landscape features like lava flows or craters, promise to increase the value of science data returned from lunar or deep-space exploration.

Genetic Programming and Intelligent Swarms

The A-Eye is Goddard oceanographer John Moisan’s latest pursuit, but he has been working for NASA in artificial intelligence and distributed missions for years.

Moisan said an artificial intelligence technique called Genetic Programming might help develop the types of AI NASA needs to solve a variety of exploration challenges, from distributed missions, to identifying high-value science data.

Infrared images like this one from a marsh area on the Maryland/Virginia Eastern Shore coastal barrier and back bay regions reveal clues to scientists about plant health, photosynthesis, and other conditions that affect vegetation and ecosystems.

John Moisan

“I’ve been doing genetic programming for 15 years,” he said, “combining segments of code to perform certain functions, analyzing their fitness capability and breeding what you’re looking for into successive generations of code.”

Much like natural selection guides biological evolution using DNA as the code, genetic programming incorporates a form of mutation by random substitutions and uses repeated versions to select the fittest programs for reproduction. Each generation of code performs better than those that came before until the desired function is achieved.

Moisan also helped NASA tackle another AI problem related to an agency priority for future explorations: training a distributed swarm to efficiently explore a region without duplicating efforts.

“We would send autonomous boats onto the surface of the ocean to collect information,” he said. “We struggled with how to give each individual craft their own ability to choose its own path without them all ending up in the same place. They needed central direction and decision making.”

Although the AI technology available at that time could not keep the boats from clumping together, Moisan is undeterred in pursuing innovations to unlock those higher functions.