to believe that they would be able to tear up the check to this opposing organization; instead the $100 would go to the organization they favored and the subject got a $75 bonus. If they lied unsuccessfully, meaning they were caught in their deceit, the organization they opposed got the money. So the stakes were high; these were liars under pressure. The challenge, then, was to teach a computer to “read” their eyes and discover who was lying.
Mining for information Enter Venu Govindaraju, PhD ’92 & MS ’88, a SUNY Distinguished Professor in the Department of Computer Science and Engineering, and founding director of CUBS; and Ifeoma Nwogu, PMCRT ’10 & PhD ’09, a CUBS research assistant professor. (Another researcher and co-author of the study, Nisha Bhaskaran, MS ’10, has left UB and is now a software developer in Los Angeles.) Together with Frank they obtained a National Science Foundation grant to pursue further work on these data. They took an optical disc containing excerpts from the 40 interviews—culled from about 130 total and chosen for diversity of age, sex and race—and wrote software that analyzed the subjects’ eye movements in excruciating detail. At issue: At the “gotcha” moment when the interviewer asked about the stolen check, did the subjects’ eyes begin to move in a different way? It was technically tricky work. The researchers had to deal with, for example, reflections from some subjects’ eyeglasses and the data disaster that resulted when someone’s hair fell over one eye. The algorithm looked at how often the person blinked and the direction of his gaze. The data was analyzed used Bayesian statistical techniques, which estimate the probability that two events (such as eye movement and lying) are related. “The baselines have to be established to see what is normal behavior for this person,” Govindaraju says, “and when the questions are being asked and there is an incentive to lie, what changes are taking place. The analysis is probabilistic in nature, so it will make some errors.” Nevertheless, the researchers found remarkable success: Better than eight times in 10, when the subjects answered at the
34 UBTODAY Fall 2012
www.buffalo.edu/UBT
“gotcha” moment, the algorithm detected telltale shifts in the subject’s eye movement.
A note of caution Frank cautions that the popular notion of a perfect “lie detector” is still a flight of fancy. Conventional systems measure heart and respiratory rates, blood pressure and perspiration, but do not directly measure lies. Researchers also have tried to use linguistic analysis, static images of facial expressions and even thermal imaging to detect deceit. The UB researchers’ algorithm points to physiological patterns that indicate something is going on with the person. That something could be a lie or it could be any of a thousand other events or emotions. Maybe, for example, one research subject stole a check in another context, and the interrogator’s “gotcha” question has brought back memories of that crime. And nobody’s completely comfortable parrying questions from a guy with a badge. “The problem,” Frank says, “is that there is no Pinocchio response to lying— there is no unique behavior that indicates a lie. Everything that co-occurs with a lie has been found to occur with other things. … What this technology is detecting at its core is not a lie. It reflects the underlying emotion or effortful thinking. What’s really happening is that we’re learning how to read people, how to detect ‘hot spots’ rather than lies. That can then be a pointer to further questions and areas of interest. It makes for more effective questioning.”
Learning from each other Both sides say working at the intersection of behavioral theory and computer analysis has been a fruitful way to collaborate. “Crossfertilization is important,” Frank says. “The big accomplishment is pairing two very diverse fields. A lot of computer science is done in a behavioral vacuum, but knowing where to look matters.” Adds Govindaraju: “Computers are good at looking at lots and lots of data and analyzing it, so there’s this notion of discovering new things. We might discover nuggets of knowledge, and we can go back to the behavioral scientists and say, does
this piece make sense and does this kind of correlation between the verbal behavior and the facial expression make sense?” The study’s success has drawn international attention. Scientific American came calling, as did the BBC. The researchers presented their findings at the 2011 IEEE Conference on Automatic Face and Gesture Recognition and published their paper, “Lie to Me: Deceit Detection Via Online Behavioral Learning,” in the proceedings of that conference.
Widening the scope The scientists are now looking at the full set of videotapes from Frank’s terrorism study and exploring the idea of expanding their algorithm to examine other facial cues: a scrunched forehead, a twitch of the lips, raised eyebrows. If the eyes are a window into the soul of a liar, might these other cues produce an even more accurate indicator? Nwogu also notes that for the algorithm to be fully useful in situations like police interrogations, it would have to produce its results instantly—something that would require further programming work. And while no one is ready to roll this technique out as a commercial application just yet, Frank says machine analysis could be used outside of law enforcement as well. For example, he says, it might help persons with autism spectrum disorders, like Asperger’s syndrome, to recognize the social cues that make for smooth human interaction, or to discover early indications of schizophrenia. Facial recognition technology “has the advantage of the human element,” Govindaraju says. “Faces are in public view, and most people can tell whether the facial images of two people are the same or different.” That’s not true, he says, for the other two major biometric instruments, fingerprint recognition and iris recognition. And as Nwogu says, “When you have someone who really knows how to hide their emotions, it would be useful to have a detector that might help law enforcement deal with the occasional expert liars they face. … These changes in the face happen in a fraction of a second. We don’t want to miss them.” Riley Mackenzie is a Buffalo freelance writer.