INDUSTRY COMMENT
ADVERSARIAL ATTACKS AND HUMAN-AI DEFENCE TEAMS
Analyst-AI teams provide faster, more thorough cyber defence than either the analyst or the AI could alone, explains Vectra’s Christopher Thissen.
D CHRISTOPHER THISSEN, Data Scientist, Vectra.
Despite the astonishing advances in AI performance over the past few years, no AI is perfect.
50
espite the astonishing advances in AI performance over the past few years, no AI is perfect. In fact, an AI’s imperfection is usually made explicit by measuring the model’s accuracy on a test dataset; perfect scores are neither expected nor common. Problematically, AIs also make mistakes in ways not captured by tests. The most famous examples use imperceptible changes to create surprising image labelling errors. Even highly sophisticated AI’s capable of defeating professional human players in complex games like DoTA II and StarCraft II have later to been shown to have vulnerabilities exploitable by lessskilled human players. In the cyber domain, Skylight Cyber recently circumvented an AI malware detector by appending strings to known malicious binaries. Troublingly, this method worked for every malicious binary they tested. These examples fall under a nascent machine learning field called adversarial attacks, defined as inputs to machine learning models that an attacker has intentionally designed to cause the model to make a mistake. There are two broad responses to adversarial attacks: l Increase an AI’s robustness to adversarial examples l Supplement AIs using defence in depth One intriguing idea is that some human skills complement those of the AI, and that human-AI teams
provide advantages over either entity alone. In his book Average is Over, Tyler Cowan documents the early success of human-AI teams in freestyle chess. In this time-limited version of the game, human-AI teams initially dominated teams comprising only AIs or only humans even against opponents that were individually much better players. In the most successful teams, the human players used intimate knowledge of their AI teammates to guide computation and exploit weaknesses. Human grandmasters, in contrast, relied more on their own knowledge and lost by failing to leverage their AIs effectively. The winning edge in these tournaments was an intuitive and rapid ability to train and corral AI teammates. Although interest in freestyle chess has waned as AI’s have become more powerful, interest in developing human-AI teams in more complex domains continues to grow. As a simple example, a human would clearly recognise the differences among the adversarial examples in Figure 2, but to an AI all the images may appear equally different from the original for the given distance metric. A human would be able to correct the labelling errors, though then the trick is to identify which examples need corrections. In these examples, a human provides complementary skills, but the caveat here being that it is possible to construct adversarial
biznesstransform.com