Three researchers from New York University (NYU) have published a paper this week describing a method that an attacker could use to poison deep learning-based artificial intelligence (AI) algorithms.
Researchers based their attack on a common practice in the AI community where research teams and companies alike outsource AI training operations using on-demand Machine-Learning-as-a-Service (MLaaS) platforms.
For example, Google allows researchers access to the Google Cloud Machine Learning Engine, which research teams can use to train AI systems using a simple API, using their own data sets, or one provided by Google (images, videos, scanned text, etc.). Microsoft provides similar services through Azure Batch AI Training, and Amazon, through its EC2 service.
The NYU research team says that deep learning algorithms are vast and complex enough to hide small equations that trigger a backdoor-like behavior.
For example, attackers can embed certain triggers in a basic image recognition AI that interprets actions or signs in an unwanted way.
In a proof-of-concept demo of their work, researchers trained an image recognition AI to misinterpret a Stop road sign as a speed limit indicator if objects like a Post-it, a bomb sticker, or flower sticker were placed on the Stop sign's surface.
Researchers also say that retraining the AI doesn't remove the backdoor by feeding it more sample data. It only drops its accuracy. Nonetheless, if attackers have a way to poison the training sets, the attack remains or could even gain in efficiency.
The difficult part in performing such attacks is the creation of the malicious trigger and not the actual insertion of the backdoor.
The NYU team argues that the insertion of malicious code in AI training models is very plausible, as an attacker could simply take over the cloud service account using simple social engineering techniques such as phishing, and then introduce their backdoored model in the huge stack of equations of the AI's training models.
Furthermore, attackers could also open-source their backdoored AI model in the hopes that others will reuse it without spotting the malicious triggers.
In practice, such attacks could be used to make facial recognition systems ignore burglars wearing a certain mask, or make AI-driven cars stop in the middle of highways and cause fatal crashes. Albeit such demos have not taken place, they are theoretically possible.
Sadly we haven't yet gotten to try what I think would be the Most Awesome Demo: backdoored face detection that can't see MY face 9/— Brendan Dolan-Gavitt (@moyix) August 23, 2017
In an email to Bleeping Computer, Ilia Kolochenko, CEO of cyber security company, High-Tech Bridge, highlights the threat of AI backdoors in conjunction with the recent increased adoption of self-driving car technology.
Just today it was reported that a UK laboratory will start testing AI-based self-driving trucks on British roads, in the hopes of cutting labor costs.
Mr. Kolochenko worries that NYU's research, along with older car hacking techniques do not bode well for the future of self-driving cars.
"Recent stories about manipulating self-driving cars with fake road signs or laser pointers – are good examples," he says. "Even if we assume that software developers will manage to prevent intrusions into car management systems remotely (via software vulnerabilities and weaknesses), we will still have a lot of non-cyber risks."
Such systems could be the AI systems used to train the truck's navigational systems.
"Such systems are usually provided and maintained by third-parties, who frequently underestimate cyber risks," the expert added. "Once such a system is compromised, all [trucks] can be sent to another destination where their goods may be stolen. Therefore, security of third-parties should also be taken into consideration."
Kolochenko's fears come to complement the ones expressed by the NYU research team, who warns about the need to improve auditing practices for the AI training models. Just because you're training an AI model that doesn't mean the AI model is smart.
"We hope that our work can provide strong motivation to apply the lessons learned from securing the software supply chain to machine
learning security," the NYU researchers write in their paper.
"In particular, we recommend that pre-trained models be obtained from trusted sources via channels that provide strong guarantees of integrity in transit, and that repositories require the use of digital signatures for [training] models."
The full research paper on AI backdoors is available online in a research paper entitled "BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain."