2.2AI system security vulnerabilities and attacks

02.03.04Software Vulnerabilities

"Programmers are accustomed to using code generation tools such as Github Copilot for program development, which may bury vulnerabilities in the program."

02.04.00Software Security Issues

"The software development toolchain of LLMs is complex and could bring threats to the developed LLM."

02.04.01Programming Language

"Most LLMs are developed using the Python language, whereas the vulnerabilities of Python interpreters pose threats to the developed models"

02.04.02Deep Learning Frameworks

"LLMs are implemented based on deep learning frameworks. Notably, various vulnerabilities in these frameworks have been disclosed in recent years. As reported in the past five years, three of the most common types of vulnerabilities are buffer overflow attacks, memory corruption, and input validation issues."

02.04.03Software Supply Chains

"The software development toolchain of LLMs is complex and could bring threats to the developed LLM."

02.04.04Pre-processing Tools

"Pre-processing tools play a crucial role in the context of LLMs. These tools, which are often involved in computer vision (CV) tasks, are susceptible to attacks that exploit vulnerabilities in tools such as OpenCV."

02.05.00Hardware Vulnerabilities

"The vulnerabilities of hardware systems for training and inferencing brings issues to LLM-based applications."

02.05.01Network Devices

"The training of LLMs often relies on distributed network systems [171], [172]. During the transmission of gradients through the links between GPU server nodes, significant volumetric traffic is generated. This traffic can be susceptible to disruption by burst traffic, such as pulsating attacks [161]. Furthermore, distributed training frameworks may encounter congestion issues [173]."

02.05.02GPU Computation Platforms

"The training of LLMs requires significant GPU resources, thereby introducing an additional security concern. GPU side-channel attacks have been developed to extract the parameters of trained models [159], [163]."

02.05.03Memory and Storage

"Similar to conventional programs, hardware infrastructures can also introduce threats to LLMs. Memory-related vulnerabilities, such as rowhammer attacks [160], can be leveraged to manipulate the parameters of LLMs, giving rise to attacks such as the Deephammer attack [167], [168]."

02.06.00Issues on External Tools

"The external tools (e.g., web APIs) present trustworthiness and privacy issues to LLM-based applications."

02.06.01Factual Errors Injected by External Tools

"External tools typically incorporate additional knowledge into the input prompts [122], [178]–[184]. The additional knowledge often originates from public resources such as Web APIs and search engines. As the reliability of external tools is not always ensured, the content returned by external tools may include factual errors, consequently amplifying the hallucination issue."

02.06.02Exploiting External Tools for Attacks

"Adversarial tool providers can embed malicious instructions in the APIs or prompts [84], leading LLMs to leak memorized sensitive information in the training data or users’ prompts (CVE2023-32786). As a result, LLMs lack control over the output, resulting in sensitive information being disclosed to external tool providers. Besides, attackers can easily manipulate public data to launch targeted attacks, generating specific malicious outputs according to user inputs. Furthermore, feeding the information from external tools into LLMs may lead to injection attacks [61]. For example, unverified inputs may result in arbitrary code execution (CVE-2023-29374)."

02.10.00Model Attacks

Model attacks exploit the vulnerabilities of LLMs, aiming to steal valuable information or lead to incorrect responses.

02.10.01Extraction Attacks

"Extraction attacks [137] allow an adversary to query a black-box victim model and build a substitute model by training on the queries and responses. The substitute model could achieve almost the same performance as the victim model. While it is hard to fully replicate the capabilities of LLMs, adversaries could develop a domainspecific model that draws domain knowledge from LLMs"

02.10.02Inference Attacks

"Inference attacks [150] include membership inference attacks, property inference attacks, and data reconstruction attacks. These attacks allow an adversary to infer the composition or property information of the training data. Previous works [67] have demonstrated that inference attacks could easily work in earlier PLMs, implying that LLMs are also possible to be attacked"

02.10.03Poisoning Attacks

"Poisoning attacks [143] could influence the behavior of the model by making small changes to the training data. A number of efforts could even leverage data poisoning techniques to implant hidden triggers into models during the training process (i.e., backdoor attacks). Many kinds of triggers in text corpora (e.g., characters, words, sentences, and syntax) could be used by the attackers.""

02.10.04Overhead Attacks

"Overhead attacks [146] are also named energy-latency attacks. For example, an adversary can design carefully crafted sponge examples to maximize energy consumption in an AI system. Therefore, overhead attacks could also threaten the platforms integrated with LLMs."

02.10.05Novel Attacks on LLMs

Table of examples has: "Prompt Abstraction Attacks [147]: Abstracting queries to cost lower prices using LLM’s API. Reward Model Backdoor Attacks [148]: Constructing backdoor triggers on LLM’s RLHF process. LLM-based Adversarial Attacks [149]: Exploiting LLMs to construct samples for model attacks"

02.10.06Evasion Attacks

"Evasion attacks [145] target to cause significant shifts in model’s prediction via adding perturbations in the test samples to build adversarial examples. In specific, the perturbations can be implemented based on word changes, gradients, etc."