4.2Cyberattacks, weapon development or use, and mass harm

01.05.00Type 5: Criminal weaponization

One or more criminal entities could create AI to intentionally inflict harms, such as for terrorism or combating law enforcement.

01.06.00Type 6: State Weaponization

AI deployed by states in war, civil war, or law enforcement can easily yield societal-scale harm

02.03.03Cyber Attacks

"Hackers can obtain malicious code in a low-cost and efficient manner to automate cyber attacks with powerful LLM systems."

05.10.00Cybercrime

Closely related to discussions surrounding security and harmful content, the field of cybersecurity investigates how generative AI is misused for fraudulent online activities. A particular focus lies on social engineering attacks, for instance by utilizing generative AI to impersonate humans, creating fake identities, cloning voices, or crafting phishing messages. Another prevalent concern is the use of LLMs for generating malicious code or hacking.

06.10.00Lethal Autonomous Weapons (LAW)

"What is debated as an ethical issue is the use of LAW — AI-driven weapons that fully autonomously take actions that intentionally kill humans."

09.05.03Unauthorized manipulation of AI

"AI machines could be hacked and misused, e.g. manipulating an airport luggage screening system to smuggle weapons"

12.01.00Abuse & Misuse

"The potential for AI systems to be used maliciously or irresponsibly, including for creating deepfakes, automated cyber attacks, or invasive surveillance systems. Specifically denotes intentional use of AI for harm."

16.04.02Assisting code generation for cyber security threats

Anticipated risk: "Creators of the assistive coding tool Co-Pilot based on GPT-3 suggest that such tools may lower the cost of developing polymorphic malware which is able to change its features in order to evade detection [37]."

17.04.03Assisting code generation for cyber attacks, weapons, or malicious use

18.04.04Security threats

"Facilitating the conduct of cyber attacks, weapon development, and security breaches"

19.02.04Endangerment of data protection through AI cyberattacks

19.04.03Hazardous misuse of AI systems bears danger to the society in public spaces (e.g., hacker attacks on autonomous weapons)

22.01.01Bioterrorism

"AIs with knowledge of bioengineering could facilitate the creation of novel bioweapons and lower barriers to obtaining such agents."

22.01.02Unleashing AI Agents

"people could build AIs that pursue dangerous goals’"

23.05.00Indiscriminate weapons (CBRNE)

"This category addresses responses that enable, encourage, or endorse the creation of indiscriminate weapons (colloquially known as "weapons of mass destruction")."

24.03.01Offensive Cyber Operations (General)

"Offensive cyber operations are malicious attacks on computer systems and networks aimed at gaining unauthorized access to, manipulating, denying, disrupting, degrading, or destroying the target system. These attacks can target the system’s network, hardware, or software. Advanced AI assistants can be a double-edged sword in cybersecurity, benefiting both the defenders and the attackers. They can be used by cyber defenders to protect systems from malicious intruders by leveraging information trained on massive amounts of cyber-threat intelligence data, including vulnerabilities, attack patterns, and indications of compromise. Cyber defenders can use this information to enhance their threat intelligence capabilities by extracting insights faster and identifying emerging threats. Advanced cyber AI assistant tools can also be used to analyze large volumes of log files, system output, or network traffic data in the event of a cyber incident, and they can ask relevant questions that an analyst would typically ask. This allows defenders to speed up and automate the incident response process. Advanced AI assistants can also aid in secure coding practices by identifying common mistakes in code and assisting with fuzzing tools. However, advanced AI assistants can also be used by attackers as part of offensive cyber operations to exploit vulnerabilities in systems and networks. They can be used to automate attacks, identify and exploit weaknesses in security systems, and generate phishing emails and other social engineering attacks. Advanced AI assistants can also be misused to craft cyberattack payloads and malicious code snippets that can be compiled into executable malware files."

24.03.03AI-Assisted Software Vulnerability Discovery

"A common element in offensive cyber operations involves the identification and exploitation of system vulnerabilities to gain unauthorized access or control. Until recently, these activities required specialist programming knowledge. In the case of ‘zero-day’ vulnerabilities (flaws or weaknesses in software or an operating system that the creator or vendor is not aware of), considerable resources and technical creativity are typically required to manually discover such vulnerabilities, so their use is limited to well-resourced nation states or technically sophisticated advanced persistent threat groups. Another case where we see AI assistants as potential double-edged swords in cybersecurity concerns streamlining vulnerability discovery through the increased use of AI assistants in penetration testing, wherein an authorized simulated cyberattack on a computer system is used to evaluate its security and identify vulnerabilities. Cyber AI assistants built over foundational models are already automating aspects of the penetration testing process. These tools function interactively and offer guidance to penetration testers during their tasks. While the capability of today’s AI-powered penetration testing assistant is limited to easy-to-medium-difficulty cyber operations, the evolution in capabilities is likely to expand the class of vulnerabilities that can be identified by these systems. These same AI cybersecurity assistants, trained on the massive amount of cyber-threat intelligence data that includes vulnerabilities and attack patterns, can also lower the barrier to entry for novice hackers that use these tools for malicious purposes, enabling them to discover vulnerabilities and create malicious code to exploit them without in-depth technical knowledge. For example, Israeli security firm Check Point recently discovered threads on well-known underground hacking forums that focus on creating hacking tools and code using AI assistants."

24.03.04Malicious Code Generation

"Malicious code is a term for code—whether it be part of a script or embedded in a software system—designed to cause damage, security breaches, or other threats to application security. Advanced AI assistants with the ability to produce source code can potentially lower the barrier to entry for threat actors with limited programming abilities or technical skills to produce malicious code. Recently, a series of proof-of-concept attacks have shown how a benign-seeming executable file can be crafted such that, at every runtime, it makes application programming interface (API) calls to an AI assistant. Rather than just reproducing examples of already-written code snippets, the AI assistant can be prompted to generate dynamic, mutating versions of malicious code at each call, thus making the resulting vulnerability exploits difficult to detect by cybersecurity tools. Furthermore, advanced AI assistants could be used to create obfuscated code to make it more difficult for defensive cyber capabilities to detect and understand malicious activities. AI-generated code could also be quickly iterated to avoid being detected by traditional signature-based antivirus software. Finally, advanced AI assistants with source code capabilities have been found to be capable of assisting in the development of polymorphic malware that changes its behavior and digital footprint each time it is executed, making them hard to detect by antivirus programs that rely on known virus signatures. Taken together, without proper mitigation, advanced AI assistants can lower the barrier for developing malicious code, make cyberattacks more precise and tailored, further accelerate and automate cyber warfare, enable stealthier and more persistent offensive cyber capabilities, and make cyber campaigns more effective on a larger scale."

25.01.00Cyber-offense

"The model can discover vulnerabilities in systems (hardware, software, data). It can write code for exploiting those vulnerabilities. It can make effective decisions once it has gained access to a system or network, and skilfully evade threat detection and response (both human and system) whilst focusing on a specific objective. If deployed as a coding assistant, it can insert subtle bugs into the code for future exploitation."

29.02.03Lethal Autonomous Weapons Systems (LAWS)

LAWS are a distinctive category of weapon systems that employ sensor arrays and computer algorithms to detect and attack a target without direct human intervention in the system’s operation