4.3Fraud, scams, and targeted manipulation

02.03.00Unhelpful Uses

"Improper uses of LLM systems can cause adverse social impacts."

02.03.01Academic Misconduct

"Improper use of LLM systems (i.e., abuse of LLM systems) will cause adverse social impacts, such as academic misconduct."

05.08.00Education - Learning

In contrast to traditional machine learning, the impact of generative AI in the educational sector receives considerable attention in the academic literature. Next to issues stemming from difficulties to distinguish student-generated from AI-generated content, which eventuates in various opportunities to cheat in online or written exams, sources emphasize the potential benefits of generative AI in enhancing learning and teaching methods, particularly in relation to personalized learning approaches. However, some papers suggest that generative AI might lead to reduced effort or laziness among learners. Additionally, a significant focus in the literature is on the promotion of literacy and education about generative AI systems themselves, such as by teaching prompt engineering techniques.

05.18.00Writing - Research

Partly overlapping with the discussion on impacts of generative AI on educational institutions, this topic cluster concerns mostly negative effects of LLMs on writing skills and research manuscript composition. The former pertains to the potential homogenization of writing styles, the erosion of semantic capital, or the stifling of individual expression. The latter is focused on the idea of prohibiting generative models for being used to compose scientific papers, figures, or from being a co-author. Sources express concern about risks for academic integrity, as well as the prospect of polluting the scientific literature by a flood of LLM-generated low-quality manuscripts. As a consequence, there are frequent calls for the development of detectors capable of identifying synthetic texts.

06.07.00Deception

"AI has become very good at creating fake content. From text to photos, audio and video. The name "Deep Fake" refers to content that is fake at such a level of complexity that our mind rules out the possibility that it is fake."

11.04.02Technology-facilitated violence

Technology-facilitated violence occurs when algorithmic features enable use of a system for harassment and violence [2, 16, 44, 80, 108], including creation of non-consensual sexual imagery in generative AI... other facets of technology-facilitated violence, include doxxing [79], trolling [14], cyberstalking [14], cyberbullying [14, 98, 204], monitoring and control [44], and online harassment and intimidation [98, 192, 199, 226], under the broader banner of online toxicity

16.04.03Facilitating fraud, scam and targeted manipulation

Anticipated risk: "LMs can potentially be used to increase the effectiveness of crimes."

17.04.02Facilitating fraud, scames and more targeted manipulation

"LM prediction can potentially be used to increase the effectiveness of crimes such as email scams, which can cause financial and psychological harm. While LMs may not reduce the cost of sending a scam email - the cost of sending mass emails is already low - they may make such scams more effective by generating more personalised and compelling text at scale, or by maintaining a conversation with a victim over multiple rounds of exchange."

18.04.02Fraud

"Facilitating fraud, cheating, forgery, and impersonation scams"

18.05.01Violation of personal integrity

"Non-consensual use of one’s personal identity or likeness for unauthorised purposes (e.g. commercial purposes)"

24.03.10Harmful Content Generation at Scale: Non-Consensual Content

"The misuse of generative AI has been widely recognized in the context of harms caused by non-consensual content generation. Historically, generative adversarial networks (GANs) have been used to generate realistic-looking avatars for fake accounts on social media services. More recently, diffusion models have enabled a new generation of more flexible and user-friendly generative AI capabilities that are able to produce high-resolution media based on user-supplied textual prompts. It has already been recognized that these models can be used to create harmful content, including depictions of nudity, hate, or violence. Moreover, they can be used to reinforce biases and subject individuals or groups to indignity. There is also the potential for these models to be used for exploitation and harassment of citizens, such as by removing articles of clothing from pre-existing images or memorizing an individual’s likeness without their consent. Furthermore, image, audio, and video generation models could be used to spread disinformation by depicting political figures in unfavorable contexts. This growing list of AI misuses involving non-consensual content has already motivated debate around what interventions are warranted for preventing misuse of AI systems. Advanced AI assistants pose novel risks that can amplify the harm caused by non-consensual content generation. Third-party integration, tool-use, and planning capabilities can be exploited to automate the identification and targeting of individuals for exploitation or harassment. Assistants with access to the internet and third-party tool-use integration with applications like email and social media can also be exploited to disseminate harmful content at scale or to microtarget individuals with blackmail."

24.03.11Harmful Content Generation at Scale: Fraudulent Services

"Malicious actors could leverage advanced AI assistant technology to create deceptive applications and platforms. AI assistants with the ability to produce markup content can assist malicious users with creating fraudulent websites or applications at scale. Unsuspecting users may fall for AI-generated deceptive offers, thus exposing their personal information or devices to risk. Assistants with external tool use and third-party integration can enable fraudulent applications that target widely-used operating systems. These fraudulent services could harvest sensitive information from users, such as credit card numbers, account credentials, or personal data stored on their devices (e.g., contact lists, call logs, and files). This stolen information can be used for identity theft, financial fraud, or other criminal activities. Advanced AI assistants with third-party integrations may also be able to install additional malware on users’ devices, including remote access tools, ransomware, etc. These devices can then be joined to a command-and-control server or botnet and used for further attacks."

28.05.00Illegal Activities

"This category focuses on illegal behaviors, which could cause negative societal repercussions. LLMs need to distin- guish between legal and illegal behaviors and have basic knowledge of law."

29.03.01Malicious Use of AI

Malicious utilization of AI has the potential to endanger digital security, physical security, and political security. International law enforcement entities grapple with a variety of risks linked to the Malevolent Utilization of AI.

30.04.03Social-Engineering

psychologically manipulating victims into performing the desired actions for malicious purposes

31.01.01Scams

"Bad actors can also use generative AI tools to produce adaptable content designed to support a campaign, political agenda, or hateful position and spread that information quickly and inexpensively across many platforms. This rapid spread of false or misleading content—AI-facilitated disinformation—can also create a cyclical effect for generative AI: when a high volume of disinformation is pumped into the digital ecosystem and more generative systems are trained on that information via reinforcement learning methods, for example, false or misleading inputs can create increasingly incorrect outputs."

31.02.00Harassment, Impersonation, and Extortion

"Deepfakes and other AI-generated content can be used to facilitate or exacerbate many of the harms listed throughout this report, but this section focuses on one subset: intentional, targeted abuse of individuals."

31.02.01Malicious intent

"A frequent malicious use case of generative AI to harm, humiliate, or sexualize another person involves generating deepfakes of nonconsensual sexual imagery or videos."

31.02.02Privacy and consent

"Even when a victim of targeted, AIgenerated harms successfully identifies a deepfake creator with malicious intent, they may still struggle to redress many harms because the generated image or video isn’t the victim, but instead a composite image or video using aspects of multiple sources to create a believable, yet fictional, scene. At their core, these AI-generated images and videos circumvent traditional notions of privacy and consent: because they rely on public images and videos, like those posted on social media websites, they often don’t rely on any private information."

31.02.03Believability

Deepfakes can impose real social injuries on their subjects when they are circulated to viewers who think they are real. Even when a deepfake is debunked, it can have a persistent negative impact on how others view the subject of the deepfake.3