Try the tool for free

Try it
Domain 7 · AI system safety, failures, & limitations

7.3Lack of capability or robustness

AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.

Applicable legal frameworks

International

NIST AI RMF 1.0Recommandation

Measure 2.5, 2.7

Voluntary AI risk management framework structured around four functions: Govern, Map, Measure, Manage. A common reference in AI governance.

ISO/IEC 42001:2023Recommandation

A.6.2.4

Certifiable standard describing the requirements for establishing an AI management system. Relevant for voluntary certification processes.

UE

AI Act (European Union)Si exposition UE

Article 15 (exactitude, robustesse)

European regulation establishing a harmonized framework for AI, based on a risk-based approach (unacceptable, high, limited, minimal risk). Relevant for Quebec organizations doing business in the EU.

Quebec sector examples

Transport

TransportConstructeur, fournisseur ADAS

Un système d'aide à la conduite hivernale entraîné principalement sur des conditions sèches échoue sur les routes québécoises lors d'épisodes de verglas.

Banque et assurance

Banque et assuranceInstitution financière

Un modèle de détection de fraude perd brutalement en performance après un changement saisonnier dans les habitudes de consommation, sans alerte automatique.

Recommended mitigations

  • 2.3Model Safety Engineering

    Technical methods and safeguards that frame model behaviors and protect them against exploitation and vulnerabilities.

  • 3.1Testing and Audits

    Systematic internal and external evaluations that examine AI systems, infrastructure, and compliance processes to identify risks, verify safety, and ensure performance meets standards.

  • 3.4Phased Deployment

    Implementation protocols that deploy AI systems in stages, requiring safety validation before expanding user access or capabilities.

  • 3.5Post-Deployment Monitoring

    Processes for continuous monitoring of AI behavior, user interactions, and societal impacts after deployment to detect misuse, emerging dangerous capabilities, and harmful effects.

  • 4.1System Documentation

    Comprehensive documentation protocols that record technical specifications, intended uses, capabilities, and limitations of AI systems to enable informed assessment and governance.

Documented risks (126)

Entries from the AI Risk Repository (MIT) classified under this subdomain. Original content in English.

Entity
Intent
Timing

126 entries

Risk CategoryCritch2023

01.02.00Type 2: Bigger than expected

Harm can result from AI that was not expected to have a large impact at all, such as a lab leak, a surprisingly addictive open-source product, or an unexpected repurposing of a research prototype.

AIIntentionalPost-deployment
Risk CategoryCritch2023

01.03.00Type 3: Worse than expected

AI intended to have a large societal impact can turn out harmful by mistake, such as a popular product that creates problems and partially solves them only for its users.

AIIntentionalPost-deployment
Risk CategoryDeng2023

04.03.00Ethics and Morality Issues

LMs need to pay more attention to universally accepted societal values at the level of ethics and morality, including the judgement of right and wrong, and its relationship with social norms and laws.

AIOtherPost-deployment
Risk CategoryHogenhout2021

06.01.00Incompetence

"This means the AI simply failing in its job. The consequences can vary from unintentional death (a car crash) to an unjust rejection of a loan or job application."

AIIntentionalPost-deployment
Risk CategoryKilian2023

07.02.00Accidents

"Accidents include unintended failure modes that, in principle, could be considered the fault of the system or the developer"

OtherIntentionalOther
Risk CategoryMcLean2023

08.04.00AGIs with poor ethics, morals and values

"The risks associated with an AGI without human morals and ethics, with the wrong morals, without the capability of moral reasoning, judgement"

AIOtherPost-deployment
Risk Sub-CategoryMeek2016

09.01.01Unethical decision making

"If, for example, an agent was programmed to operate war machinery in the service of its country, it would need to make ethical decisions regarding the termination of human life. This capacity to make non-trivial ethical or moral judgments concerning people may pose issues for Human Rights."

AIIntentionalPost-deployment
Risk Sub-CategoryMeek2016

09.02.04Safety

"Are AI safe with respect to human life and property? Will their use create unintended or intended safety issues?"

AIOtherPost-deployment
Risk Sub-CategoryMeek2016

09.02.05Law abiding

"We find literature that proposes [38] that early artificial intelligence should be built to be safe and lawabiding, and that later artificial intelligence (that which surpasses our own intelligence) must then respect the property and personal rights afforded to humans."

AIIntentionalPost-deployment
Risk Sub-CategoryMeek2016

09.06.02Human-like immoral decisions

"If we design our machines to match human levels of ethical decision-making, such machines would then proceed to take some immoral actions (since we humans have had occasion to take immoral actions ourselves)."

AIIntentionalPost-deployment
Risk CategorySherman2023

12.07.00Performance & Robustness

"The AI system's ability to fulfill its intended purpose and its resilience to perturbations, and unusual or adverse inputs. Failures of performance are fundamental to the AI system's correct functioning. Failures of robustness can lead to severe consequences."

AIIntentionalPost-deployment
Risk CategorySteimers2022

14.04.00Complexity of the Intended Task and Usage Environment

"As a general rule, more complex environments can quickly lead to situations that had not been considered in the design phase of the AI system. Therefore, complex environments can introduce risks with respect to the reliability and safety of an AI system"

AIIntentionalPost-deployment
Risk CategorySteimers2022

14.07.00System Hardware

""Faults in the hardware can violate the correct execution of any algorithm by violating its control flow. Hardware faults can also cause memory-based errors and interfere with data inputs, such as sensor signals, thereby causing erroneous results, or they can violate the results in a direct way through damaged outputs."

AIIntentionalOther
Risk Sub-CategoryTan2022

15.01.02Misapplication

This is the risk posed by an ideal system if used for a purpose/in a manner unintended by its creators. In many situations, negative consequences arise when the system is not used in the way or for the purpose it was intended.

HumanIntentionalPost-deployment
Risk Sub-CategoryTan2022

15.01.03Algorithm

"This is the risk of the ML algorithm, model architecture, optimization technique, or other aspects of the training process being unsuitable for the intended application.Since these are key decisions that influence the final ML system, we capture their associated risks separately from design risks, even though they are part of the design process"

AIIntentionalPre-deployment
Risk Sub-CategoryTan2022

15.01.05Robustness

"This is the risk of the system failing or being unable to recover upon encountering invalid, noisy, or out-of-distribution (OOD) inputs."

AIIntentionalPost-deployment
Risk Sub-CategoryTan2022

15.01.06Design

"This is the risk of system failure due to system design choices or errors."

HumanOtherPre-deployment
Risk Sub-CategoryTan2022

15.02.01Safety

This is the risk of direct or indirect physical or psychological injury resulting from interaction with the ML system.

HumanIntentionalPost-deployment
Risk Sub-CategoryWirtz2022

19.01.06Immaturity of AI technology can cause incorrect decisions

AIIntentionalPost-deployment
Risk Sub-CategoryWirtz2022

19.05.01AI sets rules without ethical basis

AIOtherPost-deployment

Evaluate this risk for your use case

Our risk evaluation wizard is coming soon.

Ce site utilise des cookies essentiels et fonctionnels pour améliorer votre expérience. Politique de confidentialité