A Classification of Safety Risks in Medical AI
A Practical Framework for Patient Safety, Medical Imaging, Clinical AI, and Lifecycle Governance
Why Does Medical AI Safety Require Its Own Risk Classification?
Artificial intelligence is becoming embedded in clinical workflows at a pace that conventional software governance was not designed to accommodate.
Medical AI can detect abnormalities on radiographs, prioritize emergency CT examinations, quantify cardiac function, identify lesions on magnetic resonance imaging (MRI), summarize clinical information, support triage, and increasingly interact directly with clinicians through generative AI.
The central safety problem, however, is not simply whether an algorithm is "accurate."
A medical AI system can achieve excellent performance in a development dataset and still become unsafe after deployment because the patient population changes, imaging equipment changes, acquisition protocols change, disease prevalence changes, the user misunderstands the output, the model becomes outdated, or the surrounding clinical workflow creates a new failure mode.
The World Health Organization (WHO) has emphasized that AI in healthcare introduces risks involving safety, cybersecurity, bias, accountability, transparency, autonomy, and equity.
The National Institute of Standards and Technology (NIST) similarly frames AI risk as a combination of the likelihood of an adverse event and the magnitude of its consequences, emphasizing continuous risk management across the AI lifecycle.
For healthcare, this suggests a fundamental principle:
Medical AI safety is not a single technical property. It is a system-level characteristic that emerges from data, algorithms, clinical users, infrastructure, workflow, governance, and time.
A useful classification therefore needs to go beyond "algorithm error."
1. What Are the Major Categories of Medical AI Safety Risk?
Medical AI safety risks can be organized into ten interconnected categories:
Data and Dataset Risks
Algorithmic and Model Risks
Clinical Diagnostic Risks
Human Factors and Cognitive Risks
Workflow and System Integration Risks
Bias, Fairness, and Equity Risks
Privacy and Cybersecurity Risks
Lifecycle, Drift, and Change-Management Risks
Governance, Regulatory, and Accountability Risks
Generative AI and Foundation-Model Risks
These categories overlap.
For example, a radiology AI system may produce a false-negative result because of poor image quality. That appears to be an algorithmic problem, but the underlying cause may actually be a dataset problem, an acquisition problem, a domain-shift problem, or a user-interface problem.
Therefore, safety investigation should ask not only:
"Did the AI make an error?"
but also:
"Why did the clinical system allow that error to reach the patient?"
2. Data and Dataset Risks
Why can training data become a patient-safety problem?
Medical AI learns from data, and the characteristics of that data strongly influence the behavior of the resulting model.
A dataset may contain:
demographic imbalance
geographic bias
institutional bias
disease-spectrum bias
scanner-specific characteristics
protocol-specific characteristics
labeling errors
missing data
duplicated examinations
inappropriate reference standards
hidden correlations
selection bias
temporal bias
Medical imaging provides particularly clear examples.
A chest radiograph AI model may learn associations between disease labels and hospital-specific image characteristics rather than the pathology itself. A CT model may perform differently when scanner vendors, reconstruction kernels, slice thickness, contrast protocols, or patient populations change.
Recent work examining public chest-radiography datasets identified label-quality problems, domain shift, population bias, and external performance degradation as important limitations for AI evaluation.
This creates an important distinction:
Large dataset ≠ safe dataset.
A million images from a narrow population may be less clinically useful than a smaller but carefully curated, diverse, and representative dataset.
Major dataset safety risks
| Risk | Mechanism | Clinical consequence | Mitigation |
|---|---|---|---|
| Sampling bias | Nonrepresentative population | Poor performance in underrepresented patients | Diverse datasets |
| Label noise | Incorrect reference labels | Misclassification | Expert validation |
| Spectrum bias | Limited disease severity | Poor real-world sensitivity | Broader case mix |
| Site bias | Single-institution characteristics | External performance degradation | Multicenter validation |
| Scanner bias | Hardware-specific features | Vendor/protocol dependence | Cross-vendor testing |
| Temporal bias | Historical practice patterns | Outdated predictions | Temporal validation |
| Hidden leakage | Information indirectly predicts label | Inflated performance | Strict dataset design |
The risk is especially important when the model appears highly accurate internally.
Internal validation answers:
"Can the model reproduce patterns in data similar to its training environment?"
External validation asks:
"Does the model remain safe when the environment changes?"
The second question is usually closer to clinical reality.
3. Algorithmic and Model Risks
What can go wrong inside the AI model?
Even with high-quality data, the model itself can fail.
Relevant mechanisms include:
overfitting
underfitting
unstable predictions
poor calibration
threshold sensitivity
class imbalance
shortcut learning
adversarial sensitivity
uncertainty that is not communicated
inappropriate extrapolation
failure outside the training distribution
A model may therefore have a respectable area under the receiver operating characteristic curve (AUROC) while still producing clinically unsafe predictions.
For clinical deployment, discrimination is only one component of performance.
A safer evaluation framework considers:
Discrimination + calibration + robustness + generalizability + clinical utility + failure behavior.
This is particularly important for decision-support systems.
If an AI model produces a probability of 0.91, clinicians may interpret that number as highly reliable. But if the model is poorly calibrated, the numerical confidence may not correspond to the actual probability of disease.
The problem is not merely that the model is wrong.
The problem is that the model may be wrong with confidence.
4. Clinical Diagnostic Risks
How can an AI error become a patient-safety event?
The most direct clinical risk is an incorrect recommendation.
Medical AI may produce:
false-positive findings
false-negative findings
incorrect classifications
incorrect prioritization
inappropriate risk scores
incorrect treatment suggestions
missed urgent findings
misleading summaries
In medical imaging, a false negative may delay diagnosis.
A false positive may trigger unnecessary imaging, biopsy, intervention, hospitalization, or patient anxiety.
The consequences therefore depend not only on model accuracy but on clinical context.
Missing a small benign lesion and missing a pulmonary embolism are not equivalent events.
A clinically meaningful risk classification should therefore include:
Severity of consequence
negligible
minor
moderate
serious
catastrophic
Probability of occurrence
rare
occasional
probable
frequent
Detectability
immediately recognized
detectable by routine review
difficult to recognize
effectively hidden
This resembles conventional medical-device risk management but must account for AI-specific behavior.
WHO specifically notes that incorrect recommendations and false-positive or false-negative outputs can create patient harm, and that widespread algorithmic errors can potentially affect many patients rapidly.
5. Automation Bias and Human Factors
Can a human clinician make AI less safe?
Yes.
One of the most important medical AI risks does not originate from the algorithm itself.
It originates from the interaction between the algorithm and the clinician.
Automation bias occurs when users place excessive trust in an automated recommendation and fail to adequately challenge it.
Consider a radiologist reviewing a chest CT.
The AI reports:
"No pulmonary embolism detected."
If the radiologist subsequently pays less attention to the pulmonary arteries, the AI has influenced the diagnostic process even if its output is not explicitly accepted as the final diagnosis.
This creates a critical distinction:
AI assistance is not neutral.
The presence, formatting, timing, confidence display, and placement of an AI result can change human behavior.
Human factors include:
alert fatigue
confirmation bias
automation bias
anchoring
overreliance
underuse of AI
misunderstanding of uncertainty
poor interface design
cognitive overload
deskilling
Recent literature has highlighted automation bias and possible clinician deskilling as important concerns in generative AI-enabled healthcare.
This is why clinical AI evaluation cannot stop at model-level accuracy.
It must examine:
AI + clinician + interface + workflow.
The DECIDE-AI framework specifically emphasizes early clinical evaluation of AI decision-support systems, including safety and human factors in live clinical environments.
6. Workflow and Integration Risks
What happens when a good AI system is inserted into a bad workflow?
A technically strong algorithm can become clinically unsafe when integrated poorly.
Consider a radiology workflow:
CT acquisition → PACS → AI processing → AI result → radiologist workstation → report → EHR → clinical team
Every interface represents a potential failure point.
Possible failures include:
examination not transmitted to AI
incorrect series selected
AI processing delay
result delivered after clinical decision
AI result displayed in an inappropriate location
alert buried among other notifications
AI result not synchronized with the correct patient
failed communication between PACS and AI platform
missing audit trail
These are not necessarily "AI model failures."
They are AI system failures.
Therefore, medical AI safety should distinguish:
Model safety
Does the algorithm perform appropriately?
System safety
Does the complete technology perform appropriately?
Clinical safety
Does its use improve or at least preserve patient safety?
A model may be safe in isolation but unsafe in a particular workflow.
7. Bias, Fairness, and Equity Risks
Why is AI bias particularly important in healthcare?
Healthcare AI may perform differently across:
age groups
sexes
ethnic populations
geographic populations
socioeconomic groups
body habitus
comorbidity profiles
disability groups
disease subtypes
WHO identifies inclusiveness and equity as core principles for AI in health and warns that models trained in one population may not perform appropriately when deployed in another.
Medical imaging research has also demonstrated that models can exploit demographic shortcuts and that correcting performance within one dataset does not necessarily guarantee fairness in a new external setting.
This leads to a subtle but important point:
Fairness is not a one-time statistical adjustment.
A model can be fair in one population and unfair in another.
Therefore, fairness assessment should be performed:
before deployment
during validation
after deployment
after major model changes
across clinically meaningful subgroups
8. Privacy and Cybersecurity Risks
Can medical AI create cybersecurity risks?
Yes.
Medical AI systems operate within highly sensitive digital ecosystems containing:
medical images
electronic health records
genomic information
laboratory data
clinical notes
biometric information
patient identifiers
Security risks include:
unauthorized data access
insecure APIs
compromised model servers
malicious input manipulation
data poisoning
model theft
privacy leakage
ransomware
supply-chain compromise
compromised third-party AI services
For generative AI, the problem may also include inadvertent disclosure of confidential information through prompts or external processing systems.
Cybersecurity is therefore not separate from patient safety.
If a security incident changes an AI output, prevents the system from operating, or exposes protected health information, the consequence is clinical as well as technological.
WHO explicitly includes safety and cybersecurity among the major risks associated with AI technologies in healthcare.
9. Lifecycle Drift and Model Degradation
Can a medical AI model become unsafe even if nobody changes the software?
Yes.
This is one of the most important concepts in clinical AI governance.
The healthcare environment changes continuously.
Examples include:
new scanners
new imaging protocols
changes in patient demographics
changes in disease prevalence
new treatment standards
changes in referral patterns
new clinical guidelines
changes in laboratory assays
seasonal variation
changes in hospital workflow
These changes can create:
Data drift
The input distribution changes.
Concept drift
The relationship between input variables and the target outcome changes.
Performance drift
Clinical performance declines over time.
A model that performed well during validation may therefore become less reliable after deployment.
This is why "FDA approved" or "validated" should not be interpreted as equivalent to "permanently safe."
Safety is longitudinal.
10. Model Updates Are Safety-Critical Events
Why can updating an AI model introduce a new risk?
Traditional software updates are often treated as maintenance.
For AI systems, a model update can change clinical behavior.
Changing:
training data
model architecture
preprocessing
threshold
calibration
segmentation
post-processing
can alter the system's output.
The U.S. Food and Drug Administration (FDA), Health Canada, and the UK's MHRA have therefore developed principles for managing planned modifications to machine-learning-enabled medical devices.
The FDA's 2025 final guidance on Predetermined Change Control Plans (PCCPs) specifically addresses planned modifications to AI-enabled device software functions and the methods used to develop, validate, implement, and assess their impact.
The underlying principle is straightforward:
AI change management is part of patient safety.
A hospital should therefore know:
which model version is running
when it changed
why it changed
what data were used
how it was validated
what performance changed
whether rollback is possible
who authorized the change
11. Governance and Accountability Risks
Who is responsible when medical AI is wrong?
This question becomes increasingly important as AI moves closer to clinical decision-making.
Potential stakeholders include:
AI developer
medical-device manufacturer
hospital
health system
radiologist
physician
clinical informatics team
AI governance committee
cloud provider
data provider
software integrator
A clinically deployed AI system therefore needs explicit accountability.
WHO's framework emphasizes responsibility and accountability alongside transparency, autonomy, safety, and equity.
A practical governance framework should define:
Who can deploy the model?
Who can modify it?
Who monitors performance?
Who investigates incidents?
Who can suspend the system?
Who informs clinicians about known limitations?
Who determines whether a model remains clinically appropriate?
Without these answers, an organization may have technology without governance.
12. Generative AI and Foundation-Model Risks
Are generative AI systems different from conventional medical AI?
Yes.
Generative AI introduces additional failure modes.
These include:
hallucination
fabricated references
incorrect clinical reasoning
prompt sensitivity
context-window limitations
ambiguous instructions
inconsistent responses
excessive confidence
inappropriate summarization
omission of critical information
privacy leakage
prompt injection
misleading natural-language explanations
The danger is amplified because generative AI can produce an answer that sounds medically sophisticated even when it is wrong.
WHO has specifically warned that large language models can generate authoritative-looking but incorrect health information and has called for rigorous evaluation, expert supervision, transparency, and evidence of benefit before widespread routine use.
This means generative AI safety requires more than measuring text quality.
Clinical evaluation must ask:
Did the output improve the decision?
Did it introduce a new error?
Did it alter clinician behavior?
Could the clinician detect the error?
What happens when the model is uncertain?
13. A Unified Classification of Medical AI Safety Risks
The following framework integrates technical, clinical, human, and organizational dimensions.
| Risk Class | Primary Failure | Example | Potential Harm | Key Control |
|---|---|---|---|---|
| R1 Data | Poor input data | Biased training cohort | Unequal performance | Dataset governance |
| R2 Model | Algorithmic failure | Poor calibration | Incorrect prediction | Robust validation |
| R3 Clinical | Diagnostic error | False-negative CT finding | Delayed diagnosis | Clinical verification |
| R4 Human | Automation bias | Blind acceptance of AI | Missed diagnosis | Human factors design |
| R5 Workflow | Integration failure | AI alert not delivered | Delayed action | Workflow validation |
| R6 Equity | Subgroup disparity | Lower sensitivity in subgroup | Health inequity | Fairness testing |
| R7 Security | Cyberattack/privacy breach | Data or model compromise | Patient harm/privacy loss | Cybersecurity controls |
| R8 Lifecycle | Drift | Scanner/protocol change | Performance degradation | Continuous monitoring |
| R9 Governance | Accountability failure | No model owner | Unmanaged incidents | AI governance |
| R10 Generative AI | Hallucination | Fabricated clinical statement | Incorrect care | Grounding + verification |
This classification is intentionally broader than an algorithm taxonomy.
The object of safety management is not merely the neural network.
It is the clinical AI ecosystem.
14. How Should Hospitals Assess Medical AI Safety?
A hospital implementing AI should evaluate the system across its entire lifecycle.
A practical architecture is:
This resembles the continuous risk-management philosophy of the NIST AI RMF, which organizes activities around Govern, Map, Measure, and Manage.
Figure 2. Continuous clinical AI safety lifecycle.
15. What Should Be Measured After Deployment?
Predeployment validation is not enough.
A clinical AI monitoring program should consider:
Technical metrics
sensitivity
specificity
positive predictive value
negative predictive value
calibration
AUROC
precision-recall performance
Operational metrics
processing latency
uptime
failed examinations
alert delivery
integration errors
Clinical metrics
time to diagnosis
time to treatment
diagnostic accuracy
unnecessary downstream testing
adverse events
Human factors
override rate
acceptance rate
alert fatigue
automation bias
clinician satisfaction
Equity metrics
subgroup performance
subgroup calibration
false-negative disparities
access disparities
Safety metrics
near misses
adverse events
model failures
unexpected outputs
incident frequency
The most important principle is that performance monitoring should be connected to clinical consequences, not merely dashboard statistics.
16. The Radiologist's Perspective: What Should Be Checked?
For medical imaging AI, the radiologist remains an essential safety layer.
Before relying on an AI output, consider:
1. Is the examination appropriate?
Was the correct modality and protocol used?
2. Is the image quality adequate?
Could motion, artifacts, contrast timing, or incomplete coverage affect AI performance?
3. Does the finding make anatomical sense?
Does the AI result correspond to the actual anatomy?
4. Is the finding clinically plausible?
Does it fit the clinical context?
5. Could the AI have learned a shortcut?
Is the output based on a genuine imaging feature or a confounder?
6. Is the model operating within its validated population?
Age, disease prevalence, scanner, protocol and institution all matter.
7. Does the AI disagree with the radiologist?
If so, disagreement should trigger reasoning—not automatic acceptance of either side.
8. What is the consequence of being wrong?
A low-risk incidental finding is different from a missed acute stroke, pulmonary embolism, intracranial hemorrhage, or tension pneumothorax.
The safest relationship is therefore not:
AI → physician
but:
AI → physician verification → clinical decision
Figure 3. Human-AI collaboration in medical imaging.
17. Multimodality Imaging and AI Safety
Medical imaging AI should also recognize that different modalities provide different information.
| Modality | Typical Strength | AI Safety Consideration |
|---|---|---|
| Radiography | Fast, inexpensive screening | Projection overlap and subtle findings |
| CT | High spatial resolution | Protocol/scanner variation |
| MRI | Excellent soft-tissue characterization | Sequence and acquisition variability |
| Ultrasound | Real-time dynamic imaging | Operator dependence |
| PET/SPECT | Functional/molecular information | Quantification and acquisition variation |
| Echocardiography | Cardiac structure/function | Operator and image-quality dependence |
A model validated on one acquisition environment should not automatically be assumed to perform equivalently across all environments.
This is particularly relevant in medical imaging because the image itself is influenced by acquisition hardware, reconstruction, protocol, patient characteristics, and post-processing.
18. Can AI Itself Help Manage AI Safety?
Yes—but with an important limitation.
AI can potentially assist with:
automated quality control
outlier detection
uncertainty estimation
data-drift detection
subgroup monitoring
model-performance monitoring
anomaly detection
audit-log analysis
automated image-quality assessment
continuous calibration monitoring
However, using AI to monitor AI does not eliminate the need for human governance.
It creates another layer that itself requires validation.
19. A Risk Matrix for Medical AI
A useful operational model combines probability, severity, detectability, and exposure.
| Risk Level | Probability | Severity | Clinical Approach |
|---|---|---|---|
| Low | Rare | Minor | Routine monitoring |
| Moderate | Occasional | Moderate | Targeted validation |
| High | Probable | Serious | Enhanced controls |
| Critical | Frequent or uncertain | Catastrophic | Restrict or suspend deployment |
NIST emphasizes that risk prioritization should reflect context and that systems presenting unacceptable negative risks may need development or deployment to stop until those risks can be sufficiently managed.
This is particularly important in healthcare.
A 1% error rate does not have a single meaning.
If the AI is used to classify harmless administrative documents, the consequence may be minor.
If the same error rate occurs in an acute stroke triage system, the clinical significance may be very different.
Risk is contextual, not purely numerical.
20. The Future of Medical AI Safety
The next generation of medical AI governance will likely move from static validation toward continuous assurance.
The important question will no longer be:
"Was the model validated?"
Instead:
"Is the model still safe in this clinical environment today?"
This requires integration of:
real-world performance monitoring
model drift detection
subgroup surveillance
cybersecurity
human factors
clinical incident reporting
version control
auditability
explainability where clinically useful
change-management procedures
regulatory oversight
The 2025 FUTURE-AI international consensus guideline provides a useful contemporary framework for trustworthy and deployable healthcare AI, reinforcing principles such as fairness, universality, traceability, usability, robustness and explainability across deployment.
The direction is clear.
Medical AI should be treated less like a static software package and more like a clinical technology that requires lifelong surveillance.
Conclusion
The safety of medical AI cannot be reduced to model accuracy.
A clinically useful AI system must be safe across the entire chain:
The most important safety risks include data bias, algorithmic error, diagnostic error, automation bias, workflow failure, inequity, cybersecurity threats, model drift, uncontrolled updates, governance gaps, and generative-AI hallucination.
For radiology and medical imaging, this becomes especially important because AI operates on complex data generated by different scanners, protocols, institutions, and patient populations.
The appropriate question is therefore not:
"How accurate is the AI?"
It is:
"Under what conditions is this AI reliable, for whom, for what clinical task, and how will we know when its safety changes?"
That question represents the transition from AI performance evaluation to clinical AI safety engineering.
For hospitals, radiologists, engineers, and AI developers, the goal should not be to eliminate every possible AI error. That is unrealistic.
The goal is to construct a clinical system in which:
errors are anticipated, detected, contained, investigated, and corrected before they cause patient harm.
That is the foundation of trustworthy medical AI.
Key Takeaways
Medical AI safety is a system-level property, not merely an algorithmic metric.
Dataset bias can become clinical harm when models are deployed in different populations.
External validation is essential because internal performance does not guarantee generalizability.
Automation bias can transform an AI error into a human-AI system failure.
Workflow integration must be validated alongside model performance.
Cybersecurity and privacy are components of patient safety.
Model drift can occur even when the underlying software has not changed.
AI model updates should be treated as safety-relevant events.
Generative AI introduces additional risks such as hallucination and misleading confidence.
Continuous post-deployment monitoring is essential for clinical AI.
AI governance must assign explicit responsibility for deployment, monitoring, incident response, and model changes.
The ultimate unit of safety is the patient-facing clinical system—not the AI model alone.
Medical Disclaimer
This article is intended for educational and professional discussion of medical artificial intelligence safety. It does not constitute medical advice, regulatory advice, or a substitute for institutional risk assessment, clinical judgment, applicable medical-device regulations, cybersecurity requirements, or professional guidelines.
Tables
Table 1. Classification of Medical AI Safety Risks
| Category | Core Risk | Typical Example | Primary Control |
|---|---|---|---|
| Data | Bias / poor labels | Nonrepresentative imaging dataset | Dataset governance |
| Model | Poor robustness | Distribution shift | External validation |
| Clinical | False prediction | Missed lesion | Human verification |
| Human | Automation bias | Blind acceptance | Human-factors design |
| Workflow | Integration failure | Delayed alert | End-to-end testing |
| Equity | Subgroup disparity | Lower sensitivity | Fairness assessment |
| Security | Cyberattack | Model/data compromise | Cybersecurity |
| Lifecycle | Drift | New scanner/protocol | Monitoring |
| Governance | Unclear responsibility | No model owner | AI governance |
| Generative AI | Hallucination | Incorrect clinical answer | Grounding and verification |
Table 2. Medical Imaging AI: Major Sources of Distribution Shift
| Source | Example | Potential Effect |
|---|---|---|
| Scanner | Different CT vendor | Performance change |
| Protocol | Different slice thickness | Detection degradation |
| Reconstruction | Different kernel | Feature alteration |
| Population | Different age distribution | Calibration change |
| Institution | Different referral pattern | Spectrum shift |
| Disease prevalence | Changing prevalence | Predictive-value change |
| Clinical practice | New guideline | Concept drift |
| Time | New technology | Temporal drift |
Table 3. AI Safety Monitoring Domains
| Domain | What Should Be Monitored? |
|---|---|
| Model | Accuracy, calibration, robustness |
| Clinical | Diagnostic and treatment consequences |
| Workflow | Latency, integration failures |
| Human | Overrides, acceptance, automation bias |
| Equity | Subgroup performance |
| Security | Access, attacks, anomalies |
| Lifecycle | Drift and model changes |
| Governance | Incidents, audits, accountability |
FAQ
What is the biggest safety risk in medical AI?
There is no single universal risk. Data bias, poor generalizability, diagnostic error, automation bias, workflow failures, cybersecurity vulnerabilities, and model drift can all produce harm. The most important risk depends on the clinical task, patient population, AI role, and consequences of an incorrect output.
Can a highly accurate medical AI still be unsafe?
Yes. High accuracy in a development or internal validation dataset does not guarantee safe real-world performance. A model may fail because of population differences, scanner changes, protocol differences, workflow problems, poor calibration, or inappropriate clinician reliance.
What is automation bias in healthcare AI?
Automation bias occurs when clinicians place excessive trust in an automated recommendation and reduce independent scrutiny. In medical imaging, for example, a radiologist may unconsciously accept an AI-generated negative finding without adequately reassessing the relevant anatomy.
Why is model drift important in clinical AI?
Healthcare environments change continuously. Patient populations, imaging equipment, acquisition protocols, disease prevalence, and clinical guidelines may change. These changes can alter model performance even if the deployed software itself has not been modified.
How can hospitals monitor medical AI safety?
Hospitals should monitor technical performance, clinical outcomes, workflow reliability, subgroup performance, cybersecurity events, human factors, model drift, and adverse incidents. Monitoring should be linked to defined thresholds for investigation, remediation, rollback, or suspension.
Is cybersecurity part of medical AI safety?
Yes. A compromised AI system can expose protected health information, disrupt clinical operations, manipulate data or affect model outputs. Cybersecurity should therefore be treated as an integral component of patient safety and AI governance.
Can generative AI be used safely in medicine?
Generative AI can potentially support healthcare, but it introduces additional risks including hallucination, misleading confidence, privacy concerns and prompt-dependent behavior. Clinical use requires appropriate validation, human oversight, monitoring and clearly defined use cases. WHO has specifically urged caution and rigorous evaluation of LLMs in healthcare.
Who is responsible when medical AI causes harm?
Responsibility depends on the technology, jurisdiction, intended use, contractual arrangements and clinical context. Regardless of legal allocation, healthcare organizations should establish explicit operational responsibility for AI deployment, monitoring, incident management, model changes and clinical oversight.
References
World Health Organization. Ethics and Governance of Artificial Intelligence for Health. Geneva: World Health Organization; 2021. ISBN: 978-92-4-002920-0.
National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. Gaithersburg, MD: NIST; 2023.
Lekadir K, Frangi AF, Porras AR, et al; FUTURE-AI Consortium. FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare. BMJ. 2025;388:e081554. doi:10.1136/bmj-2024-081554.
Vasey B, Nagendran M, Campbell B, et al; DECIDE-AI expert group. Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nat Med. 2022;28:924-933. doi:10.1038/s41591-022-01772-9.
Vasey B, Nagendran M, Campbell B, et al. Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. BMJ. 2022;377:e070904. doi:10.1136/bmj-2022-070904.
U.S. Food and Drug Administration; Health Canada; Medicines and Healthcare products Regulatory Agency. Predetermined Change Control Plans for Machine Learning-Enabled Medical Devices: Guiding Principles. 2023.
U.S. Food and Drug Administration. Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions. Final Guidance. August 2025.
Rafferty A, Rajan A. Limitations of public chest radiography datasets for artificial intelligence: label quality, domain shift, bias and evaluation challenges. Philos Trans A Math Phys Eng Sci. 2026;384(2324):20250129. doi:10.1098/rsta.2025.0129.
Tripathi S, et al. Understanding biases and disparities in radiology AI datasets: a review. J Am Coll Radiol. 2023. doi:10.1016/j.jacr.2023.06.015.
Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects. 2024.
Comments
Post a Comment