AI and Machine Learning in Healthcare: Pros, Cons and Safe Deployment

AI and machine learning in healthcare: real benefits, documented failures, and why validation, drift monitoring and human review decide safe deployment.

AI and Machine Learning in Healthcare: Pros, Cons and Safe Deployment
Written by TechnoLynx Published on 22 Nov 2023

AI and machine learning in healthcare show benefits in specific settings, such as image interpretation and skin cancer classification. They also fail in documented ways: models that look accurate at the hospitals that built them lose performance elsewhere, drift as patient populations change, and pull clinicians into following wrong suggestions. Whether a clinical model is safe to deploy is decided less by its headline accuracy than by the controls around it: external and recurrent local validation, drift monitoring, release gates, change control and genuine human review. This page covers those pros, cons and controls for clinical care; the AI use cases and disadvantages in pharma are covered on our broader page on artificial intelligence in healthcare.

AI and machine learning in healthcare: where the line sits

AI in healthcare is the application of artificial intelligence to medical and healthcare data, in disease diagnosis, treatment planning, patient monitoring, drug development and clinical decision support. AI is not one technology: it covers subfields such as machine learning and deep learning, used alone or in combination. AI/ML technologies use software algorithms that learn from real-world use and in some situations use that to improve the product’s performance.

That distinction matters in practice. A rule-based alert written by clinicians behaves the same next year as today, while a learned model’s behaviour depends on the data it was trained on and the data it now sees. The WHO puts it directly: AI systems depend not only on their code but on training data that comes from clinical settings and user interactions.

Concrete applications follow from that. Topol’s 2019 review in Nature Medicine described impact at three levels: rapid image interpretation for clinicians, workflow improvement and potential error reduction for health systems, and patients processing their own data. The FDA describes AI-enabled devices that analyse large, complex datasets to support detection, diagnosis and treatment, some of which may adapt over time as new data arrives.

The pros and cons of AI in healthcare, with their limits

The table below pairs each claimed benefit with what the evidence shows and where it stops.

Claimed benefit What the evidence shows Where it stops
Lower screening workload In the MASAI trial, AI risk scoring meant most mammograms were single-read by one radiologist instead of double-read Single centre, one mammography device, one AI system, moderately to highly experienced radiologists
Expert-level classification A 2024 meta-analysis found AI comparable to expert dermatologists in skin cancer classification The review called for real-world evaluation, and darker skin tones were underrepresented
Better human decisions Pathologists assisted by deep learning were more accurate than either humans or the AI alone on lymph node metastases Most pooled studies of AI-assisted clinicians were experimental settings
Imaging accuracy on par with clinicians Several deep learning models are roughly as accurate as professionals on imaging Few of those studies were externally validated
Using data clinicians cannot read manually Predictive modelling of EHR data reached 70–72% accuracy for individual treatment response in one Centerstone study One institute’s study, not a general accuracy figure

Small training datasets carry bias that the model inherits, which hurts generalisation and can discriminate against underrepresented groups. Some clinical prediction models have used datasets whose origin and reliability could not be verified. Barriers to using even validated tools include weak implementation infrastructure, poor workflow integration and no ongoing performance monitoring. ML4H 2024 roundtable participants noted that while some good models are never deployed, many deployed models are never evaluated.

In Beede et al.’s CHI 2020 study of a diabetic retinopathy system across eleven clinics in Thailand, the model’s data-quality thresholds clashed with the images an imperfect, resource-constrained clinic actually produced.

Why a model that validated well fails at the next hospital

Zech et al. (PLOS Medicine, 2018) trained pneumonia-detection CNNs on chest radiographs from three hospital systems. A model trained jointly on Mount Sinai and NIH data scored an internal AUC of 0.931 but only 0.815 on external Indiana data. The CNNs could identify which hospital system a radiograph came from for 99.95% of NIH and 99.98% of Mount Sinai images, and a trivial model ranking cases only by each hospital’s average pneumonia prevalence reached an AUC of 0.861 on the joint test set.

In other words, the model partly learned where an image came from, and where it came from predicted disease. The authors’ results suggest that CNNs could rely on subtle differences in acquisition protocol, image processing or compression and overlook pathology. Their warning is plain: performance estimated on data from the training hospitals may overstate real-world performance, and clinicians may then believe a system is more accurate than it is.

The Epic Sepsis Model shows the same generalisation problem in deployed software. Wong et al. published an external validation of this widely implemented proprietary model in JAMA Internal Medicine in 2021. According to the 2023 paper “All models are local”, the model was released as a one-size-fits-all solution reporting AUC of 0.73–0.83, later studies reported 0.60–0.64, and Epic subsequently announced it would localise the new version to each site’s patient mix.

External validation, then recurrent local validation

The ITU FG-AI4H recommends external analytical validation on an independent dataset, and treats randomised clinical trials as the gold standard for comparative clinical performance in the highest-risk tools. Regulators also name clinical validation metrics in the intended-use environment: Saudi Arabia’s SFDA, referencing IMDRF recommendations, cites positive predictive value and negative likelihood ratio as examples.

The authors of “All models are local” argue that external validation neither guarantees generalisability nor equates to clinical usefulness, because patient data changes across time, geography and facilities. Their proposal is site-specific validation run as a reliability test before every local deployment and repeated on a recurring basis. A 2026 Frontiers perspective makes the same point: external validation should be complemented by local evaluation under the intended data, workflow, staffing, prevalence and capacity conditions, with ongoing monitoring after deployment.

Dataset shift and calibration drift after go-live

Dataset shift is the gap between the data a model was trained on and the data it meets in clinical practice, and it can arrive suddenly or gradually. In imaging, a common cause is different scanner manufacturers or acquisition protocols, and patient populations changing over time, such as a new disease during a pandemic.

ML4H 2024 roundtable participants noted that when models move across populations, calibration (change in calibration slope) is often affected substantially more than discrimination (change in AUC). So participants judged ranking uses, such as allocating a ward’s three ventilators, more generalisable than numerical risk statements given to a patient. Populations also change over time, causing temporal shift, also known as calibration drift.

The Frontiers perspective cites a nationally deployed Veterans Health Administration model whose positive predictive values declined and false positives increased over time. It also warns that monitoring limited to discrimination metrics can miss changes in workload, delays, subgroup effects and version-associated risks.

Monitoring has practical constraints worth planning for:

  • Labels arrive late. Expected calibration error needs ground truth, so it is impractical for automated drift detection in clinical settings.
  • Label-free signals exist. Shift detection on softmax outputs with a Kolmogorov–Smirnov test can be added on top of an already trained classifier, and statistically significant variation in model inputs and outputs can indicate a likely performance decline.
  • Sample size sets sensitivity. More images in the target window improve sensitivity, which smaller institutions and rare-disease applications may not have.
  • Detection is not the end. After significant drift, expert labelling of the intervening data confirms whether performance fell, followed by retraining and revalidation.

Does a clinician still need to check what the AI says?

Yes: in one study, radiologists at every experience level followed incorrect AI suggestions. Dratsch et al. (Radiology, 2023) had 27 radiologists read 50 mammograms, 10 training cases with correct AI suggestions and then 40 test cases, with a purported AI system suggesting an incorrect BI-RADS category on 12 of the 40. When the suggestion was wrong, inexperienced readers’ correct ratings fell from 79.7% to 19.8%, and even very experienced readers fell from 82.3% to 45.5%. The authors conclude that radiologists at every experience level were prone to automation bias.

Automation bias is overreliance on a decision support system, where users stop engaging critically and follow its output, and it is documented in aviation, engineering and medicine. In mammography it produces both errors of commission (calling a normal image malignant) and errors of omission (missing a malignancy). Inexperienced readers were the most likely to follow an incorrect higher category, and the authors flag a deskilling risk if AI support is introduced early in training. Screening-specific detail sits in our article on AI in radiology.

ML4H roundtable participants named clinicians’ mistrust and misunderstanding of AI as significant barriers, with systems often perceived as black boxes. So human review has to work in both directions: catch wrong outputs and not discard right ones. Ethics work such as Maccaro (2026) describes a subtler pressure, digital paternalism, in which algorithmic output is treated as authoritative by default and the burden of proof shifts onto the clinician who disagrees.

The ten GMLP principles that Health Canada, the FDA and the MHRA identified in 2021 include a focus on the performance of the human-AI team. Their transparency principles note that a clear intended use tells users whether a device is meant to inform or replace a provider’s judgement. In MASAI, radiologists kept the final decision on recall. Dratsch et al. cite prior research on three mitigations: showing the system’s confidence levels, explaining its reasoning, and making users feel accountable for their decisions.

Release gates and change control that regulators expect

The FDA regulates AI-enabled devices with a risk-based approach and describes considerations that span development, validation, deployment, monitoring, maintenance and modification. The FDA also reviews modifications that could significantly affect safety or effectiveness, and a marketing submission may include a Predetermined Change Control Plan. The FDA, Health Canada and the MHRA have published PCCP guidance to set procedures for validating and monitoring planned post-deployment modifications.

The WHO’s 2023 regulatory considerations cover the same ground from another angle: documenting the entire product lifecycle, addressing intended use, continuous learning, human interventions and cybersecurity in risk management, externally validating data, and rigorous pre-release evaluation so systems do not amplify bias. In Europe, the EU AI Act requires technical documentation for high-risk systems and a quality management system that includes examination, test and validation procedures before, during and after development. Even so, health care AI was described as “severely under-regulated worldwide” as of 2025, so the controls a hospital or vendor actually runs matter more than the framework names.

Checklist before a clinical model goes live

  1. The intended use states whether the output informs or replaces clinical judgement.
  2. Targets, data provenance, missingness and subgroup representation are checked before deployment and after any material change.
  3. Analytical validation on an independent external dataset has been carried out.
  4. Site-specific validation is run before local go-live and scheduled to repeat.
  5. Known biases, failure modes and underrepresented populations are disclosed to users.
  6. Handling of uncertain or out-of-range inputs is defined, including escalation or abstention.
  7. Verification, override and escalation responsibilities are explicit.
  8. Downstream capacity effects of the threshold (referrals, testing, workload) are modelled, with prespecified responses up to rollback.
  9. Drift checks have set intervals and sample sizes, and confirmed drift triggers revalidation.
  10. Model, data or software version changes trigger proportionate revalidation.

The Frontiers authors frame these as gates at each transition point rather than one pre-deployment accuracy threshold. We agree with that framing: a single accuracy number at sign-off is the weakest gate in the chain.

How generative and multimodal models change the picture

Singhal et al.’s Med-PaLM paper (Nature, 2023) found clinicians judged only 61.9% of Flan-PaLM long-form answers aligned with scientific consensus, against 92.6% for Med-PaLM and 92.9% for clinicians. Yet 18.7% of Med-PaLM answers contained inappropriate or incorrect content, compared with 1.4% for clinicians. The authors still judged the model inferior to clinicians and called for more evaluation on safety, equity and bias.

The same paper says harm depends on use: using an LLM to diagnose or treat carries much greater potential harm than using it for information about a disease or medication, and deployment needs guardrails against over-reliance. Deployment data shows why. The Frontiers perspective cites a study of 16 Kenyan primary-care clinics where reviewers found actively harmful LLM recommendations in 115 of 1,469 encounters (7.8%), 67 of which reached final documentation. A follow-up cluster-randomised trial of 9,691 patients found no significant difference in 14-day treatment failure, which is exactly why monitoring has to separate outputs, their uptake and patient outcomes.

Multimodal models add their own engineering problems: misaligned data streams, varying quality across modalities and modalities missing in real-world settings. ML4H participants also noted that general foundation models may struggle to beat simpler specialised models on specific clinical prediction tasks. Generative AI may cut administrative burden, but inaccurate information can harm patients.

Reliability engineering around a clinical model

Healthcare AI safety cannot be inferred from model accuracy alone. TechnoLynx treats production AI reliability as a discipline distinct from model accuracy: task-specific evaluation harnesses, regression suites, drift detection, release-readiness reviews and validation packages for medical-imaging and perception workloads, the artefacts that catch silent regressions before patients or clinicians do. Teams building or deploying models under these constraints can see AI engineering for life sciences and healthcare.

FAQ

Will AI replace doctors in 10 years?

The evidence reviewed here does not point that way. Med-PaLM, one of the stronger medical language models evaluated, remained inferior to clinicians on human evaluation, and in the MASAI screening trial radiologists kept the final recall decision.

Which AI system is best for medical diagnosis?

There is no setting-free answer, because a fixed model’s performance changes across time, geography and facilities. The useful question is which system has been validated for your intended use and your population, and is monitored after it goes live.

Back See Blogs
arrow icon