Shadow AI is becoming a growing issue for hospitals and health systems

Lloyd Price
1 hour ago
16 min read

The Invisible Infrastructure: A Comprehensive Analysis of Shadow AI in Modern Healthcare Systems

Executive Summary

The global healthcare sector stands at a critical juncture, navigating a technological inflection point that is as transformative as the digitisation of health records, yet far more perilous due to its clandestine nature. We are witnessing the rapid, unregulated, and often invisible integration of artificial intelligence (AI) into the core workflows of medicine—a phenomenon collectively termed "Shadow AI." Unlike the Shadow IT of the past, which largely concerned the unauthorised use of software for logistical efficiency, Shadow AI involves the deployment of probabilistic, generative agents capable of synthesising medical advice, interpreting complex clinical histories, and drafting patient communications without institutional oversight or validation.

This report provides an exhaustive examination of the Shadow AI landscape within hospitals and health systems. Drawing upon data from late 2024 through early 2026, the analysis reveals a pervasive infrastructure of unauthorized intelligence that has permeated every stratum of the healthcare hierarchy, from the administrative back office to the surgical suite. Recent widespread analysis indicates that Shadow AI has infiltrated hospitals to a degree that far exceeds initial executive estimates. Surveys conducted in late 2025 reveal that approximately 40% of healthcare professionals are aware of colleagues using unauthorised AI tools, with nearly 20% admitting to personal usage, figures that likely underrepresent the true extent of adoption due to the stigma of non-compliance.

The primary driver of this trend is a systemic crisis of clinician burnout and administrative overload. In an environment where enterprise-grade solutions are often viewed as cumbersome or antiquated, consumer-grade generative AI tools like ChatGPT offer an immediate, albeit risky, mechanism for relief. However, the risks associated with this invisible infrastructure are profound. They encompass direct patient safety threats arising from algorithmic hallucinations and bias, severe legal liabilities related to medical malpractice and the evolving standard of care, and catastrophic data privacy violations under frameworks such as HIPAA and GDPR.

The financial implications are equally severe; reports suggest that data breaches involving Shadow AI in healthcare cost an average of $670,000 more than standard breaches due to the complexity of data exfiltration vectors and the involvement of third-party model training environments. This report dissects the socio-technical drivers of adoption, the specific clinical use cases being surreptitiously automated, the complex regulatory environment across the United States and the United Kingdom, and the technical and governance frameworks required to mitigate these risks while harnessing the undeniable potential of AI.

Part I: The Emergence of the Invisible Infrastructure

1.1 From Shadow IT to Shadow AI: A Categorical Shift

To understand the gravity of the current situation, one must distinguish between the legacy concept of Shadow IT and the emergent threat of Shadow AI. Historically, Shadow IT referred to the unauthorised adoption of deterministic software, tools like Dropbox for file sharing or Trello for project management. While these posed security risks regarding data leakage, the software itself functioned predictably. It did not create new content, nor did it make decisions.

Shadow AI represents a fundamental ontological shift. It involves the use of non-deterministic systems—Large Language Models (LLMs) and machine learning algorithms. that are capable of generating novel outputs that may or may not be grounded in reality. When a physician uses an unapproved app to organize their schedule (Shadow IT), the risk is strictly confined to data confidentiality. However, when a physician uses an unapproved LLM to determine a drug interaction for a pregnant patient (Shadow AI), the risk extends beyond confidentiality to immediate physical harm, diagnostic error, and professional negligence.

The defining characteristic of Shadow AI is its "agency." These tools act as "Shadow Staff," performing cognitive labor that was previously the exclusive domain of trained human professionals. This introduces a dynamic risk profile: the tool itself can hallucinate medical facts, exhibit sociodemographic bias, or inadvertently exfiltrate sensitive Protected Health Information (PHI) into public training datasets, creating a permanent privacy breach that cannot be "undone".

1.2 The Architecture of Unauthorised Intelligence

The "invisible infrastructure" of Shadow AI is built upon the ubiquity of consumer technology. It bypasses the traditional perimeter of hospital IT security not through sophisticated hacking, but through the path of least resistance: the web browser and the smartphone.

The architecture is typically tripartite:

The Input Layer: A clinician dictates a patient note into a personal smartphone or copies text from the Electronic Health Record (EHR) on a desktop.
The Processing Layer: This data is pasted into a consumer-grade AI interface (e.g., ChatGPT, Claude, Gemini) hosted on public servers.
The Output Layer: The AI processes the data, often retaining it for model training—and returns a summarised note, a diagnosis, or an appeal letter, which is then pasted back into the secure hospital environment.

This workflow effectively "air-gaps" the security protocols of the health system. The data leaves the secure enclave, traverses the public internet, is processed by an unvetted third-party algorithm, and returns, leaving virtually no trace on the hospital's internal logs unless specific deep-packet inspection tools are in place.

Part II: The Epidemiology of Unauthorised Usage

The adoption of Shadow AI is not a fringe activity but a widespread behavioral shift across the healthcare workforce. The data indicates that the "containment" phase of AI adoption has failed; the technology is already deployed at scale, largely without governance.

2.1 Prevalence and Penetration Statistics

Data from comprehensive surveys conducted by Wolters Kluwer Health in late 2025 provides a stark quantification of this trend. In a survey of over 500 healthcare providers and administrators, the findings dismantle the assumption that AI usage is limited to tech-savvy early adopters:

Widespread Awareness: 40% of healthcare staff report encountering unauthorised AI tools in their workplace.
Active Participation: Between 17% and 20% of staff admit to using these tools personally. Given the "social desirability bias" inherent in self-reporting non-compliant behaviour, the actual figure is likely significantly higher.
Clinical Impact: Perhaps most alarmingly, 10% of respondents admit to using unauthorised tools specifically for direct patient care use cases, such as diagnosis or treatment planning.

Further corroborating this is data from OpenAI, which reveals that more than 40 million people globally turn to ChatGPT daily for health-related inquiries. Within the United States, OpenAI reports that healthcare-related prompts constitute a massive volume of traffic, with 1.6 million to 1.9 million messages per week specifically focused on health insurance tasks.

2.2 The Demographics of Disobedience

The profile of the Shadow AI user challenges conventional wisdom. It is not merely the "digital native" resident or medical student who is bypassing IT protocols.

Experienced Clinicians: Providers with more than five years of experience are frequently found to be heavy users. Their motivation is often born of pragmatism and exhaustion; they have suffered through years of "click-heavy" EHR interfaces and are desperate for the efficiency that AI promises. Wolters Kluwer data suggests that 45% of providers using unapproved tools do so simply to achieve a faster workflow.
Administrators vs. Providers: While administrators are more likely to be involved in policy development (30% vs 9%), they are actually less likely to be aware of the specific AI policies in place compared to providers (17% vs 29%). This suggests a disconnect where leadership sets policies they do not fully understand, while clinicians on the ground are acutely aware they are breaking rules but do so out of necessity.

2.3 The "Trust Paradox"

A critical psychological dimension of Shadow AI is the phenomenon of misplaced trust. A report by UpGuard uncovered a "Trust Paradox" wherein nearly one-quarter of workers consider their AI tools to be "their most trusted source of information".

Hierarchy of Trust: Remarkably, these workers ranked AI tools nearly on par with their managers and higher than their colleagues or traditional search engines.
Implications for Safety: In a high-stakes environment like healthcare, this over-reliance is dangerous. If a junior doctor trusts a chatbot's drug dosing recommendation more than a senior nurse's correction, the traditional human safeguards of medicine, the "Swiss Cheese" model of error prevention—are compromised. The "human in the loop" becomes a "human asleep at the wheel," accepting algorithmic output as truth without rigorous verification.

Part III: The Psychosocial Drivers of Adoption

To effectively mitigate Shadow AI, healthcare leaders must understand that it is a symptom of deeper structural failures within the modern healthcare environment. It is a rational response by highly trained professionals to an unsustainable work environment.

3.1 The Burnout Crisis and "Pajama Time"

The primary engine driving Shadow AI adoption is the epidemic of clinician burnout. The introduction of the EHR, while beneficial for data storage, has been catastrophic for clinician workflow. Studies consistently show that for every hour a physician spends with a patient, they spend two hours on Electronic Health Record documentation.

Cognitive Load: This administrative burden forces physicians to complete documentation after clinic hours, a phenomenon known as "pajama time."
The AI Lifeline: In this context, generative AI is not viewed as a "tech toy" but as a survival mechanism. A tool that can instantly summarize a complex chart or draft a compassionate patient letter in seconds rather than minutes is a lifeline. Wolters Kluwer data underscores this: the majority of users cite "speed" and "workflow efficiency" as their primary motivation.

3.2 The Enterprise Functionality Gap

There is a widening chasm between the consumer technology clinicians use in their personal lives and the enterprise technology provided by their employers.

The "iPhone vs. Mainframe" Experience: Clinicians carry smartphones with access to state-of-the-art LLMs (like GPT-4 or Claude 3.5) that are intuitive, conversational, and incredibly powerful. In contrast, they work on hospital computers running legacy EHR software with interfaces that often date back to the 1990s.
Inadequate Tools: 24% of providers explicitly state they use unapproved tools because they offer "better functionality" than the approved enterprise alternatives. When the hospital-provided spellchecker cannot recognise medical terminology but ChatGPT can write a fluent appeal letter, the choice for the clinician is obvious, if not compliant.

3.3 The Staffing Vacuum

The global healthcare workforce shortage, projected by the WHO to reach 10 million by 2030—extends beyond clinicians to administrative support staff.

The Missing Scribe: Many hospitals have cut back on medical scribes and administrative assistants to reduce costs.
Shadow Staffing: Shadow AI fills this vacuum. It acts as a "digital scribe," a "coding specialist," and a "secretary." In rural areas and "hospital deserts", where OpenAI reports extremely high usage volumes, such as in Wyoming and Oregon—these tools may be the only support system a solo practitioner has.

Part IV: Anatomy of Shadow Workflows

Shadow AI is being utilised across a spectrum of use cases, ranging from the mundane to the clinically critical. Understanding these specific workflows is essential for identifying risk.

4.1 The "Digital Scribe" Workflow

This is the most pervasive use case. Clinicians utilise ambient listening apps on personal devices or simply copy-paste notes to generate documentation.

Mechanism: A doctor records a patient encounter using a commercially available dictation app on their phone. They then copy the transcript into a generative AI tool to "summarise this into a SOAP note format."
Risk: This workflow involves recording a patient's voice (biometric data) and processing it on unvetted servers. If the AI tool retains data for training, that patient's confidential medical consultation becomes part of the model's latent space.

4.2 Clinical Decision Support (CDS) and the "Second Opinion"

More alarmingly, clinicians are using Shadow AI as an unauthorised Clinical Decision Support system.

Differential Diagnosis: Physicians input a list of symptoms, lab values, and patient history to generate a differential diagnosis.
Drug Interactions: Clinicians ask the AI to check for interactions between multiple medications.
Case Study of Failure: A cited example involves a clinician asking an AI for treatment options for a complicated urinary tract infection. The AI correctly suggested fluoroquinolones based on general medical knowledge. However, the AI failed to ask if the patient was pregnant—a crucial contraindication. Because the clinician did not explicitly prompt with the pregnancy status, and the AI (unlike a formal CDS) did not have access to the EHR to check, the advice was clinically accurate in isolation but dangerous in context.

4.3 Administrative Coding and Revenue Cycle

In the administrative back-office, Shadow AI is used to optimise revenue.

Upcoding Risk: Staff may paste clinical notes into an AI and ask for the "best billing codes." Generative AI, driven to satisfy the user, may suggest codes that justify higher reimbursement than is warranted by the documentation (upcoding).
Fraud Liability: If a hospital submits claims based on these hallucinations, they may be liable for billing fraud under the False Claims Act, even if the error was automated. The lack of an audit trail for why a code was chosen (other than "the AI said so") makes defense difficult.

4.4 Translation and Patient Communication

In the United Kingdom, NHS England has issued specific warnings regarding the use of unapproved AI translation apps.

The Scenario: Faced with a non-English speaking patient and a 2-hour wait for a human interpreter, a clinician uses a free AI translation app to explain a discharge medication plan.
The Consequence: These apps often lack the medical vocabulary to distinguish between "take once daily" and "take once daily PRN" (as needed). Misunderstandings in translation can lead to medication errors and readmissions.

Part V: The Multi-Dimensional Risk Landscape

The deployment of Shadow AI introduces a complex matrix of risks that transcends simple IT security.

5.1 Patient Safety: The Hallucination Problem

The most immediate and catastrophic risk is physical harm to patients.

Probabilistic vs. Deterministic: Generative AI models are probabilistic engines designed to predict the next plausible word, not to verify truth. This leads to "hallucinations"—confidently stated falsehoods. There have been documented instances of AI fabricating medical citations, inventing drug dosages, or misinterpreting lab reference ranges.
Bias Amplification: LLMs trained on the open internet ingest societal biases. If a clinician relies on Shadow AI for diagnosis, the model might exhibit racial or gender bias, such as under-diagnosing cardiac conditions in women or misinterpreting dermatological conditions on darker skin tones.

5.2 Data Security: The Model Inversion Threat

Data privacy concerns are paramount, particularly for large health systems where the volume of data amplifies the risk.

The "Black Hole" of Data: When PHI is entered into a public model, it enters a "black hole." Many consumer terms of service allow the vendor to use input data to train the model.
Model Inversion Attacks: Research demonstrates that it is possible to "attack" a model to force it to regurgitate its training data. Theoretically, a hacker could prompt a model trained on shadow healthcare data to "reveal the medical history of [Patient Name]," and if that patient's data was part of a previous user's unapproved upload, the model could leak it.
Supply Chain Compromise: Shadow AI often involves "wrapper" apps, sketchy third-party interfaces for major models. These apps account for 30% of AI security incidents, often containing malware or unauthorised data harvesting code.

5.3 Legal Liability: Malpractice and Defamation

The legal landscape for Shadow AI is fraught with peril.

Malpractice and Standard of Care: The legal standard of care requires physicians to act as a "reasonable" peer would. Currently, using an unvalidated, hallucination-prone tool likely breaches this standard. If a patient is harmed because a doctor followed AI advice, the doctor is fully liable. Conversely, as AI improves, a future dilemma may arise where ignoring a superior AI diagnosis constitutes negligence.
Defamation Risks: The case of Dr. Ed Hope serves as a chilling precedent. Google's AI Overview feature falsely generated a biography stating he had been suspended by the medical council for selling sick notes, a complete fabrication that amalgamated his YouTube channel name ("Sick Notes") with a scandal involving a different doctor. This illustrates that AI risks extend to the reputation of the providers themselves.

5.4 Financial Impact

The cost of ignoring Shadow AI is quantifiable.

Breach Costs: According to IBM’s Cost of a Data Breach Report, healthcare breaches are already the most expensive, averaging $9.8 million. However, breaches involving Shadow AI cost an additional $670,000 on average compared to standard breaches. This premium is due to the difficulty in detecting the breach, the complexity of tracing data through third-party models, and the longer "dwell time" before the breach is discovered.

Part VI: Global Regulatory Frameworks

Navigating Shadow AI requires compliance with a patchwork of international regulations, each attempting to catch up with the technology.

6.1 United States: HIPAA and Emerging State Laws

HIPAA Implications: The central mechanism of HIPAA compliance is the Business Associate Agreement (BAA). Any vendor that creates, receives, maintains, or transmits PHI on behalf of a covered entity is a Business Associate. Consumer AI tools (like the free version of ChatGPT) do not sign BAAs. Therefore, any use of these tools involving PHI is a per se violation of the HIPAA Privacy Rule, exposing the health system to massive fines from the Office for Civil Rights (OCR).
California AB 3030: Effective January 1, 2025, this law specifically targets the "invisibility" of Generative AI. It mandates that health facilities must disclose to patients if they are interacting with AI-generated content (e.g., chat, letters), unless that content has been reviewed by a human provider. Shadow AI makes compliance with this law impossible, as the institution cannot disclose what it does not know is happening.
NIST AI Risk Management Framework (AI RMF): While voluntary, the NIST AI RMF is becoming the industry standard for "reasonable security." It outlines four functions: Map, Measure, Manage, and Govern. A failure to detect Shadow AI represents a fundamental failure of the "Govern" function, potentially weakening a hospital's defence in negligence lawsuits.

6.2 United Kingdom: NHS and GDPR

NHS England Guidance: The NHS has taken a proactive stance, issuing guidance on "Ambient Voice Technologies" and generative AI. This guidance warns that tools must meet DCB0129 (Clinical Risk Management) standards. It explicitly states that the guidance is "not meant for individuals seeking to use tools outside the supervision of their setting," effectively outlawing Shadow AI in the NHS context.
The Caldicott Principles: Shadow AI directly challenges the UK's Caldicott Principles, specifically Principle 7 ("The duty to share information can be as important as the duty to protect patient confidentiality"). While information sharing is vital, it must be lawful. Sharing data with an unvetted US-based AI company likely violates UK GDPR data sovereignty and subject rights requirements.
Information Commissioner's Office (ICO): The ICO is heavily scrutinizing the "legitimate interest" basis for processing personal data in AI. The use of Shadow AI often involves scraping or processing data without a clear lawful basis, and without the ability to honour "Right to be Forgotten" requests if the data is ingrained in the model.

6.3 European Union: The AI Act

The EU AI Act classifies AI systems used for "medical components" (diagnosis, treatment) as High Risk. This classification triggers onerous requirements for data governance, human oversight, accuracy, and cybersecurity. Shadow AI tools, being general-purpose and uncertified for medical use (lacking CE marking), are illegal for these high-risk use cases. Hospitals allowing their use could face penalties of up to 7% of global turnover.

Part VII: Technical and Operational Remediation

Addressing Shadow AI requires a shift from "blocking" to "enabling," supported by robust technical controls.

7.1 The "AI Firewall" Architecture

Traditional firewalls are insufficient because Shadow AI traffic looks like standard encrypted web traffic (HTTPS). Healthcare organisations are increasingly deploying "AI Firewalls" or advanced Cloud Access Security Brokers (CASB).

CASB Configuration: Modern CASBs (e.g., Zscaler, Netskope) can inspect SSL/TLS traffic to identify the unique signatures of thousands of AI applications. They can differentiate between a "Sanctioned" instance (e.g., the hospital's Enterprise ChatGPT account) and an "Unsanctioned" instance (e.g., a personal Gmail account accessing ChatGPT).
Browser-Based Discovery: Network-edge detection often misses traffic from devices off the corporate network. Deploying browser extensions allows IT to detect when a user navigates to an AI site or installs a "wrapper" plugin. Tests show browser-based tools can identify over 600 independent AI instances that edge-based tools miss.
Prompt Filtering: Advanced AI firewalls can scan outgoing prompts for patterns of PHI (e.g., regex for MRNs, SSNs, or specific clinical terms) and block the specific request while leaving the rest of the session active. This allows for "safe" use of AI (e.g., drafting a generic policy) while blocking "unsafe" use (e.g., analysing a patient chart).

7.2 The "Walled Garden" Strategy

The consensus among experts is that blocking AI entirely is futile; it merely drives usage further underground (e.g., to personal cell phones on 5G). The only viable solution is to provide a sanctioned, secure alternative, a "Walled Garden.

Enterprise Procurement: Hospitals must accelerate the procurement of enterprise-grade AI licenses (e.g., Microsoft Copilot for Health, ChatGPT Enterprise). These versions come with BAAs and "zero data retention" policies, ensuring inputs are not used for model training.
Private Instances: For the highest security, health systems are hosting open-source models (like Llama 3 or Mistral) within their own private cloud infrastructure (e.g., AWS Bedrock, Azure OpenAI). This ensures data never leaves the hospital's controlled environment.

7.3 Governance: The UVM Health Case Study

The University of Vermont (UVM) Health System offers a roadmap for effective governance. Faced with widespread Shadow AI usage, they did not issue punitive bans.

Discovery: They used Zscaler logs to map usage, finding thousands of instances across all departments.
Engagement: They formed an AI Governance Council that included clinical leaders, not just IT security.
Enablement: They used the data to justify the purchase of approved tools. By offering a safe, sanctioned alternative, they effectively converted "Shadow" users into "Governed" users. This approach acknowledges that the demand for AI is legitimate and focuses on making it safe rather than making it go away.

Part VIII: Future Trajectories (2026 and Beyond)

The phenomenon of Shadow AI is likely a transitional phase. As the technology matures, the distinction between "Shadow" and "System" will evolve.

8.1 Integration and Cannibalisation

By late 2026, it is expected that major EHR vendors (Epic, Oracle Health, Meditech) will have fully integrated generative AI features into their core platforms. When the EHR itself can draft a discharge summary, suggest a diagnosis, and reply to patient messages, all within a BAA-covered, legally compliant framework the utility of "copy-pasting" into ChatGPT will vanish. Shadow AI will effectively be cannibalized by valid, integrated System AI.

8.2 The Threat of "Agentic" AI

However, a new threat is on the horizon: "Agentic AI." These are systems that don't just generate text but execute actions(e.g., "Schedule an MRI for this patient," "Order this lab panel"). Shadow Agentic AI poses exponentially higher risks. If a physician uses an unapproved agent to manage their inbox, that agent might inadvertently promise care, admit liability, or order incorrect tests without the physician's review. Governance frameworks must evolve rapidly to manage not just content generation but autonomous action.

Conclusion

Shadow AI in healthcare is a symptom of a system under immense pressure. It represents the collision of a burnt-out, under-supported workforce with a transformative technology that offers immediate, tangible relief. While the risks, ranging from patient injury to massive financial penalties, are unacceptable, the drivers are rational and understandable.

The "containment" strategy of the past, firewalls and zero-tolerance policies, has failed. The prevalence data confirms that the invisible infrastructure is already built. The only viable path forward is radical enablement. Healthcare organizations must bring AI out of the shadows by providing secure, superior enterprise-grade tools. They must govern these tools with a council that represents clinical reality, not just IT security compliance. And they must educate their workforce to treat AI not as a magic oracle, but as a powerful, fallible intern that requires constant, vigilant supervision. Until the "official" technology of healthcare matches the speed and utility of the "shadow" technology, the invisible infrastructure will remain a potent systemic risk.

Summary of Key Recommendations

Domain	Recommendation
Governance	Establish a cross-functional AI Council including clinicians, legal, and ethics. Move from "Blocking" to "Managed Adoption."
Technology	Deploy CASB with SSL inspection and "AI Firewall" capabilities to detect and filter PHI in prompts.
Procurement	Immediately procure enterprise licenses with BAAs (e.g., ChatGPT Enterprise, Azure OpenAI) to offer a safe alternative.
Risk	Update HIPAA Risk Assessments to include GenAI specific vectors (model inversion).
Education	Train staff on specific risks (e.g., "Why your prompt might train the model") rather than generic "Do not use" warnings.

Nelson Advisors > European MedTech and HealthTech Investment Banking

Nelson Advisors specialise in Mergers and Acquisitions, Partnerships and Investments for Digital Health, HealthTech, Health IT, Consumer HealthTech, Healthcare Cybersecurity, Healthcare AI companies. www.nelsonadvisors.co.uk

Nelson Advisors regularly publish Thought Leadership articles covering market insights, trends, analysis & predictions @ https://www.healthcare.digital

Nelson Advisors publish Europe’s leading HealthTech and MedTech M&A Newsletter every week, subscribe today! https://lnkd.in/e5hTp_xb

Nelson Advisors pride ourselves on our DNA as ‘Founders advising Founders.’ We partner with entrepreneurs, boards and investors to maximise shareholder value and investment returns. www.nelsonadvisors.co.uk

#NelsonAdvisors #HealthTech #DigitalHealth #HealthIT #Cybersecurity #HealthcareAI #ConsumerHealthTech #Mergers #Acquisitions #Partnerships #Growth #Strategy #NHS #UK #Europe #USA #VentureCapital #PrivateEquity #Founders #SeriesA #SeriesB #Founders #SellSide #TechAssets #Fundraising #BuildBuyPartner #GoToMarket #PharmaTech #BioTech #Genomics #MedTech