Purpose of this webinar
Session Objectives
Introduce AI & transformer architecture – what is it?
Highlight patient & user safety considerations
Review information governance & legal rulings
Explain regulation & explore future directions
Purpose of this webinar
Transformers: Analogies & Concepts
For everyone: analogies
• Roundtable conversation – each word listens to the others and weighs their importance.
• Postal sorting – words are like letters sorted into bins by topic or role.
• Spotlight – attention shines on important words and dims the rest.
For the curious: conceptual overview
• Sentences are split into tokens and turned into numerical vectors.
• Each token creates queries, keys and values that can be compared with others.
• Attention scores measure how strongly tokens relate to one another.
• Weighted sums capture context to produce context‑aware outputs.
The patient is stable today
Purpose of this webinar
Self‑Attention: For the Curious
How self‑attention works (summary):
• Words are turned into numbers (vectors).
• The model compares words by creating small helper vectors (queries, keys & values).
• It scores how much each word should attend to others.
• Context is built by mixing information from the important words.
Limitations:
• LLMs predict words without understanding meaning.
• Susceptible to hallucinations and fabricating facts.
• Biases reflect training data and may propagate inequities.
• Not optimised for complex reasoning in health and social care.
Purpose of this webinar
Embedding Space: High‑Dimensional Intuition
How embeddings work
• Words become points in a very high‑dimensional space (hundreds of dimensions).
• Similar meanings cluster together: eg words that have similr meanings like ‘calm’ and ‘stable’ occupy nearby regions.
• The model learns patterns by exploring relationships between these clusters.
• These can be thought of as ‘Cognitive Spaces.”. The space in which the AI ‘thinks’.
• These spaces exist as mathematical objects. They are not actual spaces in a computer.
• Our brains probably work in a similar way, as these spaces have been designed to model the way that brain cells work.
This illustration is conceptual; real spaces have far more dimensions.
Purpose of this webinar
AI Workflow: From Prompt to Response
Prompt
“The patient is stable today”
Tokenise & embed Words become numbers
Attention & context
Model weighs words to build meaning
Generate output Produces a coherent response
A simple sentence like 'The patient is stable today' enters the model. It is tokenised and embedded, attention scores highlight the important words, and then the model generates an informed response.
Purpose of this webinar
Example Prompt & AI Response
User prompt
The patient is stable today
The model pays extra attention to 'stable' and 'today' to tailor its response.
Model output
Monitoring will continue; no changes in treatment are required right now.
Purpose of this webinar
Patient & User Safety: Real‑World Impact
Generative models are not optimised for complex clinical reasoning.
• They may hallucinate facts and alter context.
• Outputs can embed data biases or leave out critical information.
• Clinician oversight is essential: AI should support—not replace— human judgement.
• In Iris we have an AI policy.
• AI must not be used to
• Write care plans
• Write daily entries
• Do any task where there is any patient identifiable information.
• Do any task at all unless it is done via an Iris approved AI account or on premises AI (Co‑Pilot).
1 in 10 patients are harmed while receiving care
3+ million deaths occur annually due to unsafe care
50% of harm is preventable through safer systems
1 in 30 patients experience medication‑related harm (>25% severe)
Purpose of this webinar
Information Governance & Ethics
Key governance principles:
• Don’t upload patient‑identifiable data into public AI services.
• Consult your information governance (IG) lead and Caldicott guardian before deploying AI tools.
• Comply with UK data protection laws (UK GDPR) and NHS IG frameworks.
• Use AI – where allowed by policy to support decisions (eg research‑ but CHECK THE REFERENCES); staff retain final responsibility.
• Verify outputs, document all AI use.
Purpose of this webinar
& the Legal Profession
• President Dame Victoria Sharp (June 2025) warned that generative AI can’t conduct reliable legal research:
• “artificial intelligence is a tool that carries with it risks as well as opportunities. Its use must take place therefore with an appropriate degree of oversight, and within a regulatory framework that ensures compliance with well-established professional and ethical standards if public confidence in the administration of justice is to be maintained”
• Two cases (Ayinde & Al‑Haroun) involved fake citations; the court imposed wasted‑cost orders and referred lawyers to regulators.
• Lawyers using AI without verification risk contempt proceedings and criminal sanctions.
• Israel’s Supreme Court (Feb 2025) rejected a petition that cited nonexistent cases and held that lawyers must verify AI‑generated content.
• South African courts reprimanded lawyers for filing AI‑hallucinated citations and warned of sanctions in future cases.
Lawyers face sanctions for citing fake cases with AI Reuters (Jul 2025)
Israeli Supreme Court rejects AI‑generated citations Library of Congress (Feb 2025)
Purpose of this webinar
Legal Cases in Health & Social Care
Google, AI firm must face lawsuit over teen suicide
Reuters (2024)
Lawsuits claim insurers’ AI cuts off rehab care
Reuters/STAT (2023–24)
Case / Issue
Wrongful death (Character.AI)
UnitedHealth & Humana lawsuits
Jurisdiction Summary
Healthcare giant Cigna sued for algorithmic denials CorpWatch (2023)
Texas AG settles with AI vendor over false claims
PMC (2022)
Cigna algorithm claim denials
Texas AG settlement
Florida, USA Mother alleges that a chatbot encouraged her son’s suicide; judge allowed the case to proceed.
Minnesota & Kentucky, USA
Eastern District of California, USA
Class actions allege insurers used algorithms to prematurely cut off rehab care in Medicare Advantage plans, forcing patients to pay or go home.
Class action alleges insurer used PxDx algorithm to deny claims without physician review, breaching contracts and violating state law.
Texas, USA AI vendor settled allegations that it misrepresented accuracy of its products; healthcare entities urged to train staff and monitor AI.
Purpose of this webinar
Medical Device Regulation & UK Developments
2021
Medicines & Medical Devices Act establishes UK framework.
Jun 2024
Joint guiding principles for AI/ML transparency & change control plans (MHRA/FDA/Health Canada)
.
Nov 2024
Consultation on new medical device regulations; software classification moves to risk‑based classes.
Jun 2025
Post‑market surveillance regulations take effect; real‑world data and faster incident reporting required.
Jul 2025 Government proposes indefinite recognition of CE‑marked devices and international reliance routes.
Purpose of this webinar
Future Directions & CRITiCS Framework
CRITiCS framework for mental health AI (Hider, Wright and Needle, 2025)
CRiTICS = Clinical Reach into the Cognitive Space
Mental health care involves heterotopic and weak signals in unstructured data (eg single serious threat of violence 10 years ago, missed in formal reports)
Heterotopic = the same word to describe different things in the world or things in the world that can’t be seen. Eg ‘voice’, ‘thought’.
Generative AI could assist with formulation and care planning but must be supervised and assessed. Clinicians and patients must be involved in developing it in mental health. This is more important than in other areas of healthcare where illnesses can be measured and observed directly.
Humans must have ‘reach’ into what the AI is doing sufficient to maintain safety. Both humans and AIs are ‘agents’. AIs as agents are sometimes better than humans (eg reading a lot in a short space of time). They may make care safer.
Agent – thing that, takes in and acts upon information and may act on the world. Something that does something with intention.
Mental health care is different from other areas of healthcare:
• Contested
• Unmeasurable
• Patients can disagree with clinicians and there is no measure to say who is right (eg why someone is depressed).
• Often so complex that standard algorithms do not apply.
Purpose of this webinar
Future Directions & CRITiCS Framework
“Agentive reach” levels:
Vectorised data
AI extracts structured signals
Clinical formulation
Human formulates case; AI assists
Treatment planning
AI suggests options; clinician decides
Therapeutic alliance
Predominantly human empathetic interaction
Outcomes
Evaluate results & feedback
Purpose of this webinar
Recommendations & Risk Mitigation
Understand the limits of your AI tool; never abdicate human judgement.
Negotiate terms with anyone marketing AI to clarify liability and ensure safety. DO NOT automatically believe what a vendor or website says about what its AI doesn’t do. Ask about the model it is using. Ask if it is fine tuned and how that was done. Ask if it has any regulatory approval.
Apply risk assessment frameworks and document AI use.
Stay informed and trained as the legal landscape evolves.
Purpose of this webinar
Conclusion & Discussion
Conclusion:
• AI holds great promise but remains limited for complex clinical reasoning.
• Patient and user safety require robust governance and human oversight.
• Legal rulings worldwide warn against uncritical reliance on generative AI.
• Mental healthcare needs a different approach to AI.
• Involving service users and clinicians is critical if the field is to develop safely.
• Regulation is evolving; it is critical to stay informed and adapt as the world adopts the technology.
• It is helpful to have a basic broad understanding of how AI works – this will help you to ask the right questions as use cases emerge.
Purpose of this webinar
Discussion
ChatGPT writes a care plan
Scenario: A social worker caring for a man with a severe learning disability and autism asks ChatGPT to “write a holistic care plan” and pastes in detailed notes about his daily routine, sensory triggers and medication. The AI produces a plausible plan recommending communication aids and behavioural strategies.
Answer/Discussion: Using a public LLM exposes highly sensitive personal data and creates a non‑compliant record (no consent or data‑processing agreement). ChatGPT can hallucinate or provide outdated interventions; it is not clinically validated and should never replace the care‑planning process led by qualified staff.
Untested triage chatbot
Scenario: A mental‑health service deploys a pilot chatbot to triage overnight calls to its crisis line. The vendor claims the model can prioritise calls based on text sentiment, but no clinical staff were involved in testing it. During a busy night, a caller expressing suicidal thoughts in slang is assigned a low‑priority category, and staff don’t call back until morning.
Answer/Discussion: Introducing an unregulated AI tool without governance exposes patients to serious harm. There should have been a Data Protection Impact Assessment, risk analysis and clinical oversight. Models often misinterpret colloquial language or cultural nuances, so human triage should remain central.
Purpose of this webinar
AI “translation” of therapy notes
Scenario: An occupational therapist wants to share session notes with a patient whose first language isn’t English. She feeds the notes into a free online translation bot. Weeks later, the patient finds adverts on social media referencing their anxiety disorder and specific triggers.
Answer/Discussion: Many free translation services retain and monetise uploaded text. Uploading identifiable health information without a compliant processing agreement breaches UK GDPR and Caldicott principles. Staff should use secure, approved translation services or bilingual colleagues, not consumer chatbots.
Facial‑recognition risk tool
Scenario: A learning‑disability unit installs an AI camera system marketed as predicting aggression based on micro‑expressions. Staff begin restraining certain residents when the system flags a “high risk” alert. Later reviews show the model was trained on a non‑disabled prison population and misclassified tics and self‑stimulatory behaviours.
Answer/Discussion: Deploying a predictive device that is not a certified medical device violates medical‑device regulations. The algorithm’s biases lead to discriminatory practices, and there was no human verification. New AI products must undergo regulatory approval and ethical review before use.
Purpose of this webinar
AI scheduling ignores social context
Scenario: A service manager introduces an AI scheduling tool to allocate support‑worker hours across a caseload. The model optimises for cost and shifts, so it reduces visits to a woman with profound learning disabilities because she lives with her mother. The family is overwhelmed, but staff assume the schedule is correct.
Answer/Discussion: AI models can embed value judgements (cost over need) and may miss safeguarding risks. Without transparency, staff may defer to the tool (automation bias). Governance policies should require explainability, clinician oversight and the ability to contest and override AI‑driven decisions.