The Privacy Prognosis: AI CHATBOTS IN HEALTHCARE
Prepared by:
Tammi Farshid, Laura Freeland, and Upendra Kamble Tul Consulting, Inc.
Prepared for: ENC5237
III. GENAI 101: The Powers Behind the Bots
● What Is GenAI and What Does It Do?
● How Does GenAI Work?
● AI in Healthcare
● How AI-Based Chatbots Use Data
IV. Patient Privacy: Trust & Safety in the AI Era
● Navigating Regulations and Ethical Standards
● Trust Begins with Privacy
● The Consequences of a Privacy Breach V The Human Impact of AI: Consent, Data, and Bias
● Informed Consent
● How AI Complicates Informed Consent
● Who Owns Patient Data?
● Algorithmic Bias
● Cases of Algorithmic Bias
VI. Data Security: The Technical Side of AI Chatbot Privacy……………………..………12
● Data Security ● Data Anonymization VII. Bridging the Gap: Regulatory Protections
● The Challenges of AI and HIPAA Compliance VIII. Taking Control: Strategies for Patients Using AI Chatbots
● Ask Questions About Data Storage
● Understand the Chatbot's Scope
● Identify Personally Identifiable Information (PII)
● Carefully Read Consent Forms
● Opt-Out Options
● Patient Rights
● What to look for in Privacy Resouces
● Stay Informed
IX. Works Cited……………………………………………………………………..……………..25 X. Appendix A: Infographics
● AI Diaries: 5 Ways Chatbots are Getting Personal
● 3 Privacy Risks to Watch
● 4 Flaws of Vulnerable AI
● Privacy Rx: Don’t Chat ‘til You Ask That
● You Have the Right to Remain Private: Patient Rights
1
I. A
This report investigates the privacy implications of AI-driven chatbots in healthcare and offers strategies for patients to protect their personal health information (PHI). As healthcare systems increasingly adopt AI technologies, including chatbots for jobs like patient triage, symptom tracking, and mental health support, concerns about data security and privacy continue to grow.
This report discusses the types of chatbots most susceptible to security breaches, the privacy risks posed by generative artificial intelligence (GenAI), and the regulatory protections available to patients. It provides an overview of the technical side of PHI privacy and practical steps patients can take and questions they can ask to navigate these technologies responsibly. By empowering patients to ask questions and encouraging them to understand their rights, this report aims to bridge the gap between technological advancement and patient privacy, fostering trust in AI-based healthcare chatbots1
II. Introduction
The rapid integration of artificial intelligence (AI) in healthcare has introduced a new era of patient interaction through chatbots designed to assist with medical inquiries, mental health support, and administrative tasks. AI chatbots leverage powerful GenAI models to respond accurately to patient questions, provide medication reminders, and even facilitate symptom assessments. While these tools offer convenience and efficiency, they also raise significant concerns about data privacy. Given that healthcare chatbots handle sensitive PHI, ensuring that this information is securely managed is essential for maintaining patient trust and safety.
This report addresses the privacy risks associated with AI-powered healthcare chatbots, focusing on how patient data is collected, stored, and potentially shared. Recognizing these concerns, regulations such as HIPAA in the U.S. and GDPR in the EU set critical standards for data protection; however, the unique capabilities and complexities of AI bring new challenges that existing frameworks may not address.
As healthcare continues to embrace AI, patients must take an active role in advocating for their data security This report outlines steps patients can take to protect their data when using AI-based healthcare chatbots, including questioning data storage practices, carefully reviewing consent forms, exploring opt-out options, and consulting reliable sources for privacy guidance.
By being p rmed and secure he
III. GENAI 101: THE POWERS BEHIND THE BOTS
What is GenAI and What Does It Do?
GenAI is a type of artificial intelligence designed to generate new content based on patterns learned from existing data This technology is widely used for tasks like helping chatbots understand and respond to language, creating images, and producing content for businesses everything from internal reports to marketing materials and even academic papers
How Does GenAI work?
GenAI generates content by analyzing and mimicking patterns within large datasets. The data usually comes from a combination of publicly available information, licensed content, and data collected from user interactions Data is then processed and analyzed to identify language structures, visual patterns, and predictive behaviors (Predictive behaviors refers to GenAI's ability to anticipate what kind of content a user might expect or need based on patterns in the data with which it has been trained )
GenAI models, like GPT (the technology behind ChatGPT) and DALL-E, are known as large language models (LLMs) They use machine learning (ML), a method where computers learn from large sets of data to recognize patterns and make predictions This allows them to create text, images, and other content that feels like it was made by a person
Natural Language Processing (NLP) is a field of artificial intelligence focused on enabling computers to understand, interpret, and respond to human language in a way that feels natural This involves teaching computers to process written or spoken language so they can perform tasks like translating text, answering questions, summarizing information, or holding conversations2
NLP allows chatbots to interpret and respond to human language, resulting in conversational interactions with users. Machine learning enhances these systems by enabling chatbots to learn from data and continually refine their responses. 2
Deep Lea o work with large asharat and Shahid)
AI in Healthcare
As GenAI technology becomes more advanced, it is finding applications in various fields, including healthcare.
Beyond supporting businesses with content creation and customer interaction, AI is transforming healthcare by assisting with diagnostics, streamlining administrative tasks, personalizing treatment plans, and even predicting patient outcomes. AI is particularly impactful in areas like medical imaging, predictive analytics, and robotic-assisted surgery, where its ability to analyze large volumes of data can lead to faster and more accurate results (Topol 44-56)
While patients may never come into direct contact with most of the AI emerging in the healthcare industry right now, they have or will likely use an AI-based chatbot in a healthcare context
In healthcare, AI’s effectiveness often relies on access to extensive patient data, which enables accurate insights, but presents privacy risks if sensitive information is not adequately safeguarded.
Potential breaches or misuse of personal health records can threaten patient confidentiality, underscoring the necessity for rigorous privacy protections. Proactively addressing these risks is crucial to ensuring that the benefits of AI in healthcare are realized without compromising patient trust and privacy.
How AI-Based Chatbots use Data
Healthcar ve support. Figure 1 shows five examples of how different AI-based chatbots use data. 3
As AI amasses patient data to improve healthcare, important questions arise:
● How is patient privacy defined and protected in this new age of technology?
● What specific privacy risks do AI chatbots pose to patients?
● How can patients stay informed and take steps to protect their personal health information?
IV. PATIENT PRIVACY: TRUST & SAFETY IN THE AI ERA
Patient privacy in healthcare is about more than just keeping information confidential It is about ensuring that each patient feels secure, respected, and in control of their health information. As healthcare increasingly relies on digital tools and AI technology, managing patient data with care is vital to maintaining trust and patient safety (Thompson).
This emphasis on privacy brings key areas into focus, such as data security (protecting information from unauthorized access) and informed consent, where patients are given clear choices about how their data is used.
Patients have the right to know how their personal information is collected, used, and shared by healthcare providers and researchers. This right includes secure storage of their information, explicit consent for data collection, and safeguards against unauthorized access or data breaches.
Navigating Regulations and Ethical Standards
Healthcare chatbots that handle sensitive patient information particularly those used for triage, virtual consultations, and mental health support are at the highest risk for security breaches
The Health Insurance Portability and Accountability Act of 1996 Privacy Rule, also known as HIPAA, provides privacy for patient data and security measures for the safeguarding of medical information This includes the regulation of how healthcare providers and their affiliates utilize and govern patient data (Hansen, et al, 2023)
3
The challe hat has been stripped of personal identifiers, a process known as "de-identification." This practice, sometimes referred to as "data repurposing," involves reusing patient information for purposes other than its original intent, like improving AI algorithms. This requirement creates the possibility of exposure of private patient information to unauthorized parties. For this reason, it has been suggested that special purpose consent should be given explicitly by the patient if there is any risk of PHI being exposed (Hansen, et al, 2023)
Regulations like HIPAA in the United States, GDPR in Europe, and the Australian Privacy Act of 1988 emphasize the importance of robust data protection measures, recognizing the sensitive nature of medical information and the risks that can arise from its misuse
Ethical standards further emphasize transparency, ensuring patients understand how their data will be used and giving them options, such as deleting conversation history or opting out of data collection for AI training.
Patient privacy hinges on the responsible, confidential, and respectful handling of health information, creating a foundation for ethical healthcare practices in an era of rapid technological advancement
Trust Begins with Privacy
The ethical management of sensitive patient data impacts how comfortable patients feel sharing personal health details, which are often crucial to their care. When healthcare providers and organizations handle data with strict confidentiality, it fosters an environment where patients can speak openly, leading to more accurate and effective treatments.
For every patient who is uncomfortable sharing their personal health information, there are many more who haven’t considered the security risks posed by the sheer volume of patient information being stored and analyzed by AI-powered Chatbots.
The Consequences of a Privacy Breach
Unauthorized access whether from external cyberattacks or internal mishandling can have severe repercussions, including identity theft and data manipulation Identity theft, in particular, can trigger a cascade of issues, like fraudulent insurance claims, unauthorized use of medical services, and financial fraud
Comprom
● Privacy breaches
● Identity theft
● Digital profiling
● Bias and discrimination
● Exclusion
● Social embarrassment
● Loss of control over who sees their data and what is done with it.
Regulatory measures like HIPAA4 in the U S , GDPR in Europe, and Australia’s Privacy Act underscore the need for strict data protection policies to prevent misuse of medical information As AI becomes more prevalent in healthcare, these privacy protections are increasingly critical to mitigate risks and support the responsible use of innovative technology (Healthcare IT Today 2024; Aunger).
There are a variety of ways AI is being used in healthcare, but for the purposes of this report, we will focus on AI-based chatbots that use and/or request personal health information.
4 Additional information about the role of HIPAA with regards to AI-based Chatbots is provided in section VII of this report, Bridging the Gap: Regulatory Protections
Healthcar for triage, virtual con , pp g y eaches
High Risk Healthcare Chatbots and Their Vulnerabilities . 5
A. Diagnostic and Symptom-Triage Chatbots
Chatbots that ask patients about symptoms and provide potential diagnoses collect PHI, which, if improperly secured, could be targeted for medical identity theft.
B. Virtual Mental Health Assistants
Chatbots supporting mental health are often used for personal and highly sensitive conversations These interactions might store data about mental health diagnoses, which could be particularly damaging to a patient’s professional and social life if exposed
C. Medication Management and Adherence Bots
These chatbots assist patients in managing prescriptions, logging doses, and reminding patients to take medications They collect data on medication schedules and specific treatments, which could be exploited if accessed by unauthorized parties
D. Insurance and Billing Assistance Chatbots
Chatbots that help patients with questions related to billing, claims, and payments handle financial and PHI, making them attractive targets for attackers seeking insurance or credit information.
E. Virtual Health Assistants (for scheduling and medical records access)
These chatbots directly access and update patient information and scheduling systems If they lack robust authentication and encryption, they could be vulnerable to interception or unauthorized access.
F. CO
Screening chatbots collect recent health information, travel history, and symptoms for public health purposes. Often deployed quickly in response to health crises, they may lack rigorous security protocols, making them particularly susceptible to data breaches.
Chatbots that both store and transmit sensitive health information are at the highest risk, as the medical conditions, symptoms, medication history, and even financial details they collect could be used by cybercriminals for identity theft, medical fraud, pharmaceutical scams, targeted advertising exploitation, and even employment discrimination and blackmail (U.S. Department of Health and Human Services).
V. THE HUMAN IMPACT OF AI: CONSENT, DATA, AND BIAS
AI is always changing and with advancements come uncharted territory Thus, AI-based Chatbots inevitably come with ethical challenges
Informed consent, data ownership, and fairness in AI-driven decisions are central concerns, as the "black box" nature of AI complicates patients' ability to understand its role in their care (Shah, Parth, et al.).
Because AI relies on identifying and replicating patterns in data, it can unintentionally incorporate societal biases present in that data. These biases, often reflective of stereotypes, may then be amplified by AI algorithms, disproportionately impacting marginalized groups.
This section examines the human impact of AI in healthcare, focusing on the need for transparency, equity, and patient-centered safeguards in its development and use
Informed Consent
Informed consent can be defined as "the process in which a healthcare professional educates a patient about the risks, benefits, and alternatives of a given procedure or intervention" (Shah et al.).
It is a cornerstone of ethical medical practice, ensuring patient autonomy through clear communication about medical decisions and their potential outcomes Central to this process is a patient’s ability to make decisions voluntarily, which requires both competence and access to information they can understand. For informed consent to be valid, patients must fully understand the implications of their choices.
How AI Complicates
AI introduces unique challenges to the informed consent process due to its inherent opacity. Known as the "black box" phenomenon, this refers to the difficulty or even impossibility of understanding how an AI system arrives at its decisions (Shah, Parth, et al.).
Natural Language Processing models, including large language models like Hippocratic AI , are 6 designed to provide patient-centered responses However, these systems operate within an evolving field of medical knowledge, which complicates their reliability and adaptability. Generative AI struggles to consistently align with the rapidly changing landscape of clinical knowledge, creating potential gaps in accuracy and applicability (Zhang and Boulos 9).
For patients, this opacity means they may not fully understand how their data is being used or how AI systems make decisions about their care Without this understanding, patients cannot give truly informed consent, as they are unable to weigh the risks and benefits associated with the use of AI in their treatment
Who Owns Patient Data?
The ownership of patient data is a complex issue, with legal perspectives varying by jurisdiction There are two prevailing models for data ownership:
A. Private Ownership
Individual control and autonomy over health data, asserting that individuals have the primary right to determine how their data is used, accessed, or shared. Under this framework, health data is treated as a personal asset, similar to property, and decisions regarding its use rest solely with the individual.
B. Public Ownership
Treats health data not as something owned and controlled by individuals or private entities, but as a shared resource that benefits society as a whole (Piasecki and Cheah).
Neither model adequately addresses the complexities of data governance, including justice, autonomy, risk mitigation, and stakeholder interests What’s needed is a nuanced approach centered on governance frameworks like Data Access Committees (DACs), which balance privacy and ethical considerations with the need for equitable and effective data use
DACs are access to sensitive h allenges, including regulatory differences, costs, technological and usability limitations, data-sharing demands, and a lack of expertise (Abouelmehdi et al ; Argaw et al ; Ibrahim et al )
Transparent communication about data usage policies particularly with AI systems is essential to empower patients to make informed decisions
Algorithmic Bias
Algorithmic bias happens when AI systems produce results that unfairly favor or disadvantage certain groups, leading to outcomes that do not accurately represent the population they are meant to serve. This bias often stems from problems in the data used to train the AI, such as incomplete or unbalanced datasets, or from incorrect assumptions made by the developers during the design process (Altamimi et al)
If bias is present, these systems can unintentionally reinforce existing inequalities in healthcare, such as providing lower-quality recommendations for underrepresented or marginalized groups
Cases of Algorithmic Bias
A study examining AI chatbot responses to patient vignettes revealed discrepancies in diagnoses based on socioeconomic status, gender, race, and ethnicity
While chatbots suggested different groups for coronary artery disease, they consistently excluded Hispanic men (Kim, et al). These biases can lead to inequitable healthcare outcomes, disproportionately affecting marginalized populations and undermining trust in AI systems.
For AI to be ethically integrated into healthcare, addressing algorithmic bias is critical This requires diversifying training data, scrutinizing assumptions during algorithm development, and actively involving underrepresented communities in the AI design process Only then can AI systems move closer to equitable and accurate decision-making.
VI. Data Security: The Technical Side of AI Chatbot Privacy
As healthcare organizations increasingly rely on AI-based chatbots to manage sensitive patient interactions, data security has become a cornerstone for maintaining patient confidentiality and trust Chatbots often handle large volumes of protected health information , making them attractive targets for cyberattacks and exposing critical vulnerabilities Addressing these risks requires robust protections against unauthorized access, breaches, and misuse
This secti from authentication systems and software vulnerabilities to the challenges of balancing data utility with privacy in AI-driven healthcare tools.
Data Security
Data security is the driving force behind the privacy the healthcare industry relies on to maintain patient confidentiality and trust when utilizing AI chatbots. AI systems often handle large amounts of sensitive medical data, including PHI. Top considerations for safeguarding data include:
A. Unauthorized access to patient information collected and stored by chatbots.
AI systems process large amounts of sensitive medical data, making them frequent targets for security attacks. Without adequate protections, a breach could lead to serious consequences. Unauthorized access can happen in several ways:
i Weak authentication Systems
Weak authentication systems act like weak locks, making it easier for unauthorized individuals to access sensitive information. This can allow malicious actors to pose as legitimate users and gain entry to secure systems.
ii. Insider threats
Insider threats occur when employees with access to chatbot data misuse
in ways it
iii. Vulnerabilities in the chatbot's software or infrastructure
Vulnerabilities in the chatbot's software or infrastructure are the weaknesses or flaws in the software code that make it easier for hackers to break in.
B. Understanding Technical Vulnerabilities .
Data breaches in AI healthcare systems often happen because of technical vulnerabilities, like poorly written code or software that hasn’t been updated In these cases, “compromising data” means that someone with bad intentions is looking for weak spots in the system so they can exploit them stealing, exposing, or misusing sensitive medical information. Several common issues contribute to these vulnerabilities:
i. Insufficient Encryption Standards
Many chatbots lack end-to-end encryption for both stored and transmitted data. End-to-end encryption ensures that only the sender and the intended recipient can access the data. Without it, sensitive information is vulnerable to interception during transmission or while stored on servers.
Without this safeguard, it would be like having a private call broadcasted over a loudspeaker, allowing anyone nearby to listen in. When encryption standards are insufficient, malicious actors can exploit these gaps to intercept or misuse confidential patient information.
ii. Misconfigured Systems
Misconfigured access controls, such as overly permissive settings or improper credential management, increase the risk of unauthorized access to patient data.
iii. Software Exploits
Outdated or unpatched chatbot software introduces exploitable entry points for cyberattacks, particularly in systems reliant on legacy infrastructure Legacy infrastructure refers to outdated or obsolete technology systems ian organization uses These systems often persist because they are critical to operations, but
n
Integrating new technologies like AI-based Chatbots with legacy software can create a plethora of issues. Legacy software often lacks modern security protocols, making it an easy target for attackers. If AI systems interact with or rely on data from a legacy system, those vulnerabilities can be exploited as entry points to compromise the entire ecosystem Additionally, Forcing developers to create custom workarounds to accommodate legacy software can introduce new vulnerabilities (Partida)
iv.
Third-Party Risks
Misconfigured access controls, such as overly permissive settings or improper credential management, increase the risk of unauthorized access to patient data
Overly permissive settings are when systems grant users more access than necessary for their jobs For example, allowing all employees to view sensitive patient records instead of restricting access to authorized personnel would be entirely too much permission
Weak credential management includes practices like using default usernames and passwords, reusing passwords across systems, or failing to enforce strong password requirements, which make it easier for attackers to gain unauthorized access
Data Anonymization
Data anonymization is a common strategy employed to protect personal information in datasets used for AI training It is the process of removing or masking identifiable information from datasets to protect individuals' privacy while maintaining data utility for analysis In healthcare, this usually involves techniques like encryption, aggregation, or data suppression to ensure sensitive personal health information cannot be traced back to specific individuals However, this approach has significant limitations in the context of healthcare chatbots:
A. Anonymized data often lacks the richness and detail needed for effective machine learning, reducing its utility in developing accurate AI models.
According to the article “Why Are Legacy Data Anonymization Techniques Failing?”, modern data analytics, machine learning, and AI applications depend on high-quality,
de ch detail, ma
AI applications, including machine learning and data analytics AI models rely on nuanced patterns and relationships within data to make accurate predictions, and overly suppressed or masked data limits their ability to do so. This creates a challenging trade-off for healthcare organizations: they must prioritize privacy protection while ensuring the datasets used to train AI chatbots retain enough utility to be effective Balancing these competing demands remains a critical hurdle in developing secure and functional AI tools for healthcare (Javaid)
B. Anonymization techniques may not adequately prevent re-identification of individuals, especially when combined with other datasets.
An area of vulnerability in healthcare data is anonymization It has been suggested that anonymization is almost impossible, due to the distinct nature of each individual's data footprint To re-identify previously anonymized data is facilitated by the amount of digital information currently available (Keen and Doe) This underscores the growing challenge of maintaining true anonymity in patient data used to train AI chatbots
For AI chatbots, relying on anonymized data can still lead to situations where sensitive personal health information is at risk of being inferred or disclosed Developing and applying effective anonymization techniques while ensuring sufficient data utility remains a critical challenge for healthcare organizations leveraging AI chatbots
VII. Bridging the Gap: Regulatory Protections
Regulatory frameworks play a crucial role in safeguarding patient privacy and data security for AI chatbots. The Health Insurance Portability and Accountability Act (HIPAA) in the United States establishes stringent requirements for handling personal health data, mandating that covered entities protect individual health information from unauthorized access and disclosure (Marks and Haupt).
However, compliance becomes complex when integrating AI technologies, as many existing regulations were not designed with these advancements in mind.
The Challenges
A. Lack of clarity on how regulations apply to new technologies like AI chatbots
The rapid growth of AI chatbots in healthcare has outpaced existing regulations, leaving uncertainty about how current rules apply to these technologies. As Marks and Haupt point out in “AI Chatbots, Health Privacy, and Challenges to HIPAA Compliance,” we are only just beginning to understand the privacy risks that chatbots pose. This lack of clarity makes it difficult for healthcare organizations and AI developers to ensure compliance They also argue that even if chatbots could meet HIPAA requirements, the regulations themselves are outdated and insufficient for addressing the privacy challenges AI presents This underscores the significant gap between current laws and the unique issues created by AI chatbots in healthcare
B. Difficulty in ensuring HIPAA compliance for chatbot interactions and data handling
HIPAA compliance poses a notable challenge for AI chatbots in healthcare In the article “Security Implications of AI Chatbots in Health Care,” Li et al explain that technologies like ChatGPT, when applied in healthcare settings, must adhere to HIPAA standards to ensure the protection, integrity, and accessibility of PHI throughout its lifecycle However, companies like OpenAI, which develop these technologies, operate as private
en
This disconnect creates a complicated scenario where AI chatbot developers may not be directly obligated to comply with HIPAA, even though their products are used in contexts where HIPAA regulations apply (Li et al.).
C. Potential gaps in addressing the nuances of AI and chatbot interactions in existing privacy regulations like GDPR and CCPA.
Existing privacy regulations may not fully address the unique challenges posed by AI chatbots in healthcare. Maya Goldman, in her Axios article "States Are Writing Their Own Rules for AI in Healthcare," notes that the absence of federal guidelines has led state governments to develop their own regulations, resulting in a fragmented and inconsistent approach. This decentralized framework risks leaving gaps in privacy protection.
Goldman also points out that many healthcare providers and pharmaceutical companies are employing AI technologies with minimal oversight, further highlighting regulatory shortcomings. Similarly, Li et al., emphasize the need for innovative legal and ethical frameworks to address the complex privacy concerns associated with AI and chatbots in healthcare. These observations underscore the urgency for comprehensive policies tailored to the unique risks posed by these technologies
For regulatory protections to be effective, healthcare organizations must ensure that AI chatbots are designed with compliance in mind from the outset.
Strategies to achieve compliance:
● Conducting regular security audits
● Establishing clear data governance policies
● Prioritizing the ethical use of AI technologies
● Implementing strict data security measures for the use and disclosure of PHI (Li et al )
VIII. Ta g g vacy
Strategies for Patients using AI Chatbots
This report has highlighted the critical role of data security and privacy in the context of AI-based healthcare chatbots Patients must navigate this complex digital landscape, where understanding privacy standards is essential to safeguarding their personal health information
Building on previous discussions of privacy risks and regulatory protections, this section equips patients with practical tools to evaluate both the AI chatbots they use and the resources they rely on for privacy information By focusing on features like data encryption, transparency, and compliance with established frameworks, patients can identify credible tools and sources that prioritize their privacy and security. This proactive approach empowers individuals to make informed decisions and advocate for stronger data protections in their healthcare interactions.
Ask Questions About Data Storage
When using AI chatbots, patients might assume their data is safe, but by asking specific questions about how your data will be stored and secured, we can
According to guidelines provided by institutions like Johns Hopkins Medicine, patients should be encouraged to ask:
● What specific data will be collected?
● Who will have access to my data?
● How long will my data be stored?
● Will my data be shared with third parties, and if so, for what purpose?
● How is my data protected?
● What security measures are in place to prevent unauthorized access?
Look for answers that include secure data encryption and limited third-party sharing, as these are hallmarks of strong data protection practices. Knowing the answers to these questions can give you confidence that your data is being handled with care (Lloyd).
Understand the Chatbot's Scope
If it is not obvious, inquire whether the chatbot in question is certified for healthcare applications, or is it a general AI system? This knowledge helps us understand whether the chatbot is
designed ots usually ad to mitigate privacy risks (Chen & Benfeldt)
Identify Personally Identifiable Information (PII)
If the chatbot collects identifiable information, like names, addresses, or specific health conditions, this can increase privacy risks. Asking if the chatbot anonymizes data or limits PII collection is crucial, as de-identified data reduces the risk of misuse (U.S. Department of Health and Human Services)
Carefully Read Consent Forms
Before using a healthcare chatbot, review consent forms, which outline how their data will be used. Look for details about data sharing, retention policies, and opt-out options. If the consent forms are unclear or even if they use language you are unfamiliar with–use some of the specific questions mentioned earlier in this section for clarification.
Carefully reading these forms and asking questions ensures patients make informed choices about their data (Johns Hopkins Medicine; Chen and Benfeldt.).
Opt-Out Options
Some healthcare chatbots offer opt-out options for specific types of data collection or sharing Patients should ask if they can opt out of non-essential data sharing, especially for marketing or research purposes Understanding these options helps patients retain control over their information (Simbo)
Patients often have rights to control their personal health information, even if a healthcare provider or chatbot does not explicitly offer an opt-out option
A. Right to Request Restrictions
Under HIPAA in the United States, patients can request restrictions on how their healthcare information is used or disclosed. While healthcare providers aren’t obligated to agree, they must consider the request seriously and may accommodate it if reasonable and possible. This can be particularly relevant if a chatbot is linked to a healthcare provider, as patients can request limits on how the data collected by the chatbot is shared or stored
B Right to Withdraw Consent
If a patient initially agreed to the collection and use of their data but later changes their mind, they can withdraw their consent. This might not erase past data collected, but it typically means that future data cannot be collected or used by the provider Patients can communicate this request directly to the chatbot provider or healthcare organization
C. Right to Access and Amend Records
Patients also have the right to access and correct their health records, which includes data collected by chatbots. If a chatbot system captures inaccurate information or data the patient is uncomfortable with, they may request corrections
D. General Right to Privacy
Some jurisdictions, like the EU under GDPR, offer broader rights, such as the "right to be forgotten," where patients can request their data be deleted entirely, regardless of opt-out policies. This right
ma ivacy law
E. Contacting a Data Privacy Officer
Many healthcare organizations are required to have a designated data privacy or HIPAA compliance officer. Patients can reach out to this officer to request specific privacy accommodations, even if not explicitly mentioned in the chatbot’s terms
Evaluating Privacy Features in AI-Based Healthcare Chatbots
The list below serves as a guide for evaluating healthcare chatbots, highlighting essential privacy features such as encryption, anonymization, consent transparency, user data rights, and compliance with recognized data protection standards
A Data Encryption Standards
Check that the chatbot provider uses strong encryption to secure data both in transit and at rest, a critical aspect of data protection
B. Data Anonymization and Minimization
Opt for chatbots that anonymize data where possible and only collect the minimum necessary information.
C Transparency and Consent Procedures
Look for chatbots that provide clear, understandable consent forms, detailing what data is collected, how it’s used, and with whom it’s shared
D. Right to Access and Delete Data
Patients should verify that they can access their data, request corrections, and have their data deleted if needed, particularly in compliance with HIPAA or GDPR
E Audit Trails and Compliance
Chatbots that maintain an audit trail and comply with recognized data protection frameworks (like the NIST Privacy Framework or GDPR) demonstrate a stronger commitment to privacy and security.
Stay Informed
The list below offers a guide to trusted organizations that provide up-to-date, accurate information on data protection practices, patient rights, and privacy concerns related to AI in healthcare By consulting these resources, patients can better understand how to evaluate healthcare chatbots, identify essential privacy features, and advocate for their data security
A. Institute for Healthcare Improvement (IHI)*
The IHI offers reports on patient safety and data privacy, particularly concerning AI tools in healthcare. Patients should review IHI materials to understand the latest patient-centered practices and emerging trends in healthcare technology These resources help patients recognize which chatbot privacy practices should be in place to safeguard their information
B. Electronic Frontier Foundation (EFF)*
EFF is a digital rights organization that advocates for privacy and data security in technology. Patients can use EFF’s resources to understand general AI risks, including those related to chatbots in healthcare, and learn how to advocate for their rights if privacy issues arise. EFF publishes guides and articles on protecting personal data and navigating privacy policies in this digital world
C. The Future of Privacy Forum (FPF)*
FPF focuses on responsible data use and frequently publishes research and guidelines on privacy practices in the digital health sector, including AI-driven tools. FPF’s resources highlight what to look for in a healthcare chatbot. Their guidelines can help patients ask informed questions about how their data will be used.
D. American Medical Association (AMA)*
The AMA provides educational materials and policy guidelines on ethical AI use in healthcare, including data protection and patient consent Patients can look to the AMA’s guidance on ethical principles for AI in healthcare, which emphasize transparency, privacy, and consent as crucial considerations for any AI-driven healthcare tool AMA's resources help patients identify chatbots that adhere to these principles and avoid tools with vague or insufficient privacy practices.
E. Pr
Privacy International is a global nonprofit that advocates for privacy as a fundamental right. It provides extensive information on data rights and offers resources on protecting privacy in the age of digital health. Patients can consult Privacy International to learn more about privacy rights globally and understand the legal protections available in their regions, which is especially helpful for patients in countries with strong privacy protections, like those under the GDPR in the EU.
F. HealthIT.gov (Office of the National Coordinator for Health Information Technology)*
HealthITgov is a U S -based resource that offers guidance on health information technology, including patient privacy and data security Patients can use HealthITgov to learn about their rights under HIPAA, understand the importance of secure data storage, and find tips on protecting their health information when using AI tools like chatbots.
G. National Institute of Standards and Technology (NIST)*
NIST provides a Privacy Framework that healthcare providers and tech developers use to create secure AI applications Patients can review NIST’s guidelines for general best practices on data privacy in healthcare Knowing these standards allows patients to ask specific questions about a chatbot’s adherence to secure protocols and industry standards for data protection.
IX. Wo
Altamimi, Ibrahim; Altamimi, Abdullah; Altamimi, Abdullah S.; Altamimi, Abdullaziz; Hani Temsah, Mohamad. “Artificial Intelligence (AI) Chatbots in Medicine: A Supplement, Not a Substitute”, Cureus, 25 June 2023, https://www.cureus.com/articles/163917-artificial-intelligence-ai-chatbots-in-medicine-a-suppl ement-not-a-substitute#!/ Accessed 20 Nov 2024.
Abouelmehdi, Karim, et al. "Big Healthcare Data: Preserving Security and Privacy." Journal of Big Data, vol. 5, no. 1, 2018, pp. 1-18. SpringerOpen, https://doi.org/10.1186/s40537-017-0110-7.
American Medical Association. "Advancing Health Care AI Through Ethics, Evidence, and Equity." AMA, American Medical Association, https://www.ama-assn.org/practice-management/digital/advancing-health-care-ai-through-eth ics-evidence-and-equity#:~:text=knowledge%20and%20expertise.-,Translating%20principles %20into%20practice,Promote%20health%20equity Accessed 19 Nov 2024.
Argaw, Salem T., et al. "Cybersecurity of Hospitals: Discussing the Challenges and Working Towards Mitigating the Risks." BMC Medical Informatics and Decision Making, vol. 20, no. 146, 2020, pp. 1-10. BioMed Central, https://doi.org/10.1186/s12911-020-01161-7
Aunger, Charles. "Understanding Security and Privacy Threats in Healthcare IT." Forbes, 1 June 2021, www.forbes.com/councils/forbestechcouncil/2021/06/01/understanding-security-and-privacy-t hreats-in-healthcare-it/ Accessed 10 Nov 2024.
Basharat, Perspective on Trust and Reliability.” Journal of Health Organization and Management, vol. ahead-of-print, no. ahead-of-print, 2023. Emerald Insight, https://doi.org/10.1108/jhom-10-2023-0302 Accessed 9 Nov 2024.
Chen, J., and O. Benfeldt. "Privacy and Security of AI-based Chatbots in Healthcare: Emerging Challenges." Journal of the American Medical Informatics Association, 2022. Accessed 11 Nov 2024.
Electronic Frontier Foundation. Electronic Frontier Foundation https://www.eff.org Accessed 19 Nov 2024.
Future of Privacy Forum. Future of Privacy Forum. https://fpf.org. Accessed 19 Nov. 2024.
Goldman, Maya “States are writing their own rules for AI in healthcare ” Axios, 12 August 2024, https://www.axios.com/2024/08/12/states-regulate-ai-health-care. Accessed 18 November 2024
Hansen, Jacob; Carlin, Kelly; Hart, Anna; Johnson, Jody; Wells, Scott; Kiel, Joan. “Updating HIPPA Security to Respond to Artificial Intelligence.” Journal of AHIMA, November 13, 2023, https://journal.ahima.org/page/updating-hipaa-security-to-respond-to-artificial-intelligence
#:~:text=Under%20AI%2C%20dat Accessed 17 Nov 2024
HealthIT.gov. Office of the National Coordinator for Health Information Technology. https://www.healthit.gov. Accessed 19 Nov. 2024.
Ibrahim, A lenges in Patient Privacy." BMC Nursing, vol. 23, no. 564, 2024, pp. 1-14. BioMed Central, https://doi org/10 1186/s12912-024-02231-1
Institute for Healthcare Improvement. Patient Safety and Artificial Intelligence: Opportunities and Challenges for Care Delivery. IHI, https://www.ihi.org/resources/publications/patient-safety-and-artificial-intelligence-opport unities-and-challenges-care. Accessed 20 Nov. 2024.
Javaid, Uzair. “Why Are Legacy Data Anonymization Techniques Failing?” Betterdata, https://wwwbetterdata ai/blogs/why-are-legacy-data-anonymization-techniques-failing
Accessed 18 November 2024.
Johns Hopkins Medicine. "Informed Consent." Institutional Review Board Guidelines, https://www.hopkinsmedicine.org/institutional-review-board/guidelines-policies/guidelines/info rmed-consent-ii Accessed 11 Nov 2024.
Keen, Harry, and John Doe “The ever-growing weaknesses of anonymised data ” Hazy, 8 April 2020, https://hazy.com/resources/2020/04/08/weaknesses-of-anonymised-data.
Accessed 18 November 2024.
Kim, Jiyeong; Cai, Zhuo Ran; Chen, Michael L.; Simard, Julia F.; Linos, Eleni. “Assessing Biases in Medical Decisions via Clinician and AI Chatbot Responses to Patient Vignettes”, JAMA Netw Open, 17 Oct. 2023, 10.1001/jamanetworkopen.2023.38050 Accessed 9 Nov. 2023.
Li, Jingqu
https://pmc.ncbi.nlm.nih.gov/articles/PMC10716748/. Accessed 18 November 2024
Lloyd, M. "7 Questions to Ask Your Healthcare Cloud Provider to Avoid Headaches." LinkedIn, https://www.linkedin.com/pulse/7-questions-ask-your-healthcare-cloud-provider-avoid-heada ches-lloyd. Accessed 11 Nov. 2024.
Marks, Mason MD; Haupt, Claudia E JSD “AI Chatbots, Health Privacy, and Challenges to HIPAA Compliance ” Petrie-Flom Center, https://petrieflom lawharvard edu/resources/article/ai-chatbots-health-privacy-and-challe nges-to-hipaa-compliance Accessed 18 November 2024
McKeon, Jill “Examining Health Data Privacy, HIPAA Compliance Risks of Ai Chatbots: TechTarget.” Healthtech Security, TechTarget, 13 July 2023, wwwtechtarget com/healthtechsecurity/news/366594256/Examining-Health-Data-Privac y-HIPAA-Compliance-Risks-of-AI-Chatbots
Nathella, Ganesh. "Data Privacy in Healthcare: Balancing Innovation with Patient Security." Healthcare IT Today, 17 July 2024, www.healthcareittoday.com/2024/07/17/data-privacy-in-healthcare-balancing-innovation-with -patient-security/ Accessed 10 Nov 2024.
National Institute of Standards and Technology National Institute of Standards and Technology https://www.nist.gov. Accessed 19 Nov. 2024.
OpenAI C
https://chat.openai.com/.
Partida, Devin. "Challenges and Solutions for Integrating AI in Legacy Security Systems." Brilliance Security Magazine, 17 Sept 2024, https://brilliancesecuritymagazine.com/guest-contributor/challenges-and-solutions-for-int egrating-ai-in-legacy-security-systems/. Accessed 18 Nov. 2024.
"Patient Data Collection." JotForm, https://www.jotform.com/blog/patient-data-collection/ Piasecki, Jan, and Phaik Yeong Cheah. “Ownership of Individual-Level Health Data, Data Sharing, and Data Governance.” BMC Medical Ethics, vol. 23, 2022, pp. 1-9.
Privacy International. Privacy International https://privacyinternational.org Accessed 19 Nov 2024.
Prokhorenko, Maria “Is ChatGPT HIPAA Compliant? Privacy Concerns With Conversational AI in Healthcare ” Botscrew, 25 June 2024, https://botscrewcom/blog/privacy-concerns-with-ai-in-healthcare/ Accessed 18 November 2024
"7 Most Frequently Asked Questions about Data Privacy in Healthcare." Cassie, TrustCassie, https://trustcassie.com/resources/blog/7-most-frequently-asked-questions-about-data-privacy -in-healthcare/ Accessed 11 Nov 2024.
Shah, Parth, et al. "Informed Consent." StatPearls, StatPearls Publishing, 15 Oct. 2024, www.ncbi.nlm.nih.gov/books/NBK430827/. Accessed 19 Nov. 2024.
Simbo. "Understanding Exchanges." Simbo.ai, https://simbo.ai/blog/understanding-patient-privacy-opt-out-options-in-health-information-exc hanges-and-their-implications-for-data-sharing-4152380/ Accessed 11 Nov 2024.
Topol, E. J. “High-Performance Medicine: The Convergence of Human and Artificial Intelligence.”
Nature
Thompson, Jamie. "Data Privacy in Healthcare: Balancing Innovation with Patient Security."
Healthcare IT Today, 17 July 2024, www.healthcareittoday.com/2024/07/17/data-privacy-in-healthcare-balancing-innovation-with -patient-security/ Accessed 9 Nov 2024.
U.S. Department of Health and Human Services. HIPAA for Professionals. Office for Civil Rights, 2023, www.hhs.gov/hipaa/for-professionals/index.html.Accessed 11 Nov 2024.
Zhang, P; Boulos, Kamel MN “Generative AI in Medicine and Healthcare: Promises, Opportunities and Challenges”, Future Internet, 24 Aug 2023, wwwmdpi com/1999-5903/15/9/286 Accessed 7 Nov 2024