The co-evolution of computational processing power and neural network models has made revolutionary developments in generative artificial intelligence (GenAI) possible. One type of GenAI, large language models (LLMs), are disrupting a wide range of industries, including healthcare. LLMs are trained on large corpora of natural human language to predict and generate text (in chatbot form) that persuasively conveys contextual and semantic understandings. They are also being used to discover patterns in other types of data, such as genomic data, and enable the integration of multiple data types in ways that far surpass our previous capabilities. 

At the time of writing, the US Food and Drug Administration (FDA) has not approved any devices using GenAI or powered by LLMs. However, many GenAI applications are in development that may soon offer significant benefits for healthcare, including clinical decision support, enhanced patient communication and engagement, streamlined clinical documentation, reduced administrative workloads for healthcare professionals, assisted candidate gene prioritization and selection, medical image analysis, and accelerated drug discovery. While these emerging tools may improve care, their ethical, legal, and social implications (ELSI) remain unclear. 

Regulatory frameworks remain unequipped to address two novel features of GenAI: 1) its ability to adapt and improve performance over time or in response to changing conditions and 2) its capacity for continuous learning through unsupervised, “autodidactic” self-teaching. These features make GenAI a “moving target” that potentially requires new regulatory approaches.

The use of GenAI in the healthcare setting raises numerous ethical concerns. Notably, LLMs are known to “hallucinate,” or generate outputs not grounded in fact, logic, or their training data. However, enhanced reasoning or “chain-of-thought” models, along with advancements like retrieval augmented generation (RAG), which allows LLMs to retrieve information from search engines, databases, and other information repositories and use it to rectify their responses, are rapidly reducing hallucinations. Accuracy is especially critical, as results from GenAI chatbots are conveyed with a tone of confidence that may easily be mistaken by humans as fact.

Integrating unverified outputs into healthcare scenarios could negatively impact patient safety as well as expose practitioners to liability. Further, the quality and reliability of responses vary in line with the quality of user’s prompts, an observation that initially spawned a new focus on “prompt engineering” and hybrid intelligence, which is a broad set of approaches that aim to optimize human-machine interactions in ways that capitalize on both human and AI capacities. However, the importance of prompt engineering for quality responses is quickly diminishing as AI interfaces grow more intuitive.

A defining feature of GenAI technology is its capacity to generate new content in the style of a select portion of the training data. While this allows for the ability to generate novel outputs that emulate established styles (e.g., a new sonnet in the style of Shakespeare) or to tailor outputs for specific users or use cases, it can also reveal sensitive or personal information contained in training data sets. This is because model parameters represent specific features of data sets. The greater the number of parameters, the more features are inextricably embedded in models, and the more closely outputs mirror the training data. This capacity has drawn criticism from artists and other content creators who call for greater consent, credit, and compensation for the inclusion of their copyrighted information in GenAI training data. In healthcare settings, the potential for unintended data security breaches raises significant concerns about patient privacy and ability to consent to this use of their information.

GenAI models are also difficult to audit, in part because they are often proprietary rather than open source. Companies are reluctant to disclose which data was used to train their models. This lack of transparency has raised concerns about bias, generalizability, and fair attribution related to model outputs. Studies have revealed that LLMs reproduce racial, ethnic, gender and other biases found in human language and can produce outputs that can perpetuate “race-based” medicine or other forms of discrimination in healthcare. Further, given that the internet data on which they are trained overrepresent younger users and those from Western, English-speaking countries, LLMs have been criticized for propagating dominant rather than diverse viewpoints. Potential injustices for marginalized groups are compounded by the fact that non-Western nations may be more likely to pay the price for climate changes caused by the exploding energy requirements (representing an over 80-fold surge in just six years) of LLMs and other GenAI systems.

Some researchers are asking, “how big is too big?” and calling for greater transparency and more careful curation of training data sets for GenAI. Meanwhile, models created by critical U.S. competitors like DeepSeek, a startup based in China, are incentivizing the creation of smaller, more efficient GenAI systems. 
Finally, developers are leveraging GenAI’s capacity for processing and generating human-like content to build “agentic” AI systems that autonomously plan, make decisions, and perform actions to achieve specific goals with minimal human intervention, adapt to their environment and learn from interactions. Potential applications in healthcare include administrative task automation, personalized closed-loop treatment plans (as discussed here), emergency clinical decision support, and automated resource optimization.

This collection offers an introduction to the ELSI of GenAI that have been identified so far and highlights the need for participatory and anticipatory pathways to GenAI policy and regulation. It will continue to be updated with new literature as it is published. You are invited to email your suggestions to [email protected].