Matt Hollingsworth is the CEO and Co-founder of Carta Healthcare, which aims to improve patient care by utilizing the value of medical data.

Companies are scrambling to incorporate generative AI and large language models—such as ChatGPT—into their current and future products. Although generative AI has seemingly limitless applications across a field of industries for searches, information gathering and content draft creation, it’s not conventionally appropriate for healthcare.

Generally, ChatGPT can provide convincing query responses and content drafts, but its algorithm is only based on generating words, phrases and sentences that have a high probability of occurring together (co-occurrence). That doesn’t include the logic to review its results and check its accuracy. As a result, it could string together inaccurate information or information that doesn’t logically belong together, which could be detrimental to a healthcare application.

Generative AI is not sufficiently accurate for healthcare applications.

Large language models are being evaluated and suggested for a number of medical applications such as those listed in this article and in Table 1 of this article, but the current consensus is that it’s not yet ready.

In healthcare, we strive for 100% accuracy, and inaccurate recommendations could jeopardize the quality of patient care and, thus, human lives, as decisions based on misinformation can be fatal.

The best approach to using AI in healthcare is to have a human in the loop.

The best way to implement and move forward with AI in healthcare is to have a human in the loop to first train the AI system and then check its results. The outcome and benefits of this approach, called dialog-based AI, are an AI system that “thinks” more like a human and presents areas of its output that often require fact-checking by a human.

Generative AI systems send their output directly to the recipient with no prompts for fact- or error-checking. For this reason, in healthcare, ChatGPT and other generative AI output should not be sent directly to patients; an expert should first review its output for accuracy as done in dialog-based AI systems.

An example of dialog-based human-in-the-loop AI is data entry. Initially, the AI system may not be able to perform a data-entry role similar to a human’s role; like a new employee, it will need to observe and be trained by a human. Over time, the AI system will learn to navigate and populate screens and access data to a point where it approximates a human employee’s work. It then could be used for initial data entry, which would be reviewed, edited and approved by a medical professional.

AI is great at parsing massive data sets, doing mathematics and calculating statistical probabilities for various scenarios. When analyzing a health data set, AI can help reveal patient-health insights and potential problems, including a patient’s previous medications, current and past medical conditions, comorbidities and any prior procedures.

This application of AI can assist providers in making better-informed decisions about patient care and even save lives by realizing the value of their own data.

Data collection and change management will be the biggest inhibitors to AI adoption in healthcare.

Implementing large language models like ChatGPT in healthcare is somewhat “putting the cart before the horse.” There are many AI applications that, to date, are untapped and have the potential to transform the industry. If all AI R&D were stopped today, it would still likely take decades to get these technologies integrated into health systems. For now, the focus of AI in healthcare should be solving the issues causing these limitations in utilization.

AI adoption and usage in healthcare requires data collection, accurate and complete data sets, a cumbersome change management component and a human in the loop to review, edit and approve its results and recommendations.

Data collection involves selecting the data sets that AI will reference and integrating the data sets and real-time updates with the AI system. Specialists in data science and system integration are required for those tasks to ensure the selected data sets are accurate and complete, and a specialist from the AI vendor should ensure that the AI system is accurately reading and processing the data.

Change management requires a core team of experts to train new users and explain the AI system to users and the broader community that will benefit from it. Some best practices for system adoption and change management are documented in this article.

A human who is an expert in the tasks assigned to the AI system—ideally the human who formerly solo performed the tasks that AI now aids, and who benefits from its efficiency and time savings—should be the one to decide whether the AI system is producing accurate results, and whether to take action based on the results.

Implementing technological changes takes time and effort, including AI adoption in healthcare. It should proceed gradually to allow for governance, performance review and continuous improvement.