NLP/LLM Session 12: Outpatient reception via collaboration between nurses and a large language mode

Introduction

Let's get started. Thanks to everyone for joining us today on such short notice. This session is part of our special NP interest group. Today, it's our great pleasure to have Dr. Aring from the Chinese Academy of Medical Science and Peking Union Medical College. Dr. Long and I have been colleagues at NIH, collaborating on various AI and genomics projects related to medicine.

Dr. Long completed his PhD at the University of Michigan and his postdoc training at NIH. He is a clinician trained in hematology, and he has won several awards, including the NIH Fellows Award for Research Excellence and the NCI Informatics Tool Challenge Award. Today, Dr. Long will discuss our recent work on outpatient reception and large language model (LLM) collaboration frameworks validated in clinical settings. Please join me in welcoming Dr. Long.

Introduction

Thank you for the introduction. I'm honored to present our collaborative research here. We have been working on an innovative framework involving AI, particularly large language models (LLMs), to enhance outpatient reception.

Background

The Chinese medical system presents several unique challenges. Most high-quality medical services are centralized in government-based hospitals in big cities like Beijing and Shanghai. These hospitals are often heavily populated, leading to long waiting times and overworked staff, including reception nurses who are typically underpaid, undertrained, and have high turnover rates.

Motivation

Our research aims to address the inefficiencies in outpatient reception, focusing on using LLMs to manage queries and streamline operations. Our real-world medical dialogue dataset includes nearly 40,000 minutes of conversation data from two medical centers, offering valuable insights to train our models.

Data Collection and Analysis

We initiated a real-world medical dialogue project, capturing conversations at 10 different reception sites. The project provided valuable statistics, such as the number of cases handled per hour and specific inefficiencies like repeated Q&A and negative emotional exchanges. These statistics highlight the complexities faced by reception nurses.

The Role of LLMs

Our goal is to use LLMs to address redundant and straightforward queries, allowing reception nurses to focus on more complex issues. We developed a framework with context-specific training, showing significant improvements in response accuracy, empathy, and efficiency compared to human nurses.

Evaluation

We evaluated our system using a six-dimensional matrix focusing on factuality, integrity, empathy, readability, and safety. The LLM-based responses showed better or comparable performance across these dimensions.

Randomized Controlled Trial (RCT)

To ensure objectivity, we conducted a randomized controlled trial (RCT). The trial involved two groups: one managed by LLMs with human oversight for flagged responses, and the other managed entirely by human nurses. The LLM group showed higher patient and nurse satisfaction and reduced repeated Q&A and negative emotional exchanges.

Conclusion

Our research confirms the potential of LLMs to handle outpatient reception more efficiently and empathetically. This collaboration framework not only improves patient experience but also reduces the burden on medical staff.

We are excited about the future applications of this technology, including its deployment in more complex medical scenarios.

Acknowledgments

This research was supported by numerous collaborators and organizations, including the Chinese Academy of Medical Science and Peking Union Medical College. Special thanks to the dedicated clinical staff and technical advisors who made this project possible.

Keywords

• Large language models
• Outpatient reception
• AI in healthcare
• Medical dialogue
• Healthcare efficiency
• Patient satisfaction

FAQ

What motivated this research?

The inefficiencies and high workload in Chinese outpatient reception prompted this investigation, focusing on using AI to assist reception nurses.

How was the data collected?

Conversations at 10 different reception sites were recorded, providing nearly 40,000 minutes of dialogue data for training and analysis.

What are the key benefits of the LLM framework?

The LLM framework significantly improves accuracy, empathy, and efficiency in handling patient queries compared to human nurses.

How was the effectiveness of the LLM framework evaluated?

The framework was evaluated through a six-dimensional matrix and a randomized controlled trial, showing higher patient and nurse satisfaction.

Are there any safety measures in place for the LLM responses?

Yes, an internal alert system flags unsafe or uncertain responses, which are then reviewed by human nurses.

What are the potential future applications of this technology?

Future applications include more complex medical scenarios and broader deployment across different medical centers for improved efficiency and patient care.