Evaluating the Impact of Large Language Model AI on Acute Pancreatitis Management: A Chatgpt-Based Investigation

Abstract

Background: Evidence-based management of acute pancreatitis (AP) is important for patient outcomes. The present study evaluated suggestions by artificial intelligence (AI) chatbot system, ChatGPT, for the management of acute pancreatitis, its alignment with clinical guidelines, and assistance in clinical decision-making.

Methods: Six questions on pancreatitis management were curated by experienced RACS-qualified general surgeons and were put forth to ChatGPT. The chatbot was also asked to provide five high-level evidence references to support each of its responses. Each response was analyzed for its accuracy and comprehensiveness with respect to current internationally recognized guidelines and by two Board-Certified General Surgeons for acute pancreatitis management, as well as for its spelling, grammar, and reference quality. A five-point Likert Scale was utilized to analyze ChatGPT's responses, with scores ranging from 1 (strongly disagree) to 5 (strongly agree). Ten questions were designed to assess accuracy, consistency, informativeness, reliability, and coherence. These were independently rated by three junior doctors and two General Surgeons, with any scoring discrepancies resolved through consensus.

Results: ChatGPT successfully adhered to clinical guidelines when generating recommendations for the management of acute pancreatitis. The depth of information remained general and non-specific but was presented in an academic manner with appropriate grammar, spelling and sentence structure. ChatGPT missed pertinent references, with some being totally fabricated or erroneous.

Conclusion: ChatGPT holds promise for delivering prompt and accessible medical information to non-experts, which may benefit in situations where medical professionals and resources may be scarce or patients are reluctant to seek such services. The inclusion of aberrant or fabricated references is a challenge for researchers and clinicians and breaches academic integrity. Ethically, it is imperative for researchers to exercise prudence when utilizing ChatGPT for research purposes.