Document Type
Article
Subject Area(s)
ProKnow, AI, large language models, LLM, VMHA
Abstract
Current Virtual Mental Health Assistants (VMHAs) primarily offer counseling and suggestive care but do not assist with patient diagnosis due to their lack of training in safety-constrained and specialized clinical process knowledge, referred to as ProKnow. In this work, we define ProKnow as an ordered set of information aligned with evidence-based guidelines or categories of conceptual understanding used by domain experts. We also introduce a new dataset of diagnostic conversations guided by safety constraints and Pro- Know, known as ProKnow-data. We develop a method for natural language question generation (NLG) designed to interactively gather diagnostic information from patients, termed ProKnow-algo. Our findings highlight the limitations of state-of-the-art large-scale language models (LMs) when applied to this dataset. The ProKnow-algo method models process knowledge by explicitly incorporating safety, knowledge capture, and explainability. When used with ProKnow-algo, LMs generated 89% safer questions in the context of depression and anxiety. In contrast, without ProKnowalgo, the generated questions failed to adhere to the clinical process knowledge outlined in ProKnow-data. Furthermore, questions generated using ProKnow-algo exhibited a 96% reduction in average squared rank error. The explainability of the generated questions was assessed by measuring their similarity to concepts within depression and anxiety knowledge bases. Overall, regardless of the type of LM used, ProKnow-algo achieved an average improvement of 82% over simple pre-trained LMs in terms of safety, explainability, and process-guided question generation. We qualitatively and quantitatively evaluate the efficacy of ProKnow-algo by introducing three new evaluation metrics for safety, explainability, and adherence to process knowledge.
Publication Info
Preprint version 2024.
APA Citation
Roy, K., Gaur, M., Soltani, M., Rawte, V., Kalyan, A., & Sheth, A. (2024). ProKnow: process knowledge for safety constrained and explainable question generation for mental health diagnostic assistance in the age of large language models. [Preprint]
Rights
© 2024, The Authors