Learning to Automate Follow-up Question Generation using Process Knowledge for Depression Triage on Reddit Posts

Author(s)

Shrey Gupta, IIIT Hyderabad, IndiaFollow
Anmol Agarwal, IIIT Hyderabad, IndiaFollow
Manas Gaur, University of South Carolina - ColumbiaFollow
Kaushik Roy, University of South Carolina - ColumbiaFollow
Vignesh Narayanan, University of South Carolina - ColumbiaFollow
Ponnurangam Kumaraguru, IIIT Hyderabad, IndiaFollow
Amit Sheth, University of South Carolina - ColumbiaFollow

ORCID iD

Gaur: 0000-0002-5411-2230

Sheth: 0000-0002-0021-5293

Document Type

Article

Abstract

Conversational Agents (CAs) powered with deep language models (DLMs) have shown tremendous promise in the domain of mental health. Prominently, the CAs have been used to provide informational or therapeutic services (e.g., cognitive behavioral therapy) to patients. However, the utility of CAs to assist in mental health triaging has not been explored in the existing work as it requires a controlled generation of follow-up questions (FQs), which are often initiated and guided by the mental health professionals (MHPs) in clinical settings. In the context of `depression', our experiments show that DLMs coupled with process knowledge in a mental health questionnaire generate 12.54% and 9.37% better FQs based on similarity and longest common subsequence matches to questions in the PHQ-9 dataset respectively, when compared with DLMs without process knowledge support. Despite coupling with process knowledge, we find that DLMs are still prone to hallucination, i.e., generating redundant, irrelevant, and unsafe FQs. We demonstrate the challenge of using existing datasets to train a DLM for generating FQs that adhere to clinical process knowledge. To address this limitation, we prepared an extended PHQ-9 based dataset, PRIMATE, in collaboration with MHPs. PRIMATE contains annotations regarding whether a particular question in the PHQ-9 dataset has already been answered in the user's initial description of the mental health condition. We used PRIMATE to train a DLM in a supervised setting to identify which of the PHQ-9 questions can be answered directly from the user's post and which ones would require more information from the user. Using performance analysis based on MCC scores, we show that PRIMATE is appropriate for identifying questions in PHQ-9 that could guide generative DLMs towards controlled FQ generation (with minimal hallucination) suitable for aiding triaging. The dataset created as a part of this research can be obtained from: https://github.com/primate-mh/Primate2022.

Publication Info

Preprint version CLPsych @ NAACL 2022, Fall 2022.

© The Authors, 2022

APA Citation

Gupta, S., Agarwal, A., Gaur, M., Roy, K., Narayanam, V., Kumaraguru, P., Sheth, A. (2022). Learning to Automate Follow-up Question Generation using Process Knowledge for Depression Triage on Reddit Posts. CLPsych @ NAACL 2022 [Pre-print]