Psychol Trauma. 2025 Dec 1. doi: 10.1037/tra0002087. Online ahead of print.
OBJECTIVE: Based on the Diagnostic and Statistical Manual of Mental Disorders, fifth edition, posttraumatic stress disorder (PTSD) involves assessing whether a traumatic event meets Criterion A, which is necessary to establish symptom severity and a potential PTSD diagnosis. As research moves online, methods to establish Criterion A have varied widely, influencing the accuracy and consistency of PTSD diagnoses. Literature suggests relying solely on self-assessment of trauma experiences may be problematic. This study evaluated whether integration of large language models (LLMs) directly into online self-report data collection could enhance assessment of Criterion A for PTSD.
METHOD: The present study leveraged LLMs to test a new method for enhancing the online assessment of Criterion A from self-report. Adults (N = 110) completed the extended Life Events Checklist for the Diagnostic and Statistical Manual of Mental Disorders, fifth edition. An LLM was integrated directly into the online survey tool Qualtrics and utilized via Application Programming Interface to code text descriptions to actively follow up with participants by providing additional questions/prompts. Four clinician raters independently evaluated the text descriptions after data collection was complete to determine the proportion of individuals meeting Criterion A and to establish interrater reliability with LLMs.
RESULTS: The percentage of participants who met Criterion A based on clinician ratings was increased from an average of 65% (range: 59%-71%) at the first description to an average of 86% across all follow-up clarifications. However, interrater reliability of LLMs with clinician raters was only fair, original LLM mean κ = 0.26 (κ range: 0.18-0.46), newer LLM mean κ = 0.35 (κ range: 0.23-0.47).
CONCLUSIONS: Findings suggest that use of LLMs for enhancing Criterion A assessment led to increased information from participants, leading to greater reporting of events meeting Criterion A. However, LLMs did not provide determination of Criterion A on par with clinicians. Findings highlight the need for further assessment of integrating LLMs into online research or treatment. (PsycInfo Database Record (c) 2025 APA, all rights reserved).
PubMed:41325158 | DOI:10.1037/tra0002087