CounselReflect: A Toolkit for Auditing Mental-Health Dialogues

arXiv cs.CL / 4/1/2026

📰 NewsSignals & Early TrendsTools & Practical UsageModels & Research

共有:

Key Points

CounselReflect is an end-to-end toolkit designed to help users audit mental-health support dialogues produced by conversational systems, addressing the lack of structured, transparent evaluation methods.
It produces multi-dimensional, inspectable reports (session-level summaries, turn-level scores, and evidence-linked excerpts) rather than a single opaque quality score.
The toolkit combines 12 model-based metrics from task-specific predictors with rubric-based metrics expanded via a literature-derived library of 69 metrics plus user-defined custom metrics evaluated using configurable LLM judges.
CounselReflect is offered as a web application, browser extension, and CLI for both real-time auditing and large-scale evaluation workflows.
Initial validation includes a user study (20 participants) and an expert review (6 mental-health professionals), with demo materials and full source code provided.

Abstract

Mental-health support is increasingly mediated by conversational systems (e.g., LLM-based tools), but users often lack structured ways to audit the quality and potential risks of the support they receive. We introduce CounselReflect, an end-to-end toolkit for auditing mental-health support dialogues. Rather than producing a single opaque quality score, CounselReflect provides structured, multi-dimensional reports with session-level summaries, turn-level scores, and evidence-linked excerpts to support transparent inspection. The system integrates two families of evaluation signals: (i) 12 model-based metrics produced by task-specific predictors, and (ii) rubric-based metrics that extend coverage via a literature-derived library (69 metrics) and user-defined custom metrics, operationalized with configurable LLM judges. CounselReflect is available as a web application, browser extension, and command-line interface (CLI), enabling use in real-time settings as well as at scale. Human evaluation includes a user study with 20 participants and an expert review with 6 mental-health professionals, suggesting that CounselReflect supports understandable, usable, and trustworthy auditing. A demo video and full source code are also provided.