AI matched human judgments on some mental health posts in a small Reddit test

The researchers examined whether AI could classify possible “root causes” described in social media posts about mental health. In this study, root causes means the underlying themes people connected to their distress in their own words.

The team used Reddit posts and created an annotated dataset, meaning posts were labeled by human reviewers. Two AI systems, AutoML and PaLM 2, were trained using 800 labeled posts. The models were then tested on a separate set of 50 unseen posts. Human consensus labels were used as the comparison.

On those 50 test posts, PaLM 2 agreed with the human consensus in 40 cases, or 80%. AutoML agreed in 34 cases, or 68%. The researchers also compared the models’ errors. That analysis did not find a statistically significant difference between the two tools’ classification outcomes.

The study included a qualitative review, which looks at patterns beyond a single score. The researchers reported that PaLM 2 often produced answers that fit the broader context of a person’s narrative. AutoML appeared to rely more on particular words or language features.

Still, neither AI system was consistently better in the formal comparison of errors. Human reviewers and AI tools also found some categories hard to sort, especially when a post could reasonably fit more than one category. That ambiguity matters because personal accounts of mental health can be complex and may not fit neatly into one label.

The results are an early evaluation of text classification, not evidence that AI can provide mental health care. The study did not test the tools in clinics, compare them with usual care, or measure whether people received faster, safer, or more personalized support. It also did not report patient outcomes or follow people over time.

The test set was small, with 50 unseen posts. The study also used Reddit narratives, which may not represent people seeking care elsewhere or the wider population. A post may not contain enough context to understand a person’s circumstances, and it is not clear whether posts came from unique individuals.

Before a tool like this could be considered for use in care, it would need testing in real settings with clear safeguards, diverse participants, and careful checks for errors. Researchers would also need to show whether it helps professionals and patients without creating harmful misunderstandings or missed concerns.