Benchmarking Generative Artificial Intelligence Against Human Judgment in Eating Disorder Case Recognition and Treatment Recommendations

ChatGPT-5.4 was presented with two vignettes and prompted with questions related to problem identification and treatment recommendation. Vignette one depicted a restrictive ED consistent with anorexia nervosa or atypical anorexia nervosa, with body weight varying across low-, average-, and higher-weight conditions. Vignette two depicted a person with binge-eating disorder (BED).

Responses across 20 prompt administrations per vignette were compared against benchmark data from clinicians for vignette one and community participants for vignette two. Results: For vignette one, ChatGPT identified an ED or possible ED in 100% of responses for the low- and average-weight conditions and 90% for the higher-weight condition, compared with 47%, 21%, and 16% among human clinicians, respectively. It recommended specialized ED treatment 100% of the time across all three weight conditions, compared with 35%, 19%, and 17% of clinicians, respectively.

For vignette two, ChatGPT correctly identified BED as the problem in 100% of prompt repetitions (vs. 59% in community participants) and more consistently endorsed evidence-based treatments than community participants (100% vs. 50%). Supplementary analyses confirmed similar results when prompting Gemini-3 and Claude-4.5. Conclusion: Generative AI can accurately identify ED presentations and recommend suitable care in structured scenarios, while showing little evidence of common biases observed in human samples.