Supplemental Material for: AI-generated guidance for non-specialist physicians managing inflammatory bowel disease: expert evaluation and case-based testing of GPT-4 and DeepSeek

Twenty real-world IBD clinical questions submitted by non-specialist physicians and one standardized case vignette were entered into GPT-4 and DeepSeek using identical prompts. Seven IBD specialists independently rated response accuracy and completeness on 5-point Likert scales, and 12 non-specialist physicians rated clinical usefulness. The 12 non‑specialist physicians completed a two‑round cross‑over, pre–post case‑based test using 20 IBD clinical vignettes to evaluate whether reviewing AI outputs improves responses to common management vignettes. </p><p dir="ltr">Results: Both models produced generally accurate and complete answers (median expert scores ≥4.0/5).

Experts rated DeepSeek higher for the accuracy and completeness of diagnosis,treatment and, and follow-up. After reviewing AI answers, non‑specialist physicians’ case‑based test scores improved significantly: median per‑round score (0–10) increased from 5.5 to 8.0 for GPT‑4 (Δ=3.0, p<0.001) and from 6.0 to 8.0 for DeepSeek (Δ=2.5, p<0.001). The between‑model difference in score change was not significant (p=0.781).

Across both models, common limitations included limited patient‑level individualization, inconsistent red‑flag/referral triggers, and lack of verifiable, time‑stamped references; no harmful recommendations were identified, but safety omissions were present in both models.</p><p dir="ltr">Conclusion: GPT-4 and DeepSeek generated relatively accurate and comprehensive guidance and improved non-specialist physicians’ performance in IBD case vignettes testing, supporting their potential as adjunct educational and reference tools for clinical reasoning.

If implemented clinically, AI-based support should incorporate explicit safety prompts (red flags and referral triggers), and be validated prospectively within real-world workflows. </p>