Objective: This study aims to compare the guideline compliance of two widely used AI-based chatbot systems, ChatGPT and Bing AI, with the clinical recommendations outlined in the Disorders of Sex Development (DSD) guideline published by the Turkish Neonatal Society. Materials and Methods: A standardized evaluation set comprising 40 questions based on the DSD guideline was utilized. The questions were grouped under six main categories reflecting clinical decision-making processes and were presented to both ChatGPT and Bing AI. Responses were scored on a 5-point Likert scale by two independent experts, assessing their alignment with the guideline. Mean scores were calculated for each category, and statistical comparisons were made using the Wilcoxon signed-rank test. Results: ChatGPT demonstrated high consistency with the guideline across all categories (mean score: 4.88), while Bing AI showed lower compliance in several areas (mean score: 3.25). The differences in scores between the two systems were statistically significant across all categories (p < 0.05), with Bing AI performing particularly poorly in the areas of diagnosis/laboratory testing and multidisciplinary approach. Conclusion: ChatGPT demonstrated higher accuracy and consistency than Bing AI in providing guideline-based clinical support regarding DSD. The use of AIsupported systems aligned with current guidelines holds significant potential in supporting complex, multidisciplinary decision-making processes. Therefore, the selection of AI tools in clinical settings should be informed by such systematic evaluations.
Cite this Article As :
Uygun SS, Ozcan Siki F. A Comparative Analysis of Large Language Models in Managing Disorders of Sex Development: Evaluation Based on Clinical Guidelines. Selcuk Med J 2025;41(4): 201-204
1. Özalp Kızılay D, Özen S. Current diagnostic approaches in the genetic diagnosis of disorders of sex development. J Clin Res Pediatr Endocrinol. 2024;16(4):401-10. doi: 10.4274/jcrpe.galenos.2024.2024-3-3.
2. Yavas AZ, Guran T. Diagnosis and management of non-CAH 46,XX disorders/differences in sex development. Front Endocrinol (Lausanne). 2024;15:1354759. doi: 10.3389/fendo.2024.1354759.
3. Ndoye NA, Diallo BA, Sylla M, et al. Ovotesticular disorders of sexual development: Diagnostic, therapeutic, and evolutionary aspects. J Pediatr Surg. 2025;60(4):162187. Doi: 10.1016/j.jpedsurg.2025.162187
4. Khorashad BS, Goodarzi H, Greenfield D, et al. Recommendations for 46,XY disorders/differences of sex development across two decades: Insights from North American pediatric endocrinologists and urologists. Arch Sex Behav. 2024;53(8):2939-56. doi.org/10.1007/s10508-024-02942-1
5. Altıntaş E, Kılıç R, Gürlek C, et al. Comparative analysis of artificial intelligence chatbot recommendations for urolithiasis management: A study of EAU guideline compliance. Fr J Urol. 2024;34(7-8):102666. doi: 10.1016/j.fjurol.2024.102666
6. Alasker, A., Alsalamah, S., Alshathri, N. et al. Performance of large language models (LLMs) in providing prostate cancer information. BMC Urol 24, 177 (2024). Doi:10.1186/s12894-024-01570-0.
7. Caglar U, Tunç F, Özen B, et al. Evaluating the performance of ChatGPT in answering questions related to pediatric urology. J Pediatr Urol. 2024;20(1):26.e1-5. doi: 10.1016/j.jpurol.2023.08.003.
8. Ranjan J, Gupta A, Mehta S, et al. Assessment of artificial intelligence platforms with regard to medical microbiology knowledge: An analysis of ChatGPT and Gemini. Cureus. 2024;16(5):e60675. doi: 10.7759/cureus.60675.
9. Akyol Onder EN, Ensari E, Ertan P. ChatGPT-4o's performance on pediatric vesicoureteral reflux. J Pediatr Urol. 2025 Apr;21(2):504-509. doi: 10.1016/j.jpurol.2024.12.002.
10. Sarikaya M, Ozcan Siki F, Ciftci I. Use of artificial intelligence in vesicoureteral reflux disease: A comparative study of guideline compliance. J Clin Med. 2025 Mar 30;14(7):2378. doi: 10.3390/jcm14072378.
11. Deniz MS, Guler BY. Assessment of ChatGPT's adherence to ETA-thyroid nodule management guideline over two different time intervals 14 days apart: In binary and multiple-choice queries. Endocrine. 2024;85(2):794-802. doi: 10.1007/s12020-024-03750-2.
12. Cakir H, Kaya Z, Tunçer M, et al. Evaluating the performance of ChatGPT in answering questions related to urolithiasis. Int Urol Nephrol. 2024;56(1):17-21. doi: 10.1007/s11255-023-03773-0
13. Cil G, Dogan K. The efficacy of artificial intelligence in urology: A detailed analysis of kidney stone-related queries. World J Urol. 2024;42(1):158. doi: 10.1007/s00345-024-04847-z.
14. Lee Y, Tessier L, Brar K, et al. Performance of artificial intelligence in bariatric surgery: Comparative analysis of ChatGPT-4, Bing, and Bard in the American Society for Metabolic and Bariatric Surgery textbook of bariatric surgery questions. Surg Obes Relat Dis. 2024;20(7):609-13. doi: 10.1016/j.soard.2024.04.014.
15. Klang E, Sourosh A, Nadkarni GN, et al. Evaluating the role of ChatGPT in gastroenterology: A comprehensive systematic review of applications, benefits, and limitations. Therap Adv Gastroenterol. 2023;16:17562848231218618. doi: 10.1177/17562848231218618.
16. Ríos-Hoyo A, Shan NL, Li A, et al. Evaluation of large language models as a diagnostic aid for complex medical cases. Front Med (Lausanne). 2024 Jun 20;11:1380148. doi: 10.3389/fmed.2024.1380148.
17. Lopez-Gonzalez R, Sanchez-Cordero S, Pujol-Gebellí J, et al. Evaluation of the Impact of ChatGPT on the Selection of Surgical Technique in Bariatric Surgery. Obes Surg. 2025 Jan;35(1):19-24. doi: 10.1007/s11695-024-07279-1.
Research Paper
View as PDF
Keywords
Large Language Models, Disorders of Sex Development, Artificial Intelligence, Neonatology, Decision Support Systems
