Comparison Of The Performance Of Chatgpt-3.5, Chatgpt-4o And Gemini 2.5 İn The Management Of Rheumatoid Arthritis: Analysis Of EULAR 2022 Recommendations

Ozlem Kuculmez; Ahmet Usen

doi:10.29284/n50b4s27

Authors

Ozlem Kuculmez Department of Physical Medicine and Rehabilitation, Baskent University Alanya Hospital, Antalya, Turkey
Ahmet Usen Department of Physical Medicine and Rehabilitation, University

DOI:

https://doi.org/10.29284/n50b4s27

Keywords:

Arthritis, artificial intelligence, guideline, rheumatic disease, rheumatoid arthritis.

Abstract

Background: Clinical guidelines are essential for diagnosing and managing rheumatic conditions like rheumatoid arthritis, which can result in deformities and impairments. Large language models such as ChatGPT and Gemini have become widely popular due to their fast responses, user-friendly interfaces, and ability to improve through human feedback and reinforcement learning. Prior studies have examined how accurate and reliable these models are when answering common public health questions.

Objective: The aim of this research is to assess and contrast the answers provided by different large language models to open-ended questions based on the 2022 EULAR guideline.

Materials and Methods: This was a comparative, cross-sectional study. Five overarching principles and eleven recommendations from the 2022 EULAR guidelines were rephrased into open-ended clinical questions. These questions were presented to ChatGPT-3.5, ChatGPT-4o, and Gemini 2.5 Flash. The responses were assessed using a 7-point Likert scale for usability and reliability, readability scores (Flesch-Kincaid Reading Ease (FKRE) and Flesch-Kincaid Grade Level (FKGL)), and similarity metrics (Universal Sentence Encoder (USE) and ROUGE-L). Different models were compared statistically and p < 0.05 was revealed as statistically significant.

Results: Gemini 2.5 produced responses with a significantly higher FKGL than both ChatGPT-3.5 and ChatGPT-4o (p = 0.003 and p = 0.004). FKRE was significantly higher for ChatGPT-3.5 than Gemini 2.5 (p = 0.011), whereas the difference between ChatGPT-4o and Gemini 2.5 was not significant. Reliability did not differ between models (p > 0.05), but ChatGPT-4o had a significantly higher usefulness score than Gemini 2.5 (p < 0.05). ChatGPT-4o and Gemini 2.5 showed significantly higher semantic similarity than ChatGPT-3.5 (p < 0.05). ROUGE-L F1 scores showed no significant differences (p > 0.05). FKRE and FKGL were strongly negatively correlated, while reliability and usefulness were strongly positively correlated (p < 0.05).

Conclusions: It was determined that ChatGPT-3.5 produced more readable responses than Gemini 2.5, while ChatGPT-4o demonstrated greater usefulness in answers to open-ended questions derived from the EULAR 2022 guideline; however, no difference in reliability was observed.

References

[1] Di Matteo A, Bathon JM, Emery P. Rheumatoid arthritis. Lancet. 2023 Nov 25;402(10416):2019-2033. doi: 10.1016/S0140-6736(23)01525-8.

[2] Jang S, Kwon EJ, Lee JJ. Rheumatoid Arthritis: Pathogenic Roles of Diverse Immune Cells. Int J Mol Sci. 2022 Jan 14;23(2):905. doi: 10.3390/ijms23020905.

[3] Akgöz A, Yakut Y, Tüfekçi O, Aktaş BE, Sari EO, Barlak A et al. Biopsychosocial model-based exercise improves muscle strength, proprioception, pain, function, and quality of life in rheumatoid arthritis patients with knee involvement: a randomized controlled clinical tiral. Rheumatol Int. 2025 Sep 6;45(9):221. doi: 10.1007/s00296-025-05976-3.

[4] Smolen JS, Landewé RBM, Bergstra SA, Kerschbaumer A, Sepriano A, Aletaha D et al. EULAR recommendations for the management of rheumatoid arthritis with synthetic and biological disease-modifying antirheumatic drugs: 2022 update. Ann Rheum Dis. 2023 Jan;82(1):3-18. doi: 10.1136/ard-2022-223356.

[5] Nagy G, Roodenrijs NMT, Welsing PMJ, Kedves M, Hamar A, van der Goes MC et al. EULAR points to consider for the management of difficult-to-treat rheumatoid arthritis. Ann Rheum Dis. 2022 Jan;81(1):20-33. doi: 10.1136/annrheumdis-2021-220973.

[6] Park YJ, Pillai A, Deng J, Guo E, Gupta M, Paget M et al. Assessing the research landscape and clinical utility of large language models: a scoping review. BMC Med Inf Decis Mak. 2024,24(1):72. https://doi.org/10.1186/ s12911-024-02459-6

[7] Wang J, Tian Y, Zhou T, Tong D, Ma J, Li J. A survey of artificial intelligence in rheumatoid arthritis. Rheumatol Immunol Res. 2023 Jul 22;4(2):69-77. doi: 10.2478/rir-2023-0011.

[8] Benavent D, Carmona L, García Llorente JF, Montoro M, Ramirez S, Otón T et al. Artificial intelligence to predict treatment response in rheumatoid arthritis and spondyloarthritis: a scoping review. Rheumatol Int. 2025 Apr 7;45(4):91. doi: 10.1007/s00296-025-05825-3.

[9] Liu S, Wright AP, Patterson BL, Wanderer JP, Turer RW, Nelson SD et al. Using AI-generated suggestions from ChatGPT to optimize clinical decision support. J Am Med Inform Assoc. 2023,30(7):1237-1245. doi: 10.1093/jamia/ocad072.

[10] Sciberras M, Farrugia Y, Gordon H, Furfaro F, Allocca M, Torres J et al. Accuracy of Information given by ChatGPT for Patients with Inflammatory Bowel Disease in Relation to ECCO Guidelines. J Crohns Colitis. 2024,18(8):1215-1221. doi: 10.1093/ecco-jcc/jjae040.

[11] Liu M, Okuhara T, Chang X, Shirabe R, Nishiie Y, Okada H et al. Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis. J Med Internet Res. 2024 Jul 25;26:e60807. doi: 10.2196/60807.

[12] Tong L, Zhang C, Liu R, Yang J, Sun Z. Comparative performance analysis of large language models: ChatGPT-3.5, ChatGPT-4 and Google Gemini in glucocorticoid-induced osteoporosis. J Orthop Surg Res. 2024,19(1):574. doi: 10.1186/s13018-024-04996-2.

[13] Uz C, Umay E. "Dr ChatGPT": Is it a reliable and useful source for common rheumatic diseases? Int J Rheum Dis. 2023,26(7):1343-1349. doi: 10.1111/1756-185X.14749.

[14] Izzidien A, Fitz S, Romero P, Loe BS, Stillwell D. Developing a sentence level fairness metric using word embeddings. Int J Digit Humanit. 2022,1-36. doi: 10.1007/s42803-022-00049-4.

[15] Tang L, Sun Z, Idnay B, Nestor JG, Soroush A, Elias PA et al. Evaluating Large Language Models on Medical Evidence Summarization. medRxiv [Preprint]. 2023 Apr 24:2023.04.22.23288967. doi: 10.1101/2023.04.22.23288967. Update in: NPJ Digit Med. 2023,6(1):158. doi: 10.1038/s41746-023-00896-7.

[16] Gulden C, Kirchner M, Schüttler C, Hinderer M, Kampf M, Prokosch HU et al. Extractive summarization of clinical trial descriptions. Int J Med Inform. 2019,129:114-121. doi: 10.1016/j.ijmedinf.2019.05.019.

[17] Kılıcoglu V.S, Yurdakul O.V, Aydın T. Evaluation of Chat Generative Pre-trained Transformer’s responses to frequently asked questions about psoriatic arthritis: A study on quality and readability. Ann Med Res 2025,32(2):79–84

[18] Zhang Z, Gao X, Liu S, Wang Q, Wang Y, Hou S et al. Global, regional, and national epidemiology of rheumatoid arthritis among people aged 20-54 years from 1990 to 2021. Sci Rep. 2025 Mar 28;15(1):10736. doi: 10.1038/s41598-025-92150-1.

[19] Haville S, Deane KD. Pre-RA: Can early diagnosis lead to prevention? Best Pract Res Clin Rheumatol. 2022 Mar;36(1):101737. doi: 10.1016/j.berh.2021.101737.

[20] Hügle T. The wide range of opportunities for large language models such as ChatGPT in rheumatology. RMD Open. 2023,9(2):e003105. doi: 10.1136/rmdopen-2023-003105.

[21] Chandwar K, Prasanna Misra D. What does artificial intelligence mean in rheumatology? Arch Rheumatol. 2024 Feb 12;39(1):1-9. doi: 10.46497/ArchRheumatol.2024.10664.

[22] Bird A, Oakden-Rayner L, McMaster C, Smith LA, Zeng M, Wechalekar MD et al. Artificial intelligence and the future of radiographic scoring in rheumatoid arthritis: a viewpoint. Arthritis Res Ther. 2022 Dec 12;24(1):268. doi: 10.1186/s13075-022-02972-x.

[23] Obayya M, Alamgeer M, S Alzahrani J, Alabdan R, N Al-Wesabi F, Mohamed A et al. Artificial Intelligence Driven Biomedical Image Classification for Robust Rheumatoid Arthritis Classification. Biomedicines. 2022 Oct 26;10(11):2714. doi: 10.3390/biomedicines10112714.

[24] Alivernini S, Cañete JD, Bacardit J, Kurowska-Stolarska M. Using explainable artificial intelligence to predict and forestall flare in rheumatoid arthritis. Nat Med. 2024 Apr;30(4):925-926. doi: 10.1038/s41591-024-02818-w.

[25] Varga G, Stoicu-Tivadar L, Nicola S. Serious Gaming and Artificial Intelligence in Rehabilitation of Rheumatoid Arthritis. Stud Health Technol Inform. 2022 Jun 29;295:562-565. doi: 10.3233/SHTI220790.

[26] Ray PP. ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet Things Cyber-Phys Syst. 2023;3:121–154. doi:10.1016/j.iotcps.2023.07.003.

[27] Chen SY, Kuo HY, Chang SH. Perceptions of ChatGPT in healthcare: usefulness, trust, and risk. Front Public Health. 2024,12:1457131. doi: 10.3389/fpubh.2024.1457131.

[28] Daungsupawong H, Wiwanitkit V. Comparative performance of artificial intelligence models in rheumatology board-level questions: evaluating Google Gemini and ChatGPT-4o: correspondence. Clin Rheumatol. 2024,43(12):4015-4016. doi: 10.1007/s10067-024-07176-z.

[29] Gilvaz VJ, Reginato AM. Artificial intelligence in rheumatoid arthritis: potential applications and future implications. Front Med (Lausanne). 2023 Nov 16;10:1280312. doi: 10.3389/fmed.2023.1280312.

[30] Çelik NÇ, Kılınç EA. Assessment of ChatGPT's adherence to EULAR diagnostic criteria and therapeutic protocols for rheumatoid arthritis at two distinct time points, 14 days apart, utilizing binary and multiple-choice inquiries. Clin Rheumatol. 2025 Apr 22. doi: 10.1007/s10067-025-07417-9.

[31] Bai L, Zhang Y, Wang P, Zhu X, Xiong JW, Cui L. Improved diagnosis of rheumatoid arthritis using an artificial neural network. Sci Rep. 2022 Jun 13;12(1):9810. doi: 10.1038/s41598-022-13750-9.

[32] Valdivieso Shephard JL, Alvarez Robles EJ, Cámara Hijón C, Hernandez Breijo B, Novella-Navarro M, Bogas Schay P et al. Predicting anti-TNF treatment response in rheumatoid arthritis: An artificial intelligence-driven model using cytokine profile and routine clinical practice parameters. Heliyon. 2023 Nov 28;10(1):e22925. doi: 10.1016/j.heliyon.2023.e22925.

[33] Krusche M, Callhoff J, Knitza J, Ruffer N. Diagnostic accuracy of a large language model in rheumatology: comparison of physician and ChatGPT-4. Rheumatol Int. 2024,44(2):303-306. doi: 10.1007/s00296-023-05464-6.

[34] Is EE, Menekseoglu AK. Comparative performance of artificial intelligence models in rheumatology board-level questions: evaluating Google Gemini and ChatGPT-4o. Clin Rheumatol. 2024,43(11):3507-3513. doi: 10.1007/s10067-024-07154-5.

[35] Coskun BN, Yagiz B, Ocakoglu G, Dalkilic E, Pehlivan Y. Assessing the accuracy and completeness of artificial intelligence language models in providing information on methotrexate use. Rheumatol Int. 2024,44(3):509-515. doi: 10.1007/s00296-023-05473-5.

[36] McMaster C, Bird A, Liew DFL, Buchanan RR, Owen CE, Chapman WW et al. Artificial Intelligence and Deep Learning for Rheumatologists. Arthritis Rheumatol. 2022 Dec;74(12):1893-1905. doi: 10.1002/art.42296.

[37] Vlad AL, Popazu C, Lescai AM, Voinescu DC, Baltă AAȘ. The Role of Artificial Intelligence in the Diagnosis and Management of Rheumatoid Arthritis. Medicina (Kaunas). 2025 Apr 9;61(4):689. doi: 10.3390/medicina61040689.

[38] Garcia-Agundez A, Schmajuk G, Yazdany J. Promises and pitfalls of artificial intelligence models in forecasting rheumatoid arthritis treatment response and outcomes. Semin Arthritis Rheum. 2025 Feb;70S:152584. doi: 10.1016/j.semarthrit.2024.152584.

[39] Ataman S., Borman P., Evcik D., Aydog E., Ayhan F., Yildizlar D et al. Management of rheumatoid arthritis: consensus recommendations from the Turkish League Against Rheumatism. Turkish Journal of Rheumatology, 2011; 26.4.

[40] Aljamaan F, Temsah MH, Altamimi I, Al-Eyadhy A, Jamal A, Alhasan K et al. Reference Hallucination Score for Medical Artificial Intelligence Chatbots: Development and Usability Study. JMIR Med Inform. 2024 Jul 31; 12: e54345.