Artificial intelligence model GPT4 narrowly fails simulated radiological protection exam Journal Article
| Authors: | Roemer, G.; Li, A.; Mahmood, U.; Dauer, L.; Bellamy, M. |
| Article Title: | Artificial intelligence model GPT4 narrowly fails simulated radiological protection exam |
| Abstract: | This study assesses the efficacy of Generative Pre-Trained Transformers (GPT) published by OpenAI in the specialised domains of radiological protection and health physics. Utilising a set of 1064 surrogate questions designed to mimic a health physics certification exam, we evaluated the models’ ability to accurately respond to questions across five knowledge domains. Our results indicated that neither model met the 67% passing threshold, with GPT-3.5 achieving a 45.3% weighted average and GPT-4 attaining 61.7%. Despite GPT-4’s significant parameter increase and multimodal capabilities, it demonstrated superior performance in all categories yet still fell short of a passing score. The study’s methodology involved a simple, standardised prompting strategy without employing prompt engineering or in-context learning, which are known to potentially enhance performance. The analysis revealed that GPT-3.5 formatted answers more correctly, despite GPT-4’s higher overall accuracy. The findings suggest that while GPT-3.5 and GPT-4 show promise in handling domain-specific content, their application in the field of radiological protection should be approached with caution, emphasising the need for human oversight and verification. © 2024 The Author(s). Published on behalf of the Society for Radiological Protection by IOP Publishing Ltd. |
| Keywords: | artificial intelligence; radiation protection; artificial; health physics; intelligence; humans; human; electric power supplies; power supply; gpt4 |
| Journal Title: | Journal of Radiological Protection |
| Volume: | 44 |
| Issue: | 1 |
| ISSN: | 0952-4746 |
| Publisher: | IOP Publishing Ltd |
| Date Published: | 2024-03-01 |
| Start Page: | 013502 |
| Language: | English |
| DOI: | 10.1088/1361-6498/ad1fdf |
| PUBMED: | 38232401 |
| PROVIDER: | scopus |
| DOI/URL: | |
| Notes: | The MSK Cancer Center Support Grant (P30 CA008748) is acknowledged in the PDF. Corresponding MSK author is Michael B. Bellamy-- Source: Scopus |
Altmetric
Citation Impact
BMJ Impact Analytics
Related MSK Work