Nonexpert crowds outperform expert individuals in diagnostic accuracy on a skin lesion diagnosis task Conference Paper
| Authors: | Duhaime, E. P.; Jin, M.; Moulton, T.; Weber, J.; Kurtansky, N. R.; Halpern, A.; Rotemberg, V. |
| Title: | Nonexpert crowds outperform expert individuals in diagnostic accuracy on a skin lesion diagnosis task |
| Conference Title: | 2023 IEEE International Symposium on Biomedical Imaging (ISBI) |
| Abstract: | A recent study [1] showed that individual physicians with at least ten years of experience as dermatologists achieved 74.7% accuracy on average in labeling images from the multiclass International Skin Imaging Collaboration (ISIC) 2018 challenge dataset. Using a novel gamified crowdsourcing method, we collected 144,383 nonexpert opinions over two weeks on the medical image annotation platform DiagnosUs, and the resulting crowd consensus labels obtained by aggregating using a plurality rule achieved a significantly higher accuracy of 78.1% (p=0.0014), a multiclass ROC AUC (area under the receiver operating characteristic curve) of 0.948 (95% CI 0.936-0.959), and malignant versus benign ROC AUC of 0.928 (95% CI 0.911-0.943). These results suggest an opportunity to harness gamified methods to assist in the creation of high-quality labeled datasets that could benefit medical artificial intelligence (AI) development. © 2023 IEEE. |
| Keywords: | diagnostic accuracy; medical imaging; dermatology; skin lesion; computer aided diagnosis; skin imaging; lesion classification; international skin imaging collaboration; crowdsourcing; high-accuracy; gamification; isic; skin lesion classification; labeling image; medical image annotation |
| Journal Title | Proceedings of the IEEE International Symposium on Biomedical Imaging |
| Conference Dates: | 2021 Apr 18-21 |
| Conference Location: | Cartagena de Indias, Colombia |
| ISBN: | 1945-7928 |
| Publisher: | IEEE |
| Date Published: | 2023-01-01 |
| Language: | English |
| DOI: | 10.1109/isbi53787.2023.10230646 |
| PROVIDER: | scopus |
| DOI/URL: | |
| Notes: | Conference paper -- Source: Scopus |
Altmetric
Citation Impact
BMJ Impact Analytics
Related MSK Work