Results of the 2016 International Skin Imaging Collaboration International Symposium on Biomedical Imaging challenge: Comparison of the accuracy of computer algorithms to dermatologists for the diagnosis of melanoma from dermoscopic images Journal Article


Authors: Marchetti, M. A.; Codella, N. C. F.; Dusza, S. W.; Gutman, D. A.; Helba, B.; Kalloo, A.; Mishra, N.; Carrera, C.; Celebi, M. E.; DeFazio, J. L.; Jaimes, N.; Marghoob, A. A.; Quigley, E.; Scope, A.; Yélamos, O.; Halpern, A. C.; for the International Skin Imaging Collaboration
Article Title: Results of the 2016 International Skin Imaging Collaboration International Symposium on Biomedical Imaging challenge: Comparison of the accuracy of computer algorithms to dermatologists for the diagnosis of melanoma from dermoscopic images
Abstract: BACKGROUND: Computer vision may aid in melanoma detection. OBJECTIVE: We sought to compare melanoma diagnostic accuracy of computer algorithms to dermatologists using dermoscopic images. METHODS: We conducted a cross-sectional study using 100 randomly selected dermoscopic images (50 melanomas, 44 nevi, and 6 lentigines) from an international computer vision melanoma challenge dataset (n = 379), along with individual algorithm results from 25 teams. We used 5 methods (nonlearned and machine learning) to combine individual automated predictions into "fusion" algorithms. In a companion study, 8 dermatologists classified the lesions in the 100 images as either benign or malignant. RESULTS: The average sensitivity and specificity of dermatologists in classification was 82% and 59%. At 82% sensitivity, dermatologist specificity was similar to the top challenge algorithm (59% vs. 62%, P = .68) but lower than the best-performing fusion algorithm (59% vs. 76%, P = .02). Receiver operating characteristic area of the top fusion algorithm was greater than the mean receiver operating characteristic area of dermatologists (0.86 vs. 0.71, P = .001). LIMITATIONS: The dataset lacked the full spectrum of skin lesions encountered in clinical practice, particularly banal lesions. Readers and algorithms were not provided clinical data (eg, age or lesion history/symptoms). Results obtained using our study design cannot be extrapolated to clinical practice. CONCLUSION: Deep learning computer vision systems classified melanoma dermoscopy images with accuracy that exceeded some but not all dermatologists. Copyright © 2017 American Academy of Dermatology, Inc. Published by Elsevier Inc. All rights reserved.
Keywords: comparative study; melanoma; congresses as topic; dermoscopy; nevus; skin neoplasms; skin cancer; pathology; diagnostic imaging; algorithms; epiluminescence microscopy; skin tumor; algorithm; computer assisted diagnosis; organization; cross-sectional study; cross-sectional studies; computer vision; roc curve; receiver operating characteristic; diagnosis, computer-assisted; dermatologist; lentigo; dermatologists; machine learning; humans; human; computer algorithm; international symposium on biomedical imaging; reader study; international skin imaging collaboration
Journal Title: Journal of the American Academy of Dermatology
Volume: 78
Issue: 2
ISSN: 0190-9622
Publisher: Mosby Elsevier  
Date Published: 2018-02-01
Start Page: 270
End Page: 277.e1
Language: English
DOI: 10.1016/j.jaad.2017.08.016
PUBMED: 28969863
PROVIDER: scopus
PMCID: PMC5768444
DOI/URL:
Notes: Article -- Export Date: 3 December 2018 -- Source: Scopus
Altmetric
Citation Impact
MSK Authors
  1. Elizabeth Ann Quigley
    20 Quigley
  2. Allan C Halpern
    388 Halpern
  3. Jennifer Defazio
    15 Defazio
  4. Stephen Dusza
    261 Dusza
  5. Alon Scope
    124 Scope
  6. Ashfaq A Marghoob
    512 Marghoob
  7. Michael Armando Marchetti
    154 Marchetti
  8. Aadi Kalloo
    4 Kalloo