Does BERT pretrained on clinical notes reveal sensitive data? Conference Paper

  


Authors: Lehman, E.; Jain, S.; Pichotta, K.; Goldberg, Y.; Wallace, B. C.
Title: Does BERT pretrained on clinical notes reveal sensitive data?
Conference Title: Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2021)
Abstract: Large Transformers pretrained over clinical notes from Electronic Health Records (EHR) have afforded substantial gains in performance on predictive clinical tasks. The cost of training such models (and the necessity of data access to do so) coupled with their utility motivates parameter sharing, i.e., the release of pretrained models such as ClinicalBERT (Alsentzer et al., 2019). While most efforts have used deidentified EHR, many researchers have access to large sets of sensitive, non-deidentified EHR with which they might train a BERT model (or similar). Would it be safe to release the weights of such a model if they did? In this work, we design a battery of approaches intended to recover Personal Health Information (PHI) from a trained BERT. Specifically, we attempt to recover patient names and conditions with which they are associated. We find that simple probing methods are not able to meaningfully extract sensitive information from BERT trained over the MIMIC-III corpus of EHR. However, more sophisticated “attacks” may succeed in doing so: To facilitate such research, we make our experimental setup and baseline probing models available. © 2021 Association for Computational Linguistics.
Keywords: performance; clinical notes; condition; electronic health; clinical tasks; data access; health records; parameter sharing; personal health informations; sensitive datas; sensitive data
Journal Title Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Conference Dates: 2022 Jun 6-11
Conference Location: Virtual
ISBN: 978-1-954085-46-6
Publisher: Association for Computational Linguistics  
Location: Stroudsburg, PA
Date Published: 2021-01-01
Start Page: 946
End Page: 959
Language: English
PROVIDER: scopus
DOI/URL:

https://www.scopus.com/inward/record.uri?eid=2-s2.0-85137709416&partnerID=40&md5=2dafb927696a4216953762d52513ddaa
Notes: Conference Paper -- Export Date: 3 October 2022 -- Source: Scopus