Digital forms for all: A holistic multimodal large language model agent for health data entry Journal Article
| Authors: | Cuadra, A.; Breuch, J.; Estrada, S.; Ihim, D.; Hung, I.; Askaryar, D.; Hassanien, M.; Fessele, K. L.; Landay, J. A. |
| Article Title: | Digital forms for all: A holistic multimodal large language model agent for health data entry |
| Abstract: | Digital forms help us access services and opportunities, but they are not equally accessible to everyone, such as older adults or those with sensory impairments. Large language models (LLMs) and multimodal interfaces offer a unique opportunity to increase form accessibility. Informed by prior literature and needfinding, we built a holistic multimodal LLM agent for health data entry. We describe the process of designing and building our system, and the results of a study with older adults (N =10). All participants, regardless of age or disability status, were able to complete a standard 47-question form independently using our system - -one blind participant said it was "a prayer answered."Our video analysis revealed how different modalities provided alternative interaction paths in complementary ways (e.g., the buttons helped resolve transcription errors and speech helped provide more options when the pre-canned answer choices were insufficient). We highlight key design guidelines, such as designing systems that dynamically adapt to individual needs. © 2024 ACM. |
| Keywords: | design; user interfaces; accessibility; older adults; qualitative methods; field study; digital devices; computational linguistics; artifact or system; health - clinical; input techniques; interaction design; mobile devices: phones/tablets; prototyping/implementation; text/speech/language; user experience design; field studies; mobile device: phone/tablet; qualitative method |
| Journal Title: | Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies |
| Volume: | 8 |
| Issue: | 2 |
| ISSN: | 2474-9567 |
| Publisher: | Assoc Computing Machinery |
| Date Published: | 2024-05-01 |
| Start Page: | 72 |
| Language: | English |
| DOI: | 10.1145/3659624 |
| PROVIDER: | scopus |
| DOI/URL: | |
| Notes: | The MSK Cancer Center Support Grant (P30 CA008748) is acknowledged in the PDF -- Source: Scopus |
