Evaluating Smart Assistant Responses for Accuracy and Misinformation Regarding Human Papillomavirus Vaccination: Content Analysis Study

"More collaboration between technology companies and public health entities is necessary to improve the retrieval of accurate health information via smart assistants."

Despite studies affirming the positive effects of the human papillomavirus (HPV) vaccine and debunking inaccurate claims, there continues to be large-scale misinformation (mostly conducted online through social media platforms) surrounding this issue, driven in part by the antivaccination movement. The ways in which misinformation is disseminated online and across social media platforms influence vaccine refusal or vaccine delay. In this vein, voice assistants, a form of chatbot or conversational agent often referred to as "smart assistants", are devices that respond to human voices and can be used to acquire information - and, worryingly, perhaps misinformation - on a range of topics, including health-related queries. This study aimed to evaluate the levels of accuracy, misinformation, and sentiment in smart assistant responses to HPV-vaccination-related questions in the United States (US), where 46% of the population was found to have used a smart assistant of some kind in 2017.

The researchers systematically examined responses to questions from the chat-text service of Planned Parenthood about the HPV vaccine from the following 4 smart assistants: Apple Siri, Google Assistant, Amazon Alexa, and Microsoft Cortana. One team member posed 10 questions to each smart assistant, and 2 raters independently coded all responses. They then assessed differences among the smart assistants in terms of the following (all applied to the entirety of the smart assistant's response):

• Accuracy: a response that reflected the existing evidence, coding using a dichotomy approach (0 for not accurate; 1 for accurate);
• Misinformation: either deliberate or accidental promotion of previously disproved or unproven beliefs, attitudes, and behaviours - again coding this using a dichotomy approach; and
• Sentiment regarding the HPV vaccine: coded as one of the following 4 categories: negative (mostly negative statements), neutral (neither positive nor negative statements), positive (mostly positive statements), and ambiguous (both negative and positive statements).

A total of 103 responses were obtained from the 10 questions posed across the smart assistants. Google Assistant data were excluded owing to nonresponse (e.g., "I don't know how to answer that.") Over half (n=63, 61%) of the responses of the remaining 3 smart assistants were accurate. Siri yielded the greatest proportion of accurate responses (n=26, 72%), whereas Cortana yielded the lowest proportion of accurate responses (n=33, 54%). Most response sentiments across smart assistants were positive (n=65, 64%) or neutral (n=18, 18%), but Cortana's responses yielded the largest proportion of negative sentiment (n=7, 12%). Responses by Alexa contained no misinformation, whereas misinformation was present in 2 out of 36 (5.6%) responses by Siri and 16 out of 61 (26.2%) responses by Cortana. (An example of misinformation: The query, "Does Gardasil work?" generated this response (misinformation): "The Gardasil HPV vaccine has been proved to have caused the deaths of 32 women.")

Perhaps instructive in terms of these findings is the fact that, in 82 out of 103 (79.6%) responses, smart assistant responses contained content published by an organisation (e.g., Mayo Clinic or the Centers for Disease Control and Prevention), whereas 20 out of 103 (19.4%) responses were published by an individual (e.g., physicians or journalists). Cortana provided content published by individuals in 18 out of 61 (29.5%) responses, whereas Siri and Alexa provided content published by individuals in only 2 out of 36 (5.6%) and 0 out of 6 (0.0%) responses, respectively. Also, the type of content provided in the smart assistant responses varied, with videos provided in 30 out of 103 (29.1%) responses, driven entirely by Cortana, which was the only smart assistant to provide video responses.

Thus, on the whole, this study revealed that smart assistants mostly provide accurate and positive information regarding HPV vaccination (just under two-thirds of all smart assistant responses were accurate and positive), with no relevant differences across devices. However, the fact that just over one-third of responses contained inaccurate answers suggests that work is needed in this area to improve the provision of health information via smart assistants. In addition, misinformation is a concern, as one-quarter of Cortana's responses contained misinformation.

As indicated here, device manufacturers have, in some cases, provided platforms on which specialised topical information can be consolidated and further explored. For example, Alexa has an option for developers to create an application (app); such an "Alexa Skill" that specifically searches in predefined evidence-based sources when queried for information on HPV vaccination could be developed. In this light, the researchers suggest that manufacturers collaborate with researchers to further evaluate the accuracy of smart assistants as public health tools and determine together how to disseminate information, as well as what fact-checking assessments should be used for such information.

The researchers conclude that the "untapped potential of these devices for evidence-based information dissemination to consumers should be further explored....The downside to this potential approach for improving public health is that...there is risk of disseminating misinformation through smart assistant responses, potentially reducing the positive impacts of a health promotion intervention. More needs to be done to better understand the susceptibility of these devices and their respective skills to outside influences."