With the large number of mental health apps that have surface since the COVID-19 pandemic, I had previously written an article on the quality of mental health apps and what are the considerations when developing and choosing such apps for managing one’s mental health and well-being. However, with more people working from home nowadays, many of them are also depending on Digital Voice Assistants (DVAs) to answer their health-related queries. As we transition into the post-pandemic world in the new year whereby most organizations are starting or in the midst of their digital transformation strategies, addressing this crucial health issue is now more important than ever. The question here is whether DVAs, such as Siri and Bixby (which are the most commonly available in Singapore), are credible, trustworthy and reliable enough to answer one’s queries and/or manage their mental health conditions (e.g. stress, anxiety and depression)?

Our group has published a journal article on the quality evaluation of DVAs for the management of mental health conditions. In this study, we evaluated 4 smartphone DVAs (Siri, Bixby, Google Assistant and Alexa) in terms of their ability to answer queries on general mental health and 4 mental health conditions (i.e. anxiety, depression, obsessive-compulsive disorder (OCD) and bipolar disorder).

How is quality of Digital Voice Assistants (DVAs) determined?

The definition of DVA quality in the study was “the degree of excellence to which a DVA could fulfill the needs of mental health-related queries“.

The quality of DVAs was assessed on 3 aspects based on 6 quality domains:

• The DVAs themselves
  • Comprehension ability (how accurately the DVA could recognize and transcribe the query)
• The DVAs’ responses
  • Relevance (how adequately the DVA could address the question)
  • Comprehensiveness (how detailed and complete is the response)
  • Accuracy (how accurate are the responses)
  • Understandability (how well can the layman understand the response)
• The answer sources provided by the DVAs
  • Reliability (how credible and trustworthy are the answers and sources)

A quality evaluation scoring rubric was developed and both the web and verbal responses of the DVAs were evaluated by 3 independent evaluators. The mental health questions were compiled into 5 categories based on general mental health and the 4 conditions. These questions were sub-categorized into those asking about the disease, symptoms and treatment.

The quality evaluation rubric is shown below.

What is the quality of DVAs for addressing mental health queries?

Google Assistant fared the best overall and it was able to provide relevant and accurate responses across all the mental health conditions. However, Bixby fared the worst in terms of responding to questions on general mental health and OCD, while Alexa fared the worst for anxiety, depression and bipolar disorder.

The study showed that even though all the DVAs were able to provide relevant and appropriate responses to the mental health-related queries, the understandability of their responses were low. Although all the DVAs performed well in terms of comprehension ability, Siri and Alexa performed more poorly than Google Assistant and Bixby due to the irrelevant responses provided. For comprehensiveness, Alexa performed the worst among the DVAs. It also scored significantly lower than Bixby and Google Assistant in terms of accuracy. Interestingly, the verbal responses of the DVAs were less easily understood by the layperson, and contained more jargon than web responses.

Which DVAs should be used for which mental health condition?

While Google Assistant generally works well across the evaluated mental health conditions, Siri and Bixby can also be used for depression and anxiety. On the other hand, Alexa and Bixby may potentially be used for OCD and bipolar disorder, respectively. A recommendation list of DVAs for the various mental health conditions are shown below.

We advocate that the information provided by DVAs should be interpreted with caution and in combination with other online information sourced from authoritative healthcare organizations both locally and internationally, such as the American Psychiatric Association, National Institute of Mental Health, World Health Organization, US Centers for Disease Control and Prevention, Anxiety and Depression Association of America, etc, just to name a few. In addition, it is always wise to seek the help and advice of a healthcare professional when managing one’s mental health condition(s). More importantly, information that may relate personally to the management of the patient’s mental well-being should be shared between the patient and caregiver (or someone whom the patient trusts) in a close and private setting that is comfortable for the patient.

For more details about our evaluation study, please access the full-text article here: AIMS Medical Science 2022; 9(4): 512-530.