Imagine the year 2030… The internet has evolved into what we now call as the Metaverse. In this new digital realm, you navigate through a highly sophisticated virtual world as advanced forms of avatars – known as digital humans. Healthcare professionals access 3D images of patients’ scans, tele-collaborate with multidisciplinary teams, use patients’ digital twins to discuss diagnosis and treatment plans, and document all their medical notes in the patients’ metaverse health records.

Sounds like a sci-fi movie, you say? Well, this reality may not be as far as you think…

What are Immersive Technologies?

Although we have an idea of what immersive technologies are, many of us use the terms Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR) and Extended Reality (XR), interchangeably. There are subtle differences among these realities, and if one understands the nuances, it can better facilitate better communication between healthcare professionals and technical developers.

In order to understand these terms, let’s consider the Reality-Virtuality (RV) continuum.^[1,2] At one end of this spectrum lies the Physical/Real World. In this reality, individuals interact with physical objects/items in a real-world environment. On the opposite end of the spectrum is VR, where users engage with virtual elements within a simulated or digital environment. There are two types of VR environments:

• An external virtual environment (EVE), where users wear VR headsets/devices and interact with virtual objects while immersed in a virtual environment. However, these devices limit the user’s immersion, since they are physical objects used in the real world.
• A “Matrix-like” virtual environment (MVE), which provides a higher level of immersion because it not only stimulates the users’ exteroceptive senses (e.g., sight, smell, taste, touch, and hearing), but also their interoceptive senses (e.g., proprioception). The MVE system is aware of, and can respond to changes in real-world environments.

Between the physical world and MVEs lies a spectrum that blends real-world and virtual elements, allowing users to perceive both simultaneously. This blended experience is known as MR, which comprises of AR and augmented virtuality (AV). In AR, virtual elements are overlaid in the physical environment, offering limited interaction with virtual objects. On the other hand, AV involves integrating real-world objects into a virtual environment, enabling users to interact with physical objects that appear in a virtual space.³

On the other hand, extended reality (XR) encompasses all these different forms of reality, excluding the physical world. It combines the real and virtual environments with human-machine interactions, facilitated by computer technologies and wearables. In XR systems, users typically engage with and perform tasks in real-and-virtual environments through various sensors and input devices. The data generated is captured and can be analyzed using artificial intelligence (AI) algorithms before being relayed back to the user through output devices.

How have Immersive Technologies been used in Healthcare?

Here, I shall list some of the interesting immersive technologies that I have come across in healthcare. This is not an exhaustive list, and is purely meant for educational purposes. It is not my intention to advocate any of these products or companies, and the innovations described in this section are not meant to replace professional medical advice or consultations with healthcare professionals.

(1) VR Rehabilitation:

VR rehabilitation is increasingly used as a means to enhance motivation, engagement and adherence to traditional movement-based therapies, particularly for home-based rehabilitation. This mode of therapy has shown some effectiveness in conditions such as stroke and cerebral palsy, as well as for pain control.^4-6 For example, randomized controlled studies have showed that patients undergoing VR interventions had clinical improvements in mobility.^7,8

(2) XR Telehealth:

Telehealth gained popularity during the COVID-19 pandemic, with companies exploring the combination of XR and telehealth to provide safe and immersive healthcare experiences. For example, XRHealth, a US-based company, has combined both modalities to form a VR Telehealth Virtual Clinic. The company’s physiotherapists and occupational therapists conduct telehealth consultations through VR technologies. Patients will get a personalized 8-to-12-week care plan, in which they use VR to do gamified exercises at their own home, while being connected to their XRHeath therapist through video calls and in-app messaging services.⁹

(3) Virtual Reality Exposure Therapy (VRET):

VRET is an approach that employs VR technology for graded exposure therapy. It has been used for the treatment of mental health disorders, including specific phobias,¹⁰ and post-traumatic stress disorder (PTSD).¹¹ VRET immerses individuals in a virtual environment, allowing them to confront their phobias or trauma stimuli in a controlled and safe manner. It has demonstrated effectiveness in treating conditions such as acrophobia, fear of flying, dental phobia, agoraphobia, and panic disorder, among others.¹⁰

(4) VR in Oncology:

VR has been used as a distraction technique to alleviate pain and anxiety in patients with cancer, such as during chemotherapy sessions and in palliative and supportive care.¹² Studies have shown that VR significantly reduces pain levels, particularly when using high-fidelity VR technology, as it helps in distraction by immersing patients more deeply into the virtual environments.¹³ In one of my early VR projects, we had also developed a prototype rehabilitation system for breast cancer patients for this purpose. We used the Microsoft Xbox Kinect to enable clinicians to monitor and review their patients when they did their exercises at home.¹⁴

(5) XR-based Surgery:

Recent reviews have categorized XR applications for surgery into those for: (i) preoperative and interventional planning, (ii) intraoperative and interventional guidance, and (iii) surgical and interventional training and education.^15,16 Benefits of using XR in surgery and surgical training include improved understanding of key anatomical sites, improved visualization of structural anomalies (which can be sometimes hidden from the surgeon’s view or missed on a CT scan), improved surgical planning and execution and improved depth perception of fine and complex structures, among others, thus allowing for increased surgical precision, decreased operative time and a shorter duration of hospital stay.^16,17

(6) AR-based Digital Operations

AR can also streamline healthcare operations. Companies like Augmedix use AR and natural language processing to develop digital scribes for clinicians, allowing them to document medical notes in real-time during patient consultations, thus cutting down on administrative workload.¹⁸ Another example is AccuVein, which uses projection AR technology to visualize patients’ vein vasculature, aiding in blood draws.¹⁹

(7) Digital Education in Healthcare:

VR technologies have been used to train medical students in soft skills such as empathy and health communication.^20,21 The Royal College of Surgeons in Ireland has also used VR simulations for clinician training on managing emergencies.²² Additionally, various institutions have employed XR for various forms of training, such as septic shock management, cardiopulmonary resuscitation, medication safety, and certain procedural skills.^23-26

Conclusion

The rapid growth of telehealth during the COVID-19 pandemic suggests that virtual care is on the rise.²⁷ Just like the industrial revolutions, healthcare has also undergone a paradigm shift – from Health 1.0 to Health 4.0, which is where we aim to be.

• Health Care 1.0 – Comprises of the basic patient-clinician encounter. Through consultation, testing and diagnosis, the clinician (or care team) provides prescription for medications and a care plan for treating a disease, as well as follow-up plans (e.g. order to get a lab test or an imaging test, referral to a specialist). This model has been prevalent in health care practice for hundreds of years.
• Health Care 2.0 – New medical equipment and devices have been invented, developed and tested, and are increasingly used in health care delivery. For instance, imaging test equipment (e.g. MRI, ultrasound, CT scan), monitoring devices (e.g. pulse oximeter, arterial lines), and surgical and life support equipment (e.g. da Vinci robot, chest tubes) are increasingly used in hospitals and other care settings to support diagnosis, treatment, and monitoring.
• Health Care 3.0 – Development & implementation of information systems, electronic health or medical records (EHR or EMR) to manage care of patients across units and departments of health care organizations. These health information technologies have had major impact on clinical and operational processes. In addition, using available computer networks, remote care and telehealth have become possible, and electronic visits are beginning to replace some face-to-face encounters.
• Health Care 4.0 – Emerging in parallel with Industry 4.0. In this context, the health care delivery process becomes a cyber-physical system equipped by Internet-of-Things (IoT), RFID (radio-frequency identification), wearables, and all kinds of medical devices, intelligent sensors, medical robots, etc., which are integrated with cloud computing, big data analysis, artificial intelligence, and decision support techniques to achieve smart and interconnected health care delivery. In such a system, not only the health care organizations and facilities (e.g. hospital, clinics, and long-term care facilities) are connected; but also all the equipment and devices, as well as the patients’ home and communities are linked together. In addition, through AI techniques, a pervasive, smart, and interconnected health care community can potentially emerge.

Traditional health care treatment and delivery has gone from simple medications to more complex treatments, but will go towards intelligent disease diagnoses and treatments in the future. Similarly, in the past, patient care was primarily delivered through interactions of a patient with a single clinician, and then expanded by including multiple clinicians, teams, and communities. Initially, patient care focused on one facility or discipline, and then expanded to include multiple disciplines, units, and organizations. Now, patient care involves large networks and communities of people and organizations. With small amounts of data being used at the beginning of the health care transformation, we now have big (and potentially giant) data streams with large variations in dimension, quality, format, and characteristics. And lastly, patients and caregivers increasingly involved, and share responsibilities for monitoring their health, reporting symptoms, and participating in shared decision making for treatment and care planning.

With the younger generations embracing digital life through social media, gaming, and the metaverse, it is expected that the healthcare industry will ultimately embrace immersive technologies as part of virtual care. Despite its challenges, as we move forward in the digital era, healthcare practitioners will need to be both passionate and interdisciplinary, possessing both clinical knowledge and technological skills to navigate the immersive technologies of this emerging digital frontier.

References:

1. Skarbez R, Smith M, Whitton MC. Revisiting Milgram and Kishino’s reality-virtuality continuum. Front Virtual Real. 2021;2:647997.
2. Milgram P, Kishino F. A taxonomy of mixed reality visual displays. IEICE Trans Inform Syst. 1997;E77-D(12):1321-1329.
3. Palma G, Perry S, Cignoni P. Augmented virtuality using touch-sensitive 3D-printed objects. Remote Sens. 2021;13(11):2186.
4. Lukacs MJ, Salim S, Katchabaw MJ, Yeung E, Walton DM. Virtual reality in physical rehabilitation: A narrative review and critical reflection. Phys Ther Rev. 2022;27(4):281-289.
5. Tokgoz P, Stampa S, Wahnert D, Vordemvenne T, Dockweiler C. Virtual reality in the rehabilitation of patients with injuries and diseases of upper extremities. Healthcare (Basel). 2022;10(6):1124.
6. Massetti T, da Silva TD, Crocetta TB, et al. The clinical utility of virtual reality in neurorehabilitation: A systematic review. J Cent Nerv Syst Dis. 2018;10:1179573518813541.
7. Hassett L, van den Berg M, Lindley RI, et al. Digitally enabled aged care and neurological rehabilitation to enhance outcomes with Activity and MObility UsiNg Technology (AMOUNT) in Australia: A randomised controlled trial. PLoS Med. 2020;17(2):e1003029.
8. Metin Okmen B, Dogan Aslan M, Nakipoglu Yuzer GF, Ozgirgin N. Effect of virtual reality therapy on functional development in children with cerebral palsy: A single-blind, prospective, randomized-controlled study. Turk J Phys Med Rehabil. 2019;65(4):371-378.
9. Lovett L. XRHealth launches VR telehealth clinic. Available at: https://www.mobihealthnews.com/news/north-america/xrhealth-launches-vr-telehealth-clinic. Updated 12 Feb 2020. Accessed 30 Sep 2023.
10. Emmelkamp PMG, Meyerbroker K. Virtual reality therapy in mental health. Annu Rev Clin Psychol. 2021;17:495-519.
11. Heo S, Park JH. Effects of virtual reality-based graded exposure therapy on PTSD symptoms: A systematic review and meta-analysis. Int J Environ Res Public Health. 2022;19(23):15911.
12. Buche H, Michel A, Blanc N. Use of virtual reality in oncology: From the state of the art to an integrative model. Front Virtual Real. 2022;3:894162.
13. Yap KY, Koh DWH, Lee VSJ, Wong LL. Use of virtual reality in the supportive care management of paediatric patients with cancer. Lancet Child Adolesc Health. 2020;4(12):899-908.
14. Golby C, Debattista K, Yap KY. ‘Kinecting’ the rehabilitation of breast cancer patients through digital technologies. Support Care Cancer 2013; 21(S1): 115.
15. Vervoorn MT, Wulfse M, Van Doormaal TPC, Ruurda JP, Van der Kaaij NP, De Heer LM. Mixed reality in modern surgical and interventional practice: Narrative review of the literature. JMIR Serious Games. 2023;11:e41297.
16. Zhang J, Lu V, Khanduja V. The impact of extended reality on surgery: A scoping review. Int Orthop. 2023;47(3):611-621.
17. Mao RQ, Lan L, Kay J, et al. Immersive virtual reality for surgical training: A systematic review. J Surg Res. 2021;268:40-58.
18. Zhou C. Augmedix uses AI and remote scribes to improve clinician documentation: Interview with founder Ian Shakil. Available at: https://www.medgadget.com/2020/03/augmedix-uses-ai-and-remote-scribes-to-improve-clinician-documentation-interview-with-founder-ian-shakil.html. Updated 11 Mar 2020. Accessed 2 Oct 2023.
19. Mesko B. Augmented reality in healthcare: 9 examples. Available at: https://medicalfuturist.com/augmented-reality-in-healthcare-will-be-revolutionary/. Updated 2 Nov 2021. Accessed 2 Oct 2023.
20. Roswell RO, Cogburn CD, Tocco J, et al. Cultivating empathy through virtual reality: Advancing conversations about racism, inequity, and climate in medicine. Acad Med. 2020;95(12):1882-1886.
21. Li YJ, Ducleroir C, Stollman TI, Wood E. Parkinson’s disease simulation in virtual reality for empathy training in medical education. In: IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Lisbon, Portugal: IEEE; 2021:56-59.
22. Royal College of Surgeons in Ireland. RCSI develops world’s first virtual reality medical training simulator. Available at: https://www.rcsi.com/dublin/news-and-events/news/news-article/2016/07/rcsi-develops-worlds-first-virtual-reality-medical-training-simulator. Updated 21 Jul 2016. Accessed 2 Oct 2023.
23. SingHealth Group. Where healthcare professionals get serious about gamification. Available at: https://www.sgh.com.sg/news/tomorrows-medicine/where-healthcare-professionals-get-serious-about-gamification. Updated 31 Mar 2022. Accessed 2 Oct 2023.
24. Tsang WY, Fan P, Raj SDH, et al. Development of a three-dimensional (3D) virtual reality apprenticeship program (VRx) for training of medication safety practices. Int J Dig Health. 2022;2(1):4.
25. Case Western Reserve University. CWRU achieves global first in mixed-reality education during COVID-19 pandemic. Available at: https://thedaily.case.edu/cwru-achieves-global-first-in-mixed-reality-education-during-covid-19-pandemic/. Updated 21 Sep 2020. Accessed 2 Oct 2023.
26. Microsoft News Center. Medical education goes Holographic with mixed reality from Microsoft. Available at: https://news.microsoft.com/en-sg/2022/01/11/medical-education-goes-holographic-with-mixed-reality-from-microsoft/. Updated 11 Jan 2022. Accessed 16 Jan 2023.
27. Bestsennyy O, Gilbert G, Harris A, Rost J. Telehealth: A quarter-trillion-dollar post-COVID-19 reality? Available at: https://www.mckinsey.com/industries/healthcare/our-insights/telehealth-a-quarter-trillion-dollar-post-covid-19-reality. Updated 9 Jul 2021. Accessed 4 Oct 2023.