What is the use of AR/VR in the medical field? Also explains “HoloLens”

table of contents

What is AR/VR in the first place?

First, let’s start with a basic explanation of what AR and VR are. AR, VR, MR, etc. all partially or fully merge the real world and the virtual world. These technologies are sometimes collectively referred to as “xR.”

Meaning of AR

AR stands for Augmented Reality. A typical example is the popular smartphone game “Pokémon GO,” which adds virtual information to real-world scenery. The basic mechanism is that when real-world scenery is projected through a smartphone camera, pre-prepared 3D images and text information are also displayed at the same time.

There are two ways to specify the display position. There is a type called location-based (location-based) that specifies the location based on location information acquired by a smartphone, and a type called image recognition-based (vision-based) that specifies the location based on a reference image. Divided. Image recognition types are further divided into “marker types,” which read markers such as QR codes, and “markerless types,” which recognize specified images or objects that actually exist.

Meaning of VR

VR is an abbreviation for Virtual Reality, which means experiencing a real-life experience in a virtual world. The displayed content includes 3DCG, such as anime characters and unreal spaces, and photos taken with a 360-degree camera and processed for VR display. In most cases, 3DCG is created in the development environment called Unity.

VR goggles are divided into monocular (single-lens) goggles and compound-lens (two-lens) goggles, depending on how the image is displayed. Compound eyes allow you to experience more realistic 3D vision, but there are viewing restrictions for children under 13 as it may have a negative impact on their growth. Monocular goggles can be viewed without any problem even by children.

Compound goggles are used in the medical field. The VR goggles used are products such as Oculus Rift and HTC Vive, and the lens part of the glasses serves as a display, and by displaying images separately for the right and left eyes, you can see in 3D. Outside of the medical field, its use is increasing in the entertainment field and, recently, in the promotional field. VR goggles are not only high-performance ones that cost around 100,000 yen, but also ones that use a smartphone as a lens, and inexpensive ones made of paper.

What is the use of AR/VR in the medical field?

The medical field is the second most advanced field in the use of AR and VR after the entertainment field such as games. One of the benefits of using AR/VR in the medical field is that it can reduce the burden on doctors by displaying information that previously had to be checked on a monitor or on paper, superimposed on the patient. In addition, since bones and organs cannot be removed from the body and seen in action, using realistic 3D images as AR/VR content in teaching materials will help promote understanding for medical students. Another benefit is that showing your own CT images to patients makes it easier for them to visualize the situation inside their own bodies.

Recently, the shortage of doctors in rural areas has become a serious problem, so it is expected to be used in the field of telemedicine.

Examples of AR/VR usage in the medical field

Now, let me introduce some specific examples of what fields AR/VR are actually being used in.

Use in surgery

AR/VR is used for surgical simulations and other purposes as a means of supporting doctors performing surgery. For example, by converting a patient’s CT data into a 3D polygon image that can be viewed through goggles, it becomes easier to visualize the positional relationships of organs that are difficult to understand with 2D images, and information sharing among multiple doctors becomes easier. I will.

Use in medical examination

AR is also used to share images with patients of what will happen after treatment. A photo of the affected area is taken with a smartphone or tablet, and an image after treatment is displayed overlaid with the actual affected area. Information that is difficult to convey verbally can be conveyed visually.

Use in treatment

Doctors can also wear AR glasses and examine patients while projecting patient information through the glasses. This prevents patients from becoming distracted by doctors looking at medical records or operating a PC during a consultation.

Use in learning

AR and VR are also used as learning materials to understand the human body. By superimposing images of bones and muscles that cannot normally be seen on the surface of an actual human being, it becomes possible to create a more concrete image. Educational materials are compatible with AR/VR, and are used for a variety of educational materials other than medical purposes.

MR like “HoloLens” is also attracting attention

MR is an abbreviation for Mixed Reality . It is a technology that superimposes and displays real information in a virtual world and is a further development of AR. Since AR is a system that displays information according to a specified position, the actual positional relationship is not calculated. Therefore, in some cases, the superimposed information may deviate from the actual scenery. MR calculates the positional relationship of reality and displays it three-dimensionally, allowing you to feel more integrated with reality.

A typical MR goggle is HoloLens, developed by Microsoft. This is a device that combines a dedicated computer capable of 3D display and a head-mounted display. Not only does it display 3D hologram images, but it can also be operated using gestures.

This HoloLens is also used in the medical field. For example, there are efforts to improve the success rate by recreating three-dimensional organs from hundreds of CT images taken for treatment and allowing doctors to simulate them before surgery. This type of simulation technology is highly anticipated because failure is not an option in surgeries involving organs such as the heart and brain, and it is difficult to actually test them.

When performing remote treatment using VR or MR, the key is the existence of communication without delay, such as 5G (5th generation mobile communication system). In order to send and receive high-definition images used in medical settings, a high-capacity, high-speed communication environment is essential. Commercial 5G services will begin in Japan in 2020, and as it becomes more widespread, telemedicine is expected to further increase.

summary

AR, VR, and MR are used for various purposes in the medical field. Utilizing new technology has the effect of enabling more advanced treatment and making it easier to share information. It is predicted that in the future, as the communications environment improves, it will be used even more than it is now.