Virtual surgery: the healthcare potential of augmented reality 

One of the leaders in the field is the German company ApoQlar, which is developing a software tool named Virtual Surgery Intelligence (VSI). It uses artificial intelligence to render MRI and CT images in 3D. When a surgeon puts on the augmented/mixed reality headset, the 3D images merge virtually with the patient, giving the surgeon a new level of anatomical detail.  This scan can be controlled with gestures and speech commands.

“It displays medical images in a 3D hologram, which the user can freely manipulate round the physical environment,” says Jakub Wlasny, lead developer at ApoQlar. “Those images are automatically fixed onto the body of the patient. This means doctors can slice through those holograms in order to explore the patient’s internal structures, without having to look at any specific devices or screens.”

Alongside its surgical uses, the tool can be used preoperatively to prepare for upcoming surgeries, and post-operatively to evaluate the surgical progression. ApoQlar is also developing two related tools: VSI Patient Education (for informing patients about their condition) and VSI Education (for training surgeons). 

After extensive testing, VSI Surgery has now been introduced to a hospital in Hamburg, where surgeons in the head and neck department are using it as a postoperative and preoperative assistant. 

“We have found [the tool] gives us a much deeper and more detailed insight into the anatomical structures,” said Dr Hans von Lücken, senior physician. “But also for our interns the VSI is a big help. This way, they can orientate themselves more quickly and the more experienced doctor can better communicate their surgical approaches and procedures.”

Augmented reality technology involves ‘augmenting’ the real-world environment with various pieces of computer-generated input (sound, images or information), by way of a screen or headset. Although it is related to virtual reality (VR), there is one key difference: VR places the user in a completely immersive world, detached from reality, while AR overlays information onto the physical setting. 

Although AR is only beginning to make its way into the healthcare space, its potential is significant. 

From a medical standpoint, one key advantage is that practitioners don’t need to take their focus off the patient. Currently, surgeons need to switch between various reference points – from the person on the operating table to the medical images and patient data displayed on various screens. With applications such as VSI, they can access all the information they need while the patient remains in their field of vision.

“We want to deliver a more friendly and immersive way of working with medical images, helping doctors navigate around the surgical site and orientate those anatomical structures. This makes it quicker for them to proceed with surgeries,” says Wlasny. “We also want to eliminate as many of the other tools that are now used as we can, and put everything in one simple device.”

As many other technologies in this field, the VSI uses Microsoft’s headset, HoloLens. This device, which creates blended environments based on mixed reality, enables a live broadcast of the surgery from the surgeon’s perspective, meaning less experienced doctors could receive remote assistance.

Having already been used in many industrial settings, the HoloLens is just now beginning to make waves in surgery. In December, a team of French surgeons used HoloLens to livestream what they called the “world’s first surgical intervention performed with a mixed reality collaborative platform”. And in January, a team at Imperial College London demonstrated how surgeons can use HoloLens headsets while conducting reconstructive lower limb surgery.

"The application of AR technology in the operating theatre has some really exciting possibilities," said Jon Simmons, who led the Imperial College team. "It could help to simplify and improve the accuracy of some elements of reconstructive procedures.

"While the technology can't replace the skill and experience of the clinical team, it could potentially help to reduce the time a patient spends under anaesthetic and reduce the margin for error. We hope that it will allow us to provide more tailored surgical solutions for individual patients."

At the time of writing, ApoQlar is waiting to receive medical certification for VSI Surgery, which will enable surgeons to use the tool in the operating room. The company is also working to develop it further, adding new functionalities and areas of application.   

Since the tool uses machine-learning algorithms, it will continuously learn and improve, recognising more tissue types and acquiring more knowledge with every operation. 

“I hope that in the long-term it’ll be a full medical product presented in many different medical facilities, and we hope to deliver as many useful functionalities for different kinds of specialties as possible,” says Wlasny. 

It’s early days still, but in a few years’ time AR-enhanced surgery could well become the norm. According to a recent review of the subject, applications in this space are developing rapidly (albeit with various teething problems that need addressing). The authors concluded that, in the future, AR “will likely serve as an advanced human-computer interface, working in symbiosis with surgeons, allowing them to achieve even better results.”

From ApoQlar’s standpoint, there is no doubt that augmented and mixed reality will revolutionise current surgical options. 

As the company’s founder Sirko Pelzl recently told the Hamburg News: “It’s not a question of whether this will happen, but whether we will lead this revolution.”