Image: Hanne Kristine Fjellheim/Sopra Steria

Using Microsoft's HoloLens platform, researchers in Oslo have developed a way of turning traditional two-dimensional medical images into 3D augmented-reality models for planning surgery and navigating around organs during operations.

The project by researchers at the Intervention Centre at Oslo University Hospital, working with developers at IT consultancy Sopra Steria, was recently awarded a Microsoft Health Innovation Award.

The data that the 3D models use comes from the hospital's various image-generating scanning CT and MR machines. These scanners provide detailed views of the human body, but present these images in the form of two-dimensional picture 'slices'.

When planning surgical procedures, the surgeons have to flip back and forth through a potentially large number of these slices, when using them directly from the scanning machines.

The Intervention Centre has been working on combining data from such 2D slices into a 3D model for more than a decade. Using a method called picture segmentation, the relevant sections are extracted from the scans and used to form a 3D model of, for instance, a liver with a tumor and its surrounding blood vessels.

Until now, these models have been presented to the surgeons in two dimensions on computer monitors, but now they have been prepared for visualization and interaction by surgeons wearing the HoloLens.

The work so far has been focused on working with liver cancer patients, as well as with children and adults with heart conditions.

When treating a liver cancer, one of the results of the new technique is that more of the healthy liver tissue can be left behind, which should increase the patients' ability to withstand more surgery in the course of their treatment, according to Bjørn Edwin, surgeon at Oslo University Hospital.

However, the researchers are already looking at more applications for this technology. They are considering ways it could be used in planning and supporting replacement heart valve operations and in keyhole surgery.

In that role it could prove very useful in providing the surgeon with a larger virtual picture of the surrounding tissue when operating. The challenge is to find a way to update the imagery during the procedure.

The Intervention Centre at Oslo University Hospital is a research facility that is part of the general surgery department. It has a multi-disciplinary staff from several of the faculties at the University of Oslo and the Norwegian University of Science and Technology, NTNU, among them the Faculty of Medicine, the Department of Informatics, Department of Medical Physics, Division of Biophysics and Medical Technology and Department of Electronics and Telecommunication.

The center's role is to develop new procedures and treatments, compare new and existing strategies, improve advanced imaging techniques, study the social, economic, and organizational consequences of new procedures, and administer radiation protection for all departments.

Norwegian surgeons get 3D planning accuracy with Microsoft's VR platform Watch Now

Read more about HoloLens medical technology