A medical diagnosis can be one of the most terrifying and confusing days of a patient’s life. Doctors often struggle to fully explain a person’s illness to them because internal structures, like a tumor, can be hard to visualize.

Soon, doctors could have a new tool that helps remove some of the confusion from a diagnosis: virtual and augmented reality.

University of Washington startup Pear Med is taking advantage of that tech with VR and AR software that compiles medical scans into an interactive 3D model. The startup is currently negotiating a partnership with Seattle Children’s Hospital to further develop their technology, which could revolutionize the way doctors communicate with their patients and help surgeons prepare for complex procedures.

One of our biggest challenges is being able to separate ourselves from that hype and show this can actually provide long-term value to doctors and patients.

Co-founders Ryan James and Mark Laughery started Pear Med during their UW software entrepreneurship class in winter of 2016 and have since moved the startup to CoMotion Labs, the university’s virtual reality and augmented reality incubator.

The two combined James’ health informatics background with Laughery’s design experience to make medical data easier to visualize for patients, doctors and medical students.

“We’re still trying to find our niche, so that’s one of the reasons we’re so excited about this Seattle Children’s partnership because they really want to help us understand exactly how this can be used in a hospital,” James said.

Pear Med’s software, called Bosc, stitches together 2D images from a patient’s MRI or CT scan to create a virtual 3D model unique to their anatomy. Using the AR HoloLens or VR Oculus Rift, users can see a color-coded representation of a patient’s bones, organs, and nerves. The software is especially useful to help clinicians visualize the shape and structure of a growth or tumor.

Although various medical institutions have expressed interest in Bosc, implementing this technology into a doctor’s daily clinical workflow will be no small undertaking. VR and AR headsets are still in their early stages of development and are prohibitively expensive for some institutions to use on a large scale.

“I think there’s a lot of hype around VR and AR, so I think one of our biggest challenges is being able to separate ourselves from that hype and show this can actually provide long-term value to doctors and patients,” Laughery said.

The UW Center for Multidimensional Medicine (CMM) has provided them with the clinical connections to rise to that challenge. The UC Irvine Department of Urology was the first institution to partner with Pear Med to see how the VR software could improve surgical training and planning for procedures such as kidney stone removal.

In November of last year, the CMM connected Pear Med with a very unusual case at British Columbia Children’s Hospital. In a 1-year-old patient born with only one kidney, radiologists discovered a non-cancerous growth called a lipoma surrounding the base of her spinal cord.

Administrators at the CMM and NIH thought the Bosc technology might help the parents understand their daughter’s illness and aid her surgeon in planning for a partial removal. The growth was intricately intertwined around her nerves, making it difficult to operate on without causing further neurological damage.

James and Laughery drove to BC Children’s last November to show the family and their surgical team a virtual model of the child’s lipoma. Interacting with a 3D model of the growth helped her parents understand the problem much better than flat MRI images, James said.

James said the child’s father described himself as “a pit bull dad,” who would “do whatever he has to do to understand the problem and help in any way he can.”

We open sourced the Pear Interaction Engine on GitHub as our kind of thank you to everyone in the software community. Over the past year, much of Pear Med’s funding has come directly from James’ and Laughery’s own pockets, so the Seattle Children’s partnership will be an important source of new capital. In the past, they have received grants and awards for their work, such as $2,500 for an innovation award from the UW Business Plan Competition and $10,000 in hardware donated by UW Medicine.

Since James is still a Ph.D. student and therefore technically a UW employee, UW partially owns the intellectual property of all Bosc code. Any licensing agreement with Seattle Children’s will involve royalty payments to both Pear Med and UW. Once James finishes his doctorate, Pear Med will have exclusive ownership over the intellectual property of any new code they write for the software.

James and Laughery relied heavily on open source software, such as NodeJS, to develop Bosc, so they felt compelled to share what they’d learned over the last year for free on a popular code sharing website called GitHub.

“We open sourced the Pear Interaction Engine on GitHub as our kind of thank you to everyone in the software community,” James said. “We’ve learned a lot from the physicians that we’ve worked with about how they interact in VR and AR, the bad ways to do it and the ways that really help them.”

It could take years before Pear Med is able to fully integrate their technology into a hospital’s daily workflow, but James and Laughery believe Seattle Children’s will be the perfect environment to test their products.