We're committed to improving surgical planning and patient outcomes. Our mixed reality (MR) technology adds a new dimension to medical images: turning 2D CT and MRI scans into 3D interactive holographic simulations. By equipping surgical teams with these virtual tools, we are giving them the power of preparation, and helping usher in an era of smarter, safer surgical planning.
Vital to this is our team of designers and engineers, who design impactful software with the end-users (surgeons) in mind. Our Senior UX designer Jarl Erik Cedergren – whose background spans interaction design and user experience with virtual reality solutions – leads on our user-centred design approach to software development. His team works closely with clinicians to understand their needs, and design around their existing workflows, in order to solve real clinical problems.
While mixed and virtual reality technologies are nothing new – both have seen much hype in the worlds of gaming and entertainment – they are not yet commonplace. As a result, these innovations are only just beginning to filter into, and impact, the healthcare space. With people like Jarl Erik leading the way, we are building a new surgical future where MR finally takes a front seat.
The definition of mixed reality today goes all the way back to the 1990s, when people were first exploring and experimenting with the technology. Even at its earliest stages, people recognised the potential impact within medicine and healthcare – but the technology wasn’t ready; it didn’t yet have the power or visualisation-capacity to provide anything of value.
Today, these technologies have matured. Increased adoption of VR and MR in immersive games and entertainment has accelerated progress, with both now as valuable to a game developer as they are to a clinician. Now more commonplace, we have increasing numbers of people who are comfortable and confident using these tools, both in and out of the workplace.
The way that people interact with technology also continues to evolve. Until recently, people could only use computers by clicking a keyboard and mouse but, just ten years ago, touchscreens became relevant – now, using your fingers to interact with information directly on a screen is second nature. MR takes this one step further, bringing the information (or models) out of the screen and into the physical world, where you can touch and interact with it directly. It might feel unusual now, but in ten years time, we will take this level of interaction for granted.
With our mixed and augmented reality (AR) technology, clinicians can visualise patient-specific organs as 3D holograms, and interact with them in real time and space, from any point-of view, rather than being locked to a flat 2D screen. Surgeons say that our 3D models give a more realistic anatomical view of what they see in a patient, during a surgical procedure – enabling them to plan surgeries in a smarter, safer way. This was simply not possible ten years ago.
‘The Metaverse’ commands very differing opinions and, therefore, it is actually very difficult to define. For Jarl Erik, it is simply about bringing 2D information that is currently consumed via screens into the physical world – so that you are able to “jump into the information” and interact with it in a whole new way. Ultimately, the goal is that all this information surrounds you, at all times. Mixed and augmented reality plays a key role in the development of the Metaverse, bringing together the digital world with the physical, to create a new version of reality. Jarl Erik thinks that this future is not far away: “I think, in 20 or 30 years time, if you tell people that everyone used to have a flat device in their pocket, with their whole life inside it, people will just laugh at you”.
For now, that future might seem pretty far off, but in healthcare we are getting closer. For example, our software takes information from a two dimensional CT image, pulls that information out and turns it into three dimensions, and then “places” the 3D image in the real world where people are able to interact with it, together, regardless of where they are physically located.
For surgical teams, MR tools will be impactful both before surgery – by enabling smarter preoperative planning – and during surgery, where it can give clinicians access to patient-specific data and information, in the blink of an eye. It will also be valuable in medical training for students – offering new surgeons the opportunity to experience and practise complex surgeries and emergency situations, in virtual reality, before even stepping foot into the operating theatre.
Many healthtech tools are used in healthcare and surgery; it is critical that all are fit for clinical use, and that clinicians are trained on anything they may use during their procedures.Without adequate training, mistakes can and will happen. New surgical technologies must be designed with the surgeon and the intended use in mind to ensure they a) provide value to the clinician and patient outcomes and b) are easy to learn, with a seamless user experience. This is where UX design comes in…
UX-centred design has rapidly gained traction in software development. People used to design things that worked, rather than were enjoyable to use, which explains why older software can feel clunky and complex, requiring complex training before use. This is still common in many industries, including the healthcare sector, which is often still reliant on old legacy systems and software. A challenge is how to improve systems through a design-led process, to create something that is designed specifically for the user.
As a UX designer, a lot of Jarl Erik’s time is spent observing, in our case, surgeons to find out how they work day to day, identify pain points, and discover what could be improved and how we can address this using a device or technology. With this intel, solutions are optimised to the specific needs and workflow of the surgical team, to ensure that when it comes to using new hardware like the HoloLens headset or new software in the HoloCare suite, there is no friction, or resistance, to use. As Jarl Erik explains, this can be a challenge: “When we put the HoloLens on someone, we ask them to test it out, or test our application. If they struggle, we don't know if they struggle with the mixed reality interaction with the mixed reality device, or if they struggle with the software application itself, because nobody is used to this type of device. It’s like testing a smart phone app with someone who has never used a touch screen before. That's why it's so important to be really detailed in these kinds of workflows, to make sure that the design is good and the end-user experience is good."
Mixed reality is a vital, engaging field with so much novel innovation, and we are seeing a definite shift in the appetite for it in planning, surgery, training and other healthcare settings. As the technology continues to advance, and more clinicians explore these solutions, it is clear that mixed reality-led surgical innovation offers something new and valuable to healthcare professionals.
With user-centred design at our core, we are passionate about bringing our mixed reality technology into hospitals globally. Whether you are just starting out your career, or you’ve been a surgeon for 50+ years, everyone should feel able to use and master this new surgical reality. Within healthcare this is vital – we want to empower clinicians with technology, so they can work more optimally, ultimately saving time, money and lives.
