When Maria Chiu began her Ph.D. at Northeastern University in 2023, she never expected to become her own research subject. She also didn’t anticipate undergoing her fourth knee surgery.
That otherwise minor procedure ultimately led Chiu, a competitive snowboarder and figure skater studying interdisciplinary media as a doctoral student, on an 11-month rehabilitation journey that became the focus of her research. Over the course of that time, she crafted an experimental program to see how using immersive extended reality technology, like virtual reality and motion capture, could help her recover and get back on the ice and snow.
“There were a lot of days that were one step forward, two steps back,” Chiu said. “The technologies that I used, they were a source of comfort for those days, and they kept me going so that I never lost too much progress, whether physically or mentally.”
Chiu wasn’t alone. She had a team of doctors, trainers and physical therapists, but also her fellow researchers at the Orthopedic Augmentation, Surgical Innovation and Interactive Systems (OASIIS) Lab, a partnership between Northeastern researchers and orthopedic surgeons at Massachusetts General Hospital (MGH).
Led by MGH orthopedic surgeons Christopher Bono and Daniel Tolbert and Northeastern game design professor Casper Harteveld, the OASIIS Lab is unlike most research units at a hospital. It’s a collaboration between game designers and medical professionals to explore how to make recovery not only more effective but also more engaging.
“This is innovation like I never really visualized,” Bono said. “When I first got into orthopedics, I always thought I was going to be developing screws and implants and instruments.”
The intersection of health and immersive technologies is still fresh terrain. But growing public interest in such technology, particularly wearable devices like smartwatches that monitor everything from heart rate to sleep patterns, has helped medical professionals recognize that the technology “can be very meaningful,” Harteveld said.
“You’re not just waiting at home for your next physical therapy exercise,” Harteveld said. “You feel like you’re actively participating in the process and trying to do the best you can to recover.”
The tools being developed by OASIIS researchers are each designed for a different purpose and a different place in a patient’s recovery.
For instance, one VR project, FUNXION (pronounced ‘function’), tracks leg movements during rehabilitation using a system of sensors attached to the lower body. It allows medical professionals to take real-time measurements of a patient’s movements and provide immediate feedback and tips for how to improve through the app, said Nathan Miner, a research scientist at the OASIIS Lab who started in the lab when he was a Northeastern Ph.D. researcher.
An area in which the OASIIS Lab is making huge strides is diagnosis.
Currently, a patient will go into a clinic and a doctor will ask them to do a range of movements and basic activities, from opening a jar to cutting a loaf of bread, Bono said. Based on that assessment as well as responses to a written questionnaire, the doctor will then recommend a treatment plan.
One of the first projects the OASIIS Lab worked on, X-DASH, created a VR simulation of those assessments. Patients recruited from MGH’s orthopedics clinic would wear a VR headset and sensors on their body, and the app would provide objective data to doctors about their patients’ movements, rather than subjective data supplied by a questionnaire.
The project also highlighted why bringing game designers into health care is valuable, Miner said. If a tool, exercise technique or assessment questionnaire is unintuitive, cumbersome or boring, patients won’t continue to use it. That’s consequential in physical therapy and recovery, where keeping patients engaged in a routine long term is the key to success.
“We approach all our work with the sense that if you’re going to do it, it might as well look really good and feel really good to engage with,” Miner said. “With X-DASH, we had 80-year-old participants excited about the system and saying they would prefer to use that over paper and pencil.”
OASIIS researchers are quick to point out that XR technologies won’t replace human doctors, trainers and physical therapists. But their work highlights how these emerging technologies can complement the work being done by medical professionals in important ways.
Beyond the physical aspects of recovery, researchers at the lab are also trying to enhance patients’ mental health, a “huge gap” in the recovery process, Chiu said. Chiu’s knee injury restrained her movement and left her unable to move and compete in the same way she had before the injury. For an athlete like Chiu, she felt divorced from a key part of her identity, she said.
“For someone whose identity is, in a lot of ways, tied to their ability to move, their ability to do their sport and to compete at a high level, when you lose all of that, the mental frustration, anguish, is something that is not talked about a whole lot in my experience,” Chiu said.
Using VR fitness games in tandem with motion sensor kits, Chiu developed an extended reality recovery program that ultimately proved more beneficial for her mental health. The technology allowed her to track her movements and see how close she was to returning to a pre-injury level of movement capability.
Eventually, though, these tools weren’t pushing her physical limits like a human physical trainer would have. But by giving her more data on her progress and more of a sense of accomplishment, the tools completely changed her relationship with her medical team.
In the past, she felt like she was on the outside looking in on her own recovery, she said. Now, she could share information she got from her time with the program she had created with her physical therapists. Using that data, she could ask more informed questions of them and understand the reasoning behind the steps in her recovery.
“Now I felt like I was a part of the conversation,” Chiu said. She is now adjusting her own program to test it with groups of patients.
Bono’s vision for the OASIIS Lab is ambitious. “You might feel like you have knee pain or hip pain and [based on data from] the sensors after the module watched you walk, you have now been screened for Parkinson’s,” Bono said. “Your gait is suspicious for a neurological condition that would otherwise have never been detected because all this stuff in the background is still being monitored.”
The day when that kind of monitoring is a regular part of clinical practice might still be far away, but “the pieces to assemble that dream are actually here,” Miner said.
Key medical concepts
The content is provided for information purposes only.