HCII Researchers Ensure AR, VR Technologies Work for Everyone

When Atieh Taheri(opens in new window) first donned a virtual reality headset and started walking around a virtual world, her experience didn't match what she had imagined. Walking felt uncomfortable.

Taheri, a postdoctoral fellow at Carnegie Mellon University's Human-Computer Interaction Institute(opens in new window) (HCII), has never walked because of spinal muscular atrophy. When she first walked in VR, the technology simulated her head bobbing up and down. Such movement is natural when walking and likely seemed subtle to the designers and other users, but it was disorienting at the onset for Taheri. It felt like riding a horse, she said, and she knew it was something she wanted to customize.

"To me, it felt like people with motor disabilities weren't considered when designing VR systems," Taheri said.

Taheri and her collaborators researched how walking in VR can be expanded and customized(opens in new window) to improve and adjust the experience for people who might not be familiar with the sensation of this kind of forward locomotion.

Virtual spaces are increasingly becoming places where people interact — whether for work, play or education — and CMU researchers are studying how to improve these spaces and make them more accessible for everyone. They said that as AR and VR technologies become more ubiquitous, developers should address and incorporate accessibility features. Do this early, they said, before it becomes onerous.

"On the smartphone, accessibility felt like an afterthought. Smartphone software was developed, and then we added accessibility features. And right now, there's really cool work figuring out which features are useful," said David Lindlbauer(opens in new window), an HCII assistant professor who co-leads CMU's Extended Reality Technology Center(opens in new window). "Accessibility still feels like an afterthought for a lot of VR and, especially, AR experiences. We're not quite there yet in terms of commercial ubiquity. Not everybody has a VR or AR headset, so we can take the research into account early and bake it into the underlying systems without having to go around the barriers that we ourselves introduced."

Patrick Carrington's(opens in new window) interest in accessibility research started in web development, when a project he was working on was delayed because accessibility was an afterthought. An assistant professor in the HCII, Carrington studies wearable technology and builds assistive tech, such as a browser extension that made images accessible on X (formerly Twitter). His interest in accessibility started with a simple thought: There has to be a better way to make intuitive and accessible systems.

While immersive VR systems have had significant commercial advancements, Carrington said accessibility for people with disabilities is still an issue.

"There's been some work about access for people with low or no vision and audio alternatives in virtual spaces, but physical interactions have been largely left out of accessibility considerations in VR and AR until very recently," Carrington said. "Accessibility for people who may not be able to perform all the physical actions has been left out of the experience. But it's early stages for AR and VR. It's not too late to correct some of these things in the rollout versus trying to go back and fix them later."

Carrington's lab recently published work addressing a gap in tracking bodies in motion. The research, WheelPoser(opens in new window), arose from the increased popularity of fitness trackers. For people who use wheelchairs, there's no method to capture body movement — either for fitness or in VR.

"To properly perform detailed motion capture, you need to either have a giant studio with a bunch of cameras and wear a special suit, or wear 17 or so sensors on your body that you'd have to strap on and calibrate," Carrington said. "WheelPoser examined how we do that much more practically, how we lower the number of sensors you have to wear to make this motion capture more practical for everyday life."

The research team created a novel tracking system that used four sensors: one on each of the person's forearms, one on their forehead and one on the wheelchair. This work is ongoing.

Carrington's Axle Lab(opens in new window) is also watching closely as companies transition to headsets and other systems that move away from hand-operated controllers. Instead, they rely on cameras that detect fine-grain hand gestures, such as tapping your thumb and forefinger together. But for people with motor impairments, these movements could be difficult or even impossible. Carrington's lab studied how people with motor impairments could control a virtual hand.

"We did a study looking at how people using those input devices would want to make hand gestures and motions happen. We're identifying the taxonomy of the mappings of how these inputs work and what they might translate to. We hope we can abstract it now before all the controllers disappear and we have to remap all the hand gestures from scratch," Carrington said.

Taheri's interest in researching and working in accessibility started when she lost the fine-grain mobility in her hands and couldn't use video game controllers anymore. Similar to the work Carrington is doing now, Taheri's first published research(opens in new window) looked at how to design a hands-free video game controller using real-time facial-expression recognition.

VR and AR technologies aren't just used in gaming, and the focus on accessibility at the outset could mean more opportunities for people with a variety of impairments. Lindlbauer said as AR and VR technologies have advanced, the applications of these tools have also expanded to things like skill training and healthcare. Firefighters can now be trained with virtual reality, and augmented reality is used for remote surgical assistance. Virtual reality also facilitates new experiences for people with limited mobility.

"For example, people would like to travel more and experience things," Lindlbauer said. "For people who have visual impairments, virtual worlds can account for things like visual deficiencies and do things such as change contrasts to make things easier to perceive."

Taheri said that society creates obstacles for people with disabilities. She works to find ways to empower people with disabilities to engage with the digital world on their terms.

"Society is what truly disables people," she said. "If I build something for the disabled community, my goal is to make it accessible for them. But accessibility is not limited to only one group — it's something that can benefit everyone at some point in their life. And disability is not a sad thing; it's a reality that anyone can experience at any time."