Mixed-Reality AI Assistant Teaches Hands-On Skills

A wave of newly homebound people got the itch to knit during the COVID-19 pandemic, and in response, scores of knitters recorded how-to videos and posted them on YouTube.

"But videos don't give you feedback," said Dina El-Zanfaly(opens in new window), an assistant professor in Carnegie Mellon University's School of Design(opens in new window). She and Kris Kitani(opens in new window), associate research professor in the School of Computer Science's Robotics Institute(opens in new window), think artificial intelligence offers a better way.

El-Zanfaly and Kitani lead the development of Origami Sensei(opens in new window), an AI-powered mixed-reality system that assists beginners in creating origami. Along with graduate students from both SCS and School of Design, they recently presented their research at the Association for Computing Machinery Conference on Human Factors in Computing Systems(opens in new window) (CHI 2025) in Yokohama, Japan.

Origami Sensei employs a type of AI — computer vision — to monitor the beginner's work. The vision system is tied to a projector, which can project imagery directly on the workspace to identify a problem or demonstrate the next step.

"You don't just get feedback; you get personalized feedback," El-Zanfaly said.

The system might not be as good as a human tutor, but human tutors are a limited resource.

"If you have an instructor, they would typically teach five or 10 people," El-Zanfaly said. "Origami Sensei is like having your own tutor, one-on-one. You can do things at your own pace."

The system can't yet be paused or put in reverse, but she said those features could easily be added.

What they learned in developing the origami tutoring system applies to other domains — knitting, yes, but also sculpting and physical computing. The team plans to take on welding next.

 Learn more about Origami Sensei on the project's website(opens in new window).