University of Cambridge researchers have developed a virtual reality application where 3D modelling tools can be controlled using the user’s hand movement.
The team used Machine Learning to develop the 3D virtual reality modelling tool HotGestures. The technology is analogous to the hot keys used in many desktop applications.
The new tool allows users to build figures and shapes in virtual reality without interacting with a menu. This helps users stay focused on a specific task.
This is the first time that this ability has been made available.
The results, ‘HotGestures: Complementing Command Selection and Use with Delimiter-Free Gesture-Based Shortcuts in Virtual Reality,’ are reported in IEEE Transactions on Visualization and Computer Graphics.
Innovating virtual reality
Virtual reality has been regarded as game-changing technology for many years, but outside of gaming, its promise has not fully materialised.
Professor Per Ola Kristensson from Cambridge’s Department of Engineering and research leader, said: “Users gain some qualities when using VR, but very few people want to use it for an extended period of time.
“Beyond the visual fatigue and ergonomic issues, VR isn’t really offering anything you can’t get in the real world.”
Desktop software users will be familiar with hot keys – command shortcuts such as ctrl-c to copy and ctrl-v to paste. Although these shortcuts mean that the user does not need to open a menu to find the right tool, the user must memorise the correct command.
“We wanted to take the concept of hot keys and turn it into something more meaningful for virtual reality – something that wouldn’t rely on the user having a shortcut in their head already,” said Kristensson, who is also Co-Director of the Centre for Human-Inspired Artificial Intelligence.
About HotGestures
Inspired by hot keys, Kristensson and his team developed HotGestures. This tool allows users to gesture with their hand to open and control the tool they need in 3D virtual reality environments.
For example, a cutting motion opens the scissor tool, and a spray motion opens the spray can tool. The user does not need to open a menu to find the required tool, or remember a specific shortcut.
Users can switch between tools by performing different gestures during a task. The new virtual reality tool means that work does not have to be paused to browse a menu.
“We all communicate using our hands in the real world, so it made sense to extend this form of communication to the virtual world,” said Kristensson.
How was the VR tool built?
The researchers built a neural network gesture recognition system that can recognise gestures by performing predictions on an incoming hand joint data system.
The system recognises ten different gestures associated with 3D models. These are pen, cube, cylinder, sphere, palette, spray, cut, scale, duplicate, and delete.
The team conducted two studies where users employed HotGestures, menu commands, or a combination.
The new virtual reality tool provided fast and effective tool selection and usage shortcuts. The study found that HotGestures is fast, distinctive, and easy to use. The tool was also shown to complement conventional menu-based interaction.
The system was designed so that there were no false activations – correctly recognising what was a command and what was a normal hand movement.
The study found that the gesture-based system was faster than a menu-based system overall.
The source code and dataset are publicly available
The researchers made the source code and dataset publicly available so that VR tool designers could incorporate it into their products.
“There is no VR system currently available that can do this,” said Kristensson.
“If using VR is just like using a keyboard and a mouse, then what’s the point of using it? It needs to give you almost superhuman powers that you can’t get elsewhere.
“We want this to be a standard way of interacting with VR.
“We’ve had the tired old metaphor of the filing cabinet for decades. We need new ways of interacting with technology, and we think this is a step in that direction. When done right, VR can be like magic.”
