Back to our Insights
Back to our Insights

Hand Tracking and Gesture Recognition: What Does It Mean for Health Care Professionals And Patients?


Charlotte Mallo

Doctors can integrate hand tracking and gesture recognition into Virtual Reality and Augmented Reality procedures, training, collaboration and patient engagement. 

In the last two months we have seen the market readiness level for the adoption of new technologies accelerate. Investments that would usually take 5 to 10 years to reach market saturation, have taken just weeks to emerge as a part of everyday workflows and habits. In 2020, we are keeping a particularly sharp eye on Virtual Reality and Augmented Reality in the healthcare industry, and what hand tracking and gesture recognition means for health care professionals and patients.

Case Planning, Consultation and Training 

Hospitals are using the Oculus Rift for case planning, consultation and training. Immersive Touch’s  IVSP (ImmersiveView Surgical Planning) platform generates 3D VR replicas from patient scan data, and enables surgeons to study, assess and plan operations. Doctors can collaborate with their surgical team, and better educate patients about procedures. The integration of hand tracking and gesture recognition would facilitate the use of surgical planning tools that currently use hand controllers to direct actions such as cut, opacity, draw, scan, overlay and measure. 

Companies and institutions such as Mayo Clinic, UCLA, Stanford School of Medicine, St Joseph’s Children’s Hospital, and Mount Sinai are using similar tools via Surgical Theatre for 360 medical experiences. Patients can engage with 360 walk-throughs of their anatomy, doctor’s plan surgery, and professors direct immersive medical training. Hand tracking and gesture recognition strengthens immersion, presence and engagement in Virtual Reality and Augmented Reality experiences.  

Augmenting Surgeons 

Taking this a step-further, Proprio focuses on precision in the operating room with their breakthrough technology. They use a multi-camera system to access the data they need such as pre-operating imaging, powerful magnification, and collaborative feedback. The technology was developed to reduce distractions and notifications from multiple interfaces in the operating room that draw attention away from the patient. Navigating intricate tasks in a high stakes environment requires the most accurate response. An intuitive system with hand tracking capabilities can increase the potential of this type of technology, and reduce the transfer of germs between physical dials, equipment and patients. 

Patient Rehabilitation 

Augmented Reality and Virtual Reality also play a key role in pain management and recovery. XR Health is one company that provides headsets as a platform for therapy that is far more powerful than traditional telehealth. Their medical virtual reality includes games, exercises and environments that are designed to replace traditional therapies in addition to providing advanced data, such as movement, that enables healthcare professionals to guide patient recovery from home, and without prescription drugs. With latency lower than human perception and customizable interfaces, hand tracking can easily integrate into virtual therapy experiences to enhance immersion and to increase the data that is available for analysts to provide better healthcare services. 


For some of us, working at home today also means merging the playroom with the office. The Merge Cube is intended for younger audiences and students to interact with virtual 3D objects. With Virtual Reality headsets, users can engage with hands-on learning in the sciences, view 3D models of the human anatomy, explore our biological systems and share their experience with friends. Hand tracking and gesture recognition can be integrated into any device with a camera to bring natural interactivity to fun learning experiences. 

Why Gesture Control Technology in AR and VR is a Key Feature

Augmented Reality and Virtual Reality will continue to help with visualization, annotation, collaboration, learning, treatment, safety and additional functionalities. Hand tracking and gesture recognition is a part of the journey to becoming familiar with these virtual worlds with seamless communication and navigation. Comfort in VR and AR means accessibility to larger audiences, including patients who may be in critical care. 

In just a few years, 5G will catalyse this new wave of VR and AR adoption in health care and many other verticals. Extensive bandwidth will power high quality, low latency and real-time collaborative virtual experiences. At the heart of this performance-oriented technology is basic interactivity and immersion. To learn more about our Augmented Reality and Virtual Reality solutions for hand tracking and gesture recognition, see Clay REALITY.



Bringing natural interaction to the virtual and augmented world.