Key takeaways
- Google has released an open-source machine learning pipeline for real-time hand gesture tracking and recognition.
- The new approach addresses the complexities of hand perception due to the flexibility and articulation of human hands.
- This technology aims to enhance interactions between humans and machines through robust gesture recognition.
- The hand perception functionality is integrated into MediaPipe, a cross-platform framework for processing perceptual data.
Behind the Architecture
- BlazePalm: BlazePalm is a single-shot detector model that operates on the full image and returns an oriented hand bounding box. To achieve this, the model follows some specific strategies such as training a palm detector instead of a hand detector, using an encoder-decoder feature extractor for bigger scene context-awareness even for small objects and lastly, minimising the focal loss during training. BlazePalm is mainly used to detect the occluded and self-occluded hands. So far, this method has achieved an average precision of 95.7% in palm detection.
- Hand Landmark Model: This model operates on the cropped image region defined by the palm detector and performs precise keypoint localisation of 21 3D hand-knuckle coordinates inside the detected hand regions via regression. In order to get robust hand poses, the researchers manually annotated approximately 30,000 real-world images with 21 3D co-ordinates. A high-quality synthetic hand model is also rendered over various backgrounds and map it to the corresponding 3D coordinates.
- Gesture Recogniser: This model classifies the previously computed keypoint configuration into a discrete set of gestures. The researchers used accumulated angles of joints to determine the state of the fingers and it is then mapped to a set of pre-defined gestures. This technique helped the researchers to estimate basic static gestures with reasonable quality.
How It Is Different
Application of This New Approach
The hand perception approach can be used in various cases and few of them are mentioned below- It can be enabled to the overlay of digital content and information on top of the physical world in augmented reality.
- Will help the differently-abled with sign language
- It can be applied in VR systems to manipulate virtual objects.
- It can also be utilised to control intelligent robots.




