Hand Gesture Recognition

Natural interaction in XR depends on the device understanding not just where your hands are, but what they are doing. For Magic Leap 2, I developed a real-time hand gesture recognition system that classifies discrete hand poses and dynamic gestures directly from the headset's hand tracking data — enabling gesture-driven control flows across XR applications.

The system was designed around the constraints of on-device inference: low latency, minimal compute overhead, and robustness to the natural variability in how different users perform the same gesture. Rather than relying on predefined thresholds or rule-based logic, the classifier is learned from data — capturing the statistical structure of each gesture across users and conditions. A short per-user calibration further sharpens accuracy without requiring large individual datasets.

Beyond standalone use, the gesture system fed directly into the ML2 Avatars pipeline — triggering expressive avatar reactions and state changes from natural hand input. It was also productized as a platform-level capability for Magic Leap 2, made available to other development teams building on the device.