Efficient Egocentric Action Recognition with Multimodal Data

Recognising what a person is doing from their own point of view — egocentric action recognition — is a core building block for context-aware XR systems. The challenge on wearable devices is not just accuracy: it is accuracy under the hard constraints of portability, battery life, and real-time performance on limited hardware.

This work systematically analyses how sampling frequency across different input modalities affects both recognition performance and CPU usage. Two modalities are studied: RGB video from the headset camera and 3D hand pose from the device's hand tracking system. Rather than assuming more data is always better, the study characterises the full trade-off curve — measuring what is lost in accuracy and what is gained in efficiency as frame rates are varied for each modality independently.

The key finding is that RGB frames can be sampled at significantly lower rates when complemented with higher-frequency 3D hand pose input, preserving recognition accuracy while cutting CPU usage by up to 3× with minimal performance loss. This points to multimodal input design — not just model compression — as a practical route to real-time egocentric action recognition on XR devices.

Presented as an extended abstract at the Second Joint Egocentric Vision Workshop (EgoVis) at CVPR 2025.