Low Latency Visual Inertial Odometry With On-Sensor Accelerated Optical Flow for Resource-Constrained UAVs

Autonomous navigation is essential for modern unmanned aerial vehicles (UAVs), particularly in GPS-denied environments such as indoor spaces or disaster zones. A widely used approach for estimating drone motion is visual-inertial odometry (VIO), which fuses data from a camera and an inertial measurement unit (IMU) to continuously assess the drone’s position and orientation over time.

Conventional VIO systems depend significantly on computationally demanding tasks, especially optical flow (OF) estimation. This process entails tracking the movement of features, or reference points, between successive image frames. The computations involved typically necessitate the use of powerful processors or graphics units, which are impractical for small, battery-powered drones due to their size, weight, and energy consumption.

This research study tackles the challenge of delivering high-performance Visual-Inertial Odometry (VIO) for resource-constrained platforms, such as nano-drones, by presenting a system that offloads the optical flow computation directly to the camera.

The proposed system integrates a STMicroelectronics VD56G3 optical flow (OF) sensor for on-sensor OF vector computation and a TDK MPU6500 IMU for inertial measurements. A lightweight, quad-core embedded processor, Raspberry Pi Compute Module 4, handles the remaining VIO pipeline stages. This scalable design can be paired with various embedded processors and operates as a self-contained, battery-powered module.

The VD56G3 camera incorporates an application-specific integrated circuit (ASIC) for real-time, hardware-accelerated optical flow calculation. By embedding this functionality directly into the camera, the system avoids the need for computationally expensive feature tracking on the host processor.

The camera is integrated into a popular VIO framework called VINS-Mono to create an enhanced system termed OF VINS-Mono. By replacing VINS-Mono’s original feature tracking module with the sensor’s built-in OF data, it achieved substantial performance improvements. The OF VINS-Mono demonstrated a 49.4% reduction in latency and a 53.7% reduction in computational load compared to the original implementation.

Real-world tests in controlled environments confirmed that the improvement in efficiency was achieved without compromising accuracy. In fact, under specific parameter configurations and movement patterns, OF VINS-Mono outperformed the baseline system in terms of tracking precision.

The novel hardware-software co-design, combining a compact optical flow sensor, global shutter camera, and low-power ASIC, enables on-sensor computation at an impressive 300 frames per second. It exhibited reduced power consumption, drawing 14% less energy during operation. This is especially significant for drones with limited battery capacity, where power efficiency directly impacts flight time and mission duration.

To support the broader research community, the authors have also released a new dataset collected using this VIO setup. It includes synchronized camera, IMU, and optical flow data, as well as high-precision ground truth poses obtained from a VICON motion capture system.

This study demonstrates a compelling solution for enabling high-performance visual-inertial navigation on low-power platforms. Integrating on-sensor optical flow acceleration into the VIO pipeline can significantly reduce both latency and energy consumption while maintaining—or even improving—the tracking accuracy of future UAVs operating in GPS-denied environments.