IMU optical frame transform: verify depth camera IMU frames

A depth camera with a built-in IMU publishes two extra coordinate frames alongside its image frames: one for the accelerometer and one for the gyroscope. If the fixed transform between those frames and the color optical frame is wrong, every downstream consumer that fuses inertial data with images inherits the error.

The symptoms are subtle: a fused pose that drifts sideways, a gravity vector that points 90 degrees off, an odometry estimate that rolls when the camera yaws. This article shows how to verify the published IMU frames against the physical camera before you trust any fusion result, using static and rotation tests you can run on a desk in half an hour. It applies to any depth or ToF camera that exposes IMU streams; the geometry checks are the same whether the driver runs on an embedded SoC or a workstation.

Why the IMU frame convention trips people up

An accelerometer measures along its own silicon axes, which point wherever the chip was soldered on the board. The camera driver's job is to publish a fixed rotation from those die axes into the camera's optical convention: Z out of the lens, X right, Y down. That rotation is a statement about geometry inside the housing, and nothing about the data stream tells you whether it is true.

When the driver or its camera description file contains a mistake, the published IMU frame can end up somewhere impossible: translated in front of the lens rather than inside the housing, or rotated about two axes at once where the mechanical drawing says one. The image data is unaffected, so nothing crashes. You only notice when fusion misbehaves.

The two checks below catch both failure modes in minutes.

Check 1: print the transform and look for impossible values

Start by echoing the static transform from the color optical frame to the gyro and accel frames with whatever transform-inspection tool your middleware provides. You want two numbers: the translation and the rotation.

Translation. The IMU sits on a circuit board inside the housing, typically within 10 to 30 mm of the lens axis. A translation of, say, 70 mm along the camera's forward axis places the accelerometer clearly in front of the housing. That alone is enough to declare the published frame wrong: no MEMS part lives outside a sealed plastic shell.

Rotation. Camera makers mount IMU dies axis-aligned to the housing, so the correct rotation from die axes to optical axes is always a combination of exact quarter turns: 0, 90, 180 or 270 degrees about single axes. In quaternion form the components come out at clean values such as 0.5, 0.707 or 1.0. A published rotation of roughly +90 degrees about X and +90 degrees about Z simultaneously is suspicious, because that corresponds to no ordinary die mounting. Small deviations of a few hundredths of a degree are normal rounding; large second-axis components are not.

A quick mental model: each quaternion component relates to half the rotation angle about its axis. Components of exactly 0.5 on all four slots means two simultaneous quarter turns. Check what your camera's mechanical drawing implies and compare.

Check 2: rotate the camera and watch one axis at a time

Static printouts can look plausible and still be wrong. The physical test settles it.

1. Place the camera level on a bench and record the accelerometer reading for a few seconds. Gravity should appear almost entirely on one axis, and it should be the axis your convention says points down. With the optical convention (Z forward, Y down), a level camera should read roughly 9.81 m/s² on Y and near zero on X and Z.
2. Rotate the camera 90 degrees so the lens points straight down. Gravity should now appear almost entirely on Z. If it appears on X, the published roll axis is mislabelled by a quarter turn.
3. Yaw the camera slowly about the vertical axis. The gyro should report the rotation about its Z axis, and the integrated angle should match the angle you turned, measured against a mark on the bench.
4. Roll the camera about the lens axis. The gyro response should appear on the axis the published transform calls roll, not on another one.

If gravity lands on the correct axis but with the wrong sign, the transform is off by a 180-degree turn. If it lands on a different axis entirely, a quarter turn is missing or doubled. Both are signature errors of a wrong description file, and both are exactly the kind of thing that survives unit tests because each individual stream still looks healthy.

Where the error usually lives

Work through these in order, because the first two cover most cases:

1. The camera description file. Many drivers ship a separate model description that lists frame names, translations and rotations. A copy-and-paste error between models propagates silently. Compare the values you measured with the entries for your exact model, not the nearest one.
2. A driver branch difference. If the same hardware publishes different transforms under different driver versions or branches, one of them is wrong. Diff the two transform definitions directly; that is faster than testing fusion behaviour.
3. Axis convention mismatch in your own code. Some conventions use Y forward and Z up for IMU data while images use Z forward. If your fusion node assumes a different convention than the driver publishes, you will reproduce the symptom with a perfectly correct driver. Re-derive the expected rotation from first principles before blaming the vendor.
4. A genuinely rotated IMU mount. Rare, but some housings do mount the die at an angle to clear other components. The mechanical drawing or the vendor should confirm; do not assume a non-quarter-turn rotation is automatically an error.

Isolating the fault from fusion behaviour

Do not debug this through your visual-inertial odometry output; too many other factors move the result. Instead, feed the IMU streams into a passive plot: raw accel and gyro versus time while you perform the rotations above. The plot shows the true die axes with no transform applied, and comparing it against what the published transform claims gives you the correction directly. Once the raw data makes sense, the transform either matches it or does not, and there is nothing left to argue about.

If your end goal is inertial-visual fusion, it is worth reviewing how the fused estimate is formed before changing extrinsics, because filter tuning errors mimic frame errors closely; a guide on implementing a Kalman filter for IMU and camera fusion walks through that side. Fix the geometry first, then tune.

Worked example: judging a published transform

Suppose the echo tool prints a translation of [-70, 3, 57] mm and a rotation of about 90 degrees about X and 90 degrees about Z from color optical to gyro frame.

• Translation: the forward component of 70 mm exceeds the entire depth of a typical housing, so the gyro frame sits in front of the lens. Physically impossible.
• Rotation: two quarter turns about different axes. Real die mounts are a single quarter turn relative to the optical frame in almost every design, and the components of the quaternion (all near 0.5) confirm two simultaneous quarter turns rather than one.
• Verdict: the published extrinsics do not describe this camera. Correct them from your own measurements in Check 2, or obtain the right values from the vendor's documentation for the exact model.

Related failure modes that are not frame errors

Before rewriting extrinsics, rule out neighbours with the same symptoms:

• Timestamp misalignment. If IMU and image data are fused with wrong timing, the result drifts even with perfect geometry. Verify with the method in depth and color synchronization: check the right timestamps and its companion on camera IMU synchronization: align events with exposure.
• Timestamp jumps over long runs. Counter resets mid-recording look like intermittent extrinsic errors; see IMU timestamp jumps: diagnose resets in long recordings.
• Depth-to-color registration problems. If your issue is between two image streams rather than image and IMU, the geometry involved is different; depth to color registration: raw versus rectified frames covers that case, and multi camera extrinsic calibration in one shared coordinate frame covers the multi-camera version.

Before you trust the frames: a short checklist

• Transform echoed from color optical frame to both IMU frames; translation within the housing footprint.
• Rotation composed of exact quarter turns about single axes, matching the mechanical drawing.
• Level-camera accel reads gravity on the expected axis, correct sign, magnitude near 9.81 m/s².
• 90-degree rotations about roll, pitch and yaw each appear on the axis the transform claims.
• Driver version and camera description file identified and recorded, so the fix survives updates.

When the answer is to fix the data, not the camera

If the camera itself is fine but the published description is wrong, you have three options: correct the description and re-publish it, apply your own static correction transform downstream, or ask the driver maintainers to fix it with your measured evidence attached. Publish your correction under a distinct frame name so nobody consumes the broken default by accident.

If you are selecting hardware for a project that depends on fused inertial data, check early that the vendor publishes a mechanical drawing of the IMU position and a stated die-to-optical rotation; time-of-flight modules such as the MV-DT01WMU ToF sensing camera and structured-light modules such as the MV-DPS200P-02 projected structured-light 3D camera sit in the wider range of 3D sensors you can filter in the camera finder. A vendor who documents the geometry is a vendor whose geometry you can verify.

For depth data usage beyond IMU fusion, bounding box depth: mapping image pixels to 3D points shows how to turn depth pixels into coordinates once your frames are trustworthy.