3D visionExplainer

Depth-to-RGB transform vs IMU extrinsic: which is which

Depth-to-RGB vs IMU: the transform beside a depth camera's intrinsics is the camera-to-camera extrinsic; the gyro-to-depth one is separate and can be identity.

LumaSpec Vision11 min read Share on LinkedIn

On this page 9 sections
  1. 01What is the transform block in a depth camera's calibration list?
  2. 02How do you read the rotation and translation numbers?
    1. The worked example
  3. 03Is that transform the IMU extrinsic?
  4. 04Why can the gyro-to-depth transform be identity?
  5. 05How do you tell which extrinsic you are holding?
  6. 06What breaks when the two are swapped?
  7. 07When should you recalibrate the depth-to-RGB extrinsic yourself?
  8. 08Where does each transform go in your pipeline?
  9. 09Questions

A depth camera's SDK hands you one calibration structure: depth_intrinsic, depth_distortion, rgb_intrinsic, rgb_distortion and a single transform built from a rot block plus a translation vector. The question that costs an afternoon is which frames that transform connects — depth to RGB, or depth to the IMU. Answer it wrong and color lands on the wrong depth pixels, point clouds pick up a slow twist, and a visual-inertial filter drifts from its first frame. The numbers settle it: what the block is, how to decode a 6.38° pitch and a 32.5 mm offset out of nine rotation elements, why a gyro-to-depth transform can legitimately be identity, and a three-question test that names the extrinsic you are holding.

What is the transform block in a depth camera's calibration list?

It is the depth-to-RGB extrinsic: a 3×3 rotation matrix and a 3-element translation vector that carry a point from the depth camera's frame into the RGB camera's frame. In the worked example below it encodes a pitch of about -6.4° and a 32.5 mm sideways offset.

The list is the device's factory calibration, measured on the production line. depth_intrinsic and depth_distortion describe the depth imager; rgb_intrinsic and rgb_distortion describe the color imager. transform is the rigid mount measurement between the two. You read it through a device-level calibration getter of the get_calibration_camera_param_list() kind; get_camera_param(0) returns the first parameter set in that list. The structure carries no inertial field: no imu, gyro or accel entry sits anywhere beside it.

That absence is the tell. A parameter set scoped to cameras connects cameras. The IMU extrinsic travels separately — a dedicated SDK call, or a static transform the driver publishes on its own. Structured-light and ToF depth cameras across the 3D sensors range publish factory calibration of this kind, read through the vendor SDK; the 3D camera finder filters them by type.

3D sensorsHikrobot MV-DPS200P-02 Projected structured-light 3D cameraMV-DPS200P-02
Type
Stereo / structured light
$8,829.00 · quotedView and quote
3D sensorsHikrobot MV-DT01WMU ToF sensing cameraMV-DT01WMU
Type
Depth / ToF camera
Price on requestView and quote

How do you read the rotation and translation numbers?

Read rot as the matrix R and transform as the vector t in the mapping p_rgb = R · p_depth + t, with the translation printed in millimeters. The worked example below decodes to a 6.38° pitch about the x-axis and a total offset of 32.6 mm.

rot lists nine numbers row by row, R[0][0] through R[2][2]. Apply the pair as rotate first, translate second — reversing that order is its own classic bug, covered step by step in why extrinsic transforms rotate then translate.

p_rgb = R · p_depth + t θx = asin(-R[1][2]) for a mount tilted essentially about one axis

The worked example

A real parameter list from a stereo depth camera with a 640×576 depth stream and a 640×480 RGB stream, printed in the SDK's own order:

rot       = [0.993813, 0.000713588, -0.00148801, -0.00054392, 0.993814, 0.111057, 0.00155805, -0.111056, 0.993813]
transform = [-32.5317, -1.03079, 1.90774]

Rows two and three hold the large off-diagonal pair: R[1][2] = 0.111057 and R[2][1] = -0.111056. That pair is the sine of the tilt about the x-axis, so θx = asin(0.111057) = 0.1113 rad, and 0.1113 × 57.296 = 6.38°. The sign records which way the color axis pitches relative to depth; the magnitude is what you sanity-check.

Translation magnitude: |t| = √(32.5317² + 1.03079² + 1.90774²) = √(1058.31 + 1.06 + 3.64) = √1063.01 = 32.6 mm.

Decode: the color aperture sits 32.5 mm off the depth axis along x, about 1 mm along y, about 2 mm along z, tilted 6.38°. The magnitude also settles the unit: two apertures in one housing sit tens of millimeters apart — 32.6 cm is absurd on a camera head, 32.6 µm impossible. Millimeters is the only scale that fits, which is how you confirm the unit when the header comment is missing.

Is that transform the IMU extrinsic?

No — a parameter list that also carries rgb_intrinsic connects two camera frames, so its transform is depth-to-RGB. The IMU extrinsic is a gyro-to-depth transform that the SDK exposes through a separate call and that your driver publishes as its own static transform.

Frames named depth and rgb on both ends settle it. The example structure prints camera intrinsics and nothing inertial, so the transform maps depth-frame points into the RGB frame — exactly what a color-on-depth projection consumes.

You can prove it in one toggle. Switch on depth-to-RGB alignment in the pipeline and read the list again: on the camera used for the worked example, the SDK then returns an identity rot and a zero translation, because depth pixels are already registered into the RGB frame and the extrinsic has nothing left to do. An IMU extrinsic cannot respond to a pixel-alignment mode — no software setting moves an inertial chip inside its housing.

Why can the gyro-to-depth transform be identity?

Because the vendor defines the IMU frame that way: the coordinate-system documentation for this camera class places the gyroscope origin at the same physical point as the depth camera origin. Gyro-to-depth therefore legitimately prints translation [0, 0, 0] with an identity rotation — a convention, not a missing calibration.

Identity admits two readings, and only one is safe to accept on faith. Here the driver is obeying its device's frame definition: with the gyro origin pinned to the depth origin, the translation is zero by construction, and the published transform is a unit quaternion — rotation 0, 0, 0, 1 — printed straight from the device rather than a placeholder for unset values. The coordinate-system notes are where such a definition is written down, so they are the first place to look before treating an identity as suspicious.

The accelerometer is the exception to carry forward. Its origin is defined at the chip's physical location, so a gyro-to-accelerometer relationship is not identity, and IMU axis conventions can still differ from the camera's axes. Verify the frames before you feed a filter: how to verify depth camera IMU frames walks that check, and building a camera-IMU rototranslation covers the case where you must measure one yourself. When the IMU stream itself looks noisy on a stereo rig, frames are rarely the fault — troubleshooting stereo depth jitter and IMU drift works that failure mode instead.

How do you tell which extrinsic you are holding?

Run the three-question test: does the same structure carry depth_intrinsic and rgb_intrinsic; does the transform collapse to identity when depth-to-RGB alignment is switched on; is the translation a few tens of millimeters? Three yes answers identify the depth-to-RGB extrinsic. An IMU extrinsic fails the alignment test.

Each question maps to evidence you already have on the bench, and the two transforms disagree in exactly one place: the alignment toggle.

Property Depth-to-RGB extrinsic Gyro-to-depth IMU extrinsic
Published in the camera parameter list, beside depth_intrinsic and rgb_intrinsic a separate SDK call or the driver's static-transform publisher
Frames connected depth camera frame to RGB camera frame gyroscope/IMU frame to depth camera frame
Translation unit millimeters in the SDK print millimeters in the SDK; meters in ROS static transforms
Effect of the alignment toggle collapses to identity once pixels are registered none — pixel alignment cannot move an inertial frame
Worked-example values rot: 6.38° pitch; t = [-32.53, -1.03, 1.91] mm; |t| = 32.6 mm translation [0, 0, 0]; rotation identity (quaternion [0, 0, 0, 1])

What breaks when the two are swapped?

Projection error grows with range: at 2 m, the example's 6.38° pitch puts a co-located assumption out by 2000 mm × tan 6.38° ≈ 223 mm. Color lands on the wrong depth pixels by a margin no acceptance threshold hides.

The 223 mm scales with the tangent: 56 mm at 0.5 m working distance, 335 mm at 3 m. Keeping the translation while dropping the rotation still leaves on the order of 190 mm at 2 m (223 − 32.5), so the tilt term dominates with range — the reason a color overlay can look clean up close and fall apart at two meters.

Two further swaps bite. Applying R and t when the direction you need is the inverse — R^T and -R^T·t — mirrors the offset to the wrong side of the depth axis; direction and units when inverting a camera-IMU transform works through that trap in detail. And hard-coding identity for gyro-to-depth on a device that does not pin the gyro to the depth origin quietly decorrelates your inertial data: identity there is a convention to read from the coordinate-system notes, not a default to assume.

When should you recalibrate the depth-to-RGB extrinsic yourself?

Recalibrate only when the measured overlay is off — color edges displaced by several pixels against the depth image across your working distance — and do it with your own checkerboard procedure. The factory extrinsic stands until your measurement proves otherwise.

Judge the factory extrinsic by overlay, never by the printed numbers. Project a target of known depth through the extrinsic and count the pixel offset between color edges and depth edges at your near, middle and far working distances. A pixel or two at the field border is the normal cost of two separate apertures; an offset that grows with range — tens of pixels by mid-range — means the rotation term is wrong for your unit.

Replace, don't patch: run your own checkerboard calibration and load the result in place of the factory values, the way depth-to-RGB alignment off? run your own checkerboard calibration sets it out, then re-check dimension trueness with the infield procedure in verify dimension trueness on the line. Frame-level fixes of this family are gathered on the 3D vision topic page.

Where does each transform go in your pipeline?

The depth-to-RGB extrinsic feeds every color-on-depth projection and RGB-textured point cloud; the gyro-to-depth transform feeds the odometry filter's IMU path. Publish each once as a static transform under an explicit frame name, and never substitute one for the other.

In practice the SDK hands the depth-to-RGB extrinsic straight to your alignment or projection function, and the driver publishes gyro-to-depth itself. Static transforms carry no advancing timestamp — if tf_echo prints time 0.000 for one, that is correct behavior, not a stale tree, as why static camera transforms show time 0.000 explains. Name the frames explicitly and prove any extrinsic on a static target before it drives a live robot cell; the compatibility checks that matter there are collected in 3D camera robot integration: check compatibility.

Both streams run on the machine-vision computer beside the camera. A compact controller such as the MV-VC2140-128G40-1T — an Intel N-series unit with 4 built-in light channels and 3-4 LAN ports — drives the camera and your filter from one chassis. The vision controllers range and the vision system configurator cover the rest of the chain: camera ports, PoE and light channels in one pass.

Vision controllersHikrobot MV-VC2140-128G40-1T VC2100 Vision Controller, N97, 8GB+128G SSD+ 1THDDMV-VC2140-128G40-1T
LAN ports
3-4 LAN ports
USB 3 ports
2 USB 3 ports
Light outputs
4 built-in channels
Memory
8 GB
$1,679.00View partBuild a system with it

The decision, in one line: a transform that ships beside depth_intrinsic and rgb_intrinsic is depth-to-RGB — apply it to points, in millimeters, rotate then translate. A transform named gyro-to-depth belongs to the IMU path, and identity there is the device telling you the gyro already lives at the depth origin. Read the names first; the numbers then confirm.

Questions

What units are the rotation and translation in?

The rotation matrix is dimensionless — it is built from sines and cosines, so every element is bounded by 1. The translation in a depth SDK's calibration structure is printed in millimeters; you can confirm that from the magnitude alone, because a depth-to-RGB baseline on one housing is tens of millimeters. ROS static transforms use meters by convention, so convert once at the boundary where the SDK value enters your transform tree.

My transform reads as exact identity — is the calibration broken?

Usually not. Identity is legitimate in two cases: depth-to-RGB alignment is active, so the extrinsic is already consumed by the pixel registration, or you are reading gyro-to-depth on a device that defines the gyroscope origin at the depth origin. A genuinely broken extrinsic looks different: a rot block whose rows are not unit length, or a translation far from the housing's physical baseline. Re-read the list with alignment off to separate the cases.

Does an identity gyro transform mean the IMU data is unusable for VIO?

No. It means angular velocity arrives already expressed on the depth camera's axes, so the filter needs no extrinsic composition for the gyro. The accelerometer is the exception: its origin sits at the chip's physical location and its axes follow the IMU frame convention, so treat it as its own frame. Most visual-inertial stacks also estimate gravity and biases online, which absorbs small residuals — but check the published frame names before trusting it.

How accurate is the factory depth-to-RGB extrinsic?

Treat it as a starting point measured per unit rather than a spec value: project a target of known depth through it and read the color-to-depth edge offset in pixels at the ranges your application uses. If your task tolerates, say, a 2 px fringe at 1.5 m, test at 1.5 m and accept on that number. A factory extrinsic that passes your overlay tolerance needs no recalibration; one that fails it goes through a checkerboard session.

Do I need to convert the rotation matrix to a quaternion?

For most SDK calls, no: pass the nine elements as printed. ROS and most SLAM stacks want a quaternion, which the identity case prints as [0, 0, 0, 1] in x, y, z, w order. Convert once at the boundary and keep the matrix as your source of truth — a matrix is easier to sanity-check, because its tilt axis and angle are readable at a glance.