Transform depth point clouds between optical frame and camera frame

You subscribe to a depth camera's point cloud, read the frame_id, and find that it is not the frame you expected: the points are tagged with an optical frame such as camera_depth_optical_frame, while your robot or cell model lives in a body or base frame. The x, y and z values look wrong for the real world, and the temptation is to edit the driver so it publishes directly in the frame you want. That is almost never the right move. The data is not wrong; it is expressed in a different, well-defined convention. This article explains the two conventions, gives the rotation that converts between them, works a numeric example, and shows where transforms go wrong.

Why the optical frame exists

A depth sensor does not measure a world grid. It measures, for each pixel of the image sensor, one range value along that pixel's viewing ray. The natural axes of that measurement follow the image: x to the right across the columns, y down the rows, and z forward along the optical axis, growing with range. That right-down-forward set is the optical convention, and the driver tags the cloud with an optical frame to say so.

A body frame describes the housing instead: typically x forward, y left, z up, the way a mobile robot or a mounting bracket is described. Neither convention is more correct. The driver publishes the cloud in the frame where the numbers are produced, and publishes the fixed transform between the optical frame and the body frame alongside it. Your job is to apply that transform once, on your side, rather than asking the driver to burn CPU rotating every point for every subscriber.

Stereo and ToF cameras in the 3D sensors category all follow this pattern, whatever their ranging principle:

The rotation between the two conventions is a constant of the housing design, so you apply it once and forget it.

The rotation, concretely

For the usual case, body axes are x forward, y left, z up, and optical axes are x right, y down, z forward. Matching the axes gives this rotation matrix, applied to every point p expressed in optical coordinates:

In words: the body x (forward) is the optical z (depth), the body y (left) is the negative optical x, and the body z (up) is the negative optical y. If your software already provides a transform tree, prefer its transform over a hand-written matrix; the matrix here is for checking that what the tree gives you is sane.

For a full pose you also need the translation from the body origin to the optical origin, which the driver publishes as part of the same transform. A depth camera's optical origin sits at the principal point of the depth sensor, not at the mounting hole on the bracket, so the translation is rarely zero and is often tens of millimetres.

Many drivers instead publish the inverse pose, from body to optical. Check the direction: a common bug is composing the transform backwards and getting a mirror image of the scene.

A worked example

Take one pixel from a depth cloud in optical coordinates:

That reads: 40 mm right of the optical axis, 25 mm above it, 1.2 m ahead. Apply the matrix term by term:

So the point sits 1.2 m in front of the housing, 40 mm to its right, 25 mm above the optical origin. Sanity checks: the forward distance moved from the optical z slot to the body x slot unchanged, and the two lateral signs flipped, because the image's down is the world's negative up. If your result instead shows the point 1.2 m above the robot, you picked the wrong axis slots or the inverse transform.

A point at the image centre has optical x = y = 0, so it maps to body (z, 0, 0): straight ahead at its range. If centre points do not come out straight ahead, your rotation is wrong. That centre-pixel trick is also a fast field check, and it is the same idea behind reading a fixed image point described in center pixel depth measurement: read a fixed image point.

Applying the transform in practice

You have three sane options, in order of preference.

1. Let the middleware do it for display and merging. A transform listener that knows the optical-to-body transform can re-express any cloud on request. Set your viewer's fixed frame to the body or world frame; the cloud will appear correctly placed without you touching the data.

2. Transform the cloud once in your own code. Look up the transform with the cloud's timestamp, apply the rotation and translation to the point buffer, and relabel the frame_id afterwards. One rotation per point, done once, then the cloud is usable everywhere.

3. Never relabel without transforming. Overwriting the frame_id string alone makes every downstream consumer misread the numbers. Robots will reach to mirrored or rotated positions, and merged clouds will double up on themselves.

If you work with depth and colour streams from the same device, note that each stream has its own optical frame; the extrinsics between them come from the factory calibration. The conversion between those parameter sets is covered in camera intrinsic parameter conversion for depth and texture images. When two clouds must land in one world frame, the same transform mechanics apply, as described in multi camera extrinsic calibration in one shared coordinate frame. For merging two consumer depth clouds step by step, an outside reference on merging two stereo depth camera point clouds walks the frame arithmetic in detail.

Timestamps: the second half of a transform

A transform is only valid at one moment. The optical-to-body rotation never changes, but the body-to-world pose changes whenever anything moves. If the camera rides on a robot arm or a moving platform, capture the transform stamped at the cloud's acquisition time, not the latest one on file. A 100 ms lag on a platform moving 0.5 m/s puts every point 50 mm off, which is enough to miss a bin. This is the same failure mode treated in depth and color synchronization: check the right timestamps.

For stationary cameras on a fixed stand, one static transform captured at commissioning is enough, but re-check it after any bump, re-mount or bracket replacement, and after any recalibration of the sensor itself.

Choosing and mounting with frames in mind

The transform work gets easier if you plan the frames before the hardware goes up:

• Mount the camera so the body frame is close to the frame your cell already uses; a small yaw offset is easier to trust than a compound rotation.
• Record the mounting offset at commissioning, from the housing's mechanical datum to the robot or cell origin, and store it with the cell documentation.
• Prefer depth cameras whose driver publishes the full transform tree, so your code reads transforms instead of hard-coding housing dimensions.

The shop finder for depth and stereo cameras filters by ranging type and interface if you are still selecting hardware, and the vision configurator walks camera, lens and light choices in order for a complete station.

Checks before you trust a transformed cloud

Run these on the bench, before the cell runs live:

• Centre check: the point at the image centre must land straight ahead of the body origin at its measured range.
• Box check: place a box at a known offset, transform the cloud, and confirm the dimensions and signs on all three axes.
• Floor check: with the camera level, a ground-plane fit of the transformed cloud should be level in the world frame; a tilt usually means a leftover mounting rotation, a case examined in why a laser triangulation point cloud is tilted: calibration checks.
• Round trip: apply the transform and its inverse to a cloud and confirm you recover the original numbers to within float rounding.

A short checklist for the day you commission a new camera:

• Note the frame_id the driver actually publishes; do not assume it.
• Confirm the transform direction (optical-to-body or body-to-optical) from the driver's documentation or a centre-pixel test.
• Capture the mounting transform with timestamps and store it in version control.
• Verify with the box and floor checks before the first live pick.

When the usual answer is wrong

Occasionally the optical-frame convention is not your real problem. If the cloud looks wrong even in the optical frame, the cause is usually calibration, mounting flex or a violated assumption about the scene rather than frames: fuzzy or thin surfaces dropping out of range are covered in close range depth dropout on fuzzy surfaces and thin parts, and coverage differences between the colour and depth views in depth image field of view: why stereo coverage is smaller. Fix the data first, then the frame. And resist patching the driver to bake the transform in: it couples your code to one driver version, doubles the rotation work, and hides a transform your downstream tools will need anyway.

When you are ready to pick hardware, start from the 3D sensor category, or contact the applications team with your mounting and frame questions and we will help you spec the station.