Point cloud extrinsic transform order: rotate then translate

Two depth clouds that should land on top of each other sit a fixed distance apart, or every board-frame measurement comes out short by the same amount. The calibration is usually innocent. The bug is the order in which the rotation and the translation are applied, and each tool hides its convention somewhere different. This guide names the two conventions, shows the residual a mixed order leaves, and gives a validation test to run on the calibration board before the transform touches production data. It is written for anyone merging depth clouds from 3D sensors or handing poses to a robot from code running on a vision controller.

Why does my merged point cloud sit a fixed distance off target?

The offset is the order bug: applying a translation in the wrong order for its convention leaves a residual the size of the translation itself, so a 100 mm baseline becomes an error of the same 100 mm order even with a perfect rotation matrix.

The signature lives in the residual, not in the scene. Transform a point, subtract where that point should be, and repeat across the cloud. An order bug returns the same vector for every point. A calibration error returns random scatter. A unit or scale error returns a residual that grows with distance from the sensor.

Divide the residual's magnitude by your calibrated baseline for a verdict before touching code: near 1 convicts the order, near 1000 convicts units, small and random convicts the calibration itself. When the baseline comes from merging several cameras into one frame, the frame design behind it is covered in multi-camera extrinsic calibration in one shared coordinate frame.

Rotate then translate, or translate then rotate: which comes first?

Neither order is right on its own: the numbers you hold decide it. If your tool outputs the world-to-camera pair p_c = R · p_w + t, subtract the translation first and rotate with the transpose. If it stores a camera pose, rotate first, then add.

The rotation can be exact to the last bit and the residual stays put, because the mixed-order error is (R - I) · t and its size follows only θ and |t|. One subtlety: place t along the rotation axis — (0, 0, 100) mm here — and the residual is exactly zero. The bug hides until the camera is re-mounted off the axis, which is why a rig can pass on the bench and fail in the cell.

Which convention do calibration tools and camera SDKs actually output?

OpenCV-style output — rvec and tvec from a solver such as calibrateCamera or solvePnP — maps board points into the camera, so a camera-to-board cloud transform subtracts tvec first and rotates with Rᵀ, or converts once with t_w = -Rᵀ · tvec.

The camera's position in the world is therefore -Rᵀ · tvec, not tvec itself, a detail that matters the moment you reason about where the sensor sits on the rig. Transpose R and negate the converted translation once, at load time, and the rest of the pipeline can assume rotate-then-add forever.

Depth SDKs add a second layer: factory extrinsics that map depth points into the color camera, measured when the sensor was built. Apply them through their own call and never rebuild them from your board fit — the factory pair and your board frame are different transforms. The pixel-to-point step feeding all of this has its own arithmetic, set out in depth map to point cloud: the deprojection calculation.

An integrated projected structured-light sensor such as the MV-DPS200P-02 collapses that layer: it projects the pattern and returns depth in one measurement, so the only extrinsic left in your code is the one from its frame to the cell.

Why did swapping the order not fix the offset?

Order is only the first of five causes; if a correct swap leaves the offset behind, work down the list — rotation direction, units, the wrong calibrated stream, distortion, row-vector habit — in that order of likelihood.

1. Rotation direction. The cloud lands rotated the wrong way about the board origin. Test: replace R with Rᵀ; the inverse of a rotation matrix is its transpose, so the swap costs one character and should collapse the residual.
2. Units. A tvec in millimeters against a cloud in meters — depth pixel times depth scale — leaves a residual 1000 times too large or too small. Test: print one component of t next to a tape-measured distance and compare unit for unit.
3. Wrong stream calibrated. The board detection ran on color images while the cloud is deprojected from depth or infrared pixels; color optics are not depth optics. Fix: calibrate the exact stream you transform, or move depth points into the color frame with the factory extrinsic first.
4. Distortion. Pixels from an unrectified lens bend toward the edges, so the residual grows with distance from the image center. Test: split the residual by pixel radius — a radial trend convicts distortion. Rectify pixels before deprojecting.
5. Row-vector habit. Writing p · R instead of R · p applies the transpose and silently inverts the rotation. Symptom: the code only works where a transpose has been inserted "to make it fit". Standardize on column vectors and one helper.

How do I prove a transform before running it on live data?

Re-run the calibration board through your own code: project its known circle centers into the image and transform the measured cloud points back to board coordinates. A healthy chain scatters a few millimeters; a constant vector the size of your baseline convicts the order.

Two numbers set the pass line. On the images, a sound calibration reprojects the board's circle centers at an RMS under about 0.5 px. In 3D, depth noise for consumer-class structured-light and stereo sensors runs near 1 % of the working distance, so at 500 mm the transformed centers should scatter a few millimeters about the known coordinates with a mean near zero. Industrial coded-light sensors land several times tighter again.

Anything outside that pattern fails with a name: a constant vector at the baseline size means order, rotation about the origin means a missing transpose, error that scales with range means units or depth scale. No board at hand? The same check runs on measured scene points instead of a checkerboard, and a tape-measured distance pins the Z translation as in camera calibration with a known distance.

A ToF sensing camera such as the MV-DT01WMU passes through the identical test: deproject with the camera's own intrinsics, transform with your extrinsic, compare each circle center to its board coordinate.

How do I make the order mistake impossible in my pipeline?

Convert every calibration once, at load time, into a single stored pose — rotate then add, p_w = R_w · p_c + t_w with t_w = -Rᵀ · t — and route every transform through one helper, so the order question is answered in exactly one place.

import numpy as np

# From the calibration tool: R is (3, 3) float (world -> camera), t is (3,) float in mm
R_w = R.T                       # camera -> board rotation, (3, 3) float
t_w = -(R.T @ t)                # camera origin in the board frame, (3,) float, mm

def to_board(p_c):              # p_c: (3,) float, mm, camera frame
    return R_w @ p_c + t_w

# Test on the board points already measured: pairs of (camera-frame point, board coordinate)
res = np.array([to_board(p_c) - p_w for p_c, p_w in board_pairs])   # (N, 3) float, mm
rms = np.sqrt((res ** 2).sum(axis=1).mean())                        # float, mm
print(rms)                      # about 1 mm passes; about 141 means order is still wrong

Where does the same trap appear in a robot cell?

The same convention trap waits in the robot cell, where a hand-eye result handed over as Euler angles meets the controller's own fixed extrinsic rotation order, and a mismatch rotates the pose about the wrong point.

Robot controllers keep their own stored conventions. One pendant family applies its WPR workpiece-coordinate rotation in a fixed extrinsic order. Another sets out, in its MULMAT coordinate transform, whether translation or rotation acts first.

Axis conventions flip closer to the sensor too: depth clouds arrive in an optical frame whose axes the mounting does not share, and the cure is a fixed conversion as in transforming depth point clouds between optical and camera frames. A hand-eye result that still lands the tool centimeters off is its own investigation — hand-eye calibration translation error: is 1-2 cm normal? walks the likely magnitudes.

Which fix matches my residual?

Read the residual and the fix names itself: a constant vector near the translation magnitude is order, a rotation about the board origin is a missing transpose, a 1000-scaled error is units, and zero-mean scatter of a few millimeters is a pass.

Match the signature, apply the one-line fix, rerun the board test, and expect a pass on the first retry; a second failure means a second signature, not a harder version of the first. To choose the hardware around this code, filter depth sensors in the camera finder or start at the camera configurator. When a residual survives all five causes, bring the vector, both matrices and the units to the contact page and we will work the chain with you.