URDF nominal extrinsics vs calibrated camera offset

You fused a LiDAR and a depth camera on one rig, registered the overlapping point clouds, and got the same offset every run: a few centimetres of translation and about a degree of tilt. The URDF disagrees. It says the camera cannot sit that far back, and no bracket is that sloppy. In this standoff the measurement is usually right, and the model is describing a different frame than you think. Below you rank the likely causes in order of probability, run one test on each, and then encode the measured offset as a fixed joint that survives SDK updates. The same workflow holds whether the camera rides a wrist or a static frame, and it starts by deciding which side to trust.

The symptom: a repeatable offset the URDF calls impossible

You calibrate camera to LiDAR by registering two overlapping scans of one scene, and a wall meeting a ceiling is the classic target: it pins two rotation axes at once. Then the URDF spoils it. Its nominal transform puts camera_link at the housing, and applying your measured correction pulls the clouds apart instead of together. Before you touch the model, spend two minutes confirming the measurement deserves the fight:

• The same correction returns at several depth ranges, say 0.5 m, 1.5 m and 3 m. Noise does not repeat; geometry does. If the clouds scatter rather than offset, clear that first with the hardware versus software registration causes checklist.
• A known object lands where it belongs. An arm you can pose from the controller is ideal: read the pose, apply the correction, and check that the arm's patch in the camera cloud sits on the arm in the LiDAR cloud.
• The camera agrees with itself. Depth and colour clouds overlay cleanly, so the camera's internal registration is not the culprit.

When all three hold, the registration has earned its 59 mm and its degree. Stop auditing the measurement and audit the frame tree instead: the model is answering a different question than the clouds are.

Rank the causes before you touch the model

Five failure modes produce this exact standoff. Table them, then work top down:

Work rank 1 first: it is the most common and the cheapest to settle. In one public calibration saga, the rig owner found a bug in their own URDF, and the fix removed the vertical displacement while a smaller horizontal remainder stayed. Rank 2 is the structural one: the model was never wrong about the housing, it was describing a frame the registration never measured. Rank 3 is real too, because depth-camera SDKs have shipped corrected URDF trees between releases, and calibrating on top of a stale tree wastes the day. Ranks 4 and 5 you exclude once, in that order, and rarely revisit.

Why nominal extrinsics disagree with your measurement

A depth camera in a ROS-style tree carries two frames that matter here. camera_link is the mechanical reference: X forward, Z up, sitting where the description puts the housing. camera_depth_optical_frame follows the optical convention: Z forward along the optical axis, X right, Y down, with its origin at the depth reference point inside the housing, often 20-60 mm behind the mounting face. The joint between them carries the fixed axis rotation that converts one convention into the other, plus that housing-to-imager offset.

Nominal extrinsics are the design values of that joint, taken from CAD. They carry the housing geometry exactly and your mounting error not at all. Your registration carries both:

A clamped bracket repeats to about 1-3 mm and a few tenths of a degree; a doweled or dial-indicated fixture reaches a few tenths of a millimetre. When the difference comes back at tens of millimetres, with the largest part along the optical axis, the subtraction itself is suspect: you compared a housing frame against an optical frame, or a sign flipped in your own description. And a rotation can never absorb a translation, so tuning only rpy against a centimetre-scale residual is time spent in the wrong term.

Two toggles decide which values your tree even carries. One depth-camera SDK uses the URDF argument use_nominal_extrinsics and the launch parameter enable_publish_extrinsic; whatever yours calls them, confirm the switch before assuming the nominal joints exist. While you are in the tree, check every frame, not just the depth pair: one SDK release shipped with the IMU frames, camera_gyro_frame and camera_accel_frame, parked outside the housing while the depth frames were fine. The IMU optical frame verification routine extends to every auxiliary frame. For the axis rotation itself, the optical frame to camera frame transform walkthrough works the math end to end.

Worked example: is 59 mm and 1° a mounting error?

A static-rig registration returns this correction, from housing frame to data frame:

xyz = (-0.030, -0.006, -0.051) m, rpy = (-0.018, 0.017, 0.010) rad

Translation magnitude:

|t| = sqrt(0.030² + 0.006² + 0.051²) m = sqrt(0.000900 + 0.000036 + 0.002601) m = sqrt(0.003537) m = 0.0595 m, about 59 mm

Rotation per axis, at 57.3 degrees per radian:

0.018 rad gives 1.03°, 0.017 rad gives 0.97°, 0.010 rad gives 0.57°

What one degree does to the clouds at range:

e = L × tan(1°) = 2.0 m × 0.0175 = 35 mm of lateral shift at 2 m

Now the verdict. A clamped bracket explains 1-3 mm; this correction is twenty to sixty times that, and 51 mm of it lies along the optical axis, a direction where the mount has no authority: a bracket moves the whole housing and cannot shift the optical origin inside it. The description therefore compared two frames that are not the pair it claims, with the internal offset missing from one side. Trust the clouds, repair the tree, and only then read the remainder as mount error.

Decision: millimetres of difference, fold it in as mount error. Tens of millimetres with an along-axis component, fix the description first.

Tests that isolate the cause

1. Draw the TF tree over the clouds. Confirm every hop's parent, child and direction: camera_link must read as parent of camera_depth_optical_frame, not the reverse. A swapped pair inverts the transform.
2. Check the axis hop. The fixed rotation that turns X-forward-Z-up into Z-forward-X-right-Y-down must sit in its own joint. If you edited or deleted it while chasing the offset, restore it before calibrating: a missing axis turn is worth 90°, not 1°.
3. Repeat the registration at three ranges and two scene types. The same answer each time is geometry; scatter means the scene needs work first. The extrinsic calibration accuracy targets article sets what 'same' should mean numerically.
4. Land a known object under the corrected transform. Pose the arm from the controller, apply the correction, and confirm the arm's patch lands on the arm in both clouds.
5. Update the SDK description and diff the frame tree. A corrected upstream URDF costs one download; calibrating on a stale one costs the day.
6. Then calibrate, using the scene when the rig cannot take a target: aligning depth and tracking cameras step by step, or extrinsics from measured scene points.

Encode the fix as a measured fixed joint

Keep the SDK's visual model where the housing is, and carry the measured correction in one fixed joint of your own. The frame names below follow the names the SDK publishes; the values are the worked example's, rounded to the millimetre and to 0.001 rad:

<link name='camera_visual_link'/>
<joint name='camera_visual_to_camera' type='fixed'>
  <parent link='camera_visual_link'/>
  <child link='camera_link'/>
  <origin xyz='-0.030 -0.006 -0.051' rpy='-0.018 0.017 0.010'/>
</joint>

Three habits keep the correction alive:

• Comment the joint with the date, both serial numbers, the SDK version and the calibration ranges. The next engineer should not have to reverse-engineer your millimetres.
• Round to what the measurement supports. A value like 0.05113393129377081 m claims sub-nanometre precision that no point-cloud registration delivers; three decimals in metres and three in radians are honest.
• Never fold the values into the SDK's own joint. The next description update overwrites it, and the correction disappears without an error message.

For quick experiments, publish the same numbers as a static transform from the launch file, but land them in the URDF before the cell ships: launch-file corrections do not follow the project between workstations. If your tree runs the other way, visual link as child of camera_link, attach the geometry with the inverse transform instead: negate the translation, transpose the rotation.

Fix the mount instead when the rig is loose

A measured correction absorbs whatever the mechanics did, including what the mechanics will undo. On a loose bracket, vibration and the day's temperature swing walk the mount by millimetres, and the joint you calibrated in the morning quietly stops being true by the afternoon. Two symptoms point this way: the residual drifts across a shift, or re-measuring after someone touched the rig returns a different offset. Then the fix is hardware, not software.

Give the camera a position it can repeat. The LS-MNT-0040 camera stand, 700 mm height with a 400 x 450 mm base, holds a bench rig steady while you calibrate camera against LiDAR. The LS-MNT-0220 manual XY and rotation stage, dia. 60 mm base, trims roll, pitch and yaw by hand while you watch the residual collapse. Both come from the camera stands category, and the accessories finder filters the rest.

Once the mount repeats to a few tenths of a millimetre, re-measure and re-enter the joint. The values carried over from the loose era are wrong by definition.

Building the rig from scratch

If you are specifying the rig rather than repairing it, pick the depth side from the 3D sensors category and budget a calibration session at commissioning: every depth camera ships a nominal frame tree, and your measurement is what makes it true. The class covers projected structured light with the MV-DPS200P-02, ToF with the MV-DT01WMU and stereo with the MV-DLS300P-04:

Mount camera and LiDAR to one plate, overlap their views of the scene, and run the same register-and-encode workflow. When two depth cameras share the scene without a LiDAR, the multi-camera point cloud fusion with depth camera extrinsics article applies the same reconciliation between them.

Checks before you sign the transform off

• Re-run the registration at two ranges with the joint in place; the residual holds at a few millimetres.
• View the TF tree over the clouds one last time; every hop's direction reads the way the clouds do.
• Overlay the coloured cloud on the raw depth cloud; they agree where they should.
• The correction joint carries its comment: date, serials, SDK version, calibration ranges.
• Downstream consumers behave in a dry run. The vision-guided robot offset correction article shows what a stale or mirrored transform does to a pick.
• The measured values, the scene and the ranges live in version control next to the URDF, not in a notebook.

If you are assembling the hardware now, start at the stand step of the configurator, filter sensors in the camera finder, and keep the 3D guides open for the rest of the frame and calibration fixes. When a transform still refuses to sit still, send the frame tree and both clouds through expert support and calibrate from the reply.