Point cloud differs in rviz: isolate calibration from transforms

You check the depth camera in the vendor viewer and the point cloud is clean: flat floor, square corners, straight walls. Open rviz on the same stream and the scene is bowed, tilted or crushed. That mismatch carries real information. The camera has already proved its calibration and depth engine are good, so the error lives downstream, in the ROS driver, the frame conventions or the units. This guide gives you three checks in order of cost, ending with a one-pixel deprojection you can do with a calculator. If your hand-computed point matches the vendor viewer but not rviz, you stop debugging the camera and repair the transform chain instead. Most benches clear this inside an hour.

Read the distortion pattern before you change anything

The shape of the warp narrows the cause to one layer. Screenshot the rviz view, note what the vendor viewer shows for the same scene, and match the pattern:

Rows one and two account for most viewer-versus-viewer mismatches. Row three is rare when the vendor viewer is clean, because modern stereo pipelines correct lens distortion ahead of the disparity stage, feeding a rectified pair straight into the stereo depth engine. Straight walls in the viewer mean distortion is already handled. Row four is a camera problem to solve before anything in this guide applies.

Rank the suspects: the camera is usually innocent

Both tools consume the same sensor. If they disagree, the difference cannot originate inside the camera; it comes from how each side converts pixels into 3D points. Rank the causes by how often they appear on a bench:

1. Missing optical-frame rotation. Deprojection emits points on optical axes, so the cloud's frame_id must name a frame with those axes or carry the static transform to one.
2. Unit mismatch. Millimetre depth published into a metre-based pipeline stretches every coordinate a thousandfold.
3. Intrinsics from the wrong stream. fx, cx, cy borrowed from another camera or another rectification warp the scene.
4. Preset or mode difference. The viewer's pipeline and the published topic can run different depth settings; one common stereo ROS driver, for example, exposes stereo_i_depth_preset, and setting it to DEFAULT aligns the published depth with the viewer's.
5. Genuine calibration error. Last, because a clean vendor viewer rules it out; it only returns if the viewer itself is wrong.

Work down the list. Each cause has a quick test, and the tests run in the order below.

Check the depth image and its header first

Before touching transforms, confirm the published depth image is healthy. Display the depth topic in rviz and look at three things:

• Content. The image should read as the scene, with edges where edges are. If the depth image itself is already broken, stop: the fault sits in the driver or the camera link, and the fix starts at the driver, not the cloud.
• Encoding. The ROS depth-image convention carries 16UC1 as millimetres and 32FC1 as metres, so the encoding names the unit before any arithmetic.
• Header. Note the frame_id and the fx, cx, cy values the driver publishes alongside. The pixel test in the next section needs exactly those numbers.

A clean depth image with sane headers proves the driver's perception path and moves the investigation to frames and units.

The decisive test: deproject one pixel by hand

Pinhole deprojection turns a pixel plus depth into a 3D point:

f is the focal length in pixels, B the stereo baseline in metres, d the disparity in pixels, (cx, cy) the principal point, and fx, fy the focal lengths in pixels from the camera_info message. Pick the pixel nearest the image centre so cx and cy subtract to roughly zero.

Worked example, with illustrative numbers that are not any camera's calibration: fx = fy = 400 px, cx = 640 px, cy = 360 px on a 1280 × 720 px image, baseline B = 75 mm, and a pixel at u = 840 px, v = 360 px returning d = 15 px:

• Z = 400 × 0.075 / 15 = 2.000 m
• X = (840 − 640) × 2.000 / 400 = 1.000 m
• Y = (360 − 360) × 2.000 / 400 = 0.000 m

The pixel sits 200 px right of centre, so the point lies 1.000 m to the right and 2.000 m ahead, an angle of arctan(1.000 / 2.000) = 26.6° off-axis. Now compare three readings: your hand result, the vendor viewer's value for that pixel, and rviz's.

• Your numbers match the viewer; rviz differs. The camera side is correct, and the ROS driver or transform chain is wrong. Apply the frame and unit fixes below.
• Your numbers match rviz; the viewer differs. The viewer is applying an extra correction or a different stream. See the next section but one on the viewer lying.
• Neither matches your arithmetic. The intrinsics you used belong to another stream or unit. Recheck fx, cx, cy against the exact image you measured.

Fix the frames: the optical-frame rotation rviz expects

The ROS frame conventions define two axis sets: body frames run x forward, y left, z up, while optical frames run z forward, x right, y down. Deprojection emits optical-axes points, so the cloud must carry the optical frame_id, or a static transform must connect the two. The canonical connection is zero translation with yaw −90° and roll −90° (−1.5708 rad each, pitch 0), conventionally published between camera_link and camera_optical_frame — frame names vary by driver, so check the frame_id yours actually publishes:

0 0 0 -1.5708 0 -1.5708 camera_link camera_optical_frame

Check whether the driver already publishes an optical frame before adding your own; duplicate static publishers make the tree ambiguous. Verify with tf2_echo camera_link camera_optical_frame, point rviz's Fixed Frame at a stable parent such as map or odom, and re-display: the floor should read as a floor again. The full axis algebra is in transform depth point clouds between optical frame and camera frame, and a cloud that collapses flat in XY is usually the same mix-up (fix the optical frame).

Fix the units and the silent scale errors

A millimetre stream in a metre pipeline multiplies every coordinate by 1,000: the 2 m wall from the worked example spans 2,000 units, rviz's default 10 m grid drowns inside it, and the scene reads as crushed or absurdly tall. Convert once at the driver — divide the depth channel by 1,000 or republish as 32FC1 — rather than patching each consumer. The unit-conversion fix for scaled point clouds walks the same repair.

Next, match intrinsics to stream. Deprojection must use the calibration of the exact image you measured: rectified and raw streams carry different maps, and aligned-depth conventions differ per camera (rectified vs raw stereo IR images: which intrinsics apply, which camera to deproject with). Mixing them bows straight walls — the row-three pattern in the table.

Presets are the last silent mover. A depth preset changes disparity filtering, so the viewer and the published topic can disagree; set the same preset on both paths and re-run the pixel test.

When the vendor viewer is the one that lies

Sometimes the usual answer is backwards:

• After a mode or preset change, a viewer can keep rendering with cached calibration until restarted. Restart it before trusting its verdict.
• Heavy clouds lag, and rviz drops or delays points until the geometry looks torn. Cut density at the source first (reduce the point cloud before processing lags).
• Saved or exported clouds can differ from the live render by design, through scaling, decimation or colour mapping applied on export (why exported point clouds differ from the viewer render).
• Fused scenes inherit each camera's extrinsics, so a wrong inter-camera transform twists the merged cloud even when every single camera is clean (multi-camera point cloud fusion with depth camera extrinsics).

If the vendor viewer itself warps a plain wall at a taped distance, the calibration is genuinely in question, and no ROS repair will fix it.

Prevention checklist

At set-up

• Confirm the depth image, camera_info and cloud headers all name the frame the deprojection assumes.
• Publish the static optical-frame transform in the launch file, not by hand in rviz each session.
• Set the encoding deliberately: 32FC1 metres unless you convert on purpose.

Before you trust a bench

• Hand-deproject the centre pixel against a taped wall and record all three numbers.
• Compare the vendor viewer and rviz on the same wall and note the offset.

After any change

• Re-run the pixel test after firmware, preset or driver updates; settings drift silently.
• Watch tf warnings when the camera moves: a delayed transform on a moving base bends the live view.

Choosing hardware you can verify this way

The pixel test needs only a depth camera whose driver exposes the raw depth stream with the intrinsics used to build it. The stereo and structured-light types in the 3D sensors category both qualify, and the camera finder filters them by type and interface:

The projected structured-light 3D camera projects its own pattern, which keeps depth dense on textureless parts. The laser galvanometer stereo camera steers its laser line across the scene instead of flooding it.

Running both viewers side by side takes LAN headroom on the bench computer:

The MV-VC5525-256G70-NN vision controller combines PoE camera ports and 5+ LAN ports with 32 GB of memory and eight built-in light channels, so depth cameras publish over the same cables that power them. Browse the wider vision controllers range if the cell needs more camera ports.

To plan the whole chain, start at the vision system configurator, skim the 3D vision topic page for related fixes, or contact the applications team with your wall-test numbers and we will help you read them.