3D visionExplainer

Non-zero distortion in camera_info: is image_raw rectified

Non-zero distortion in camera_info means image_raw is not rectified. Learn to read D, K, R and P, compute the pixel shift, and rectify with one remap.

LumaSpec Vision11 min read Share on LinkedIn

On this page 9 sections
  1. 01The three-field test that decides it
  2. 02What D, K, R and P each mean
  3. 03How far non-zero distortion moves a pixel
  4. 04How to rectify image_raw yourself
  5. 05Is aligned_depth_to_color rectified?
  6. 06When the usual reading is wrong
  7. 07Checks before you trust the calibration
  8. 08The decision for each case
  9. 09Questions

You integrated a depth camera, opened the camera_info topic next to the color image_raw, and the D vector is full of numbers while the topic name says raw. The question is concrete: is that stream rectified, is the aligned depth stream rectified, and what do you do about it? This explainer gives you the three-field test that settles it in seconds, the plumb_bob math with a worked pixel-shift calculation from a published camera_info, and the rectification path for color and depth.

The reading is identical whatever publishes the message: depth and 3D sensors, time-of-flight units, or area scan cameras behind a standard vision driver. Laser-based sensors hold their stated laser class only under intended use, so leave interlocks and housings in place when you reposition one.

3D sensorsHikrobot MV-DT01WMU ToF sensing cameraMV-DT01WMU
Type
Depth / ToF camera
Price on requestView and quote
3D sensorsHikrobot MV-DLS300P-04 Laser galvanometer stereo cameraMV-DLS300P-04
Type
Stereo / structured light
$16,579.00 · quotedView and quote

The three-field test that decides it

Open camera_info and read three things in order: D, R, and the upper-left 3 × 3 block of P. A typical published example for a 1280 × 720 color stream looks like this:

distortion_model: "plumb_bob"
D: [0.16470, -0.49777, -0.00217, 0.00126, 0.45043]
K: fx 912.10  fy 912.27  cx 654.50  cy 351.96
R: identity
P: upper 3x3 equal to K, fourth column zero
binning_x: 0   binning_y: 0
do_rectify: False

All three readings agree, and any one of them alone is enough:

  • D is non-zero, so the stream carries lens distortion.
  • R is the identity and the upper 3 × 3 of P equals K, so nothing has been re-projected; this is the measured calibration of the imager as it sits.
  • The pixels are untouched sensor output. image_raw is not rectified, and the driver never claimed otherwise.

Raw here means minimally processed sensor data, in the standard sense of a raw image format: no geometric correction has been applied. A rectified stream publishes the opposite: D as five zeros, and P rebuilt for the rectified image plane, where cx and cy may move to keep valid pixels in frame. In a rectified stereo pair, R leaves identity and the right imager's P carries the baseline in its first row, fourth column: P[0][3] = -fx × b, with b the baseline.

The topic name is the second confirmation — image_rect and image_rect_color mean rectified — and when D is zero the two streams look pixel-identical, which is why both exist. That case is covered in image_raw versus image_rect_color when distortion is zero.

What D, K, R and P each mean

For the plumb_bob model, D holds five coefficients in a fixed order: k1, k2, t1, t2, k3. The first, second and fifth are radial — they stretch the image along rays from the principal point. The third and fourth are tangential — they come from a lens sitting slightly off square to the sensor. K holds the pinhole part: focal lengths fx and fy in pixels, and the principal point cx, cy where the optical axis meets the sensor. R and P describe life after rectification: R is the rotation applied to the camera frame, P is the 3 × 4 matrix to project with once rectified.

Field What it holds Raw stream Rectified stream
distortion_model name of the coefficient set in D plumb_bob same name, coefficients unused
D k1, k2, t1, t2, k3 non-zero five zeros
K fx, fy, cx, cy measured lens and imager the values before rectification
R 3 × 3 rectifying rotation identity on a single camera rotation that aligns a pair
P 3 × 4 projection after rectification upper 3 × 3 equals K new matrix, may move cx and cy
binning_x, binning_y decimation the matrix already includes 0, 0 at full resolution 0, 0

Two fields deserve one line each. do_rectify belongs to the roi sub-message; with an empty roi — x_offset 0, y_offset 0, width 0, height 0, as in the example — it is inactive and says nothing about your image. binning_x and binning_y record decimation: at binning 2 a correct driver halves fx, fy, cx and cy, because every pixel then covers twice the angle.

How far non-zero distortion moves a pixel

This is the number that decides whether rectification matters for you. The plumb_bob model maps an ideal pinhole pixel to where the lens actually puts it:

x = (u - cx) / fx y = (v - cy) / fy r2 = x*x + y*y s = 1 + k1*r2 + k2*r2*r2 + k3*r2*r2*r2 xd = x*s + 2*t1*x*y + t2*(r2 + 2*x*x) yd = y*s + t1*(r2 + 2*y*y) + 2*t2*x*y u_dist = fx*xd + cx v_dist = fy*yd + cy

Run it on the far corner of the example, pixel (1280, 720), with the K and D above:

  • x = (1280 - 654.50) / 912.10 = 0.6858
  • y = (720 - 351.96) / 912.27 = 0.4034
  • r2 = 0.6858² + 0.4034² = 0.4703 + 0.1627 = 0.6331
  • r4 = 0.6331² = 0.4008, r6 = 0.6331 × 0.4008 = 0.2537
  • s = 1 + 0.16470 × 0.6331 - 0.49777 × 0.4008 + 0.45043 × 0.2537 = 1.0191
  • x × y = 0.6858 × 0.4034 = 0.2766
  • xd = 0.6858 × 1.0191 - 2 × 0.00217 × 0.2766 + 0.00126 × (0.6331 + 2 × 0.4703) = 0.6997
  • yd = 0.4034 × 1.0191 - 0.00217 × (0.6331 + 2 × 0.1627) + 2 × 0.00126 × 0.2766 = 0.4097
  • the corner lands at u = 912.10 × 0.6997 + 654.50 = 1292.7 and v = 912.27 × 0.4097 + 351.96 = 725.7

The straight-line position (1280, 720) is displaced to about (1293, 726): 12.7 px sideways, 5.7 px down, 14 px overall — 1.1% of the image width. At the centre the shift is exactly zero, because s = 1 where r = 0; that is why a centred target looks perfect while a corner target misses. Fourteen pixels is noise to a detector or classifier, and fatal to stereo matching, structured-light decoding, or any pixel-to-millimetre conversion. To see the whole field rather than one corner, the lens distortion vector map method plots this displacement per pixel across the frame.

How to rectify image_raw yourself

Rectify downstream; the driver has handed you everything the job needs. In the order that matters:

  1. Take K and D exactly as published. The example runs at binning 0, so no scaling applies; at binning 2 a correct driver has already halved the matrix for you.
  2. Choose the output matrix. Keep K when downstream code expects the same field of view and a few corner pixels may go invalid; choose a shorter-focal-length matrix when every source pixel must land inside the frame.
  3. Build the undistort map once, from K, D, R and the output matrix. Every vision library exposes it as a precomputed per-pixel lookup that depends on the calibration, never on the frames.
  4. Remap each frame with that map. Bilinear interpolation is right for color; the per-frame cost is one lookup per pixel, about a millisecond for a 1280 × 720 image on a desktop CPU.
  5. Treat the output matrix as your projection matrix from here on. If you republish camera_info, it belongs in P with D zeroed, so the next consumer does not rectify twice.

The map-and-remap code with the matrices in this order is walked through in rectifying color images yourself with a remap and calibration. For stored calibration files and offline batches, see offline image rectification from stored calibration data; for continuous streams, live stereo rectification for calibrated camera video covers the pipeline.

Is aligned_depth_to_color rectified?

Aligned depth is depth re-projected onto the image plane of the stream it was aligned to, and it inherits that geometry completely. If the color camera_info carries non-zero D, the aligned depth matches the raw color image pixel for pixel, distortion included. The answer is therefore the same as for color: aligned_depth_to_color is rectified exactly when the color image it was aligned to is rectified. How to confirm which frame you actually hold is the subject of checking depth image_raw registration against RGB; the memory layout of aligned streams is in aligned depth streams: data layout and intrinsics.

Two consequences follow:

  • Rectify both streams or neither. A rectified color image beside a raw-aligned depth pair mis-registers by the same 14 px class of error computed above.
  • Never fix depth with a color-style remap and bilinear interpolation. Interpolating across a depth edge averages foreground with background and manufactures flying pixels at every silhouette. Deproject each valid depth pixel with the color K and D, then project with the rectified matrix — the choice of matrix is covered in aligned depth point cloud intrinsics: which camera to deproject with. A nearest-neighbour remap with the same map is the low-effort fallback, at half a pixel of edge accuracy.

For depth computed from a stereo pair, rectification happens inside the camera — disparity exists only on rectified infrared images — but the published result is mapped back onto the color frame as published, so the reading above still decides.

When the usual reading is wrong

  • D is five zeros on a stream named image_rect. The work is done; undistorting again is an identity map that costs a millisecond per frame. The same camera can publish image_raw with non-zero D and image_rect with zeros side by side — that is the design, not a contradiction.
  • D is zeros everywhere but straight edges bow on screen. The driver shipped a placeholder calibration. Treat every matrix as unknown and calibrate; camera info full of zeros walks the process, and image distortion low or high in camera calibration settles how many coefficients to enable.
  • The resolution changed under you. Enable 2 × 2 binning or an ROI in the driver and K must follow: fx, fy, cx and cy all halve at binning 2. Guard with cx against width / 2 — 654.50 against 640 confirms the 1280-wide example is full resolution; 654.50 on a 640-wide image means the matrix was never rescaled.
  • A wide-angle lens with plumb_bob coefficients. Five terms cannot fit a strong fisheye. distortion_model names the coefficient set; if the model cannot represent the lens, D is meaningless rather than approximate, and remapping with it bends lines that were straight.
  • Firmware folklore. "This product line rectifies in firmware" is true until the next firmware. The three-field test survives every update — read D, not the release note you half-remember.

Checks before you trust the calibration

Thirty seconds with the numbers catches most bad calibrations before they cost a commissioning day:

  • Principal point near centre. cx = 654.50 against width / 2 = 640, cy = 351.96 against 360: a few pixels off is a normally mounted lens, tens of pixels off means the wrong resolution or the wrong matrix.
  • fx and fy within a fraction of a percent. 912.10 against 912.27 is 0.02% — square pixels and a sound fit. A gap near 1% points to a poor calibration or anisotropic scaling upstream.
  • Focal length consistent with the field of view. hfov = 2 × atan(width / (2 × fx)) = 2 × atan(1280 / 1824.2) ≈ 70°. A lens that is obviously narrower or wider than that means the matrix belongs to another imager.
  • A straight edge through a corner. A bow near the computed 14 px confirms D describes this lens; a bow with D at zero means the calibration is stale.
  • One imager, one set of numbers. Every stream from the same imager must agree on K and D; if color and its aligned twin differ, one was rescaled and the other was not.

The decision for each case

  • Non-zero D, R identity, P upper 3 × 3 equal to K — the example, and the common case: image_raw is raw. Rectify downstream for stereo matching, structured light, hand-eye work and measurement; skip it for display, detection and classification.
  • D all zeros on a rectified-named stream — already rectified. Use P as the projection matrix and spend the millisecond elsewhere.
  • aligned_depth_to_color — it follows the color stream it was aligned to. Keep the pair consistent, and move depth by reprojection or nearest neighbour, never by an interpolated remap.
  • D all zeros but edges bow — placeholder calibration. Calibrate, republish, then apply the first case.

If the camera is still on the bench rather than on the network, the camera finder narrows area scan models by sensor format, interface and resolution, and the system configurator walks camera, lens and light together. Free expert support comes with every order if you want the rectification pipeline reviewed against your parts list.

CamerasArea scan cameras

GigE, USB3, 10GigE and CoaXPress

1,287 models · from $149.00

Questions

Does do_rectify: False mean my image is not rectified?

No. do_rectify sits inside the roi sub-message of camera_info and only matters when a region of interest is set; with an all-zero roi it is unused. The flag describes driver behaviour, not your pixels. Read D instead: non-zero D with R at identity and a P whose upper 3 × 3 equals K means the stream is raw, whatever do_rectify says. If D is five zeros, the stream is rectified.

Why are fx and fy slightly different if the pixels are square?

Calibration noise. Each is fitted independently from the same target images, and a fraction of a percent of spread is normal: 912.10 against 912.27 is 0.02%. Use both values exactly as published rather than averaging them; the undistortion and projection math expect the matrix that was fitted, and collapsing it to a single focal length moves corner geometry by whole pixels at wide fields of view.

Can I apply the color stream's K and D to the infrared or depth streams?

No. Each imager has its own intrinsics and its own distortion, which is why the driver publishes a separate camera_info per stream. The depth stream's geometry is the rectified stereo pair of the infrared imagers, not the color matrix. To relate a depth pixel to a color pixel, use registration or an aligned stream rather than copying matrices between topics.

What does rectification cost per frame?

Almost nothing once set up. The undistort map depends only on K, D, R and the output matrix, so you compute it once at start-up. Per frame the work is one lookup per pixel; a 1280 × 720 color frame remaps in about a millisecond on a desktop CPU, faster on a GPU. Depth is the exception: reproject it rather than interpolate it, or accept nearest-neighbour accuracy.

Should I rectify before or after my detection network?

After, in most cases. Object detectors tolerate a 1% corner stretch, and a full-frame remap either crops or blanks a thin wedge at the edges. Rectify before anything geometric: stereo matching, structured-light decoding, hand-eye calibration, or converting pixels to millimetres. If the network was trained on raw frames, keep feeding it raw frames and rectify only the measurements you extract from its output.