Inverse distortion coefficients: why rectified depth looks warped

Rectification is supposed to make depth and infrared frames geometrically clean: straight edges straight, flat surfaces flat. When the opposite happens and the rectified image bends more than the raw one, the usual culprit is a set of inverse distortion coefficients published where standard software expects forward ones. This article explains the two coefficient sets, shows why the plumb_bob model only makes sense in one direction, and gives you a flat-target test that settles the question in minutes. It is written for anyone wiring a depth or structured light camera into standard vision pipelines.

What rectification actually does

A raw depth or IR frame carries lens distortion: straight lines in the scene arrive bent, typically with barrel distortion near the edges. Rectification resamples the image so that pixels obey the ideal pinhole model described by the camera matrix. Every standard node downstream, from stereo matching to point cloud generation, assumes this has already happened correctly.

The recipe for that resampling lives in the distortion coefficients, published as the D vector alongside the camera matrix. If D describes the distortion in the wrong direction, the resampling pushes distortion in instead of pulling it out, and the result looks worse than the raw frame.

Forward versus inverse coefficients

The standard distortion model (the plumb_bob convention used in most robot vision frameworks) is a forward model. Given an ideal, undistorted pixel position, the coefficients predict where that point actually lands on the distorted sensor. Undistortion inverts that mapping numerically: it takes each measured pixel and solves for the ideal position that would have produced it.

The inverse coefficients are a different parameterisation. They directly map distorted measured pixels to undistorted ones, which is convenient if your code wants a one-step lookup, but they are not what the plumb_bob field means. The two sets look deceptively similar: five numbers of similar magnitude, with some signs flipped. Swap them and nothing fails, no error is thrown, and the image is simply warped the wrong way.

This is the forward map: ideal coordinates in, distorted sensor coordinates out. A driver that stores an inverse model and prints it into D makes every standard undistortion routine run this equation backwards, so distortion is added on top of the raw frame instead of removed.

Why the numbers look so similar

For mild distortion, inverting the radial polynomial roughly flips the signs of the tangential terms and slightly rescales the radial ones. Comparing a device's stored forward and inverse vectors, you will typically see matching magnitudes with a few signs reversed. That is exactly why the mistake is hard to spot by eye in the metadata: five plausible-looking coefficients either way.

The visual signature is easier. If a scene full of straight lines (door frames, shelves, a checkerboard) is straight in the raw image and bowed in the rectified image, with the bow direction matching the raw lens distortion rather than opposing it, the undistortion ran in reverse.

A flat-target verification you can run today

The decisive test needs no special tooling:

1. Print or display a checkerboard or grid and mount it as flat as you can; rigid foam board works well.
2. Capture the raw depth or IR frame and the rectified frame of the same target, in the same session.
3. Fit lines or fit the checker corners in both images and measure how much each grid line deviates from straight, in pixels.
4. Compare. Correct rectification should leave flat-panel edges straighter than the raw image, typically reducing line deviation by several times.

If the rectified deviation is larger than the raw deviation, the published D vector is not a forward plumb_bob set. Confirm by reading the coefficients the device itself reports through its native API, where the vendor documents which vector is which, and compare term by term with what your driver publishes.

Fixing the pipeline without breaking everything else

Once you have confirmed an inversion, resist the temptation to hand-edit coefficients in one node. Every consumer, from undistortion nodes to point cloud builders to the extrinsics chain described in the depth to color registration guide, reads the same camera_info. Fix it at the source:

• Check the driver layer first. The fault is often in the code that copies calibration out of the device, and a corrected release may already exist.
• Look for a configuration option that selects forward versus inverse output; some drivers publish both on separate topics.
• If you must patch it yourself, invert the polynomial numerically and validate with the flat-target test before deploying, then note the change prominently, because anyone re-running your calibration later needs to know.

Never silently replace a stored device calibration with your own numbers. If the underlying intrinsics are genuinely wrong, recalibrate properly using the vendor tooling and verify the result the same way.

What this means when choosing a depth camera

Modern depth and laser profiler products handle calibration and rectification internally, on-device or in a maintained SDK, so the raw-versus-rectified mismatch that plagued early structured light and time-of-flight devices is far less common. The verification method above is still worth running on any new camera: it costs minutes and catches reversed extrinsics, wrong model selection and stale calibrations too.

For profile-based 3D measurement, a device such as the MV-DP4180-03P 3D laser profile sensor outputs calibrated profiles directly rather than raw distorted frames. Where a full surface map from one shot is the goal, a projected structured-light camera like the MV-DPS200P-02 likewise ships its calibration as part of the supported SDK, so downstream tools consume consistent, documented parameters.

Related geometry checks

Inverted coefficients are one of several ways a rectified pipeline quietly lies to you. The same flat-target discipline catches most of them, and these guides cover the neighbouring failure modes:

• If you are debugging an image that never went through rectification, the interpretation of distortion coefficients for an unrectified camera image explains how to read D and K directly.
• For stereo pairs, the live stereo rectification for calibrated camera video walkthrough shows how rectification parameters interact with the rotation matrix R.
• When disparity maps misbehave after a calibration change, the stereo disparity map troubleshooting guide lists the checks in order of probability.
• Registration between depth and colour frames adds another layer; the comparison of raw versus rectified frames in depth to color registration explains which stream each convention applies to.
• If your undistorted image shows black borders at the edges, that is a scaling and cropping question, not a distortion one; see black borders after undistortion.

Quick checklist before you trust a rectified depth stream

• Raw grid straight, rectified grid bent: expect inverted or misdirected coefficients.
• Verify the published D vector against the device's own reported calibration, term by term.
• Fix at the driver or configuration layer, never in a single downstream node.
• Re-run the flat-target test after any firmware or SDK update.
• Check the depth-to-colour chain afterwards, since registration inherits the same parameters.

If you are selecting a 3D sensor and want the calibration behaviour confirmed for your application, browse the 3D sensors range or use the camera finder. Our applications team can review your pipeline and sensor choice before you order; the system configurator is a good starting point, and you can always reach us through the contact page.