Offline image rectification from stored calibration data

You have saved colour frames from a calibrated stereo or depth camera and the unit is no longer on the bench. You still want rectified frames, because every downstream step, from disparity matching to pixel-to-3D lookup, expects the epipolar lines horizontal. Offline image rectification solves this: you build a remap table once from stored calibration, then apply it to every saved frame with two lookup arrays and no camera connected. This guide walks through the table, the one parameter that trips most people up, and the blank-border artefact that follows.

Before you start, make sure you have captured the right calibration in the first place. If you are producing those intrinsics yourself from checkerboard pairs, the paired-image procedure in our stereo camera calibration from paired checkerboard images guide is the foundation. If the camera firmware supplies both raw and rectified intrinsics, store both sets per unit; they differ.

What you need

• The intrinsics of the raw (unrectified) stream: focal lengths fx, fy, principal point cx, cy, and distortion coefficients.
• The intrinsics of the rectified stream, as the driver would produce them.
• The extrinsics (rotation and translation) between the two frames, from the camera's own calibration or from your stereo calibration.
• A saved frame, its width, height and pixel format.

If your coefficients come from a different tool or a different camera SDK, convert them rather than guessing: our guide on camera intrinsic parameter conversion for depth and texture images lists the conventions that silently break a remap.

Sensors that publish this data natively make the storage step simple. A projected structured-light camera such as the MV-DPS200P-02 exposes its stream calibration through the driver, and a laser galvanometer stereo camera like the MV-DLS300P-04 does the same for its rectified outputs. Grab and persist those values once per unit, in a versioned file, before the camera ever leaves the bench.

The remap table, parameter by parameter

A rectification table maps each destination pixel to a source location. Its three inputs are:

1. The destination stream intrinsics you are emulating, with distortion removed. For a rectified colour output, that is the rectified colour stream's own intrinsics, not the raw ones.
2. The extrinsic transform between the two frames.
3. The source stream intrinsics, unrectified, with its distortion model, because the lookup must know how to undo the lens.

The geometry is a standard pinhole reprojection. For each destination pixel you compute a normalised ray, rotate it into the source frame, and project it back through the source intrinsics:

The scale of the translation only affects the result through its ratio to the scene distance, so a unit-vector extrinsic with a baseline value is enough for the rotation-dominated part of the remap.

The extrinsic direction: the one that gets it wrong

Here is the mistake that costs an afternoon. Intuition says: I am going from colour to rectified colour, so I need the colour-to-rectified-colour transform. The table function expects the opposite. It walks destination pixels back to the source, so it needs the transform from the rectified frame to the raw frame.

Symptom of the wrong direction: the output looks recognisably like the scene but sheared, rotated by a small angle, or shifted horizontally, even though the intrinsics are verifiably correct. Test: reverse the extrinsic, recompute the table, and compare. If the corrected version's epipolar lines sit level on a checkerboard capture, the direction was the problem. The same direction logic applies when you later align depth to colour offline, and our depth to colour registration: raw versus rectified frames guide covers the frame-pairing rules there.

Worked example: building the table in code

The flow, in pseudocode independent of any SDK:

1. Load raw intrinsics, rectified intrinsics and the extrinsic (rotation R, translation t) from your stored calibration file.
2. Compute the inverse direction: R_inv = Rᵀ, t_inv = −Rᵀ t. Pass R_inv and t_inv to the table builder together with the rectified destination intrinsics and the raw source intrinsics.
3. The builder returns two maps, one per axis, sized to the rectified resolution. Interpolation is bilinear; store them as floating point.
4. For each saved frame, call a remap with the two maps and a border constant.
5. Verify once on a checkerboard or structured target, then batch.

Do the arithmetic on buffer sizes before you start. For a 12 MP frame, 4000 × 3000 pixels at 3 bytes per pixel gives 36,000,000 bytes of output per frame, and the two float maps add 4000 × 3000 × 4 × 2 ≈ 96 MB, computed once and reused. If you want to sanity-check the resolution against your field of view first, the camera resolution calculator gives you the pixel count needed for a target detail size.

Blank corners and border handling

A rectified frame almost always contains a wedge with no valid data, typically at a corner or edge, because the rotation pulls content from outside the original sensor area or pushes it off. Three ways to deal with it:

• Fill with a border constant and carry a validity mask alongside the image.
• Crop to the common valid region shared by all frames of that unit; this is what most stereo pipelines do.
• Leave the border and make the consumer ignore pixels outside the valid polygon.

Resizing is a separate question. If the rectified and raw streams share the same resolution, a fixed-size output buffer is correct and no resize is needed after computing the table. Resize only when the destination resolution differs, and size the buffer from the rectified intrinsics, not from habit.

If your rectified output feeds a stereo depth pipeline, artefacts in the remap show up downstream as streaks and holes; the checks in our stereo disparity map troubleshooting after rectification guide help you tell a calibration problem from a rectification problem. And if you ever see the raw image itself looking wrong, start with the distortion terms using our guide on interpreting distortion coefficients of an unrectified image before blaming the table.

Choosing hardware that makes offline rectification easy

The offline route works for any calibrated camera, but some sensor classes reduce the work. Stereo and structured-light cameras publish per-unit calibration you can dump to file. Laser profilers deliver rectified profiles directly and skip the step entirely. Browse the 3D sensors category or use the camera finder to filter by type, and compare against plain area scan cameras if you would rather run your own calibration on a standard sensor. The system configurator walks the full camera, lens and light selection if you are specifying a new cell.

Checklist before you rely on offline results

• Store raw and rectified intrinsics per unit, in a versioned, timestamped file.
• Record the extrinsic with an explicit direction label, so the inverse is unambiguous later.
• Verify rectification once on a checkerboard: matched corners share a row within about one pixel.
• Confirm output buffer width, height and stride against the rectified intrinsics before the first remap.
• Decide border policy (mask, crop or constant) and document it for the consumer of the frames.

When offline rectification is the wrong tool

If frames arrive live and you own the driver, rectify in the acquisition path instead and save only rectified data; you carry one less calibration artefact. If your inspection tolerates a few pixels of skew, skip the remap and calibrate your measurement step against the raw image. And if you are re-projecting pixels into 3D rather than just flattening epipolar lines, the pixel-to-point mapping in our bounding box depth: mapping image pixels to 3D points article is the piece you actually need.

To check that a saved calibration matches the unit it came from, compare the reprojection residuals against the guidance in camera extrinsic calibration accuracy beyond reprojection error. If you are specifying hardware for a new offline-friendly pipeline and want help matching sensor class to the task, contact us and the applications team will work through the requirements with you.