Passive infrared image registration: when depth is needed

A passive infrared image can look distorted even when its pixels contain no depth information. That does not prevent lens correction. The difficulty appears when you want the same pixels expressed in a separate camera's viewpoint, or as positions on a screen.

For infrared spot localization, choose the coordinate system before choosing a transformation function. This explainer separates lens undistortion, planar mapping and depth-assisted reprojection, then shows how to estimate the alignment error each approach can leave behind.

Choose the transformation by the output you need

The word “transformation” covers several different operations. They do not require the same inputs.

Start with the simplest operation that answers your actual measurement question. Straightening an image does not require constructing a point cloud. Conversely, a visually straight image does not automatically align with another camera.

The separate guide to thermal and color image fusion covers overlay design and preserving measurement data. Here, the focus is deciding which geometry makes an overlay or spot coordinate valid.

Lens undistortion corrects directions, not distances

A calibrated central-camera model associates each image pixel with a viewing ray. Lens distortion changes the relationship between pixel coordinates and ray direction. Undistortion corrects that relationship without deciding where along the ray the observed object lies.

The usual inputs are the camera intrinsics, distortion coefficients and the image dimensions used during calibration. A wide-angle lens may need a fisheye model rather than a conventional radial-and-tangential model. Coefficients from one model cannot simply be inserted into another.

Conceptually, the calculation is:

Here, p_IR is the measured infrared pixel, K_IR contains the source intrinsics, and D_IR represents its distortion model. The output intrinsics determine how the corrected image is framed. Neither step requires a scene-distance measurement.

For a spot detector, you can often undistort only the detected center instead of resampling the entire image. This reduces processing and avoids changing every intensity sample. Test that approach against distorted spot shapes near the image boundary, where detection bias can become significant.

Use interpreting distortion coefficients to check the model conventions. Then use cropping versus field of view during undistortion when deciding the corrected image boundaries.

A separate viewpoint generally requires depth

Two cameras mounted beside each other observe an object along different rays. Where an infrared ray lands in the color image depends on the object's distance.

For a central-camera model, normalize the infrared ray so its third component is one. With axial depth measured along the infrared camera's optical axis, reprojection becomes:

R and t describe the rigid transform from infrared coordinates to color-camera coordinates. Z_IR is axial depth, not necessarily the distance measured along the ray. Confusing those definitions creates errors away from the image center.

Without depth, the corresponding color pixel is generally constrained to an epipolar line rather than a unique location. A software wrapper cannot resolve that missing geometric information merely by exposing another function.

There are useful exceptions. Known object geometry can supply the missing surface intersection. With coincident optical centers and pure relative rotation, depth cancels from the ideal projection. At sufficiently long distances, a small camera separation may also produce an acceptable approximation—but calculate the error before accepting it.

If a depth image supplies the geometry, verify its coordinate system before indexing it. Mapping image pixels to 3D points explains why matching array coordinates alone is insufficient.

Depth-assisted warping also needs visibility handling. Several source points may project into one output pixel, while other output pixels receive nothing. Keep the nearest visible surface where appropriate, and mark unsupported regions rather than treating interpolated holes as measured geometry.

Use a planar mapping for a fixed screen or work surface

When the infrared feature lies on a known flat surface, a homography can map its undistorted pixel position to coordinates on that surface or in another camera's image.

This is often the practical choice for locating a spot on a fixed screen. The plane supplies the missing geometric constraint, so a separate depth measurement is unnecessary during each frame.

The matrix H maps between the chosen planar coordinate systems. The scale s is removed by dividing the first two output components by the third.

Use at least four well-spread point correspondences, with no three collinear. In practice, collect more points across the working area and reserve additional positions for validation. Correct lens distortion first; a homography does not generally represent nonlinear lens curvature.

A suitable calibration sequence is:

1. Fix the camera mounting, focus, image dimensions and target plane.
2. Present recognizable reference positions across the plane, including near its boundaries.
3. Measure their infrared coordinates and corresponding target coordinates.
4. Fit the mapping and evaluate separate validation positions.
5. Repeat after any mounting, focus or image-geometry change.

Do not judge the result using only the points used to fit it. A low fitting residual can hide poor coverage or a systematic error between reference positions.

The mapping stops being valid when the feature leaves the plane. A raised component, bowed surface or moving handheld emitter can all violate that assumption.

Estimate parallax before accepting a fixed mapping

A fixed-distance approximation may be adequate when the depth variation is small compared with the working distance. A simplified rectified stereo model gives a useful first estimate:

Here, f_px is focal length expressed in pixels, B is camera baseline, Z0 is the calibrated plane distance, and Z is the actual point distance. Use consistent length units. This estimates parallax in simplified geometry, not the complete calibration error.

Suppose a hypothetical setup has a focal length of 800 pixels, a baseline of 60 mm and a reference plane at 1,000 mm. A feature moves to 1,050 mm:

If your permitted error is one pixel, that fixed-plane approximation already fails before including calibration residuals, spot-detection error or mechanical movement.

This calculation also explains why a mapping can look excellent on a wall but fail on objects held in front of it. More calibration points on the wall will not remove the depth-dependent error.

Separate spot localization from pointing direction

An infrared camera observing an emitter measures the emitter's apparent position. That does not generally reveal where the emitter is aimed.

For example, translating a handheld emitter while maintaining its orientation changes its image location. Rotating it without moving its emitting point may leave that location almost unchanged. One observed point therefore cannot uniquely determine pointing orientation.

Choose a measurement arrangement that matches the interaction:

• Spot on a known surface: detect the surface spot and map it to planar coordinates.
• Emitter position in space: use depth, multiple calibrated views or known geometric constraints.
• Pointer orientation: use suitable tracked geometry or another orientation measurement, then intersect its pointing ray with the screen plane.

Also distinguish passive acquisition from wavelength. “Passive” means the camera is not supplying the illumination for that acquisition; it does not identify the spectral band. Reflected near-infrared signals and long-wave thermal emission require different detector responses.

Verify wavelength sensitivity, filters, exposure behavior and surface response before designing the geometry. A transformation cannot recover a spot that the detector does not see. Do not increase emitter output beyond applicable eye-safety limits to compensate for an unsuitable sensor.

Choose hardware for the signal and the geometry

Browse infrared cameras by spectral suitability before comparing interfaces or resolution. Use the camera finder to narrow the acquisition options, then confirm the specific measurement and calibration requirements.

The LS-IRC-0044 is a thermal camera in the LWIR category. It illustrates a thermal acquisition option, not evidence of suitability for a reflected near-infrared control signal.

If changing scene geometry requires measured depth, consider the 3D sensors category separately. The MV-DPS200P-02 is identified as a projected structured-light camera. That classification alone does not establish alignment, synchronization or software compatibility with another camera.

For any proposed combination, request confirmation of overlapping working volumes, accessible calibration data, depth conventions and timestamp behavior. Also test whether projected illumination interferes with the infrared signal you want to detect. Neither a shared interface nor physically adjacent mounting establishes a calibrated relationship.

Check sampling independently of registration. For a hypothetical 400 mm-wide working plane, suppose you want a 2 mm spot to span four pixels:

That is a sampling target, not a guarantee of centroid accuracy. Focus, saturation, background contrast, noise and spot shape still matter.

Keep raw measurement data separate from display warps. Interpolation may be useful for an overlay, but it changes sample values and can mix signals across object boundaries.

Validate the complete acquisition chain

A correct spatial model can still fail when the source frames represent different moments. Sequential infrared and depth captures are a static-scene approximation, not simultaneous measurements.

Suppose an object moves laterally at 200 mm/s and the captures are separated by 20 ms. Its displacement is 200 × 0.020 = 4 mm. At a sampling density of two pixels per millimeter, that becomes an eight-pixel mismatch before geometric errors are considered.

Check exposure intervals as well as frame timestamps. Checking depth and color timestamps helps distinguish acquisition timing from software delivery time.

Use this acceptance checklist before committing to hardware:

• Define the output: corrected infrared pixels, screen coordinates, color pixels or spatial points.
• Confirm spectral response: the actual signal must be detectable with usable contrast.
• Match calibration to image geometry: account for resizing, cropping, binning and rectification.
• Test withheld locations: include corners, boundaries and the full required depth range.
• Record meaningful errors: report median, high-percentile and worst observed localization errors, not only averages.
• Test motion: use the intended speed and capture sequence rather than only a stationary target.
• Reject unsupported coordinates: handle missing depth, occlusions and positions outside the calibrated plane explicitly.
• Check repeatability: repeat after warm-up, remounting and any focus adjustment relevant to the installation.

For altered image sizes, follow converting camera intrinsic parameters rather than reusing an unchanged calibration matrix.

Choose undistortion when you need corrected rays, planar mapping when the surface is known, and depth-assisted reprojection when viewpoint translation meets varying scene depth. Before ordering, send the wavelength, working distances, permitted error and capture timing through application support so the proposed components can be checked against the actual measurement.