Rectified image pose estimation: choosing PnP intrinsics

A chessboard can look straight after rectification while its estimated pose remains wrong. The usual cause is inconsistent geometry: corners come from one image, intrinsics describe another, or the reported pose uses an unexpected camera frame. This procedure is for engineers estimating target pose from a rectified amplitude image. You will identify the correct camera matrix, account for image transformations, validate the pose, and convert it back to the original optical frame when necessary.

1. Identify the rectified camera model

Rectification creates an image associated with a chosen output camera model. For a rectified pinhole output, supply that output camera matrix to the PnP solver, together with zero distortion coefficients in the solver's accepted format.

Do not reuse the original distorted-image matrix merely because the output dimensions stayed unchanged. Rectification can change focal lengths, principal point, orientation, and the amount of retained scene coverage.

The rectified intrinsic matrix has this form, assuming zero skew:

Here, focal lengths and principal-point coordinates are expressed in pixels. The principal point is where the optical axis intersects the image plane; it is not automatically the image center.

Obtain the output matrix from the same configuration that generated the rectification map. Save the output dimensions and rectification rotation alongside it. A matrix returned by a separate example or a different stream configuration is not sufficient evidence.

The related explanation of intrinsic conversion between depth and texture images covers why calibration must match the image stream. For the distinction between original calibration and corrected geometry, read interpreting distortion coefficients in an unrectified image.

If the software exposes a projection matrix instead, identify its convention before extracting intrinsics. A fourth column represents additional projection information, not another intrinsic parameter. A translated virtual viewpoint also requires more than the rotation-only frame conversion described below.

2. Keep corners and intrinsics in the same pixel coordinates

Run corner detection on the exact image whose geometry your matrix describes. Record any processing between rectification and detection, especially cropping, resizing, rotation, or mirroring.

For a crop beginning at pixel coordinate (x0, y0), with no resizing, subtract that origin from the principal point. The focal lengths remain unchanged.

A subsequent resize changes both focal lengths and principal-point coordinates. The exact offset depends on the resampler's pixel-coordinate convention. For a resize using the pixel-center mapping below:

Some pipelines instead use an origin-aligned coordinate scaling. Use the mapping implemented by your image-processing operation, rather than mixing conventions from different functions.

More generally, if a supported pixel-coordinate transformation is represented by matrix A, the transformed projection is K_out = A * K_rect. Rotation or mirroring can produce a matrix outside a solver's expected intrinsic form. Mapping detected coordinates back into the rectified image is often simpler in those cases.

Cropping away invalid borders is common, but it does not preserve principal-point coordinates. The discussion of undistortion scaling and retained field of view explains the coverage trade-off behind those borders.

3. Define the chessboard coordinates and units

PnP estimates pose from corresponding object points and image points. For a flat chessboard, define the board plane as Z = 0, choose an origin, and assign coordinates using the measured corner pitch.

For example, a target with 9 by 6 inner corners and a 20 mm pitch provides 54 correspondences. Its detected corner grid spans:

Those spans describe the inner-corner grid, not the target's outer edges. Measure the physical target rather than assuming its printed dimensions survived scaling and mounting.

Keep detected corner order consistent with object-point order. A reversed ordering can produce a different board orientation while still giving a convincing overlay. Mark a physical origin or use an identifiable target if board-axis direction matters.

The translation returned by PnP uses the same length unit as the object points. Coordinates entered in millimeters produce translation in millimeters; the intrinsic matrix remains in pixels.

Use a flat, rigid target with clear corners in the amplitude image. Saturation, weak contrast, motion blur, and mixed pixels near depth boundaries can move apparent corner positions. Improve acquisition before trying to compensate through the pose solver.

4. Supply a consistent input set to the solver

Build the processing chain so each acquisition carries its own geometry metadata. Avoid relying on an unrelated matrix stored in a global configuration file.

1. Acquire the amplitude frame and identify its stream configuration. Record dimensions and whether rectification has already been applied.
2. Rectify with the documented source model and retain the output matrix, output dimensions, and frame rotation.
3. Apply any crop or resize, updating the camera matrix or mapping detected coordinates back to the rectified coordinate system.
4. Detect the chessboard corners and refine them on that same image. Reject incomplete or visibly unreliable detections.
5. Pair corners with board coordinates in a fixed order. Call a PnP solver that supports planar targets, using the matching matrix and zero rectified-image distortion.
6. Retain candidate poses where supported. Check visibility, reprojection, physical orientation, and consistency before accepting a result.

A planar target can admit competing pose solutions, especially when it is small or nearly front-facing. A solver reporting success does not prove that the selected orientation is physically correct. Check that visible target points lie in front of the camera and compare plausible candidates.

Do not feed normalized coordinates into a solver configured for ordinary pixel coordinates and nonidentity intrinsics. Normalization is valid only when the solver's input contract matches it.

5. Work through a crop-and-resize example

Consider a hypothetical rectified image measuring 1280 by 960 pixels. Its output calibration has focal lengths of 800 pixels and principal point (640, 480).

Crop a 960 by 720 pixel region beginning at (160, 120), then resize it to 480 by 360 pixels. Both scale factors are 0.5. Assume the resizer uses the pixel-center convention given earlier.

The processed-image intrinsics become:

Now take a corner at (720, 520) in the original rectified image. Its processed coordinates are:

The normalized viewing direction must agree before and after processing:

This is a useful unit test because it checks coordinate consistency without needing a pose estimate. Test several points, including corners near the image boundaries.

An origin-aligned resize would instead produce principal point (240, 180) and correspondingly different corner coordinates. Either convention can be consistent; combining the matrix from one with corners from the other introduces an error.

6. Verify reprojection and physical pose separately

Project the board points through the estimated pose and the same processed-image camera matrix. Compare projected points against detected corners in that processed image, using zero distortion.

Report RMS, maximum error, and a residual-vector overlay. A single average can hide localized errors. The following patterns are diagnostic clues, not unique proofs:

Compare errors at a common image scale. An error of 0.3 pixels after halving image dimensions corresponds to 0.6 pixels in the original rectified image. Smaller displayed residuals do not automatically mean better physical accuracy.

Validate at several board positions, distances, and tilts. Include the image perimeter, where model mismatch can become more apparent. Check a known physical displacement or an independently measured target location as well as reprojection.

The article on extrinsic accuracy beyond reprojection error explains why a visually accurate overlay can coexist with an unsuitable physical pose.

7. Convert from the rectified frame when required

A typical PnP result maps board coordinates into the camera frame represented by its input matrix. With rectified intrinsics, that is the rectified camera frame, not necessarily the original optical frame.

Suppose the documented rectification rotation maps original camera coordinates into rectified coordinates, with an unchanged optical center:

Both rotation and translation require conversion. Rotating only the orientation leaves translation expressed along the wrong axes.

Confirm the documented direction of R_rect before using these equations. If the supplied rotation maps rectified coordinates back to original coordinates, the required operation is reversed. If processing changes the virtual camera center, use the full documented rigid transform instead.

Also distinguish board-to-camera from camera-to-board pose. To invert a pose, use the inverse rotation and rotate the negated translation; simply negating translation is insufficient.

Stereo processing adds another naming risk: a rectified camera frame is not automatically the stereo rig's reference frame. The explanation of live stereo rectification provides context for keeping those geometries separate.

8. Choose hardware around accessible geometry

For an existing camera, first establish whether its documented interface exposes the amplitude image and matching calibration. Replacing hardware is unnecessary when the fault is an inconsistent crop origin or an incorrect frame conversion.

When selecting 3D sensors, ask specifically about amplitude availability, intrinsic parameters, rectification outputs, and coordinate-frame definitions. A depth image alone does not establish how another intensity stream is projected.

The MV-DT01WMU is a ToF sensing camera. Confirm the required stream and calibration access before selecting it for this workflow.

If the task only needs chessboard pose from intensity images, area scan cameras are another acquisition category. They require their own lens calibration and image-processing geometry rather than parameters borrowed from a depth stream.

The LS-CMR-0166 is a USB3 area scan camera. Evaluate its suitability against the target contrast, motion, and calibration requirements.

Use the camera finder to compare acquisition options. These catalog examples identify different camera types; they do not establish support for a particular amplitude or rectification API.

9. Check the complete chain before deployment

Keep a short acceptance checklist with the integration:

• Confirm the selected stream and its actual output dimensions.
• Save the rectification map configuration and output camera matrix together.
• Record crop origins, resize factors, and pixel-coordinate conventions.
• Verify board pitch, corner ordering, origin, and measurement units.
• Test normalized-ray agreement before and after image transformations.
• Inspect reprojection residuals across the field, not only near the center.
• Validate pose direction and axis definitions against a physical movement.
• Convert the pose into the required optical or machine frame explicitly.
• Repeat validation after changes to resolution, optics, mounting, or processing.

Do not apply this recipe unchanged to an image that retains fisheye distortion, uses a nonpinhole projection, or mixes streams with different optical centers. Such cases need the documented projection model or an additional geometric mapping.

For changes to an established installation, follow the checks in calibration recovery after camera or robot changes. For a new selection, start with the camera configurator, or request application support with your stream dimensions, processing chain, and required pose frame.