Camera intrinsics after binning: reconstruct depth correctly

A depth image can look correct while its reconstructed point cloud bends, stretches or shifts sideways. One common cause is using calibration parameters expressed in full-sensor coordinates with pixels delivered after binning or cropping. Another is treating radial distance as optical-axis depth.

For depth-camera integration, the first decision is whether the supplied calibration already describes the acquired image. This guide shows how to establish that coordinate contract, transform intrinsics when appropriate, and compare your reconstruction with camera-generated XYZ. Keep image geometry, distance interpretation and coordinate-frame conversion separate until each passes its own check.

Decide whether the intrinsics need changing

Use calibration supplied for the active acquisition mode whenever the manufacturer documents it. Do not divide focal lengths or principal-point coordinates merely because the image dimensions look smaller than expected.

An intrinsic matrix normally contains focal lengths and a principal point expressed in image coordinates. Those coordinates might belong to the full sensor, a cropped sensor region, a binned image or a rectified output. The principal point is where the optical axis intersects the image plane; it need not coincide with the image center.

A principal point near an image boundary is therefore a clue, not proof of incorrect calibration. Cropping can legitimately move it toward a boundary or outside the delivered image.

Start with the distinction explained in intrinsic parameter conversion for depth and texture images. Calibration for a texture stream is not automatically valid for a depth stream, even when their dimensions match.

Record the image and calibration contract

Save the following metadata alongside one depth frame and its corresponding camera-generated XYZ frame:

• Delivered width and height, including any software resize.
• Calibration reference width and height.
• Horizontal and vertical binning or sampling factors.
• ROI origin and whether its offsets precede binning.
• Pixel-coordinate convention, including the location of pixel centers.
• Raw versus rectified image status and the matching distortion model.
• Distance units, invalid-value encoding and axial-depth versus radial-range meaning.
• XYZ axis directions and the physical origin used for distance measurement.

Also record whether a calibration query occurred before or after the final acquisition settings were applied. An earlier response may describe a different mode.

If the delivered dimensions change unexpectedly, follow the checks in depth image resolution and decimation. A reduced image might result from sensor binning, selecting every second sample, or host-side processing. These operations do not necessarily share the same pixel-center mapping.

Transform the intrinsic matrix through the image mapping

For an ideal pinhole model, write the intrinsic matrix as follows. Here, s is a skew term expressed in pixel units.

The safest general rule is to derive the image-coordinate mapping first. If a matrix A maps source image coordinates to delivered image coordinates, then:

Consider a specific convention: integer coordinates identify pixel centers, the ROI starts at source pixel index (x0, y0), and each delivered pixel is assigned the center of its source block. Let the block dimensions be bx and by.

These expressions describe that mapping, not every camera's implementation. If the device selects the first source pixel of each block instead, the half-pixel terms differ. If ROI offsets are reported in delivered coordinates, convert them before using these equations.

A documented corner-origin convention may permit simple division of principal-point coordinates. Mixing that convention with integer-centered coordinates introduces a fixed offset. Transform pixel coordinates and intrinsics consistently rather than adding an unexplained correction afterward.

Do not scale a dimensionless normalized skew parameter as though it were the pixel-valued s defined above. Parameter names alone do not establish their mathematical definition.

Work through a cropped and binned example

Consider a hypothetical calibration referenced to a 640 × 480 image. These numbers illustrate the arithmetic and do not describe a catalog product.

Suppose the acquisition selects a 480 × 360 source ROI beginning at (80, 40), then combines source blocks by a factor of 2 along both axes. The delivered image is:

Using the block-center convention above:

The principal point is not the geometric image center, which is (119.5, 89.5) under integer-centered coordinates. That difference is valid for the specified calibration and crop.

For a rectified pixel at (150, 100), the normalized coordinates are:

Notice what simple division would miss: the ROI subtraction. Conversely, applying these equations to intrinsics already expressed for the delivered image would transform the calibration twice.

Convert pixels into rays using the correct distortion model

For an undistorted or rectified image, a viewing ray follows directly from the inverse intrinsic matrix:

With zero skew, this reduces to q = [x, y, 1], using the normalized coordinates from the example.

For a distorted image, applying the inverse matrix only produces distorted normalized coordinates. You must then invert the documented distortion mapping to recover the undistorted ray. Depending on the model, this can require iteration or a manufacturer-provided conversion routine.

Radial and tangential coefficients defined in normalized coordinates generally remain unchanged under a matching image-coordinate transformation. Do not divide those coefficients by the binning factor. Other calibration models may define their parameters differently, so verify the equations rather than inferring behavior from coefficient labels.

A rectified output instead needs its rectified intrinsic matrix. Applying raw-image distortion correction again changes rays that were already corrected. The distinction is also central to choosing intrinsics for rectified-image pose estimation.

If a device supplies calibrated ray tables, those may represent its intended reconstruction more accurately than a generic pinhole approximation. Confirm their resolution, normalization, origin and relationship to the delivered depth mode.

Distinguish axial depth from radial range

The same viewing ray produces different XYZ coordinates depending on what the distance image measures.

Axial depth Z is distance along the optical axis. Radial range r is distance along the viewing ray from a specified origin. Under the ideal central-camera model, with range measured from the ray origin:

At the example pixel, x = 0.1 and y = 0. An axial depth of 1000 mm gives X = 100 mm and Z = 1000 mm. A radial range of 1000 mm instead gives approximately X = 99.5 mm and Z = 995.0 mm.

The difference becomes more obvious farther from the principal point. At delivered pixel (210, 100):

Mistaking range for axial depth therefore creates an error that grows toward the image edges. A center-only comparison can miss it.

Validate against corresponding camera-generated XYZ

Use a stationary, matte target with depth variation and coverage across the image. Capture depth and XYZ from the same acquisition, without temporal filtering differences if the documented interface permits that comparison.

Compare identical pixel locations first. Unorganized point clouds may discard invalid samples or reorder points, so array position alone may not establish correspondence.

1. Decode distance values into the documented physical units.
2. Exclude invalid, saturated and low-confidence samples according to the interface documentation.
3. Reconstruct points in the calibration's native coordinate frame.
4. Apply only the documented transformation into the XYZ output frame.
5. Compute component errors and Euclidean error for corresponding valid points.
6. Summarize median and upper-percentile errors separately for central and edge regions.

Choose acceptance limits from documented output precision and your application's measurement requirement. There is no universal millimeter threshold suitable for every depth technology and working distance.

Treat these patterns as diagnostic clues rather than unique signatures. The optical-frame and camera-frame transformation guide covers the axis-conversion stage separately.

Choose hardware around documented geometric outputs

When comparing 3D sensors, establish the required depth product before choosing a sensing technology. Ask whether your application needs an organized XYZ image, a distance image with documented calibration, or manufacturer-provided per-pixel rays.

The MV-DPS200P-02 is cataloged as a projected structured-light 3D camera. Confirm its available geometric outputs and active-mode calibration documentation before selecting it for custom reconstruction.

The MV-DT01WMU is cataloged as a ToF sensing camera. Its catalog classification alone does not establish whether an exported distance image represents radial range or axial depth.

These are different technology candidates, not a claim that either supplies every interface discussed here. Use the camera finder to narrow the hardware options, then request confirmation of output semantics and calibration access.

Check the intended target distance and scene coverage before ordering. The 3D camera working-distance guide addresses those physical constraints; correcting intrinsics cannot recover measurements outside a sensor's usable geometry.

Recognize when simple scaling is the wrong answer

A crop followed by regular binning can often be represented by an affine pixel-coordinate mapping. Other processing requires more care.

Selecting every second point from an existing cloud changes sampling, not the coordinates of retained points. It is not equivalent to constructing a smaller depth image and assigning new pixel coordinates. Keep that distinction separate from the issues described in point-cloud downsampling distortion.

Depth aggregation also need not equal averaging image coordinates. A delivered sample may combine different surfaces, confidence decisions or nonlinear distance measurements. A transformed intrinsic matrix can describe a nominal viewing direction without reproducing the device's complete depth-processing pipeline.

Likewise, alignment into another camera's viewpoint may involve reprojection and occlusion handling rather than simple resizing. Use calibration associated with the aligned output, if documented, instead of applying a resolution ratio to the original matrix.

If disagreement appears only after reconnecting or restarting, investigate calibration selection and cached ray data. The checks in unit-vector inconsistency after restart help separate configuration changes from reconstruction mathematics.

Complete the preflight checklist

Before accepting the reconstruction or ordering hardware, confirm these items:

• Calibration identity: the parameters belong to the depth stream and active acquisition mode.
• Coordinate mapping: crop origin, sampling factors and pixel-center convention are documented.
• Single transformation: intrinsics have not already been adjusted by the acquisition interface.
• Distortion status: raw and rectified data use their corresponding calibration models.
• Distance semantics: units, invalid values, axial depth and radial range are explicitly identified.
• Reference origin: any offset between sensing and XYZ origins is included as documented.
• Frame agreement: axis order, signs and transformations match before points are compared.
• Validation coverage: comparisons include edges, multiple distances and depth boundaries.
• Regression coverage: saved frames reproduce the same geometry after configuration changes.

If any item remains unknown, send the acquisition settings, calibration metadata and a representative depth/XYZ pair through application support. Confirming that contract is the next step before changing equations or committing to a sensor.