Map detections between two fixed cameras in one calibration

You can read the barcode in the zoom camera, but the wide camera's image is what the operator and the rest of the line see, and the box needs to appear there. The plane-to-plane homography only holds while the part sits at one depth. This how-to replaces it with a route through 3D: one intrinsic calibration per camera, one stereo calibration for the pair, then per detection a solvePnP lift, a rigid transform and a projection. It runs on two cameras bolted to one plate, it is about twenty lines of code, and the worked example shows how many pixels each geometric term is worth at a 1.5 m working distance.

Why does a homography between two fixed cameras fail when the part moves in depth?

A homography maps one plane to another, so it is exact only on the plane you calibrated it on; when the code moves 300 mm closer, its projection jumps about 55 px for a 1100 px focal length, and no single matrix can follow.

Parallax is the reason. The two lenses sit at different origins, so every depth change slides the code's projection against the background, and one 3 × 3 matrix absorbs exactly one depth plane. That is why the map needed recalibrating for every new scene: it was fitting the scene, not the cameras.

A homography stays the right tool when everything of interest lives on one fixed plane, such as labels passing a scanner at constant height; mapping pixel points through a homography covers that case. Once depth varies across the working volume, lift each detection into 3D. The pattern used here is one of the standard pair-geometry fixes on the systems topic page.

What do you need before any of the math works?

Four things: intrinsics for both cameras, one board session filmed by both cameras at the same time, a mount rigid enough that nothing moves between calibration and production, and locked focus and iris on the zoom lens. With those four, the extrinsic calibration is a single function call.

• Intrinsics per camera from a charuco or chessboard: 15-20 poses covering corners and tilts, reprojection RMS under about 0.3 px, yielding the camera matrix and distortion coefficients.
• One simultaneous board session: the board must be still in both frames of every pair, so trigger both cameras from the same line or strobe the scene.
• One shared plate, shimmed once; if a camera can be bumped during the shift, you do not have a fixed pair.
• Focus and iris locked on the zoom lens, with no autofocus in production.

If you are specifying bodies rather than calibrating what you have, the area scan cameras category is where both halves of the pair come from.

How do you get the rotation and translation between the cameras?

Run cv2.stereoCalibrate on the simultaneous board pairs with cv2.CALIB_FIX_INTRINSIC set, and it returns R and t that carry a point from camera A's millimetre frame into camera B's; accept the pair when reprojection error stays under 0.5 px RMS.

Convert the returned rotation vector to a matrix with cv2.Rodrigues. If you already hold per-camera board poses, the rotation_vector and trans_vector each cv2.solvePnP returned, compose them directly:

Those vectors are board poses inside each camera's frame, not camera-to-camera values, which is why no combination of dot products turns them into a transform. The acceptance test is the same either way: project the board corners from camera A into camera B's image and measure against the corners detected there. Mean under 1 px, RMS under 0.5 px — pass. The stereo calibration with fixed intrinsics guide walks this call in detail.

What are objectPoints in solvePnP when the target is a barcode?

objectPoints are the barcode's corner coordinates in a frame attached to the barcode itself, in real millimetres — for a 60 × 40 mm label, corners at (±30, ±20, 0) — and imagePoints are those same corners located in the image, in pixels.

Units are the whole trick: whatever goes into objectPoints comes out of tvec, so millimetres in, millimetres out. Your decoder already hands you the symbol's corner points in pixels; feed those as imagePoints and mark the planar case with cv2.SOLVEPNP_IPPE. Set the objectPoints origin at the code's centre and tvec is exactly the 3D point you want to map.

Depth is PnP's soft term. With the code spanning roughly a thousand pixels in the zoom camera, Z lands within a few percent of true distance, and the error section below shows why a few percent barely moves the drawn box. To also read the label's printed size from the same pose, measuring planar distances with a calibrated camera uses the same planar geometry.

How do you draw the box in the second camera's image?

Lift all four corners into camera A's frame, map each with X_B = R·X_A + t, project with cv2.projectPoints, and join the four resulting pixels into a polygon; nothing in that chain looks at the scene, so it holds anywhere in the working volume.

The per-frame chain, in order:

1. Detect the code in camera A and take its four corner pixels.
2. Run cv2.solvePnP on those corners to get the code's pose in A's frame.
3. Map each corner with X_B = R_AB · X_A + t_AB.
4. Project the four points with cv2.projectPoints into B's pixels.
5. Draw the polygon, and read B's depth at the centre as the cross-check.

Worked example. The wide camera sits 120 mm to the left of the zoom camera, so t_AB = (120, 0, 0) mm and R_AB = I. PnP puts the code centre at X_A = (60, -40, 1500) mm. Then X_B = (60 + 120, -40, 1500) = (180, -40, 1500) mm — a code 60 mm right of the zoom axis is 180 mm right of the wide axis. Example wide-camera intrinsics: f = 1100 px, centre (960, 540). u = 960 + 1100 × 180 / 1500 = 1092 px. v = 540 + 1100 × (-40) / 1500 = 511 px. Ignoring the 120 mm baseline would draw the centre at u = 1004 px, an 88 px error (1100 × 120 / 1500) — exactly the residual the homography could not absorb.

import cv2
import numpy as np

# Example values; substitute your calibrated numbers.
R_ab = np.eye(3, dtype=np.float64)               # (3, 3) rotation A -> B, from cv2.stereoCalibrate
t_ab = np.array([[120.0], [0.0], [0.0]])         # (3, 1) translation, mm
K_b = np.array([[1100.0, 0.0, 960.0],
                [0.0, 1100.0, 540.0],
                [0.0, 0.0, 1.0]])                # (3, 3) wide camera, pixels
dist_b = np.zeros((5, 1), dtype=np.float64)      # (5, 1) distortion coefficients
X_a = np.array([[60.0], [-40.0], [1500.0]])      # (3, 1) code centre from cv2.solvePnP, mm

X_b = R_ab @ X_a + t_ab                          # (3, 1) point in camera B's frame, mm
uv_b, _ = cv2.projectPoints(X_b, np.zeros(3), np.zeros(3), K_b, dist_b)
print(uv_b.ravel())                              # -> [1092. 510.67]
assert np.allclose(R_ab.T @ (X_b - t_ab), X_a)   # round trip recovers X_a

No hand undistorting anywhere: cv2.solvePnP absorbs camera A's distortion on the way up and cv2.projectPoints applies camera B's on the way down. The proof runs on the board: project camera A's corners into B and compare with B's detections; a mean under 1 px passes. If the wide camera reports depth, read it at the projected centre and compare with Z — agreement within a few percent at 1.5-3 m confirms both calibrations at once. A 3D sensor supplies that depth as part of the image.

For the wide role, a depth camera such as the MV-DT01WMU is exactly that part.

Projecting a 3D point to pixel coordinates works the projection step on its own if you want the full derivation.

How much zoom does the barcode camera need?

Size the focal length so the smallest code fills a quarter of the frame width at the far end of the range; for a 240 mm field at 3000 mm on a 6.2 mm-wide sensor, f = 3000 × 6.2 / 240 = 77.5 mm.

An 80 mm lens class fits, and the quarter-frame rule leaves margin for focus drift and mounting tolerance. Run the check in pixels too: 80 mm on a 4000 px-wide sensor gives 4000 / (3000 × 6.2 / 80) = 17.2 px per millimetre at 3 m, so a 60 mm code spans about 1030 px — comfortably past what decoding through shrink wrap needs.

USB3 bodies that fit the zoom role: the LS-CMR-0166 reads 4000 × 3000 at 30 fps, and the MV-CS200-10UM pairs a 1" mono sensor with 20 MP — resolution and mono both help on small, glossy codes.

How much error does the pair geometry add between 1.5 m and 3 m?

Budget the projected shift as f·δ/Z: a 5 mm baseline error costs about 4 px at 1.5 m for f = 1100 px, a degree of uncalibrated rotation about 19 px, and a 2 % PnP depth error under 1 px for a point 60 mm off-axis.

The rotation row is the one that bites, and it does not shrink with distance the way the baseline row does. Calibrate R_AB; never assume mounts that look parallel are parallel. The same arithmetic explains the original problem: the 60 mm code spans 1100 × 60 / 1500 = 44 px in the wide camera at 1.5 m, and 44 px of shrink-wrapped glare is not a decodable barcode — the zoom camera earns its place.

When do the calibrated numbers stop being true?

Recalibrate after any event that moves either camera — a bump, a refocus, a re-zoom, a lens swap, a seasonal temperature swing on a bare mount — and between events a weekly board check keeps the pair honest; left alone, a rigid pair holds for months.

The weekly check costs a minute: one board pair, project, compare, ship if the mean residual is under 1 px. Over that, recalibrate — twenty minutes with the board, and the boxes are honest again. Specifying the pair from scratch starts at the camera finder; configure a camera steps through sensor, interface and mount, and the contact page reaches the applications team when your geometry has a twist this guide did not cover.