Extrinsic calibration without checkerboard: use measured scene points

You have K from a one-time checkerboard calibration, but the images of the part cannot contain the board, so the per-view extrinsics R and t look out of reach. They are not. R and t only describe where a chosen world coordinate frame sits relative to the camera, and any scene points whose 3D coordinates you know and whose image positions you can detect define that frame as well as a board ever did. This guide shows you how to place and measure four to eight reference points around the part, solve the pose for every view with a homography or PnP, verify it by reprojection, and assemble the projection matrix P = K[R|t] that a multi-view reconstruction needs.

Why the checkerboard is not magic

An extrinsic pose answers one question: given a point X_world in the frame you chose, where does it land on the pixel grid?

The checkerboard contributes nothing but convenient X_world values: its corners sit at known spacings on a plane, and a detector finds them to sub-pixel precision. Reamed hole centres, machined corners or printed dot targets supply the same two ingredients. Four point pairs are the mathematical minimum — eight equations for the eight unknowns — and six or more make the least-squares solve shrug off one bad detection. The habit of reaching for a board usually comes from distance: a small board far from the lens stops resolving corners, the checkerboard target size problem at distance. Reference points sized to fill the view remove that limit entirely.

Choose reference points you can measure and detect

• Known coordinates. Measure each point once, in millimetres, in a single world frame you write down: origin, X axis, Y axis, Z axis.
• Detectability. Circle-fitted hole centres, printed dots or machined corners all work; contrast and a repeatable detector matter more than the feature type.
• Spread. Fill at least half the frame width. Tilt error ≈ 2 × point error ÷ baseline, so a short reference amplifies every pixel of noise.
• Stability. The points stay fixed for every view — bolted to the fixture, or marked on the part's base so they travel with it.

Put numbers on the spread rule. At 0.15 mm/px, a corner detected to ±0.5 px carries ±0.075 mm of position error; over a 300 mm baseline that is 2 × 0.075 ÷ 300 = 0.0005 rad ≈ 0.03° of tilt uncertainty. Squeezed into a 50 mm reference, the same detection error would cost six times more. Surveying scene points instead of waving a board is also how stereo rigs get calibrated without a chessboard.

The reference view needs enough pixels that a hole centre is a disc, not a smudge. The LS-CMR-0166 captures 4000 × 3000 frames over USB3 on an IMX226 sensor:

Two routes to R and t

Both routes start from the same input — matched pairs of world coordinates and detected pixels — and differ only in the geometry they assume.

Route A: four coplanar points

Four pairs determine the 3 × 3 homography H — eight independent ratios, and every vision library ships an estimator. With K known, the pose falls out of the decomposition:

The scale s drops out in the normalisation. Divide K⁻¹H by s, read r1 and r2 from the first two columns after normalising their lengths, and complete the right-handed rotation with r3 = r1 × r2. When your four points are the floor or table under the part, the ground-plane pose method from four known points works this exact case in detail.

Route B: six or more 3D points

PnP solves the pose directly from any spread of known 3D points. cv2.solvePnP takes the object points (N × 3), the detected image points (N × 2), K and the distortion coefficients, and returns an axis-angle rotation plus a translation. Three points already determine candidate solutions but leave an ambiguity; four resolve it; six or more let least squares average the detection noise, so a single mis-detected corner cannot steer the pose. If detections may contain outliers, use the RANSAC-style variant of the same call.

Two practical notes. Iterative solvers accept an initial guess, and getting a pose initial guess's reference frame right prevents sign surprises. And the intrinsics you pass matter enough that choosing PnP intrinsics for pose estimation is its own subject.

From images to projection matrices, step by step

1. Define the world frame. Origin on reference point 1, X along the plate's long edge, Y along the short edge, Z out of the surface. Write the definition down; every later number refers to it.
2. Measure the reference coordinates once. Hole centres on a 300 mm plate read to 0.1 mm with calipers, and the pose can never be better than this measurement.
3. Calibrate K once and freeze it. Keep the distortion coefficients together with the lens, the focus setting and the resolution; the hand-eye workflow fixes intrinsics before any pose solve for the same reason.
4. Capture every view with the reference points in frame and the exposure fixed.
5. Detect to sub-pixel precision, then undistort the pixel coordinates with the calibration's distortion coefficients before solving.
6. Solve per view: homography plus decomposition for four coplanar points, cv2.solvePnP for six or more general points.
7. Check, then store. Run the checks below, then save R and t per view and build P = K[R|t] for the reconstruction.

Worked example: four holes on a fixture plate

The one-time board calibration gave fx = fy = 4000 px, cx = 2000 px, cy = 1500 px on 4000 × 3000 images; the pixel positions below are after undistortion. The rig: a 300 × 200 mm fixture plate with reamed holes at its corners, camera 600 mm above, looking straight down, world frame on hole P1.

These four pairs solve to R = diag(1, −1, −1) — the half-turn about the camera x-axis that every straight-down view produces — and t = (−150, 100, 600) mm. Check the scale both ways: the 300 mm edge spans 3000 − 1000 = 2000 px, i.e. 6.67 px/mm, and fx/Z = 4000 ÷ 600 = 6.67 px/mm. The recovered camera height, 600 mm, equals the tape-measured stand height — the sanity check that costs nothing.

Reproject P2 with the recovered pose. R·P2 = (300 × 1, 0 × (−1), 0 × (−1)) = (300, 0, 0), so X_cam = (300 − 150, 0 + 100, 0 + 600) = (150, 100, 600) mm. Then u = 4000 × 150 ÷ 600 + 2000 = 3000 and v = 4000 × 100 ÷ 600 + 1500 = 2166.7 — both match the detection, a reprojection error of 0.0 px in this constructed example; real detections scatter within a few tenths of a pixel.

Now use the pose for its actual job, projecting a new point. The cylindrical part stands centred on the plate with its top face at Z = 120 mm, so the top centre is X = (150, 100, 120). R·X = (150, −100, −120), hence X_cam = (0, 0, 480) mm, u = 4000 × 0 ÷ 480 + 2000 = 2000 and v = 1500: the top centre lands exactly on the principal point, because the optical axis pierces the plate centre and the part is centred on it. Run the same projection for every view and the rays intersect on the part's surface — that intersection is the 3D reconstruction.

At this geometry the frame covers 4000 px × 0.15 mm/px = 600 mm of width at 600 mm distance. On a 1/1.7-inch sensor about 7.4 mm wide, the pinhole estimate gives f ≈ 7.4 mm × 600 ÷ 600 ≈ 7.4 mm — a distinctly wide lens, the price of seeing a 300 mm plate whole from a 600 mm column:

The stand carries the geometry. The LS-MNT-0040 slide stand holds the camera on a 700 mm column over a 400 × 450 mm base:

A 600 mm working distance leaves about 100 mm of column travel to spare for framing, which is why height adjustment earns its place on this kind of rig.

Checks before you trust a pose

Targets with sub-pixel detection on a machined reference: below 0.3 px RMS is normal; 0.3-1 px means re-check pair ordering and the undistortion; above 1 px the solver matched a wrong pair, not a pose.

Two more checks cost seconds and catch what RMS misses. Physical: the recovered camera height must match the tape measure — 600 mm in the example — and the recovered pixel scale must equal fx/Z. Held-out: solve with all but one pair and predict the point you kept back (with only four coplanar points, add a fifth measured point to check against). A point the solver never saw, reproduced within half a pixel, is the strongest evidence the pose is real.

Common mistakes that wreck extrinsics

• Reference cluster too tight or centred. Tilt and distance trade off along the viewing axis; spread the points toward the frame edges.
• Pair-ordering errors. Keep one printed table of world points and one of image detections in a fixed order. A symmetric reference can even accept a wrong labelling and return a valid-looking pose in a rotated frame — one asymmetric mark removes the ambiguity.
• Skipped undistortion. A wide lens bends points near the frame edges by several pixels; undistort before the solve, every time.
• Refocused or swapped optics after the K calibration. fx and the distortion coefficients changed, so every pose is wrong until you recalibrate — and re-check the board square size you typed in, the classic grid-pitch calibration error.
• References moved between views without re-measuring. Each view then solves against a different world frame, and the reconstruction cannot close.

Set the geometry once, keep it fixed, and every new view is one detection and one solve away. If you are building the rig itself, start from the area scan cameras, the 3D sensors for depth-first designs, the camera stands, or the camera finder; the 3D vision topic page collects the calibration guides.