Camera calibration residual values: average and max targets

You ran the calibration and the tool reports two numbers: an average residual and a maximum residual. The pass line is simple. An average reprojection residual of 0.1 pixels is what a careful set-up delivers, and anything up to 0.3 pixels still holds for pixel-to-millimetre robot guidance. The maximum should stay under about 0.5 pixels with no single point standing out. And when the numbers improve because a few edge points went undetected, that improvement is a trap: the fit no longer sees the worst part of the field. This guide gives the targets, converts them into millimetres, and settles the dot-grid-versus-checkerboard choice.

The residuals that pass

A residual is the distance, in pixels, between a detected calibration point and the place the fitted model projects it. The tool averages every point into its 'average residual' and reports the single worst point as the 'maximum residual'. Pixels are the native unit, so 0.1 px means different things on a 300 mm field and a 1200 mm field; the next section does that conversion.

Hold a calibration to these numbers:

Add one ratio rule: the maximum should sit within about five times the average. An average of 0.15 px with a 1.2 px maximum is not a good calibration with one unlucky point; it is a local fault — a mis-located corner, a smudge on the target, one patch of glare — and the fit is wrong exactly there.

Residuals measure how well the model explains your calibration points, nothing more. They do not by themselves prove the camera-to-robot result; that wider argument is made in camera extrinsic calibration accuracy beyond reprojection error. The corner-scale test in the checklist at the end closes that gap.

What a residual costs in millimetres

Work through a common case: a 20 MP camera with a 1-inch sensor watching a 300 mm wide field. The MV-CS200-10UM is an example of the class, one of many area scan cameras. That sensor is typically 5472 × 3648 pixels of 2.4 µm, or 13.1 mm of sensor width.

• Sensor width: 5472 × 2.4 µm = 13.1 mm.
• Magnification: m = 13.1 / 300 = 0.044.
• With a 35 mm lens, working distance: WD = 35 × (1 + 0.044) / 0.044 ≈ 830 mm.
• Object-space pixel size: 300 / 5472 = 0.055 mm, so 55 µm per pixel.
• Average residual 0.1 px means 0.1 × 55 µm ≈ 5.5 µm of error; the 0.5 px worst point means 27 µm.

A robot placing parts to ±0.5 mm spends 27 µm at the worst calibrated point — about 5% of the tolerance. Pass. Now stretch the same camera over a 1200 mm conveyor: one pixel is worth 1200 / 5472 = 0.22 mm, so a 0.3 px average costs 66 µm and a 0.5 px worst point 110 µm. Fine for ±2 mm placement; too coarse to double as a measuring system.

Run the same arithmetic in both directions — field of view from focal length, focal length from field of view — with the calculator below. The derivation of pixel size from magnification is set out in object space pixel size from magnification.

Why residuals fall when edge points go missing

Here is the classic pattern. A dot-grid calibration with every point found reports 0.05 px average and 0.18 px maximum; five edge dots drop out and the tool reports 0.04 and 0.09 instead. The second run looks better because the fit is no longer scored in the most distorted corner of the image. Keep the first calibration, then fix whatever lost the dots.

A fit is constrained only where points exist. Lose one interior dot among a hundred and its neighbours interpolate across the gap; the fitted scale moves by under 0.1%. Lose a corner dot and you remove the longest lever arm the fit owns, so the model extrapolates into a region nobody measured. The statistics improve because the difficult region left the data set, while the real error at that corner — often several pixels of unmodelled distortion — stays right where your part lands.

The misses themselves are a lighting and focus problem, not a statistics problem. Glare and shadow hide dots, and a wide-open iris with poor focus blurs them past detection. The exposure side is covered in automatic iris, exposure and gain control, and the detection-side fixes in checkerboard calibration not detecting corners apply to dot grids unchanged.

Dot grid or checkerboard

Both targets map pixels onto millimetres and give the camera its origin. They differ in what the fit is allowed to model, and that difference lives at the edges of the field.

The decision: the two agree only when the lens is nearly distortion-free and the target sits square to the sensor. Every other case — tilted camera, wide-angle lens, measurement duty — goes to the checkerboard, because a plane fit has nowhere to put distortion except the corners where parts actually sit. Target choice and sizing are covered in choosing a camera positioning grid target and calibration target field of view and lens coverage margins.

When a checkerboard result looks far worse than the dot result on the same camera — 0.54 px average against 0.05 px, say — the two numbers do not measure the same fit. The checkerboard run honestly estimates focal length, principal point and distortion; a lax plane fit can look clean while being wrong at the edges. Before rejecting it, check three things. First, the grid pitch entered in the tool must match the printed target — the trap dissected in checkerboard camera calibration error: fix the grid pitch. Second, the target must be flat, and paper waves; camera calibration target flatness: when paper is enough quantifies when that matters. Third, the poses must be real poses: tilt the board 20–45°, cover all four corners of the field, and collect 10–20 images, because all-frontal views leave the distortion estimate starved.

When the standard targets are the wrong targets

The 0.3 px pass line is written for guidance work. Three situations need different numbers.

• Dimensional measurement. A gauging task that locates edges to 0.1 px needs a calibration quieter than its own noise: demand 0.1 px average or better from a multi-pose checkerboard and lock the set-up down. At 0.3 px the calibration is three times the measurement noise.
• Wide-angle and fisheye lenses. A standard polynomial cannot describe an equidistant lens; the symptom is a modest average with a blown maximum at the edge. Switch the model, not the target — the failure mode and the fix are in fisheye distortion model matching for undistortion.
• Heat. A lens that warms through the day drifts in scale and focus, so a morning calibration reads wrong by afternoon. Camera focus drift with temperature covers the diagnosis; the cure is a locked iris, a rigid mount and recalibration after heat soak.

Rolling-shutter cameras add one rule of their own: calibrate with camera and target stationary, because a moving target warps rows and bakes that warp into the intrinsics.

Hardware that keeps the numbers true

The calibration is only as stable as the mechanics under it. A bracket that flexes between calibration and production moves the pixel-to-millimetre scale without touching the residuals, so bolt the camera to a rigid column from camera stands — the LS-MNT-0040 is a slide-table type with a 700 mm height on a 400 × 450 mm base — and lock every joint.

Distortion you do not have to model is accuracy you get for free. Low-distortion machine vision lenses leave the fit little to absorb: the LS-FFL-0567 is a 16 mm C-mount lens for 1.1-inch sensors, rated 20 MP, with distortion under 0.3%. Reading distortion curves before you buy is the other half of that decision: how to read lens performance curves.

Two habits finish the job. Lock the iris and focus rings the moment calibration passes — refocusing changes effective focal length and distortion, which silently invalidates the fit — and focus at the working f-number, because aberrations and edge behaviour shift with aperture.

The acceptance checklist

Run all six checks before sign-off; the first failure tells you which section above to reread.

• Every dot or corner detected, with nothing missing in the outer fifth of the field.
• Average residual 0.3 px or less, aiming for 0.1 px; maximum 0.5 px or less; maximum within 5 × average.
• A known length — a gauge block, a ruled scale — placed near a corner measures inside tolerance in the live image. This is the one test a residual cannot fake.
• A repeated calibration reproduces the pixel-to-millimetre scale within 0.1%.
• Grid pitch in the tool matches the printed target; the target is flat; checkerboard poses tilt 20–45° and cover all four corners.
• Iris and focus rings locked after calibration, and the mount still tight.

If the numbers still refuse to line up, send the residual report and one calibration image to the applications team — most of these cases close in one exchange. To build the station around a passing calibration, start from the lens finder, the camera finder, or run the system configurator from the lens step.