Lens distortion correction for accurate pixel-to-mm measurements

A 600 mm field shot from inside a meter of working distance leaves no optical choice: the focal length lands near 5 mm, and lenses that short bend straight lines badly. The single px/mm factor you scaled your measurements with is then true only near the image center, and dimensions taken at the edge miss by tens of millimeters. The fix is not a different lens — it is a geometric calibration that measures the distortion and converts every pixel position to true millimeters. This guide gives the arithmetic that forces the short lens, the calibration procedure that removes its error, the accuracy you can hold afterwards, and the cases where you should change the optics instead.

The decision up front

Calibrate the distortion out with a grid (chessboard) calibration, and run every measured point through the model before converting to millimeters. On this field with a 4.8 mm lens this holds sub-millimeter accuracy edge to edge: residuals under 0.5 px, and at 0.94 mm per pixel that is under 0.5 mm of geometric error. Commit to this route whenever your tolerance is ±1 mm or looser on a field of this size.

Change the optics instead when any of these holds: the tolerance is tighter than about 0.1% of the field (±0.6 mm here), the part-to-camera distance varies by more than a few millimeters, or the field is small enough — roughly 150 mm or less — for a telecentric lens. All three cases are worked through below.

Why a short lens wrecks a single px/mm factor

The field of view, the sensor width and the working distance fix the focal length:

With a 1/3-inch class sensor 4.8 mm wide — the format built into many compact smart cameras — a field this wide sets m = 4.8 / 600 = 0.008. A 4.8 mm lens then places the camera at about 605 mm, inside a 500-1000 mm window; no longer focal length fits. Sensor formats and their widths are summarized in lens sensor format: 1/1.8 inch, 2/3 inch and 1 inch; the field-by-field arithmetic is in blister inspection working distance and field of view formula and the trade-offs in how to choose a machine vision lens focal length.

Barrel distortion on lenses of the 4.5-5 mm class typically reaches 10-40% at the image corner. Distortion shifts a point radially, in proportion to its distance from the optical axis:

At 25% corner distortion, an edge point at r = 300 mm sits 0.25 × 300 = 75 mm from where a single px/mm factor predicts it. On a lens dominated by the k1 coefficient the displacement grows with roughly the cube of the radius, so at half-field (r = 150 mm) the error is still single-digit millimeters. Longer focal lengths behave far better: the LS-FFL-0567 is a 16 mm C-mount lens for 1.1-inch sensors, rated 20 MP, with distortion under 0.3%.

Worked example: 4.8 mm lens at 605 mm working distance

Inputs, each with its source:

• Field of view 600 mm in X, working distance window 500-1000 mm (the application).
• Sensor 4.8 × 3.6 mm with 640 × 480 pixels — a 1/3-inch class sensor with 7.5 µm pixels.
• Lens 4.8 mm, the focal length the geometry allows.
• Corner distortion taken as 25%, mid-range for this lens class.

The arithmetic:

1. Magnification: m = 4.8 / 600 = 0.008 (1:125).
2. Working distance: WD = f × (1 + m) / m = 4.8 × 1.008 / 0.008 = 604.8, so about 605 mm — inside the window.
3. Scale: 640 px across the field = 1.07 px/mm; object-space pixel = 7.5 µm / 0.008 = 937 µm, so 0.94 mm per pixel (the mapping is derived in object space pixel size from magnification).
4. Uncorrected error at the field edge: 0.25 × 300 mm = 75 mm. At r = 150 mm: about 9 mm.
5. After grid calibration: residual ≤ 0.5 px, so 0.5 × 0.94 = 0.47 mm anywhere in the field.
6. Aperture: set F5.6-F8 for corner sharpness and lock the ring; at 0.008x the depth of field runs to hundreds of millimeters, so F8 costs nothing.

Decision: with a tolerance of ±1 mm (0.17% of the field), calibrate and ship — the geometric error budget is half the tolerance. If the tolerance is ±0.1 mm, no polynomial correction of a 4.8 mm lens reaches it; change the optics as described later.

Cross-check at another scale: a rig covering 1200 mm from 4 m gives m = 4.8 / 1200 = 0.004 and f = 4000 × 0.004 / 1.004 = 15.9 mm — which is why that set-up runs a 16 mm lens and still needs the same calibration.

Calibrate the distortion out: the procedure

1. Rigidify the mount. Put the camera on a stand with a coarse focus slide and bolt the base to the table — a top-heavy camera that tips is both a hazard and the end of every calibration before it. The LS-MNT-0040 is a slide-table machine vision stand with a 700 mm height and a 400 × 450 mm base.

2. Set focus and aperture, then lock both. Focus at the part plane, stop down to F5.6-F8, tighten the lock rings and witness-mark them. Distortion belongs to the lens design and the focus setting — refocus after calibrating and the model no longer matches the lens.

3. Build the target. A chessboard of known squares: measure the square pitch with calipers to ±0.1 mm. For a field this wide, 50 mm squares at 9 × 6 inner corners make a 500 × 350 mm board that fills most of the frame. Keep it flat — glass or aluminum backing, not loose paper. Camera calibration target flatness: when paper is enough gives the flatness limits, and calibration target field of view and lens coverage margins how much of the field the board must occupy.

4. Capture 15-25 frames. Center, all four corners and the edges; the board tilted in two axes and rotated 90 degrees in one frame; and through the working depth if part height varies. Fix exposure, switch gain off and set the iris to manual — automatic iris, exposure and gain control explains why a floating exposure poisons a calibration.

5. Solve the model. Intrinsics plus radial k1-k3 and tangential p1, p2 — OpenCV's calibrateCamera, or the grid-calibration function in a smart camera. On a lens this wide keep k3; dropping it leaves systematic error at the corners. Accept the fit only with an RMS reprojection residual under 0.5 px and no single corner over about 1 px: camera calibration residual values: average and max targets sets the thresholds.

6. Correct the points, not the picture. Undistort each measured point or blob centroid, then scale to millimeters — faster than warping whole frames, and it keeps edge contrast intact. If the camera views the part plane off-axis, add a homography to that plane; measure object size from a camera image works through the mapping.

7. Verify with hardware. Lay a ruled scale, or two marks at a known spacing, on the part plane and measure at the center and all four corners. Pass mark: every reading within 1 px in object space (0.94 mm here). How to measure barrel distortion correction accuracy extends this into a full error map, and checking the artifact itself against a calibrated reference is the metrology-grade version (CMM ring gauge calibration for accurate OD measurements).

Three ways out when calibration is not enough

Bigger sensor, longer lens, same working distance. An 8 mm lens on a 2/3-inch sensor (8.8 mm wide) covers the same field at m = 8.8 / 600 = 0.0147 and WD = 8 × 1.0147 / 0.0147 ≈ 552 mm — still inside the window. Distortion on the 8 mm class is usually under 2%, so the calibrated residual shrinks with it. The LS-FFL-1582 is an 8 mm C-mount FA lens for 2/3-inch sensors with a 57.6° field of view.

Stand further back. With the field fixed, doubling the working distance doubles the focal length that fits, and distortion falls roughly with the square of focal length. The 1200 mm-from-4 m case above runs a 16 mm lens — already a low-distortion regime — and calibrates to the same sub-pixel residual.

Telecentric for small fields. The LS-TCL-1422 is a 0.166x bi-telecentric C-mount lens for 1-inch sensors with a 258 mm working distance. On a 1-inch sensor 12.8 mm wide, 0.166x covers 12.8 / 0.166 = 77 mm of part, and magnification holds as the part moves through depth — no distortion term, no perspective scaling. When you need a telecentric lens for gauging gives the decision rule, and telecentric lens distortion and telecentricity explained the residual errors that remain.

Which route, by the numbers

When the calibrated wide-angle is the wrong answer

The standoff varies. A wide-angle scales with distance: 10 mm of part-height variation at 605 mm changes scale by 10 / 605 = 1.65%, which is 10 mm across the full measurement. No distortion model absorbs that; fixture the part or go telecentric.

The tolerance is tighter than ~0.1% of the field. That is 0.6 mm here. A polynomial correction holds about 0.5-1 px, not 0.1 px. Split the field over two cameras with 8 mm lenses, or gauge the critical dimension telecentrically.

The parts are not on one plane. Perspective folds height into X and Y; each measurement plane needs its own homography, and the world frame needs extrinsics — camera extrinsic calibration accuracy beyond reprojection error covers what that costs.

The lens is a true fisheye. Above about 100° horizontal field the radial polynomial diverges at the rim; switch to a fisheye (equidistant) model instead of k1-k3.

Checklist before you commit

• Focus and aperture set at the part plane, locked and witness-marked.
• Board pitch measured with calipers; board flat on glass or aluminum.
• 15-25 frames captured, all four corners of the field covered, several tilted.
• RMS residual under 0.5 px, max corner residual under 1 px.
• Gauge check passes at the center and four corners within 1 px in object space.
• Recalibration triggers posted: refocus, remount, shock, temperature swing of 20 °C or more.

Then order against the numbers: the lens finder filters by mount and format, the system configurator walks camera, lens and controller together, and the rest of the lenses topic covers the geometry in depth. If your field geometry is unusual, send us the numbers and we will run them with you.