Image distortion low or high in camera calibration: which to pick

Switching image distortion between Low and High in a calibration tool is not a quality setting — it selects the projection the solver will fit: the pinhole model of a normal lens, or the equidistant fisheye model of an ultra-wide. Anyone calibrating board cameras, machine-vision lenses or fisheyes for measurement, guidance or depth needs this decision right: the correct switch converges in seconds and undistorts cleanly, while the wrong one leaves straight lines bowed, residuals an order of magnitude too high, and a depth pipeline downstream rejecting frames. This guide gives you the 90° field-of-view decision line, the two projection formulas behind it, worked examples for four catalogue lenses, and the checks that confirm the choice.

What does the image distortion setting change?

The setting selects between two projection models, not two quality grades, and both fit the same kinds of parameters: focal length, principal point, and a set of distortion terms describing how image radius r grows with the ray angle θ measured from the optical axis.

A rectilinear lens keeps straight scene lines straight, so r follows the tangent; a fisheye trades that straightness for coverage, and r grows linearly with the angle. The two curves, and the field-of-view formula that separates them:

f is the focal length in millimetres, w the sensor's active width in millimetres, θ the ray angle, and r the distance from the image centre in millimetres. Those two curves drive everything that follows: where tangent and angle nearly agree, either setting works; where the tangent runs away, only the fisheye model can follow the lens.

When should you set image distortion to Low?

Choose Low for any rectilinear lens — every fixed-focal machine-vision lens and nearly all M12 board lenses sold for factory work — because fields of view stay below roughly 90° and a radial polynomial corrects the residual barrel or pincushion to well under a pixel.

The whole fixed-focal machine-vision range sits on this side. The LS-FFL-0679 is a 35 mm C-mount machine vision lens for a 2/3" sensor rated 6 MP.

The pinhole field of view on the 8.8 mm width of a 2/3" format is 2·atan(4.4 ÷ 35) = 2 × 7.17° ≈ 14°, an angle at which tangent and θ differ by under half a percent, so the polynomial has almost nothing to correct and the fit converges without complaint.

Compact cameras follow the same rule. The M12 board lenses tagged low distortion are pinhole designs by intent, even when they are wide: the LS-MBL-0159 pairs a 3.3 mm focal length with a 1/2.3", 16 MP sensor and is listed as a low-distortion M12 lens.

The horizontal field of view here is 2·atan(3.085 ÷ 3.3) = 2 × 43.1° ≈ 86° — genuinely wide and still inside the tangent's usable range, which is exactly what the low-distortion label and the Low setting agree on.

When is the High setting the right choice?

Set High only when the lens is a fisheye or an ultra-wide whose field of view runs past about 100°, because the calibrator then fits the equidistant model r = f·θ, which stays finite out to 180°, instead of a pinhole tangent that diverges at 90°.

Fisheye M12 lenses for embedded and inspection cameras are the main case. The LS-MBL-0041 is a 1.45 mm M12 fisheye rated 10 MP on a 1/2.3" sensor.

Feed the same numbers into the pinhole formula and it claims HFOV = 2·atan(3.085 ÷ 1.45) = 2 × 64.8° ≈ 130°. Near 65° of ray angle the tangent steepens toward vertical, a polynomial needs ever-larger coefficients to chase it, and fits typically stall at a high error or fold the outer frame during undistortion. Reframe with the equidistant relation: if the image circle fills the 7.66 mm diagonal of a 1/2.3" format, the corner sits at θ = (7.66 ÷ 2) ÷ 1.45 = 2.64 rad ≈ 151° — a lens photographing roughly 150° of the scene, a geometry only the fisheye model describes. Designs covering a full hemisphere, about 180°, exist for surround view and SLAM, and the tangent cannot represent them at any polynomial order.

For how image circle, back focal length and field of view interact on these short lenses, see fisheye lens selection for M12 board cameras: circle, BFL and FOV.

What field of view decides between Low and High?

Use about 90° of horizontal field of view as the decision line: below it the pinhole model fits comfortably, from 90° to about 100° try Low first and check the corner residuals, and above roughly 100° the equidistant High model is the reliable choice.

The line follows from the two curves. At a 25° ray angle, tan(θ) and θ differ by about 7%, so either projection nearly agrees and the pinhole polynomial is the better-conditioned fit. Toward 90° the tangent climbs without limit while the equidistant curve stays perfectly linear out to 180°, which is why ultra-wide lenses calibrate only on the High side. The grey band between belongs to lens design, not to the number alone: a well-corrected rectilinear lens at 90-100° still calibrates on Low, while an ordinary ultra-wide may already behave like a fisheye at 95°. Settle those cases by residuals on the same image set, not by the box label.

Field of view belongs to the lens–sensor pair, not to the lens: the same focal length crosses the 90° line sooner on a bigger sensor, which same lens on a different sensor: how HFOV changes works through with example numbers.

How do you calculate the field of view before calibrating?

Compute the pinhole horizontal field of view with HFOV = 2·atan(w ÷ 2f), where w is the sensor's active width and f the focal length, both in millimetres; the result places the lens on one side of the 90° line before you shoot a single target.

Worked example, a 2.8 mm lens on a 1/2.5" sensor:

1. Sensor width: a 1/2.5" format measures 5.76 mm across.
2. Half-width: 5.76 ÷ 2 = 2.88 mm.
3. Ratio to focal length: 2.88 ÷ 2.8 = 1.0286.
4. Arctangent: atan(1.0286) = 45.8°.
5. Field of view: 2 × 45.8° = 91.6°, call it 92°.
6. Decision: grey band — start on Low and let corner residuals confirm.

The LS-MBL-0018 is a 2.8 mm, F2.8 M12 board lens rated 8 MP on a 1/2.5" sensor and is exactly this borderline case.

If the undistorted edges stay straight and corner residuals sit under about a pixel, the Low fit was right; if the solver struggles, switch that one lens to High and compare the error on the same images. Any other focal length and sensor pairing runs through the field-of-view calculator below the same way:

What about a lens that sits just past 90°?

Keep a rectilinear lens on Low even at 92-95° of field of view, because the fisheye model would force its linear curve over what is really a tangent curve and leave a systematic pincushion in every undistorted frame; what changes is how closely you check the corners.

Rectilinear behaviour is a design property, not something the focal length tells you. Wide board lenses in the 2.8-3.3 mm range are engineered to keep straight lines straight out to their corners, so they belong on Low despite the width. Two misclassification signatures separate the cases: a rectilinear lens pushed through the fisheye model comes out pincushioned, edges bowed outward, while a fisheye-geometry lens pushed through the pinhole model fits its centre but piles residuals in the corners. When a lens genuinely sits between the designs, fit both models on the same target set and keep the one with the lower, more even error; the comparison takes minutes and settles the case on evidence.

One caution for measurement: undistortion amplifies noise fastest where ray angles are steepest, so past about 45° of ray angle, accuracy per pixel falls off quickly. If the task only needs the middle of a 92° frame, crop the ROI instead of chasing corner accuracy, or step up in focal length and pull the camera back.

What breaks if you pick the wrong model?

The wrong model announces itself: the calibration refuses to converge or plateaus at a reprojection error an order of magnitude too high, and the undistorted image bends straight scene edges into a visible bow.

The two parameter sets describe different mappings between image radius and ray angle, so they do not translate in either direction. Load a set saved under the other switch and downstream code produces garbage coefficients or fails outright — the failure signatures covered in distortion model unknown: garbage depth camera coefficients explained and, on the matching side, in fisheye undistortImage assertion error: matching the distortion model.

Depth pipelines inherit the error silently: wrong intrinsics shift every triangulated point, so stereo and structured-light depth maps drift even when the RGB view looks fine. The same discipline holds for laser profilers and 3D sensors, whose factory calibration assumes the documented projection — a home-grown remap in front of them invalidates it.

How do you verify the fit after calibration?

Judge the fit with the mean reprojection error and the corner map: keep the mean below about half a pixel, treat corner residuals above roughly a pixel as a sign of the wrong projection, and finish with a straight-edge test scene after undistortion.

Work in that order. First the number: reproject the target corners with the fitted parameters; a working rule for measurement work is a mean under half a pixel, the level at which the calibration stops adding visible noise to a dimension. Second the map: on the correct switch the error spreads evenly across the frame, while corner-only residuals point at the wrong projection. Third the picture: undistort a frame containing known straight edges — a doorway, a shelf upright, the bench edge — and look before trusting any measurement downstream.

Shoot the target set the same way for either model: 10 to 20 frames, target in every quadrant, tilted up to roughly 45°, in focus at the working distance. Size the target against the field of view with the margins covered in calibration target field of view and lens coverage margins, and when a board camera needs its profile rebuilt from images alone, build a lens distortion profile for a board camera without an LCP file walks that end to end.

The decision, one line per case: rectilinear and under about 90° HFOV — Low; rectilinear in the 90-100° band — Low, then verify corners; fisheye or past about 100° — High; borderline — fit both and keep the smaller, more even error. When the choice is made and the glass is next, the machine vision lenses and the lens finder filter by focal length, sensor format and resolution, the lens step of the configurator narrows the match to your camera, and the 3D topic page collects the depth-side guides that consume the result.