Fisheye undistortion: how the pixel mapping works

Run a fisheye image through an undistortion function and you get a rectilinear image, black corners and stretched edges, usually with no hint of what the function computed to get there. This page walks the mapping pixel by pixel, the way an implementation actually runs it, so you can write it yourself, debug one that misbehaves, and predict the black corners before the first run. We use the equidistant model that most calibration toolboxes fit, show the arithmetic on real pixel values, and close with the decision the mapping forces: which field of view to keep, and when a rectilinear lens is the better fix.

The mapping in one paragraph

The function does not push source pixels to new positions. It loops over the pixels of the output image and asks, for each one: where in the fisheye frame does this rectilinear pixel's view come from? It computes that source coordinate and copies the fisheye value from it. This is a backward, or inverse, mapping, and nearly every undistortion routine works this way, whether it is a library call such as undistortFisheyeImage or twenty lines of your own code.

For an output pixel (u, v), with the lens calibrated as principal point (cx, cy) and fisheye focal length f_fish in pixels, and with f the output focal length you choose:

The output pixel takes the bilinearly interpolated fisheye value at (u_src, v_src). That is the entire function. Calibration coefficients and projection-model variants only change how r is computed from theta.

Forward or backward: which loop runs

Two directions are possible and the first guess is usually the wrong one. A forward mapping loops over source pixels, computes where each lands in the rectilinear image, and writes it there. At wide angles the fisheye compresses, so many source pixels land on the same destination while neighbouring destinations get none: the output is full of holes and streaks. A backward mapping loops over output pixels, so every one is filled exactly once.

One trap before you write a line of code: calibration can hand you a polynomial that maps fisheye radius to rectilinear radius, or the reverse. Feed the wrong direction into the backward loop and the output comes out more bent than the input, not less. Depth pipelines fail the same way, for the same reason — inverse distortion coefficients: why rectified depth looks warped works through that diagnosis.

Step 1: output pixel to a ray

Treat the output image as an ordinary pinhole camera. Each pixel (u, v) becomes a slope on the z = 1 plane: x = (u - cx)/f and y = (v - cy)/f. Nothing fisheye has happened yet; this is plain perspective geometry. The principal point starts at the image centre and is refined by calibration, and the output focal length f is yours to choose — it only sets the scale of the result (the FAQ covers choosing it).

Step 2: ray to incident angle

The ray with slopes (x, y, 1) meets the optical axis at angle:

This atan is where rectilinear physics enters, and only here: a pinhole camera places a point at radius f·tan θ, and you are undoing exactly that. Every output pixel is now reduced to one number, its incident angle θ.

Step 3: angle to source radius

The fisheye lens does not project r proportional to tan θ. It compresses angle into radius, and the model you fit decides how:

The equidistant model is linear in θ, so the inverse the backward loop needs is closed-form: θ = r / f_fish. The open-source fisheye calibration most vision code uses adds a softening polynomial, r = f·(θ + k1·θ³ + k2·θ⁵ + ...), with k1 and k2 the fitted terms; invert it numerically, or drop it for a first approximation. The omnidirectional polynomial model from Scaramuzza's calibration toolbox goes further: r = a0 + a1·θ + a2·θ² + a3·θ³ + a4·θ⁴. Invert that once by Newton iteration into a lookup table of θ(r), one entry per pixel of radius, and interpolate the table for every output pixel.

Step 4: sample the source image

u_src and v_src land between source pixels, so blend the four neighbours by the fractional parts: at u_src = 300.7 you take 70% of column 300 and 30% of column 301. Where r exceeds the calibrated coverage circle, or (u_src, v_src) leaves the frame, write black and set a mask bit. Stretch is the honest limit of this interpolation: once each source pixel must feed several output pixels, the extra detail is invented, not measured.

Worked example: two pixels, all the arithmetic

Calibration: a 1024 x 1024 fisheye frame, cx = cy = 512 px, equidistant f_fish = 400 px, full 180° coverage. The coverage circle then has radius 400 × π/2 = 628 px, inside the roughly 724 px half-diagonal of the sensor. Output image same size, f = 400 px.

Pixel A, (632, 512), near the centre:

• x = (632 - 512)/400 = 0.300, y = 0
• θ = atan(0.300) = 0.2915 rad = 16.7°
• r = 400 × 0.2915 = 116.6 px
• u_src = 512 + 116.6 = 628.6, v_src = 512.0
• Value = 0.42 × I(512, 628) + 0.58 × I(512, 629)

Pixel B, (912, 512), well out on the axis:

• x = (912 - 512)/400 = 1.000
• θ = atan(1.000) = 0.7854 rad = 45.0°
• r = 400 × 0.7854 = 314.2 px
• u_src = 512 + 314.2 = 826.2, v_src = 512.0
• Value = 0.8 × I(512, 826) + 0.2 × I(512, 827)

A point the rectilinear image places 400 px from the centre sits 314 px out in the fisheye — 21% compression, growing with angle. Since v_src lands exactly on 512, only two pixels blend; off-axis pixels blend four. Everything else in the output is this computation repeated a million times.

Black corners and edge stretch

Black corners have one cause: an output pixel whose source radius falls outside the calibrated circle or the frame. The rule:

with R_out the output pixel's distance from the principal point. Black wherever r_needed > r_circle, and r_circle = f_fish × θ_max, where θ_max is half the fisheye's real coverage. Worked numbers: a 160° fisheye at f_fish = 400 px has r_circle = 400 × 1.396 = 558 px. An output widened to f = 120 px asks its corner (R_out = 723 px) for r = 400 × atan(723/120) = 562 px — about 4 px of black right in the corner. Keep f near f_fish and the problem disappears: atan never reaches π/2, so with true 180° coverage nothing in a same-size output can sample outside the circle.

The second cost is stretch. Rectilinear radius grows as tan θ while the fisheye stores θ, so with the output focal length equal to the fisheye's, each source pixel must feed this many output pixels:

Commit to this cutoff: undistort out to a 60° half-angle, a 120° field, and no further. At that angle each source pixel already feeds about seven output pixels; past it, the majority of edge detail is interpolation and measurement there is unreliable.

Undistort or replace the lens

The mapping tells you when it should not be used at all:

• You need 120° or less and the camera sits at a fixed distance: fit a rectilinear wide-angle lens. No resampling, no invented pixels, and later pixel-to-millimetre measurements keep full resolution.
• You need more than 120°, or one camera must watch a whole workcell: keep the fisheye. Undistort only the region you process, or skip image resampling and map feature coordinates instead.
• The task is metrology at one working distance: use a low-distortion or telecentric lens, where correction is a few tenths of a pixel rather than a remap of the whole frame.

For the first case the numbers are ordinary lens selection. On a 2/3-inch sensor the LS-FFL-1582 is an 8 mm C-mount FA lens with a 57.6° field of view.

Pick the rectilinear field you actually need first, then let the calculator below turn sensor and field into the focal length to search for.

The lens finder then filters by that focal length, mount and sensor format, and the lenses topic page groups the lens guides on one page.

Where the coefficients come from

cx, cy, f_fish and the distortion terms come from a calibration: 15 to 30 photographs of a flat target, tilted and turned so corners spread across the whole field, including the edges where the model is most sensitive. A dot grid such as the LS-LXR-0029 board, 0.5 mm dots across 76 x 76 mm, gives the solver a dense, unambiguous pattern.

Size the board to the field: it must span a large fraction of the frame at the working distance, or the edge coefficients are unconstrained — calibration target field of view and lens coverage margins works through that sizing. When the solver returns, interpreting the coefficients of an unrectified camera image tells you what each term does.

Checks before you ship the mapping

• Straight-line test. Photograph a straight edge crossing the field. After undistortion, fit a line to it: a good calibration leaves under 0.5 px RMS residual over the region you use, under 0.3 px is excellent.
• Principal point sanity. cx and cy should sit within a few tens of pixels of the frame centre for a centred lens. A principal point at (0, 0) or in a corner means the pose set was bad, not that the lens is exotic.
• Stretch budget. Confirm the output half-angle stays at or under 60°, checked on the widest region you actually process, not the nominal field.
• Mask fraction. Count black output pixels and compare with the r_needed > r_circle prediction above; a mismatch means the coverage or focal length is not what you think.
• Precompute the map. (u_src, v_src) depend only on calibration, so build the arrays once and remap every frame. Real-time image undistortion in embedded vision pipelines covers the cost, and GPU lens undistortion mapping: coordinate and sampling checks the fast path.

If the decision lands on new optics, build the station in the system configurator; if the fisheye-versus-rectilinear call needs another opinion, the contact page reaches the applications team, and expert support comes free with every order.