Fisheye undistortion field of view loss: stitch on a sphere

You calibrated a fisheye camera, ran the undistortion, and got back a small, narrow picture with the rim gone — the very rim a stitch needs. Nothing is broken. The output field of view is set by the new camera matrix handed to the remap, and its default keeps only the undistorted center. This guide shows where the crop comes from, how much view a rectified frame can physically hold, how to drive the balance parameter and the output size, and when to stop rectifying altogether and put both cameras on a shared spherical canvas.

The new camera matrix sets the output view

The lens did not change; the projection did. Flattening a fisheye frame needs a destination camera matrix, and estimateNewCameraMatrixForUndistortRectify from the cv2.fisheye module builds it. Its balance argument picks the new focal length between two ends. Near 0 the matrix shows the largest region with no empty pixels, which on a wide fisheye is a modest center crop. Near 1 it shows every pixel the lens captured, at a lower pixel scale, with empty corners. The default is 0, so an unmodified call crops hardest.

A rectified frame is a pinhole projection, and pinhole geometry sets a hard wall. A pinhole image places a ray at radius f·tan(theta), so a ray 90° off axis would need infinite radius. No rectified output at any resolution holds it: content beyond 90° is unreachable in principle, and content between roughly 60° and 90° survives only as heavy stretch.

Commit to one of three paths:

• Lens 120° or narrower: rectify normally. The table below shows you lose almost nothing.
• One camera, rim needed: raise balance toward 1 and accept the scale drop, or enlarge the output canvas.
• Two lenses around 150° or wider, built for stitching: skip pinhole rectification and remap both cameras onto one spherical canvas.

For non-fisheye lenses the same trade appears as the free scaling factor of the remap; undistortion scaling parameter: cropping versus field of view works through that case.

How much view a rectified frame can hold

The numbers for an output the size of a 640 px input, with a 224 px source focal length:

This table is pure geometry: only tan and the output width set it, whatever the lens and whatever the library. Past a 90° half-angle there is no entry at all.

The same arithmetic, run backwards from a target view to the focal length that delivers it, is what the lens calculator on this site does:

Worked example: a 224 px focal length on a 640 px frame

Take an example calibration of the kind a 640 px board-camera fisheye produces: fx = 223.66 px, fy = 259.93 px, cx = 290.87 px, cy = 309.10 px, four distortion coefficients k1 through k4 each below 0.016 in magnitude, fitted from 43 checkerboard poses.

First, a sanity check that fails. fx and fy differ by 36 px, which is 16%. On a square-pixel sensor those two numbers are the same physical length, and a healthy fit lands within about 2%. Refit with more poses tilted 30° to 45° before trusting anything below: the field of view swings with that gap, 141° through fy against 164° through fx, and the fitted k terms push the fx figure toward 176°. Evaluated with these coefficients, the mapped radius also stops growing near 350 px, so the frame corners lie outside what this fit describes at all.

Now the design estimate from the first-order model, using fx. The usable circle touches the half-width at r = 320 px, so theta = 320/224 = 1.43 rad = 82.0°: a 164° lens. Two outputs to compare:

• Default rectification at the source focal length: the fully visible cone is 2·atan(320/224) = 110°, with corners reaching 127° along the diagonals. The lens delivers 2π(1 − cos 82°) = 5.4 steradians; the rectified rectangle holds about 3.5 sr, so 65% survives. The missing 35% is the rim — the seam material.
• Full circle at balance near 1: f_new = 320/tan(82°) = 45 px. Pixel scale falls to 45/224 = 20%, five times fewer pixels per object at the center, and the blend samples a heavily stretched rim.

Keep the view: the procedure

1. Refit until fx and fy agree within about 2%. Add tilted poses, not more frontal ones, and re-check the reprojection error. A board camera ships with no lens profile, and building a lens distortion profile for a board camera covers that step. Use a proper calibration target rather than a printed sheet: flatness and pitch accuracy are the point.
2. Write down the field of view you must keep, then the seam requirement. For a stitch, plan at least 15° of shared view; below that the blend has too little common texture to align on.
3. Sweep balance and print the returned focal length each time:

import cv2
import numpy as np

K = np.array([[223.66, 0.0, 290.87],
              [0.0, 259.93, 309.10],
              [0.0, 0.0, 1.0]], dtype=np.float64)            # float64, from your calibration
D = np.array([-0.0157997, -0.00707966,
              0.00217689, -0.000716948], dtype=np.float64)   # float64, k1..k4
dim = (640, 640)                                             # (width, height), pixels
balance = 0.5                                                # float, 0.0 to 1.0

new_K = cv2.fisheye.estimateNewCameraMatrixForUndistortRectify(
    K, D, dim, np.eye(3), balance=balance)

print(new_K[0, 0])   # new focal length in pixels:
                     # ~224 = sharp center only, ~45 = the whole 164-degree circle

The printed number, read against the table above, tells you exactly what you kept — no eyeballing required. Pick the smallest balance whose empty corners your software tolerates. To keep scale instead, pass a larger newSize in the same call and let the focal length follow the canvas.

4. For a single inspection camera, stop here. Run the remap with the matrix you chose, count the empty pixels, and mask the corners explicitly if your measurement region touches them, so edge detection does not fire on the boundary.
5. For a stitch, do not rectify at all. Map fisheye pixels straight onto the spherical canvas of the next section. The mapping arithmetic — distortion polynomial, rotation, bilinear sampling — is laid out in fisheye undistortion: how the pixel mapping works. If the library rejects your coefficients outright, the distortion-model mismatch behind the fisheye undistortImage assertion error names the fit you need.

Stitch on a sphere: canvas size and overlap

An equirectangular canvas stores the whole sphere on a 2:1 image: longitude across, latitude down. For each output pixel, compute the ray, rotate it into each camera's frame with that camera's rotation, and read the source pixel through the fisheye model:

Size the canvas to the source sampling, not to a round number. The example camera delivers 1 px per 1/224 rad along any radius, so a full 360° needs W = 2π·224 = 1407 px: use 1408 × 704. A 4096 × 2048 canvas upsamples about three-to-one in each direction and wastes memory; a 512 × 256 canvas throws away most of what the sensor paid for.

Overlap decides whether a stitch is possible at all. With two 164° lenses the shared band is 82° + 82° − alpha, where alpha is the angle between the axes:

• Back to back, alpha = 180°: 82 + 82 − 180 = −16°. No overlap at all, and a 16° blind band runs exactly where the seam should be.
• Perpendicular, alpha = 90°: a 74° shared band around the bisector, comfortable to stitch, but a 53° blind cone sits behind the bisector (135° − 82°).
• Full sphere from two cameras: both lenses must reach 180°. The 190° class gives a 10° band (95 + 95 − 180); aim that band at scenery with texture, not at a fixture pole.

To register more than two frames into one frame of reference, three image stitching with a shared coordinate frame extends the same idea.

Pick the lens for the path you chose

For the spherical path you want a true fisheye on each camera. The LS-MBL-0041 is a 1.45 mm M12 fisheye lens for 1/2.3-inch, 10 MP sensors.

Image circle, back focus length and mount clearance matter more than focal length here; fisheye lens selection for M12 board cameras works those numbers, and the M12 board lenses category and the lens finder list what fits your sensor.

For the rectify-normally path at 120° and below, a low-distortion wide lens avoids the whole problem. The low-distortion 3.3 mm LS-MBL-0159 and the 12 mm LS-MBL-0012 sit in the catalog's low-distortion series.

Run your own numbers through the lens configurator before ordering, and see the lenses topic page for mounting and focus guides.

Checks before you order the second camera

• fx and fy within about 2% after the refit; if not, every number downstream is soft.
• The printed new focal length matched against the table, so the kept cone is known before anything is bought.
• The overlap band computed from both lenses' measured half-angles at your mount angle — 15° minimum for a blend with shared texture.
• A calibration target large enough to fill the seam region; a small board calibrates only the center of the frame.
• The empty-corner fraction at your chosen balance, with a mask plan if measurements touch the corners.
• One resampling step in the pipeline: fisheye to sphere, never fisheye to pinhole to sphere.

When the usual fix is the wrong fix

Setting balance to 1 keeps every angle but is the wrong tool for a two-camera stitch past about 150°: the output still lives in a pinhole frame at 20% scale, and a later spherical remap resamples it a second time. Enlarging the canvas holds scale only by multiplication: an 82° rim at a 224 px focal length lands 224·tan(82°), about 1594 px from the center, so the canvas grows to 3188 px wide — the FAQ prices what that costs per frame. And a lens at 180° or wider carries rays at or beyond 90° off axis, which need infinite image radius: it can never be fully rectified at any size, and it belongs on the sphere from the start.