getOptimalNewCameraMatrix zero ROI: calibrate at the size you use

You calibrate a wide-angle camera, call getOptimalNewCameraMatrix, and the function hands back a new camera matrix that squeezes the whole frame into one corner plus a validPixROI of [0, 0, 0, 0]. Undistort with that matrix and you get a small fragment of the scene on a black canvas. Undistort with the calibration matrix alone and the picture looks almost right, which makes the fault feel random. It is not random. Strong barrel distortion, a resized calibration set and the function's valid-region estimate interact in a way you can predict. Here is the whole chain for vision engineers feeding resized streams into SfM, optical flow or any pinhole geometry: read the ROI correctly, calibrate at native resolution or scale the intrinsics to the stream size, choose alpha deliberately, and run four checks before you build on the matrix.

What does an all-zero ROI from getOptimalNewCameraMatrix actually mean?

validPixROI is the axis-aligned rectangle of the undistorted frame that stays fully valid, and all zeros is the function's degenerate-rectangle answer: the distortion remap folded the corrected corners out of the image, so no valid rectangle exists to report.

The function estimates, for the alpha value you pass, how far the undistorted frame's valid region reaches, and it returns that region as a crop rectangle next to the new camera matrix. On a strong barrel lens the corrected corners wrap around past the frame border, the rectangle computes to a negative size, and the function clamps the result to all zeros. Nothing about your scene changed; the estimate collapsed.

Read the value as a diagnosis rather than a crop. The combination of distortion coefficients, camera matrix and target size has no single valid rectangle at that alpha, and each signature in the table sends you to a different fix.

Why does the new camera matrix collapse the frame into one corner?

The collapse is honest arithmetic on a broken estimate: the function's search for a usable scale wrapped past the frame border, the new focal length comes out far below the lens geometry, and undistort faithfully maps the frame into a tiny corner.

You can predict fx from plain lens geometry, which is what makes this failure easy to spot:

A 640-pixel-wide frame from a dashcam lens with a 120° horizontal field of view needs fx = 320 / tan(60°) = 320 / 1.7321 = 184.7 px. Run the formula backwards on a calibration that returned fx = 469.9 px and it gives 2 · atan(320 / 469.9) = 68.5°, an ordinary lens. If calibrateCamera hands you fx = 47 px or fx = 4700 px for the same 640-wide frame, the dataset is broken: flipped board corners, motion blur on a rolling shutter, or an imageSize argument that does not match the images. No alpha rescues a broken dataset. Fix it first; the checkerboard troubleshooting guide walks the tests in order.

The collapse itself follows from the estimate. getOptimalNewCameraMatrix searches for a scale at which the undistorted valid region fills the requested frame, and when the distortion map folds the corners through the border the search runs away. The fx it returns lands far below lens geometry, undistort applies it faithfully, and the corrected frame lands in a corner of the canvas. The remap is innocent; it did exactly what the matrix told it.

Should you calibrate on full-resolution images or scale the intrinsics?

Calibrate once on native-resolution frames, then scale the four pixel-valued intrinsics by s = stream width divided by calibration width: resizing only shrinks the pixel grid, so every pixel-unit value divides by the same factor while the distortion coefficients never move.

Worked example: calibrateCamera on 1920 × 1080 chessboard frames returns fx = 1450.0 px, fy = 1447.0 px, cx = 960.5 px, cy = 540.4 px, and you stream at 640 × 360, so s = 640 / 1920 = 0.3333.

• fx' = 1450.0 × 0.3333 = 483.3 px
• fy' = 1447.0 × 0.3333 = 482.3 px
• cx' = 960.5 × 0.3333 = 320.2 px
• cy' = 540.4 × 0.3333 = 180.1 px

The coefficients stay at [-0.32, 0.11, 0.0, 0.0, -0.05] in both matrices. They are dimensionless functions of the normalized image radius, a property of the lens rather than the pixel grid, and resizing cannot touch them.

import cv2
import numpy as np

# Illustrative values; the real numbers come from your calibrateCamera() run
camera_matrix = np.array([[1450.0, 0.0, 960.5],
                          [0.0, 1447.0, 540.4],
                          [0.0, 0.0, 1.0]])            # calibrated on 1920 x 1080
dist_coeffs = np.array([-0.32, 0.11, 0.0, 0.0, -0.05]) # k1, k2, p1, p2, k3

s = 640 / 1920                   # resize factor 0.3333
scaled = camera_matrix.copy()
scaled[0, 0] *= s                # fx 1450.0 -> 483.3
scaled[1, 1] *= s                # fy 1447.0 -> 482.3
scaled[0, 2] *= s                # cx  960.5 -> 320.2
scaled[1, 2] *= s                # cy  540.4 -> 180.1
# dist_coeffs stay unchanged: they are dimensionless

new_matrix, roi = cv2.getOptimalNewCameraMatrix(
    scaled, dist_coeffs, (640, 360), 0.0, (640, 360))  # alpha = 0.0
print(roi)
if roi[2] == 0 or roi[3] == 0:   # the test that proves the fix
    raise SystemExit("ROI still degenerate: sweep alpha or check sizes")

Calibrating directly on the 640-wide frames is also valid, and the scaling law lands within noise of the same matrix. Calibrating high and scaling down is still the better habit: cornerSubPix works on a finer pixel grid at native size, so the native-route matrix is better conditioned.

Cropping is the other half of the rule. A sensor or host-side crop shifts the principal point instead of scaling it: cx' = cx - x0, cy' = cy - y0, with fx and fy untouched. The same law decides pose accuracy in SolvePnP when a crop rescales the intrinsics, and camera ROI cropping at the source covers why you crop at all. Stereo rigs meet the identical choice as rescale or recalibrate on a resolution mismatch.

Factory cameras turn the whole question into a commissioning decision: fix the streamed resolution once, calibrate once at it, and every later scale is four multiplications. The LS-CMR-0166, for example, is a 12 MP USB3 camera that delivers 4000 × 3000 at 30 fps — one calibration at 4000 × 3000 serves any smaller preview you derive from it.

Which alpha value should you pass?

The free scaling parameter alpha sets what the undistorted frame keeps: alpha = 0 zooms in until every output pixel is valid and no black areas remain, alpha = 1 keeps every source pixel and accepts black corners, values in between interpolate.

The semantics are the function's documented contract. OpenCV's calibration and 3D reconstruction reference spells out the alpha behavior, and the official camera calibration tutorial shows the call inside the standard pipeline.

For detection and optical flow, alpha = 0 is the right default: black wedges seed false corners. For metrology, where dropped scene content is unacceptable, use alpha = 1 and mask the border. Pass centerPrincipalPoint as true only when your geometry assumes a centered pinhole. Compute the new matrix once per camera — it depends only on the original matrix, the coefficients and the two sizes — and recompute only when the stream size changes.

If you inherit a zero ROI, sweep alpha = 0, 0.5 and 1.0, print the ROI at each step, and take the first value that yields a positive-width rectangle. On strongly distorted lenses the step between neighbouring alpha values is large, not subtle.

Do you still need getOptimalNewCameraMatrix for SfM?

Yes if you track features with optical flow across frames: a pinhole-undistorted stream keeps the motion model linear and the trajectories stable, and the new matrix is computed once per camera, never per frame.

Structure from motion assumes near-pinhole projection between frames. Distortion adds a position-dependent warp that optical flow mistakes for motion, so undistorting once, before keypoint detection, is the low-cost fix.

Two alternatives are legitimate. Feed distCoeffs into your own projection and matching code and skip the undistorted stream entirely; feature matching does not require rectangular pixels. Or use the fisheye model for very wide lenses: cv2.fisheye.calibrate fits the equidistant model and cv2.fisheye.undistortImage maps it to a pinhole view without going through getOptimalNewCameraMatrix at all.

The deciding number is the field of view. Up to about 120°, rectilinear undistortion keeps edge detail usable. Past that, the rectilinear frame stretches the periphery until edge keypoints cost multiples of their center resolution, and the fisheye model wins on accuracy and speed. For the 120° to 170° lenses that produce these bugs, the fisheye route is usually the real fix: the zero ROI was a symptom of forcing a rectilinear model onto a lens that is not one.

Which checks prove the fixed calibration is usable?

Four checks decide it: a nonzero ROI, an fx that matches the lens's field of view, a reprojection RMS below about 0.5 px from a flat-board dataset, and straight structures that stay straight after undistortion.

Run them in order; the first one to fail names the stage to fix.

1. ROI sane. The call returns a rectangle with positive width and height at the stream size, and the same call at calibration size returns its own sane rectangle.
2. fx matches geometry. The calibrated fx agrees with the formula above within a few percent of the datasheet field of view; a mismatch indicts the dataset, not alpha.
3. Reprojection RMS low. With cornerSubPix converged to 0.001 and a rigid flat board, a healthy 1920-wide set lands below about 0.5 px RMS. Between 0.5 and 1 px, suspect board flex or rolling-shutter tilt; above 1 px, fix the dataset before touching alpha.
4. Lines stay straight. Undistort a frame with long straight structures and overlay the result: residual bow near the edges asks for more tilted board poses, not a different alpha.

Watch cx and cy as well. Both should sit within a few tens of pixels of the frame center, and a principal point that wanders is its own red flag, as when to distrust a principal point explains. Then apply the discipline from hand-eye calibration accuracy verification: prove the matrix on a scene you did not calibrate on before the line runs it.

What other faults produce a strange ROI?

Three further bugs mimic the degenerate ROI: an imageSize argument that does not match the matrix's calibration size, the new matrix passed into undistort's original-matrix slot, and the new matrix recomputed every frame with a drifting alpha.

Size mismatch is the classic follow-up to the fix above. getOptimalNewCameraMatrix takes imageSize and newImgSize, undistort takes dsize, and all three must state the same stream size. One leftover (1920, 1080) rescales the result around the wrong grid.

Recomputing per frame buys nothing. The new matrix is a constant of the camera and the output size, so call once, cache it, and recompute only on a resolution change. Per-frame calls slow the loop, and sweeping alpha per frame makes the edges breathe.

Next step: filter by resolution and interface in the area scan cameras range or let the camera configurator assemble a kit, and see 3D sensors for the stereo rigs where this same resize decision appears. The related guides sit on the 3D topic page and in the learning hub, and the contact page reaches the applications team.