Camera calibration returns five distortion coefficients: fix k3

You pass a four-element vector into cv2.calibrateCamera, and five coefficients come back. You try the CV_CALIB_FIX_K3 flag from the old cv module, and Python replies with 'an integer must be assigned'. Both symptoms share one cause: the length of the input vector is an initial guess, not a model size, and the flags that control the model live in the cv2 namespace as plain integers. The fix, for anyone calibrating area scan cameras or board-level cameras with OpenCV: pin k3 with cv2.CALIB_FIX_K3, prove the result with two asserts, and decide from your lens's field of view whether k3 should stay free.

What the input vector actually controls

The distortion vector you pass in sets starting values, not the number of coefficients the solver fits. With cv2.CALIB_USE_INTRINSIC_GUESS, OpenCV takes your four zeros as the starting values for k1, k2, p1 and p2, starts k3 at zero as well, and then optimizes every parameter you did not pin. The call returns five coefficients — (k1, k2, p1, p2, k3), packed as a (1, 5) array.

The line in the documentation about a four-element vector meaning k3 = 0 describes what a four-element vector means when you hand one to undistort or solvePnP. It is a statement about the exchange format, not a promise that the solver returns four elements. That is why np.zeros(4) changed nothing: the length of your guess never reaches the model selection.

Test it in one line before and after the call:

print(dist_init.shape)   # (4,)   — your guess
print(dist.shape)        # (1, 5) — the solver's output, five slots by default

If the fitted values look absurd rather than merely five long — k values in the hundreds, coefficients flipping sign between runs — the cause is thin or inconsistent data, not the vector length. The garbage depth camera coefficients article walks through diagnosing coefficient sets that cannot be real.

Why the legacy flag fails

CV_CALIB_FIX_K3 belongs to the old C API, exposed to Python through the legacy cv module. The cv2 binding type-checks its flag argument as plain integers, and a constant from that other API layer is what failed the check and produced the error text. The durable fix removes the ambiguity: build every flag set from cv2.CALIB_* constants and nothing else.

import cv2                      # never: from cv2 import cv
flags = cv2.CALIB_USE_INTRINSIC_GUESS + cv2.CALIB_FIX_K3
print(type(cv2.CALIB_FIX_K3))   # <class 'int'> — what the binding expects

Recent OpenCV releases removed the legacy cv submodule from the Python package entirely, so from cv2 import cv now fails at import time on an upgraded machine. Deleting the import costs nothing and removes both failure modes at once.

The same discipline carries into stereoCalibrate RMS problems, where a flag built from the wrong constants shows up as a failed assertion or an implausible RMS instead of a type error.

The fix, with the test that proves it

import numpy as np
import cv2

size = (1680, 1050)              # image width, height in pixels

# Per-view float32 corner arrays: 20 views of a 48-corner target
obj_points = [np.zeros((48, 3), np.float32) for _ in range(20)]  # target corners, mm
img_points = [np.zeros((48, 2), np.float32) for _ in range(20)]  # detected corners, px

# Initial guesses, float32 (example values)
camera_matrix = np.zeros((3, 3), np.float32)
camera_matrix[0, 0] = 2200.0     # fx, pixels
camera_matrix[1, 1] = 2200.0     # fy, pixels
camera_matrix[0, 2] = 750.0      # cx, pixels
camera_matrix[1, 2] = 750.0      # cy, pixels
camera_matrix[2, 2] = 1.0

dist_init = np.zeros(4, np.float32)   # a guess, not a model size

flags = cv2.CALIB_USE_INTRINSIC_GUESS + cv2.CALIB_FIX_K3

rms, K, dist, rvecs, tvecs = cv2.calibrateCamera(
    obj_points, img_points, size, camera_matrix, dist_init, flags=flags
)

# The test that proves the result:
assert dist.reshape(-1).size == 5    # five slots, every default run
assert dist.reshape(-1)[4] == 0.0    # k3 was pinned, not fitted
print(rms)                           # RMS reprojection error, in pixels

cv2.CALIB_USE_INTRINSIC_GUESS tells the solver to start from your camera_matrix; cv2.CALIB_FIX_K3 holds k3 at its initial zero through the whole optimization. The output is still a five-slot array, and the last slot stays exactly 0.0 — the four-coefficient model, expressed in the five-slot format your undistort calls already expect.

Should k3 be free? The corner arithmetic

Radial distortion scales the normalized image coordinate x, y by a factor S:

Everything k3 can do lives in the r⁶ term, which only grows large near the frame edges. Two examples, both on a 1680 × 1050 sensor.

Example 1 — the guesses from the failing call: fx = 2200 px, cx = 750 px, cy = 750 px. The farthest corner from the principal point sits at pixel (1679.5, 1049.5):

• x_n = (1679.5 − 750) / 2200 = 0.423, y_n = (1049.5 − 750) / 2200 = 0.136
• r² = 0.423² + 0.136² = 0.197, so r = 0.444 and r⁶ = 0.197³ = 0.0077
• an extreme k3 = 0.05 gives ΔS = 0.05 × 0.0077 = 3.8 × 10⁻⁴
• shift ≈ 2200 × 0.444 × 3.8 × 10⁻⁴ ≈ 0.4 px

For comparison, k1 = 0.1 gives ΔS = 0.1 × 0.197 = 0.0197 and a 19 px shift at the same corner: on this lens, k1 does the work and k3 is noise.

Example 2 — the same sensor behind a wide lens: fx = 1000 px, cx = 840 px, cy = 525 px. Corner pixel (1679.5, 1049.5):

• x_n = 839.5 / 1000 = 0.840, y_n = 524.5 / 1000 = 0.525
• r² = 0.840² + 0.525² = 0.981, so r = 0.991 and r⁶ = 0.981³ = 0.944
• a plausible k3 = 0.01 gives ΔS = 0.01 × 0.944 = 9.4 × 10⁻³
• shift ≈ 1000 × 0.991 × 9.4 × 10⁻³ ≈ 9.3 px

In pinhole terms, Example 1's lens has a horizontal field of view of 2 × atan(840 / 2200) = 41.8°, Example 2's 2 × atan(840 / 1000) = 80.1°. The formula ignores distortion, which is close enough for choosing a model.

The decision, from those inputs:

• Horizontal FOV below 60°: pin k3 with cv2.CALIB_FIX_K3 whenever the data is thin. Example 1's 41.8° lens cannot produce a corner shift above about half a pixel even for an extreme k3, so the free parameter only adds variance.
• Horizontal FOV above 70°: leave k3 free. Example 2's 80° lens shows a plausible k3 moving edge points by roughly 9 px; pinning it biases every undistorted measurement near the edges.
• In between: calibrate twice, once with and once without cv2.CALIB_FIX_K3, same images. Corner detection on a decent printed target repeats to about 0.1 px, so if the pinned RMS is within 0.1 px of the free RMS, keep it pinned — the same accuracy with one fewer parameter.

Model sizes beyond five

Five slots is the default, not the ceiling. The flags that grow the model live in the same namespace:

Downstream, undistort and solvePnP accept 4, 5, 8, 12 or 14 element vectors — a four-element vector is simply the k3 = 0 case. To read the values you actually got, including which signs and magnitudes are plausible for which lens, see interpreting distortion coefficients.

The OpenCV fisheye module is a different model with four coefficients of its own, a polynomial in the viewing angle θ rather than powers of r. Its distortion vector is not interchangeable with the one calibrateCamera returns, and mixing the two is the classic fisheye undistortImage assertion error.

Prevention: one namespace, plain integers, data that reaches the corners

• Delete from cv2 import cv from every calibration script, and build flags only from cv2.CALIB_* constants, one comment per flag.
• Keep the two asserts from the fix section in the script permanently. They cost microseconds and catch a wrong-flag refactor before it produces a plausible-looking but wrong calibration.
• Feed the solver 10 to 20 views of a planar target: straight-on plus tilted poses, with the target corners reaching into all four quadrants of the frame. The r⁶ term is only measurable near the edges, so a target that never enters the corners leaves k3 unconstrained — a single-view, 16-point set is exactly that case. Grid pitch and detection errors on the target have their own write-up in the checkerboard calibration grid pitch fix.
• Save the result at full float precision, and load it with locale-independent parsing. Calibration files that round-trip through locale-sensitive number parsing produce the calibration file locale error, where the coefficients will not load as numbers at all.

Checks before you trust the result

• Shape and pin: dist.reshape(-1).size is 5, and with cv2.CALIB_FIX_K3 the fifth element is exactly 0.0.
• RMS in pixels, printed, and compared against a run with k3 free — apply the 0.1 px rule from the decision above.
• Edge residuals: project your target into the corners of a test image and inspect residuals there, not just the frame-average RMS, because k3 lives at the edges.
• Straightness: undistort one image containing a known straight edge — a machine rail, a door frame — and confirm it stays straight to within about a pixel at the frame edges.
• Downstream: if the calibration feeds a robot pick or a measurement, verify against a known artefact before production; the hand-eye calibration accuracy verification method is built for exactly that step.

If you are still choosing the hardware these scripts will calibrate, the camera finder filters by resolution, shutter and interface, and the camera configurator walks the rest of the chain. The 3D topic page gathers the calibration and depth articles that pair with this one.