System designTroubleshooting

cv::undistort output garbage or blank: Mat pointer wraps your array

Why cv::undistort output comes back garbage or blank when your Mat pointer wraps a stack array, how to test for it and the fixes that cannot alias.

LumaSpec Vision10 min read Share on LinkedIn

On this page 8 sections
  1. 01The symptom: raw frame fine, undistorted frame destroyed
  2. 02The cause: a Mat built from a pointer wraps memory, it never copies
  3. 03Likely causes in order of probability
  4. 04Worked example: the 640 × 480 loop that overwrote fx
  5. 05Fix it in five steps
  6. 06Keep it from coming back
  7. 07Where this bites in a vision build
  8. 08Questions

An undistorted frame that comes out blank gray, pure noise, or patterned garbage while the raw frame looks perfect is usually not a calibration problem. It is a memory problem: a cv::Mat constructed from your own array wraps that array without copying it, so any later write into the array silently edits the camera matrix that undistort reads. This page shows the symptom, the Mat behavior behind it, the causes in order of probability, a worked 640 × 480 px example you can check by hand, and the fixes that make the aliasing impossible.

The symptom: raw frame fine, undistorted frame destroyed

The raw frame shows the scene as the sensor delivers it. The undistorted frame is uniform gray or black, or carries noise and stray symbols that have nothing to do with the scene. Two details tell you this is overwritten memory, not a bad calibration. First, changing the type argument in the Mat constructor, CV_64F to CV_32F for example, changes the pattern of the garbage: the header is re-reading the same bytes a different way. Second, printing the camera matrix with std::cout << CameraMatrix shows values you never assigned, typically fx = 1 px where the calibration output printed 953.276 px. A bad calibration warps or zooms the image; overwritten memory gives you no image at all.

The cause: a Mat built from a pointer wraps memory, it never copies

OpenCV splits every Mat into two parts: a header that stores rows, cols, type and a data pointer, and the element memory itself. The constructor that takes a user data pointer assigns the pointer and stops there: no allocation, no copy, and the Mat will not free your array either. Passing &data instead of data is not the problem; both are the same address. From construction to destruction, the array and the Mat are the same bytes.

double data[3][3] = { {953.276, 0.0, 320.0},
                      {0.0, 953.276, 240.0},
                      {0.0, 0.0, 1.0} };
cv::Mat CameraMatrix(3, 3, CV_64F, &data);   // wraps data: header only, no copy
data[0][0] = 1.0;                            // camera matrix silently becomes fx = 1 px
std::cout << CameraMatrix;                   // prints fx = 1 at [0,0], not 953.276

The diagonal write above is a reduced version of the classic accident: a loop meant to zero a separate 3 × 3 array sets the calibration array's diagonal to 1 instead. The same header-versus-memory distinction decides whether a saved stereo frame holds real pixels, which copying Mat data from the SDK buffer walks through frame by frame. The cure is to own the memory:

cv::Mat CameraMatrix = cv::Mat(3, 3, CV_64F, &data).clone();  // header + private buffer
data[0][0] = 1.0;                                             // matrix now unaffected

clone() and copyTo() allocate a buffer the Mat owns. After a clone, the original array is just your scratch space and the matrix keeps its values no matter what happens to the array.

Likely causes in order of probability

Before blaming aliasing at all, confirm the calibration numbers themselves were stored correctly: calibrateCamera returning wrong intrinsic matrix values covers the read-with-wrong-depth trap. Then work down this table; the first row explains most cases.

Rank Cause Test Decision
1 Code writes into the wrapped array after construction Print the Mat right before undistort; search the source for the array name Clone at construction or make the array const
2 The wrapped array was a local that went out of scope Values print correctly inside the function that built the Mat, garbage at the call site Clone before returning; never store a Mat that wraps a local
3 Type or row-step mismatch: a float array wrapped as CV_64F, or padded rows Swap the type and watch the garbage pattern change; compare the element size with sizeof(double) Match the type to the array (CV_32F for float) and pass the step when rows are padded
4 Distortion coefficient count or order wrong for the model Print it: undistort accepts 4, 5, 8, 12 or 14 coefficients Use the count the calibration wrote; the 5-element order is k1, k2, p1, p2, k3
5 newCameraMatrix argument misused Remove the argument: empty means undistort reuses the camera matrix Leave it empty for a like-for-like result; pass a getOptimalNewCameraMatrix result only for a deliberate crop

Worked example: the 640 × 480 loop that overwrote fx

Inputs, taken from the failing session itself:

  • Frame 640 × 480 px; calibration fx = fy = 953.276 px, cx = 320 px, cy = 240 px.
  • Distortion, 5-element model: k1 = -0.119190, k2 = -2.947282, p1 = 0, p2 = 0, k3 = -19.208490.
  • Mean reprojection error 0.356 px, so the calibration fit itself was sound.
  • The bug: a nested loop meant to zero a separate 3 × 3 array sets data[i][j] = 1 on the diagonal of the wrapped calibration array instead. After the loop, fx = fy = 1 px; cx and cy are unchanged.

The rule: undistort back-maps every output pixel. It converts the output pixel to normalized camera coordinates with the camera matrix, evaluates the radial polynomial there, and samples the source image at the result:

x_n = (u - cx) / fx y_n = (v - cy) / fy r2 = x_n^2 + y_n^2 u_s = fx * x_n * (1 + k1*r2 + k2*r2^2 + k3*r2^3) + cx

Healthy matrix, at the right-edge middle pixel u = 640 px, v = 240 px: x_n = (640 - 320) / 953.276 = 0.33567 and y_n = 0, so r2 = 0.33567² = 0.11268. The polynomial scales the coordinate by 1 - 0.119190 × 0.11268 - 2.947282 × 0.11268² - 19.208490 × 0.11268³ = 1 - 0.013430 - 0.037421 - 0.027480 = 0.92167, and the source pixel lands at u_s = 953.276 × 0.33567 × 0.92167 = 294.9 px, call it 295 px, comfortably inside the frame.

Corrupted matrix, same pixel: x_n = (640 - 320) / 1 = 320, so r2 = 320² = 102,400 and the terms explode. k1 × r2 = -0.119190 × 102,400 = -12,204.7. r2² = 102,400² = 1.048576 × 10¹⁰, so k2 × r2² = -2.947282 × 1.048576 × 10¹⁰ = -3.09062 × 10¹⁰. r2³ = 102,400³ = 1.073742 × 10¹⁵, so k3 × r2³ = -19.208490 × 1.073742 × 10¹⁵ = -2.0625 × 10¹⁶. The scale factor is about -2.0625 × 10¹⁶, and the source coordinate lands at u_s = 1 × 320 × (-2.0625 × 10¹⁶) = -6.6 × 10¹⁸ px.

Every output pixel asks for a source pixel roughly 6.6 × 10¹⁸ px from the image center, about 10¹⁶ frame widths outside a 640 px frame. Nothing valid gets sampled and the output fills with black, gray or noise: exactly the destroyed frame on screen.

The decision: a radial polynomial is only valid inside the range it was fitted on. This calibration reaches r2 = 0.17606 at the image corner (0.33567² + 0.25176² = 0.11268 + 0.06338), while the corrupted matrix evaluates it at r2 = 102,400, an extrapolation by a factor of about 5.8 × 10⁵. fx must print 953.276 px before every call. If it prints 1 px, garbage output follows from this arithmetic alone, and no type, flag or ROI change can rescue it.

Fix it in five steps

  1. Print both matrices immediately before the failing call: std::cout << CameraMatrix << ' ' << distCoeffs << std::endl;, and compare fx, fy, cx and cy with the calibration output.
  2. Search the source for the wrapped array's name. Every write after the Mat constructor is a suspect; in the worked case it was data[i][j] = 1 inside a loop aimed at a different array.
  3. Remove the alias by giving the Mat its own buffer, with one of the two patterns below.
  4. Declare the calibration array const double data[3][3] = {...}: a later write then fails to compile instead of failing on the line.
  5. Load the calibration from the file the calibration tool wrote, through FileStorage (OpenCV's YAML and XML reader and writer), and print what was loaded at start-up.
// Option A: clone the wrapped array once; the Mat then owns its buffer
cv::Mat CameraMatrix = cv::Mat(3, 3, CV_64F, &data).clone();

// Option B: build an owning matrix in one line with the Mat_ comma initializer
cv::Mat CameraMatrix = (cv::Mat_<double>(3, 3) << 953.276, 0.0, 320.0,
                        0.0, 953.276, 240.0, 0.0, 0.0, 1.0);

Verify by repeating the edge-pixel check: with the calibrated fx back in place, the pixel at u = 640 px samples the source at about 295 px, and the undistorted frame differs from the raw one only where lines are straightened.

Keep it from coming back

  • Treat every Mat built from a pointer as a view, never an owner. Owning buffers come from clone(), copyTo(), or constructors that allocate their own storage.
  • Wrapping the first element of a std::vector has the same trap one level up: a resize can reallocate the vector's buffer and leave the header pointing at freed memory. Reserve the size once or clone.
  • For per-frame work, build the maps once with initUndistortRectifyMap and call remap. This is also where an aliased matrix does its longest damage: maps computed from a wrong matrix keep feeding wrong frames until you rebuild them, so rebuild after every calibration change.
  • The pointer-only habit fails elsewhere too; zero copy image buffer access on GigE covers the buffer-lifetime version on the acquisition side.
  • Silent value drift has hardware cousins as well: camera float values are silently snapped to the nearest increment is the acquisition-side version of values that look right and are not.
  • Assuming a copy happened when only a reference was made is not OpenCV-specific either: on the controls side the same failure class shows up as ControlLogix COP array copy garbage data after a copy instruction that moved less than the whole array.

Where this bites in a vision build

Everything measured downstream inherits fx and fy. Calibrate a lens on an area scan camera, undistort or rectify every frame, then measure edges, codes or scales: a matrix that silently changes fx corrupts every measurement, not just the picture. The same path runs inside 3D sensors whenever a raw stereo pair is rectified before depth is computed; check the streams first with raw stereo image calibration.

A 12 MP GigE model such as the LS-CMR-0053, or a 20 MP USB3.0 mono model such as the MV-CS200-10UM, is typical of the frames moving through this path.

Area scan cameras12 MP GigE area scan camera, IMX226, 1/1.7" rolling shutter, 9 fps (LS-CMR-0053)LS-CMR-0053
Sensor format
1/1.7"
Lens mount
C-mount
$560.00View part
Area scan camerasHikrobot MV-CS200-10UM 20 MP Area Scan Camera, USB3.0, IMX183, 2nd Gen., MonoMV-CS200-10UM
Sensor format
1"
Lens mount
C-mount
$759.00View part

Stereo hardware such as the MV-DLS300P-04 laser galvanometer stereo camera sits one step further down the same road: where you rectify a raw pair yourself, an aliased matrix bends the robot's world, not only the picture.

3D sensorsHikrobot MV-DLS300P-04 Laser galvanometer stereo cameraMV-DLS300P-04
Type
Stereo / structured light
$16,579.00 · quotedView and quote

The one-line decision for the bench: when the undistorted frame comes back as garbage, do not touch the distortion coefficients, print the Mat. fx reading 1 px instead of the calibrated value is the whole diagnosis; clone the matrix or rebuild it with the comma initializer and the frame returns. To pick the camera side of that pipeline, start at the camera step of the system configurator or the camera finder; the systems topic page collects the rest of the runtime traps.

Questions

Does a Mat built from a pointer ever copy the array?

No. The constructor Mat(rows, cols, type, data) wraps external memory: the header stores rows, cols, type and the data pointer, nothing else, and the Mat does not free that memory either. The same applies to wrapping the first element of a std::vector. Call clone() or copyTo() when the Mat must own its buffer; after that, later writes to the original array leave the matrix untouched.

Why did changing the type argument change the garbage pattern?

The type tells the header how to interpret the bytes it points at: element size, channel count and row step. CV_64F reads 8-byte doubles and CV_32F reads 4-byte floats, so the same memory prints as different values. If swapping the type changes the pattern of your garbage, the Mat is wrapping memory instead of owning it.

Should I pass newCameraMatrix to undistort?

Leave it empty for a like-for-like result: undistort then reuses the camera matrix itself, with an identity rotation. Passing an identity 3 × 3 matrix there does not produce garbage; it scales the image down by roughly the focal length in pixels. Passing a getOptimalNewCameraMatrix result crops or zooms on purpose. None of that replaces checking that fx and fy still print their calibrated values.

How can I catch a corrupted camera matrix before it reaches the line?

Print the matrix once per session and compare it element by element with the calibration output. In the worked case, fx = 953.276 px and fy = 953.276 px must appear before every undistort call; a printed 1 px is the entire diagnosis. Storing calibration in YAML through FileStorage and logging the loaded values at start-up makes the check run automatically.

Does the same aliasing corrupt initUndistortRectifyMap?

Yes, and more permanently. initUndistortRectifyMap computes the maps once from the camera matrix and distortion coefficients; maps built from an aliased matrix keep feeding wrong frames to every later remap call, even after you correct the array. Rebuild the maps whenever calibration values change, and spot-check one edge pixel against a hand calculation.