Colour frame comes out doubled: fix Mat type pixel format mismatch

A colour frame that arrives doubled, stretched or shuffled is rarely a camera fault. It is almost always a mismatch between three numbers that must agree: the Mat's depth and channel count, the stream's pixel format, and the size of the byte buffer you copy into. This article is for anyone wiring an SDK colour stream into OpenCV, on a depth camera or on an area scan camera, who sees the same picture twice in one saved PNG. In ten minutes you will be able to name the mismatch, fix the declaration, and verify the saved image.

Symptom and what it tells you

The classic failure looks like this: the log prints sensible width and height, the save completes without an exception, and the file shows the scene content twice side by side, or in vertical stripes of green and magenta.

That pattern points to a Mat type pixel format mismatch. If OpenCV expects more bytes per pixel than the frame delivers, it keeps filling each row by wrapping into the following rows, so two logical frames land inside one image. If it expects fewer, the image is cropped or diagonal-shifted. Neither case corrupts the camera data; the wrapper is simply reading the buffer with the wrong shape.

The three numbers that must agree

Before touching code, write down three values for your stream.

1. Bytes per pixel. RGB8 and BGR8 deliver 3 bytes per pixel. YUYV packs two pixels into 4 bytes, so 2 bytes per pixel on average. A 16-bit mono or depth format delivers 2.
2. Stride. The byte width of one row, which is bytes per pixel × width rounded up to the driver's alignment, commonly 4 or 8 bytes.
3. Mat type. The OpenCV depth and channel count, which must consume exactly stride bytes per row.

If expected_mat_bytes and buffer_bytes differ, the resolution is not aligned and you must copy row by row or strip padding after the copy.

Worked example: 640 × 480 BGR8

A 640 × 480 colour stream in BGR8 carries 640 × 3 = 1920 bytes per row, which is already a multiple of 8, so stride equals 1920. The full frame is 1920 × 480 = 921,600 bytes.

Now look at the wrong declaration. A Mat of type CV_16UC3 at 640 × 480 expects 640 × 480 × 6 = 1,842,720 bytes, exactly twice the frame. OpenCV fills the first half of the Mat from the frame, then wraps and fills the second half with the same data again. The result is the face shown twice, which is precisely the doubling in the log's total() × elemSize() line. Switch the type to CV_8UC3 and the expected size matches the buffer, and the doubling disappears.

For comparison, a 16-bit depth stream at the same resolution needs CV_16UC1, consuming 640 × 480 × 2 = 614,400 bytes. The depth type and the colour type differ in both depth and channels, so never reuse one declaration for both streams.

RGB8, BGR8 and YUYV

OpenCV's convention is BGR, so the safe default for a colour stream is BGR8: wrap the bytes straight into an 8UC3 Mat and display. If the pipeline requests RGB8 instead, the geometry stays correct but red and blue swap, giving blue faces and orange greens; fix it with one cvtColor call using COLOR_RGB2BGR before matToBitmap or imwrite.

YUYV is different: it is not three bytes per pixel at all. Convert it with COLOR_YUV2BGR_YUYV immediately after the copy, then work in 8UC3. Wrapping YUYV bytes into an 8UC3 Mat without conversion produces the familiar green-and-magenta striped pattern.

The stride trap

Sizing the receive buffer as height × width × 3 works only when the resolution happens to be aligned. Consider a 1366 × 768 RGB8 stream with 8-byte row alignment: width × 3 is 4098 bytes, but the stride rounds up to 4104 bytes per row. A buffer sized from width is 4,608 bytes short over the frame, so the last rows arrive black or the copy overruns.

The robust habit is to allocate from the stride the frame object reports, copy the whole padded frame, and then either construct the Mat row by row or let the SDK hand you tightly packed data. When you control the camera side, choosing a resolution whose row width is naturally aligned, such as 1280 or 1920 pixels, removes the problem entirely. The LS-CMR-0053 is a 12 MP GigE area scan camera with a 4000-pixel-wide sensor; check its stride value in the SDK rather than assuming, and log it alongside the width so the buffer arithmetic stays predictable.

Fix in numbered steps

1. Log the configured stream format. If it is not RGB8 or BGR8, either re-enable the colour stream as BGR8 or plan the matching conversion (YUV2BGR for YUYV).
2. Declare the Mat to match: CV_8UC3 for BGR8, plus one cvtColor for RGB8, and only CV_16UC1 for a 16-bit depth stream.
3. Allocate the receive buffer as height × stride, taking stride from the frame object, not from width × 3.
4. Copy the frame into the buffer, then put or wrap it into the Mat in one call. Verify Mat.total() × Mat.elemSize() equals the buffer length before the copy.
5. If the source is RGB8, convert channels once with cvtColor before display or saving.
6. Save and inspect one frame at full size. The scene should appear once, at the right colours, with no vertical striping.

If the colour stream must line up with a depth stream afterwards, the timestamps and registration questions are covered in depth and color synchronization: check the right timestamps, and pixel-to-3D mapping continues in bounding box depth: mapping image pixels to 3D points.

Checks before you trust the pipeline

• elemSize() reports 3 for an 8UC3 colour Mat and 2 for a 16UC1 depth Mat. Anything else means the declaration drifted.
• The saved PNG's dimensions equal the stream's width and height, not a multiple of them.
• A flat white or grey target saves with equal R, G and B values; unequal values mean an RGB/BGR swap remains.
• Two consecutive frames differ; identical frames suggest the buffer is being reused without a fresh copy.

For capture-side settings that affect what the sensor delivers before any conversion, see camera pixel format not changing: why it is fixed and how to set it, and if frames stop arriving altogether, camera preview freezes after first frame: channel checks walks the acquisition path.

When the usual answer is wrong

Doubling can also come from things other than the Mat type. A put call in a loop that runs twice per frame, a preview callback that saves into the same Mat while a second thread reads it, or a file writer called from two stream callbacks all produce two images in one file. If the Mat type, stride and format all check out, serialise the save path behind a single owner, or clone the Mat at hand-off. Persistent memory trouble on the acquisition side has its own write-up in C# camera memory leak: manage image buffer ownership.

Quick checklist

• Stream format logged and known: RGB8, BGR8, YUYV or 16-bit.
• Mat type matches: 8UC3 for packed colour, 16UC1 for depth.
• Buffer sized from stride × height, verified against total() × elemSize().
• One channel-order conversion at most, applied after the copy.
• Saved frame inspected at full resolution, not in a scaled preview.
• Save path owned by one thread.

When the fix is confirmed, the fastest next step is to re-run your capture against a known-good camera: browse the area scan cameras or 3D sensors categories, use the camera finder to filter by interface and resolution, or build the rest of the chain in the system configurator. For help matching a camera to your SDK and buffer layout, contact us — free expert support comes with every order.