Stereo frames saved wrong: copy Mat data from the SDK buffer

You capture 100 stereo frames, push them into two vectors of cv::Mat, and settle in to save a clean image set. What lands on disk is something else: live frames mixed with captured ones, left and right imagers swapped in runs, and, once the pipeline stops before the save loop, files that fail to write at all. The vector still reports 100 images, so the bug hides well. This guide traces the failure end to end for a C++ capture loop: why a Mat built from a frame pointer aliases the SDK's buffer, why stop() finishes the damage, and the one-line fix that makes every saved file match the frame you actually saw. The same rule holds for any SDK that hands you a raw frame pointer, mono or colour, area scan or stereo.

What you will see

The capture loop behaves, then the save loop betrays you:

• During capture, the live window shows correct left frames.
• In the save loop, the same window shows the current live scene, not the stored frames.
• The saved files alternate imagers: two from one, two from the other, in runs, with ascending file numbers.
• Move the pipeline stop ahead of the save loop and the saves break entirely, yet left_images.size() still prints 100.

One root cause explains all four. The Mat objects in your vectors never owned pixels. Each one wraps a buffer that belongs to the SDK, and that buffer returns to rotation the moment its frameset goes out of scope. Everything after that is consequences.

The root cause: a Mat header points, it never owns

A cv::Mat is a small header plus a data pointer. The constructors that take a data pointer do not allocate pixel memory; they fill in rows, columns and type and store the pointer you handed them. That makes this line a wrapper, not a copy:

// Wraps the SDK's buffer, copies nothing
cv::Mat left_image(cv::Size(1280, 720), CV_8UC1, (void*)left_frame.get_data());
left_images.emplace_back(left_image);   // header copied, pixels shared

emplace_back copies the header into the vector, which is quick and still moves zero pixels. When the iteration ends, the frameset is destroyed, the SDK releases both frames back to its ring of buffers, and the next wait_for_frames() hands that memory out again. After 100 iterations you hold 100 headers per vector, all pointing into a handful of buffers that never stop turning over.

The SDK has to work this way: the camera writes each frame into pre-allocated transfer memory while you are still processing the previous one, so ownership stays with the pipeline unless you take it. Frame storage gets recycled unless your code explicitly holds it, and that contract repeats across SDK after SDK. Managed runtimes hit the identical wall from the wrapper side, as the managed-code frame lifetime guide shows.

Why each symptom follows

The live scene in your saved files: imwrite reads whatever the wrapped buffer holds at save time. By then the ring has recycled that memory many times, so most headers see the newest frame. File 0007 can carry frame 0093's pixels.

The left and right runs: both imagers deliver through the same buffer pool, served alternately as the streams arrive. Your headers were taken from that pool in delivery order, so the saved files group into short runs per imager, exactly the two-left, two-right pattern you watched.

The stop() failure: stopping releases the pool for real. Every Mat keeps the old address, so each read is undefined behaviour rather than an empty image. size() prints 100 because the vector owns its headers, and freed pixel memory says nothing about them.

The correct live display inside the capture loop is the last clue: during iteration i the buffer still held frame i. The corruption appears only once frames start outrunning their framesets.

Fix it: copy inside the capture loop

Three moves, in order:

1. Deep-copy each frame inside the iteration that produced it. clone() allocates fresh memory and copies every byte, so the vector owns pixels the SDK may recycle freely:

std::vector<cv::Mat> left_images, right_images;
left_images.reserve(100);
right_images.reserve(100);

for (int i = 0; i < 100; i++) {
    auto frames      = pipeline.wait_for_frames(); // buffers still owned by the SDK
    auto left_frame  = frames.get_infrared_frame(1);
    auto right_frame = frames.get_infrared_frame(2);

    cv::Mat left_wrap (cv::Size(1280, 720), CV_8UC1, (void*)left_frame.get_data());
    cv::Mat right_wrap(cv::Size(1280, 720), CV_8UC1, (void*)right_frame.get_data());

    left_images.push_back(left_wrap.clone());   // deep copy: 921,600 B you own
    right_images.push_back(right_wrap.clone());
}

2. Any deep copy counts. copyTo() into a preallocated Mat or a std::memcpy into your own buffer works the same; the requirement is that the bytes land in memory you allocated before the frameset's scope closes.

3. If your SDK exposes a keep-frame or retain call, use it as the alternative: mark each frame before the frameset dies and the SDK leaves that buffer alone, then copy when it suits the pipeline.

4. Read rows, columns and type from the SDK's frame description instead of hard-coded literals. A wrong type distorts the image even after an honest copy; the Mat type versus pixel format fix covers that second failure in detail.

5. Leave imwrite in the second loop. Encoding 100 PNGs takes seconds, and it now reads memory you own, so seconds are fine.

What the copy costs

So one clone costs about 0.05 ms, and two per frame is 0.1 ms of a 33.3 ms budget: 0.3%. Memory for the full set is about 184 MB, trivial on a capture workstation and worth reserving with reserve(100) so the headers never reallocate. The expensive step is PNG encoding, roughly 5-50 ms per textured frame on a desktop CPU, which is exactly why saving stays in its own loop or on a writer thread. Where the rest of the per-frame time goes, trigger to saved file, is the trigger versus exposure timing guide.

Cameras this loop fits

The fix is a property of the SDK contract, so it transfers unchanged: stereo depth cameras delivering paired frames and mono area cameras delivering single frames both need the clone at capture. For depth dataset work, the projected structured-light stereo camera below hands you the pair through one frameset, and each half needs its own clone:

For plain mono capture, the same 6 MP series comes in a GigE and a USB3.0 version, so the choice is mostly which port your host has free:

When each frame needs more detail, the 20 MP mono camera below saves through the identical clone-and-write path:

Browse the wider ranges in the stereo and structured-light 3D cameras and the area scan cameras, or filter by interface, shutter and resolution in the camera finder.

Prevention checklist

Set these once and the whole class of bug disappears:

• Copy at capture, save later. The clone belongs inside the frame's iteration; the save loop can run seconds or minutes later with no risk.
• Read the frame's real geometry. Take rows, columns and type from the SDK's frame description; literals that match today's stream break quietly when you switch formats.
• Watch 16-bit and 12-bit streams. Same rule, more bytes per pixel; the 12-bit image saving guide keeps intensity data alive through conversion and file formats.
• Put saving on a writer thread with a bounded queue. The frame grabber ring buffer guide shows the pattern, and the Python frame queue fix shows what happens when the queue grows unbounded. A bound of 16 frames at 0.92 MB is about 15 MB.
• Stop the stream last, after every stored frame is copied; the SDK shutdown order guide covers the teardown sequence.
• Match shutter to motion. Rolling shutter suits a static rig; global shutter when parts move quickly through the field.
• Skip the code when you can. For a quick sequence dump, the no-code consecutive image capture guide writes the files without a loop.

Tests that prove the fix

1. Capture a changing scene: wave a hand through the view during the 100 frames, which is 3.33 s at 30 fps (100 ÷ 30 = 3.33 s).
2. Open left_image_0.png and left_image_99.png. The hand must sit in different places; bit-identical frames of a changing scene mean the headers still alias.
3. Print the data pointers once: left_images[0].data, left_images[99].data and right_images[50].data must be three distinct addresses after the fix, instead of a few addresses cycling.
4. Open a left/right pair at the same index. Same instant, same scene, shifted by the stereo baseline; different moments mean the pairing broke.
5. Stop the pipeline before saving on purpose. With owned copies this now succeeds, which is the exact case that failed before.

The whole fix is one word, clone, and the rule behind it decides most capture code: a frame is yours only once its bytes sit in memory you allocated, or the SDK has been told to keep them. If you are picking hardware for this loop, the system configurator starts at the camera step, and our contact page takes your frame rates and pixel formats. More capture-side fixes are collected on the cameras topic page.