Mixing depth frames from two cameras: find the buffer bug

Two depth cameras on one host, and the streams start bleeding into each other: part of scene one appears inside scene two, half a frame at a time. The reflex is to blame the cameras, the driver or USB bandwidth. In most cases the cameras are fine. The bug sits in how your program owns frame memory, and it shows up in depth before RGB for a specific, predictable reason.

This article is for engineers integrating two or more depth cameras on a single host, whether that is a pair of structured-light cameras on a station or a ToF camera next to a profile sensor. You will learn why depth mixes while RGB does not, how to reproduce the fault in ten lines of code, how to isolate it in minutes, and the buffer ownership rule that prevents it coming back.

Why depth mixes and RGB does not

Depth data is usually delivered as a 16-bit buffer, and most SDKs deliver it zero-copy: the pointer you get points straight into a buffer the driver reuses for the next frame. Nothing protects that memory once you hold the pointer. If you queue the pointer, send it on another thread, or push it into a container, the next frame from the same camera overwrites the data behind your back.

With two cameras the failure looks stranger: frames from camera A and camera B interleave in one buffer, so the image appears to blend both scenes. In reality both devices usually write one shared or wrongly-indexed buffer, or one camera's pointer sits in a slot that the other camera's callback refills.

RGB survives for mundane reasons. It is often converted to another format, compressed, drawn to a preview or copied into an encode pipeline, and any of those copies hides the reuse. Depth goes straight from pointer to network or disk, so it is the first stream to expose the bug.

Reproduce it in a minimal test

Before changing your application, prove the platform is clean. Run the vendor's own multi-camera example with both cameras streaming depth on the same host. If the example runs clean for several minutes, enumeration, drivers, ports and cameras are all fine, and the fault is yours. That single test once saved an integration team days of firmware work.

Then write the smallest program that shows your bug:

1. Enumerate devices and open two, exactly as your application does.
2. For each camera, wait for a frame and take the depth pointer.
3. Instead of copying, store the pointer and its timestamp in a vector per camera.
4. One second later, copy all stored pointers out and save the images.
5. Inspect: if the saved depth images contain geometry from the wrong scene, you have reproduced buffer reuse by reference.

Swap one variable at a time to isolate it: store a deep copy instead of the pointer, and the mixing disappears. That is the diagnosis and the fix in the same experiment.

The likely causes, in order

1. Frames stored by reference. The pointer from the SDK is kept past the next wait-for-frame call. Probability: highest by far. Test: copy immediately after acquisition and compare.

2. A buffer shared between devices. One global staging buffer that both camera callbacks write into, or an array indexed by a loop variable instead of the device handle. Test: give each camera its own buffer and log the device index used in every write.

3. Callback and polling mixed. One camera polled in the main loop, the other driven by a callback that runs on the SDK's thread. The two paths race for the same container. Test: use one acquisition pattern for all cameras.

4. A container reallocated under a held pointer. A std::vector of frames that grows invalidates earlier pointers. Test: reserve capacity up front or store copies, not pointers.

5. A transport layer copying lazily. Your RTP or queue layer takes a reference for a deferred send. By the time the packet is built, the buffer holds the next frame. Test: serialize the frame into the packet before returning from acquisition.

For background on why holding camera memory past its lifetime crashes or corrupts even in a single-camera setup, see camera frame memory lifetime in managed code: avoid access violations and the related C# camera memory leak: manage image buffer ownership.

The fix: one owned buffer set per camera

The rule is short: copy the frame at acquisition time, into memory your program owns, before doing anything else with it. Copy once, immediately, on the acquisition thread.

Per-frame allocation works but fragments the heap on a long-running station. Better is a small ring of pre-allocated buffers per device:

Worked example: two cameras at 1280 × 1024, four frames in flight each. Buffer bytes = 1280 × 1024 × 2 = 2,621,440 bytes (2.5 MB). Buffers per camera = 6. Total = 2 × 6 × 2.5 MB = 30 MB. Trivial for any host, and it removes both the reuse bug and allocation jitter from your acquisition loop.

If your pipeline genuinely cannot afford the copy, the alternative is a double-buffer handshake where the SDK writes buffer N while you read buffer N−1, with an explicit swap you control. That is more code and more ways to get it wrong; start with the copy and optimize only if profiling says you must.

Choosing cameras so the integration stays simple

Buffer discipline is your code's job, but the camera choice affects timing margins. Structured-light cameras expose a full depth frame per capture and suit static or slow scenes on a shared host. ToF cameras stream continuously and reward a tight acquisition loop. Both behave identically with respect to buffer ownership.

For a second, independently timed depth source on the same host, a compact ToF camera is a common pairing; just give it its own ring of buffers and its own wait loop.

If your depth cameras share a host with conventional 2D inspection, the same ownership rules apply to area scan frames too; the area scan cameras category covers the GigE and USB3 models, and the 3D sensors page lists profilers, stereo and ToF families in one place. When several devices land on one machine, check host-side capacity early with the camera resolution calculator and our multi-camera acquisition guide on callbacks and buffer queues, which covers the queueing side of the same problem.

Checks before you redeploy

• Run the vendor multi-camera example clean for at least five minutes before blaming hardware.
• Log the device index and buffer address on every acquisition write; two devices writing one address shows up instantly.
• Store copies or ring-buffer slots in queues, never raw SDK pointers.
• Exercise the failure path: drop the network while streaming and confirm the queue does not grow past your ring.
• Test with cameras on different ports and in swapped order; a correct program is order-independent.

When the usual answer is wrong

Occasionally it really is not your code. Red flags that point upstream: the vendor's own example also mixes streams, corruption survives a full rewrite of your acquisition path, or frames from one camera appear in the other's handle after a device reconnect. Then the productive moves are a different USB controller or port set to rule out host-controller quirks, current firmware from the manufacturer, and a report to the vendor with your minimal reproduction. Even then, per-device owned buffers remain the right defensive design on your side.

Quick checklist

• Copy depth at acquisition time, before any queue, thread or network hop.
• One ring of pre-allocated buffers per camera, never shared.
• Index everything by device handle, not loop position.
• Verify with the vendor example first; your application second.
• Watch for RGB hiding the same bug in a different stream.

Next step

If you are planning a multi-camera depth station, start from the 3D sensors category, use the system configurator to match cameras to host and bandwidth, or contact us with your scene and frame rates. For the sync side of two-camera capture, see hardware sync two USB cameras: trigger both cameras together and multi-camera RGB synchronization: verify exposure timing.