Sync RGB and depth frames by device timestamp, not queue order

Your RGB stream and your depth stream start in step, then the delay between them grows minute by minute and only a restart clears it. You are pairing frames by queue order: one get() per stream, keep what comes back. Any rig that streams two outputs from one device drifts this way once the host loop misses a beat — RGB plus disparity, a mono pair plus point cloud, colour plus depth. This page shows how to prove the cause with the device timestamps already carried on every frame, how to pair with a half-frame-period tolerance, and the arithmetic that predicts how fast the offset grows and why reconnecting resets it.

Symptom: the gap grows and a restart clears it

You print the difference between the two device timestamps every second. For the first few minutes it reads single-digit milliseconds. Then it starts climbing in steps, and after an hour the RGB picture shows a scene seconds away from the disparity picture. Nothing in the loop brings it back; only stopping the program, or reconnecting the device, resets the gap.

Two observations narrow the cause fast. The sign of the difference tells you which stream is falling behind in your pairing. The shape of the growth separates lost frames from clock problems: clocks drift smoothly, lost frames ratchet in whole frame periods. Two stamps read from one device share one oscillator and cannot wander apart by themselves, so on a single-device rig a growing gap is a pairing problem until proven otherwise. Even a poor crystal, off by 50 ppm, pulls two separate devices apart by 50 µs per second — 180 ms per hour, far too slow to explain a gap of seconds, and a reconnect would not reset it.

Likely causes, most likely first

Queue-order pairing with silent drops

The failing pattern pairs by alternation: one get() on the RGB queue, one get() on the depth queue, store both results. Both outputs are configured to queue size 1 and non-blocking (setQueueSize(1), setBlocking(false)), so each queue holds only the newest frame and discards older ones without any error. The moment the loop takes longer than a frame period, drops begin on whichever stream delivers while you are busy.

Every one-sided drop shifts the pairing by one frame and adds one frame period to the gap — 41.7 ms at 24 fps. Drops on the two streams rarely alternate perfectly, so the gap ratchets in one direction, grows without limit and never recovers on its own. Reconnecting flushes both queues and restarts the device, which is exactly why the reset appears to fix it for a while.

Test: log both device timestamps for every pair you emit and divide the difference by the frame period. Healthy pairs sit well inside ±0.5 periods; one-sided drops show as jumps to ±1.0 periods or a whole multiple. Then count delivered frames per stream over a fixed minute — 1,440 per stream at 24 fps, no matter what your loop consumed.

One stream runs slower than the other

Depth costs more than RGB: the stereo stage — left-right check, subpixel — runs per frame on the device, and exposure time can cap the frame rate before your code does anything wrong. If the depth side delivers 20 fps while RGB delivers 24, no scheme matches one-to-one: alternating pops drift by 4 frames every second, and 4 × 41.7 ms = 166.8 ms per second, about 10 s of drift per minute.

Test: count frames per stream for 60 s while doing no other work. A stream that returns 1,200 frames instead of 1,440 is running at 20 fps, and long exposure is the usual reason — the depth camera frame rate drops: exposure settings cap the FPS guide walks that diagnosis. Fix: run both sensors at the same fps, keep exposure below one frame period, and still pair by timestamp as shown below.

The stamps come from different clocks

A difference only means something when both values are stamps from the same device clock. If one value is a device timestamp and the other is host arrival time, or the RGB comes from a separate camera, you are measuring transfer and scheduling, not capture offset.

Test: confirm both values come from the same accessor and the same device — in the SDK where this failure appeared, that is getTimestampDevice() on each ImgFrame. For two-device rigs, share a clock first: hardware trigger lines, or network time sync such as PTP where the cameras support it. The multi depth camera sync configuration and validation guide covers the set-up, and pair frames from two synced cameras by timestamp covers the matching afterwards. IMU stamps are yet another domain; align them to depth by interpolation as in the IMU and depth image timestamps: interpolation for alignment guide.

An SDK timestamp regression

Upgrade before you engineer. Capture SDKs have shipped timestamp regressions, and one rig with this exact monotonic drift ran correctly again on the next release. Read the release notes of the version you deploy for fixes to device timestamps or queue behaviour, then re-run the drift plot from the symptom section. If the growth is gone, stop there.

The fix: pair on device timestamps with a half-period tolerance

Both streams carry stamps from the same device clock, so the comparison is direct and no mapping to host time is involved. Accept a pair when the stamps differ by at most half a frame period; capture SDKs use the same convention when they label two frames synchronized.

1. Read the device timestamp of every frame as it arrives — getTimestampDevice() on this SDK, a frame or chunk timestamp on GenICam cameras. Never stamp arrival time yourself.
2. Buffer each stream in its own bounded deque of five frames, about 208 ms of cover at 24 fps, and count every frame you discard. Bound and reuse the buffers the way the Python frame queue memory leak: bound the queue and reuse frames guide describes.
3. When both buffers hold frames, compare the head stamps. Accept the pair when the difference is within tolerance; otherwise discard the older head and add one to that stream's unmatched counter.
4. Emit only matched pairs, and log the stamp difference of every pair.
5. Give every get() a timeout so a stalled stream cannot freeze the loop — the Frame didn't arrive within timeout: one wait call per frame pattern.

The matcher, as pseudocode with the types and initial values declared:

TOL_US = 20800                      // int; acceptance window, 20.8 ms at 24 fps
BUFSIZE = 5                         // int; frames kept per stream
rgb_buf = []                        // list of (ts_us: int64, frame), oldest first
depth_buf = []                      // list of (ts_us: int64, frame), oldest first
unmatched = {"rgb": 0, "depth": 0}  // int counters, start at 0

on_frame(stream, frame):
    ts_us = frame.getTimestampDevice()      // int64 microseconds, device clock
    append (ts_us, frame) to buf[stream]
    while length(buf[stream]) > BUFSIZE:
        pop oldest of buf[stream]
        unmatched[stream] += 1
    try_pair()

try_pair():
    while rgb_buf and depth_buf both non-empty:
        dt = rgb_buf[0].ts_us - depth_buf[0].ts_us
        if abs(dt) <= TOL_US:
            emit (rgb_buf[0].frame, depth_buf[0].frame); pop both heads
        else:
            pop the older head; unmatched[its stream] += 1

The test that proves it: run the rig for 10 minutes and check two things. Every emitted pair satisfies |dt| ≤ 20,800 µs, and the logged differences stay flat across the run. The unmatched counters grow only while the loop is slower than 41.7 ms per cycle — the honest signal the silent drops used to hide.

Worked example: the 24 fps rig that drifted

Both sensors run at 24 fps, so T = 1000 ms / 24 = 41.7 ms and the tolerance is 20.8 ms. Suppose the per-pair work — colour conversion, copy, one detection — takes 50 ms (illustrative). The loop then consumes 1000 ms / 50 ms = 20 pairs/s against 24 pairs/s offered, so each stream sheds 4 frames/s: roughly 240 silent drops per stream per minute inside the size-1 queues.

If just 4 of those drops per minute land on one stream without a matching drop on the other, the gap grows 4 × 41.7 ms = 166.8 ms every minute, and 166.8 ms × 60 ≈ 10 s per hour. After a two-hour run the emitted pair is about 20 s apart — the visible lag, and the reason the drift seemed to appear randomly after hours: it starts the moment one-sided drops begin, not on a schedule.

After the fix, the same 50 ms loop still sheds about 4 frames/s per stream — you cannot consume frames you cannot process — but the shed frames land in the unmatched counters and every emitted pair stays within 20.8 ms for as long as the rig runs. If you need every frame, the loop must fit inside 41.7 ms or the frame rate must come down.

Prevention

• Pair on device timestamps from the first commit; never on queue alternation.
• Set the tolerance to half a frame period and alert when any pair exceeds it.
• Count delivered frames per stream for 60 s at startup, then hourly: 1,440 per stream at 24 fps.
• Keep matching buffers bounded and count discarded frames; an unbounded buffer trades drift for a memory leak.
• Block with a timeout on every get(); a non-blocking size-1 queue is a drop machine.
• Pin the SDK version, read its release notes, and re-run the drift plot after every update.

When pairing by timestamp is not enough

Recording changes the requirement: matching cannot conjure frames the loop already dropped. If the task needs every frame of both streams, either fit the loop inside the frame period by cutting per-frame work, or write each stream straight to disk with its device timestamp and pair offline during analysis.

When depth legitimately runs slower than RGB and every RGB frame matters, attach the nearest depth to each RGB frame, mark pairs beyond tolerance as depth-less, and match analysis results back afterwards the way the asynchronous vision result frame matching with timestamps guide describes.

With a separate RGB camera beside a 3D sensor, do the shared-clock work first and only then pair; the set-up is the one in the two-clock section above.

Specifying the rig

The matcher travels with your code, so it applies to any multi-stream rig you build next. When you specify the depth side of a texture-plus-depth station, stereo and projected structured light cameras cover surface and profile work, and each is a single camera whose own streams share a single device clock — which reduces the job to the matcher above. The shop's examples of that depth side:

The decision, case by case. A gap that grows in frame-period steps and resets on reconnect, on a single device: match on device timestamps with a half-period tolerance. A gap that keeps growing after the fix: the streams are not at the same fps — equalize them first, then match. Every frame needed for recording: fit the loop under one frame period, or record per stream to disk and pair offline. Two separate devices: share the clock, then pair.

To size the hardware, start at the camera finder or the configurator; the 3D sensors and area scan cameras category pages list the depth and RGB sides. For a second opinion on a planned rig, use the contact page, and the systems topic page collects the neighbouring timestamp and streaming guides.