Color lags depth by one frame: sync RGBD frames by sequence number

A point cloud that looks correct on a static bench and smears the moment anything moves usually has a frame-pairing fault, not a calibration fault. If the pipeline builds the cloud inside the depth callback and grabs whatever color frame is newest, it silently assumes color always arrives before depth. Plenty of rigs deliver color after depth, so the sequence numbers never line up and every moving scene registers color to the previous depth frame. This page shows how to confirm the offset, pair frames by the camera's own sequence number under one mutex, and handle mismatched stream rates.

What a one-frame offset looks like on the bench

On a static scene the cloud is perfect: color sits exactly on its depth pixels. Start the belt or swing a target and the color edges trail the depth edges along the direction of motion. A printed face lands downstream of its geometric edge; a logo floats off the part. The offset scale is set by the frame period alone: at 30 fps one frame lasts 33.3 ms, at 60 fps it lasts 16.7 ms. Multiply by scene speed and you have the displacement in millimetres.

The trap is that static-scene acceptance passes while every moving frame is wrong. The fault survives commissioning and only surfaces later as intermittent pick errors or smeared inspection overlays.

Likely causes and the test for each

Work down this list in order; each cause comes with a test that takes minutes.

1. The cloud publishes from the depth callback with the newest color frame. The pipeline assumes color precedes depth, which is how most sample pipelines are written. Test: log both sequence numbers where the cloud is built. A constant difference of one is this cause.
2. Your rig delivers color after depth. Delivery order is set by sensor start-up, internal pipeline lengths and driver buffering, not by physics. Test: time both callbacks on a monotonic host clock; a consistent gap says the order is reversed.
3. The streams run at different rates. Depth at 30 fps against color at 15 fps means the two counters can never be equal at all. Test: plot the sequence difference over 200 frames; a difference that grows linearly is a rate mismatch.
4. Host queueing reorders frames. A deep receive queue or a slow consumer makes arrival order differ from capture order. Test: compare the camera's own timestamps, if the driver exposes them, rather than host arrival times; if the offset appears only when the CPU is loaded, queueing is the cause.

If the offset shows on a static scene too, pairing is not your fault. The color-to-depth registration geometry or a stream resolution mismatch is, which is the subject of the depth-to-color alignment and stream resolution guide.

The fix: pair on sequence numbers under one mutex

Pair on the frame counter the camera stamps, not on arrival order. Most machine-vision drivers expose a per-stream sequence number or block counter. Publish the cloud only when the depth frame's number equals the color frame's number, and publish nothing while one side of a pair is missing. The change is small and local:

1. In each frame callback, copy the frame and its sequence number first. Callbacks own the buffer only briefly; the torn frames and callback buffer guide covers what happens if you skip the copy.
2. Store the copy in a per-stream cache under one mutex shared by both callbacks.
3. After storing, run one integer comparison; build and publish the cloud only when the cached sequence numbers match.
4. If they do not match, return and wait for the other stream. Do not fall back to newest-color silently.
5. Gate strict pairing behind a boolean in your pipeline configuration, so you can switch modes on site and compare clouds.

// Names below are illustrative; use the frame counter your SDK exposes.
std::mutex pair_mutex;              // guards both caches and the publish decision
uint64_t color_seq = 0;             // newest color frame counter
uint64_t depth_seq = 0;             // newest depth frame counter
ColorFrame color_cache;             // copied on entry, never held by reference
DepthFrame depth_cache;

void on_color(const ColorFrame& f) {        // driver's color thread
  std::lock_guard<std::mutex> lock(pair_mutex);
  color_cache = f;                          // copy before any heavy work
  color_seq = f.sequence_number;
  try_publish();
}

void on_depth(const DepthFrame& f) {        // driver's depth thread
  std::lock_guard<std::mutex> lock(pair_mutex);
  depth_cache = f;
  depth_seq = f.sequence_number;
  try_publish();
}

void try_publish() {                        // called with pair_mutex held
  if (color_seq != depth_seq) return;       // no matched pair yet
  enqueue_cloud_build(depth_cache, color_cache);  // hand to a worker, stay short
}

Two rules keep this sound. Keep the locked region to copies and the comparison, and hand the heavy cloud build to a worker afterwards: a task must never exit with a mutex held, and memory under a held lock must never be freed. And never re-introduce a newest-color fallback inside the same run; a silent fallback recreates the fault with extra steps.

The test that proves it: run 200 frames of a moving target with the log on. Three things confirm the fix. Every published cloud carries equal color and depth sequence numbers. The publish count equals the matched-pair count. The color-edge offset you measured in the unpaired run collapses to the static registration error.

Worked example: color one frame late on a 30 fps belt at 0.5 m/s

Inputs, and where each comes from: both streams run at 30 fps, the typical rate for an RGBD rig of this class and a setting you would read off the camera's streaming page; color is delivered one frame after depth, the delivery-order fault this page fixes; the belt runs at 0.5 m/s, declared here as the example speed, so substitute your own line speed; and the part carries a ±2 mm tolerance band, declared so the decision has a number to fail against.

Rule in words: one frame of offset lasts one frame period, and the scene travels speed × period between the depth exposure and the color exposure that the pipeline wrongly joins.

Arithmetic:

• t_frame = 1 / 30 fps = 0.0333 s = 33.3 ms per frame
• s_offset = 0.5 m/s × 0.0333 s = 0.0167 m = 16.7 mm

Result: every cloud built from the newest color frame carries color displaced 16.7 mm along the belt direction relative to the depth surface. That is more than eight times the ±2 mm band, so the unpaired pipeline fails on every moving frame, not occasionally.

Decision: with sequence pairing, a published pair shares one exposure instant, so the cross-frame offset returns to 0 mm and the residual is the belt's motion within a single exposure, not a frame-to-frame jump. Threshold: whenever scene speed × frame period exceeds about a tenth of your tolerance band, strict pairing stops being optional. Here 16.7 mm against a 0.2 mm line (a tenth of ±2 mm) clears it by more than a factor of eighty.

Rate branch, same rig with color throttled to 15 fps: equality never fires, so pair through the ratio. Depth at 30 fps and color at 15 fps pair through depth_seq = 2 × color_seq, so depth frame 42 joins color frame 21, because 42 / 2 = 21. Integer ratios stay locked for the whole shift; fractional ones such as 25 against 30 drift each time a frame drops.

Wraps, gaps and timestamp cross-checks

Three details decide whether the pairing survives a long shift.

• Counter wrap. Sequence counters are fixed width. A 16-bit counter at 30 fps wraps every 65,536 / 30 ≈ 2185 s, about 36 minutes; a 32-bit counter takes years. Compare with modulo arithmetic, (a − b) mod 65536 for 16 bits, and treat the counter as free-running. A huge negative-looking jump is a wrap; a jump of two or more is a drop.
• Dropped frames. A gap in one stream means that pair never existed; skip it and log the gap. Occasional drops are normal on a shared link. If gaps exceed roughly one in a hundred, fix the transport before trusting any statistics; the point cloud camera processing load guide separates host load from stream faults step by step.
• Timestamp cross-check. A matched pair should carry near-identical camera timestamps, because the camera captured it as one event. If matched pairs differ by a systematic frame period, the counter you are reading is not the one gating exposure, and you are pairing metadata rather than frames.

Put the modulo check in the cloud publisher, not only in the logger. A wrapped counter must not wedge the pipeline into publishing nothing.

When sequence pairing cannot help

Pairing selects frames; it cannot make two frames share an instant. If a camera exposes color and depth at different times with no shared trigger, equal sequence numbers still label frames that saw different light, and the moving-scene residual stays whatever the inter-exposure gap dictates. Confirm with a pulse test: flash an LED or strobe in view and check which frame in each stream caught it. For two separate cameras, pair by camera timestamp instead of a shared counter, as the pair frames from two synced cameras by timestamp guide sets out; when the scene moves within the inter-frame gap and you need sub-frame alignment, resample to a common instant as covered in the IMU and depth timestamp interpolation guide.

What the host needs

A rig streaming color and depth at 30 fps each raises 60 callbacks per second, and pairing adds one mutex and one integer comparison to each. Keep callbacks to copy-compare-enqueue and the load stays trivial. Let the cloud build run inside the lock, and the queues shift the very offsets you just fixed. If you merge several depth frames into one gap-free cloud downstream, budget for that stage too; the merge multiple depth frames guide shows its cost shape.

The vision controllers category is the host side of this build. A current example: the MV-VC5525-256G70-NN prints 5+ LAN ports, PoE camera ports, 8 built-in light channels and 32 GB of memory, with PCIe expansion supported.

When the same box also runs inference or feeds several cameras, compare what each controller family prints:

All 3 share: Certification CE, KC. Values as printed in each product's data sheet; open a part for its drawing and full specifications.

Prevention checklist for the next rig

Carry these to the next integration; each one exists because it failed someone before.

• Log sequence numbers per frame in production, not only while debugging. It is the cheapest diagnostic you will ever install.
• Run both streams at the same rate, or at integer-divisible rates, from day one.
• Copy frames inside callbacks and process outside the lock, every time.
• Commission with a moving target at line speed, never with a static chart alone.
• Re-run the moving-target check after any driver or firmware update, because delivery order can change with the software stack.
• Keep strict pairing behind a configuration switch, so a field engineer can compare modes without a rebuild.

If you are still choosing hardware, the camera finder filters the 3D sensors by type and series, and industrial mini PCs cover the host class when your build outgrows a vision controller. The 3D vision topic page collects the rest of the integration guides, and you can always reach the team through the contact page.