Depth color timestamp drift that grows while running

A depth and color pair that starts aligned and drifts apart while it runs will break every time-based frame pair downstream, and no amount of sync-tolerance tuning holds it. An offset that grows — tens of milliseconds after startup, 700 ms after ten minutes — is a time-source fault, not a calibration fault. This guide is for anyone integrating a depth camera or a depth-plus-color rig whose synchronization log shows the gap climbing during a run. You will read the three stamp gaps, compute a drift rate, match it to one of three causes, and put both streams back on a single clock. The numbers use a 30 fps pipeline, the most common case; halve the periods for a 60 fps rig.

What a growing offset means

A constant offset and a growing offset have different causes and different fixes.

A constant offset of a few milliseconds, present from the first frame, is geometry or stamping point: the depth map is aligned to the wrong color frame, or one topic is stamped one stage later than the other. That is the territory of the raw versus rectified registration question and the offset-in-pixels fix.

A growing offset is a clock or a queue. Two stamps diverge at a steady rate because they come from sources that do not agree: a device clock and host arrival time, or a producer and a consumer running at different effective rates. Calibration cannot do that — registration error is static while the camera sits still, but your offset climbs anyway. The depth and color synchronization guide covers choosing the right stamp pair; this one covers the case where the pair slides apart while you watch.

Read the three gaps first

A drift log usually prints three gaps per sample. Here is a representative one, taken minutes into a run on a ROS 2 rig whose topics are /camera/color/image_raw, /camera/depth/image_raw and /camera/color/camera_info, printed by a small probe (this one tags itself sync_time_diff) that writes all three differences per line:

The step size gives the mechanism away. Consecutive samples move by 734.182 - 700.809 = 33.373 ms, and 1000 / 30 = 33.3 ms is exactly one frame period at 30 fps. An offset that advances one whole frame per processed message is a queue feeding frames through in order: each new pair examined is one frame older than the last. Clock drift moves gaps by microseconds per sample, never by 33 ms jumps.

Read the ordering too. CameraInfo sits nearest wall clock, color trails it by a few frames, depth trails it by 700.8 / 33.3 ≈ 21 frames. One topic at wall clock with the image topics lagging behind at frame-rate steps means capture-time stamps feeding something that consumes them slower than the camera produces.

Cause 1: a stage that runs slower than the frame rate

This is the most common cause and the one that log matches. The camera produces frames at a fixed rate; one stage on the depth path — registration, cropping, colorization, compressed transport, a network hop — takes slightly longer than the frame period per frame. The queue between producer and consumer grows without bound, delivered frames carry capture stamps that fall behind wall clock by exactly the backlog, and the measured offset climbs at a constant rate until frames drop or the process restarts.

Worked example: a 30 fps depth stream whose registration stage takes 34.0 ms per frame.

• Frame period: T = 1000 / 30 = 33.3 ms
• Excess per frame: 34.0 - 33.3 = 0.7 ms
• Drift rate: 0.7 × 30 = 21 ms/s
• Time to the 700 ms in the log: 700 / 21 ≈ 33 s

The stage barely has to be slow. For the offset to hit 700 ms after ten minutes, the drift is 700 / 600 = 1.17 ms/s, which at 30 fps is 1.17 / 30 = 0.04 ms of overrun per frame — 0.1% of the period. One slow frame of 40 ms every ten seconds contributes (40 - 33.3) / 10 s = 0.67 ms/s by itself. That is why the rig runs cleanly at startup and fails later: the overrun only shows as an offset once the backlog is hundreds of milliseconds deep.

Test: log each topic's age — receive time minus stamp — once per second for two minutes. The topic whose age rises is the one accumulating queue. Then time one frame through the suspect stage; 34 ms against a 33.3 ms period is a conviction.

Fix, in order of preference: get the stage under the period with ROI, decimation or a compiled loop, or stop queuing — process keep-latest and drop stale frames so the queue stays at one. Dropping frames is the correct engineering choice when the compute is not affordable; a queue that never drains is latency, not buffering. The frame-rate bottleneck checklist locates the slow stage, and running one stream at a time isolates the topic that drags the rest.

Cause 2: two clock sources

Two devices that stamp independently — a depth sensor on its own clock, a color camera stamped at host arrival — drift at the ratio of their clock errors, typically tens of ppm.

At a realistic 50 ppm relative error: 50 × 0.000001 × 1 = 50 µs per second, which is 3 ms per minute, and 50 × 0.000001 × 3600 = 180 ms per hour. Pure clock drift therefore reaches hundreds of milliseconds in hours, not minutes. If your log climbs to 700 ms within minutes, clock drift alone is not the story — a queue is at work too — but the slow residual drift left after fixing Cause 1 is usually this.

Test: log one stamp pair every 10 s for 30 minutes and compute drift per minute. A 3 ms/min drift corresponds to 3 / 60,000 × 1,000,000 = 50 ppm, which is a clock domain. 20 ms/s or more is more than half a frame period per second — a queue.

Fix: one time source. Hardware-trigger both cameras from a single pulse generator and stamp from the trigger, so each stamp marks the same exposure instant; the hardware-trigger frame-rate guide shows how the pulse rate then defines the stamp rate. GigE Vision 2.0 cameras can share one clock over the wire with PTP (IEEE 1588) instead. Or map device time to host time once in the driver and stamp every topic from that clock — the timestamp-to-PC-time mapping pattern, including its periodic re-sync, is the model to copy.

Cause 3: topics stamped at different pipeline stages

The third signature is in that log too: CameraInfo stamped near wall clock while the image topics trail behind. One publisher writes arrival time — the moment the driver handed the message to the middleware — while the others carry capture time. The gap between them is the sum of every queue between capture and publish, so it moves with CPU load and network traffic: it grows while the pipeline is busy and shrinks when idle.

Test: take one frame and compare the stamp on /camera/color/camera_info with the stamp of the image it describes. If the driver exposes the device's own exposure timestamp, compare against that as well. A CameraInfo consistently one to three frames newer than its image confirms mixed stamp points.

Fix: stamp every topic at the same point — capture time. Most depth-camera wrappers expose a time-source or stamp-mode parameter; set all three topics to the device clock. Where the stamp source cannot change, re-stamp CameraInfo in your sync node from the image stamp it belongs to, before the synchronizer sees it.

One clock, one stamp point: the fix that holds

Whatever the cause, the durable fix is the same: one clock and one stamp point across all three topics.

A single-device sensor already has that clock. Its depth and image streams are stamped from one internal timing, so a growing gap points at the driver's stamp path or at host backlog, not at two disagreeing devices. Structured-light and projected-speckle sensors such as the MV-DPS200P-02 fit this pattern.

A two-camera rig needs the agreement to come from outside. One pulse generator drives the exposure of both devices and both drivers stamp from the trigger instant; neither free-running clock is used, so ppm differences never accumulate. For the color side, the MV-CS200-10GC is a 20 MP GigE color camera that fits this pattern.

Then set the pair tolerance in your synchronizer to about half a frame period — 16.7 ms at 30 fps — so a mismatched pair is rejected and counted instead of silently paired 700 ms apart. A rejection counter stuck at zero while the offset stays flat is the acceptance test. When the trigger fires matters as much as when exposure starts; the trigger call versus exposure timing guide covers that difference.

Keep it from coming back

• Budget the CPU against the frame period: keep worst-case per-frame stage time under 80% of the period. At 30 fps that is 26.7 ms of the 33.3 ms available; at 60 fps, 13.3 ms of 16.7 ms. The margin absorbs the occasional slow frame instead of starting a backlog.
• Log age per topic and alarm when it exceeds one and a half frame periods: 50 ms at 30 fps. The alarm fires while the gap is still tens of milliseconds, long before pairing breaks.
• Record the drift rate at commissioning. After the fix it should be zero: a flat offset below half a frame period, not a slope.
• Treat restarts as a mask, not a cure: an offset that resets to zero on restart and regrows at the same rate is the same fault running again.
• Keep transport symmetric: if one topic travels compressed and the other raw, the slower path adds per-frame latency that receive-time pairing reads as drift. Compress all topics or none.

Decision guide

Decisions, per case: if the offset grows in 33.3 ms steps at 30 fps, treat it as a queue today — add the age log, time the depth stage, then get it under 33.3 ms or drop to the latest frame. If it grows at microseconds per second, put both devices on one pulse generator or PTP and re-check. If one topic sits at wall clock while the others trail, fix the stamp point in the wrapper parameters. If the offset is constant from startup, stop chasing clocks: that is registration, and the guides linked above are the right reading.

If drift survives unified stamps and empty queues, take the log to the wrapper's maintainers — a reproducible rate in ms per second plus the three-topic gaps is exactly the report they can act on. If you are specifying hardware for a new depth-and-color station instead, the 3D sensors category filters single-device sensors by type, the area scan cameras category covers the color side of a two-camera rig, and build a camera system pairs the camera with the trigger hardware that keeps both stamps on one clock. For a rig already specified, bring your frame rate and stage timings to the contact page and the applications team will sanity-check the budget. More timing and integration guides sit in the Learn guides hub, and the camera finder filters 3D sensors by type and interface.