3D visionExplainer

Depth camera hardware timestamps vs software frame sync

Where a depth camera's frame timestamps really come from, how software sync matches depth and color frames, and the match tolerance that keeps pairs correct.

LumaSpec Vision10 min read Share on LinkedIn

On this page 8 sections
  1. 01Where a depth camera's timestamps come from
  2. 02One shared clock is not simultaneous capture
  3. 03How software frame sync pairs depth and color
  4. 04Worked example: pairing 30 fps depth and color for mesh texturing
  5. 05When timestamp matching is not enough
  6. 06Depth cameras that shrink the sync problem
  7. 07Four checks before you trust a pair
  8. 08Questions

Pair a depth frame with the wrong color frame and the damage is quiet: a textured mesh with ghost edges, a dataset whose labels sit one frame off the geometry, a gripper that closes where the part was. The question underneath is simple: does the timestamp on a depth frame come from the camera, or from the computer that received it? On a real depth camera it comes from the camera. An onboard counter stamps every stream it produces in one shared time domain, and everything about syncing depth with color follows from that fact. Here is where the stamp originates, what it does and does not promise, how to compute a match tolerance for software frame sync, and when matching stops being good enough.

Where a depth camera's timestamps come from

A depth camera is several imagers in one housing: a depth and IR pair that computes range, and usually a separate RGB imager for texture. The imagers free-run at their own rates, set by resolution, exposure and processing load. Every frame from every imager is stamped in hardware, by a counter on the camera itself. The stamp is not a wall-clock reading; it is a counter value that travels with the image data, has a finite width and wraps around, and the driver tracks the wraps so the sequence stays monotonic. On some cameras the counter is even carried in the first pixel values of each frame, which is how literal the stamping gets.

Because every stream is stamped from that one counter, an IR frame and a color frame carrying the same value were captured within the same increment of the same clock. That shared domain is the entire reason software sync works without extra wiring.

What the stamp is not: the moment the frame arrived at the host. Arrival time carries bus transfer and driver scheduling, jitters with system load, and can reorder relative to capture. The split between acquisition time and arrival time is the same for a USB depth camera as for any networked camera, and it decides which value you may pair on — see camera frame timing jitter: acquisition vs arrival time.

Expect one wrinkle: SDKs anchor the same counter differently per platform. On one OS the values look like ticks counted from an epoch-style base; on another they start near zero and count up from boot. Neither is wrong — they are presentations of one clock, not different clocks. Map camera stamps to PC time once, at recording, and keep every comparison afterwards in that single timeline: map camera timestamps to PC time.

One shared clock is not simultaneous capture

A shared clock answers when each frame was captured, not whether two frames were captured together. The depth/IR pair and the RGB imager still free-run independently, so their exposure instants sit wherever their frame rates and exposure times leave them. Identical stamps mean the same increment of one clock, nothing more.

True hardware sync — commanding both imagers to expose in the same instant — exists only on cameras that implement it, and they are the minority. The capability usually rides on per-frame metadata in the transport protocol, and that metadata is exactly what some operating systems' drivers never deliver. A hardware sync mode can therefore be advertised, work on one OS, and fail on another.

A working trigger is not automatically better than matching, either: a badly distributed hardware trigger can calibrate a multi-camera rig worse than plain timestamp matching did — see hardware sync gives worse calibration than software.

How software frame sync pairs depth and color

The fallback, and for most single-camera tasks the right answer, is matching in the camera's own stamp domain. A frame-sync stage, in the SDK or in your own code, groups frames whose timestamps differ by less than a tolerance. Four pieces decide whether it works:

  1. One domain per camera. Depth, IR and RGB stamps all come from the onboard counter, so their differences mean something as written. Matching on device timestamps beats matching on arrival order, because the receive queue reflects USB timing, not capture time — see sync RGB and depth frames by device timestamp, not queue order.
  2. A constant offset per stream. Each pipeline adds its own latency, so one stream's stamps sit a few milliseconds off the other's. Measure the offset once from a recording and subtract it.
  3. Residual jitter around that corrected offset, of whatever size your rig adds.
  4. A tolerance that decides which frames group together.

Match on stamps and use sequence numbers only as a tiebreak. Stamps say when a frame was captured; sequence numbers say only in what order a stream delivered its frames. A color stream lagging depth by one frame shows up as a delta of almost exactly one period, which is the signature to check — see sync RGBD frames by sequence number.

Two rules size the tolerance:

T = 1000 / fps frame period in ms tolerance < T_min / 2 keeps every match unique to one neighbor

The recording-side view of the same decision — which timestamps to trust when you sync depth and color — is covered in depth and color synchronization: check the right timestamps.

Worked example: pairing 30 fps depth and color for mesh texturing

The case: a USB depth camera free-running depth and color, each at 30 fps, feeding a mesh texturing step.

Inputs:

  • Frame rate: 30 fps on depth, 30 fps on color, so one period governs both streams.
  • Offset between the two stamp domains: 5-15 ms has been measured on this class of camera; size the correction for the worst case, 15 ms.
  • Constraint: every accepted pair must be unique, meaning the matcher must never see two depth candidates within tolerance of one color frame.

The rule in words: the frame period is 1000 divided by the frame rate; a tolerance below half the shortest period puts each color frame inside exactly one depth frame's window; and the constant offset must be corrected first, because it silently eats the window.

The arithmetic:

  • Period: T = 1000 / 30 = 33.3 ms on both streams.
  • Uniqueness ceiling: 33.3 / 2 = 16.7 ms.
  • Uncorrected: a 15 ms domain offset makes every stamp gap read 15 ms short, so a pair whose stamps differ by 1.7 ms can really be 16.7 ms apart, and near the ceiling the matcher grabs the wrong neighbor.
  • Corrected: subtract the 15 ms offset, then match at a 10 ms tolerance — below the 16.7 ms ceiling with margin.

Result: every accepted pair is unique, and its true skew is at most 10 ms.

Decision: at 30 fps, correct the offset once and run the matcher at 10 ms. If you raise the depth stream to 60 fps, the period halves to 16.7 ms and the ceiling drops to 8.3 ms, so an uncorrected 15 ms offset exceeds the entire window — correction stops being optional and becomes step one.

When timestamp matching is not enough

Matching bounds the skew; it does not remove it. Whether the bound matters is one line of motion arithmetic: a scene moving 100 mm/s shifts 1.0 mm during a 10 ms skew (100 mm/s × 0.010 s = 1.0 mm), which is invisible on a turntable scan, while a 300 mm/s conveyor moves a feature 3.0 mm in the same window and the texture smears. For moving scenes, halve the period by running 60 fps, shorten exposure to freeze motion, or trigger both streams from one hardware event.

Second limit: one counter per housing. Two cameras mean two counters and two time domains with unrelated zero points. Fusing or calibrating across cameras needs a shared clock: a trigger line, a PPS pulse from a GPS receiver — stamped at exposure start or end, and the choice matters, as PPS GPS camera time sync explains — or network time, which holds loose tolerances but not pixel-tight ones (NTP time sync).

Third limit: everything else on the timeline. IMU samples arrive on their own pipeline with their own stamps, so interpolate across the depth stamps to bring both onto one timeline — see IMU and depth image timestamps: interpolation for alignment.

Depth cameras that shrink the sync problem

Half the machinery above disappears when depth and its companion frames come out of one housing. A one-box depth camera stamps its internal streams from one counter by construction, so there is no cross-device offset to measure and no second domain to reconcile; the problem reduces to the single-camera tolerance math. Two families worth a look when sync is your bottleneck:

3D sensorsHikrobot MV-DPS200P-02 Projected structured-light 3D cameraMV-DPS200P-02
Type
Stereo / structured light
$8,829.00 · quotedView and quote

A projected structured-light camera triangulates depth from a pattern it projects itself, which keeps every frame the device emits on its own clock. For dense depth on low-texture or reflective surfaces, a stereo camera with a galvanometer-steered laser pattern such as the MV-DLS300P-04 does the same across a stereo pair.

3D sensorsHikrobot MV-DT01WMU ToF sensing cameraMV-DT01WMU
Type
Depth / ToF camera
Price on requestView and quote

A time-of-flight camera returns depth per pixel from light travel time, again stamped internally. The 3D sensor category holds all the families — laser profilers, stereo and structured light, ToF, logistics volume — and the camera finder filters them by type, resolution and interface. Laser-projected sensors are laser products: install them as their classification requires and never bypass an emitter interlock. Profilers add a third stamp to the pair, encoder position per profile, so motion sync comes from hardware rather than estimation.

Four checks before you trust a pair

  • Record device stamps, not arrival times, on both streams; keep arrival time only to flag stalls and dropped frames.
  • Histogram the accepted deltas. A pile-up at the tolerance edge means the offset is drifting or frames are dropping; re-measure the offset and tighten the tolerance.
  • Verify on a static scene. Capture a static target for a few seconds, run the matcher, and check texture-to-geometry alignment across the pairs; drift over the run is residual skew showing itself.
  • Exercise any strong-sync mode before committing to it. Confirm frames keep flowing and the stamp deltas actually tighten; where the mode is unsupported it can stop delivery outright.

If you are sizing a new rig rather than fixing one, the camera configurator walks camera, lens, light and controller in order, and the 3D vision topic page collects the guides around this one. If a pairing still fails the checks above after the offset is corrected, the contact page reaches the applications team; two short recordings of the same scene, one per stream, are usually enough to read the offsets straight off the stamps.

Questions

Is a depth camera frame timestamp the exposure time or the arrival time?

On a depth camera with an onboard counter, the stamp is generated in hardware at the sensor, in one counter domain shared by the depth, IR and color imagers. It is not the OS arrival time: the host adds transport and driver delay that varies with bus load. Pair frames on device stamps and use arrival time only to detect drops or stalls.

Why do the same depth frames show different timestamp values on Windows and Linux?

The counter is the same; the SDKs anchor it differently. One reports large values counted from an epoch-style base, the other values near zero counted from boot. Neither is wrong, and each drifts against PC wall time in its own way. Map camera stamps to PC time at recording and compare everything in that one timeline, never as raw values across machines.

What tolerance should I use to match depth and color frames in software?

Less than half your shortest frame period: a 16.7 ms ceiling for 30 fps streams, 8.3 ms at 60 fps. Correct the constant offset between the streams first — offsets of a few up to 15 ms have been measured on USB depth cameras. Uncorrected, a 15 ms offset at 30 fps can leave a matched pair nearly a full period apart in reality.

Can any depth camera hardware-sync its depth and RGB exposures?

No. Most depth cameras free-run their imagers and rely on timestamp matching. A minority can hardware-synchronize exposures, and the capability usually depends on per-frame metadata that the operating system's driver must deliver, so the same mode can work on one OS and fail on another. Where it is unsupported, enabling it can stop frames entirely, so test before you commit.

Does a higher frame rate make depth and color sync easier?

It cuts the time error of a matched pair, because nearest-neighbor matching skews at most half a period: 16.7 ms at 30 fps, 8.3 ms at 60 fps. But it also halves the tolerance ceiling, so an uncorrected stream offset that matched fine at 30 fps can grab the wrong neighbor at 60 fps. Correct the offset first, then raise the rate.