Depth camera latency measurement with a reference sensor

Your closed-loop positioning runs late and the evidence points at the depth camera: its reported position trails a reference tracker by 100-150 ms, while a plain webcam on the same host sits near 20 ms. Your program is not the culprit, and neither is marker detection. The delay lives in the depth pipeline itself — several capture-and-compute stages, then the frame queues the operating system keeps between camera and application. This guide defines what an end-to-end latency figure contains, shows why arrival timestamps understate it, walks through a cross-correlation measurement that produces a number you can act on, and ends with the fixes in order of effect: queue depth, exposure, frame rate, then sensor choice.

What latency means for a depth camera

Latency is the time between the scene state that produced a depth value and the moment your code can use that value. It starts at exposure — the middle of the exposure window is the true capture instant — and ends when your process reads the frame. Everything in between counts: exposure time, internal capture and depth computation, transfer over USB or GigE, the driver's frame queue, SDK processing and your own first processing step.

Latency is not frame rate. A camera at 30 fps delivers a frame every 33.3 ms, but each of those frames can describe a scene far older than 33.3 ms. It is not jitter either: jitter is the frame-to-frame variation of the delay and gets its own section below.

The N_pipeline term is the whole story: close to one frame for a plain camera, two or more frames for a structured-light depth camera.

Why a depth pipeline spans several frames

A webcam or a plain area scan camera exposes once, conditions the image and outputs it. A structured-light depth camera cannot work that way: it projects a sequence of near-IR patterns onto the scene, captures several frames of the deformed pattern, correlates them against the projected code, and only then computes depth per pixel. Every captured frame in the burst occupies one frame period, so the pattern sequence alone puts two or more frame periods on the clock before depth computation starts.

The technology sets the floor. Stereo needs one synchronized exposure pair plus correlation. ToF computes depth from modulated light in a single exposure cycle. Structured light pays for its pattern burst. A laser profiler emits one profile per scan line. That is also why the plain camera sits near 20 ms: 20 ms is under one 33.3 ms frame period — one exposure, ISP and transfer, nothing queued. How projected pattern geometry turns into 3D points is covered in structured light 3D cameras: projector pixel mapping to 3D points.

Where the delay hides: driver and framework frame queues

The camera itself has almost no memory — it cannot park five frames while your loop catches up. Queueing happens on the host: the USB or GigE driver and the capture framework above it (V4L2 on Linux, Media Foundation on Windows) keep ring buffers of completed frames. Whenever your application reads slower than the camera produces, frames stack up, and each buffered frame adds one frame period to the gap between exposure and arrival.

That is why arrival-time stamping misleads. A frame that reaches your callback at time t may have been exposed 100 ms earlier; the receive stamp records the end of the queue, not the exposure. What a V4L2 buffer timestamp actually marks is worked through in v4l2 buffer timestamp: know when a frame was really captured, and the same host-side reading problem in camera buffer timestamp latency: interpret frame timing.

The arithmetic is blunt: 150 ms of lag at 30 fps is 150 / 33.3 ≈ 4.5 frame periods — four to five frames parked somewhere between exposure and arrival. On a host that keeps up, those buffers should hold at most one frame.

Measure latency by cross-correlating two position streams

You do not need a device timestamp to get an end-to-end number. Measure the position stream you actually use:

1. Track the same target with both devices: the depth camera derives position from its depth output, the reference tracker from its own sensors.
2. Move the target irregularly — random steps, sudden reversals, no fixed period. Periodic motion produces correlation peaks one period apart and an ambiguous lag.
3. Log both streams, each with its own timestamps, then resample onto a common time grid.
4. Cross-correlate the two position signals over candidate lags from 0 to 300 ms. The lag with the highest correlation is the depth camera's delay relative to the tracker; subtract the tracker's own latency where that is not negligible.
5. Repeat the capture. A lag that reproduces within one frame period is a real pipeline delay; a lag that wanders means queueing (next section).

Worked example. A depth camera streams 640 × 480 at 30 fps, so T_frame = 1000 / 30 = 33.3 ms. The reference logs at 240 Hz, so lag steps are 1 / 240 = 4.2 ms. Both watch a plate moved by hand for 60 s; the cross-correlation of the two Z signals peaks at 127 ms. That is 127 / 33.3 = 3.8 frame periods — read as two frames of pattern capture and onboard depth computation, plus two parked in the driver queue. After setting the capture layer to hold a single buffer, the same test peaks at 64 ms: 64 / 33.3 = 1.9 frames, the internal pipeline alone. The queue therefore cost 127 - 64 = 63 ms, consistent with 2 × 33.3 = 66.7 ms for two frames; the 3.7 ms gap sits inside the 4.2 ms lag step.

Without a second tracker, step a target — or a light it can see — on a known command and record when each stream first reports the step; averaged over dozens of repetitions this converges to the same figure. Step the target in Z, not merely across the view: an RGB-only stimulus measures the RGB pipeline, and that is not the one that lags.

Separate latency from jitter and clock drift

Two independent clocks run any such test — the tracker's and the host's — and a small rate difference drifts the apparent lag across a minute-long capture. If the lag estimate trends steadily over the record, split it into 5-10 s windows, compute the lag per window and average the early ones. A shared trigger pulse on both devices removes the drift entirely; the same latching discipline as in chunk data timestamp: camera time at exposure end and how to latch it applies.

Then read the shape of the result. A constant lag with a few milliseconds of spread is pipeline delay, and you compensate it by time-shifting the stream. A lag with large frame-to-frame spread is queue jitter, and buffering discipline fixes that, not compensation. For stereo cameras specifically, where depth noise also comes from correlation quality and IMU integration, the stereo depth jitter and IMU drift troubleshooting guide works through those jitter sources step by step.

Where the milliseconds live, and what cuts them

The transfer row, spelled out: one 640 × 480 depth frame at 16 bits per pixel is 640 × 480 × 2 B = 614,400 bytes ≈ 0.6 MB; at a realistic 300 MB/s of USB 3 throughput, 614,400 / 300,000,000 ≈ 2 ms.

Exposure first. Auto-exposure in a dim scene can stretch the exposure itself toward a full frame period, delaying every frame before the pipeline even starts. The same exposure-plus-readout accounting that caps frame rate — as in maximum frame rate on a hardware trigger: exposure plus readout time — sets the floor under your latency.

Queue second. Capture layers let you set how many frames the driver retains; set that count to the minimum your loop sustains and consume every frame promptly. When the CPU stalls — logging, visualization, inference on the same machine — the queue grows and the measured lag grows with it, the same mechanism behind 3D camera frame rate low on software trigger: mode and data streams.

Frame rate third. Every pipelined or queued frame costs one T_frame, so the four-frame pipeline of the worked example drops from 4 × 33.3 = 133 ms at 30 fps to 4 × 16.7 = 66.7 ms at 60 fps — bandwidth and lighting permitting, the cheapest global win. When a part of the loop needs only the 2D image, running that stream with the projector off, as in disable the projector for a 2D live feed: brightness 0.0, takes the pattern burst off its path entirely.

Choosing the sensor when latency is in the spec

Technology choice sets N_pipeline — the one term software cannot tune away. Industrial 3D camera selection: stereo, ToF or structured light compares the three frame-based families; a laser profiler fits surface scanning better than closed-loop positioning.

In the shop, the 3D sensors category splits by type — laser profiler, stereo / structured light, depth / ToF camera, logistics / volume — and the camera finder filters it further.

Three cameras, one per pipeline length:

The ToF sensing camera sits on the shortest pipeline of the three, because ToF computes depth from modulated IR exposures with no pattern sequence to capture. The projected structured-light camera trades some pipeline length for dense surface detail. The laser galvanometer stereo camera steers a laser line for scanned reconstruction. Count each technology's capture stages in frame periods, multiply by your frame rate, and you have the pipeline budget before the first measurement.

When the measurement misleads

Two further traps. A periodic trajectory correlates almost as well at the true lag as at the lag one period away, so drive the target with random steps. And if the camera outputs a fused pose rather than raw depth, you are measuring the filter's group delay as well as the pipeline — which depth cameras output a pose stream and why most do not explains what that stream is; measure the depth-derived position and the filter stays out of your number.

Checklist: a latency figure you can act on

Set-up:

• Irregular target motion with sharp steps — no fixed period, or correlation peaks multiply.
• Reference logging at several times the camera's frame rate: 240 Hz against 30 fps gives 4.2 ms lag steps.
• Fixed exposure below one frame period; auto-exposure off.
• Driver buffers at the minimum; heavy logging and visualization off the capture host.

Measurement:

• Cross-correlate depth-derived position against the reference over 0-300 ms; repeat the whole capture three times.
• Compute the lag per 5-10 s window and check for a clock-drift trend.
• One shared trigger pulse if the hardware offers it.

Interpretation:

• Constant lag within a frame period: pipeline delay; shift the stream in time.
• Lag that wanders with host load: queue jitter; fix buffering, not compensation.
• Depth values dropping out only while the tracker runs: IR interference; separate the bands.

The decision, case by case

Lag constant near a whole multiple of the frame period with buffers already at minimum: that is the pipeline — raise the frame rate or change technology, software will not remove it. Lag that grows when the host is busy: queueing — cap background load, set buffers to the minimum, read every frame. Lag that appears only against an IR tracker: interference, not latency — separate the wavelengths or switch to a passive reference for the test. A loop that must respond within one or two frame periods: ToF or stereo at the highest frame rate the link sustains, and structured light kept for offline surface work.

If the latency budget is in the specification, configure from the camera step:

For a budget you cannot close on paper, contact us with your frame rate and measured lag, and we will help you pick the sensor. The rest of the depth-timing guides sit on the 3D vision topic page.