A recording timeline can show depth and color messages arriving at different times even when their exposures are closely aligned. Conversely, matching timestamps can hide a clock conversion error or a poorly chosen frame pair. For recorded-data debugging, you need to identify what each timestamp represents before judging synchronization. The checks below separate recording delay, clock-domain differences, pairing mistakes, and motion-related errors, so you can decide whether the fault belongs to acquisition, transport, or processing.
1. Separate acquisition time from recording time
Start with the most common explanation: the timeline displays when messages reached the recorder, not when the sensors acquired their images. Recording timestamps are useful for diagnosing delivery behavior, but they do not necessarily describe exposure timing.
Inspect the timestamps stored inside individual image messages. A field such as header.stamp may contain the timestamp supplied by the driver. However, its name alone does not establish its meaning. Check the manufacturer's documentation for the timestamp source, units, clock domain, and reference event.
| Timestamp or field | What it may represent | Appropriate use |
|---|---|---|
| Sensor timestamp | Exposure start, midpoint, end, or another documented acquisition event | Comparing acquisition events after checking clock compatibility |
| Image header timestamp | A sensor timestamp, converted timestamp, or driver-assigned time | Useful only after confirming how the driver populates it |
| Host arrival timestamp | When the host receives a frame or message | Measuring delivery behavior |
| Recording timestamp | When the recorder accepts or stores a message, depending on implementation | Inspecting recording order and pipeline delay |
| Frame counter | An image sequence identifier | Detecting gaps, repeats, and ordering problems |
For example, suppose both images describe an acquisition event at the same instant. One message reaches the recorder after 8 ms, while the other arrives after 20 ms. Their recording timestamps differ by 12 ms, although this example contains no acquisition-time offset.
Export several frames with all available timestamps and counters. Preserve the original values rather than replacing them during conversion. The related guide to multi-camera RGB synchronization explains why exposure timing requires evidence beyond simultaneous delivery.
2. Check the clock domains before subtracting timestamps
Two timestamps are comparable only when their units, reference epochs, and clock rates are compatible. One stream might use a device clock measured from startup, while another uses host time. Even timestamps expressed in seconds can belong to unrelated clocks.
Check these items before calculating a depth-to-color difference:
- Units: distinguish seconds, milliseconds, microseconds, and raw clock ticks.
- Epoch: establish whether time starts at device initialization or an absolute reference.
- Clock source: determine whether both streams share a clock or use separate oscillators.
- Reference event: identify whether the timestamp describes exposure start, midpoint, or another event.
- Continuity: check for resets, wraparound, and discontinuities after reconnection.
When independent clocks need conversion, a common model is:
This model requires a justified estimate of both parameters. Use documented clock conversion, shared timing events, or another validated synchronization method. Subtracting each stream's first timestamp removes its starting value, but does not establish that those first frames were acquired together.
Do not estimate a precise acquisition offset from a single pair of host arrival times. Unknown transport and processing delays are mixed into that observation.
Plot the mapped timestamp difference throughout the recording. A steady offset suggests a different problem from a gradually changing offset or a sudden jump. For the wider distinction between sensor events and clock relationships, consult camera–IMU synchronization.
3. Pair frames by acquisition time, not file position
After establishing a common time basis, check how your processing code selects frame pairs. Pairing the first depth image with the first color image is unreliable when streams start independently, run at different rates, or lose frames.
For each candidate pair, calculate a signed difference:
Keep the sign during diagnosis. It shows which acquisition reference is later and helps reveal consistent pairing bias.
For offline inspection, a nearest-timestamp search is a useful starting point. Reject a pair if its absolute difference exceeds the application limit. Also define whether a frame may be reused: nearest-neighbor matching can assign one color frame to several depth frames when rates differ.
Prefer a documented acquisition-pair identifier when the system provides one. Independent frame counters are not automatically shared pair identifiers, even when their values happen to match.
Log the selected frame identifiers, signed differences, rejected pairs, and reused frames. If the relationship changes after a missing frame, investigate the pairing algorithm before changing hardware. If one stream appears progressively shorter, use the checks for two-camera recording duration mismatch to distinguish missing frames from clock-rate differences.
4. Rule out spatial mismatch with a stationary target
A displaced color edge is not, by itself, evidence of a timing fault. Separate viewpoints create parallax. Incorrect calibration, lens distortion handling, cropping, and resizing can also move corresponding features.
First inspect a stationary target at several depths. If the mismatch persists without motion, investigate registration before estimating temporal error. Do not assume that equal image dimensions mean that corresponding pixels view the same scene point.
Use the correct intrinsics for each stream and the transformation between their coordinate systems. If images were resized or cropped, follow the principles in camera intrinsic parameter conversion. For projecting image locations into depth geometry, the bounding box depth mapping guide addresses the spatial part of the problem.
Next, move the target laterally across the field of view at a controlled speed. A mismatch that changes with speed, and reverses direction when motion reverses, supports a temporal explanation. It is not conclusive alone: rolling-shutter distortion and depth reconstruction artifacts can also depend on motion.
5. Turn the allowed image error into a timing budget
Define synchronization by the inspection requirement, not by whether timestamp values are identical. A useful first estimate relates image displacement to motion speed and time offset.
For approximately constant lateral motion, with a local image scale:
Consider a hypothetical image covering 600 mm across 1200 pixels. The sampling is:
Here, a 2 px displacement allowance gives a 5 ms temporal budget. A measured acquisition-reference difference of 3 ms would contribute approximately:
This is an estimate, not a complete error budget. Calibration residuals, exposure blur, depth uncertainty, and occlusion require separate allowances. Image scale also changes with distance in a perspective view.
Use the resolution calculator to check the spatial sampling portion of this example:
Frame rate alone does not establish synchronization. At 30 frames/s, nominal spacing is about 33.3 ms. With ideal, evenly spaced frames, nearest-frame selection can still leave approximately 16.7 ms of mismatch for an event midway between frames. That does not satisfy the example's 5 ms limit.
6. Inspect exposure and depth-generation timing
Once clocks and pairing are correct, inspect the acquisition modes. Equal frame rates do not imply equal exposure start times, exposure durations, or timestamp conventions.
A global-shutter image represents an exposure interval rather than an instantaneous sample. A rolling-shutter image adds row-dependent timing: the top and bottom of the image may describe different moments. A depth result may also depend on several illumination patterns or exposures, depending on the sensing method and mode.
Check the documented behavior for the exact configuration:
- Whether the timestamp references exposure start, midpoint, end, or another event.
- Whether automatic exposure changes the effective acquisition interval.
- Whether resolution or frame-rate changes alter readout timing.
- Whether depth processing combines measurements from different instants.
- Whether temporal filtering carries information from previous frames.
- Whether a supported synchronization input coordinates the required acquisition events.
If both timestamps describe exposure start, unequal exposure durations can still produce different midpoints. Comparing start times alone may therefore understate the mismatch relevant to moving objects.
Record exposure metadata when available, and follow the checks for per-frame exposure and gain. Change one mode at a time, then repeat the moving-target test. Do not assume a trigger input guarantees aligned exposures across every supported mode.
7. Isolate transport and recording delays
If acquisition timestamps look consistent but recording times vary, investigate the delivery pipeline. Compare operation with preview rendering, image conversion, compression, and recording enabled separately.
Begin with a short acquisition that logs metadata without expensive image processing. Then add processing stages individually. Look for increasing queue occupancy, frame-counter gaps, and growing differences between arrival and acquisition times.
You may calculate arrival-minus-acquisition delay only after expressing both timestamps in a common clock domain. Otherwise, the result contains an arbitrary clock offset and has no direct latency meaning.
A useful diagnostic sequence is:
- Confirm that acquisition timestamps remain monotonic within each continuous clock segment.
- Check frame counters for gaps or repeats, accounting for documented counter behavior.
- Compare acquisition intervals against the configured operating mode.
- Compare arrival intervals and recorder throughput under load.
- Check whether frame pairing still meets the temporal limit after delivery jitter.
Increasing buffers can absorb short bursts, but it does not correct mismatched exposures. It can also allow old frames to remain queued longer. Treat buffer changes as transport adjustments, not synchronization fixes.
8. Choose hardware around documented timing behavior
Do not replace a sensor because its messages appear on separate recording tracks. First establish whether its acquisition behavior actually fails your timing budget.
When evaluating 3D sensors, request timestamp definitions, clock-domain information, acquisition duration, synchronization support, and behavior in the intended depth mode. The category provides a starting point for comparing sensing approaches.
Laser profilers and stereo 3D cameras
170 models · from $699.00The MV-DPS200P-02 is a projected structured-light camera. Its inclusion here identifies a sensing approach, not a claim about a supported depth-and-color timing arrangement.
- Type
- Stereo / structured light
If your design uses a separate color channel, evaluate area scan cameras against the same timing requirements. The LS-CMR-0166 is an area scan camera; it is not presented as a verified synchronized partner for the depth camera above.
- Sensor format
- 1/1.7"
- Interface
- USB3
- Resolution
- 12 MP
- Shutter
- Rolling
Use the camera finder to narrow candidate hardware, then confirm the required timing features and operating modes. Available catalog descriptions alone do not establish compatibility between separate devices. The camera configurator can organize the camera selection after those requirements are defined.
9. Verify the fix and preserve the evidence
Repeat the original test after each change. A successful repair should improve measured acquisition pairing and the moving-target result, not merely make the recorder timeline look tidier.
Keep this acceptance checklist with the recording:
- Timestamp source, units, clock domain, and reference event are documented.
- Original timestamps survive export without unexplained replacement or precision loss.
- Frame pairing uses compatible acquisition times or documented pair identifiers.
- Accepted pairs remain within the application-derived time limit.
- Missing frames are detected rather than silently shifting later pairs.
- A stationary target passes the spatial-registration check.
- Moving-target error remains acceptable at the required speed and scene depths.
- Long recordings show no unexplained drift, resets, or growing delivery backlog.
- Software versions, acquisition modes, exposure settings, and filtering settings accompany the results.
For hardware selection help, send the required motion speed, allowed image error, stream modes, and a short timestamp log through application support. Those details make the next step a timing assessment rather than a guess based on message arrival order.
Questions
Do different recording timestamps prove that depth and color are unsynchronized?
No. A recorder can receive or store corresponding messages at different times because their processing and transport paths differ. Inspect the original image timestamps, verify what they represent, and establish a common clock domain before deciding whether the acquisitions exceed your application's timing allowance.
Can I assume an image header contains the exposure timestamp?
Not without checking the driver's documented behavior. An image header may carry a sensor-generated acquisition timestamp, a converted device timestamp, or a host-assigned value. You also need to know the timestamp's units and whether it references exposure start, midpoint, end, or another event.
Will matching depth and color frame rates synchronize the images?
No. Frame rate describes how often frames are produced, not their relative acquisition phase. Streams can run at equal nominal rates with a fixed offset, and independent clocks can drift. Exposure duration, rolling readout, and depth-generation timing can introduce additional differences relevant to moving objects.
How should I choose an acceptable synchronization tolerance?
Measure image motion at the operating distance and divide the allowed displacement by that speed. Reserve part of the total error allowance for calibration, blur, and depth uncertainty. Validate the resulting time limit with a controlled moving target rather than relying only on timestamp statistics.
When should I consider different camera hardware?
A replacement decision needs documented evidence about clock sharing, exposure timing, synchronization inputs, and behavior in the required acquisition modes. A catalog category or matching interface is not enough. First rule out timestamp interpretation and pairing errors, because different hardware will not automatically correct those software problems.