3D visionTroubleshooting

Color image and point cloud synchronization: pair frames by timestamp

Why a fast part gets XYZ from an older point cloud than its color frame, how to measure the skew in milliseconds, and how to pair frames by timestamp.

LumaSpec Vision10 min read Share on LinkedIn

On this page 9 sections
  1. 01Symptom: the XYZ error grows with part speed
  2. 02Likely causes, most probable first
  3. 03Measure the skew before you tune anything
  4. 04Worked example: two frames of skew on a part moving at 1 m/s
  5. 05Fix: pair frames by capture timestamp
  6. 06Skip the cloud: project one point instead
  7. 07Hardware that keeps both views in one capture
  8. 08Keeping it fixed
  9. 09Questions

You detect the part in the color image, then read its XYZ from the point cloud, and on a fast part the coordinates land behind the detection. At 60 fps the frames are 16.7 ms apart, and the cloud built from a depth frame finishes later than the color frame captured at the same instant. Your detector therefore works on a newer color image than the cloud it reads coordinates from, and the error grows with part speed. This guide is for anyone who filters with color and measures with a generated cloud on a moving line. You will measure the skew in milliseconds, pair each color image with the cloud captured closest to it, and decide when to skip a detection or project one point instead of generating a cloud.

Symptom: the XYZ error grows with part speed

The signature is easy to recognize. At standstill or low speed, coordinates check out against a gauge or a calibrated target. As speed rises, the reported XYZ drifts backwards along the direction of motion, and the offset grows in proportion to speed. Moving the camera changes nothing; changing conveyor speed changes everything.

That scaling is the fingerprint of a time skew rather than a calibration fault. A calibration error produces the same offset at every speed. A time skew produces an offset equal to the distance the part travels between the color frame you detect in and the depth frame the cloud was built from:

T = 1 / fps Δt = N × T e = v × Δt

Here e is the along-motion error in mm, v the part speed in mm/s, N the number of frame periods between the color capture and the cloud's depth capture, and fps the stream rate. At 60 fps one frame period is 16.7 ms, so each frame of skew costs about 17 mm per m/s of part speed — a part at 1 m/s detected two frames ahead of its cloud reads about 33 mm off.

Likely causes, most probable first

Work down this list and run each test before changing a setting.

  1. Arrival-order pairing. The detection fires on the newest color frame and the code grabs the newest cloud in the queue. Cloud generation finishes behind the color path, so the newest cloud belongs to an older instant; two color frames of skew is common. Test: count how many color frames arrive between two consecutive clouds. One or more means every detection can read a stale cloud.
  2. Mixed timestamp domains. The capture stamp (assigned at the sensor), the host arrival stamp (assigned when the driver hands over the buffer) and the processing-complete time are three different clocks. Queues on one path can even invert arrival order. Test: print capture and arrival stamps for the same frame pair and compare them.
  3. Cloud pipeline slower than the stream. Depth and color deliver at 60 fps, but generation cannot sustain that rate, so a backlog builds and the delivered cloud rate sags. Published real-time capture measurements put depth-to-color registration alone at about 29 ms for unbinned 720p depth and about 42 ms at 1080p, before your own cloud code runs. Test: count clouds delivered per second over one minute and compare with the set rate; if it sags, see 3D camera frame rate low on software trigger: mode and data streams.
  4. Loose synchronizer policy. An approximate-time policy accepts pairs whose stamps are merely close, and a generous queue lets close mean several frames apart. Test: log the capture-time difference of every accepted pair. The aligned-stream version of this lag is covered in depth and RGB out of sync: alignment latency and the fps fix.

Measure the skew before you tune anything

Tuning buffer sizes first hides the number you need. Measure it:

  • Stamp at capture. Carry the sensor's capture timestamp and sequence number with the color frame, the depth frame and every cloud generated from them. The cloud inherits the depth frame's stamps; generation time stacks on top, which is exactly what you must not pair on.
  • Log the delta per detection. For each detection your code actually uses, compute Δt = t_color_capture − t_depth_capture of the cloud read. A histogram over one minute of running tells you N.
  • Calibrate with motion. Run a part at a known speed v, record the along-motion error e, and compute Δt = e / v. A part at 1 m/s reading 33 mm long gives 33 ms, which is two frames at 60 fps.

Worked example: two frames of skew on a part moving at 1 m/s

Inputs. Both streams run at 60 fps, so T = 1/60 s = 16.7 ms per frame. The skew is N = 2, the case where two color frames arrive while the cloud for an older depth frame is still queued. The part moves at v = 1 m/s = 1000 mm/s, conveyor-class speed for a pick handoff. The placement tolerance is ±10 mm.

Rule. e = v × N / fps: the part travels one frame period's worth of distance for every frame the cloud lags.

Arithmetic. Skew time = 2 × 16.7 ms = 33.3 ms. Error = 1000 mm/s × 0.0333 s = 33 mm, directed along the motion. That is three times the ±10 mm tolerance.

Decision. With N = 2 at 60 fps, the tolerance caps the part at v = 10 mm / 33.3 ms ≈ 300 mm/s. Above 0.3 m/s you must cut N to zero by pairing the detection with the cloud of its own instant, raise the frame rate — at 90 fps the same two frames cost 2 × 11.1 ms = 22.2 ms, an error of 22 mm — or read XYZ only while the part is stopped. Pairing exactly removes N altogether; slower parts and higher rates only shrink it.

Fix: pair frames by capture timestamp

  1. Stamp every frame at capture. Keep the device timestamp and sequence number with the color frame, the depth frame and each cloud; the cloud inherits the stamps of the depth frame it was built from.
  2. Pair color to depth once. Establish the color-to-depth mapping for the same capture instant at start-up, from the intrinsics and the device's own alignment, so sequence k in color resolves to sequence k′ in depth.
  3. Resolve the detection through the pair. A detection at color sequence k maps to depth sequence k′; accept only the cloud stamped k′, within a tolerance of half a frame period — 8.3 ms at 60 fps.
  4. Skip, don't substitute. No cloud within tolerance means skip that detection and let the next instant carry it. Reading the newest cloud instead reintroduces exactly the error you measured.
  5. Bound the queues. Keep the cloud queue one or two frames deep and drop the oldest on overflow, so a backlog cannot grow unnoticed.
  6. Re-run the moving test. The along-motion error should collapse to your calibration residual. If your SDK numbers frames rather than stamping them, sequence-number pairing works identically — see color lags depth by one frame: sync RGBD frames by sequence number.

Skip the cloud: project one point instead

If the deliverable is the XYZ of one detected object, a full cloud per frame buys delay, not information. Detect the object in the color frame, map its pixel or mask centroid into the depth frame of the same capture pair, read Z there, and unproject the pixel with the depth intrinsics for full X, Y, Z. The generation stage leaves the latency budget entirely, and the only remaining requirement — that the depth frame belongs to the same instant — is what the pairing above already guarantees.

Plan for two failure modes. No valid depth at the projected pixel (dark part, shiny surface, beyond range): take the median valid depth in a 5 × 5 pixel window, or skip. And masks that spill past the part edge: erode the mask a few pixels before projecting. Both cost less than a stale cloud. The mapping mechanics are in color pixel to depth projection: one point instead of a whole frame and the pixels-to-XYZ conversion in 2.5D height map to 3D point cloud: pixels to XYZ.

Hardware that keeps both views in one capture

Two choices in the hardware plan shrink the problem before any code runs. First, the camera: when color and depth are captured in one event inside one device, both frames share a timestamp base and N starts at zero. Structured-light 3D cameras work this way — the 3D sensor category lists them under the stereo / structured-light type, and the 3D camera finder filters by type. A projected structured-light model such as the MV-DPS200P-02 is that class of part:

3D sensorsHikrobot MV-DPS200P-02 Projected structured-light 3D cameraMV-DPS200P-02
Type
Stereo / structured light
$8,829.00 · quotedView and quote

Second, the host: run the color path, the depth path and the pairing code on one machine, so every buffer lands on one host clock and the sequence counters stay comparable. A vision controller with PoE camera ports carries the 3D camera's data and power over a single cable and leaves the remaining LAN ports for PLC and robot traffic:

Vision controllersHikrobot MV-VC5525-256G70-NN VC5000 Vision Controller, 12th-I5, 32GB+256G SSD, Supported PCIEMV-VC5525-256G70-NN
LAN ports
5+ LAN ports
USB 3 ports
3-4 USB 3 ports
PoE
PoE camera ports
Light outputs
8 built-in channels
$4,479.00 · quotedView and quoteBuild a system with it

Whether to fuse the two views in one process or split them across nodes is an architecture call with sync consequences; 2D and 3D capture strategy: combine or split RGB and point cloud compares the two.

Keeping it fixed

The skew follows the settings: change a setting and it moves.

  • Frame rate. Error scales with 1/fps. Dropping the streams from 60 to 30 fps doubles the frame period from 16.7 to 33.3 ms and doubles every error above. Re-run the moving test after any rate change.
  • Resolution. Generation cost climbs with pixels — the published registration figures above rise from about 29 ms at 720p to 42 ms at 1080p. A sharper color stream can widen N.
  • Delivered rate. Log clouds per second against the set rate and alarm on divergence; a widening gap is a growing backlog. Drift that accumulates over a run is its own failure mode — depth color timestamp drift that grows while running.
  • Pair log. Record the capture-time delta of every pair a detection consumed. One week of logs settles any future dispute with a histogram instead of an anecdote.

More frame-pairing fixes sit on the 3D topic page. If you are building a station rather than repairing one, shortlist the camera in the 3D camera finder, add the host from the vision controller category, and run the whole chain through the system configurator. And if a skew survives correct pairing, contact the applications team with your pair-delta histogram — it is the first thing we will ask for.

Questions

How much position error does one frame of delay cause at 30 fps?

At 30 fps the frame period is 33.3 ms, so each frame of skew adds about 33 mm of along-motion error per m/s of part speed — double the 60 fps figure of roughly 17 mm. A part at 0.5 m/s with two frames of skew reads about 33 mm off at 30 fps and about 17 mm off at 60 fps. Error scales linearly with speed and inversely with frame rate.

What tolerance should I use when matching a color frame to a point cloud?

Half a frame period. At 60 fps that is 8.3 ms: a cloud whose depth capture time sits within 8.3 ms of the color frame's paired depth time is accepted, and anything older is rejected as stale. Keep the tolerance at half a frame whatever the rate — 16.7 ms at 30 fps, 4.2 ms at 120 fps.

Can I pair color and point cloud with an approximate-time synchronizer?

Yes — message_filters with an approximate-time policy in ROS joins topics whose stamps are close, but it will still hand you pairs several frames apart unless you tighten the queue and inspect the stamps. Synchronize on capture timestamps, log the accepted deltas, and treat any pair beyond half a frame period as a miss. Queue size changes which frames pair, not how stale the cloud pipeline is.

Why is my point cloud latency different every frame?

Because the delay is queue-dependent, not fixed. Cloud generation competes with the color path and other processes for bandwidth and CPU, so the backlog depth moves frame to frame — sometimes two color frames of skew, sometimes three. That is why arrival-order pairing fails intermittently and looks like a random vision error. Logging the capture-time delta per pair turns the variance into visible rejections instead of silent coordinate errors.

Does a camera that outputs color and depth together remove the problem?

Mostly, yes. When one device captures both views in the same event, the frames share one timestamp base and the pairing reduces to reading the stamps, so cross-stream skew starts at zero. What remains is your own downstream delay: detection and coordinate read must still reference the same captured pair, and taking the newest output instead of pairing by stamp can still hand you a stale result.