3D visionExplainer

Track stereo features between frames for visual SLAM pose

How temporal descriptor matching tracks stereo features between frames, how to size the search window from speed and frame rate, and when to re-detect.

LumaSpec Vision11 min read Share on LinkedIn

On this page 11 sections
  1. 01The two matches every tracked feature needs
  2. 02The per-frame pipeline
  3. 03Size the search window from your own motion
  4. 04Sizing the window for a 30 fps rig moving 0.5 m/s at 1 m depth
  5. 05Frame rate is the control you turn
  6. 06Descriptor matching or KLT: choosing the tracker
  7. 07When the usual answer is wrong
  8. 08Checks that keep the tracker honest
  9. 09Building the rig: what the hardware owes the tracker
  10. 10The decisions, in one place
  11. 11Questions

Your first stereo pair gave you the map: corner features detected in the left image, matched across the pair, triangulated to 3D points. The rig now moves and you need a pose for every new frame, which means finding those same world points again. The efficient way is temporal descriptor matching: match the previous left image against the new left image inside a small search window, keep only features that also match across the stereo pair, and solve the pose from the survivors. Point-cloud overlap cannot do this job, because an independently detected cloud carries no feature identity. This page gives the pipeline step by step, the formula that sizes the search window from your speed and frame rate, a worked example for a 30 fps rig moving 0.5 m/s with the scene 1 m away, and the checks that keep the tracker healthy.

The two matches every tracked feature needs

Feature tracking in a stereo SLAM front end is two associations, not one.

The stereo match runs inside one frame pair. A feature found in the left image is searched along its epipolar line in the right image; the offset gives disparity and the disparity gives depth. That is how the first pair produced your 3D points, and the mechanics are identical in every later pair — the disparity-to-3D math does not change from frame to frame.

The temporal match runs between successive left images. It answers the question your map poses: which corner in the new frame is the same physical edge as the point triangulated two frames ago? The descriptor stored with each feature is the identity. Two detections describe one world point when their descriptors agree and their positions fit the predicted motion.

Asking whether the two point clouds overlap skips both associations. The new cloud is a fresh set of detections with fresh descriptors and no linkage to the old points, and aligning clouds to each other needs the very pose you are trying to estimate. Association happens in image space, where the descriptor lives. Stereo also settles scale: with the baseline known, depth is metric from the first pair, while a monocular rig must find observable scale before its pose is metric.

The per-frame pipeline

  1. Carry state. Every tracked point keeps its 3D coordinate, its descriptor and the frame where it was last seen.
  2. Predict. Project the 3D point into the new left image with the predicted pose: the last pose plus a constant-velocity and constant-turn-rate model, or wheel odometry where the rig has it. A point at (X, Y, Z) in camera coordinates lands at u = f_x·X/Z + c_x, v = f_y·Y/Z + c_y.
  3. Search a window. Compare descriptors only within a radius of r pixels around the prediction, and reject a candidate whose descriptor distance is not clearly better than the second-best; the common ratio cutoff sits at 0.7 to 0.8.
  4. Match the stereo pair. Search the right image along the epipolar line for the survivors and triangulate, or reuse depth for points the map already trusts.
  5. Solve the pose from the 3D-to-2D pairs, rejecting outliers by reprojection error. The back end then fuses these noisy per-frame measurements in a pose graph into a smooth trajectory.
  6. Grow the map. Unmatched points stay in the map for later frames. When tracked inliers drop below a floor — 50 to 100 is a common band — detect new features, match them across the pair, triangulate and add them.

KLT optical flow replaces steps 2 and 3 with patch tracking: it follows each patch frame to frame by local refinement, cheaper than computing descriptors, but a dead track loses the point's identity. Production front ends commonly run KLT between frames and keep descriptors from detection time for re-localization and loop closure.

Size the search window from your own motion

The radius comes from one quantity: how far the image content moves between frames. Translation moves near objects more than far ones, so pixel motion grows with focal length and speed and shrinks with depth.

ΔX = v / fps translation per frame (m) m_t = f_x · ΔX / Z pixel motion from translation (px) m_r = f_x · ω / fps pixel motion from rotation (px), ω in rad/s r = 1.5 to 2 × (m_t + m_r) search radius (px), margin for prediction error

The 1.5 to 2 margin covers odometry drift and the motion the model did not predict. Compute it with your own f_x and depth: halve the depth and the window doubles.

Sizing the window for a 30 fps rig moving 0.5 m/s at 1 m depth

Inputs, and where each comes from:

  • v = 0.5 m/s — rig speed, a typical indoor mobile-robot pace.
  • fps = 30 — the stereo capture rate the rig delivers.
  • Z = 1.0 m — distance to the dominant surface in view.
  • f_x = 1200 px — focal length in pixels, read from your calibration; a 1920 px-wide image with a 77° horizontal field of view sits near this value.

Arithmetic:

  • Translation per frame: ΔX = 0.5 m/s ÷ 30 fps = 0.0167 m.
  • Pixel motion from translation: m_t = 1200 px × 0.0167 m ÷ 1.0 m = 20 px.
  • With the rig also panning at 15°/s: ω/fps = 0.0087 rad per frame, so m_r = 1200 px × 0.0087 ≈ 10 px.
  • Radius with a 1.5 margin: r = 1.5 × 20 px = 30 px, or 1.5 × 30 px = 45 px with the pan included.

Decision: a 30 px radius — a 60 × 60 px search window — covers this rig. It absorbs up to 0.75 m/s at 1 m depth, since 30 px × 1.0 m ÷ 1200 px × 30 fps = 0.75 m/s, half as much headroom again above nominal. Past that speed, or when the scene closes to 0.5 m (motion doubles to 40 px), double the frame rate instead of the window: at 60 fps the translation motion is 10 px and the radius drops to 15 px. Match cost scales with window area, so 60 fps with a 15 px radius beats 30 fps with a 60 px radius on CPU time and on false matches.

Exposure belongs to the same sizing. Blur in pixels is v · t_exp · f_x / Z, so keeping it at or under 1 px needs t_exp ≤ Z / (v · f_x) = 1.0 m ÷ (0.5 m/s × 1200 px) ≈ 1.7 ms. The 33 ms frame period at 30 fps leaves room; a 10 ms exposure would smear features by 6 px and the descriptors degrade before the window even matters.

Frame rate is the control you turn

Everything above scales with 1/fps. Halve the frame rate and the per-frame motion, the search radius and the blur budget all double, while the window area — and with it match time and the false-match rate — grows about fourfold.

One dropped frame does the same damage as half the frame rate for that interval: at 30 fps the next interval stretches to 67 ms, the expected motion to 40 px, and a 30 px window is too small. Measure what arrives under full CPU load with the delivered-frame-rate check; a rate that alternates between two values is dropped frames, not the configured number.

Where the budget is tight, spend it on frame rate rather than resolution: the window scales with f_x, which scales with image width for the same lens, so a modest image at 60 fps is a steadier SLAM input than 4K at 10 fps. Scaling thresholds to the image is its own tuning task.

Descriptor matching or KLT: choosing the tracker

KLT optical flow Descriptor matching (SIFT- or ORB-style)
Cost per frame Low: patch refinement only Higher: detect and describe each frame
Motion per frame it tolerates Small, a few pixels Large, limited by descriptor repeatability
Identity after a lost track Gone; the point is abandoned Recoverable by re-detection and re-match
Re-localization and loop closure Needs stored descriptors anyway Native: old descriptors match new frames

Choose KLT when frames arrive fast and motion per frame stays within a few pixels — the regime the worked example lands in at 60 fps. Choose descriptor matching when frames are sparse, when the rig stops and restarts, or when re-localization matters. The hybrid — KLT between frames, descriptors stored at detection — is the common production compromise.

When the usual answer is wrong

The rig never moves. A fixed camera needs no temporal tracking; every frame is only a stereo match against a static scene. That is the operating mode of a projected structured-light camera such as the MV-DPS200P-02, which reconstructs depth from its projected pattern and stereo pair in one capture.

3D sensorsHikrobot MV-DPS200P-02 Projected structured-light 3D cameraMV-DPS200P-02
Type
Stereo / structured light
$8,829.00 · quotedView and quote

The scene has no texture. Both KLT and descriptors need corners or gradient. A bare wall or a polished panel starves the tracker at any window size; add texture, light the surface so detail appears, or move to a depth sensor that measures range directly — whose intensity image still needs texture for tracking.

The scene moves. People and vehicles generate candidate matches that violate the epipolar and reprojection geometry; outlier rejection discards them while static features are the majority. Where dynamic objects dominate, recent visual-SLAM work detects and filters dynamic instances before matching, because the static minority can no longer carry the pose.

The shutter rolls. A rolling-shutter sensor exposes each row at a different instant, so rig motion skews the image and a single prediction per point cannot fit every row. Use global-shutter cameras on anything that moves.

Checks that keep the tracker honest

  • Log tracked inliers per frame and trigger re-detection below the floor; a slow decline over minutes usually means lighting drift or texture leaving the view.
  • Lock exposure and gain once lighting is settled: auto exposure shifts brightness between frames and descriptor values move with it, so set it per stereo camera in the rig.
  • Plot predicted against matched positions. A systematic offset points at calibration, most often residual lens distortion; the same signature shows up in aligned point clouds.
  • Record stereo pairs and replay them to tune the ratio threshold, radius and floors offline; the recording workflow for visual SLAM is built for that.
  • Watch the stereo half too. The temporal window can be perfect and the depth still wrong if the left-right search fails; a depth camera that streams left and right pairs explicitly makes both halves inspectable.

Building the rig: what the hardware owes the tracker

The pipeline assumes two synchronized global-shutter cameras on a link that sustains the frame rate. Two C-mount global-shutter GigE cameras such as the MV-CU020-90GC, hardware-triggered from one source, form the variable-baseline rig; a mono 20 MP USB3 alternative in the same class is the MV-CS200-10UM.

Area scan camerasHikrobot MV-CU020-90GC 2.5 MP Area Scan Camera, GigE, GMAX4002, ColorMV-CU020-90GC
Sensor format
1/1.7"
Lens mount
C-mount
$309.00View part
Area scan camerasHikrobot MV-CS200-10UM 20 MP Area Scan Camera, USB3.0, IMX183, 2nd Gen., MonoMV-CS200-10UM
Sensor format
1"
Lens mount
C-mount
$759.00View part

Global shutter exposes every row at the same instant, so one prediction fits the whole image. Hardware sync keeps the two captures within microseconds of each other; a 1 ms offset at 0.5 m/s is a 0.5 mm error in the stereo triangulation, and that is depth error straight into the map.

Build the whole station

The configurator matches a camera, lens, light, light controller and cables to each other, step by step, and lists only parts that fit what you already picked.

Start with the camera

The decisions, in one place

  1. Associate in image space. Descriptors are the identity; point-cloud overlap needs the pose you are still estimating.
  2. Size the window from your numbers: r = 1.5 to 2 × f_x × v / (fps × Z), plus f_x × ω / fps for rotation.
  3. For the 30 fps, 0.5 m/s, 1 m rig with f_x = 1200 px: a 30 px radius and exposure at or below 1.7 ms.
  4. Buy frame rate before window size. Past the window's speed budget — 0.75 m/s in the example — double fps rather than the radius.
  5. Re-detect on a floor, not on failure. Keep the inlier floor, add new points when it is crossed, and let the map grow.

To select hardware, start in the stereo and structured-light 3D sensors and area scan cameras categories, filter by shutter, interface and resolution in the camera finder, or let the configurator walk the chain end to end. More depth-side reading lives on the 3D topic page, and the contact page reaches the applications team when you want a second opinion on a build.

Questions

Can I align the two point clouds instead of matching in the image?

Cloud-to-cloud alignment exists (ICP-style nearest-neighbour refinement) but it needs a good initial pose to converge, which is exactly what you are trying to estimate. It also gives no per-feature identity, so the map cannot grow from it. Use image descriptors for association and reserve cloud alignment for refining a pose you already have.

What if the rig has no odometry to predict motion?

Predict from the previous solution itself: assume constant velocity and angular rate between frames, which works well when the frame rate is high relative to the motion. With no prediction at all, full-frame descriptor matching still works, but the search cost grows with image area and repetitive scenes produce false matches, so expect a lower inlier rate and a slower loop.

How many features do I need to track per frame?

The pose solve needs on the order of tens of good 3D-to-2D pairs; production front ends hold hundreds of tracked points to survive occlusion and outliers. Set a floor on tracked inliers, with 50 to 100 a common band, and trigger new detection and map growth whenever the count drops below it.

What frame rate should a stereo SLAM rig run?

Aim for pixel motion in the low tens of pixels per frame, then solve the window formula for the rate: fps ≥ v × f_x ÷ (r × Z). At 0.5 m/s, f_x = 1200 px, Z = 1.0 m and a 20 px budget, that gives 30 fps. Size it from the delivered rate under load, not the configured one.

Which ratio threshold should I use when matching descriptors?

For SIFT-style float descriptors the common ratio threshold between the best and second-best match is 0.7 to 0.8; tighten toward 0.7 in repetitive scenes, loosen toward 0.8 when texture is sparse and you need more candidates. Binary descriptors such as ORB are usually filtered by an absolute Hamming-distance cutoff instead.

How does a variable baseline change the tracking?

The temporal match is unchanged; the baseline only affects the stereo half. A wide baseline buys depth accuracy but grows the left-right search and can push near points out of the right image entirely, so expect fewer close-range stereo matches whenever the rig widens the baseline mid-run.