Stereo camera fps drops in pose tracking: find the real bottleneck

A stereo pose-tracking rig that holds 25 fps on the bench and drops the moment the rig or the part shakes is a common fault, and almost never a dying camera. Shake does not slow the sensor; it blurs the image, stereo matching fails, and the tracker falls back to full feature detection every frame — work that spikes the processor. A desktop absorbs that spike; an embedded GPU box drops frames instead. This guide splits the problem with one measurement — the delivered frame rate taken from per-frame timestamps — then works the four causes in order of likelihood: processing load, exposure growth, thermal throttling, and the cable or mount. You finish with the numbers for your rig, the cause that explains your drop, and the fix, including the mounting hardware that removes shake at its source.

Measure two frame rates, not one

Your pipeline has two rates, and they fail for different reasons. The delivered rate is what the camera hands over, read from the timestamp already attached to every frame. The output rate is how many pose transforms the tracker publishes per second. Log both over the same 60 seconds while you reproduce the shake.

import statistics

# Illustrative: delivered fps from the timestamps already attached to each frame.
# ts: list[int] of per-frame timestamps in microseconds, read from your SDK's
# frame objects, in capture order.
gaps = [b - a for a, b in zip(ts[:-1], ts[1:])]
delivered_fps = 1_000_000 / statistics.median(gaps)
# test: a 30 s log with 750 frames gives gaps near 40000 µs → delivered_fps = 25.0

The decision falls out of the pair:

• Delivered holds near 25 fps while the output rate falls: the camera is fine and the processing side is starving — work causes 1 and 3.
• Both fall together: acquisition is the limit — work causes 2 and 4.

This is the same split behind measuring the delivered frame rate on a low-fps stereo camera, and the same gap you close when acquisition outruns processing.

The frame-period budget

Each frame passes through exposure, readout, transfer and processing, pipelined. The slowest stage sets the frame rate; the sum of the stages is your latency:

At 25 fps your budget is 40 ms per frame. One stage crossing 40 ms caps the rate: an exposure of 50 ms gives 1000 / 50 = 20 fps, and processing of 70 ms gives about 14 fps. On a triggered stereo pair the camera cannot start the next frame until exposure and readout both finish — the same exposure-plus-readout sum that sets the ceiling on a hardware trigger. Find which term grew when the shake started and you have your cause.

Why shaking costs frames

Shake couples into the pipeline through blur. A blurred patch gives the stereo matcher no clean edges to correlate between left and right images, so disparity fails there and the depth map grows holes. The pose tracker then loses its feature tracks and drops back to full-scene detection and re-initialization, typically several times the cost of holding a lock. Depth filtering makes it worse: hole-filling and temporal filters run longest exactly when the map has the most holes.

None of that touches the camera. It is pure compute, and headroom decides who notices. A published performance comparison of motion-detection methods on embedded hardware measured the GPU about four times faster than the CPU for high-compute methods at high resolution, and an embedded box still carries a fraction of a desktop's headroom — the spike a desktop absorbs costs real frames there.

Worked example: how much blur the shake makes

Take a rig that holds 25 fps and starts dropping under a 20 Hz shake. Assumptions, all typical:

• Shake amplitude ±0.4 mm at 20 Hz
• Working distance 500 mm over a 300 mm field of view
• A 1/1.8-inch class stereo sensor, about 7.2 mm wide, with 3 µm pixels
• Exposure 10 ms

The peak velocity is v = 2π × f × A = 2π × 20 × 0.4 = 50 mm/s. Magnification is m = 7.2 / 300 = 0.024, so the image sweeps the sensor at 50 × 0.024 = 1.2 mm/s. In 10 ms that is 12 µm, or 12 / 3 = 4 pixels of blur — far past the half-pixel budget that sub-pixel disparity needs.

Holding 0.5 pixels needs t = 0.5 × 3 µm / 1.2 mm/s = 1.25 ms, call it 1.3 ms. Dropping from 10 ms to 1.3 ms costs 10 / 1.25 = 8 times the light, about three stops, at equal noise. Doubling the shake velocity halves that exposure budget again.

Decision: at this shake you will not buy 8× more light. You remove the shake at the mount, or you accept failed matching and the load spike that follows.

Cause 1: the tracker re-detects every frame

Signature: delivered fps holds at 25, the pose output rate falls, and the fall tracks shake amplitude.

Tests: watch processor load through the shake — pinned near full while static scenes idle lower — and replay a recorded shaky sequence offline with a timer; if per-frame processing exceeds 40 ms, the math already explains the drop.

Fixes, in the order that pays:

1. Cap exposure so blur stays under about half a pixel; tracking keeps its lock and re-detection never starts. Capping the maximum auto-exposure time does exactly this.
2. Drop the depth resolution one step; disparity search cost scales with resolution.
3. Turn hole-filling and temporal filters off while diagnosing, and add back only what the application needs.
4. Where the SDK offers the mode, run re-detection on an interval and track in between.

Decision: if the output rate recovers the moment blur is capped, this was your cause.

Cause 2: exposure grows past the frame period

Signature: both rates fall together, and the per-frame exposure readout grows when the drop starts.

Test: log exposure time next to the timestamps. Any exposure over 40 ms caps you below 25 fps; 50 ms means 20 fps. Some depth cameras run an auto-exposure priority mode that frees exposure to cross the frame period, and the rate then steps down, often by half — why auto-exposure priority halves the frame rate covers the mechanism. A rig that sweeps the scene changes brightness frame to frame, so the loop hunts exactly when it moves.

Fix: set a maximum exposure of half the frame period, 20 ms at 25 fps, and let light and gain carry the rest. Exposure settings capping a depth camera's FPS walks the same cap on a depth stream.

Decision: exposure over budget at the instant of the drop is the whole answer; nothing downstream can fix it.

Cause 3: the embedded box throttles

Signature: the drop builds over minutes of sustained load and recovers after idle, whatever the shake does.

Test: read the module's clock and temperature telemetry during a run. Falling clocks on rising temperature is throttling; the drop is time-dependent, not motion-dependent. Remove screen capture and any viewer while measuring — recording video is sustained GPU load of its own.

Fix: cool the box to its sustained power limit and cut standing load: lower depth resolution, cap the frame rate at what the tracker consumes, log timestamps instead of recording video. Sustained recording FPS drops on an embedded vision box covers the thermal path in depth.

Decision: a clean first few minutes that decays later is thermals, and no camera change will fix it.

Cause 4: the link drops frames under vibration

Signature: delivered fps falls only while the rig shakes; on the bench it holds 25.

Test: swap in the shortest cable that reaches, shielded, with strain relief at both ends, and shake again. A connector working loose micro-interrupts the link, the driver resets, and each reset costs hundreds of milliseconds. If several cameras share the bus, budget the USB bandwidth before blaming the camera.

Decision: a bench cable that holds 25 fps under the same shake convicts the installed cable, not the camera.

Stop the shake at the mount

The least expensive fix in the chain is mechanical: remove the excitation instead of paying for it in exposure, light and compute. Bolt the camera to a rigid stand with a short column and a wide base, fixed to machine structure rather than to a sheet-metal panel that rings with it:

Keep the moment arm short: the same base shake turns into several times more motion at the tip of a long arm than at the camera flange. To separate the machine from the camera, reproduce the fault with the rig on a bench stand such as the LS-MNT-0006; if the drop follows the camera to the bench, the fault travels with the rig, not the machine.

Two logs confirm the mechanical side. An IMU in the rig gives the shake frequency directly — check that its read rate holds while video streams, or you are diagnosing with a sagging sensor. You can also detect camera vibration from the frames themselves with no extra hardware. Shake corrupts the data you do deliver as depth jitter and IMU drift, and measuring stereo depth jitter and IMU drift covers that half. For columns, bases and stages, browse camera stands.

Pick hardware that survives motion

When the fix is a new camera rather than a mount, shutter type decides how vibration reaches the image. A rolling shutter reads its rows at different instants, so shake shears the frame even at a short exposure; a global shutter exposes every pixel at once:

All 2 share: Binning Supports 1 × 1, 2 × 2, 4 × 4; Decimation Supports 1 × 1, 2 × 2, 4 × 4; Lens mount C-mount; Humidity 20% RH to 95% RH (no condensation); Certification CE, RoHS, KC. Values as printed in each product's data sheet; open a part for its drawing and full specifications.

The GMAX4002 in the MV-CU020-90GC is the global-shutter choice for a moving rig; the IMX183 in the MV-CS200-10UM rolls its rows, the right sensor for a static station where its 20 MP of detail does the work. Low-texture parts add a second failure, because passive stereo has nothing to match. Projected texture fixes it — the MV-DPS200P-02 casts a structured-light pattern on the part, so matching survives plain surfaces:

Browse 3D sensors for stereo and structured-light options, or area scan cameras filtered to global shutter for the tracking camera. Standoff matters as much as the sensor: the 3D camera working distance and positioning guide keeps the field of view and magnification where the blur math assumed them.

When the usual answer is wrong

• More light fixes only what light can fix. It buys shorter exposure, which cuts blur and holds tracking, but it does nothing for a throttling box or a resetting link.
• A faster box fixes cause 1 only with thermals under control; a bigger chip in the same enclosure throttles the same way.
• A new camera is wasted while delivered fps holds 25 through the drop — the camera you own is not the bottleneck.
• A tighter tripod clamp does not replace geometry: a long arm amplifies the same base shake no matter how hard it is cranked down.

Checklist before you order parts

• Delivered fps from timestamps, with and without shake — the split that picks your branch.
• Exposure logged per frame against the drop; anything past the frame-period budget caps the rate.
• Clock and temperature telemetry over a ten-minute run; falling clocks mean thermals.
• Cable swap under the same shake, with strain relief at both ends.
• Mount: short column, wide base, bolted to structure; IMU log for the shake frequency.
• Shutter type on any replacement camera: global for a moving rig.

When the branch is known, start at the stand step of the configurator for mounting, the camera finder filtered to global shutter, or the accessories finder for stands and strain-relief hardware. Send your two measured rates through the contact page and we will tell you which branch you are on.