Stereo depth without CUDA: rectify and match on embedded boards

Your stereo camera streams a clean synchronized pair, and the vendor SDK will not install: it needs CUDA, and your board offers a CPU and nothing else. You can still get metric depth. Anything that runs OpenCV can rectify the pair with the factory calibration, match it with semi-global block matching and convert disparity to XYZ. This guide gives you the pipeline, the depth-error formula with worked numbers, a frame-rate budget to check against your own board, the matcher settings to start from, and the point at which buying a camera that computes depth on board becomes the better decision. It is written for robotics engineers deploying on ARM single-board computers and low-power x86 controllers.

Can you compute stereo depth without CUDA?

Yes — any board that runs OpenCV can produce metric stereo depth with no CUDA at all: rectify a synchronized left/right pair with the factory calibration, match it with SGBM and divide, because depth is just focal length in pixels times baseline divided by disparity. That is also, at its core, all a vendor depth library does. What you must bring is a synchronized pair — hardware-triggered global-shutter frames, or a rig that transfers both sensors through one cable — plus the calibration: focal length in pixels, principal point, distortion coefficients and the sensor-to-sensor baseline. A factory stereo rig ships both; a pair you build from two area scan cameras needs one chessboard calibration session to replace them.

What does a CPU-only stereo pipeline look like?

Four stages and nothing more: capture a synchronized pair, rectify both images with the factory maps, run block matching on the rectified pair and reproject every matched pixel to XYZ with the Q matrix — no GPU stage anywhere in the chain. Each stage maps to a standard OpenCV function; the OpenCV documentation lists the signatures: initUndistortRectifyMap and remap for rectification, StereoBM or StereoSGBM for matching, stereoRectify for Q, reprojectImageTo3D for the cloud.

Two subtleties bite early. Use the intrinsics that match the frames you actually have — rectified and raw images each need their own set. And the conversion is per pixel and independent: one matched point pair is enough to compute one 3D point, so the cloud costs exactly one formula evaluation per matched pixel.

Which matcher should you run on embedded silicon?

Start with StereoSGBM: it survives weak texture far better than plain BM, and the 5- or 8-direction dynamic-programming smoothing over the cost volume is what you pay for — roughly 5-8× the compute of StereoBM at the same resolution and disparity range. The cost buys a lower rejection rate: plain block matching — the SSD cost window compared pixel by pixel — discards so many ambiguous pixels on real scenes that you re-do much of the work in post-filtering.

Reported cases of SGBM artifacts on high-resolution pairs begin once blockSize climbs above 23, one more reason to keep the block small.

How fast does depth error grow with distance?

Depth error grows with the square of distance: at a 120 mm baseline and f = 700 px, one disparity pixel is worth ±3 mm at 0.5 m, ±11.9 mm at 1 m, ±107 mm at 3 m and ±1.19 m at 10 m, so CPU stereo is a near-field sensor, not a 10 m one.

The lever is f·B: doubling the baseline halves the error at every distance — how a wide baseline buys long-range stereo accuracy — at the price of close-range overlap and a doubled disparity range, which doubles matching time. The decision this produces: inside about 3 m, CPU SGBM returns grasping-grade numbers; past 5 m, treat it as an obstacle detector, not a measuring instrument.

What frame rate can an embedded board sustain?

The matcher, not the camera, sets your frame rate: budget roughly 15-30 fps for SGBM at 640×480 with 64 disparities on a quad-core x86 vision controller, and 3-6 fps on a quad-core ARM board. Halving resolution buys about 4×, halving the disparity range another 2×, and cropping the ROI pays the area ratio.

These are typical measured ranges for this class of hardware; the scaling law behind them is what you apply to your own numbers. Work grows with width × height × disparity count × direction count, so dropping 640 × 480 × 64 to 320 × 240 × 32 leaves (320 × 240 × 32) / (640 × 480 × 64) = 1/8 of the work. Capture is not the constraint: dual global-shutter rigs stream 1440 × 320 at 193 fps from an ARM board, while the matcher on the same board sits in single digits at VGA.

Two controller classes cover both budgets:

At these rates the smaller host is enough; move up a class when you add cameras, resolution or disparity depth. Size the host by its LAN port count, PoE support and memory, matched to the cameras you actually have.

How do you build and verify it step by step?

Seven steps take you from a raw synchronized pair to a filtered XYZ cloud, and each carries a check you can run in minutes.

1. Synchronize capture. Fire both shutters from one hardware trigger, or use a rig that transfers both frames together. Check: swing a fast object through the scene — no ghosting between views.
2. Load the calibration. Factory intrinsics and baseline come with a stereo rig; for a self-built pair, one chessboard session with cv::stereoCalibrate. A negative baseline means your cameras are flipped.
3. Rectify. Build the maps once with initUndistortRectifyMap, then remap every frame. Check: a feature at mid-depth sits on the same row in both images, within about a pixel.
4. Set the disparity range. numDisparities = f·B / Z_min, rounded up to a multiple of 16. On the example rig, depth down to 0.5 m needs 700 × 120 / 500 = 168 px, so set 176.
5. Match. StereoSGBM with the starting values above; keep the reject mask, not just the disparity image.
6. Reproject. Apply Q from stereoRectify through reprojectImageTo3D, or compute Z = f·B/d per pixel; set the depth window to your working range so far-field noise is clipped before it reaches the robot.
7. Verify against truth. Tape-measure a matte target at 1.0 m and 3.0 m; on the example rig the errors should land near ±12 mm and ±107 mm and grow as Z². Static-scene flicker points to matcher ambiguity — this depth-jitter troubleshooting guide walks the usual causes.

When should you buy depth instead of computing it?

When you need more than single-digit fps on an ARM board, dense depth past 3-5 m, or depth on textureless parts, stop computing and buy a camera that matches on board and streams finished depth. An onboard-depth stereo camera offloads the expensive stage, so the host only reads finished points — even a modest controller keeps up. A projected structured-light camera such as the MV-DPS200P-02 casts its own pattern, so matching never starves on featureless surfaces; a laser galvanometer stereo camera such as the MV-DLS300P-04 covers scenes a projector cannot safely reach; and the MV-DT01WMU returns time-of-flight depth directly. If your roadmap later adds GPU-accelerated ROS nodes, the failure to expect is accelerator negotiation — how that negotiation failure presents.

The decision in one line: run the CPU pipeline above while your range stays inside 3 m, your parts carry enough texture and your fps target is single-digit on ARM or 15-30 on x86; move to onboard depth the moment any one of those breaks. To size the whole stack, start the system configurator or open the camera finder.