Stereo depth from host images: feed rectified pairs and calibration

You have stereo pairs on disk and a script that pushes them to the stereo depth node on the device, and every frame comes back with the same line: Left input image is nullptr! Skipping frame! The images are fine; the pipeline is not configured for pairs fed from the host. This guide is for integration engineers who run stereo depth from host images — offline datasets, recorded captures, custom matching — and need frames accepted on the first pass. You will set the two arguments the example requires, build a calibration file with the right intrinsics, prove the pair is rectified, and put an error bar on the depth that comes back.

Why does my stereo script report "Left input image is nullptr"?

The stereo node never received a usable left buffer: your send loop ran, but the pipeline was never configured to accept pairs from the host, so it drops every frame before matching. The fix is two arguments.

The log proves the point. It prints each send — Sent frame: Left\frame_0000.png, timestamp_ms: 0 — and then the node answers with the nullptr error, so the files left the host and died at the handshake. Two conditions make that handshake work: the pipeline must be told that the input arrives from the host as a rectified pair, and it must hold the intrinsics of the cameras that produced the images. Miss either one and the node has no valid way to match rows, so it skips the frame rather than return wrong depth.

What do --rectify and --calibration actually change?

--rectify is the flag that makes the example accept pairs fed from the host, and --calibration hands it the intrinsics file. Without the first, the pipeline rejects every frame whatever the images contain; without the second, it has no depth scale.

Rectification re-projects both views onto one common image plane, so a physical point lands on the same row number in the left and right images. That is what makes stereo search tractable: the matcher scans along a single row instead of the whole image. The calibration file supplies what the reprojection and the depth conversion consume — focal length in pixels, optical center and distortion coefficients per camera, plus the stereo extrinsic.

You can rectify offline from stored maps or let the node do it from the file; either way, the matcher only sees clean rows once that reprojection has run. Two related guides go deeper on the inputs: which intrinsics apply to rectified versus raw stereo images, and why rectification output changes when distortion correction is applied.

How do I run the example so frames are accepted?

Run stereo_depth_from_host.py with --rectify enabled and --calibration pointed at your intrinsics file: the script then accepts pairs from its Left and Right folders and returns disparity maps instead of the nullptr error.

1. Stage the pair. Put matching-name PNGs in the Left and Right folders the script walks; the console confirms each send with Sent frame: and the pair's timestamp in milliseconds.
2. Set both flags. Launch with --rectify and --calibration pointing at the intrinsics file. Both are required; the flag alone, without real intrinsics, still leaves the node without a depth scale.
3. Match resolution to calibration. Both images share one resolution, and it must be the resolution the intrinsics were computed at.
4. Check versions before debugging code. The bracketed log fields carry the device firmware; one host SDK release, v2.21.2, shipped a regression that broke host-fed stereo outright, so pin a host library and firmware pairing the example runs on.
5. Verify on the console. After the first accepted pair the nullptr lines stop and the disparity window opens; the startup keys then tune matching live — census kernel size on c, disparity search range on d, subpixel mode on 2.

Which intrinsics go into the stereo calibration file?

The file needs, per camera, the focal length in pixels (fx, fy), the optical center (cx, cy) and the distortion coefficients, plus the stereo extrinsic that carries the baseline and the relative rotation. Rectification and the depth conversion consume exactly those numbers.

Get them from a checkerboard calibration of your own rig, or from the factory calibration of a pre-aligned stereo module. Once they are good, keep the intrinsics fixed and re-solve only the extrinsics; stereo calibration with fixed intrinsics, extrinsics only shows that split. The baseline that reaches the depth formula is the calibrated one, not the nominal distance printed on a module's data sheet: half a millimetre of baseline error on a 100 mm baseline biases every depth by half a percent.

How do I check an image pair is rectified before sending it?

Pick a strong feature near the left edge of the left image and find it in the right one: its row must match within about a pixel. Repeat on two more features at mid-frame and the right edge; rows that drift point to residual rotation or a wrong optical center in the calibration.

Public stereo datasets ship rectified pairs, so they pass this test by construction; raw captures from your own cameras do not. Rectify them offline from stored calibration before the send — the procedure is in offline image rectification from stored calibration data — and confirm the result with the full check in verifying that recorded stereo frames are truly rectified. Beyond a pixel of residual row error the matcher searches a sheared line, and the disparity map frays at the edges first, so treat the row test as a gate rather than a formality.

What does one pixel of disparity error cost in depth?

Depth scales with one over disparity, so the relative depth error equals the relative disparity error: 0.5 pixels of noise on a 20-pixel measurement is 2.5 percent — about 122 mm at a depth of 5,000 mm — while the same half pixel at 100 pixels disparity costs 0.5 percent.

Worked example on a short-range rig, with baseline B = 100 mm and rectified-image focal length f = 1,000 pixels — substitute the values your calibration file prints. A feature matched at d = 20 px sits at Z = 100 × 1,000 / 20 = 5,000 mm. One at d = 100 px sits at 100 × 1,000 / 100 = 1,000 mm. Half-pixel quantization then sets the error floor: the d = 20 feature reads Z = 100,000 / 20.5 = 4,878 mm, which is 122 mm low against the true 5,000 mm and agrees with the 2.5 percent estimate; the d = 100 feature reads 100,000 / 100.5 = 995 mm, a 5 mm error.

Close the loop by aiming at a target at a known 1,000 mm: the rig must return d = 100 × 1,000 / 1,000 = 100 pixels. When you quote the result, separate scatter from bias — precision, trueness and spatial resolution explains that split. The example's spatial and temporal filter keys, w and t, trade a little latency for a steadier map; tune them after the geometry is right. If a learned matcher replaces the classical one and the map starts flickering, troubleshooting noisy DNN stereo disparity depth walks through the usual noise sources.

Why do frames still drop when the flags are right?

Work down the handshake in order: version pairing first, then resolution mismatch between pair and calibration, then pixel format, then a disparity search range narrower than the scene needs.

• Send succeeds, node still skips. Version regression or a pipeline built for device cameras rather than host pairs: the v2.21.2 host release broke host-fed stereo outright, so move to a release where the example runs.
• First pair lands, later pairs skip. Timing between sends: push the left and right of one timestamp back to back and keep timestamps increasing, so the matcher always sees a coherent pair.
• No error, empty disparity. Search range too narrow. Compute the expected disparity first — on the 100 mm, 1,000-pixel example, a part at 1,000 mm returns 100 px, so a range capped below that finds nothing. Setting that window deliberately on a live rig is covered in disparity shift and minimum range in your script.
• Left accepted, right reported null. The right file is missing or its name does not mirror the left; the log prints the exact path it tried, in Right\frame_0000.png style.

When should the camera compute depth instead of my host script?

Keep depth on the host while you develop a matcher, mine recorded pairs or need a custom algorithm, and move it onto the camera for live production, where the device returns finished depth maps at its own frame rate without pair transport over your host link.

For live inspection — height, volume, presence — the 3D sensors category is built for that hand-off: cameras that do the matching themselves and hand your code a finished depth map. The MV-DPS200P-02 is a projected structured-light 3D camera, and the MV-DLS300P-04 is a laser galvanometer stereo camera; both are depth cameras, so the host consumes maps instead of feeding pairs. Filter the full range by type and series in the camera finder.

Host-side depth stays the right answer for dataset mining, algorithm prototyping and matchers you tune per part family; device-side depth wins on frame rate, latency and a quiet host link. Size the on-device route from the configurator below, and keep the 3D camera guides at hand for calibration and playback topics, including decoding confidence-image bit flags.