You calibrate a stereo camera behind a housing window, the fit succeeds, and the script's rectified preview bends straight lines at the borders exactly the way the port demands. Then the camera's own rectified pair comes back with straight borders, and depth near the frame edge disagrees with your hand measurements. Both rectifications read the same calibration store, so the math should be identical — it is not. On embedded stereo depth nodes the distortion-correction option ships disabled, and without it the on-device rectifier skips the coefficients your script fitted. This guide shows how to confirm that is your case, the one-line fix, how much depth error the flag removes, and when rectifying on the host is the better route.
What the symptom tells you
Three images tell the story. The raw pair shows the world warped by lens and window together. Your script's rectified preview bends straight lines near the borders, because a strong distortion model must pull hard to straighten them. The device's rectified output looks nearly straight — and that is the problem: it is not applying the same pull.
Run one captured frame through both paths. Rectify it on the host with your script's maps, the same operation as live stereo rectification for calibrated camera video, and difference the result against the device's output. If the two agree in the centre and drift apart toward the edges, the device is not applying your distortion model. That is a pipeline setting, not a calibration failure.
Why the two rectifications differ
Calibration writes a camera matrix, distortion coefficients and a stereo rotation and translation into the device's calibration store (EEPROM). The script's preview applies all of it: undistort, then rectify.
The stereo depth node rectifies on the chip. With distortion correction off — the default — it rectifies with the matrix and the extrinsics but leaves your coefficients out. For a bare machine-vision lens with mild distortion the difference stays sub-pixel and nobody notices. Behind a refractive port the fitted coefficients are large, so the two rectifications separate quickly with field angle, exactly where your depth accuracy lives.
If your coefficients read as nonsense, the fault may sit in the data rather than the flag: Distortion model unknown: garbage depth camera coefficients explained shows what each stored value should look like.
What the mismatch costs in depth
A rectified-pair mismatch is a horizontal disparity error, and disparity sets depth directly:
Here f is the focal length in pixels from your calibration, B the baseline in millimetres, d the disparity in pixels, Z the distance in millimetres. Pixels cancel, so ΔZ comes out in millimetres. The error grows with the square of distance and in proportion to the pixel mismatch, so the border — where the mismatch is largest — is where depth lies hardest. Even a healthy rig deforms edge distances somewhat: Stereo depth edge accuracy: why distance deforms edges covers that baseline effect, and the 3D camera working distance and positioning guide shows how mounting distance drives the Z² term.
Rule out these causes first
Work down the list; the first cause matches the symptom exactly.
- Distortion correction off on the stereo node. Script curved, device straight, same stored calibration. Test: enable the option (next section) and repeat the overlay difference.
- Resolution mismatch. Calibration stored for one resolution while the pipeline requests another rescales the matrix and shifts everything. Test: compare the calibration entry's resolution with the node's input resolution; if they differ, follow Stereo calibration resolution mismatch: rescale or recalibrate.
- Different frames compared. The script ran on stored stills while the device streamed live frames; exposure and lighting changed the matching. Test: feed both paths the same stored pair.
- Coefficients never written. A flash that reports success can still leave factory values for a different optic in the store. Test: read the store back and compare matrix and coefficients with the script's report, value for value.
- Thermal or pressure shift. A housing squeezes optics as temperature and depth change. Test: repeat the overlay check after a full temperature soak and compare.
The fix: one option on the stereo node
Enable distortion correction when you build the pipeline, before it starts. The node is the one the SDK calls StereoDepth, and in its C++ form the call reads:
// stereo: the StereoDepth node you created from your pipeline
stereo->enableDistortionCorrection(true);
That switch makes the node apply the distortion coefficients already stored on the device. Nothing is recalibrated or rewritten: the coefficients from your housing calibration were there all along; the rectifier was ignoring them. Other language bindings expose the same option on the same node — set it wherever you construct the pipeline, not after start.
The node's rectifiedLeft and rectifiedRight outputs then bend at the borders the way your script's preview does, and depth inherits the same correction automatically.
Verify the fix on the bench
Three checks, in increasing strictness:
- Overlay difference. Rectify one stored frame on the host and difference it against the device output. The border band that diverged before should now sit within about 1 px everywhere.
- Row alignment. Pick five features along the top border and five along the bottom in the raw left image, match them in the raw right, and check row positions after rectification. A vertical residual above 1 px at the borders means the models still differ. The full procedure is in Verify recorded stereo frames are truly rectified.
- Known distance. Put a target at a measured 1,000 mm and read depth at the centre and at all four corners. The centre barely moves; the corners should fall inside your calibration's reprojection band — about 5.7 mm at 1 m for the example geometry above.
If both rectified pairs agree but the depth map still bows, that is the depth grid's rendering, not a remaining calibration error — Inverse distortion coefficients: why rectified depth looks warped explains the distinction.
Flat port or dome port: why the housing dominates
Every refractive interface bends rays that strike it off-normal. Behind a flat window, rays cross water, glass and air at increasing angles toward the frame edge, and the angular compression grows steeply with field angle — far faster than the lens's own radial distortion. A radial-tangential fit absorbs much of it, which is why your script's preview bends so hard at the borders.
A dome port removes most of the effect by geometry. When the dome's centre of curvature sits at the camera's entrance pupil, rays meet each surface close to normal incidence and pass through almost unbent, so the calibration behaves close to in-air and whatever the fit leaves is small.
Two habits follow. Calibrate with the housing fitted, in the medium you run — never calibrate in air and move the assembly underwater. And if residuals stay large behind a flat port after the flag, the pinhole model itself is the limit: refractive-stereo research ray-traces the port geometry for such cases, and only a host-side rectifier can run that model.
When to rectify on the host instead
The flag is right when your fitted model is a standard radial-tangential one the device can apply. Rectify on the host instead when either holds: you need a refractive, ray-traced model the device cannot evaluate, or you must match offline results pixel-for-pixel for metrology.
The architecture is simple. Stream the raw pair, apply your own undistort-and-rectify maps on the host — the Offline image rectification from stored calibration data workflow — and run disparity there. The cost is throughput: disparity search is the heavy step, so size the host for your frame rate and disparity range instead of getting them from the device.
Decision rule: standard model at full frame rate, use the flag. Exotic model or traceable results, rectify on the host. A mixed rig can do both — device depth for the live view, host rectification for measured frames.
Stereo hardware that ships with its model
Most of this problem class starts when someone rebuilds an optical path. If your task runs in open air — a bench, a line, a gantry — an integrated stereo or structured-light 3D camera arrives with its distortion model measured for the exact optics inside, so on-device rectification and fitted coefficients agree by construction and there is nothing to enable or forget.
- Type
- Stereo / structured light
- Type
- Stereo / structured light
When the part is better scanned in profile than matched in stereo — height, cross-section, angle — a laser profiler ships the same way: optics, emitter and calibration as one factory unit.
- Type
- Laser profiler
The decision, per case
Committing, case by case:
- Script rectified pair curved, device pair straight, same calibration. Enable distortion correction on the stereo node before start. Border depth error collapses from the Z²-scaled figure to your reprojection error — from about 229 mm to about 5.7 mm at 1 m in the worked example.
- Residuals stay large after the flag, flat port, wide angle. Rectify on the host with a refractive model, or restrict measured features to the central band where both models agree.
- You can still choose the window. Specify a dome port concentric with the entrance pupil; the calibration then behaves close to in-air and the flag absorbs the remainder.
- No housing in the requirement. Buy stereo depth as an integrated, factory-calibrated camera and skip the model-matching problem entirely.
Prevention is one habit: the flag in the template, and an overlay check after any optical change — new window, knocked lens, new resolution entry.
For a new depth station, start at the 3D sensors category or the camera finder; check working distance first, because the Z² term above is what punishes the wrong choice.
The configurator matches a camera, lens, light, light controller and cables to each other, step by step, and lists only parts that fit what you already picked.
An unusual port, extreme angles or a traceability requirement: the contact page reaches an applications engineer who builds these rigs.
Questions
Does enabling distortion correction change depth values that already work in the centre?
Barely in the centre, substantially toward the borders. Radial distortion is near zero on the optical axis, so centre disparities and depths stay as they were. At the frame edges the unapplied coefficients were shifting disparity by your measured residual — in the worked example, 12 px at 1 m meant about 229 mm of error — so border depths move by that kind of amount when the flag goes on. Re-validate against a known distance after enabling.
Do I need to recalibrate or reflash the device after enabling the option?
No, provided the calibration currently stored was fitted with the optics and window you actually run. The option changes how the stereo node interprets that stored data, not the data itself. Recalibrate only when the optical path changes — a different port, a knocked lens, a new resolution entry — and then write the fresh result before repeating the overlay and row-alignment checks.
Why is distortion so much worse inside an underwater housing?
A flat window adds a refractive interface that compresses ray angles increasingly toward the frame edge, so effective distortion grows far faster than the lens's own. A dome port with its centre of curvature at the entrance pupil meets each surface near normal incidence and cancels most of the bend, which is why underwater housings for wide optics use domes. Calibrate with the housing fitted, in the running medium.
Can I keep the device output raw and rectify on the host?
Yes. Stream the unrectified pair, apply your own undistort and rectify maps on the host, and run disparity there. You gain the freedom to use any model the device cannot evaluate — ray-traced refraction behind a flat port, for instance — at the cost of host processing for the disparity search. Choose this route when metrology traceability or an exotic model matters more than frame rate.
How small should the border mismatch be before I call the fix successful?
Work to numbers. With the flag on, the overlay difference against your script's rectification should be near zero across the frame, and matched features at the borders should sit within about 1 px of their row. The known-distance check then decides: corner depths at a measured 1,000 mm should land inside your calibration's reprojection band, a few millimetres for a typical bench geometry.