A stereo pair you calibrated yourself can hand you a point cloud with a will of its own. Matched points sit properly in the middle of the frame, but the cloud bends, bowls or curls toward the edges, and recalibrating with extra distortion coefficients changes nothing. This guide is for anyone triangulating sparse feature matches from a calibrated pair in millimetres who needs the geometry to hold still. You get the ranked causes of the bow, a capture recipe that constrains the radial coefficients, the reprojection-error numbers that mark a calibration trustworthy, the sign rule behind all-negative Z values, and the point where a factory-calibrated depth sensor is the better buy.
Why does a stereo point cloud bow toward the edges of the image?
The bow is almost always unconstrained radial distortion rather than a broken algorithm. Your calibration poses never carried the checkerboard into the image corners, so the coefficients are extrapolating there, and every triangulated point at large radius inherits that extrapolation as curvature.
The leading radial term grows with roughly the cube of radius. A coefficient set fitted on center-weighted samples looks perfect across the middle third of the frame and fails exactly where your edge matches live. After undistortion and rectification, the leftover edge error displaces matched points along the epipolar direction, and triangulate turns those sub-pixel displacements into depth shifts that grow with the square of range — the bowl shape. Both cameras contribute, because each undistortion map is applied to its own image before matching.
| Likelihood | Cause | Signature |
|---|---|---|
| Most likely | Poses never reached the corners, so radial terms are extrapolated | Smooth bowl growing with radius; identical for sparse and dense matching |
| Common | Too few pairs (under about 15), every parameter weak | Mean error above 0.5 px; focal length moves between runs |
| Common | Board not flat | Depth bias that copies the board's own bow |
| Occasional | Mechanics moved after calibration | Low error, but the same warp survives recalibration |
| Rare | Decentered or damaged lens | Asymmetric bow that will not fit below about 0.5 px |
Your dense reconstruction reproducing the warp is the most useful clue you have. Matching outliers scatter randomly; they do not organize into smooth curvature. A warp that survives both sparse and dense reconstruction is geometry, not noise.
How many checkerboard poses fix it, and where do they go?
Plan on 20-30 good image pairs, placed so the target touches all four corners and all four edges of both frames at least once and spans your nearest-to-farthest working distance.
The radial coefficients have almost no leverage from centered samples. A board in the middle of the frame constrains focal length and principal point; k1, k2 and k3 are observed where corner positions change with radius. The regions most affected by radial distortion are the edges of the image, which is precisely where calibration practice says the grid corners must land.
Capture recipe:
- Mount the pattern on glass or an aluminium-composite plate so it cannot flex.
- Capture 20-30 pairs: one pressed into each corner and each edge of both frames, one centered, two or three at every working distance, tilted up to about ±45°.
- Keep squares at 20 px or more on the sensor; below roughly 10 px corner localization degrades fast. The checkerboard target size guide shows how range shrinks the board.
- Confirm both cameras see the full board in every pair. A stereo calibrator drops pairs where one camera misses corners, and silently thinning your corner coverage is how edge poses disappear. If corners go missing, the corner detection checklist lists the usual fixes.
- Judge every capture at 100% zoom on a corner crop; a blurred corner passes in the thumbnail and lands straight in k2 and k3.
Sharp corners matter as much as placement. A square blurred across a few pixels moves its detected corners in a biased direction, and that bias feeds the radial terms. The thin-lens near/far limits set your sharp capture band (an estimate, good to roughly ±10%):
Here s is focus distance, N the f-number, and c a circle of confusion budgeted at 2 px (2 × pixel pitch). Worked example: f = 12 mm, N = f/4, pixel pitch 3.45 µm, so c = 6.9 µm = 0.0069 mm, focused at s = 600 mm. Then N·c·(s - f) = 4 × 0.0069 mm × 588 mm = 16.2 mm².
near = (600 × 144) / (144 + 16.2) = 86,400 / 160.2 ≈ 539 mm, and far = 86,400 / 127.8 ≈ 676 mm: a sharp band about 137 mm deep. At f/8 the band widens to roughly 490-775 mm, about 285 mm. No aperture of a 12 mm lens covers a 400-1200 mm pose span sharply, so split the capture into distance bands and refocus between bands.
On the capture side, resolution is what buys you 20 px squares at range. The MV-CS200-10UM is a 20 MP mono USB3 camera with a 1" sensor and a C-mount, and mono sensors avoid demosaicing around the corners your detector has to find.
- Sensor format
- 1"
- Lens mount
- C-mount
- Interface
- USB3
- Resolution
- 16-25 MP
What reprojection error says the new calibration is good?
Accept a mean reprojection error below about 0.3 px for a pair you will triangulate in millimetres. The 0.3-0.5 px band is usable for coarse geometry, and anything above 0.5 px usually indicts the pose set rather than the lens.
Sub-pixel corner detectors localize sharp, high-contrast squares to around 0.1 px, so a healthy full-frame set lands at 0.1-0.3 px mean. The mean hides geography, though: sort per-point errors by radius and look at the outermost points. A 0.25 px mean with 1 px residuals stacked at the frame border is an unfinished calibration, not a good one.
| Mean reprojection error | Verdict |
|---|---|
| Under 0.25 px | Good for millimetre triangulation |
| 0.25-0.5 px | Coarse geometry only; inspect the corner residuals |
| Above 0.5 px | Fix pose coverage before touching anything else |
Two independent captures on the same rig should repeat the focal length within about 1%. If it moves more, the parameter set is swimming — add poses, not coefficients. If your errors cluster by board square instead of by radius, the grid pitch is the likelier culprit, and the grid pitch fix guide covers that case. Treat the error as a floor, not a verdict: the extrinsic accuracy guide explains what reprojection error cannot tell you about real-world geometry.
Does adding more distortion coefficients fix the warp?
No. A third radial coefficient, tangential terms and skew cannot repair a calibration whose samples never reached the edges — the optimizer merely redistributes the same under-constrained error, and the bow survives recalibration unchanged.
Each added parameter needs independent information to be identifiable. With corner samples clustered near the center, k1, k2 and k3 trade against focal length and principal point, so the solver returns a family of solutions with similar mean error and different edge behavior. That is also why a second toolbox agrees with the first on the same images: identical results across estimators are evidence that the data, not the algorithm, is the limit.
Leave skew fixed unless you have measured a reason to free it — factory-mounted sensors and sub-pixel corners rarely identify it. Keep tangential terms in the model but expect them small, since they encode only lens-to-sensor tilt; tangential coefficients that come out comparable to your radial terms point back at the pose set.
Why are the Z values in the point cloud all negative?
Nothing is bent or broken. Triangulation expresses points in camera 1's frame with Z along its optical axis, and when you assigned the right camera as camera 1 the baseline points the other way, so every Z comes out negative.
The usual convention sets camera 1 as the left camera, with X pointing toward camera 2, so points in front of the pair get positive Z and disparities compute as left minus right. With the right camera as camera 1, camera 2 sits along minus X and the triangulation basis flips; magnitudes stay correct, only signs move. The clean fix is to rebuild the stereoParameters object with the left camera as camera 1; the quick fix is to negate X and Z in one documented place downstream. Either way, validate with a tape-measured target at a known distance before you trust the millimetres.
How do you prove the warp is calibration, not the algorithm?
Run three quick tests on images you already have: check for bowed straight lines after undistortion, reconstruct a flat target and fit a plane, and confirm your outliers scatter — curvature that survives both sparse and dense reconstruction is geometry, because matching noise never organizes itself into smooth shapes.
- Straight-line test: undistort a frame containing known straight structures — door frames, shelf edges, the board's own rows near the border — and overlay a fitted line. Residual bow above about 1-2 px at the edges means the intrinsics are still wrong.
- Plane fit: capture a flat target at a known distance, triangulate its points, fit a plane, and plot residual Z against radius. A bowl that grows smoothly toward the edges is distortion residue; scatter of both signs is matching noise.
- Rectification check: after rectification, the vertical offset of every match should sit within about ±1 px across the full image width. A systematic vertical drift at the edges means intrinsics are leaking into the extrinsics.
The plane-fit numbers come straight from stereo depth sensitivity:
dd is residual disparity error in pixels; f, B and Z are focal length, baseline and range in the same unit, and dZ comes out in that unit.
Example rig: f = 16 mm, B = 400 mm, scene at Z = 1500 mm, left-over edge residual dd = 0.3 px. Then dZ = 1500² × 0.3 / (16 × 400) = 2,250,000 × 0.3 / 6,400 ≈ 105 mm. Halve the range and the same residual gives dZ = 750² × 0.3 / 6,400 = 562,500 × 0.3 / 6,400 ≈ 26 mm. Depth error grows with the square of range, which is the bowl in one line: near-field clouds look flat while the far field curls.
Different edge symptoms have their own guides: ragged missing points at silhouettes are match sampling, covered in the depth holes guide, and mixed-depth halos along edges are the flying pixels artifact. If a rectified depth image itself looks warped rather than your sparse cloud, see the rectified-depth warping guide.
When is the warp real hardware instead of a sampling problem?
If a fresh 25-pair, corner-covering calibration with 0.2 px mean error still leaves the plane-fit bowl, stop blaming software and inspect mechanics. A loose setscrew, a focus ring that drifted, or a lens decentered in its mount reproduces the same geometry after every recalibration.
Work through the suspects in this order: mount screws and the lens ring (a ring that can rotate will rotate); focus and iris rings, locked with tape or a drop of thread binder after calibration; the bar carrying both cameras — one machined plate, not two separate tripods; thermal shift, because a rig moving from a 20 °C bench to a warm line changes focus, so re-check after warm-up; and shock from a nearby axis stopping.
Prevention costs far less than a second mystery week:
- Calibrate on the mounted rig, in its final position, at operating temperature.
- Log calibration date, pair count, mean error and the per-corner error plot next to the parameter file.
- Re-run a 60-second acceptance capture after any handling: one flat target, one plane fit; recalibrate when the edge residual doubles.
When is it smarter to buy a factory-calibrated stereo camera?
If the deliverable is depth data rather than a stereo rig you maintain, a factory-calibrated structured-light or laser-stereo sensor removes the whole capture-and-recalibrate loop: intrinsics, extrinsics and rectification ship locked, tested corner to corner.
The MV-DPS200P-02 is a projected structured-light 3D camera built around a stereo pair, and the MV-DLS300P-04 is a laser galvanometer stereo camera with a steered projection. Browsing the 3D sensors range shows the profiler, structured-light and ToF families side by side.
- Type
- Stereo / structured light
- Type
- Stereo / structured light
Decision by case:
- Research or SLAM rig you control end to end: fix the calibration with the coverage recipe above; the bow disappears and you keep full control of the geometry. If the sparse cloud feeds a visual SLAM stack and the map breaks downstream, resolving point cloud failures in visual SLAM is a useful next read.
- Production measurement at a fixed station: buy the factory-calibrated sensor and spend your time on the application, not the calibration loop.
- Close-range picking on a tight budget: a self-calibrated pair is fine, because the depth-error formula works for you at short range. With f = 8 mm, B = 200 mm, Z = 500 mm and a good 0.1 px calibration, dZ = 500² × 0.1 / (8 × 200) = 25,000 × 0.1 / 1,600 ≈ 16 mm — plenty for a gripper, useless for a gauge.
For the rig-versus-sensor decision on a moving conveyor, the 2D or 3D camera choice guide walks the system-level trade. To spec a pair yourself, start at the camera finder or the configurator below; for a second opinion on baseline and range, use the contact page. Depth geometry continues on the 3D vision topic page.
The configurator matches a camera, lens, light, light controller and cables to each other, step by step, and lists only parts that fit what you already picked.
Questions
Can I keep my existing extrinsics and recalibrate only the distortion coefficients?
Practically no. Intrinsics and extrinsics are estimated together in a stereo calibration, and an extrinsic set fitted to the old, biased intrinsics will not fit corrected ones. Re-run the full stereo calibration with the new pose set, then check the recovered baseline against a tape-measured value to confirm the whole parameter set moved coherently.
How large should the checkerboard squares be at my working distance?
Aim for squares of 20 px or more on the sensor, and never below about 10 px, where corner localization degrades quickly. Worked example: at 1.2 m with a 12 mm lens and a 3.45 µm pixel pitch, image scale is (1200/12) × 3.45 µm ≈ 345 µm/px, so 20 mm squares give about 58 px — comfortably above the limit.
Does the pattern have to be a checkerboard?
No, but the checkerboard is the most used pattern because its corners are small, high-contrast features that sub-pixel detectors localize well and that stay invariant under viewing angle. Circle grids find centers but inherit a perspective bias, and coded patterns such as ChArUco identify each corner so partial views still count. Whatever you choose, the corner-to-edge coverage rule stays the same.
How flat does the calibration board need to be?
Flat enough that its bow is small against the depth accuracy you want, because the triangulator faithfully reproduces the board's own shape as ground truth. For millimetre-class work, mount the pattern on glass or an aluminium-composite plate and keep measured bow under about 0.2 mm across the board; a paper sheet taped to cardboard can bow several millimetres and quietly becomes your error source.
How often should a stereo rig be recalibrated?
After any mechanical or thermal event: a lens cleaned or removed, a focus ring touched, a trip or impact, or a move from bench to line. Otherwise run a quick acceptance capture monthly or quarterly — one flat target, one plane fit, edge residual logged — and recalibrate when the residual roughly doubles from your acceptance value. The log tells you whether drift is gradual (thermal) or stepped (mechanical).
What does tangential distortion do to the cloud compared with the radial terms?
The radial terms create the symmetric bowl or pincushion you see at the edges, growing with radius. Tangential terms encode a small lens-to-sensor tilt, so they shear the cloud asymmetrically rather than bowing it, and in a factory-mounted camera they are usually a fraction of the radial magnitude. If your tangential coefficients come out comparable to k1, suspect the pose set rather than the lens.