A stereo camera rolled sideways to fit a housing, or a pipeline that rotates the left and right streams to match that mounting, will report depth that reads correctly at the frame center and falls apart toward the edges. This page is for anyone integrating a stereo depth camera, or a pair of area scan cameras, into hardware that forces an unusual orientation. You will get the causes ranked by how often they turn up, the pipeline order that fixes the reading, and the box-remapping arithmetic to copy into your code.
Symptom: depth reads right at the center and wrong off-center
The object in front of the lens sits near the middle of the frame, and the distance reads sane. Slide the same object toward either side, and the reported depth drifts, scatters, or collapses onto the background distance. The preview looks fine and detection boxes land on the object, but the depth attached to them is wrong. The trouble started when the camera was mounted at 90 degrees and a matching rotation was added to the image streams.
Other depth faults look similar but root elsewhere: stereo disparity troubleshooting after rectification covers noisy output once the rectification maps change, and an invalid band at the frame edges has a different cause again. Rotation damage is distinct: it corrupts readings wherever the true disparity is not zero.
Causes, most likely first
Work down the list; the first cause accounts for most rotated-camera cases.
- Rotation applied before the stereo stage. After rectification a physical point sits on the same row of the left and right frames, and disparity is the horizontal offset between them:
Z is depth in meters, f the focal length in pixels, B the baseline in meters, d the disparity in pixels. In most depth cameras the matching runs in a fixed-function block on the silicon, and that block compares windows along rows only. Rotate both frames by 90 degrees and every correspondence moves from its row to its column. The matcher still searches along rows, locks onto the best wrong window, and d becomes fiction. Test: bypass the rotation step and feed raw frames to the stereo stage. If depth becomes consistent across the whole frame, this was the fault.
-
Detection boxes or 3D coordinates taken in the wrong frame. The detector runs on the rotated preview, but the depth query uses the displayed pixel indices; or a spatial node is fed rotated depth with a camera model built for the unrotated grid. Mapping image pixels to 3D points only works when both steps share one coordinate frame, and the imager-center origin of a depth camera is defined on the unrotated grid. Test: aim a target at a measured distance near the frame edge and read depth at the remapped indices, then at the displayed indices. The second reading will be off.
-
Stale calibration after a mechanical change. A factory stereo module is calibrated as one rigid unit, so pure remounting does not touch it. Two separate cameras on a bracket are different: any change in baseline or roll invalidates the extrinsics, and uncalibrated rectification gives far objects large disparity. Test: hold a checkerboard target at three distances and three lateral positions. Error that grows toward the frame edges means recalibrate.
-
Disparity search range too small for near objects. Near, off-center points carry the largest disparity; when the searchable range tops out below the true value they read as far away or invalid. Stereo disparity maps that come out mostly black point at baseline and matching range, and this fault usually announces itself more loudly than rotation damage.
Learning-based stereo stages add failure modes of their own; troubleshooting noisy learning-based stereo disparity is a separate exercise from a broken pipeline order.
The fix: match first, rotate last
The pipeline order that survives any mounting:
- Capture raw, unrotated frames from both imagers.
- Rectify with the stored calibration.
- Run the stereo stage on the rectified pair; the depth map is then valid across the whole frame.
- Compute spatial or 3D coordinates from the unrotated depth. If the on-device spatial node cannot be pointed at an unrotated map, do this step on the host.
- Remap every detection box into depth-map indices before reading a value (arithmetic in the worked example below).
- Rotate finished outputs, the depth map for the operator screen and the annotated video, and nothing earlier.
A rigid stereo rig rolled sideways in its bracket is not a problem for this pipeline. Both frames roll together, the calibration still describes their relative geometry, and the depth map arrives simply rotated with every value intact. What an orientation sensor reports after such a roll is a separate correction at the transform step, as with an IMU yaw offset after robot mounting.
For a 90 degree clockwise display rotation of a depth map with height H rows, a displayed pixel (x_D, y_D) came from:
Apply that to both box corners, then read the median depth inside. One padding trap: some on-device rotation blocks round the rotated width up to a multiple of 16 pixels. Do the index arithmetic against the true depth-map size, not the padded one.
Worked example: reading depth for a detection box on a camera rolled 90 degrees
Inputs (the sizes are illustrative, substitute your own; the arithmetic is identical at 300 × 300 or 4K):
- Depth map: 1280 × 800 px (width × height), straight out of the stereo stage.
- Display and detection stream: the same frames rotated 90 degrees clockwise, so 800 × 1280 px.
- Detection box in the displayed stream: x_D = 300, y_D = 520, w = 140, h = 180 px.
- Stereo geometry, illustrative: f = 960 px, B = 75 mm.
- Median disparity read from the true region: d = 36 px.
Rule in words: undo the rotation on the box corners, then compute depth from the median disparity with Z = f·B/d.
Arithmetic. First corner (300, 520): x = 520, y = 800 - 1 - 300 = 499. Second corner (440, 700): x = 700, y = 800 - 1 - 440 = 359. The box occupies columns 520 to 700 and rows 359 to 499 of the depth map, 180 px wide and 140 px tall, width and height swapped exactly as a 90 degree rotation demands. Reading the displayed indices (300 to 440, 520 to 700) instead lands at least 80 px away from the true region, on whatever else is in the scene.
Depth: Z = f·B/d = 960 px × 0.075 m / 36 px = 2.0 m. Sensitivity: one pixel of disparity error moves the result by Z²·Δd/(f·B) = 2² × 1 / 72 m ≈ 5.6 cm, or 2.8% of the reading. A rotation-fed matcher that returns 3 px of accidental correlation reports 72/3 = 24 m.
Decision: with the remap in place the reading tracks the object to within about one disparity pixel, roughly ±6 cm at 2 m for this module, and no threshold tuning is needed. If off-center objects still read far after reordering, stop adjusting thresholds and check the disparity search range instead.
When the mounting keeps winning: triangulate light instead of images
If the depth pipeline is closed inside a camera and always rotates before matching, no amount of host-side remapping repairs the disparity itself. The clean way out is depth hardware with no passive left-and-right matching stage at all. Sensors that triangulate projected light, a speckle or stripe pattern from a projector or a scanned laser line, compute each pixel depth from a single imager view of light whose geometry they know. Rolling such a unit sideways cannot break a row-matching stage because there is none; the mounting becomes a pure output transform, the same rotate-after step as above.
Both families sit in the 3D sensors range: structured-light cameras that project their own pattern, and laser profilers and galvanometer-scanned stereo units that sweep a line across the scene.
- Type
- Stereo / structured light
- Type
- Stereo / structured light
For plain 2D inspection beside the depth task, the same logic applies to area scan cameras: orient them however the housing demands and rotate the result, not the pixels before processing. The camera finder filters both ranges by interface, resolution and sensor type.
Checks before you redeploy
- Trace the pipeline. The rotation step may only appear after the depth map and after any pixel-to-3D calculation. A rotation before the stereo stage is the fault, whatever else you also fix.
- Edge-of-frame target. Place a target at a measured distance near the left and right edges. A correct pipeline reads within one disparity pixel of truth, about 6 cm at 2 m for the f = 960 px, 75 mm module above; a rotation-damaged one misses by meters.
- Box remap. Read depth at the remapped indices and at the displayed indices for the same detection. The remapped value must match a tape-measured distance; if both are wrong, the problem sits upstream of the remap.
- Padding. Compare the rotated display size against the depth-map size and account for any multiple-of-16 rounding before remapping.
- Mechanics. Secure the mount so roll cannot creep, and record the roll direction so the remap sign survives maintenance.
If you are choosing hardware for a right-angle mounting rather than rescuing an existing rig, start at the camera configurator. When the depth pipeline is closed and the rotation happens inside it, contact the applications team before committing the mounting design. The 3D vision topic page gathers the rest of the depth-pipeline fixes.
Questions
Can I physically mount a stereo camera at 90 degrees?
Yes. Rolling the whole rigid rig leaves the distance between the two imagers and their relative angle untouched, so factory calibration still lines the epipolar lines up along the sensor rows and matching works. The finished depth map simply arrives rotated relative to your workspace, which you undo with an output rotation or a coordinate transform. What breaks depth is rotating the image streams in software before the matching stage, not the mounting itself.
Why is depth correct at the image center but wrong at the edges?
After an input rotation the true correspondence for every point leaves its row, so the row-by-row matcher returns the best wrong window. How wrong the reading looks depends on local texture and on the true disparity: flat, low-texture areas often yield a small disparity that reads as a plausible distant surface, while regions with real disparity return scattered nonsense. Plausible at the center, wrong at the edges is the signature of geometry broken upstream of matching, not a calibration or exposure fault.
Where does the rotation step belong in the pipeline?
After the stereo stage and after any node that converts pixels to 3D coordinates. Capture, rectification, disparity matching and spatial calculations must all see the frames your calibration describes, unrotated. Rotate the finished depth image and the annotated video for operators, and remap detection boxes from the displayed orientation into depth-map indices before you read a depth value.
Does rotating the depth map change the depth values?
No. A rotation only rearranges pixels; each pixel keeps its value, so a correctly matched depth map that is rotated afterwards holds the same depths, re-ordered. Values change only when rotation happens before matching, or before the pixel-to-3D calculation, because both steps depend on the row geometry and the camera model of the unrotated frames.
Why must the rotated output width be a multiple of 16 pixels?
Hardware-accelerated image manipulation blocks move memory in aligned chunks, and some SDKs document that a rotated output width must be a multiple of 16 pixels, padding it when it is not. Do your index arithmetic against the true depth-image size, and allow for any padding when you remap boxes, or every read lands a few pixels off, enough to miss a small part at close range.

