The first RGBD pipeline you write usually starts ambitious: open the color camera and the stereo mono pair at the same high sensor mode and display both streams. On many stereo depth cameras that start-up dies with a fatal sensor-driver error naming the mono sensor's driver. Nothing is broken and no unit is defective. The color sensor and the mono pair are different imagers with different native mode lists, and the high mode you asked for exists on one side only. This guide gives the rule that fixes it, the scale and bandwidth arithmetic to size the streams before deployment, and the alignment step that puts a depth value on every color pixel: run each sensor at its native mode, downscale the color output in the ISP, and align depth to color.
Why does forcing color and depth to one sensor mode fail?
Because the two sides are different imagers with different mode tables: a typical mono stereo pair tops out at a 1280×800 sensor mode, while the color sensor runs 1920×1080 or higher, so a 1920×1200-class request exists on one side only and the driver aborts at start-up.
A pipeline graph opens every sensor before the first frame arrives. The color side opens because its mode table holds a 1920×1200-class entry; the mono driver receives a mode its table does not contain, the firmware logs a critical line naming the mono sensor driver, and the device tears the pipeline down. Run the same pipeline at a low common mode and both sides open, which is why the failure looks random rather than systematic.
Tables, not preferences, decide this. A mono stereo pair is usually built around small, fast global-shutter imagers whose native list runs from 720p-class (1280×720) up to XGA-class (1280×800). A color sensor is a higher-resolution imager whose useful modes start at 1080p-class (1920×1080) and climb through 4K-class to full-sensor output. Where the two lists overlap at all, it is at the legacy bottom end — and that overlap is the next trap.
Can I just pick 720p for both cameras instead?
Usually no. Modern color sensors have dropped legacy 720p and 800p modes from their tables; the driver prints an unsupported-resolution warning, names the modes it does support, and silently starts a 1080p-class native mode instead, so the two streams still do not match.
The tell is a warning in the start-up log rather than a crash. The driver reports the color sensor it detected, lists the modes that sensor actually supports and states the native mode it fell back to. The pipeline runs, your color frames arrive at a size you never requested, and every downstream step that assumed a matched size — alignment, cropping, overlay — is quietly wrong.
Stop shopping the mode lists and start scaling instead. A sensor mode table is fixed hardware: if no mode contains a setting, no phrasing of the request creates one. The pattern is the same as a pixel-format control that stays read-only until the scan type is set first — 3D camera pixel format locked read-only: set the scan type first covers that variant of the trap.
How does ISP scaling give both streams the same resolution?
Each sensor runs its own native mode and the color camera's ISP applies a downscale to its processed output, so the color stream leaves the camera at the mono pair's resolution; the stereo node computes depth at mono resolution and the streams now match exactly.
The scale sits inside the color camera's image signal pipeline, after demosaic and color processing and before the output. Because it resamples the processed frame rather than binning the sensor, the color camera still runs its full native mode and detail from every native pixel is averaged into the smaller output. The mono pair keeps its native mode too, which is what depth quality wants: depth comes from mono pixels.
Two settings do the work on a typical stereo depth node. The color camera node carries an ISP scaling property with separate horizontal and vertical ratios; the depth node carries an alignment property that names the camera frame the depth output should be reprojected onto. The working order is: native mode for the mono pair, native mode for color, scale set so the color output equals the mono size, depth alignment pointed at the color output. After start, read back every stream's actual size rather than trusting the request.
Keep the two mono cameras identical — same mode, same trigger — or disparity falls apart before scaling matters. If either side runs auto exposure, set it per camera the way setting auto exposure per stereo camera in a depth rig describes. And if a later resolution edit changes the color output without updating the scale, the aligned frame tears and offsets; the depth-to-color alignment fix for a stream resolution mismatch walks that exact symptom.
What scale factor and output size should I set?
Compute the factor from the mode table: divide the color sensor's native height by the mono height. 1920×1200 to 1280×800 is a factor of 1.5 on both axes, and 1920×1080 to 1280×720 is also 1.5, so both classic pairings keep the aspect ratio intact.
| Color native mode | Mono native mode | Divisor d | Aspect kept |
|---|---|---|---|
| 1920×1200 | 1280×800 | 1.5 | 16:10 |
| 1920×1080 | 1280×720 | 1.5 | 16:9 |
| 3840×2160 | 1280×720 | 3.0 | 16:9 |
| 3840×2160 | 1920×1080 | 2.0 | 16:9 |
All four classic pairings divide exactly on both axes, which is why they make dependable defaults. Divide by the same d on both axes or the aspect ratio drifts and the calibration no longer fits the scaled frame.
Work from the measurement backwards. Disparity precision is a fraction of a pixel and range error grows with the square of the distance in a stereo pair, so the mono resolution sets the depth resolution you can defend at range. Run the mono pair at the highest native mode your frame rate and depth compute allow, then scale color down to meet it — never shrink the mono side to flatter the color sensor. When RGB texture decides more than dense depth, invert the setup: keep color near native and align depth onto color, accepting resampled depth.
The scaled grid also tells you what one pixel means on the part. A 200 mm wide field at 1280 pixels across is 200 / 1280 = 0.156 mm per pixel, and the ISP-scaled color stream lands on that same grid — every color pixel and every depth sample then refer to the same 0.156 mm cell on the surface.
How much bandwidth do matched color and depth streams need?
About 154 MB/s at 1280×800 and 30 fps: 16-bit depth costs 61.4 MB/s and 3-bytes-per-pixel color adds 92.2 MB/s. One USB3 link carries that with headroom, while a 100 Mbps Ethernet link misses it by about twelve times.
Worked example: 1280×800 matched pair at 30 fps
The mono pair runs at its native 1280×800 and never crosses to the host; the on-board stereo engine consumes it. Only depth and color leave the device, and in the matched design both are 1280×800 frames.
- Frame area: 1280 × 800 = 1,024,000 pixels.
- Depth, 16-bit: 1,024,000 px × 2 B = 2,048,000 B per frame; × 30 fps = 61,440,000 B/s = 61.44 MB/s.
- Color scaled to 1280×800, RGB at 3 B/px: 1,024,000 px × 3 B = 3,072,000 B per frame; × 30 fps = 92,160,000 B/s = 92.16 MB/s.
- Host link total: 61.44 + 92.16 = 153.60 MB/s, or 153.60 × 8 = 1,228.8 Mbit/s ≈ 1.23 Gbit/s.
Skip the downscale and color costs more than the depth it serves. At native 1920×1080: 1920 × 1080 = 2,073,600 px; × 3 B = 6,220,800 B per frame; × 30 fps = 186,624,000 B/s = 186.62 MB/s. The link total climbs to 186.62 + 61.44 = 248.06 MB/s ≈ 1.98 Gbit/s.
- USB3 link, roughly 350-400 MB/s sustained: carries both variants with room to spare.
- GigE link, about 115 MB/s usable: carries neither total at 30 fps. Keep the scaled color and drop to 15 fps: 153.60 / 2 = 76.8 MB/s, leaving roughly a third of the link free.
- Pull 8-bit disparity instead of 16-bit depth and the depth term halves: 61.44 / 2 = 30.72 MB/s.
How do depth-to-color alignment and the scaled intrinsics work?
Alignment reprojects depth onto the color camera's pixel grid with the factory extrinsics and color intrinsics, so the aligned depth frame matches the color frame size. After an ISP downscale by factor d, divide fx, fy, cx and cy by d.
Alignment is what turns two same-size streams into registered RGBD. For each pixel of the color frame, the warp looks up the depth the mono geometry sees there, using the extrinsics from the factory calibration. The aligned depth frame comes out with the color frame's dimensions; where the mono field of view does not reach — typically the border strips — aligned pixels carry an invalid value instead of a distance.
Matching sizes is not registration: two streams at the same resolution stay unaligned until the warp runs, which depth and color at the same resolution are not aligned demonstrates. Resampling adds no depth detail either — why aligned depth to color keeps the depth resolution explains what survives the reprojection.
The device's alignment stage applies that intrinsic scaling internally when it produces aligned depth, so deprojecting the aligned stream is a one-step job. If you deproject the ISP-scaled color frame yourself from raw calibration, divide first. Dividing an already-scaled set a second time is the classic double-scale bug.
Two details catch teams later. The aligned stream's memory layout and applicable intrinsics differ from the raw mono stream — aligned depth streams: data layout and which intrinsics apply covers the layout. When you build the point cloud, pick the deprojection camera deliberately; aligned depth point cloud intrinsics: which camera to deproject with works the choice through. And if registration drifts after a lens swap or a mechanical knock, recalibrate rather than rescale — stereo camera RGB calibration: fixing depth-to-color alignment errors covers that repair.
When should you skip the matching problem entirely?
Skip it when the camera pairs the sensors for you: an integrated stereo or structured-light 3D camera computes depth from its own matched sensors on board, and a laser profile sensor emits intensity and height per line, so the host never has to make two different sensors agree.
Integrated depth cameras exist because making two different sensors agree is engineering. A stereo or structured-light 3D camera pairs its sensors at the factory, computes depth on board and hands the host matched outputs, so the pipeline work above disappears into firmware you never write. For dimensioning, volume and presence checks that is the shorter path: the MV-DPS200P-02 is a projected structured-light 3D camera and the MV-DLS300P-04 is a laser galvanometer stereo camera.
- Type
- Stereo / structured light
- Type
- Stereo / structured light
Dense height profiles on moving material are the same story: a laser profiler emits its own matched pair, intensity and height for every scan line, so there is no second sensor to reconcile. The MV-DP4180-03P is a 3D laser profile sensor of that type.
- Type
- Laser profiler
If the task genuinely needs a custom baseline, custom optics or a color sensor of your own choosing, discrete area scan cameras buy that freedom at the price of doing everything in this guide yourself: native modes, ISP scaling, alignment, and a bandwidth budget on one trigger. The integrated route lives in the 3D sensors category.
Laser profilers and stereo 3D cameras
170 models · from $699.00What should you check before you commit the pipeline?
Six checks: read back actual stream sizes, confirm an exact size match on both axes, confirm one scale factor per axis, budget the link with 20% headroom, deproject a known artifact, and read the depth units from stream metadata.
- Read back, do not trust. Query each stream's actual frame size after start; any size you did not set means the driver substituted a native default.
- Match on both axes. Color output width and height must equal the mono pair's exactly; a one-line mismatch shows up as a sheared overlay after alignment.
- One factor per axis. Scale horizontal and vertical by the same d. Image a square part in both streams and confirm it stays square.
- Budget the link. Apply width × height × bytes per pixel × fps to every stream that crosses to the host and keep 20% headroom below the measured link throughput.
- Deproject something you know. A shim or gauge of known size settles the intrinsics: lateral size reading exactly 1.5 times large means the intrinsics were scaled twice.
- Read the depth units once. Depth metadata commonly carries 1 mm per unit, but the metadata field decides — store its scale next to your calibration.
The decision in one line: two different sensors never share a mode at the resolution you want, so stop requesting and start scaling — native mode on every sensor, divisor d = color native height ÷ mono height (1.5 in the classic pairing), depth aligned to color, and a link budget of width × height × bytes × fps per stream. Where a matched stream matters more than a custom rig, an integrated 3D camera ships it out of the box. For the surrounding pipeline — triggering, buffering, exposure — the 3D topic page collects the rest of the sequence, and the camera finder is where the hardware choice starts.
The configurator matches a camera, lens, light, light controller and cables to each other, step by step, and lists only parts that fit what you already picked.
Questions
Does ISP downscaling make the color stream soft compared with the depth map?
A downscale is a resample of the processed frame, so high-frequency detail at the native size is averaged into the smaller output; at a factor of 1.5 the practical loss is small and usually below what your inspection needs. Judge sharpness on the scaled stream you will deploy, with your final exposure and focus, not on a native-resolution preview.
Should I align depth to color, or color to depth?
Align depth to color when every RGB pixel needs a depth value — overlays, RGBD segmentation, per-pixel measurement against texture. Align color to depth when depth fidelity leads, for example textured point clouds for picking: the depth map stays at mono resolution and the color image is resampled to meet it. Pick the stream whose resolution you cannot afford to lose.
Do I need a new calibration after I change the color scale factor?
No. Scaling is a pure image operation: the extrinsics between the sensors do not move, and only the color intrinsics divide by the scale factor — fx, fy, cx and cy. If your pipeline's alignment stage scales intrinsics internally, feed it nothing extra. A point cloud that stretches by exactly the scale factor is the signature of the division applied twice.
Can the color stream run at a lower frame rate than the stereo pair?
Yes, and it is a useful bandwidth saver. The two mono cameras must run at one rate on one trigger, but the color camera can run slower, for example 15 fps against a 30 fps depth output. Alignment then pairs each depth frame with the nearest color frame, so allow up to one color period of extra latency between the streams.


