Infrared vs RGB stereo input for visual SLAM: train and deploy match

You prototype the SLAM stack in a simulator that renders the stereo pair in color, then the physical robot arrives and its depth camera speaks a different language: two infrared mono streams, one color stream, and a launch file asking for a left and a right image. The question is whether the graph you tuned on rendered color can run on near-infrared greyscale. It can. Feature-based stereo SLAM never reads color; it reads two synchronized grey images from a known baseline plus the intrinsics you calibrated. What breaks is the mismatch around the modality: an emitter painting speckle into your features, contrast that vanished with the wavelength, or a twin simulated at the wrong baseline. Below you get the reasons, the depth arithmetic that sizes a baseline, the three ways to align simulation with deployment, and the decision for each case.

Match the modality end to end

Commit to one image modality and carry it through the whole chain: simulation, recordings, launch file, runtime. If the robot runs on a depth camera's two infrared sensors, the rules are:

• Grey-convert the simulated color pair, or simulate two mono cameras at the physical baseline. Both are legitimate; the front-end cannot tell them from the real streams once every image is single-channel.
• Never remap the single color topic into both stereo inputs. Zero disparity means zero metric scale.
• Force the infrared projector off before you record any run that feeds odometry, and light the scene passively.

Do these three and the same parameters run in simulation and on the robot. Skip the third and the estimate looks healthy while it drifts; the geometry is below.

Why the front-end never looks at color

A feature-based visual odometry front-end works on intensity only. It finds keypoints where the image has corners and strong gradients, describes each one as a pattern of intensity comparisons in a small patch, and matches those descriptors between the left and right images and between consecutive frames. Every step consumes a single-channel array. An RGB triplet collapses to luminance before the gradient filters run, and the descriptors compare luminance values regardless.

So a grey-converted render and a near-infrared photo of the same textured scene feed identical math. What differs is gradient quality: real 850 nm-class exposure under indoor lighting is noisier than a rendered frame, and materials that separate in color, like black fabric on dark grey carpet, can merge to the same luminance in the infrared band. That gap is a tuning problem, not a modality blocker: set your feature and matching thresholds on recordings from the real camera, as visual SLAM hyperparameter tuning: scale settings to your image walks through.

Color earns its place at a higher level, in place recognition, semantic labels and operator video. The odometry core does not want it.

One color camera is not a stereo pair

Stereo depth comes from parallax: the same physical point lands at different pixel columns in the left and right images, and the disparity converts to range through the baseline. One camera has no baseline with itself. Feed the same color topic into both inputs of a stereo node and every pixel shows exactly zero disparity, the matcher returns maximum range for everything, and the pose graph loses the only constraint that fixes metric scale.

That is why remapping the left color stream over both inputs fails, as many have found the hard way. The depth camera carries one color sensor because color is there for the output image and overlays, not for odometry; the two mono sensors exist for the pair. If your node reports a healthy image pair but no depth, the usual causes sit on the mono side; the stereo camera that works for RGB but returns no depth guide works them in order. And if a single-camera design is on the table at all, be clear about what a depth map from a single image can give you: monocular SLAM runs, but scale stays unobservable until an IMU, wheel odometry or a known object size anchors it.

What the infrared pair actually delivers

The two infrared sensors on a depth camera are a real stereo rig: two mono global-shutter sensors behind a band-pass window in the near-infrared, typically around 850 nm, at a fixed factory baseline with matched lenses. The driver rectifies both streams from factory calibration and publishes them as 8-bit greyscale, under topic names on the pattern infra1/image_rect_raw and infra2/image_rect_raw in the naming most depth-camera drivers use. Geometrically, the SLAM node cannot tell them from two grey cameras you bolted together yourself.

Two practical notes. If a preview shows grey where you expected color, you are almost certainly looking at an infrared stream; depth camera RGB shows black and white: you are on the IR stream covers the mix-up. The streams also come in fixed formats, and the stereo infrared camera streams: Y8 greyscale versus RGB modes guide explains when the plain 8-bit variant is the right pick.

Watch contrast at this wavelength. Dark textiles and matte plastics reflect weakly in the near-infrared, so a scene that looks rich in color can arrive with thin gradients. Add passive light or move the camera before you blame the algorithm.

The speckle projector is the trap

Active-stereo depth cameras fire a built-in projector that paints a dense speckle pattern over the scene. Depth mode wants exactly that: a textureless white wall gains disparity and returns a solid range map. Odometry images do not.

The geometry is unforgiving. The pattern is rigid with respect to the camera, so the same speckle dot occupies the same image position in both eyes and stays there frame after frame: zero disparity, zero apparent motion. Every keypoint parked on the projector's pattern behaves like a feature sitting at optical infinity, infinitely far away. The stereo match returns maximum range for it, the tracker reports it as standing still, and the estimator absorbs a constraint that contains nothing about your motion. Enough such features and the trajectory drifts while confidence stays high.

Most drivers expose the emitter as a setting: forced on, forced off, or automatic. Force it off for SLAM. If the depth map then starves on a genuinely textureless surface, that is the correct trade: run depth from a separate configuration with the emitter on and odometry from the clean passive pair, or put texture in the workspace.

Three ways to align training with deployment

Three routes, in rising order of setup effort:

The first route is the answer when your simulator has no greyscale camera model at all, which is still common. Whichever you take, match four things between the twin and the robot: single-channel 8-bit output, the deployed rectified resolution, the deployed baseline and focal length, and locked exposure. Then tune thresholds on real recordings rather than on renders; the stereo camera recording workflow for visual SLAM shows how to capture replayable runs.

What your baseline buys you: the depth math

Baseline and focal length decide how precisely the pair locates features in depth, and that depth feeds the pose graph's scale. The relations:

with W the rectified width in pixels, B the baseline in metres, Z the range in metres and dd the disparity error in pixels.

Worked example, with illustrative inputs: W = 1280 px, HFOV = 90 degrees, B = 0.060 m, Z = 2.0 m, dd = 0.5 px as a conservative half-pixel:

• f_px = (1280 / 2) / tan 45 degrees = 640 / 1 = 640 px
• d = 640 * 0.060 / 2.0 = 38.4 / 2.0 = 19.2 px
• Z_err = (2.0^2 * 0.5) / (640 * 0.060) = 2.0 / 38.4 = 0.052 m, about 52 mm

So at 2 m, half-pixel disparity quantization alone puts roughly 5 cm of depth noise on each feature: coarse for measurement, ample for odometry scale. The scaling matters more than the absolute value. Double the range to 4 m and the error grows fourfold: d falls to 38.4 / 4 = 9.6 px and Z_err = 8.0 / 38.4 = 0.208 m, about 21 cm. Halve the baseline and every error doubles. Commit accordingly: for indoor work inside 3 m, a 60 mm baseline is comfortable; if the robot wanders to 5-8 m, spec 90-120 mm. Identical-looking cameras can still differ in true scale, and the stereo depth scale differs between identical cameras guide shows the verification.

Building the pair from two mono cameras

If the deployment case lets you choose hardware, a purpose-built pair beats repurposing a depth camera: you set the baseline, there is no projector to neutralize, and every sensor parameter is yours.

• Two mono cameras with global shutters, so both frames integrate the same instant; rolling shutters work only with careful synchronization and slow motion.
• Hardware or shared-clock synchronization; exposure drift between the eyes reads as depth noise.
• A rigid baseline sized from the math above: 60 mm inside 3 m, 90-120 mm out to 8 m, the range where commercial depth modules also sit.
• Matching lenses, and identical band-pass filters if you illuminate in the near-infrared.
• Calibration as one stereo rig, with exactly that calibrated baseline copied into the simulator.

Synchronization is where these builds wobble; the guide on syncing two CSI cameras for stereo SLAM works the problem end to end. Shortlist sensors from the area scan cameras range, filtering for mono sensor and global shutter, or buy the pair pre-calibrated from the 3D sensors page. The camera finder filters both at once.

Checks before you train

Run this list before the first training replay:

1. Both pipelines publish single-channel 8-bit images at the same rectified resolution.
2. Baseline and focal length in the deployment calibration equal the twin's; check the calibrated values, not the nominal ones.
3. Projector forced off in the driver; passive light present and steady, with no flickering PWM-dimmed lamps.
4. Exposure locked, not automatic: auto-exposure differs between the eyes and pumps your feature positions.
5. Scene texture present: the grey images should show gradients across most of the workspace, because a bare white wall starves the front-end once the emitter is off.
6. Real-camera recordings from at least three routes through the workspace, kept for threshold tuning and regression.
7. Histogram spread on both sides: rendered grey and deployed grey should both use most of the 0-255 range.

If odometry still drops to zero with all seven satisfied, the cause is rarely the modality: ignored images and replay mismatches top the list in RGB-D SLAM odometry drops to zero: fix ignored images and replay.

Where color earns its keep

Grey runs the odometry; color can still ride along:

• Place recognition and loop closure. Appearance-based loop detection works on grey, but color descriptors add discrimination where luminance repeats. If your stack accepts a side color stream for this, publish the camera's color topic there and keep the pair for odometry.
• Semantic labels and inspection. Any classifier running on the map output wants color.
• Operator video. The overlay everyone trusts is the color image with the trajectory drawn on it.

If you run color-driven loop closure, mind the intrinsics plumbing: a wrong intrinsics type in the loop-closure module is a common silent failure, covered in monocular vSLAM error: fix the loop closure intrinsics type. A color stream beside the pair is not a third odometry camera either; geometry comes only from the synchronized pair, as the visual odometry versus depth mapping comparison lays out.

The decision, per case

Deploying on a depth camera's infrared stereo pair, the common case: grey-convert the simulated color pair if you want the change in one place, or simulate two mono cameras at the physical baseline if you want disparity statistics to match. Force the projector off, lock exposure, tune thresholds on real recordings. The same parameters then run in simulation and on the robot.

Deploying with a single color camera: do not fake a stereo pair by duplicating the topic. Either add the mono pair, which every serious depth camera has, or accept a monocular-plus-IMU design where the IMU supplies scale, and use color only where it discriminates.

Integrating your own rig: two synchronized mono global-shutter cameras, 60 mm baseline for indoor ranges to 3 m or 90-120 mm out to 8 m, no projector, calibrated as one rig with that baseline in the twin.

Shortlist hardware through the camera configurator, compare modules on the 3D sensors page, and when the images are right but the map still fails, the point-cloud failure guide for GPU visual SLAM stacks walks the remaining causes. For a build review or a volume quote, contact the applications team. Every live 3D guide, this one included, gathers under the 3D topic page.