RGBD SLAM with a depth camera: rtabmap setup and good depth data

An RGBD SLAM stack fails on camera configuration far more often than on package choice. The mapper needs one registered depth-plus-RGB pair at a steady rate, with exposure settings that hold depth range while the robot moves. Every fault you chase later — drift on turns, leaning walls, a map that stops growing — traces back to one of those inputs. This guide is for engineers bringing a depth camera onto a mobile robot or inspection platform. It follows the setup order that works: pick a maintained SLAM package, set depth-to-RGB registration, lock exposure for range, size the data link, and bench-check the stream before the first drive. The numbers assume a typical indoor setup at 30 Hz.

Pick the SLAM package, then fix the camera feed

Use rtabmap — in ROS terms rtabmap_ros, which installs from the package manager and is under active development. The older RGB-D SLAM projects still floating around on code hosting sites are archived and pinned to ROS releases nobody ships, so start from the maintained one. With that settled, the package is the easy half: rtabmap consumes whatever registered depth-plus-RGB pair your driver publishes together with camera_info intrinsics. Its appearance-based loop closure with bounded working memory is what keeps long robot sessions from outgrowing RAM.

Set the stack up in an order where each step hides the faults of the next. Prove the driver builds and streams — the ROS camera package availability check shows how to prove the stream — then get the camera on the network (subnet and link fix for a depth camera on a Linux controller). Only then configure registration, exposure and mapping.

Registration: put depth and RGB in one frame

RGB-D odometry finds features in the RGB image and reads their distance from the depth map. A fixed offset between the two images bends every 3D point the same way: walls lean, floors curve, loop closure degrades. Enable the driver's depth-to-RGB registration once and leave it on; the mapper should never see unregistered depth.

Check it on a flat target at your working distance. Registered edges should sit within a few pixels; that tolerance is what keeps 3D points trustworthy. A constant residual shift is a rectification problem — the depth-to-RGB registration offset fix walks the pixel offset — while bands at object edges in close-range work are occlusion physics, not misconfiguration (occlusion bands in depth-to-color alignment).

Exposure: hold range without smearing features

Depth and RGB pull exposure in opposite directions. Range needs signal: a ToF camera trades integration time against frame rate, and too little signal shows up as invalid pixels at distance — the ToF exposure and frame-rate trade-off gives the mechanism. RGB needs sharpness, and on a moving robot exposure converts directly into smear.

Lock exposure once the lighting is fixed. Auto modes hunt under changing light and reset gain mid-map, which the tracker reads as scene motion; the same class of surprise covers emitter idle behavior such as an IR flood light that switches itself off. In genuinely dark zones, add lighting or average frames at a fixed rate (averaging multiple frames on a fixed-frame-rate camera).

Sizing the link for a 640 × 480 RGB-D pair at 30 Hz

Inputs: 640 × 480 depth at 16 bits (2 bytes per pixel; depth in millimetres saturates at 2^16 − 1 = 65,535 mm, which is why 16-bit is the standard indoor depth format), 640 × 480 RGB at 3 bytes per pixel, a 30 Hz target, and usable link throughput of about 940 Mbps (roughly 117 MB/s) on Gigabit Ethernet and about 400 MB/s on USB 3.0. The throughput figures are typical usable values, below line rate; drivers that compress depth only add margin, so size uncompressed.

The rule in words: bytes per frame = width × height × bytes per pixel, summed over the two streams; link load = bytes per frame × frame rate × 8 bits per byte.

Arithmetic:

• Depth: 640 × 480 px × 2 B = 614,400 B
• RGB: 640 × 480 px × 3 B = 921,600 B
• Pair: 614,400 + 921,600 = 1,536,000 B ≈ 1.54 MB per frame
• Rate: 1,536,000 B × 30 Hz = 46,080,000 B/s ≈ 46.1 MB/s, and 46.08 MB/s × 8 = 368.6 Mbps

Against a usable 940 Mbps on Gigabit Ethernet, 368.6 / 940 ≈ 39% of the link, so 940 / 368.6 ≈ 2.5× headroom for the rest of the robot's traffic on the same wire.

The same pair at 1280 × 720 changes the answer: 1280 × 720 px × 5 B = 4,608,000 B ≈ 4.61 MB per frame, and at the same rate that is 138.2 MB/s ≈ 1,106 Mbps, which no longer fits. The exits, in the order they usually get picked: depth-only at the target rate (1280 × 720 × 2 B = 1,843,200 B; × 30 Hz = 55.3 MB/s ≈ 442 Mbps, fits), the pair at 20 Hz (138.2 MB/s × 20 / 30 = 92.2 MB/s ≈ 737 Mbps, fits), or USB 3.0, where 138 MB/s sits well under 400.

Decision: a 640 × 480 depth-plus-RGB pair rides one Gigabit Ethernet link at 39% utilization; a 720p pair forces USB 3.0, a cut to about 20 Hz, or depth-only streaming. A 100 Mbps link fails outright — about 94 usable Mbps against 369 needed.

Which depth mechanism to mount

For indoor fleets mapping corridors and work cells, projected structured light is the default — the stereo, ToF or structured-light selection guide ranks all three by range, accuracy and budget, and the 3D sensors category with its camera finder narrows the field to what is orderable today.

For close-quarters indoor mapping, a projected structured-light camera is the pick:

Routes that cross dock doors, glazing or mixed light favor time of flight:

Logistics robots that also estimate parcel volume can pair mapping with measurement in one sensor: a speckle structured-light 3D camera built for volume measurement shares the same registered-depth geometry a warehouse map needs. Browse the 3D sensors category for the logistics series.

Mounting and bench checks before the first drive

Mount the camera rigidly to the chassis; a mast that rings at driving frequency adds the vibration blur familiar from fixed-focus lenses on mobile robots. Set height and pitch from the mapping geometry rather than convenience — the 3D camera working distance and positioning guide covers the trade between near-ground detail and far-wall coverage.

Run five checks at the bench, in this order:

• Flat target at working distance: registration offset within a few pixels, no constant shift.
• Chessboard sweep: focus holds across the depth range — diagnosing an out-of-focus stereo camera with a chessboard applies to any depth camera.
• Far wall at the depth limit: invalid-pixel share and confidence values behave (masking invalid pixels with the IR image).
• Both streams hold the target rate sustained, then again with the mapping process running.
• If you remount or re-pitch afterwards, recalibrate — depth range and accuracy move with mounting (when to recalibrate).

A bench stand holds the camera at the planned height and pitch while you run those checks:

The camera stands range and its accessories finder cover taller columns, glass trays and X/Y stages for bigger sensors.

When the standard answer is wrong

• Long, feature-poor aisles starve RGB-D odometry of visual features. Fuse wheel odometry and IMU so the filter carries rotation between landmarks, and budget for the jitter-and-drift coupling described in stereo depth jitter and IMU drift troubleshooting.
• Glass, mirrors and matte black plastic return no depth. Mask by confidence rather than raising exposure; raising exposure costs range elsewhere and fixes nothing here.
• Bright doorways and windows collapse a projected pattern's range. Re-route the sensor, shade the glass, or move that stretch of map to a ToF camera.
• If onboard compute cannot hold the full frame at the target rate, cut the processed area instead of the rate: camera-side ROI and z-layer limits do exactly that (depth camera ROI and z-layer scan configuration).
• Fast yaw shears a rolling-shutter frame mid-rotation. Cap yaw rate during mapping runs rather than debugging the tracker afterwards.

Pre-drive checklist

• Driver publishes registered depth, RGB and camera_info at the target rate, sustained with the mapping process running.
• Link sized uncompressed (369 Mbps for the VGA pair) on Gigabit Ethernet or USB 3.0, kept clear of other traffic during mapping.
• Registration offset within a few pixels on a flat target; intrinsics match the registered stream.
• Exposure locked; smear at top speed under about 5 mm.
• Focus and invalid-pixel share checked at both ends of the working range.
• Wheel odometry and IMU fused when the route runs through long featureless aisles.
• Yaw rate capped on the drive profile for mapping runs.

Next step: start the camera configurator from your working distance and interface, or request a quote with the route profile attached. The 3D topic page collects the registration, exposure and calibration guides this setup leans on.