A driver-generated depth point cloud carries exactly one texture: either RGB or infrared intensity, chosen by a parameter. Standard point cloud messages have one color field per point, so there is nowhere to put a second value, and no launch setting will change that. The fix is to stop consuming the packaged cloud and build your own: subscribe to the depth, color and infrared streams separately, align them into one coordinate frame, and project each 3D point onto both images so every point ends up with RGB and IR intensity. This guide walks through that pipeline for a robotics or inspection application, with the checks that catch the classic failures: zero intensities, ghost colour at edges, and points sampled from the wrong image.
Why the packaged cloud cannot carry both textures
The usual driver node builds its cloud by taking depth-aligned-to-color coordinates and copying one pixel from the texture stream you configured into each point's color field. When you set the texture to infrared, the same field is filled with the IR pixel instead of RGB. The structure simply has one slot.
Two consequences follow. First, asking for both via launch parameters cannot work; there is no such option. Second, converting the packaged message into a typed cloud drops whatever does not fit its fields, which is why a reader built for RGB warns that the rgb field is absent when it receives an IR-textured cloud.
The exit is to treat the packaged cloud as a convenience, not a data source. Subscribe to the raw images and do the projection yourself.
The streams you need and how they relate
A typical active depth camera publishes four streams you care about:
- Depth image, 16-bit, values in millimetres.
- Color image, 8-bit RGB or BGR, its own sensor and field of view.
- One or two infrared images, 8-bit (Y8) or 16-bit, from the stereo pair or the projector-side imager.
- Intrinsics and extrinsics for each stream, read from the device at startup.
The color and infrared sensors sit at different physical positions, so they see the scene from slightly different viewpoints. That difference is a rigid transform plus per-stream intrinsics, and it is the whole geometry problem. If your pipeline already registers depth to color, you have done half the work; the depth to color registration guide explains raw versus rectified frames and when each is the right input.
If you use a structured-light or stereo camera that exposes IR and depth natively, such as the MV-DPS200P-02, the infrared stream is the texture the matching algorithm consumed, so its intensity correlates directly with surface quality. You can browse the shop's 3D sensors or use the camera finder to shortlist devices that expose all three streams you need.
The projection math in one page
For each depth pixel (u, v) with depth value z:
Here fx, fy, cx, cy are the depth stream's intrinsics. That gives you a point in the depth camera frame.
To colour the point, project the same 3D point into the color image:
where (X_c, Y_c, Z_c) is the point transformed by the extrinsic transform from the depth frame to the color frame. Sample the RGB pixel at (u_c, v_c). Repeat the same two lines with the infrared intrinsics and the depth-to-infrared transform to sample the IR intensity.
Worked example: a depth pixel at (320, 240) with z = 1000 mm, depth intrinsics fx = fy = 382, cx = 320, cy = 240 gives X = 0, Y = 0, Z = 1000. With color intrinsics fx = fy = 640, cx = 640, cy = 360 and a small baseline offset of 15 mm along x, the color projection lands at u_c = 640 * (15 / 1000) + 640 = 649.6, v_c = 360. Sample RGB there, then repeat with the IR pair.
Sanity-check the pixel scale in advance with the camera resolution calculator: it tells you how many pixels cover your target, which is also how much resolution the texture sampling preserves or loses.
Step-by-step: build the fused cloud
- Read the device intrinsics and extrinsics at startup, once, from the SDK. Do not hard-code factory values; units and rectification differ between models.
- Subscribe to three image topics: depth, color and infrared. Request the raw (non-rectified where your SDK expects it) formats and note each stream's timestamp.
- Match frames by timestamp before projecting. If your color and depth frames come from different instants on a moving part, the colour lands on the wrong surface. The depth and color synchronization guide covers which timestamps to compare and what tolerance to allow.
- Align color to depth once with the SDK's alignment function, or apply the extrinsic transform yourself. Aligning depth to color instead is fine too, but then compute XYZ in the color frame and stay consistent.
- For every valid depth pixel, compute (X, Y, Z) with the depth intrinsics. Reject zeros and saturated range values.
- Transform the point into the color frame, project, bounds-check, and copy RGB into the point's color field.
- Transform the same point into the infrared frame, project with the infrared intrinsics, bounds-check, and copy the IR pixel into a new intensity field you added to the point structure.
- Publish or store the cloud with the extended fields. If a downstream tool only reads the standard color field, publish two clouds from the same XYZ set: one RGB-textured, one IR-textured.
If you keep the depth frame fixed and add textures in a second pass, you can debug each projection independently, which matters when you chase edge artefacts.
Common failure modes and how to test each
- Intensities near zero. The texture was read as float metres or the field was filled but never sampled. Test: print a 10 by 10 block of IR values straight from the raw image and from your cloud; the ranges should match (0 to 255 for Y8).
- Colour offset by a few pixels that grows toward the image edge. The extrinsic transform is missing or uses the wrong direction (depth-to-color versus color-to-depth). Test: project a checkerboard corner from depth into color and overlay; it should sit within a pixel or two.
- Ghost colour on slanted edges. Depth and color frames are not time-matched, or alignment was done with rectified intrinsics on raw images. Test: capture a static scene; if ghosting persists on a static scene it is geometry, not timing.
- IR black in half the image. The IR imager saturates from ambient light or the projected pattern is disabled during that exposure. Test: subscribe to the IR image directly before projecting; if the raw image is fine, the fault is in your transform. The stereo infrared black image guide lists further causes in order of likelihood.
- Points outside the texture image. Depth field of view is usually wider than color. Always bounds-check before sampling; discard or clamp.
If you need to see the result in 3D before trusting the numbers, the stereo point cloud visualization guide shows how to render the cloud and spot missing points early.
Choosing hardware for a three-stream pipeline
The pipeline above works with any depth camera that exposes depth, color and IR streams with readable calibrations. Structured-light models give the strongest IR pattern, which makes IR intensity a useful quality signal for matte and dark surfaces alike. Logistics volume-measurement cameras target dimensions and pallet sizing rather than dense texture work, so match the sensor type to what the IR channel must tell you. If you also need a conventional 2D camera in the same cell for appearance checks, the area scan cameras category covers GigE and USB3 models that can share the trigger with the depth device, and the camera configurator narrows the choice by interface, resolution and mount.
Checks before you trust the output
- Reconstruct a flat, matte target at a known distance and confirm the fitted plane is within a fraction of a millimetre of expected.
- Project a known 3D marker into both images; each reprojection should land within one to two pixels.
- Compare IR histogram of your cloud against the raw IR image histogram; they should match closely.
- Rotate a textured object between two captures and confirm colours stay attached to surfaces, which proves the timestamps and transforms agree.
When the usual answer is wrong
Sometimes the answer is not to fuse everything into one cloud. If your consumer is a learning-based detector that only needs RGB plus depth, two separate aligned clouds are simpler and preserve more precision than packing values into nonstandard fields. And if the IR channel's only job is to detect projector drop-out, a downscaled IR grid sampled every fifth point is enough. Build the full fused cloud only when a downstream algorithm genuinely reads both textures per point.
Quick checklist
- Read intrinsics and extrinsics from the device at startup, never hard-coded.
- Timestamp-match depth, color and IR before projecting.
- Bounds-check every projected pixel before sampling.
- Add a dedicated intensity field, or publish two textured clouds from one XYZ set.
- Verify with a flat target and a known marker before running on parts.
If your application needs a device or sensor combination you do not see here, tell us the scene through the contact page and we will help you specify it.
For related reading on mapping image pixels to 3D values, see the bounding box depth guide and the camera intrinsic parameter conversion guide, which covers converting intrinsics between depth and texture resolutions, the same conversion step 4 relies on.
Questions
Why does the driver's point cloud topic not include both RGB and IR intensity?
The point cloud message type most drivers emit stores one four-byte color field per point. The wrapper fills that field from whichever texture stream you selected, so you get color or infrared intensity, never both at once. There is no launch parameter that adds a second texture channel, because the message structure itself has nowhere to put it.
Which streams should I align to which frame?
Align color to the depth frame, then compute XYZ from the depth image and sample color at each depth pixel. Keep the infrared image in its own frame and sample it with its own intrinsics after transforming the 3D points into the infrared camera's coordinate system. Aligning infrared to color first loses depth resolution and couples two calibrations into one step.
Why are my IR intensity values near zero after projection?
Near-zero values usually mean the texture was sampled as a float in metres or normalized instead of an 8-bit pixel value, or that the points landed outside the infrared image because the frames were not transformed with the correct extrinsics. Check the scale, the ordering of the transform, and that every point is tested against the image bounds before sampling.
Can a single point cloud message carry both textures?
Yes, if you build the message yourself. Add extra fields to the point structure, for example r, g, b and intensity as separate channels, and fill them during projection. Standard visualizers may ignore the extra channel, but any reader you write can access both values per point.
Do I need an extra camera for RGB and IR at the same time?
Most depth cameras expose a color sensor and infrared imagers simultaneously, so one device is enough. Structured-light models such as the MV-DPS200P-02 publish IR and depth streams together, and the color stream comes from the same housing. Only very compact depth modules without a color sensor need a separate RGB camera.