Object detection runs on the color stream at full frame rate, and one bounding-box corner has to become a depth coordinate. Running the full depth-to-color alignment to get it warps every pixel on every cycle, and the loop's frame rate drops with it. The cheaper path is to project the single point. This how-to lists the inputs the projection needs and where each one comes from, gives the call order, works the arithmetic for one point end to end, and shows the two failures that put the answer on the wrong surface, plus a round-trip test that catches both. It applies to any stereo RGB-D module whose Python binding exposes per-stream intrinsics, extrinsics and a depth scale. If the open question is whether color helps the pick at all, start with the 2D or 3D camera for conveyor pick and place comparison.
Why does aligning the whole depth frame slow the loop down?
The align step maps all of them: a 1280×720 depth stream is 921,600 pixel projections per frame, roughly 27.6 million per second at 30 fps, while a bounding box needs five points at most.
Alignment also writes a second full-resolution image and interpolates across depth discontinuities, and that memory traffic is what you feel as dropped fps. The single-point helper rs2_project_color_pixel_to_depth_pixel exists for exactly this case and is presented as an alternative to performing the align: it touches one ray, not 921,600 pixels. Five projections in place of the warp are 5 out of 921,600 mappings, a 99.9995% cut in the coordinate-transform work. If what you actually want is a distance under a color feature rather than a coordinate to index the depth buffer at, the color pixel to depth lookup route covers that variant.
Which inputs does the single-point call need?
Nine arguments, seven of them fetched once per session: the raw uint16 depth buffer, depth_scale, depth_min, depth_max, depth_intrin, color_intrin, the color→depth extrinsic, the depth→color extrinsic and the input pixel itself.
- depth_scale comes from the depth sensor and reads in meters per LSB, typically 0.001 for 1 mm units. Every depth value you derive multiplies by it; mixing raw LSB with meters is the classic wrong-scale bug behind aligned point clouds at the wrong scale.
- depth_intrin and color_intrin come from each stream's video profile via get_intrinsics() and are valid only for that stream at that resolution. Which set describes which buffer is the subject of aligned depth streams and their intrinsics.
- The two extrinsics come from get_extrinsics_to() called between the stream profiles. Keep both, because the helper wants them in opposite directions.
- depth_min and depth_max bound the search in meters. Only the pixel buffer changes per frame.
What are the steps to project one point?
Six steps, and only the last two run per frame: the calibration values are fetched once after you configure the streams, then each detection is one call plus one validity check.
- Set the depth and color streams to the resolutions you will run, then fetch depth_scale with the depth sensor's get_depth_scale().
- Fetch depth_intrin and color_intrin with get_intrinsics() from each stream's video profile.
- Fetch both extrinsics with get_extrinsics_to(): color profile to depth profile, and depth profile to color profile. Name them c2d and d2c.
- Set depth_min and depth_max in meters around the working band.
- Per frame, take the raw uint16 depth buffer and call rs2_project_color_pixel_to_depth_pixel with the header's order: depth intrinsics, color intrinsics, c2d, d2c, then the input pixel.
- Round the result to integers, read the raw depth there, multiply by depth_scale, and reject the point if the raw value is 0.
# calibration, once per session
depth_scale = depth_sensor.get_depth_scale() # m per LSB, e.g. 0.001
depth_intrin = depth_profile.get_intrinsics() # depth stream resolution
color_intrin = color_profile.get_intrinsics() # color stream resolution
c2d = color_profile.get_extrinsics_to(depth_profile) # color -> depth
d2c = depth_profile.get_extrinsics_to(color_profile) # depth -> color
depth_min, depth_max = 0.25, 1.0 # m, working band
# per detection; depth_data is the raw uint16 buffer of the current depth frame
corner = [400.0, 150.0] # u, v in pixels, floats
out = [0.0, 0.0] # result, floats
rs2_project_color_pixel_to_depth_pixel(out, depth_data,
depth_scale, depth_min, depth_max,
depth_intrin, color_intrin, c2d, d2c, corner)
u_d, v_d = int(out[0]), int(out[1])
What does the projection do with the numbers?
It deprojects the color pixel to a 3D ray in the color frame, walks candidates along that ray between depth_min and depth_max, projects each into the depth image, and settles where the measured depth agrees — for the pixel below, (400.0, 150.0) maps to (417, 165) at 0.5 m.
In effect the helper evaluates this chain along the color ray:
Work an illustrative 640×480 stereo module (example values, not a catalog product): depth fx 500.0 px and principal point (320.0, 240.0); color fx 600.0 px, principal point (320.0, 240.0); identity rotation and color→depth translation (0.030, 0.000, 0.000) m, a 30 mm baseline; depth_scale 0.001 m per LSB; band 0.25–1.0 m; detector corner at (400.0, 150.0) with the part near 0.5 m.
- Start from the detector's range guess Z = 0.500 m: X_c = (400.0 − 320.0) × 0.500 / 600.0 = 0.0667 m and Y_c = (150.0 − 240.0) × 0.500 / 600.0 = −0.0750 m.
- Color→depth with the 30 mm offset: X_d = 0.0667 + 0.030 = 0.0967 m, Y_d = −0.0750 m, Z_d = 0.500 m.
- Project into the depth image: u_d = 500.0 × 0.0967 / 0.500 + 320.0 = 416.7 and v_d = 500.0 × (−0.0750 / 0.500) + 240.0 = 165.0, so pixel (417, 165).
- The raw value there is 502 LSB, or 0.502 m after depth_scale. Recomputing with Z = 0.502 m gives u_d = 416.5 — the same integer pixel, so the answer has converged after one pass.
Two checks confirm it. The baseline alone should shift the image by fx·b/Z = 500.0 × 0.030 / 0.500 = 30.0 px, and the focal-length ratio should scale the 80 px corner offset to 80 × 500 / 600 = 66.7 px; 320.0 + 66.7 + 30.0 = 416.7. And with parallel optical axes the v mapping needs no depth at all, which is why v_d came out exactly 165.0. One depth pixel at this range spans Z / fx = 0.500 / 500.0 = 1.0 mm laterally, so a converged projection is good to about one pixel footprint.
How tight should depth_min and depth_max be?
Search the working band, not the sensor's full range: for parts arriving 0.30–0.50 m out, set depth_min = 0.25 m and depth_max = 1.0 m even if the module ranges 10 m, because fewer candidate depths mean faster convergence and no far background pixel can win the match.
The helper enumerates candidate depths across the band, so a full-range band spends its iterations on distances your cell never produces and admits matches on a racking beam meters behind the part. Keep depth_min at or above the module's shortest working range, and leave margin on both sides of the detection's distance estimate — a band tighter than the detector's range error can exclude the true surface. When part classes sit at different distances, per-class bands are one dictionary away.
Why does the projected point land in the wrong place?
Almost every wrong-pixel report comes down to two causes: a zero depth value at the target, which makes the search snap to a nearby valid pixel, and the two extrinsics passed in swapped order, which moves the answer by roughly the whole baseline.
Zero depth is common on shiny edges, dark plastic and the occlusion border of a stereo pair. The helper does not return an error; it walks to the nearest valid pixel and reports that coordinate, which can sit several pixels from the true point. Read the raw value at the returned pixel and treat 0 as unmeasured.
A swapped extrinsic pair reverses the 30 mm translation in the worked example: the answer moves from u_d = 416.7 to 500.0 × (0.0667 − 0.030) / 0.500 + 320.0 = 356.7, a 60 px error worth 60 mm at 1.0 mm per pixel. Name the two extrinsics at fetch time and pass them in the header's order, and the symptom disappears.
One third cause is self-inflicted: changing resolution or binning a stream after startup leaves cached intrinsics describing buffers they no longer match — scale and refetch them as in the guide to camera intrinsics after binning.
How do you prove the projection is right before trusting it?
Run two checks: a software round trip that must close within one pixel, and one physical target at a measured distance — in the example, a target taped at 0.500 m must project back to u_c = 400.5, within 0.5 px of the input.
Round trip the worked example: deproject (417, 165) at 0.502 m with the depth intrinsics to X_d = (417 − 320) × 0.502 / 500.0 = 0.0974 m and Y_d = (165 − 240) × 0.502 / 500.0 = −0.0753 m; apply the depth→color extrinsic, X_c = 0.0974 − 0.030 = 0.0674 m; reproject with the color intrinsics to u_c = 600.0 × 0.0674 / 0.502 + 320.0 = 400.5 and v_c = 600.0 × (−0.0753 / 0.502) + 240.0 = 150.0. The 0.5 px residue is the integer pixel rounding and nothing more; a residue over a pixel points at stale calibration values, not at the math.
Then one bench check: tape a target at a tape-measured 0.500 m, project its color pixel, and read depth. An offset beyond the module's depth accuracy at that distance means the extrinsics are stale — the camera was bumped, remounted, or the profiles were fetched before a resolution change — so refetch rather than recompute.
When is a full aligned frame still the right call?
Align once when many points need color registration: dense overlays, segmentation masks, or a downstream consumer that already wants the aligned stream — past roughly fifty projected points per frame, one bulk warp beats hundreds of individual searches in both code and runtime.
The decision is the count of points you genuinely need per frame. Four corners plus a center is five calls; keep the single-point path and its zero per-frame allocations. A color overlay across most of the image is the warp's home ground. When you do align, do it once per frame and index the aligned buffer for every detection; the color-to-depth projection walkthrough covers that per-pixel mapping step by step. Mixed setups work too: align a downsampled stream for the overlay and run single-point calls on the full-resolution depth for the measurement.
If you are still choosing the sensor itself, the 3D sensors range covers laser profilers and stereo modules, the camera finder filters by type, resolution and interface, and the configurator pairs the camera with the rest of the station. For help matching a module to this projection workflow, use the contact page.
Questions
How accurate is one projected point?
Laterally, about one depth pixel: at 0.5 m with a 500 px depth focal length one pixel spans 1.0 mm, so expect ±1–2 mm, growing linearly with range — at 1.0 m the footprint doubles. In depth, expect the module's stated depth accuracy at that distance, plus any offset from a snap to a neighboring valid pixel when the raw value was 0. Check the raw depth at the returned pixel before trusting the point.
Why is my point off by about one baseline?
That is the swapped-extrinsics symptom: the header's argument order is depth intrinsics, color intrinsics, color→depth extrinsic, depth→color extrinsic. With the pair reversed, the translation flips sign — in the worked example the answer moved from depth pixel 416.7 to 356.7, a 60 px error. Fetch both extrinsics, name them c2d and d2c where you get them, and pass them in that order.
Can I project a point without a live depth frame?
The projection reads real depth values out of the uint16 buffer, so it needs one. A synthetic buffer filled with a constant works and converges to that range, but the answer is then only as good as the guess. For a known fixed distance, skip the buffer: deproject the color pixel, apply the extrinsics by hand, and project into the depth image with the three-line formula.
Do intrinsics change when I switch stream resolution?
Yes — intrinsics belong to one stream at one resolution, and binning scales them too. Values fetched from a 1280×720 color profile do not describe an 848×480 buffer; the projection then lands a few pixels off with no error raised. Configure the streams first, fetch from the active profiles, and refetch whenever resolution or binning changes.
How do I steady a flickering distance reading?
Average over frames: depth noise falls with the square root of the frame count, so nine frames read about 3× quieter than one. Keep the projection per frame — intrinsics and extrinsics do not move — and average the resulting distances; or, with a static scene and a fixed returned pixel, average the raw values at that pixel and multiply by depth_scale once.