You exported a point cloud from a stereo depth camera, projected the points back into your images with the published intrinsics, and got scatter that matches nothing: sizes wrong, positions shifted. The cloud is fine; the projection used the wrong frame. This page pins down where the origin of a stereo depth cloud sits, why the depth image and the reference infrared image align pixel for pixel, and which intrinsics reproduce each stream. You get the deprojection and projection formulas, a worked round trip with the arithmetic shown, and the baseline rule that explains drift growing as distance shrinks. Everything here applies to any depth camera that publishes a depth image plus intrinsics, structured light or ToF alike.
The short answer: the cloud lives in the depth frame
The exported point cloud sits in the depth frame's own coordinate system. Its origin is the optical center of the reference imager — the camera the stereo pair was rectified to, conventionally the left IR camera of a binocular module. X points right, Y down, Z forward through the lens, and Z carries the depth unit, almost always millimeters on industrial SDKs.
Because depth is computed per pixel of that rectified frame, depth and reference IR align pixel for pixel. That alignment is the decision rule: project the cloud with the depth stream's intrinsics and you reconstruct the depth image exactly. The imager-side view of this origin is in depth camera coordinate origin: where is the imager center; the reverse projection, cloud to pixels, is overlay 2D image on depth map.
Every depth camera that publishes depth as an image plus intrinsics behaves the same way, whatever generates the depth inside — projected structured light, a laser galvanometer stereo pair, speckle patterns for volume rigs. The classes below all hand you a cloud built from that depth frame:
- Type
- Stereo / structured light
- Type
- Stereo / structured light
- Type
- Laser profiler
How stereo builds the cloud from a depth image
The module rectifies both raw images onto a common pixel grid, then matches features across the pair to produce a disparity d wherever the fields of view overlap. Depth follows from similar triangles: Z = fx·b / d, with b the baseline between the two imagers. Each valid depth pixel then unprojects through the pinhole model:
The unprojection runs on the rectified grid's intrinsics, so the cloud inherits the depth frame's geometry with nothing else hidden in the export. Two consequences matter. The cloud only covers pixels with disparity: outer columns beyond the stereo overlap, occluded edges and untextured surfaces come back empty. And each X and Y is proportional to its own Z, so a bad depth at one pixel never drags its neighbors — errors stay per pixel.
Use the rectified intrinsics the SDK publishes with the stream, not a raw lens calibration: rectified vs raw stereo IR images sorts out which numbers belong to which image type.
The round trip that must work: project into the depth image
Take a 1280 × 720 depth stream with fx = fy = 954.32 px, cx = 639.97 px and cy = 363.41 px. These values are illustrative — read the ones your SDK publishes for the depth stream at its own resolution. Pick a pixel at u = 800, v = 400 whose measured depth is Z = 1500 mm.
Deproject it. X = (800 − 639.97) × 1500 / 954.32 = 160.03 × 1500 / 954.32 = 251.5 mm. Y = (400 − 363.41) × 1500 / 954.32 = 36.59 × 1500 / 954.32 = 57.5 mm. The PLY stores that point as (251.5, 57.5, 1500.0) mm, expressed at the depth frame's origin.
Project back with the same intrinsics. u = 954.32 × 251.5 / 1500 + 639.97 = 160.01 + 639.97 = 799.98. v = 954.32 × 57.5 / 1500 + 363.41 = 36.58 + 363.41 = 399.99. The point lands 0.02 px from home; rounding is the only loss. A round trip missing by whole pixels means the intrinsics are not the depth stream's.
Sanity-check fx itself while you are there: tan(FOVx / 2) = (W / 2) / fx = 640 / 954.32 = 0.6706, so FOVx = 2 × arctan(0.6706) ≈ 68°. An fx implying a field of view far off the datasheet's is the wrong stream's calibration.
Why the same formula drifts on the IR images
Project the cloud into the reference IR image at the same resolution and the depth intrinsics still work, because depth is aligned to that frame. The secondary imager is different: it sits a baseline b away, so each point needs the translation X → X − b before projecting. Skip it and every point lands off by fx·b / Z pixels.
The offset scales with 1/Z, which is why it reads as a scale mistake. With fx = 954.32 px and b = 60 mm for this example, fx·b = 57,259 px·mm. A point at Z = 1500 mm shifts 57,259 / 1500 = 38.2 px; at Z = 750 mm it shifts 76.3 px; at Z = 3000 mm only 19.1 px. Near points drift about twice as far as points twice as distant, plus a small constant offset from differing principal points — the scale-plus-position symptom, exactly.
Three routes into the secondary image all work. Apply the extrinsic translation before projecting — rotate then translate, in that order. Use the disparity directly: u_secondary = u_reference − d, with d = fx·b / Z, so the u = 800 point at 1500 mm lands at 800 − 38.2 = 761.8. Or have the SDK re-express the cloud in the target frame. A wide baseline makes the offset punishing fast; what a wide baseline did to stereo overlap shows the cost at the pixel level.
Which intrinsics to read, and at which resolution
- Cloud exported from the depth stream at width W and height H: use the depth stream's intrinsics at W × H. This is the case that must reproduce the depth image.
- Reference IR at the same resolution: the same numbers work unchanged.
- Any stream at another resolution: rescale fx and fy by the width ratio and shift cx and cy the same way — rescaling intrinsics after resizing shows the arithmetic.
- Color or any other camera: only after the points have been moved into that camera's frame. Aligned depth streams and their intrinsics sorts out which stream owns which numbers, and which camera to deproject with settles the choice when several streams are aligned.
Into the color image: two clean routes
Route one is alignment. Run the SDK's depth-to-color alignment so the cloud is re-expressed in the color camera's frame, then project with the color intrinsics; registering RGB and depth images walks that path. Route two is a manual transform: move the points into the target camera's frame with its extrinsics first, then apply the target intrinsics. The color imager is a separate physical camera with its own focal length and principal point, so depth intrinsics on color pixels are wrong twice over.
Holes, overlap edges and far-outlier points
Expect empty regions with everything configured correctly. Disparity exists only where both imagers see the surface, so occlusions, frame edges and untextured patches leave gaps — missing points in a point cloud lists the fixes. Stereo matching also throws occasional far outliers, points at tens of meters from poor matches on shiny or featureless areas. Clip Z to your working envelope before projecting: a 5,000 mm ceiling on a 1,500 mm bench cell removes them silently. If your overlay disagrees with the viewer's render of the same file, suspect the intrinsics source, not the export — exported clouds vs the viewer render lists the usual causes.
When the origin is not an imager
Depth aligned to color before export moves the origin to the color camera's optical center, and the color intrinsics own the projection — worth checking before you assume the reference-imager default. Direct ToF skips stereo matching and still publishes a depth image plus intrinsics, so its cloud sits in the depth frame the same way:
- Type
- Depth / ToF camera
Line-scan 2.5D systems and laser profilers are the real exception: their geometry is a triangulation fan assembled from profiles plus the travel axis, so the native frame belongs to the scanner's calibration, not one imager's pinhole. A system such as the MV-HLVS-16K100-L60 samples 16K-resolution profiles at a 120 kHz sampling rate across a 600 mm emitting area; projecting those points with a camera's intrinsics needs the system's hand-eye calibration first.
- Type
- 2.5D inspection
One bench note: projected-pattern and laser-based depth heads are rated eyesafe at their intended standoff, so keep faces out of the window while a housing is open during set-up.
Checks before you trust a projection
- Round-trip the depth image first: deproject a few pixels, project them back, and expect sub-pixel error as in the worked example.
- Cross-check one point by reading depth at a fixed pixel with the center-pixel method and matching it to the cloud point that projects there.
- Measure a known dimension in the cloud to pin the unit before any tuning.
- Sort the residuals by depth: flat errors point at the principal point, error growing with 1/Z at the missing baseline shift, one clean multiplier at a unit mix-up, and scatter concentrated at the frame borders at the stereo overlap limit.
Choosing hardware is the same decision in reverse
If you are specifying rather than debugging, run the rule backwards: pick the sensor whose native frame matches where your measurements must live. The 3D sensors category and the camera finder filter by type, resolution and interface, and area scan cameras add full-rate 2D texture alongside a depth module when the inspection needs both. Depth-frame guides beyond this one sit on the 3D topic page, and the team answers sizing questions through the contact page.
Laser profilers and stereo 3D cameras
170 models · from $699.00The configurator matches a camera, lens, light, light controller and cables to each other, step by step, and lists only parts that fit what you already picked.
Questions
Where exactly is the point cloud origin measured from?
The optical center of the reference imager — the principal point of the camera the pair was rectified to — not the lens glass or the housing. Some SDKs also publish a separate mechanical datum, often the front of the reference lens, for mounting and hand-eye calibration. Use the mechanical datum for robot poses and the optical origin for projections; mixing them shifts every projected point by the fixed offset between the two.
What units does the exported cloud use?
Whichever the SDK writes — mostly millimeters on industrial depth SDKs, meters on many robotics stacks. The export header or the API call that builds the cloud names it. Check once by measuring a part of known size: a board that comes out 1,000 times too large or too small is a millimeter-versus-meter mix-up, and no intrinsics change will fix it.
Does the origin move when I change resolution, bin, or crop an ROI?
No. The origin stays at the reference imager's optical center; only the intrinsic values change with the pixel grid. fx and fy scale with resolution, 2 × 2 binning halves them, and an ROI crop shifts cx and cy by the crop offset. Rescale the intrinsics by the same ratios you applied to the image and the projection stays exact; the axes and the origin do not move.
Why does the cloud match the depth image but land off the color image?
The color imager is a separate physical camera with its own position, focal length and principal point, so a cloud expressed at the depth frame's origin cannot land on color pixels unaided. Run the SDK's depth-to-color alignment and project with the color intrinsics, or transform the points into the color frame first. Alignment costs some latency per frame; the manual extrinsic route keeps the pipeline leaner.
How close should a correct round trip land?
Within a pixel. Reprojecting the cloud into the depth image loses only rounding, so 0.2 px is normal and anything past 1 px means the wrong intrinsics or units. Classify the residual against Z: a constant offset is a stale principal point, growth as Z shrinks is an unhandled baseline, a uniform scale is a unit mismatch, and scatter at the frame edges is the stereo overlap boundary.




