Recompute normals on a rendered point map for textured 3D data

You computed normals on your stereo point map, rendered the cloud into the RGB camera's view to texture it, and now the shading is wrong: the normals belong to the original grid, not to the rendered image. The fix is not to transform the old normals but to recompute them on the rendered point map, passing that map explicitly as the input. This guide walks through the workflow, the occlusion problem that makes naive re-projection fail, and the node-path mistakes that silently fall back to per-camera normal maps. It is for anyone building textured, shaded point clouds from a 3D sensor with an RGB camera, for inspection, picking or archiving.

Why the original normals stop matching after rendering

A stereo or structured-light sensor outputs its point map in its own perspective, typically the left camera's rectified view. Each pixel holds an XYZ point, and a normal computed at pixel (r, c) describes the surface around that exact point.

Rendering into the RGB camera changes three things at once:

1. Perspective. The new image is sampled from the RGB camera's viewpoint, so the same surface occupies different pixels.
2. Resolution. The RGB image usually has a different pixel count, so the render resamples the cloud.
3. Occlusion. Several original points can project onto one RGB pixel; the renderer keeps the visible one and discards the rest.

After that, pixel (r, c) in the rendered map is a different surface location than pixel (r, c) in the original map. Every downstream use of the original normal image — shading, edge angle filters, plane fitting per pixel — is now offset and possibly scaled. The mismatch is largest on curved parts and at depth discontinuities, where the resampling is most aggressive.

For background on how stereo depth and its rectified views behave, see the guide on stereo disparity map troubleshooting after rectification.

Transform the points, recompute the normals

The point coordinates must move into the color camera's coordinate system; the normals should be estimated fresh in that system. The chain of transforms is straightforward:

You invert the stereo camera's link transform into the workspace, apply the color camera's own 2D/3D transform, and then project with the rectified color camera matrix to find each point's row and column in the rendered image.

What you must not do is rotate the original normal vectors through the same chain and attach them to the rendered pixels. That produces vectors in the right coordinate frame but at the wrong sample positions, and it gives you no rule for occluded pixels. A renderer already solves visibility correctly — z-buffering picks the nearest point per pixel — so let the render do its job, then estimate normals on its output. If you tried to hand-roll this, you would end up reimplementing the software render path yourself, with all its edge cases.

The workflow, step by step

1. Acquire and compute the point map from the 3D sensor in its native perspective. Verify the cloud is complete before any rendering; dropout patterns that start here only get worse downstream.
2. Compute normals on the original map if you still need them there, for example for native-perspective edge inspection. Keep them in their own node; do not overwrite.
3. Render the point map into the RGB camera's perspective, with the color texture attached. Confirm the rendered output is written to its persistent result node in the image tree.
4. Recompute normals with the rendered map as the explicit input. The normal command accepts a point-map parameter; pass the rendered map's node path. Do not also pass a camera list — see the next section.
5. Verify alignment. Shade the rendered cloud with the new normals and tilt the view: highlights should sit where the surface faces the light. Compare a flat reference face; its normal should be uniform within a few degrees.
6. Export one cloud per pixel carrying XYZ, RGB and normal together, so downstream consumers never have to reconstruct the correspondence.

If your sensor and host pipeline drop frames or stall during this chain, the recovery patterns in camera acquisition stalls under load: isolate recovery faults apply directly.

The per-camera fallback trap

The most common failure after upgrading to rendered-map normals: you request normals for the rendered map and receive two or three normal images — one per camera, plus perhaps a general one. The command used its default camera mode instead of your point map.

Three causes, in order of likelihood:

• You specified both a camera list and a point map. When a point map is given, the camera list is unnecessary, and supplying both can produce the general map plus per-camera maps.
• The point-map path was wrong or not yet valid, so the command fell back to all open cameras.
• The rendered result was requested inside the same execution slot before its node path existed. In many SDK trees the render result is written to a fixed node such as the rendered image node under the images branch; reference that stored path, not a transient handle.

A quick test: run the normal command with only the rendered map path, nothing else, and check the result node count. One merged map in the render's perspective — including a telecentric or combined view rendered from several calibrated sensors — means you got it right.

Multi-camera workspaces

When two or more sensors are calibrated into a common workspace and the render fuses them into one image, the same rule holds: compute normals on the fused, rendered map. If you instead let the command run per camera, each normal map comes back in that camera's own perspective, which is exactly the mismatch you started with. See the multi-camera point cloud synchronization for mobile robots guide for the synchronization side of fused setups.

Hardware that fits this pipeline

A projected structured-light camera is one option for depth acquisition. Confirm whether your application requires a separate RGB camera and color-to-depth calibration.

Rendering, normal estimation and texturing are all host-side work, so the PC matters. A controller with enough cores, PoE camera ports to feed the sensor and the RGB head, and built-in light channels keeps the pipeline on one machine.

Browse the full range on the 3D sensors category page or the camera finder, and the vision controllers category. Use the system configurator to plan your camera and lighting components, and confirm controller compatibility separately.

Worked example: checking the result on a flat reference

Place a matte flat plate at a known tilt, say 15 degrees from the RGB camera's optical axis, and run the full chain.

• Extract the normal vectors inside a 100 × 100 pixel patch in the center of the rendered plate.
• Convert to angles: for each normal (nx, ny, nz), tilt from the viewing axis is acos(nz).
• Compute the mean over the patch. On a correct result the mean sits near 15 degrees with a standard deviation under about 2 degrees on a matte surface.
• Repeat after re-running the render. If the mean shifts by more than a degree between runs, your render or your node path is not deterministic — usually a stale handle.

If the standard deviation is large but the mean is right, look at the estimation neighborhood before blaming calibration: a radius that is too small on a stereo-derived surface amplifies noise. For reading depth at a fixed location as a sanity input, the technique in center pixel depth measurement: read a fixed image point is a fast check.

When the rendered-map answer is wrong

Recomputing on the rendered map is the right default, but not always:

• Native-perspective inspection. If your defect detection runs on the sensor's own point map, compute normals there. Rendering first would only add resampling noise. Keep two normal sets if you need both.
• Thin or fuzzy parts. Stereo dropout and translucent surfaces produce sparse clouds where the rendered map has holes; normals estimated around holes lean on the fill and mislead. The failure modes are covered in the close-range depth dropout guide.
• Extreme curvature. A neighborhood of two to three point spacings can straddle a sharp edge. Shrink the radius or segment edges first.

Pre-order checklist

• Confirm the SDK version on the host actually supports a point-map input for normal computation; older releases only accept camera mode.
• Check the RGB camera's resolution against the depth map: a large ratio means aggressive resampling and wider normal neighborhoods.
• Verify the color-to-depth extrinsic calibration exists and is current; it is the transform chain the whole method rides on.
• Plan storage: XYZ plus RGB plus normals roughly triples the per-pixel payload of a plain cloud.
• Decide once whether each consumer needs the rendered or native perspective, and label the nodes accordingly.

If you are specifying a complete scanning cell rather than fixing one pipeline, the planning approach in 3D scanner prototype planning for limb surface capture carries over. To map pixel positions back into 3D space after rendering, see bounding box depth: mapping image pixels to 3D points. For anything not covered here, contact us and an applications engineer will help you scope the set-up.

Quick reference

• Original normals belong to the original grid; rendering invalidates the pixel correspondence.
• Recompute normals with the rendered point map as the explicit input; never rotate old normals through the render transform.
• Pass the point map alone — no camera list — or you will get per-camera fallback maps.
• Reference the render result by its stored node path, and render before computing within the same execution slot.
• Validate on a tilted flat plate: mean tilt matches, spread under a few degrees.