Structured light 3D cameras: projector pixel mapping to 3D points

A structured light 3D camera is usually bought as a depth sensor: projected pattern in, point cloud out. The projector inside it is also a calibrated output device, and that changes what the camera can do for you. Because the factory calibration ties every projector pixel to the camera's coordinate frame, the SDK can take any 3D point you detect — a pick point in a bin, a checker corner on a fixture, a hole center on a casting — and return the projector pixel that illuminates it. This guide explains that mapping, works out a marker-size budget with the arithmetic shown, and lists what your SDK must expose before you build pick-point projection or operator guidance on top of it.

The projector is a second, calibrated camera

Inside the housing sit two optical systems on one rigid frame: a camera sensor and a projector. The camera's 2D image pixels correspond to sensor pixels, the sites that collect photons. The projector image pixels correspond to projector pixels, the sites that emit them. Depth comes from the fixed offset between the two viewpoints: the projector casts a known pattern, the sensor sees it deformed by the surface, and triangulation turns that deformation into distance. It is stereo geometry with the projector playing the second camera. The structured-light 3D scanning overview covers the pattern families — stripes, grids, speckle — and the triangulation principle in more depth.

The same factory calibration runs in both directions. Forward, a projector pixel plus its depth gives a 3D point — that is the depth map to point cloud calculation. Backward, a 3D point plus the calibration gives the projector pixel that hits it. Backward is the direction that matters here, because it lets the camera draw on the scene instead of only watching it.

The first line is the depth principle: B is the projector-to-camera baseline, f the focal length in pixels, d the disparity the pattern deformation encodes; B and f are fixed by the factory calibration. The second line maps your 3D point P, expressed in the camera frame, through the calibrated rotation R and translation t between projector and camera and the projector's intrinsic matrix K_proj, giving the projector pixel (u, v). You never assemble these matrices yourself: the SDK's mapping call applies the internal calibration for you.

On a projected structured-light camera such as the MV-DPS200P-02, the same optics that carry the depth pattern can carry your overlay.

How a projected overlay gets built

The flow through a structured-light SDK with a projection interface looks like this:

1. Connect and query the projector resolution. It is a fixed property of the model, it is not the sensor's resolution, and it should be read rather than hard-coded.
2. Build the projector image at exactly that pixel grid. SDKs accept the frame as a plain pixel buffer, typically BGRA byte order; load an image sized to match, or resize your artwork to fit.
3. Capture a first 3D frame and detect the features the overlay must land on: checker centers on a calibration board, grasp points from a bin-picking model, hole centers from a CAD match.
4. Call the mapping that converts 3D points in the camera frame to projector pixels; it returns integer pixel coordinates, one per point.
5. Draw the overlay at those coordinates — circles at pick points, crosses at corners, a colored zone around a defect.
6. Start projection. The overlay stays on the scene for as long as the SDK's projected-image handle stays alive.

An overlay that ignores the 3D data needs none of the mapping: any image at the projector resolution can be thrown onto the scene as-is. The mapping earns its keep the moment the drawing must land on geometry the camera detected.

Worked example: marker budget for a bin pick at 600 mm

The inputs, each with its source:

• Projector grid: 1000 × 720 px — the projector image resolution of the example camera class this article uses; read your own model's value from the SDK query in the list above.
• Pattern span at the target: 300 mm across, measured once by projecting a full-white frame onto a steel ruler lying where the part sits.
• Standoff: 600 mm from the camera face to the bin surface.
• Marker: a 20 mm diameter circle, sized to mark the grasp zone for a parallel-jaw gripper.

The rule in words: footprint per projector pixel equals pattern span divided by projector pixels across that span, and a drawn feature reads as a shape only when it spans at least about three projector pixels, because fewer cannot carry both an edge and an interior.

The arithmetic:

• footprint = 300 mm ÷ 1000 px = 0.30 mm per projector pixel
• marker width = 20 mm ÷ 0.30 mm/px ≈ 67 px across
• smallest drawable feature = 3 px × 0.30 mm/px ≈ 0.9 mm
• vertical span = 300 mm × (720 ÷ 1000) = 216 mm

The 20 mm circle is 67 projector pixels wide, so it renders as a clean ring with a hard edge. Anything drawn smaller than about 1 mm at this standoff will not read.

The decision: keep markers at or above 1 mm when the bin sits at 600 mm. To mark a 0.5 mm feature, halve the standoff — projected size scales roughly linearly with distance, so the span drops to about 150 mm, the footprint to 0.15 mm/px, and the feature spans 0.5 mm ÷ 0.15 mm/px ≈ 3 px, the minimum with no margin left. Doubling the standoff to 1200 mm instead widens the footprint to about 0.6 mm/px, which is why marker work lives at bench distances; the 3D camera working distance and positioning guide covers choosing the standoff itself.

What runs alongside, and what stops the overlay

The projector and the 2D color camera are controlled separately, so a 2D capture during projection records the scene with the overlay in place — operator feedback, or an archived proof image of what was marked. A 3D capture is the exception: the decoder needs the camera's own projected pattern, so starting one ends the overlay. Order the cell's cycle as overlay first, 2D capture second, 3D capture last, and keep the handle alive exactly as long as the overlay should show. Capturing both streams from one scene is its own topic; see capturing 2D and 3D images together from one scene.

Where it earns its keep, and where it washes out

If you are still choosing between stereo, ToF and structured light, sort the trade-offs first with the industrial 3D camera selection guide; the rest of this section is about getting an overlay out of a structured-light camera. Projected overlays pay off in four places:

• Pick-point projection: show the grasp the vision system computed, at a manual bench or as a spot-check in a robot cell. On conveyors the camera choice itself comes first — see 2D or 3D camera for conveyor pick and place.
• Fixture loading: project the outline of where the part belongs, so placement is checked by eye before the cycle starts.
• Rework outlines: a detected defect's contour projected onto the casting shows the operator where to work, with no screen swivel.
• No-go zones around clamps and fixtures, visible from any side.

Ambient light is the usual killer: the overlay competes with the cell lighting, so shade the target or dim the lights above it. Surface finish matters as much as it does for depth itself — matte mid-gray faces carry the overlay cleanly, mirror-like faces scatter it, and very dark matte faces swallow it. Wavelength matters too: cameras that depth-code with an infrared pattern may not offer a visible overlay at all, which is the situation described in IR projector and depth camera: when the IR-cut filter blocks your plan. A fixed-pattern projector also cannot draw anything: speckle designs emit one pattern for depth only, with no overlay interface.

When the task needs only depth — parcel cubing, bin fill level, volume measurement — a fixed-speckle logistics camera covers it without any overlay interface, and the 3D sensors category's depth and volume types sort those models for you.

Where the pattern must sweep across the scene rather than sit still, a laser galvanometer stereo camera such as the MV-DLS300P-04 steers its projected light with fast steering mirrors.

Checks before you build on it

• Projection section in the SDK: a resolution query, a points-to-pixels mapping, and start/stop calls. Without the middle call there is no overlay on detected geometry, only a static projection.
• Overlay built at the queried resolution: a rescaled image shifts and blurs every drawn feature.
• 3D points in the camera's own frame: a transform applied twice, or not at all, moves every marker by the same offset — the cross check on a calibration board exposes it immediately.
• One measured span at the real standoff: the footprint budget above is arithmetic, but its input is your measurement with a ruler.
• Contrast checked with the cell lighting on: the overlay has to survive the lights production runs under.
• Overlay stopped before 3D capture: otherwise the first production frame comes back empty.

If the projection section of your SDK is empty, the camera cannot draw, and no amount of code will change that — the projector must be addressable from the start. The 3D sensors category and the camera finder filter by projector and measurement type, and the configurator started at the camera step walks the rest of the chain. More on the geometry behind this lives on the 3D topic page. If you are unsure whether a specific camera exposes the projection interface, send us the model number through the contact page and build the architecture on the answer, not on a guess.