3D visionApplication note

3D camera 2D template matching: locate before height inspection

Use the 3D camera's 2D image for template matching and position correction, then run the height check at a fixed ROI once 2D and depth are pixel-aligned.

LumaSpec Vision13 min read Share on LinkedIn

On this page 11 sections
  1. 01The decision: locate on the 2D image, inspect at a fixed ROI
  2. 02The one test that decides it: are 2D and depth the same pixels?
  3. 03The workflow: from texture to height result
  4. 04Worked example: the ROI margin your alignment error costs
  5. 05Tilt and the Z datum: what a 2D locate cannot fix
  6. 06Which camera family carries this pattern
  7. 07Profilers and line scan: the cross-section and web variants
  8. 08Pre-build checks and production pitfalls
  9. 09When to skip the 2D image and locate on the height map
  10. 10Where to go next
  11. 11Questions

Your part arrives at the station wherever the upstream process left it, but the height check needs to look at the same few square millimetres every cycle. The 3D camera at the station already captures a texture image alongside its depth map, and that texture can carry the locate. What follows is the decision rule for running template matching on a 3D camera's 2D image, correcting the part position, then inspecting height inside a corrected region: the alignment test that decides whether the pattern works, the ROI-margin arithmetic that sizes the region, and the camera families that carry it best.

The decision: locate on the 2D image, inspect at a fixed ROI

Yes — this is a standard pattern, and it fails for exactly one reason: a 2D image and a depth map that do not describe the same pixels. When the two streams are registered, template matching on the texture is the cheapest accurate locate on the station. The 2D image holds the sharpest edges the camera can produce, the matcher resolves in tens of milliseconds, and the height measurement code never moves. When the streams are not registered, the locate is true in one coordinate frame while the height region sits in another, and every bit of that mismatch lands on the measurement.

The whole decision therefore reduces to one question, answered by the test in the next section: do the 2D image and the depth map come from one optical path, or from two?

The one test that decides it: are 2D and depth the same pixels?

Two architectures exist, and the camera's type tells you which one you have before you power it up.

One optical path. The same sensor reads both the projected pattern and the part's texture, so every depth pixel corresponds to the same pixel in the 2D image, pixel for pixel. A template offset measured in the 2D image then applies directly to the depth map, with no extra calibration. Depth in these cameras is computed from the pattern inside the image itself — the projector pixel mapping behind structured-light depth walks through that chain. Projector-and-speckle structured-light families are the usual case, and the projected 3D camera below is the type to shortlist first for this pattern.

3D sensorsHikrobot MV-DPS200P-02 Projected structured-light 3D cameraMV-DPS200P-02
Type
Stereo / structured light
$8,829.00 · quotedView and quote

Two optical paths. A stereo camera reconstructs depth from a pair of sensors while the texture or colour image arrives from a separate sensor; a ToF camera ranges with its own modulator and sensor. Nothing in the raw streams lines up pixel for pixel. The SDK's registration function — the depth-to-texture alignment step in the stream set-up — has to run before a template offset means anything in depth. The paired IR and texture streams from a structured-light depth camera show what a registered pair looks like once that step is done.

Whichever architecture you have, measure instead of assuming. Clamp a sharp target where the part sits — a machined step, a checkerboard, even a coin. Capture both streams on one trigger, overlay the depth edges on the 2D image, and read the residual offset in pixels at the field centre and at the four corners.

Read the result against three bands. Under 1 px of residual: use the native alignment as it comes. Between 1 and 3 px and constant across the field: subtract the constant offset once and store it with the camera. Varying across the field: run the SDK registration or recalibration first, then re-test. Re-run the whole test after anything that moves the optics — a knocked bracket, a swapped lens, a large temperature swing.

The workflow: from texture to height result

  1. Trigger both streams as one event. One trigger, both grabs, part still. If a robot presents the part, it must be clear of the field before the capture — the snapshot position rule exists because a gripper in frame corrupts both the match and the reference plane.
  2. Train the template at nominal position. Pick a region of the 2D image with real edges: part corners, a machined step, a mark with geometry. Two corners at different distances from the template centre resolve rotation unambiguously; a single circle does not. A 100 to 200 px square is the practical band — large enough to average sensor noise, small enough to keep the search fast.
  3. Match with a rotation search bounded by the feeder spread. Parts arriving within ±15° need a ±15° search, not ±180°, and the search area dominates the matcher's run time. The match returns an offset dx and dy, an angle θ, and a score.
  4. Gate on the score. Record scores from 30 known-good parts at nominal, set the reject line just under the worst, and route low scores to a reject path. A weak match that passes can sit half a feature off, and the height check then measures the wrong place with full confidence.
  5. Move the ROI, not the measurement code. Rotate the fixed ROI by θ about the located datum and translate by dx and dy — or transform the point cloud the other way and keep the ROI fixed. Choose one convention and never mix them; the 2D versus 3D ROI article covers where each kind of ROI belongs. The match runs on the camera's own processing engine or on a vision controller beside it, so the height code stays frozen either way.
  6. Fit, measure, log. Inside the corrected ROI, fit a plane to the surface around the feature, measure the feature height against that plane, and write the match score next to the height result for every part.

Little here is custom code: template matching with rotation search is a stock function in every industrial vision library, and the height check is a plane fit plus a maximum. If you prototype the locate step in LabVIEW, the walkthrough on locating the LabVIEW IMAQ vision examples and testing a camera covers the first half hour — a live image on screen before any matching logic.

Watch the capture side of the cycle. Some cameras expose the 2D frame in the same window as the pattern frames; others take it afterwards, and the cycle then carries a second exposure plus transfer time. The 2D stream exposure and gain defaults article covers the settings that keep that gap short. If the texture on your part comes from glare rather than geometry — polished or plated surfaces — fix the lighting before trusting any score: a diffuse dome light over the station gives the matcher edges that stay put when the room lights change.

Worked example: the ROI margin your alignment error costs

Inputs for a realistic bench job. The feature is a 3 mm wide ridge on a machined bracket, height tolerance ±0.10 mm against the bracket's own flange. The camera resolves 2448 × 2048 px over a 120 mm wide field of view. Locate repeatability on the 2D texture is ±1 px, a normal figure for gradient-based matching on clean edges. The camera is a stereo type with registration enabled, and the step-target residual measured ±2 px.

s = FOV_width / pixels_width u = s × sqrt(u_locate_px² + u_align_px²) ROI_margin = 3u ROI_width = feature + 2 × ROI_margin

Pixel scale: s = 120 mm / 2448 px = 0.049 mm/px. Combined XY uncertainty: u = 0.049 mm/px × sqrt(1² + 2²) px = 0.049 × 2.24 = 0.110 mm. Margin at three sigma: 3 × 0.110 mm = 0.33 mm per side. The 3 mm ridge therefore gets a 3 mm + 2 × 0.33 mm = 3.66 mm ROI — program 4 mm, which is 82 px across at this pixel scale, plenty for the plane fit around the ridge.

Run the same numbers on a single-path camera: u = 0.049 × 1 = 0.049 mm, margin 0.15 mm, ROI 3.3 mm. The second optical path triples the margin, from 0.15 mm to 0.33 mm. On a 3 mm ridge both fit comfortably. On a 1 mm rib the stereo margin alone would span two thirds of the feature, and the right response is a tighter field of view or a stronger template — not a looser tolerance.

That pixel-scale trade — pixels per millimetre against the smallest feature you must resolve — is exactly what the resolution calculator runs, so use it when picking the camera:

Camera resolution for the smallest feature
Units
Pixels needed
–
Sensor size
– MP
Smallest sensor that fits
–
Pixel on the part
– mm
Pixels across the feature, that sensor
–

Pixels needed = field / feature × pixels per feature on each axis (field incl. margin). The lens must resolve the pixels too: Nyquist = 1000 / (2 × pixel µm) lp/mm.

Find cameras with enough pixels Open the full calculator

Tilt and the Z datum: what a 2D locate cannot fix

A template match corrects X, Y and in-plane rotation. It cannot see tilt, and tilt is a Z error. A 40 mm long part sitting 1° off level raises one end by 40 mm × tan(1°) = 40 × 0.0175 = 0.70 mm — seven times the ±0.10 mm tolerance of the worked example, with the locate score still perfect.

Two handles, in order of preference:

  • Fit the reference plane inside the corrected ROI, on the surface surrounding the feature. A plane fit absorbs any common tilt, so the height result reads the feature against its own flange — which is what most drawings specify anyway.
  • When the drawing measures against the machine datum instead, tilt has to leave the process: fixture the part, or measure tilt from the point cloud and compensate in software.

The Z datum itself drifts too — temperature, mount creep, projector aging — so put a trueness check on a schedule rather than on suspicion. For the artifact, a gauge block clamped in the fixture works on the line, and the guide on selecting a precision height gauge for shop layout and inspection covers the reference tools and their classes for bench work.

Which camera family carries this pattern

The four families differ in the same two dimensions the decision turns on: how much alignment work the pattern needs, and what kind of height result the depth hardware delivers.

Family 2D and depth paths Set-up work Height-check character
Projected structured light Pattern and texture usually share one sensor Step-target test only Fine Z detail at short working distance
Stereo (binocular) Two depth sensors, separate texture path Enable registration, re-test after knocks Depth error scales with the square of range
Time of flight Separate modulator and sensor Enable registration Millimetre-class depth: coarse bands and volume
Laser profiler Depth built line by line from laser triangulation Encoder timing when parts move Highest Z resolution of the four, along each line

Stereo's range law deserves a look before you pick a working distance: disparity noise is roughly constant, so depth error grows as Z² over baseline times focal length — double the distance, about four times the error. The same physics seen from the image side is in the structured-light versus stereo detail comparison, and the full family trade is in Industrial 3D camera selection: stereo, ToF or structured light.

A ToF camera turns phase into distance — millimetre-class results, right for volume and presence, wrong for a ±0.10 mm band. The indirect time-of-flight explainer covers the phase-to-depth chain, and the unambiguous range setting is the one to match to your depth span. Browse the shelves in the 3D sensor category once the family is settled.

The two cards below are the stereo and ToF types from the table's registration rows:

3D sensorsHikrobot MV-DLS300P-04 Laser galvanometer stereo cameraMV-DLS300P-04
Type
Stereo / structured light
$16,579.00 · quotedView and quote
3D sensorsHikrobot MV-DT01WMU ToF sensing cameraMV-DT01WMU
Type
Depth / ToF camera
Price on requestView and quote

Profilers and line scan: the cross-section and web variants

A laser profiler builds depth one line at a time, so the pattern changes shape rather than validity: the locate runs on the profile's intensity image or on the height data of the scanned strip, and the fixed ROI becomes a fixed window in profile coordinates. For a stationary spot check, a profile sensor such as the two below sweeps the part under the line and returns cross-section after cross-section.

3D sensorsHikrobot MIS-DP3850-01P 3D laser profile sensorMIS-DP3850-01P
Type
Laser profiler
$12,379.00 · quotedView and quote
3D sensorsHikrobot MV-DP4180-03P 3D Laser Profile SensorMV-DP4180-03P
Type
Laser profiler
$12,389.00 · quotedView and quote

When parts move, the encoder stitches profiles into a continuous height map and the locate moves into that map's machine direction. For web and extrusion work the snapshot pattern does not apply at all — a line-scan 2.5D system images the moving surface line by line instead. The MV-HLVS-16K100-L60 pairs 16K resolution with a 120 kHz sampling rate and a 600 mm emitting area length, which is the scale class for wide webs.

3D sensorsHikrobot MV-HLVS-16K100-L60 Line-scan 2.5D, 16K resolution, Sampling rate 120 kHz, Emitting area length 600 mmMV-HLVS-16K100-L60
Type
2.5D inspection
Price on requestView and quote

Pre-build checks and production pitfalls

Before you write code:

  • One trigger delivers both streams, and trigger-to-result time is measured on your hardware, not assumed.
  • For two-path cameras, registration runs and the step-target residual sits in your log. The SDK support check for depth cameras is the pre-purchase version of that question.
  • The field of view is sized on the swept envelope, not the bare part: a 60 × 40 mm part turning ±15° sweeps a 68 × 54 mm box (60·cos 15° + 40·sin 15° = 68.3 mm; 60·sin 15° + 40·cos 15° = 54.2 mm), an 87 mm diagonal instead of 72 mm.
  • The camera and bracket hold still through the exposure — mounting stillness during capture decides whether everything else in this list survives contact with the floor.

Once it runs:

  • A locate that passes in teach and drifts at run time is almost always a difference in how the image is acquired, not a matcher fault — why locate passes in teach and fails at run time is the first place to look.
  • The score gate is audited whenever the part, the lighting or the supplier changes; glare moves textures between batches more than wear does.
  • The trueness check runs on schedule — the infield correction procedure gives the on-line method.

When to skip the 2D image and locate on the height map

Some parts refuse to give a matcher a texture: polished surfaces that mirror the room, black plastics that swallow the projected light, mouldings whose only edges are soft radius transitions. Match on the height data instead — gradient and surface matching against a reference patch, covered in 3D surface pattern matching: locate parts from height data. Height matching is immune to reflectance and slightly coarser laterally, and it runs on the camera you already own. The broader 3D versus 2D system decision is worth a read before the hardware order either way.

And price the simple alternative honestly. If the 3D requirement collapses to presence, level, or a coarse height band, a fixture plus a smart camera or vision sensor with a backlight silhouette answers it with less integration work. The 3D path earns its keep once the tolerance heads into the 0.1 mm class and the reference has to come from the part itself.

Where to go next

Shortlist hardware in the camera finder, or walk the station end to end in the system configurator, which sequences camera, lens, light and mount in the order the decisions actually come. If you want the alignment numbers and the ROI margin checked against a real part, send the drawing and tolerance through the contact page. The remaining depth-camera guides, including the alignment and stream topics this page touches, sit on the 3D topic page.

Questions

Can the template match run on the height map instead of the 2D image?

Yes, and it is the plan B for shiny or black parts. Matching edges and gradients in the height data ignores reflectance entirely, so glare cannot move the locate. The trade is lateral sharpness: a clean 2D texture locates to a fraction of a pixel, while height-map edges are softer by roughly the camera's depth noise. Start on 2D texture; keep the height-map matcher configured and tested.

How much 2D-to-depth misalignment can a locate tolerate?

Treat the residual as measurement margin. At 1 pixel or less the locate carries the error alone; on a 120 mm field over 2448 pixels that is 0.049 mm per pixel, so a ±1 px locate needs about a 0.15 mm ROI margin. A ±2 px registration residual raises the margin to about 0.33 mm. Above 3 px, or any residual that changes across the field, run registration before production.

What does the 2D locate step add to the cycle time?

The match itself is small: a 150 px template inside a search window bounded to the feeder spread resolves in 10-50 ms on a typical controller. Capture matters more: if the 2D frame is exposed after the pattern frames rather than in the same window, the cycle gains a second exposure plus transfer time. Measure trigger-to-result on your hardware before quoting a cycle.

Where should the match score reject line sit?

Derive it from your own parts. Capture 30 or more known-good parts at nominal, record the score for each, and set the reject just under the worst score observed. On clean textured parts correlation scores commonly sit above 0.8. A score drifting toward the line usually means glare, contamination or a changed part — route those to the reject path instead of letting the height check measure a misplaced ROI.