Find 3D coordinates of a known template with a stereo camera

A textured template — a printed label, a dial marking, a sticker on a moving part — has to be found in 3D, marker-free, at 20-50 cm from the camera, for robot navigation or pick positioning. Stereo gives you that coordinate in one pass: match the template's natural SIFT or ORB features between the rectified images, take the median disparity, and convert it to X, Y and Z in millimetres. This guide gives you the pipeline step by step, the four calibration numbers it consumes, a worked example with the arithmetic shown, the accuracy you can quote at 200, 300 and 500 mm, and the checks that catch a bad median before it reaches the robot.

What you need

Four calibration numbers, one template image, and the standard functions of any vision library:

• Calibration: focal length in pixels f, principal point cx and cy, baseline B. A factory-calibrated 3D camera publishes them; a self-built pair gets them from a checkerboard stereo calibration, where B is the x-component of the translation between the two cameras.
• The template as a flat reference image with visible print texture — a 40 mm × 40 mm label in the worked example. Run the detector once, offline, and store keypoints plus descriptors.
• Matchers: SIFT or ORB detection, a brute-force descriptor matcher, RANSAC, and subpixel disparity refinement. Every OpenCV-style vision library ships all four.
• A working band that covers your distances. The example rig is comfortable from 150 mm to 500 mm and usable beyond; the accuracy section shows what the extra distance costs.

Read the calibration numbers first

f, cx, cy and B are the entire interface between your matcher and the millimetre world — every formula below is two multiplications and a division around them.

If you are sizing lenses before a calibration exists, predict f in pixels from the optics:

A 2.9 mm lens on a 1920 px wide image from a 7.44 mm (1/1.7"-type) sensor: f_px = 2.9 × 1920 / 7.44 = 748, which rounds to 750 px. Use the prediction to size the lens, then run the calibration and take its value.

The worked example uses an illustrative desk rig: f = 750 px, B = 75 mm between lens centres, cx = 960, cy = 540 on a 1920 × 1080 rectified image. Your calibration file prints your own four numbers, and the arithmetic that follows is identical.

Rectify the pair so a physical point lands on the same row in both images, then keep matches only when |v_left - v_right| ≤ 2 px. Sloping rows mean stale rectification or swapped camera order — stereo rectification: fix camera order and baseline. Depths that come out negative mean the baseline sign in your extrinsics is flipped — negative baseline after stereo calibration. Take f and B from the calibration, not from the lens barrel: stereo depth scale factor and focal length shows why the per-camera values differ from nominal optics.

The pipeline, step by step

1. Grab a synchronized, rectified pair. Both frames from the same instant, with exposure matched between the two cameras so descriptors stay comparable — set auto exposure per stereo camera in a depth rig covers the split.
2. Build the template set once, offline. Run SIFT (or ORB) on the flat template image and store keypoints with descriptors. ORB is faster and rotation-aware; SIFT holds up better under scale change as the camera approaches.
3. Locate the patch in the left frame. Detect features and match them against the template descriptors; the average pixel of the matched template keypoints is the patch centroid (u, v).
4. Match left keypoints to the right frame. Brute-force descriptor matching, keeping a pair only when its descriptor distance is at most 0.8× the runner-up (the ratio test; tighten to 0.7 on cluttered scenes).
5. Filter, then compute disparities. Drop matches failing the row check |v_L - v_R| ≤ 2 px, run one RANSAC pass or trim outliers by median and median absolute deviation, then compute d_i = u_L - u_R for each inlier.
6. Take the median. The median disparity shrugs off the few bad matches that always slip through. Set a floor: fewer than about 30 inliers on this patch means starved texture, blown highlights or motion blur — fix the frame or reject it, never steer on a thin median.
7. Convert with the calibration. Refine the median with the matcher's subpixel step, then apply Z = f·B/d and the two back-projections below. The single-point version of the conversion is worked through in depth from a matched stereo point pair.

Worked example: a 40 mm label at 300 mm

Keep the units paired: f, cx, cy and d in pixels; B and the results in millimetres. Mixed units are the classic silent error here.

• Patch size on the image. 750 × 40 / 300 = 100 px of print at 300 mm — enough for dozens of repeatable keypoints. At 500 mm it falls to 750 × 40 / 500 = 60 px, the floor of what a stable median wants; below about 50 px, lengthen the lens or move closer.
• Predicted disparity. d = f·B/Z = 750 × 75 / 300 = 187.5 px. Across the band: 281 px at 200 mm, 113 px at 500 mm. A dense block matcher needs its disparity window set over that span — stereo disparity to depth: set the depth window; feature matching needs no window, only overlap.
• Depth. Say the measured median, after subpixel refinement, comes back at 187.5 px: Z = 750 × 75 / 187.5 = 56250 / 187.5 = 300 mm.
• Back-projection. Centroid pixel (1030, 560): X = (1030 - 960) × 300 / 750 = 70 × 0.4 = 28 mm; Y = (560 - 540) × 300 / 750 = 20 × 0.4 = 8 mm.

One assumption to keep honest: the template is flat, so one depth describes it. A label wrapped around a can returns the median over a curved surface — right for locating the centre, wrong for measuring the can.

Accuracy you can quote

Take σd = 0.2 px from subpixel refinement (the jitter check below measures your own value); f·B = 750 × 75 = 56250 px·mm on the example rig:

• Depth error grows with Z²: double the distance, four times the error.
• Whole-pixel disparities would cost 1.6 mm of depth per pixel at 300 mm (90000 / 56250), which is why the subpixel step carries most of the accuracy — and why integer-disparity bias shows up as height overestimation (disparity quantization).
• Lateral error is gentler: σX ≈ Z·σu/f, so 0.5 px of centroid noise at 300 mm is 300 × 0.5 / 750 = 0.2 mm.
• The table is precision, not total error: calibration bias adds on top, and the ruler check in the last section bounds the sum.

Buy a calibrated head or build a pair

Buying the head buys the calibration: the camera outputs depth aligned to its own image, and the pipeline collapses to match-template, mask, median-of-depth. The MV-DPS200P-02 is a projected structured-light 3D camera and the MV-DLS300P-04 is a laser galvanometer stereo camera; both live in the 3D sensors category.

Building the pair keeps the form factor free: two identical global-shutter C-mount cameras on a rigid bar, hardware-triggered together, one checkerboard session. Shortlist from the area scan cameras category — a pair of MV-CU020-90GC heads puts a 2.5 MP, 1/1.7" global-shutter sensor with C-mount and GigE on each side of the bar.

Fit matched wide lenses; the example's 2.9 mm on a 1/1.7"-type sensor sees about 104° horizontally, so the calibration's distortion model does real work. Alternatives are one filter away in the camera finder or the C-mount lenses page.

When the natural-feature route lets you down

• Blank print. No gradients, no keypoints. Print texture onto the part, read depth from a dense stereo map under the label's mask, or move to a projected structured-light head that supplies contrast optically.
• Past about a metre. σZ at 2 m is 2000² × 0.2 / 56250 = 14 mm on the example rig. Holding 1 mm at 2 m needs B = 2000² × 0.2 / (750 × 1) = 1067 mm of baseline — a different machine, and long-range stereo depth accuracy with a wide baseline plus wide-baseline stereo camera selection are the route in.
• Occlusion and tilt. The median absorbs a few hidden keypoints; when half the patch is hidden, the inlier count falls under the floor and the frame should be rejected, holding the last pose instead.
• Repetitive print. Grids and barcode-like patterns self-match; the ratio test and the row check remove most of it, and the median plus the inlier floor catches the rest.
• You control the artwork. Then a coded fiducial marker beats natural features on speed and robustness in a single library call. The feature route above earns its keep exactly when the target must stay untouched.

Checks before you trust the numbers

• Ruler at 300 mm. Measure the camera-to-label distance and read Z; agreement within about ±1 mm on the example rig bounds calibration bias on top of the 0.32 mm precision.
• Distance sweep. Repeat at 200, 300 and 500 mm; error should grow as Z² grows (0.14 → 0.32 → 0.89 mm), and a jump at the near end usually means the wide lens needs a stronger distortion model.
• Thirty-frame jitter. Log the standard deviation of Z at a fixed pose; it should sit near the table value (0.3 mm at 300 mm), and jitter several times larger points to flickering exposure or starving texture.
• 45° rotation. Turn the label and re-shoot; a flat template holds its centroid within a few tenths of a millimetre laterally, and a drift of millimetres means the reference image does not match the print on the part.
• Production logging. Record |v_L - v_R| and the inlier count per frame and alert under the ~30 floor, so the robot never steers on a thin median.
• After any bump. Re-run the row check first, the ruler second; mechanical shock is the routine cause of a rig that used to be accurate.

Next step

Size the rig from the two formulas: pick the near edge, set the disparity ceiling at f·B/Z_min, then read σZ at the far distance and compare it with what the task needs. For a head-and-lens combination matched to a specific template, contact the applications team, or start at the camera step of the system configurator. The 3D topic page collects the rest of the stereo material.