Vision metrology: the 10 to 1 accuracy and repeatability rule

A vision station asked to gauge a part must answer one question before any camera is chosen: how good is good enough? The answer comes from the tolerance on the drawing, not from megapixels. The rule that has sized measuring instruments for a century applies to machine vision unchanged: the gauge should be about ten times better than the tolerance it polices. This page turns that rule into numbers you can order parts against — the tick (your pixel or subpixel increment), the accuracy a calibration can hold, and the one-sigma repeatability the station must demonstrate — then matches them to the optics and light that deliver them: a telecentric lens and a collimated backlight, with a worked example and the arithmetic shown.

The rule: ten ticks across the tolerance band

Whatever measures a part should be roughly ten times better than the tolerance it checks — ten times as accurate and ten times as repeatable. A vision system's scale is made of pixels, so the rule first fixes how small one tick of that scale must be.

Span means the full band, not the ± value: a ±0.005 in tolerance is a 0.010 in span. Calibration can only be nudged one tick at a time, and the worst case is half a tick of mismatch at each end of a dimension — that is where the span/20 accuracy figure comes from. Repeatability gets the full tick because nothing downstream corrects it.

Two relaxations circulate on real lines. Some shops allow repeatability at 5 to 1 — span/5 — where marginal parts can be pulled and re-gauged. Others quote the composite rule, accuracy plus repeatability under a third of the span; the 10 to 1 split already satisfies it (span/20 + span/10 = 0.15 of the span). Design to 10 to 1 for both, and spend a relaxation only where a re-measurement path genuinely exists.

What the tick is: pixels, subpixels and the edge profile

The tick is the smallest increment your scale can express. In vision that is the object-space pixel: sensor pixel pitch divided by magnification on a telecentric set-up — calculate object space pixel size from magnification runs it in one line — or field of view divided by pixel count on any lens.

A part edge almost never lands on a pixel boundary. Each pixel reports the average brightness across its area, so the edge shows up as a gray ramp spread over three or four neighbouring pixels. Edge algorithms fit that ramp — gradient peak, centroid, second derivative — and quote a position to a fraction of a pixel; 1/5 to 1/10 of a pixel is realistic on a clean backlit edge, because the profile is what the fit consumes.

Budget so the raw pixel carries the tolerance and subpixel is margin: aim for a pixel at or below one fifth of the tick, so at least five pixels fall inside each tick. Interpolation then refines a measurement the sampling already supports. Estimate edge position accuracy from image sampling shows how much of that refinement survives real noise.

Worked example: ±0.005 in on a 0.100 in pin

A turned pin, nominal 0.100 in (2.54 mm), tolerance ±0.005 in (±0.127 mm). The budget in one pass:

• Span: 0.010 in = 0.254 mm.
• Tick: 0.254 mm / 10 = 0.0254 mm = 25.4 µm.
• Accuracy: 25.4 µm / 2 = 12.7 µm.
• Repeatability (1σ): 25.4 µm, or 50.8 µm at the 5 to 1 relaxation.
• Per edge of a two-edge diameter: 25.4 µm / √2 = 18.0 µm, because both edges scatter independently.

Now the optics, top down. A common 5 MP machine-vision camera delivers 2448 × 2048 pixels at 3.45 µm pitch; a 1x telecentric lens makes the object-space pixel 3.45 µm. The pin spans 2540 µm / 3.45 µm = 736 pixels, one tick spans 25.4 µm / 3.45 µm = 7.4 pixels, and the field of view is about 8.4 mm across — three times the part, with room for fixturing. Subpixel at 1/5 pixel would place an edge to 0.69 µm, but the environment, as later sections show, stops you near 2 µm long before sampling does.

The decision these numbers make: a 5 MP-class camera behind a 1x telecentric passes this tolerance on sampling with margin. Do not buy sharper sampling; spend the margin on the light, the fixture and the calibration standard.

The calculator behind that button runs the same arithmetic in one step — field of view and smallest feature in, pixel count out — and hands the result to the camera finder.

Accuracy is calibrated, repeatability is earned

Accuracy is a procedure. Place a certified standard — a gauge pin or a glass scale — exactly where the part sits, adjust the pixels-to-millimetres scale until the reading matches the certificate, and the residual is at most half a tick at each end of the dimension. That is why accuracy earns the tighter allowance: a good standard and a disciplined routine buy it, and the routine repeats whenever temperature or mounting changes.

Repeatability is earned on the floor. Sensor noise, vibration, seating variation, edge contrast and the algorithm's own variance all land here, and no calibration cancels them. Measure it without flattery: present the same part 30 times, re-fixtured every time, and take one sigma of the readings. That scatter — not the pixel pitch — is the station's repeatability.

Machine-tool spec sheets draw exactly this line between the two properties; positioning accuracy vs repeatability on CNC specs is a clear refresher on reading both, and the discipline transfers one-to-one to a vision station.

The lens: position error is magnification error

An ordinary entocentric lens changes image size with distance: a part seated 1 mm closer grows by Δz / WD. At a 108 mm working distance, 1 mm of seating scatter is 1/108 = 0.93 %, and on a 2.54 mm pin that is 23.5 µm — 93 % of the whole 25.4 µm tick spent before noise gets a vote.

Run that check with your worst-case seating variation. At gauge tolerances it fails almost every time, so the lens goes telecentric: constant magnification through the working range, so height variation stops converting into size error. Before ordering, confirm the lens's telecentricity range covers your Δz, and that its distortion is small and stable enough to calibrate out. When you need a telecentric lens for gauging walks the borderline cases, and entocentric vs telecentric lenses: when to use each lays out the underlying trade.

For the 2/3" example above, the 1x choice is the LS-TCL-2118, a C-mount telecentric lens with a 108 mm working distance:

When the cell needs standoff from a moving or hot process, the LS-TCL-1972 is also a 1x C-mount telecentric for 2/3" sensors, with a 440 mm working distance and a 177.5 mm long body — lay out both in the cell before committing floor space.

Distortion does not vanish with telecentricity; it becomes small and repeatable, which is exactly what calibration needs. Fit it instead of assuming zero, as lens distortion correction for pixel-to-mm measurements shows, and compare candidates on geometric distortion rather than TV distortion when the number feeds a tolerance. Browse telecentric lenses by magnification, mount and working distance, or filter the lens finder.

The light: a collimated backlight makes the edge

Gauging wants a silhouette. Backlight the part and the edge becomes a full-contrast light-to-dark transition: you measure the physical shadow — no radius highlight, no surface-finish bias, no question about which surface the software found.

Collimate that backlight. A diffuse panel emits over a wide cone, so the shadow edge carries a penumbra and the gray ramp spreads across extra pixels. A collimated panel emits near-parallel rays, the ramp stays as steep as diffraction allows, and the subpixel fit works on a clean profile. Collimated backlight or diffuse backlight for edge gauging works through the trade — collimated for thin parts and steep edges, diffuse when a thick part would throw its own shadow onto the field.

The LS-BKL-0830 is a 120 x 120 mm collimated backlight on 24 V:

Sources built to sit opposite a telecentric lens are the sharper pairing at small ticks. The LS-CLM-0079 is a telecentric collimated source with a 90 mm beam:

Colour and pulse width matter as much as geometry. Blue near 450 nm cuts the diffraction spot about 29 % against red at 630 nm — the Airy radius scales linearly with wavelength — and a blue band-pass filter on the lens rejects shop ambient. Motion sets the strobe: at 100 mm/s the edge walks 3.45 µm in 34.5 µs of exposure, so pulse the light at 30 µs or less and the smear stays inside one object pixel, with the camera's global shutter open only for the pulse.

One safety habit: a collimated source concentrates its power into a narrow beam, so never sight along the beam path while a station is powered, and interlock the enclosure on the line. Both families sit in backlights and collimated lights; the lights finder filters by size, colour and voltage.

Where pixels stop helping: diffraction and the floor

Two limits sit below the pixel arithmetic. The first is diffraction: a lens focuses a point of light to an Airy disc, not to a point.

At f/2 with 500 nm light, R = 1.22 × 0.5 µm × 2 = 1.22 µm, about 0.000048 in. Blue at 450 nm through an f/0.8 optic pushes the theoretical radius to 0.44 µm. Extra megapixels do not move either number.

The second limit is the factory. Vibration, thermal drift, part finish and presentation scatter stack on top of diffraction, and the practical repeatability of a production vision station lands near 0.00008 to 0.0001 in — 2 to 2.5 µm at one sigma. That floor is why the 5 MP camera in the worked example is already sufficient: sharper sampling refines a number the environment has blurred.

The floor also sets the smallest tolerance a single station can police at 10 to 1, because a 2 µm tick sits at the floor and spans near 20 µm mark the boundary:

Checks before you order

The whole method in seven lines:

1. Write the tolerance as a span — ±0.005 in is 0.254 mm — before dividing anything.
2. Compute tick, accuracy and 1σ repeatability from the formulas above; a two-edge dimension carries tick/√2 per edge.
3. Check sampling: five pixels per tick minimum, object pixel = sensor pitch divided by magnification.
4. Check position error: (Δz / WD) × dimension below tick / 3, otherwise go telecentric with a range covering Δz.
5. Size the light: collimated backlight covering the field of view, blue, strobed at 30 µs or less for moving parts.
6. Order a certified standard near the nominal dimension; calibrate in the fixture, at line temperature.
7. Run the 30-presentation study and sign off only when 1σ lands at or under the tick.

With the drawing in front of you, run the chain the other way in the system configurator or start from the light step. If a tolerance sits near the 25 µm boundary and you want the station plan checked, send the drawing through the contact page. More optics guidance is collected on the lenses topic page.