Subpixel edge measurement of cutting tools to micrometer precision

Tool presetters and grinding stations judge a tool's edge to a couple of micrometers from a single camera image. The claim is real, but it does not come from a bigger sensor or a pricier lens alone: it comes from an object-side pixel of five to seven micrometers, subpixel edge extraction running near a tenth of a pixel, a telecentric lens, and a calibration that holds across the whole field. This guide works the numbers for a 12 mm end mill, shows where each micrometer of uncertainty comes from, and ends with the parts decision: which pixel scale, which lens type and which checks to run before you order.

The pixel floor and what subpixel extraction buys

An edge located to the nearest whole pixel carries up to ±1 px of error, so the pixel sets the floor. Measure the 12 mm part with 6.5 µm object-side pixels and the best integer-pixel reading is ±6.5 µm before anything else contributes. Subpixel extraction fits the grey-value profile across the edge and interpolates where it crosses its midpoint. Published subpixel edge studies converge on 0.05-0.2 px of repeatability for a clean, well-lit edge; 0.1 px is a fair planning value for a backlit silhouette. The floor drops to a tenth of a pixel, and the field-of-view arithmetic becomes the design:

Work the formulas backwards before choosing parts. For a ±2 µm target, keep the object-side pixel between 4 and 7 µm so edge noise sits near half a micrometer and the rest of the budget has room. The guide to calculating object space pixel size from magnification walks the conversion step by step, and estimating edge position accuracy from image sampling shows what image noise does to that tenth of a pixel.

Where the micrometers go: the five-term error budget

Edge noise is one term of five, and the largest term - rarely the sensor - decides pass or fail:

1. Edge extraction: 0.1 px × object pixel, times √2 because a diameter uses two edges.
2. Calibration residual: a field calibration fitting to 0.05 px leaves about 0.05 px × object pixel of scale error.
3. Depth-induced magnification error: telecentricity in degrees × depth error; an entocentric lens runs near 1.3% per millimetre of depth error instead.
4. Thermal growth: steel expands about 12 ppm/°C, roughly 0.14 µm/°C across the part.
5. Fixturing: the axial placement of the measured feature, which multiplies term 3.

Combine independent terms as a root-sum-square and hold the total under 10-20% of the tolerance you must judge.

Worked example: the end mill at ±2 µm

The case: the 12 mm carbide end mill read as a backlight silhouette, the classic presetter job.

Inputs - part and goal from the job, sensor figures standard for the class, the rest planning values for a well-built station:

• Field of view: 16 mm, the part plus margin so edges sit off the border.
• Sensor: 2/3" 5 MP class, 2448 × 2048 pixels at 3.45 µm pitch, width 8.45 mm.
• Subpixel factor: 0.1 px.
• Calibration residual: 0.05 px.
• Telecentricity: 0.1°; spec sheets commonly run 0.05-0.2°.
• Depth slack: ±1 mm as first built, ±0.5 mm with the tool tip referenced to focus.
• Temperature: ±2 °C, steel part.

The arithmetic:

1. Magnification: m = 8.45 mm ÷ 16 mm = 0.53x.
2. Object-side pixel: 3.45 µm ÷ 0.53 = 6.5 µm.
3. Edge noise: 0.1 px × 6.5 µm = 0.65 µm per edge; two edges give 0.65 µm × √2 = 0.92 µm.
4. Calibration: 0.05 px × 6.5 µm = 0.33 µm.
5. Telecentricity: tan 0.1° ≈ 0.0017, so ±1 mm of depth shifts an edge by 1.7 µm.
6. Thermal: 12 mm × 12 ppm/°C × 2 °C = 0.29 µm.
7. Stack: √(0.92² + 0.33² + 1.7² + 0.29²) = √3.93 ≈ 2.0 µm, exactly on the limit.
8. Reference the tip so measured features sit within ±0.5 mm of focus: the depth term halves to 0.87 µm and the stack falls to √(0.85 + 0.11 + 0.76 + 0.08) = √1.80 ≈ 1.3 µm.

Decision: ±1.3 µm passes the ±2 µm goal with a third of the budget in reserve. The depth × telecentricity term is about half the stack, so fixturing, not the camera, is where the next improvement lives. Against a ±25 µm grind tolerance the 10:1 rule allows 2.5 µm of uncertainty, and the station qualifies. A 1.1" 12 MP sensor halves the pixel to 3.9 µm (14.1 mm ÷ 16 mm = 0.88x; 3.45 µm ÷ 0.88 = 3.9 µm) and trims the edge and calibration terms, but the 1.7 µm depth term still dominates. Pixel count alone cannot buy the last micrometer.

Why the measuring lens is telecentric

Term 3 is why presetters image through telecentric optics. An ordinary lens magnifies by perspective: nearer features image larger. Take a 35 mm lens at 110 mm working distance: m = f ÷ (WD − f) = 35 ÷ 75 ≈ 0.47x. Move the feature 1 mm nearer and m = 35 ÷ 74 ≈ 0.473x, a 1.3% change, ±150 µm on this part - two orders of magnitude over budget. A telecentric lens accepts only the near-axial cone of rays, so magnification stops depending on depth; the residue is telecentricity, quoted in degrees. For viewing rather than gauging, a 35 mm, 2/3", 6 MP C-mount lens is the economical choice.

Telecentrics trade that stability for size, cost and clearance at the front cone; the entocentric vs telecentric comparison sets out when a standard lens still wins. If a station stays entocentric, size the focal length from the field: at 0.53x and 200 mm working distance, f = 200 × 0.528 ÷ 1.528 ≈ 69 mm, about 70 mm on a 2/3" sensor.

For gauging, two 2/3" C-mount telecentrics bracket the mounting options:

The LS-TCL-2118 works at 108 mm and keeps the station compact. The LS-TCL-1972 holds the same 1x magnification at a 440 mm working distance to clear spindle and clamping hardware, with a barrel 177.5 mm long. Whichever you pick, validate depth of field against the tolerance - the telecentric depth of field guide gives the test - then browse the telecentric lens category or the lens finder for other magnifications.

The camera: mono, pixel scale and shutter

Shoot mono for gauging: demosaicing a colour sensor invents pixel values between photosites and softens exactly the gradient a subpixel fit feeds on. The sensor sets the pixel scale from the example - 5 MP on 2/3" for the compact build, or 12 MP on 1.1" (4096 × 3.45 µm, 14.1 mm wide) for a 3.9 µm pixel over the same 16 mm field. Shutter follows motion: parked in the spindle, rolling and global shutter image a static scene identically; anything moving or spinning wants a global shutter and an exposure short enough that motion stays under one object pixel. Two USB3 C-mount cameras cover both builds:

The MV-CH120-11UC is the 12 MP, 1.1" global shutter option from the arithmetic above. The MV-CS200-10UM is a 20 MP mono USB3 camera on a 1" sensor. Image a 1" 20 MP class sensor at 0.5x magnification and its 2.4 µm pitch yields a 4.8 µm object pixel over the field the arithmetic gives. Filter the area scan camera category by mono, resolution and mount, or start from the camera finder.

Calibration and backlight hold it across the field

Camera calibration makes a reading true anywhere in the image: it maps lens distortion and sensor misalignment out of the pixel grid, so an edge at the field corner measures as true as one at the centre. Use a target that spans the whole field, not a small slide at the middle - the calibration target coverage guide sizes it - and hold the fit residual near the 0.05 px the example assumes; the calibration residual targets article gives average and maximum values worth accepting. The distortion spec tells you how much there is to map: telecentrics commonly sit at 0.01-0.1%, which is 1-12 µm across this part before correction, and the distortion correction guide covers the remap.

Lighting decides whether a tenth of a pixel exists to extract. A collimated backlight turns the tool into a silhouette whose grey gradient spans one or two pixels - the profile subpixel fitting wants. The collimated vs diffuse backlight guide compares the choices and the backlight finder lists the hardware. Set exposure from the edge histogram: bright enough that the gradient crosses mid-grey, short of clipping.

Checks before you order

• Field of view at least 1.2 × the largest feature - 16 mm for this tool - so edges stay inside the calibrated area.
• Object-side pixel of 4-7 µm for ±2 µm work; under a third of whatever tolerance you must judge.
• Telecentricity low enough that depth slack × tan(angle) stays under a third of the budget: 0.1° at ±0.5 mm gives 0.87 µm.
• Mono sensor behind a collimated backlight, exposure set from the edge histogram.
• Calibration target larger than the field, residual accepted at 0.05 px or better.
• Tip-referenced fixturing holding measured geometry within ±0.5 mm of focus; in the example this halved the whole stack.

The decision, restated

For ±2 µm class tool measurement: a 2/3" 5 MP or 1.1" 12 MP mono camera, a 0.5-1x telecentric lens on a field about a third larger than the part, a collimated backlight, a field-spanning calibration held near 0.05 px, and fixturing that keeps measured geometry within half a millimetre of focus. That stack landed at ±1.3 µm on the example - the same band as a bench micrometer, which the micrometer vs dial caliper accuracy guide puts numbers to - but it repeats across every feature in one image without touching the part.

If your field or tolerance differs, redo the eight lines of arithmetic above; five minutes tells you which part to spend on. The small-part dimensional measurement guide takes the same logic to smaller fields, and the learn hub collects the rest of the measuring set. To specify optics, start the lens configurator at your field of view.