3D flatness inspection: fitting a plane to a point cloud

A drawing calls out flatness of 0.04 mm, the 3D sensor hands you a million XYZ points, and nothing in the software is labelled flatness. The gap between the two is one least-squares fit and one subtraction. Here is the number you report, the plane that has to be fitted first, the tolerance it is judged against, and the three things that make a good part read badly: sensor noise, edge artefacts and missing data. The working math fits on one page, and every number below is one you can reproduce on your own cloud in an afternoon.

What number does a flatness inspection report?

A flatness result is one length in millimetres: the separation of two parallel planes that sandwich every measured point on the surface. Computed from a point cloud it is the peak-to-valley spread of the point-to-plane residuals, F = r_max − r_min, and nothing else belongs on that line.

In the drawing's language, flatness is a two-parallel-plane zone around the surface, and unlike parallelism it needs no datum — the surface judges itself. The strict geometric definition is the minimum zone: the smallest such separation over every possible plane orientation. A least-squares plane gives the spread for one particular orientation, which always equals or slightly exceeds the minimum-zone value. That bias points in the safe direction: if the least-squares number sits inside tolerance, the minimum-zone flatness does too. Report the value in millimetres to three decimals — 0.025 mm, not 25 µm — so it reads directly against the callout.

How do you fit the plane and compute the residuals?

Solve one least-squares problem, then take every point's height above the fitted plane. The residual of point i is r_i = z_i − (a·x_i + b·y_i + c) after fitting z = a·x + b·y + c, in the units of your cloud.

The 3 × 3 normal equations solve in microseconds even for a million points, so the fit is never the bottleneck. The residuals are signed heights above the plane — the same arithmetic as measuring a height difference from a point cloud, just against a reference you fitted instead of one you measured. If the sensor delivers a height map, convert it to calibrated XYZ as the height map to point cloud conversion shows, and fit in one frame and one unit, millimetres: the optical-frame-versus-camera-frame transform is where mixed frames and mixed units usually sneak in.

One division is worth knowing about. The perpendicular distance is r_i divided by sqrt(1 + a² + b²); on a machined face the slopes a and b sit around 10⁻⁴, so the factor is 1.00000001 and a 25 µm reading moves by less than a thousandth of a micrometre. Apply the division only to scans tilted more than about 1°, where a or b exceeds 0.017.

What tolerance do you compare the number against?

The tolerance printed on the drawing wins: a flatness callout of 0.05 mm means every measured point must sit inside two parallel planes 0.05 mm apart. Where the drawing invokes a general-tolerance standard instead, ISO 2768-2 class H allows 0.2 mm on a surface from 30 to 100 mm.

ISO 2768-2 straightness and flatness general tolerances; a specific flatness callout overrides them.

A die face carrying a 0.04 mm callout is judged at 0.04 mm even though ISO 2768-2 class K for that size allows 0.4 mm. Keep the units on one side: the fitted cloud is in millimetres, so a 0.04 mm limit is 40 µm of peak-to-valley, and the final comparison is one subtraction you can do in your head.

Which sensor should capture the surface?

Match the sensor to the surface and the tolerance: a laser profiler on a rail gives the lowest Z noise, a projected structured-light camera captures the whole face in one shot, and a line-scan 2.5D system judges parts while they move on a conveyor.

Z noise, not sensor resolution, decides what you can measure. Profiler-class sensors typically reach a few micrometres of per-point repeatability at short stand-off, which supports tolerances down to about 0.04 mm by the capability rule below; structured-light area systems trade some of that for full-field coverage with no motion stage, and line-scan systems add conveyor speed. The cameras inside a 2.5D head are ordinary area scan cameras; the heads themselves live in the 3D sensors range.

Current examples of all three classes:

For wide surfaces the line-scan option scales differently: the MV-HLVS-16K100-L60 reads 16K points across a 600 mm emitting area at a 120 kHz sampling rate, so a plate sweeping underneath at speed still gets a dense height profile.

A shiny machined face is a lighting problem before it is a fitting problem: direct highlights punch holes in the height map, and holes turn into fake peaks at the fit stage. A diffuse dome lights the face from every direction and keeps the map complete; the MV-PSA-12UM-C4D30-WDM pairs a 12 MP camera with a 300 mm white-light dome for exactly these parts.

How much does sensor noise raise the measured flatness?

Plan on a peak-to-valley noise floor near six times the sensor's per-point standard deviation: with σ = 2 µm, an ideally flat plate reads about 12 µm of flatness, in a band from 8 to 16 µm. Keep it under a third of the tolerance.

For independent Gaussian samples the extreme spread grows like σ·sqrt(2 ln N), about 7.5σ at 10,000 points; real scans have correlated neighbours, which pulls the effective sample count down. The practical band runs from 4σ on tightly sampled clouds to 8σ on sparse ones. The fitted plane itself is far steadier than the points: with N = 10,201 points over a 100 mm plate, the tilt uncertainty is about σ·sqrt(12/N)/L = 2 µm × 0.034 / 100 mm ≈ 0.0007 µm per mm, or 0.07 µm of rise across the whole plate. Noise lives in the residual spread, not in a wobbly plane.

t is the flatness tolerance. With σ = 2 µm the instrument is capable of a 0.04 mm decision; a depth camera whose per-point scatter is measured in centimetres is not, whatever its pixel count.

Which points should you delete before fitting?

Remove the points the sensor never trusted — zero and flagged returns, plus the 1–2 mm band at the field edge where stereo and structured-light depth typically rolls off — then fit, take sigma = 1.4826 × MAD of the residuals, cut points beyond ±5 sigma once and refit.

On a sound plate that cut removes well under 1% of the points, nearly all at corners and silhouettes; the stereo corner-noise article shows how those artefacts look in a corner region. Keep the cut at 5 sigma rather than the 3 sigma you would use on pure noise: once the plane is removed, a real crown lives within a few sigma of the median residual, and a tighter cut starts deleting true geometry and flattering the part.

Never interpolate across holes: filled holes read flatter than the truth, and the fit cannot tell a filled hole from a polished patch. If artefacts survive the first pass, a RANSAC fit rejects them by consensus rather than by threshold; the least-squares-versus-RANSAC comparison shows when that extra machinery earns its keep.

What does a full worked measurement look like?

A 100 × 100 mm milled plate scanned on a 1 mm grid: the least-squares fit gives a = 0.00012 and b = −0.00008, the trimmed residuals span −0.011 to +0.014 mm, so F = 0.025 mm — a pass against the 0.04 mm callout with 0.015 mm of margin.

1. Cloud: 101 × 101 = 10,201 points, calibrated, in millimetres.
2. First fit: a = 0.000120, b = −0.000080, c = 49.997 mm — the fixture adds a 0.012 mm tilt across the plate (0.00012 × 100 mm).
3. Robust spread: MAD = 6 µm, so sigma = 1.4826 × 6 µm = 8.9 µm and the cut sits at 5 × 8.9 ≈ ±44 µm — wider than the noise alone because the crown is in the spread too.
4. Raw worst residual: −0.090 mm on one corner, so F_raw = 0.090 + 0.014 = 0.104 mm. The cut removes 3 corner points; the +0.014 mm crown survives.
5. Refit: residuals now span −0.011 mm to +0.014 mm, and F = 0.014 + 0.011 = 0.025 mm.
6. Decision: 0.025 mm against 0.04 mm — pass, with 0.015 mm of margin. Coverage is 10,198 of 10,201 points and the σ = 2 µm noise floor is about 12 µm, under the 13.3 µm one-third threshold, so the margin is real rather than noise luck.

When is the least-squares flatness number the wrong answer?

Least-squares flatness always meets or exceeds the true minimum-zone value, so a pass is safe while a narrow fail may still pass a minimum-zone evaluation. It also absorbs missing data silently: every unmeasured hole is a gap the plane leans into, hiding the real peak.

Three habits cover the risks. Quote coverage with the number — the worked cloud reports 99.97%, and a cloud below about 95% coverage is geometry you have not measured, not flatness you have. Re-evaluate before scrapping: a minimum-zone fit minimises the spread over all orientations and never returns more than least squares, and on the worked cloud it gives 0.021 mm, closer to the truth and still inside tolerance. Watch the surface type: flatness only means something on a nominally flat face, and a bowed plate reports its bow as flatness — often exactly the defect you are hunting, and the point-cloud plane curvature article separates a smooth bow from local waviness. If the drawing controls that bow as profile of a surface instead, flatness is the wrong characteristic to quote.

Parts larger than one field of view are stitched from several placements, and each placement adds the motion error of the stage to the combined cloud; the point-cloud stitching guide walks that pipeline and its error budget.

What belongs on the measurement report?

A report someone can re-run has five lines: flatness against tolerance, fitted area and coverage, sensor and per-point noise floor, grid spacing, and fit method — least squares with one trim, or minimum zone. Anything less is a number without a pedigree.

• Flatness F = 0.025 mm against a 0.04 mm callout — the decision on the first line.
• Fitted area 100 × 100 mm minus a 1 mm edge band; coverage 10,198 of 10,201 points.
• Sensor and σ = 2 µm noise floor — this sets the 13.3 µm capability threshold.
• Grid 1 mm, finer than the shortest waviness the drawing cares about.
• Fit: least squares with one ±5 sigma MAD trim; minimum zone on request.

To size a sensor before you buy, invert the capability rule — σ ≤ t/18 for your tolerance — then filter the camera finder by type and interface or step through the camera configurator from the part you need to judge. The other 3D measurement guides build on this fit, from height maps to stitched clouds, and if the part resists every configuration above, the contact page reaches the applications team.