You have a point cloud of a nominally flat part and three numbers to produce: tilt, flatness, and the height of each feature above the surface. One operation gets all three — select a few disjoint regions, pool their points, and fit a single plane by least squares. The fit returns the plane coefficients, a residual RMS in mm, and, once a feature region is compared against that plane, a height difference in mm. This how-to covers the region choice, the two equivalent ways to pose the fit, the Z-image shortcut that keeps the whole job in 2D, and a worked example that lands at 0.05 mm RMS on a 100 mm plate.
What you need
Four things:
- A point cloud or height map with metric Z in mm and a known pixel scale in mm per pixel. Grid and spacing choices are their own subject; see 3D point cloud x y z resolution settings without distortion.
- Three to five disjoint regions on the surface, each holding at least 100 points.
- A least-squares routine. A spreadsheet fits z = ax + by + c for a bench demo; your vision library or ten lines of Python handle millions of points.
- The sensor's single-point Z repeatability from its data sheet — the yardstick your RMS is judged against.
Any sensor in the 3D sensors category that exports metric XYZ works, and the 3D camera finder filters them by type. Laser profilers deliver the height map natively, so the Z-image shortcut below is their natural path; structured-light and stereo cameras deliver a full cloud you project to a Z image yourself or fit directly in XYZ.
- Type
- Laser profiler
- Type
- Stereo / structured light
Both routes end at the same deliverables — coefficients, RMS, height difference.
Two ways to pose the same fit
If the data is a height map — one z per grid cell, noise in Z only — fit vertically:
If the data is a raw cloud with comparable noise in all three axes, fit point-to-plane with a unit normal:
The second form is solved by the eigenvector of the smallest eigenvalue of the covariance about the centroid. The forms differ only in how they weight residuals: a vertical offset r_z is a perpendicular offset of r_z·cos θ, with θ the tilt from horizontal. At 10° tilt, cos 10° = 0.985, so the fits differ by about 1.5%; below roughly 10° of tilt they return the same plane for practical purposes.
Decision: height map from a profiler — vertical form. Raw cloud with isotropic noise — point-to-plane. Either way, fit all regions in one pooled pass: four separate fits averaged together throw away the 100 mm baseline between patches and weight every patch equally regardless of point count.
The Z-image shortcut: keep the job in 2D
Convert the cloud to a Z image and region selection becomes ordinary 2D: draw rectangles or polygons on the image, and the XY of every selected pixel comes from the grid itself — x = column·s, y = row·s, with s the mm-per-pixel scale plus the cloud's origin offsets. There is no separate XY transform to construct; the grid is the transform. That removes exactly the step where implementations stall: the question of how to build a matrix for X and Y has no answer because none is needed.
The shortcut has one hard limit — the surface must be single-valued in Z. Near-vertical walls, folded edges and stacked levels either miss Z or alias into spiked columns that pollute the fit, so keep patches two pixels clear of them. While total tilt stays under about 10°, the Z-grid fit matches the raw-XYZ fit to the 1.5% level worked out above; beyond that, or with vertical features in play, fit the raw cloud.
The reverse conversion, a height map back into a cloud with true XYZ spacing, is covered in 2.5D height map to 3D point cloud: pixels to XYZ. And if a nominally flat plane comes out not flat, suspect the frame first: depth camera point cloud flat in XY: fix the optical frame.
Steps from cloud to accepted plane
- Load the data and set the scale. Confirm Z is metric mm and note the pixel scale s. If the cloud passed through an export, spot-check points against the viewer render — exports can diverge for frame reasons covered in why exported point clouds differ from the viewer render.
- Draw the regions. Three to five disjoint patches, each at least 50×50 pixels, centers spread across the part in both X and Y, every patch at least two pixels clear of edges, holes and walls. Pulling the points inside a mask is the same operation as cutting a 2D ROI — see extract a region of interest from an image after detection.
- Collect the triples. Export each patch as (x, y, z) in mm. A 50×50 patch is 2,500 points at minimum.
- Fit once on all patches pooled. Height map: z = ax + by + c by vertical least squares. Raw cloud: point-to-plane via the covariance eigenvector.
- Normalize and read the tilt. Unit normal n = (−a, −b, 1)/√(a² + b² + 1); tilt angle = atan √(a² + b²).
- Score the residuals. rᵢ = zᵢ − (a·xᵢ + b·yᵢ + c); RMS = √(Σrᵢ²/N); record max |r|; flag every point with |r| > 2×RMS.
- Inspect the flags, refit once. Flags ringing a hole or edge mean the mask was generous: shrink two pixels and refit. Flags clustered on actual damage stay — that is measurement, not noise.
- Measure the feature. Height difference = mean z of the feature region minus the plane height at the feature's centroid.
- Verify the fit is honest. Split the patches into two halves and refit both: slopes should agree within about 10% on a flat part, and a bigger gap means one patch sits on a different surface. Repeat the capture a few times — the RMS should hold within a few percent.
Worked example: 0.05 mm RMS on a 100 mm plate
Setup: a machined aluminum plate under a laser profiler, height map at s = 0.1 mm/px. Four patches of 60×60 pixels — 6×6 mm, 3,600 points each — centers spread over roughly 100×80 mm. Pooled: N = 14,400 points.
The pooled vertical fit returns:
z = 0.0020·x − 0.0010·y + 12.500 mm
Tilt: √(0.0020² + 0.0010²) = √0.000005 = 0.00224 mm/mm, or 0.13° — far under the 10° mark, so the height-map form is valid.
Spot-check one point at (x, y) = (40.0, 20.0) mm with measured z = 12.620 mm:
plane height = 0.0020 × 40.0 − 0.0010 × 20.0 + 12.500 = 0.080 − 0.020 + 12.500 = 12.560 mm residual = 12.620 − 12.560 = +0.060 mm
RMS across the pooled set, with Σrᵢ² = 36.0 mm²:
Max |r| = 0.12 mm. The 2×RMS flag at 0.10 mm catches 47 points — 0.3% of the set — ringing the coolant hole; the mask shrinks two pixels and the refit settles at RMS = 0.049 mm.
Feature measurement: the raised pad's region averages z = 12.946 mm at centroid (20.0, 30.0). Plane height there = 0.0020 × 20.0 − 0.0010 × 30.0 + 12.500 = 12.510 mm, so the height difference = 12.946 − 12.510 = 0.436 mm.
Decisions: the plate is flat to 0.05 mm (1σ) after tilt removal, and the pad stands 0.44 mm proud. Against a drawing calling 0.10 mm flatness and a 0.40 ± 0.10 mm pad, both pass; the line gate runs at |r| > 0.10 mm.
Judge the fit: RMS, max deviation and gates
RMS after the fit is flatness at one standard deviation, with tilt and level removed. Never compare it to the raw Z spread of the cloud — that still contains the tilt.
Three checks turn RMS into a verdict:
- Stability. The relative standard error of an RMS over N Gaussian residuals is about 1/√(2N). At N = 14,400 that is 1/√28,800 ≈ 0.6%; at the 100-point floor it is 7%. Two captures agreeing within a few percent means the surface, not the mask, is being measured.
- Sanity against the sensor. Expect RMS within about 2-3× the data-sheet single point Z repeatability; that multiple absorbs curvature inside the patches and leftover outliers. A 10× gap means something structural — wrong units, a wall inside a patch, or a real bow.
- Curvature. A patch of width L on a surface of radius R carries sagitta L²/(8R). A 6 mm patch on R = 1 m carries 0.0045 mm — invisible against 0.05 mm. On R = 100 mm it carries 0.045 mm and the RMS is measuring the bow, which no refit removes; the diagnosis continues in point cloud plane curvature: diagnosing a bowed flat surface.
Set the reject gate from the drawing, not from the fit: warn at |r| > 2×RMS, fail at the tolerance band. The same fit aimed at a flatness specification instead of a pad is the workflow in 3D flatness inspection: fitting a plane to a point cloud.
When least squares is the wrong tool
Least squares assumes every pooled point belongs to one plane. Three situations break it:
- Outliers beyond a few percent — dust, multipath reflections, background leaking through an edge. A stray cluster drags the plane toward itself and inflates RMS everywhere. Segment with RANSAC or a trimmed fit, then apply least squares to the inliers for the final coefficients; the trade-offs are laid out in plane fitting on 3D point clouds: least squares vs RANSAC.
- Curved surfaces — a plane answers flatness and tilt, not shape. Once the RMS says the bow dominates, move to curvature or a second-order surface fit.
- Vertical or multi-level geometry — no single z per (x, y). Fit the raw cloud point-to-plane, region by region on the wall itself.
Common mistakes
- Fitting each region separately and averaging the planes. The average has no least-squares property, ignores point counts, and wastes the baseline. Pool the points.
- Regions strung along one line. The cross-line tilt is then set by the 6 mm patch width instead of the 100 mm spread — about 16× less sensitivity. Spread centers in 2D.
- Mixed units. Z in metres with x, y in mm shrinks every slope coefficient by 1,000×; the tilt reads near zero. Read the tilt angle back and sanity-check it against the part.
- Border pixels inside the patches. They mix with background and post the largest residuals in the set; two pixels of margin removes them.
- Comparing RMS between different masks. The number tracks the mask as much as the surface; freeze the mask once accepted.
- Fitting on an 8-bit viewer preview. Fit only on the float metric Z.
Kit that runs this job
In production the fit should run where the capture runs. The MV-VC5525-256G70-NN vision controller pairs a 12th-gen Intel Core i5 with 32 GB of memory, 5+ LAN ports with PoE camera ports and 8 built-in light channels — capture, fit and gate in one box. A pooled fit over tens of thousands of points is single-digit-millisecond work; the cloud transfer over GigE dominates the cycle.
- LAN ports
- 5+ LAN ports
- USB 3 ports
- 3-4 USB 3 ports
- PoE
- PoE camera ports
- Light outputs
- 8 built-in channels
Browse the vision controllers for the run-time box, the 3D topic page for capture-side techniques, and the contact page when a cloud refuses to flatten.
Questions
How many points does a region need for a stable plane fit?
Treat 100 points as the floor. The relative standard error of an RMS estimate runs about 1/√(2N), so 100 points give ±7% and a 50×50 patch at 0.1 mm/px — 2,500 points — about ±1.4%. Extra points sharpen the slope estimate too, but the real accuracy gain comes from spreading patches across the part, not from densifying one corner.
Can I fit each region separately and average the planes?
No. Averaging plane parameters has no least-squares property, weights every patch equally regardless of its point count, and discards the long baseline between patches that makes tilt accurate. Pool all regions and fit once. To validate the pooled fit, split the points into two halves, fit each, and expect the slopes to agree within about 10% on a flat part.
Where should the pass gate come from?
From the drawing, not the fit. If flatness is specified at 0.10 mm, fail any point beyond the 0.10 mm band and warn at 2×RMS. A gate derived from the fit itself, say 3×RMS, only proves the part matched its own average — it accepts slow drift and never alarms on it. Gates belong to the tolerance; RMS belongs to the health check.
When must I abandon the Z image and fit the raw point cloud?
When the surface is not single-valued in Z: vertical walls, folds or stacked levels produce missing and aliased columns that spike the fit. Switch too for tilt beyond roughly 10°, where vertical and perpendicular least squares diverge past the 1.5% level. On those parts, fit the raw XYZ point-to-plane with a unit normal, patch by patch on the steep faces.
Does patch size change the plane I get?
Patch size sets how well you know the numbers, not the numbers themselves. With Gaussian residuals, larger patches narrow the RMS estimate per the 1/√(2N) rule while the fitted slope and intercept converge to the same values; a 60×60 and a 90×90 patch on one surface return matching coefficients within their error bars. Placement, not size, is what can bias a fit.


