3D visionHow-to

Align a point cloud base plane to Z zero and fix upside-down clouds

Fit a plane to a flat board, align the point cloud so its base plane reads Z = 0 at height zero, and fix an upside-down view with one rotation.

LumaSpec Vision12 min read Share on LinkedIn

On this page 11 sections
  1. 01The base plane is a rotation, not an offset
  2. 02What you need before the fit
  3. 03The calibration, step by step
  4. 04Worked example: a 1 m sandbox, 0.83° of tilt
  5. 05The upside-down view is the optical frame
  6. 06Map the projector from measured corners
  7. 07Pick the camera for the contour step you need
  8. 08Checks before you trust Z = 0
  9. 09When a plane fit is the wrong tool
  10. 10Field checklist
  11. 11Questions

You have a depth camera looking down at a surface — a sandbox, a tray, a conveyor bed — and two annoyances. The flat surface that should be your zero arrives in the camera's own tilted frame, and the point cloud renders in Open3D or OpenCV as if you were viewing it from below. Both have single, durable fixes. Fit a least-squares plane to a rigid board lying on the surface, rotate its normal onto +Z, shift its centroid onto the origin, and every later capture reads true heights from Z = 0. The upside-down view is the camera's optical-frame convention, not a depth fault, and it disappears with one rotation. Below: the math, about twenty lines of Python, a worked example on a 1 m sandbox, and the projector mapping that closes the loop.

The base plane is a rotation, not an offset

Resist the instinct to subtract an average Z. A camera tilted barely one degree off vertical spreads a dead-flat board across many millimetres of raw height, and tilt cannot be subtracted away — only rotated away. The base-plane calibration is one rigid transform: a rotation that puts the fitted plane's normal on the +Z axis, and a translation that puts the plane's centroid at the origin.

The fit is the standard total-least-squares eigenvector form:

c = (1/N) Σ pᵢ (centroid, mm) qᵢ = pᵢ − c C = (1/N) Σ qᵢ qᵢᵀ (3×3 covariance) n = eigenvector of C with the smallest eigenvalue (plane normal) if n·c > 0: n = −n (normal faces the camera) a = (n × ẑ) / |n × ẑ|, θ = atan2(|n × ẑ|, n·ẑ) R = I + sin θ·[a]× + (1 − cos θ)·[a]ײ (Rodrigues rotation) p′ = R (p − c) (Z′ = signed height above the board, mm)

Facing the normal toward the camera is what makes +Z point up: after the transform, Z is exactly the signed distance from the board, so material piled on it reads positive height. Because the transform is rigid, distances and angles survive — a circle drawn on the board stays a circle, which the projector calibration reuses.

Several stereo 3D camera SDKs ship this exact pipeline as three commands: fit a primitive plane to the current point map, run a workspace calibration that makes that plane the Z = 0 reference, and store the resulting link in the camera's EEPROM so every capture opens already level, on any host. In the viewers that ship with these cameras the same action usually sits on a Workspace Calibration tab as Align plane with depth data. Use those commands when they exist; the math above is what they compute, and keeping it in the camera ties the transform to the sensor. When the capture is cluttered instead of a clean board, plane fitting on 3D point clouds: least squares vs RANSAC covers the robust variant.

What you need before the fit

  • A rigid board larger than the area you will project, supported on three points so it cannot rock, clean and dry.
  • A camera mount that does not move between the fit and production: 3D camera mounting: stillness during capture decides image quality sets the stiffness bar.
  • A gauge block or a known-height stack for the acceptance check.
  • Any point-cloud library; the whole fit is a few numpy calls.
import numpy as np

pts: np.ndarray = np.asarray(board.points)   # (N, 3) float64, millimetres
c = pts.mean(axis=0)                         # centroid (3,)
q = pts - c
cov = (q.T @ q) / len(pts)                   # 3x3 covariance
evals, evecs = np.linalg.eigh(cov)           # ascending eigenvalues
n = evecs[:, 0]                              # normal: least variance
if n @ c > 0.0:
    n = -n                                   # face the camera (origin)
z_axis = np.array([0.0, 0.0, 1.0])
cross = np.cross(n, z_axis)
axis = cross / np.linalg.norm(cross)         # plumb camera: rotate 180 deg about X instead
angle = float(np.arctan2(np.linalg.norm(cross), n @ z_axis))
K = np.array([[0.0, -axis[2], axis[1]],
              [axis[2], 0.0, -axis[0]],
              [-axis[1], axis[0], 0.0]])
R = np.eye(3) + np.sin(angle) * K + (1.0 - np.cos(angle)) * (K @ K)
levelled = (R @ (pts - c).T).T
assert abs(levelled[:, 2].mean()) < 0.01     # board mean Z = 0 mm

The eigh call returns eigenvalues in ascending order, so the first eigenvector is the normal — the direction of least variance. The assert is the test that proves the transform: after it, the board's mean Z sits at zero to float precision, and its scatter stays at the sensor's noise level.

The calibration, step by step

  1. Prepare the reference. Rake the sand flat or clear the tray, lay the board over the projection area and support it on three points.
  2. Settle the camera. Mount it, let the frame reach shop temperature, tighten everything you will not touch again.
  3. Capture one cloud of the board alone. No hands, cables or sleeves in the frame — and keep eyes out of the projected beam while it runs; the illuminator's safety class on the data sheet sets the controls the station needs.
  4. Crop to the board and fit the plane. Record the centroid, the unit normal and the RMS residual; that residual is your Z = 0 trueness budget.
  5. Face the normal toward the camera. In code, negate it when its dot product with the centroid is positive. The rotation angle then lands near 180°, not near zero — that flip is what makes +Z point up and piles read positive.
  6. Apply the transform and refit as the test. The refit must show tilt under 0.05° and mean Z at 0.0 mm; then put a gauge block on the board and read its top face.
  7. Store the result. Write the 4×4 matrix to disk with the date, or push it through the SDK's fit-plane, workspace-calibration and store-to-camera chain so it lives in the EEPROM.

Worked example: a 1 m sandbox, 0.83° of tilt

Example station: a structured-light camera looks straight down at a sandbox 1000 mm × 750 mm from about 1100 mm away, delivering a 1280 × 960 grid over roughly 1200 mm × 900 mm — one point every 1200 / 1280 = 0.94 mm. A rigid board covers the sand and the cropped capture holds 412,000 points.

The fit returns centroid c = (−12.4, 8.6, 1050.2) mm and, before the sign fix, normal (0.0120, −0.0080, 0.99989) with an RMS residual of 0.35 mm. Faced toward the camera, n = (−0.0120, 0.0080, −0.99989). The board tilt is

θ_tilt = asin(√(0.0120² + 0.0080²)) = asin(0.0144) = 0.83°.

That is all the camera is off vertical, and it is plenty. Uncorrected, the raw depth of a dead-flat board spans 1000 × 0.0120 = 12.0 mm along X and 750 × 0.0080 = 6.0 mm along Y — anywhere from −9 mm to +9 mm corner to corner. Contour lines every 10 mm would paint themselves onto flat sand.

The correction rotates about a = (0.0080, 0.0120, 0) / 0.0144 = (0.56, 0.83, 0) by θ = atan2(0.0144, −0.99989) = 179.17° — the 0.83° of tilt riding on the 180° flip that makes +Z point up — then shifts by the centroid: p′ = R (p − c). After the transform the refit shows 0.02° of residual tilt and mean Z = 0.0 mm, with scatter still 0.35 mm RMS. A 60 mm dune reads Z ≈ +60 mm, and a 10 mm contour step stands about thirty times above the noise — solid lines, no shimmer.

The upside-down view is the optical frame

Levelling fixes Z; X and Y still follow the optical frame, with x right in the image and y down the image. Aim a camera at the floor and y runs horizontally along the surface, not vertically. A viewer whose default viewport assumes y is up renders that cloud from below or rotated 180° in plan view. The depth values are correct; only the axis labels differ from your expectation. Depth camera point cloud flat in XY: fix the optical frame walks the same convention from the flattened-plot side.

For display, rotate 180° about Z: (x, y, z) → (−x, −y, z). That is a proper rotation — determinant +1 — so it preserves handedness and every geometric quantity you compute later.

When the levelled cloud must live in a world or robot frame, compose the conventions instead of improvising signs: transform depth point clouds between optical frame and camera frame shows the axis order that keeps orientations intact.

Map the projector from measured corners

The sandbox projector draws on the same surface the camera measures, so the last piece is a fixed 2D mapping from levelled world X, Y to projector pixels. Mark four non-collinear points whose levelled coordinates you know — the board's corners are ideal — record where the projector lands them, and solve the homography once. Rasterize the levelled cloud onto a grid at the projector's pixel pitch and render from that grid. The 180° display rotation and this homography stay separate, and both stay fixed; no per-frame hand flips anywhere in the pipeline.

Pick the camera for the contour step you need

Height noise picks the family. Typical figures at about a metre of standoff:

Camera family Typical height noise Verdict for 10 mm contours
Projected structured light 0.2–1 mm First choice for sand and matte surfaces
Stereo with projected pattern 0.5–2 mm Good, needs pattern or texture
Time of flight 5–15 mm Volume and presence only
Laser line profiler 0.05–0.3 mm Finest detail, part must move under it

Commit to what the table implies: structured light or stereo for 10 mm contour lines, time-of-flight where a centimetre of noise is acceptable, a laser line profiler when the surface moves and the finest height detail pays for the mechanics.

For a matte granular surface such as sand, projected structured light is where to start, and the MV-DPS200P-02 is that type:

3D sensorsHikrobot MV-DPS200P-02 Projected structured-light 3D cameraMV-DPS200P-02
Type
Stereo / structured light
$8,829.00 · quotedView and quote

Stereo and time-of-flight units cover the same overhead mount at coarser height detail:

3D sensorsHikrobot MV-DLS300P-04 Laser galvanometer stereo cameraMV-DLS300P-04
Type
Stereo / structured light
$16,579.00 · quotedView and quote
3D sensorsHikrobot MV-DT01WMU ToF sensing cameraMV-DT01WMU
Type
Depth / ToF camera
Price on requestView and quote

Lateral pitch matters as much as height noise. A projector with 1280 columns across a 1000 mm box paints 1000 / 1280 = 0.78 mm per pixel, and the camera needs points at least that fine or the contour lines go blocky. The camera resolution calculator runs the same division for any 2D camera you pair with the depth stream:

Camera resolution for the smallest feature
Units
Pixels needed
–
Sensor size
– MP
Smallest sensor that fits
–
Pixel on the part
– mm
Pixels across the feature, that sensor
–

Pixels needed = field / feature × pixels per feature on each axis (field incl. margin). The lens must resolve the pixels too: Nyquist = 1000 / (2 × pixel µm) lp/mm.

Find cameras with enough pixels Open the full calculator

To browse by family, the 3D sensors category lists profilers, structured-light and stereo cameras, the camera finder filters them by type, and the configurator builds the whole station starting from the camera step.

Cameras3D sensors

Laser profilers and stereo 3D cameras

170 models · from $699.00

Checks before you trust Z = 0

When a plane fit is the wrong tool

Calibrating to the sand itself. Sand is the measurement, not the reference: it is raked, rebuilt and shifted hourly. Fit to the rigid board or the box floor and let the sand read as positive height.

Fitting a cluttered capture with plain least squares. Least squares follows outliers — a hand, a cable, a toy left on the board bends the plane. Fit only the cropped board, or switch to the RANSAC variant for dirty scenes.

A camera that moves between stations. The stored link is tied to the camera's pose at fit time. An eye-in-hand camera needs the plane recomputed per pose, so keep that transform in software, not in the EEPROM.

A reference that is not your zero. The board defines Z = 0; if product sits on a fixture 40 mm above the bed, add that offset once, in one place, instead of refitting.

Field checklist

  • Board clean, rigid, three-point support, larger than the projection area.
  • Camera warm, tight and untouched after the fit.
  • One cropped cloud of the board only.
  • Centroid, unit normal and RMS residual recorded.
  • Normal faced toward the camera.
  • Refit after transform: tilt under 0.05°, mean Z at 0.0 mm.
  • Gauge block reads its height within tolerance.
  • Matrix dated on disk, or stored to the camera EEPROM.
  • Homography solved from four measured corners.
  • Daily gauge check; refit after any bump, drift or remount.

If the box is wider than one camera, or the surface is glossy rather than matte, send the layout through the contact page and we will size the station with you. More frame conventions and calibration workflows sit on the 3D vision topic page.

Questions

How flat does the calibration board have to be?

Flat to a good margin below the height trueness you need: for ±1 mm work, choose a panel whose bow is well under 0.5 mm across the span and support it on three points so it cannot rock. The fit itself tells you the quality: a bowed board leaves smooth, cupped residuals far above the sensor noise floor, while a good board leaves random scatter at the noise level. Treat the residual pattern as a board test, not just a fit score.

Should the calibration be stored on the camera or on the PC?

Store it on the camera when the SDK offers the fit-plane, workspace-calibration and store chain: the transform then travels with the sensor and every capture opens already level, on any host PC. Keep a dated copy of the same 4×4 matrix on disk as a backup. The stored link survives power cycles and host changes, but not a knock to the mount; recheck with a gauge block after any physical disturbance.

Is an upside-down point cloud a sign of broken depth data?

No. The depth values are correct; the axes are labelled in the camera's optical convention, with x right in the image, y down the image and z away from the camera. A viewer whose default viewport assumes y is up renders a downward-looking camera's cloud from below or rotated in plan view. Rotate 180° about Z for display, which is a proper rotation that keeps handedness, and map the projector from measured corners rather than hand flips.

Least squares or RANSAC for the base plane fit?

Least squares on a clean, dedicated board: it uses every point and there is nothing to reject. Switch to RANSAC when the capture is cluttered with hands, cables or product left on the surface. Set the RANSAC inlier distance to two or three times the sensor's height noise, about 1 mm for the example camera in this guide, so real board points survive and debris does not drag the plane.

How often should the base plane calibration be redone?

Gauge-check at every shift start and refit on events: a bumped bracket, a remounted camera, a frame that has heated or cooled by several degrees. Slow, steady drift usually points to mount creep or thermal growth rather than sensor wear, so tighten and re-shim instead of refitting endlessly. The copy stored in the camera does not update itself; after a refit you must store the new link again.

Can the projector be driven straight from the raw depth image?

Only after the base-plane transform. Raw depth is distance along the camera's tilted optical axis, so a flat board carries the camera tilt as false height, millimetres to tens of millimetres across the box. Level the cloud first, rasterize it onto a grid at the projector's pixel pitch, then apply a homography solved from four measured corners. Skipping the levelling step bakes the tilt into every contour line you project.