3D visionGuide

ToF camera calibration: checkerboard to a z=0 base plane

ToF camera calibration: keep the factory intrinsics, turn the radius depth image into a point cloud, fit the checkerboard, rotate the base plane to z=0.

LumaSpec Vision10 min read Share on LinkedIn

On this page 10 sections
  1. 01What calibration means for a ToF depth stream
  2. 02Radial depth comes before corner detection
  3. 03One checkerboard pose, sized for the pixels
  4. 04Fit the plane: least squares, RANSAC when the mask leaks
  5. 05Rotate first, translate second
  6. 06Worked example: 40 mm squares at 600 mm, radius to z = 0
  7. 07Acceptance numbers and what moves them
  8. 08When one pose is not the answer
  9. 09Checklist before you trust the cloud
  10. 10Questions

You have a ToF camera serving a luminance image and a depth image, and you want the surface of a table to become the z = 0 plane of your measurements. Conventional practice splits that into two stages, and only one of them needs a checkerboard. Keep the factory intrinsics that already shaped the depth stream, capture one checkerboard pose on the table, fit its plane, and apply a single rigid transform so that plane reads z = 0. This guide gives the formulas, a worked example with every arithmetic step, and the residual number that decides whether the calibration passes.

What calibration means for a ToF depth stream

A depth camera is calibrated in two halves. Intrinsics map a pixel to a viewing ray; extrinsics map the camera frame onto your base frame. For a ToF camera the intrinsics are already inside the product: the depth image is generated through the factory geometric model, so fx, fy, cx and cy belong in your conversion, not on your to-do list. Read them as the ToF camera intrinsic parameters guide shows, then build XYZ points with the depth map to real distance workflow.

What is missing is the extrinsic: one rotation and one translation. A visible plane fixes all six degrees of freedom of that transform, so one checkerboard pose on the table is a complete extrinsic calibration. Do not re-fit fx and fy from the ToF luminance image. It is low-resolution, infrared-lit and noisy, so the refitted focal length disagrees with the one the depth stream was built from, and the cloud bends toward the image edges. The re-derivation makes the geometry worse, not better.

Any depth camera in the 3D sensors category serves this luminance-and-depth pair, and the MV-DT01WMU is the ToF entry.

3D sensorsHikrobot MV-DT01WMU ToF sensing cameraMV-DT01WMU
Type
Depth / ToF camera
Price on requestView and quote

Radial depth comes before corner detection

A raw ToF depth image stores radius: the range from the camera origin to each pixel's scene point, not the axial distance along the optical axis. That distinction changes the arithmetic. For pixel (u, v):

d = ((u − cx) / fx, (v − cy) / fy, 1) P = r · d / ‖d‖

The point P lands at range r along its unit ray. When a stream already stores axial Z, drop the division; an SDK that hands you XYZ directly has done it for you. The penalty for mixing the two grows with ray angle, because r exceeds Z by 1/cos of that angle, about 23% at the edge of a 70° field of view. Treated wrongly, a flat wall curls away at the corners, and every plane you fit afterwards inherits the curvature.

Tool support is direct. MATLAB's pcfromdepth converts a depth image and intrinsics into a point cloud; OpenCV and Open3D do the same back-projection in a few lines. Depth normally arrives as 16-bit values beside an 8-bit luminance frame, a pairing the depth and infrared stream guide untangles.

One checkerboard pose, sized for the pixels

Mount the pattern rigidly, ink on film bonded to aluminium composite rather than loose paper, because a warped board writes its warp into your base plane. Lay it where the parts will be measured, filling a third to a half of the image, with the camera within about 30° of the board normal. Detect the corners in the luminance image and use their bounding region as the mask that selects depth points for the fit.

Size the squares to span at least 20 pixels: pixels per square = square edge × fx / distance. At 600 mm with fx = 450 px, a 40 mm square covers 40 × 450 / 600 = 30 px, comfortable for corner detection. The luminance frame is the navigation layer of the pair; teams that also need colour pair the ToF unit with an area scan camera.

Fit the plane: least squares, RANSAC when the mask leaks

With only board points inside the mask, a least-squares fit, one SVD of the centred coordinates, returns the plane n̂ · P = d0, with n̂ the unit normal and d0 the offset. When the mask also catches the table edge or a clamp, RANSAC on the same points rejects them; the least squares versus RANSAC comparison trades off the two estimators with code. Set the inlier distance to roughly three times your target residual, 2 mm here. That keeps noisy board pixels in and drops the table edge.

Rotate first, translate second

Build R so that R · n̂ = ẑ: its rows are any two orthonormal axes in the board plane plus n̂ itself. After the rotation every board point reads the same z, equal to d0. Subtract that offset, plus the board thickness when z = 0 must be the table face rather than the board face:

P_base = R · P − (0, 0, d0 + t_board) residual RMS = √( mean( (n̂ · P_i − d0)² ) )

Order matters: rotate, then translate. Translating first shifts the offset the rotation is about to reuse, and the plane ends up tilted again; the extrinsic transform order guide walks the sequence. With n̂ mapped to +z, the axis points down into the table and parts read negative until you flip it; the base plane alignment walkthrough covers that flip. Store R and the offset and apply them to every frame you capture, offsetting the plane rather than only the snapshot, so the calibration survives every later capture.

Worked example: 40 mm squares at 600 mm, radius to z = 0

The case: a fixed ToF camera over a table. Depth and luminance both 640 × 480 px; intrinsics from the calibration model fx = fy = 450 px, cx = 320 px, cy = 240 px, typical values of the kind the model prints. Board: 9 × 6 inner corners, 40 mm squares, inner area 8 × 5 = 320 × 200 mm, about 600 mm from the camera. Mask: the corner bounding box, 240 × 150 px, holding about 36,000 depth points, of which 5,000 are sub-sampled for the fit.

  1. Back-project one pixel. (u, v) = (440, 190) reads r = 610 mm. du = 440 − 320 = 120 px and dv = 190 − 240 = −50 px, so d = (0.2667, −0.1111, 1) and ‖d‖ = √(0.0711 + 0.0123 + 1) = 1.0409. Then P = (610 / 1.0409) · (0.2667, −0.1111, 1) = 586.1 · (0.2667, −0.1111, 1) = (156.3, −65.1, 586.1) mm.
  2. Fit the plane. Least squares over the 5,000 points returns n̂ = (0.087, 0.174, 0.981) and d0 = 576.5 mm. The board sits arccos(0.981) = 11.2° off the camera axis: that is the mounting, not an error.
  3. Check one residual. n̂ · P = 0.087 × 156.3 + 0.174 × (−65.1) + 0.981 × 586.1 = 13.60 − 11.33 + 574.96 = 577.2 mm, so the residual is |577.2 − 576.5| = 0.7 mm. The RMS over the whole mask lands at the same 0.7 mm.
  4. Transform. With R · n̂ = ẑ, board points read z = 576.5 mm. The board is 3 mm thick, so subtract 576.5 + 3 = 579.5 mm and the table face is z = 0.
  5. Check the scale. Square edges in the transformed cloud measure 40.3 mm and 39.8 mm, which is 0.75% and 0.5% error.

Decision: 0.7 mm RMS is under the 1 mm pass threshold at this distance, so accept the transform and log R, d0, the thickness and the date. Skipping the division in step 1 gives the same pixel P = (162.7, −67.8, 610.0) mm and a residual of 24 mm. A residual in the tens of millimetres is the radial-to-axial mix-up, not a bad fit.

Acceptance numbers and what moves them

Pass is a residual RMS under 1 mm for a ToF camera working between 300 and 1000 mm, the span where these sensors hold their noise floor. When the number is higher, it reads almost as a diagnosis:

Residual RMS Likely cause Fix
under 1 mm none accept, log R and offset
1–3 mm glossy board, multipath, mask edge matte target, shrink mask, RANSAC at 2 mm
tens of mm (24 mm in the example) radial depth read as axial divide by ‖d‖ and refit
under 1 mm, squares off by ~2% intrinsics not the pair the depth stream uses reload fx, fy from the calibration model

A glossy laminated target returns multipath signal, and board-edge pixels average board depth with table depth; shrinking the mask a few pixels inside the corner box removes both. A plane that passes while the scale check fails points at the intrinsics you typed, not the fit, so reload them from the model. Once the plane is trusted, keep dimension trueness honest on the line with the infield correction check.

When one pose is not the answer

A plane transform is only as permanent as its mounting. Bolted brackets hold the calibration for months; re-fit after any collision, remount or table move. When you need robot-base coordinates rather than table coordinates, the z = 0 plane is still the right first step, but the full frame comes from hand-eye calibration; a cloud sitting off the robot base plane afterwards has a known cause and fix. And when the target itself defeats ToF, mirror-finish tables and shiny foils, residuals stay high no matter what you fit. A projected structured-light camera takes the plane from a single shot on surfaces like that.

3D sensorsHikrobot MV-DPS200P-02 Projected structured-light 3D cameraMV-DPS200P-02
Type
Stereo / structured light
$8,829.00 · quotedView and quote
3D sensorsHikrobot MV-DLS300P-04 Laser galvanometer stereo cameraMV-DLS300P-04
Type
Stereo / structured light
$16,579.00 · quotedView and quote

Checklist before you trust the cloud

  • Depth convention confirmed as radial or axial; the 24 mm example residual is what the mix-up costs.
  • fx, fy, cx and cy loaded from the calibration model, not refitted from the luminance image.
  • Squares spanning at least 20 px, board rigid and flat, mask shrunk inside the corner box.
  • z = 0 defined as the table face, with the board thickness subtracted.
  • Residual RMS under 1 mm recorded, square scale check under 1%.
  • R and offset stored and applied to every frame, not baked into one capture.

More point-cloud workflows are collected under the 3D vision topic page. If the residuals will not come down, bring the numbers to our applications team; the residual, the tilt and the square measurement carry most of the diagnosis. You can also start a build or browse the camera finder when the depth stream itself is the bottleneck.

Questions

How many checkerboard poses does a ToF camera need?

One, for the extrinsic base plane. The intrinsics are already inside the depth stream, so the checkerboard acts only as a flat, locatable target that fixes the six degrees of freedom of the transform. Extra poses earn their keep only when you re-derive intrinsics, which a ToF depth stream does not need and is worse off for.

How do I tell whether the depth image is radial or axial?

Capture a wall roughly parallel to the image plane and back-project the values as if they were axial Z. If the corners fall behind the centre, meaning the cloud bows away from the camera, the stream is radial; repeat with the division by ‖d‖ and the wall goes flat. The plane residual drops from centimetres to millimetres when you get it right.

Can I fit the bare table and skip the checkerboard?

Yes. The plane fit needs only depth points, so a hand-drawn mask over a clean, matte table works. You lose the two things the board adds: a corner-based mask that follows the target automatically, and the known square edge for the scale check. For a one-off alignment that trade is often acceptable.

What residual RMS should pass after the transform?

Under 1 mm for a ToF camera between 300 and 1000 mm working distance. Between 1 and 3 mm, suspect gloss, multipath or mask edges before the maths; in the tens of millimetres, the radial depth was read as axial. Log the number with the date so later re-checks have a baseline.

How often should the z=0 calibration be redone?

After any event that can move the camera or the table: a collision, a remount, a new bracket. Between events, refit the empty table weekly and compare the residual and tilt with the logged baseline. A rigid bolted mounting typically holds the plane for months; magnetic or clamped mounts ask for more frequent checks.