ToF camera intrinsic parameters: read fx fy from the calibration model

Deserialize a time-of-flight camera's calibration and you get a model id and a wall of floats: focal lengths in pixels, a principal point, four distortion terms and a field-angle limit. None of them are labelled. This guide maps the layout — which entry is which, what each does in the projection, and how to turn one depth pixel into a metric X, Y, Z point with four lines of arithmetic. It is written for anyone wiring a ToF camera into robot picking, palletizing or volume measurement. By the end you can read the array from your SDK, project pixels to metres, and run four checks that catch a mis-sliced array before it moves a robot.

What the ToF info chunk holds

ToF SDKs deliver each depth frame with metadata buffers alongside. In the layout this article decodes, the calibration rides in the buffer named TOF_INFO. Deserialize that chunk and two fields come back: a model_id, which selects the parameter layout that applies to your head, and a modelParameters array. The array often runs 32 floats or more, but the first ten carry everything the projection needs:

• fx — horizontal focal length, in pixels
• fy — vertical focal length, in pixels
• mx, my — principal point column and row, in pixels
• α — axis skew
• k1, k2, k3, k4 — distortion terms
• θmax — the field angle out to which the model is calibrated

Slots past the tenth read back as zeros; they are reserved. Those ten numbers are the complete pinhole description of the depth camera. Where the head sits in your cell — mount height, tilt, rotation — is extrinsic calibration, and this chunk says nothing about it.

The ten entries and the projection they drive

The projection itself is the standard pinhole camera model, the same one taught in every computer-vision textbook: every 3D point in front of the camera lands on exactly one pixel of the frame, and the depth value Z closes the loop back to metres.

fx and fy are focal lengths measured in pixels, so they carry the frame size with them. The scale comes from the field of view: a 90° horizontal FOV needs fx = (W/2) / tan(45°), which is 120 px on a 240-pixel-wide frame and 480 px on a 960-pixel frame. Small fx values on a small ToF frame are therefore normal, not a misread.

mx and my mark where the optical axis pierces the frame. Calibration rarely lands it dead centre, so a readout tens of pixels off centre is correct data, not noise — project with the calibrated values, never with W/2 and H/2.

α is skew between the pixel axes; essentially every modern head reports 0.0.

The k terms describe the lens distortion the calibration removes. When all four are zero, the SDK hands you rectified depth and the four-line projection above is exact. When they are non-zero, the head delivers raw optics and you should call the SDK's own undistort or deproject routine. The polynomial convention behind k1 to k4 — radial in radius or angular in field angle — varies between SDKs, and the array does not say which. θmax, the tenth entry, is the field angle out to which that fit was made; pixels beyond it are extrapolation.

Reading the chunk in your SDK

Your SDK owns the device plumbing: open the device, start its frame grabber, and subscribe depth and TOF_INFO in the same schema so both arrive from one frame. Deserialize the calibration chunk with the deserializer the SDK ships for that buffer id — the chunk comes back as an object, not raw bytes. Then slice the first ten entries into named variables immediately:

# Illustrative snippet: our variable names; model_id and modelParameters are
# the fields the deserialized ToF info chunk exposes.
model_id: int = int(calib.model_id)                 # which parameter model applies
params: list[float] = list(calib.modelParameters)   # flat float array

fx: float = params[0]          # horizontal focal length, px
fy: float = params[1]          # vertical focal length, px
mx: float = params[2]          # principal point column, px
my: float = params[3]          # principal point row, px
alpha: float = params[4]       # axis skew
k1: float = params[5]          # distortion terms
k2: float = params[6]
k3: float = params[7]
k4: float = params[8]
theta_max: float = params[9]   # field-angle limit, degrees

Print model_id at start-up and log it with the run data. The meaning of the array belongs to that id, so a script that serves more than one head type should branch on it before touching params.

Anything that consumes pinhole parameters takes them as a 3 × 3 matrix: K = [[fx, 0, mx], [0, fy, my], [0, 0, 1]]. Fold α into the top-middle cell if a head ever reports non-zero skew.

Worked example: one depth pixel to a 3D point

The values below are an illustrative readout for teaching the arithmetic, not a data-sheet value for any camera in the catalogue. The frame is 240 × 172 px, and the chunk reads: fx = fy = 128.6 px; mx = 119.5 px, my = 85.5 px, the frame centre; α = 0.0; k1 to k4 all 0.0, so the depth is rectified; θmax = 49.0°.

Take one pixel at u = 200, v = 130 whose depth reads Z = 2.000 m, taken from the depth image at that same pixel. Pulling the centre pixel's value instead is its own classic bug, covered in Center pixel depth measurement: read a fixed image point.

X = (200 − 119.5) × 2.000 / 128.6 = 161.0 / 128.6 = 1.252 m right of the axis

Y = (130 − 85.5) × 2.000 / 128.6 = 89.0 / 128.6 = 0.692 m below the axis

Row 0 sits at the top and Y points down, the usual camera convention. Flip the sign on Y once, in one place, if your robot frame wants it up.

Run the projection backwards to prove the arithmetic: u = 128.6 × 1.252 / 2.000 + 119.5 = 200.00 px and v = 128.6 × 0.692 / 2.000 + 85.5 = 130.00 px. With unrounded intermediates the round trip closes exactly; the residues above are display rounding.

Distance and field angle come free: √(1.252² + 0.692² + 2.000²) = √(1.567 + 0.479 + 4.000) = √6.046 = 2.459 m, at atan(√2.046 / 2.000) = atan(0.715) = 35.6° off-axis — inside θmax = 49.0°, so the pixel sits in calibrated territory.

Four checks on a fresh readout

Run these once per head at start-up; each catches a different failure.

• fx and fy agree. ToF pixels are square and depth normally ships in its native geometry, so equal values are the healthy case. A large gap means the SDK resampled the depth, or you sliced the array one slot out.
• mx and my sit near the frame centre. Tens of pixels off: normal calibration. Equal to fx: your slice offset is wrong. Hundreds of pixels off: wrong head, wrong model id, wrong array.
• The FOV that fx implies matches the manual. For the example frame, FOV_h = 2 · atan(120 / 128.6) = 86.0° and FOV_v = 2 · atan(86 / 128.6) = 67.6°. If your arithmetic lands near 10° or 170°, stop and re-slice.
• θmax covers the frame corner. Half-diagonal = atan(√(120² + 86²) / 128.6) = atan(147.6 / 128.6) = 48.9°, just inside the example's 49.0°. A smaller θmax means corner pixels are extrapolated; mask them out.

The same geometry runs the other way when you size a lens for a standard area scan camera: focal length from the field of view you want to cover.

Where the shortcut fails

• One head, two calibrations. A ToF camera with a 2D imager publishes separate intrinsic sets for depth and texture, and depth's fx is not the RGB image's fx. Project depth with depth intrinsics and convert or register through the SDK; the conversion itself is laid out in Camera intrinsic parameter conversion for depth and texture images.
• Distortion applied twice. If the SDK already rectifies depth, the k slots hold zeros and hand-rolled undistortion bends a straight rack into a curve. Test with a straight edge in view and check the cloud stays straight; the coefficient conventions behind this are sorted out in Distortion model unknown: garbage depth camera coefficients explained.
• Intrinsics mistaken for placement. The chunk describes the lens, not the mount. Height above the bin, tilt and roll live in your extrinsic transform — a separate exercise with its own numbers in the 3D camera working distance and positioning guide.
• The wrong camera deprojected. Aligned depth plus texture tempts you to deproject with whichever camera the API names first; the choice moves every point. The options are laid out in Aligned depth point cloud intrinsics: which camera to deproject with.

Depth cameras for this pipeline, and the decision

All three depth families — time of flight, projected structured light, stereo — end at the same projection in your code; only the chunk name and the distortion convention change. The depth/ToF line in our 3D sensors range:

And the structured-light and laser-galvanometer stereo lines, which range by projected pattern or a second imager:

The decision, per case:

• Rectified depth with zero k terms: slice the ten entries, project with the four-line formula, ship it.
• Non-zero k terms: feed the chunk to the SDK's undistort or deproject call instead of hand-rolling; your only job is the right buffer.
• Depth plus RGB in one view: two intrinsic sets, registered through the SDK, then converted as in the article linked above.
• Detections to pick points: the same projection per pixel; the box-to-point path is walked in Bounding box depth: mapping image pixels to 3D points.

Shortlist hardware in the camera finder, spec the camera side in the camera configurator, and browse the rest of the depth setup guides on the 3D vision topic page. Expert support comes with every order — send us your head model and frame rate and we will check the projection with you.