Depth camera intrinsics and distortion parameters: read, don't guess

You have a depth camera on the bench, a SLAM or point-cloud pipeline waiting, and the pipeline wants fx, fy, cx, cy and the distortion coefficients. The question that costs people a day is whether to print a chessboard and calibrate the unit by hand. The answer is almost always no. A depth camera leaves the factory individually calibrated, the values are stored on the device, and the SDK prints them on request. This guide covers what the five numbers mean, where to read them, which parameter set applies when you run the camera as a monocular sensor, and the few cases where recalibrating genuinely earns its hour.

What the intrinsics describe

The model behind every depth source is the pinhole, and four numbers pin it down: fx and fy, the focal length in pixel units along each axis, and cx, cy, the principal point where the optical axis meets the sensor. The principal point sits near the image centre but almost never exactly at it, which is why pipelines carry the values explicitly. The fifth item is a name rather than a number: the distortion model, followed by its coefficient list - radial terms k1, k2, k3 and tangential terms p1, p2 in the common formulation.

Project a point X, Y, Z from the camera frame and u, v is the pixel it lands on. Deprojection runs the same equations backwards: the depth value replaces Z and the intrinsics produce X and Y, which is how a depth map becomes a point cloud.

Depth cameras differ from a bare board camera in one useful way: the published streams are usually rectified, so most distortion is corrected before the data reaches you. Expect a simple model on depth and infrared streams - often none at all, or a single radial term - and a fuller set on a wide colour lens. If the model field reads unknown or empty, that is a parsing problem rather than a lost calibration; distortion model unknown: garbage depth camera coefficients explained walks through that failure.

Where the factory numbers live

Each unit is calibrated at production and the result is written to the camera, so it survives power cycles and re-installs. Three read paths cover most SDKs:

• Enumeration tool. The SDK's enumerate or calibration mode prints per-stream intrinsics, the distortion model with its coefficients, and the extrinsics between imagers. It is the fastest route to values for a report or a config file.
• Stream profile at runtime. Every stream you open carries its intrinsics alongside the frames; read them there and they always match the live resolution.
• Calibration model on ToF. Time-of-flight units publish fx and fy through the calibration model; the arithmetic is identical, as set out in reading fx and fy from a ToF calibration model.

Intrinsics belong to a stream profile, not to the camera as a whole. Change the resolution and fx, fy and the principal point scale roughly with the pixel grid, so values read at 640 x 480 do not describe a 1280 x 720 stream. Binning applies the same rule with the binning factor; camera intrinsics after binning: scale them correctly works that arithmetic through. The safe habit: re-read the values whenever the profile changes instead of scaling by hand.

Monocular use changes nothing

Feed the camera to a monocular visual-odometry or SLAM stack and every coefficient stays as the enumeration printed it. Monocular use ignores the second imager and the extrinsics between the pair; nothing about the first imager's geometry changes. The only decision is which parameter set to feed the estimator, and one rule covers it: use the intrinsics of the stream whose frames you actually consume. Colour frames take the colour stream's values, infrared frames take the infrared stream's values, and mixing them displaces features toward the frame edges enough to poison loop closures.

If depth runs alongside, deproject it with the depth stream's own values; aligned depth point cloud intrinsics: which camera to deproject with sorts out the common mix-ups.

Worked example: field of view from fx on a 1280 x 720 stream

The fastest sanity check on a readout converts fx into a field of view. The inputs here are illustrative round numbers, not one unit's calibration - the method is what you take away:

• Stream width W = 1280 px, the profile you opened.
• fx = 640 px, from the readout.
• Imager width = 4.0 mm, from the imager's data sheet, needed only for the millimetre step.

The rule in words: a lens's field of view and the pixel-scale focal length are two views of the same angle, so the horizontal field follows as FOV = 2·atan(W / (2·fx)).

The arithmetic: 1280 / (2 × 640) = 1.0, atan(1.0) = 45°, and 2 × 45° = 90.0°. An fx of 640 px on a 1280 px stream is exactly half the pixel width, and that is what a 90° horizontal field looks like in intrinsic form. The millimetre step: f = 640 px × 4.0 mm / 1280 px = 2.0 mm, which is why the factory lens is not a stock machine-vision lens in disguise.

The decision, with its thresholds: run the same division on your own readout. fx at half the image width means a 90°-class stream; 1,000 px on the same width gives 2·atan(1280/2000) = 65.2°; 1,920 px gives 36.9°. If the value in your config file implies a field of view far from the lens you see on the unit, you are reading another profile's values - go back to the runtime readout. To run the conversion the other way, a target field of view in and a focal length out, use this:

When a recalibration is worth the hour

Recalibrate when the optics or mechanics have changed, and otherwise leave the stored values alone:

• A lens was replaced, or focus was moved and re-locked at a new position.
• A protective window, filter or fold mirror now sits in the optical path.
• The head was opened for repair, or the unit took a knock hard enough to shift an imager.
• You need corner-to-corner undistortion of a wide colour image and the factory model stops at the first radial term.
• Metrology asks for per-unit error numbers you can trace, rather than a factory statement.

For depth quality after thermal or mechanical stress, prefer the vendor's on-device recalibration utility where the SDK ships one; it re-tunes the depth engine without touching the colour calibration. A chessboard session of your own measures lens distortion well, but it will not beat the factory stereo baseline, so keep the extrinsics from the device unless the hardware actually changed.

What each sensor type hands you

Depth is not one product family, and the calibration you consume differs by type:

Structured-light and stereo area cameras are the family this workflow applies to, and the 3D sensors range is where they sit side by side. For time-of-flight work, the MV-DT01WMU is the shop's ToF sensing camera.

Stereo-family units from the range, side by side:

Checks before you trust the readout

Four bench checks take minutes and catch the errors that cost days:

• Ratio check. Divide fx by the image width. Around 0.5 means a 90°-class wide stream, about 0.8 gives near 64°, and 1.0-1.5 is the band from 53° down to 37°. A ratio far outside your lens's territory means the values came from another profile.
• Flat plane. Scan a flat plate and fit a plane; curvature from centre to edge beyond your tolerance says the distortion model does not match the data path, as worked through in depth camera flat plane distortion: quantify edge error.
• Reproject a known target. Place a target at a measured distance, deproject, and compare coordinates; the residual is your end-to-end error budget.
• Round trip. Project the deprojected cloud back through the same intrinsics; it must land on the original pixels exactly. If it drifts, something in the pipeline rescales frames without rescaling values.

When the usual answer is wrong

Read, don't recalibrate, breaks in four situations. The optics changed - swapped lens, added window, refocus and re-lock - and the list above is the test. The resolution or binning changed and the values on hand belong to the old profile, so re-read rather than scale. The frames come from a different stream than you assumed: colour and infrared imagers have their own cx and cy and often their own field of view, so colour values applied to infrared frames are wrong at the edges even when the image looks fine. And a camera mounted on a moving axis accumulates knocks and thermal cycles, so a periodic depth-quality recalibration pays even when nothing looks damaged.

Bench checklist

Work through this once per unit, and again after any hardware change:

• Read intrinsics from the live stream profile, never from a copied config file.
• Record the resolution next to the values; the pair travels together and only together.
• Convert fx to field of view and compare it against the lens you expect on the unit.
• Fit a flat plane and log the centre-to-edge deviation as the unit's baseline.
• Note the SDK and firmware versions alongside the numbers, so a later change is traceable.
• After a lens swap, an added window or a knock: recalibrate, then re-read the extrinsics from the device.

To pick the hardware and the rest of the chain around it, start at the camera finder. More depth-camera reading sits on the 3D vision topic page, and if a build stays stuck the contact page reaches the applications team.