Project a 3D point to pixel coordinates with a calibrated camera

You have a calibrated camera, a part at known world coordinates, and a user interface that must draw a crosshair on the exact pixel where a feature sits. The point arrives in millimetres, referenced to the image centre, and the overlay has to land on the raw image, not a rectified copy. Forward projection — 3D point to pixel — does this in three steps: pose to homogeneous matrix, world point to camera coordinates, then the pinhole division into pixels. The worked example below uses a 25 mm lens, 3.45 µm pixels and a point 10 mm off-centre at 300 mm working distance; it lands at pixel (1466, 1266). You will also see where the one-constant millimetre-to-pixel shortcut breaks, how to test your own code against the example, and which camera and lens choices keep the overlay accurate across the whole field.

How do you project a 3D point to pixel coordinates?

Project in three steps: convert the calibrated pose to a 4×4 homogeneous matrix, transform the world point into camera coordinates with that matrix, then divide by depth and add the principal point to get u and v in pixels.

1. Pose to matrix. Take the calibrated pose — rotation R and translation t — and stack them into the 4×4 matrix T.
2. World to camera. Write the point in homogeneous coordinates (X_w, Y_w, Z_w, 1) and multiply by T: P_c = R·P_w + t.
3. Project. Divide the in-plane offsets by the depth and add the principal point: u = f_x·X_c/Z_c + c_x, v = f_y·Y_c/Z_c + c_y.
4. Draw. Keep u and v as floats for a sub-pixel crosshair, or round to integers if the drawing API demands them.

Conventions differ between libraries, so pin yours down: here the camera looks along +Z_c, x points right, y points down, and only points with Z_c > 0 project sensibly. Machine-vision SDKs wrap the same arithmetic in three calls; one common library names them pose_to_hom_mat3d, affine_trans_point_3d and project_3d_point — pose to matrix, 3D affine transform, 3D-to-image projection. Find the equivalents in your SDK and everything below maps one to one.

What do f_x, f_y, c_x and c_y mean in pixels?

They come from your calibration: f_x and f_y are the focal length expressed in pixels (focal length in mm divided by pixel pitch in mm), while c_x and c_y locate the principal point, which for a centred sensor sits at half the pixel counts.

f_x converts an angle into pixels; it is not the number printed on the lens barrel, it is that number in pixel units. With 3.45 µm pixels and a 25 mm lens, f_x = 25 / 0.00345 = 7246 px. Square pixels make f_y equal f_x, which covers nearly every machine-vision sensor. The principal point is where the optical axis pierces the sensor — near the geometric centre but not guaranteed, and whether the centre is W/2 or (W−1)/2 is a half-pixel question your calibration file settles. Read all four numbers from the calibration rather than computing them from resolution. This pixel scale is the same quantity that decides what a set-up can resolve, as in the guide on resolving 25 µm features at 10 cm working distance.

How does the camera pose become a 4×4 transform matrix?

Stack the pose's rotation and translation into T = [R t; 0 1] and multiply it by the world point in homogeneous form; P_c = R·P_w + t is the same point re-expressed in camera coordinates, in the units you chose, usually millimetres.

Worked pose for this example: camera axes parallel to world axes, camera centre 300 mm in front of the world origin. R is the identity matrix and t = (0, 0, 300) mm, so the world point (10, 10, 0) mm — 10 mm right and 10 mm down from the origin on the work plane — becomes P_c = (10, 10, 300) mm. With no rotation mixing the axes, you can check every number by hand. A camera tilted 30° or bolted to a robot flange carries a real 3×3 in place of the identity, and its pose comes from chaining the mounting transforms — the procedure in eye-in-hand pixel to world coordinates. Keep one unit across pose, point and focal length: millimetres throughout this example.

How many pixels is 10 mm off-centre at the example depth?

With f_x = 7246 px and Z_c = 300 mm, a point 10 mm off-centre lands 242 px from the principal point, so the crosshair goes to (1466, 1266) on a 2448 × 2048 sensor whose principal point is (1224, 1024).

The inputs, one per line:

• Sensor: 2448 × 2048 with 3.45 µm pixels (a 5 MP 2/3"-class sensor), giving c_x = 1224 px and c_y = 1024 px.
• Lens: f = 25 mm, so f_x = f_y = 25 / 0.00345 = 7246 px.
• Point: X_c = 10 mm, Y_c = 10 mm, Z_c = 300 (units already established).

The scale at the work plane is f_x / Z_c = 7246 / 300 = 24.15 px/mm, so each 10 mm offset projects to 24.15 × 10 = 241.5 px:

• u = 1224 + 241.5 = 1465.5 → 1466
• v = 1024 + 241.5 = 1265.5 → 1266

One pixel at the work plane covers 300 / 7246 = 0.0414 mm, about 41 µm, which makes the field of view 2448 × 0.0414 = 101 mm wide. This is the same working-distance and field-of-view relation used in the blister inspection geometry guide.

Does the projection land on the raw image or a rectified one?

It lands on the raw image: project with the intrinsics your calibration returned and the pixel is valid exactly as the sensor delivers the frame, before any rectification map is applied.

Intrinsics describe one specific image formation. Your calibration was computed from raw frames, so its parameters — including distortion — project correctly into raw pixels, which is exactly what an overlay needs. If you remap the frame through a rectification first, the delivered image is a resampled camera, and you must project into it with the rectified parameters; the same raw-versus-rectified split confronts stereo users, as in which intrinsics apply to rectified vs raw IR images. A warped image can alternatively be described by a homography, the route taken in mapping pixel coordinates into a warped image. Readout settings move the parameter pair too: 2×2 binning doubles the pixel pitch, halving f_x and f_y and scaling the principal point with them, so recalibrate after any binning, ROI or flip change.

Why does the same 10 mm land elsewhere when the part sits deeper?

The pixel offset scales as 1/Z_c: the same 10 mm offset seen at 600 mm instead of 300 mm lands 121 px instead of 242 px, because the mm-to-pixel factor halves whenever the depth doubles.

At Z_c = 600 mm the scale is 7246 / 600 = 12.08 px/mm and 10 mm draws 121 px. Depth error propagates as f_x·X_c/Z_c² per millimetre — 0.8 px per mm in this geometry — so a 3 mm depth error shifts the crosshair 2.4 px. The single-constant shortcut, multiplying all millimetres by 24.15, is therefore valid only for parts on one plane parallel to the sensor at exactly the calibrated depth. For parts with height variation or a tilted presentation, transform each point to camera coordinates first and let Z_c come from the pose, never from a stored constant. The inverse mapping — pixels to millimetres on a known plane — is the axis scaling in mapping pixels to real-world sizes with imagesc axis coordinates, and a pixel plus a depth value becomes full 3D through the deprojection calculation.

What does the projection code look like, and how do you test it?

About ten lines cover it: one function taking the intrinsics and camera coordinates as floats and returning rounded pixels, plus a test asserting the 10 mm example lands at pixel (1466, 1266) — run it whenever you touch the calibration.

def project(fx, fy, cx, cy, xc, yc, zc):
    # fx, fy: focal length in px (float); cx, cy: principal point in px (float)
    # xc, yc, zc: point in camera coordinates in mm (float); zc must be > 0
    if zc <= 0:
        raise ValueError('point at or behind the camera')
    u = fx * xc / zc + cx
    v = fy * yc / zc + cy
    return round(u), round(v)

fx = 25000.0 / 3.45  # 25 mm focal length, 3.45 um pixels -> 7246.38 px
u, v = project(fx, fx, 1224.0, 1024.0, 10.0, 10.0, 300.0)
assert (u, v) == (1466, 1266), (u, v)

The assert is the test: it fails loudly if a refactor, a unit slip or an argument swap breaks the geometry. If your SDK's equivalent returns anything else for these inputs, check argument order first — many take (row, column) or (y, x) — then units. Three mistakes account for most wrong overlays:

1. Mixed units. A pose in metres with a point in millimetres moves the crosshair a thousandfold.
2. Rectified parameters on a raw image, or the reverse — shifts by roughly the distortion amount.
3. Convention slip. v is the row and u is the column, and rows grow downward.

Which camera and lens make the overlay land where it should?

Match three things: a lens whose image circle covers the sensor, a focal length that puts the whole part inside the field of view, and distortion low enough that corner overlays stay within a few pixels — for example 0.3%-class optics on a 20 MP sensor.

The focal length follows from the geometry you already have: FOV = sensor width × WD / f, so f = WD × sensor width / FOV. To frame 100 mm on a 2/3" sensor (8.8 mm wide) at 300 mm, f = 300 × 8.8 / 100 = 26.4 mm, and a 25 mm lens on that sensor delivers a 105.6 mm field. The calculator does this arithmetic and filters the lens finder to the focal length it computes:

Distortion is the second number that matters for overlays. At 0.3% radial distortion the worst corner pulls 0.3% of the half-diagonal — 7.5 px on a 4000 × 3000 sensor — and the calibrated distortion model pulls that back to a fraction of a pixel. The LS-FFL-0567 is a 16 mm, 20 MP C-mount lens for 1.1" sensors held below that 0.3% figure.

For the geometry in the worked example — a 2/3" sensor at a moderate field — the LS-FFL-0679 is a 35 mm C-mount lens rated for 6 MP on 2/3" sensors. A 35 mm lens on a 2/3" sensor frames about 75 mm wide at that working distance (8.8 × 300 / 35 ≈ 75 mm).

On the camera side, sample the field as finely as the overlay tolerance demands: the 41 µm pixel of the example shrinks in proportion as pixel count rises. Area scan cameras from 2.5 to 20 MP over GigE and USB3:

If the part moves during exposure, choose a global shutter model — the MV-CU020-90GC is a 2.5 MP GigE camera with a global shutter and C-mount — because rolling-shutter smear drags a moving feature off its projected pixel.

Start from the camera finder, browse the machine vision lenses, or pair sensor, lens and working distance in the camera configurator and the lens configurator.

How accurate is a pinhole projection in practice?

Plan for single-digit pixels: a good calibration reprojects checkerboard points at 0.1–0.3 px, uncorrected distortion adds up to about 7 px at the corners of a 4000 × 3000 sensor with 0.3%-class optics, and a 1% depth error moves the result by 1%.

Budget the error in layers. Calibration quality comes first — most checkerboard tools print the reprojection error, and a value above roughly 0.5 px means the board images or the lens model deserve another pass. Uncorrected distortion contributes zero at the centre and the corner figures from the last section at the edges, collapsing once the coefficients are applied. Depth error scales the whole offset by ΔZ/Z, so 1% of the depth (3 mm here) moves the 242 px crosshair 2.4 px. Mechanics show up last: a bumped focus ring or a re-seated mount changes the flange distance by fractions of a millimetre and costs pixels until you recalibrate — the failure mode covered in the C-mount flange focal distance guide.

Before you trust the overlay in production:

• Reprojection error from the last calibration below 0.5 px.
• Intrinsics read from the calibration file, not computed from resolution.
• Pipeline unchanged since calibration: no binning, ROI shift, flip or rectification added.
• One unit everywhere: pose, point and focal length consistent before the division.
• Z_c sign-guarded, and each point's depth taken from the pose transform, not a stored px/mm constant.

More coordinate-chain and calibration guides sit under the 3D vision topic and in the learn hub. If the overlay has to land tighter than your calibration allows, describe the sensor, lens and working distance through the contact page and we will check the geometry with you.