Project a 3D mesh to a 2D image and turn it into a mask

You have a triangle mesh - a CAD model fitted to a depth capture, a fitted hand or object model, a triangulated point cloud - and the 3x3 intrinsic matrix of the camera that took the 2D image. Plotting the projected vertices gives you a scatter of dots, not the solid segmentation mask a mesh rendering shows. The fix is two ideas: project every vertex with the pinhole equations, then draw each projected triangle as a filled polygon instead of a point. This how-to covers the projection, the renderer-to-camera axis flip that mirrors everything if you skip it, the rasterization in five numbered steps, and an area check that proves the mask before you ship it. The arithmetic is worked end to end, so you can check your own matrix and mesh against it in minutes.

What you need

• Mesh vertices: a float array of shape (N, 3), one row per vertex, in any one unit - metres or millimetres both work.
• Triangle indices: an integer array of shape (M, 3), each row listing three indices into the vertex array.
• Intrinsic matrix K: 3x3, with the focal lengths fx and fy in pixels on the diagonal and the principal point (cx, cy) in the last column. It comes from the calibration your 3D sensor SDK reports, from a chessboard calibration, or from the annotation files shipped with a dataset.
• Image size: the height and width of the 2D image in pixels, because the mask canvas must match it exactly.
• Python with NumPy and OpenCV: cv2.projectPoints does the geometry and cv2.fillPoly does the drawing. Their references live in the OpenCV calib3d module and the OpenCV drawing functions.

The projection, one formula

A vertex already in camera coordinates (X, Y, Z), with Z positive, lands on the pixel

fx and fy scale angles into pixels; cx and cy move the origin from the optical axis onto the pixel grid. The division by Z is what makes the result perspective-correct: parts of the mesh farther from the camera compress toward the principal point exactly as they appear in the photograph. cv2.projectPoints applies these equations to a whole (N, 3) array at once; when your vertices are already in the camera frame, pass a zero rotation vector and a zero translation.

Flip render axes to camera axes first

Mesh tools and viewers almost all use the render convention: X right, Y up, and the camera looking along -Z. Image coordinates and OpenCV use X right, Y down, camera looking along +Z. Convert before projecting:

In code that is one line: verts_cam = verts @ np.diag([1.0, -1.0, -1.0]). Skip it and every depth is negative, so the projection folds the mesh through the principal point: the mask comes out mirrored and upside down, sitting on the wrong side of the image. Vertices that come straight out of a depth camera SDK in its optical frame are already camera coordinates and need no flip.

Project and rasterize in five steps

1. Load the arrays. verts as float64 with shape (N, 3), tris as int32 with shape (M, 3). Check tris.max() < len(verts) first, so a bad index fails here with a readable error.
2. Convert to camera coordinates. Multiply by diag(1, -1, -1) when the mesh came from a renderer; use the vertices as they are when they are already in the camera frame.
3. Project. imgpts, _ = cv2.projectPoints(verts_cam, np.zeros(3), np.zeros(3), K, np.zeros(5)), then reshape the (N, 1, 2) result to (N, 2) with imgpts.reshape(-1, 2).
4. Filter by depth. Keep only triangles whose three corners have Z above zero: depth_ok = (verts_cam[tris][:, :, 2] > 0).all(axis=1). A partly behind-camera triangle projects inside out and paints a large wrong region.
5. Fill. Allocate mask = np.zeros((H, W), np.uint8) and make one call, cv2.fillPoly(mask, px[tris][depth_ok], 255), where px[tris] is the (M, 3, 2) int32 array of per-triangle corner pixels. The canvas-and-fill pattern is the same one used to create a black mask image with a filled circle ROI; only the polygon list differs.

Put together:

import numpy as np
import cv2

K = np.array([[800.0, 0.0, 320.0],
              [0.0, 800.0, 240.0],
              [0.0, 0.0,   1.0]])           # 3x3 intrinsic matrix, pixels
H, W = 480, 640                           # image height, width, pixels

verts = np.array([[ 0.10,  0.05, -1.00],  # (N,3) vertices, render frame, metres
                  [-0.10,  0.05, -1.00],
                  [ 0.00, -0.10, -1.00]])
tris = np.array([[0, 1, 2]], dtype=np.int32)  # (M,3) corner indices
assert tris.max() < len(verts)

verts_cam = verts @ np.diag([1.0, -1.0, -1.0])      # (N,3) camera frame, Z > 0

imgpts, _ = cv2.projectPoints(verts_cam,            # (N,1,2) pixel coordinates
                              np.zeros(3), np.zeros(3), K, np.zeros(5))
px = imgpts.reshape(-1, 2).astype(np.int32)         # (N,2)

depth_ok = (verts_cam[tris][:, :, 2] > 0).all(axis=1)   # (M,) boolean
mask = np.zeros((H, W), dtype=np.uint8)
cv2.fillPoly(mask, px[tris][depth_ok], 255)

Worked example: one triangle end to end

Take a 640x480 image with fx = fy = 800 px, cx = 320 px, cy = 240 px, and one triangle in render coordinates, in metres: A = (0.10, 0.05, -1.00), B = (-0.10, 0.05, -1.00), C = (0.00, -0.10, -1.00).

The flip puts the camera-frame vertices at A = (0.10, -0.05, 1.00), B = (-0.10, -0.05, 1.00), C = (0.00, 0.10, 1.00). Now the pinhole equations, one corner at a time:

• A: u = 800 × 0.10 / 1.00 + 320 = 400 px, v = 800 × (-0.05) / 1.00 + 240 = 200 px, so pixel (400, 200)
• B: u = 800 × (-0.10) / 1.00 + 320 = 240 px, v = 800 × (-0.05) / 1.00 + 240 = 200 px, so pixel (240, 200)
• C: u = 800 × 0.00 / 1.00 + 320 = 320 px, v = 800 × 0.10 / 1.00 + 240 = 320 px, so pixel (320, 320)

The shoelace formula gives the expected area: ½ × |400 × (200 - 320) + 240 × (320 - 200) + 320 × (200 - 200)| = ½ × 19,200 = 9,600 px². Cross-check by base and height: the base from (240, 200) to (400, 200) spans 160 px, the apex sits 120 px below it, and ½ × 160 × 120 = 9,600 px².

The test that proves the polygon filled rather than three dots landing:

area = np.count_nonzero(mask)          # filled pixel count
assert abs(area - 9600) < 300          # shoelace area plus boundary pixels
assert mask[200, 400] == 255           # corner A
assert mask[200, 240] == 255           # corner B
assert mask[320, 320] == 255           # corner C

Three plotted points would leave a handful of lit pixels; roughly 9,600 means the triangle is solid.

Three checks before you trust the mask

1. Vertex containment. mask[v, u] == 255 for every projected corner that falls inside the frame. A zero points at a flip, a bad index or a transposed (row, column) pair.
2. Area against the shoelace sum. Add the per-triangle pixel areas and compare with np.count_nonzero(mask). Agreement within the boundary-pixel tolerance means the depth filter let nothing inside out.
3. Extent against physics. A feature 0.20 m wide at 1.00 m depth with fx = 800 px must span 800 × 0.20 / 1.00 = 160 px in the mask. If the bounding box is half or twice that, Z or fx is in the wrong units.

Common mistakes

• Skipping the axis flip, which folds the whole mask through the principal point.
• Projecting world-frame vertices with a zero pose: cv2.projectPoints expects camera-frame points, so either apply the mesh pose first or pass it as rvec and tvec.
• Filtering depth after the fill, when the inside-out triangles have already painted.
• Handing fillPoly float coordinates: cast the projected pixels to int32 and keep the canvas uint8.
• Zero distortion on a lens you calibrated yourself: pass the k1, k2, p1, p2 vector from your calibration, because an un-rectified wide lens moves corner points by many pixels - enough to slide the mask off the object.

Where this fits on a 3D camera line

On a live station the same code becomes the verification overlay: register your CAD mesh to the captured point cloud to get the pose, project the mesh onto the 2D image from the same device, and confirm the fit before a robot acts on it. The surrounding pieces are covered elsewhere: projecting a 2D image onto a depth map goes in the reverse direction, averaging depth over the pixels a segmentation mask selects turns the mask into a measurement, and mapping image pixels to 3D points casts the inverse ray. Structured-light 3D cameras return the depth map and the registered 2D image from one device, so the overlay runs on a single capture:

Masks travel well beyond inspection code. If one ever has to live in a dashboard or report layer, cropped to a shape, the same fill idea applies - a walkthrough of circular image masks for report dashboards shows that side.

Shortlist cameras for the station in the 3D camera finder, work the system configurator to match sensor, lens and controller to your field of view, and browse the 3D vision topic page for the rest of the projection and masking toolbox. For a second pair of eyes on a fit, contact the applications team.