Point cloud after resizing RGB and depth images: rescale intrinsics

You cut an RGB and depth pair down to a quarter of the linear size to save bandwidth and CPU on an embedded obstacle detector, deproject, and the point cloud lands in the wrong place: compressed to one side, or thin as a wedge. The geometry did not change; your intrinsics are still expressed in full-resolution pixels. Divide fx, fy, cx and cy by the same factor you resized by and the cloud comes back exact. This guide gives the scaling rule, the pipeline order (align, then resize, then deproject), the interpolation rule for depth maps, and the checks that prove the rescale is right. It applies to any depth camera that exposes a pinhole profile — stereo, structured light or ToF — and to rigs where the color frame comes from a separate area scan camera.

Why does my point cloud collapse after I resize the frames?

Because the deprojection still runs on full-resolution intrinsics: divide fx, fy, cx and cy by the resize factor k, so a k = 4 resize of a 640 × 480 depth frame needs fx 384 px rescaled to 96 px before a single point is computed.

Deprojection is pinhole geometry: pixel (u, v) at depth Z maps to X = (u − cx) · Z / fx and Y = (v − cy) · Z / fy, with all four intrinsics in pixels of the original frame. The pinhole camera model reference covers where those numbers come from. After a resize by k your pixel coordinates live in a grid k times smaller, but the intrinsics still describe the old grid. The rescue is that the projection equation is scale-invariant: (k·u − k·cx) / (k·fx) = (u − cx) / fx. Divide all four parameters by k and every pixel rides the same ray as before.

Exactly which intrinsics do I divide?

Divide the intrinsics of the camera whose pixel grid the resized depth values sit on: the depth camera for an unaligned cloud, the color camera after depth-to-color alignment, because the aligned frame lives on the color grid and its fx, fy, cx and cy are the ones that must scale.

Unaligned, the depth frame sits on the depth camera's grid, so the depth profile's parameters are the ones to scale. Alignment resamples the depth values onto the color camera's grid, so the deprojection must then run on the color intrinsics — the full reasoning is in which camera to deproject an aligned depth frame with. Read the numbers from each camera's own profile and never copy one camera's focal length to the other; reading the depth scale and focal length per camera covers that discipline.

The rule is camera-agnostic: stereo, structured-light and ToF sensors all expose a pinhole profile, and a rig pairing a depth sensor with a separate color camera is covered too, because the aligned depth lands on that camera's grid. A safety note that belongs here: structured-light and laser-stereo models project infrared or laser patterns, so hold to the manufacturer's laser class and viewing-distance limits — the pattern is invisible while the source is not automatically harmless. For a new rig, this is the stereo end of that hardware class:

Projected structured-light sensors, and a separate USB area scan camera carrying the color side of a rig, follow the same rule:

Should I align depth to color before or after resizing?

Align first, at native resolution, then resize both frames by the same factor — an aligned depth frame is written onto the color stream's resolution, so resizing before alignment either wastes the reduction or breaks the pixel correspondence between the two images.

The align processing block uses the two streams' factory extrinsics and their native intrinsics. Decimate the depth frame with the SDK's post-processing decimation filter first and alignment still works, because the filter recalculates the frame's intrinsic parameters to compensate for the resolution change, exactly as the SDK documentation describes. But the aligned output is written at the target stream's resolution: decimate depth and then align to a full-size color frame, and you get a full-size frame back. The reduction bought processing savings, not a smaller cloud. To actually shrink the cloud, align at native resolution, then resize the aligned depth and the color frame together. Read the aligned frame's width and height from the frame rather than assuming them — depth frame resolution changes after alignment catches that trap. Aligning color onto a decimated depth grid instead keeps the small cloud and costs color sharpness; that direction suits visualization, not measurement.

How do I resize a depth map without inventing surfaces?

Pick, never blend: use nearest-neighbour or minimum pooling over each k × k block, because bilinear or area interpolation averages two real surfaces into one false surface between them, which appears as ghost geometry along every depth edge.

Minimum pooling is the conservative pick: it keeps the closest surface in the block, so a near obstacle survives when it covers just 1 of the 16 pixels at k = 4. A median is stricter — a near surface must cover 9 of 16 pixels to own the median, so one covering 8 disappears. Nearest picking is the fast middle ground and is enough for visualization. The color frame can take bilinear or area interpolation without harm; averaging RGB is fine, averaging depth is not. Pick raw depth codes or metres, whichever you hold — picking copies values, so converting with the stream's depth_scale before or after changes nothing.

Is there a filter that rescales the intrinsics for me?

Yes: the SDK's post-processing decimation filter, rs2::decimation_filter, reduces the depth frame by an integer factor and recalculates the frame's intrinsic parameters to compensate for the resolution change, so pc.calculate deprojects the filtered frame correctly with no manual scaling.

Set the filter's decimation factor to 4 and a 640 × 480 depth frame comes out 160 × 120 with corrected intrinsics attached. The catch is what happens inside each block: the rs-depth-filter sample reduces with a median, which is exactly the behaviour an obstacle detector should not inherit blindly. If nearest-surface safety matters, keep the reduction in your own code with minimum pooling and the four divisions from the formula above. The camera-side cousin of this problem is hardware binning — camera intrinsics after binning shows why the binned profile's own intrinsics must carry the reconstruction.

What does the rescale look like with real numbers?

A 640 × 480 aligned depth frame with color intrinsics fx = 384.0 px, fy = 384.0 px, cx = 320.0 px and cy = 240.0 px, resized by k = 4 to 160 × 120, deprojects correctly with fx' = 96.0 px, cx' = 80.0 px and cy' = 60.0 px.

The arithmetic, step by step:

• Grid: 640 / 4 = 160 and 480 / 4 = 120, so the resized frame is 160 × 120.
• Scaled intrinsics: fx' = 384.0 / 4 = 96.0 px, fy' = 96.0 px, cx' = 320.0 / 4 = 80.0 px, cy' = 240.0 / 4 = 60.0 px.
• A scene point at full-resolution pixel (400, 300), Z = 2.00 m: X = (400 − 320) × 2.00 / 384.0 = 0.42 m and Y = (300 − 240) × 2.00 / 384.0 = 0.31 m.
• The same point after resize sits at (100, 75): X = (100 − 80) × 2.00 / 96.0 = 0.42 m and Y = (75 − 60) × 2.00 / 96.0 = 0.31 m. Identical, as it must be.
• Forget the rescale and push the resized pixel through the original intrinsics: X = (100 − 320) × 2.00 / 384.0 = −1.15 m. One missing division moves the point 1.56 m, from 0.42 m right of the axis to 1.15 m left of it.

# Illustrative values: color-stream intrinsics, resize factor, one pixel.
k = 4                                # int, resize factor
fx, fy = 384.0, 384.0                # float, px, from the color profile
cx, cy = 320.0, 240.0                # float, px
u, v, z = 100, 75, 2.00              # int px, int px, float metres

fx_s, fy_s = fx / k, fy / k          # 96.0, 96.0
cx_s, cy_s = cx / k, cy / k          # 80.0, 60.0
X = (u - cx_s) * z / fx_s            # 0.4167 m
Y = (v - cy_s) * z / fy_s            # 0.3125 m

# Test: the full-resolution pixel (400, 300) must give the same point.
X_full = (400 - cx) * z / fx         # 0.4167 m
assert abs(X - X_full) < 1e-9

If your cloud comes out in the wrong units rather than the wrong place, that is the depth-to-metres factor, not the intrinsics — depth image to point cloud per-axis scale factors covers that scaling.

How much accuracy do I give up at one quarter size?

Lateral resolution drops by exactly k while depth accuracy and field of view are unchanged: at 2.00 m one resized pixel spans 2.00 / 96 = 20.8 mm across the scene, against 5.2 mm at full resolution, and the cloud shrinks from 307,200 points to 19,200.

Per-pixel footprint is Z / fx': 2.00 m / 96 px = 20.8 mm, against 2.00 m / 384 px = 5.2 mm at full size. Field of view does not move: horizontal FOV is 2 · atan(W / (2 · fx)), and 640 / 768 = 160 / 192 = 0.833, so both grids give 79.6°. Depth values keep their measured accuracy, and minimum pooling biases each block toward the nearest surface — the safe direction. The practical limit is small objects: a 10 mm post at 2.00 m spans 96 × 0.010 / 2.00 = 0.48 resized pixels against 1.92 pixels at full resolution, so nearest picking can drop it between samples while minimum pooling over its block still catches it. The cloud itself falls from 307,200 points to 19,200, a 16-times cut in memory and deprojection time.

What is the full pipeline, step by step?

Align depth to color at native resolution, min-pool or nearest-pick both frames down by the same integer k, build scaled intrinsics from the color profile, deproject every valid pixel, then test — six steps, and the test step is one assertion long.

1. Open the depth and color streams at native resolution; read each profile's intrinsics (fx, fy, cx, cy) and the depth stream's depth_scale.
2. Align depth to color, then read the aligned frame's width and height from the frame itself, because they follow the color stream, not the depth stream.
3. Resize the aligned depth by integer k — minimum pooling for obstacle work, nearest picking otherwise — and resize the color frame by the same k with any interpolation. Crop first if either dimension is not divisible by k.
4. Build the scaled intrinsics fx' = fx / k, fy' = fy / k, cx' = cx / k, cy' = cy / k, keeping the distortion model and its coefficients exactly as reported.
5. Deproject every pixel with valid depth: X = (u − cx') · Z / fx' and Y = (v − cy') · Z / fy', with Z = raw × depth_scale in metres.
6. Test before shipping: the pixel at (cx', cy') must deproject to X = Y = 0; a landmark pixel must match its full-resolution deprojection within one resized pixel's footprint (20.8 mm at 2.00 m here); and 2 · atan(W' / (2 · fx')) must still equal 79.6°.

The center-pixel test catches an unscaled principal point in seconds, the cross-resolution test catches an unscaled focal length, and the FOV test catches a wrong grid size. All three are one-line assertions in the snippet above.

When is a quarter-size resize the wrong way to shrink the cloud?

Shrink at the source when you can: if the camera lists a native depth mode at the resolution you need, request it instead of resizing in software, because a native 160 × 120 mode moves 16 times less data over the link and needs no intrinsics arithmetic at all.

Three cases argue against software resizing. First, the native mode above — setting a listed depth mode is the whole procedure, and the mode's own intrinsics arrive matched to its grid. Second, when you need full lateral resolution but fewer points, crop to the region of interest before deprojecting, or decimate the finished cloud in 3D: reducing stereo point cloud size before processing lags covers the timing, and why decimation shifts points covers the geometry cost. Third, when the target is a feature smaller than one resized pixel's footprint, the resize is the defect — keep native resolution and cut cost elsewhere.

Specifying the rig rather than the code? The camera finder filters depth-capable sensors by type, the configurator walks camera, lens and light, and sizing questions go to our team through the contact page. Everything else on depth capture sits on the 3D topic page.