Crop a depth ROI in 2D before the pointcloud reaches OpenCV

You ran the point cloud on a depth frame, cut what looked like a region from the vertices, and handed it to OpenCV - and the tracker, the statistics, everything downstream came back empty. The array itself looks suspicious: negative numbers and long runs of zeros. Nothing is corrupt. A vertex list is an (N, 3) float array, one row per pixel, and cv2.selectROI is a mouse tool that displays a 2D image and waits for a drag; it has nothing it can render a point cloud into. The fix is order of operations: pick the rectangle on the depth image, mask the depth band you want (1.5-3 m in the example here), and touch the vertices last, only for the pixels that survived. This guide gives the full pipeline, the arithmetic that predicts every coordinate, and the checks that prove the crop worked.

Why the vertices array broke your pipeline

An OpenCV region is a rectangle on an image: four integers, x, y, w, h. cv2.selectROI earns them by opening a window, drawing the image you pass it and recording your mouse drag. Pass it a vertex buffer and there is no image to draw - the buffer is an (N, 3) float32 array with N = width × height, one vertex per pixel in reading order. Runs of (0, 0, 0) are the pixels where the sensor returned no depth; they are placeholders, not coordinates sitting at the camera. That is why the downstream call comes back empty: nothing in that array is a picture.

The negative values are correct geometry. The vertex origin sits at the depth sensor's optical center: X grows to the right, Y grows down the rows, Z grows away from the lens. Every pixel left of the principal point produces a negative X, every row above it a negative Y, so about half of any centered scene is negative on each axis. Z never goes negative on a healthy stream. Each vertex is just the depth image run through the pinhole model:

with Z the depth at pixel (u, v) and fx, fy, cx, cy the depth stream's intrinsics in pixels. Depth image to point cloud: apply per-axis scale factors covers the same deprojection with the per-axis scale factors spelled out.

What you need

• A depth stream as a 2D NumPy array: uint16 millimeters is the common layout, float32 meters also appears. Read the unit out of the stream's scale field before you pick thresholds.
• The depth intrinsics: fx, fy, cx, cy in pixels, supplied by the SDK alongside the stream profile.
• Python 3 with OpenCV and NumPy; no particular versions are needed for these calls.
• Optionally, a color frame registered to the depth resolution for a more readable selection view - How to register RGB depth images for a colored point cloud shows that step.

To make the uint16 depth viewable, normalize it to 8 bits for display only and keep the raw array untouched, the approach in Normalize a depth map for JET colormap display, keep the values. One safety note: structured-light and ToF depth cameras carry an infrared projector - an active illuminator. Do not run the camera with its housing open, and do not stare into the emitters at close range.

The pipeline: rectangle in 2D, band mask, vertices last

The rule, once and for all: never hand OpenCV a vertex array. Crop the 2D depth image, mask the band, deproject last.

1. Build a viewable image. Normalize the depth to 8-bit and apply a color map, or use the registered color frame. The window is only for your eyes; the numbers stay in the raw array.
2. Draw the rectangle. Call cv2.selectROI on that view; Enter or space confirms, c cancels and returns (0, 0, 0, 0) - treat an all-zero rectangle as an abort, not as a crop of the top-left pixel. Where the rectangle should sit is its own design question; 2D vs 3D camera ROI: where to set it and which one wins works through it.
3. Slice the depth image. NumPy ranges are half-open: depth[y:y+h, x:x+w] covers columns x to x+w-1, the same convention the rectangle means.
4. Mask the depth band. keep = (d >= 1500) & (d <= 3000) for a 1.5-3 m band on a millimeter stream. Any floor above zero deletes the invalid (0, 0, 0) pixels as a side effect.
5. Keep a 2D image for the tracker. Optical flow, templates and blob trackers want pictures: band = np.where(keep, d, 0) is a clean image with out-of-band pixels zeroed.
6. Deproject only the survivors. Build X, Y for the rectangle's pixels and select with the same mask. If your SDK already returns vertices, reshape the flat buffer to (height, width, 3) before slicing - slicing an (N, 3) buffer with 2D ranges cuts along the wrong axis; Numpy depth image to point cloud: fix strides and dtypes dissects that trap.
7. Hand the result downstream. Trackers take band; anything metric takes the (n, 3) points.

import cv2
import numpy as np

# Example inputs for a 640 x 480 millimeter stream - swap in your frames
depth = np.zeros((480, 640), dtype=np.uint16)   # mm, 0 = no depth
depth[160:320, 214:426] = 1876                  # target plane, 1.876 m
depth[300:320, 300:340] = 940                   # clutter edge, 0.94 m

view = cv2.normalize(depth, None, 0, 255, cv2.NORM_MINMAX, dtype=cv2.CV_8U)
view = cv2.applyColorMap(view, cv2.COLORMAP_JET)
x, y, w, h = cv2.selectROI("select target", view, showCrosshair=True,
                           fromCenter=False)     # -> (214, 160, 212, 160)

d = depth[y:y + h, x:x + w].astype(np.float32)   # shape (160, 212)
keep = (d >= 1500.0) & (d <= 3000.0)             # 1.5-3 m band
band = np.where(keep, d, 0.0).astype(np.uint16)  # 2D image for trackers

fx = 580.0                 # pixels, from the depth intrinsics
fy = 580.0
cx, cy = 319.5, 239.5      # principal point, pixels

u, v = np.meshgrid(np.arange(x, x + w, dtype=np.float32),
                   np.arange(y, y + h, dtype=np.float32))
X = (u - cx) * d / fx      # mm; negative left of the optical axis
Y = (v - cy) * d / fy      # mm; negative above the optical axis
pts = np.stack([X[keep], Y[keep], d[keep]], axis=1)   # (n, 3) float32

# Same crop on an SDK vertex buffer, if you use one:
# verts3 = verts.reshape(480, 640, 3)[y:y + h, x:x + w]
# pts = verts3[keep]

Worked example: one target at 1.876 m

The example scene above yields the following numbers. The drag returns x = 214, y = 160, w = 212, h = 160, so the rectangle holds 212 × 160 = 33,920 pixels. The target plane sits at 1876 mm; a clutter edge at 940 mm crosses its lower rows. The band mask keeps 33,920 - 800 = 33,120 pixels, because the 800 clutter pixels fail the 1500 mm floor.

Corner coordinates, from the formula with fx = 580 px and cx = 319.5 px:

• Left edge, u = 214: X = (214 - 319.5) × 1876 / 580 = -341 mm
• Right edge, u = 425: X = (425 - 319.5) × 1876 / 580 = +341 mm
• Top edge, v = 160: Y = (160 - 239.5) × 1876 / 580 = -257 mm
• Bottom edge, v = 319: Y = (319 - 239.5) × 1876 / 580 = +257 mm

So the kept patch spans about 682 mm × 514 mm at a 1.876 m standoff, every kept Z equals 1876 mm, and the point spacing is 682 / 212 = 3.2 mm per pixel (514 / 160 = 3.2 on the other axis - square pixels, as they should be). The horizontal field of view behind those numbers is 2 × atan(320 / 580) = 57.7°; swap in your own fx and the arithmetic transfers unchanged. The signs carry information: columns 214-319 give negative X and columns 320-425 give positive X - 106 columns each, exactly the left and right halves of a centered rectangle.

The zeros are missing depth, not bad coordinates

A depth camera does not measure every pixel: dark surfaces, glossy highlights, out-of-range geometry and occlusions behind structured-light shadows all return nothing. The vertex buffer carries those as (0, 0, 0) and the depth image as 0 mm. Keep them and every mean, minimum or fit sinks toward the floor. A band mask with any floor above zero removes them for free, which is why the 1.5 m floor and the invalid-pixel problem solve each other in one line. If your band starts at zero - say 0 to 0.5 m for a near-field gesture task - add a d > 0 term to the mask. For grading pixel quality rather than a binary keep/drop, decode the camera's confidence map; 3D camera confidence image values: decode bitflags and mask bad pixels shows the bit flags.

When the camera has no min/max depth option

Not every depth camera exposes a depth-range control, and on some models the control clips at a ceiling the sensor cannot reach anyway. The software band above is the complete substitute: it acts on the delivered frame, costs one boolean mask, and runs the same comparison the camera would run internally.

The one thing a mask cannot do is add range. The short-range structured-light class is built for desk distances: expect the specified accuracy inside about 1 m and hard ceilings near 1.2-2 m. If your band is 1.5-3 m, the top half of the band is simply not measured and no filter recovers it - the decision there is the sensor class, not the code. Stereo structured-light and ToF cameras are the classes that work at those distances. The MV-DPS200P-02 is a projected structured-light 3D camera, and the MV-DT01WMU is a ToF sensing camera; both hand you the per-pixel depth map this pipeline runs on.

Standoff and mounting then follow from the band: place the camera so the target crosses the band with margin, using the sizing steps in the 3D camera working distance and positioning guide. For tight Z layers the same mask scales down to millimeter bands - configuring one-inch Z-layer scans on a depth camera works that variant end to end.

Seven mistakes that produce an empty result

1. selectROI on the vertex buffer. There is no image to draw, so the drag never happens and the return is empty. Select on the depth view or the registered color frame.
2. Slicing a flat (N, 3) buffer with 2D ranges. verts[y:y+h, x:x+w] cuts rows of vertices, not pixels of image. Reshape to (height, width, 3) first.
3. Thresholds in the wrong unit. 1500 as a meter bound keeps everything; 1.5 as a millimeter bound keeps nothing. Print d.min() and d.max() once and the unit is obvious.
4. A color-frame rectangle applied to the depth frame. Different resolutions shift the rectangle; register the streams, or scale x, y, w, h by the resolution ratio.
5. Expecting a metric box from a 2D rectangle. The rectangle is a frustum: at 1 m the example rectangle spans ±182 mm, at 1.876 m it spans ±341 mm. When the bounds must be constant in millimeters, deproject first and filter X and Y - Crop XY region of interest on a 3D depth camera point cloud is that variant.
6. Masking after deprojection. Building N points and discarding most of them wastes the frame budget on arithmetic you throw away; Pointcloud filter slows depth camera FPS: fix the CPU load measures the cost. Mask the 2D depth, deproject the survivors.
7. Treating a cancelled drag as data. An all-zero (0, 0, 0, 0) rectangle means the operator pressed c; abort the cycle instead of tracking the top-left pixel.

Verify the crop before you trust it

Run the asserts against the example scene; they generalize to any rectangle and band:

assert pts.shape == (int(keep.sum()), 3)
assert pts[:, 2].min() >= 1500.0 and pts[:, 2].max() <= 3000.0
assert (pts[:, 2] > 0).all()
print(pts.shape, round(float(pts[:, 0].min()), 1), round(float(pts[:, 0].max()), 1))
# -> (33120, 3) -341.2 341.2

Three checks carry the meaning: the point count equals the mask count (33,120 here), every Z sits inside the band, and the X minimum is negative while the maximum is positive - the signature of a rectangle straddling the optical axis. Visually, the clutter edge at 940 mm must vanish from band while the target plane keeps its texture; if the whole view goes black, the mask is empty and mistake 3 above is the usual cause.

Checklist: from selection to production

• Read the depth unit from the stream's scale field before writing thresholds; write the band once as named constants.
• Give the mask a floor above zero, or add d > 0, so invalid pixels cannot enter any statistic.
• Confirm the camera's range class covers the top of your band - a software mask cannot extend it.
• Select the rectangle on a stream registered to the depth resolution, or scale the rectangle to it.
• Mask the 2D depth before deprojecting; hand trackers the band image and metric code the pts array.
• Abort cleanly on a cancelled selection instead of running the cycle on a zero rectangle.

The same rectangle-and-band logic is what our 3D sensors category feeds, what the camera finder filters by sensor type, and what the configurator's camera step walks through with lens and mount choices. If your band, target speed or camera shortlist needs a second opinion, the contact page reaches the applications team - free expert support comes with every order.