Connected components: 3D point cloud segmentation by size

A depth camera hands you one cloud: parts, fixtures and background in a single point set. To count parts, check sizes or feed a picker, you need that set split into individual objects, each with a volume and a diameter you can test. The standard pipeline has four steps. Threshold the depth in Z so only the parts' surfaces remain. Join points that sit within a distance tolerance of each other, which is the connected-components step, so every part becomes its own cluster. Measure each cluster's volume against a reference plane and its maximum point-to-point diameter. Then keep the clusters whose numbers fall inside a window sized from the part. This page gives the numbers for one worked scene: a 10 mm clustering tolerance, a 0.35-1.0 liter volume window, a 185-300 mm diameter window, and the arithmetic behind all three.

The four steps and what each decides

The example scene is a row of dome-shaped covers on a table, captured by a time-of-flight camera and stored as X, Y and Z channels. Any depth cloud from any sensor runs through the same four steps.

Machine-vision libraries differ in spelling, not in substance. The operators below use the 3D object-model naming that several libraries share; the Point Cloud Library carries the same clustering algorithms in its pcl_segmentation module, again driven by a distance tolerance. If the same cloud is processed repeatedly, preparing the model once for segmentation (prepare_object_model_3d with 'segmentation') speeds up every later call.

Step 1: threshold the depth in Z

Depth data arrives as three channels, X, Y and Z. Convert to meters first: the example scene multiplies raw values by 0.001, so a stored 670 becomes 0.670 m. In device units, every volume below is wrong by the same unknown factor; the fixed mapping comes from the depth-to-distance calibration for XYZ point clouds.

Then one call does the cut: select_points_object_model_3d with the feature 'point_coord_z', a minimum of 0.500 m and a maximum of 0.670 m keeps only the points inside that band. The band does two jobs. The lower limit throws out stray points closer than 0.5 m: dust, ghost reflections, a fixture leaning into view. The upper limit cuts the scene at the background plane. Keep the pairing in mind: the upper limit of 0.670 m returns in step 3 as the reference plane for every volume.

Two details decide whether the threshold means what you think. The axis first: in the optical frame, z grows with distance from the camera, so a point below the plane in the image sits farther from the lens. If your cloud lives in another frame, convert it first, as in transforming depth point clouds between the optical frame and the camera frame. A flat cut is also the fast way to drop a table or floor; fitting and subtracting the ground plane is the general version, as in removing ground points from a depth point cloud for obstacle detection.

Step 2: cluster the remaining points by 3D distance

connection_object_model_3d with the method 'distance_3d' and a threshold of 0.010 m walks the point set: any two points closer than 10 mm in straight-line 3D distance join the same component, and the rule chains. A is 8 mm from B and B is 9 mm from C, so all three land in one cluster even though A and C sit 17 mm apart. The output is one 3D object model per connected component.

The tolerance is the one number that decides whether a part arrives as one object or three. Size it from two measurements taken on a few test captures: the smallest gap between neighboring parts, g, and the largest hole inside a single part, such as a black or glossy patch that returned no depth. Set the tolerance to g/3. The covers in the worked scene sit at least 30 mm apart, so 30/3 = 10 mm.

A hole wider than the tolerance splits one cover into two undersized clusters that both fail the volume window, which shows up as a false reject. Fix that at capture: recover the holes (missing points in a 3D camera point cloud: fix the depth holes) or fill them by stacking several frames (merging multiple depth frames into one gap-free point cloud).

Two limits stay in view. Parts that touch have a gap of zero, and no tolerance can split them; that needs surface fitting or matching, covered at the end of this page. And clutter outside the work area costs compute time for nothing: an XY crop first, as in cropping an XY region of interest on a 3D depth camera point cloud, removes background clusters and shortens the run.

Step 3: measure volume and diameter per cluster

Volume needs a plane. volume_object_model_3d_relative_to_plane takes the connected components, a reference plane written as a pose — [0, 0, 0.670, 0, 0, 0, 0], that is, position (0, 0, 0.670) m with zero rotation — the mode 'signed' set to 'true', and returns one volume per cluster in m³. With the plane at the threshold's upper edge, that number is the space enclosed between the plane and the part's surface: the cover's volume below its rim.

Diameter needs no plane. max_diameter_object_model_3d returns the largest distance between any two points of a cluster, in m. For a dome cover that is the rim diameter; for a fastener, its length.

Both values come out in meters. Report them after multiplying by 1,000: diameter in mm, volume in dm³. The conversions are exact, 1 m³ = 1,000 dm³ and 1 dm³ = 1 liter, so 6.7e-4 m³ prints as 0.67 dm³ and 0.240 m as 240.0 mm.

Worked example: three objects against the window

Predict the window from CAD before touching the cloud. For a dome of rim radius r and crown height h, the spherical-cap formula is:

Cover: rim Ø 240 mm, so r = 120 mm, and crown 45 mm below the plane. V = (π/3) × 45² × (3 × 120 − 45) = (π/3) × 2,025 × 315 ≈ 668,000 mm³ = 0.67 dm³. Largest point-to-point distance = rim diameter = 240 mm. Both inside the window.

Shallow tray: Ø 310 mm, 15 mm deep, r = 155 mm. V = (π/3) × 15² × (3 × 155 − 15) = (π/3) × 225 × 450 ≈ 106,000 mm³ = 0.11 dm³, diameter 310 mm. Under the volume floor and over the diameter ceiling: rejected twice.

Fastener: Ø 8 mm, 20 mm long, roughly a cylinder, so V ≈ π × 4² × 20 ≈ 1,000 mm³ = 0.001 dm³, diameter about 20 mm. Rejected by both tests.

The cap formula only predicts the window. The plane-relative volume integrates whatever shape the points describe, so the final decision always uses measured numbers.

Step 4: select by volume and diameter, and mind the plane

select_object_model_3d takes the clusters, the features ['volume', 'diameter_object'], the logic 'and', minima of 0.35e-003 m³ and 0.185 m, maxima of 1.0e-003 m³ and 0.300 m. It returns the passing clusters as ready-to-use 3D models.

The trap is the plane. The volume inside this operator is measured against the default plane at z = 0, a detail the example programs that ship this pipeline flag in their own comments. Clusters living between 0.500 m and 0.670 m are far from z = 0, so a direct call measures against the wrong plane and the window rejects everything. Two ways out:

1. Translate first. Build a translation of (0, 0, −0.670) with hom_mat3d_translate, apply it with affine_trans_object_model_3d, and the old plane lands on z = 0, where the operator's volumes agree with step 3. Tag each translated cluster with an '&Index' attribute so a selected model traces back to its scene position.
2. Skip the operator. Step 3 already produced correct volumes and diameters; compare them with the window in a loop and keep the passing indices. Same decision, no transform, no trap.

Choosing your four numbers

Tolerance from gaps, not taste. Measure the smallest part-to-part gap and the largest in-part hole on a few captures, set the tolerance to a third of the smallest gap, and check that no hole exceeds it. Worked scene: gap 30 mm, holes under 10 mm, tolerance 10 mm.

Volume window from CAD plus repeatability. Start centered on the CAD volume and widen until ten repeated captures of one static part all pass. The example window, 0.35-1.0 dm³ around a 0.67 dm³ cover, does double duty: it absorbs capture variation and it rejects the 0.11 dm³ tray and the 0.001 dm³ fastener. When wrong-part rejection is the goal, a wide window is a feature.

Diameter window by the same logic. With 3 mm point spacing, a Ø 240 mm rim reads slightly low when rim points drop out and slightly high from noise, so a good part lands near 234-246 mm. The example's 185-300 mm window therefore does not inspect a good cover's diameter; it excludes other object classes. When the diameter is itself the inspected dimension, tighten to about twice the measured spread instead of roughly ±60 mm.

Sampling decides what the tolerance can see at all. Put at least three samples across the smallest gap you must honor: a 10 mm gap needs 10/3 ≈ 3.3 mm spacing. Over a 480 mm wide field that takes 480/3.3 ≈ 145 pixels across, so a 640-pixel-wide depth image covers it with about 4× margin, and 1,280 pixels tighten the spacing to roughly 0.4 mm on the same field. Size your own field against your own gap:

Cameras that capture clouds worth segmenting

Match the sensor type to the scene, because density and holes drive the tolerance. Projected structured light typically gives the densest single-shot cloud of the common sensor types, which keeps tight windows honest; the 3D sensors category groups all of them by type and series. The MV-DPS200P-02 is a projected structured-light 3D camera of this kind, at home on a static scene.

Logistics dimensioning runs the same measure-volume-then-filter logic at package scale, and speckle structured light is the usual engine there: multi-DOE speckle projectors built for volume measurement sit in the logistics and volume group of the 3D sensors category.

Where speed beats density, time of flight covers a larger volume faster and leaves wider holes, so the tolerance and the windows grow with it. The worked scene above is itself a time-of-flight capture. The MV-DT01WMU is a ToF sensing camera of that type.

On a moving conveyor, a laser profiler sweeps one profile after another under the parts and stacks them into a cloud, and the segmentation runs per stacked frame. Mounting height and angle set the profile quality — the 3D camera working distance and positioning guide covers the geometry, and 2D or 3D camera for conveyor pick and place: choosing the system covers when this pipeline beats a 2D one.

When connected components is the wrong tool

Touching or nested parts end the method: a zero gap defeats any tolerance, and cutting a merged cluster apart by volume is guesswork. Move to fitting planes, cylinders or complete surfaces (segmenting planes and cylinders in a point cloud), or to match-based location, which returns each part's pose as well as its identity (3D fine matching: check clouds and poses). If counts wobble between identical captures, the cloud itself is bent; diagnosing wavy point clouds from depth filters and camera intrinsics traces that to filter and intrinsic settings before you blame the clustering.

Checks before you trust the count

• Units, first of all. A 100 mm gauge block in the scene must read 0.100 m in z; if it reads 100, nothing downstream means anything.
• Plane pairing. The volume plane equals the threshold's upper edge, 0.670 m here. Route 2 in step 4 is immune to this trap by construction.
• Stability. Re-cluster the same scene at half and double the tolerance, 5 mm and 20 mm here; the cluster count must not move.
• Repeatability. Ten captures of one static part give ten volumes; their spread is the honest width for the volume window.
• No decimation before measuring. Downsampling shifts points, and the volume error follows (point cloud downsampling distortion: why decimation shifts points).
• Metric view for judging. Shaded renders hide scale, and exports can differ from the render (why exported point clouds differ from the viewer render).

Set the tolerance from the smallest gap, the windows from CAD, and prove both on one known part before the first production run. Shortlist sensors in the camera finder, then configure a camera around field and mounting; the 3D topic page collects the rest of this workflow, from calibration to transforms.