2D bounding box to 3D: estimate object extent from depth

A detection rectangle identifies where an object appears in an image; it does not reveal the object's hidden surfaces. With registered depth and calibrated camera geometry, you can turn valid pixels inside that rectangle into spatial points and calculate their visible extent. This workflow is useful for localization, coarse size screening and downstream geometric fitting. The key decision is what your output means: a box around observed points, or an estimate of the complete object. Those outputs require different evidence and should never share an ambiguous label.

Decide what the spatial box must represent

A 2D detection box defines a viewing region, not a solid object. Without depth, it corresponds to a set of rays extending from the camera. Adding depth locates measured surfaces along some of those rays.

Even perfect depth does not normally reveal the back of an opaque object. Calculating minimum and maximum coordinates therefore produces an envelope of the selected visible points. It does not automatically recover the object's physical length, width and height.

Start with the mapping from image pixels to spatial points if you need a single object location first. Extent estimation adds a harder requirement: the selected samples must represent the surfaces that determine each reported dimension.

Verify registration, timing and calibration

Before calculating coordinates, establish which image grid owns the depth values. Equal image dimensions do not prove that color and depth pixels describe the same viewing rays.

Check these inputs:

• Detection coordinates: Map resized, cropped or padded inference coordinates back to the acquisition image.
• Registered depth: Use depth explicitly registered to that image, or project native depth points into it using the calibrated relationship between cameras.
• Camera intrinsics: Use the parameters associated with the actual registered image geometry.
• Depth meaning: Determine whether values represent optical-axis depth or radial distance.
• Depth scale: Convert stored values into one consistent length unit.
• Frame association: Match detections to the correct depth acquisition, especially when objects move.

The registration checks for color and depth address the first geometric dependency. For moving scenes, matching asynchronous frames using timestamps helps prevent a correct rectangle from selecting a different object position.

For a simple resize, focal lengths and principal-point coordinates scale with the image dimensions. Cropping also shifts the principal point. Rectification may introduce a new camera matrix, so do not reuse the original parameters automatically. Follow the intrinsics guidance for rectified images when deciding which matrix applies.

Back-project each accepted depth sample

For a rectified pinhole image, use the following relationship when the depth value is optical-axis distance. Pixel coordinates and focal lengths must use the same image grid.

Here, u and v are pixel coordinates; fx and fy are focal lengths in pixels. The principal point is (cx, cy). If Z is expressed in metres, the resulting coordinates are also in metres.

For an illustrative calibration, let fx = fy = 600 pixels and the principal point be (320, 240). A sample at pixel (380, 270) with axial depth 1.20 m becomes:

This is one measured surface point, not the object's center or a cuboid corner.

If the sensor reports radial range, substituting it directly for Z introduces an off-axis error. Instead, normalize the viewing ray and multiply it by the range. The distinction is explained in axial depth versus true camera distance.

Do not assign a single center depth to all rectangle corners and call the result a complete object box. That construction describes a rectangle on an assumed depth plane. It supplies no evidence about the object's thickness.

Select object pixels before calculating bounds

A detection rectangle commonly contains background, supporting surfaces and neighboring objects. Raw coordinate minima and maxima are particularly sensitive to these unwanted samples.

Use a selection sequence that preserves the features relevant to your measurement:

1. Clip the rectangle to the image and verify that it remains nonempty.
2. Reject invalid depth using the documented validity flags and sentinel values.
3. Apply an object segmentation mask when one is available.
4. Exclude a known support plane or background region where the geometry justifies it.
5. Separate disconnected spatial clusters and select the cluster associated with the detection.
6. Review isolated samples before allowing them to determine an extreme coordinate.

The object-mask depth selection workflow explains why a rectangular crop and an object mask answer different questions. A mask improves pixel selection, but it cannot recover depth that was never measured.

A central patch can provide a useful cluster seed for an isolated object. It is unreliable for hollow parts, objects with central openings, overlapping objects or detections whose centers fall on the background. Likewise, a narrow depth band can remove background while also discarding genuine sloped surfaces.

Record valid coverage against a clearly defined denominator. Suppose an illustrative object mask contains 12,000 pixels, but only 6,000 provide accepted depth. Coverage is 6,000 / 12,000 = 50%. That percentage alone does not establish accuracy: missing points along an outside edge are more damaging to extent measurement than an equally large interior gap.

When coverage is poor, diagnose the causes of missing point-cloud measurements rather than filling holes merely to make the box look complete.

Calculate the visible envelope and its corners

For accepted object points, a camera-axis-aligned bounding box uses independent coordinate extrema:

Its eight corners are every combination of the two limits on each axis: {xmin, xmax} × {ymin, ymax} × {zmin, zmax}. They are mathematical corners of the envelope; none needs to coincide with a measured surface point.

Consider an illustrative filtered cloud with these limits:

The envelope center is (0, 0, 1.21 m), because (1.18 + 1.24) / 2 = 1.21. Its observed depth extent is 1.24 - 1.18 = 0.06 m. That value could reflect a sloping front surface, visible side surfaces or residual noise. It is not automatically the object's front-to-back thickness.

Percentile limits can reduce isolated outlier influence. However, a box using the 2nd and 98th percentiles deliberately excludes some accepted points. Label it a trimmed extent, not an enclosing box. Validate that trimming does not remove real tabs, pins or other dimension-defining features.

Choose the reporting frame deliberately

Camera-axis bounds change when an object rotates, even if its physical dimensions remain constant. A long part viewed diagonally can have large extents along several camera axes.

If you need bounds in a fixed station frame, transform the accepted points first and calculate extrema afterward:

Rotation is dimensionless; translation must use the same length unit as the points. Check axis directions against the optical-frame transformation guidance. A sign error can make plausible dimensions appear in the wrong location.

Rotating a camera-aligned box's corners gives the same box expressed in another frame. It does not generally give the tight axis-aligned envelope of the original points in that frame. For that result, transform the points and recalculate their bounds.

An oriented box introduces another decision: where its axes come from. Principal-component analysis estimates axes from point distribution, but partial visibility and symmetric shapes can make those axes unstable. Known object geometry or a validated pose fit may provide a better reference.

Once orientation is established, rotate points into the object frame, calculate bounds there and transform the resulting corners back. Keep the orientation method and frame identifier with the output.

Match the sensing architecture to the measurement

Choose acquisition hardware around surface visibility, motion and measurement tolerance. The 3D sensors category and camera finder provide starting points; the 3D sensing-method comparison explains the broader acquisition choices.

The MV-DPS200P-02 is a projected structured-light 3D camera. Treat it as a candidate requiring application-specific validation, rather than assuming that its classification guarantees the required output geometry.

The MV-DT01WMU is a ToF sensing camera. Before selection, verify the documented depth convention, calibration access, validity information and acquisition behavior needed by your processing pipeline.

A separate color camera adds another calibration and timing relationship. Browse area scan cameras only when an additional image stream has a defined purpose, such as improving detection detail.

The LS-CMR-0166 is a USB3 area scan camera with a rolling shutter. It provides an image stream, not an independent depth measurement.

With separate acquisition devices, verify overlap, occlusion differences and exposure timing. Do not assume that any listed camera pair provides factory registration or synchronized operation. Start the camera configurator only after those interface and geometry requirements are recorded.

Test uncertainty and recognize unsuitable cases

Depth error propagates into lateral coordinates through the back-projection equations. Holding calibration and pixel location fixed, the contribution from axial-depth error is approximately:

At an offset of 120 pixels with fx = 600 pixels, an illustrative 5 mm depth error contributes approximately (120 / 600) × 5 = 1 mm of lateral error. This is only one contribution. Pixel selection, calibration, registration and pose errors remain.

Repeated captures reveal variability, but averaging cannot restore an unseen back face. A stationary flat front surface can produce an excellent point cloud while leaving physical thickness completely undetermined.

Use another measurement strategy when:

• Hidden surfaces define the required dimension.
• Touching objects cannot be separated reliably.
• Reflective or transparent surfaces prevent dependable depth acquisition.
• Small protrusions are missing from the accepted cloud.
• Motion misaligns the detection and depth measurements.
• The required tolerance is smaller than the validated measurement uncertainty.

Known dimensions can support a pose-based cuboid estimate. Multiple viewpoints can improve surface coverage. Neither approach removes the need to validate calibration and document which parts of the output are measured versus inferred.

Acceptance checklist and next step

Before using the output for production decisions, complete these checks:

• Reference geometry: Compare against independently measured parts throughout the intended viewing region.
• Repeatability: Repeat captures without moving the part, then repeat after repositioning it.
• Orientation: Test rotations that change which surfaces are visible.
• Selection: Review accepted points on overlays, especially at dimension-defining edges.
• Validity: Reject empty or insufficient clouds explicitly; never return a zero-sized object as a successful measurement.
• Metadata: Store units, frame, timestamp, selection method, point count and measured-versus-inferred status.
• Failure behavior: Confirm that missing depth produces an invalid measurement rather than a stale box.

Set acceptance thresholds from your measurement tolerance and validation data, not from an arbitrary universal coverage percentage. Keep representative successful and failed captures so later calibration or filtering changes can be checked against the same cases.

For the next step, send the required dimensions, tolerance, working distance, motion conditions and representative image/depth pairs through application support. Identify whether you need a visible-surface envelope or a complete object cuboid before choosing hardware or committing to a geometric model.