Full resolution color with depth point cloud

A colored point cloud does not have to use the color camera's pixel grid. You can retain a large, unregistered color image for marker tracking while generating geometry from a smaller depth image. The key is separating acquisition from registration. This approach helps when a convenience pipeline resizes both streams together, leaving your tracker with fewer pixels or producing more cloud entries than your application needs.

Separate the native image from the registered output

Treat the native color image, depth image, and colored point cloud as distinct outputs. Here, native color means the selected color stream before cross-camera registration, not necessarily raw sensor data.

Branch the acquired color frame before applying registration. Send one branch to the marker tracker with its matching calibration. Use the other branch to supply texture for points generated from depth.

Registration should create a derived output rather than replace the only available color image. If your driver exposes only a registered image, check its documented native-stream outputs first. Otherwise, use the manufacturer's supported SDK interfaces to branch the data before the alignment stage.

Understand which grid determines cloud size

A depth-grid cloud starts with one candidate point per depth pixel. Invalid depth, filtering, and region selection can reduce that count. Sampling a larger color image does not require adding more geometry points.

By contrast, a pipeline that registers depth onto the color grid may create a much larger depth raster. A subsequent cloud generator can then allocate an entry for each output pixel, depending on its implementation. More entries do not necessarily mean more independent depth measurements.

The distinction between image size and measurement density also matters when stereo depth output dimensions appear unchanged. Output resolution, registration target, and depth filtering are separate decisions.

For additional texture channels, the same separation principle applies. The guide to combining depth, color, and infrared in a point cloud explains the broader data association problem.

Work through the point-count and memory arithmetic

Consider an illustrative application with a 3840 × 2160 color stream and a 640 × 576 depth stream. These are application examples, not specifications for the products below.

The candidate cloud sizes are:

Keeping the cloud on the depth grid therefore gives 22.5 times fewer candidate entries than using the color grid. The full-resolution color image remains available separately.

For a hypothetical cloud layout using 16 bytes per entry, including coordinates and packed color:

That is approximately 5.90 MB versus 132.71 MB, using decimal megabytes. Actual storage depends on field types, padding, organization, and whether invalid points remain in the buffer. These figures exclude images, processing buffers, and transport overhead.

Notice that the image dimensions also have different aspect ratios. Their horizontal scale ratio is 6, while their vertical scale ratio is 3.75. Resizing coordinates independently still does not account for camera separation, lens distortion, or different fields of view.

Generate geometry from depth and sample color by projection

For each valid depth pixel, first reconstruct its position in the depth optical frame. The following pinhole equations assume an undistorted depth pixel and a depth value representing axial distance, not radial range.

Here, the intrinsics describe each selected image mode. The rotation and translation transform depth-frame coordinates into the color camera frame. Translation and point coordinates must use consistent units.

Apply the color camera's documented distortion model when projecting into a distorted color image. If the color image is already rectified, use its rectified calibration instead. Manufacturer projection functions are useful because they handle the supported camera model and stream conventions.

Reject projections behind the color camera or outside its image. Sample the remaining color coordinates using a declared interpolation method. Nearest-neighbor sampling preserves discrete pixel values; bilinear sampling can smooth texture but mixes neighboring colors.

The projection can be used directly to attach color to cloud points. An intermediate color-on-depth image is optional.

If you export points in another frame, keep that transformation separate from texture projection. The guide to optical-frame and camera-frame transformations covers the coordinate conventions that commonly cause confusion.

Keep marker coordinates tied to their original image

Run the marker tracker on the full-resolution color branch. Supply calibration for that exact image size, crop, distortion state, and orientation.

A marker corner reported at a color-image coordinate cannot be used directly as an index into the smaller depth image. The two coordinates belong to different projection systems. Even matching image dimensions would not make their pixels interchangeable.

Choose the mapping method according to what the tracker needs:

• Color-camera pose only: estimate marker pose using color calibration and the known marker geometry. Depth registration is not required for this calculation.
• A marker pose expressed in the depth frame: transform the estimated pose using the calibrated relationship between the cameras, with the transform direction checked explicitly.
• Measured depth near a marker: project valid depth samples into the color image, then associate samples with the marker region. Account for occlusion, boundary mixing, and missing measurements.

Do not assume the marker center has reliable depth. A small printed target may straddle depth discontinuities, and its surface may not produce valid measurements everywhere. Inspect the associated samples rather than accepting a single pixel without checks.

The guide to mapping image detections to depth points develops this association process for image regions.

Branch synchronized frames without corrupting their metadata

A useful processing sequence is:

1. Acquire the selected native color and depth modes.
2. Record timestamps, frame identifiers, and calibration identifiers.
3. Retain the native color frame for the tracker.
4. Generate points from valid native depth samples.
5. Project those points into the associated color frame for texture sampling.
6. Publish each output with its own dimensions, coordinate frame, and timing information.

Branching does not always require copying the full color image. Immutable shared buffers can work when the SDK and processing framework support them. However, a retained reference is unsafe if acquisition later reuses its memory. Follow the documented buffer ownership rules.

Time alignment matters even when calibration is correct. As an estimate, an object moving laterally at 0.5 m/s travels 10 mm during a 20 ms mismatch:

Trigger synchronization and exposure timing affect the actual association. Matching arrival times alone is not proof of simultaneous capture. Avoid pairing a delayed depth frame with whichever color frame happens to be newest.

Check hardware capability before choosing an architecture

Start with the required outputs, not a registration setting. Confirm that the device and supported software can deliver the selected native color and depth streams simultaneously. Also verify access to calibration, timestamps, and projection functions.

The 3D sensors category includes several sensing methods, so a product's category alone does not establish native color support. The camera finder helps narrow the hardware options; simultaneous modes and SDK behavior still require confirmation.

The MV-DPS200P-02 is cataloged as a projected structured-light camera. Its listing does not establish the native-color branching capability discussed here.

If the depth device cannot supply the required color image, a separate color camera is another architecture to evaluate. The area scan cameras category provides that selection route.

The MV-CU020-90GC is cataloged as a color area scan camera with a global shutter and GigE interface. It is a separate-camera candidate to evaluate, not a verified pairing with the depth device above.

Separate cameras require mechanical stability, suitable timing, overlapping coverage, and extrinsic calibration. The guide to placing multiple cameras in a shared coordinate frame addresses that additional work.

Verify the outputs before trusting detections

Test a static scene first, then introduce motion. Save representative images and clouds together with their metadata so you can compare the processing branches.

Use this acceptance checklist:

• Native image dimensions: confirm that the tracker receives the selected color mode, not the registered texture image.
• Unchanged tracker calibration: verify that registration has not replaced the native intrinsics or distortion metadata.
• Depth-grid organization: check cloud dimensions or candidate count before invalid points are removed.
• Projection accuracy: inspect interior surfaces and depth edges separately, across the overlapping view.
• Transform direction: verify a known displacement along each axis rather than relying on a visually plausible overlay.
• Timing: compare frame timestamps and exposure information during motion.
• Buffer lifetime: confirm that saved tracker images do not change after later frames arrive.
• Restart behavior: repeat the test after reconnecting and restoring the selected stream modes.

For calibration capture and stream selection, use the stereo target and stream setup guide. Where geometry itself has gaps, diagnose missing point-cloud measurements before treating the problem as texture registration.

Set acceptance tolerances from the inspection task. A visually attractive colored cloud can still be unsuitable for associating a small marker with a nearby surface.

When the depth-grid approach is not enough

Keeping geometry on the depth grid saves processing and storage, but it cannot recover detail the depth sensor never measured. A high-resolution marker detection does not create equally detailed surface geometry.

You may need another approach when the required feature is smaller than the available depth sampling, falls outside overlapping coverage, or is frequently hidden from one camera. Moving parts may also require tighter synchronization than the existing hardware provides.

Color-grid registration can still be useful when downstream software requires depth indexed by color pixels. Keep that as another derived output, retain validity information, and do not confuse a denser raster with additional measured detail.

Before ordering, document the native color size, depth size, required measurement tolerance, motion conditions, and acceptable latency. Use the camera configurator to organize the component choices, or request application support to confirm the required stream modes and calibration access before selecting hardware.