2D and 3D capture strategy: combine or split RGB and point cloud

A structured-light 3D camera hands you two data products from one position: a color image and a point cloud. You can take them in a single combined capture, or shoot the 2D image on its own, with its own exposure and its own light. Which 2D and 3D capture strategy wins follows from three questions: do you need the color image before the depth data, must color and depth share pixels one to one, and how many milliseconds the cycle has left. Here are the decision rules, the settings that avoid the camera's mode-switch penalty, and a worked cycle-time budget with the arithmetic shown.

What the camera does in each mode

Two modes, one sensor. In 3D mode the internal projector throws a coded pattern across the scene, the sensor records a short sequence of pattern images, and the SDK triangulates them into X, Y, Z points with an intensity value. In 2D mode the projector stays dark and the sensor behaves like a plain area-scan color camera: you set exposure and gain and receive an RGB frame.

A combined capture runs both phases in one call and returns a single frame that holds the image and the cloud. That is also why the camera may reconfigure between the phases: readout, aperture and pixel sampling can all differ between a 2D and a 3D acquisition. Every difference you remove from the settings is reconfiguration time you remove from the cycle.

Three ways to get color, and what each costs

1. Separate captures: 2D first, then 3D. The 2D shot uses its own settings, its own exposure and, if needed, its own light. You get color at the earliest possible moment. If the two captures run back to back, a settings mismatch adds a reconfiguration step between them.
2. One combined capture, color taken from the point cloud. The cloud carries its own color buffer at the cloud's resolution. Copy from there and alignment is guaranteed by construction, even when the 2D and 3D sampling differ.
3. One combined capture, color taken from the frame's 2D image. The image is identical to what a separate 2D shot returns; only the call differs. Use it when you want the plain full frame rather than cloud-resolution color.

The cost of mismatched settings is real and measurable. One structured-light camera family documents a 35 ms capture-time penalty for changing the sampling setting between the 2D and 3D acquisitions, and asks you to keep the aperture equal between them as well. Treat tens of milliseconds as the price of a mismatched pair, then design it away: equal aperture, equal sampling, or a switch that hides behind processing you had to do anyway.

A projected structured-light camera of this class, such as the MV-DPS200P-02, returns image and cloud from a single call whenever you choose the combined path.

External light: better color, noisier depth

Color wants strong, diffuse light. When the projector is the only source, shiny and curved parts bloom: the pattern reflects into the lens and the color washes out. Mounting the camera at an angle reduces blooming; a diffuse external light removes most of it. A short exposure under your own bright light also swamps ambient variation, so daylight shifts, doors and ceiling lamps stop mattering in the color image.

Depth wants darkness. Ambient light is noise for a structured-light measurement, so the external light must be off, or dimmed far below the projector's contribution, during the 3D exposure.

That sequencing is what an LED lighting controller is for: the camera's exposure output fires the light for the 2D frame only, and the why your vision light needs a controller guide gives the full argument. For the flat field that color segmentation likes, put a diffuser in the beam: the 170 x 30 mm bar light diffuser plate spreads a narrow bar into an even strip, and the lighting accessories range covers the other sizes and cabling. The ambient light variation guide shows how to prove the cell holds up across a full day.

Pixel mapping when resolutions differ

The combination that serves most cells: full-resolution 2D for segmentation-grade color, subsampled 3D for a compact, fast cloud. On a 5 MP sensor at 2448 × 2048, a 2x2 subsampled 3D capture returns 1224 × 1024 = 1,253,376 points, a quarter of the full grid. Reuniting the two grids takes one of three routes:

• Cloud color. Copy the color from the point cloud object. It is already cloud-resolution and aligned; the cost is that this color is subsampled too.
• Upsample the cloud. Resample the 3D data back onto the full grid; you keep the fast subsampled acquisition and restore one-to-one correspondence. The sub-pixel depth sampling guide covers the interpolation choices.
• Index mapping. Leave the cloud subsampled and map 2D indexes into it. The sampling pattern fixes a stride (2 for 2x2) and a small constant offset for the sensor site the 3D path reads; the SDK returns both, so you never have to guess the Bayer alignment.

With H = 2048, stride = 2 and offset = 0, that is (2048 - 0) / 2 = 1024 rows, matching the cloud exactly. When you project with intrinsics instead, scale them to the target grid first: fx' = fx × W'/W and cx' = cx × W'/W, likewise fy and cy with the height ratio. The projector pixel mapping guide works the same machinery one level down, from projector column to sensor pixel, and the RGB and depth registration guide builds the colored cloud end to end.

How many pixels the color image needs

Segmentation wants 3 to 5 pixels across the smallest color feature. Work it backwards: a 1 mm print mark at 4 px per mark needs 0.25 mm per pixel, and over a 300 mm field that is 300 / 0.25 = 1,200 pixels across. A 2448-pixel sensor covers that twice over; 2x2 subsampled color gives 1,224 pixels and still clears it. Right at that boundary, drop the subsampling for 2D and keep it for 3D.

Worked example: the cycle-time budget

Scene: fixed-mount pick cell, 250 ms of camera time available, color needed first for segmentation, pose from the cloud. Assumptions: 5 MP sensor (2448 × 2048), 4 bytes per RGBA pixel, 16 bytes per point (XYZ as float32 plus RGBA), 350 MB/s effective read over USB 3, 35 ms sampling-switch penalty. Measure the transfer rate on your own host; the rest is arithmetic.

Strategy A — combined capture, full-resolution 3D, equal settings. Points: 2448 × 2048 = 5,013,504. Bytes: 5,013,504 × 16 = 80,216,064 B ≈ 80 MB. Transfer: 80 / 350 ≈ 0.229 s = 229 ms. Color arrives with the cloud, lit by the projector.

Strategy B — separate 2D at full resolution, then 3D at 2x2. 2D: 5,013,504 px × 4 B = 20,054,016 B ≈ 20 MB, so 20 / 350 ≈ 57 ms. Switch: 35 ms worst case. 3D: 1224 × 1024 = 1,253,376 points, × 16 B = 20,054,016 B ≈ 20 MB, again 57 ms. Total: 57 + 35 + 57 = 149 ms, or about 114 ms once the switch hides behind segmentation. Diffuse-lit color is in hand after the first 57 ms.

Strategy C — combined capture, 3D at 2x2, color from the cloud. One call with differing sampling: 35 + 57 = 92 ms, aligned color at cloud resolution. Pull the full-resolution frame as well and the total lands at 149 ms — the same bytes as strategy B, one call instead of two.

Add each phase's exposure to these totals: a strobed 2D shot runs at 1 to 5 ms, and the 3D pattern sequence takes longer, at a duration the camera's specification fixes.

Decision. C is the fastest fused result, 92 ms against 229 ms for A. B returns usable color 172 ms earlier than A (57 ms against 229 ms) and lights it diffusely, and its quarter-size cloud cuts every downstream step roughly fourfold. Take C when color-on-points is the deliverable; take B when segmentation runs on color or the parts are shiny; take A when the full-resolution cloud itself is required and the scene is matte.

Which strategy when

One coupling decides several rows on its own: the moment the color shot needs external diffuse light, you are on the separate-capture path, because that light must be dark during the 3D exposure. Matte parts under bright hall lighting can take the combined path and skip the strobe entirely. If the deeper question is whether the cell needs depth at all, the 2D or 3D camera guide for conveyor pick and place makes that call first.

When the camera class changes the answer

Stereo and time-of-flight cameras split the job across two sensors behind one window: one for depth, one RGB. Registration is a factory calibration rather than a sampling choice, so there is no mode switch to optimize, but parallax remains and color and depth can disagree slightly at object edges and occlusions. How the extra channels ride along on a cloud is covered in the RGB and IR intensity guide, and the structured light vs stereo comparison shows when fine detail pushes you toward projected patterns. On stereo or time-of-flight hardware the strategy question shrinks to registration quality:

A laser profiler returns intensity line by line alongside each profile; there is no full-frame color shot to schedule. When the application also wants a color overview, that frame comes from a separate area-scan camera registered to the profile frame by calibration. And where the deliverable is the cloud itself — pallet and parcel volume, for instance — color is a nice-to-have, which is exactly the flow the logistics and volume class of structured-light cameras in the 3D sensors range is built around.

Checks before you freeze the design

1. Benchmark on the target host. Run the capture benchmark sample your SDK ships with, load your settings files, and record the 2D, 3D and back-to-back times. The 350 MB/s transfer figure above is a planning value, not a measurement.
2. List the legal settings pairs. Confirm which sampling and aperture combinations the SDK allows for the 2D and 3D acquisitions, and which of your pairs triggers the switch penalty.
3. Prove the mapping once. Shoot a marker at a known pixel, map it onto the cloud, and require it to land within 1 pixel at the center, an edge and a corner of the field.
4. Check point density at the smallest 3D feature. 2x2 halves the linear sampling: a hole spanning 4 full-resolution pixels spans 2 subsampled ones, which can be too few for a clean pose edge.
5. Size the buffers to the strategy. The example moves 80 MB per full-resolution cloud against 20 MB subsampled; set the ring buffer and processing queue from your own bytes per frame, not a guess.

Pick the capture order before writing the integration: it fixes the lighting hardware, the thread layout and the buffer sizes in one stroke. Browse the 3D sensors range, filter the camera finder by resolution and interface, and use the configurator to tie camera, light and controller into one timing plan. The 3D topic page collects the capture-side guides. If the cycle budget looks tight on paper, bring the numbers to the contact page and we will check the strategy with you.