ApplicationsExplainer

3D vision guided robot applications: depth data drives the pick

3D vision guided robot applications: how depth data becomes a grip point and approach vector, and the capture-and-solve budget that sets your pick rate.

LumaSpec Vision12 min read Share on LinkedIn

On this page 10 sections
  1. 01What the robot must get from the depth data
  2. 02The cycle budget from scan to robot motion
  3. 03Coverage, depth range and point spacing
  4. 04The accuracy budget
  5. 05Choosing the capture technology
  6. 06The 2D refinement shot after the pick
  7. 07Where the usual answer is wrong
  8. 08Checks before you order
  9. 09Bring it together
  10. 10Questions

Random bin picking is the application that proves a 3D vision guided robot cell: parts arrive loose, stacked and touching, and there is not a taught point in sight. A 2D camera answers "where in X and Y"; a 3D sensor answers "where in X, Y and Z, which way up, and what the surface does where the gripper lands". That answer is what turns a pick offset into a grip point and an approach vector. This explainer follows the data path — point cloud, grip point, approach vector, collision model, robot command — and puts numbers on the two budgets that decide viability: capture-to-solve time and pick accuracy. One worked example runs throughout: 80 mm steel brackets, four layers deep, in a 600 × 400 mm bin, picked at 10 per minute.

What the robot must get from the depth data

A 2D guided pick hands the robot an offset — X, Y and a rotation applied to a taught position; that is the whole contract in how a 2D vision guided pick cycle works. Bin picking needs three more deliverables from the cloud:

  1. A grip point in six numbers. X, Y, Z plus orientation — the pose variables rx, ry and rz the robot consumes. Z is the one 2D cannot give: the grip point sits on a part that may be 25 mm or 100 mm below the rim.
  2. An approach vector. Fit a plane to the points around the grip point; its normal is the direction the gripper travels along for the final 30-50 mm. A vacuum cup holds to about ±10° of that normal; a two-finger grip instead wants the part's own axes.
  3. A collision model. The same cloud contains the bin walls, the rim height and the neighbouring brackets. The solver plans an approach that clears them, which is what stops the first pick of the morning from being a crash.

The solver itself is template matching: your CAD model is matched into the live cloud and every match carries its pose. Part CAD quality therefore caps pose quality before any sensor specification enters the picture.

The cycle budget from scan to robot motion

A structured-light snapshot head builds one cloud from a burst of projected patterns; as a planning figure, capture plus solve lands near 0.5 s for a full bin. Budget the whole vision chain, then hold it against the pick rate:

t_cycle = t_capture + t_solve + t_transfer + t_approach + t_grip + t_retract + t_place + t_return picks per minute = 60 / t_cycle

Worked example at a 10 picks/min target (t_cycle = 60 / 10 = 6.0 s):

Segment Budget
Capture + solve 0.5 s
Cloud to controller 0.1 s
Approach + descend 2.5 s
Grip (vacuum settle) 0.4 s
Retract + place 2.0 s
Return over bin 0.5 s
Total 6.0 s

Vision takes 0.6 s of the 6.0 s — 10%, the right share. Push to 15 picks/min (4.0 s) and robot motion must shed 1.4 s; vision still fits at 15%. Past that, stop buying speed and start overlapping: the sensor scans and solves pick N+1 while the robot still carries pick N, so vision adds no serial time and motion alone sets the ceiling. One measurement trap: cycle time read in the sensor's own viewer software is a single-frame best case; in your API, stream start-up and host-side copies can stretch it several times over — the gap and its fixes are covered in capture time slower in the API than in the viewer.

Coverage, depth range and point spacing

Three numbers size the sensor to the bin, all taken from the example:

  • Measurement volume. Bin interior 600 × 400 mm with a 50 mm margin per side so the wall points stay usable: at least 700 × 500 mm, at the standoff your gantry actually allows.
  • Z range. Four layers of 25 mm brackets give 100 mm of stack, plus 20 mm of bin-to-bin height variation: at least 120 mm.
  • Point spacing. The smallest grip-critical feature is a 6 mm flat before the radius, and you want it sampled three times: spacing ≤ 6 / 3 = 2 mm. Across the volume that is (700 / 2) × (500 / 2) = 350 × 250 = 87,500 points — modest, since snapshot heads deliver millions. Spacing at this volume is never the limiting specification, so do not pay for 0.2 mm spacing a vacuum cup cannot use.

The accuracy budget

The rule: total pick uncertainty stays under one third of the gripper's usable clearance. The example cup self-centres within ±2.0 mm, so the budget is 2.0 / 3 = ±0.67 mm. Combine the sources in quadrature:

U_total = sqrt(U_sensor² + U_calib² + U_robot²) U_total ≤ grip clearance / 3

With robot repeatability of ±0.05 mm and a hand-eye calibration residual of ±0.3 mm RMS across the volume, the sensor may contribute sqrt(0.67² − 0.30² − 0.05²) = sqrt(0.449 − 0.090 − 0.003) = sqrt(0.356) = 0.60 mm. Call the sensor requirement 0.5 mm to keep margin — that is the number to hold it to at your volume, not on a best-patch region of a spec sheet.

Eye-to-hand — the sensor fixed above the bin — is the default layout: one calibration transform, no cable dress on the arm, and the robot can move while the next scan runs. Eye-in-hand wins when bins change position every cycle or several stations share one robot; you pay with a calibration that must hold across the whole working envelope. In both layouts, recheck the residual after any collision, and when measured trueness drifts, run in-field correction on a known gauge part before blaming the matching.

Choosing the capture technology

Planning figures for a 700 × 500 mm bin:

Capture type Best for Vision time per pick Blind-spot behaviour Limit
Projected structured light, multi-camera head Static bin, random mix, tall stacks ≈0.5 s Cameras spaced around the projector see past tall stacks Specular surfaces saturate
Laser galvanometer stereo High-accuracy poses in a static bin 1-3 s dense, less with fewer patterns One viewpoint: shadowing behind tall parts Cycle cost at full density
ToF / depth camera Cartons, totes, coarse picks Tens of milliseconds Wide single-shot volume Z noise ≈1% of range: 5-10 mm at 1 m
Laser profiler + encoder Parts already moving on a belt Continuous, no snapshot Needs motion Wrong tool for a static bin

For the example brackets, the first row wins:

3D sensorsHikrobot MV-DPS200P-02 Projected structured-light 3D cameraMV-DPS200P-02
Type
Stereo / structured light
$8,829.00 · quotedView and quote
3D sensorsHikrobot MV-DLS300P-04 Laser galvanometer stereo cameraMV-DLS300P-04
Type
Stereo / structured light
$16,579.00 · quotedView and quote

The MV-DPS200P-02 is that snapshot type: a projected structured-light 3D camera. When a pose must land within about 0.3 mm — assembly insertion rather than bin-to-tray — move to laser galvanometer stereo. The MV-DLS300P-04 is a laser galvanometer stereo camera. Its cost is seconds of scan per pick, so spend it only where the cycle has those seconds.

The 2D refinement shot after the pick

Bulk picking leaves the part in the cup approximately right; assembly needs it exact. One 2D image taken during the transfer move corrects the residual grip misalignment and confirms orientation — extracting that angle is covered in finding object orientation from a camera — and the same frame doubles as presence and damage inspection at no cycle cost. This station is where smart cameras earn their place: sensing, lighting and decision in one housing.

Sizing it: the bracket is 80 mm wide and the correction tolerance is ±0.5 mm, so measure the offset to ±0.25 mm; two pixels on that is 0.125 mm per pixel; framing part plus gripper takes a 150 mm field, so 150 / 0.125 = 1,200 pixels across. A 5 MP sensor delivers more than twice that, so sampling is never the question — run the same sum for your part with the camera resolution calculator.

Camera resolution for the smallest feature
Units
Pixels needed
–
Sensor size
– MP
Smallest sensor that fits
–
Pixel on the part
– mm
Pixels across the feature, that sensor
–

Pixels needed = field / feature × pixels per feature on each axis (field incl. margin). The lens must resolve the pixels too: Nyquist = 1000 / (2 × pixel µm) lp/mm.

Find cameras with enough pixels Open the full calculator

Freeze residual part sway with a strobe pulse of 100-500 µs, brighter and shorter than any continuous setting. The MV-SC6050C-08M-XBN is a GigE smart camera pairing a 5 MP color sensor with an 8 mm M12 lens and a multi-color light source, so refinement and inspection are one device.

Smart cameras & vision sensorsHikrobot MV-SC6050C-08M-XBN SC6000 Series Smart Camera, 5 MP, Color, 8 mm focal length, M12-mount, Multi-color light sourceMV-SC6050C-08M-XBN
Interface
GigE
Resolution
5 MP
Sensor format
1/1.45"
Shutter
Global
$3,449.00 · quotedView and quote

Where the usual answer is wrong

  • Shiny bare metal. Specular saturation deletes the top surfaces — exactly where the cup must land. Multi-exposure capture and a head mounted off the surface normal recover most of it; the working methods are in bin picking reflective metal parts.
  • Flat, thin parts. Below roughly 10 mm of height the cloud becomes a plate and pose about Z turns noisy. Choosing a vision approach for flat parts works the alternatives, and 2D guidance on a feeder often wins — see flexible feeding architecture for random pick.
  • A moving stream. Snapshot heads need a still scene. Parts on a conveyor call for a profiler swept by the belt's motion, with the trade-offs in 2D or 3D camera for conveyor pick and place. For that case the MV-DP4180-03P, a 3D laser profile sensor, builds the height map line by line as the belt advances.
3D sensorsHikrobot MV-DP4180-03P 3D Laser Profile SensorMV-DP4180-03P
Type
Laser profiler
$12,389.00 · quotedView and quote
  • Parts near the sensor's floor. Every depth technology has a smallest resolvable object, and below it you are picking noise. Check your smallest feature against minimum object size for depth cameras before committing to tiny fasteners.
  • The robot refuses the pose. The cloud looks fine but the controller faults the result as a bad robot position — the CVIS-662 class of alarm. The cause is almost always a transform or user-frame mismatch between solver and robot, not the sensor. Settle which side commands the gripper with who drives what between vision and gripper, then walk the frame and handshake checks in bad robot position alarm troubleshooting.

Checks before you order

  • Measurement volume ≥ bin plus 50 mm per side, and Z range ≥ stack plus bin variation, both at the standoff you actually have.
  • Point spacing ≤ one third of the smallest grip-critical feature.
  • Sensor Z repeatability in the 0.5 mm class at your volume, from the accuracy budget above.
  • The solver takes your CAD, outputs pose plus surface normal, and speaks your robot's handshake — native driver, TCP string or fieldbus; 3D camera robot integration checks list what to confirm.
  • Something rigid holds the sensor: bracket sway converts directly into Z noise, the failure mode behind mounting stillness during capture.
  • A vision controller that can absorb the cloud and still serve the robot on time. The MV-VC5525-256G70-NN is one such unit: 32 GB of memory, PoE camera ports, 8 built-in light channels and PCIe expansion for whatever accelerator or grabber you add later.
Vision controllersHikrobot MV-VC5525-256G70-NN VC5000 Vision Controller, 12th-I5, 32GB+256G SSD, Supported PCIEMV-VC5525-256G70-NN
LAN ports
5+ LAN ports
USB 3 ports
3-4 USB 3 ports
PoE
PoE camera ports
Light outputs
8 built-in channels
$4,479.00 · quotedView and quoteBuild a system with it

Bring it together

  1. Size volume and Z range from your bin, with the 50 mm and stack margins carried through.
  2. Split the cycle: vision under 10%, robot motion the rest; overlap past about 12 picks/min.
  3. Split the accuracy: one third of grip clearance total, 0.5 mm of it on the sensor in the example.
  4. Choose capture technology by surface and part size, not by headline resolution.
  5. Plan the 2D refinement shot and the handshake frames before any hardware is ordered.

The decision for the worked example: a projected structured-light head over a static bin, 700 × 500 mm volume, 120 mm Z range, 2 mm point spacing, 0.5 mm Z repeatability, hand-eye residual 0.3 mm RMS, one 5 MP-class smart camera on the transfer path, and a controller with PoE camera ports running the solve. That cell makes its 10 picks per minute with 0.6 s of vision in a 6.0 s cycle.

Next step: browse 3D sensors and the camera finder, start the system configurator at the camera step, or read more cells like this one on the machine-vision applications topic page.

Build the whole station

The configurator matches a camera, lens, light, light controller and cables to each other, step by step, and lists only parts that fit what you already picked.

Start with the camera

Questions

How much of the pick cycle should 3D vision consume?

Keep the vision chain — capture, solve, transfer — under 10% of cycle time. At 10 picks per minute that is 0.6 s, which a structured-light snapshot head meets with its roughly 0.5 s solve. Above about 12 picks per minute, overlap instead of shortening: scan and solve pick N+1 while the robot still carries pick N, and vision stops adding serial time. Robot motion, gripper settle and place accuracy then set the ceiling, not the sensor.

Should the 3D sensor sit fixed above the bin or on the robot arm?

Fix the sensor above the bin (eye-to-hand) whenever the bin holds still: one calibration transform, no arm cable dress, and the robot can move while the next scan runs. Mount it on the arm (eye-in-hand) when bins change position every cycle or several stations share one robot; the cost is a hand-eye calibration that must hold across the whole working envelope plus a cable package that survives arm motion. Either way, recheck the residual after any gripper collision.

How accurate does the 3D sensor need to be for bin picking?

Hold total pick uncertainty to one third of the gripper's clearance. A vacuum cup that self-centres within ±2.0 mm gives a ±0.67 mm total budget; with ±0.3 mm calibration residual and ±0.05 mm robot repeatability, the sensor keeps about ±0.5 mm. Gripping machined features for insertion pushes the need toward 0.1-0.2 mm, which moves you to laser galvanometer stereo. Accuracy beyond your gripper's self-centering ability is money spent on nothing.

Does a 3D bin-picking system still need a 2D camera?

Yes, for two jobs at once. After the bulk pick, one 2D image measures residual grip misalignment and part orientation so placement or insertion is exact, and the same frame doubles as presence, damage and colour inspection during the transfer move, at no added cycle time. Sizing: an 80 mm part with a ±0.5 mm correction tolerance needs about 1,200 pixels across the field — comfortable for a 5 MP-class sensor with more than twice that sampling.

Can I plan several picks from a single 3D scan?

Not from a random bin. Every grip shifts, tilts or drops neighbouring parts and reveals new faces, so solved poses age the moment the gripper retracts; re-scan before each pick and the 0.5 s budget already covers it. The exception is a fixtured or nested presentation where parts physically cannot move — a full tray emptied in one cycle — where a single snapshot legitimately feeds many picks.

Why does the robot miss picks on shiny parts?

Usually specular saturation: the projected pattern bounces straight off the metal and the cloud grows holes exactly on the top surfaces the cup must land on. Turn on multi-exposure (HDR) capture, mount the head 15-25° off the surface normal so reflections leave the camera cone, and let the solver reject poses whose confidence image flags the grip region. If the holes persist, the part needs a different capture technology, not a bigger point count.