RoboticsHow-to

Overlap robot motion and 3D processing to cut bin picking cycle time

Overlap robot motion and 3D processing: move to place the moment capture returns, run pose estimation in a parallel thread, use previous-cycle poses.

LumaSpec Vision11 min read Share on LinkedIn

On this page 11 sections
  1. 01The rule: move the robot the moment the capture call returns
  2. 02Time the five phases of the cycle first
  3. 03Implement the overlap in two threads
  4. 04Worked example: 4.0 s pose estimation against 3.0 s of place motion
  5. 05When processing outlasts the motion: pick the previous cycle's pose
  6. 06Stationary camera: capture while the robot is out of view
  7. 07Which set-up to pick
  8. 08Verify that the robot never waits
  9. 09Mistakes that put the seconds back
  10. 10Checklist before you run the cell
  11. 11Questions

In a 3D bin-picking cell the robot often parks at the capture pose for seconds while software turns a point cloud into a pick pose. That stop is scheduling, not physics: the only moment the arm must hold still is the acquisition itself. This how-to shows you how to start the place move and the 3D processing in two parallel threads the moment the capture call returns, and how to pick from the pose computed one cycle earlier when processing still runs longer than the motion. You will time each phase of the cycle, size the overlap with one formula, work through a full cell, and prove from timestamps that the robot's wait is zero.

The rule: move the robot the moment the capture call returns

With the camera on the arm, the cycle is pick, travel to the capture pose, hold still, capture, place, retreat. The capture call returns when the acquisition completes, not when the point cloud is ready; the copy happens later, on demand. So the robot's only unavoidable stop is the acquisition. Command the place move the instant the call returns, and ask for the point cloud afterwards from a second thread. The arm then moves through the copy and the matching, and the only serialization left is the shot itself.

Order beats threading. A program that requests the point cloud before it commands the place move parks the arm for the whole copy-and-match interval even if both threads exist. The return of the capture call is also not the moment the frame is fully readable; the distinction is laid out in capture call returns vs image ready. The same pattern covers a plain 2D capture, and when you need both a photo and a cloud, settle the capture strategy once; 2D and 3D capture strategy works through that split.

On the arm, a compact unit keeps the capture pose a fixed stand-off from the bin; a laser galvanometer stereo camera such as the MV-DLS300P-04 is one example.

3D sensorsHikrobot MV-DLS300P-04 Laser galvanometer stereo cameraMV-DLS300P-04
Type
Stereo / structured light
$16,579.00 · quotedView and quote

Time the five phases of the cycle first

You need five times, in seconds, measured on the running cell from the robot's motion logs and timestamped code, not from anyone's estimate:

  • t_pick: approach plus grip
  • t_to_cap: travel from the pick to the capture pose
  • t_capture: acquisition with the arm still
  • t_process: point-cloud copy plus matching to a ranked pose
  • t_place + t_out: place move plus retreat out of the camera's view

Keep t_process separate from t_capture; they are different phases, and point cloud with normals timeout shows how often they get conflated. Motion times come from the controller's own logs, and over short moves acceleration dominates rated speed, which robot move time for small moves quantifies. The request you send the robot controller sits on the same critical path; vision picking request timing covers that half of the cycle. When the measured cycle disagrees with the programmed one, troubleshooting cycle time against robot motion is the methodical route.

The robot waits only for the shortfall between processing and the motion it can hide under:

t_wait = max(0, t_process - t_hidden) t_hidden = t_place + t_out (pose used this cycle) t_hidden = t_place + t_out + t_pick + t_to_cap (pose used next cycle) cycle = t_to_cap + t_capture + t_place + t_out + t_pick + t_wait

For a pose consumed next cycle, the following acquisition adds still more cover, because the worker keeps running while the arm holds still for the shot.

Implement the overlap in two threads

What you need: a 3D camera whose capture call returns a frame handle you can process later, a robot controller reachable from a worker thread, and one CPU core the matcher can own.

  1. Split the program into one motion thread, the only writer of robot commands, and one processing worker.
  2. Move to the capture pose and confirm the arm has settled before triggering; camera stillness during capture decides the cloud quality.
  3. Capture. The moment the call returns, send the place-and-retreat move from the motion thread.
  4. Hand the frame to the worker for cloud extraction and pose matching; the motion thread never touches frame data.
  5. Log three timestamps per cycle: capture returned, pose ready, robot ready for the pick move.
  6. If the logged wait is positive cycle after cycle, switch to the previous-cycle pose (next section) instead of accepting the stop.
// Illustrative C++; Frame, Camera and the helper names are ours, not any
// camera SDK's. The times in comments are the example cell's.
struct Pose { double x; double y; double z; double yawDeg; };

Pose capturePose{0.20, 0.0, 0.55, 0.0};   // m, m, m, degrees
Pose placePose  {0.45, -0.30, 0.10, 90.0};

Pose computePickPose(const Frame& frame);  // matcher: 4.0 s on this cloud
void moveRobotTo(const Pose& goal);        // the only robot-command writer

void runCycle(Camera& camera) {
    moveRobotTo(capturePose);              // arm settled before the shot
    Frame frame = camera.capture();        // returns when acquisition ends

    auto placing = std::async(std::launch::async, [&] {
        moveRobotTo(placePose);            // 2.0 s place ...
        moveRobotTo(capturePose);          // ... 1.0 s retreat: 3.0 s motion
    });
    auto nextPose = std::async(std::launch::async,
                               computePickPose, frame);  // 4.0 s, parallel

    Pose pickPose = nextPose.get();        // arm idles 1.0 s: 4.0 - 3.0
    placing.wait();
    moveRobotTo(pickPose);                 // 2.0 s to the next pick
}

The worker needs a core of its own, or matching steals time from the robot server. A vision controller such as the MV-VC5525-256G70-NN with 32 GB of memory runs both without contention.

Vision controllersHikrobot MV-VC5525-256G70-NN VC5000 Vision Controller, 12th-I5, 32GB+256G SSD, Supported PCIEMV-VC5525-256G70-NN
LAN ports
5+ LAN ports
USB 3 ports
3-4 USB 3 ports
PoE
PoE camera ports
Light outputs
8 built-in channels
$4,479.00 · quotedView and quoteBuild a system with it

Worked example: 4.0 s pose estimation against 3.0 s of place motion

Inputs, all from the example cell: t_pick = 2.0 s, t_to_cap = 2.0 s, t_capture = 0.5 s for one structured-light shot with the arm still, t_process = 4.0 s for full-cloud copy plus template matching, t_place = 2.0 s and t_out = 1.0 s. The motions are the durations the pipelined reference cycle uses; the acquisition and processing times are what a one-shot capture and full-cloud matching cost this controller. Put your own measured numbers in their place; the arithmetic does not change.

The rule in words: processing hides under motion, and the robot stops only for the acquisition plus any shortfall.

  • Serial code: 2.0 + 2.0 + 0.5 + 4.0 + 2.0 + 1.0 = 11.5 s per pick, which is 3600 / 11.5 = 313 parts an hour.
  • Same-cycle pose: t_wait = max(0, 4.0 - 3.0) = 1.0 s; cycle = 2.0 + 0.5 + 2.0 + 1.0 + 2.0 + 1.0 = 8.5 s.
  • Previous-cycle pose: t_wait = max(0, 4.0 - 7.0) = 0 s; cycle = 2.0 + 0.5 + 2.0 + 1.0 + 2.0 = 7.5 s, the motion floor of 7.0 s plus the 0.5 s acquisition. That is 3600 / 7.5 = 480 parts an hour.

Decision: 4.0 s of processing against 3.0 s of place motion leaves a 1.0 s stop on every pick under the same-cycle pose, so this cell goes previous-cycle: 7.5 s and 53 percent more parts than serial. Stay same-cycle only while t_process fits under three quarters of its window; matching time moves with point count, and a full bin is the slow case.

When processing outlasts the motion: pick the previous cycle's pose

The robot picks the object whose pose was computed from the previous capture, and the current capture's processing runs through the place, the retreat, the pick and the travel back to the capture pose: at least 7.0 s of cover on the example cell. Three conditions make the strategy valid, and they belong to the strategy itself. The solver must rank several candidate poses per capture, not one. The object picked this cycle must not have moved when the previous object was lifted. And consecutive picks should come from different regions of the bin, which keeps the second condition true and raises grasp success.

When a cached pose is wrong, the robot's own position check rejects it; the failure looks like a bad robot position alarm in a vision bin-picking cell. On any miss or slip, recapture immediately and rebuild from one known state rather than consuming the next cached pose.

Stationary camera: capture while the robot is out of view

Fixed over the bin, the camera sees a still scene, so the acquisition no longer needs the arm to stop; it needs the arm out of the view. The cycle becomes pick, exit the view, then capture-and-process in parallel with the place and return motion. The robot never stops while t_capture + t_process fits under t_place + t_return. On the example cell that is 0.5 + 4.0 = 4.5 s needed against 2.0 + 2.0 = 4.0 s of place-and-return, a 0.5 s stop. Previous-cycle posing widens the window by the 2.0 s pick move to 6.0 s and the stop disappears: cycle = 2.0 + 1.0 + 2.0 + 2.0 = 7.0 s, with no robot stop at all.

For a fixed mount over the bin, a projected structured-light 3D camera such as the MV-DPS200P-02 frames the whole bin in a single shot.

3D sensorsHikrobot MV-DPS200P-02 Projected structured-light 3D cameraMV-DPS200P-02
Type
Stereo / structured light
$8,829.00 · quotedView and quote

Which set-up to pick

Set-up Robot stops for Cycle on the example cell
Serial code, on-arm camera acquisition plus all processing 11.5 s
Overlap, same-cycle pose, on-arm acquisition plus 1.0 s shortfall 8.5 s
Overlap, previous-cycle pose, on-arm acquisition only, 0.5 s 7.5 s
Stationary, same-cycle pose 0.5 s shortfall 7.5 s
Stationary, previous-cycle pose nothing 7.0 s

One bin with a clear overhead view: stationary camera with previous-cycle poses, 7.0 s here. Deep bin with occluding walls, several bins on one arm, or a capture of the part in the gripper: on-arm with previous-cycle poses, 7.5 s here. Processing comfortably inside place-plus-retreat: the same-cycle pose, the simplest state machine to recover after a fault. If t_process exceeds even the 7.0 s window, shrink the cloud before buying compute; reduce stereo camera point cloud size shows the cuts.

Verify that the robot never waits

Prove the wait is zero from the logs, not from the cell looking busy. Per cycle compute t_wait = max(0, t_pose_ready - t_robot_ready_for_pick). Acceptance: across 100 cycles, t_wait = 0 everywhere except cycle 1, which primes the pipeline serially. A run that logs capture returned at 10.000 s, robot ready for the pick at 13.000 s and pose ready at 14.000 s carries a 1.000 s wait, the signature of a same-cycle pose with processing over the window. One more check before trusting the numbers: drain the camera buffer before each capture, or you time the processing of a stale frame; latest image only returns duplicate frames covers the drain.

Mistakes that put the seconds back

  • Requesting the point cloud before commanding the place move; the copy alone can exceed the acquisition.
  • Waiting on the processing result before the motion command is queued. The order of two lines is the whole trick.
  • Two threads writing robot commands. One writer, or the controller queues motions you cannot predict.
  • Triggering the capture while the arm is still settling; the retry costs more than the stop you saved.
  • Previous-cycle posing with a single-candidate solver or consecutive picks in the same bin region.
  • Letting the matcher share a core with the robot server. Pin the threads apart.

Checklist before you run the cell

  • Five times measured from logs: t_pick, t_to_cap, t_capture, t_process, t_place + t_out.
  • t_process checked against its window with the formula above, with margin for the full-bin case.
  • Motion thread is the only robot-command writer; the place move is queued before any cloud request.
  • Solver returns ranked candidates; consecutive picks are separated by bin region.
  • Timestamps logged, with t_wait = 0 from cycle 2 onward over 100 cycles.
  • Fault path drops both threads' state and re-handshakes before the next pick.

Size the camera, controller and triggering around this pipeline in the system configurator, compare 3D sensors and vision controllers, browse the camera finder, or send the five measured times through the contact page and we will check the arithmetic with you.

Questions

How much cycle time does overlapping robot motion with 3D processing actually save?

The saving is the share of processing time that fits under motion, capped by the motion window. On the example cell: serial code 11.5 s per pick, 313 parts an hour; same-cycle overlap hides 3.0 of the 4.0 s and runs 8.5 s; previous-cycle hides all 4.0 s and runs 7.5 s, 480 parts an hour. Faster processing buys nothing once it fits whole under the window.

When does the robot still have to stop in a 3D bin-picking cycle?

Three cases. With the camera on the arm, during the acquisition itself, because the scene moves with the gripper, 0.5 s on the example cell. Then whenever the pose is not ready when the pick move should start, by exactly the shortfall. And after a failed grasp, which costs a recapture and a rerun of matching, the most expensive stop in the cell. A stationary camera removes the first case, because the bin stays still while the arm places.

What breaks previous-cycle pick posing?

Three assumptions: the solver ranks several candidate poses per capture, the part picked this cycle did not move when the previous part was lifted, and consecutive picks come from different regions of the bin. A disturbed bin, nested parts or a single-candidate solver break it. Detect it from the grasp result: on a miss or a slip, recapture before the next pick instead of consuming the next cached pose.

Should the 3D camera sit on the arm or stay stationary?

Stationary over the bin when one camera sees the whole bin and the arm leaves that view on the way to place: acquisition then overlaps motion completely, and the example cell runs 7.0 s. On the arm when bin walls block a fixed view, several bins share one camera, or the part must be captured in the gripper; budget the extra travel to the capture pose, 7.5 s on the example cell.

Is a faster controller the better fix for a long pose-estimation time?

Only after the window arithmetic fails. Crop the cloud to the bin interior and downsample it first, because point count drives matching time, then overlap as described here. Buy compute when t_process still exceeds the previous-cycle window, 7.0 s on the example cell, after those cuts; below that the robot hides the processing and a faster matcher changes nothing.