PBVS control law stability with task sequencing

A position-based visual servoing (PBVS) control law is straightforward to prove stable when it drives one task to one target pose. Real cells rarely run one task. A picking application sequences tasks: center the part, align the gripper, then approach. The question that follows is whether switching between task sets preserves the stability proof, or whether the switches reintroduce oscillation and drift. The short answer: a sequenced PBVS controller is stable under conditions you can check and demonstrate, and this article sets out those conditions, the math behind them, and a practical test procedure on a real cell.

This assumes you have already chosen your features and closed the loop; if not, start with visual servoing loop design for eye-in-hand picking and visual servoing feature selection for perpendicular approach.

The single-task PBVS control law and its Lyapunov argument

PBVS estimates the pose error between the current and desired camera pose from image features, then commands a Cartesian velocity proportional to that error. In the classical formulation with a Pose Interaction Matrix, the control law takes the form:

where e is the pose error, L_e the interaction matrix, and lambda a positive gain. The standard proof uses the candidate Lyapunov function:

Differentiating along the closed-loop trajectory gives a negative definite derivative as long as L_e stays full rank and the gain is positive. The error decreases monotonically. Two assumptions carry the whole proof: the feature errors (and therefore the pose estimate) remain bounded, and the interaction matrix is well conditioned along the trajectory. Everything about task sequencing builds on these two assumptions.

Note the proof relies on pose reconstruction, which is only as good as your calibration; the article on camera extrinsic calibration accuracy beyond reprojection error covers where calibration error enters the loop.

What task sequencing actually changes

A sequenced controller activates different task sets in stages. A typical sequence:

1. Task set A: center the target in the image (2 translational degrees of freedom).
2. Task set B: align orientation to the gripper approach axis.
3. Task set C: control the approach distance along the approach vector.

Between stages the control law switches: a new error vector e, a new interaction matrix L_e, and often a new gain. The individual control laws can each carry a valid Lyapunov proof. The subtlety is that stability of each mode does not automatically give stability of the switched system. Switched linear systems can diverge even when every mode is stable, if the switching itself injects energy faster than each mode removes it.

In visual servoing this shows up as a recognizable symptom: the robot converges toward the stage-A goal, switches to stage B, and the stage-B error initial condition kicks the camera so that the stage-A error regrows. The composite error never settles.

The stability conditions for a sequenced PBVS law

Three conditions restore the proof.

Condition 1: each task set is stable on its own. For every task set in the sequence, V_i = (1/2) e_i^T e_i must decrease under the active law, with L_e full rank across the reachable set of that stage. Check the conditioning of L_e over the stage's expected error region, not just at the goal.

Condition 2: the switch must not increase the composite Lyapunov function. With common features across stages, you can often form a composite error and show that the switch is a projection: the new task set activates on an error no larger than the old. If the switch necessarily introduces a new error component (say, orientation was uncontrolled during centering), its magnitude must be bounded and known, and the dwell condition below must allow the next stage to remove it.

Condition 3: dwell time. The controller must remain on each task set long enough that the error contracts by a guaranteed margin before the next switch. Contraction per dwell period compounds geometrically, which is what makes the sequence converge overall. A lower bound on dwell time relative to the loop's convergence rate is the standard dwell-time argument from switched-systems theory, applied here to the visual loop.

Bounded feature errors: the assumption that fails first

The interaction matrix is a local linearization of how feature motion responds to camera motion. Its entries stay reasonable while features sit well inside the image and the pose error is moderate. As a feature approaches the image border, or the pose error grows past the calibration's validity range, the linearization degrades and the commanded velocity can grow or point wrongly.

In a sequenced design this bites hardest at switches, because each switch resets which features matter. Practical mitigations:

• Add visibility constraints as a highest-priority task, with later tasks projected into the null space. The controller then steers away from image borders before a feature can leave.
• Cap the commanded velocity norm. A saturated law preserves stability for large errors if the cap keeps the camera within the region where L_e is well conditioned; the convergence rate degrades gracefully instead of diverging.
• Weight tasks so the dominant task consumes the available degrees of freedom first. The article on line features for visual servoing: errors and geometry shows how feature geometry drives the error terms you are bounding here.

Demonstrating stability on a real cell

A formal proof is the right exercise on paper; on the factory floor you demonstrate the same three conditions.

1. Measure the single-task response. Command one task set and record the error norm per cycle. Fit the exponential decay rate; this is your contraction rate and it sets the minimum dwell time for that stage.
2. Log the composite error across switches. Plot the norm of the composite error vector over a full sequence. It must be monotonically decreasing except for bounded jumps at switch instants, with the post-switch trend downward. Persistent growth or a plateau across switches is a stability violation.
3. Verify feature bounds. Track every feature's image position across the whole workspace the sequence runs in. Confirm margins from the image border and that no interaction matrix crossed a singularity threshold.
4. Sweep the dwell time. Shorten the dwell time in steps. Below a threshold the sequence will oscillate; that threshold, with margin, is your minimum dwell setting.

where lambda_effective is the measured error decay rate and k a margin factor of roughly 3 to 5. Treat this as an estimate from measured data, not a derivation.

Computing the loop fast enough for the proof to matter

The dwell-time argument assumes the visual loop runs at a fixed, known rate. If the servo rate varies, the effective gain varies and your measured contraction rate no longer describes the system. Keep the perception and control pipeline deterministic.

A typical split runs feature extraction and pose estimation on a dedicated controller, streaming pose errors to the robot over a real-time link at a fixed rate. The vision controllers category covers machines built for this: for example, the MV-VC3301H-128G66 gives you an Intel Core i3 processor with 8 GB of memory, 5+ LAN ports and 4 built-in light channels, which suits a single-camera PBVS cell with deterministic Ethernet throughput. When the sequenced task needs heavier feature processing (learned features, dense matching) alongside the servo loop, the MV-VC5525-256G70-NN provides an Intel Core i5-class processor with 32 GB of memory, PoE camera ports, 8 built-in light channels and 5+ LAN ports.

The camera matters too: a global shutter sensor avoids the rolling-shutter distortion that corrupts pose estimates during fast servo motion, and the area scan cameras finder filters by shutter type and interface. The camera acquisition software compatibility guide covers how to get a fixed-rate acquisition pipeline out of common SDKs.

Choosing gains and dwell times: a worked example

Suppose your centering stage shows an error decay rate of 2 per second (the error norm falls to about 13% in one second, since e^(-2) ~= 0.135). With a margin factor of 4:

Set the stage-A dwell to 2 seconds, or better, switch on an error threshold crossed and stable for a short confirmation window, with 2 seconds as the backstop minimum. If the alignment stage runs a slower rate of 1 per second, its dwell becomes 4 seconds. Total sequence time is then bounded and predictable, which matters for the cell's cycle budget. If that budget cannot absorb the dwells, raise the gains instead of shortening dwell below the bound; higher gain raises the decay rate that the dwell calculation divides by, at the cost of velocity saturation sooner.

When the usual answer is wrong

A few situations break the standard reasoning:

• Rapid switching or hysteresis-free thresholds. If two tasks alternate every few cycles near a boundary, you have effectively no dwell on either. Add hysteresis: require the switching condition to hold for N consecutive cycles.
• Interaction with the robot's own velocity loop. The Lyapunov argument covers the visual loop; a poorly tuned robot axis can oscillate independently. Verify the Cartesian velocity commands track cleanly before blaming the servoing law.
• Time-varying targets. A moving target adds a feedforward term; the error dynamics and the dwell bound change. The conveyor target availability article discusses the timing side of tracking moving parts.

Pre-deployment checklist

• [ ] Each task set verified stable in isolation, with measured decay rate.
• [ ] Interaction matrix conditioning checked across each stage's error region.
• [ ] Feature visibility margins confirmed across the full workspace.
• [ ] Dwell times computed from measured decay rates with a margin factor of 3 or more.
• [ ] Switching thresholds carry hysteresis.
• [ ] Servo loop rate measured and constant; jitter logged and within tolerance.
• [ ] Composite error logged over a full sequence and confirmed decreasing between switches.
• [ ] Commanded velocity saturation set below the robot's limits and inside the linearization's validity range.

Next steps

If you are specifying a controller for a sequenced servoing cell, the vision controllers category filters by LAN port count, PoE and processing power, and the system configurator walks the build step by step. For the loop structure itself, the visual servoing loop design guide pairs with this article: it sets the architecture whose stability this article proves. And if the stability behaviour you see does not match the analysis, the multi-camera image corruption isolation guide is a useful companion for ruling out hardware faults before re-deriving the control law.