IBVS convergence on small targets: diagnose incorrect poses

A tilted eye-in-hand camera can drive image residuals downward while the tool finishes at the wrong position or orientation. This is especially difficult when tracked points occupy a small patch of the image: different tool motions can produce nearly indistinguishable feature changes. For a planar task that permits translation and yaw but forbids tool roll and pitch, diagnose the constrained image-to-motion model before changing gains. The tests below separate frame errors, weak geometric sensitivity, depth errors and genuinely unreachable image references.

Start by separating image convergence from pose convergence

A small image residual proves agreement with the selected image measurements, not necessarily agreement with the required tool pose. Perspective ambiguities, poor conditioning, tracking bias and an incorrect motion model can all weaken that connection.

Record these signals together throughout each trial:

• Image residual in pixels, including the largest individual point error.
• Tool translation and orientation error from an independent reference.
• Commanded and measured tool velocities, with their coordinate frames identified.
• Feature spread, depth estimates, damping and acquisition-to-command latency.
• Singular values of the scaled, constrained image Jacobian.

Independent pose verification must not simply reuse the same feature model and assumed depth. Depending on the task, use a calibrated fixture, separate measurement system or validated reference positions.

Treat low residual with persistent pose error as a diagnostic symptom, not immediate proof of a local minimum. First establish that the reference pose is reachable under the permitted motions. The visual servoing loop design guide covers the surrounding timing and feedback architecture; the focus here is the geometry inside that loop.

Check the likely causes in a controlled order

Use the same initial pose and reference image when comparing changes. Otherwise, a smaller starting displacement can make an ineffective modification appear successful.

A wider calibration target is a useful diagnostic because it changes feature geometry. Its success does not demonstrate that the small production part provides equivalent pose information.

For feature alternatives, use the perpendicular-approach feature selection guide. Changing feature type should follow identification of the weak motion direction, rather than replace that diagnosis.

Put the four permitted tool motions into the Jacobian

Suppose the tool may translate along its own axes and rotate about its own z-axis. Tool roll and pitch remain fixed. Define the permitted velocity vector as u = [vx, vy, vz, wz]^T, expressed at the tool origin in tool coordinates.

For a stationary target and fixed desired features, the local model is:

Here, L is the feature interaction matrix, C maps the tool-origin twist into the camera-origin twist, and A is the constrained Jacobian. For N point features, their dimensions are L: 2N × 6, C: 6 × 6, S: 6 × 4 and A: 2N × 4.

Solve using A. Do not solve for an unrestricted camera twist, transform it into tool coordinates, and then zero tool roll and pitch. That discards components the original solution relied upon. The remaining motion generally no longer produces the predicted image correction.

The velocity mapping must include both camera tilt and camera-to-tool offset. For linear-then-angular twist ordering, one consistent convention is:

The cross-product block accounts for camera translation caused by rotation about the offset tool origin. Confirm that your software uses this same transform direction, origin definition and component order.

The selection matrix describes instantaneous allowed motion. It does not restore an already incorrect tool tilt. Establish the required initial orientation separately, and verify the robot executes tool-frame rather than base-frame commands.

Verify the model with isolated, bounded motions

Before closing the loop, compare the predicted image derivative with measured feature movement. Use undistorted normalized coordinates consistently with the interaction matrix.

For normalized point coordinates (x, y) and positive camera-axis depth Z, a conventional point interaction matrix is:

This convention assumes camera motion relative to a stationary scene. Reversing that interpretation changes signs.

1. Hold the target stationary and capture a baseline feature vector.
2. Apply a small positive and negative displacement along each permitted tool coordinate.
3. Estimate each column of the constrained image Jacobian with a central difference.
4. Compare measured and predicted column directions and magnitudes.
5. Repeat at another depth and image location before enabling feedback.

For example, translations of plus and minus 0.05 mm give a central-difference denominator of 0.10 mm. Choose the actual excursion from clearance, measurement noise and the valid local-motion range, not from this example alone.

A sign reversal suggests a convention error. A consistent magnitude mismatch suggests scale, depth or calibration error. A badly predicted yaw column often points toward an incorrect rotation axis, omitted camera offset or frame mismatch.

Use the hand-eye calibration pose-selection guide if the camera-to-tool transform itself needs validation. More gain cannot correct a wrong transform.

Measure conditioning with meaningful scaling

Many points in a compact cluster can provide repeated measurements of nearly the same motion signature. They may reduce random noise without adding much independent geometric information.

Compare singular values only after defining translation, rotation and feature-noise scales. Raw columns mix translation and angular units; changing millimeters to meters changes their numerical comparison without changing the physical system.

Let D contain characteristic permitted displacements, and let W scale feature residuals by their measurement uncertainty. Define:

For an illustrative diagnostic, characteristic translations might be 1 mm and characteristic yaw 0.01 rad. These are coordinate scales, not commanded excursions. Keep them fixed across comparisons.

If the scaled singular values are 1.0, 0.6, 0.15, 0.002, the ratio is 1.0 / 0.002 = 500. This indicates a relatively weak direction for those scales; it is not a universal rejection threshold. Inspect the corresponding right singular vector to identify the combination of tool motions that produces little image change.

A damped solution can then be written as:

Use a numerically stable linear solve or singular-value decomposition rather than explicitly forming an inverse. Damping limits sensitivity near weak directions, but it also reduces correction along those directions. It does not create missing information.

If the residual remains nonzero while the command approaches zero, inspect its projection onto the constrained Jacobian's column space. A residual outside that space cannot be removed by the current local set of permitted motions.

Work through the small-target sampling numbers

Consider an illustrative 4 mm by 3 mm planar target viewed across an 80 mm horizontal field. With 4000 horizontal pixels, nominal object-plane sampling is:

The height estimate assumes equal object-plane sampling in both directions. Tilted viewing changes local scale and foreshortening, so evaluate actual feature coordinates rather than treating this estimate as a calibrated projection.

The target occupies only 4 / 80 = 5% of the horizontal field. Adding points inside that footprint does not widen the geometric baseline. To estimate a sampling requirement, a 0.08 mm detail represented by four pixels requires 80 × 4 / 0.08 = 4000 horizontal pixels.

Field of view determines how widely the target spans the sensor. Narrower coverage can improve sampling, but it also reduces allowance for initial position error and target motion.

Use object-space pixel size from magnification to relate optical magnification to sampling. A localization uncertainty of 0.2 pixel would correspond nominally to 0.2 × 0.020 = 0.004 mm in this simplified plane. It does not establish equivalent tool-pose accuracy, particularly in depth or weakly observed rotations.

Compare the small target against a wider target at the same camera pose. If the weak singular value improves substantially, the evidence favors a geometry limitation rather than insufficient point count alone.

Test depth estimates without changing everything else

Point-feature translational sensitivity depends on inverse depth. A depth assumption that works near one pose may produce misleading coupled corrections elsewhere.

For a tilted planar surface, points generally do not share identical camera-axis depth. If a validated plane is available in camera coordinates, intersect each normalized viewing ray with that plane:

Use this only with consistent units, the intended ray intersection and positive valid depths. Near-zero denominators make the estimate unreliable.

Compare three models while holding feature tracking, gain and damping fixed:

• A single constant depth for all points.
• Per-point depths from a calibrated plane.
• A locally measured constrained Jacobian from bounded perturbations.

For scale, changing assumed depth from 100 mm to 110 mm changes the magnitude of a 1/Z coefficient by a factor of 100 / 110 ≈ 0.91. The resulting controller behavior depends on the complete coupled matrix, not just this coefficient.

If measured derivatives agree with the plane model but not the constant-depth model, depth variation is a supported explanation. If neither agrees, return to calibration, transform conventions and correspondence quality.

Do not add heavy smoothing during this comparison. Filtering can reduce noise while adding delay, obscuring whether the geometry improved.

Choose components around the diagnostic requirements

Select area scan cameras for sufficient target sampling, stable acquisition and accessible timing information. Use the camera finder to narrow interface and shutter requirements, then verify operating modes in the manufacturer's documentation.

The LS-CMR-0053 is a GigE area scan camera with a C-mount. It is a catalog candidate for acquisition evaluation, not evidence of suitability for a particular feedback rate.

Select machine vision lenses by sensor coverage, required field, working distance and measurable image quality. The lens finder helps narrow optical options before testing the actual target.

The LS-FFL-0567 is a C-mount lens with a 16 mm focal length. Confirm field coverage, focus range and target contrast in the proposed arrangement before committing to an optical pairing.

Evaluate vision controllers against acquisition software compatibility and measured end-to-end latency. Camera-port availability alone does not establish deterministic control performance. The camera integration guide addresses adapting image input without losing the metadata needed by the loop.

Before ordering, document sensor dimensions, lens mount, image circle, working distance, target footprint, exposure time, timestamp availability and software licensing requirements. Keep illustrative sampling calculations separate from verified component performance.

Verify the fix and recognize when constraints must change

Repeat translation-only, yaw-only and combined-offset trials from both signs of each displacement. Include the expected starting extremes, not just a favorable central pose. Judge success using independent tool-pose limits as well as image residuals.

A practical acceptance checklist is:

• Predicted and measured feature derivatives agree in sign and reasonable magnitude.
• The constrained Jacobian contains only the motions the task permits.
• Forbidden tool rotations remain fixed in measured motion, not merely in software variables.
• Weak singular directions remain within an explicitly accepted noise and motion budget.
• Depth estimates remain valid across the tested approach range.
• Tracking loss, stale frames and excessive residuals invoke the validated stop or recovery behavior.
• Final translation and yaw satisfy independently measured task tolerances.

The usual four-motion solution is wrong when the desired image requires a different tool tilt, the surface is not represented by the assumed plane, or the available observations cannot distinguish required poses. Capture a reachable reference, revise the feature geometry, add independent measurements or redesign the task sequence. Do not simply increase gain until a weakly observed motion becomes aggressive.

Start with the camera configuration step once sampling and timing requirements are measured. For a geometry review, send target dimensions, permitted tool motions, calibration conventions and diagnostic traces through applications support.