Learned features for visual servoing: choosing CNN outputs

A pretrained convolutional neural network can supply observations to a visual servoing system, but its output is not automatically a usable control feature. A bounding box may locate the correct object while moving unpredictably as the viewpoint changes. A pose estimate may look convincing while switching between symmetric solutions.

For vision-guided manipulation, choose the representation from the corrections you must control, then validate its behavior across the approach. The practical task is to connect an inference adapter to your controller, establish coordinate and timing conventions, and reject observations that cannot support a reliable correction.

Choose the output from the required correction

Start by listing the motions the vision system must resolve. Centering an object laterally is different from aligning its surface normal or estimating its complete spatial orientation. More network outputs do not necessarily provide more independent geometric information.

A box center is not necessarily the projection of a fixed physical point. Likewise, the centroid of a visible mask can move when an object becomes partially hidden, even if the object itself remains stationary.

For approach geometry rather than learned representations, use the companion guide to feature selection for a perpendicular approach. Here, the decision is whether a network output preserves the information that geometry requires.

Separate the neural network from the control law

The network should produce observations; a separate adapter should convert them into validated features. That separation lets you change inference software without silently changing the controller's coordinate conventions or failure behavior.

Check the documented capabilities of your installed inference and control libraries. Support for a model file format does not guarantee support for every operator, preprocessing sequence or output tensor layout.

A useful observation record contains:

• Capture timestamp and frame identifier.
• Source image dimensions, crop and resize transformation.
• Object class and association identifier.
• Feature values, validity flags and confidence information.
• Depth provenance or pose coordinate frame, when applicable.
• Model, preprocessing and calibration version identifiers.

The guide to detection messages between processing components covers the transport boundary. Preserve those semantics even when inference and control run in the same process.

Image-based control

Image-based visual servoing uses image features and their difference from a desired observation. A common local formulation is:

Here, s is the feature vector, L_estimated relates camera motion to feature motion, and lambda sets the correction rate. This is a local model, not a stability guarantee for arbitrary network outputs.

For conventional point features, translational terms depend on depth. Using image coordinates therefore does not eliminate depth assumptions. Bounding boxes, masks and learned descriptors need a justified feature-motion model rather than an interaction matrix copied from unrelated point features.

Pose-based control

Pose-based visual servoing forms a position and orientation error from the estimated object pose and the desired relationship. It requires consistent camera, object and tool frames, together with the appropriate calibrated transforms.

A visually smooth pose can still use the wrong axis direction, distance unit or symmetry convention. Check those definitions before tuning gains. The broader eye-in-hand loop design guide addresses the surrounding control architecture.

Measure feature stability before moving the mechanism

Evaluate the exact pretrained weights, input preprocessing and output decoder you intend to deploy. A model trained to recognize an object category may not provide stable instance identity, semantic landmarks or metric pose.

Use recorded sequences covering the full intended approach, not only isolated images. Include viewpoint changes, partial occlusion, background variation, motion blur and the nearest intended working position.

Separate noise from geometric drift

First hold the camera and object stationary. Measure feature spread, outliers, identity changes and invalid observations. Then change viewpoint by known increments and inspect whether each feature follows the same physical structure.

A feature may have low stationary jitter but large viewpoint-dependent bias. Conversely, a meaningful landmark may follow the correct geometry while remaining too noisy for the final correction tolerance.

Suppose a network processes a resized crop that is 640 pixels wide, taken from a 1280-pixel-wide source region. The horizontal scale factor is 1280 / 640 = 2. A measured feature spread of 0.8 pixels at network input scale becomes 0.8 × 2 = 1.6 pixels in source coordinates.

That conversion matters when the controller's acceptable image error is only 2 pixels. Include crop offsets and any padding when restoring coordinates; scaling alone is insufficient after letterboxing.

Check observability and identity

A single image point provides two scalar coordinates. It does not independently constrain every component of a general spatial motion. Additional features must supply independent information, not merely duplicate the same center estimate.

For keypoints, check their spatial distribution and visibility throughout the approach. For masks, test centroid and contour behavior under occlusion. For poses, inspect orientation continuity and alternative solutions for symmetric parts.

Treat a sudden object association change as a new observation sequence, not as ordinary feature motion.

Decide what depth information the representation needs

Depth may come from a sensor, known geometry, a pose estimator or an approximation over a restricted operating region. Each option carries assumptions that should remain explicit in the observation record.

For image-based control, an approximate depth can sometimes support a limited correction, but its adequacy must be demonstrated over the intended range. A changing approach distance can invalidate a fixed-depth approximation that worked during lateral alignment.

For pose-based control, confirm whether translation is metric and what establishes its scale. A monocular pose network may require known object dimensions or a corresponding object model; do not infer metric accuracy from a pose visualization.

When combining image detections with depth, use the bounding-box-to-depth mapping guide. A rectangle can include foreground, background and invalid depth, so its central sample is not automatically the object's surface distance.

Compare 3D sensors through the camera finder when measured depth is part of the design. The MV-DPS200P-02 is a projected structured-light 3D camera. Its listed category does not establish suitability for your surface, distance or motion conditions.

Validate depth availability on representative parts before choosing it as a control input. Also verify spatial registration and depth-to-image timestamp alignment; matching array dimensions do not establish either relationship.

Budget observation age rather than inference time alone

The controller acts on an observation of the past. Exposure, transfer, preprocessing, inference, queueing and command delivery all contribute to that age.

Use a documented timestamp convention and map timestamps into a common clock domain. Record the observation age at the point where a command is accepted, not merely when inference finishes.

Consider an illustrative pipeline budget:

At a relative speed of 80 mm/s, the corresponding travel is:

This estimates travel during the observation delay. It is not a complete positioning-error prediction because geometry, feedback and any validated prediction also affect the result.

A stream arriving every 20 ms can still contain observations that are much older. Throughput, inference duration and observation age are different measurements.

Log median, high-percentile and maximum observed ages during representative load. Test startup, recording, display updates and competing processing. Set a maximum usable observation age from the motion and error budget, then verify what happens when that limit is exceeded.

Match image sampling to the learned feature

A larger input image is useful only if it preserves relevant detail and the model can use that detail. Resizing a high-resolution acquisition to a small network input may remove the very landmark you selected.

As an illustrative sampling requirement, suppose a landmark detail is 0.6 mm wide and you want it represented by 6 pixels across a 120 mm horizontal field of view:

This is a sampling estimate, not a localization-accuracy guarantee. Blur, contrast, optical quality, training data and preprocessing remain limiting factors. Check the landmark size after the final resize, not only in the original image.

For the distinction between pixel sampling and measurement capability, read choosing resolution for dimensional inspection.

Browse area scan cameras by interface, shutter and image requirements. The LS-CMR-0166 is a USB3 area scan camera with a rolling shutter. Treat it as a candidate to evaluate, not as an established fit for moving-camera control.

Rolling-shutter acquisition can distort feature geometry during motion because different image rows are exposed at different times. A global-shutter alternative may be preferable when that distortion exceeds your feature budget, but it does not remove exposure blur or processing delay.

Control illumination and exposure during representation testing. Otherwise, changes attributed to the network may actually come from changing image formation. The guide to automatic exposure and gain control explains those acquisition tradeoffs.

Select compute by measured end-to-end behavior

Benchmark the complete exported model on the intended runtime and hardware. Include preprocessing, memory movement, output decoding and synchronization with acquisition.

Compare execution options with the same image sequence and numerical settings. Quantization or a different backend can change feature positions, confidence scores and invalid-result behavior even when average inference becomes faster.

Use vision controllers when their camera connections and integration facilities suit the system. The MV-VC5525-256G70-NN has PoE camera ports. That connection capability does not establish model compatibility, inference performance or timing determinism.

Before selecting compute, obtain documented runtime compatibility and measure the intended workload. Do not assume an accelerator is present or supported because a device is described as a vision controller.

Keep acquisition, inference and command generation separated by explicit ownership and freshness rules. The guide to adapting camera image input covers the acquisition boundary without treating the inference result as a control-ready observation.

Validate rejection behavior before closing the loop

An observation rejection policy is part of the controller design. Confidence alone is insufficient: a high-confidence output can still be geometrically inconsistent, stale or associated with the wrong object.

Use staged validation with recorded evidence at each stage:

1. Replay recorded acquisitions. Verify decoding, coordinate restoration, object identity and repeatability without connected motion commands.
2. Run live acquisition without motion authority. Record proposed corrections, observation ages and rejection reasons under representative processing load.
3. Test deliberate failures. Hide landmarks, remove the object, introduce a second object, interrupt depth and delay inference. Confirm that invalid observations cannot produce fresh correction commands.
4. Check command direction in a controlled setup. Use the machine's approved commissioning procedure, restricted motion and appropriate safeguarding. Confirm transform signs before increasing motion authority.
5. Expand the validated operating envelope gradually. Add viewpoints, approach distances and operating speeds only after reviewing errors and loss behavior.

Use separate thresholds for accepting a new observation sequence and continuing an established one where appropriate. Validate any hysteresis so it does not prolong acceptance of a wrong association.

After detection loss, require a documented reacquisition policy. Reset or reconcile filters, target identity and accumulated controller state before resuming correction. A reappearing box should not automatically inherit the previous target's state.

Check the shortlist and recognize when to change approach

Before ordering or committing to a representation, complete this checklist:

• Task geometry: identify which motion components must be observed and which are constrained elsewhere.
• Output meaning: document whether features represent fixed landmarks, a visible silhouette or an estimated object frame.
• Model suitability: verify target classes, training assumptions, deployment rights and required object models.
• Image formation: confirm sampling after resize, motion blur, shutter behavior and lighting stability.
• Depth and calibration: establish scale, registration, coordinate frames and applicable working limits.
• Timing: measure observation age and jitter with the complete production workload.
• Failure behavior: test stale results, identity changes, occlusion, depth loss and reacquisition.
• Acceptance evidence: define tolerances before testing, and retain sequences showing both passes and failures.

Choose boxes when coarse location is enough and other constraints resolve the remaining motion. Choose keypoints when stable semantic locations support the required geometry. Choose masks when the visible silhouette contains useful information that survives the expected occlusion.

Choose pose outputs when the task needs an object-frame position and orientation, and you can validate calibration, scale and symmetry handling. Pose-based control is not automatically preferable simply because the network returns more values.

The usual answer may also be wrong: if conventional edges, fiducials or geometric fitting already provide stable features, a CNN can add deployment effort without improving control. A useful hybrid is learned object localization followed by geometric refinement, provided both stages meet the timing and validity requirements.

Start the camera configuration workflow once the feature and acquisition requirements are defined. For an application review, use expert support with representative image sequences, feature tolerances, depth needs and the measured timing budget.