Line features for visual servoing: errors and geometry

A detected straight edge tells you where a line appears, but not how far that edge lies from the camera. Confusing those two kinds of information leads to oversized error vectors, arbitrary plane equations and misleading expectations about motion control. This explainer is for developers connecting line detection to an image-based visual servoing controller. You will learn how to build the feature error, distinguish projection geometry from physical geometry, and check whether your selected lines can constrain the requested camera motion.

Start with two feature errors per matched line

An infinite straight line in a calibrated image has two independent parameters. A common representation uses its signed distance from the normalized image origin, rho, and the angle of its normal, theta:

Here, x and y are normalized image coordinates, not pixel coordinates. Theta describes the line's normal, not the direction along the edge. Check your detector's convention before constructing the feature.

Each matched line contributes two scalar errors. With four matched lines, the stacked feature vector and error vector each contain eight components. The camera velocity can still contain six components: three translations and three rotations.

These dimensions describe different things. Measurement count determines the number of interaction-matrix rows; controlled motion determines its columns. For unrestricted camera motion, four lines produce an 8 × 6 interaction matrix, not a 24-component error vector.

Finite segment endpoints, segment length or additional appearance measurements can form other features. They are not extra components of the two-parameter infinite-line representation. For broader choices between points, lines and other measurements, use the visual servoing feature-selection guide.

Convert detected pixel lines into calibrated image lines

Suppose your detector reports a homogeneous pixel line. For an undistorted pinhole image, convert it using the intrinsic calibration matrix:

This follows from the point relation p_pixel = K p_normalized and the line constraint l_pixel^T p_pixel = 0. The transpose is deliberate: lines and points do not use the same coordinate transformation.

Lens distortion requires separate treatment. A physical straight edge can appear curved in a distorted image, so transforming an already fitted distorted-image line is generally insufficient. Undistort the edge samples before fitting, or detect the line in a consistently rectified image. The guide to interpreting distortion coefficients explains the distinction between raw and corrected coordinates.

If you resize, crop or rectify the image, use the calibration associated with that resulting coordinate system. Keep the same convention for current and desired features.

The line representation also has a sign ambiguity. The pairs (rho, theta) and (-rho, theta + pi) describe the same line. Align each current representation with its desired representation before subtraction, then calculate a continuous angular difference. Otherwise, a small physical change can produce a large numerical jump.

Work through one line and a sampling budget

Consider a hypothetical undistorted image with focal calibration values fx = fy = 1000 pixels and principal point coordinates cx = 640 pixels, cy = 480 pixels. A vertical edge currently lies at u = 840 pixels; its desired position is u = 800 pixels.

That line contributes two errors even though only its distance error is nonzero. Repeat the calculation for the other matched lines, retaining a consistent ordering in the stacked vector.

Do not interpret the distance error as a physical translation without geometry. A normalized displacement of 0.04 corresponds to different object-space distances at different depths.

Camera sampling is a separate design calculation. Suppose the reference-plane field width is 160 mm, and a 0.5 mm separation between neighboring edges should span at least 4 pixels. The horizontal sampling requirement is:

This is an example sampling target, not a guaranteed detection threshold or pose-accuracy specification. Contrast, blur, line length and fitting quality also matter. The guide to choosing camera resolution for dimensional inspection provides more detail on separating pixel count from measurement performance.

Distinguish the viewing plane from physical line geometry

A calibrated image line defines a plane through the camera center. Every camera ray associated with that image line lies in this back-projection plane:

Its constant plane coefficient is zero because the plane passes through the camera origin. It tells you the directions along which the physical edge could lie. It does not identify the edge's depth or unique position in space.

Converting three collinear image samples into rays does not supply the missing depth. Those rays remain in the same viewing plane. Assigning an arbitrary common depth constructs an assumed scene; it does not recover the actual one.

A physical spatial line can instead be modeled as the intersection of two independent planes. For example, the object-frame planes X_object = 0 and Z_object = 0 intersect along the object's Y axis. They are useful only if that axis represents the actual modeled edge.

Some implementations accept two supporting planes; others accept spatial line coordinates, endpoints or equivalent depth quantities. Use the documented representation. Do not substitute the viewing plane for physical geometry simply because both use plane coefficients.

Transform object geometry with an explicit pose convention

Suppose the transform T_camera_object maps object-frame points into camera coordinates. A plane transforms by the inverse transpose:

Apply the transform to both supporting planes when your line representation uses a plane pair. Their intersection then describes the physical line in the camera frame.

Three non-collinear surveyed points can define a physical supporting plane. Three points sampled along a single straight edge cannot uniquely define that supporting plane, because they are collinear. A planar face surrounding the edge, known object dimensions or another independent geometric constraint supplies the missing information.

Several edges on the same object may share a supporting plane. You therefore do not necessarily need unrelated plane estimates for every detected line. You do need an unambiguous association between each image line and its modeled spatial edge.

A desired pose can generate desired image features and geometry for a desired-pose interaction matrix. It does not automatically provide the current geometry. Current, desired and approximate interaction matrices are different controller design choices. Follow the documented method and validate its useful operating region.

Use the extrinsic calibration accuracy checks to test transformed geometry independently of the image-fit residual. Preserve frame names, units and transform direction with the calibration, as described in saving calibration parameters for reuse.

Understand how geometry changes coupled camera motion

The interaction matrix maps camera velocity to feature velocity. For a stacked line feature vector, the local relationship and a common control form are:

This expression assumes a static target and compatible velocity, coordinate and sign conventions. Joint-space commands also require the appropriate robot kinematics and frame transformations.

The coupling becomes clearer by differentiating the line constraint at a projected point on the edge:

The projected point velocities depend on both camera translation and rotation. Translation sensitivity also depends on depth. Combining the constraints from distinct points along the physical edge gives the two line-feature derivatives.

Consequently, rho is not an isolated translation command, and theta is not an isolated rotation command. Forward translation can change line distance, while rotations can change both distance and orientation. Geometry changes these sensitivities throughout the matrix.

Plane coefficients therefore do not control particular motion axes. They describe geometry used to predict feature changes. The controller combines all errors and sensitivities to calculate a coupled velocity request.

For timing, velocity limits and convergence behavior beyond this geometry question, use the visual servoing loop-design guide.

Check rank and scaling before trusting the solution

Eight measurements do not guarantee control of six independent motion components. The stacked matrix must have sufficient rank for the motion you intend to control, and its conditioning must be acceptable.

Repeated observations of the same physical edge do not add independent scene geometry. Nearly redundant line arrangements can also provide weak sensitivity to some motions. A single line cannot observe camera translation parallel to that spatial line under the ideal infinite-line model.

Inspect singular values using a declared scaling convention. Normalized rho and angular theta have different meanings, so their raw numerical magnitudes should not determine their relative importance accidentally. Camera translation and rotation columns also require a deliberate scaling convention when assessing conditioning.

A practical approach is to weight feature errors using estimated measurement uncertainty. Apply those weights to both the residual and corresponding interaction-matrix rows. Reject unstable detections rather than giving them the same authority as clean, long edges.

Check sensitivity over the expected approach range, not only at the desired pose. A configuration that behaves well near convergence can become poorly conditioned farther away. If the mechanism permits fewer motion components, construct the reduced mapping for those actual degrees of freedom.

Choose acquisition hardware around feature repeatability

Start with area scan cameras and the camera finder when evaluating shutter type, interface and sampling. For moving scenes, shutter behavior and exposure duration affect how faithfully an image represents a single geometric configuration.

The MV-CU020-90GC is a global-shutter camera candidate. The LS-CMR-0053 is a rolling-shutter alternative to evaluate for stationary captures or sufficiently slow motion. Global shutter avoids row-dependent exposure timing, but it does not eliminate motion blur.

Choose machine vision lenses through the lens finder after defining coverage, working distance and edge sampling. The LS-FFL-0567 is a lens candidate to assess against the selected camera and scene requirements, not a guarantee of a suitable field of view.

Check lens sensor-format compatibility before ordering, then verify mechanical clearance and usable image coverage. Fix focus and aperture before calibration; changing the optical setup can invalidate the model used by the controller.

If integrating a dedicated acquisition and processing unit, compare vision controllers for the required interfaces and software compatibility. Controller hardware cannot compensate for missing scene geometry or inconsistent feature definitions.

Verify the model before enabling automatic motion

Use a staged check rather than treating initial convergence as proof that every convention is correct:

1. Confirm each detected edge is matched to the correct desired and modeled line.
2. Overlay projected model lines on a stationary image and inspect the entire visible edge.
3. Log pixel lines, normalized features, representation signs and the final stacked error.
4. Perturb a simulated camera pose separately along each motion component and compare finite-difference feature changes with the interaction matrix.
5. Check matrix rank, scaling and conditioning across representative poses.
6. Validate permitted low-speed motion under the machine's approved commissioning procedure, with safety functions active.

A finite-difference test changes the pose by a small known amount, recomputes features and divides their change by that perturbation. Repeat with smaller perturbations to distinguish derivative behavior from numerical noise or feature reassociation.

The usual two-parameter answer is wrong when your task genuinely uses segment endpoints, curves or other feature definitions. It is also insufficient when lines cannot constrain the required motion, correspondences are ambiguous, or the target moves outside the controller's assumptions. Add appropriate measurements or change the task formulation rather than inventing extra line-error components.

Bring your target geometry, motion range, image samples and calibration conventions to applications support, or start a camera configuration once the required sampling and acquisition conditions are defined.