Eye to hand object pose transform for robot picks

A fixed camera can locate a part accurately while the robot still receives an unusable pick pose. The missing information is often the relationship between the detected object, the intended grasp, and the active tool center point. This procedure is for integrating an already calibrated camera with a robot. You will compose the transforms, distinguish a current tool-relative observation from a stored grasp relationship, and validate the calculation before commanding motion.

Establish the frame contract and required equipment

Start with a valid camera-to-base calibration, a measured object pose, and a defined grasp. Calibration alone does not specify where the gripper should contact the production part. For the broader workflow, see applying an existing camera-to-base calibration.

Use these frame names throughout the calculation:

The notation T_B_C means a transform that converts coordinates expressed in C into coordinates expressed in B. The destination frame comes first. Establish this convention in your integration code rather than relying on variable names such as camera pose.

For acquisition hardware, compare 3D sensors through the camera finder. The MV-DLS300P-04 is a laser galvanometer stereo camera.

The MV-DPS200P-02 is a projected structured-light camera.

These catalog descriptions identify imaging methods, not robot-interface compatibility. Confirm accessible pose data, coordinate definitions, supported software, and calibration procedures before selection. The camera and robot compatibility checklist covers that integration boundary.

If processing runs on vision controllers, verify software support and communication requirements independently of the camera connection.

Normalize every pose representation

1. Convert each incoming pose into a homogeneous transform with documented frame direction, rotation convention, and translation units.

A homogeneous transform contains a rotation matrix and a translation vector:

Here, p_C and p_B describe the same physical point in different frames. A translation vector alone cannot transform an object orientation.

Before converting any six-value pose, record:

• Translation units: millimeters or meters, for example.
• Rotation representation: Euler angles, rotation vector, quaternion, or matrix.
• Euler axis sequence and intrinsic or extrinsic interpretation.
• Angular units and coordinate-system handedness.
• Quaternion component order, when applicable.
• Whether the pose describes a frame in its parent or the reverse mapping.

Do not assume all three-angle representations are interchangeable. A rotation vector is not an Euler-angle triplet. Keep matrices or correctly implemented quaternions internally, then convert to the exact representation required by the robot interface.

Also confirm that the calibration and detection use the same camera frame. An optical frame and a camera mounting frame can have different axis directions. If those differ, insert the documented fixed transform; the guide to optical-frame and camera-frame conversions explains this boundary.

Compose the object pose in the robot base

2. Obtain the camera-to-base mapping, then multiply it by the detected object pose in camera coordinates.

If calibration supplies T_C_B, it describes the base frame expressed in camera coordinates. Invert it before applying a camera-frame measurement:

If calibration already supplies T_B_C, use it directly. Never add an inversion solely because an example elsewhere includes one.

For a rigid transform, inversion changes both orientation and translation:

Negating translation without rotating it is generally incorrect. Reversing the multiplication order is also incorrect: matrix multiplication is not commutative.

Read the frame subscripts as a chain. In T_B_C × T_C_O, the adjoining C frames match, leaving a mapping from O to B. This is a useful implementation check, although it cannot detect a mislabeled input.

The resulting T_B_O locates the object's model frame. It is not automatically a valid TCP target.

Convert the object pose into the intended TCP pose

3. Apply a deliberately defined grasp relationship, not merely whichever tool-relative pose the calibration software exported.

Define T_O_T_grasp as the desired TCP frame expressed in object coordinates. It contains both the contact offset and the gripper orientation relative to the part:

Some systems store the opposite relationship: the object's pose in the tool at the intended grasp, written here as T_T_O_grasp. In that case:

A calibration output described as object-in-tool may instead refer to the calibration target attached to the robot. That transform is not necessarily the production part's grasp relationship. Reuse it only when the physical frame relationships actually match, or when a separately established conversion connects them.

A current tool-relative observation answers a different question

To express a detected object relative to the TCP at the time of observation, use:

This answers where the object is relative to the current tool. It does not independently define how the object should sit in the gripper after grasping.

If that measurement is consumed by a relative-motion interface, establish whether the interface expects a displacement expressed in base axes, current tool axes, or another reference frame. Those commands are not interchangeable. For an absolute Cartesian target, calculate the desired base-frame TCP pose explicitly.

Convert to a flange target only when required

If the command expects the flange pose and T_F_T is the configured flange-to-TCP transform:

Do not perform this conversion when the robot already accepts TCP targets using its active tool definition. Otherwise, the tool offset can be applied twice.

Check the arithmetic with a worked example

4. Verify position and orientation using a small synthetic example before inserting live detections.

Assume the camera frame is rotated 90 degrees about the base Z axis. Its origin is located at (500, 100, 800) mm in the base frame:

Rotate the detected position, then add translation:

For this arithmetic example, assume the object axes align with the camera axes. Therefore, R_C_O is the identity matrix and R_B_O = R_B_C.

Now define the desired TCP offset as (20, 0, 60) mm in object coordinates, with TCP axes aligned to object axes:

Notice that the object-local X offset becomes a base Y offset. Adding (20, 0, 60) mm directly to the base-frame object position would produce the wrong target.

These coordinates demonstrate composition only. They are not an assertion of reachability, clearance, or a suitable gripper orientation. Define the actual grasp and approach using the tool geometry and the machine's approved operating procedures.

Validate the calculation without commanding motion

5. Compare predicted frames and physical references before enabling any movement based on the calculated target.

Use several checks, because a plausible-looking coordinate is weak evidence:

• Inverse check: multiply a transform by its inverse. The result should approximate the identity matrix within numerical precision.
• Rotation check: confirm transpose(R) × R approximates identity and the determinant approximates positive one.
• Direction check: move a test object along a known physical axis and confirm the expected base-coordinate change.
• Offset check: rotate the object model and verify that its local grasp offset rotates with it.
• Round-trip check: convert the final orientation into the robot representation and back. Compare rotation matrices, not only displayed angles.
• Independent position check: compare the predicted grasp frame with a physically established reference that was not used to construct the same calculation.

A round trip proves internal consistency, not physical calibration accuracy. For independent acceptance checks, use the camera-to-base calibration verification procedure.

Capture the image identifier, object-model revision, transform revision, active tool, and relevant timestamps with each computed target. If you use a current TCP pose, associate it with the measurement time rather than an unrelated later robot state.

For a fixed camera observing a moving part, correct frame composition does not compensate for travel after acquisition. Stop the part or use an explicitly designed tracking and timing method.

Send the pose through the correct command interface

6. Confirm the command's reference frame and controlled point, then follow the machine's approved commissioning procedure.

If the interface expects an active user-frame target instead of a base-frame target, convert accordingly:

Here, U is the active user frame. Never send base coordinates while silently leaving a different command frame selected.

Check the active TCP definition, orientation format, units, robot configuration, workspace limits, and allowed approach direction. A Cartesian pose can have multiple joint solutions or no usable solution. Reachability and collision checking remain separate from pose arithmetic.

When planning a pre-grasp position, define whether the approach offset belongs to object, tool, or base axes. A base-Z retreat is not necessarily aligned with a tilted gripper's withdrawal direction.

Diagnose common composition mistakes

For position drift linked specifically to orientation, the rotation-dependent pick-error diagnostic provides a focused next test.

Complete the integration handoff

Before ordering hardware or releasing the application, record:

• [ ] Camera measurement frame and physical axis directions.
• [ ] Calibration direction, units, and revision.
• [ ] Object-model origin and orientation definition.
• [ ] Desired grasp relationship, separate from any calibration-target relationship.
• [ ] Command reference frame, active tool, and rotation representation.
• [ ] Processing software compatibility and supported communication path.
• [ ] Independent acceptance criteria, timing assumptions, and commissioning responsibilities.

If the required object orientation is ambiguous because the part is symmetric, resolve that ambiguity in detection or grasp planning. Transform multiplication cannot recover an orientation the measurement never established.

Bring this frame contract and a representative recorded detection to applications support. For component selection, start with the camera configuration step after defining the measurement and interface requirements.