Fiducial tag pose estimation in OpenCV from detected corners

A tag detector hands you four corner coordinates in pixels. What your robot or stage needs is a 6-DoF pose: the tag's position in metres and its orientation in 3D. The bridge between the two is OpenCV's cv2.solvePnP with the SOLVEPNP_IPPE_SQUARE flag, which solves the planar four-point perspective problem directly for a square target. This guide walks through the whole path: building the 3D object points that match your detector's corner order, supplying camera intrinsics and distortion, running the solve in Python, and checking whether the result is good enough for your task. It is for anyone doing fiducial tag pose estimation with OpenCV who has detections but not yet a pose.

What you need before you can solve a pose

Three inputs feed solvePnP, and all three must be consistent:

1. 2D image points. The four detected corners, in the order your detector reports them. A typical detector returns the corners counterclockwise starting from the lower-left; confirm this for your library rather than guessing, because a wrong order silently produces a rotated or mirrored pose.
2. 3D object points. The same four corners expressed in the tag's own coordinate frame, in metres, with Z = 0. For a tag of side s centred at the origin, the half-side is h = s/2.
3. Camera intrinsics. The 3×3 camera matrix (fx, fy, cx, cy) and the distortion coefficients, from a calibration of the exact camera and lens you are using.

If you have detections but no calibrated camera yet, the pose will be scaled or bent wrong. For the intrinsics step, the camera intrinsic parameter conversion for depth and texture images article covers how intrinsics differ between streams, and the article on rectified image pose estimation and choosing PnP intrinsics goes deeper into which intrinsics a PnP solve actually needs. Any global-shutter area scan camera works here; we stock a wide range of area scan cameras and 3D sensors if you are still selecting hardware.

Build the object points to match the corner order

The single most common failure is an object-point list that does not correspond, corner for corner, to the detector's output. Write them together, in one place:

In Python, with s the printed tag width in metres and assuming a detector that returns corners as lower-left, lower-right, upper-right, upper-left in image coordinates (y down):

import numpy as np

s = 0.10                      # printed tag width, metres
h = s / 2.0
obj = np.array([[-h, -h, 0],
                [ h, -h, 0],
                [ h,  h, 0],
                [-h,  h, 0]], dtype=np.float32)

# corners from the detector, shape (4, 2), same order
img = corners.astype(np.float32)

If your detector returns a different order, permute obj, not img, so the mapping stays readable. Getting the sign of the y-axis wrong shows up as the pose pointing into the camera; if that happens, flip the sign of every y in obj first.

Feed intrinsics and run the solve

With the camera matrix K and distortion vector D from calibration:

K = np.array([[fx,  0, cx],
              [ 0, fy, cy],
              [ 0,  0,  1]], dtype=np.float64)
D = np.array([k1, k2, p1, p2, k3], dtype=np.float64)

ok, rvec, tvec = cv2.solvePnP(obj, img, K, D,
                              flags=cv2.SOLVEPNP_IPPE_SQUARE)
R, _ = cv2.Rodrigues(rvec)

solvePnP returns True on success, a rotation vector (convert with Rodrigues to a matrix) and a translation vector giving the tag origin in camera coordinates. Because IPPE is built for planar squares, it internally evaluates two plane hypotheses; if you want to catch ambiguous views, use solvePnPGeneric with the same flag and inspect the reprojection error of both solutions yourself.

Where do fx, fy, cx and cy come from? From a checkerboard calibration, or as a first estimate from the lens geometry. The relationship is:

where f is the lens focal length and p the sensor pixel pitch. This estimate ignores distortion and is only a starting point — a real calibration always beats it. The lens focal length calculator helps you pick the focal length for a given field of view in the first place, and the same page's assumptions are what this estimate rests on.

Worked example: a 100 mm tag at 500 mm

Take a 100 mm tag (s = 0.10, so h = 0.05 m) imaged by a 12 MP camera with a 4000 × 3000 sensor, 1/1.7" format, and a 16 mm lens. The pixel pitch is roughly 1.1 µm, so:

At 500 mm working distance, the tag's apparent width is:

A tag spanning nearly 3,000 pixels is ideal — corner localisation error is a fraction of a pixel, so pose error stays small. Now move the same tag to 2 m: the apparent width drops to about 725 pixels, and the same one-pixel corner error now produces roughly four times the relative size error, which multiplies straight into the depth estimate. This is why single-tag depth is trusted at close range and treated as a rough number at distance.

For the camera side, a 12 MP USB3 model such as the LS-CMR-0166 covers this kind of bench set-up well; its 4000 × 3000 rolling-shutter sensor gives plenty of corner resolution for static tags.

How the OpenCV path compares to a tag library's own solver

Many tag libraries ship their own pose estimator, and it is fair to ask whether the generic OpenCV route loses anything. The two solve the same problem with different machinery:

The practical reading: solver choice is rarely your accuracy bottleneck. Corner detection quality, calibration quality and tag size in pixels dominate. If you can call the library's solver from your language without writing bindings, either path is defensible; if the wrapper is missing, SOLVEPNP_IPPE_SQUARE from Python is the shorter road and loses nothing that matters.

Verify the pose before you trust it

Run these four checks on your first working build:

1. Reproject and overlay. Use cv2.projectPoints to draw the solved object points back onto the image. They should sit on the detected corners to within a pixel or so.
2. Measure a known distance. Fix the tag at a measured distance (say 500 ± 0.5 mm) and compare tvec's norm. Repeat at 250, 750 and 1000 mm; the error should grow roughly with distance squared for depth.
3. Rotate the tag slowly. Watch for pose flips between the two IPPE hypotheses, which show as sudden 180°-style jumps. If you see them, enlarge the tag or avoid near-frontal views.
4. Check rotation against a physical angle. Tilt the tag by a known amount, for example 15°, and read the angle from the rotation matrix.

If depth accuracy at range matters more than a single tag can give, stereo is the natural upgrade; the stereo point cloud visualization guide shows how to check that a stereo system's 3D output is actually complete, and mapping image pixels to 3D points covers the pixel-to-3D math that replaces the tag geometry.

When a single tag is the wrong tool

A square tag gives you four constraints for six unknowns, so the solve leans entirely on the planar assumption and your intrinsics. Know its limits:

• Depth is the weak axis. For solid depth at 1 m or more, prefer a stereo pair, a structured-light camera or a laser profiler — see the 3D sensors category for the range.
• Rolling shutter bends fast poses. If the camera or tag moves during exposure, corners smear and rotation reads wrong. Use a global-shutter camera for moving targets.
• Distortion at the image edges matters. Keep the tag near the centre, or calibrate well, or both.
• Multiple tags beat one big tag for orientation stability, since the solve averages over a larger baseline.

Quick checklist before production

• [ ] Corner order in code matches the detector's documented order.
• [ ] Tag size in metres matches the physical print, measured with a ruler.
• [ ] Intrinsics calibrated on the exact camera and lens in use.
• [ ] Distortion coefficients non-zero unless verified negligible.
• [ ] Reprojection overlay checked at three distances and three tilts.
• [ ] Pose flip behaviour tested near frontal views.
• [ ] Depth error measured against a known distance and within tolerance.

When you are ready to pick hardware, the camera finder filters by resolution, shutter and interface, and the system configurator steps through camera, lens and light together so the intrinsics story stays consistent end to end. If the wider geometry questions keep coming up, the 3D vision articles collect the rest of the series, and our team is reachable through the contact page for help matching a camera to a pose-estimation task.