Image to map homography: map tracking points onto a ground plane

You have a fixed camera watching people, AGVs or totes cross a floor, and you want their positions on a plan or map view in metres, not pixels. You do not need a depth camera to get there. If the targets stay on one flat plane and the camera never moves, four surveyed points are enough: they define a ground-plane homography that converts every tracked pixel into map coordinates directly, with no lens model and no depth map. This guide walks through the calibration, the solve, a fully worked example with the arithmetic on show, the error budget, and the point at which a 3D sensor becomes the better tool.

When a ground-plane homography is the right tool

A camera that watches a flat plane sees that plane through a projective mapping called a homography: a 3x3 matrix that turns every pixel (u, v) into plan coordinates (X, Y) in metres.

The matrix carries 8 degrees of freedom: the overall scale is arbitrary, so h33 is set to 1 and 8 coefficients remain to find. Each surveyed point pair supplies 2 equations, which is why 4 pairs solve the system exactly, and why 6-10 pairs let the solver average your clicking and survey errors instead of inheriting them.

Three conditions make this the right tool: a fixed camera, everything you track touching one plane, and access to survey that plane. A camera over a load-in bay, a pallet zone or a floor intersection meets all three. It is the same mathematics as mapping points into a warped image, pointed at the world instead of a rectified picture: there the image is warped, here the point coordinates are. And if you do know the lens geometry — focal length, mounting height, tilt — you can derive the same matrix from the pose; ground-plane pose from one image using four known points works that variant.

What you need

• A fixed camera, bolted to a wall or mast, with focus and zoom locked. The method never reads intrinsics, so any machine-vision camera qualifies; the camera finder filters by resolution and interface if you are fitting a new one.
• Six to ten surveyable marks on the plane: floor-tape corners, bolt heads, expansion-joint crosses. A tape measure or laser distance meter good to 1 cm is plenty.
• A viewer that reports pixel coordinates, so you can click each mark and read its (u, v).
• A solver: OpenCV's findHomography, or roughly 20 lines of numpy around a linear least-squares solve.
• A detector whose output you refine to a contact point — feet, tyre patches, tote bottoms.

Calibrate and map, step by step

1. Fix the camera and its lens. Bolt the mount, set focus and iris, and log the pose. Rerun the calibration after any physical change; see the warning above.
2. Place and survey 6-10 points on the plane. Spread them to the edges of the tracking area, not a tidy cluster in the middle. Measure X and Y in your plan frame to about 1 cm with a tape or laser distance meter.
3. Click the matching pixels. Zoom into each mark and record its pixel position to 1-2 px. Keep a written pairing table so pixel and metre values stay matched.
4. Build the two coordinate lists. src holds the clicked pixels, dst holds the surveyed metres, both as floating-point pairs (Point2f in the classic API).

Each pair (u, v) mapped to (X, Y) contributes two linear equations in the 8 unknowns:

5. Solve for H. Four pairs give 8 equations for 8 unknowns; extra pairs are averaged as least squares. In code, with the values from the worked example below:

import numpy as np
import cv2

# Illustrative values from the worked example below.
src = np.array([[300., 900.], [1700., 900.], [1700., 1900.], [300., 1900.]])      # clicked pixels
dst = np.array([[0.000, 0.546], [6.135, 0.526], [5.641, 3.707], [0.000, 3.837]]) # surveyed metres

H, _ = cv2.findHomography(src, dst)   # 3x3 matrix
H = H / H[2, 2]                       # normalise so h33 = 1

u, v = 960.0, 1400.0                  # a tracked feet point, pixels
q = H @ np.array([u, v, 1.0])         # homogeneous plan coordinates
X, Y = q[0] / q[2], q[1] / q[2]       # divide by w -> metres
# -> X ~= 2.82 m, Y ~= 2.22 m on the plan

6. Check a point you did not calibrate on. With exactly four pairs the solve reproduces them exactly, so a zero residual there proves nothing. Hold one marker back and map it through H; it should land within two to three times your survey accuracy.
7. Map contact points, not box centres. Run each detection's floor contact point through H every frame — the two divisions in the code above. The next section shows what a head point costs you.
8. Leave a permanent check in view. Paint a floor target, map it every shift, and recalibrate when its logged error passes twice the survey tolerance.

Worked example: one camera over a load-in bay

A 1,920 x 1,080 fixed camera watches a load-in bay. Four floor-tape corners sit near the frame corners, and a laser distance meter places them in the plan frame:

The quad spans the frame corners, so nothing is collinear. Solving on these pairs returns, to the digits that matter:

A detector fires on a person at pixel (960, 1400); the feet point maps like this:

The person stands 2.82 m across and 2.22 m out from the far edge of the plan. One pixel is worth 4.03 mm at the near markers (5.641 m over 1,400 px) and 4.38 mm at the far markers (6.135 m over 1,400 px), so a 2 px click error alone is worth 8-9 mm.

Track the contact point, not the box centre

The homography is only true for points that lie on the calibrated plane. A detector box's centre rides at roughly half the object's height, and height buys parallax error through:

Here h is the camera height above the plane, z the point's height above it, and R the point's true horizontal distance from the camera's ground point. Work the numbers for a camera 6 m up, a true range of 4 m and a head point 1.7 m above the floor: the head appears to stand at 64/(6-1.7) = 24/4.3 = 5.58 m, an error of 1.58 m. Use the centre of the same box, about 0.9 m up, and the error is still 40.9/5.1 = 0.71 m.

So the rule is mechanical: track feet, tyre contact patches or the bottom edge of the box, and feed those pixels to H. Bottom-edge tracking needs no extra model and removes the dominant error in this chain.

Verify the calibration before go-live

1. Hold-back check. Keep a fifth surveyed marker out of the solve. In the example, a pallet corner taped at plan (3.20, 1.80) m sits at pixel (1041, 1272). Mapping it through H: w = 0.000031041 + 0.000101272 + 1 = 1.158, X = 3.705/1.158 = 3.198 m, Y = 2.088/1.158 = 1.802 m — within 2 mm of the tape on both axes. That 2 mm is the number to demand from your own build; with a real tape survey and real clicks, expect 10-30 mm instead of this noise-free figure.
2. Far field. Metres-per-pixel grow with range (4.03 to 4.38 mm/px across the example bay), and tilt errors scale with ground distance: the 0.3 degree bump from the warning is worth 2 cm at 4 m and 5 cm at 10 m. Quote accuracy per zone, not for the frame.
3. Vanishing-line margin. w = 0 is the horizon, where the mapping divides by zero. Across the example frame w ranges from 1.00 to 1.17 — comfortable. If your tracked region pushes w below about 0.2, errors explode; crop the region away from the horizon.
4. Flatness. Walk the plane with the parallax formula: every millimetre the floor deviates from the plane becomes millimetres of position error at mid-range. A dip, a ramp or a dock leveller mid-travel needs its own zone calibration.

The same discipline applies to depth sensors — quantifying flat-plane error at the frame edges walks the equivalent check for a 3D camera.

When the usual answer is wrong

The floor refuses to stay flat

A drain slope at the bay door, stock piled against a wall, a leveller in motion: any of these puts real floor outside the calibrated plane. Split the scene into zones, each with its own four points, or change sensor — the depth route below. Do not fold a sloped area into one solve; the residuals will look acceptable near the calibration marks and read decimetres away from them.

Wide lenses bend straight lines

A short focal length bows straight floor lines near the frame edges. The four-point solve absorbs some of that bend around the calibration points and none of it elsewhere, and tens of pixels of residual bow cost centimetres at mid-range. If expansion joints look curved in the viewer, undistort the stream with a lens calibration before solving; every common library ships the model.

The destination is a Mercator map

The solver does not care what dst means, so a Mercator view works: feed projected easting and northing in metres (EPSG:3857 for web maps) and the same four pairs produce the same kind of matrix. Subtract a local origin from the coordinates first — easting values in the millions wreck the conditioning of the linear solve — and add it back afterwards. One conversion matters: Mercator metres are not ground metres. Multiply by the cosine of your latitude; at 51.5 degrees N the factor is 0.62, so 100 projected metres is 62 m on the ground. Across one scene of tens of metres that factor barely changes, and if you need ground distances anyway, calibrate in local metres and convert once with a fixed translation-rotation-scale fitted from two known points.

The depth-camera route

If targets leave the floor, the floor will not stay flat, or the camera cannot be locked, a depth camera ends the debate. It returns a calibrated X, Y, Z for every pixel, so a head 1.7 m above the floor maps as correctly as the feet beneath it, and the transform to plan coordinates becomes one fixed rotation and translation with no survey at all. The deprojection calculation from depth map to point cloud shows how pixel plus depth becomes 3D points, and mapping image pixels to 3D points from a bounding box covers attaching that depth to detections.

For a fixed overhead view of a working zone, a ToF sensing camera such as the MV-DT01WMU captures the depth of the whole scene in one shot:

For close-in indoor zones such as a pallet drop area or a robot cell approach, a projected structured-light 3D camera such as the MV-DPS200P-02 does the same job from a fixed mount:

Where this leaves you

Fixed camera, flat plane, contact points: solve a homography from four surveyed pairs, hold one point back to prove it, and map feet — the whole chain needs no depth camera and lands within 1-3 cm over a bay-sized scene. Points above the plane, slopes or a moving camera: change the sensor, not the maths, and let a 3D camera give you Z per pixel. Both routes end in the same place — tracked objects plotted in metres on your plan.

Compare depth options in the 3D sensors range, or read the systems guides for the surrounding coordinate-transform work. If a scene refuses both routes, send the applications team your layout with a frame attached. Fitting a new fixed camera to a zone? The configurator below starts at the camera step.