Bounding box offset on new images: the double letterbox fix

You trained a detector on images you letterboxed to 640×640 by hand, and the boxes were tight on those images. Feed it a phone photo at its native 1280×720 and the detections still fire, but every box sits a band of pixels off, so the distances you compute between items come out wrong. The cause is double letterboxing: your manual pad plus the letterbox the detector applies internally before every forward pass. What follows is the one-image test that proves it, the scale-and-pad arithmetic that undoes it, and the decision between retraining on raw images and rescaling in post-processing.

The symptom: right objects, wrong coordinates

Detections are not the problem. The model finds every item on the new photo at any size, with healthy confidence. The coordinates are the problem: each box lands shifted toward one edge, and the further an object sits from the image origin, the further its box drifts.

Two signatures identify the fault. On square 640×640 inputs the boxes are pixel-tight, because a square input letterboxes with scale 1 and no pad. On anything wider or taller, the error has two parts: a constant shift equal to the pad, and a proportional part equal to the scale error. You notice it the moment the boxes become measurements — the results.plot() overlay shows the drift, and a centre-to-centre distance between two items no longer matches the value the same pair gave on the training canvas.

Why it happens: two letterboxes, one coordinate frame

Letterboxing resizes an image with a single uniform scale so it fits a square canvas, then fills the remainder with padding. It preserves aspect ratio, which is why detection pipelines use it: a 1280×720 photo becomes a 640×360 image with bars above and below.

A YOLOv8 pipeline letterboxes every image itself, in training and again at inference, then maps the network's boxes back to the frame of the image you passed in. That return map rests on one assumption: the model learned its coordinates on raw images, so its frame is the raw photo's frame.

Your hand-letterboxed training set broke that assumption. The model learned coordinates on a 640×640 canvas with your pad baked in. When your script pasted the resized image in a different place than the internal letterbox puts it — top-left paste against centred placement is the common pair — the two conventions differ by exactly the pad, and every coordinate the model produces inherits that difference.

The pad terms above assume centred placement. Match them to the script that built your training set: a top-left paste puts all the pad on the right and the bottom, and the correction changes with it.

The one-image test that proves it

1. Take one new photo at native size, plus a copy letterboxed to 640×640 with your training script.
2. Run inference on both. Tight boxes on the letterboxed copy and drifting boxes on the raw copy mean the model is healthy and the coordinate frame is the fault.
3. Crop the raw photo to a square and infer again. Scale is 1 and both pads are 0, so a letterbox error cannot exist; if the square crop is tight and the full photo is not, the diagnosis is confirmed.
4. Compare the offset on two objects at different positions. A constant part plus a part that grows with distance from the origin is scale and pad; a random scatter is annotation quality.

Fix A: retrain on raw images

Retraining is the clean fix, and it is the one to choose whenever a training window exists. The pipeline letterboxes internally, batch by batch, so the model never depends on your pad convention, and inference coordinates come back in the raw photo's frame with no post-processing to maintain.

You do not need to re-annotate. Your labels live on the 640×640 canvas; undo the manual letterbox on each coordinate with the same arithmetic as Fix B, then write the labels in your trainer's format. YOLO label files store normalised centre coordinates between 0 and 1, so one more division by the raw dimensions finishes the conversion:

Width and height convert the same way: divide the canvas width by scale × img_w and the canvas height by scale × img_h. If your annotations start life in spreadsheets, the indexing traps of that conversion are covered in Convert CSV bounding box annotations to YOLO without index errors.

Fix B: keep the model, undo the letterbox

When the model must stay in service today, invert the letterbox in post-processing. Run inference on an image you letterboxed yourself, because then you know the exact transform between the canvas and the photo instead of trusting a return map built for raw-image training. Three steps, and the order of the arithmetic matters:

1. Letterbox the new photo to 640×640 with the script that built the training set — same scale rule, same pad placement, same fill value.
2. Run inference on that letterboxed image. The boxes come back in the canvas frame the model was trained on.
3. Subtract the pad, then divide by the scale. Subtract first: dividing first leaves the pad error at pad / scale on the full-size photo.

Worked example, a 1280×720 phone photo on a 640×640 canvas with centred placement:

• scale = min(640 / 1280, 640 / 720) = min(0.500, 0.889) = 0.500
• resized image: 1280 × 0.500 = 640 px wide, 720 × 0.500 = 360 px tall
• pad_x = (640 - 640) / 2 = 0 px and pad_y = (640 - 360) / 2 = 140 px
• a box at (240, 220, 400, 300) on the canvas maps to x1 = (240 - 0) / 0.500 = 480 px, y1 = (220 - 140) / 0.500 = 160 px, x2 = 400 / 0.500 = 800 px, y2 = (300 - 140) / 0.500 = 320 px
• its width grows from 400 - 240 = 160 px on the canvas to 160 / 0.500 = 320 px on the photo
• two item centres 320 px apart in x on the canvas sit 320 / 0.500 = 640 px apart on the photo
• skip the pad subtraction and y1 lands at 220 / 0.500 = 440 px instead of 160 px — a 280 px error from one dropped term

The same steps in code, with the test that proves the mapping:

import numpy as np
import cv2
from ultralytics import YOLO

CANVAS = 640  # int: canvas side in px, as used for the training images

def letterbox(img):
    # uniform scale to fit CANVAS, pad the rest: the training script's rule
    h, w = img.shape[:2]                               # example: 720, 1280
    scale = min(CANVAS / w, CANVAS / h)                # 0.5
    new_w, new_h = round(w * scale), round(h * scale)  # 640, 360
    pad_x, pad_y = CANVAS - new_w, CANVAS - new_h      # 0, 280
    left, top = pad_x // 2, pad_y // 2                 # 0, 140 (centred)
    out = cv2.copyMakeBorder(cv2.resize(img, (new_w, new_h)),
                             top, pad_y - top, left, pad_x - left,
                             cv2.BORDER_CONSTANT, value=(114, 114, 114))
    return out, scale, left, top

model = YOLO('best.pt')                # trained on letterboxed 640x640 images
photo = cv2.imread('test.jpg')         # example input: 1280 x 720 px
square, scale, pad_x, pad_y = letterbox(photo)
results = model(square)[0]             # boxes in the 640x640 canvas frame

for box in results.boxes.xyxy:
    x1, y1, x2, y2 = box.cpu().numpy()
    x1, x2 = (x1 - pad_x) / scale, (x2 - pad_x) / scale
    y1, y2 = (y1 - pad_y) / scale, (y2 - pad_y) / scale  # photo-frame pixels

# test: a 720x1280 blank must yield scale 0.5 and pads (0, 140),
# and the canvas box (240, 220, 400, 300) must map to (480, 160, 800, 320)
blank = np.zeros((720, 1280, 3), dtype=np.uint8)
_, scale, pad_x, pad_y = letterbox(blank)
assert (scale, pad_x, pad_y) == (0.5, 0, 140)
assert round((240 - pad_x) / scale) == 480
assert round((220 - pad_y) / scale) == 160
assert round((400 - pad_x) / scale) == 800
assert round((300 - pad_y) / scale) == 320

From pixels to millimetres

Undoing the letterbox returns distances in photo pixels. A distance in millimetres needs one more number: the pixel scale of that photo. Put a known dimension in the scene — a calibration target, a machined edge, a fixture — and measure it in the same frame.

A 100.0 mm reference that spans 400 px gives 100.0 / 400 = 0.25 mm/px. The 640 px centre distance above is then 640 × 0.25 = 160 mm. With handheld photos the scale changes with every shooting distance, so the reference must appear in each image; a fixed camera mount turns the pixel scale into a constant you calibrate once.

Box corners sit on integer pixels. When you need better, work from centroids and set the origin deliberately — the 0.5 px bias is explained in half-pixel offset in blob centroid: pixel coordinate origin. For a complete small-part measurement chain, measuring points on a 5 mm part in under one second covers the camera, optics and timing end to end.

Keep one geometry in production

The fault class is two preprocessing paths. Close it at the source: annotate raw, train raw, let the pipeline letterbox, and keep one version-controlled preprocessing module imported by both the training code and the inference code.

Then fix the image geometry. An area scan camera outputs the same sensor format every frame, so scale and pad become constants you compute once; the camera finder filters by sensor format and mount. The LS-CMR-0166 delivers 4000 × 3000 frames over USB3 at 30 fps.

On smart cameras the detector runs onboard, and the same rule holds: annotate and train on raw frames, because the letterbox inside the device should be the only one in the chain.

Give training and inference one home. A vision controller such as the MV-VC3301H-128G66, an Intel Core i3 with 8 GB of memory and 128G SSD storage, runs the preprocessing, the training and the inference in one place, so the module cannot drift between a workstation and the line.

When the boxes are still off

Work down this list in order; each cause has its own signature.

1. EXIF orientation. Phone photos carry an orientation flag, and a decoder that ignores it hands you a landscape array for a portrait shot, with width and height swapped. Every scale and pad computed from those numbers is wrong. Apply or strip the orientation before anything else touches the pixels.
2. Drawing on a resized canvas. The coordinates are correct and the overlay is wrong: you drew on a thumbnail, or an HMI image object resizes itself at runtime. Scale the drawing to the displayed size. The display-side version of this trap is covered in HMI coordinate container resize and bounding box offset. Indexing mistakes when drawing overlays from detection coordinates have their own short guide in barcode detection coordinate indexing: prevent overlay errors.
3. Label format mix. Pixel corners pasted into a format that expects normalised centres — or the reverse — corrupts a retraining run quietly. Check the label format before the next training job.
4. Annotation drift. If the offset is random rather than constant-plus-proportional, the labels are at fault, not the geometry. Audit a sample of annotations at full resolution before touching preprocessing.

Checklist: one letterbox, owned once

• Annotate and train on raw images; the pipeline letterboxes internally, so a second letterbox can only subtract accuracy.
• Keep one version-controlled preprocessing module, imported by training and inference alike, so the two paths cannot diverge.
• Hold a regression pair — one scene as raw photo and as letterboxed canvas — and diff the boxes after every preprocessing change.
• Undo the transform in order: pad subtraction before division by the scale.
• Fix the camera geometry so scale and pad are constants, and calibrate the pixel scale against a known dimension.
• Check EXIF orientation on every image that came from a phone.

Retrain on raw images when a training window exists; that removes the offset class entirely. Run the letterbox inversion while the current model must stay in service. For a fixed-geometry station, the configurator below starts at the camera step, and the contact page reaches the applications team for a review of your coordinate chain. More guides in this vein sit on the systems topic page.