Mixed resolution images in object detection training: pad, not stretch

Machine-vision datasets rarely arrive at one size. Frames come from several cameras at different resolutions, crops exported from review software, and screenshots with whatever aspect ratio the operator had; one production-style set spans 49 × 27 up to 1920 × 1122 pixels with aspect ratios from 1:1 to 6:1. A detector that expects one fixed input refuses to train on that, and the tempting fix, stretching every image to a square, quietly degrades the model. This guide gives the standard preprocessing: one fixed canvas, an aspect-preserving scale, centered padding, and box coordinates carried through the same transform. You get the canvas-size rule, the arithmetic with real numbers, a step-by-step conversion pass, and the decision for tiny frames, panoramas and 1080p inference streams.

How do I train one detector on images of many different sizes?

Scale each image with one aspect-ratio-preserving factor until it fits a fixed square canvas, 640 × 640 by default, then pad the empty border with a constant grey and transform the boxes with the same factor and offsets.

Training wants uniform tensors because a batch stacks images into one array; mixed heights and widths abort the run with a shape error, usually a few epochs in. Padding solves that without lying about geometry: nothing is stretched, objects keep their true proportions, and the constant border tells the network where content ends. For what changes inside a network when tensor height and width change, read object detector input dimensions: height, width, and padding.

At inference the constraint relaxes: the same weights letterbox whatever frame arrives and map the boxes back. Train on one canvas, run on anything, because that asymmetry is normal and by design.

What does force-resizing to one square actually destroy?

It scales the two axes by different factors: a 1920 × 1122 frame squeezed into 640 × 640 scales x by 0.33 and y by 0.57, and an extreme panorama distorts one axis six times more than the other.

A circle becomes an ellipse and stays one, because the network sees that warp in every sample of that aspect ratio. Each aspect ratio applies its own warp, so one model is asked to learn a different geometry per ratio in the same class. Thin features go first: a vertical edge 3 pixels wide in a 6:1 panorama becomes 1 pixel when the frame is forced into a square.

Clean, proportional scaling has its own limit, which is pure size. A 40 px object in a 2560-px-wide frame lands at exactly 10 px on a 640 canvas, right on the floor. Size loss is what the next section budgets for; distortion never is.

How big should the training canvas be?

Make it 640 × 640 unless your smallest labelled object falls under 10 pixels after scaling, then move to 832 × 832, which costs 1.69 times the compute per image and buys 30 percent more pixels on an object's side.

Two numbers set the canvas: the longest side in the dataset and the smallest object you must detect. After scaling, the smallest object keeps at least 10 px on its shorter side; 8 px is the floor, and below that a label trains noise.

With a 1920 px longest side and a 40 px smallest object, canvas_min = 10 × 1920 / 40 = 480, so 640 already carries margin. Round any canvas up to a multiple of 16: four stride-2 downsampling stages divide the input by 16, and 640 / 16 = 40 keeps every feature map whole.

Decisions from the table: 1920-class sources with 40 px objects sit comfortably at 640 with 13 px. A 2560-class set needs 832 to clear the floor. A 4000 px source with 40 px objects fails even at 832, and the fix is cropping tiles at native resolution, not a 1664 canvas that costs (1664 / 640)² = 6.76 times the memory. Activation memory scales with canvas area, so a move from 640 to 832 cuts a batch of 32 to 32 / 1.69 = 19 images.

The same smallest-feature arithmetic, run forward, sizes a sensor: the camera resolution calculator and the guide to camera resolution for defect detection work this pixel rule on the camera side.

How do the box coordinates change when I pad?

Every coordinate scales by s and shifts by the pad offsets, x' = s × x + dx and y' = s × y + dy, where s = min(canvas_W / W, canvas_H / H) and dx, dy are the centered border widths.

Worked example, corner-format labels, the largest frame of the set into the default canvas:

• Source 1920 × 1122, canvas 640 × 640
• s = min(640 / 1920, 640 / 1122) = min(0.3333, 0.5704) = 0.3333
• Scaled size: 1920 × 0.3333 = 640 wide, 1122 × 0.3333 = 374 tall
• dx = (640 - 640) / 2 = 0, dy = (640 - 374) / 2 = 133
• Label (512, 280, 608, 340)
• x1' = 0.3333 × 512 + 0 = 170.7, round to 171
• y1' = 0.3333 × 280 + 133 = 93.3 + 133 = 226.3, round to 226
• x2' = 0.3333 × 608 + 0 = 202.7, round to 203
• y2' = 0.3333 × 340 + 133 = 113.3 + 133 = 246.3, round to 246

The new label (171, 226, 203, 246) is 32 px wide and 20 px tall, above the 10 px floor. A smaller landscape frame shows the upscale case: 480 × 360 gives s = min(1.333, 1.778) = 1.333, lands at 640 × 480, and dy = (640 - 480) / 2 = 80. Upscaling to 1.33× invents no detail but costs nothing at that ratio; past about 2× it is blur, and tiny frames belong to the exceptions below.

Keep coordinates as floats through augmentation and round once at the end; rounding corners independently can shave a pixel off a 20 px box, which is 5 percent of it.

How do I convert the whole dataset step by step?

Run one offline pass that reads every image and its label file, applies the transform above, writes the padded image and rescaled labels to a new folder, and saves the per-image scale and offsets beside them.

1. Audit first: script every image for width, height and aspect ratio, and every label for its smallest box per class. The longest side and the smallest object are all the canvas decision needs.
2. Pick the canvas with the 10 px floor and round up to a multiple of 16; on the example set that is 640 × 640.
3. Write one transform function: compute s, resize with bilinear interpolation, letterbox to the canvas with constant mid-grey (114, 114, 114).
4. Push every box through the same s, dx and dy, clamp the corners to the canvas, then drop and log any box under 8 px on its shorter side.
5. Write to a new folder tree, leave the originals untouched, and save s, dx and dy per image in a sidecar file.
6. Check the output the blunt way: draw the transformed boxes over 100 random padded images. A 2 px systematic offset on one border means the offset math is wrong, not the labels.
7. Split train and validation by capture session or camera, never randomly across near-duplicate frames, otherwise validation scores a memory test.
8. Freeze the same letterbox, same constants, in the inference path, with the inverse mapping ready for reporting.
9. Train, and if the smallest class recalls poorly, regenerate at 832 from the sidecar files instead of hand-editing anything.

Should training resolution match the camera's native resolution?

Match the object scale, not the pixel count: a 4000 × 3000 frame letterboxed to 640 shows objects at the pixel size the network trained on, and the boxes map back to full resolution for reporting.

Training close to deployment scale is the goal, which is why the canvas rule runs on object pixels, not frame pixels. At the line, the same letterbox runs on each live frame whatever its size, and the inverse transform returns boxes in camera coordinates, where measuring object size from a camera image takes over on the calibration side.

The stronger fix is upstream: one camera with one fixed output ends the mixed-size problem at capture. Compare area scan cameras in the camera finder, or start configuring the system around a sensor. The LS-CMR-0166 is a 12 MP USB3 area scan camera delivering 4000 × 3000 frames at 30 fps, so every training image arrives at one size. Letterboxed into a 640 canvas, that frame takes s = 0.16 and lands at 640 × 480 with 80 px borders top and bottom: one predictable geometry for the whole set.

Two more fixed-output cameras across the resolution range:

All 2 share: Gain 0 dB to 24 dB; Binning Supports 1 × 1, 2 × 2, 4 × 4; Decimation Supports 1 × 1, 2 × 2, 4 × 4; Lens mount C-mount; Certification CE, RoHS, KC. Values as printed in each product's data sheet; open a part for its drawing and full specifications.

What hardware should training and inference run on?

Train wherever a GPU lives, a workstation or a training server, and run inference at the line on a vision controller with PCIe expansion for a GPU, or on the CPU when the model is light enough.

The training pass is offline and batch-shaped, so book the biggest GPU available for the night. Deployment is the real decision: vision controllers mount at the machine, switch cameras and lights, and host the inference process. The MV-VC3301H-128G66 pairs an Intel Core i3 with 8 GB of memory, drives 4 built-in channels for lights, and carries PCIe expansion plus a VM6200Pro license.

When the model is small and the cycle allows tens of milliseconds per frame, CPU inference holds the line: object detection without a GPU sizes that path. On the fully integrated end, smart cameras run the model on the device; the LS-SMC-0017 is an x86-platform smart camera with an 8.8 MP global-shutter sensor on a C-mount, so trained weights load onto the sensor unit itself.

When is pad-to-square the wrong answer?

When the frames are tiny or extreme panoramas: a 49 × 27 frame upscaled 13 times into a 640 canvas gains no detail, and a 5:1 or wider panorama letterboxed into a square wastes four fifths of the canvas or more.

Three exceptions, each with its fix:

• Frames under about 128 px on the long side: do not upscale them alone. Group four into one canvas as a mosaic, or crop patches around the objects at native scale, so objects keep their real pixels.
• Panorama-heavy sets: use a rectangular canvas with both sides multiples of 16, 1152 × 192 for 6:1 content, where 1152 / 16 = 72 and 192 / 16 = 12. Batch tensors stay uniform; only the canvas shape changes.
• Smallest objects under 8 px even at 832: crop tiles at native resolution around object clusters and train on the tiles. A bigger canvas multiplies memory by the area ratio before it buys a single pixel of detail.

And the standing exception: if you control acquisition, fix the size at the camera. One fixed output beats every downstream preprocessing decision in this guide.

What should I check before the first training run?

Check five things: overlays drawn from the transformed labels, the smallest box size per class, the padding fraction of the batch, identical letterboxing at inference, and a validation split scored at the same canvas.

• Overlays: boxes drawn over 100 random padded images, corners on the object, with 2 px the tolerance ceiling.
• Smallest box: every class above 10 px at the canvas, and anything at 8 px logged rather than silently kept.
• Padding fraction: median padded area under 40 percent of the canvas; higher means the canvas shape fights your aspect ratios.
• Inference path: the live letterbox uses the same constants, and the inverse mapping returns camera-coordinate boxes.
• Augmentation order: flips and crops happen in native space before the letterbox, and boxes transform once, at the end.

Pick the canvas from your own audit this week, or send the resolution table and smallest-object numbers through the contact page and we will size the canvas and the camera side with you.