YOLO letterbox padding: what happens to a 16:9 image

You train YOLO on frames from a factory camera and the numbers refuse to match: the camera delivers 1920 × 1080 or 4000 × 3000 pixels, the network wants a square 640. Below is the transformation that bridges the two — uniform scale, gray padding, stride alignment — with the arithmetic worked for 16:9 and 4:3 frames. By the end you can predict the padded size of any frame at any input size, say what the gray borders do to accuracy, and choose input size and camera resolution from the smallest defect the application must catch. The same math holds for training on a PC and for inference on an embedded vision device.

What does YOLO actually do to a 16:9 frame?

YOLO never crops a 16:9 frame and never stretches it to a square. It letterboxes: the 1920 pixel long side of a 1080p image scales to the input size, and the leftover rows fill with neutral gray, so every object survives. The transform runs in three steps:

1. Scale. One factor for both axes: the input size divided by the frame's long side. The long side lands exactly on the target; the short side follows proportionally.
2. Pad. The gap left on the short side is filled with neutral gray, split evenly between the two edges.
3. Align. Both final dimensions are brought up to a multiple of 32, the network's largest stride.

At inference the short-side target is the scaled size rounded up to the next multiple of 32, which keeps inputs small and fast. In square training the target is the full input size on both axes, so the padding is much larger. No center crop exists anywhere on this path and no step distorts geometry, so every object in the original frame reaches the network intact. For the wider picture on rectangular inputs in deep learning, see letterbox or rect input handling; the rest of this page stays on the letterbox math itself.

Where does the padding go on a 1920 × 1080 image?

At a 640 input a 1920 × 1080 frame scales to 640 × 360. Square training pads 280 pixels of gray in total — 140 above and 140 below — while inference pads only the 24 pixels needed to reach the next multiple of 32. Run the numbers once and they hold for every frame that camera geometry produces:

• scale = 640 / 1920 = 0.333
• scaled height = 1080 × 0.333 = 360 px
• square training: padding = 640 - 360 = 280 px, split 140 top and 140 bottom
• inference: 360 rounds up to 384, so padding = 24 px, split 12 and 12

The inference case shows how little overhead 16:9 carries at deployment: only 24 of 384 rows, about 6 percent, are gray. Square training is the expensive case, and it is the default.

The 4:3 rows explain why factory sensors are convenient: 480 is exactly 15 × 32, so a 4000 × 3000 frame at a 640 input needs no padding at inference. A 1080p frame is less tidy because 360 sits between the multiples 352 and 384, so it rounds up to 384.

Why must the input be a multiple of 32?

The network downsamples its input by a total factor of 32, so both input dimensions must be multiples of 32 — 512, 640, 960, 1280 all work — and the detection heads read grids at strides 8, 16 and 32. Each convolution stage halves the feature maps; after five halvings the deepest map is 1/32 of the input. A 640 input therefore gives grids of 80 × 80, 40 × 40 and 20 × 20 cells.

A value that is not a multiple gets corrected before the network runs: set 650 and the pipeline rounds it up to 672, quietly adding the difference as padding. Every true multiple of 32 is legal, and the choice among them is the speed-versus-small-object trade settled two sections down. Input dimensions and padding covers how these numbers reach the detector's configuration.

Does the gray padding cost you detections?

The gray padding costs almost nothing: training sees the same borders inference sees, boxes ride the identical scale and offset, and no object is cropped — the real cost is scale, because the 1080 pixel height of a 16:9 frame drops to 360 rows at a 640 input. Training letterboxes exactly the way inference does, so the network learns gray borders as background. The common implementation fills them with a midtone gray of 114 per channel; why the padding value is 114 explains that choice. And because letterbox scales rather than crops, the padding never contains part of an object — content stops where the gray begins.

The cost lands on the short side of any wide frame. At a 640 input a defect spanning the height of a 16:9 frame is judged on a third of its original pixels; a 4:3 frame drops harder, from 3000 rows of height to 480.

What input size does your smallest defect need?

Keep the smallest object at least 6 pixels across at the network input. A 3 mm defect seen by a 4000 pixel sensor over a 600 mm field of view spans 20 sensor pixels, which shrinks to 3.2 pixels at a 640 input — too small to rely on. The floor has a structural reason: the finest detection grid works at stride 8, so an object of only 2 to 3 input pixels falls inside a single 8 × 8 cell and the head cannot separate it from background texture. Six pixels gives the head a center, an extent and a few texture edges; below 3, recall falls off fast.

Worked example — a 3 mm defect, a 600 × 450 mm field of view, a 4000 × 3000 sensor:

• mm per pixel = 600 / 4000 = 0.15
• px on sensor = 3 / 0.15 = 20
• at a 640 input: 20 × 640 / 4000 = 3.2 px — below the floor
• at a 1280 input: 20 × 1280 / 4000 = 6.4 px — passes

Three ways out of 3.2, each with its own cost:

1. Run 1280 and accept roughly three to four times the inference time of 640.
2. Halve the field of view to 300 × 225 mm: mm per pixel drops to 0.075, the defect covers 40 sensor pixels and 6.4 px at a 640 input — free at run time, but one shot now covers a quarter of the tray area, so four shots replace one.
3. Move to a sensor with more pixels at the same field of view, which the next section covers.

Decision for this case: a line that pauses for imaging should narrow the field of view and stay at 640; a one-shot-per-tray line running at cycle rate should run 1280. The 4000 × 3000 sensor in the example is the LS-CMR-0166, a 12 MP USB3 model.

Which camera sensor keeps the defect above the floor?

Count sensor pixels rather than megapixels. A 12 MP camera with 4000 × 3000 output over a 600 mm field of view resolves 0.15 mm per pixel, and that single number decides whether the defect stays above the 6 pixel floor after the letterbox scale. Millimetres per pixel follows from two inputs alone: field of view and sensor pixel count. A narrower field or a higher pixel count lowers it, and the defect's sensor-pixel count rises in direct proportion. The calculator below runs that relation and opens the camera finder filtered to sensors with the pixel count the arithmetic asks for.

When the mm-per-pixel budget needs more sensor pixels at the same field of view, step up the resolution:

The MV-CS200-10UM pairs a 20 MP rolling-shutter sensor with a USB3 interface. The MV-CH120-11UC stays at 12 MP and adds a global shutter — the shutter type to choose when parts move during exposure, because a rolling shutter smears moving parts. Both live in the area scan cameras category; for inference on the device itself, smart cameras run the same stride arithmetic on board. Filter the camera finder by resolution and shutter, or start the configurator at the camera step and work forward to lens and light.

When is letterbox the wrong tool?

Letterbox is the wrong tool for extreme aspect ratios. A square 640 × 640 training input on 16:9 content carries only about 56 percent real image, so very wide or very tall frames deserve rect mode, tiling or a custom aspect-preserving pipeline instead. The waste has a number: 640 × 360 of content inside a 640 × 640 training input is 230,400 of 409,600 pixels, so roughly 44 percent of the training compute processes gray. Three ways to spend less:

1. Rect mode. Batches share one non-square shape close to the data's aspect, cutting most of the padding; rect versus square training covers the trade.
2. Tile. Cut a very wide or very tall frame into near-square tiles; each tile letterboxes with almost no padding and small objects keep their pixels.
3. Accept it at inference. Deployment pads only to the next multiple of 32 — 24 rows on 1080p — so the waste argument bites training compute, not run time.

What should you check before the first training run?

Grab one raw frame from the camera and run five checks on it — scale factor, padded dimensions, smallest-object pixels, border value and inference time — before you label a single image, because every one of them is arithmetic you can do in a minute.

• Compute scale = input ÷ long side for one real frame and write down the padded size; both dimensions must be multiples of 32.
• Run the two-line formula on the smallest defect; require 6 px or more at the chosen input.
• Save one preprocessed image and read the border pixels: a value near 114 gray confirms the letterbox is active and identical across training and deployment.
• Time 20 inferences at 640 and at 1280 on the target hardware before committing to 1280 — the 3-4× estimate is a guess until your own hardware answers.
• Keep annotations in original pixel coordinates and let the training pipeline apply the letterbox transform.
• After the first run, score recall on the smallest class on its own; aggregate metrics hide a 3 px object problem.

When the checks pass, the remaining question is hardware: which sensor delivers the millimetres per pixel the arithmetic demands. Bring the field of view and the smallest defect to the quote request and we will match camera, lens and light to them, or browse the cameras topic page for the training-side guides that build on this one.