Most machine-vision frames are not square. A 1080p camera hands you 1920 × 1080, a 12 MP industrial sensor something near 4:3, and a line-scan frame is a strip thousands of pixels wide and a few hundred tall. Detectors such as YOLOv8 default to a square input near 640 × 640, so the question of training on the camera's native rectangle comes up on every project. The answer is yes, and the choice between letterboxing into the square and training rectangular comes down to three numbers: fill fraction, padded rows and pixels per millimetre. By the end you can read your sensor resolution and your defect spec off the same page and commit to an input size.
The decision in one line
Train letterboxed squares by default. A 1920 × 1080 frame enters a 640 × 640 input as a 640 × 360 image with 140 rows of padding top and bottom, and training runs on that without complaint. Switch to rectangular training only when the frame is wider than about 2:1, when a line-scan strip makes a square input absurd, or when the smallest feature you must catch falls below about 8 pixels at the square size. Inference needs no decision at all: predict and val already run rectangular inference on their own, letterboxing each frame to the nearest stride-compatible rectangle instead of a square.
For how detectors treat input height, width and padding across several input choices, read object detector input dimensions, height, width and padding alongside this.
What letterboxing does to the frame
Letterboxing scales the whole frame by one uniform factor and fills the leftover area with a constant grey, so circles stay circles and aspect ratios survive into the network. The arithmetic:
For 1920 × 1080 at S = 640: scale = 640 / 1920 = 0.333, height = 1080 × 0.333 = 360, and padded rows = (640 - 360) / 2 = 140 on top and 140 on the bottom. The frame occupies 640 × 360 = 230,400 of the tensor's 409,600 pixels, a fill fraction of 0.5625, which is exactly 1080 / 1920.
| Frame | Aspect | Fills at S = 640 | Padded rows | Fill fraction |
|---|---|---|---|---|
| 640 × 640 | 1:1 | 640 × 640 | 0 | 1.00 |
| 1600 × 1200 | 4:3 | 640 × 480 | 2 × 80 | 0.75 |
| 1920 × 1080 | 16:9 | 640 × 360 | 2 × 140 | 0.5625 |
| 1920 × 960 | 2:1 | 640 × 320 | 2 × 160 | 0.50 |
The fill fraction follows the aspect ratio alone, not S: a 16:9 frame wastes 43.75 percent of the pixels at 640 and at 1920 alike. Padding costs compute, never accuracy, because no object can land in the pad.
What rect=True changes in training
Default training runs on square mosaics, collages assembled from several frames, and every batch is the same square tensor. That uniformity is what keeps batching, augmentation and memory stable. With rect=True the loader sorts frames by aspect ratio and builds rectangular batches instead, each side rounded up to a multiple of the 32-pixel stride. Batch shapes then differ from batch to batch, feature maps vary in size, and per-batch memory stops being predictable, which is why the setting is marked supported but not recommended for general training. You reach it with rect=True on the train command, or by passing a two-value imgsz such as imgsz=1920,1080.
The payoff is the padding you stop paying for. At the same long side, a 1920 × 1088 tensor holds 2,088,960 pixels against 3,686,400 for a 1920 × 1920 square: 43 percent fewer activations for identical image content. Predict and val take that discount automatically; in training you trade it for the batching simplicity above.
The number that picks your input size
Input size is not a free choice; the smallest feature sets it:
GSD is ground sample distance, millimetres per input pixel. A feature needs about 10 pixels across for a detector to find and localize it reliably; below about 8, misses climb fast. The downsample factor is camera width divided by input width, so 1920 to 640 is a factor of 3 and each input pixel summarizes 9 camera pixels.
The camera resolution calculator runs the same arithmetic from the camera end and gives you the pixel count a feature size demands.
Worked example: 240 mm field of view, 4 mm defect
Take a 1920 × 1080 camera framing a 240 mm × 135 mm field of view, so the camera GSD is 240 / 1920 = 0.125 mm per pixel. The spec: catch every flaw of 4 mm and larger.
At a 640 letterbox the input GSD is 240 / 640 = 0.375 mm per pixel, so a 4 mm flaw spans 4 / 0.375 = 10.7 input pixels, above the 10-pixel floor. The frame enters as 640 × 360 with 140 padded rows. Decision: train at 640.
Tighten the spec to 2 mm and 640 fails, because 2 / 0.375 = 5.3 pixels. Two fixes. A 1280 letterbox gives 240 / 1280 = 0.1875 mm per pixel and 2 / 0.1875 = 10.7 pixels, at four times the pixels of 640, so quarter the batch on the same GPU. A rectangular 1920 × 1088 input restores the camera's own 0.125 mm per pixel and 16 pixels for the flaw. Decision: for 2 mm on this camera, train at 1280 letterbox with a quarter batch; go 1920 × 1088 rectangular only if the frame is also wider than 2:1 or you need full camera resolution in one tensor.
Your camera sets the rectangle
Frames arrive from area scan cameras at their native aspect ratio, and that is where you should capture: crop or letterbox in software, where it is reversible, never in the camera. Catalog cameras span the aspect spread:
- Sensor format
- 1/1.7"
- Interface
- USB3
- Sensor format
- 1"
- Lens mount
- C-mount
The LS-CMR-0166 outputs 4000 × 3000 frames over USB3. That 4:3 frame fills 640 × 480 of a square input with 80 padded rows, 25 percent waste. The MV-CS200-10UM is a 20 MP camera on a 1-inch sensor. A 3:2 frame from that sensor class fills 640 × 427 and wastes a third of the tensor. Wider sensors pay more padding, which is the whole content of the 2:1 threshold in the decision.
Where the letterbox lives in your system
In a PC-side pipeline the frames reach the machine over GigE or USB3 through network cards and frame grabbers. A dual-port PoE card such as the MV-GE2002P carries two GigE camera links into the training or inference machine.
- Interface
- GigE
- Ports
- 2 ports
Do the letterbox in your own preprocessing and run the identical transform in training and inference. When throughput matters, move the resize onto the GPU: feeding GPU tensors to a vision model without a CPU round trip shows the path. If the camera ships 10-bit data, unpack it to 16-bit planes first and resize after, and confirm what the camera actually delivers with camera pixel format not changing.
Smart cameras settle the question before it reaches you. A 6 MP smart camera such as the MV-SC5060XM-08M-WBN runs detection on the device behind an 8 mm lens, with preprocessing matched to its own sensor.
- Interface
- GigE
- Resolution
- 6-10 MP
- Sensor format
- 1/2.4"
- Shutter
- Rolling
Four options, one verdict
| Method | 1920 × 1080 becomes | Geometry | Verdict |
|---|---|---|---|
| Letterbox to square | 640 × 360 in a 640 × 640 pad | preserved | the default |
| Rectangular | 1920 × 1088 | preserved | wider than 2:1, line scan, memory-bound long side |
| Stretch | 640 × 640 | distorted | never |
| Centre crop | 1080 × 1080 taken from the frame | preserved, 56 percent of the frame kept | only for fixtured, centred parts |
The crop row keeps 1080 × 1080 of 1920 × 1080, which is 1,166,400 of 2,073,600 pixels, 56 percent of the frame. It is legitimate when a fixture guarantees the part sits inside that square; otherwise you are gambling with part placement.
Checks before you train
- Capture at native aspect ratio, with any in-camera crop or anamorphic scaling turned off.
- Run one letterbox implementation, unchanged, in training and in inference.
- Keep input sides multiples of 32, the network stride.
- Budget memory by pixels: raising S from 640 to 1280 quadruples them.
- Validate on native frames; val already letterboxes to rectangles, so its numbers reflect deployment.
- Unpack 10-bit data before resizing, and lock the pixel format you trained on.
When the square answer is wrong
Line scan kills it outright. A 16384 × 200 strip letterboxed to 640 gives scale = 640 / 16384 = 0.039 and a 200 × 0.039 = 7.8-pixel-tall image, with nothing detectable in it. Cut the strip into overlapping square tiles, 2048 × 2048 windows with 10 to 20 percent overlap, or train a rectangular input sized to the strip, such as 2048 × 224.
Long, thin objects are fine: a pencil lying across a 16:9 frame still fills the wide direction, and letterboxing preserves it. The failure mode is small features after downsampling, which the 10-pixel rule catches before you train.
One more trap is re-training after a sensor change. Letterboxed models survive a 16:9 to 4:3 swap because the pad absorbs the difference; a stretched model has the old aspect baked into its weights and degrades silently.
The decision, case by case
- 16:9 or 4:3 frame, smallest feature at 10 pixels or more at 640: letterbox at 640.
- 16:9 or 4:3 frame, smallest feature at 5 to 10 pixels at 640: letterbox at 1280, batch divided by four.
- Frame wider than 2:1, or a line-scan strip: rectangular input with rect=True, sides at multiples of 32.
- Fixture-centred parts: a centre crop is defensible and wastes nothing.
- Inference: set nothing; predict and val run rectangular on their own.
- Stretching: never.
Next step: size the camera for your defect spec in the camera finder, then walk the configurator from the camera step. More integration pieces are on the systems topic page.
Questions
Can YOLOv8 train with imgsz=1920,1080 directly?
Predict and val accept a two-value imgsz and run rectangular inference automatically. Training on a two-value imgsz needs rect=True, which swaps square mosaic batches for aspect-ratio-sorted rectangular batches. It is supported but marked not recommended for general training, because batch shapes, feature-map sizes and memory then vary between batches. For a 16:9 frame, letterboxed square training at 640 or 1280 is the supported default.
How much of a 640 × 640 tensor does a 16:9 frame waste?
The image fills 640 × 360 and the remaining 280 rows are padding, so 43.75 percent of the tensor carries no image. A 4:3 frame wastes 25 percent, a 2:1 frame 50 percent. Padding costs compute, not accuracy, and rectangular inference at 1920 × 1088 needs 2.09 MP against 3.69 MP for a 1920 square, about 43 percent less.
Why not stretch the frame to fill the square input?
Stretching resizes the axes by different factors: 1920 × 1080 becomes 640 × 640 by scaling x by 0.333 and y by 0.593. Circles turn into ellipses, and defects defined by shape or aspect ratio shift with them. The camera aspect also gets baked into the weights, so a sensor change at upgrade time degrades the model silently. Letterbox instead.
When is rectangular training the right call?
Three cases: frames wider than about 2:1, where half a square tensor would be padding; line-scan strips, which are useless letterboxed, since a 16384 × 200 strip shrinks to 640 × 8; and memory-bound long sides, where 1920 × 1088 activations fit and 1920 × 1920 does not. Otherwise train letterboxed squares and let inference run rectangular on its own.
Do smart cameras need the same aspect-ratio handling?
No. A vision sensor runs its detection model on-device, with preprocessing already matched to its own sensor, so you feed it native frames and the aspect question never reaches you. The letterbox-versus-rect decision applies when you train your own model on frames from area scan cameras flowing into a PC.
What input size should a 1920 × 1080 frame train at?
640 by default: the 3× downsample keeps a 4 mm feature at 10.7 input pixels on a 240 mm field of view, above the 10-pixel floor for reliable detection. Move to 1280 when the smallest feature falls below about 8 pixels at 640, and divide the batch by four because pixels quadruple. Keep both sides multiples of 32.