Letterbox padding value 114: why neutral gray beats black borders

A trained detector keeps missing parts near the edges of your camera frames, or you open the preprocessing source and find the constant 114 — gray, not black — where the letterbox pads the image. This page explains where that fill value comes from, what it becomes after normalization, when a black border really does hurt, and how much padding your sensor's aspect ratio adds. You will leave with the arithmetic to map boxes back to camera pixels and a five-minute border dump that proves your deployed pipeline pads exactly the way training did. It is written for engineers wiring machine-vision cameras into detection models on a PC or on the device.

What does letterbox padding do to a camera image?

Letterbox scales your frame by one factor so the longest side fits the network input, then fills the leftover rows or columns with a constant color until the image reaches a fixed shape such as 640 × 640.

A 4000 × 3000 camera frame has a 4:3 shape and a 640 × 640 input is square, so letterbox reconciles the two without stretching: one scale factor for both axes, and the short axis padded up. The network always consumes the same shape. During training, every image it saw carried the same constant fill in the border, so the border is learned background — whatever value you deploy there is read as background too. That is why the fill value is an engineering decision rather than cosmetic detail, and why a mismatch shows up only in the numbers, never as an error. For how the input shape itself gets chosen, see object detector input dimensions, height, width and padding.

Why is the letterbox padding value 114 and not black?

Gray 114 is the ImageNet dataset mean pixel value averaged over the red, green and blue channels, so a border at 114 reads as ordinary background rather than a strong dark edge.

The published ImageNet means are about 0.485, 0.456 and 0.406 for red, green and blue on the 0-to-1 scale. Average the three and you have the neutral gray of the training corpus as a whole:

A byte cannot store 114.5, so the constant lands on 114, and the YOLOv8 letterbox applies it as value=(114, 114, 114) through cv2.copyMakeBorder with cv2.BORDER_CONSTANT. Black is not the mean of any training corpus; it is the minimum of the pixel range. The number is an inherited convention, and it works because training and inference share it.

What happens to a 114 border after normalization?

The divide-by-255 normalization maps the 8-bit image to [0, 1], so a 114 border becomes 0.447, almost the mean of natural image content, while black becomes 0.0, the extreme low end.

These detection pipelines apply exactly one preprocessing step after the letterbox: divide every pixel by 255. No mean subtraction, no per-channel standard deviation. The border arithmetic is 114 / 255 = 0.447, against 0 / 255 = 0.0 for black and 1.0 for white, so gray padding sits near the center of the input range and black sits at its edge.

Convolution layers respond to local differences, so the step between border and content is what counts. A machined aluminum part at gray level 160 enters the network at 160 / 255 = 0.627. Against a 114 border the step is 0.627 − 0.447 = 0.180; against black it is 0.627, about 3.5 times stronger, at a position where the model has only ever seen neutral background.

What goes wrong when inference pads with black?

Black padding hands the network a step from any mid-tone part to 0.0 at the frame edge, an edge stronger than anything the model saw at that position during training.

The model learned its border statistics from gray 114. Swap the fill to black and every border row shifts by 114 gray levels, so scores and boxes near the frame edge drift: parts touching the edge get missed, and some models answer with detections hugging the border. The failure is quiet — no exception, just worse numbers.

Downstream classical stages inherit the problem. A constant black band beside live content creates a variance spike at the seam that a uniform-region method such as local variance filtering for defect detection will flag, and thresholds calibrated on gray borders misfire on black ones.

Why do some pipelines pad with 127 instead of 114?

The fill value follows the normalization: a pipeline that scales pixels to [−1, 1] needs a border at the middle of the range, and 127 lands there at −0.004.

Apply the scaling v′ = 2 × (v / 255) − 1 to each candidate. 127 gives 2 × 0.498 − 1 = −0.004, 128 gives +0.004, and exact zero sits at 127.5, which no 8-bit pixel can hold. The same formula sends 114 to −0.106, close to neutral but not the midpoint. Compute the fill from your own normalization instead of copying a constant:

• Divide by 255 only: the neutral value is the dataset mean, about 114 for ImageNet-era training.
• Rescale to [−1, 1]: the neutral value is the middle of the range, so pad with 127 or 128.
• Subtract a per-channel mean and standard deviation: pad each channel with its own dataset mean.

Older tutorials carry 127 for the second case and 114 for the first; each is correct inside its own pipeline and wrong outside it.

How much padding does your camera actually add?

Sensor aspect ratio decides it: a 4000 × 3000 frame letterboxed into 640 × 640 leaves two 80-pixel bands and 25 percent of the input is padding, a 1920 × 1080 frame leaves 140-pixel bands and 44 percent.

The letterbox scale factor is the smaller of the two axis ratios, and the leftover axis is split evenly between the two sides:

Worked example, a 4:3 frame into a square input:

• r = min(640 / 4000, 640 / 3000) = 0.16
• scaled size: 4000 × 0.16 = 640 px wide, 3000 × 0.16 = 480 px high
• vertical pad: 640 − 480 = 160 px, split 80 px top and 80 px bottom
• padding area: 640 × 160 = 102,400 px of the 409,600 px canvas, or 25%

The same input with a 16:9 frame: r = 640 / 1920 = 0.333, scaled size 640 × 360 px, pad 280 px split 140 and 140, padding area 640 × 280 = 179,200 px = 44% of the canvas.

Shares use the exact scale factors, then round. Many area scan cameras output 4:3 or 5:4, which keeps the bands near 50 to 90 px at this input size; a 16:9 output spends nearly half the network canvas on padding. The LS-CMR-0166 captures 4000 × 3000 over USB3.

How do you map detections back through the padding?

Subtract the padding first, then divide by the scale: a corner at (200, 320) in the 640 × 640 letterbox of a 4000 × 3000 frame maps back to (1250, 1500) in true camera pixels.

With r = 0.16 and an 80-pixel top band from the worked example above:

• x_cam = x_lb / r = 200 / 0.16 = 1250 px
• y_cam = (y_lb − pad_top) / r = (320 − 80) / 0.16 = 240 / 0.16 = 1500 px

Run the forward direction as the test: 1250 × 0.16 = 200 and 1500 × 0.16 + 80 = 320, both exact. Box sizes divide by r the same way, and integer rounding leaves about ±1 px of slack. Skip the padding subtraction and every box lands 80 / 0.16 = 500 px too low. When boxes still shift on new images, work through the letterbox fix for bounding-box offsets on new images, and mind the coordinate origin: the half-pixel offset in blob centroids applies to network boxes just as it does to classical detections.

How do you keep training and deployment padding identical?

Dump one preprocessed frame from the training pipeline and one from the deployed camera path and compare the border rows: both must read 114 on every channel, with the same band widths and scale factor.

The reference letterbox applies the fill with cv2.copyMakeBorder and cv2.BORDER_CONSTANT, value=(114, 114, 114), so that is exactly what a healthy dump shows:

# frame: uint8, 3000 x 4000 x 3, straight from the camera
# lx: uint8, 640 x 640 x 3, the letterboxed image your pipeline feeds the model
lx = letterbox(frame, (640, 640), fill=(114, 114, 114))
print(lx[0, 0], lx[0, 639], lx[639, 0], lx[639, 639])  # expect [114 114 114]

letterbox stands for your own preprocessing function, and the printout at the four corners is the test. Run the full check once per deployment:

• Training side: save one letterboxed training batch as an image and read its border rows; they must show 114 on every channel.
• Deployment side: run the corner dump on a live frame; the values and the band widths must match the training dump.
• Acquisition: feed the letterbox the raw camera frame; any resize or pad in the acquisition layer adds bands twice and shifts every box.
• Channels: confirm channel count and order end to end, because a camera image encoding that falls back to Mono8 changes the frame before padding even runs.
• Regions: keep inspection regions off the gray bands; bind each ROI to the image the detector consumed, as in ROI image source binding.

On a PC-based line your own code owns the letterbox, so the dump runs in your process. When inspection runs on the device, the fill value lives in the device configuration instead — smart cameras and vision sensors such as the MV-SC3050XC-16M-ATN-H and MV-SC5060XM-08M-WBN run the whole inspection on the device, with the preprocessing configured alongside the detection tooling.

If you are choosing hardware for a detection cell, start in the camera finder or at the camera step of the system configurator; for a question about a specific line, the contact page reaches the applications team. More systems-level checks sit on the systems topic page.