Segmentation noise from overfitting in custom dataset training

You trained a two-class fruit segmentation model on 500 images, and it passes on the bench but masks shadows, neighboring fruit and bits of the table the moment the scene changes. That is the classic single-condition dataset failure, and the fix spends the images you already have rather than a bigger model or more epochs. This guide shows how to confirm the dataset is the cause, how to re-budget 500 images across lights, backgrounds and background-only frames, and how to verify the retrained model before it returns to the line. The same steps apply whether you train on a workstation and deploy to a smart camera, or train and run everything on a vision controller.

Why does the model segment shadows and background objects?

The backdrop is the only thing that never changed in your set: with 500 frames on one white surface under one light, roughly 70-90% of the pixels in a typical shot are identical background, so the strongest pattern the model found is the table rather than the fruit.

A segmentation model splits every frame into shaped object regions against a background region, and it decides where that boundary sits from the statistics of the images you give it. Your statistics say: objects sit on pure white, and any dark, textured or green patch against the white must be an object. Shadows produce the strongest gradients on a white table, so they inherit masks. Neighboring fruit and tray edges never appeared as background either, so the model segments them as willingly as the fruit. Nothing is wrong with the network; it answered the question your images asked.

Is the noise the dataset or the small model?

The dataset is the cause: YOLOv8n and YOLO11n have more than enough capacity for two fruit classes, and an undersized model fails by missing objects, not by painting confident masks over things it was never shown, which is exactly what your pictures show.

Run two tests before touching anything.

1. Empty-scene test. Capture 10 frames of the bare white table under the training light, no fruit, and run them through your trained weights. Masks coming back on an empty table confirm the backdrop habit; keep those frames, because they become your first negatives.
2. New-condition test. Shoot 30 frames under a different light angle or on a different surface and label 10 of them. Crisp masks on the original white validation frames that collapse on the new ones is overfitting in one picture.

Both results point at the data. Keep the pretrained weights as well: the from-scratch .yaml run cannot fix this and usually lands worse, because 500 images are not enough to teach edges, textures and objectness from zero. The pretrained start already knows those and only needs to unlearn the backdrop.

How much can augmentation fix?

Very little: crop-zoom of 0-30% and exposure shifts of ±5% keep the same white backdrop in every frame, and augmentation edits pixels inside a scene without ever inventing a new lighting geometry or a new surface.

Augmentation multiplies conditions you captured; it cannot create ones you skipped. Zoom, crop, flips and small brightness shifts all keep the white table in frame, so the habit survives every augmented copy. Do raise exposure and brightness jitter well beyond the ±5% used so far — training tools commonly default to around a ±40% value shift — and add saturation jitter, but treat hue separately.

With a re-spread dataset these stronger settings add tolerance to lighting drift. Without one they only re-shuffle the same white-table frames, and the noise returns on the first new background.

How should the same 500 images be re-budgeted?

Treat your original 500 as one over-weighted condition: keep the best 150, capture 275 new part shots across the eleven remaining lighting-and-background combinations, add 75 background-only frames, and the largest condition lands at 30% of the set instead of 100%.

Retire the other 350 originals: they are repeats of a condition the model already knows too well, and repetition is what taught it the backdrop. The full matrix is 4 lights by 3 surfaces (12 cells); your training light on white occupies one, leaving 11 for the new shots. About a third of the 275 should show clutter anyway — neighbors touching, fruit partly out of frame, tray edges in view.

The 75 negatives teach the complement: white table plus shadows alone is not an object. List them with no labeled objects and spread them over all three surfaces, 25 per surface, under several lights. Why background images need no labels explains the mechanism, and the negative-samples guide covers balancing them in depth. For the size question in general, see how many images you need to train a custom YOLO dataset.

What lighting and background variety is enough?

Four lighting conditions across three backgrounds is the minimum that breaks a single-surface habit: diffuse bright, dimmed to about half, hard single-sided light that casts shadows, and one colored or polarized pass — shot on white, a dark surface, and the real conveyor.

Shadows need a hard light specifically: your failure mode includes masked shadows, so at least one cell must contain strong directional light with its shadows present and labeled as background. The production pair is not optional either — the real light and the real surface must each appear somewhere in the set, or you have optimized for everything except the line.

Exposure discipline keeps the cells honest. Lock exposure and white balance manually per condition; auto modes silently re-light every cell. Keep the brightest highlight on the fruit below about 90% of full scale — under 230 of 255 in 8-bit — so specular peaks do not clip into flat white patches that mimic the backdrop. Save each cell as a camera user set so recaptures repeat exactly; saving camera settings across power cycles shows how. For the wider discipline of stable training conditions, the lighting for AI vision training guide is the companion piece.

Which hardware captures the varied set and runs the model on the line?

Repeatable per-frame switching of lights and exposure is what the fix demands, and a smart camera with a built-in multi-color light or a vision controller with four built-in light channels delivers it, turning twelve capture conditions into saved settings instead of a day of re-rigging.

Two routes cover it. A smart camera with an integrated light holds several lighting cells inside one fixture and runs the finished model on-device; the camera finder filters by resolution, shutter and mount. Where you need separate fixtures and more compute, a vision controller drives the lights and hosts the model.

The Hikrobot MV-SC6050C-08M-XBN pairs a 5 MP color sensor with an 8 mm lens and a built-in multi-color light source, so ripeness classes that depend on color can be captured under different illumination from one device.

The MV-VC2140-128G40-1T switches 4 built-in light channels from its own outputs, takes the cameras on its LAN ports, and its N97 processor with 8 GB of memory and a 128 GB SSD hosts the retrained weights when a smart camera cannot.

How do you verify the retrained model before it runs on the line?

Hold out 20% of the new set for validation, run 30 frames from a condition that is rare in training, and accept the model when mask IoU on the fruit stays above 0.8 while background frames return fewer than one false mask in ten.

Two checks make that acceptance number trustworthy. First, scale. Worked example: the smallest boundary you must segment is a 3 mm color transition between green and orange, and you want at least 4 pixels across it, so the pixel scale must be 3 ÷ 4 = 0.75 mm/px or finer. At a 120 mm wide field of view, a 5 MP sensor of roughly 2592 × 1944 pixels delivers 2592 ÷ 120 ≈ 21.6 px/mm, which is 0.046 mm/px; the same 3 mm transition spans about 65 pixels, sixteen times the minimum.

Capture training frames close to the mm/px you will run in production: a model trained at 0.05 mm/px and deployed at 0.4 mm/px meets edges it never saw, and mask boundaries turn ragged.

Second, read per-class results. With two classes, a mean mask score can hide one class near 0.5 IoU while the other flatters the average, and Immature is the usual loser because green against shadows resembles the backdrop gradients the old set taught.

When is more data the right answer instead of re-budgeting?

Buy more data when the failure is misses rather than noise: if the re-budgeted model skips ripe fruit under one specific light, capture 50-100 more images of that failing condition and continue from the checkpoint — targeted additions beat blanket volume.

Misses mean scarcity, not habit: a condition with too few examples loses objects under exactly that light, while the ten healthy cells keep their balance. Continue training from the checkpoint rather than restarting, so the spread you built is preserved.

The single-condition dataset is not always a defect. If the camera stares into a closed, matte enclosure where the surface and the light never change and ambient is blocked, the backdrop is constant at deployment too, and training on that one condition is correct. The habit only costs rework when conditions change between the bench and the line — which is what happened here.

What do you check before hitting train again?

Run the list in order; every line guards against one of the failure modes above, and the first three decide whether the retrained model still paints the table.

• Run 10 empty-background frames through the current weights: masks there confirm the habit.
• Keep the best 150 originals; retire the other 350 repeats.
• Capture 275 new part shots spread over the 11 remaining cells, production light and surface included.
• Add 75 background-only frames with no labels, spread over surfaces and lights.
• Lock exposure and white balance per cell; keep peaks below about 230 of 255.
• Hold out 20% from every cell plus 30 production frames.
• Accept only above 0.8 per-class mask IoU and below one false mask in ten background frames.

When you are ready to spec the capture rig or the line, the configurator above walks camera, lens, lights and controller in order, and the systems topic page collects the companion training guides. Our applications team helps with the lighting and dataset plan — expert support is free with every order.