Validate a vision model without a real test set

You trained an inspection model on the frames you collected, the quality department asks what accuracy it will hold on the line, and there is no independent test set — every labelled image is already in the training pool. Quoting the training accuracy is not an answer: the model has seen each of those frames. One random train/validation split is little better, because the entire estimate then rides on one small slice of data and moves with the split. K-fold cross-validation fixes this with the data you already have. Split the collection into k folds, train k models, and let every frame be scored exactly once by a model that never saw it. Below: the arithmetic, a worked example, where the estimate holds, where it breaks, and the leaks that inflate it.

Scoring a model on its own training frames proves nothing

A model evaluated on its training frames reports memorization, not skill. With enough capacity it can reach 100% there and still miss defects under the line's lighting. That number says the recipe converged; it says nothing about deployment.

The usual fix, holding out 20% as a test set, fails when the collection is small or when every labelled frame is needed for training. With 400 frames, a 20% holdout leaves 80 test frames, and a single mistake moves the accuracy estimate by 1/80 = 1.25 percentage points. Change the random seed and the estimate can swing several points either way. A decision the size of a camera order should not rest on a number that volatile — and once you tune the model against that one split, it quietly becomes training data too.

K-fold cross-validation uses every frame for both jobs, without letting any frame do both at once.

What k-fold cross-validation does

Pick k — five is the working default. Split the labelled collection into k folds of equal size, keeping the good/defect ratio in every fold (stratified splitting). Train a fresh model on k − 1 folds and score the held-out fold. Repeat until each fold has been held out once. Every frame is then scored exactly once, always by a model that never trained on it, and the pooled predictions form one confusion matrix over the whole collection.

The deployed model is not one of the k fold models. Once the estimate is in, retrain on all N frames with the frozen recipe and ship that one; the fold models exist only to produce the numbers.

How many folds, and what five runs cost

Each fold costs one full training run, so k-fold multiplies training time by k. The return is bias: with k folds, every estimate model trains on (k − 1)/k of the frames, so 5 folds train on 80%, 3 folds on 67%, 10 folds on 90%. More folds move each estimate closer to the final model's data budget; fewer folds save compute but leave the estimate more pessimistic.

Commit to this: 5 folds for collections up to a couple of thousand frames, 3 when a single run takes hours, 10 only when frames are scarce and a run takes minutes. The number of scored predictions stays at N whatever you pick — k changes the bias and the compute bill, not the sample size.

Stratify every fold. With 40 defect frames and 5 folds, each fold must hold 8; an unstratified split can hand one fold 3 and another 14, and per-fold escape rates become noise.

When k full runs stretch a training evening into a training week, batch jobs belong on server-class hardware rather than a desktop. The card above is an AI training server for exactly that stage: it holds the dataset, runs the folds while you sleep, and leaves the runtime device at the line out of it.

A worked example: 5 folds over 400 labelled frames

The inputs, and where each comes from:

• Collection: 400 labelled frames of one part number and one defect (a missing seal) — 360 good, 40 defective. A typical mid-size collection for a single-defect class; the method transfers, the counts change with your dataset.
• k = 5, stratified (the default from the previous section).
• Budgets for this case: false rejects at most 2% of good parts, escapes at most 5% of defective parts.

The rule in words: hold out each of the five folds in turn, train on the remaining four, score the held-out fold, then pool all five runs into one confusion matrix.

The arithmetic:

• Fold size: 400 / 5 = 80 frames per fold → 72 good + 8 defective in each.
• Training set per run: 400 − 80 = 320 frames (288 good + 32 defective).
• Predictions pooled: 5 × 80 = 400, each frame scored once.
• Run outcome (an illustrative result): 4 good frames rejected, 36 of 40 defects caught.
• False reject rate: 4 / 360 = 0.011 → 1.1%.
• Escape rate: 4 / 40 = 0.10 → 10%.
• Resolution: one defect frame moves the escape estimate by 1/40 = 2.5 points; one good frame moves false rejects by 1/360 ≈ 0.28 points.

The result: an estimated 1.1% false rejects and 10% escapes for this recipe on this collection. The decision: false rejects pass the 2% budget with margin; escapes fail the 5% budget at double — do not ship this recipe. Find the four escaped frames. If they cluster in one fold, that fold's capture session (a lighting change, a new lot, a different handler) is under-represented; collect frames of that condition, retrain, and re-run all five folds. Before ordering hardware, read the estimate as a floor: each fold model trained on 320 frames while the shipped model trains on all 400, so the final model should score slightly better than this on the collection — and the collection still says nothing about the line.

Where the estimate holds and where it breaks

Within the population the frames came from — same part family, same camera and lens, same light, same presentation — the pooled numbers are an unbiased estimate, because no scored frame trained the model that scored it.

The estimate is silent about anything the collection does not contain. A lighting change in intensity or spectrum voids it, which is why a machine vision lighting stability test before deployment stays on the checklist. Ambient light bleeding past the shield shifts contrast in ways the lab never showed; if the station sits near windows or shop-floor lights, plan for ambient light variation in pick-and-place vision. Synthetic frames in the folds measure the recipe against the synthetic distribution; whether that matches real parts is exactly what the synthetic training data question examines. And features that vanish at the line — a new reflection, a handled part, a shaken fixture — are the target of pattern-matcher validation when features stay hidden.

Cross-validation measures the model. The station still owes you a golden-part run after install.

Four leaks that inflate the score

1. Twins across folds. Burst capture — ten frames of one part in one pose — puts near-identical frames on both sides of the split. The model recognises the twin, not the defect, and the pooled score climbs. Assign folds by part or by capture session, never by frame.
2. Tuning after scoring. Adjusting thresholds once the fold scores are read, then quoting those same scores, burns the estimate. Tune on splits inside the training folds, or freeze the recipe and re-run all k folds once on the frozen version — quote only that run.
3. Thin or unstratified defect classes. With 10% defects, an unstratified fold can hold anywhere from 2 to 14 of them — stratify. When a class holds fewer frames than 3 × k, the per-fold escape estimate rests on scraps; see how many samples deep-learning defect inspection needs and, for detection models, how many images to train an object detection model.
4. Label errors. A defect frame labelled good poisons both rates: scored as a reject it inflates false rejects, scored as an accept it hides an escape. If a trained model returns nothing at all, the labels are the first suspect — fix labels before retraining.

From validated recipe to ordered hardware

Validate the recipe first, then size the device that runs it, because the recipe decides the latency and interface budget. A single-station check lives at the camera; a multi-camera cell runs on a controller; heavier models get an inference box:

The MV-SC5060XM-08M-WBN SC5000X smart camera carries a 6 MP mono sensor with an 8 mm M12-mount lens and inspects on the device. The MV-VC3301H-128G66 vision controller pairs an Intel Core i3 with 8 GB of memory, includes a VM6200Pro license and feeds multi-camera cells over its 5+ LAN ports. The MV-VAC3504X-128G01 AI inference terminal adds 4 built-in light channels for stations that switch illumination between shots. If the frames already come from cameras installed on the line, adding AI surface defect inspection to existing cameras covers the retrofit path, and AI vs rule-based smart camera inspection sets what each part of the recipe contributes.

Before you quote the number

• Stratify folds by class and by capture session, so every fold sees every condition.
• Pool the runs into one confusion matrix; quote each rate with its count, because a percentage hides how thin the sample behind it is.
• Report the worst fold beside the pooled rate; it bounds a bad shift better than the average does.
• Freeze the recipe, score once, quote that run — never re-quote after tuning.
• Keep at least three defect frames in every test fold, or lower k until you do.
• After install, run golden parts through the real station before sign-off.

Quote the pooled rates with counts, carry one honest caveat — the collection, not the line, is what was measured — and then order the hardware. When a second pair of eyes would help, the contact page reaches the applications team.