Planning vision inspection error rates for the worst case

A vision system that scores 99% on its validation set can still stop your line in week three. Industrial inspection repeats the same decision up to a million times a day, with a physical part on the other side of every call, so the exceptions your error rate creates — not the average accuracy — decide whether production flows. This guide gives you the arithmetic that turns an error rate into parts per day, splits false accepts from false rejects, and shows the routing pattern for parts the system is unsure about: reject, recirculate or hold. It is written for anyone specifying an inline inspection or code-reading station, from a smart camera deciding on the device to a model running on a vision controller.

Why does an average accuracy number stop your line?

Because an inspection that is right 99% of the time is wrong or unsure 10,000 times a day at one million decisions, and those 10,000 parts still need a physical decision the moment they pass the camera.

Consumer AI optimizes the average across millions of tolerant users: the wrong city's weather costs a repeat question, the wrong song costs a skip. A production line has no such tolerance. The same model may fire the same judgment a million times before lunch, and every judgment moves a physical part — a battery cell, a pharmaceutical package, a machined housing — one step closer to a customer or to a bin.

Mean accuracy also hides where the mistakes sit. A system at 99.5% overall can hold 99.9% on the easy 90% of parts and something far worse on the hardest 10%, and the hard cases are exactly what a changeover or a new supplier batch delivers. You are judged on those cases, and on how fast the system adapts when they arrive. That is why the design target is a bounded worst case rather than a polished mean, and why the choice between trained rules and learned models matters as much as the score — see AI vs rule-based smart camera inspection: how to choose.

How many parts a day will your error rate send sideways?

Multiply daily decisions by the share the system gets wrong or refuses to commit on: 1,000,000 decisions at a 1% error-or-uncertainty rate returns 10,000 parts a day, and at a 6 second manual recheck that is about 17 operator-hours of clear-up.

Work it for a real line: about 20 units per second across 16 production hours at 87% uptime. That is 20 × 57,600 s × 0.87 ≈ 1,000,000 decisions per day. At a combined 1% error-or-uncertainty rate, 1,000,000 × 0.01 = 10,000 units a day leave the decision path needing something done to them. A person clearing them one at a time at 6 seconds each needs 10,000 × 6 s = 60,000 s ≈ 16.7 operator-hours; at a realistic 10 seconds with handling it is 100,000 s ≈ 27.8 hours, more than the line's whole production day. The route has to be automatic, with people clearing only what the route cannot.

The rate is also leverage. At this volume every 0.1% on it is another 1,000 parts a day, so a validation result reading 0.5% against 1.5% is a staffing decision before it is a quality one.

Which rate decides your risk: false accept or false reject?

A false reject costs you recheck labor — 10,000 extra parts a day at a 1% false reject rate — while a false accept ships defects, so the false accept rate sets your risk and the false reject rate sets your workload.

Count the two streams separately. False accept rate (FAR) is defective parts passed divided by defective parts presented, and escapes never appear in your reject data, so it stays invisible until you audit for it. False reject rate (FRR) is good parts failed divided by good parts presented, and it is your reject-lane load, your lost yield and, in food or pharma, your write-off.

A single accuracy figure, (TP + TN) ÷ (TP + TN + FP + FN), averages the two and tells you neither. Take the line above with a 0.4% defect rate, so 4,000 defective units a day pass the camera, and move the decision threshold:

Each escape row is 4,000 × FAR and each recheck row is 1,000,000 × FRR. Where an escape can trigger a field recall, the bias in factory automation runs toward zero escapes: take the reject-close row and absorb its workload — but 41.7 operator-hours cannot exist as individual 6-second checks, so that row means batch review or an automated disposition, not a person with a monitor. Lighting and optics move both rates down together; the threshold only trades them, and at this volume each 0.5 point of FRR is 5,000 rechecks a day.

Two exceptions keep the worst-case rule honest. When the retry is free — a logistics code that misses recirculates and reads again at almost no cost — a 99.5% first-pass rate with automatic retry beats weeks of pushing toward 99.99% on pass one. And never trust accuracy on an imbalanced set: at a 0.4% defect rate, a system that passes everything scores 99.6% while catching nothing. Report per-class and worst-case rates, and do not tighten a similarity threshold until everything routes as unknown — that is the line stopping by another name.

What counts as an error when the model still gives an answer?

A model forced to choose between its trained classes assigns a part it has never seen to whichever class looks closest, so a revised part with a new tab still reads pass, and the mistake never shows in your accuracy metric.

Sort every non-ideal outcome into four buckets: false accept, false reject, no-result — no code found, part missing, exposure blown — and the quiet one, a part from outside the training distribution forced into the nearest trained class. The first three announce themselves in your counts. The fourth hands you a confident-looking wrong answer that no validation metric flags, because the part that triggers it does not exist in the validation set.

Defenses, in deployment order:

1. Prefer tools that expose a similarity or distance measure, so unknown is a legal output rather than a forced choice.
2. Band the confidence score: results falling inside a margin of the decision boundary route as uncertain, not as pass.
3. Cross-check with an independent tool — a rule-based presence or geometry check that disagrees with the learned classifier is an exception, not a tie to break.
4. Watch the no-result rate shift by shift: lighting drift, contamination and focus drift all show up there before they show up as escapes.

The scoring question behind all four — how much to trust a result — has a code-reading twin: judging whether a barcode decode is trustworthy works the same problem for read grades.

Where should uncertain parts go?

Into a route you designed before day one: reject to a lockable bin, recirculate past the camera a second time, or divert to a hold station for human review — never back into the good stream.

Pick the route from consequence, then check it against the rate:

• Reject — uncertainty counts as failure. The route when an escape is unacceptable: food safety, pharma, brake components. The bin is the answer.
• Recirculate — the part passes the camera a second time. The fit for transient triggers: glare, foam, a marginal read. Give the loop an exit; anything that fails twice routes to reject, or the return belt fills with repeats.
• Hold — divert to a bench or bin for human review. The fit for judgments a person genuinely adds: a new variant, a process change, a defect pattern missing from training.

Size each route for the worst-case row of the table, not the average: a hold lane built for 500 parts a day that receives 10,000 becomes the new bottleneck inside a shift. Wire it like any other automation outcome — pass, fail and uncertain as distinct states into the PLC, diverter owned by controls — as laid out in routing a single-camera inspection to a robot and reject pusher. The model's job is to say that a part is novel, never to decide what a novel part is.

A route also needs somewhere to send the part. On a single-pass line with no diverter there is no route, so the accuracy has to come from physics — lighting contrast, optics, part presentation — because the misses have nowhere to land. Fixing the station before tuning the model is the right order in exactly that case.

How do you stop the model from guessing on parts it never saw?

Train on the variation you expect, hold back tagged images to measure the worst case, and give the system a similarity threshold that can return unknown instead of a guess.

Coverage beats cleverness. Collect training images across every condition the station will meet: morning, afternoon and night light if the hall has windows; packages folded and unfolded; the defect at every size and position it has appeared in; the variant you know is coming. Tag the strange cases too, because parts you do not want classified need their own label — and most training-set failures are folder and file housekeeping before they are algorithm problems, as custom dataset training: fix folder and mask errors catalogs.

Then hold back. Keep 20-30% of tagged images out of training and score the holdout per class, reporting the worst class rather than the mean. A model at 99.9% on three classes and 97% on the fourth carries a 3% exception rate on that class, and 3% — not 99.9% — is what sizes your reject lane. If that gap is unaffordable, the fix is more examples of the hard class, not more epochs on the easy ones.

What hardware does the exception route need?

Three building blocks: an on-device inspection camera that returns pass, fail or no-result, a code reader whose no-read output is a first-class result, and a controller with enough Ethernet ports to log every flagged image for review.

The decision starts at the camera, on the line. Smart cameras and vision sensors run inspection where the pixels land and return the result over Ethernet, which keeps the pass-fail-uncertain call inside the same deterministic trigger cycle as the motion.

The MV-SC3050XC-16M-ATN-H is a 5 MP color vision sensor with a global shutter, Fast Ethernet and an M12 mount. Global shutter matters once parts move or divert, because every pixel is sampled at the same instant; color pays off when the failure is a stain, a wrong film or a misprint rather than a shape change.

Code reading solved the three-state problem long ago: a fixed code readers result is pass, fail or no-read, and no-read is a first-class outcome wired to its own PLC state — copy that pattern for uncertain classifications.

The MV-ID3013PM-12M-WBN carries the pattern in hardware: a 1.3 MP reader with a 12 mm lens, global shutter, Fast Ethernet and an M12 mount. Its no-read is a result, not a fault, and an uncertain classification should travel through your controls the same way.

Flagged images need somewhere to land with score, threshold and software version attached, and several stations should feed one review point. Vision controllers collect that stream on a single machine.

The MV-VC5525-256G70-NN provides 5+ LAN ports, PoE camera ports, 3-4 USB 3 ports, 8 built-in light channels and 32 GB of memory — enough ports for the inspection stations, the review client and the line PLC to land on one controller.

How do you keep the error rate down after day one?

Treat every flagged image as training data and every shift as a test: review the uncertain bin weekly, confirm or correct it, retrain on confirmed cases, and re-audit a sample of passed parts against ground truth.

Log every decision with enough context to reconstruct it months later: image, result, score, threshold, recipe and software version. When a claim arrives about a part that reached a customer, that log is the difference between a root cause and an argument, and it is what lets the station join your traceability chain as in inline traceability validation: connect codes, OCR and rejects.

Run three loops off that log. The uncertain bin: review it weekly, confirm or correct each disposition, and fold confirmed new cases into the next training set — the only mechanism that permanently lowers the uncertainty rate. The escape audit: pull a sample of passed parts for full manual verification, because escapes never appear in reject data; one escape in 2,000 audited passes extrapolates to 1,000,000 ÷ 2,000 = 500 escapes per million decisions, a number you act on with a lighting or optics change rather than a threshold shuffle. The condition watch: track no-result rate and score distribution per shift, because both move before the error rate does.

What do you check before you sign off the system?

Before release, confirm the worst case on paper: at your line's decisions per day, exceptions stay inside what the reject lane and review rota can absorb — 10,000 a day needs an automated lane, not a person with a monitor.

• Per-class holdout scores. Worst class reported, not the mean; it sets the lane size.
• Three-state wiring proven. Pass, fail and uncertain each reach the PLC as distinct states, and uncertain has a tested destination.
• Route forced past its limit. Tighten the threshold for one shift and confirm the reject lane, recirculation exit and review queue absorb the resulting rate.
• Audit log enabled. Image, score, threshold and software version stored per decision and kept long enough to answer a claim.
• Escape audit scheduled. A weekly sample of passed parts checked by hand, with the FAR estimate written where the team sees it.
• Golden parts sealed. A known-good and known-bad set to re-verify after every retrain or software update.

Pass those six and the worst day has a defined cost instead of a surprise. Browse the camera finder with your exceptions-per-day number in hand, start from scoping a machine vision system before you call an integrator if the application is still taking shape, or send us your decisions per day and the cost of one escape — we will help you pick the camera, reader and controller that turn the worst case into a planned line item.