Independent review of a vision application: what to prepare

You have a vision system in production, the process around it varies naturally, and someone — a customer, an auditor, your own quality manager — has asked whether the system really catches what it should. Bringing in an independent reviewer is the right move. The review, though, is only as strong as the evidence you put in front of it, so the preparation is the project. This checklist covers what to assemble: how many labelled samples the reviewer needs, how to label them so the numbers mean something, which process extremes the set must span, and how to turn the measured false-accept and false-reject rates into an accept-or-fix decision.

What can an independent review actually tell you about your vision application?

A well-prepared review returns two measured numbers — the false-accept rate and the false-reject rate on your labelled sample set — plus a judgement on whether the optics, lighting and decision logic behind those numbers can hold as the process varies.

Those are measurements, not impressions. From 100 known-defect samples, a system that misses one part has a measured miss rate of 1%, with the arithmetic on the page. From 25 demonstration images it has an anecdote. Independence matters as much as the arithmetic: a reviewer who sells no hardware has no reason to steer you from a threshold change towards a replacement camera.

Know the limits of the exercise. A review quantifies and diagnoses; it does not replace your regulatory file, and it cannot prove a rate lower than the sample size supports. On complex or naturally variable parts, rule-based inspection commonly runs false-reject rates of 5–15% or higher, so a finding in that band is a diagnosis to act on, not a verdict against the tool.

What do you prepare before the reviewer arrives?

Prepare four bundles before day one: a labelled image set of at least 100 known-good and 100 known-defect samples, the gold-standard disposition behind every label, the system's settings and hardware list, and its production history of alarms and escapes.

Image set

• At least 100 known-good and 100 known-defect raw images, untouched by filters, with original file names and timestamps. Below 100 per class, a single mislabel moves the measured rate by a full point.

Gold standard

• An accept or reject disposition for every sample from a method that never touched the vision system, plus the written defect library used for labelling. The labels are the ruler of the whole measurement.

System records

• Current inspection parameters with the date each was last changed, the software version, and the camera, lens, light and controller model numbers. The reviewer connects rates to causes only through this page.

Production history

• Alarm and reject logs from a recent period, escape complaints, containment actions. Live history shows behaviour at true defect prevalence, which the enriched sample set deliberately distorts.

If the reference set must be shot fresh — new variant, changed presentation, a light replaced since the last validation — shoot it on a bench set-up you control rather than through the production tool. A mono GigE camera with a C-mount such as this one keeps the captures raw:

How many known-good and known-defect samples does the review need?

Use 100 per class as the floor and the rule of three as the sizing tool: to demonstrate a false-accept rate no worse than p with zero misses at 95% confidence, you need about 3 / p known-defect samples — 600 for 0.5%, 3,000 for 0.1%.

The rule of three is ordinary statistics: with n trials and no failures, the upper 95% confidence bound on the failure probability is roughly 3/n. The 100-sample floor gives about 1% resolution — enough to show a system is grossly broken, not enough to certify a tight escape limit.

Worked example, on a line running 10,000 parts per day with a true defect rate of 2%:

• Defects: the 0.5% escape target gives n = 3 / 0.005 = 600 known-defect samples. The system misses 2, so FAR = 2 / 600 = 0.33%. Expected escapes: 200 defective parts per day × 0.0033 ≈ 0.7, or about 2 every 3 days.
• Goods: 500 known-good samples spanning the extremes in the next section. The system rejects 15, so FRR = 15 / 500 = 3.0%. Expected false rejects: 9,800 good parts per day × 0.03 = 294.

Run at least 1,000 consecutive live parts through the system too, and reconcile its verdicts against the gold standard; at 2% prevalence that yields only about 20 defects, which is why the enriched set above carries the escape statistics. Where sampling is contractual, lot sample sizes come from ISO 2859-1 inspection levels (special levels S1–S4, general levels I–III, normal, tightened or reduced severity).

These numbers measure an existing system; training a new classifier needs larger counts again, as set out in how many samples deep-learning defect inspection really needs.

How do you label samples so the review measures truth rather than opinion?

Label every sample with a disposition from a method that never touched the vision system — metrology, a functional test, or a three-person consensus panel working to a written defect library — because labels made by the tool's own tuners measure agreement, not accuracy.

Get the ruler right before anyone counts anything:

• Write the defect library first: one page per defect class with a photograph, the physical cause, and the accept or reject boundary in measurable terms. The panel grades against this, not against instinct.
• Keep the grey zone. Hard-to-call parts are exactly where the system will fail, so disposition them with the reference method instead of deleting them from the set.
• Blind the scoring. Nobody assigning dispositions sees the system's verdict; unblind only after both sets are frozen, or the labels drift towards the machine.
• Measure the ruler itself. Re-present 10% of samples to the panel a week later; if the panel disagrees with itself by more than a few percent, the reference method is the first thing to fix.

On naturally variable product, the reference method carries most of the review's credibility. Spend the effort here rather than on more images.

Which process extremes must the image set cover?

Cover every axis the process really varies on — illumination, part presentation, material lots, contamination state and ambient light — because a set shot in one afternoon at one brightness proves only that one corner of the process works.

Sample each axis at its extremes and record the condition with every image, so the reviewer can attribute failures:

• Illumination: log illuminance at the part plane with a meter, and shoot both ends of the observed drift — if the reading moves ±10% across a shift, the set includes −10% and +10%.
• Presentation: position, rotation and spacing at the limits of what the feeder produces, not the nominal.
• Lots: at least three material lots including the newest; lot-to-lot appearance drift is the classic escape cause.
• Contamination: optics and diffusers clean, then again after a week of production dust or film transfer.
• Ambient light: doors open versus closed, hall lights on versus off. The levers are in controlling ambient light variation in pick and place vision.
• Natural appearance spread: the full colour and texture range good product shows. AI defect inspection with natural appearance variability covers how reviewers reason about that spread.

An image without its conditions is scenery. The metadata is what turns the 1,100-image set of the worked example into a test.

What does the reviewer need on the day?

Hand over the complete optical chain on one page: camera model and sensor format, lens focal length and f-number, working distance, lighting type and controller settings, inspection parameters and software version — without the settings, the review audits a ghost.

• Optics record: focal length, f-number, working distance, sensor format, field of view at the part.
• Lighting record: light model, controller channel outputs, strobe duration, pulse timing relative to the trigger.
• Parameter record: every threshold and check, its current value, and the date it last changed.
• Version history: software and firmware versions with change dates, so a rate change traces to a tool change.
• Access: read-only export of raw images and verdict logs, so the reviewer can re-score the same images through the current logic.

If any two of these records disagree, reconcile them before day one; otherwise the review stalls on whose numbers are true.

How do you turn the results into an accept or fix decision?

Set the limits before the review runs — a maximum false-accept rate from your risk analysis and a maximum false-reject rate from what the line can absorb — then the verdict is arithmetic: a measured rate above its limit means fix, both inside means accept with monitoring.

Typical starting points: 0.1% maximum false-accept when an escaped defect reaches a customer or patient; 1% maximum false-reject when rejects go to a manual re-check, 0.5% when every reject is scrapped. Tighten or loosen from your own risk analysis.

Price the limits before the meeting, on the worked line above:

• False rejects at the measured 3.0% cost 9,800 × 0.03 = 294 parts per day. At an assumed 30 seconds per manual re-check, that is 294 × 30 s = 8,820 s ≈ 2 h 27 min of labour daily — usually cheaper than scrapping, and an argument for a re-check path.
• Escapes at 0.33% cost 200 × 0.0033 ≈ 0.7 defective parts per day. One escape reaching a customer can cost more than a year of that labour, which is why the accept limit sits below the reject limit in any risk-bearing application.

After any fix, re-score the identical labelled set and log the change as you would a software release — the machine vision software upgrade validation checklist is the pattern. If the verdict is replacement, carry the labelled set and the limits into the machine vision system selection by application requirements brief, so the new system faces the same test.

When is the right answer to rebuild the inspection instead of tuning it?

Rebuild when the defect signature is not in the image: if the critical feature is 0.05 mm and one pixel covers 0.12 mm, no threshold, filter or retraining will separate good from bad — change the optics, lighting or resolution first, then re-run the same labelled set.

The arithmetic decides it. A 2,448-pixel-wide sensor over a 45 mm field resolves 45 / 2,448 ≈ 0.018 mm per pixel, so a 0.05 mm feature spans about 2.8 pixels — detectable. The same sensor stretched to a 300 mm field resolves 300 / 2,448 ≈ 0.12 mm per pixel, and the feature sinks below half a pixel. Design for at least 3 pixels across the smallest feature that must be detected: pixel size ≤ 0.05 / 3 ≈ 0.017 mm.

In a rebuild, change one variable at a time, wavelength and lighting geometry first: on shiny or curved surfaces the lighting choice decides whether the defect exists in the image at all, the pattern shown in glass bottle defect detection: fix lighting before thresholding. Re-shoot, re-label and re-score the same way. If the physics is fixed and the escape limit still cannot be met, run the vision feasibility checklist before you teach and treat the outcome as a selection decision judged by the same labelled set and limits. For the final on-device check in a rebuilt cell, an integrated vision sensor keeps the verdict where the parts are — this 5 MP colour model talks Fast Ethernet through an M12 mount:

Where do you start this week?

Start with the sample bank, because it is the long pole: define the defect library, begin banking known-good and known-defect samples with their conditions recorded, and freeze the acceptance limits in writing before the reviewer is booked.

With the set growing and the limits fixed, the review becomes a scheduling matter. When the rebuild path needs parts, browse smart cameras and area scan cameras or the camera finder; walk the camera configurator if you are sizing a new chain; send a labelled-set plan through the quote page for a parts review. The applications topic page gathers the related guides.