Anomaly detection without ground truth: test and heatmap setup

You have a model trained on good images only, and a test set with no ground truth: no defect masks, maybe not even a settled split between good and bad parts. That is the normal situation in production vision, and it blocks less than you might expect. Detection never needs labels. Labels feed only the metrics table, and the folder layout decides which metrics still compute. This walkthrough sets up the test folders, tells you which numbers to expect — including the n/a pixel rows — shows how to read an anomaly heatmap, and derives a pass/fail threshold from your own normal-image scores, with the arithmetic worked through for a 40-good / 30-bad test set.

What ground truth is and is not needed for

One-class models such as PatchCore-style memory-bank methods learn a single thing: what your part looks like when it is good. At inference they compare patches of the new image against that memory of normal patches and return two outputs — an image-level anomaly score and an anomaly map rendered as a heatmap. Neither output touches a label.

Labels enter in exactly three places:

• Training folder membership. The trainer treats everything in the training folder as good, so a defective image there silently poisons the model.
• Image-level metrics. The evaluator takes each test image's label from its subfolder name, so a split into a good folder and defect folders is enough to compute image-level AUROC, the area under the ROC curve. Masks are not required.
• Pixel-level metrics. Pixel AUROC and PRO (per-region overlap) compare the heatmap against an annotated mask pixel by pixel. With no masks there is nothing to compare, and those rows print n/a.

The decision follows: train now, split the test folders wherever you trust the split, read heatmaps for localization, and set the pass/fail threshold from your normal scores. When the metric rows and the task disagree, align them deliberately — aligning metrics with the task covers that mapping.

What you need

• A training collection of good images only — a few hundred frames is a typical start, chosen to cover every acceptable appearance: part position in the fixture, finish variation, and the lighting drift your station produces. Parts with natural appearance variability need extra coverage; that case has its own guide on AI defect inspection for products with natural appearance variability.
• 40 to 70 test images you can vouch for, split into known-good and known-bad. The worked example below uses 40 good and 30 bad.
• A trainer that follows the standard folder convention and exports per-image scores plus anomaly maps. Anomalib and similar one-class frameworks do this out of the box.
• Compute in two places: a GPU machine for the one-off training run and a compact inference box at the line. Training happens once, on server-class GPU hardware such as the following.

Memory-bank methods grow with the training set, so if you plan thousands of images, read why anomaly detection memory banks blow up on large datasets before you collect.

Lay out the test folders so labels exist without masks

The evaluator reads labels from folder names, so the split is the annotation. Build the tree like this:

1. Training folder: good images only, pulled from golden parts and visually confirmed production runs. The training-side folder and mask errors have a dedicated walkthrough at anomaly detection custom dataset training: fix folder and mask errors.
2. Test good folder: only images you have personally checked are defect-free. When in doubt, move the image out — a wrong good label flatters the false-alarm count you will quote later.
3. One test folder per defect type, named for the defect: scratch, dent, missing pin. The name becomes the label in reports and per-class metrics; spelling and grouping do not affect detection itself.
4. No mask folder at all. Leave ground truth out of the tree. The pixel-level rows then print n/a, which is the documented result of missing masks, not an error to fix.
5. If the split cannot be settled, put every test image in a single folder. You lose image-level AUROC, but scoring and heatmaps still work; the verification section below shows how to judge such a run.

Run inference and read the heatmaps

Scoring the test set returns, for every image, a normalized anomaly score (0 to 1 in most trainers) and an anomaly map drawn over the image, brightest where the image deviates most from the normal patches. With the layout above and no masks, expect the metrics table to show image-level AUROC computed from the folder labels and the pixel-level rows n/a.

Read each heatmap against three questions:

• Does the hot spot sit on the part surface, where a defect would be?
• Is the hot area compact, or smeared across background, fixture or edges?
• Does the score agree with what you see?

A hot area smeared over the background is a capture problem, not a threshold problem: restrict the region of interest to the part, re-check lighting consistency, then retrain. Capture settings that hunt under their own detection feedback are covered in auto exposure ROI instability from detection feedback.

Worked example: 40 good and 30 bad test images at a mean + 3 sigma threshold

The inputs are an illustrative but typical run; your first inference pass replaces every one of them with measured values.

• Training data: 200 good images, whose per-image scores give a mean of μ = 0.12 and a standard deviation of σ = 0.02 on a 0-1 normalized scale.
• Test set: 40 confirmed-good images in the good folder, 30 confirmed-bad images in the defect folders.
• Measured outcome: after scoring, 28 of the 30 bad images and 3 of the 40 good images land above the threshold computed below.

Rule. Set the pass/fail threshold three standard deviations above the mean normal score. For a Gaussian distribution that line sits above about 99.7% of normal images, which starts you near the low-false-alarm end.

Arithmetic.

• 3σ = 3 × 0.02 = 0.06
• t = 0.12 + 0.06 = 0.18
• Detection rate = 28 / 30 = 0.93, so 93% of bad images are caught.
• False-alarm rate = 3 / 40 = 0.075, so 7.5% of good images are flagged — about 75 false flags per 1,000 good parts (1,000 × 0.075).

Result and decision. Adopt t = 0.18 as the starting operating point: any image scoring above 0.18 is flagged for review. A 7.5% false-alarm rate is workable when clearing a flag costs an operator a five-second glance at the heatmap; if review floods, raise the threshold and accept a lower detection rate — the size of that trade comes from your two score histograms, not from the formula. Note the gap between theory and this run: a Gaussian tail predicts about 0.3% of good images above t, against the 7.5% observed, which tells you the normal-score distribution has heavy tails. That gap is common, and it is why the threshold is confirmed against measured test outcomes instead of trusted from the rule alone. Once about 30 confirmed-bad parts are archived, replace the sigma rule with a measured operating point: pick the t where the cost of missed bad parts equals the cost of clearing false flags.

Verify the model with nothing to score against

Work through four checks in order; each uses images, not labels:

1. Golden parts. Run 10 or more fresh known-good images. Every score must stay below the threshold; a flag means the training set missed that appearance, so add the image to training and retrain.
2. Seeded defects. Score two or three images carrying a defect you introduced — a scribed scratch, a missing component. Each must flag, and the heatmap hot spot must sit on the seeded defect rather than anywhere else.
3. Rank review. Sort all test images by score, highest first, and review the top 10 heatmaps; expect mostly real defects. Then review the bottom 10; expect all clean. If the ordering feels wrong, fix capture or training before quoting any number.
4. Archive per run. Save inputs, scores and heatmaps in a per-run folder layout — organize inference output folders for per-run detection results — so the threshold can be re-checked later against the same files.

Where an end-of-line functional test already exists, compare its verdict with the model's on the parts that pass through both; that agreement is a second, label-free sanity check.

Common mistakes that look like no ground truth problems

• A defective image in the training folder. The model learns the defect as normal and the heatmap stays cold on real defects. No metric catches this; only the spot-check does.
• One mixed test folder, then expecting AUROC. With no folder split the evaluator has no label, so the image-level row goes n/a too. The split is the annotation.
• Deleting the good test folder because only defects seem to matter. Image-level AUROC needs the negatives; without them the number cannot exist.
• Reading the n/a pixel rows as a broken model. Missing masks make those rows n/a by definition; the headline number is image-level.
• Tuning the threshold on the test set and quoting that run's AUROC as honest performance. Either report it as tuned-on-test, or freeze the threshold on one subset and re-score on another.

From test run to line-side pass/fail

Freeze the threshold, export the model, and run it where the parts are. Training stays on the GPU server; at the station you want a compact box that takes the camera stream, drives the light and returns pass/fail:

The MV-VAC3504X-128G01 AI inference terminal provides 5+ LAN ports, PoE camera ports and 4 built-in channels of light outputs, so trigger, illumination and inference sit in one enclosure. The MV-VC3301H-128G66 vision controller pairs an Intel Core i3 with 8 GB of memory and includes the VM6200Pro license, which fits cells already standardized on that vision software suite.

Capture discipline decides everything downstream: a global-shutter camera freezes every part identically, which is what a memory-bank model needs from the station. At fixed small-part stations a smart camera such as the MV-SC6050C-08M-XBN — 5 MP color, 8 mm M12-mount lens, global shutter — covers capture in one unit, and the smart cameras category filters by resolution, shutter and mount. The deployment path from exported maps to a pass/fail bit is walked through step by step in ONNX anomaly detection inference: from maps to pass/fail, and vision controllers is the category to shortlist station boxes from. Not sure which combination fits the cell? The build-a-system configurator steps through camera, lens, lighting and controller choices in order.

Checklist before you trust the numbers

• Training folder spot-checked: 20 random images reviewed, all good.
• Test good folder non-empty, so image-level AUROC stays computable.
• Golden parts all below threshold; seeded defects all flagged, with hot spots on the defects.
• n/a pixel rows accepted as expected, with image-level AUROC read as the headline.
• Threshold, μ and σ written into the run report, so the next engineer can re-derive t.
• Any threshold tuned on test images marked as such in the report.