High accuracy but no segmentation mask in anomaly detection

You trained an anomaly-detection model on your own parts, the test log reports 0.98, and the inference run still saves a blank or speckled mask where the scratch should be. That combination — high accuracy with no usable segmentation mask — is the most common surprise when PatchCore-style models move from benchmark folders onto the line, and it is not a broken install or a bad checkpoint. The 0.98 is an image-level AUROC; the mask is decided by a pixel-level F1Score and a threshold. This page separates the two numbers, works the arithmetic that links them to mask quality, and lists the settings to check before the overlay reaches your HMI. It is aimed at engineers putting anomaly inspection into a production cell.

Why does a model scoring 0.98 save an empty mask?

The 0.98 you are reading is image-level AUROC, a threshold-independent ranking score for whole frames, so a model can rank defective images almost perfectly and still draw a blank mask once the pixel threshold sits above every score it produces.

The training configuration already separates the two families: a metrics block with image and pixel entries, each listing AUROC and F1Score, and a threshold block that is either adaptive or manual. The number quoted in a report is almost always the image AUROC, because it is high and it looks like accuracy. The mask on disk, though, is the anomaly map after normalization and one fixed pixel threshold. When that mask is empty while the image score shines, the pixel F1Score in the same log is usually low — read it before touching anything else. A blank mask and a missing mask are different faults: if no mask file appears at all, the run was trained without task: segmentation, and no threshold will create pixels.

What does an image AUROC of 0.98 actually measure?

An image AUROC of 0.98 says that a randomly chosen defective frame outscores a randomly chosen good frame about 98 times in 100, and that is all it says: it is computed over whole images and never touches individual pixels.

AUROC sweeps every possible threshold and integrates the trade-off between image-level hits and false alarms, so no threshold appears in the result. Two properties follow. First, the score can be high even when the anomaly map is poor: PatchCore-style models collapse the map into one image score, often its maximum, so a single strong hot spot ranks the frame correctly while the rest of the map is mush. Second, AUROC is blind to how much of the frame the defect fills. A scratch covering 8 % of the frame can coexist with a map that flags 40 % of the frame at some threshold, and the ranking number never notices. Pixel AUROC flatters for the same reason in reverse — background pixels dominate — so an impressive pixel AUROC can sit next to an unusable mask. F1Score at the operating threshold is the honest number.

Which score tells you if the segmentation mask is usable?

The number that decides mask quality is the pixel-level F1Score at your operating threshold: roughly 0.2 to 0.4 produces blotchy noise, about 0.5 means half the flagged pixels are wrong, and production overlays want 0.7 or better.

The 0.5 row is arithmetic, not taste: with precision and recall balanced, an F1Score of 0.5 leaves half of the flagged pixels false and half of the true defect unflagged. The bands on either side are working targets for surface-inspection overlays.

Check the imaging scale before blaming the model. Among area scan cameras, a 12 MP model reading 4,000 × 3,000 pixels is the example here.

Spread the 4,000-pixel axis across a 200 mm field of view and one camera pixel covers 0.05 mm (200 / 4,000). This configuration resizes images to 256 and center-crops to 224, so one network-input pixel covers 0.78 mm (200 / 256) and a 0.5 mm scratch arrives 0.64 pixels wide. Features are computed on a strided grid, so a defect spanning fewer than about 3 input pixels falls between samples and is lost before any threshold is chosen. Crop the region of interest to the part first: a 40 mm window gives 256 / 40 = 6.4 pixels per millimetre, and the same scratch spans 3.2 pixels.

How do you set the pixel threshold so the mask means something?

Set the pixel threshold from scores measured on line-captured images: keep the adaptive method only while validation matches production, then fix manual_pixel between the highest good-part score and the lowest defect score — 0.30 in the worked example below.

The configuration carries both values separately: a threshold block with method adaptive or manual, plus manual_image and manual_pixel fields for fixed values. The adaptive method derives its values from the validation split, which is sound only while the line reproduces validation conditions — same optics, same lighting, same part presentation. When production drifts, fixed values measured on live parts win.

Measuring the gap takes minutes. Run inference on 10 to 20 known-good and the same number of known-defective parts from the line, record the anomaly-map values, and read two numbers: the highest score any good part reaches, and the lowest score inside any defect. Suppose good parts peak at 0.12 and defect interiors start at 0.45; manual_pixel = 0.30 sits in the gap, about 2.5 times the good-part maximum. Set manual_image the same way from the image scores, independently. The heatmap-based validation workflow behind this step is covered in the anomaly detection without ground truth guide.

Why is the heatmap sensible but the mask blank or fully flagged?

A blank mask means every pixel scored below the threshold and a fully flagged mask means every pixel scored above it; both point to a threshold-and-distribution mismatch, and the fix is re-deriving the threshold on line images rather than retraining.

Work down this list before retraining. First, normalization: with normalization_method: min_max, anomaly maps are scaled into a 0 to 1 range against the minima and maxima recorded at training time, so production images scoring outside that recorded range collapse to 0 or 1, and the same fixed threshold flags nothing or everything. Rebuild the statistics from line images after any optics or lighting change. Second, an adaptive threshold computed on a validation split that no longer matches the line — switch to manual values measured as above. Third, lighting drift: an aging lamp or an inspection lamp switched on nearby shifts every score, and the threshold that worked on Monday fails on Friday; controlling ambient light variation at the station removes most of this class of failure. Some runtimes make the failure explicit, raising an attribute error when a mask output is enabled but comes back empty — this knowledgebase entry on enabled-but-empty output masks follows that path.

When is image-level pass/fail enough without a mask?

If the decision is pass or fail per part, the image-level score against the image threshold is enough, and you can drop the mask: segmentation earns its cost only when you need defect location, size, or an operator overlay.

Gating a capsule, a casting or a connector needs one number per part, and the image score plus the image threshold deliver it. Pixel ground truth, threshold tuning and overlay rendering all drop out of the project with the mask. Keep segmentation when the mask feeds something downstream: rejecting by defect area, steering a robot to the flaw, or showing an operator where to rework. The distinction between the task families — and the outputs each one produces — is set out in the anomaly detection versus classification, detection and segmentation guide, and the deployed-model path from map to decision is in the ONNX anomaly detection inference walkthrough. For a single-view pass/fail station, smart cameras that run the gate on the device are often the shorter path.

What hardware runs anomaly inference at the line?

A single-camera PatchCore-style model working on 224 × 224 inputs fits a compact fanless controller with an Intel Core i5-class processor, while multi-camera lines or frame rates above roughly 10 fps want the single-GPU AI inference terminal.

Two costs decide the host: the backbone forward pass (wide_resnet50_2, reading layer3) and the nearest-neighbour search against the coreset, sampled at 10 % with 9 neighbours. On a current Core i5-class CPU the whole pass takes about 0.1 to 0.5 s per frame — comfortable for a one-camera station inspecting one or two parts per second, tight beyond that. Coreset memory grows with the training set and shows up as slow inference long before it fails; the anomaly detection memory bank article puts numbers on that growth. Score stability matters as much as compute: a controller whose built-in light channels drive the lamp from the same clock as the camera keeps scores comparable frame to frame.

For a one-camera station, the MV-VC3301H-128G66 carries an Intel Core i3 with 8 GB of memory and a 128 GB SSD, with PCIe expansion supported. The MV-VC5525-256G70-NN steps up to an Intel Core i5 with 32 GB of memory, PoE camera ports and eight built-in light channels. The MV-VAC3504X-128G01 is the AI inference terminal route, single-card, with five or more LAN ports and four built-in light channels.

Which checks do you run before trusting masks in production?

Before the mask goes into production, read the pixel F1Score from the training run, run inference on defective parts photographed at the line, fix both thresholds from those images, and confirm the drawn mask covers each known defect with margin on 20 or more frames.

1. Confirm task: segmentation in the training configuration — an image-level model never emits a mask.
2. Read the pixel F1Score at the saved threshold from the test log, not the image AUROC; hold the 0.5 floor and ask 0.7 or higher for overlays.
3. Confirm every defect class spans at least 3 pixels at the network input after resize and crop.
4. Record score distributions from 10 to 20 good and 10 to 20 defective line images, then set manual_pixel and manual_image inside the gap.
5. Rebuild normalization statistics and re-measure both thresholds after any lighting, optics or presentation change.
6. Save the output set on mode: full so the overlay, heatmap and input image land in one folder for review.

Run those six and the decision writes itself: a pixel F1Score of 0.7 or better on line-captured defects, with a fixed threshold sitting in a measured gap, deploys as-is; 0.3 means collecting more defect examples before anyone sees the overlay. For the imaging side, the camera finder filters by resolution, sensor format and interface; the compute side starts at vision controllers. The applications topic page collects guides like this one, and contact reaches the applications team when you want a second pair of eyes on your score distributions.