Anomaly detection vs classification, detection and segmentation

You have a new inspection job, a pile of sample parts, and a decision that fixes weeks of work: which deep-learning task to train. Pick anomaly detection and you label nothing; pick segmentation and you trace defect contours by hand on hundreds of images. Pick wrong and you re-annotate everything. This comparison sets out what anomaly detection, classification, object detection and semantic segmentation each return, what each costs in training images, annotation time and hardware, and gives a decision rule for each case, so you commit to a task and size the area scan cameras and computer before opening an annotation tool.

The decision in one pass

Train anomaly detection when you can image good parts but defects are rare, subtle or not yet catalogued: about 100 pass images, zero annotation, and training that runs on a CPU in industrial toolsets.

Train classification when the line needs a named verdict per part — OK/NG, defect type or grade — and where the defect sits does not matter. You need labeled examples of every class, including each defect type.

Train object detection when the verdict needs a location: a rectangle around each defect, with a class and a confidence score.

Train semantic segmentation when shape or measured size drives the decision — scratch length in millimetres, deposit area, or touching defects that one box cannot separate.

The ladder rule: take the simplest task that answers the line's question. Everything above it costs more images, more annotation and more GPU time, and you can add a finer task later on the same captured set.

What each task returns

Anomaly detection learns only the appearance of good parts and flags what departs from it. The output is an anomaly score for the image, usually with a heatmap showing the offending region, and your decision variable is the score threshold. Proving that threshold with no labeled defects is its own procedure, covered in anomaly detection without ground truth: test and heatmap setup. The same good-samples-only logic reaches past surfaces: it is the backbone of predictive maintenance beyond anomaly detection on machine signal data.

Classification returns one class label per image or per region of interest: pass, or which of several defect types. You judge it with a confusion matrix — the table of true versus predicted class that shows exactly which defect gets mistaken for which.

Object detection returns rectangles, each carrying a class and a confidence value. Some toolsets add a rotation angle to the rectangle, which helps when parts arrive in varying orientation.

Semantic segmentation returns a class for every pixel. That gives defect area in pixels — convertible to square millimetres with your imaging scale — and separates defects that touch.

Training data each task costs you

Anomaly detection starts at about 100 good images. Variety matters more than count: cover the part poses, lighting states and supplier lots the line will actually see, because the model treats unseen-but-good variation as an anomaly.

Classification needs labeled examples of every class. Work from 100 to 300 images per class as the scoping range — and note the trap: the defect classes are exactly the images you rarely have.

Object detection needs boxes: 300 to 1,000 labeled images is the working range, each defect boxed and named. Include clean images with no target; they teach the model what "nothing" looks like, and background images in object detection training explains why they carry no labels.

Segmentation needs the same image counts, but the annotation is contour tracing — an order of magnitude more clicks per image than a box. The polygon workflow in draw and fill segmentation label polygons on training images is what that looks like in practice.

Training and inference hardware

Training the three supervised tasks needs a CUDA-class GPU; plan on it from day one, and why GPUs beat CPUs for deep learning training quantifies the gap. Anomaly detection is the exception: toolsets that ship it train on a CPU as well as a GPU, slowly. For sizing both sides, see machine vision deep learning setup: choosing CPU or GPU.

Inference is lighter than training, and all four tasks run on a CPU. Classification and anomaly scoring sit comfortably in tens of milliseconds on a current industrial processor; detection and segmentation land in the hundreds of milliseconds at a few megapixels. When the cycle needs faster, object detection without a GPU shows what CPU inference realistically delivers — or the controller takes the GPU card instead.

That controller carries PCIe expansion, so the GPU that trained the model can move into the cabinet, and its PoE camera ports power the cameras over the same cable.

When the inspection should live in the device, smart cameras host the trained model on board and free you from a separate computer; whether that beats rule-based inspection is its own comparison in AI vs rule-based smart camera inspection.

The camera side is decided by the smallest defect you must catch, not by the task. Sampling theory puts the floor at 2 pixels across a feature; work from 3 as your floor and 5 when the model must grade severity. Anomaly heatmaps and segmentation masks both go blind below that, and no training run recovers it — run your numbers through the camera resolution calculator.

When the defect is small relative to the field of view, start from resolution: the MV-CS200-10UM is a 20 MP USB3.0 camera that leaves pixels to spare on fine defects.

How a detection is scored: IoU

Object detection is judged on intersection over union: the overlap between the box the model drew (the detection result, DR) and the box your annotator drew (the ground truth, GT), running from 0 for no overlap to 1 for a perfect match.

A detection counts as correct when IoU reaches the threshold you set. 0.50 is the accepted default; move to 0.75 when the box steers a robot gripper or feeds a measurement, because a loose box still passes at 0.50. That robot hand-off has its own architecture questions, mapped in deep learning integration with industrial robots.

Classification's confusion matrix and segmentation's per-pixel agreement sit in the same evaluation step; for the anomaly side of it, read anomaly detection metric configuration: align task and metrics.

Worked example: one scratch box at the 0.50 threshold

Take one scratch on an anodized surface, boxed on a 12 MP frame. The annotator's ground truth starts at x = 610, y = 300, and measures 230 px wide by 50 px tall. The model's box starts at x = 620, y = 310, and measures 210 px wide by 48 px tall. These positions are illustrative inputs; the arithmetic is the part you reuse.

Edges first. The ground truth ends at 610 + 230 = 840 in x and 300 + 50 = 350 in y; the prediction ends at 620 + 210 = 830 in x and 310 + 48 = 358 in y.

Intersection: width = min(830, 840) − max(620, 610) = 830 − 620 = 210 px; height = min(350, 358) − max(300, 310) = 350 − 310 = 40 px; area = 210 × 40 = 8,400 px².

Union: prediction area = 210 × 48 = 10,080 px²; ground-truth area = 230 × 50 = 11,500 px²; union = 10,080 + 11,500 − 8,400 = 13,180 px².

IoU = 8,400 ÷ 13,180 = 0.64. That clears the 0.50 default, so the box counts as a true positive; at a robot-grade 0.75 threshold it would count as a miss. Same arithmetic, different threshold, opposite verdict — set the threshold from what the box is for.

The four tasks side by side

How to choose before you annotate

1. Write the verdict the line needs in one sentence: pass/fail, defect name, or defect name plus position plus size.
2. Count your defect samples. Under about 50 examples of a defect type, the choice is made: anomaly detection, and collect defects as the line produces them.
3. Verdict only, no location: classification, with a confusion matrix as the acceptance test.
4. Location needed, shape not: object detection, with the IoU threshold fixed by the downstream use.
5. Size in millimetres or touching defects: segmentation, and budget the contour-tracing time.
6. Then size the hardware: pixels per defect on the camera, GPU or CPU inference from the vision controllers category, and the enclosure and cooling that go with them.

When the usual answer is wrong

Anomaly detection fails on parts with natural appearance variation — wood grain, cast surfaces, leather — because the normal spread reads as anomalous. Products like that need the supervised route described in AI defect inspection for products with natural appearance variability.

Classification is wrong when two defects can share one image: a single label hides how many and where, and the operator learns nothing to fix.

Detection is wrong when defects touch or merge: two scratches inside one rectangle read as one finding, and measured area is meaningless.

Segmentation as a reflex is wrong too. If a box plus a simple measurement answers the question, the extra annotation and GPU time buy precision nobody uses.

Checklist before you annotate

• Verdict defined in one sentence per task — it fixes the task type.
• About 100 good images spanning real variation for anomaly training — unseen good variation becomes false alarms.
• 50+ defect examples per type if you go supervised — below that, classes stay untrainable.
• Smallest defect spans 3+ pixels, 5 for severity grading — below the floor, no model recovers it.
• GPU identified for supervised training; the CPU-only path is for anomaly detection.
• IoU threshold written down before detection training starts, with the reason attached.
• Score threshold validated on held-back defect parts before the line trusts the model.

Next step: size the camera in the camera finder, walk the system configurator, or send us the part and its defect set and we will name the task and the hardware.