Anomaly detection custom dataset training: fix folder and mask errors

Training an anomaly detection model on a public benchmark dataset usually works on the first try. Training on your own defect images often fails after a few steps, with a missing-file warning, a NaN loss, or a crash inside the data loader. The cause is almost never the model. It is the hand-off between your folder structure and what the training pipeline expects. This article walks through the failures in the order they appear, the test that isolates each one, and the fix, so you can move from captured images to a trained, validated model on the line.

Symptom: training starts, then dies reading a mask

The most common crash pattern with a from-folder dataset looks like this: the first training steps run, the model summary prints, then validation starts and a loader worker throws a type error deep inside the dataset code, right after a warning that an image file could not be opened. The error usually lands in a line that divides a mask by 255, and the division fails because the mask path was empty.

Read the sequence, not the traceback. Training ran, which means normal images loaded fine. The crash happens at validation, on an abnormal image. The pipeline was asked for that image's ground-truth mask, found nothing, and passed an empty path to the file reader.

Test: set the task to classification instead of segmentation in your dataset configuration and rerun. If training completes, the model, the paths and the splits are all fine. Only the mask expectation was broken.

Fix: choose one of two paths. If you only need good/bad decisions, stay with classification and delete the mask entry. If you need defect localization, set the task to segmentation and provide one mask per abnormal image, with file names that match the abnormal images one to one. A segmentation pipeline cannot guess which pixels are defective; the mask is the label.

Why NaN loss appears in the first epoch

A loss that prints as NaN within the first epoch has three usual causes, in order of likelihood.

1. Empty or unreadable training images. A zero-byte file, a wrong extension, or a path that resolves differently from where you launched training produces empty tensors. The warning can't open/read file in the log is the tell; treat any such warning as fatal, not cosmetic.
2. Images with an unexpected number of channels. Grayscale captures saved alongside color ones, or images with an alpha channel, break transforms written for three-channel input. Convert everything to one format before training.
3. A batch containing no valid samples after filtering, which makes normalization divide by zero. This happens with very small datasets and aggressive split ratios.

Test: run a five-line script that opens every image under your dataset root with an image library and prints the shape of each. Any file that fails to open, or prints a shape other than the size and channel count you expect, is your defect. This catches in seconds what otherwise costs an hour of debugging inside the trainer.

Getting the split settings right

From-folder pipelines build their own train, test and validation splits from the folders you point at. Three settings do the work, and getting them wrong produces either empty loaders or a model scored on images it trained on.

• Test split. A ratio of 0.2 holds out 20 percent of the normal images for testing. With fewer than 50 normal images, drop the ratio to 0.1 so training keeps enough data.
• Validation split. A ratio of 0.1 of the training remainder works well. If your version supports a synthetic validation mode, prefer it when abnormal images are scarce: it perturbs normal images instead of eating your only defect samples.
• Do not reuse defect images across splits. Every abnormal image must be in the test set only. A defect the model saw during training makes later scores meaningless.

A worked example: you capture 200 normal and 30 abnormal images of one part. With a test ratio of 0.2, 40 normal images move to test. The 30 abnormal images all land in test as well. A validation ratio of 0.1 then holds 16 of the remaining 160 normal images for validation, leaving 144 for training. If your framework reports a training set larger than that, the split did not apply, usually because the folder paths pointed at the wrong level of the tree.

The same discipline applies when you move from offline training to deployed inference; the anomaly detection early stopping with high training loss article covers how validation behaviour during training predicts on-line performance.

Checklist: preparing the dataset before you train

Work through this list once per product category. Each item exists because skipping it has cost someone a debugging session.

• One folder per category, with subfolders for normal, abnormal and, for segmentation tasks, masks. The structure should mirror the benchmark format the pipeline documents.
• Consistent extensions. Pick PNG or JPEG and convert everything. Mixed extensions break name-to-mask matching.
• Every abnormal image has a mask with a matching name, if the task is segmentation. Count both folders with a file listing; the counts must be equal.
• No defects in the training set. Inspect the normal folder with the same care you give the defect folder; one mis-sorted defective part poisons the whole model.
• Images capture real production variation: lighting drift, part rotation, surface finish change. For acquisition-side decisions that affect what the model sees, see industrial camera image quality settings: exposure, gain, balance.
• All images open, same bit depth, same channel count. Verify with a script, not by eye.
• Paths relative to the working directory where you launch training, and test with a dry run on a copied subset first.

Background and negative samples matter here too: images of an empty belt or a different product belong in training only if the pipeline explicitly supports them, which the background images in object detection training article explains for detection pipelines.

Where training runs: from workstation to the line

Train on a GPU workstation; the first epoch tells you within minutes whether the data pipeline is sound. Once the model validates, deployment usually moves to a controller on the line. A vision controller combines the compute, the camera ports and the light channels in one fanless box, so the trained model, the cameras and the strobes share one enclosure next to the station.

For heavier categories, larger image sizes or several cameras per station, a controller with more memory and more I/O keeps inference latency flat as the feature memory bank grows. The model below pairs PoE camera ports with eight built-in light channels and 32 GB of memory.

Compact stations with one or two cameras and a modest image size fit a smaller controller without giving up the built-in light channels.

If you are still choosing the capture side, browse the smart cameras category, where integrated inspection devices run models on the device, and the vision controllers category for PC-class inference. The camera finder filters by interface and resolution, and the system configurator steps through camera, controller and lighting together.

Once a model is in production, the question shifts from training errors to keeping the pipeline healthy; the machine vision software upgrade validation checklist covers how to revalidate after any software change. And anomaly detection is one tool among several for asset health; for the wider picture, see this predictive maintenance overview.

When the usual answer is wrong

Sometimes the configuration is genuinely correct and training still fails. Three less common causes to check before you rewrite the dataset again:

• Version drift. Dataset configuration keys move between releases of the training framework. A config written for one version silently ignores keys in another and falls back to defaults that expect different folders. Diff your config against the template shipped with your installed version, not against an older tutorial.
• Symlinks and network drives. A dataset that lists fine in a file browser can fail in a loader worker when the mount drops or symlinks resolve outside the sandbox. Copy the dataset locally for training runs.
• Mask value range. Masks saved as 0/255 and masks saved as 0/1 both exist in the wild. If your loss behaves oddly but nothing crashes, inspect a decoded mask: the pipeline expects 0/255 and divides by 255 itself.

A related failure mode, where segmentation output exists but is mostly noise, is treated in crack segmentation in images: separating cracks from noise.

Quick reference: symptom to fix

Fix the data pipeline first, and the model almost always trains on the second attempt.