Anomaly detection metric configuration: align task and metrics

An anomaly-training run can finish feature extraction and still fail when validation collects its first metrics. The configuration may request pixel measurements for an image-only task, monitor a metric that never appears, or combine incompatible software components. This troubleshooting workflow helps you separate those cases. Start with the final exception, check the evaluation contract, and reproduce the failure with the smallest supported configuration before changing the model or acquisition hardware.

Identify which stage actually failed

Read the final exception first, then locate the last completed stage. Dataset loading, feature extraction, metric updates, threshold selection and stopping callbacks are different failure boundaries. A warning printed earlier may describe a real configuration problem without causing the eventual crash.

Use this order to decide what to inspect:

A missing internal metric attribute does not, by itself, prove that the stopping callback caused the failure. First establish whether metric updates complete at all. If validation succeeds but stopping behaves unexpectedly, continue with early stopping when training loss remains high.

Match the task, predictions and ground truth

Treat evaluation as a contract between model outputs, target labels and metric inputs. Check that all three describe the same unit of inspection.

For image classification, each image needs an image-level target and an image-level anomaly score. The question is whether the entire image is normal or anomalous. Pixel metrics should be disabled through the configuration mechanism supported by your installed release.

For segmentation evaluation, each evaluated image needs an aligned target mask and a predicted anomaly map. The question is where the defect appears. A model producing a heatmap does not automatically make pixel evaluation valid: the dataset loader must also supply meaningful masks.

Check the binary convention explicitly. The positive class should consistently mean the defect class throughout scoring, thresholding and reporting. Follow the installed metric implementation's documented input requirements rather than assuming it accepts arbitrary floating-point labels.

Resizing and cropping must transform masks consistently with images. Mask interpolation should preserve discrete labels rather than introduce fractional classes. If the loader cannot find annotations, use the checks in dataset folder and mask troubleshooting.

Audit the resolved validation split

Configuration ratios are not evidence of the dataset that actually reached validation. Some workflows create missing normal evaluation samples automatically; others reuse a test set for validation. Inspect the resolved file lists and class counts after dataset setup.

For every split, record:

• The number of normal and anomalous images.
• Whether masks exist for every image requiring pixel evaluation.
• The physical part, acquisition session or production batch associated with each image.
• Any overlap between training, validation and final testing.
• The resize, crop and normalization applied during evaluation.

Consider a planning example with 800 normal images and 200 anomalous images. Reserve 20% of the normal images for evaluation, then divide that reserve equally between validation and testing. Divide the anomalous images equally between those two evaluation sets.

This example describes an intended allocation, not universal split-setting semantics. A ratio may apply to the original dataset, an existing test subset or only the normal images. Verify its meaning in the documentation matching your release.

Keep related captures together. Adjacent frames of the same part should not cross split boundaries merely because their filenames differ. Use validation for threshold selection and configuration decisions, then reserve the independent test set for the final assessment.

If the inspection mixes different part families, address that separate design question with training across multiple product classes. Changing metric names does not resolve a poorly defined product population.

Separate metric construction from stopping configuration

A metric's display name, configuration identifier and internal attribute name are not necessarily identical. Do not invent a prefix or rename a metric until an exception disappears. Use the release's supported configuration examples, then inspect the values actually emitted during validation.

There are two distinct checks:

1. Metric construction: Can the configured metric accept predictions and targets, update its state, and compute a result?
2. Callback monitoring: Does the stopping or checkpoint callback reference that emitted result using the exact documented key?

If construction fails, changing callback patience will not repair it. If construction succeeds but the callback reports a missing monitor, inspect naming, logging scope and validation timing.

F1 depends on a decision threshold. AUROC evaluates score ranking across thresholds and requires both target classes for a meaningful binary result. Do not feed thresholded decisions into a score-ranking metric unless the metric's documented interface specifically requires them.

For a separate sanity-check fixture, suppose there are 20 defective images and 20 normal images. At a chosen threshold, the detector finds 18 defects, misses 2 and falsely rejects 6 normal images:

Recompute this from saved predictions and labels. A plausible scalar alone is insufficient; verify that the confusion counts match the intended positive class.

For F1 or AUROC, improvement normally means increasing the value. A monitored loss normally improves by decreasing. Select the direction appropriate to the actual quantity. Also check whether the model has meaningful iterative training: a one-pass feature-fitting workflow may not benefit from conventional patience-based stopping.

Isolate configuration errors from compatibility errors

Change one variable at a time and preserve the failing configuration. Use this sequence rather than repeatedly editing unrelated sections:

1. Record the environment. Save the application release, dependency versions, interpreter location, launch command and resolved configuration. Check that the launched process uses the intended environment.
2. Start from a release-matched example. Obtain it through the project's official distribution or documentation. A configuration copied from another release may contain renamed or removed fields.
3. Reduce to image-level evaluation. Use a small, valid dataset with both normal and anomalous validation labels. Select one supported image metric and disable unrelated pixel evaluation and optional exports.
4. Run through validation. Confirm that loading, model fitting, metric updates and metric computation all complete. Do not stop the test immediately after feature extraction succeeds.
5. Add metrics individually. Introduce the second image metric, then pixel metrics only if the task and masks support them. Record the first addition that reproduces the failure.
6. Add monitoring last. Use the emitted validation key and the correct optimization direction. Restore optional logging and export features afterward.

If a supported single metric still produces a missing internal attribute, investigate dependency compatibility and stale configuration rather than assuming the dataset task is responsible. Install the documented compatible dependency set in an isolated environment. Avoid replacing one package with its newest release without checking the application's supported versions.

Do not edit installed library internals as the first remedy. Such changes can hide the symptom while leaving metric state or reporting incorrect.

Classify warnings by their consequences

Warnings deserve review, but they do not all explain the fatal exception.

A seed helps repeat a split but does not guarantee identical results across devices and software releases. Save the actual split membership as well.

For pixel evaluation, storage can grow much faster than image-level evaluation. Retaining one score for every pixel across 100 images of 512 × 512 pixels creates 26,214,400 values. At 4 bytes per value, that score array alone occupies about 105 MB, before targets and other buffers. Reducing batch size does not necessarily reduce memory retained across the full evaluation run.

Verify the repair beyond a successful exit

A run that finishes is only the first acceptance check. Confirm that it measured the intended inspection task:

• Both normal and anomalous examples reached validation.
• Prediction counts match target counts at the intended evaluation level.
• Requested metrics are present, finite and associated with the correct split.
• The monitored key exists whenever the stopping callback evaluates it.
• Threshold selection uses validation data, not the final test set.
• Saved results identify the configuration, environment and split manifest.
• A manually checked subset produces the expected confusion counts.

Inspect false accepts and false rejects separately. A strong aggregate score can still conceal an unacceptable missed-defect pattern. For deployment decisions after evaluation, use turning anomaly outputs into pass/fail decisions.

Keep acquisition provenance separate from the software repair

Replacing a camera will not repair a missing metric attribute. Hardware becomes relevant when image scale, exposure, focus or acquisition settings differ across datasets. Record those settings so a software repair is not confused with a changed image population.

For a new capture station, compare area scan cameras through the camera finder. The LS-CMR-0053 provides a GigE interface.

The LS-CMR-0166 provides a USB3 interface. Select acquisition hardware for the imaging requirement, independently of evaluation-library troubleshooting.

Keep exposure and preprocessing consistent when collecting diagnostic samples. If acquisition changes are necessary, create a new dataset revision rather than silently mixing captures into the previous evaluation set.

Prevent the mismatch from returning

Before accepting another model configuration, complete this short audit:

• Task: Image classification or segmentation is explicit.
• Targets: Image labels or masks match the requested evaluation level.
• Splits: Membership is saved, with physical-group separation checked.
• Metrics: Each metric works independently before the collection expands.
• Monitor: Its exact key and optimization direction are verified.
• Environment: Versions match a documented compatible installation.
• Outputs: Each run has its own configuration and result directory.

For an unresolved failure, prepare the final exception, a minimal configuration, version information and split counts without including confidential production images. Use application support to discuss the capture and evaluation requirements before changing the inspection hardware.