Verify object detection model generalization: per-condition metrics

A model that scores well on one fresh batch has proved one thing: it works on the conditions that batch contains. Whether it also detects on a background it has never seen, under similar lighting, is a question an aggregate score cannot answer, because an aggregate blends every condition together and can hide one that fails. This guide gives you the test design that answers it: choose the condition axes your production varies, size each condition cell from the margin you need to act on, keep whole capture sessions on one side of the split, and publish precision, recall and mAP per condition. A worked example with 4,000 labeled instances walks the arithmetic from split ratio to interval width, and the closing checklist turns the design into a sign-off sheet.

What the test can prove, and what it cannot

A held-out test set proves the model on the condition combinations it contains, and only those. That sets the design rule: a condition counts as covered when production runs it, the test set holds labeled samples of it, and its measured recall clears the production threshold by more than the interval of the estimate. Anything short of all three is a gap, and the honest response is targeted data collection at that condition, not a lower threshold.

The deliverable is a table, not a score. For every cell you ship, the report carries the labeled instance count, recall and precision at the deployed score threshold, mAP at your match level, and a pass or fail verdict. One aggregate mAP over a mixed set answers no production question: a cell at 0.82 recall and a cell at 0.96 average to 0.89, and both disappear inside the average.

No finite test proves the conditions it does not contain. A model measured under LED banks and gray fixtures says nothing direct about sunlight or bare wood, and closing that gap means capturing those conditions. What the statistics can do is make each measured cell trustworthy enough to act on, which is the rest of this guide.

Choose the condition axes and define the cells

An axis earns a place in the grid when two things hold: production actually varies it, and the failure mode differs across its values. On a factory line the usual candidates are lighting (level and color temperature), background (material and clutter), camera or sensor (the shipping model versus a replacement), and the runtime device (PC versus on-device inference). Pose, scale and occlusion usually stay inside the cells, because they mix freely under every lighting and background, so tag them per instance instead of gridding them.

A part inspected under three lighting levels against four background groups gives a 3 × 4 grid of 12 cells. Cap the grid at what you can fill: 12 to 30 cells covers most lines, and backgrounds that share material and clutter belong in one group. The background axis deserves its own design pass — Object detection background distribution for inspection shows how to characterize what surrounds the part before you grid it.

Size each condition cell from the margin you can act on

Recall inside one cell is an instance-level proportion, so its uncertainty follows the binomial interval:

Here p is the recall you expect, n the labeled test instances in the cell, and Δ the half-width you can act on. At p = 0.95:

The sizing decision follows from the rejection line. If the target is 0.95 recall and anything below 0.90 triggers rework, the margin is Δ = 0.03, and n ≈ 0.95 × 0.05 × (1.96 / 0.03)² = 0.0475 × 4268 ≈ 203, so plan on about 200 labeled instances per cell. With 200, a cell measured at 0.94 spans 0.91 to 0.97 and clears the 0.90 line. With 67, the same 0.94 spans 0.89 to 0.99 and touches it, so no verdict is possible.

Cell size sizes the test; training-set size is its own question — How many images to train an object detection model covers the training side.

Worked example: 4,000 instances over 12 condition cells

Say the labeled set holds 4,000 instances across the 12-cell grid, an average of 4,000 / 12 = 333 instances per cell before any split.

Path one, hold out 20 percent: 4,000 × 0.20 = 800 test instances, so 800 / 12 = 67 per cell. The half-width at p = 0.95 is 1.96 × sqrt(0.0475 / 67) = 1.96 × 0.0266 = 0.052, about ±5.2 points. A cell measured at 0.93 then spans 0.88 to 0.98 and straddles the 0.90 rejection line, so this test cannot separate pass from fail on this budget.

Path two, merge the grid. Combine the four background groups into two material groups and the three lighting levels into two, keeping merges that share a failure mode: 12 cells become 4, and 800 / 4 = 200 per cell. The half-width drops to 1.96 × sqrt(0.0475 / 200) = 1.96 × 0.0154 = 0.030, about ±3.0 points, and a cell at 0.94 now spans 0.91 to 0.97, clear of the line. When the merged pairs genuinely fail the same way, this is the right call. When they do not, collect instead: 200 × 12 = 2,400 test instances, and at a 20 percent hold-out that means 2,400 / 0.20 = 12,000 labeled instances in total.

The decision for a 4,000-instance set: run the 4-cell merged grid at ±3 points per cell, and spend new capture effort on the merged pairs whose failure modes differ, because those are the cells worth splitting again once more data exists.

Split by capture session, not by frame

Images captured minutes apart on one sitting share fixture pixels, lamp spectrum and near-identical poses. A frame-random split sprinkles those near-duplicates across train and validation, and the score then measures how well the model remembers a session, not how it handles a new one. Give every capture session — one camera, one lens, one background, one lighting setup, one sitting — a single group label, and keep each session wholly on one side of any split.

Grouped K-fold applies the same rule inside development. K-fold cross-validation divides the data into k parts, each serving as validation once; the grouped variant keeps each session inside one part. With k = 5 you train five models, each held out from a fifth of the sessions, and report mean and spread. Five folds at 0.91, 0.89, 0.92, 0.88 and 0.90 mAP give a mean of 4.50 / 5 = 0.90 and a range of 0.88 to 0.92, which is a stable model. A fold at 0.78 against 0.88 to 0.92 elsewhere is a finding, not noise: that fold's sessions hold a condition the others do not, and the per-condition table names it.

Held-out sessions are also the honest rehearsal. Production feeds the model sessions it has never seen, and grouped validation measures exactly that.

The per-condition report and the aggregate trap

Freeze the operating point before scoring: the score cutoff and the IoU match level come from the deployed configuration, not from the test results.

The table below is an illustrative example of one production report: two lighting levels against three background groups, about 200 labeled instances per cell, scored at the deployed threshold with IoU 0.50.

The same predictions averaged across cells read recall 5.41 / 6 = 0.90 and mAP 5.19 / 6 = 0.87, shippable-looking numbers that hide two failing cells. Read by the verdict rule instead: a cell passes when measured recall minus 3.0 points clears the production threshold. Amber-light bare metal sits at 0.84 with a span of 0.81 to 0.87, below 0.90 with no overlap, which is a real failure with a real fix. When a cell fails, Interpret an object detection background confusion matrix is the next read: it separates missed detections from wrong-class and localization errors, which fail for different reasons.

Test on the hardware that runs in production

The runtime device is a condition axis like lighting: the same weights produce different scores on a desktop GPU, on the processor inside a smart camera, and on an exported runtime with quantization. Score the model that ships. For on-device inspection that means running the exported network on the camera itself and building the per-condition table from its output; if you export for the target, Non square YOLO training: image size, rect mode and ONNX export covers the input-size rules that survive an export. On embedded accelerators, first check whether the runtime supports every operator your exported model uses, because an unsupported operator silently changes what the device computes.

Smart cameras run the network on the device, so the camera belongs inside the test loop: capture the stratified set through the camera the line will use and read the metrics from the device itself. Two routes cover most lines, mono and color:

When inference runs on a PC, the camera is still the front end that sets the appearance the model sees, and a GigE global-shutter color area scan camera is the common choice; the model below is a 2.5 MP C-mount color camera:

The capture rule holds either way: the test set is shot with the production sensor and lens. A batch recorded on a phone or a bench camera screens the concept; only the production sensor measures the line.

When the usual answer is wrong

• One aggregate mAP on a mixed set. The table above is the counterexample: 0.90 overall with two cells below 0.85. Aggregate for reporting, verify per condition for decisions.
• Frame-random splits. If sessions land on both sides of the split, the score mostly measures memory of the sitting. Regroup by session and re-run before believing any number.
• Augmented test data. Darkening test images in software measures robustness to your darkening, not the low-light cell. The low-light cell is a lighting condition on the line; capture it there.
• Inferring from a similar condition. A new background under familiar lighting is exactly the cell the grid exists to measure. Add the cell; never extend a pass.
• Thresholds chosen late. A cutoff tuned on the test set reports the model you tuned, not the model you ship.

Checks before you trust the numbers

• Every shipping condition maps to a cell, and merges are labeled as merges in the report.
• Each kept cell holds at least 200 labeled instances, counted as instances rather than images.
• Session IDs were assigned before any split, and no session appears on both sides.
• The score cutoff and IoU level were frozen from the deployed configuration before scoring.
• Ground truth was audited, because one wrong label moves recall while every metric looks healthy; Trained YOLO model has no detections: fix labels first is the anatomy check when a cell fails against expectation.
• The same table was produced from the deployment device's output, not only from the training machine.
• Borderline cells were re-measured with more sessions rather than re-argued.

Next step

Pick the camera the test will run on and shoot the grid through it: the camera finder filters by interface, resolution and shutter, and the camera configurator walks the build step by step. For the surrounding methods — training data size, metric reading, deployment quirks — the systems guides collect the rest.