Training an image classification model on subjective labels

Some inspection calls have no number behind them. Whether a cover design, a painted finish or a plated surface looks appealing is a subjective judgment, and the only ground truth available is what your reviewers say. The usual move is to collect like and dislike labels and train a binary image classification model, hoping the inconsistency averages out. It does not: the model reproduces the labelers' habits as faithfully as their intent. This guide gives the sequence that holds up: measure agreement between labelers with Cohen's kappa before committing to a dataset, repair the disputed images against a written rubric, then ship the trained model as a ranked probability score instead of a pass/fail verdict, with the full arithmetic worked through.

What changes when the label is an opinion

For a defect classifier the label points at something measurable: a scratch wider than a limit, a missing character, a wrong color. The binary image classification for defect detection guide assumes that kind of ground truth. With a subjective quality, the label is the spec. There is nothing else to compare the model against, so system accuracy is capped by how consistently humans apply the label itself.

Two consequences follow. First, an accuracy ceiling: if two trained reviewers disagree on roughly a quarter of images, that disagreement is noise no network architecture removes, and a model that appears to beat it is usually memorizing one reviewer's idiosyncrasies. Second, bias transfer: a reviewer's private preferences — a color they dislike, a layout style they associate with low-end products — enter the weights as if they were physical facts. That makes annotation design as much a part of the engineering as the model choice. If you can dodge explicit labels entirely, anomaly methods do, and the anomaly detection without ground truth walkthrough covers that route. When you need a named, two-sided call — appeal, acceptability, style match — you need labels plus a way to audit them.

Measure agreement before you collect the full dataset

Cohen's kappa measures agreement between two labelers on the same images, corrected for the agreement that chance alone would produce; it ranges from -1 (systematic disagreement) to +1 (complete agreement) (Wikipedia's overview). Raw percent agreement flatters you whenever one class dominates: two reviewers who both say "like" most of the time agree a lot without sharing any standard beyond the base rate.

Read κ against three bands, and treat each band as an instruction:

• κ ≥ 0.6 — the labelers share a standard. Continue to full annotation and training.
• 0.4–0.6 — moderate agreement. The classes are real but the boundary is fuzzy. Adjudicate every disputed image with a third reviewer or a written rubric, then recompute.
• Below 0.4 — the two classes overlap so much that no model can out-vote the labelers' own inconsistency. Stop, and rewrite the rubric or the class split before spending another hour annotating.

Compute κ on the first 100 to 300 images, not after the full set is labeled. Repairing 200 labels costs an afternoon; repairing 10,000 costs the project its schedule.

Worked example: two labelers score 300 cover designs as like or dislike

Say a publisher wants to rank candidate cover designs by appeal, and two reviewers independently label a pilot batch of 300 covers, each with a binary like or dislike call. The counts below are the illustrative inputs, in images; the kappa formula and the three bands above are the fixed rules the decision runs on, and κ itself is dimensionless.

The arithmetic, step by step:

• Observed agreement: p_o = (150 + 80) / 300 = 230 / 300 = 0.767
• Reviewer A's shares: like (150 + 30) / 300 = 0.600, dislike 0.400. Reviewer B's shares: like (150 + 40) / 300 = 0.633, dislike 0.367.
• Expected agreement: p_e = 0.600 × 0.633 + 0.400 × 0.367 = 0.380 + 0.147 = 0.527
• Kappa: κ = (0.767 − 0.527) / (1 − 0.527) = 0.240 / 0.473 = 0.51

Decision: κ ≈ 0.51 lands in the moderate band. The 76.7 percent raw agreement looked shippable, but 52.7 points of it is what chance would produce given the 60/40 lean — the signal above chance is about half the raw number, and the reviewers disagreed outright on 70 of 300 covers (23 percent). Do not train on these labels as they stand. Adjudicate the 70 disputed covers with a third reviewer or the rubric, fold the settled calls back in, and recompute: proceed to training once κ reaches 0.6, and if it stays below 0.4 after a genuine rubric pass, the like/dislike split itself is the defect to fix.

Repair the labels before you tune the model

The order of operations is rubric, adjudication, then training. In practice that means:

• Write the rubric in images, not adjectives. "Clean" and "appealing" drift between mornings; ten pinned anchor images per class at the boundary do not. Every rule gets one sentence and one example image.
• Adjudicate disputes, do not average them. A third reviewer or a defined committee call settles each disputed image. Those settled disputes are gold: keep them as the hard validation set, because they sit exactly where the deployed model will be judged.
• Quarantine the middle. If annotators keep reaching for a third bucket ("not sure"), record it rather than forcing it silently. Either force the binary call with the rubric, or hold the middle band out of training and let the score sort it later.
• Compare per-annotator rates. If one reviewer's like-rate is 75 percent and another's is 45 percent on the same batch, they are applying different standards. Recalibrate in a joint session before trusting either.
• Design the negative class on purpose. What counts as a plain "dislike" or a well-chosen background steers the boundary as much as the positives do; the negative samples and false positives guide shows the effect on the error profile.

Ship a score, not a verdict

Train on the adjudicated labels and preserve the model's probability output. A subjective classifier earns its keep as a ranker: sort candidates by score and act on a slice — the top 10 percent for a design shortlist, the bottom decile for an automatic reject — rather than forcing one threshold on the whole distribution. Band the extremes only where the score is confident, for example above 0.9 or below 0.1.

Set any operating threshold on the validation score distribution, not on intuition: rank the held-out adjudicated set and choose the cut where precision on the preferred class meets your tolerance for letting a wrong call through.

Set expectations with the kappa you measured. With κ = 0.51, a perfect model still agrees with any single reviewer only about 77 percent of the time, because 23 percent of the time the reviewers themselves split. Quote accuracy against the adjudicated test set, never against raw training labels — the raw number measures the committee, not the model.

Deployment keeps its usual shape. Send the score downstream instead of the image: the vision system to PLC pass-fail pattern carries over with the score in the boolean's place, and the AI versus rule-based smart camera comparison helps you decide where inference should run.

Keep the capture fixed so taste is the only variable

Book covers arrive as flat files with no capture noise. On a production line the images come from a camera, and every capture variable — exposure, white balance, working distance, light angle — adds disagreement that has nothing to do with the judgment under study. Lock all of it: same lens and distance, manual exposure and white balance, a diffuse light at a fixed position, and a global shutter wherever parts move.

For the capture station, a 2.5 MP color global-shutter GigE camera with a C-mount, such as the MV-CU020-90GC, holds exposure and framing identical across the entire training set. Choose the lens for the field of view the part needs; the shop's area scan cameras span the common sensor formats, and the camera configurator pairs body and lens in one pass.

At deployment, production images have to match the conditions the labels were made under. A smart camera runs the inspection on the device with no PC in the loop, and one with the light built in removes the largest capture variable altogether: the LS-SMC-0063 combines a 1.6 MP sensor, a fixed 16 mm lens and an integrated white light over GigE. The smart cameras category and the cameras finder filter the rest by resolution, mount and interface.

When the subjective approach is the wrong tool

• Kappa stays below 0.4 after a genuine rubric pass. The decision needs a specification, not a model: write an objective measurement (gloss units, color difference, a dimension) and classify on that.
• The score would gate production on a regulated or contractual criterion. A committee's taste is not validation evidence. Keep the subjective score advisory — ranking, design feedback — and pass/fail on measurements.
• One annotator supplies every label. You are modeling one person's taste with no consistency check at all; the single-rater question in the FAQ gives the partial workaround.
• The "subjective" label is tracking a physical defect reviewers happen to dislike. Measure the defect directly; the subjective route is a noisier path to the same answer.

Pre-flight checklist

Labels

• Two or more independent labelers on every image, with κ computed on the first 100 to 300 images — percent agreement alone hides chance agreement.
• A written rubric with pinned boundary images, and every disputed call adjudicated and kept as hard validation cases.
• Per-annotator like-rates compared before training, so one reviewer's bias is not trained in as fact.

Model

• Binary classes with the probability output preserved for ranking, and thresholds read off the validation score distribution.
• Evaluation against the adjudicated set, with the kappa value quoted next to accuracy.

Deployment

• Capture variables locked — distance, exposure, white balance, light position — so production images match training images.
• Score, not image, sent downstream, with the operating band re-checked whenever the rubric or the reviewer group changes.

The kappa check costs an afternoon and prevents the classic failure: a polished model that reproduces one reviewer's mood. The hardware links above size the capture side; for anything stranger — mixed lighting, moving parts, a judging panel with a deadline — send the part and the decision to the contact page and we will spec the camera, lens and light around it. The applications topic page collects more build sequences like this one.