Multispectral object detection with RGB pretrained weights

You point a YOLO11n run at a 10-channel multispectral dataset, load the 3-channel RGB checkpoint, and the training just runs: no error, and the log prints Transferred 499/499 items from pretrained weights. Nothing is broken. The trainer reads a channel count from your dataset YAML, rebuilds the first convolution to that width, and tiles the pretrained RGB filters across the extra planes. This guide is for engineers wiring band-stacked data into a detector: RGB plus NIR, dedicated multispectral sets, or band subsets selected from hyperspectral capture. You will set the count correctly, prove it from the printed summary, and decide with numbers whether tiled weights or a from-scratch run deserve the GPU time.

Why the channel mismatch never throws an error

The band count comes from one key: channels in the dataset YAML you pass to the train call. Absent, it defaults to 3. Set channels: 10 and the model builder shapes layer 0 — the stride-2 stem convolution — to take ten planes per pixel before a single weight loads.

When you start from a .pt checkpoint, the loader does not drop the stem or refuse the run. It rebuilds layer 0 at the new width and fills the wider tensor by repeating the pretrained RGB filters until every band has one. Band 4 starts as a copy of a pretrained filter, band 7 as another. Every layer after the stem keeps both its shape and its checkpoint values, because their sizes follow layer 0's 16 outputs, not its 10 inputs.

That is also why the model YAML is the wrong place to fix anything. Whether you load yolo11n.pt or build from yolo11.yaml, the stem's input width comes from the dataset key. An edited Conv line in the model YAML either does nothing or plants a second source of truth that loses to the dataset.

The proof lands in the log before epoch 1: for ten channels, layer 0 prints as Conv [10, 16, 3, 2] — 10 inputs, 16 output filters, 3×3 kernel, stride 2 — with 1472 parameters, inside a summary of 181 layers and 2,625,088 parameters.

Set channels in the dataset YAML, then read the proof

The whole change is one key plus one habit: confirm the stem line before a run gets long.

1. Count the planes your loader emits per pixel. RGB is 3; RGB plus NIR from one sensor is 4; a 10-band subset chosen from a hyperspectral cube is 10.
2. Add the key at the top level of the dataset YAML, beside the existing path, train, val and names keys:

# top level of the dataset YAML (for example coco8-multispectral.yaml)
channels: 10   # planes per pixel; 3 is assumed when the key is absent

3. Load the checkpoint and train exactly as you would for RGB:

from ultralytics import YOLO

model = YOLO("yolo11n.pt")  # 3-channel checkpoint, loaded as usual
results = model.train(data="coco8-multispectral.yaml", epochs=100, imgsz=640)

4. Read the stem line in the summary the trainer prints. The first argument must be your band count — Conv [10, 16, 3, 2] for ten channels — and layer 0 must show 1472 parameters for yolo11n. A 3 in that position means the channels key never reached the trainer, usually because the code passes a different dataset YAML than the one you edited.
5. Start the control run in parallel, same data, epochs and seed, with the architecture file instead of the checkpoint:

model = YOLO("yolo11.yaml")  # architecture only: random weights everywhere

The .yaml-for-.pt swap is the entire change; the dataset key still drives the stem.

6. Zero the color augmentations for band stacks. The printed defaults hsv_h=0.015, hsv_s=0.7 and hsv_v=0.4 act on hue, saturation and value — a 3-channel color space. Keep the geometric set (mosaic, scale, translate, flip) at its defaults and reintroduce color gains only if an ablation shows a gain on your bands.

What Transferred 499/499 does and does not tell you

The line is a load report, not a shape check. It counts the parameter tensors that received values from the checkpoint, and layer 0 counts because its rebuilt tensor was filled — with tiled copies of the RGB filters, not with random numbers. Expect the same 499/499 on a 3-channel and a 10-channel run: the message cannot distinguish them, and it cannot judge whether tiling is a good initialization for your bands. The only instrument that answers that is the control run's validation mAP on a fixed split at a fixed seed.

Worked example: what 10 channels cost a 2.6-million-parameter yolo11n

The inputs are printed by the trainer itself for the 10-channel run: layer 0 is Conv [10, 16, 3, 2] with 1472 parameters, and the network totals 181 layers and 2,625,088 parameters. The rule: a convolution's weights hold out_ch × in_ch × k × k values, and this trainer's Conv module ships without a conv bias but with a following BatchNorm that adds 2 × out_ch.

Check the rule against the printed value first: 16 × 10 × 3 × 3 = 1440 weights, plus 2 × 16 = 32 BatchNorm parameters, so 1440 + 32 = 1472. That matches the printed 1472 and confirms both the formula and that the ten channels reached the builder.

The same stem at the default 3 channels: 16 × 3 × 3 × 3 = 432 weights, plus 32 = 464 parameters. Seven extra bands therefore add 1472 − 464 = 1008 parameters. Cross-check from the other side: each band contributes 16 filters × 3 × 3 = 144 parameters, and 7 × 144 = 1008. In float32 that is 1008 × 4 B = 4032 B, about 3.9 KiB.

The 3-channel twin of this network would print 2,625,088 − 1008 = 2,624,080 parameters, so the seven bands grow the model by 1008 / 2,625,088 = 0.038 %.

Decision: capacity is not the constraint. Even a 30-band stack adds 27 × 144 = 3888 parameters, under 0.15 % of the network, so widening the stem is never the cost that matters. Spend the next GPU hour on the tiled-versus-scratch control from step 5, not on the architecture.

Tiled filters or scratch: run both and keep the winner

Tiling leaves 2,623,616 of the 2,625,088 parameters — everything after layer 0 — starting from pretrained values. Those layers already carry edges, textures and part shapes at every scale, so your bands inherit them from epoch 1.

As a working rule, tiled weights win when the labeled set is small — under about 2,000 instances — and the bands behave like colors: visible bands, or NIR planes with continuous texture. Scratch wins when the band physics sit far from RGB, as with strong shortwave-infrared contrast or narrowband illumination; when instances number in the tens of thousands; and when you can afford the extra epochs.

Run both whenever the second run costs less than a day, and keep the higher validation mAP at the same seed and split — the fixed-split, repeated-seed discipline behind the spectral imaging material classification validation test. If scratch loses by less than about 1 mAP50-95 point, take the tiled model for its faster convergence. If it wins by more, audit normalization and augmentation before concluding that transfer does not help your bands.

Tiling is the free baseline, not the ceiling. Published multispectral detectors add structure to the stem instead: a 2025 IEEE design integrates Sobel and Laplacian edge operators into the convolution stream to sharpen target contours before fusion and detection (multispectral object detection with an edge-enhanced module). If your tiled baseline plateaus, band fusion in the stem is the next lever, not a bigger head.

For very small datasets there is a middle path: the trainer's freeze option, printed as freeze=None in the args dump, locks chosen layers, so you can hold the pretrained trunk and train the tiled stem plus the head first, then release.

Pipeline checks before you blame the weights

Run these five checks on one batch and a few hundred frames before reading anything into a weak mAP.

• Band order. Print per-channel means for one sample and compare them with the sensor's band list; a swapped pair reads as a color cast and costs accuracy silently. The RGB version of this failure is in red and blue channels swapped: why RGB images look wrong.
• Bit depth and scale. Most capture paths deliver 8-bit planes, so a 10-bit feed dropped into an 8-bit assumption lands dark and low-contrast. Convert every band to one consistent scale before training; the failure mode is explained in ADC bit depth vs pixel format: why 10-bit sensors ship 8-bit images.
• Per-band statistics. Take the mean and standard deviation of every band over a few hundred frames. A band with a standard deviation near zero is dead or duplicated: drop it and lower channels to match. The input-audit habit for color models is the color normalization training pipeline.
• What the loader emits. Plane count, dtype and scale on one real batch — validate frames first, before the first epoch, not from file names.
• Band selection before the detector. Hyperspectral capture hands you hundreds of bands; the detector wants a curated subset. At imgsz 640, one band in a batch of 16 costs 16 × 640 × 640 × 4 B ≈ 26 MB in float32, so 10 bands run about 0.26 GB per batch while 1200 bands run about 31.5 GB. Choose planes with the hyperspectral imaging guide, then set channels to the count you kept.

Cameras that feed a multispectral detector

Capture decides how many channels you even have. For the RGB baseline and classic 3-channel lines, a color area scan camera is the front end; the area scan cameras page filters the range by interface, resolution and shutter.

The Hikrobot MV-CS200-10GC is a 20 MP color GigE camera with a 1-inch sensor. It covers RGB control capture with room to crop.

For band-rich work, a hyperspectral camera replaces filter wheels and per-band optics. The LS-HSC-0004 records 400-1000 nm with 1200 bands over USB3. You then select the ten or twelve informative planes for the detector from its cube.

Both paths feed the training pipeline unchanged, so choose by band count rather than pixel count. The hyperspectral cameras page lists the band-rich sensors, and the camera finder filters both families side by side. To size the interface, trigger and storage around the camera, start the configurator at the camera step.

Common mistakes

Six mistakes account for most wasted multispectral runs:

1. Editing the first Conv line in the model YAML. The dataset's channels key wins in every path, so the edit does nothing for .pt loads and only misleads the next reader for .yaml loads.
2. Editing one YAML while passing another. The args dump at the top of the log prints the data= path that actually ran; read it once per run.
3. Reading 499/499 as proof that pretrained weights fit your bands. The line reports loading, not suitability.
4. Leaving the hsv_* gains at their RGB defaults on a 5-to-10-band stack.
5. Mixing bit depths across planes, so half the stack sits dark relative to the rest.
6. Stacking unaligned cameras. RGB from one sensor plus NIR from another needs per-pixel registration first; the NIR-to-RGB alignment walkthrough is the starting point.

Once the stem line, the split and the band audit are clean, the remaining variance is data and initialization — and both are settled by the control run. If the open question is capture rather than training, start the system configurator at the camera step, or contact the applications team with your band list, target sizes and lighting plan.