Cameras & sensorsExplainer

Multispectral camera band registration: align channels for fusion

Why multispectral bands land on different pixels, what a sensor offset becomes in pixels and millimetres, and how to register bands for per-pixel fusion.

LumaSpec Vision9 min read Share on LinkedIn

On this page 10 sections
  1. 01Where band-to-band offset comes from
  2. 02The registration model
  3. 03What an offset means in pixels and on the part
  4. 04What each architecture needs
  5. 05Calibrating on a target
  6. 06What invalidates the calibration
  7. 07When you can skip registration
  8. 08Building a registered rig from stock parts
  9. 09The decision, per set-up
  10. 10Questions

A multispectral camera returns several images of one scene in different wavelength bands, but "same scene" is not "same pixels". Each band lands on its own sensor or its own moment in time, and mechanical tolerances leave the images offset by anything from a fraction of a pixel to tens of pixels. Fuse bands per pixel — spectral ratios, defect overlays, a model that takes all bands as input channels — and that offset becomes ghost edges and wrong readings. This page shows where band-to-band offset comes from, how to turn a sensor tolerance into pixels and millimetres on the part, and what registration work each camera architecture actually needs.

Where band-to-band offset comes from

Three architectures deliver multiple bands. A prism multi-sensor camera sends one optical path through a dichroic prism that splits the spectrum onto two or three sensors bolted to the same block: bands are simultaneous and co-boresighted by design, with whatever residual offset the sensor mounts leave. Separate cameras behind a beam-splitter cube are simultaneous too, but each carries its own mount tolerances, so alignment is entirely yours. A filter wheel, or narrowband lights strobed in sequence on one mono camera, uses the same pixels for every band — geometry matches by construction, at the price of sequential capture that only suits a still scene.

If the bands themselves are not settled yet, the SWIR, NIR and visible band comparison covers that choice; here the bands are fixed and the question is how well they line up.

One more source: lateral chromatic aberration. Even through a single lens, each wavelength refracts slightly differently, so the band shift grows from near zero at the center to its worst in the corners, and lenses built for wavelengths outside the visible band enlarge the effect again. The offset is a field, not a constant, unless you work in a small centered ROI.

The registration model

Registration maps every band onto the pixel grid of a reference band. Start with the simplest model that fits and add terms only when residuals demand them:

x' = a·x + b·y + tx y' = c·x + d·y + ty

A pure translation (a = d = 1, b = c = 0) suits a prism camera worked near the field center, where the residual is close to a constant shift. The full six-coefficient affine adds rotation, scale and shear — what a two-camera rig needs, since no two mounted cameras share pixel-exact magnification and roll. When bands show different distortion at wide fields or strong lateral color, a second-order polynomial with 12 coefficients is the next step. Fit by least squares on matched target points, then judge the fit by RMS and maximum residual, never by eye.

The identical problem of one rig carrying sensors that must agree is walked through for a depth-plus-RGB pair in fixing a depth-to-RGB registration offset in pixels; the diagnosis carries over.

What an offset means in pixels and on the part

Mechanical tolerance is quoted in millimetres at the sensor; your inspection tolerance lives on the part. Convert before deciding whether a rig is acceptable as delivered.

Example: a 5 MP-class sensor with 2448 × 2048 pixels at a 3.45 µm pitch, a 25 mm lens, a 300 mm working distance.

  • Sensor width: 2448 × 3.45 µm = 8.45 mm
  • Magnification: m = f / (WD − f) = 25 / (300 − 25) = 0.091
  • Field of view: 8.45 mm / 0.091 = 93 mm wide, so 93 × 78 mm
  • Object-side pixel: 3.45 µm / 0.091 = 0.038 mm

Now give two bands a 0.05 mm relative sensor offset:

  • Offset in pixels: 0.05 mm / 0.00345 mm = 14.5 px
  • Offset on the part: 14.5 px × 0.038 mm = 0.55 mm

One 50 µm mounting tolerance moves one band 14 pixels against the other: an edge in the NIR image lands 0.55 mm from the same edge in color, which any per-pixel fusion reads as a false feature.

The rule runs the other way too: keep band offset under about one tenth of the smallest feature you fuse. A 0.2 mm defect at 0.038 mm per pixel allows 0.02 mm, or 0.5 px — sub-pixel registration is mandatory. A 5 mm blob tolerates about 13 px, which a factory-aligned prism camera may meet without software. After a successful fit, residuals of 0.2 px mean 0.2 × 3.45 µm = 0.69 µm at the sensor and 7.6 µm on the part — measurement grade.

m = f / (WD − f) object pixel (mm) = pixel pitch (mm) / m offset (px) = sensor offset (mm) / pixel pitch (mm) offset on part (mm) = offset (px) × object pixel (mm)
Lens focal length, field of view and working distance
Units
Solve for
Focal length
– mm
Field of view
– mm
Working distance
– mm
Magnification
– x
Angle of view
– °
Pixel on the part
– mm

Thin-lens model: m = sensor width / field of view, f = WD × m / (1 + m). Accurate to a few percent above about 150 mm; check the lens's minimum object distance.

Find lenses with this focal length Open the full calculator

What each architecture needs

Set-up Capture Raw offset Registration work
Prism multi-sensor camera simultaneous, one lens small, factory-set, largest in corners verify on a target once; fuse when the residual beats your pixel tolerance
Filter wheel or strobed bands, static part sequential, one sensor zero by construction none while the scene stays still
Beam splitter plus separate cameras simultaneous tens of pixels full affine fit; re-fit after every optical change

The last row's number is not pessimism: half a degree of roll between two mounted cameras across a 2,000-pixel field is already tan(0.5°) × 2000 ≈ 17 px, before any translation. The first row's demand is discipline: measure the delivered alignment, do not assume it.

Calibrating on a target

One capture session per optical configuration:

  1. Place a planar checkerboard or dot-grid target that fills the field, corners included — lateral color concentrates its error there.
  2. Trigger all bands on one command with the scene static; per-band exposure and gain may differ, geometry does not care.
  3. Detect corners in each band, match them to the reference band, and fit the model over the full set of matches spread across the field.
  4. Compare RMS and maximum residual with the pixel tolerance derived from your feature size; add polynomial terms only if the affine fit stays above it.
  5. Resample with bilinear or bicubic interpolation — nearest-neighbour re-creates one-pixel steps — then verify the overlay on a real part with known edges.

What invalidates the calibration

Refocus, zoom, a working-distance change or any lens or filter swap changes the geometry, so re-fit. A bandpass filter seated at a slight angle steers its own band by pixels, so a new filter set means a new calibration. A magnification change leaves a mechanical offset the same in pixels but scales what it means on the part: halve the object pixel and the 14.5 px error from the example becomes 14.5 × 0.019 mm ≈ 0.28 mm. Drift from temperature and vibration is checked the same way as everything here: re-run the target, compare the fit against the baseline, re-fit when the residual moves.

When you can skip registration

Three cases need none. Per-band independent inspection — a presence check in color, a defect rule in one NIR band — never compares pixels across bands. Coarse band ratios over a generous ROI work unregistered as long as the offset stays small against the ROI. And a still scene captured band by band is registered by construction.

When the bands do feed one fused image — agriculture runs on per-pixel band indices, and multispectral camera bands for crop versus weed shows the band-selection side of that workflow — registration comes before the index math. For tens of contiguous bands rather than two to four, a pushbroom hyperspectral camera builds its cube on a single detector, so spatial alignment is inherent.

Building a registered rig from stock parts

If sequential capture fits your part flow, the simplest registered system is one mono area scan camera behind a filter wheel or under strobed narrowband lights: geometry matches by construction and the effort goes into lighting, not alignment.

If bands must be simultaneous, build a twin-camera head: a beam-splitter cube, the same lens model on both ports — a 35 mm, 2/3", 6 MP C-mount lens on each — and both cameras triggered from one hardware line, the pattern in triggering two cameras from one frame-grabber input. Identical lenses keep the distortion curves identical, so one affine model fits both bands without polynomial padding.

Both cameras below are mono sensors on USB3:

Area scan cameras12 MP USB3 area scan camera, IMX226, 1/1.7" rolling shutter, 30 fps, 4000 × 3000 resolution (LS-CMR-0166)LS-CMR-0166
Sensor format
1/1.7"
Interface
USB3
$560.00View part
Area scan camerasHikrobot MV-CT0MU0060/R/NN/NN 6 MP Area Scan Camera, 3rd Gen., USB3.0, Mono, BASEMV-CT0MU0060/R/NN/NN
Sensor format
1/1.8"
Lens mount
C-mount
$969.00View part

Browse the area scan cameras range or the camera finder, pair them in the machine vision lenses category or the lens finder, and let the camera step of the configurator assemble the chain.

The decision, per set-up

  • Prism multi-sensor camera: capture the target once, fit a translation model, fuse directly when the maximum residual sits under your pixel tolerance — under 0.5 px for the 0.2 mm defect case above.
  • Filter wheel or strobed bands on a static part: skip registration; your only enemy is motion between captures.
  • Two cameras behind a beam splitter: affine registration is part of the build. Fit it on the target, expect residuals under your tolerance in one session, and re-fit after any optical change.

If a fused multispectral image is the requirement and the architecture is undecided, start from the cameras topic page or send the part, its feature sizes and the bands you need through the contact page, and we will size the registration work with the optics.

Questions

Can I register bands that were captured with different exposure or gain settings?

Yes. Registration matches geometric features such as checkerboard corners and dot centers, and a corner is a corner in a dark or a bright image as long as it clears the noise floor. Do the radiometric work, flat-field and reflectance normalization, after resampling so the interpolation weights are not applied to your measurements twice.

How often should a prism multispectral camera be re-verified?

Run the target check after any refocus, working-distance change, lens or filter swap, and after any mechanical shock. Between events, a weekly check is a sensible starting cadence: fit the model, compare the maximum residual against the baseline, and stretch to monthly once several checks come back unchanged. Stop stretching if residuals drift by more than roughly 0.2 px.

How many target points does an affine band registration need?

Three non-collinear points solve the six affine coefficients, but never fit with exactly three: use 20 to 50 corners spread over the whole field, corners included, so the residuals reveal distortion differences between bands. If the RMS residual stays above your tolerance, move to a second-order polynomial with 12 coefficients and confirm the improvement is not just noise fitting.

Do I need band registration if I only compute band ratios over a large ROI?

Usually not. A ratio of mean intensities over the same ROI tolerates a shift as long as the offset is small against the ROI and no feature sits on its border. Keep the offset under about a tenth of the ROI width and you can skip per-pixel registration entirely for coarse indices; register only when ROI edges cut through features.

Does band misregistration hurt a CNN that takes all bands as input channels?

Yes. A network treats its input channels as spatially aligned, so an offset band teaches it ghost edges and reduced contrast at every boundary, and the error grows with the offset in pixels. Register the bands before training and apply the same transform at inference; a model trained on misaligned data is not repaired by aligning only the live images.