Image stitching for machine vision: offsets, matching and merge

One camera rarely covers a wide part. A 624 mm panel inspected at 0.06 mm/px needs 10,400 pixels across. That means either a specialist large-format sensor or several standard cameras stitched into one image. Stitching earns its place when the arithmetic points that way: trigger two to four overlapping camera fields, copy each frame onto one large canvas at a computed offset, and refine every offset by matching the shared strip between neighbours. This guide gives the numbers that make the mosaic trustworthy — the 20% overlap default, the canvas and camera-pitch formulas with a three-camera worked example, search-window sizing, seam handling for measurement versus display, and the memory, bandwidth and cycle-time budget. It is written for engineers wiring a multi-camera cell around a vision controller.

How much overlap do stitched machine-vision images need?

Leave 20% of the field width as overlap between adjacent frames: at a 4000 px camera width that is 800 px of shared scene. Drop to 10% only with a machined fixture and rich texture. Raise it to 25-30% when stage repeatability, thermal drift or lens distortion eats the margin.

The overlap is not wasted resolution; it is the matcher's working material. The seam between two frames lives inside this shared strip, so the strip must hold a template with genuine texture plus a search window wide enough for the worst-case mechanical error.

For the 4000 px camera used throughout this guide:

• 20% default: 0.20 × 4000 = 800 px of shared scene between neighbours.
• 10% floor: a 200 × 200 px template, a ±32 px search and a 100 px safety margin need 200 + 64 + 100 ≈ 400 px, which is 10% of 4000 px. Below that, the match fails on the first vibration.
• 25-30% band: when the two lenses distort differently, the template should sit away from the field edges where distortion residuals grow, so give it that room.

Fix the mechanical pitch early and let the matcher absorb only the residual. In the worked example the pitch lands on 192 mm; mounting plates like round numbers.

How do you work out how many cameras a wide part needs?

Total pixels needed are part width divided by pixel size, and the camera count follows from the overlap. A 624 mm part at 0.06 mm/px needs 10,400 px, and at 20% overlap three 4000 px cameras deliver exactly 10,400 px, so the answer is three, not two.

Work from the feature, not from the sensor:

1. Required pixel size: to measure a 0.3 mm feature with 5 px across it, 0.3 mm / 5 = 0.06 mm/px.
2. Pixels needed across the part: 624 mm / 0.06 mm/px = 10,400 px.
3. Camera count at 20% overlap with w = 4000 px: each added camera contributes 4000 − 800 = 3200 px, so N = ceil((10,400 − 4000) / 3200) + 1 = 3.

Three cameras deliver 4000 + 2 × 3200 = 10,400 px, exactly the budget, at 180 mm of height per field (3000 px × 0.06 mm/px). The single-sensor alternative is a 10,400 × 3000 ≈ 31 MP device with optics that still fill 624 mm; three standard sensors on ordinary lenses usually win on working distance and cost. Scan sensor options by resolution in the area scan cameras category or the camera finder, and the camera resolution calculator runs the two divisions above for any part size.

The example cell uses three of the 12 MP USB3 model below, which prints a 1/1.7" rolling shutter sensor and 4000 × 3000 resolution:

How do you size the stitched canvas and place the nominal offsets?

Canvas width is N × camera width minus one overlap per seam: 3 × 4000 − 2 × 800 = 10,400 px, which is 624 mm at 0.06 mm/px, and camera k starts at x = k × 3200 px, so the mechanical pitch between camera centres is 192 mm.

Worked numbers: canvas = 3 × 4000 − 2 × 800 = 10,400 px; offsets 0, 3200, 6400 px; physical pitch 3200 px × 0.06 mm/px = 192 mm; canvas footprint 624 × 180 mm. If the part is taller than one field, repeat the geometry in a second row, or move the part under one camera row on a stage and stitch row by row with encoder-derived offsets.

Set the nominal offsets from the mechanical pitch, never from the first grabbed image. The matcher then corrects a small residual instead of hunting the whole field, which keeps cycle time flat part after part. When the mosaic feeds a robot or stage coordinate system, give every camera one shared coordinate frame from the start — the set-up is worked through in three-image stitching with a shared coordinate frame.

What are the steps to build the stitched image?

Build the mosaic in eight steps: size the canvas, compute the nominal offsets, create the canvas image, copy each frame in, pattern-match every seam, apply the refined offsets, write measurement and display outputs, then prove the result on a calibrated artifact before the cell runs.

1. Fix the geometry. Overlap 20%, pitch 192 mm, canvas 10,400 × 3000 px, nominal offsets 0 / 3200 / 6400 px.
2. Match the cameras. Same lens family, focus and f-number set on a target; identical exposure and gain; one flat-field reference per camera.
3. Create the canvas. One 8-bit mono image, 10,400 × 3000 px = 31.2 MB.
4. Copy each frame in at its nominal offset. Some libraries combine this with step 3 in a single call — the IMAQ ImageToImage 2 function is the classic example; elsewhere it is a canvas allocation followed by a copy with a destination offset.
5. Pattern-match every seam. Template from the left frame's overlap strip, bounded search in the right frame, sub-pixel interpolation on.
6. Apply the residuals. Re-copy or shift each frame by the matched Δx, Δy, which stays within the ±32 px window by design.
7. Write the outputs. A measurement mosaic split at each matched seam, plus a feathered display copy.
8. Prove it. Image a calibrated artifact spanning two seams, then log seam offsets over 30 parts before release.

If match boxes draw in the wrong place or never appear, the geometry is usually innocent and the overlay path is guilty — pattern matching overlay not showing: draw the results on the image walks that fault end to end.

How do you refine each seam to sub-pixel accuracy?

Refine each nominal offset by matching a template from one frame's overlap strip inside a bounded search window in its neighbour: the score peak, interpolated, gives a sub-pixel shift. On textured, evenly lit parts expect 0.1-0.3 px repeatability — here a 200 × 200 px template in a ±32 px window.

The standard tool is template matching: the matcher scores a normalized correlation of the template against every window position, then interpolates the score peak for the sub-pixel result. Size the window from mechanics, not hope. Placement tolerance ±0.6 mm at 0.06 mm/px is ±10 px; doubling it for thermal drift and trigger jitter gives 20 px, rounded up to a 32 px half-search.

Keep the window that tight because match cost grows with template area × search area: a 264 × 264 px window gives 65 × 65 = 4,225 candidate positions, about 170 million correlation operations, which pyramidal matchers turn into tens of milliseconds. A loose window is the usual root of a slow update rate, analysed in pattern matching low fps: shrink the search to speed the update rate.

Place the template on the strongest, least repetitive texture in the strip: a grid of identical holes returns many near-equal peaks and the match walks between them. Score every production match and hold acceptance at 0.7 normalized correlation on textured industrial surfaces; below it, keep the nominal offset, flag the part and alarm — an unmatched seam is never merged silently. The matcher-side tuning, from score weighting to interpolation mode, is detailed in pattern search sensitivity vs accuracy: tuning position detection.

Should you blend the seam or keep one image authoritative?

For measurement mosaics do not blend: split at the matched seam so every output pixel comes from exactly one camera, and blend only a separate display copy. Where the image is for humans, a 100 px linear feather inside the 800 px overlap hides the join without touching the data.

For measurement, split at the matched seam: output columns left of it come from the left camera, the rest from the right. Every pixel keeps exactly one origin, so defect counts and edge positions cannot double-count ghosted content.

For display, feather inside the overlap. With alpha = (x − seam start) / 100 px, the output pixel is (1 − alpha) × left + alpha × right; a 100 px blend inside the 800 px overlap leaves 700 px of untouched matched material for spot checks.

Both outputs need the cameras brightness-matched first: identical exposure and gain, then per-camera flat-field against a uniform white target. Run that before matching too, because a shading step across the strip biases the correlation and drags the peak.

What does stitching cost in memory, bandwidth and cycle time?

The worked canvas is 10,400 × 3000 px = 31.2 million pixels, 31.2 MB as 8-bit mono — small for any current controller. The real costs are stream bandwidth while capturing and CPU while matching: budget roughly 200 ms per part end to end in this three-camera example.

Memory first. The canvas is 31.2 MB as 8-bit mono and 93.6 MB in colour — trivial for a current controller. Scale it up and it bites: a 16,000 × 30,000 px colour mosaic is 16,000 × 30,000 × 3 B = 1.44 GB, near enough to a process ceiling that allocation discipline matters; stitching large images past the 2 GB memory limit handles that regime.

Bandwidth second. One 4000 × 3000 mono frame is 12 MB, so 30 fps over USB3 is 360 MB/s — at the practical USB3 ceiling of roughly 350-400 MB/s. That is fine for a triggered frame per part and tight for a continuous stream. Over GigE the same stream divides to about 9 fps, which is why 12 MP GigE models sit near that frame rate.

Cycle time third. Trigger all three cameras in parallel on one star-wired line, then add one exposure plus one frame transfer — about 34 ms on USB3, about 111 ms at 9 fps GigE — two matches at roughly 20 ms each, and three copies at a few milliseconds: call it 90 ms on USB3, 160 ms on GigE, and budget 200 ms. A single thread sustains 1000 / 200 = 5 parts per second; one merge thread per seam raises the ceiling with the core count.

Which cameras and controller fit a stitching cell?

Pick cameras by the pixel budget, then size the controller by ports. The three-camera GigE example needs one camera port per camera plus one port for the PLC, so a controller exposing 5+ LAN ports covers the layout with room to spare and no network switch in the image path.

For a stop-and-shoot cell a rolling shutter sensor is fine, because nothing moves during exposure; specify global shutter only when the part travels under strobe light. Keep every stitching camera the same model, so one calibration and one mm/px value serve all offsets. Continuous web is a different machine: stitched stills cannot follow it, so move up to a line scan camera or the area scan line scan mode covered in area scan camera line scan mode for moving-web inspection.

The vision controllers category lists these machines with their LAN port count, USB ports, memory and built-in light channels, so you can filter straight to the port layout your cell needs. Three that each expose 5+ LAN ports, with built-in light channels for the strobe:

Values as printed in each product's data sheet; open a part for its drawing and full specifications.

When a larger sensor can swallow a seam, this 20 MP mono USB3.0 camera with a 1" sensor removes one camera and one match step from the example:

What checks keep a stitched mosaic trustworthy in production?

Prove geometry and brightness separately, with numbers: a calibrated artifact spanning two seams must read within ±0.12 mm — 2 px at 0.06 mm/px — and logged seam offsets over 30 parts must stay within ±2 px, both tighter than the 0.3 mm feature the mosaic exists to resolve.

Run these before release and after any mechanical or optical change:

• Scale: image a glass ruler spanning two seams; the known span must read within ±0.12 mm.
• Repeatability: log per-seam Δx, Δy over 30 cycles; a spread beyond ±2 px means a loose mount or thermal drift — re-torque, re-home, re-check.
• Brightness: white-reference every camera per shift; histogram means within a few gray levels across cameras before matching starts.
• Match health: keep the production score distribution at or above 0.7; a slow slide precedes defocus, dust or lamp ageing, and cleaning a lens beats scrapping a batch.
• Feature placement: dimensions toleranced tighter than the 0.3 mm feature size are measured away from the 800 px seam bands.

Size the cell from the feature, leave 20% overlap, and let the matcher own the last few pixels. To have the ports, memory and light channels checked against your own part, start at the cell configurator, or send the drawing through the contact page and we will run the pixel budget with you. More applied cell layouts sit on the applications guides topic.