Match the same object across two cameras: re-ID after detection

You detect objects in two camera views and need to know which detection in the first view is the same physical part as a detection in the second. Keypoint matching such as SIFT is the obvious first try, and it fails the moment the viewpoints diverge, which is why it only looked good while the cameras sat almost side by side. The working method is object re-identification: one detector on both views, a CNN embedding per crop, cosine similarity between embeddings, a threshold near 0.8, and a one-to-one assignment step. This guide gives the formulas, the field-of-view arithmetic that decides where the cameras mount, the labeled-pair count that sets the threshold, and the failure cases to check before you trust a match.

Why does SIFT fail when the cameras view the part from different angles?

SIFT is invariant to scale and rotation but not to viewpoint: once the two viewing directions differ by more than roughly 20 to 30 degrees, too few keypoints land on the same physical structures and the match count sinks into noise.

The scale-invariant feature transform describes small local gradient neighborhoods, and its invariance covers in-plane changes: zoom and rotation of the image. A second camera pointing from another direction changes those neighborhoods wholesale. Edges shorten, shading flips, whole faces of the part appear or disappear. Two cameras mounted a few centimeters apart behave like a stereo pair, the angle change stays small, and SIFT appears to work; move one camera and it stops. That is exactly the symptom to recognize: detection held up in both views while keypoint matching did not.

SIFT still earns its place when the pose change between reference image and live image is small — same station, same angle. Cross-view identity is a different problem, so the descriptor has to change, not the tuning.

What replaces keypoint matching across two views?

Treat it as object re-identification: describe each crop with a CNN embedding of roughly 128 to 2048 values, L2-normalize it, and compare embeddings between views with cosine similarity, starting from a threshold of 0.8.

Any pretrained backbone gets you a first working version. Feed each crop to the network at its expected input size — 224 × 224 pixels for a classification model — and keep the penultimate feature vector as the embedding. A model re-trained for re-identification separates same-object from different-object pairs wider and buys a stabler threshold, but start with the network you already run.

If the cameras stream continuously, the tracking literature fuses exactly this appearance distance with a motion model, DeepSORT being the best-known combination. For two captures per part, skip the track machinery: score the similarity matrix and assign it, as the next two sections show.

The running a custom CNN for object recognition in robot picking guide shows the crop-to-network path end to end, and auto-pick taught objects with a vision model applies the same embedding idea to matching parts against taught references.

How do you score the similarity between two crops?

Cosine similarity between the two normalized embeddings decides the match: the score runs from -1 to 1, same-object pairs sit well above 0.8 in a consistent rig, and every score below the threshold counts as no match.

With both vectors L2-normalized the formula collapses to a plain dot product, so scoring an N × M matrix costs microseconds on a CPU. The detector is the expensive half of the pipeline, not the matcher. Use cosine rather than Euclidean distance: brightness and scale changes stretch an embedding's length more than its direction, and cosine ignores the length.

The threshold carries the decision and belongs to your rig, not to the algorithm. The section on setting it below places it from labeled pairs instead of a guess.

How do you pair detections in one view with the other?

Score every crop of camera A against every crop of camera B in an N × M matrix, blank entries below the threshold, then take the highest-scoring one-to-one assignment; the Hungarian algorithm finds it exactly and greedy maximum-first selection matches it when matches are clear.

Blanking means setting the entry to a large negative value so the solver never picks it. The Hungarian algorithm costs O(n³), milliseconds for tens of detections. Greedy selection — take the global maximum, consume its row and column, repeat — gives the same result when best and second-best scores are well separated. Every detection ends with exactly one partner or the label unmatched, and those unmatched detections with their best scores are the tuning data for the threshold section.

When one frame holds many parts, the same matrix drives the pairing. Picking a full tray of detections in one snapshot shows the many-detection frame handled on a single view first.

How do you build the matcher end to end?

Eight steps cover the build, and each carries a number from the rest of this guide: overlap the fields, capture paired views, detect with one network, crop with padding, embed and normalize, score, assign, and log the misses.

1. Mount both cameras so their fields overlap at the part plane by at least 1.2 × the widest part — 96 mm of overlap for an 80 mm part. The arithmetic is in the next section.
2. Capture both views of the same part: one photo-eye triggering both cameras, or a software handshake, so the two frames hold the part at the same instant.
3. Run the same detector on both frames — same weights, same 640 × 640 input, same confidence floor. A detector tuned per view produces crops the embedding cannot compare fairly.
4. Crop each detection with about 10 percent padding on each side, so borderline detections stay embeddable.
5. Resize each crop to the embedding model's input, 224 × 224 for a classification backbone, run one forward pass, and L2-normalize the output vector.
6. Fill the N × M cosine-similarity matrix across the two views.
7. Blank entries below the threshold — start at 0.8 — and assign one-to-one with the Hungarian algorithm, or greedy selection when counts are small.
8. Publish each pair as (id_A, id_B) and log every unmatched detection with its best score; that log is both the verification record and the threshold-tuning set.

Before production, run 20 parts through with the true pairing written down. A matcher ready for the line pairs 20 of 20; any miss points at lighting asymmetry or a too-tight threshold, both covered below.

How much field of view must the two cameras share?

Plan an overlap at the part plane of at least 1.2 × the widest part that must appear complete in both views — 96 mm for an 80 mm part — calculated as half of each field of view added together, minus the distance between the two lens centers.

Worked example, arithmetic shown:

• Parts 80 mm wide cross a belt. Required overlap: 1.2 × 80 mm = 96 mm.
• The frame allows d = 260 mm between lens centers, so each field must cover 260 + 96 = 356 mm. Target 400 mm for margin.
• Sensor: 2/3-inch format, 8.8 mm nominal width. Working distance 350 mm. Magnification m = 8.8 / 400 = 0.022. Focal length f = 350 × 0.022 / 1.022 ≈ 7.5 mm, so the nearest stock lens is 8 mm.
• With the 8 mm lens the field at 350 mm works out near 380 mm (376 by the exact magnification, 385 by the linear one), giving overlap = 380 - 260 = 120 mm, comfortably past the 96 mm rule.
• Pixels: a 5 MP 2/3-inch sensor has 2448 × 2048 pixels, so the ground sample is 2448 / 380 = 6.4 px/mm. The part spans 80 × 6.4 ≈ 515 px. Letterboxed into a 640 × 640 detector input at a scale of 640 / 2448 = 0.26, it presents about 134 px to the network — enough to detect — while the matcher embeds the full-resolution 515 px crop.

Which cameras suit a two-view matching cell?

Choose two identical color cameras with global shutter when parts move, one lens model per side, and pick resolution by crop size: a 2.5 MP pair carries a 640-input detector comfortably, while 12 MP pays off when parts sit small inside a large field.

Color matters because the embedding reads appearance, and mono throws away the strongest re-ID features. Global shutter matters on moving belts; rolling shutter is fine for stationary parts. Identical cameras and lenses keep color response and perspective consistent, which is what holds the same-object score up.

Where a PC does the matching, two ordinary area scan cameras over GigE or USB3 feed it, and the camera finder filters by exactly those interfaces. When inspection must run at the device with no PC, the smart camera class carries detector and matcher onboard, and one camera or a small controller hosts the assignment step.

If both cameras must hang off one embedded host instead of a PC — two GMSL2 cameras onto a single carrier board, for example — the deserializer wiring is its own design task. The MAX9296A two-camera wiring guide walks the harness and register work.

The MV-CU020-90GC is a 2.5 MP color camera with a 1/1.7-inch global-shutter sensor and C-mount over GigE. Two of them, one per side, are the matched-pair baseline behind the worked example.

When parts sit small in the frame, step up to the MV-CH120-11UC: 12 MP crops from a 1.1-inch global-shutter color sensor over USB3, C-mount as well. Reach for it when the part fills a small share of the field.

All three cameras below are color C-mount models; resolution, sensor format, shutter and interface separate them:

All 3 share: Binning Supports 1 × 1, 2 × 2, 4 × 4; Lens mount C-mount; Certification CE, RoHS, KC; Mono/color Color. Values as printed in each product's data sheet; open a part for its drawing and full specifications.

The MV-CS200-10GC extends the crop budget to 20 MP on a 1-inch rolling-shutter color sensor over GigE, the pick for stationary parts where detail runs thin.

How do you set the threshold on your own parts?

Collect at least 50 same-object pairs and 50 different-object pairs under production lighting, plot the two cosine-similarity histograms, and place the threshold in the gap; for most rigid-part rigs the gap falls between 0.7 and 0.9, so start at 0.8 and let your histograms move it.

Label pairs straight from production. The unmatched log of step 8 hands you candidate negatives and near-miss positives for free. If the two histograms overlap, no threshold separates them, and the fix is upstream: same light model and intensity on both sides, matched white balance, no polarizer on just one view.

If you would rather compare crops with explicit color numbers than with an embedding, the metrics in compare color between two images with meaningful metrics apply to the same crops.

What breaks cross-camera matching in practice?

Two failures account for most dead matchers: parts that are identical by design, which no appearance embedding can separate at any threshold, and appearance drift between views, which drags same-object scores down into the different-object band.

• Identical parts. Two identical castings differ only in position, so both candidate pairings score the same high similarity. Calibrate the two cameras and associate by projected position, or mark the parts. When the downstream task only needs size or count, one camera with a measurement routine is enough — see identify cylinder diameter for robot picking with one camera.
• Lighting asymmetry. A 6500 K panel over one view and a 3000 K panel over the other shifts color embeddings further than any threshold margin. Run the same light model and intensity on both sides.
• Motion blur. Streaked crops score low on both sides of the matrix. Keep exposure at or below 3 ms at a belt speed of 300 mm/s to hold blur under 1 mm; faster belts scale that budget down linearly.
• Edge and occlusion crops. A part half outside the overlap embeds as a different object. Reject crops touching the frame border or falling under the detector's confidence floor.
• Two USB cameras on one host. Identical models enumerate under one name, and opening the wrong one silently pairs a view with itself. Opening two identical USB cameras by port shows the fix.

Set up this way, the matcher returns (id_A, id_B) pairs that downstream code can treat as one part. To spec a pair for your part sizes and belt speed, start the camera configurator or reach us through the contact page. The systems topic page collects the neighboring guides, and vision locate set up to survive rotation covers the single-view half of a pick cell that depends on this pairing.