Image similarity search: match a query image against a saved library

A station has thousands of saved inspection images and a new frame just arrived: which saved image is it closest to? Comparing that query pixel by pixel against every stored image takes tens of seconds once the library passes a few thousand frames, so the brute-force route dies early. The working answer is the one reverse image search is built on: reduce every image to one short descriptor when you save it, then search those descriptors. This page gives you the descriptor choices, the distance thresholds and the arithmetic that sizes the search for your own library.

What the search actually compares

A brute-force comparison walks every pixel of the query against the matching pixel of every stored image. At 5 MP that is five million operations per stored image, before any normalization, and it repeats on every query. It also answers the wrong question: raw pixel values move with exposure, registration and noise even when the scene is identical.

A descriptor fixes both problems. It is a fixed-length vector — a hash, a histogram, an embedding — computed once per image and stored beside it. A query then costs one extraction plus N comparisons of short vectors, and the library side never re-reads the images. Which descriptor you pick decides how much variation the search absorbs; for two specific images at a time, the histogram-versus-pixel-value trade-off is its own decision with its own numbers.

Descriptor options and what each tolerates

Each family trades descriptor size against the variation it absorbs:

The 64-bit hashes are the smallest useful descriptor, and the 8 × 8 idea behind them is literal: shrink the image to 8 × 8 grayscale, threshold at the mean, keep one bit per cell. Two hashes compare with a Hamming distance — count the differing bits — and each costs 8 bytes to store. They absorb brightness shifts because the mean threshold removes them, but not rotation or scale, which move pixels between cells.

Color histograms sit one step up: count pixels in an 8 × 8 × 8 RGB grid and you get a 512-number vector that ignores position, rotation and scale entirely, and shape just as entirely. Good for sorting parts by overall appearance; useless for telling two shapes apart at the same color mix.

Keypoint methods describe local patches — 128 floats per keypoint in the SIFT family — and match sets of them, so rotation, scale and partial occlusion survive; flat, textureless surfaces give them nothing to grip. CNN embeddings compress the whole image into 128–2048 floats tuned so same-class images land close together, the same idea that powers AI surface defect inspection added to existing cameras. Pick by the variation your parts actually show.

The distance rules

Float vectors rank by Euclidean distance, and cosine similarity ranks identically once both vectors are normalized to unit length (for unit vectors, d² = 2 − 2·cos, so the orderings coincide). Normalize at save time and either formula gives the same answer.

Binary hashes rank by Hamming distance, and chance sets the baseline: two unrelated 64-bit hashes differ in about 32 bits on average — half the bit count — while near-duplicates land in single digits. Start the accept cut at 10 differing bits, then calibrate on your own data: run fifty known-same and fifty known-different pairs from the line, record the worst distance among the known-different pairs, and place the cut just below it. For float descriptors do the same with distances instead of bits. If the two groups overlap, the descriptor is wrong, not the threshold.

Linear scan or an approximate index

Searching N descriptors one by one is O(N), and no exact method removes that: in high dimensions every exact nearest-neighbor scheme degenerates toward a full scan. What exists are approximate indexes. Locality-sensitive hashing (LSH) buckets vectors so a query touches only its bucket; graph indexes such as HNSW walk a pre-built nearest-neighbor graph. Both answer in a fraction of the full scan, and both can miss the true best match — recall becomes a tuning knob traded against speed.

Whether you need the knob is arithmetic, not fashion. The worked example below puts a 10,000-image linear scan at about 2.6 ms per query, and the cost scales linearly, so the scan stays under 26 ms out to 100,000 images. Keep the linear scan while the answer fits the cycle budget: it is exact, needs no index to build or rebuild, and cannot miss. Add LSH or a graph index when the library passes roughly 100,000 descriptors, when vectors exceed about 1,000 floats, or when query rate times library size outruns the budget.

Worked example: one query against 10,000 saved 5 MP images

Inputs. Query and library images: 2448 × 2048 pixels = 5,013,504 pixels each, a 5 MP frame (typical inline camera). Library: 10,000 saved images (a round figure for a season of production at one station). Descriptor: 256 floats per image (a common embedding width). Hardware basis: about 10^9 simple operations per second on one desktop CPU core, roughly 1 ns per operation — conservative, and SIMD only widens the gap.

Rule. A pixel scan costs pixels × images × operations per pixel. A descriptor scan costs floats × images × operations per float. Extraction runs once per image, not once per query.

Arithmetic. Pixel scan: 5,013,504 × 10,000 = 5.01 × 10^10 pixel operations, which at 1 ns each is about 50 s per query. Descriptor scan: 256 × 10,000 = 2.56 × 10^6 multiply-adds, about 2.6 ms per query. The search step shrinks by 5.01 × 10^10 / 2.56 × 10^6 ≈ 19,600 times. The index itself needs 10,000 × 256 × 4 bytes = 10.2 MB of RAM.

Result. A query costs extraction plus 2.6 ms of search: under 1 ms of extraction for a hash or histogram, tens of milliseconds for a CNN embedding on a desktop CPU, then the scan.

Decision. The pixel scan is dead past a few hundred images — 100 stored frames already cost about 0.5 s per query (100 × 5.01 × 10^6 ops ≈ 0.5 s at 1 ns). At 10,000 images the descriptor linear scan answers in under 3 ms, so do not add an index at this size. Revisit at about 100,000 images (≈ 26 ms per query, 102 MB of index) or with vectors beyond roughly 1,000 floats; that is where LSH or a graph index starts to pay. At 10,000 images the lever that matters is descriptor choice, not search machinery.

Where the library lives: camera, controller, disk

A few hundred references fit on the device. On-device matching keeps the whole cycle inside the camera and returns the matched ID over the same link as any other result — the smart cameras category is one integrated camera-plus-processor after another. The MV-SC5060XM-08M-WBN is a 6 MP mono smart camera that connects over GigE.

Past a few thousand descriptors the index belongs on PC-class hardware, where RAM and CPU are not shared with the inspection engine and the library survives camera swaps. The vision controllers are that box: camera ports and processor in one fanless unit. The MV-VC5525-256G70-NN pairs an Intel Core i5 with 32 GB of memory, PoE camera ports and 8 built-in light channels.

Keep raw images on disk and only descriptors in RAM. The index is 10 MB per 10,000 descriptors at 256 floats, while a 5 MP mono frame is about 5 MB uncompressed on its own. The MV-VC2140-128G40-1T records to a 128 GB SSD plus a 1 TB hard drive, with 8 GB of memory for the working set.

Size the library first, then filter hardware: the camera finder sorts smart cameras by resolution and interface.

Build it once: save time and query time

Save time runs offline, once per image: acquire, confirm the frame is usable, extract the descriptor, append (descriptor, image ID, file path) to the store. Query time runs inside the cycle: acquire, extract the same descriptor with the same code, search, compare the best distance to the threshold, return the ID or no-match.

Two details bite in the query loop. Descriptor extraction runs on every frame, so reuse image buffers instead of per-frame allocation or extraction itself becomes the bottleneck. And keep the store in a database keyed by image ID, pulling the matched record with a filtered query — a database search needs a SELECT query with a WHERE clause to stay fast as the table grows. What happens with the matched ID — parameters from PLC memory or a PC database — is the same decision as for barcode part-number parameter lookup, and showing the matched frame to the operator runs into the same limits as a smart camera image in a PLC HMI.

Checks before you build

• Calibrate the threshold on your own parts: fifty known-same and fifty known-different pairs, distances recorded, the cut placed between the two groups with margin.
• Deduplicate before you index: hash the whole library at 64 bits first; duplicate frames inflate match confidence and slow every scan.
• Freeze the imaging recipe: a descriptor extracted under one light and exposure sits far from the same part under another, so save-time and query-time lighting must match.
• Budget the worst case: cycle rate × (extraction time + search time) against the cycle budget, using your own pixel counts and library size in the worked-example formulas.
• Log every match with its distance; thresholds drift as parts and optics age, and the log is the only way to see it.

When similarity search is the wrong tool

The search answers which saved case this most resembles. It does not answer the neighbouring questions, and each has a better tool:

• Is this exactly part X? Read the code — barcodes and data matrix symbols carry the answer with no library at all.
• Where is this known feature, and is it present? Template matching; when the template image is too small the failure is explicit and fixable — see pattern match error: template image too small.
• Does this symbol match the golden reference? That is golden template symbol comparison, a pass/fail tool, not a ranking one.
• Is this part defective, and do you have labeled defect images? That is a classification decision — AI versus rule-based smart camera inspection sets out the split.

Similarity search earns its keep on variant sorting, recipe or setup recall, library deduplication and re-finding the last known-good frame of a part.

Size the library, pick the descriptor from the table against the variation your parts show, then run the worked-example arithmetic with your own pixel counts — the decision falls out in milliseconds. The systems topic page collects the neighbouring patterns, and the configurator below walks the camera-to-controller chain.