Frame differencing with two image buffers for change detection

A camera watching a mostly still scene ships the same pixels again and again, and your link, recorder or PLC loop pays for every repeat. Frame differencing cuts that cost with subtraction: hold the last trusted frame in one image buffer, hold the current frame in a second, subtract, and keep only the pixels whose difference clears a noise-based threshold. The resulting changed-pixel mask gates transmission, triggers a recording or starts an inspection. Set up wrong — with both inputs pointing at the same memory — the difference is zero on every frame and the mask never fires. This guide gives you the working loop, a threshold derived from measured noise, and the bandwidth arithmetic that decides when the trick pays.

What frame differencing gives you

Subtract the reference from the current frame, take the absolute value, and flag every pixel above a threshold T. What remains is a binary mask of the pixels that changed. Three uses pay for themselves in production:

• Bandwidth gating. Send only the changed pixels and an occasional full keyframe instead of every frame.
• Motion and presence triggers. Start recording or call the inspector when the mask count crosses a limit.
• Inspection pre-check. Skip frames whose mask is empty; run the full inspection only where something moved.

The 4 in the last line is bytes per changed pixel: a 12-bit x coordinate, a 12-bit y coordinate and an 8-bit value pack into 32 bits, which covers sensors up to 4,096 pixels across. For most quiet 8-bit scenes a threshold of 10 to 15 grey levels clears sensor noise and catches real change. The section below derives it from a measurement instead of guessing it.

What you need

• Two image buffers per stream — one the camera fills, one that holds the reference. Separate allocations, never the same handle.
• A noise measurement sigma, in grey levels, taken on your scene at your final exposure and gain.
• A threshold T and two action limits: the mask fraction that triggers the action, and the fraction that means send everything instead.
• A camera that delivers stable 8-bit frames at a known rate. Mono is the simplest; colour works too (the FAQ covers Bayer).
• An area scan camera feeding a PC, or a smart camera that runs the whole loop on the device.

Set it up step by step

1. Capture the reference. Let the scene settle, grab one frame, and copy it into the reference buffer. A copy, not a second pointer to the acquisition buffer.
2. Acquire the next frame into the acquisition buffer only. The grab-loop pattern in live edge detection from a video stream keeps acquisition and processing from fighting over one buffer.
3. Subtract and take the absolute value, clamping at 255 so the result stays 8-bit.
4. Threshold at T. Every pixel above T becomes 1; everything else becomes 0. Edge detection without a vision library: threshold basics covers the mechanics.
5. Clean the mask. Remove isolated single pixels and keep blobs of at least 3-5 pixels; the passes in dark speck detection with thresholding and morphology do this in two operations.
6. Count and act. Below the limit: transmit the (x, y, value) entries or skip the frame. Above it: send the full frame, trigger the recorder, raise the alarm.
7. Refresh the reference every 100 frames and after any exposure, gain or lighting change.
8. Log changed_fraction per frame. A slow upward drift with a still scene means the noise grew or the optics moved.

Keep the reference frame in its own buffer

The classic failure: the difference image is black on every frame, the mask count is exactly zero, and the camera looks fine. The cause is buffer aliasing. Vision libraries commonly reuse a single acquisition buffer, so if you pass that buffer, or a saved handle to it, as both subtract inputs, the result is A minus A = 0 for every pixel, every time. Nothing downstream can recover a difference that was never computed.

The copy costs almost nothing: 12 MB moves in about a millisecond on a modern PC, against the 33 ms that a 30 fps frame period gives you. Keep the same discipline everywhere else — the reason text overlay on a live camera image without altering the frame exists is that processing steps love to write into the buffer the camera is still filling.

Two variants, two jobs. Adjacent-frame differencing (current minus previous frame, ping-ponging between two rotating buffers) detects motion. Baseline differencing (current minus a stored good frame) detects change from a known state. Pick adjacent frames for speed and direction; pick the baseline for has anything appeared, moved or gone.

Set the threshold from measured noise

Measure, do not guess. With the camera aimed at the real scene at final exposure and gain, capture 30-50 frames with nothing changing, compute the per-pixel standard deviation, and take the median across pixels. That sigma is temporal noise; fixed per-pixel offsets cancel because both frames come from the same pixel sites. Set:

Quiet 8-bit pairs land at sigma of about 3-5 grey levels, so 3 sigma puts T at 10-15 grey levels. What the choice costs you, for Gaussian noise on a perfectly still pair:

At 3 sigma the residue is scattered single pixels; the morphology pass in step 5 removes it. If you want a quiet mask without cleanup, run 4-5 sigma. Re-measure sigma after any exposure or gain change — gain raises noise directly, and a threshold set in daylight will not survive a dimmer cell or a shorter exposure.

Worked example: a 12 MP changed-pixel stream over GigE

Inputs, with their sources:

• Frames: 4000 x 3000 pixels, 8-bit mono, 30 fps — the printed spec of the camera below.
• Noise: sigma = 4 grey levels — measured over 50 static frames at final gain; your value comes from the same procedure.
• Threshold: T = 3 sigma = 12 grey levels.
• Mask trial: 1.5% of pixels changed — from a 100-frame pilot run on the real scene.
• Entry format: 4 bytes per changed pixel (12-bit x + 12-bit y + 8-bit value).
• Link: GigE line rate 1,000 Mbit/s; budget 800 Mbit/s for this stream, the customary 80% that leaves room for protocol overhead and other traffic.
• Keyframe: one full frame every 100 frames.

The camera class the example assumes:

The arithmetic, line by line:

1. Pixels per frame: 4000 x 3000 = 12,000,000. A raw 8-bit frame is 12,000,000 bytes = 12 MB.
2. Raw stream: 12 MB x 30 fps = 360 MB/s = 2,880 Mbit/s. Nearly three times the 1,000 Mbit/s line, so raw does not fit — the problem differencing solves.
3. Changed pixels per frame: 0.015 x 12,000,000 = 180,000.
4. Diff payload per frame: 180,000 x 4 bytes = 720,000 bytes = 0.72 MB.
5. Diff rate: 0.72 MB x 30 fps x 8 = 172.8 Mbit/s.
6. Keyframe amortised: 12 MB / 100 frames = 0.12 MB per frame, times 30 fps times 8 = 28.8 Mbit/s.
7. Total: 172.8 + 28.8 = 201.6 Mbit/s — about 25% of the 800 Mbit/s budget. Fits, with margin.
8. Where the budget runs out: 800 Mbit/s / 30 fps = 26.7 Mbit per frame = 3.33 MB; at 4 bytes per entry that is 833,000 changed pixels, or 6.9% of the frame.

Decision: below about 7% changed pixels, the diff stream rides GigE at 30 fps with margin. Between 7% and 25%, the diff still beats raw pixels (a 4-byte entry only loses to a 1-byte pixel above 25%) but needs USB3, a second link or a lower frame rate. Above 25%, stop differencing for transport: send the full frame or a compressed one, because entries now cost more than the pixels they replace.

Size the pixels for the smallest change you must catch

The mask can only flag what the sensor resolves. Put at least 3 pixels across the smallest feature you must catch, 5 if you also intend to measure it. Say the target is a 0.3 mm speck inside a 300 mm field: at 4 pixels per speck the image needs 0.3 / 4 = 0.075 mm per pixel, and the sensor needs 300 / 0.075 = 4,000 pixels across the field. Feed your own field width and feature size to the calculator:

Picking the camera: shutter, colour and where the loop runs

Shutter first. A global shutter sensor freezes the whole frame at once, so a real change maps cleanly into the mask. A rolling-shutter frame of a fast change is captured row by row, and the row skew itself reads as change — workable while scene motion stays slow against the readout, wrong once things speed up.

Where the loop runs is the second fork. On a PC the mask logic is a few lines in any vision library and buffer memory is plentiful on a modern workstation. On the device, an x86-platform smart camera runs acquisition, subtraction and thresholding with no controller in the middle. The LS-SMC-0017 is that class of part: an 8.8 MP sensor, global shutter and a C-mount, with the loop living inside the camera.

For PC-based setups, three cameras cover the common differencing cases:

The MV-CU020-90GC runs a trigger line on GigE with a 2.5 MP global-shutter sensor. The MV-CH120-11UC is the fast-change pick at 12 MP, global shutter, on USB3. The MV-CS200-10UM goes the other way: 20 MP mono on a 1" sensor, where mask detail matters more than motion speed.

When differencing is the wrong tool

• Dense motion. If most of the frame changes every cycle, (x, y, value) entries cost more than the pixels they replace; a standard video codec such as H.264 compresses motion far better than raw differencing.
• Light that drifts or flickers. Unstable illumination changes nearly every pixel and the mask lights up everywhere. Drive the lights from a steady controller, lock the exposure, and refresh the reference after any lighting service.
• A moving camera. Vibration or a bumped bracket misaligns every edge against the reference, and the mask explodes even though the scene stood still. Mount rigidly and re-capture the reference after any mechanical work.

Checks before you trust the mask

1. Static run. 100 frames with a still scene at 3 sigma should show the mask under about 0.3% of pixels — about 32,000 on a 12 MP frame. A rising fraction means gain, exposure or temperature moved; re-measure sigma.
2. Injected change. Put a target of known area in view. At the worked example's 0.075 mm per pixel, a 20 mm x 20 mm plate is 267 x 267 = 71,000 pixels, about 0.6% of the frame; the mask should land within plus or minus 20% of that.
3. Zero check. Exactly zero changed pixels on every frame is the aliasing signature from the buffer section, not a quiet scene.
4. Rate and load. Measure delivered fps the way video FPS returns zero in OpenCV: read frame rate reliably shows, and watch the link under load — the tactics in camera burst capture faster than link bandwidth: use the buffer apply when bursts exceed it.
5. Eyes on the mask. Preview the binary mask during commissioning with the pattern in live binary image preview for defect inspection.

Decision and next steps

The rule that survives contact with the floor: two buffers, T = 3-5 sigma from a measured sigma (10-15 grey levels on a quiet 8-bit pair), a transmit limit at the fraction your link budget allows — about 7% of a 12 MP frame on GigE at 30 fps — and a keyframe every 100 frames to reset drift. Above 25% changed, send whole frames through a codec instead.

If the mask feeds a machine stop, treat it as process control only: personnel protection belongs on a certified safety circuit, sized by the machine's risk assessment, not on a vision result.

Shortlist from the area scan camera or smart camera category pages, or open the camera finder and start the configurator at the camera step. More set-up material sits on the systems topic page, and the contact page reaches us when a scene resists every threshold.