Image sequence segmentation and tracking from a machine vision cell

Tracking and video segmentation models are built around one idea their users often miss: the frames form a sequence, not a set. A predictor that segments objects and follows them over time expects frames in capture order, at a steady rhythm, captured the same way. Feed it a folder of saved stills or an array of frames gathered ad hoc, and it either refuses to start or produces masks and track IDs that jump between objects. This guide explains why that is, and how to structure the output of a machine vision cell so temporal segmentation and tracking work reliably on the line.

What "video mode" actually requires

Many video segmentation and tracking predictors check at start-up that their data source is a video stream rather than an image list. That is not an arbitrary restriction. The tracker maintains state between frames: remembered object positions, motion estimates, appearance models and identity registers. Every one of those assumes that frame n and frame n+1 show the scene a short, known time apart.

When the source is an unordered list of images, three things break:

• Temporal ordering. The tracker links a detection in one frame to the nearest matching detection in the next. Shuffle the order and it links a part to the wrong neighbour.
• Uniform spacing. Motion models predict where an object went between frames. Irregular gaps between used frames make the prediction wrong by a variable amount.
• Consistent appearance. Frames shot with different exposure, zoom or resize make the same object look different frame to frame, so appearance-based matching drops it and starts a new ID.

The practical answer for a vision cell is simple: do not hand the model loose images. Build a genuine sequence first, then feed that.

Turning camera acquisitions into a sequence

An area scan camera on a line already delivers what you need: frames in order, timestamped by arrival. The work is in how you capture, buffer and pre-process them. Four rules cover most cases.

1. One exposure for the whole sequence. Lock exposure, gain and white balance manually before capture begins. The industrial camera image quality settings needed for stable sequences are the same as for any serious inspection: fixed values, no auto modes. Auto exposure responds to scene content, so a bright part entering the frame changes every parameter mid-sequence.

2. One size for every frame. Resize every frame to the same dimensions with the same interpolation before it reaches the model, not once per frame with a different result. Do it once in your capture pipeline.

3. A fixed stride. If the camera runs faster than the model can process, skip frames on a fixed rule — every second, every third — counted on the frame index, never on elapsed time alone. Timers drift with CPU load; frame counters do not.

4. No mid-sequence changes. Do not reconnect the camera, change resolution, rotate the image on the fly or restart acquisition inside a sequence. If you need orientation changes, see how camera image rotation without losing acquisition speed handles rotation inside the pipeline; the same principle applies — the stream must stay continuous.

The numbers: stride and sequence length

The stride decides whether tracking works at all. A workable rule of thumb: a tracked object should move no more than about a quarter of its own size between two frames the model actually sees.

Worked example. A conveyor runs at 300 mm/s. The object is 80 mm along the direction of travel. Your camera captures at 30 fps and your model processes about 10 frames per second.

• Stride = 30 / 10 = 3, so the model sees every third frame, effectively 10 fps.
• Shift per used frame = 300 mm/s × (3 / 30 s) = 30 mm.
• Limit = 0.25 × 80 mm = 20 mm. 30 mm exceeds it, so tracking will drift.
• Fix: drop the stride to 2. Shift becomes 300 × (2/30) = 20 mm, exactly at the limit — tighten further to a stride of 1 or slow the conveyor if the line allows.

For sequence length, aim for at least 20 to 30 consecutive used frames covering the whole motion you care about. Below that, trackers have too little history to separate a real object from noise; far above that, memory use grows and most models only use a sliding window anyway.

If you are deciding how many pixels the model really needs after resizing, the camera resolution calculator gives you the pixel count your field of view implies — resize to that or a clean fraction of it, and never upscale.

Hardware that keeps the stream continuous

A sequence is only as good as its weakest dropped frame. Two hardware choices matter most.

Global shutter, enough pixels. A moving part captured with a rolling shutter shears vertically, and shear changes the object's appearance frame to frame — the tracker sees a different shape every frame. A global shutter camera avoids this. The MV-CH120-11UC, for example, is a 12 MP USB3 area scan camera with a global shutter in a 1.1" format, which covers both requirements for line-speed tracking of medium parts.

A controller that absorbs the load. Buffering a sequence while a model runs on the frames is memory and CPU work on top of acquisition. A vision controller separates acquisition from inference: the MV-VC3301H-128G66 pairs a Core i3 with 8 GB of memory and five LAN ports, enough to keep camera streams alive while a GPU or inference device handles the sequence model.

For a broader choice, browse the area scan cameras and vision controllers categories, or use the camera finder to filter by shutter type and interface directly.

Where sequences beat single images, and where they do not

Temporal segmentation earns its cost when:

• objects overlap or occlude each other, and identity across frames disambiguates them;
• the part appearance is ambiguous in one frame but clear in motion (folds, translucent film, tangled parts);
• you need per-object counting across a whole passage, not one verdict per frame.

Single-image inspection remains the better tool when the part sits still, the defect is visible in one frame, and per-image cycle time dominates. A smart camera doing per-frame inspection is simpler, cheaper to run and easier to validate than a sequence pipeline. Video replay systems show the same split from the other side: recorded video replay for machine vision pipelines exists precisely because replaying a genuine stream behaves differently from replaying loose frames.

Checks before you commit to a sequence pipeline

• Capture a 60-frame sequence and confirm the frame counter increments by exactly 1 with no gaps; dropped frames show up as double-size object shifts.
• Verify the per-frame object shift on the real line, with real speed, against the quarter-size rule above.
• Freeze all camera parameters and confirm in two captured frames, seconds apart, that histograms match.
• Track one object through a full sequence by hand and confirm the ID never changes.

Run these before model tuning. Most unstable tracking traces back to the capture side, not the model.

A short checklist

• Fixed exposure, gain and white balance across the sequence
• Every frame resized to the same dimensions, once, in the capture path
• Stride counted on the frame index; shift per used frame under a quarter of object size
• Global shutter for any moving part
• No reconnection, resolution change or rotation change inside a sequence
• 20–30 consecutive usable frames minimum over the motion of interest

If you are assembling a new cell around this, the system configurator walks through camera, lens, light and controller choices in order, and our expert support team can review a capture set-up before you order.