Color blob tracking algorithm: segmentation and PID control

A cart that chases a colored ball, a gimbal that holds a marker at frame center, a camera that hands a centroid to a PLC: all of them run the same two-stage loop. Stage one finds the target — threshold the image in a color space and keep the one clean blob. Stage two moves the hardware — feed the centroid error in pixels to a PID controller. Here you get the full pipeline, threshold starting points for HSV and LAB, a proportional gain sized from your own frame width and full-scale command, and the exposure and lighting settings that keep the loop stable frame after frame.

The pipeline from frame to command

Blob detection finds connected regions that differ in brightness or color from their surroundings — exactly what a tracking front end needs. The whole tracker is seven steps, and on a smart camera they all run on the device at 30-60 fps:

1. Acquire a frame, global shutter if the target moves fast.
2. Convert RGB to HSV or LAB.
3. Threshold the color window or windows into a binary mask.
4. Open the mask — erode, then dilate — to strip speckle.
5. Label connected components and keep blobs that pass area and shape filters.
6. Choose one blob: the largest, or the nearest to last frame's centroid.
7. Compute the centroid, form the error, run the PID, send the command.

Steps 1-5 cost most of the frame budget; steps 6-7 are the loop that closes every frame. Published work on mobile-robot visual tracking commonly wraps this PID in a cascade with an inner velocity loop on the drive — which is where a fast platform should close its own loop anyway. Keep the camera-side controller at frame rate.

Pick the color space: HSV, LAB, and why not RGB

Raw RGB fails first, so cross it off: all three channels move together when brightness changes, and a window tuned under one light dies under another.

HSV keeps chroma in the hue channel, largely independent of brightness. In the common 8-bit convention hue runs 0-179, so red wraps around both ends: start with two windows, 0-10 and 170-179, plus a saturation floor near 60 to reject gray pixels. Treat those as starting points and tune on your parts.

LAB separates lightness (L) from the two chroma axes (a, b). One target color forms a tight cluster in a-b that barely moves when brightness changes, which makes LAB the steadier pick for a single known color under variable light. Gate out very dark pixels first — L below roughly 32 — then window a and b.

Decision: one known color under changing light, take LAB. Several hues at once or a quick demo, take HSV. RGB only when the lighting is truly fixed, which in production it rarely is.

Clean the mask and reject false blobs

Thresholding leaves speckle and fragments. Open the mask: erode by 2-3 pixels, then dilate by the same amount. Erosion deletes single-pixel noise, and the matching dilation restores the real blob's size almost exactly.

Filter what remains. Set the minimum area near half your smallest expected target: a 12-pixel-wide ball covers about 110 pixels, so a floor of 60 pixels rejects noise without eating the real one. For a ball, add a shape gate — keep blobs whose width-to-height ratio sits between 0.7 and 1.4, which drops elongated reflections and merged pairs.

Define the error in pixels before millimeters

Fix the definition once and keep the sign: e = x_blob - x_aim, with the aim point at frame center for a following platform. Prove the sign early — put the target left of center and watch the platform steer toward it, not away.

Convert to millimeters only if the mechanism needs physical distances. Pixel size follows from magnification, and the object-space pixel size calculation works it step by step. For example: 320 pixels across an 80 mm field of view is 4 px/mm, so a 20 mm ball covers 80 pixels. A pure follower can stay in pixels end to end — the loop closes on error either way.

Size the field of view from the target's travel, not from optical detail: travel plus two or three target diameters of margin. A blob at the frame border has a clipped centroid and a saturated error, so keep the path comfortably inside the frame. The calculator below sizes a lens for that field at your working distance:

Worked example: proportional gain for a 320-pixel frame and a 64-pixel band

The inputs, and where each comes from:

• 320-pixel processing ROI, aim point at pixel 160 — a typical on-device ROI for a following camera.
• Proportional band B = 64 px — a design choice: the command should reach full scale while the blob is still 64 px off center.
• Full-scale command U_max = 255 counts — the span of an 8-bit PWM or DAC output to the drive.
• Sample time T = 16.7 ms — one frame at 60 fps.

The LS-SMC-0063 smart camera runs 1.6 MP at 60 fps, which sets that sample period. The per-frame exposure and gain check confirms the rate the frames actually deliver.

Rule: the proportional gain is full-scale output divided by the band over which you want proportional action.

Kp is the gain, U_max the full-scale command (255 counts), B the band (64 px), e the centroid error in pixels.

Arithmetic: Kp = 255 counts / 64 px = 3.98, so set Kp = 4 counts per pixel.

Saturation check: a blob at the frame edge gives e = 160 px, and 4 × 160 = 640 counts — well past the 255 the output can deliver. The command clamps at 255, and that is correct: any error beyond 64 px earns full command, so 64 px is exactly where saturation begins. Halve the band to 32 px and the same division gives 255 / 32 ≈ 8 counts per pixel — twice the loop gain, and enough to overshoot without derivative action.

Decision: Kp = 4 counts per pixel, output clamped to 0-255 counts, sample time 16.7 ms. Every error past 64 px is a saturated, full-speed pursuit; the control happens inside the band.

Integral and derivative without destabilizing the loop

Add integral action last and keep it small, in counts per pixel per second. Its job is closing a steady residual — an off-center aim point, an asymmetric drive — not chasing noise. Clamp or freeze the integrator whenever the output saturates; otherwise it winds up through every edge-to-center transit and the platform overshoots on arrival.

Derivative action damps that overshoot but amplifies centroid jitter. A centroid hopping 2 px between frames changes at 2 / 0.0167 s = 120 px/s; with Kd = 0.5 counts·s per pixel, that alone injects a 60-count spike at every jump. Low-pass the error with a two- or three-tap moving average, or use the velocity form, before trusting derivative action.

Attack order: P-only at the worked-example gain. Oscillating around center — add derivative. Settling a few pixels off center — add integral with anti-windup.

Lock the light, or the thresholds move

Auto exposure and auto gain re-balance every frame, and your color windows move with them: the mask flickers, shrinks or dies. Lock exposure, gain and white balance once the scene is set. The auto-exposure pixel-count article shows which pixels drive that decision, so you can lock it deliberately instead of by accident.

Drive the LEDs at a fixed operating point as well. An unregulated supply or mains ripple shifts brightness, and with it your saturation and hue floors; the closed-loop lighting control article shows how to hold the level steady. Pick the LED color for contrast against the target instead of defaulting to white — the LED colour and contrast article ranks the options.

A light controller holds that operating point. The LS-CTL-0322 digital controller drives four channels of 24 V constant-voltage lighting with 8 A total output from a 100-240 V AC supply.

For a self-contained tracker, the LS-SMC-0063 integrates a polarized white light with a 16 mm lens and runs 1.6 MP at 60 fps. Polarized output cuts specular glare off shiny targets — one fewer hue shift to chase.

Camera: global shutter, then frame rate

A rolling-shutter sensor reads its lines sequentially, so a fast ball skews: the blob stretches along the motion and the centroid biases with speed. For a moving target, buy a global shutter. Both color parts below have one.

The MV-SC6050C-08M-XBN pairs a 5 MP color sensor with an 8 mm M12 lens and a built-in multi-color light source, over GigE.

The MV-SC3050XC-16M-ATN-H vision sensor runs 5 MP color over Fast Ethernet, global shutter, M12 mount.

Then check blur and per-frame travel. Blur is speed times exposure: a target crossing at 500 mm/s under a 0.5 ms exposure smears 0.25 mm — one pixel at 4 px/mm, acceptable; at 2 ms it smears four pixels and edge pixels mix hue with the background. Hold blur under 1-2 pixels; the exposure time versus shutter speed article works the full trade.

Per-frame travel sets how far the blob jumps between updates; keep it near 3 px. At 60 fps and 4 px/mm that allows 3 × 60 / 4 = 45 mm/s of target speed. Widen the field so the same 320-pixel ROI spans 160 mm — 2 px/mm — and the loop follows 90 mm/s. Tracking rewards field of view, not resolution. For larger fields or several targets at once, step up to area scan cameras with a C-mount lens and run the identical pipeline.

When the usual answer is wrong

Occlusion: something crosses the target and the thresholds return nothing or a fragment. Constant-velocity prediction bridges one or two frames, not a full second of blockage.

Same-color distractors: a second object in the target color defeats largest-blob selection and eventually stalls nearest-to-last tracking. Change the marker color, or move to shape- or learning-based detection — the rule-based versus AI comparison lays out that fork with its costs.

Hue that shifts with position is a lighting-geometry fault, not a software one: fix the illumination before widening windows into uselessness.

Metrology: a centroid guides, it does not measure. Do not quote part dimensions from blob edges.

Checks before you deploy

Run these six before the loop drives a real platform:

• Windows tuned on 500+ sampled pixels per class, target and background, from real frames.
• Mask stable over 100 frames with exposure, gain and white balance locked.
• P-only first, at your own U_max / B gain, settling without oscillation.
• Edge test: drive the target to the frame border and confirm the command saturates instead of reversing.
• Lost-target test: cover the ball mid-run and confirm the output decays to zero in about 200 ms.
• Loop audit: the PID period equals the delivered frame period.

Parts for this build

Four parts cover the loop end to end:

• Camera: MV-SC6050C-08M-XBN — 5 MP color, global shutter, 8 mm M12 lens, integrated multi-color light source.
• Vision sensor: MV-SC3050XC-16M-ATN-H — 5 MP color, Fast Ethernet, global shutter, M12 mount.
• All-in-one tracker: LS-SMC-0063 — 1.6 MP, 60 fps, 16 mm lens, integrated polarized white light.
• Lighting control: LS-CTL-0322 — four channels, 24 V constant voltage, 8 A total.

Shortlist hardware in the camera finder, size the optics with the calculator above, and walk the camera configurator to finish the chain. The applications topic page collects the neighboring guides, and the team answers specific set-up questions through /contact/.