2D image filters for robot vision: which ones to use and when

A robot cell that mispicks or false-rejects parts usually has a preprocessing problem, not a detection problem. The pixels coming off the camera carry dust specks, sensor grain, glare and uneven light, and the locate or inspect tool downstream inherits every one of them. Five 2D image filters cover nearly all of it: median, Gaussian, Sobel or Canny, morphological open and close, and threshold. This guide is for integrators wiring a camera to a robot or PLC. It gives the noise-to-filter match, the kernel sizes to start with, the arithmetic that keeps the chain inside the cycle time, and the cases where no filter helps.

Match the filter to the noise

Start from what is wrong in the frame, not from habit. Each filter removes one class of defect; run the wrong one and you pay taps for nothing while the real problem survives.

Everything else is a variant of these five. The sections below give the numbers for each, then a worked budget for a 1.0 s robot cycle.

Median: the first filter for specks

Dust on the part, a glint, a dead pixel: these are single-value outliers, and the median filter is built for them. It sorts the values inside the window and keeps the middle one, so an isolated speck is discarded while a real edge, which has support on both sides, survives untouched. Start at 3×3, which already removes any speck one pixel wide; step to 5×5 only when specks cluster two or three pixels across. The cost is 9 taps per pixel at 3×3 and 25 at 5×5, and the median is not separable, so there is no shortcut. For a floor case with the images at every stage, see Remove dust noise from screw thread images: filter before threshold.

Gaussian: the first filter for grain

Grain is the opposite failure: every pixel wobbles a little, so no single pixel is an outlier. The Gaussian filter replaces each pixel with a weighted average whose weights fall away from the window centre. Common libraries pair a 3×3 kernel with σ ≈ 0.8, 5×5 with σ ≈ 1.1 and 7×7 with σ ≈ 1.4; keep the kernel at least about twice σ wide so the curve is not cut off. The Gaussian is separable: a 7×7 runs as two 7-tap one-dimensional passes, 14 taps instead of 49. That makes it the default before any detector — threshold on grainy frames, gradient on faint edges. Its weakness is the same property: it blurs true edges too, so run it on the gray frame before detection, never on a binary mask you want crisp.

Edge filters: Sobel and Canny

Edge filters respond to change rather than to brightness. The Sobel operator applies two 3×3 kernels, one per axis, and combines the responses:

A threshold on G turns that gradient magnitude into a binary edge map. Canny wraps the same gradient with three extras: a Gaussian pass first, non-maximum suppression that thins the ridge to a single pixel, and hysteresis with two thresholds — strong edges seed the map, weak pixels join only when connected to a seed. Set the low threshold between one half and one third of the high one. The result is the cleanest edge per tap, and it is what most subpixel locators consume. One trap: Canny edges are thin and almost never closed, so they are the wrong input for a fill or a blob area count — threshold into a mask for that. The failure mode is worked through in Canny before findContours leaves open contours: threshold the mask.

Morphological open and close: repair the binary

Thresholding carries the gray frame's defects into the binary image, so repair shape there. An opening, erosion then dilation, erases bright specks smaller than the kernel; a closing, dilation then erosion, fills dark pinholes and reconnects thin bridges. Start at 3×3 with a single pass and repeat rather than enlarging the kernel: two 3×3 dilations reach as far as one 5×5 for 18 taps against 25. Before counting blobs on a dusty line, open first, or every surviving speck becomes a phantom part. Before measuring a shape's area, close first, or pinholes shave the count. Keep the sequence threshold, open, close, and only one morphology family per defect: an over-opened mask eats thin features that no later pass restores.

Threshold: fixed, automatic or adaptive

Binarization is where preprocessing pays out, and the simplest choice usually wins. A fixed threshold marks every pixel above one value as white; under a controlled light on a matte part, a value read off the histogram holds all shift long. An automatic, histogram-based threshold picks that value from the histogram itself, which absorbs batch-to-batch brightness drift but still assumes two separable peaks.

An adaptive threshold computes a local value per window and survives uneven illumination, at a cost multiplied by the window area, and it invents edges in flat regions. In a robot cell the fixed or automatic threshold is almost always right, and the effort is better spent on a stable light than on adaptive code. If the light cannot be tamed, shield it before coding around it; the controls are laid out in Ambient light variation in pick and place vision: control it.

Worked example: pixel budget and filter cost

The scenario: a fixture pauses 1.0 s per part, the robot waits at most 250 ms for vision, the field of view is 60 mm wide, and the smallest feature that matters is a 0.3 mm edge defect you want at least 4 pixels across.

The calculator below runs the same pixel budget the other way, from a required feature size to cameras with enough pixels.

A compact global-shutter GigE camera with a C-mount covers that budget:

The MV-CU020-90GC is a 2.5 MP color camera with a global shutter on GigE and a C-mount. A 2.5 MP 4:3 frame is about 1600 × 1200 px: twice the 800 px the defect needs, so 8 px across 0.3 mm, at 60 mm ÷ 1600 px = 0.0375 mm/px. A 3×3 kernel spans 3 × 0.0375 mm = 0.11 mm, small against a 0.3 mm feature, so a single pass cannot erase it.

Cost, at an assumed 5 ns per tap on one embedded-class core (conservative; a desktop core is several times faster): 1,920,000 px × 9 taps = 17.3 million taps per 3×3 pass, and median plus Gaussian is 34.6 million, about 173 ms of the 250 ms budget. Crop to an 800 × 800 ROI around the part instead: 640,000 px × 9 = 5.8 million taps per pass, 11.6 million for both, about 58 ms — 23% of budget — leaving room for threshold and blob locate inside a 3×3 median → 3×3 Gaussian → fixed threshold → locate chain.

If the defect shrinks to 0.15 mm, the pixel budget doubles to 1600 px across and the margin is gone. The next pixel step up is the 20 MP class, on USB3.0 instead of GigE:

All 2 share: Binning Supports 1 × 1, 2 × 2, 4 × 4; Decimation Supports 1 × 1, 2 × 2, 4 × 4; Lens mount C-mount; Humidity 20% RH to 95% RH (no condensation); Certification CE, RoHS, KC. Values as printed in each product's data sheet; open a part for its drawing and full specifications.

The MV-CS200-10UM moves the same decision to 20 MP, mono, on USB3.0. For a paused fixture either camera holds; where the part moves during exposure, take the global-shutter class and recover the margin with an ROI crop.

Where the filters run: on device or on a controller

A 2D filter chain does not need a PC in the cell. A smart camera runs median, Gaussian, threshold and blob analysis on the device and ships coordinates and pass/fail over Ethernet; no image crosses the wire. For one camera and one job, that is the shorter path.

For a single-camera locate and inspect step, the on-device route:

The LS-SMC-0063 runs a 1.6 MP sensor with a 16 mm lens at 60 fps over GigE. A 60 fps rating means the whole pipeline, filters included, must finish inside 16.7 ms per frame, so a cropped 3×3 pass runs in single-digit milliseconds — far inside the 250 ms budget.

Cells with several cameras, or heavy full-frame chains with Canny plus matching, go on a controller mounted beside the robot, which also drives the lights:

The MV-VC2140-128G40-1T runs an Intel N-series processor with 8 GB of memory, 3-4 LAN ports, 2 USB 3 ports and 4 built-in light channels: filters, light strobing and robot communication in one box. Where the time goes on either platform, and what counts as fast enough, is broken down in Smart camera image processing time: what is fast enough.

When a filter is the wrong fix

Three failures no spatial filter repairs:

• Motion blur. Blur in pixels is part speed × exposure ÷ mm per pixel. At 100 mm/s and a 5 ms exposure the smear is 0.5 mm, which is 0.5 ÷ 0.0375 = 13 px on the example setup, and no filter reconstructs it. Hold blur under 1 px with exposure ≤ 0.0375 mm ÷ 100 mm/s = 0.375 ms, then recover brightness with more light or a strobe. The exposure side of this is worked through in Reduce wrist camera snapshot time: fix focus and exposure.
• Missing contrast. Glare and a washed-out edge are illumination problems; a median cannot create a gradient the light never made. Change the light geometry first.
• Missing height. A 2D filter works on one gray plane. Volume, flushness and dent depth need height data from motion or a 3D sensor: a 2.5D blob builds volume by integrating a surface across frames as the part moves, which is why volume measurement needs motion across frames, not another filter.

The order that survives the shop floor

1. Set light and exposure first; blur and glare are not filter problems.
2. Crop to the region of interest around the part.
3. Median 3×3 for specks; Gaussian 3×3 for grain.
4. Threshold, fixed or automatic; read the value off a histogram rather than typing one in.
5. Open, then close the binary at 3×3, one or two passes.
6. Locate or measure, then log the worst-case time per part, not the average; the worst case sets the robot's wait.

The commit test before freezing the recipe: run 50 floor images through the chain, dusty and glary ones included. If worst-case processing is under 60% of the vision budget — under 150 ms of the example's 250 ms — ship it. Above that, crop tighter or drop a pass before adding hardware.

Parts and the next step

Every camera above runs the same five-filter chain; what changes is where it executes. Compare smart cameras for on-device chains, area scan cameras when the pixels must feed a PC-side pipeline, and vision controllers for the multi-camera box beside the robot. The camera finder filters by resolution, interface and shutter, and the configurator walks sensor, lens and light in one pass. More system-level fixes sit on the systems topic page.