Word boundary detection in an image with OpenCV morphology

You have a page or photo that mixes printed text with pictures, logos and speckle, and you need one rectangle per word — not the single block that edge detection hands back. The fix is two operations: a threshold that turns the text into a solid binary mask, and a horizontal morphological closing that grows each letter sideways until the members of one word touch. Contour filtering then throws out the pictures and the noise. Below are the kernel sizing rule, the full Java pipeline with a test, the contour filters with numbers, and the camera, lens and exposure numbers that keep the word gaps alive at capture time, plus the cases where classical morphology gives up and a trained detector is the honest answer.

Why does Canny box the whole paragraph instead of each word?

Because Canny returns an edge map, not a mask: contours traced on it outline strokes and whole blocks, never words, so binarize with a threshold instead and close the mask horizontally with a kernel of roughly 15 × 3 px to make each word one solid blob.

Canny marks gradients one pixel wide, doubled on both faces of every stroke, so a contour pass returns broken rings around individual letters plus edge noise — the failure traced in Canny before findContours leaves open contours: threshold the mask. The single box around all the text that starts most projects like this is the outer contour of a thresholded block whose letter gaps filled at the wrong cutoff. Either way, nothing in an edge image records which strokes belong together.

Membership is geometry, and morphology supplies it: letters of one word sit closer to each other than to the next word, so growing every letter sideways by a fixed amount fuses exactly the intra-word gaps. It is the same recipe as Threshold then morphology: a segmentation workflow for part detection, run at text scale instead of part scale.

How wide should the morphological kernel be to merge letters but not words?

Set the kernel width between the two gaps you measure on the binary mask — wider than the largest letter gap, narrower than the smallest word gap — which works out at 1.5 to 3 times the average letter gap: 15 px when letters sit 7 px apart.

A closing with a k-pixel-wide rectangular kernel fuses every background run narrower than k and leaves anything wider open. Print gives you the margin for free: in Latin body text the white gap between letters is about 0.15 em and the word space about 0.4 em, so word gaps run 2.5 to 4 times letter gaps. Measure rather than guess — scan a few rows through the text band of the mask, collect the background runs, and take the two medians.

For 12 pt print at 300 dpi, one em is 12/72 inch = 4.23 mm = 50 px, so letter gaps land near 7 px and word gaps near 20 px. A 15 × 3 kernel fuses the 7 px gaps and still holds 5 px of margin before the 20 px word gaps. The white band between lines is about 30 px, and the height rule caps the kernel at 10 px, so 3 px is comfortable.

Use a closing — dilation followed by erosion — rather than bare dilation. The closing fuses the words but restores the letter outlines, so each box hugs the x-height instead of growing by half a kernel on each side.

How do I drop the pictures and noise from the word boxes?

Filter the surviving contours on two medians: keep blobs whose height is 0.5 to 1.5 times the median blob height and whose area is at least 0.3 times the median blob area — photographs fail the height test, speckle fails the area test, and words pass both.

Compute both medians over all contours from a first pass, then filter; because the thresholds are relative, they survive a font change or a reprint. Words also have a recognizable shape: boxes come out 1.5 to 10 times wider than tall, a photo stands taller than the text band, and a rule or border fails the aspect test on the high side. A 3 × 3 opening before the closing removes speck smaller than a letter stroke, which keeps the median estimates honest on dirty scans. What survives is one contour per word, plus the occasional underline — those drop out on aspect, since a full-width rule is wider than 15 times its height.

What does the word-blob pipeline look like step by step?

Eight steps in Java: grayscale, median blur, adaptive threshold inverted so text is white, 3 × 3 opening, horizontal 15 × 3 closing, external contours, median filters, then boundingRect and draw — fewer than 25 lines of Imgproc calls end to end.

1. Convert to grayscale: Imgproc.cvtColor(src, gray, Imgproc.COLOR_BGR2GRAY).
2. Denoise with a 3 × 3 median blur; speck must not survive into the threshold.
3. Binarize with Imgproc.adaptiveThreshold(...) using THRESH_BINARY_INV, block 31, constant 10, so text is white on black.
4. Open with a 3 × 3 kernel to remove residue smaller than a stroke.
5. Measure letter and word gaps from the background runs; set kernel width to about twice the letter-gap median, height 3.
6. Close the mask: Imgproc.morphologyEx(mask, words, Imgproc.MORPH_CLOSE, kernel).
7. Collect external contours with Imgproc.findContours(...) and RETR_EXTERNAL; the hierarchy Mat stays empty.
8. Apply the two median filters, take Imgproc.boundingRect(c) for each survivor, and draw it.

// input: Mat src (BGR page); types from org.opencv.core, calls from org.opencv.imgproc
Mat gray = new Mat();
Imgproc.cvtColor(src, gray, Imgproc.COLOR_BGR2GRAY);            // 1. grayscale
Imgproc.medianBlur(gray, gray, 3);                              // 2. speck removal

Mat mask = new Mat();
Imgproc.adaptiveThreshold(gray, mask, 255, Imgproc.ADAPTIVE_THRESH_MEAN_C,
        Imgproc.THRESH_BINARY_INV, 31, 10);                     // 3. text = white

Mat kernel = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(15, 3));
Mat words = new Mat();
Imgproc.morphologyEx(mask, words, Imgproc.MORPH_CLOSE, kernel); // 4. fuse the letters

List<MatOfPoint> contours = new ArrayList<>();
Imgproc.findContours(words, contours, new Mat(),
        Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE);    // 5. one blob per word

The test that proves the sizing rule builds its own page — four words of five letters, letters 8 × 24 px, letter gaps 6 px, word gaps 26 px:

// run with -ea so the asserts fire
Mat test = new Mat(64, 400, CvType.CV_8UC3, new Scalar(255, 255, 255));
for (int w = 0; w < 4; w++)            // four words
    for (int l = 0; l < 5; l++)        // five letters each
        Imgproc.rectangle(test, new Rect(20 + w * 90 + l * 14, 20, 8, 24),
                new Scalar(0, 0, 0), -1);

// run steps 1-5 on 'test', then:
assert contours.size() == 4;   // 15 px closes 6 px gaps, leaves 26 px gaps open

Mat wide = new Mat();
Imgproc.morphologyEx(mask, wide, Imgproc.MORPH_CLOSE,
        Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(31, 3)));
// rerunning findContours on 'wide' returns 1 contour: 31 px bridges the 26 px word gap

Between the two kernels sits the whole method: pick a width inside the gap band and words separate; step past the word gap and the page collapses to one blob.

How many pixels does each word gap need at capture time?

Give the text three pixel floors at the sensor: at least 3 px across the smallest letter gap, at least 8 px across the word gap, and at least 15 px of x-height; for 12 pt print that caps ground sample distance at 125 µm per pixel.

Twelve-point print gives an x-height near 1.9 mm, letter gaps near 0.6 mm and word gaps near 1.7 mm. The floors demand: 3 px on 0.6 mm → 200 µm, 8 px on 1.7 mm → 212 µm, 15 px on 1.9 mm → 127 µm. The x-height floor binds, so anything finer than 125 µm per pixel passes all three — the same 15 px rule that makes text readable to OCR, argued from the reading side in Minimum font size for OCR: why small text fails and how to fix it.

A 300 mm wide target — an A4 page sideways — on a 4096 × 3000 grid gives 300 ÷ 4096 = 73 µm per pixel, a 1.7 × margin: letter gaps image at 8.7 px, word gaps at 23 px, x-height at 26 px, and the kernel becomes 17 × 3. Run your own numbers in the camera resolution calculator.

Among area scan cameras, a 1.1-inch 12 MP class sensor is the sweet spot for a full page in one shot; the camera finder lists the current range. The MV-CH120-11UC is a 12 MP USB3 camera with a 1.1" sensor, a C-mount and a global shutter — the high-speed version, so the frame is in the buffer while the closing runs.

Which lens and working distance frame the page at that scale?

A 16 mm lens at 356 mm working distance: magnification is sensor width over field of view, 14.1 ÷ 300 = 0.047, and f = WD × m ÷ (1 + m) turns a 300 mm starting distance into 13.5 mm, so the 16 mm catalog focal length wins.

With a 14.1 mm sensor width (4096 px × 3.45 µm) over a 300 mm field, m = 0.047. From a 300 mm working distance, f = 300 × 0.047 ÷ 1.047 = 13.5 mm; the catalog step above that is 16 mm, which sets the real working distance at 16 × 1.047 ÷ 0.047 = 356 mm. The short side then covers 10.35 ÷ 0.047 = 220 mm, more than the 210 mm page height. Cross-check your own geometry in the lens focal length calculator.

In the machine vision lenses range, the LS-FFL-0567 is a 16 mm C-mount lens rated 20 MP for a 1.1" sensor, F2.8-22 — a format and resolution match for the sensor above.

Shoot it near f/4: the Airy disc is 2.44 × 0.55 µm × 4 = 5.4 µm, about 1.6 pixels, small next to the 8.7 px letter gap. At f/8 the disc reaches 10.7 µm and begins to eat the gaps the closing needs. Depth of field at f/4 is 2 × 4 × 0.00345 × 1.047 ÷ 0.047² mm ≈ 13 mm, so a bowed page is no threat. If the page rides a belt at 0.2 m/s, hold blur under half a pixel: 0.5 × 73 µm = 36 µm of smear, so exposure ≤ 36 µm ÷ 200 mm/s = 180 µs — strobe the light and open the aperture rather than raising gain.

Can the pipeline run on a smart camera without a PC?

Yes — an x86 smart camera runs the same OpenCV calls on the device and hands the PLC only the box coordinates, which removes the PC, the grabber cabling and most of the latency budget from the cell.

Morphology and contour passes cost a few hundred operations per blob, so compute is never the constraint; the sensor format is. The LS-SMC-0017 is an 8.8 MP smart camera on an x86 platform, with a 1" sensor, a C-mount and a global shutter. Fit a short prime such as the LS-FFL-0602, an 8.5 mm, 1", 10 MP C-mount lens. At the same 73 µm scale, a 12.8 mm sensor width frames about 175 mm — a full label panel or half a page per shot, usually the right trade against a PC in the cabinet. More options live under smart cameras.

When does morphological word segmentation break down?

It breaks on touching lines, scripts without spaces, and heavy skew: when the vertical gap between lines falls under the kernel height, or the language joins characters (CJK, Arabic, cursive handwriting), no kernel separates words and you need projection profiling or a trained detector.

• Lines that touch. If the white band between lines drops under the kernel height, the closing welds the page into a block. Keep height at or under a third of the band; when photocopy bleed has already joined the lines, split them first with a row-sum projection profile — sum the mask by row, cut at the valleys — then close each band on its own.
• Scripts without word spaces. Chinese, Japanese and Thai set characters at near-uniform pitch, so there is no word gap for the kernel to respect; box phrases or characters, or move to a detector.
• Cursive and handwriting. Joined scripts are one blob per line by nature; a word with no visual boundary cannot be found by geometry.
• Skew. A few degrees of rotation shear the word gaps, so a horizontal kernel cuts long words and joins neighbours across lines. Deskew first — the routine in Deskew a rotated table before OCR with OpenCV and Python works on pages too — or close along each line's baseline angle.
• Washed-out print. When the binarization eats the word gap before morphology runs, no kernel recovers it; fix it at capture with the pixel floors above and flat, even lighting.
• Complex layouts. Text wrapped around photos, multi-column pages and unknown orientation are detector territory: crop each detection and read it, the pattern in Detection to OCR pipeline: crop detected regions then read text.

One reassurance: letters that already touch inside a word cost nothing here, because the closing was going to fuse them anyway. They matter to the character reader — the fixes are in OCR touching letters: the segmentation fix without a new camera.

The decision, then: printed, roughly horizontal text on a plain background — morphology, no training set. Anything else — budget the annotation time for a detector.

How do I verify the word boxes before shipping?

Run the pipeline on 20 or more production images and assert the box count against the expected word count: any pair of boxes overlapping by more than 30% means the kernel has crept past the word gap, so step it down 2 px at a time.

Log three numbers per image: box count, median box height, and the smallest word gap measured on the raw mask. Commissioning goes faster with a live overlay of mask and boxes, the loop used in Live binary image preview for defect inspection.

If the count runs low and boxes run wide, shrink the kernel 2 px; if single words split, widen it 2 px and re-measure — the margin to the word gap must stay above 2 px. When the numbers hold, size the rest of the station in the system configurator. For pages outside the assumptions — unusual scripts, moving webs, mixed media — send samples through the contact page and we will work the numbers with you. More build walkthroughs sit on the systems topic page.