OCR on an embedded AI accelerator: the detect-then-read pipeline

If you are sizing OCR for an embedded accelerator inside a smart camera or an AI inference terminal, split the job before you pick a model. The architecture that works is a two-stage pipeline: a one-stage detector finds where the text is, then a small CRNN recognizer reads each crop against a frozen character set with a CTC decoder. This guide gives you the arithmetic behind that split: a weights budget you can check, a per-form latency formula worked through a 14-field example that lands at 143 ms, the crop and input sizes that keep both stages accurate, and the decoder settings that stop a correct model returning gibberish. It closes with the hardware split — what runs on the camera, what runs on the inference terminal, and when the reader belongs on a vision controller with PCIe expansion.

Why one end-to-end model does not fit the accelerator

Run the weights arithmetic before you pick a model. Weights in memory equal parameter count times bytes per parameter: 1 B for an int8 graph, 2 B for fp16.

The two-stage stack is small. A YOLO-class detector with about 9 M parameters exports to 9 MB in int8. A CRNN reader with about 12 M parameters exports to 12 MB. Together that is 21 MB of weights plus activation buffers — small enough for both graphs to stay resident on the accelerator at all times.

The end-to-end alternative, one document transformer that takes the image and emits text, sits in the 200-300 M parameter class. At 250 M parameters in fp16 that is 250 × 2 = 500 MB of weights, about 24× the two-stage pair, before activation buffers and decode state. It also emits text token by token: hundreds of small sequential launches that an NPU compiler, built for static graph execution, pushes back to the CPU. Latency stops being predictable exactly when a cycle time needs it to be predictable.

Keep both small graphs on the module and host anything bigger elsewhere. The same asymmetry between model size and memory shows up in general deployments — see why small neural network models use large GPU memory.

The two-stage pipeline, end to end

The pipeline runs six steps per frame:

1. Grab one frame at native resolution — 2592×1944 for a 5 MP camera.
2. Letterbox it into the detector's 640×640 input: the canvas holds the frame at 640×480, padded top and bottom.
3. Run the detector once; it returns a box and a class label for every text region.
4. Scale each box back to the original frame (multiply by 2592/640 ≈ 4.05 on both axes) and pad it by about 10% of its size.
5. Resize each crop to 32 px height, keep the aspect ratio, pad the width to 320 px, and run the recognizer once per crop.
6. Decode each output with CTC, attach per-character confidence, and pass or flag the field with a grammar check.

One formula carries the latency and memory arithmetic for the whole chain:

This is the crop-then-read shape described in detection to OCR pipeline: crop detected regions then read text, and the same shape a license plate OCR pipeline uses: one detector that sees the whole frame, one reader that sees only the crops.

Stage 1: a detector that finds text, not reads it

A YOLO-class one-stage detector fits: one pass per frame, boxes out, no region proposals. Train one class per field type — date box, serial number, amount, code — when the fields need different downstream grammar, and a single text class when they share one grammar. Parsing the output of a YOLO-class detector is one result list per processed frame.

Watch the geometry. The letterboxed canvas of a 2592×1944 frame is 640×480, so one canvas pixel covers 2592/640 ≈ 4.05 original pixels. Detection stays reliable while text stands at least 40 px tall in the original, which is about 10 px on the canvas. That sets the capture rule: 10 pt text at 300 dpi is 10/72 in × 300 dpi ≈ 42 px tall and passes; the same text at 200 dpi is about 28 px and sits below the floor — the same size floor behind why small text fails OCR. Capture at 300 dpi equivalent, or read small fixed text with ROIs at native resolution.

Deploying the detector on the camera itself is often the cleanest split, as covered in running custom object detection models on embedded smart cameras.

Stage 2: read each crop against a frozen charset

The reader is a CRNN: a convolutional front end feeding a bidirectional LSTM, trained with CTC loss. Standard input geometry is 32 px height with the aspect ratio kept; a width up to 320 px covers most single-line fields. The network emits roughly one time step per 8-16 px of width, so a 32×320 pass yields 20-40 time steps. Compute scales with width: a 32×640 pass costs about twice a 32×320 pass, so split long strings at word gaps instead of raising the width.

Freeze the character set first. A practical industrial charset is 10 digits + 26 uppercase letters + 9 punctuation marks + 1 blank token = 46 classes. Every class needs training examples, and a character added later means retraining and revalidating the reader — cover every field you might inspect, then stop editing the list.

Training data is mostly synthetic: render your real fonts over photographed backgrounds with random skew, blur and contrast, then hold out real crops to score against. The cropping half of that job is covered in build your own OCR training data from cropped character images.

Decoding the CTC output without gibberish

Greedy CTC decoding takes three steps and runs in microseconds:

1. Take the argmax class at every time step.
2. Collapse consecutive repeats into one label.
3. Remove the blank token.

Raw labels 3, 3, blank, 7, 7, blank, A collapse to 3, blank, 7, blank, A, then strip to 37A. Field confidence is the weakest kept character's probability; start accepting at 0.90 and flag anything below until your holdout gives you a better threshold.

When a correct crop still decodes to nonsense, the network is almost never the cause. Three mismatches account for nearly all of it: the blank token's index, which exports place first or last; the order of your charset array against the training order; or preprocessing that drifted from training, usually the 32 px input height or the normalization range. This failure is well known enough that the first thing to check is the charset mapping, not the model: dump the raw label sequence for one known crop, and if it looks structured but decodes wrong, the mapping between class indices and characters is at fault.

Worked example: a 14-field form on a 13 TOPS-class module

Put the formula to work on a typical job: one 5 MP camera, one form with 14 printed fields, both networks quantized to int8 on a 13 TOPS-class embedded accelerator.

143 ms leaves headroom: the detector keeps running at frame rate on the same module while the reader steps through the crops.

What moves the result, and by how much:

• Each extra field adds about 7 ms. A 30-field form lands at 45 + 30 × 7 = 255 ms, about 4 forms/s.
• Doubling crop width to 640 px doubles t_read: 14 fields then cost 45 + 196 = 241 ms.
• Raising the detector input to 960×960 multiplies its pixel count by 960²/640² = 2.25, so expect t_detect near 90-100 ms.
• Quantizing an fp16 graph to int8 typically returns 2-4× on NPU-class silicon; if your first measurement came from fp16, re-measure after quantization.

The 45 ms and 7 ms values are the class of number this hardware returns with int8 graphs. Measure your own two values once; the formula carries everything else.

Skip the detector when the form cannot move

If the form sits in a fixture at a repeatable position, teach the 14 ROIs once and drop the detection pass entirely. The pipeline becomes 14 × 7 = 98 ms, about 0.10 s per form, with no detector to train and no letterboxing step. That is the fastest and cheapest variant, and the right default for label-in-nest and nameplate-in-fixture stations.

ROIs hold while placement repeats within about a quarter of the character height — roughly 10 px of drift at 42 px text. Past that, box edges start clipping characters, and a detector that relocates the fields every frame pays for itself. Mixed part types, hand-placed parts and variable layouts always belong behind the detector.

Which hardware runs which stage

The capture stage wants a global shutter when parts move, and 5 MP keeps 10 pt text above the detection floor at 300 dpi. The MV-SC6050C-08M-XBN supplies 5 MP color frames over GigE; the smart camera finder filters by resolution, shutter and lens mount.

Both networks run on one AI inference terminal. The MV-VAC3504X-128G01 takes the camera stream on its 5+ LAN ports and drives the light from its 4 built-in channels, so camera, light and accelerator share one chassis.

When two cameras feed one station, or a burst of forms must clear inside one cycle, split the stages across two accelerators. The MV-VAC5724-256G02 is the dual-card build with PoE camera ports and 8 built-in light channels.

Handwriting, many scripts and free layout overflow a frozen charset. Keep the detector on the accelerator and host the reader on a vision controller with PCIe expansion, so the recognizer gets the CPU headroom it needs while detection stays on real-time silicon.

When the usual answer is wrong

Three cases break the two-stage pattern, and each has a clean redirect:

• One code per part. If the job is really one barcode or QR code per part, a code reader vision sensor does detection, decoding and decision on the device; a two-network pipeline is overhead.
• Curved or rotated print on the part. A bigger reader does not fix geometry; preprocessing does. Deskewing curved cap print into a single line, as in deskew curved cap text into one line for OCR, recovers accuracy without touching either model.
• Questions about the document, not just the fields. Classifying the form type, finding fields with no fixed label, reading handwriting — these need language context a CTC reader does not carry. Keep detection on the accelerator and run the reader on a host; the two-stage split still applies, only the reader's home changes.

Checks before you commit

Five checks, in order:

1. Freeze the charset and record the blank index beside the model build.
2. Golden-crop test: run 20 known crops through the exported graph and your decoder. Exact strings pass; gibberish means charset order or blank index, not the network.
3. Measure t_detect and t_read once on your module, then compute t_form = t_detect + N × t_read with your real field count.
4. Hold out 100+ real crops per character class and score them skewed ±5°, at worst-case contrast and lighting; the per-condition method in verify object detection model generalization transfers directly.
5. Gate on grammar first, confidence second: a date format or a checksum rejects wrong reads that score above 0.95.

Pass all five and the pipeline is production-shaped: two graphs totaling 21 MB of weights, a per-form time you computed rather than guessed, and a decision for every field.