MIPI CSI RAW ISP inline processing on embedded boards

Continuous 4K video from a RAW Bayer sensor on MIPI CSI-2, with white balance and debayering done before anything touches DRAM: that is the natural wish on an embedded imaging board, and the silicon rarely grants it. What an embedded ISP can and cannot do with a RAW stream is decided in hardware, not software. The pipeline options you actually have are four, and the memory numbers for the case that matters — 4K RAW10 at 30 fps — are computable in a few lines. You will finish able to pick a pipeline, size its DRAM traffic with the arithmetic shown, and name the checks that separate a board that fits from one that does not.

What on-the-fly processing means at the silicon level

On-the-fly processing has one specific meaning: the ISP consumes the RAW stream while it travels from sensor to memory. The path runs sensor → MIPI CSI-2 receiver → ISP → DRAM, and DRAM only ever holds processed YUV or RGB; no RAW frame exists in memory at any point. The common alternative is memory-to-memory: the CSI-2 receiver writes RAW frames into DRAM, and a processing engine — a hardware ISP driven as a memory-to-memory device, a GPU, or the CPU — reads them back afterwards. That round trip is exactly what you were trying to avoid.

On the class of quad-core imaging boards this design usually starts from, the answer is no: the CSI-2 receiver writes RAW to DRAM and nothing in the silicon consumes the stream as it goes past. Newer SoC generations add an ISP, but which side of DRAM it sits on is fixed in hardware. Read the media pipeline graph in the board support package (BSP): a direct CSI-2-to-ISP link means inline processing is real; a standalone memory-to-memory ISP node means the RAW frame stops in DRAM first; no ISP node at all means a GPU or CPU kernel does the debayering. One graph decides the architecture, and no software stack can change it.

The pipeline options and where the RAW frame stops

Every working design falls into one of four shapes. The table shows where the RAW frame lives and what each shape costs.

The first two rows share the attractive property: the RAW frame never reaches DRAM, so there is no bandwidth problem and the processor only handles display, encoding or analysis. They cost tuning control, because a sensor ISP or a BSP's fixed pipeline processes pixels the way its authors decided. The last two rows write the RAW frame to DRAM exactly once; from there, the difference between a hardware ISP and your own kernel is who does the pixel work, not how many memory passes it takes.

Zero-copy handoff: V4L2 buffers and DMABUF

Zero-copy means each RAW frame is written to DRAM once by the capture driver and every consumer reads that same memory. The Linux kernel makes the handshake concrete: the V4L2 capture device allocates buffers and exports them as DMABUF file descriptors, and a GPU compute stack, a memory-to-memory ISP driver, or an encoder imports those descriptors and works on the buffer in place. The contract is documented in the kernel's V4L2 DMABUF I/O specification and the dma-buf framework pages in the mainline kernel documentation. Both ends must agree on allocator, memory type and pixel format; when they disagree, the import falls back to a per-frame copy.

The pipelines that hurt are the accidental ones: capture into a driver buffer, a memcpy into a library's own image container, conversion there, another copy into the display or inference stage. Each hop is a full-frame copy on a CPU core. Converting a raw camera buffer into a processing-ready image handles the format step itself, and buffer ownership in frame conversion explains why a misplaced copy also produces hangs, not only slowdowns.

Worked example: DRAM traffic for 4K RAW10 at 30 fps

The case is continuous video: 3840 × 2160 pixels (4K UHD), CSI-2 RAW10 packed at 10 bits per pixel, 30 fps, and an RGB888 output of 3 bytes per pixel, with debayering and white balance fused into one kernel.

The rule: count DRAM traffic as frame bytes × fps for every full-frame write and for every pass that reads or writes a full frame. One term per pass; nothing else moves.

1. RAW frame: 3840 × 2160 × 10 bits = 82,944,000 bits = 10,368,000 bytes ≈ 10.4 MB.
2. Capture write: 10.4 MB × 30 fps = 311 MB/s ≈ 0.31 GB/s.
3. RGB888 output: 3840 × 2160 × 3 bytes = 24,883,200 bytes ≈ 24.9 MB.
4. Fused kernel, one read of RAW and one write of RGB: 10.4 + 24.9 = 35.3 MB per frame; 35.3 MB × 30 = 1,059 MB/s ≈ 1.06 GB/s.
5. Zero-copy total: 0.31 + 1.06 ≈ 1.37 GB/s.
6. CSI-2 link: 82,944,000 bits × 30 = 2.49 Gbit/s of payload. Over four lanes that is 622 Mbit/s per lane, well inside the 1.5 Gbit/s per lane MIPI D-PHY v1.2 allows. Over two lanes it needs about 1.3 Gbit/s per lane before packet and blanking overhead — no margin. Choose a four-lane sensor for this stream.
7. The naive chain — a separate white-balance pass reading and writing the 24.9 MB RGB frame adds 49.8 MB × 30 = 1.49 GB/s — runs near 2.9 GB/s before a display or encoder copy adds another 0.3-0.75 GB/s.
8. A memcpy of every RAW frame moves 10.4 MB per frame; at the roughly 5 GB/s an embedded-class core sustains for copying, that is about 2 ms per frame, some 6% of one core at 30 fps.

The result: a zero-copy pipeline with one fused kernel runs this stream on about 1.4 GB/s of DRAM traffic. Each extra full-frame stage adds 0.75-1.5 GB/s, and a chained pipeline can triple the total.

Decision: fuse white balance into the debayer kernel, keep the RAW frames in the DMABUF ring, and hold the total near 1.4 GB/s. Read the same figures from the SoC's bandwidth counters on your own board; when measured traffic passes roughly half of usable DRAM bandwidth, cut a pass, drop to RAW8 or YUV where the task allows, or crop the ROI at the sensor — a region half the width and height quarters every term above.

Fuse white balance into the debayer pass

White balance on RAW is one gain multiply per pixel: each pixel carries a single channel's value, so multiply it by that channel's gain, then interpolate. Riding in the debayer kernel keeps the pass count at one. Bilinear interpolation reads four neighbor taps per output pixel and is the cheapest; Malvar-He-Cutler reads a 5×5 neighborhood and recovers most of the sharpness bilinear loses, still in a single pass over the same RAW buffer. Produce the gains from per-channel sums computed on every tenth frame at full resolution, or every frame at one-eighth scale — either way the statistics cost a rounding error next to the debayer itself. Where color fidelity dominates, weigh an optical light-balancing filter against software gains; the filter-versus-software white balance comparison walks that choice.

When to skip the ISP fight: sensor ISP or a standard camera interface

If your sensor module can output YUV422 or RGB565 through its own ISP, the board never sees RAW and the whole question disappears. Accept two trades: you tune color through the sensor's registers rather than your own kernel, and most global-shutter machine-vision sensors are RAW-only, so this path usually means a rolling-shutter module — fine for static scenes, wrong for moving parts.

There is a second exit: keep the sensor off the CSI-2 link altogether. A board-level camera over USB3 hands the host finished image buffers through the camera's own SDK, so the SoC ISP and the RAW frame never enter the design; the board camera selection guide for embedded machines works through that trade. The MV-CB060-10UM-B is a 6 MP bare-board USB3 camera with an IMX178 sensor, 1/1.8" format, rolling shutter, mono. The LS-BCM-0003 pairs a 3.1 MP IMX252 global-shutter sensor with 120 fps over USB3 on a C-mount.

For camera-to-processor runs of a few meters and up, GigE gives the same finished-buffer model on standard cabling. The LS-BCM-0020 delivers 6.3 MP over GigE at 18 fps with a CS-mount; the LS-BCM-0056 carries a 12 MP sensor with 1.6 μm×1.6 μm pixels on a C-mount. Both use rolling shutters and sit comfortably at inspection frame rates that are not chasing 4K video.

For an inspection station rather than an embedded video path, buy the acquisition solved: a standard area scan camera with its SDK starts your code at the image, not at the pipeline. The MV-CU020-90GC is a 2.5 MP GigE color camera with global shutter and C-mount, a frequent anchor for such stations; the camera finder filters the rest by resolution, interface and shutter, and the embedded camera interface comparison from CSI-2 to GMSL2 settles the link when the camera sits far from the processor.

Checks before you commit

1. Read the media pipeline graph in the BSP and find the RAW path: a direct CSI-2-to-ISP node means inline exists; a standalone memory-to-memory ISP node means one DRAM round trip; no ISP node means the kernel is yours to write.
2. Confirm the receiver accepts the sensor's format and resolution before ordering hardware: resolution limits live on the host side, and a mismatch usually announces itself as a format negotiation error at startup.
3. Benchmark the fused kernel on the target board at the target frame rate with the worst-case scene — low light pushes gain and noise — reading CPU load per stage and DRAM bandwidth from the SoC's performance counters.
4. Match shutter to motion: rolling for static scenes, global for moving parts. That is a sensor choice, not a software setting.
5. Budget the link before the board: lanes × lane rate must exceed payload plus overhead. Four lanes carry 4K RAW10 at 30 fps; two do not.

The decision, restated: do not design around on-the-fly RAW processing, because on most embeddable boards it does not exist. Write the RAW frame to DRAM once, share it as DMABUF, and fold debayering and white balance into one kernel. If the product is an inspection machine rather than a video pipeline, start from a finished-frame camera and let the SDK carry acquisition. The camera configurator steps through sensor, interface and lens in that order, and the systems topic page collects the rest of the pipeline work.