Embedded ISP vs GPU for raw image processing on an SoC

You have an embedded board whose datasheet advertises an image signal processor, and a raw sensor to feed it. The natural question — where is the API to program that ISP? — usually gets an unwelcome answer: there is no public per-stage API. The ISP runs under the vendor's camera drivers, and your code sees finished frames. If you build cameras into custom boards and need to know which path your pixels take and what each path costs, the three routes below cover it, with the bandwidth arithmetic that picks between them and the checks that settle the decision before you order anything.

What the ISP block does, and why its registers stay closed

An ISP is a chain of fixed-function stages: black-level clamp, defect pixel correction, white balance and gain, demosaicing of the raw Bayer pattern, lens shading correction, denoise, and color conversion into YUV formats such as NV12. Higher-end blocks add scaling and frame-rate conversion for a display path. The chain works line by line, so internal stages never write full frames to memory; only the finished output lands in DRAM.

Vendors keep the register map out of public documentation because image quality is a product. Each sensor module is tuned with proprietary image-quality (IQ) files, and the register map is the recipe. Two consequences follow. Your user-space handle is the vendor camera stack, not the hardware, and the stages you get are the stages the vendor tuned, in the order they chose.

How the ISP is actually reached: the vendor camera stack

On a Linux board the camera path is a media-controller graph. Sensor, CSI-2 receiver and ISP appear as nodes that a pipeline configuration links; user space then captures through the V4L2 APIs, and many vendor stacks wrap the chain in GStreamer elements so a pipeline string is your whole program. The device tree decides which channels exist at all: one vendor's application notes describe enabling the ISP channels and disabling the direct capture channels in the device tree before anything streams.

So the practical API to an ISP reads: configure the media graph, open the device, read finished NV12 or RGB frames. Where a vendor documents driver bring-up, that procedure is the documentation to follow — for example a step-by-step ISP driver set-up that captures raw sensor frames. For how the capture-and-process split looks on real boards, see MIPI CSI RAW ISP inline processing on embedded boards, and for the sensor-side half of the question, what you can offload to on-camera processing.

Path one: let the ISP finish the image

The default path takes what the stack delivers: NV12 or RGB, already debayered, shaded and denoised. You write no pixel code, the fixed-function stages cost little power, and the vendor's tuning gives a consistent look from unit to unit.

The limits are the flip side of the convenience. You inherit the tuned demosaic, the denoise and the color response. For gauging, colorimetry or defect contrast, those choices can be wrong for your part, and there is no setting that changes them.

Path two: capture raw Bayer past the ISP

Where the vendor documents the media pipeline, raw capture is a legitimate configuration. One processor vendor's SDK documentation puts it plainly: "Once the media pipeline is configured, you should be able to capture raw frames from the sensor using any tool compliant with V4L2 APIs."

Raw means packed Bayer at the sensor's bit depth — 10 or 12 bit — not 8-bit RGB. The bit depth is a format and memory question of its own: see why 10-bit sensors ship 8-bit images and how to store 12-bit frames without losing intensity data.

Not every driver exposes raw formats; some deliver only processed output, so enumerate what the capture device actually offers before planning around raw frames. On boards without a usable ISP you get raw sensor output whether you asked for it or not — what raw output means when a CSI camera has no ISP covers that case.

Path three: debayer and process on the GPU or CPU

This is the industrial-camera default. A USB3 or GigE camera hands the host raw mono or Bayer frames, and the pixel pipeline is your code; the same applies when you capture raw Bayer from a CSI sensor past the ISP. The cost is data movement:

Worked example, 4000 × 3000 at 8-bit Bayer on GigE:

• Raw frame: 4000 × 3000 × 1 B = 12 MB.
• RGB frame: 12 MB × 3 = 36 MB.
• GigE moves 125 MB/s on the wire; after protocol overhead, plan on about 110 MB/s of payload. The stream caps at 110 ÷ 12 ≈ 9 fps.
• A debayer kernel reads the raw frame and writes the RGB frame: 12 + 36 = 48 MB per frame. At 30 fps that is 48 × 30 = 1,440 MB/s of memory traffic, about 1.4 GB/s.

The ISP path writes only the 36 MB output: 1,080 MB/s at the same rate. With one camera, both fit a memory subsystem comfortably. Latency: the bandwidth arithmetic sets a lower bound of microseconds per frame; in practice a debayer kernel on a shared-memory SoC GPU lands nearer 1-3 ms per 12 MP frame once launch overhead, cache behaviour and memory contention are counted — an estimate from typical measured ranges, so measure on your board. If a GPU pipeline underperforms, work through why GPU image processing runs slow; for lens correction in the same stream, real-time image undistortion in embedded pipelines.

ISP versus GPU, stage by stage

The pattern: fixed stages belong in the ISP, custom stages belong in your code, and the tiebreaker is memory bandwidth.

Sizing the data path before you commit

Two more numbers settle most builds.

Memory ceiling. Current embedded parts share one LPDDR subsystem between CPU, GPU and video engines. A 32-bit bus at LPDDR4-3200 moves 3200 MT/s × 4 B = 12.8 GB/s theoretical, and kernels reach a fraction of that in practice. Four cameras at 100 fps through GPU debayer demand 4 × 48 MB × 100 = 19,200 MB/s ≈ 19 GB/s of raw reads plus RGB writes — over that bus's ceiling before anything else runs. Work on mono or Bayer instead of RGB, cut the rate, or move to wider or faster memory.

Link ceiling, USB3 edition. A 3.1 MP USB3 module at its 120 fps maximum moves 3.1 MB × 120 = 372 MB/s of raw data. USB 3.0's 5 Gbps line rate pays 8b/10b encoding, which leaves 400 MB/s of payload before packet overhead: it fits, with under 10 percent margin. On MIPI the same accounting runs lane by lane — what limits frame rate on a MIPI camera lane — and pixel format trades against exposure and rate on color sensors in its own way (how pixel format and exposure cap the frame rate). If acquisition keeps outrunning processing, that gap has known fixes: acquisition faster than processing.

What this means for the hardware you order

If you are choosing a board, read the camera block's documentation the way you read a lens datasheet: codec and connectivity numbers are easy to compare, but documented ISP access is what decides the architecture — how SoC spec sheets trade ISP capability against codecs and connectivity. Choosing the sensor module is its own exercise: board-level camera selection for embedded vision machines.

When the host is an x86 vision controller, the ISP question disappears: the pixel pipeline is entirely your code on the CPU or integrated GPU. An entry build that keeps debayer on the CPU:

The MV-VC2140-128G40-1T is an Intel N-series controller with N97 processor, 8 GB memory, a 128 GB SSD plus a 1 TB HDD, 4 built-in light channels, 3-4 LAN ports and 2 USB 3 ports. The two USB 3 ports carry USB3 camera modules such as the ones below.

Once a GPU joins the plan, step up:

The MV-VC4712-256G10-2TH pairs an Ultra 7 255H with 2 × 16 GB memory and a 256 GB SSD plus a 2 TB HDD, PCIe expansion, PoE camera ports and 6 built-in light channels. Its integrated graphics run the debayer and pre-processing kernels, and the PCIe expansion takes a discrete GPU if the stage count grows.

For the camera side, three modules cover the common cases. First, a color module for the raw GigE path in the worked example:

The LS-BCM-0056 is a 12 MP GigE board camera with 1.6 μm×1.6 μm pixels. On GigE, a resolution of this size moves 12 MB raw frames and caps near 9 fps — the worked example above, in hardware.

Second, a high-rate USB3 module:

The LS-BCM-0003 is a 3.1 MP USB3 board camera with IMX252 sensor, 1/1.8-inch format, global shutter, 120 fps — the link-ceiling example above, in hardware.

Third, the mono shortcut that removes debayer from the conversation:

The MV-CB016-10GM-M-W is a mono GigE board camera with IMX296 sensor, 1.6 MP, global shutter, M12 lens mount. Mono skips the demosaic stage: one byte per pixel from sensor to measurement, with no RGB inflation behind it.

Checks before you commit to a board

• Media graph in the documentation. The vendor describes sensor → CSI-2 → ISP as a configurable media-controller pipeline. Without it, you are adapting a reference design, not using a supported path.
• Raw formats enumerated. The capture device advertises packed Bayer at the sensor's bit depth, not only RGB or YUV output.
• Driver sources and IQ files. Published driver sources with per-module image-quality files mean shading and white balance can be re-tuned; without both, the ISP stages are fixed.
• Memory budget. Sum (raw + output) × fps across every camera and compare with the memory subsystem's realistic bandwidth, not its datasheet peak.
• Thermal headroom. GPU debayer spends watts that ISP stages do not; check the budget at 100 percent duty cycle.
• Latency budget. The GPU path adds frame-quantized stages; a control loop that needs sub-frame response has to count them.

The decision, case by case

• Sensor wired to the board's CSI-2 input and the vendor documents the media pipeline: use the ISP for display and power-tight paths; capture raw Bayer when the measurement needs your own pipeline, and budget its memory traffic with the formulas above.
• Camera on USB3 or GigE into an x86 host: the SoC ISP is out of the conversation entirely, and the pipeline is yours. One camera at modest rates runs on the CPU; multi-camera or heavy pre-processing goes to the GPU.
• Mono sensor: skip the debayer question and spend the freed bandwidth on resolution or frame rate.
• Pipeline ends at a display: let the fixed-function scaling and frame-rate conversion do the work; GPU copies only spend memory.

Shortlist the camera side in the board-level cameras category, filter by interface and resolution in the camera finder, or start a build in the camera configurator. More on the processing side sits on the systems topic page. If the trade is still open after that, contact the applications team.