MIPI CSI-2 image capture pipeline: sensor to host step by step

Connecting a camera module to a microcontroller or SoC over MIPI CSI-2 means designing half of a high-speed serial link and then configuring both halves. The documentation is scattered across sensor briefs, SoC reference manuals and a long industry standard. The whole route is hard to see in one place, and this page is that route: the five stages from photosite to memory, the lane-rate arithmetic with a worked 1080p60 example, where the ISP fits, and what the firmware does frame by frame.

It also gives the bring-up order that separates a wiring fault from a firmware fault. It is written for engineers adding a camera to an embedded board for the first time, and for anyone debugging a link that probes but never delivers a clean frame. The standard itself is published by the MIPI Alliance; the receiver registers come from your SoC manual.

What happens between a sensor's pixels and a frame in host memory?

The chain is always the same five stages: the sensor reads out Bayer pixels, its transmitter serializes them over MIPI CSI-2 lanes, the host receiver deserializes and unpacks the packets, an ISP converts Bayer to RGB, and DMA writes the finished frame to memory.

Every stage is owned by one chip. The first two live in the camera module: the sensor die and its transmitter, sometimes on the same package, sometimes joined by a short flex. The last three live in the host: the PHY and CSI-2 receiver built into the SoC or a bridge chip, the ISP block beside it, and DMA hardware that moves frames into RAM without the CPU touching the pixels.

CSI-2 over D-PHY is the pairing you will meet most often; current transmitter IP commonly advertises CSI-2 v1.3 over D-PHY v2.1. The same D-PHY electrical layer also carries the display link, which is why one SoC often shares PHY blocks between camera input and display output. This chain runs phones, drones and inspection rigs alike, and in machine vision it appears whenever the camera is built into the machine rather than bolted on. For how the connectors and wiring differ from the older parallel camera buses, see DVP versus MIPI CSI-2: why the connectors don't match.

What does a camera module actually send down the lanes?

Most bare camera modules transmit raw Bayer in 8-, 10- or 12-bit packets, while modules with a built-in ISP transmit finished YUV or RGB, so identify the module's CSI-2 data type before you plan any debayer stage of your own.

A module is a sensor plus whatever the manufacturer added in front of its transmitter. The minimal module is a sensor die, a clock source and the CSI-2 transmitter, and it sends exactly what the sensor produced: raw Bayer, usually in an RGGB pattern, as RAW8, RAW10 or RAW12. A module with an on-board ISP performs black level, debayer and white balance before the transmitter and sends YUV422 or RGB888 instead. Neither choice is better in general: raw gives you control and a shorter pipeline; processed gives you a working color image with no ISP work of your own. Monochrome sensors have no Bayer pattern to decode, so raw mono is the simplest format of all: one value per pixel, no color conversion anywhere.

What raw output means, and how to confirm a camera is streaming it, is covered in CSI camera with no ISP: what raw sensor output means. Before mating module to host, match the receiver's supported data types against the module's — MIPI CSI-2 camera compatibility: beyond the connector walks that check.

Packaged board cameras take the pairing question off your desk: the sensor, lane count and bridge are already matched, and the interface leaving the board is USB3 or GigE.

How many lanes do you need, and how fast must each run?

Budget one clock lane plus one to four data lanes at up to about 2.5 Gbit/s each: compute width × height × frame rate × bits per pixel, add roughly 20 percent for packet overhead and blanking, then divide by the lane count to get each lane's bit rate.

The overhead factor covers packet headers, per-line framing and the blanking your timing needs; 1.2 is a fair planning value, and sensor modes with long vertical blanking push it toward 1.3. The clock lane runs at half the per-lane bit rate because D-PHY captures data on both edges of the clock, and it travels its own lane beside the data — a source-synchronous design, so the receiver aligns to the sensor's clock rather than recovering one.

Worked example, 1920 × 1080 at 60 fps in RAW10 on two data lanes:

Every row sits inside the common D-PHY planning maximum of 2.5 Gbit/s per lane, and a four-lane link tops out around 10 Gbit/s aggregate. What moves the number: RAW12 instead of RAW10 adds 20 percent, 120 fps doubles it, and cropping the readout cuts it in proportion — which is why cropping at the source is the first lever to pull when a link is tight. The same arithmetic carried through to frame-rate limits is in MIPI CSI-2 frame rate estimation: lanes and data rate, and what limits frame rate on a MIPI lane covers the receiver side of the budget.

The link is board-level hardware: tens of centimetres of flex or PCB is the comfortable run, which is why lane count and lane rate, not cable, are the design variables here.

Where does the ISP sit, and what does it do to the raw Bayer data?

The ISP runs on the host in most embedded designs: it takes the raw Bayer frames the receiver wrote to memory and applies black-level correction, defect-pixel repair, debayer, white balance, gamma and noise reduction as hardware stages, turning them into usable RGB.

The order matters because each stage needs the previous one's output: black level first so the zero point is right, defect-pixel repair before debayer so a bad photosite cannot smear into its neighbours, white balance on the debayered pixels, gamma last to fit the display range. Noise reduction can sit on either side of debayer; sharpening always comes last.

For measurement work, decide deliberately which stages to keep. Gamma and aggressive noise reduction change gray values, so a metrology pipeline keeps the image linear and takes the raw path instead. White balance across several cameras is a consistency question of its own — multi-camera white balance: test before rejecting shows the test that keeps rejects defensible.

A host without a usable ISP block has two ways out: debayer in software, which keeps a processor core busy at 1080p60, or choose a module with its own ISP and receive finished color. Inline ISP processing on embedded boards compares those two placements.

What does the firmware do, step by step, to pull one frame?

Firmware brings up the sensor over its I2C control bus, configures the CSI-2 transmitter for lane count and bit rate, enables the receiver PHY, then waits for frame-start packets and copies payload packets into memory as finished frames.

1. Power and reset. Bring the rails up in the sequence the sensor brief specifies, hold reset long enough, and confirm the oscillator runs before the first register write; when even the address ACK fails, start with the module-connection checks.
2. Configure the sensor over I2C. Set the internal PLL to the lane bit rate you computed, plus lane count, output size, data type and frame rate. Until these writes land, the transmitter has nothing to send.
3. Enable the receiver. Program the host PHY for the same lane count and bit rate, confirm lane mapping, and clear the error and CRC counters.
4. Wire the capture path. On a Linux-class host this means the sensor appears as a video device; when the sensor probes but no video node appears, the fault is usually in that chain, not in the lanes.
5. Run the capture loop. Queue your buffers and wait for one frame per wait call with a timeout; treat a timeout as a link event rather than a code bug — the one-wait-per-frame pattern shows why.

The two buses are independent. I2C reaching the sensor proves wiring and address and nothing more: a sensor that answers every register read can still be silent on the lanes because its PLL never locked.

What bandwidth does the frame need in host memory, and where does it go next?

Plan on roughly 250 MB/s of memory bandwidth for a 1080p60 YUV422 stream — 124.4 million pixels per second times 2 bytes per pixel — plus whatever your storage or streaming output adds, and size the memory path so capture, ISP and output fit together.

The input side, by format, for the same stream: RAW10 packed costs about 156 MB/s; RAW10 unpacked into one 16-bit word per pixel costs about 249 MB/s; YUV422 costs 249 MB/s; RGB888 costs 373 MB/s. The ISP output and your processing copies multiply these again, so plan memory bandwidth as input plus output plus working copies, never input alone.

Downstream, the image leaves one of two ways. Stored: a modern SD card or eMMC part writes a 1080p60 YUV422 stream with room to spare — check the sustained write figure, not the peak. Streamed: a CSI-2-to-USB bridge turns frames into USB3 traffic, and sizing that data path is its own calculation; radio links carry a fraction of their advertised rate once shared, so budget measured throughput. On the bench, the run from host to PC is short: a 3 m high-flex USB3 cable covers it. The camera data cables page lists camera-side lengths from under 1 m to over 20 m.

How do you bring the link up and prove each layer works?

Bring the link up in layer order — power, then the I2C control bus, then the clock lane, then the data lanes, then packets, then images — and confirm each layer before blaming the next, because a sensor that answers on I2C can still send no CSI-2 data at all.

The test for each layer:

• Control bus: an I2C read of the sensor's identification register returns the expected value.
• Clock lane: receiver status shows the clock lane in high-speed mode at the frequency your PLL targets.
• Data lanes: lane error and CRC counters stay at zero while frames flow.
• Packets: a frame-start, payload, frame-end sequence arrives at the expected frame rate.
• Image: a flat gray card produces a flat image; banding or striping points at lane mapping or bit alignment.

When an image arrives but is wrong — zeros, noise, stripes — probing the pipeline stage by stage separates a link fault from a driver fault faster than swapping parts. For the same discipline applied to a real sensor-to-bridge pair, this layer-by-layer CSI-2 bring-up walkthrough shows each check in order.

When should you skip CSI-2 and use a camera with the interface already built?

Skip CSI-2 when you build one-offs or prototypes: a finished board camera with USB3 or GigE already implemented gets you imaging in days, while a custom D-PHY receive chain costs weeks of register-level debugging before its first clean frame.

Build the chain when the sensor you need has no packaged equivalent, when production volume will amortize the integration work, or when the camera must live inside your own carrier electronics for size or power reasons. Under a few hundred units a year, integration hours dominate unit cost, so the finished camera wins on total engineering spend. The common wrong turn is the reverse: choosing a raw module to save money, then discovering the host ISP needs a full tuning cycle per lens and lighting change before color is trustworthy.

Three finished cameras cover the usual cases:

The LS-BCM-0003 takes motion: 3.1 MP at 120 fps, global shutter, C-mount, over USB3. The MV-CB060-10UM-B is the bare-board route for a custom carrier: 6 MP mono on the IMX178, USB3 off the board. The LS-BCM-0020 is the GigE option for longer runs: 6.3 MP at 18 fps, rolling shutter, CS-mount.

When the host itself has no CSI-2 controller, bridge silicon can feed one from another source — for example, capturing an HDMI signal into a CSI-2 host interface.

Before ordering, check four things: the receiver's maximum lane count and lane rate against your computed bit rate; the module's data type against the receiver's supported list; the flex pin order against the host connector, lane by lane; and the memory budget from the section above. The wider selection process is in board level camera selection for embedded vision machines, and the evaluation checklist for inspection systems turns it into a sign-off sheet. To see what exists before committing to a custom chain, start at the board-level cameras page, filter by interface at the camera finder, or spec the whole chain in the camera configurator. More system-level pieces of this topic sit on the systems topic page.