Bayer decode from raw image data when the driver demosaics anyway

A colour camera driver is happy to hand you a finished RGB image, and most of the time that is exactly what you want. The trouble starts when the buffer is not a single Bayer frame: many stereo cameras pack the left sensor's byte into the low half of each 16-bit word and the right sensor's byte into the high half, and the driver's image call still demosaics across the whole thing. The two mono planes never reach the RGB output and cannot be recovered from it. This guide shows you how to pull the untouched buffer from the acquisition layer, split it into two images, decode colour yourself, prove the result is right, and pick cameras and capture hardware that cooperate.

Why does the driver's decoded image lose packed pixel data?

The driver's image call assumes your buffer is one Bayer frame, so it interpolates colour across neighbouring pixels; a frame that instead packs two mono sensors, or two 8-bit samples per 16-bit word, comes out smeared into a meaningless picture and the packed layout cannot be recovered from the RGB output.

A Bayer sensor puts one colour filter on each photosite: one red, one blue and two green in every 2 × 2 tile. Colour reconstruction is therefore interpolation, because the driver must estimate the two missing channels at every pixel from the neighbours its pattern table predicts. Feed it a word that is really two cameras, and it blends the left sensor's low byte with the right sensor's high byte, returning a green-and-magenta mosaic that contains neither image.

The loss is one-way in practice: the image call returns only its finished RGB, never the bytes it started from. Even for a genuine Bayer frame, that decode carries the driver's own white balance and interpolation kernel, so colour numbers move whenever the SDK is updated. If you match colour against a reference, run a decode you control.

A cousin of this problem is 10-bit RGB delivered in 16-bit buffers, which needs its own unpack: Access 10-bit RGB camera data: unpack to 16-bit planes first.

Which acquisition call returns the raw Bayer buffer?

Ask the acquisition layer for the buffer itself, not for an image: the raw-data call returns the untouched bytes and their length, and with the pixel format set to a Bayer format such as BayerRG8 the bytes arrive still interleaved in the sensor's own pattern.

Nearly every SDK ships the pair side by side. The high-level image call formats, white-balances and demosaics for you; the raw-data call, usually named something like get image data or get buffer, hands back a pointer and a byte count. Work from the byte count: it tells you the stride, the padding and whether the frame really holds width × height samples.

On a GenICam camera the buffer's meaning is the PixelFormat feature value. BayerRG8, BayerGR8, BayerGB8 and BayerBG8 carry one 8-bit sample per pixel, while 16-bit names such as BayerRG16 carry a 12-bit sensor's full depth. If the camera reports Mono8 where you expected colour, the format fell back when acquisition started, so request the Bayer format explicitly as described in Camera image encoding falls back to mono8: request Bayer instead.

Before you build a decoder around the buffer, confirm the stream really is raw Bayer: Bayer raw stream from an RGB camera: how to know you have it.

How do you split a 16-bit buffer into two 8-bit images?

Read the buffer as little-endian 16-bit words and take alternate bytes: in a 640 × 480 stereo frame of 614,400 bytes, the even offsets give one sensor's 307,200-byte image and the odd offsets give the other's.

Each sensor's pixel occupies exactly one byte, so the two images sit interleaved in the same words. Split first, decode second; never decode a still-interleaved buffer.

The split itself is one shift per word:

// Illustrative variables and names, not a documented API.
int width = 640;                 // pixels per row, per sensor
int height = 480;                // rows, per sensor
uint8_t raw[614400];             // one acquired stereo frame
uint8_t left[307200];            // plane A, even byte offsets
uint8_t right[307200];           // plane B, odd byte offsets
unsigned word;                   // one little-endian 16-bit sample
for (int i = 0; i < 307200; i++) {
    word     = raw[2*i] | (raw[2*i + 1] << 8);
    left[i]  = word & 0xFF;      // low byte
    right[i] = word >> 8;        // high byte
}

The test that proves the split: cover one lens, grab one frame, and check that the covered sensor's plane is a flat dark 640 × 480 array while the open sensor's plane is a flat grey image. If the dark plane comes out as noise, the byte order is swapped, so exchange the two planes and carry on.

Worked example:

• Buffer: 640 × 480 × 2 = 614,400 bytes, or 307,200 words.
• Plane A: 307,200 bytes, reshaped to 480 rows × 640 columns.
• Plane B: the remaining 307,200 bytes, same shape.
• Nothing left over: 307,200 + 307,200 = 614,400 bytes.

Which sensor owns the low byte depends on the camera, so fix it once with the covered-lens test. Turn the planes into processing-ready images with the method in Convert a raw camera buffer to a processing-ready image. Note that packed formats are a different job: 10- and 12-bit packed data squeezes several samples into uneven byte groups, so unpack first as in Packed Bayer 12-bit conversion to a 16-bit image.

How do you decode an 8-bit Bayer plane into colour yourself?

Decode each plane with the classic three-step copy: keep each pixel's own value in its own colour channel, then fill the other two channels from same-colour neighbours, choosing the neighbour by the camera's documented Bayer phase, one of RGGB, GRBG, GBRG or BGGR.

The phase, meaning which colour sits at the first pixel, fixes every neighbour choice in the plane. The four phases and their 2 × 2 tiles are:

GenICam names encode the phase directly, so BayerRG8 declares an RGGB tile. Write the decode against the phase, not against a camera model:

1. Read the phase from the negotiated pixel format name: BayerRG8 means RGGB, BayerBG8 means BGGR.
2. Create an RGB output of width × height × 3 bytes and copy every pixel's own byte into its own channel.
3. At a red or blue site, copy green from an adjacent left/right or up/down neighbour and copy the remaining channel from the diagonal pixel of the same 2 × 2 tile. At the top-left R of an RGGB tile, green comes from the pixel to the right or below, and blue from the tile's bottom-right corner.
4. At a green site, copy red and blue from the left/right and up/down neighbours; the phase decides which axis supplies which colour.
5. Average the two candidate neighbours where both exist to move from nearest-neighbour to bilinear quality, or keep nearest-neighbour when the output feeds a measurement rather than an eye.
6. Apply your own fixed per-channel white-balance gains so the colour scale is yours, not the driver's.

Precompute a 256-entry table for each phase position and channel, and the whole decode becomes table lookups and byte copies — the idea behind the Bayer-decode lookup tables in classic vision libraries.

What does manual Bayer decode cost in memory and time?

Budget about 2.8 MB of memory traffic per frame: 614,400 bytes read in, 614,400 written as two split planes, 614,400 read back for decode, and 921,600 written as RGB — 83 MB/s at 30 fps, which is small for a PC.

Where that traffic comes from:

• Split: read 614,400 bytes, write 2 × 307,200 = 614,400 bytes.
• Decode: read both planes again, 614,400 bytes, and write 640 × 480 × 3 = 921,600 bytes of RGB.
• Total: 2,764,800 bytes ≈ 2.8 MB per frame, so at 30 fps, 2,764,800 × 30 = 82,944,000 bytes/s ≈ 83 MB/s.
• Mono shortcut: green samples are half of all pixels, 307,200 ÷ 2 = 153,600 per frame, so a luminance measurement can skip the colour decode entirely.
• Archive: a 30 fps, 8-hour shift is 30 × 3,600 × 8 = 864,000 frames; at 614,400 bytes each, that is about 531 GB as raw Bayer8 against 1.6 TB as RGB — keep raw on disk and decode on demand.

Allocate the ring of buffers once and reuse it: per-frame allocation on this path churns about 5 GB per minute for nothing, as explained in Reuse image buffers instead of per-frame allocation.

When should you decode Bayer yourself instead of the driver?

Let the driver decode whenever the buffer is one genuine Bayer frame and you only need colour for matching or display; decode yourself when the frame carries two sensors in one word, when you also need the mono planes for measurement, or when you want a deterministic white balance.

The driver's automatic white balance is the quiet trap: the same part can score differently after an SDK update because its gains moved. A decode under your control with frozen gains gives the same RGB numbers for the same scene, which is what colour matching needs.

How do you verify your Bayer split and decode are correct?

Verify on a uniform grey card first: every pixel of a correctly split and decoded plane should show the same value within a few counts, and the red and blue channels of a colour card shot should swap exactly when you feed the wrong Bayer phase.

Run these checks on every camera you add:

1. Flat grey card on both sensors: each plane is a constant array with a spread of a few counts; stripes mean interleaving survived.
2. Red test card: R sites high, G and B low. Red and blue exchanged means the phase is off by one, so change the phase, not the code.
3. Green-and-magenta checkerboard over the whole image: the decoder ran on data that is not Bayer, such as a mono plane or an unsplit buffer.
4. Cross-check against the driver's decode of a single-sensor frame: per-channel means should agree within a few counts; pixel values will differ because the interpolation kernels differ.
5. If the raw frame itself looks wrong, debug the acquisition before the decode: Debug packed Bayer acquisition before image processing.

Which cameras and capture hardware deliver raw Bayer data?

Any GenICam colour camera can hand you raw Bayer: pick the camera by resolution and link, set its pixel format to a Bayer variant, and on GigE give it a dedicated network port or a multi-port card so the raw stream has the bandwidth it needs.

Confirm the PixelFormat enumeration before ordering: a Bayer 8-bit name covers the split-and-decode path shown above, and a 16-bit name such as BayerRG16 preserves a 12-bit sensor's depth. The camera finder filters colour models by interface, resolution and shutter, and the area scan cameras category is the full list.

For a compact colour camera on GigE, the 2.5 MP MV-CU020-90GC pairs a global shutter with a C-mount.

Budget the link before you buy. A Bayer8 frame is width × height bytes, so 4000 × 3000 is 12,000,000 bytes per frame. At 30 frames per second that stream is 12,000,000 × 30 = 360 MB/s, and GigE carries roughly 118 MB/s of payload at best, so it belongs on USB 3. The 12 MP, 30 fps LS-CMR-0166 is a USB3 camera built for exactly that stream.

Large fields on GigE work at proportionally lower rates: a 20 MB Bayer8 frame leaves about 5 to 6 frames per second of headroom against that payload. The 20 MP MV-CS200-10GC is the colour camera in that class.

Each camera stream needs its own bandwidth: give every GigE camera a dedicated port, or use a dual-port PoE card such as the MV-GE2002P when cameras multiply. It sits in the frame grabbers category beside the multi-port cards for CoaXPress stacks.

To build the chain, start at the camera step of the configurator, or send us your frame format through the contact page and we will check the decode path with you. More on raw acquisition lives on the systems topic page.