Unencoded capture to a NumPy array: skip JPEG encode and decode

A capture loop that saves JPEG snapshots and decodes them again pays three taxes: encoder time on the camera, decoder time on the host, and a fresh multi-megabyte allocation per frame. Streaming the raw pixel format into a pre-allocated NumPy array removes all three: one transfer, one reshape, one view. This guide shows how to switch a USB3 or GigE board-level or area scan camera to unencoded capture, size the buffer correctly — including the padded block dimensions that break naive reshaping — and prove the array before inspection code touches it. The arithmetic is worked for a 4000 × 3000 sensor, and the same steps apply to any camera whose SDK hands you raw bytes. If you are still choosing hardware, start with the board level camera selection guide.

What do you need for unencoded capture into a NumPy array?

You need three things: a camera that streams an unencoded pixel format such as Mono8 or BGR8 over USB3 or GigE, its SDK or GenICam driver to hand you the raw byte buffer, and NumPy to reshape those bytes in place.

Expanded into a working list:

• A camera that exposes an uncompressed format. Mono8 or Mono10/12 on a mono sensor, RGB8 or BGR8 on color. A module that only serves a compressed stream cannot skip the decoder — the encoder has already run on the device.
• A capture path that hands you bytes. The SDK's buffer callback, a GenICam driver, or a frame-grabber call that returns the raw image data as a flat array of unsigned 8-bit values. Frame grabber APIs expose exactly this so you can reshape without a conversion step.
• NumPy, and OpenCV downstream. OpenCV functions accept an interleaved BGR array directly, so the reshaped frame needs no color conversion before filtering, thresholding or decoding.
• A link budget. The next section's arithmetic decides whether the loop runs at the rate you want.

Both compact board-level cameras and full-size area scan cameras serve this pattern; the camera finder filters by interface, resolution and shutter.

How do you switch the camera from JPEG snapshots to unencoded frames?

Set the camera's pixel format to an uncompressed one — Mono8 on a mono sensor, BGR8 or RGB8 on a color sensor — and grab frames as raw buffers rather than snapshots, which run a JPEG encoder before the bytes ever reach your process.

1. Set the pixel format before acquisition starts. On GenICam cameras the feature is PixelFormat; choose Mono8 for a mono sensor, BGR8 or RGB8 for color, and avoid every entry with JPEG, MJPEG or a compressed suffix.
2. Replace the snapshot or still-capture call with the SDK's streaming acquisition object. The snapshot call is the stage that runs the encoder; the stream hands you the sensor's bytes.
3. Read the format back after acquisition starts and compare it with what you set. Formats can be fixed per sensor or firmware build, and a silent fallback is a known failure — the guide to pixel formats that refuse to change lists the checks.
4. Capture into one reused byte buffer per frame instead of a fresh allocation, so the same memory serves every iteration and the allocator never enters your timing.

Board cameras that deliver these formats straight over USB3 or GigE:

How big is the raw buffer, and can your link carry it?

The buffer is width × height × bytes per pixel — 12 MB per Mono8 frame and 36 MB per BGR8 frame at 4000 × 3000 — so a 10 fps loop puts 360 MB/s on the link, which USB3 carries and GigE does not.

Mono8 spends 1 byte per pixel and RGB8 or BGR8 spends 3. As denominators, GigE carries about 118 MB/s of payload (a 125 MB/s line rate less protocol overhead) and USB3 sustains about 380 MB/s streaming — typical sustained host-capture figures. MB here means one million bytes.

At 4000 × 3000:

The arithmetic: 4000 × 3000 × 1 B = 12,000,000 B = 12 MB, and × 3 gives 36 MB. Then 118 ÷ 12 = 9.8 fps, 118 ÷ 36 = 3.3 fps, 380 ÷ 12 = 31.7 fps and 380 ÷ 36 = 10.5 fps.

Decision: Mono8 unencoded fits GigE to about 10 fps at this resolution. BGR8 at 10 fps demands 36 MB × 10 = 360 MB/s — inside USB3's 380 MB/s, far over GigE's 118. To keep color on GigE, crop the ROI at the source — halving both dimensions quarters the frame to 9 MB — or take the compressed path in the last section.

The LS-CMR-0166 is a 12 MP USB3 area scan camera listing 4000 × 3000 with a 30 fps ceiling. Its Mono8 mode sits just under the 31.7 fps the USB3 column allows.

Area scan cameras that stream the same formats for standard inspection cells:

How do you reshape the padded buffer into a (height, width, 3) image?

Reshape the flat uint8 buffer to the padded dimensions, then crop back to the set resolution: round width up to a multiple of 32 and height up to a multiple of 16 on block-aligned pipelines, or use the row stride your SDK reports.

Two geometries cover nearly every camera:

• Block-aligned pipelines — embedded boards and module ISPs — write the image in fixed blocks: width rounds up to the next multiple of 32 columns, height to the next multiple of 16 rows. The padding is real bytes in the payload, not slack.
• Stride-reporting SDKs — GenICam drivers and frame grabbers — state it outright: padded width = row stride ÷ bytes per pixel, padded height = payload ÷ stride. Prefer those numbers when the SDK prints them.

The block case at 4000 × 3000:

import numpy as np

width, height, channels = 4000, 3000, 3   # int: the resolution you set
fwidth  = (width  + 31) // 32 * 32        # 4000 — 32 × 125, no column padding
fheight = (height + 15) // 16 * 16        # 3008 — 8 padded rows

# raw_bytes: one frame from the SDK, len == fheight * fwidth * channels == 36,096,000
frame = (np.frombuffer(raw_bytes, dtype=np.uint8)
         .reshape(fheight, fwidth, channels)
         [:height, :width, :])            # crop the padding off
print(frame.shape)                        # (3000, 4000, 3)

The slice [:height, :width] is the step people skip: here it drops 8 rows, and on a sensor with 3096 active columns it would drop the 8 pad columns that round 3096 up to 3104. When the alignment is unknown, derive it from one frame: 36,096,000 ÷ (4000 × 3) = 3008 rows, which reproduces the block rule exactly.

Why does np.frombuffer refuse to reshape the frame?

The reshape throws a ValueError when the buffer's byte count does not equal height × width × channels: at 4000 × 3000 BGR8 the padded buffer holds 36,096,000 bytes, but the naive (3000, 4000, 3) shape needs 36,000,000, so NumPy refuses.

Work down the list; the first cause covers most frames.

1. Naive shape. The payload carries the padded image. Reshape to (fheight, fwidth, 3) and crop as above; the 96,000-byte gap is 8 padded rows of 4000 × 3.
2. Packed bit depth. Mono10 and Mono12 packed formats store pixels below one byte each on average, so a width × height uint8 view cannot exist. Take the SDK's unpacked 16-bit format, or unpack per the documented packing rule. If color arrives as mono, the encoding fell back silently — see when the encoding falls back to Mono8.
3. Planar order. YUV-family formats store channels as planes, not interleaved. Reshape to the single-channel padded shape and split the planes; a (height, width, 3) view scrambles them.
4. Appended data lines. Some cameras add metadata lines per frame, so the payload exceeds the image by a documented offset. Strip the tail before reshaping.

And one symptom that survives a successful reshape:

How do you prove the array is right before processing it?

Feed the pipeline a synthetic payload of known values once, reshape and crop it, and assert the corners: if channels 0, 1, 2 land in row 0, column 0 and the last pixel reads 255 in channel 2, the mapping is right.

import numpy as np

raw = np.arange(fheight * fwidth * channels, dtype=np.uint8)  # synthetic frame: 0, 1, 2, ...
frame = raw.reshape(fheight, fwidth, channels)[:height, :width, :]

assert frame.dtype == np.uint8
assert frame.shape == (height, width, channels)
assert frame[0, 0, 0] == 0                      # first pixel, channel 0
assert frame[0, 0, 1] == 1                      # channels interleaved, not planar
assert frame[height - 1, width - 1, 2] == 255   # flat index 35,999,999; % 256 = 255

The last assert is the proof. The last in-bounds pixel sits at flat index ((2999 × 4000) + 3999) × 3 + 2 = 35,999,999 inside the padded buffer, and np.arange wraps modulo 256, so channel 2 there must read 255. When it does, rows, columns and channel order all line up; had a planar layout slipped in, channel 1 at pixel (0, 0) would not be 1. Run the same asserts on live frames for a few hundred iterations before trusting the loop: a failure that appears only after minutes of running points at buffer reuse, not the reshape.

How do you keep the loop fast frame after frame?

Reuse one pre-allocated buffer, keep the reshape and crop as views, and copy only when the algorithm must write, because np.frombuffer returns a read-only array and every extra copy moves 72 MB per frame — at 10 fps that is 720 MB/s of pure memory traffic.

Four habits decide whether the reshape stays free:

• Reuse one buffer. Rewind the byte stream or recycle the SDK's buffer objects each frame. Allocating 36 MB per frame at 10 fps pushes 360 MB/s through the allocator for nothing.
• Stay on views. np.frombuffer wraps the bytes read-only, and reshape and slicing stay views. Copy once, and only when your algorithm writes into the image.
• Split capture from processing. A producer thread reshapes and queues frames; a worker consumes them, so the grab rate survives a slow stage. The acquisition-faster-than-processing fix shows the pattern, and burst capture through the camera buffer covers the stalls you cannot split.
• Budget for the slowest stage. The loop rate is the minimum of sensor, exposure and link limits — the color frame-rate guide works that minimum out — and when processing is the slow stage, shrink before decoding, as in downscaling before barcode decoding.

More loop patterns live on the systems topic page.

When is JPEG still the better choice?

Stay compressed when the link, not the CPU, is the limit: JPEG shrinks a 4000 × 3000 frame roughly tenfold, so about 3.6 MB crosses a 118 MB/s GigE link at 30 fps, where the raw 36 MB frame manages barely three.

The decode still runs on the host — one JPEG decode per frame at full frame rate — and compression costs a few percent of sharpness at block edges, plus a decode library whose quirks surface as corrupt JPEG data errors. The decision per case:

• Mono inspection to 12 MP at up to about 10 fps on GigE: Mono8 unencoded. No decoder, no compression artifacts, and the link fits.
• Color at 4000 × 3000 on GigE above about 3 fps: a compressed stream or a faster link; unencoded color needs USB3 at this size.
• Embedded boards with the sensor on the same memory bus: unencoded BGR8 wins outright — the link is memory bandwidth and there is no decode stage to pay for.
• Long GigE runs to a modest host: MJPEG trades pixels you can spare for bandwidth you cannot.

Pick the camera by formats and link budget rather than by snapshot demos: the configurator builds the chain around your sensor, and the applications team answers format questions at /contact/.