Stream camera images to disk at high frame rate

Your camera acquires at full speed until you add saving, and the rate collapses — 250 fps in memory, 50 fps to disk. The camera is rarely the culprit; per-image files, on-the-fly compression and thousands of small write calls are. This walkthrough sizes the sustained write rate from resolution, bit depth and frame rate, then builds the pipeline that keeps up: raw frames appended to one open file, written in large blocks from a queue the camera thread never touches. The same arithmetic picks the disk, the interface card and the recording volume, and proves afterwards that no frame was lost. It applies to area scan cameras logging short high-rate runs; for several cameras on one machine, start from multi-camera recording to disk without dropped frames and come back here for the single-stream pipeline.

How much write speed does the camera need?

Multiply width by height by bytes per pixel by frames per second: at 640 × 480, 8-bit mono and 250 fps that is 640 × 480 × 1 B × 250 = 76,800,000 bytes per second, about 76.8 MB/s (73 MiB/s) of sustained sequential writing.

The worked example this page keeps returning to: one 640 × 480 mono frame is 640 × 480 × 1 B = 307,200 bytes, close to 0.3 MB, and the rate above is what the disk must absorb continuously. A 30-minute capture is 307,200 B × 250 fps × 1,800 s = 138 GB. Budget in bytes as the camera delivers them, never in compressed file sizes.

A 10- or 12-bit sensor usually reaches you as 16-bit words unless you pack it, so the first row is the realistic case for saving 12-bit camera images without losing intensity data. One field check on the same math: an 8-bit 640 × 480 PNG that truly stores one byte per pixel lands near 300 kB. Files arriving far larger mean the pipeline is delivering more bytes per pixel than you planned for.

Why does one PNG per frame drop the frame rate?

Each saved image pays three taxes at once — a file create, open and close through the OS, a directory and journal metadata update, and CPU time to losslessly encode the pixels — and at 250 fps you have 4 ms per frame to pay them all in.

One representative case: a 640 × 480 camera acquired at 250 fps, fell to about 50 fps while every frame was written as its own PNG, and returned to about 180 fps once the same loop wrote into a single open video-file session. The disk was never the limit; the file management was.

• Per-file overhead: thousands of new files per minute make filesystem metadata the bottleneck, and each create and close costs milliseconds.
• Compression: lossless encoding spends CPU on every pixel of every frame.
• Small writes: one 300 kB write per frame is 250 kernel write transitions per second; batching into multi-megabyte blocks drops that below ten.
• Background scanners: on Windows, indexing and real-time scanning commonly inspect every new file, adding work exactly where you can least afford it.

The decision: keep per-image files below about 20 fps, where a 50 ms per-frame budget swallows a few milliseconds of overhead without noticing. Above that, write a single file.

What file format should the frames go into?

Write raw, unconverted frames into one file you open once and append to sequentially, with a small header recording width, height, pixel format and frame count; that removes compression and per-file overhead completely.

In the raw route the pixels land exactly as the camera delivered them, in fixed-size records, with nothing recomputed. You write a header of a few dozen bytes, then append 307,200-byte records until the run ends. Reading simply reverses it — which is why the geometry lives in the header instead of in filenames.

An AVI-style video file is the middle ground: still one open file and one append per frame, but playable in standard viewers. It routes frames through the OS media layer and assumes 8 bits per channel, so it suits quick 8-bit review more than archival capture. When 12-bit intensity must survive, record a 12-bit grayscale video file without dropping to 8 bits covers the format side; the raw stream sidesteps the question by never converting at all.

How do you keep the camera thread free of disk work?

Split the work in two: the acquisition callback copies each frame out of the driver buffer, stamps it and pushes it onto a queue, while one writer thread drains that queue in large blocks — the camera thread never calls the filesystem.

1. Open the output file and write the header with the frame-count field set to 0, then preallocate the length.
2. Set the driver ring buffer to hold at least 1 s of frames — 250 frames here — so a slow write never blocks acquisition.
3. In the frame callback, copy the pixels out of the driver buffer, append a 4-byte running counter, enqueue the record, return. No file calls in the callback.
4. The writer thread accumulates records until the block reaches the size chosen below, then issues one write call.
5. On stop, flush the partial block, seek back to the header, write the true frame count, and close.
6. To read, parse the header and walk the fixed-size records.

# example variable names; types and starting values shown
BYTES_PER_FRAME: int = 640 * 480 * 1   # 307200 bytes as delivered
BLOCK_MIN: int = 8 * 1024 * 1024       # flush threshold, 8 MiB
pending: bytearray = bytearray()       # writer-thread accumulation

def on_frame(driver_buffer):           # acquisition thread
    queue.put(bytes(driver_buffer) + counter_bytes())  # 4-byte index

def writer(stream):                    # the only thread touching disk
    while True:
        pending.extend(queue.get())    # blocks until a frame arrives
        if len(pending) >= BLOCK_MIN:
            stream.write(pending)
            pending.clear()

The test that proves the writer correct is arithmetic: after N frames, the file size must equal header size + N × 307,200 bytes, exactly. The queue is the shock absorber — when the disk hesitates, frames pile up in RAM instead of the driver buffer, the same gap described in acquisition faster than processing: fix the frame rate gap. The identical producer/consumer structure in a graphical language is laid out in a producer/consumer loop for streaming camera images to disk.

How big should each write call be?

Aim for 8-32 MB per write: at the worked-example rate an 8.3 MB block (27 frames of 307,200 bytes) leaves roughly every 0.11 s, cutting kernel write calls from 250 per second to about 9 while adding only a tenth of a second of stop latency.

Smaller blocks bring the per-call overhead straight back; larger ones raise memory use and stop latency for almost no gain, because past a few megabytes the disk is already writing sequentially. The queue behind the block costs little memory: 1 s of 8-bit backlog is 250 frames of 307,200 bytes — well under 100 MB — and even 10 s of the 153.6 MB/s 16-bit stream is about 1.5 GB. Commit to 8 MB blocks below 300 MB/s and 32 MB above it.

Which disk, interface and controller keep up?

A single SATA SSD sustains about 550 MB/s — near the ceiling of its 6 Gbit/s link — and clears the 8-bit worked-example stream with a 7× margin; a 7200 rpm hard disk manages 100-200 MB/s, enough for the 8-bit stream but marginal once 16-bit demand passes 150 MB/s.

The hard-disk figure is an outer-track number that falls as the platter fills, so keep the capture volume mostly empty; a two-drive stripe roughly doubles it when one disk cannot reach the rate. Capacity is arithmetic, not preference: 2,000 GB ÷ 0.0768 GB/s ≈ 26,000 s ≈ 7.2 hours of the 8-bit stream fits a 2 TB volume. In the vision controllers family, the MV-VC4712-256G10-2TH pairs a 256 G SSD for the OS with a 2 TB HDD for capture, and the MV-VC5525-256G70-NN brings 32 GB of RAM. That RAM gives the queue room to ride out a stall: 10 s of even the 16-bit stream sits around 1.5 GB, a rounding error next to it.

The camera link matters before the disk does. GigE tops out at 1 Gbit/s ÷ 8 = 125 MB/s of raw link, about 100 MB/s usable, so the 8-bit worked-example stream fits with little to spare. A 2.5 MP global-shutter camera such as the MV-CU020-90GC is the class that outgrows the cable. At 1 byte per pixel, 2.5 million pixels and 60 fps, the link demand is about 150 MB/s — that is 10GigE work. A quad-port copper 10GbE NIC such as the MV-GT2104 then gives each camera a link of its own. Multi-port NICs and PoE cards sit with the frame grabbers in the accessories finder.

Three checks before ordering: the disk's sustained rate at least 1.5× your row of the table; capacity at least 1.2× the planned run; camera link ceiling above the sensor's pixel rate. Then start at the camera step or the camera finder to match the chain in one pass.

How do you prove that no frame was dropped?

Check three numbers against each other after every run: the frames the driver reports acquired, the frames your writer counted, and the file size divided by bytes per frame — when file size equals header plus N × 307,200 bytes exactly, nothing was lost.

The per-frame counter from step 3 makes gaps findable without decoding anything: read the last 4 bytes of each record and look for a jump in the index. Timestamps answer the same question from the camera side, the method behind camera frame rate measurement from timestamped frames, and the wider discipline is collected in high-speed camera recording without dropped frames. Before the first long run, work this list:

• Acquisition-only rate, saving disabled, reaches the target.
• Free space at least 1.2× the planned capture — 166 GB for a 138 GB run.
• Indexing and real-time scanning excluded from the capture volume.
• A full-duration soak: a rate that holds for a minute can fail at minute 20 when a consumer SSD's fast write cache fills.
• Stop path tested: partial block flushed, header count written, clean close.

When is the disk not the problem at all?

Before blaming storage, disable saving and watch the acquisition rate alone — if the camera does not reach 250 fps with nothing being written, the limit is the link, the exposure setting or the trigger, and no disk change will fix it.

Exposure is the quiet one: 250 fps allows at most 1/250 s = 4 ms per frame before sensor readout takes its share, so a 10 ms exposure caps the camera at 100 fps no matter how fast the storage is. The link is next: sharing a GigE cable or a USB hub between cameras splits the ceiling between them, the failure mode in USB hub drops camera frame rate: budget the bandwidth first and the sizing question in GigE Vision for a high speed camera: how much frame rate fits?. And when the target climbs to thousands of fps, where the camera's own buffer becomes part of the plan, high frame rate camera data rate: plan 2000 fps recording takes over from here.

Size the stream first — resolution, bit depth, fps — and the hardware falls out of the table above. When the numbers get awkward, run the configurator to match camera, link and storage in one pass, or send the rate and format through the contact page and we will size the chain with you.