Camera stuck at 30 fps in a Python GenTL loop: fix the link first

You ask a thermal camera for 94.8 fps and the acquisition loop delivers 29.3, whatever you write into the frame-rate setting. The natural suspect is the Python side — the per-frame copy, the buffer handling, the GenTL producer — and the natural move is to optimize the loop. In one real case, removing the copy changed the rate by 0.05 fps, a second camera and a freshly installed PC changed nothing, and a different application on the same driver reached 92 fps: the ceiling was in the transport. What follows computes the stream's real data rate, compares it with what a gigabit link carries at 1500-byte and 9000-byte packets, and separates a link cap from an exposure cap, a camera throughput limit and packet loss, with the decision for each case.

Why is my camera stuck at 30 fps when I request 94.8?

Because the transport is moving about 19 MB/s while the camera produces 62 MB/s: a ceiling that ignores the requested rate, survives a change of PC and camera, and costs the copy loop only 0.05 fps is a link or driver limit, and the fix is packet size, not code.

The measured case is a 640 × 512 Mono16 thermal stream whose specification lists 94.8 fps at that region of interest. Timing 1000 frames gave 29.3 fps with the per-frame copy removed and 29.25 fps with it. An identical second camera and a clean Windows installation produced the same 29 fps. Then a different application, using the same acquisition driver, reached 92 fps on the same hardware — which moves the fault out of Python and into how that driver's transport was configured. You can write AcquisitionFrameRate into the remote device's node map all day; a transport ceiling never reads it.

Two classic caps must fall first, because they masquerade as this fault. A long exposure pins the sensor: the ceiling is 1000 ÷ exposure in milliseconds, so 33 ms means 30.3 fps regardless of the network. A camera waiting on an external trigger returns exactly the trigger rate — see camera FPS drops suddenly: exposure, USB link and bandwidth and hardware trigger configuration per camera. Neither survived the tests here: the rate ignored exposure, and the camera was free-running.

How much bandwidth does a 640 × 512 Mono16 stream need?

655,360 bytes per frame, 62 MB/s at the full 94.8 fps: width × height × 2 bytes per pixel, times the frame rate. Compute this number before touching acquisition code, because every decision that follows compares against it.

For this stream: 640 × 512 × 2 B = 655,360 B, about 0.66 MB. At 94.8 fps that is 655,360 × 94.8 = 62,128,128 B/s ≈ 62 MB/s, or 497 Mbit/s. What the loop actually moved was 655,360 × 29.3 = 19,202,048 B/s ≈ 19.2 MB/s, roughly 154 Mbit/s — 31 percent of the camera's output, with every frame arriving complete.

A catalog example in the same class is the LS-IRC-0044, a 640 x 512 thermal camera on GigE running at 50 Hz.

Halve the bit depth and you halve the load: the same geometry at 8 bits is 327,680 B per frame and 31 MB/s at 94.8 fps, which standard Ethernet frames carry without argument. At 16 bits, a 50 Hz stream of that frame size is 655,360 × 50 = 32,768,000 B/s ≈ 32.8 MB/s — carried, but with little margin on a weak receive path. More infrared cameras publish frame size and rate the same way, so the arithmetic transfers directly.

How many frames per second can a gigabit link really carry?

119.6 MB/s of image data is the absolute wire-rate ceiling, and about 110-118 MB/s is what 9000-byte jumbo frames sustain in practice; standard 1500-byte packets reach roughly 60-70 MB/s on a healthy host and can fall into the 20-40 MB/s band on a weak receive path.

The wire math, which no setting can beat: one 1500-B packet occupies 1538 B on the wire — 8 B preamble, 14 B Ethernet header, 1500 B payload, 4 B checksum, 12 B inter-frame gap. One gigabit is 1 × 10⁹ bit/s ÷ 12,304 bit = 81,272 packets/s, and at 1472 B of IP payload per packet that is 119.6 MB/s. Packet size does not change this ceiling; it changes how many packets per second your host must catch, reassemble and hand to the producer.

Count packets and the table explains itself. At 1500 B, each frame is 655,360 ÷ 1472 = 445.2, so 446 packets — about 450 once the stream protocol's own header bytes are counted — and 94.8 fps needs 446 × 94.8 ≈ 42,281 packets/s. At 9000 B, each packet carries 8972 B past the IP and UDP headers, so a frame is 74 packets and the full rate needs 7,015 packets/s: six times fewer interrupts and buffers. The measured 19.2 MB/s with intact frames is the third row — the host received everything and could not sustain the packet rate. Sharing a link between cameras makes the same arithmetic bite sooner, so divide it first as the multi-camera GigE bandwidth budget guide shows.

The decision for this stream: 62 MB/s sits at the top of what standard frames deliver and above what a weak receive path delivers. Run it with 9000-byte jumbo frames end to end, or move it off gigabit; there is no third option worth tuning.

How do I set jumbo frames across camera, card and switch?

Put 9000 bytes in three places that must agree — the network card's driver, the camera's stream packet size feature, every switch in between — then reconnect the camera and re-measure; one element still at 1500 bytes silently pulls the whole path back.

1. Read the negotiated link speed first. Gigabit uses all four cable pairs; one damaged pair renegotiates to 100 Mbit/s, everything still works, and this stream tops out near 18 fps. A card reporting 100 Mbit/s is a cable or port fault, not a tuning problem — the same failure mode behind GigE camera packet size errors when your link drops below gigabit.
2. Raise the adapter's jumbo-frame or MTU value in its driver's advanced settings to 9000 bytes, then reconnect.
3. Raise the camera's packet size to 9000 bytes through the feature tree before acquisition starts — the order matters, as setting the GigE camera packet size before starting acquisition explains. Where auto-negotiation overshoots what the path supports, cap it back: GigE Vision packet size exceeding MTU covers that failure.
4. Prove the path with a do-not-fragment ping to the camera's address carrying an 8972-byte payload (9000 − 20 B IP header − 8 B ICMP header). A reply means every element forwards jumbo frames; a "must fragment" answer names the element still at 1500.
5. Raise the camera's device link throughput limit where it exposes one. Cameras that implement it hold the frame rate down to the bandwidth allocated, so that limit, not the link, can be your 30 fps ceiling.

For a machine whose only job is acquisition, the GigE frame grabbers range carries PoE and multi-port cards; these three compare as follows:

Values as printed in each product's data sheet; open a part for its drawing and full specifications.

Whichever card carries the stream, its driver decides the packet size the producer can negotiate, and it is the one component in the path you fully control.

What else pins the frame rate at a fixed value?

Five suspects, in the order they bite: exposure time (1000 ÷ exposure in ms = fps cap), a link negotiated at 100 Mbit/s (about 18 fps here), a camera-side link throughput limit, packet loss with retransmission, and a producer-level bandwidth setting — each has a one-minute test.

• Exposure. The ceiling is 1000 ÷ ms: 33 ms gives 30.3 fps, and this camera needs the 10.5 ms period of 94.8 fps. If the achieved rate tracks exposure changes, the sensor, not the link, is the ceiling — the mechanism depth camera frame rate drops: exposure settings cap the FPS walks through.
• Trigger. An externally triggered camera returns the trigger rate and nothing more. Count triggers with a scope before blaming transport.
• 100 Mbit/s negotiation. Covered above: read the adapter state; about 18 fps is the signature.
• Link throughput limit. Raise the camera's limit and the rate follows it — that is the whole test.
• Packet loss with resends. The producer's statistics report lost and retransmitted packets. Resends mean a path fault that looks like a fixed low rate with clean frames — diagnose it as GigE Vision stream packet loss or as the pattern in GigE Vision stream timeout: check packets, buffers and MTU.
• Producer settings. Two applications on one driver can sit at 29 and 92 fps with identical camera settings, because producers keep their own packet-size and bandwidth defaults. Copy the working application's transport settings into your loop's set-up and re-measure.

How do I measure the achieved rate without fooling myself?

Time a fixed count of frames with a monotonic clock and divide: 1000 frames in 34.1 s is 29.3 fps, and running the same count with and without the per-frame copy turns the copy's cost into a number — here 0.05 fps — instead of a suspicion.

import time

n_frames = 1000              # int: frames to time; 1000 frames gives about 0.1 fps resolution
frames = None                # numpy array, shape (n_frames, 512, 640), dtype uint16; None skips the copy
t0 = time.perf_counter()     # float: monotonic clock reading in seconds

for i in range(n_frames):
    with ia.fetch_buffer() as buffer:      # ia: the image acquirer returned by create_image_acquirer()
        component = buffer.payload.components[0]
        image = component.data.reshape(component.height, component.width)
        if frames is not None:
            frames[i] = image              # copy now, inside the context, before the buffer is requeued

elapsed = time.perf_counter() - t0         # float: seconds
print(n_frames / elapsed)                  # achieved fps: 29.3 failing, 90+ once the link is fixed

The test that proves the point is the pair of runs: 29.3 fps without the copy, 29.25 fps with it. Keep the copy inside the fetch context, before the buffer returns to the queue, or you will read torn frames instead of a frame rate. Save to disk after the timed section, and use the method in how to measure camera frame rate in Python or measuring camera frame rate reliably so the number is comparable run to run. When acquisition outruns processing, that gap has its own fix, covered in acquisition faster than processing: fix the frame rate gap.

When should I stop tuning gigabit and change the interface?

Above roughly 115 MB/s sustained there is nothing left on one gigabit link: run one 62 MB/s camera per link with jumbo frames, put two or more such cameras on 5GBASE-T or 10GigE, and use USB3 or CoaXPress when the camera must not share a network at all.

5GBASE-T gives about four times a tuned gigabit link over existing cabling — the 5GBASE-T camera interface: bandwidth, cables and distance guide has the numbers. 10GigE reaches about 1 GB/s and keeps one Ethernet fabric for several cameras. USB3 delivers roughly 400 MB/s with no switch, no MTU and no packet size to negotiate, and CoaXPress about 1.2 GB/s on a CXP-12 link with trigger lines built in.

At this frame size, the LS-IRC-0016 is a 640×512 LWIR module on USB3.

For short captures above the link rate, the camera's own memory buys the difference: a 1000-frame Mono16 burst at 0.66 MB per frame is 0.66 GB, streamed out afterwards at whatever the link sustains — the pattern in camera burst capture faster than link bandwidth: use the buffer. Continuous monitoring gets no such holiday; there the link rate is the honest ceiling.

What do I check before replacing any hardware?

Run the numbers in this order — measured MB/s, negotiated link speed, packet size at all three ends, exposure cap, camera throughput limit, producer statistics — and only then spend money; in most 30 fps ceilings the fix is a setting, not a card.

• Compute frame bytes and MB/s. Here 655,360 B and 62 MB/s; every other number is compared against these two.
• Read the negotiated speed. 100 Mbit/s alone explains an 18 fps ceiling with the camera working perfectly.
• Confirm 9000-byte packets at card, camera and switch. The do-not-fragment ping with an 8972-byte payload settles it in one command.
• Check the exposure cap: 1000 ÷ ms. A 33 ms exposure is a 30.3 fps ceiling no network change can lift.
• Raise the camera's link throughput limit. Where the feature exists, the frame rate follows it directly.
• Read the producer's loss and resend counters. Resends point at the path; a clean path points back at bandwidth.
• Time 1000 frames with and without the copy. One run separates the loop from the transport.

If the numbers still refuse to close, take the measured MB/s, the nominal rate and the camera model to the contact page — a bandwidth check is part of the free expert support that comes with every order. New builds can start from area scan cameras and the camera finder, or run the system configurator with the data rate as the first constraint.