Extract SPS and PPS NAL units from an H.264 camera stream

You ask the encoder for H.264, the SDK hands you byte buffers, and your first attempt — reading the NAL type at a fixed offset — returns 9 on every packet. The stream itself is fine: a saved file plays, your live parser does not. The reason is that one encoded-frame message carries a whole access unit, several NAL units back to back, and the first of them is almost always an access unit delimiter. This guide shows you how to walk the Annex B start codes, split those units apart, extract the SPS and PPS (types 7 and 8), and hand them to whatever consumes your stream: a recorded file, a software decoder or an RTSP session. The same loop works for any camera SDK that delivers Annex B H.264, including area scan cameras streamed over GigE or USB3.

Why does every buffer report NAL unit type 9?

Byte 4 is the header byte of the first NAL unit in the buffer: bytes 0 to 3 are the start code, and the first unit nearly every encoder emits is the access unit delimiter, type 9. A saved file plays because the player scans the whole file for start codes instead of trusting one position; your parser reads one fixed index, so it sees the delimiter again on every message and never reaches the units behind it.

An Annex B H.264 buffer is a chain of NAL units. Each unit begins with a start code, the 4-byte 00 00 00 01 or the shorter 00 00 01, followed by a one-byte header whose lower five bits name the type. Here is the head of a first message, walked byte by byte:

The SPS header sits at index 10 in this message and at a different index in every other one, because the units ahead of it vary in length. No fixed offset can track that. Encoders that omit the delimiter put the SPS header straight at index 4, which is why the same one-line read appears to work on one camera and never on another.

What is inside one encoded-frame message?

One encoded-frame message carries a whole access unit in Annex B form: a start code, then several NAL units back to back — typically the delimiter (9), the SPS (7), the PPS (8), an SEI (6) and the slices (5 on the IDR frame, 1 on the rest).

So the first message of a stream usually reduces to the sequence 9, 7, 8, 6, 5, and every message until the next IDR reads 9, 1. A frame coded as multiple slices carries several type 1 or type 5 units in one message, and encoders that skip the AUD or the SEI produce shorter chains, so the parser should tolerate any order rather than expect one exact pattern.

Two consequences follow. First, any code that parses "the header of the message" is really parsing the header of whichever unit happens to come first, which is why a fixed-offset read looks broken. Second, the parameter sets are usually present in the very first message you receive, so extraction means pulling them out and remembering them, not waiting for them to appear.

How do you extract SPS and PPS from an H.264 stream?

Scan the buffer for the byte pattern 00 00 01, extend the match to a 4-byte start code whenever the preceding byte is also 00, split at every code, and read the header byte that follows each one: & 0x1F returns 7 for SPS and 8 for PPS. Six steps cover every buffer:

1. Pull one buffer from the SDK's queue as plain bytes; one message should hold one frame, and if the SDK batches several, the same loop still works.
2. Search for every 00 00 01 in the buffer.
3. When the byte before a match is also 00, step back one byte: the start code is the 4-byte form.
4. Split the buffer at each start code; every piece is one NAL unit, start code included.
5. Read the first byte after the code and compute byte & 0x1F; keep the pieces that return 7 and 8, verbatim.
6. Cache those two pieces and prepend them wherever a consumer starts: the head of a recording, a decoder's initialisation, a new client session.

START = b"\x00\x00\x01"

def split_nals(buf):
    # yields (nal_type, unit_bytes) for each Annex B NAL unit in one buffer
    marks = []
    pos = 0
    while True:
        pos = buf.find(START, pos)
        if pos < 0:
            break
        marks.append(pos - 1 if pos > 0 and buf[pos - 1] == 0 else pos)
        pos += 3
    for i, s in enumerate(marks):
        e = marks[i + 1] if i + 1 < len(marks) else len(buf)
        yield buf[s + 3] & 0x1F, buf[s:e]

sps = pps = None
for nal_type, unit in split_nals(frame):
    if nal_type == 7:
        sps = unit          # begins 00 00 00 01 67
    elif nal_type == 8:
        pps = unit          # begins 00 00 00 01 68

Both units keep their start codes, which is the form Annex B consumers want. Write the pair once and append every buffer in arrival order:

with open("capture.h264", "wb") as f:
    f.write(sps + pps)     # parameter sets once, at the head
    f.write(frame)         # then every buffer, first frame onwards

If the first buffer already carries the sets inside it, writing them twice is harmless for Annex B readers. Decoder libraries that expect length-prefixed units want the start codes replaced instead: a 25-byte SPS becomes 00 00 00 19 (25 = 0x19) followed by its 25 payload bytes.

If the queue backs up while you parse, you are adding buffering latency on top of the encode; the frame-drop behaviour is the same one described in encoded image pipeline latency: find hidden frame buffering.

Which NAL unit type numbers are SPS, PPS and the slices?

The lower five bits of the header byte name the unit, so 0x67 & 0x1F = 7 is the SPS, 0x68 & 0x1F = 8 is the PPS, 0x65 & 0x1F = 5 is an IDR slice and 0x41 & 0x1F = 1 is a non-IDR slice. The list you will meet in a camera stream is short:

The upper bits of the header byte are the reference priority, read as (byte >> 5) & 0x03; that is why non-IDR slices show up as 0x41, 0x61 or 0x01 and all reduce to type 1. Types 14 to 20 belong to SVC, MVC and Annex G extensions that streaming encoders rarely emit; if you meet one, log it and move on.

Does H.264 have a VPS?

It does not: H.264 defines exactly two parameter sets, the SPS at type 7 and the PPS at type 8, and a VPS exists only in H.265/HEVC, where it is type 32. An H.264 Main or High profile encoder will never produce one, whatever the variable names in an example suggest, so a missing VPS in a genuine H.264 stream is not a fault — the two sets you already have are complete.

For a real HEVC stream the loop changes in two places: the header type field is six bits, read as (byte >> 1) & 0x3F, and the parameter sets are types 32 (VPS), 33 (SPS) and 34 (PPS), with IDR slices at types 19 and 20. Start codes and the delimiter behave the same.

Over RTSP the sets ride in the session announcement rather than in every frame: the SDP carries both, base64-encoded, in its sprop-parameter-sets attribute, and the session side of that delivery is covered in reducing RTSP frame latency in vision applications. Raw RTP packets are not Annex B — they carry payload headers and fragment large units — so parse after depacketisation, not on the wire bytes. If parameter sets feel like overhead you do not need, MJPEG streams skip them entirely, which is one reason SCADA viewers take MJPEG readily; the SCADA MJPEG stream configuration guide walks that setup.

How often does the encoder resend the parameter sets?

Parameter sets travel with every IDR frame, so a 60 fps stream with a one-second GOP resends the SPS and PPS every 60 frames — cache the first message that contains them and update it whenever a newer copy arrives. The cadence is your GOP: 60 fps at a one-second GOP gives 60 × 1 = 60 frames between sets, 30 fps at two seconds gives the same 60, and halving the GOP to 0.5 s halves the gap to 30 frames.

If you attach to the stream late and cannot wait a full GOP, request a key frame from the encoder — every SDK exposes a forced-IDR request under some name — and the next message arrives with a fresh pair. A resolution or bitrate change re-emits the sets as well, so treat any new SPS content as the start of a new sequence and re-initialise the decoder if the resolution moved.

Why do 00 00 03 bytes appear inside the parameter sets?

Those bytes are emulation-prevention code: the encoder inserts 0x03 after every payload 00 00 pair that would otherwise imitate a start code or a reserved sequence, and only parsers that read payload fields need to strip it back out. The undo rule is one-directional: scan the payload and delete any 0x03 that follows 00 00 when the next byte is 0x00 to 0x03.

Everything this guide reads — the type byte, the AUD payload, the SPS profile and level bytes — sits in the first bytes of a unit, before emulation prevention has anything to interfere with. You only need the removal step when you parse deeper: timing or colour metadata in the SPS, or the slice-level fields a decoder consumes.

How far can the encoded stream travel to the host?

The encoded frames leave the camera as ordinary IP packets on its Ethernet link, and standard Gigabit Ethernet carries that traffic up to 100 m over Cat 5e or better, so the run itself is rarely the limit inside a cell. At a common 1080p60 encoder setting of 8 Mbit/s, the stream takes 8 ÷ 1,000 = 0.8% of a Gigabit link; four cameras at 32 Mbit/s together still sit near 3.2%. The parser, not the wire, is where streaming setups usually lose frames — the protocol side of that link is covered in camera compatibility with vision software: what GigE Vision guarantees.

Beyond 100 m the options are fibre with media converters or 2.5G/5G Ethernet, compared in 5GBASE-T camera interface: bandwidth, cables and distance. On a bench any patch lead works; on a moving axis use a high-flex drag-chain cable, and on a vibrating rig insist on a locking connector. The camera data cables range covers both cases:

How do you check the parameter sets actually decode?

Write SPS, then PPS, then one IDR buffer to a file, hand the same three pieces to your decoder, and read the SPS payload's first bytes: 0x4D 0x00 0x2A means Main profile, Level 4.2. Three checks, in increasing strength, prove the extraction:

1. Sizes. A Main-profile 1080p SPS without custom timing metadata runs about 15 to 30 bytes and a PPS about 4 to 8, start codes included. Units far larger than that mean you caught a slice by mistake.
2. Profile and level. The first three payload bytes after 0x67 are profile_idc, constraint flags and level_idc. 0x4D = 77 is Main, 0x64 = 100 is High, and the level number is level × 10, so 0x2A = 42 is Level 4.2. A 1080p60 stream must carry it, because the level caps the macroblock rate:

3. The decode test. Feed sps + pps + one IDR buffer to your decoder and require a picture. If a frame comes out, the extraction is proven end to end.

Those macroblock rates are the H.264 Annex A level limits, so a stream advertising 60 fps under level_idc 40 has no conforming budget for it — expect exactly that mismatch when an encoder preset is set lower than the frame rate.

Once the parser holds types 7 and 8, the streaming side is done: cache the pair, prepend it at every consumer, and re-capture on any resolution change. If you are still selecting hardware for the camera side, the area scan camera finder filters by interface and resolution, the systems topic page gathers the stream-side guides, and the team answers integration questions through the contact page.