Unplayable camera recording file after an unclean shutdown

You come back to a capture station in the morning and the recording from the night before will not open. The file has a plausible size — hundreds of megabytes or several gigabytes — and its first bytes look identical to a known-good clip from the same rig, yet every player, and the SDK's own playback sample, calls it invalid. Nine times out of ten the frames are fine and the map that finds them is not: the capture process was killed, and the writer never reached the index it writes at close. What follows is the five-minute test that proves the cause, the recovery path for the frames already on disk, and the shutdown ordering that keeps every later run clean — for area scan cameras, 3D sensors and frame grabber capture loops alike.

What the symptom tells you

A recording with a valid-looking header, a plausible size and an 'invalid file' error is a file that was never closed. Commit to that diagnosis first, because it changes what you do next: you stop reinstalling players, swapping codecs and re-flashing cameras, and you start rebuilding one small index and fixing one exit path.

Recording formats split their metadata in two. The header — magic number, image geometry, frame rate, codec — is written when the file is opened, which is why a hexdump of the broken file's first bytes matches a working clip almost field for field. The index — the table mapping each frame number to its byte offset, size and timestamp — is written when the writer closes. The player reads the header, reaches for the index, finds nothing coherent, and reports the whole file as invalid. The error text will not say 'missing index'; almost none of them do.

Why the index is written last

Frame data streams to disk as it is captured, one chunk per frame. The index cannot precede that stream, because its length is unknown until the last frame lands; each entry is roughly 16 to 32 bytes of offset, size and timestamp. The layout of nearly every container follows:

The named cases: AVI keeps its frame table in an idx1 chunk at the end. MP4 in its default layout writes the moov atom after the last frame. Matroska writes the cues element at close. Many vendor SDK recording formats — depth-camera formats included — keep an end-of-file index for the same reason. Kill the process, by SIGKILL, a power cut or the kernel's out-of-memory killer, and the close never runs: header and payload are on disk, the table is not. That is the entire failure.

Likely causes, most probable first

1. Killed before close. The dominant cause at unattended stations: Ctrl-C with no handler, a service stop that escalates to SIGKILL, an OOM kill, a power cut. The confirmation section below proves it from the exit code and the logs.
2. Disk full mid-frame. The writer truncates the final frame, the index is never written, and the process can still exit looking normal. Test: free space at the moment of failure and the writer's error log.
3. Codec the tool cannot parse. The file is intact, but the repair tool or player lacks support for the compression you chose; tooling often lands a release or two behind a new codec. Test: a short control clip, same codec, properly closed.
4. Geometry changed mid-recording. Resolution or frame rate altered while the writer ran, so the header disagrees with the frames. Test: the settings log from the recording session.

The five-minute confirmation

1. Compare headers. Hexdump the first 64 bytes of the broken file and of a known-good clip from the same rig. Matching magic and geometry fields prove the header is intact, which puts the damage at the other end of the file.
2. Check how the run ended. Exit code 137 is SIGKILL (128 + 9); 143 is SIGTERM (128 + 15) with no handler installed. journalctl and dmesg show an OOM kill naming your process. A clean exit 0 with the same symptom points at the full disk instead.
3. Size-check the payload against the camera's numbers — the worked example below shows the arithmetic. A file within a factor of two of the estimate holds real frames; a few kilobytes does not, and that is a different failure.
4. Record a control clip: ten seconds, same camera, same codec, closed properly. If it plays, the pipeline and the codec are exonerated in one move.
5. Run the SDK's repair tool on a copy. A successful re-index confirms the diagnosis; a refusal on a compressed recording usually means the tool lacks that codec, not that the data is worse.

Recover the frames already on disk

The payload is intact, so recovery is an indexing job, not a decoding job. Match the path to the container:

• Vendor SDK formats. The SDK's repair or re-index tool is the only safe path, because frame boundaries belong to the SDK. Re-test the tool after every SDK update instead of writing old files off; codec support in repair tooling tends to arrive in later releases.
• MP4. Index-rebuild tools that use a reference recording from the same device reconstruct the moov atom by scanning frame data. With inter-frame compression you get everything up to the last intact keyframe — at a 1–2 s keyframe interval, at most the final couple of seconds.
• AVI. Only the idx1 chunk is missing, and readers that scan the frame lists can demux without it. Copying the streams into Matroska — ffmpeg -i input.avi -c copy output.mkv — usually yields a playable file in seconds.
• Matroska. Players already tolerate a missing cues element and stop at the last complete cluster; a -c copy remux rebuilds a clean index.

The yield: every frame to the last complete one for uncompressed or lossless recordings, and every frame to the last keyframe for H.264 or H.265.

Fix the cause: close the writer before the camera

Recovery saves this file; the shutdown path saves every file after it. The rule fits in one sentence: the writer closes first, completely, and the camera, SDK, frame grabber and network close after it. The same ordering is what keeps an SDK from hanging at exit — shutting camera streams down in the right order covers the teardown side.

1. Install handlers for SIGINT and SIGTERM before the first frame. The handler sets a flag and returns; it does no I/O.
2. Break the capture loop on that flag and let the current frame write finish.
3. Call the writer's close and wait for it to return — that call writes the index. Give it room, with no short timeout wrapped around it.
4. Stop the camera stream, close the SDK handle, release the frame grabber, and tear down the network last.
5. Under systemd, raise TimeoutStopSec above your worst-case flush. A unit stop sends SIGTERM, waits — 90 s by default — then escalates to SIGKILL, and that escalation is often the very kill that destroys the index.

import signal

stop = False
def request_stop(signum, frame):
    global stop
    stop = True

signal.signal(signal.SIGINT, request_stop)
signal.signal(signal.SIGTERM, request_stop)

while not stop:
    image = camera.grab()
    writer.write(image)

writer.close()   # writes the index — let it finish
camera.stop()
camera.close()

Script errors deserve the same exit path: wrap the loop so an exception still reaches the writer's close, the pattern in cleaning up camera handles after script errors.

Prevention: containers and rotation that survive a kill

Three habits, in order of value:

1. Rotate files every 5 to 10 minutes. A kill then costs one segment instead of the whole shift; most SDKs take a segment length, and FFmpeg-based writers get it from the segment muxer.
2. Record kill-tolerant containers where you control the format: fragmented MP4 at roughly one fragment per second, or Matroska. Both play a truncated file up to the cut, which turns this failure from a crisis into a shrug.
3. Keep one session of disk headroom plus 10 percent, with an alert at the threshold. A full disk produces the identical symptom with a clean exit code and is the cause most often misread as corruption.

On multi-camera GigE stations, put each camera on its own link. A card such as the MV-GE1104 is a 4-port GigE frame grabber, so one camera's restart never disturbs another camera's writer.

When the recording lives on an HMI next to the PLC logic, the same close-order rule applies — the pattern in camera capture with PLC communication. IP cameras feeding the same disk over the network add a second writer that must close just as cleanly; IP camera integration with HMI display and recording walks that set-up. Depth and 3D streams make the loss hurt most, since payloads are expensive to recapture — reducing depth recording file size keeps rotation lean.

Worked example: sixty seconds of 12 MP mono

Put numbers on it with a bench rig: a USB3 area scan camera feeding a capture PC. The LS-CMR-0166 is a 12 MP USB3 area scan camera with a 4000 × 3000 resolution at 30 fps.

Assume 8-bit mono and do the arithmetic:

One minute of that stream is the formula's 21.6 GB; a lossless codec typically lands at 5–11 GB per minute on natural scenes. Sixty seconds at 30 fps is 1,800 frames, so the unwritten index is roughly 1,800 × 16–32 bytes — call it 30–60 KB, about three parts per million of the file. Kill the process at the one-minute mark and that missing 0.0003 percent is why nothing opens. Rebuild the index and, lossless, all 1,800 frames come back; with H.265 at a 2 s keyframe interval you recover to the last keyframe, about 1,740 frames. The stream also explains the interface choice: one gigabit link carries roughly 110 MB/s of payload, so a 360 MB/s camera belongs on USB3, 10GigE or CoaXPress.

When it is not the shutdown

Four neighbours are worth telling apart:

• Zero-byte or few-kilobyte files. The writer never produced frames — a path, permission or first-frame problem, not a close problem. Work through checking writer setup for an empty camera recording file.
• Infrared and thermal recordings. Same container logic in formats of their own; fixing empty or unplayable infrared recordings collects those remedies.
• Plays in the recording app, fails downstream. Codec or pixel-format mismatch between writer and consumer; the index is fine, so convert the file instead of repairing it.
• Playback stops at a fixed frame near the end. A truncated tail from a full disk: everything before the cut is good, and the size check in the confirmation section flags it.

Checklist for the next run

• SIGINT and SIGTERM handlers installed before the first frame is grabbed.
• Close order fixed and tested: writer, camera stream, SDK handle, frame grabber, network.
• Stop timeout raised above the worst-case flush; SIGKILL never the planned stop.
• A ten-second control clip recorded and played back this shift.
• Disk headroom of one session plus 10 percent, with an alert wired to the threshold.
• Files rotated every 5–10 minutes.
• Repair procedure rehearsed once on a copied broken file, so it is known before it is needed.

If the station needs rebuilding around this rule — camera, capture link, cabling — start at the system configurator or the area scan camera finder, and bring the recording symptoms to the contact page for a second opinion on the set-up.