Extract all frames from a video in JavaScript: seek every frame

A recorded clip is the easiest way to develop a vision routine, and the browser is where these tests often start: OpenCV.js, one MP4 file, no install. The catch is that an HTML video element does not behave like the desktop VideoCapture. A timer loop that samples it once per frame period returns the same frame every time, because nothing advances the playback position. This guide gives you the seek loop that moves the element frame by frame, exports every frame to a canvas, and proves you captured all of them, with a 10 s, 10 fps, 1920 × 1280 MP4 as the worked case.

Why a timer loop returns the same frame

In OpenCV.js, cap.read(src) copies whatever frame the video element is displaying at the moment of the call. The element only advances while it is playing or when you set currentTime. A capture loop that never calls play() and never seeks stays at position 0, so every read hands back frame 0. Desktop cv2.VideoCapture is the opposite: it owns the file, and each read() decodes the next frame and advances an internal pointer. A while loop over read() works in Python for that reason, and cannot work here — the browser build has no pointer of its own to advance.

The starter code most projects copy also has a unit bug: delay = 1 / fps computes 1 / 10 = 0.1, but setTimeout expects milliseconds, so the browser rounds 0.1 down to 0 and fires the callback as fast as timers allow. Even a corrected 100 ms interval would not rescue the loop: hidden tabs clamp timers, and the media clock drifts against setTimeout, so samples fall between frames. The timer pattern in the official OpenCV.js video tutorial is fine for preview; it is the wrong tool for frame capture.

The black output has a third, separate cause: imshow(canvas, dst) paints dst, a CV_8UC1 Mat that nothing ever wrote to. Paint src right after cap.read, and call canvas.toBlob only after the draw, because toBlob is asynchronous and exports whatever the canvas holds when it runs.

One frame repeated forever is a familiar failure in capture pipelines. On live GigE streams it appears when the transport layer keeps handing over the newest buffered frame, and the fix there is draining the buffer queue, as in Latest image only returns duplicate frames: drain the buffer queue. On a recorded clip the fix is the seek loop below.

What you need

• The OpenCV.js WebAssembly build, loaded and initialized before the first cv. call.
• A video element whose src is an object URL of the clip, which keeps the pixels same-origin; the cross-origin case has its own section below.
• The clip's frame rate: 10 fps in the example. If your library reports 0, read it reliably as shown in Video FPS returns zero in OpenCV: read frame rate reliably.
• One canvas and one CV_8UC4 Mat sized from videoWidth and videoHeight, read after the loadedmetadata event; VideoCapture errors while those sizes are still 0.
• A frame fingerprint for dedupe: a 64 × 64 corner crop through getImageData, summed to one number.

Extract every frame: the seek loop

1. Create the video element, set preload to "auto", and await loadedmetadata so videoWidth, videoHeight and duration hold real values.
2. Compute the frame period: period = 1 / fps.
3. Size one canvas and one Mat from the video, then create new cv.VideoCapture(video).
4. For frame index i, seek to the frame's midpoint: currentTime = (i + 0.5) * period. Midpoints dodge the boundary rounding covered in the next section.
5. Await the seeked event, which fires when the seek has completed and the displayed frame has changed.
6. Call cap.read(src), then cv.imshow(canvas, src), then canvas.toBlob, in that order, each step consuming the previous one.
7. Increment i and repeat while the next midpoint is still inside duration.
8. Free the Mat with src.delete() and revoke the object URL when the run ends.

// Documented calls: HTMLVideoElement currentTime/onseeked/onloadedmetadata/
// videoWidth/videoHeight/duration, canvas.toBlob, and the OpenCV.js calls
// cv.Mat, cv.VideoCapture, cap.read, cv.imshow. clipUrl, FPS, blobs are ours.
const FPS = 10;                                 // frames per second, from the clip
const PERIOD = 1 / FPS;                         // frame period: 0.1 s at 10 fps
const video = document.createElement("video");  // HTMLVideoElement
video.preload = "auto";
video.src = clipUrl;                            // same-origin object URL of the MP4
await new Promise(done => { video.onloadedmetadata = done; });

const canvas = document.createElement("canvas");
canvas.width = video.videoWidth;                // 1920 in the example
canvas.height = video.videoHeight;              // 1280 in the example
const cap = new cv.VideoCapture(video);
const src = new cv.Mat(video.videoHeight, video.videoWidth, cv.CV_8UC4);

const seeked = () => new Promise(done => { video.onseeked = done; });
const blobs = [];                               // array of Blob, one per frame
for (let i = 0; (i + 0.5) * PERIOD < video.duration; i++) {
  video.currentTime = (i + 0.5) * PERIOD;       // 0.05, 0.15 ... 9.95 s
  await seeked();                               // the decoder is now on frame i
  cap.read(src);                                // copy the displayed frame
  cv.imshow(canvas, src);                       // draw BEFORE exporting
  blobs.push(await new Promise(done => canvas.toBlob(done, "image/png")));
}
src.delete();                                   // one reused Mat, freed at the end

Reuse one Mat and one canvas for the whole run, and export each frame immediately; a fresh Mat per frame exhausts the WebAssembly heap fast. toBlob snapshots the canvas at call time, so queuing it before the next seek is safe. A camera callback gives you no such snapshot — there you copy the buffer before you queue it, as in Torn frames in a camera callback: copy the buffer before you queue it.

Worked example: 100 frames out of a 10 s, 10 fps clip

Inputs, taken from the clip's metadata: duration 10 s, frame rate 10 fps, frame size 1920 × 1280 pixels, PNG export per frame.

• Frame period: 1 / 10 fps = 0.1 s = 100 ms.
• Seek targets: 0.05, 0.15, 0.25, ... 9.85, 9.95 s — the midpoints of frames 0 through 99.
• Frame count: 10 s × 10 frames/s = 100 frames. The loop runs i = 0 through 99, because 99.5 × 0.1 s = 9.95 s lies inside the 10 s duration while 100.5 × 0.1 s = 10.05 s does not.
• Raw size per frame as RGBA: 1920 px × 1280 px × 4 bytes = 9,830,400 bytes ≈ 9.8 MB per frame.
• All 100 frames held as Mats: 100 × 9,830,400 bytes = 983,040,000 bytes ≈ 0.98 GB of live pixel data — the reason the loop ships one Blob per frame and reuses a single Mat.

Decision: expect exactly 100 blobs with 100 distinct fingerprints. Fewer than 100 means a seek or a read was skipped, so log i on every iteration to find where. More means two timestamps landed on one frame; the fingerprint dedupe removes the duplicate. The whole run should finish in a few seconds, well inside the 10 s the clip takes to play — a run slower than playback points to a missing seeked await.

When a seek lands off-frame: nudge and dedupe

Frame-exact seeking is not something the HTML specification guarantees, and browsers differ. A decoder can snap to the frame in progress rather than to the frame that starts at the requested time. As an illustrative example on a 20 fps clip: seeking currentTime to 0.05 s, the nominal start of frame 2, may keep showing frame 1 until the seek reaches roughly 0.07 s — an offset of about 20 ms, close to 40 % of the 50 ms frame period. Treat any such offset as browser- and build-specific and verify it on your own target browser before you trust frame-boundary seeks.

Midpoint seeks sidestep most of that: the midpoint of frame i lies strictly inside frame i's display interval, so a decoder snapping in either direction still lands on frame i. If fingerprints still repeat, add a nudge of a quarter to half a period — 25 to 50 ms at 10 fps — and re-run. Cap the nudge at half a period; past that you start skipping frames. Surviving duplicates are dropped by the fingerprint check, and the count after dedupe must still equal duration × fps.

Cross-origin clips: clear the canvas taint first

When the clip loads from another origin, the draw succeeds but every pixel read fails with SecurityError: Failed to execute 'getImageData' on 'CanvasRenderingContext2D': The canvas has been tainted by cross-origin data., and toBlob throws the same error. Pixels from another origin taint the canvas, and the browser refuses to read tainted pixels back.

Two clean fixes. Serve the clip from the same origin as the page — an object URL built from a locally chosen Blob already is. Or have the clip's server send an Access-Control-Allow-Origin header, and set video.crossOrigin = "anonymous" before you assign video.src.

When to prefer requestVideoFrameCallback

Seeking wins when you need every frame, indexed, faster than real time. When you can simply play the footage — long clips you would watch anyway, or variable-frame-rate encodings where a constant fps assumption breaks — use requestVideoFrameCallback. The browser fires it once per presented frame, drawing each frame onto a canvas at exactly the video's own frame rate; the callback's metadata carries the frame's media time, so you label frames by measured time instead of a computed index. Check the HTMLVideoElement page of the official web-platform API documentation for the exact callback signature and metadata fields in your target browser.

Live pipelines have the same one-callback-per-frame shape, and the same latency accounting: by the time your code runs, the frame you hold is already the previous one, as laid out in Set exposure every frame: latency and the one-frame delay.

Prove you captured every frame

• Count: Math.round(duration × fps) — 100 for the example. Assert it.
• Fingerprints: 100 distinct sums expected. A repeat means two seeks hit one frame; apply the midpoint or nudge fix, dedupe, recount.
• Spot check frames 0, 50 and 99 against manual seeks at 0.05 s, 5.05 s and 9.95 s in a player.
• Wall clock: the seek run should finish well inside the clip's 10 s runtime. Slower than playback usually means the seeked await is missing and the loop is racing the decoder.

When the video is a stand-in for a live camera

A recorded clip proves the routine; production wants the sensor live, and then the browser loop gives way to an acquisition chain. Budget the link first. A 1920 × 1280 mono8 stream at 10 fps moves 1920 px × 1280 px × 1 byte × 10 fps = 24,576,000 bytes/s ≈ 24.6 MB/s ≈ 197 Mbit/s. Four such cameras on one port carry 786 Mbit/s of payload against a 1 Gbit/s line rate, which leaves no headroom, so give each camera its own port.

Reading recorded runs at full capture rate is its own problem, covered in Bag file playback dropped frames: read every frame offline; when acquisition outpaces processing instead, the fix is in Acquisition faster than processing: fix the frame rate gap.

A two-camera rig needs one port per camera. The MV-GE2002P is a dual-port PoE GigE NIC.

The interface decides the card:

A camera list also needs the machine that runs the inspection around the clock. The MV-VC5525-256G70-NN is a VC5000 vision controller with a 12th-generation Intel Core i5, 32 GB memory, a 256 GB SSD, PoE camera ports and 8 built-in light channels.

Prototype in the browser until the routine is right, then move the loop onto the capture chain: pick the interface from the camera side, match ports to cameras, size the machine to the pipeline. The frame grabbers and vision controllers category pages list the range, the capture configurator walks the chain in order, and more capture-side guides sit on the systems topic page. Send a camera list through the contact page and we will check it against the link budget before you order.