OpenCV capture from an HDMI-to-USB capture device

You have an HDMI signal — from a microscope, a test-rig display or a legacy camera — and a small USB dongle that turns it into a video stream. OpenCV should read that stream, and it can: to the host, the dongle is just another UVC camera. This guide gives you the backend flags to force, the way to pin down the device index, the bandwidth arithmetic behind most opens-but-no-frames failures, and the fixes for the permission and crash problems that come up on Linux and Windows. Work through the sections in the order you hit them; each one ends with the decision for that symptom.

Does an HDMI to USB capture device need special OpenCV code?

No — an HDMI-to-USB capture stick enumerates as a standard UVC camera, so once its stream plays in VLC the same /dev/videoN node opens with cv2.VideoCapture, and no brand-specific code exists or is needed.

OpenCV never sees the dongle's brand. It talks to the operating system's capture layer: the uvcvideo driver and V4L2 on Linux, DirectShow and Media Foundation on Windows. That layer presents the stick as an ordinary camera with an index, a set of pixel formats and a frame rate. So if VLC, ffplay or the OS camera app shows the picture, the OS half of the chain works and whatever remains is an OpenCV selection problem: index, backend or format. Brand SDKs exist for vendor extras — overlay burn-in, encoder settings, firmware updates — never for plain frame grabbing. The same logic covers every UVC device: a UVC camera streams over USB with no config file, and a capture stick is no exception.

Decision: skip the vendor SDK entirely and open the stick like any other camera.

Which backend flag should you pass to cv2.VideoCapture?

Pass an explicit backend rather than 0: CAP_V4L2 on Linux, CAP_DSHOW as the first try on Windows with CAP_MSMF as the fallback, because CAP_ANY makes OpenCV probe every driver in order and can open the wrong device or stall on a consumer capture stick.

The backend decides which OS API OpenCV uses, and the constants have stayed stable across OpenCV 4.x:

CAP_ANY probes backends in a fixed internal order and hands you whichever opens first. On a machine that also has a built-in webcam, index 0 can resolve through a different backend than you meant, or a half-initialized Media Foundation handle can block for seconds before failing. The official OpenCV Getting Started with Videos tutorial shows the bare VideoCapture(0) call — fine for one webcam, wrong for a box with several capture paths — and the VideoCapture class reference documents the index and device-path signatures. The same lesson shows up on industrial hardware: why VideoCapture cannot open a GigE Vision camera is the identical fight on a different transport, and backend fallback diagnosis explains what happens when OpenCV silently tries several openers in a row.

import cv2  # OpenCV 4.x

index = 2                                    # int, from the enumeration step below
cap = cv2.VideoCapture(index, cv2.CAP_V4L2)  # Linux; cv2.CAP_DSHOW on Windows
print(cap.isOpened())                        # True once the backend owns the device

Decision: hardcode the backend constant on both platforms; treat CAP_ANY as a diagnostic, never a default.

How do you find the device index for the capture stick?

Unplug the stick, run v4l2-ctl --list-devices, plug it back in and run it again: the entry that appears is your stick, and on Linux a UVC device usually registers two nodes, so pick the one whose --list-formats-ext output shows the MJPEG and YUYV capture formats.

The two-node detail matters: current kernels add a second node for stream metadata, it lists no pixel formats, and opening it in OpenCV yields a valid handle with zero frames — which masquerades as the no-frames bug. The probe that settles it:

v4l2-ctl --list-devices                       # run, unplug, run again: the new entry is the stick
v4l2-ctl -d /dev/video2 --list-formats-ext    # the node listing MJPG/YUYV is the capture node

Then make the choice survive replugs. On Linux, pass the /dev/v4l/by-id/ symlink for the stick as the index string: the V4L2 backend accepts a device path, and the by-id name encodes the device serial, so the index survives hotplugs and reboots. On Windows there is no lister inside OpenCV, so sweep the indices once and keep the one that reports the HDMI source's width:

for i in range(4):                            # indices 0-3; use range(10) if nothing opens
    cap = cv2.VideoCapture(i, cv2.CAP_DSHOW)
    if cap.isOpened():
        print(i, cap.get(cv2.CAP_PROP_FRAME_WIDTH),   # float, e.g. 1920.0
              cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
    cap.release()

If the stick shows in lsusb or Device Manager but no node appears at all, that is an enumeration failure, not an OpenCV one — work through diagnosing the USB host path first. On a box with several cameras, resolving the device path conflicts before hardcoding any index saves hours later.

Decision: identify by unplug-replug once, then commit to a by-id path on Linux or the verified index on Windows in your config.

Why does VideoCapture open but return no frames?

An open handle with no frames usually means the negotiated format does not fit the link: a 1920×1080 YUYV frame is 4.1 MB, 30 fps of it is 124 MB/s, and a USB 2.0 port carries only about 35-40 MB/s of real video payload, so request MJPEG first.

The arithmetic works for any UVC device:

Worked example, the most common case — a USB 2.0 stick fed 1080p30 over HDMI. One YUYV frame is 1920 × 1080 × 2 = 4,147,200 bytes, about 4.1 MB. At 30 fps that is 4,147,200 × 30 = 124,416,000 bytes/s, or 995 Mbit/s of payload counting the 8 bits per byte. The USB 2.0 bus runs at 480 Mbit/s and sustains roughly 35-40 MB/s of real isochronous video in practice, so the request is over budget by a factor of about three. UVC handles that by advertising 1080p30 only as MJPEG and capping YUYV at 5 fps. If your code asks for YUYV at 30 fps, negotiation fails and every read returns (False, None). Ask for MJPEG and the same stream costs roughly a tenth — about 12 MB/s — which fits with room to spare.

cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*"MJPG"))
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 1920)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 1080)
cap.set(cv2.CAP_PROP_FPS, 30.0)
print(cap.get(cv2.CAP_PROP_FOURCC))   # 1196444237.0 is "MJPG" packed as an integer

Confirm what the stick really supports with v4l2-ctl -d /dev/video2 --list-formats-ext; if it lists YUYV 1920x1080 at 5 fps maximum, no OpenCV call will raise that — the limit is the link, not the library. Two other causes share this symptom: no HDMI source connected (the handle stays open and every read returns False), and having opened the metadata node instead of the capture node. For frames that arrive but come out black, walk the pipeline stage by stage; when several USB cameras share one controller and its buffer overruns, raise the host USB buffer.

Torn or duplicated frames point at the physical link before software: move the stick to a rear-panel host port and avoid hubs. If it shares a root hub with other USB devices, give it a private lane — the MV-GU2204 quad-port USB3.0 adaptor puts it on its own PCI Express controller — and take the wire off the suspect list with a purpose-built 3 m high-flex USB3.0 camera cable.

Decision: negotiate MJPEG on any USB 2.0 link; run YUYV only on USB 3.0, where even 1080p60 YUYV — 4,147,200 × 60 = 249 MB/s — fits.

Why does the stick crash, hang or bluescreen on Windows?

On Windows, open with CAP_DSHOW before CAP_MSMF, because the Media Foundation path stalls or bluescreens with some consumer capture drivers, and remember that a stick with no HDMI source connected delivers nothing: cap.isOpened() stays True while every read returns False.

The causes, in order of probability. First, the backend: if MSMF hangs on open or the process dies inside Media Foundation, switch the single argument to CAP_DSHOW — that clears most reports. Second, the driver: update it in Device Manager and prefer the stick vendor's own driver over the generic "USB Video Device" entry; a bluescreen is kernel-level and no user-space code will mask it. Third, the handshake order: connect and power the HDMI source before opening the device, because some sticks sit in a dead state when opened with no signal present. Fourth, mid-stream source changes: when the HDMI source switches resolution, close and reopen the capture rather than resetting properties on a live pipe.

One more discipline: open once at program start and keep the handle. Reopening per frame re-runs format negotiation, and that is where most hangs begin. If the crash follows a bandwidth problem instead — a busy root hub shared with other cameras — the host-controller fix in the previous section applies unchanged.

Decision: CAP_DSHOW first, vendor driver if it still crashes, HDMI source connected before open, one open per program run.

Why is /dev/video permission denied on Linux?

A "Permission denied" error on Linux means your user is not in the video group: the UVC nodes are owned root:video with 0660 rights, so add yourself with sudo usermod -aG video $USER, log out and back in, and use a udev rule on a production machine.

ls -l /dev/video2 shows crw-rw---- root video — that 0660 mask is what udev applies to video4linux nodes by default. The tempting chmod 0666 /dev/video2 gets you through the afternoon and dies at the next replug, because udev re-applies its rules on every hotplug event. The durable fixes, in order:

1. Bench machine: add your user to the group — sudo usermod -aG video $USER — then log out and back in; group changes do not reach running sessions.
2. Production machine: a udev rule in /etc/udev/rules.d/99-uvc-capture.rules containing SUBSYSTEM=="video4linux", MODE="0666", then sudo udevadm control --reload-rules && sudo udevadm trigger. Narrow it with the idVendor/idProduct values from lsusb if the same box hosts cameras that must stay locked down.
3. Container: pass the verified node explicitly — --device /dev/video2 — instead of mounting the first node blindly; passing the V4L2 media nodes, not just /dev/video0 covers the full mapping.

The Linux camera device permissions walkthrough covers the node checks in depth, and when rights are correct but the SDK still cannot see the stick, the UVC capture path guide is the next stop.

Decision: video group on the bench, udev rule in production, an explicit device flag in containers.

When is a capture stick the wrong tool for an inspection job?

If the job needs hardware triggering, sub-frame timing or repeatable measurements, the stick is the wrong part: it adds one to three frames of latency (33-100 ms at 1080p30), exposes no trigger I/O, and its exposure controls live in the HDMI source, not in OpenCV.

Sticks earn their place where the HDMI signal already exists and you only need to see or record it — a microscope's output, a test-rig display, a converter on the lab bench. Machine vision asks for four things a stick cannot give:

• Trigger and strobe I/O — you cannot tie exposure to a part-present signal.
• Bounded latency — one to three frames is the range for a UVC stick, and there is no timestamp you can defend in a cycle-time claim; measure your actual figure with the method in camera buffer timestamp latency.
• Camera controls — exposure, gain and white balance belong to the HDMI source; cap.set(cv2.CAP_PROP_EXPOSURE, ...) writes go nowhere.
• Delivery guarantees — UVC over a consumer dongle drops frames silently under load.

For any of those, put a real camera on the part: GigE Vision and USB3 Vision cameras expose GenICam controls, hardware triggers and trustworthy timestamps, and it is worth reading what GigE Vision guarantees to your software. Where no camera link carries the bandwidth — high line-rate line scan or multi-camera cells — frame grabbers with Camera Link, CoaXPress or 10GigE interfaces take over, and the accessories finder filters them by interface and port count.

Decision: display and record through the stick; measure, trigger and guide robots through a machine-vision camera.

How do you prove the capture works end to end?

Prove the path with a ten-line probe that opens the node by index with a forced backend, sets MJPEG at 1920×1080 and 30 fps, reads sixty frames and prints the shape and the time between the first and last, then keep it as your bring-up test.

Run it before any application code, on any new device, port or kernel:

1. Connect and power the HDMI source, then plug the stick into a rear-panel USB port on the host.
2. Verify the index or path — v4l2-ctl --list-devices on Linux, the sweep loop on Windows.
3. Open with the forced backend; cap.isOpened() must print True.
4. Set FOURCC MJPG, then 1920, 1080 and 30.0 fps; read them back with cap.get() — expect 1196444237.0, 1920.0, 1080.0 and 30.0.
5. Read 60 frames in a loop, count the True return flags and time the loop — expect 60/60 after the first frame and about 2 s, since 60 frames at 30 fps is 60 / 30 = 2 s.
6. Print frame.shape and frame.dtype — expect (1080, 1920, 3) and uint8, the BGR Mat OpenCV promised.

Keep that probe forever; it is the difference between "the stick changed" and "my code changed" on a bad day. Prevention, condensed:

• Force the backend constant in every open call.
• Negotiate MJPEG first on USB 2.0 links; reserve YUYV for USB 3.0.
• Keep the USB run short and shielded — camera cables covers the purpose-built options.
• Ship the udev rule, not a chmod.
• Open once per program run and watch the return flag forever: a stick that loses its HDMI source keeps the handle open and returns False.

If the numbers above push you toward a proper camera or a frame-grabber path, the accessories configurator starts at the right step, and we help with the stick-versus-camera decision through the contact page — expert support is free with every order.

Decision: if VLC shows the stream and the probe prints 60/60 with (1080, 1920, 3), ship it; if you need triggers or defensible timing, order a machine-vision camera instead.