VIDIOC_REQBUFS busy with multiple cameras: serialize the start-up

You are bringing up eight serializer/deserializer (serdes) cameras on one embedded carrier. Each camera runs as its own process, launches go out one after another, and an earlier camera is often already streaming when the next one starts. Randomly, one of them fails at ioctl(fd_, VIDIOC_REQBUFS, &req) with errno 16, Device or Resource Busy. The file descriptor is healthy — other ioctls on the same node succeed — and MMAP and UserPTR buffer types fail identically. What follows is the diagnosis in order of probability, the test that confirms each cause, and the fix: a start-up gate that waits for each camera's first frame before the next launch.

What errno 16 on the buffer request means

The V4L2 API gives VIDIOC_REQBUFS two documented busy conditions: another open file handle already owns the driver's buffer allocations, or the device is streaming and the request would change them mid-flight. Check both against the V4L2 ioctl documentation for your kernel version before looking anywhere else. Neither holds here. lsof | grep /dev/video shows no second holder, and the failing camera has never streamed a frame.

So the practical meaning on a multi-camera serdes carrier is this: the driver refuses the allocation because that camera's upstream pipeline — sensor, serializer, deserializer, capture engine — has not finished reaching the state where buffers can be sized and pinned. The call is refused, not queued, and an identical retry meets the same state. That is exactly what the while-loop experiment showed: spinning on the ioctl until it returns 0 bought nothing.

Cause 1: the next camera starts before the previous pipeline settles

First in probability, and the one your own evidence convicts. Each serdes camera needs its own bring-up — the deserializer locks, the sensor enumerates, the media-pipeline stages connect — and on shared carrier drivers that work runs serialized across the whole capture subsystem. With one process per camera, camera N+1's buffer request lands while camera N is still mid-bring-up, and the driver answers busy because, from its side, it is.

The test is decisive, and you have already run half of it. Launch the same eight cameras, same nodes, same order, with v4l2-ctl: the error disappears, because v4l2-ctl's launches wait — the next camera does not begin initializing until the previous one completes. The 10-second sleep passes the same test for the same reason: it guarantees that no two cameras initialize at once. Two positive results on one hypothesis close the case.

Cause 2: eight cameras pull inrush current at the same instant

Second in probability, and it compounds the first. Cameras that power together draw inrush together; a carrier rail that sags can delay one camera's link lock or drop it mid-initialization, and the failure then surfaces at the buffer request instead of as a clean power fault. Clocks pinned high and about 28 GB of RAM free at the moment of failure leave power as the physical suspect.

Test: stagger, and watch the kernel log. A first-frame gate staggers power draw automatically — one more reason to install it. If failures persist even serialized, size the camera rail: sum steady draw plus the inrush of every camera that can power at once, and compare with the carrier's budget.

Cause 3: memory is ruled out, and two numbers prove it

First, the errno: allocation exhaustion reports the out-of-memory errno (12, ENOMEM), not 16. Second, the pool: MMAP buffers come from the boot-time contiguous reservation, which system free RAM does not show — yet the set-up had about 28 GB free and MMAP and UserPTR failed identically, the signature of a sequencing refusal rather than a memory shortfall. If you ever do meet the out-of-memory errno on this call, raise the contiguous reservation; on embedded boards the kernel's CONFIG_CMA_SIZE_MBYTES setting is the one to change.

The fix: gate each launch on the previous camera's first frame

Run this sequence per camera, one camera at a time:

1. Open the camera's /dev/video node.
2. Set the format first; buffer size follows the format.
3. Request the buffers — the VIDIOC_REQBUFS call that fails today.
4. Map or register them (MMAP or UserPTR), queue every buffer, and switch the device to streaming.
5. Block for one frame with a timeout of 10 s. That matches the delay already proven to work in this set-up, and a serialized camera typically delivers its first frame in under 2 s.
6. Only now start the next camera.

Across processes, make the gate a contract: the orchestrator spawns camera 1, waits for a readiness signal that the process emits only after its first frame — one line on stdout, a pipe write, a socket message — then spawns camera 2. If a camera misses its gate, log it and retry that camera alone; never relaunch the whole set at once. For the wait itself, Frame didn't arrive within timeout: one wait call per frame is the pattern to copy: one blocking wait per frame, no polling loop.

Worked example: start-up time and buffer memory

Budget about 0.2 s per camera for open, format and buffer setup, plus up to 2 s for the first frame.

• Serialized, worst case: 8 × (0.2 s + 2.0 s) = 17.6 s — call it 18 s for eight cameras.
• Fixed 10 s sleep: 8 × 10 s = 80 s.
• The gate saves about 62 s and never launches a camera into a settling pipeline.

Buffer memory, with 1080p-class frames at 2 bytes per pixel and four buffers per camera:

1920 × 1080 × 2 = 4,147,200 bytes, or about 3.96 MiB per buffer. Times four buffers: 16,588,800 bytes, about 15.8 MiB per camera. Times eight cameras: 132,710,400 bytes, about 127 MiB. Double the buffer count and the total sits near 253 MiB — still nowhere near the free memory logged at the time of the failure. The contiguous reservation must cover it on an embedded board, but a shortfall there announces itself as the out-of-memory errno, never as 16.

Checks before you blame the driver

• Run lsof | grep /dev/video before the run and confirm there is no second holder.
• Bring the same nodes up with v4l2-ctl in the same order; Capture raw Bayer RG10 frames from six V4L2 cameras with v4l2-ctl shows the working pattern on a six-camera rig.
• Read the kernel log at the failure timestamp for link, power or probe messages naming the failing node.
• Pin node assignment so enumeration order cannot silently swap cameras after a reboot — the failure mode behind Multiple frame grabber device order: fix swapped camera channels.
• Separate the siblings: the same errno on the format ioctl usually means two processes on one node (VIDIOC_S_FMT device or resource busy: two nodes opening one camera); a busy state that survives a restart is a stale handle (V4L2 device busy after restart: the old node still holds the camera).

Give each camera its own host port

When the camera count grows or the carrier keeps misbehaving, stop sharing the bring-up path entirely. A multi-port host card gives every camera its own port and link, so per-camera initialization cannot contend — the architecture most large multi-camera lines land on eventually. The bandwidth arithmetic that forces the move is in How many USB cameras one controller can feed before drops.

Put each camera on its own port with a multi-port card:

All 3 share: Camera quantity supported 4 at most; Stream 4; Humidity 10% RH to 95% RH (no condensation). Values as printed in each product's data sheet; open a part for its drawing and full specifications.

The frame grabbers finder lists all three interfaces. Cable each run as one continuous lead with no mid-span joint, and pick a drag-chain rated lead where the camera moves — the LS-CBL-0199 is a 3 m GigE camera cable, high-flex, built for drag-chain use.

When serialized start-up is the wrong answer

• v4l2-ctl fails on the same node every time: that is hardware — the camera's link, its connector, or the rail from cause 2. Fix the link before touching sequencing.
• One camera, nothing else running, still gets a busy buffer request: a real second holder or leftover driver state; the two sibling articles in the checklist cover both.
• The errno is the out-of-memory one: pool sizing, per cause 3 — raise the contiguous reservation.
• Start-up is clean but frames drop at full rate: the gate fixed bring-up, not bandwidth. Redistribute cameras across ports or cut the per-camera data rate.

The decision, case by case

• Eight cameras, one carrier, random errno 16 at VIDIOC_REQBUFS: install the first-frame gate. Start-up lands near 18 s, with a 10 s timeout per camera.
• Failures that survive serialization: fix power or the failing link first; sequencing cannot repair a sagging rail.
• Clean start-up, drops at rate: redistribute cameras onto per-port host cards.
• A set that keeps growing: move to per-port links now, and the race stops existing.

To re-spec the camera side while you are at it, start from the camera finder or the camera step of the configurator, and browse area scan cameras for the resolutions you need. For a power-rail question on a custom carrier, write to us through the contact page.