Merge CSI and USB camera feeds into one overlay output

Two cameras, one screen. A CSI camera carries the main view at native resolution, a USB camera feeds a detector that runs an HEF model, and the operator needs both in a single picture: the CSI image full-frame, the USB image inset in the bottom-right corner. One composited output means one encoder, one recording and one stream to the network, and it lets the model run on the USB branch without ever touching the display path. This guide builds the merge as a GStreamer compositor pipeline: bring each camera up on its own, tee the USB branch so inference runs on unscaled frames, size the inset with arithmetic you can repeat on any canvas, and hold a steady 30 fps on a small board.

What you need

A board with one free CSI camera connector and one free USB port, both cameras in hand, and GStreamer 1.x with the compositor element — modern 1.x builds ship it; very old stacks knew the same job as videomixer. You also need the element that runs your HEF model; it comes with the accelerator's own plugin package, not with GStreamer. Everything else is caps strings and queues.

• CSI camera — the main view, full-frame at native resolution.
• USB camera — the detector branch; the example build uses a webcam mode of 640×360 px at 30 fps in MJPEG.
• The HEF model file and its runtime element.
• A display sink or an encoder for the merged canvas.

Webcams are fine for a preview inset. When the small picture also has to carry inspection value — stop motion on a moving line, fixed focus, known optics — choose the USB branch camera from the shop's USB3 board-level cameras instead:

The LS-BCM-0003 streams 120 fps at 3.1 MP with a global shutter, so fast lines stay sharp in the inset. The LS-BCM-0047 is the lighter 1.3 MP colour option when the inset only checks presence and orientation.

Bring each camera up on its own

1. Stream the CSI camera by itself with your board's CSI source element and write down the exact mode it delivers — resolution, pixel format, frame rate. Those values become the branch caps later. If the source delivers no frames at all, clear that first with the CSI camera no-images checklist. A raw sensor without an ISP needs debayering or colour conversion before it can merge — see what raw sensor output means.
2. Stream the USB camera by itself through v4l2src; the UVC bring-up guide needs no configuration file. List the modes the device offers: most webcams give 1280×720 or 640×480 at 30 fps in MJPEG, plus uncompressed YUY2 at 640-class sizes. Pick one mode and stay in it.
3. Pin the device paths. /dev/video indexes can swap after a replug or a reboot, so point each branch at a stable by-id or by-path entry instead of a bare index. Two cameras that quietly trade places after a power cycle are the classic cause of a model inspecting the wrong scene.
4. Decide the canvas now: the main camera's native mode, 1920×1080 at 30 fps in this build. Both branches must end in caps the compositor can agree on — the same pixel format and the same frame rate on every pad.

Two branches into one compositor

The compositor element stacks raw video streams: pad sink_0 paints first, pad sink_1 paints over it. Each pad takes position properties — xpos and ypos, the top-left corner of that pad's picture on the canvas — plus alpha for translucency. Every leg gets its own queue so a blocked leg stalls only itself.

The pipeline below is the whole merge. csi-src is a stand-in for your board's CSI source element: on boards that expose the CSI camera as a V4L2 device it is another v4l2src, otherwise it is the vendor's camera-stack source. model_in is a stand-in for the element that runs your HEF model. /dev/video2 is the example USB index — use the stable path from step 3.

gst-launch-1.0 -e \
  compositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_1::xpos=1416 sink_1::ypos=786 \
    ! video/x-raw,width=1920,height=1080,framerate=30/1 ! videoconvert ! autovideosink \
  csi-src ! video/x-raw,width=1920,height=1080,framerate=30/1 ! videoconvert ! queue ! comp.sink_0 \
  v4l2src device=/dev/video2 ! image/jpeg ! jpegdec ! videoconvert ! videoscale \
    ! video/x-raw,width=640,height=360,framerate=30/1 ! tee name=t \
  t. ! queue ! videoscale ! video/x-raw,width=320,height=320 ! appsink name=model_in \
  t. ! queue ! videoscale ! videorate ! video/x-raw,width=480,height=270,framerate=30/1 \
    ! videoconvert ! queue ! comp.sink_1

Run it with -v the first time. The log prints the caps negotiated on every pad: the compositor's src pad should show the 1920×1080 canvas, sink_0 the CSI mode, sink_1 the 480×270 inset at 30/1. A branch that negotiates nothing means a caps mismatch — drop back to the solo pipelines from the previous section and re-read each camera's modes.

Sizing the inset: 480×270 on a 1920×1080 canvas

Inputs for this build:

• Canvas: 1920×1080 px at 30 fps — the CSI camera's native mode.
• Inset width: one quarter of the canvas width — the rule of thumb for an inset that stays readable without hiding scene detail.
• Margin: 24 px on each edge — an example value; 16–32 px reads well on a monitor.
• USB camera mode: 640×360 px, 16:9, at 30 fps in MJPEG.
• Model input: 320×320 px, the size the example HEF declares.

The rule: the inset width is a quarter of the canvas width; its height follows the camera's aspect, never a stretch; the corner offset places the top-left corner at canvas minus inset minus margin.

The arithmetic: overlay_w = 1920 / 4 = 480 px. overlay_h = 480 × 360 / 640 = 270 px. x = 1920 − 480 − 24 = 1416 px. y = 1080 − 270 − 24 = 786 px. Check the covered area: (480 × 270) / (1920 × 1080) = 129,600 / 2,073,600 = 6.25% of the canvas.

Result: a 480×270 px inset at (1416, 786), covering 6.25% of the frame. Decision: use exactly that on a 1920×1080 canvas. Keep the inset between one fifth and one quarter of the canvas width — above one quarter it starts covering detail the main camera exists to show; below about one sixth (320 px on this canvas) boxes and labels crowd. The same rule scales down: a 1280×720 canvas takes a 320×180 inset at (936, 516).

Run the HEF model on the USB branch only

The tee sits after decode and colour conversion and before either scaler. One leg scales to the size the HEF declares — 320×320 px here — and feeds the inference element; the other scales to 480×270 px for the display. The model therefore sees raw camera frames at its native input size, and the canvas never reaches the model. videoscale adds borders by default, so the 16:9 frame letterboxes into a square model input rather than stretching.

The leg is light on the wire: 320 × 320 px × 3 bytes = 307,200 bytes per frame, about 9.2 MB/s at 30 fps. Point the model at the composited canvas instead and it consumes 1920 × 1080 px × 3 bytes × 30 fps ≈ 186.6 MB/s of pixels — about twenty times the pixel rate — and the detections would include the inset itself, which is the wrong scene.

Draw the detector's boxes and labels as overlay graphics on top of the canvas instead of burning them into frames — see text overlay on a live camera image. Keep the consuming application allocation-free by reusing image buffers rather than allocating per frame.

Hold 30 fps: rates, queues and USB bandwidth

Frame-rate matching: both pads at the compositor must run the same rate, so cap each branch at 30/1 and put videorate on the USB leg; videorate duplicates or drops frames to hold the cap, and the compositor holds a pad's last frame if it runs late. Rates that drift between cameras show up as judder in the inset.

USB bandwidth, worst case: uncompressed YUY2 carries 2 bytes per pixel, so the 640×360 branch moves 640 × 360 × 2 × 30 = 13,824,000 bytes/s ≈ 13.8 MB/s ≈ 110.6 Mbit/s. That fits a USB 3 port with room to spare. On a USB 2 bus — about 35 MB/s practical for bulk, less for isochronous video — the same branch takes roughly 40% of the bus, so keep the camera on its own port or run MJPEG, which moves roughly a tenth of the YUY2 rate. When frames still drop, work through budgeting a USB hub's bandwidth and the frame-rate collapse diagnosis.

Compositing itself costs memory, not network: a 1920×1080 px canvas at 4 bytes per pixel is 8.3 MB per frame, about 249 MB/s of memory traffic at 30 fps. Desktop-class boards shrug; a small ARM board may not. If 30 fps will not hold, drop the canvas to 1280×720 — 1280 × 720 × 4 × 30 ≈ 110.6 MB/s — before blaming the model. The habit is the same one behind two GigE cameras on one host: give every interface its own budget.

Common failures and the fix for each

• Nothing renders: one branch negotiated no caps. Read the -v log; the compositor pads and the src pad must agree on format and frame rate, and each camera must still stream in its solo pipeline.
• The output stalls when the model starts: the inference leg has no queue, so its start-up block backs up into the compositor. Add a queue ahead of the inference element.
• The inset sits in the wrong place: xpos and ypos locate the inset's top-left corner. Recompute them as canvas minus inset minus margin.
• The inset looks stretched: the height was picked without the camera's aspect. Set both width and height from the formula — 480×270 from a 16:9 branch, never 480×270 from a 4:3 one.
• Green or torn frames on the USB leg: the decoder is fighting a format the camera switched into. Lock the camera to its MJPEG mode instead of letting the driver pick.

When not to merge

• The operator only watches live: two windows from two independent pipelines beat one composite on CPU cost.
• The display lives on another machine: send both streams and composite at the operator PC, where the pixels are effectively free.
• Both cameras feed algorithms that need full frame rate and full resolution: compositing adds a rescale and a copy on the merge path that those algorithms never asked for.

The decision rule: merge when one output is the requirement — one encoder, one recording, one stream to the network. Then the composite pays for itself; otherwise run two pipelines. If the main view later needs measurement-grade optics, swap the CSI module for a USB3 or GigE area scan camera and change one branch's source element; the compositor topology stays the same.

Before you ship: the merge checklist

• Sources. Each camera streams alone at its final mode, and both device paths are pinned by-id or by-path so a reboot cannot swap them — a swapped pair feeds the model the wrong scene.
• Merge. Both compositor pads negotiate the same pixel format and frame rate; a queue sits on every leg; xpos and ypos come from the sizing formula.
• Model. The inference leg's caps match the HEF's declared input, and the tee sits before the scalers, so the model sees raw frames and never the canvas.
• Output. Hold 30 fps for ten minutes with the model running and the display attached; the USB camera sits on its own port; the board has CPU headroom for a second encoder if you record.

Pick both cameras in the camera finder, work the merge out step by step in the system configurator, and if a branch refuses to negotiate, contact the applications team. More build-level guides are on the systems topic page.