Rotate a camera feed before neural network detection

A camera mounted 90° to fit a cramped bracket, a tall part or a portrait display breaks a detection network in three ways at once: the model sees sideways objects, the crop stage resizes the wrong geometry, and the boxes come back mapped to a frame you never display. This guide fixes all three on a concrete example — a 640 × 400 preview feeding a 300 × 300 detector. You will put the rotation before the network in a single crop-and-resize stage, keep a stereo depth path intact, and draw boxes that land on the faces. The same wiring applies to any fixed-input detector on any portrait mount.

Why does a rotated camera feed break a detection model?

Because the model was trained on upright objects at a fixed input size, a 90° feed must be rotated and resized to that size before inference, or confidence drops and the boxes it returns land on the wrong pixels.

A detector is a fixed-size function. A typical face model consumes exactly 300 × 300 pixels; feed it a full sensor frame and the node rejects the size, feed it a naive squash and the geometry lies. Training data is overwhelmingly upright, so sideways faces score low even though the pixels are identical up to rotation. The output compounds it: coordinates come back normalized 0–1 against the model's own input, so every mismatch between what it saw and what you draw on shows up on screen. Keep the example numbers — 640 × 400 in, 300 × 300 inferred — in mind through the wiring.

The example only asks for one clean landscape stream, which any of the shop's area scan cameras provides; the camera finder filters by resolution and interface. The LS-CMR-0166 sends 4000 × 3000 frames at 30 fps over USB3. Scaling that sensor frame down to the example's 640 × 400 preview discards nothing the detector needs.

Where should the rotation run: camera, pipeline, or host?

Rotate inside the acquisition pipeline, in the image-manipulation stage between the camera and the network, so the detector sees upright frames at full frame rate and the host CPU never spends its frame budget moving pixels.

Three places can do the job, in order of preference. Re-mount the camera upright when the bracket allows: zero code, and model, crop and display agree by construction. Otherwise run the rotation on the pipeline's manipulation node: it executes in the sensor's data path, costs the host nothing, and the network's input is correct by wiring rather than by discipline. A host-side rotation (OpenCV, for example) is the flexible bench option, but every frame pays a full copy and you must keep both paths rotated identically by hand. If the rotation serves measurement rather than detection, the guide on rotating a camera image 90 degrees before measurement covers the coordinate side.

When the pipeline lives on a dedicated box instead of a desktop, pick from the vision controllers: the MV-VC3301H-128G66 pairs an Intel Core i3 with 8 GB of memory and five or more LAN ports, a common pairing for a camera-and-display bench.

How do you crop and rotate in one stage to the network input?

Define a rotated rectangle centred on the frame's middle with width and height swapped, set its angle to 90°, and let the same manipulation stage resize the result to the network input: one stage, one copy, correct orientation.

Concretely: preview 640 × 400, detector input 300 × 300. Define the rectangle on the stream you link into the node — centre (640 / 2, 400 / 2) = (320, 200), size swapped to 400 × 640, angle 90°. The node extracts the upright portrait crop and resizes it in the same pass. With aspect preservation off, the two axes scale differently:

A 0.75 horizontal scale against 0.47 vertical visibly squashes faces. The alternative keeps the aspect ratio: resize at 0.75 to 300 × 480, then centre-crop to 300 × 300, discarding (480 − 300) / 480 = 37.5% of the vertical field. Where faces cross mid-frame — a door, a bench — the crop wins; keep the distortion version only if faces can sit near the frame edges.

One caution prevents most miswiring: the centre and size are coordinates of the input frame. Define them from the stream feeding the node, not from the sensor's native frame, or the crop lands partly outside the image and detection dies quietly.

How do you wire the pipeline, step by step?

Run two paths out of the camera: a rotated, resized stream that feeds only the detector, and a full-size stream you rotate separately for display — never wire the detector's input stream to the display queue.

1. Set the camera's preview to the display size, 640 × 400 here, and leave the sensor in its native orientation. Nothing rotates yet.
2. Insert one manipulation node between the preview and the detection network. Configure the rectangle from the previous section — centre (320, 200), size 400 × 640, angle 90° — with resize to 300 × 300.
3. Declare the output frame type to match the model's channel layout. Planar RGB against interleaved BGR is the classic swap, and a grayscale tensor into a 3-channel input fails in its own quieter way — the grayscale-to-3-channel-input guide covers that variant.
4. Link the manipulation output to the network only. The 300 × 300 stream is the model's plate, not your viewer's.
5. Rotate a second copy of the full stream for display: the same rectangle resized to 400 × 640, or a host-side transpose of the landscape frame. Both rotations must agree, or the boxes will not line up.
6. Draw on the stream that shares the detector's orientation, scaling normalized coordinates by that stream's width and height — the arithmetic sits in the mapping section below.

Verify with a red object and a face-sized target at mid-frame: colors prove the channel order, the centred target proves the crop, and a box hugging the target proves the mapping.

Do you rotate the stereo depth pair too?

No: leave the left and right images in their native orientation until depth exists, because stereo matching searches along horizontal epipolar lines that a rotation destroys; rotate the finished depth image afterwards instead.

A stereo block finds each pixel's match on the same scanline of the other image. Turn either image 90° and the search line rotates with it, so disparities come back meaningless — and left-right consistency checks and subpixel refinement inherit the same assumption. Depth, once computed, is an ordinary image and rotates freely; nothing upstream re-runs.

Among the shop's 3D sensors, a projected structured-light camera such as the MV-DPS200P-02 outputs an already-computed depth map, so on that class of hardware the rotation question lives entirely on the host side of that output. Spatial coordinates attached to detections are the second trap: they are computed against the depth frame's geometry, so map them with the same discipline as the boxes — the guide on passing bounding-box coordinates to a depth function shows the pairing.

Rotate depth and color by the same transform and their registration survives. If the overlays drift after a rotation change, the depth-to-RGB registration offset guide is where to look.

How do you map detection boxes back onto the rotated image?

Detector outputs are normalized 0–1 against its own input, so multiply xmin and xmax by the displayed frame's width, ymin and ymax by its height, and draw on the stream that carries the same rotation as the network input.

Take a detection at xmin = 0.25, ymin = 0.10, xmax = 0.75, ymax = 0.60, drawn on the upright display copy from step 5 (400 × 640):

Rectangle corners: (100, 64) to (300, 384). The widespread helper that fills a normalizer array with the frame's height and overwrites the even entries with the frame's width implements exactly this — width for the x pair, height for the y pair. Draw those same pixel values on the unrotated 640 × 400 frame and they land on the wrong pixels: the classic "detection doesn't work" symptom with a perfectly healthy model.

For per-face crops — age, gender, attributes — crop after detection, not before. The rotated-bounding-boxes guide walks the detect-then-crop order and the per-box rectangle math.

Why did the preview shrink to 300 × 300 and swap colors?

The shrunken preview and the swapped colors are two independent wiring bugs: the viewer was fed the detector's 300 × 300 input instead of the camera stream, and planar RGB frames reached a viewer that expects interleaved BGR.

Fix the first with one link change: route the display queue from the camera stream or its rotated full-size copy (step 5), never from the manipulation node feeding the network.

Fix the second with a frame-type declaration. Set the manipulation node's output type to what the consumer expects, or convert once on the host; blue faces and orange lips mean the channels arrived swapped, and the BGR-to-RGB conversion guide shows the one-line repair.

Neither bug shows in a screenshot of the wrong stream. Check colors on something red and geometry on a centred target before trusting the pipeline.

When is rotating the feed the wrong fix?

If the model has only ever seen upright, landscape training images, perfect wiring still leaves weak detections, so the durable fix is retraining on frames that match the mounted camera — or mounting the sensor upright in the first place.

Sort your case before writing code. Bracket movable: re-mount upright, done, no accuracy loss. Mount fixed and the model tolerant — most general object detectors survive 90° with an aspect-preserving crop — the pipeline rotation above is the right answer. Detections vanished entirely, or accuracy must be maximal: the model never saw sideways examples, so fine-tune on captures from the mounted orientation until validation confidence matches the upright baseline. The wiring in this guide delivers frames the model can read; only training decides what it recognizes.

Bench the chain in order — camera stream, manipulation node, network, separate display path — before anything goes on a bracket. Start the camera configurator below, and the systems topic page collects the surrounding pipeline fixes, from frame-rate gaps to packet loss.