Rotated bounding boxes in live inference: detect first, crop after

Your rotated-box detector already cuts labels cleanly out of saved images. The next step is a live stream, above 8 FPS, with every label cropped upright for the reader or classifier downstream. The exported ONNX file is not the obstacle; the gap is post-processing, because most embedded runtimes and video-analytics pipelines ship no rotated-box decode, no angle-aware NMS and no rotated crop. This guide gives you the working route: run the graph in a generic runtime, then decode, convert and crop in your own Python post-step. You get the frame-time budget for 8 FPS, the box-to-corner arithmetic for a 180 × 60 px label at 30°, the warp recipe with its test, and the checks that catch sign and convention errors early.

Decide: run the detector as exported, add the angle afterwards

Keep the ONNX file unchanged and stop hunting for a runtime with a native rotated-box head. Underneath the rotation output, a rotated-box detector is ordinary convolution and pooling; any runtime that executes standard ONNX ops runs the graph, ONNX Runtime included. What embedded stacks and video-analytics frameworks lack is the glue after the graph: decoding (cx, cy, w, h, θ) from the head, NMS that measures box overlap with the angle included, conversion to corner points, and the upright crop. Rotated-detection toolkits perform that glue in their training-time post-processing; the exported graph stops at raw head output, so the glue becomes a post-step you write once in Python.

The post-step is small: NumPy box math, one warp per kept box, one polygon draw for review. The plain capture-inference-draw loop is already covered step by step in ONNX detection on a camera stream in Python: the full loop; everything there still holds, and this guide adds only the angle handling. If the whole pipeline must sit on the camera itself, check the on-device smart cameras; for a fanless box that drives cameras, lights and this loop, see the vision controllers.

The alternative routes — hunting a framework plugin that does not exist, or writing custom runtime ops in C++ — cost weeks and buy nothing here. Detect with the model you trained; crop rotated after.

What you need

Hardware: one compute box for the loop, your camera, and the exported ONNX file. The detector sets the compute class: a ResNet18-FPN backbone clears 8 FPS on a small embedded GPU module, so a controller in that class leaves headroom for a post-step whose cost the budget section works out below. An AI inference terminal exists for exactly this shape of loop — camera ports in, detections out, at the line rather than in an office.

If you plan to add an accelerator card later, pick a controller with PCIe expansion so the card drops in without replacing the box.

Software: Python 3, with onnxruntime for the session, opencv-python for the warp and overlay, and numpy for the box math. All three libraries install from the standard Python package index and none needs a license server. Keep the training toolkit's export notes at hand: they define the head's output layout and the angle convention that step 3 pins down.

Most of these loops land on one of two controller points. The MV-VC5525-256G70-NN carries a 12th-gen Intel Core i5-class processor with 32 GB memory and PoE camera ports; the MV-VC3301H-128G66 runs an Intel Core i3 with 8 GB memory and includes a VM6200Pro license.

The frame-time budget for 8 FPS

Work the budget before touching code. The target is above 8 FPS, so each frame owns 1000 ms ÷ 8 = 125 ms. The detector is the expensive stage: measure it alone first, at the input resolution and batch size you plan to run, because the runtime choice alone can move that number by a factor of two (ONNX inference slower in OpenCV DNN: pick the right runtime). Whatever the detector leaves from the 125 ms is the post-step's allowance.

The post-step's arithmetic is tiny by comparison. Corner conversion costs 4 corners × (2 multiplications + 2 additions) = 16 multiply-adds per box, so 50 candidate boxes before thresholding come to 800 multiply-adds — noise. Bandwidth is the real cost: one upright crop of the worked-example box writes 180 × 60 = 10,800 px, which is 10,800 / 921,600 ≈ 1.2 % of a 1280 × 720 frame. Eight labels per frame write 86,400 px, still under 10 % of the frame's pixels.

Decision: if the detector alone clears 8 FPS on your box, the post-step fits, provided it stays vectorised. Per-pixel Python loops are the classic way to burn the allowance; the fix pattern is in slow Python OpenCV code: fix per-pixel loops that kill FPS. If you need 30 FPS or more, batch the warps and keep tensors on the accelerator instead of round-tripping through main memory.

Worked example: one 180 × 60 px label box at 30°

Inputs, in your head's own output format — the task fixes the shape: labels on parcels, randomly rotated, live stream, above 8 FPS. Centre cx = 640 px, cy = 400 px; width w = 180 px; height h = 60 px; angle θ = 30° from the image x-axis to the box's long side; frame 1280 × 720 px.

Rule: each corner is the box centre plus its half-extent pair rotated by θ.

Arithmetic, with cos 30° = 0.8660 and sin 30° = 0.5000. Corner (−90, −30): x' = 640 + (−90)(0.8660) − (−30)(0.5000) = 640 − 77.94 + 15.00 = 577.06 px; y' = 400 + (−90)(0.5000) + (−30)(0.8660) = 400 − 45.00 − 25.98 = 329.02 px. The other three follow the same rule: (732.94, 419.02), (702.94, 470.98) and (547.06, 380.98) px.

Result: the polygon (577.1, 329.0), (732.9, 419.0), (702.9, 471.0), (547.1, 381.0) px — the label's true footprint on the parcel.

Decision: hand that polygon to the crop step with a canvas of w × h = 180 × 60 px, and pad the frame only if a corner leaves 0…1280 × 0…720 px — here all four corners sit inside, so no padding. The threshold in plain terms: any corner outside the frame bounds means pad (border replicate) before the warp.

Steps: from ONNX file to upright crops

1. Serve the model once. Create the runtime session at start-up, not per frame; per-frame allocations are the usual source of creeping memory in live loops, the pattern in out of memory in a video loop: fix the per-frame image leak.

2. Undo the letterbox before using any coordinate. If the input pipeline scales or pads the frame, every raw head coordinate lives in model-input space. Map boxes back to full-frame pixels first, scale and offset both, or the NMS, the polygons and the crops all drift; the failure mode and fix are in bounding box offset on new images: the double letterbox fix.

3. Decode the head and pin the angle convention. Print the first rows of the output tensor and overlay one box as a polygon built from (cx, cy, w, h, θ). If the polygon crosses the short side, swap w and h and shift θ by 90°. If it tilts the wrong way, negate θ — once, in the decoder, nowhere else. Radians fed where degrees are expected shows up the same way. Reading the orientation back into image coordinates is the same mapping problem as in rotated barcode result points: map orientation back to the image.

4. Threshold, then NMS with the angle. Keep boxes above your operating score — 0.5 is a starting point; tune it on your rejects — then suppress overlaps with an intersection-over-union that sees rotation, polygon IoU on the corner points. Where labels sit far apart, NMS on the axis-aligned extent of each polygon keeps the same survivors for a fraction of the work.

5. No angle output? Measure it on the crop. Run the same loop with an axis-aligned detector, crop the rectangle, threshold or edge-detect the label region, and read the angle from a minimum-area rectangle fit (the OpenCV call is cv2.minAreaRect). Levelling an image from a measured angle is worked through in detect screw orientation angle and level the image.

6. Convert kept boxes to corner points with the function below.

7. Crop each box upright with the matrix below, and check its test line.

8. Draw and store. Overlay the polygon for review and save every upright crop with its frame number and angle; per-run foldering for that is in organize inference output folders for per-run detection results.

The corner function, with the worked example as its test

import numpy as np

# np.deg2rad / np.cos / np.sin / np.array: documented NumPy API names.
# Everything else is an illustrative variable your detector supplies per frame:
cx: float = 640.0        # centre x, px
cy: float = 400.0        # centre y, px
w: float = 180.0         # width, px
h: float = 60.0          # height, px
theta_deg: float = 30.0  # angle, image x-axis to long side, degrees

def rotated_box_corners(cx: float, cy: float, w: float,
                        h: float, theta_deg: float) -> np.ndarray:
    t = np.deg2rad(theta_deg)
    R = np.array([[np.cos(t), -np.sin(t)],
                  [np.sin(t),  np.cos(t)]])
    half = np.array([[-w / 2, -h / 2],
                     [ w / 2, -h / 2],
                     [ w / 2,  h / 2],
                     [-w / 2,  h / 2]])
    return np.array([cx, cy]) + half @ R.T   # (4, 2) corner points, px

print(rotated_box_corners(cx, cy, w, h, theta_deg)[0])
# -> [577.05771366 329.01923789]  = the worked example's first corner (577.06, 329.02)

The crop matrix, with its own test

The matrix rotates the frame about the box centre, then shifts so the centre lands on the canvas centre; the canvas is the box's own w × h.

import cv2  # cv2.getRotationMatrix2D and cv2.warpAffine: documented OpenCV API names

crop_w, crop_h = 180, 60                    # canvas, px = the box's w, h
M = cv2.getRotationMatrix2D((cx, cy), theta_deg, 1.0)
M[0, 2] += crop_w / 2 - cx                  # shift: box centre -> canvas centre
M[1, 2] += crop_h / 2 - cy
# frame: np.ndarray, shape (720, 1280, 3), the current BGR frame
crop = cv2.warpAffine(frame, M, (crop_w, crop_h))

print(M @ np.array([cx, cy, 1.0]))          # -> [90. 30.] = (crop_w/2, crop_h/2)

The printed [90. 30.] proves the label lands centred; only a real frame proves it lands level. If the crop is centred but the label inside is still tilted, the detector's angle sign is the mirror of the corner formula's — go back to step 3 and flip the sign there, not here.

Verify it worked

Three checks, in order. First, the unit test: the corner function on the worked-example values must return (577.06, 329.02) px as its first corner, within 0.01 px. Second, the matrix test: the centre must map to (90, 30) px on the 180 × 60 px canvas. Third, a synthetic end-to-end run: draw a filled rectangle on a black 1280 × 720 frame, rotate the whole frame by a known angle, run the pipeline, and require the saved crop to be upright and its polygon within about a pixel of the drawn truth. Only the synthetic run catches sign and convention errors — the first two tests pass either way.

Time the post-step separately from the model call. On the worked geometry — 10,800 px per crop, 16 multiply-adds per box — the post-step's share of the 125 ms budget should sit well below the detector's time; if it does not, per-pixel Python has crept back in.

Common mistakes

• NMS or drawing on raw head coordinates while the input was letterboxed: everything drifts by the pad and the scale (step 2).
• Degrees into a radians call or the reverse: a 30° label arrives as 0.52° and comes out almost axis-aligned.
• Cropping the polygon's axis-aligned rectangle from a separately rotated copy of the frame instead of warping by the matrix: double rotation, clipped corners.
• Flipping the angle sign in the crop step only, so the overlay and the saved crops disagree; keep the one conversion in the decoder.
• Pixel-by-pixel Python for the draw or the crop: vectorise with NumPy slicing or let the warp do it.
• Thresholding after NMS: threshold first, NMS second, or you spend polygon time on boxes you would have discarded anyway.

When detect-then-crop is the wrong route

Three cases argue against it. Dense scenes with hundreds of heavily overlapping rotated objects make polygon-IoU NMS slow and fragile; in that case train the rotated NMS into the model's graph and export it end-to-end instead. Presence-only questions — is a label there, is the carton sealed — need no angle at all; an axis-aligned box with a classifier costs less compute per frame. And when the angle itself is the measurement, a sub-degree skew check say, a detector box is not the instrument: fit the minimum-area rectangle to the label's pixels on the upright crop and read the angle from that.

Everything else in this family — labels on parcels, print on component reels, tags on tote bins — follows the route above.

Next step: size the box that runs the loop. The vision controllers cover the range from a few cameras to many, with PoE ports and built-in light channels; the system configurator builds the station around your camera; and if you send the detector's input size, the camera count and the target FPS through the quote request, we will come back with a controller and built-in light channels to match.