System designHow-to

YOLO OBB ONNX inference in C++: decode rotated boxes

Run YOLO OBB inference in C++ with ONNX Runtime: tensor shape, angle decode, rotated NMS, and the diff test that proves your boxes.

LumaSpec Vision9 min read Share on LinkedIn

On this page 10 sections
  1. 01What you need
  2. 02The output tensor: one extra channel
  3. 03Preprocess: letterbox, then keep gain and pad
  4. 04Run the session
  5. 05Decode each row into a rotated rectangle
  6. 06Rotated NMS that respects the angle
  7. 07Prove it matches the Python output
  8. 08Common mistakes and their symptoms
  9. 09Where the decode runs on the floor
  10. 10Questions

You trained an oriented bounding box model, exported the ONNX file, and the Python predict call draws clean rotated boxes. The line software is C++, and the reference C++ example the framework publishes targets plain detection: four numbers per box, no angle. An OBB model adds one angle channel per candidate, so the session call barely changes and the decode step is where everything breaks. This guide builds the whole pipeline in C++ with ONNX Runtime and OpenCV: the output tensor layout, the letterbox arithmetic, the rotated-rectangle decode, a rotated NMS loop, and the test that proves your boxes match the Python output to within half a pixel. It is written for whoever has to put OBB inference on a factory-floor machine without dragging the Python training stack along.

What you need

  1. ONNX Runtime C++ libraries — the CPU execution provider is enough at 640 px; link the static or dynamic build into your console project.
  2. OpenCV — for resize, padding, color conversion and drawing. You will not use its dnn module for the model itself.
  3. The exported model — for example yolov8n-obb.onnx, exported from the OBB task at a fixed 640 px input. If you exported dynamic shapes, the code below reads the real shape at run time; the rect-versus-square export choice is covered in non-square image size in YOLO training: rect vs square.
  4. One test image with its Python answer — run the training framework's predict on it once and keep the boxes. That is your ground truth for the diff test near the end.
  5. The Python loop this replaces — it is the same one in ONNX detection on a camera stream in Python: the full loop; only the decode changes.

The output tensor: one extra channel

Plain detection exports [1, 4+nc, N]: four box numbers plus class scores per candidate. The OBB head appends one more channel, the rotation angle, so the shape becomes [1, 5+nc, N]. At a 640 px input, N = 80² + 40² + 20² = 6400 + 1600 + 400 = 8400 candidates. The example model is trained on DOTA with 15 classes, so the tensor is [1, 20, 8400] and one frame of raw output is 20 × 8400 × 4 bytes = 672,000 bytes. The layout is channel-major: candidate i, channel c sits at out[c * 8400 + i], so you never need to materialize a transpose.

Channels (of 20) Content Example row
0-3 x, y, w, h in network pixels 312.5, 200.0, 36.2, 18.4
4-18 15 class scores, sigmoid already baked into the export 0.87 at channel 6
19 angle in radians 0.35

Confidence is simply the largest of the class scores — there is no objectness channel on this export. The angle is radians, not degrees, and it arrives unwrapped. A rectangle is unchanged by a half-turn, so fold it into [-π/2, π/2] with an atan2 before you compare or draw boxes; the drawn box is identical either way.

Preprocess: letterbox, then keep gain and pad

The input is 640×640 RGB, values scaled to 0-1, planar channel-first. Letterbox keeps the aspect ratio: scale by the smaller ratio, then pad the short side with gray 114, the same filler color the training-side letterbox uses, so the border pixels look like the ones the model saw in training. Whatever gain and pad you apply here comes back out in the decode, so hold them in variables, not in your head.

gain = min(640 / W, 640 / H) pad_x = (640 - W * gain) / 2 pad_y = (640 - H * gain) / 2 x_src = (x_net - pad_x) / gain y_src = (y_net - pad_y) / gain w_src = w_net / gain h_src = h_net / gain angle_deg = angle_rad * 180 / pi

A 1280×720 camera frame: gain = min(640/1280, 640/720) = 0.5, scaled size 640×360, pad_x = 0, pad_y = (640 - 360) / 2 = 140 top and bottom.

const int   netW = 640, netH = 640;
const float gain = std::min((float)netW / src.cols, (float)netH / src.rows);   // 0.5 for 1280x720
const int   newW = cvRound(src.cols * gain), newH = cvRound(src.rows * gain);  // 640 x 360
const int   padX = (netW - newW) / 2, padY = (netH - newH) / 2;                // 0 and 140

cv::Mat resized, rgb;
cv::resize(src, resized, cv::Size(newW, newH));
cv::copyMakeBorder(resized, rgb, padY, netH - newH - padY, padX, netW - newW - padX,
                   cv::BORDER_CONSTANT, cv::Scalar(114, 114, 114));
cv::cvtColor(rgb, rgb, cv::COLOR_BGR2RGB);

std::vector<float> input(3 * (size_t)netH * netW);                  // CHW planar, 0-1
for (int c = 0; c < 3; ++c)
    for (int y = 0; y < netH; ++y)
        for (int x = 0; x < netW; ++x)
            input[(size_t)c * netW * netH + y * netW + x] = rgb.at<cv::Vec3b>(y, x)[c] / 255.0f;

If the tensor you feed does not match the exported input size, boxes land wrong before any decode runs — that failure and its fix are in YOLO inference image size mismatch: fix visual accuracy. Running at the original image size instead of letterboxing is a different, deliberate choice; its trade-offs are in YOLO predict at original image size: set imgsz at run time.

Run the session

YOLO exports use stable tensor names: input "images", output "output0". If your exporter differs, read the names off the session once at start-up. After the first run, take nc from the output shape (channels minus 5) instead of hardcoding 15, so the same decode serves a 5-class and a 15-class model.

Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "obb");
Ort::Session session(env, L"yolov8n-obb.onnx", Ort::SessionOptions{});
Ort::MemoryInfo mem = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);

std::array<int64_t, 4> inShape = {1, 3, netH, netW};
Ort::Value inTensor = Ort::Value::CreateTensor<float>(mem, input.data(), input.size(),
                                                      inShape.data(), inShape.size());
const char* inNames[]  = {"images"};
const char* outNames[] = {"output0"};
auto outputs = session.Run(Ort::RunOptions{nullptr}, inNames, &inTensor, 1, outNames, 1);

auto shape = outputs[0].GetTensorTypeAndShapeInfo().GetShape();     // {1, 20, 8400} here
const float*  out   = outputs[0].GetTensorData<float>();
const int64_t ch    = shape[1], n = shape[2];                       // 20 channels, 8400 rows
const int64_t angCh = ch - 1;                                       // angle is the last channel

Decode each row into a rotated rectangle

Start from the framework's own predict defaults: 0.25 confidence and, in the NMS below, 0.7 IoU. Keep them until a measured reason moves them. The loop reads the buffer column-wise, folds the angle, un-letterboxes, and builds a cv::RotatedRect directly — no minAreaRect round-trip, whose angle convention differs.

struct Detection { cv::RotatedRect box; float conf; int cls; };
std::vector<Detection> keep;

for (int64_t i = 0; i < n; ++i) {
    float best = 0.0f; int bestCls = -1;
    for (int64_t c = 4; c < ch - 1; ++c) {                          // class channels 4 .. ch-2
        float s = out[c * n + i];
        if (s > best) { best = s; bestCls = (int)(c - 4); }
    }
    if (best < 0.25f) continue;

    float cx = out[0 * n + i], cy = out[1 * n + i];
    float w  = out[2 * n + i], h  = out[3 * n + i];
    float r  = out[angCh * n + i];                                  // radians
    r = std::atan2(std::sin(r), std::cos(r));                       // fold to [-pi/2, pi/2]
    if (r >  1.5707963f) r -= 3.1415927f;
    if (r < -1.5707963f) r += 3.1415927f;

    Detection d;
    d.box  = cv::RotatedRect(cv::Point2f((cx - padX) / gain, (cy - padY) / gain),
                             cv::Size2f(w / gain, h / gain), r * 57.29578f);  // degrees
    d.conf = best;  d.cls = bestCls;
    keep.push_back(d);
}

Rotated NMS that respects the angle

cv::dnn::NMSBoxes compares axis-aligned rectangles only. Two parallel strips tilted 20° overlap almost totally as bounding rects; two crossing thin boxes overlap barely. Both verdicts are wrong for OBB, so compute IoU from the rotated polygons themselves: box.points() gives the four corners, cv::intersectConvexConvex gives the intersection area. Greedy is enough: sort by confidence, keep a box unless a stronger kept box overlaps it above 0.7.

std::sort(keep.begin(), keep.end(),
          [](const Detection& a, const Detection& b) { return a.conf > b.conf; });

std::vector<Detection> results;
for (size_t a = 0; a < keep.size(); ++a) {
    std::vector<cv::Point2f> pa; keep[a].box.points(pa);
    bool suppressed = false;
    for (const Detection& r : results) {
        std::vector<cv::Point2f> pb; r.box.points(pb);
        std::vector<cv::Point2f> inter;
        double interArea = cv::intersectConvexConvex(pa, pb, inter);
        double unionArea = keep[a].box.size.area() + r.box.size.area() - interArea;
        if (unionArea > 0.0 && interArea / unionArea > 0.70) { suppressed = true; break; }
    }
    if (!suppressed) results.push_back(keep[a]);
}

One nuance when you diff against Python: the reference implementation scores rotated overlap with a probabilistic-IoU variant, so a pair sitting at IoU 0.70 ± 0.02 can flip sides. Pairs away from the threshold agree exactly. From here the detections go wherever the line wants them — saving YOLO OBB detections to CSV with timestamp and confidence shows one sink, and rotated bounding boxes in live inference: detect first, crop after covers the crop order downstream.

Prove it matches the Python output

  1. Run the training framework's predict on the test image and print boxes.xywhr — center x and y, w, h, and the angle in radians.
  2. Fold both angle sets into the same half-turn interval, then convert the model's radians to degrees.
  3. Run your C++ binary on the same image.
  4. Compare: centers and sizes within half a pixel, angles within half a degree.
// expected values from the worked row above, test_1280x720.png, top result
const float tolPx = 0.5f, tolDeg = 0.5f;
assert(std::abs(results[0].box.center.x - 625.0f) < tolPx);
assert(std::abs(results[0].box.center.y - 120.0f) < tolPx);
assert(std::abs(results[0].box.size.width  - 72.4f) < tolPx);
assert(std::abs(results[0].box.size.height - 36.8f) < tolPx);
assert(std::abs(results[0].box.angle       - 20.1f) < tolDeg);

When the numbers disagree, the fault is in the letterbox bookkeeping or the angle channel index, almost never in the session call. If confidences also sag, the preprocessing drifted between training and deployment; OpenCV ONNX confidence lower than training: match the preprocessing walks that checklist.

Common mistakes and their symptoms

  • Angle from the wrong channel — every box shares one tilt, or class scores look like angles. The angle is the last of the 5+nc channels; index ch - 1, with ch read from the output shape.
  • Radians fed to a degrees field — a 0.35 rad box drawn as 0.35° sits near axis-aligned instead of tilted 20.1°. Multiply by 180/π exactly once.
  • Pad or gain skipped — every box pulls toward the frame center by pad/gain pixels: 140 / 0.5 = 280 px vertically on the 1280×720 example.
  • Axis-aligned NMS — long, thin, tilted boxes survive as stacked duplicates. Replace cv::dnn::NMSBoxes with the rotated loop above.
  • BGR fed as RGB, or the /255 dropped — confidence collapses or drifts; the preprocessing-match guide above lists each divergence.
  • Batch dimension missing — the export wants [1, 3, 640, 640]; a [3, 640, 640] tensor fails the shape check before inference. Feeding one channel instead of three fails the same way — feeding grayscale images to a 3-channel model input covers that case.

Where the decode runs on the floor

The pipeline is a plain console process: CPU execution provider, one frame buffer, no GPU. At 640 px an n-size model generally lands in the tens to low hundreds of milliseconds per frame on a current four-core x86 CPU — time the Run call plus decode with std::chrono and divide by the frame count rather than quoting a number you never measured. If the cycle budget misses, a GPU-class edge target with a TensorRT export is the usual route; fixing YOLO performance on Jetson-class hardware with TensorRT export walks that path. If you built on cv::dnn and it trails the standalone runtime, the reasons and the pick are in ONNX inference slower in OpenCV DNN: pick the right runtime.

On the floor this is one smart camera streaming GigE frames to one vision controller, which runs this decode and answers with pass/fail bits or part coordinates; the camera finder narrows the camera choice. Both of these controllers carry the pipeline as written:

Vision controllersHikrobot MV-VC2140-128G40-1T VC2100 Vision Controller, N97, 8GB+128G SSD+ 1THDDMV-VC2140-128G40-1T
LAN ports
3-4 LAN ports
USB 3 ports
2 USB 3 ports
Light outputs
4 built-in channels
Memory
8 GB
$1,679.00View partBuild a system with it
Vision controllersHikrobot MV-VC3301H-128G66 I3-8100, 8GB+128G SSD, No PoE, PCIe expansion supported, Incl. VM6200Pro licenseMV-VC3301H-128G66
LAN ports
5+ LAN ports
USB 3 ports
3-4 USB 3 ports
Light outputs
4 built-in channels
Memory
8 GB
$2,419.00View partBuild a system with it

The MV-VC2140-128G40-1T runs an Intel N-series processor with 8 GB of memory and a 1 TB HDD for image logging. The MV-VC3301H-128G66 steps up to an Intel Core i3-8100 with 5+ LAN ports and PCIe expansion supported. Log every frame with its detections while you tune thresholds; the CSV format in the OBB-detection-to-CSV guide above is ready for it. When the whole chain needs sizing, start at build a system; the systems topic page collects the sibling deployment guides, and the applications team can tell you which smart cameras load user models if you want inference inside the camera housing.

Questions

Where is the angle channel in a YOLO OBB ONNX output?

In the last channel. For a model with nc classes the tensor is [1, 5+nc, N]: x, y, w, h, then the nc class scores, then the angle in radians. Read the channel count from GetTensorTypeAndShapeInfo and index ch - 1; never hardcode it. In the 15-class DOTA example the angle is channel 19 of 20, so row i reads out[19 * 8400 + i].

Can I reuse my plain YOLOv8 detect decoder in C++?

Almost. The letterbox, the 0-to-1 scaling, the column-major indexing and the max-class confidence are all unchanged. Three things must change: read the extra angle channel, build a cv::RotatedRect instead of an axis-aligned cv::Rect, and replace cv::dnn::NMSBoxes, which only compares axis-aligned rectangles, with the rotated-IoU loop shown above.

Why do my C++ boxes sit a few pixels away from the Python output?

Almost always letterbox bookkeeping. The network sees the padded image, so every box must subtract the pad and divide by the gain before drawing. Rounding matters as well: split each padding remainder between the two sides exactly the way the training-side letterbox does. After the fix, centers agree within about a pixel and angles within a degree, once both sides are folded into the same half-turn interval.

Which confidence and IoU thresholds match the Python defaults?

0.25 confidence and 0.7 IoU, the same values the predict call uses when you do not override them. Start there, then tune to the part: lower confidence to 0.10-0.15 when small or low-contrast boxes get missed; raise the IoU cut toward 0.8 when two close parallel parts collapse into one detection; lower it toward 0.5 when one part survives as stacked duplicates.

Can the exported OBB model run directly on a smart camera?

Only where the camera firmware lets you load your own ONNX and run custom post-processing; many smart cameras execute their own model formats inside a fixed pipeline instead. The common layout is a camera streaming frames to a vision controller, where this decode runs and answers over digital outputs or the network. Ask which cameras accept user models before you plan the cell around it.