Scale the ISP frame for full FOV previews in ROS

You wired a 12 MP camera into a ROS Noetic machine, and the preview crawls at about one frame per second while the teach-and-repeat routine waits for images. What that routine needs is width, not pixels: the whole scene in frame, at a rate a small host can publish. Scale the frame inside the camera pipeline, where the image signal processor already touches every pixel, instead of resizing on the host. This guide sets a 1/5 scale that turns 4056 × 3040 pixels into 812 × 608, shows the bandwidth arithmetic behind it, wires the output into a ROS topic, and repeats the same move with binning and decimation on GigE and USB3 machine cameras.

The decision: scale before the frame leaves the camera

Set the downscale in the camera pipeline whenever the preview needs field of view rather than detail. A host-side resize still receives, converts and copies every pixel of the full frame before shrinking it: the embedded CPU does 12 million pixels of work to display half a million. Scaling in the ISP happens while the frame is produced, so the host receives 812 × 608 pixels and publishes them directly. On the embedded host this guide follows, that one change took the preview from about 1 fps to about 4 fps — a 4× rate gain from a 25× payload cut, because the publish path then became the next limit.

The arithmetic that forces the decision

A 12 MP frame, 4056 × 3040 pixels in RGB8, carries 3 bytes per pixel:

• Pixels per frame: 4056 × 3040 = 12,330,240
• Bytes per frame: 12,330,240 × 3 B = 36,990,720 B ≈ 37 MB
• At 25 fps: 36,990,720 × 25 = 924,768,000 B/s ≈ 925 MB/s ≈ 7.4 Gbit/s

The scaled stream after setIspScale(1, 5):

• Pixels per frame: 812 × 608 = 493,696 (4056 / 5 = 811.2, which the scaler rounds up to 812; 3040 / 5 = 608)
• Bytes per frame: 493,696 × 3 B = 1,481,088 B ≈ 1.5 MB
• At 25 fps: 1,481,088 × 25 = 37,027,200 B/s ≈ 37 MB/s ≈ 0.30 Gbit/s

12,330,240 / 493,696 = 25.0, so twenty-five times fewer bytes leave the camera. The measured preview rate rose about 4× rather than 25× for two reasons: the ISP works hardest at full 12 MP input, and the host's convert-and-publish step costs CPU on every message. To chase the next limit, find the frame-rate bottleneck in ROS 2 publishing walks the same search on the host side.

Set the ISP scale, step by step

The ROS wrapper of that generation resized previews by changing the aspect ratio, with letterboxing on the roadmap. Scaling in the ISP needs neither once the ratios match, and it works on ROS Noetic today without waiting for a wrapper update.

1. Create the color camera node and select the 12 MP sensor mode: camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP).
2. Set the scale before you link any output: camRgb.setIspScale(1, 5) — numerator 1, denominator 5.
3. Fix the output geometry and format: camRgb.setPreviewSize(812, 608), camRgb.setInterleaved(False), camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.RGB).
4. Cap the rate: camRgb.setFps(25). An uncapped stream queues faster than the host drains it, and latency grows without bound.
5. Create the output node: xoutIsp = pipeline.create(dai.node.XLinkOut), then xoutIsp.setStreamName("isp").
6. Link the ISP stream: camRgb.isp.link(xoutIsp.input).

# camRgb: color camera node; xoutIsp: output node; dai: the camera SDK module
camRgb = pipeline.create(dai.node.ColorCamera)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)
camRgb.setInterleaved(False)
camRgb.setIspScale(1, 5)      # 4056 x 3040 -> 812 x 608
camRgb.setPreviewSize(812, 608)
camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.RGB)
camRgb.setFps(25)             # ISP load peaks at 12 MP input

xoutIsp = pipeline.create(dai.node.XLinkOut)
xoutIsp.setStreamName("isp")
camRgb.isp.link(xoutIsp.input)

On the host, the SDK's bridge converts each ImgFrame into a sensor_msgs/Image and your node publishes it. Convert RGB to gray after the bridge, at preview size — 493,696 pixels per frame instead of 12,330,240.

Full width: scale, crop or letterbox

Three ways to fit a wide scene into a small preview, and they are not interchangeable:

4056 / 3040 = 1.334 and 812 / 608 = 1.336 — both effectively 4:3, so the scaled frame fills the preview with no bars. If the display is 16:9, pad rather than stretch: stretching changes angles and distances in the image and quietly breaks anything calibrated in pixels. Cropping to hit a frame size is the one move that defeats the purpose, because the width you kept is the width you cut; undistortion scaling: cropping versus field of view makes the same argument in a calibration pipeline. When a smaller stream matters more than the edges, camera ROI cropping at the source shows where to cut before the host ever sees the frame.

What a 1/5 scale does to your pixel scale

Take a 12 MP board-level camera with 1.6 μm × 1.6 μm pixels and a C-mount:

A 12 MP grid of square pixels is 4000 × 3000. At 1.6 μm per pixel the sensor spans 4000 × 1.6 μm = 6.4 mm across and 4.8 mm high. Put a 16 mm lens on it and work at 300 mm:

Full-resolution pixel scale: 120 mm / 4000 px = 0.030 mm per pixel. After a 1/5 scale: 120 mm / 800 px = 0.15 mm per pixel. The same 120 mm of scene stays in frame; each preview pixel covers a 5 × 5 block of sensor pixels instead of one. Sensor resolution versus scaled video output covers the relation from the camera's side, and simulated camera resolution and pixel scale shows how to set it when the camera is simulated.

Keep at least 2 pixels across the smallest feature the routine must find. At 0.15 mm per pixel that is any feature 0.3 mm or larger — comfortable margin for a preview that only has to locate the machine in the scene.

The same move on GigE and USB3 machine cameras

An ISP scale is one implementation of a general rule: spend the camera's silicon, not the host's CPU. Across area scan cameras and board-level cameras, the GenICam feature tree gives you three levers:

1. Binning merges pixels before readout. 2×2 binning quarters the pixel count and, in the shot-noise limit, roughly doubles signal-to-noise.
2. Decimation skips columns and rows after readout. It costs no extra silicon, and it aliases on fine repeating detail such as meshes, threads and displays.
3. ROI crops the frame: full detail on a smaller window, paid for in field of view.

For a 12 MP USB3 camera with a 4000 × 3000 grid at up to 30 fps:

Do the same arithmetic for any camera in this class. Full-rate RGB8 from a 4000 × 3000 grid at 30 fps is 4000 × 3000 × 3 B × 30 = 1,080,000,000 B/s ≈ 1.08 GB/s ≈ 8.6 Gbit/s — beyond the 5 Gbit/s line rate of a USB 3 link, so something gives before the frame reaches ROS. With 2×2 binning the same stream is 2000 × 1500 × 3 B × 30 = 270,000,000 B/s = 270 MB/s ≈ 2.2 Gbit/s, which fits with headroom.

Plan on roughly 400 MB/s usable per USB 3 link after encoding and protocol overhead; when cameras share a controller, budget the USB bandwidth first, and when two run full resolution together, raise the host USB buffer. Sensors without binning still reach the same frame size through decimation, at the cost of the aliasing above.

Where camera-side scaling belongs

Reach for it when the host is small, the link is shared, or the preview feeds a person rather than a measurement. Publish two streams when both consumers exist: a scaled preview topic for monitoring and a full-resolution topic triggered on demand — multi-ROI capture and its frame-rate arithmetic sizes the second stream. Skip host resizing when a workstation with hardware resize sits downstream; the trick earns its keep on embedded CPUs. When the whole job fits on one device, smart cameras run inspection on board and put pass/fail decisions on the network, so the frame never crosses to ROS at all. If you draw targets on the live view, update the overlay every frame at the preview rate keeps the two in step.

Two 6 MP mono options for small hosts and on-board inspection:

The board camera mounts inside the machine with no housing around it; the smart camera does its inspection on the device and reports over the network. More host-side plumbing is collected under the systems topic.

Verify the change

1. Size. Assert on the message your test subscriber receives:

# image_msg: the sensor_msgs/Image your test subscriber received
assert image_msg.width == 812, image_msg.width
assert image_msg.height == 608, image_msg.height

2. Rate. Measure the topic rate at the publisher and again at the subscriber; a gap between them points at transport or a slow consumer, not the camera.
3. Field of view. Put a marker at the frame edge before the change; after the change it must still be visible, at the edge.
4. Latency. Compare capture time with arrival time. Rate without latency is how previews look live and steer late.
5. Geometry. No letterbox bars and no stretched circles: the output ratio must match the sensor's.

Checklist before you commit

• Scale set in the camera pipeline, before the output link — never a host resize for an embedded preview.
• Output ratio matches the sensor ratio: 4:3 in, 4:3 out.
• Frame rate capped, setFps(25), so queues cannot grow.
• mono8 published when the routine is grayscale.
• A test asserts 812 × 608 on the live topic.
• Rates and latency measured before and after, at both ends.
• Full-resolution path still reachable for teaching and measurement.

Pick the sensor with the camera finder, start a build at the camera step of the configurator, or write to us through the contact page when the host side fights back.