Downscale images after capture and match on-camera resolution

You feed one RGB stream into two consumers: a 3D reconstruction step that wants every pixel, and a machine-learning model trained at a lower resolution. Capturing twice at two resolutions wastes bandwidth and can desynchronize the two views, so the obvious fix is to capture full-size and downscale in software. The catch is that a naive resize changes the aspect ratio and shifts pixels, so the two paths no longer agree. Here is how to downscale a captured frame so it approximates a native low-resolution stream, and how to measure the difference before you trust it.

Why the two paths differ at all

A camera set to a lower output resolution does not simply resize its full frame. Depending on the sensor it may bin adjacent pixels, skip rows and columns, crop the sensor area, or run a scaler in the camera FPGA. Each of those has its own filtering and noise behaviour, so pixel values will never match a software resize exactly. What you can demand is that the two are close enough for your downstream task, and that the geometry — field of view and aspect ratio — is preserved exactly.

The aspect-ratio problem is arithmetic. A 1920 × 1080 frame is 16:9; 640 × 480 is 4:3. A plain resize to 640 × 480 compresses the horizontal axis by a different factor than the vertical one, and everything looks stretched. That is the distortion you saw, and it is a geometry decision before it is a filtering decision.

The relevant comparison for depth work is covered from another angle in depth to color registration: raw versus rectified frames, which explains why registration results depend on which frame variant you feed the mapper.

Choose the target geometry first

You have three legitimate options, and mixing them is what breaks pipelines:

1. Keep the source aspect ratio. Resize 1920 × 1080 to 640 × 360. Nothing is distorted and the full field of view survives; your ML model must accept 16:9 input.
2. Letterbox to 4:3. Resize to 640 × 360, then paste onto a 640 × 480 canvas with 60-pixel bars top and bottom. Distortion-free, but the model sees black bars and you must crop them before any geometric measurement.
3. Center-crop then resize. Crop the source to 1440 × 1080 (4:3), then resize to 640 × 480. No bars, but you lose 25 percent of the horizontal field of view, and depth-to-color alignment breaks unless you crop the depth data identically.

For a pipeline where the same frames also drive 3D reconstruction, option 1 or 2 is usually right, because every source pixel remains represented. Option 3 silently changes the extrinsics between the two processes.

Resize with the right filter

Downscaling is a sampling operation, and dropping pixels aliases. Use area-weighted averaging, which integrates every source pixel contributing to each destination pixel:

In Python terms:

1. Load the full-resolution frame as a NumPy array; keep it in its native dtype so you do not clip highlights.
2. Resize with cv2.resize(src, (640, 360), interpolation=cv2.INTER_AREA). INTER_AREA is the area-weighted average and the correct default whenever the scale factor is below about 0.5.
3. If you letterbox, allocate the canvas with np.zeros((480, 640, 3), dtype=resized.dtype) and paste the resized frame at the computed offset — 60 rows top and bottom for the 16:9-to-4:3 case.
4. Do the resize once per frame, in a worker process if the main loop is busy, and reuse the same code path for training data generation so train and inference see identical preprocessing.

Bilinear and nearest-neighbour resize are faster but visibly alias fine texture; bicubic is acceptable for mild factors. scikit-image's downscale_local_mean and resize with anti-aliasing enabled give results consistent with area averaging.

When the camera can do it for you

Many cameras reduce resolution on the sensor or in the camera, which saves interface bandwidth before any software runs. On-sensor binning also improves signal-to-noise, because summed pixels average out read noise — something a software resize of an already-digitized frame cannot recover. The trade-off is that the two resolutions then come from different readout modes, and matching them in software becomes harder, not easier.

For example the LS-CMR-0166 delivers 4000 × 3000 at 30 fps over USB3, and a lower-resolution mode of the same camera will not behave like an area-average of that 12 MP frame — the readout and any binning differ. Frame rate interacts with resolution and pixel format too, as RGB camera frame rate limits: pixel format and exposure explains; dropping resolution is often how you buy frame rate.

If your low-resolution consumer is a trained model and you would rather not run a PC pipeline at all, a smart camera such as the LS-SMC-0063 runs inference on-device at its native resolution, sidestepping the comparison entirely.

For 3D sensors the same question appears on the depth side; see change 3D camera depth resolution and field of view at capture time for what a capture-time resolution change does to coverage.

The comparison workflow

This is the part most teams skip and later regret. Run it once per camera per scene type:

1. Fix the scene and lighting; do not move anything between captures.
2. Capture a sequence at the native low resolution (for example 640 × 480 4:3, or your camera's nearest mode).
3. Capture the same scene at full resolution and downscale it with the code above, using your chosen geometry option.
4. Scale the camera intrinsics to the low-resolution grid. Every intrinsic — focal lengths and principal point — scales linearly with its axis:

For letterboxed or cropped targets you must also shift the principal point by the crop or bar offset. The conversion is laid out in camera intrinsic parameter conversion for depth and texture images. 5. Compute mean absolute difference between the two low-resolution images over at least ten frames and three scenes. Single-digit intensity levels (0-255 scale) is a realistic target for a static scene. 6. Compute a structural-similarity score; expect high scores on smooth regions and dips on high-frequency texture, where anti-aliasing choices diverge. 7. Feed both versions through the actual downstream process — the ML inference and the 3D reconstruction — and compare outputs, not just pixels. A model's predictions can be stable across pixel differences that would fail a per-pixel test, and vice versa.

If the metrics pass but outputs diverge, the mismatch is usually geometric: a one-pixel principal-point error from a letterbox offset, or a crop applied to RGB but not to depth. Check those before blaming the interpolation.

Checks before you commit

• Aspect ratio of the target matches the convention you picked, with no stretch.
• The same resize code path generates training images and runtime frames.
• Intrinsics rescaled and principal point shifted for any letterbox or crop.
• Error metrics recorded per scene, so a camera firmware change later has a baseline.
• Downstream outputs compared, not just pixels.

When native low resolution is the better answer

Downscale-after-capture is right when you need one synchronized feed, identical field of view for both consumers, and you have CPU to spare. Choose native low-resolution capture instead when interface bandwidth is the limit, when you want binning's noise advantage, or when the low-resolution consumer runs standalone — for example a vision sensor on the line. If you are also shopping for the full-resolution camera, the area scan cameras finder filters by resolution and interface, and the camera resolution calculator tells you the minimum pixels your inspection actually needs before you pay for resolution you will throw away. For combined depth-plus-RGB work, browse 3D sensors. A modest-resolution part such as the LS-CMR-0053, a 12 MP GigE camera, may already give you headroom to serve both consumers from one stream.

If you would rather have the comparison workflow run against specific hardware before you commit, contact us and we will go through it with your camera and scene.