OpenCV image upscale comparison: which interpolation flag

You downsized the frame so the processing fits the time budget, and now the image on the operator screen looks soft. The fix is a one-flag choice inside cv2.resize, not a neural network: this comparison covers the classic OpenCV interpolation methods — INTER_NEAREST, INTER_LINEAR, INTER_CUBIC and INTER_LANCZOS4 — what each computes, what each costs per output pixel, and which to pass for display, for masks and for measurement. It also covers the pixel-art scalers behind those crisp block edges on retro graphics, EPX and xBR, and why they stop working the moment the source is a camera image. The examples assume a PC pipeline fed by area scan cameras; on a smart camera the resize happens inside the device, so the flag is not yours to pick.

The decision, in one call

Pass INTER_LANCZOS4 when a person looks at the upscaled frame and sharpness is the point. Pass INTER_LINEAR when the upscale runs every frame at camera rate. Pass INTER_NEAREST when the values must stay real — masks, label maps, anything you will threshold or count. Use INTER_AREA for the downscale before processing, never for the upscale: enlarging with INTER_AREA falls back to nearest-neighbor behavior, so you get blocks without choosing them.

The call, with a test that proves the behavior:

import cv2
import numpy as np

small = np.zeros((750, 1000), dtype=np.uint8)  # processed image, 750,000 px
small[:, 500:] = 255                           # one hard edge for the test
fx = fy = 4.0                                  # upscale factor, float

sharp = cv2.resize(small, None, fx=fx, fy=fy,
                   interpolation=cv2.INTER_LANCZOS4)
blocks = cv2.resize(small, None, fx=fx, fy=fy,
                    interpolation=cv2.INTER_NEAREST)

assert sharp.shape == (3000, 4000)             # 4x on each axis
assert np.array_equal(np.unique(blocks), np.array([0, 255], dtype=np.uint8))
assert len(np.unique(sharp)) > 2               # interpolation adds levels

INTER_NEAREST reproduces exactly the two source levels; every other flag invents intermediate ones. That property decides most cases on its own: legal values or pleasing edges.

What each flag actually computes

Every resampler answers one question: which source pixels feed one output pixel, and with what weights. Nearest copies one pixel. Bilinear weighs the surrounding 2×2 block, and a weighted average can never overshoot the source range. Bicubic fits a curve through the 4×4 neighborhood, which restores edge slope and overshoots a little. INTER_LANCZOS4 applies a windowed sinc over the 8×8 neighborhood — 64 source pixels, negative lobes included — which is why it reconstructs edges most faithfully of the classic flags and why it rings: bright and dark halos beside hard edges, clamped at 0 and 255 once you write 8-bit output.

INTER_AREA earns its row because it owns the other half of your pipeline. Shrinking with INTER_AREA averages every source pixel under each output footprint, so edges antialias; shrinking with nearest or bilinear drops pixels, and the later upscale displays that aliasing permanently. The antialiasing guide for downsampling and warping works through the mechanics.

What sharpness costs, in arithmetic

Cost scales with the tap count: 1, 4, 16 and 64 multiply-adds per output pixel. Upscaling to 12,000,000 output pixels therefore means 12,000,000 operations with nearest, 48,000,000 with bilinear, 192,000,000 with bicubic and 768,000,000 with Lanczos4, which is 12,000,000 × 64 taps. At a 30 fps display rate Lanczos4 asks 768,000,000 × 30 ≈ 23 billion multiply-adds per second; at a 5 Hz operator refresh, enough for a screen a person watches, 768,000,000 × 5 = 3.84 billion.

Measured wall-clock gaps come out below the tap ratio — commonly two to four times from bilinear to Lanczos4 on a desktop CPU — because resize vectorizes well and memory bandwidth dominates. Budget the taps for the arithmetic and about threefold for the clock. The lever that matters most is the rate, not the flag: upscale what is shown, at the rate it is shown.

Worked example: 12 MP in, ×4 down, ×4 back up

Take the LS-CMR-0166: a 12 MP USB3 camera on the IMX226 sensor, 1/1.7" rolling shutter, 4000 × 3000 output at 30 fps.

At 8 bits per pixel one frame is 4000 × 3000 × 1 B = 12,000,000 B, about 12 MB, and 12 MB × 30 fps = 360 MB/s of pixel data. Divide each axis by four and the processed image is 1000 × 750 = 750,000 pixels, one sixteenth of the count: an algorithm that needs 40 ms on the full frame needs about 40 / 16 = 2.5 ms on the small one.

Upscaling that result back to 4000 × 3000 with INTER_LANCZOS4 writes 12,000,000 output pixels at 64 multiply-adds each — 768,000,000 operations against 48,000,000 for INTER_LINEAR. As an order-of-magnitude estimate, a desktop core sustaining a few billion resize multiply-adds per second puts the Lanczos pass around a hundred milliseconds, next to the 2.5 ms of processing you just protected. Three patterns fix that:

1. Upscale at display refresh, 5 Hz, not at camera rate.
2. Upscale only the ROI the operator asked for, at native resolution.
3. Keep the split architecture — capture at full resolution while showing a low-resolution preview — and upscale nothing nobody looks at.

If the sharp frames feed a line HMI rather than a window on the vision PC, the display stack loads its own copy of the library; OpenCV load failures in an HMI image-streamer module are the recurring snag on that side.

Pixel-art scalers: EPX and xBR

The block-crisp upscales you see on retro sprites come from a different family. EPX, also known as Scale2×, turns every source pixel into a 2×2 block and lets the neighbors beside each corner decide whether that corner takes the neighbor's color; xBR and HQx apply the same idea with gradient tests. All of them assume an integer factor — ×2, ×3, ×4 — large regions of exactly equal color, and a small palette.

Camera images break all three assumptions at once. Sensor noise leaves two pixels of one flat surface a few counts apart, so the exact-match tests fire at random and the output collapses toward nearest-neighbor blocks with speckled corners. OpenCV ships none of these algorithms, so there is no flag to pass; you would implement EPX yourself, and it would still be the wrong tool on sensor pixels.

One vision-pipeline case does qualify: a binarized result, a classified overlay, any few-level image. There INTER_NEAREST at an integer factor already gives the pixel-art look exactly, because it returns pure blocks of real values. EPX and xBR earn their keep on palette graphics — sprites, UI art, flat-fill renderings — and nowhere else in this pipeline.

When upscaling is the wrong fix

Resampling adds pixels, not information. The downscale kept one pixel in sixteen, and the upscale interpolates between the survivors; no flag changes that. Before committing to INTER_LANCZOS4, check whether the viewer needs detail that only the sensor and the lens can supply.

Use the rule from camera resolution for defect detection: 3 to 5 pixels across the smallest feature a person must judge. For a 300 mm field of view and 0.25 mm features, that is 300 / 0.25 = 1200 feature widths across the field, and 1200 × 3 = 3600 pixels needed across the sensor. A 4000-pixel-wide sensor meets it with margin; a camera short of that needs more pixels or a tighter lens, not a better interpolator.

For more true pixels, the MV-CS200-10UM is the step up: 20 MP on a 1" sensor over USB3, IMX183, mono.

Set the two side by side:

Values as printed in each product's data sheet; open a part for its drawing and full specifications.

The lens has to deliver those pixels too. Lens selection by feature size and MTF and the Nyquist budget method for lens and pixel size cover the pairing; in the machine vision lenses category the LS-FFL-0567 is a 16 mm, 1.1", 20 MP C-mount lens.

The measurement rule is separate: never measure on the upscaled copy. After a ×4 resample every edge is a kernel product, so sub-pixel positions and edge widths carry the interpolation's smoothing rather than the sensor's. Run geometry on native pixels — or on the processed small image in its own coordinates — and multiply overlay coordinates by four when you draw them on the display. If you conclude you want a neural upscaler after all, read ONNX inference in OpenCV DNN and picking the runtime first; for per-frame work the classic flags above stay the performance answer.

Checks before you commit

• Shrink with INTER_AREA. Nearest or bilinear downscaling aliases, and the upscale afterwards shows that aliasing permanently.
• Masks, label maps and classified overlays: INTER_NEAREST, always.
• Human-viewed frames: INTER_LANCZOS4 for stills and slow refresh, INTER_LINEAR when the upscale runs per frame.
• Above about 10 million output pixels, upscale at display refresh rather than camera rate — at Lanczos4 that is 64 multiply-adds per pixel, every frame.
• Threshold and count on the processed small image; the upscaled 8-bit copy carries ringing clamped at 0 and 255.
• Show a native-resolution crop when a person must judge small features.
• The CUDA resize path implements the simpler flags only — there is no Lanczos4 there — so keep the Lanczos upscale on the CPU or step down a flag before porting.

Next step: run your own numbers. The pixel rule tells you whether the upscale stays cosmetic or the sensor has to move; when it moves, the camera finder and the configurator started at the camera step take you to a shortlist, and the lens finder keeps the optics matched to the pixel pitch.