How virtual try-on works in OpenCV: pose, segmentation and warping

A clothing app that shows a garment on the shopper's own body is not a photo filter. It is a measurement pipeline: find the body, decide where the torso is, place the garment there, and keep it there on the next frame. This guide explains how a virtual try-on works in OpenCV, stage by stage — pose, segmentation, warping — with the numbers that decide whether the loop runs smoothly: keypoint counts and tolerances, milliseconds per stage, and the pixel sampling behind them. It is written for developers adding try-on to a fashion app, and it closes with the case where a machine-vision shelf beats the phone: an in-store kiosk with a fixed camera, controlled light and no variables left to chase.

Can OpenCV really run a virtual try-on on a phone?

Yes — a try-on is three stages, pose estimation, person segmentation and garment warping, and a lightweight pose network alone runs in roughly 20-50 ms per frame on a mid-range phone GPU, while the full serial loop lands at 15-30 fps.

OpenCV is the engine for two of the three stages and the glue for the third. Its DNN module loads a pretrained pose network — Caffe, TensorFlow or ONNX formats — and runs the forward pass on CPU or GPU (official OpenCV documentation). Warping and alpha compositing are pure geometry and pixel math, which is OpenCV's home ground. Only the learned stages need network weights, and public training sets already provide them.

The part developers underestimate is not speed but discipline: the garment has to follow the body across frames, which turns a one-shot filter into a small real-time system with a frame budget, a pose update rate and a render loop that runs faster than the inference.

What happens in each of the three stages?

Pose estimates body keypoints with a confidence per joint, segmentation separates the person from the background as a soft mask, and warping draws the garment onto the torso — pose finds where, the mask decides what shows, the warp renders it.

The stages feed each other in that order. Pose output is a short list of coordinates plus confidences; segmentation output is an 8-bit mask the size of the frame; the warp consumes both plus the garment texture, an RGBA image whose anchor points you measure once per garment.

Treat the last two rows as budgets rather than stopwatch facts: pose dominates because it is a full convolutional network, the mask model is smaller, and the warp is plain geometry that the GPU does almost for free. A mask one frame behind the pose is invisible in practice — cloth moves far slower than joints.

How many keypoints does it take to fit a garment?

Four keypoints place a top — the two shoulders and the two hips give position, scale and rotation in one step — and sleeves pull in the elbows and wrists, so a shirt lives on six to eight of COCO's 17 body points.

The COCO layout — nose, eyes, ears, shoulders, elbows, wrists, hips, knees, ankles — is the common language of pretrained pose models because the benchmark labels over 200,000 images and 250,000 person instances with those 17 points. Extensions such as COCO-WholeBody add face and hand keypoints, but a torso fit needs only the core set.

Map the garment before the person: for each SKU, measure the shoulder-seam span and the hem line on the flat texture, in pixels, and store them beside the image. Trousers and dresses swap the shoulder line for the hips, plus knees or ankles. Then gate on confidence: when a shoulder or hip scores below about 0.3, skip the frame rather than warp to a guess, because a wrong anchor looks worse than a frozen garment.

How do you segment the person from the background?

Use a lightweight person-segmentation model rather than colour thresholding: it returns a soft mask that stops the garment spilling onto the background and lets arms crossed in front of the torso stay in front — something flat thresholding cannot resolve.

Compositing is then one line per pixel: output = α × garment + (1 − α) × frame, where α is the garment's own alpha channel multiplied by the person mask. Feather the mask edge by two or three pixels, because a hard cut reads as a sticker. The mask also settles occlusion: pixels where the person's arms sit in front of the torso keep the frame, not the garment.

Classical thresholding still earns its keep when you control the scene. A matte, evenly lit wall behind a kiosk subject separates cleanly in Lab colour, the same logic as automatic threshold segmentation in Lab colour with Otsu's method. For the compositing mechanics themselves, cropping through a polygon mask with a transparent background shows how the alpha path behaves.

How do you warp the garment so it follows the body?

Anchor the garment with a similarity transform from its own shoulder line to the detected shoulder line — that fixes position, scale and tilt in one step — then refine with a mesh warp so the fabric bends at the elbows and hips instead of hanging like a flat sticker.

Worked example: the shirt texture stores a shoulder-seam span of 420 px; the detected shoulders come back 180 px apart in a 720 px frame. The scale is 180 / 420 = 0.43. The right shoulder sits 12 px above the left across 176 px of horizontal run, so theta = atan2(-12, 176) = -3.9° and the garment tilts with the shoulders. Hem length and sleeve span follow the same 0.43 factor; nothing else needs fitting.

The mesh stage divides the garment into a coarse grid, pulls nodes near each keypoint toward it and splines the rest; OpenCV's remap call consumes exactly that per-pixel source map. Because the texture is drawn at 0.43 of its size, filter while shrinking or the seams shimmer — the standard fixes are in antialiasing when you downsample or warp images. The geometry is the same point-correspondence math that places fiducial tags in OpenCV, with shoulders standing in for printed corners.

How fast can the loop run frame by frame?

Budget the loop around the pose stage: a lightweight network takes 20-50 ms per pass on a mid-range phone GPU, so running every stage on every frame lands near 15-30 fps — instead, update pose every second or third frame and keep the warp on every frame.

The arithmetic for a 30 fps target, with every stage at its slow end:

• Frame budget: 1000 ms / 30 fps = 33.3 ms.
• Pose, 50 ms, amortised over three frames: 50 / 3 = 16.7 ms.
• Segmentation, 20 ms, amortised over three frames: 20 / 3 = 6.7 ms.
• Warp plus composite on every frame: 5 ms.
• Total: 16.7 + 6.7 + 5 = 28.4 ms, under the 33.3 ms budget, so 30 fps holds in the worst case.

Between pose updates the warp reuses the last keypoints, smoothed so the garment glides instead of snapping; an exponential average across the last few poses is enough. The rendering side is the same pattern as updating overlay graphics every frame at full preview rate: draw the expensive pose rarely, redraw the overlay every frame.

Is a phone camera good enough?

For a try-on preview, yes: the pipeline needs silhouette and keypoints, not measurement, and a front camera covering about 700 mm of width at 0.5 m delivers 1920 / 700 ≈ 2.7 px per millimetre, so a 2 mm seam still reads at about 5 px.

What the phone costs you is stability. Auto-exposure hunts under shop lighting and shifts every colour the mask depends on, and rolling shutter skews a waving arm. Lock the exposure, ask the shopper for a beat of stillness during capture, and accept that the model input, not the optics, sets placement accuracy. What a consumer module gives up next to a machine-vision camera — deterministic timing, locked exposure, honest frame rates — is catalogued in consumer camera vs machine vision camera: what you give up.

When does an in-store kiosk need machine-vision hardware?

Build a kiosk when the result must be repeatable: a fixed camera at a fixed distance with controlled light removes the variables a phone cannot, and a 12 MP USB3 camera then samples a 1,000 mm-wide frame at 4 px per millimetre.

Fix the geometry first. A 1,000 mm horizontal field of view covers shoulders plus arm swing on a standing subject, and pixels divided by coverage set the sampling: 4,000 px over 1,000 mm gives 4,000 / 1,000 = 4 px per millimetre, so a 2 mm seam reads at 8 px — and the figure never drifts, because neither distance nor lens changes. Budget the pose input separately: typical heatmap models accept 256-368 px, so over a 1,000 mm view one input pixel is about 2.7 mm of body and a two-pixel keypoint wobble is about 5 mm of placement; tightening that is a sampling question, worked through in estimate edge position accuracy from image sampling.

Lock the light and the exposure together. Retail lighting that drifts in colour wobbles the mask and the keypoint confidences in the same direction, so treat it like any ambient light variation problem: fixed fixtures, a matte backdrop, an exposure that never renegotiates.

The capture camera then follows straight from the arithmetic:

The LS-CMR-0166 is a 12 MP USB3 camera on an IMX226 sensor with a 1/1.7" rolling-shutter format that reads 4000 × 3000 at 30 fps. Rolling shutter costs nothing while the shopper holds a pose, and a capture that lasts a second or two needs no faster readout.

Shoppers who keep moving during capture need the whole frame in one instant:

The MV-CH120-11UC is a 12 MP USB3 camera on an IMX304 sensor with a 1.1" format, a C-mount and a global shutter. A global shutter grabs all rows at once, so a swaying subject records without the skew a rolling scan paints across moving arms.

When the kiosk column has to hide every component, run the pipeline on the camera itself:

The LS-SMC-0017 is an 8.8 MP X86-platform camera with a 1" sensor, a C-mount and a global shutter. The X86 platform runs the same OpenCV pipeline on the device, so pose, mask and warp never leave the column — no PC to house, cool or cable.

Shortlist in the area scan cameras and smart cameras categories, narrow by pixels in the camera finder, or step through the camera configurator with the framing numbers above. More build notes sit on the applications topic page, and if the catalogue is large or the promise is fit-level sizing, contact us with the framing geometry before ordering.