Synthetic training data: does photorealism help your model?

Synthetic images promise unlimited labeled training data for pose estimation and segmentation, and the first question every team asks is how photorealistic the renders must be. The evidence points one way: randomized lighting, texture and pose close the sim-to-real gap more reliably than render fidelity, and photorealism pays only for texture-bound tasks such as defect segmentation. This comparison is for vision engineers deciding where to spend effort between a high-fidelity renderer and a disciplined randomization pipeline. You will get the published findings, a six-axis randomization recipe, a worked camera-matching example, and a per-task decision table.

Does photorealism make synthetic training data work better on the real line?

On its own, no: published comparisons since 2016 report that models trained on moderately realistic renders score about the same on real images as models trained on real photo sets, and the gains come from randomized lighting, texture and pose rather than render fidelity.

The clearest published test is from 2016: an object detector trained on rendered images scored on par with the same detector trained on real photographs, and both beat a detector trained on a small, narrow real set. Later transfer work reached the same verdict from the other side: randomizing lighting, texture and pose reduces or removes the need for large hand-labeled real sets and slow photorealistic rendering.

The reason sits in how networks read images. A render that is too perfect forms a narrow distribution: one noise signature, permanently clean edges, every highlight in the same place. The model memorizes the renderer's fingerprint, then meets a camera that has never produced it. Baked shadows become shortcuts; when the line's light moves a few degrees, the shortcut breaks.

So treat fidelity as a budget you spend on named gaps, not a baseline. Work from 2015 already showed effective training scenes can be synthesized around a small set of real images rather than a full renderer. The systems-side reading that pairs with this decision lives on the systems topic page.

When does a high-fidelity render pay for itself?

Pay for real render quality only when the feature your model must detect is itself a fine-image property: surface scratches, print defects, dent shading, texture direction; for pose, presence and coarse segmentation, low-fidelity geometry with wide randomization transfers as well and renders faster.

Texture is the dividing line. Scratches, dents, print faults, surface finish and grain are image properties in their own right: if the render cannot produce them, no number of extra images will teach the cue. For presence checks, pose and coarse segmentation, the cue is geometry, and geometry survives a flat-shaded render untouched.

Hybrid pipelines close even that texture gap without a full photorealistic pass: they map real surface textures onto synthetic geometry and report zero-shot part inspection, where the model trains without real images of the part at all. That result matters twice, because photoreal frames also render slowly, so every fidelity point you avoid is generation throughput you keep.

The practical test: name the visual cue your model must fire on. If the cue survives a diagram, randomize. If the cue is the texture, buy realism or hybridize.

What should you randomize in a synthetic training set?

Randomize at least these six axes across the envelope your station can physically produce: light direction, light intensity, exposure, pose, texture and background clutter; a rule of thumb from the transfer studies is to span slightly beyond that envelope, because production drift always exceeds your worst planned day.

Work axis by axis, and let the station decide the width. In an enclosed, light-tight cell the envelope is narrow, so randomize narrowly; on an open cell fed by ambient light, the envelope widens and the renderer must cover it. Typical starting ranges:

• Light direction: sweep azimuth and elevation over the whole hemisphere the geometry exposes.
• Light intensity: 70–130% of nominal.
• Exposure and gain: the shutter range the line actually runs, for example 0.1–10 ms if that is the station's span.
• Part pose: full yaw, tilt to ±20° unless the fixture physically fixes the part.
• Texture and finish: every material and wear variant you ship.
• Background: neighboring parts, conveyor texture, drop shadows.

Calibrate the ranges against reality rather than guessing: measure what the cell drifts in production, and read how ambient light variation in pick and place vision is controlled before you set the envelope. Rendering exposure variants from a small real set extends the same idea to captured images, as synthetic exposure augmentation for realistic camera images shows.

How do you match the render to your real camera?

Match four things before you render a single training image: image scale, resolution, noise and exposure response; a 12 MP camera that sees 0.05 mm per pixel needs synthetic images at the same 0.05 mm per pixel, not a 1920-pixel render stretched to size.

Fix the optics first, because image scale is a property of the lens and sensor, not of the renderer. For a required field of view and working distance:

Worked example: the station needs a 200 mm field at 250 mm working distance on a 1/1.8-inch sensor, about 7.2 mm wide. Magnification m = 7.2 / 200 = 0.036, so f = 250 × 0.036 / (1 + 0.036) = 8.69 mm, which means an 8.5–9 mm lens. A 4000-pixel-wide sensor then resolves 200 / 4000 = 0.05 mm per pixel, so the renderer must output 0.05 mm per pixel: a 40 mm part spans 800 pixels in training and on the line alike. A 1920-pixel render over the same field gives 200 / 1920 = 0.104 mm per pixel, and upscaling cannot restore the missing detail.

Then match sensor behavior. Render brightness response across the real shutter and gain span, and add defocus if the line stops down between f/4 and f/8, because depth of field moves with f-number. An all-sharp render trains the model on a focus cue the optics never produce.

Capture the real reference and validation set with the camera family you will deploy — area scan cameras span every interface and resolution class:

The MV-CU020-90GC is a compact 2.5 MP GigE camera. The LS-CMR-0166 records 4000 × 3000 over USB3 at 30 fps. The 20 MP MV-CS200-10UM covers fine-defect scale on a 1-inch mono sensor. Render to the pixel scale of whichever one ships.

How much real data do you need alongside the synthetic set?

Budget a real validation set of 200 to 500 images, because at 90% accuracy that size puts the 95% confidence interval near ±3 points, small enough to see whether a randomization change actually moved the sim-to-real gap.

The interval comes from ordinary binomial statistics:

At p = 0.90 and n = 400: √(0.09 / 400) = 0.015, times 1.96 = ±2.9 points. Halve the set to 100 images and the band widens to ±5.9 points, too coarse to tell whether a randomization change helped. Set the size by the decision you need to make, not by render convenience.

Use the same real set to anchor preprocessing: verify that color statistics are normalized identically for synthetic and real inputs before judging the gap, as the color normalization training pipeline article lays out. The real captures also carry the sensor noise, flare and mounting artifacts the renderer can only approximate.

Which data recipe fits which inspection task?

Match the data recipe to the failure mode: pose and presence tasks take low-fidelity geometry with wide randomization; defect and texture tasks take hybrid data, real textures mapped onto synthetic geometry; fully controlled enclosed stations narrow the randomization and make mid-fidelity renders the faster, correct choice.

Presence checks earn their accuracy from negatives, empty scenes, neighboring parts and shadow shapes, which is exactly what background images in object detection training covers. Code reading randomizes the same axes you fight on the line: rotation, blur and glare, as 1D barcode preprocessing for rotation, blur and glare details.

Training for on-device deployment changes nothing in the recipe, only where the model runs. A compact smart camera such as the MV-SC5060XM-08M-WBN trains offline and serves the line over GigE:

The MV-SC5060XM-08M-WBN pairs a 6 MP mono sensor with an integrated 8 mm M12 lens. Randomize around that fixed field, since the lens will not change between training and production. Per case: an enclosed cell with one part family takes narrow randomization at mid fidelity; an open picking cell with varied parts takes wide randomization before any fidelity spend; textured-defect work takes the hybrid route first.

How do you verify the sim-to-real gap actually closed?

Score the model on real images it has never seen, and treat the gap as closed only when real accuracy is within 2 points of renderer accuracy; a model that scores 95 points on synthetic tests and 78 on the line still has a randomization problem, not a rendering problem.

Run these checks in order, and stop at the first failure because each isolates a different cause:

1. Real hold-out. Capture 200–500 images across at least a week of shifts, parts and lighting the model never trained on.
2. Renderer hold-out. Score on synthetic scenes from an unused seed; if renderer-only scores also dropped, the training run is at fault, not the transfer.
3. Statistics check. Compare exposure histograms and color means between the synthetic and real sets; a mismatch here predicts a mismatch in predictions.
4. Artifact probe. Perturb at test time, with light angle, slight blur and added clutter; a model leaning on renderer fingerprints collapses under small perturbations.
5. Post-deployment drift. Keep reviewing live images after launch: pose estimation data drift: find images for review shows how to find which captures deserve a human look.

If the gap stays open after these checks, widen the randomization before you raise fidelity; the transfer studies point the same way from both directions. When the parts list settles, the area scan camera finder narrows the reference camera, the camera configurator steps through the chain, smart cameras cover the on-device route, and [/quote/] handles a station-level parts list.