Fisheye depth model domain shift: mix domains, build sparse depth GT

You trained a monocular depth model on RGB-D image pairs, and the RGB-D validation numbers are good. Point the same network at rectified fisheye frames and the predictions collapse: flat depth, wavy object boundaries, mean absolute error (MAE) several times worse. That is camera domain shift, not a broken loss or a bad rectification map, and it has a known fix. Train across both camera domains at the ratio you actually deploy, fine-tune gently from the pretrained weights, and manufacture fisheye depth labels by projecting a calibrated RGB-D or LiDAR point cloud into the fisheye image. This guide gives the tests that confirm the diagnosis, the mix arithmetic, the label pipeline step by step, and the two cases where you should stop training and change hardware instead.

Symptom: fine on RGB-D, collapsed on rectified fisheye

The signature is clean. RGB-D validation error holds at its usual level, so the network, the loss and the training loop all work. The same weights on rectified fisheye frames return structure that looks melted: surfaces merge, distant walls pull forward, and MAE rises by a large multiple.

One test separates domain shift from a deeper fault: run the original pretrained model, before your fine-tune, on the same fisheye frames. If it still returns roughly correct object structure, with worse absolute error, the fisheye domain is learnable and your fine-tune simply never saw it. If even the pretrained model produces garbage, suspect the rectification, the input format or the depth scale first, and unify those before touching training data.

Note what has not happened yet: a model trained only on RGB-D has seen zero fisheye pixels, so nothing has been forgotten. Forgetting becomes the risk only after you add fisheye data.

Causes to test, in order of probability

Four causes explain nearly every case. Work down the list; each test takes minutes.

1. No fisheye images in training. The network never saw one pixel of the target domain. The test is your dataset list: if the fisheye count is zero, this is the cause.
2. Preprocessing mismatch between the streams. Different rectification parameters, resolution, color space or depth units shift the input distribution on their own. Push one RGB-D training frame through the exact fisheye pipeline and compare, using the preprocessing-match checklist for confidence drops and the depth-scale check for close-range cutoffs.
3. Edge distortion the model never saw. Rectification puts the fisheye on pinhole geometry, but the periphery is resampled, interpolated and stretched, so its statistics differ from any true perspective image. Score MAE separately for a center crop and an outer band: a much worse outer band confirms it. This is the same edge effect that leaves an invalid band at the edges of stereo depth images, and it is why side-view perspective pictures, the tempting shortcut, fail to cover it.
4. Forgetting after the fix. Once fisheye data exists, a fisheye-only fine-tune erodes RGB-D performance. Keep both validation sets and evaluate both every epoch.

Fix 1: mix domains at the deployment ratio and fine-tune gently

Do not retrain from scratch, and do not fine-tune on fisheye alone. Start from the pretrained weights, keep the RGB-D pairs, and add fisheye images at the share the deployed system actually serves. Mixed training at the deployment ratio protects the domain you already work in while teaching the one you are adding. Unify distortion handling, resolution, color space and depth scale across both streams first, or the network learns your pipeline differences instead of depth.

Here N is the image count per domain in the training set and r is the fraction of inference frames each camera serves in production. Fine-tune at a small learning rate — AdamW with lr0 = 1e-4 is a proven starting point — and augment to match the real differences between the cameras: brightness, contrast, blur and mild perspective.

Worked example: sizing the fisheye share for an 80/20 cell

Inputs, illustrative for a typical rig of this kind: one overhead fisheye camera serves 80% of inference frames, a close-range RGB-D camera serves the other 20%, and you hold 12,000 labeled RGB-D pairs. Labeling a dense depth map by hand takes even an optimistic 10 minutes.

Rule: mix at the deployment ratio, N_fisheye = N_rgbd × r_fisheye / r_rgbd.

Arithmetic:

• N_fisheye = 12,000 × (0.80 / 0.20) = 12,000 × 4 = 48,000 fisheye images
• Hand-labeling cost: 48,000 maps × 10 min = 480,000 min ≈ 8,000 hours, about 200 working weeks

Decision: at a fisheye-to-RGB-D ratio of 4:1, dense hand labels are not a plan. Fill the 48,000-image fisheye share with sparse depth projected from the reference camera, keep all 12,000 RGB-D pairs, and fine-tune as in Fix 1, evaluating both validation sets every epoch. Set the acceptance target at fisheye MAE within about twice your RGB-D MAE. If the fisheye curve stalls above it, fix the label pipeline next; if RGB-D MAE climbs above its baseline, add RGB-D data until both curves hold.

Fix 2: project the reference point cloud into the fisheye image

This is how the fisheye share above gets labels without a labeling team. You need the fisheye camera, a reference depth source rigidly mounted beside it — a second RGB-D camera or a LiDAR — and the extrinsics between the two.

1. Calibrate the pair once. Estimate the fisheye intrinsics and the rigid transform from the reference camera to the fisheye camera; the two-depth-camera extrinsics calibration guide covers the procedure, and the same transform later lets you fuse both cameras' point clouds for other checks.
2. Capture synchronized pairs. For a static rig, any co-timed pair works. If the scene moves, fire both cameras in the same instant, or projected labels smear across surfaces.
3. Back-project the reference depth map to a point cloud in the reference camera's frame; the depth-image-to-point-cloud conversion notes cover the dtype and stride details.
4. Transform and project. Apply the extrinsics to bring the cloud into the fisheye frame, then project every point through the fisheye camera model — or the rectified pinhole model, if the network sees rectified frames — and write the point's depth at the pixel it lands on.
5. Mask invalid pixels to 0. A projected point whose depth disagrees with the nearest surface along the ray is an occlusion: write 0, not the false depth. Drop points whose reference-camera confidence flags are bad, decoded with the confidence bitflag guide, and carry over the depth-hole handling from the reference cloud. Zero is the conventional ignore label, so a depth loss that masks zeros trains on valid pixels only.
6. Check real frames before training. Inspect 20-50 projected-label frames against the fisheye images by eye. If the labels land on the right surfaces there, they land on the right surfaces wherever the geometry holds.

Expect sparsity, not density, from this pipeline. A one-megapixel reference depth map holds 1,228,800 points; with half falling inside the fisheye overlap and half of those surviving the masks, about 307,000 labeled pixels remain — roughly 15% of the 2,073,600 pixels in a 1,920 × 1,080 rectified frame. A masked depth loss trains well on that. A second low-effort source of pairs: apply the fisheye distortion offline to your existing RGB-D images and depths, but tune that synthesis against the real projected labels above, because synthetic pairs alone do not reproduce the nonlinear distortion at the frame edge.

When a one-shot depth calibration is enough

Some inference stacks expose a per-camera depth calibration — a scale, sometimes an offset, applied to predicted depth. It is the right tool in exactly one case. Fit a single scale factor between predicted and reference depth on a labeled fisheye set; if that one multiply drops MAE close to your RGB-D baseline, structure was intact and only absolute depth drifted, so a calibration is enough.

If structure itself is wrong — surfaces merged, geometry warped — no global scale recovers it, and the only fix is mixed-domain training with real fisheye labels. Decide with the center-crop and outer-band test from the causes list: a calibration can help the center, and it will never rescue the outer band.

When the right answer is hardware, not training

Step back to why the fisheye is there. In most of these projects the RGB-D camera's field of view simply does not cover the scene, and the fisheye plus a retrained network is a work-around for that gap. If a 3D sensor with a wide native view covers the same area, depth arrives measured instead of inferred, and the domain-shift problem disappears with the fisheye.

For a fixed, close-range cell, a projected structured-light 3D camera gives full-frame depth directly; for a scene that moves or lighting you cannot control, a ToF sensing camera measures depth per frame. Size the coverage against mounting position with the 3D camera working distance and positioning guide, then shortlist in the camera finder.

If the fisheye stays in the cell, the mix-and-project pipeline above is the plan instead.

Prevention checklist

Run this list on every new camera domain before the first training run.

Data

• Both domains present at the deployment ratio, with the counts written down
• Two validation sets, one per camera, frozen before training
• Distortion handling, resolution, color space and depth scale identical across streams

Training

• Fine-tune from pretrained weights for a domain add, never from scratch
• Small learning rate from the pretrained start; cut it in half if validation oscillates
• Augmentation limited to real camera differences: brightness, contrast, blur, mild perspective

Labels

• Projected sparse depth with occlusion and confidence masking to 0
• The hand-checked frame set from the projection steps signed off first

Deployment

• Per-domain MAE logged in production, never one blended number
• A quarterly scale-fit test: a drifting scale factor is a calibration task, a drifting structure error is a retraining task

Where to go next

If the fisheye stays, the plan is set: unify preprocessing, mix at the deployment ratio, fine-tune at a low learning rate, and label the fisheye domain with projected sparse depth. If field of view is the real constraint, cover the scene with a sensor that sees it natively:

More depth-image and point-cloud techniques are collected on the 3D topic page. To spec the capture rig — fisheye, reference depth camera, mounting and trigger — start the configurator at the camera step, or send the layout through the contact page if you want the mix ratio and the sensor choice checked before you order.