6D pose dataset annotation mismatch: objects outside the view

A 6D pose dataset in the BOP scene-wise layout stores ground truth in two files per scene: scene_gt.json holds one pose entry per object placement, and scene_gt_info.json holds one bounding-box record per object that made it into the image. When the two lists have different lengths, most dataloaders either crash on the index error or silently train on misaligned annotations. This explainer gives you the cause behind almost every case, the projection that proves it, and a preprocessing pass that reconciles the counts before training. It is written for anyone building 6D pose or bin-picking models on synthetic or mixed datasets.

Why the pose file lists objects the box file does not

The extra entries are objects outside the camera field of view. A synthetic renderer places every object in the scene and writes a pose for each placement, whether or not the camera can see it. The bounding-box file is different by construction: a box only means something for geometry that projects into the frame, so fully out-of-frame objects get no record. The count difference is exactly the number of objects that landed outside the image.

That is what one public train_pbr split shipped: an image in scene 000000 carried 88 pose entries and 87 bounding-box entries, and the missing one was the last placement, sitting outside the field of view. The maintainers removed that object from the pose file and the mask folder when they regenerated the data, which is the same action your preprocessing should take. The dataset was not corrupt; the two files answer different questions.

So the decision up front: the box file is right about what is trainable in the image, the pose file is right about what was placed in the scene, and training wants the intersection, every pose whose object leaves a footprint in the pixels.

How a pose becomes a bounding box

Each pose entry stores a rotation and a translation that map the object's model coordinates into the camera frame. The intrinsics matrix K turns camera-frame points into pixels:

Project the model's 3D bounding-box corners, eight points are enough for this check, then take the minimum and maximum of u and v: that is the axis-aligned box the info file should carry. Clip the box to the image border and you have the visible box. An object earns an entry when at least one projected point lands inside the image; when every point falls outside, or the object sits behind the camera plane where the projection is undefined, the footprint is zero and the entry disappears.

The lens decides how much of the scene reaches the sensor: a longer focal length narrows the field of view and pushes more placements out of frame, a shorter one pulls them in. Before you render or annotate, calculate the camera angle of view from calibration and leave coverage margins at the frame edges, the same margin discipline you apply to a calibration target. Coverage narrows further in depth sensing, where stereo overlap shrinks the usable field of view, and placement itself starts from 3D camera working distance and positioning.

Worked example: 88 poses, 87 boxes

Take a camera with a 1-inch sensor, 5472 × 3648 pixels at 2.4 µm. The sensor is 5472 × 0.0024 = 13.1 mm wide and 3648 × 0.0024 = 8.8 mm tall. With a 16 mm lens the principal point sits near cx = 2736, cy = 1824, fx = 16 / 0.0024 ≈ 6667 pixels, and the horizontal half-angle is atan(6.57 / 16) = 22.3°. At a 600 mm working distance the frame spans 13.1 × 600 / 16 = 491 mm of width, so the right-hand image edge corresponds to about +246 mm in x at that depth.

Object A is 150 mm wide, centered at x = +500 mm, y = +50 mm, z = 600 mm. Its projected center sits 16 × 500 / 600 = 13.3 mm from the principal point, which is 13.3 / 0.0024 = 5555 pixels: u = 2736 + 5555 = 8291, past the 5472-pixel edge. The half-extent projects to 16 × 75 / 600 = 2.0 mm = 833 pixels, so the footprint runs from u = 7458 to u = 9124. Not one pixel lands on the sensor, while v = 1824 + 556 = 2380 sits comfortably in frame: the exclusion is purely horizontal. Zero footprint means no box and no info entry, and this is the object behind the 88-against-87 count.

Object B, the same size at x = +250 mm, projects its center to 16 × 250 / 600 = 6.67 mm = 2778 pixels from the principal point, so u = 2736 + 2778 = 5514 with the footprint spanning 4681 to 6347. That straddles the right edge, and close to half the object's width is visible: keep the entry and let the visible box clip at column 5472. Borderline objects are the hardest training cases; fully invisible ones are noise.

The rule the arithmetic leaves you with: drop only objects whose entire projected footprint clears the image, with a small margin, and treat anything within about 5 pixels of the border as still visible.

The validation pass that reconciles the counts

Run this once over every scene before the first training epoch:

1. For each image, compare the entry counts of scene_gt.json and scene_gt_info.json. Log the image id and both counts wherever they differ.
2. For each mismatched image, project all eight corners of every posed object through K · [R | t], using the intrinsics the dataset ships.
3. Drop a pose when every corner clears the image rectangle by more than 5 pixels, or when the object has Z ≤ 0. Delete the matching mask in the same pass so indices stay aligned.
4. Re-count. Every image must now agree. If one still does not, the files were edited by hand or truncated, so rebuild both from the rendered masks instead of deleting more entries.
5. Assert the invariant inside the dataloader: equal counts, every box non-empty after clipping. A future regeneration of the dataset then fails loudly instead of training on misaligned data.

What else produces a count mismatch

The archive case is real, not theoretical: one split was updated in place, and downloads that overlapped the update produced conflicting files until they were fetched again. A geometry mismatch and a broken download never look alike once you read the counts.

When dropping entries is the wrong fix

Three boundaries. Published benchmark test splits: run the count check and log the result, but leave the files untouched, because scores are only comparable on annotations everyone shares; report the mismatch to the maintainer. Partially visible objects: never drop them, even when a sliver shows, since the visible box is the supervision your model needs for cluttered bins. Someone else's training set you must use today: work on a copy, keep the original checksums, and record every entry you remove so the run is reproducible.

The same geometry resurfaces at run time when the part overfills the sensor; template matching a part larger than the field of view covers that failure and its remedies.

Real images to anchor a synthetic split

Photorealistic renders transfer well, but a model destined for a production cell still needs real images with real poses, captured with the camera and light you will ship. Structured-light and stereo sensors deliver depth and pose ground truth in one capture, and 3D sensors cover the full range:

The annotated 2D image itself comes from an area scan camera such as the MV-CS200-10UM, a 20 MP USB3 model on a 1" sensor:

The intrinsics that validate the dataset are the same ones that guide the robot at run time, and choosing PnP intrinsics for rectified-image pose estimation picks them apart. Most consumers of these datasets are picking cells: see 3D bin picking for flat parts and bin picking non-planar parts with 3D vision for the cell-side decisions.

Checklist before you train

• Entry counts equal in every image, asserted in the dataloader.
• Every box non-empty and clipped to the image rectangle.
• Out-of-frame poses dropped together with their masks, logged by image id.
• Originals kept with checksums; edits made on a copy.
• Published test splits untouched; mismatches reported, not rewritten.
• Archive extracts cleanly; ERROR: Headers Error : means a truncated download, not an annotation problem.

If a count still disagrees after the projection pass, send the scene id and both counts through the contact page and we will work it through with you. To move from dataset to hardware, the camera step of the configurator walks sensor, mount and interface, and the camera finder lists every model it can use.