Fine-tune a YOLO pose model on your custom keypoints

A pose model that reports a strong box mAP next to a near-zero pose mAP is not usually a data problem: the keypoint branch of the network started from random weights, because the checkpoint loaded into it was a detector, not a pose model. This guide shows how to fine-tune a YOLO pose network on your own keypoint set so the pose head inherits trained weights instead. You will see which checkpoint to load, how to declare kpt_shape, which augmentation to switch off, how many epochs to budget, and which log number tells the truth. It is written for machine-vision engineers adding keypoint-based inspection or robot guidance — gripper corners, hole centers, joint angles — to a camera system, and it closes with where such a model runs in production.

Why box metrics pass while pose metrics fail

Loading weights into a freshly built network transfers only the tensors whose layer name and shape match the architecture you built. A detector checkpoint such as yolo12n.pt contains no keypoint branch, so if you build yolo12n-pose.yaml and load the detector file into it, the backbone, the neck and the box and class branches of the head inherit trained weights — and the whole keypoint branch starts from noise. Nothing in the console flags it: the run starts cleanly, and the box numbers look pretrained from the first validation.

That is the same transfer logic that decides what carries over and what breaks when you reuse YOLO weights: matching names and shapes move, everything else resets. Even a genuine pose checkpoint transfers only what matches. If kpt_shape declares a different keypoint count than the checkpoint was trained with, or two values per point where it expects three, the final keypoint layer reshapes and starts fresh, while the convolutions below it keep their training.

The decision follows from the mechanics: start every custom fine-tune from the -pose.pt checkpoint for your model size, never from the plain detector file, and declare kpt_shape to match your labels exactly.

What you need

Everything here is software plus one GPU, and the label format causes most of the failures. Start there: each labeled instance carries a bounding box plus K points, with the coordinates normalized to the image:

Mark clean, visible points with visibility 2 and occluded or out-of-frame points with 0. If your annotations live in a spreadsheet, converting CSV annotations to YOLO format covers the file layout and the index errors that come with it.

Then gather:

• A data YAML declaring the train and validation image paths and kpt_shape: [K, 3] — K is your keypoint count, 3 means x, y and visibility. Keep three values per point even when nothing is ever occluded: COCO-pretrained pose checkpoints are three-valued, and matching the layout keeps the transfer path alive.
• The -pose.pt checkpoint for your model size, from the same source you train with — not a detector.pt file.
• A GPU with 8 GB of memory or more for a nano-size model at 640 pixels, batch 8-16.
• Enough labeled instances. A transferred pose head makes a few hundred labeled instances a workable start, while from-scratch pose training needs several thousand; how many images a custom YOLO dataset really needs goes deeper on the counting.

Fine-tune in eight steps

1. Declare the skeleton. Write the data YAML with kpt_shape: [K, 3] and your K. Every label file must carry exactly K points per instance.
2. Check the labels, not the YAML. Open five label files and confirm that point 0 is the same physical feature in every one, through point K-1. Ten minutes here prevents the most expensive failure in this guide.
3. Load the pose checkpoint. Start from the -pose.pt file and read the transfer report the loader prints — it states how many weight tensors moved across. If the count suggests only the backbone moved, stop and check which file you actually loaded.
4. Overfit-test on 20 images. Train a short run on a tiny slice at 640 pixels. By roughly 100 epochs the losses should collapse toward zero and the pose mAP on that same slice should approach 1. A model that cannot overfit 20 images has a label or ordering defect, not a hyperparameter problem.
5. Set the flip for your skeleton. Turn the horizontal-flip augmentation off for asymmetric parts and for skeletons whose left and right points are not mirrored in index order; keep it on for genuinely symmetric layouts.
6. Train the real run at 640 pixels. Budget 100-300 epochs. Raise the image size only for small keypoints, and price it honestly: doubling the side length multiplies memory and training time roughly by four.
7. Watch pose_loss and Pose mAP50-95. The box numbers will look good from the first validation; the keypoint numbers are the ones your data is training.
8. Stop on the plateau. When pose_loss has stopped falling for 20-30 epochs, take the checkpoint with the best Pose mAP50-95 and stop.

Pre-flight list before the long run: kpt_shape matches the label files; visibility is three-valued; the checkpoint is a -pose.pt file; the transfer report shows head tensors moved; the flip setting matches your symmetry; the overfit test passed.

Worked example: a detector checkpoint in a pose network

Build yolo12n-pose.yaml, load yolo12n.pt into it, and train on COCO keypoints — the configuration this guide warns about — and the first validation pass reads like this (epoch 1 of a 200-epoch schedule, 640-pixel images):

Two bits of arithmetic pin the diagnosis. The pose score sits at 0.146 / 0.82 = 0.178, about 18% of the box score. The keypoint loss opens at 8.14 / 0.9935 = 8.2 times the box loss — the signature of an untrained regression branch beside a transferred one.

None of this says the dataset is bad. It says the keypoint branch has no trained parents, and that epoch 1 of 200 is far too early to judge a pose run anyway. Restart from a -pose.pt checkpoint and the same plot opens with pose_loss well below 8 and a nonzero Pose mAP50 after the first validation, because the keypoint convolutions now inherit.

Which numbers to trust while the run progresses

Five loss columns scroll past: box_loss, pose_loss, kobj_loss, cls_loss, dfl_loss. Box_loss and dfl_loss regress the box, cls_loss scores the class, pose_loss regresses your keypoint coordinates, and kobj_loss learns whether a keypoint is present at all. Only the last two train on what makes your part special.

Judge the run on the Pose metrics, not the Box metrics. Pose mAP50-95 is the headline number, Pose mAP50 is the one to compare against a pass/fail threshold, and the precision and recall columns set that threshold per part. High loss values beside healthy validation scores are usually harmless; the val-mAP-versus-loss explainer covers when to worry and when to let the run continue.

One inference-time trap belongs here: if your pipeline crops the part before pose inference, keypoints can fall outside the crop and vanish from the output. Getting all 17 keypoints from a cropped person image handles that case at prediction time.

Common mistakes that sink a custom keypoint run

Ordered by how often they bite:

• Judging at epoch 1. The table above is a healthy log for a broken start. Read metrics once the loss has clearly begun to fall.
• Detector checkpoint into a pose YAML. The silent one: the file loads without error and the defect only shows as a stuck pose_loss. Real load refusals are a different family — truncated downloads, format mismatches and Darknet-era files dropped into a modern build — and Darknet custom-weights troubleshooting guide lists those classics, as does the checklist for a truncated weights file.
• kpt_shape with two values per point against a three-valued checkpoint: the whole keypoint branch reshapes and the head start disappears. Keep three.
• Flip augmentation on an asymmetric skeleton. Symptom: predictions swap left and right under mirroring. Fix: flip off, retrain.
• Inconsistent keypoint index order across annotators. Only the overfit test in step 4 catches this; nothing else will.
• Training on crops, deploying on full frames. Keypoints are normalized to the trained frame, so a framing change silently moves every label. Train on the framing you will run.

Where the fine-tuned model runs

Training is a desk job; inference happens beside the line, and the host decides the export path. Three placements cover most inspection cells.

In the camera. When the model must live in the camera, an X86-platform smart camera is the fit: the LS-SMC-0017 pairs that platform with a global shutter and a C-mount, so the network you trained runs on board and the lens stays your choice. The smart cameras range covers smaller and larger sensors for the same job.

In a vision controller. One model serving several cameras belongs in a controller. The MV-VC5525-256G70-NN pairs a 12th-gen i5 with 32 GB of memory, PoE camera ports and 8 built-in channels — acquisition and inference in one box; browse the vision controllers for lighter builds.

On an AI inference terminal. Multi-camera pose guidance with tight cycle times belongs on a dedicated inference terminal: the MV-VAC5724-256G02 is a dual-card terminal with PoE camera ports and 8 built-in channels:

Whichever hosts it, export the trained weights instead of running the training framework live. On an embedded accelerator the export is the usual failure point: this TensorRT export troubleshooting guide for YOLO models covers the export and performance causes. To trigger the inspection from your line controller, see starting an inspection automatically from a custom program.

When the standard advice is wrong

Three deviations are worth knowing:

• Symmetric skeletons keep the flip. If the part is genuinely mirror-symmetric and the index order mirrors with it, flipping is free augmentation, and turning it off slows convergence for no gain. The rule targets asymmetry, not flips in general.
• Map your skeleton onto COCO slots where the correspondence is real. Two shoulders and a wrist are COCO points: define them in the COCO slot order and the keypoint convolutions inherit their trained meaning. Seventeen bolt holes are not a skeleton — do not force a mapping; with any keypoint count other than 17, only the final keypoint layer reshapes and the rest still transfers.
• From-scratch training rarely wins. Even on imagery far from natural photographs, the pretrained backbone's edge and texture filters transfer. One from-scratch run is worth trying only with tens of thousands of labeled frames and weeks of patience; expect the fine-tune to match it.

Once the model validates, the hardware decides whether those keypoints survive production lighting and cycle times. Size the camera, lens and light around the points you chose to track, or bring a skeleton and a cycle time to the contact page and talk it through.