YOLO training runs but no h5 file is created: checkpoint timing

You retrained a YOLOv3 detector on your own annotated inspection images, the console scrolls through epochs, and logs/000/ still holds a single file: events.out.tfevents.<timestamp>.<hostname>. No ep003-...h5, no trained_weights_stage_1.h5, no trained_weights_final.h5. That events file is written by the TensorBoard callback seconds after the script starts, before the first epoch ends, so it proves the run began and nothing more. The .h5 files appear only when three conditions line up: a save epoch, an improvement in val_loss, and a validation loss that exists and is finite. Work through the causes below in order; each has a console sign, a test and a fix.

What the script writes, and when

The retraining script produces two files unconditionally and one family of files conditionally. Stage 1 (epochs=50, initial_epoch=0) ends with a model.save_weights call that writes trained_weights_stage_1.h5 whether or not any checkpoint fired. Stage 2 (epochs=100, initial_epoch=50) ends the same way, writing trained_weights_final.h5 even when EarlyStopping cut the fit short. The ep-numbered files come only from ModelCheckpoint, configured with monitor='val_loss', save_weights_only=True, save_best_only=True and period=3, writing into the template ep{epoch:03d}-loss{loss:.3f}-val_loss{val_loss:.3f}.h5. Stage 2 also attaches ReduceLROnPlateau (factor=0.1, patience=3) and EarlyStopping (patience=10).

That splits the symptom cleanly. If the console has returned to the prompt and neither unconditional file exists, stage 1 never finished: the process hit a traceback or was stopped mid-fit. If training is still running, missing ep-numbered files may simply mean no save epoch has arrived yet. Read the last console lines before anything else; they usually settle it at a glance.

The three gates every checkpoint passes

1. The epoch is a multiple of period=3. The callback evaluates at epochs 3, 6, 9 and so on. The filename number is the finished epoch counted from one, so ep003 is the earliest file that can ever appear.
2. val_loss improved. With save_best_only=True, the write happens only when the current val_loss is lower than the best value seen at a previous evaluation. That tracked best updates only at evaluations, not every epoch, so expect gaps between files.
3. val_loss exists and is finite. If validation never ran, Keras prints a warning that best-only saving needs val_loss and skips the write. If val_loss is NaN, the lower-than comparison is false, and the write is skipped again.

First suspect: no save epoch has arrived yet

Most reports end here: the run is simply young. Do the arithmetic for your dataset before suspecting the callback. Take 480 annotation lines in 2007_train.txt, with val_split = 0.1 and batch_size = 32:

Stage 1 therefore runs 50 epochs × 13 steps = 650 steps before trained_weights_stage_1.h5 is written unconditionally. The first checkpoint evaluation lands after epoch 3 — 3 × 13 = 39 steps — and writes only if val_loss has improved by then. Stage 2 continues the numbering from initial_epoch=50, so its files read ep051, ep054 and so on, and each unfrozen step costs more compute than a frozen one. With a few hundred images, an epoch or two of patience is often the entire fix; watch the progress bar's epoch counter, not the folder.

Empty validation split: the skip warning in your console

num_val = int(len(lines) × val_split) decides whether validation runs at all. int() truncates: with val_split = 0.1, nine lines or fewer give num_val = 0 (int(9 × 0.1) = 0), and ten lines give 1. When num_val is 0, the validation generator returns None, no val_loss key enters the logs, and gate 3 blocks every write for the rest of the run. The console shows it plainly: each third epoch repeats the warning that best-only saving needs val_loss, while training otherwise looks healthy.

Count the lines in 2007_train.txt before anything else. Ten validation images technically unblock the callback, but a val_loss computed on a single batch is mostly noise, and improvements become rare. Hundreds of images are the realistic floor for a detector that must hold tolerances at the line; how many images you need to train a custom YOLO dataset gives per-class counts to aim for. Annotation faults live next door: empty boxes reaching preprocess_true_boxes produce the empty YOLO anchor dataset error, which has its own distinct fix.

NaN val_loss never counts as an improvement

When val_loss: nan appears in the progress bar, gate 2 silently blocks every write: NaN compared against any best value fails the lower-than test, so save_best_only never triggers and no file appears at any evaluation. In small custom datasets the usual source is divergence. Stage 1 trains at Adam(lr=1e-3) with batch_size = 32, which is aggressive for a few hundred images; drop stage 1 to Adam(lr=1e-4) and rerun. Also check the annotations for empty or degenerate boxes, which push losses out of range during preprocess_true_boxes. Judge progress by validation metrics rather than the training loss; why a good val mAP can sit next to a high YOLO loss explains what those two numbers do and do not tell you.

The run died mid-fit: nothing saves during an epoch

No file is written while an epoch is in progress, so a traceback that ends the process leaves exactly your symptom: one events file, zero weights. The two killers, in order of how often they turn up:

• Out of GPU memory. batch_size = 32 at (416, 416) input is heavy, and stage 2 unfreezes every layer, raising memory demand further. Drop batch_size to 16 or 8; steps per epoch double (432 ÷ 16 = 27) and training still converges, just slower.
• A bad line in 2007_train.txt. One path pointing at a moved or unreadable image throws an exception mid-batch. Loop over the file and open every listed image before training; it takes a minute and removes this class of failure entirely.

EarlyStopping is not the culprit here. With patience=10 it only ends the fit_generator call; the save_weights line after it still runs, so stage 2 writes trained_weights_final.h5 even when early stopping fires. A missing final file means the process never got that far. The same early-stopping mechanics appear in early stopping during anomaly-detection training, and a weights file that exists but refuses to load has its own truncated-file checks.

A two-minute test that proves writing works

Separate a callback problem from an environment problem with one short run: set period=1 and save_best_only=False, then train two or three epochs. You should get an ep001 file after the first epoch and one per epoch after. If even this writes nothing, the training logic is innocent — check that the drive holding logs/000/ is not full and that the process has write permission there. Restore period=3 and save_best_only=True for the real run; best-only saving is what keeps the strongest weights instead of merely the newest.

Prevention, deployment, and when to change stacks

Prevention starts on the capture side: training images should come from the same area scan cameras, lens and lighting the cell will use, so runtime statistics match the training set.

The first card above is a 6 MP mono model on USB3 with a C-mount, sized for everyday inspection frames. Where small defects need more pixels, the LS-CMR-0166 captures 4000 × 3000 frames at 30 fps over USB3.

Once trained_weights_final.h5 exists, move inference next to the line. A vision controller loads the weights and runs the detector inside the inspection program, so results reach the PLC over I/O instead of a training workstation.

The MV-VC3301H-128G66 includes a vision software license and 4 built-in light channels, so the same box drives the lights and runs inference. If a deployed model disappears after a controller restart, follow the missing vision models recovery checklist. And if the inspection is simple enough, smart cameras with built-in tools skip custom training entirely: no dataset, no checkpoint logic to debug.

Two notes if you are starting fresh rather than finishing an old script. Maintained YOLOv3 implementations checkpoint every epoch out of the box, which removes this failure mode at the source — read the training script of the port you adopt and confirm where it saves weights. And if you port the old script to current Keras, expect breakage before the first epoch: fit_generator no longer exists (Model.fit accepts generators), the period argument is replaced by a save-frequency setting, and HDF5 weight saving is treated as deprecated in the current Keras documentation. The decision: squeeze the current script dry with the checks above, and start new detection projects on a stack that saves each epoch.

Checks before you restart training

1. Count the lines in 2007_train.txt and confirm int(lines × 0.1) ≥ 1, ideally a few dozen per class.
2. Run two epochs with period=1 and save_best_only=False to prove the write path.
3. Watch the progress bar for val_loss: nan; drop stage 1 to lr 1e-4 if it shows up.
4. Confirm free disk space and write permission on log_dir.
5. Read the last console lines for a traceback before calling a run clean.
6. Give each run its own log_dir so checkpoints never mix.

Sizing the capture side for a new dataset? The camera finder and the camera configurator narrow models by resolution and interface, the systems topic page collects the neighbouring retraining checks, and the team can review a parts list before you order — contact us.