The benchmark that lied for two weeks

Our throughput numbers were about 30% better than they had any right to be. Nothing in the metrics said why.

The tell

Throughput was flat with respect to batch composition. Shuffling differently changed nothing. Turning augmentation off changed nothing. A number that refuses to move when you change its inputs is not a good number, it is a constant.

The cause

The dataloader was caching decoded batches, and a change to the sampler meant it was handing back the same cached batch every step. The model was training on one batch, very fast, and reporting excellent throughput for doing so.

Loss still went down, which is what made it survive two weeks. It was memorising a single batch, and the training curve for memorising a single batch looks encouraging right up until you evaluate.

What I check now

1. Hash the first tensor of each batch for the first fifty steps and assert the hashes are distinct.
2. Plot throughput against batch size early. If it is flat, something is not doing the work you think it is.
3. Evaluate on held out data within the first few hundred steps, not at the end. A silent data bug shows up there and nowhere else.

Takeaway

A suspiciously good number deserves the same scrutiny as a bad one. In practice it gets far less, because nobody debugs good news.