Quantization, and where the accuracy actually goes

Almost every quantization paper reports average degradation across a benchmark suite. Almost none report which layers absorbed it. The second number is the one that predicts whether the method will work on your model.

The averaging problem

A method that loses half a point on average can be losing nothing on most layers and a great deal on two of them. If those two happen to be load bearing for your task, the average told you nothing useful.

Placeholder numbers, but the shape is the point: a three percent slice carries most of the loss.

What follows from it

Mixed precision assignment beats uniform precision at equal average bit width, and it is not close. Keeping a small number of sensitive layers at higher precision costs very little memory and recovers most of the gap.

What I still do not know

Whether sensitivity can be predicted from the weight distribution alone, before running any calibration. If it can, the assignment step becomes free. If it cannot, every new model needs its own sweep and the method is less practical than it appears.