Hardware

MXFP4 quantization is a microscaling, 4-bit floating-point compression scheme designed to shrink the memory footprint of deep-learning models without hurting accuracy.

You run a model in INT8 and it is 30% faster than FP32. You run the same model in INT8 with a different framework — same hardware, same batch size — and it is 2% faster. You run it again with a different calibration dataset and the accuracy drops by 4 points. Quantization is not one thing.

Your quantized model is listed at 4 GB on the model card. You load it onto an NVIDIA Jetson Orin with 8 GB of unified memory. It OOMs at 3.2 GB loaded. You reduce batch size to 1. Still OOM. You switch to INT4 weights. Finally it runs — but the throughput is half of what you expected.