LLM memory: 1-bit vs FP16 vs INT8
How quantized weights affect footprint—and what still uses RAM during inference
When people compare 1-bit LLMs to FP16 baselines, they usually care about how many gigabytes a model needs on disk and in VRAM. BitNet targets extreme compression of weights; runtime memory also includes activations, KV cache, and framework overhead.
Order-of-magnitude weight storage
For a rough intuition only (actual packing and metadata vary):
- FP16: 2 bytes per parameter → higher baseline VRAM/RAM for the weight tensor.
- INT8: ~1 byte per weight → often ~2× smaller than FP16 weights.
- 4-bit (GPTQ/AWQ-style): often ~0.5 bytes effective with grouping—not identical to BitNet’s layout.
- 1-bit / BitNet-style: dramatically fewer bits per weight for storage; marketing often cites large multiples vs FP16 for weights—always profile your build.
Why your GPU still fills up
Long contexts allocate large KV caches. Embedding tables may remain higher precision. Batch size multiplies activation memory. Use benchmarks on your checkpoint and context length instead of relying on weight math alone.
Choosing a stack
If your goal is minimum hardware for acceptable quality, compare BitNet against INT8/4-bit in BitNet vs other methods. If your goal is maximum accuracy at fixed hardware, larger models in higher precision may win despite size.