LLM Quantization State of the Art: Comparing AWQ, GPTQ, GGUF, and ExLlamaV2
The Necessity of Post-Training Quantization
When a 70-billion parameter model is trained, its weights are stored in 16-bit floating point precision (FP16 or BF16). Serving a 70B model in uncompressed FP16 requires 140 Gigabytes of VRAM for weights alone, plus tens of gigabytes of additional memory for KV caches and activations. This mandates an expensive multi-GPU setup (e.g., 2x A100 80GB or 4x A6000).
Post-Training Quantization (PTQ) compresses weights from 16 bits down to 4 bits or 8 bits: - Reduces Memory Footprint by 70% to 75%: A 70B model shrinks from 140GB down to ~38GB, fitting comfortably onto a single consumer RTX 3090/4090 or single A100. - Dramatically Increases Throughput: Because LLM inference is memory-bandwidth bound, reading 4-bit weights transfers 4x fewer bytes over the memory bus, directly increasing generation speeds.
However, naive round-to-nearest (RTN) quantization causes severe perplexity degradation. The machine learning community has developed four dominant advanced quantization frameworks: AWQ, GPTQ, GGUF, and ExLlamaV2.
The Core Algorithms Explained
``` 1. GPTQ (Frantar et al.): Second-Order Hessian Inversion ──> Quantizes weights column-by-column, compensating errors in remaining weights.
2. AWQ (Lin et al.): Observes Activation Outliers (top 1% salient weights) ──> Protects salient channels from aggressive clipping.
3. GGUF / llama.cpp (Gerganov): K-Quants block quantization ──> Unified binary format optimized for CPU, Apple Silicon, and mixed GPU offloading.
4. ExLlamaV2 (turboderp): Custom FP4 / EXL2 variable bit-rate kernels ──> Tuned for maximum decode tokens/sec on NVIDIA consumer GPUs. ```
#### 1. GPTQ (Generalized Post-Training Quantization) GPTQ is based on second-order Taylor expansion approximations. It processes weight matrices column by column, utilizing the inverse Hessian matrix ($H^{-1}$) to dynamically adjust unquantized weights to compensate for the rounding error introduced by each quantized weight. - Strengths: Fast calibration (quantizes a 70B model in ~30 minutes); supported across vLLM and Hugging Face. - Weaknesses: Treats all weights relatively equally, occasionally clipping critical outlier channels that govern reasoning.
#### 2. AWQ (Activation-aware Weight Quantization) Developed by MIT researchers, AWQ recognized a crucial insight: not all weights are equally important. By observing activation distributions during a calibration pass, AWQ discovered that protecting just 1% of salient weights from quantization error eliminates almost all perplexity degradation. - AWQ applies per-channel scaling to protect salient weights before quantizing the remainder to 4 bits. - Strengths: Superior perplexity retention on math, coding, and reasoning; excellent integration in high-throughput enterprise servers like vLLM.
#### 3. GGUF (K-Quants / llama.cpp) Pioneered by Georgi Gerganov for llama.cpp and Ollama, GGUF is a self-contained single-file format. It introduces K-Quants (e.g., Q4_K_M, Q5_K_S), which dynamically assign different bit-depths across model layers (e.g., using 6-bit for attention and 4-bit for feed-forward layers). - Strengths: Runs on CPUs, Apple Silicon Metal, and supports partial GPU layer offloading when VRAM is insufficient.
#### 4. ExLlamaV2 (EXL2)
ExLlamaV2 was engineered from scratch by developer turboderp for pure raw speed on modern NVIDIA GPUs. It features hand-tuned CUDA assembly kernels that bypass standard PyTorch runtime overhead and supports non-integer bitrates (e.g., 3.2-bit or 4.65-bit quantization).
Benchmark Comparison Across 70B Models
Comprehensive evaluation on Llama 3 70B on an NVIDIA RTX 4090 / A100 SXM5:
| Format / Kernel | Typical Bit Depth | Memory Footprint (Weights) | WikiText-2 Perplexity | Decode Speed (RTX 4090) | Target Environment | | :--- | :---: | :---: | :---: | :---: | :---: | | Uncompressed FP16 | 16-bit | 140.0 GB | 5.68 (Baseline) | N/A (OOM) | Enterprise Multi-GPU | | AWQ (4-bit GEMM) | 4.0-bit | 38.5 GB | 5.74 (+0.06) | 38 tokens/sec | Production Enterprise (vLLM) | | GPTQ (4-bit ActOrder) | 4.0-bit | 38.5 GB | 5.81 (+0.13) | 35 tokens/sec | General Python / Hugging Face | | ExLlamaV2 (4.0 bpw) | 4.0-bit | 38.0 GB | 5.76 (+0.08) | 54 tokens/sec | Single-User Local Inference | | GGUF (Q4_K_M) | 4.5-bit | 41.2 GB | 5.75 (+0.07) | 32 tokens/sec | Desktop / Mac / Ollama |
Architectural Decision Guide
- For Enterprise Cloud Serving (vLLM / Triton): Choose AWQ. It delivers the cleanest balance of low perplexity degradation and massive concurrency support across batched users.
- For Maximum Single-User GPU Speed (RTX 3090/4090): Choose ExLlamaV2 (EXL2) for unprecedented decode throughput.
- For Cross-Platform Local Desktop (Mac / Windows): Choose GGUF via Ollama for seamless memory offloading.
To explore how quantized local runtimes operate, review our guide on The Local LLM Revolution: Ollama and llama.cpp.