Anyone who has browsed a model library in Ollama or LM Studio has seen labels like Q4, Q5, or Q8 attached to model names. These refer to quantization, the single technique most responsible for making it possible to run capable AI models on an ordinary laptop instead of specialized data-center hardware.
What Quantization Actually Does
A neural network’s knowledge is stored as billions of numerical parameters, or weights. In their original, uncompressed form, these weights are typically stored using 16 or 32 bits of precision per number, the kind of precision used during training on powerful data-center hardware. Quantization is the process of reducing the precision used to store those weights — representing them with fewer bits, such as 8, 5, or 4 — which shrinks the model’s total file size and memory footprint roughly in proportion to how much precision is removed.
The key insight behind quantization is that neural network weights don’t need extreme precision to still produce useful output. Rounding each weight to a coarser value loses some information, but for most models and most tasks, that loss is small enough that the model’s actual output quality barely changes, while the practical benefits — smaller files, less memory used, faster inference — are substantial.
Reading Quantization Labels
The Q-numbers you see in model names refer to bits per weight, with some variation in exact method. Q8 uses 8 bits per weight and stays very close to the original full-precision model’s quality, at roughly half the uncompressed size. Q4 uses about 4 bits per weight, cutting size roughly to a quarter of the original, with a slightly more noticeable but usually still acceptable quality trade-off for most everyday tasks. Q4 has become something of a default sweet spot for general use, which is why it’s so often the pre-selected recommendation in tools like Ollama.
Why This Matters for Your Hardware
Quantization directly determines whether a given model fits in your available memory, which is the single biggest factor covered in our guide to what hardware you need to run AI models locally. A 7-billion parameter model at full 16-bit precision needs roughly 14GB of memory just to load — out of reach for most laptops. The same model at Q4 quantization needs closer to 4GB, comfortably fitting on a typical consumer machine with RAM to spare for everything else running at the same time. Without quantization, running even modest-sized models locally simply wouldn’t be practical for most people.
The Quality Trade-Off in Practice
Quantization isn’t free — it does introduce some degradation, and the effect becomes more noticeable at the most aggressive compression levels (below Q4). For everyday use, most people genuinely cannot tell the difference between a full-precision model and its Q8 or Q5 quantized version in casual conversation. At Q4, the difference becomes slightly more detectable on tasks demanding precise reasoning or exact factual recall, though it’s still usually acceptable for general use. Below Q4, quality degradation becomes more apparent, which is why heavily compressed models are typically reserved for situations where memory is extremely constrained and some quality loss is an acceptable trade-off for being able to run the model at all.
Choosing the Right Quantization Level
As a practical rule, Q4 is a reasonable default for most consumer hardware and most everyday tasks. If you have memory to spare and want the best possible quality from a given model size, moving up to Q5 or Q8 is worthwhile. If you’re working with especially constrained hardware and need to fit a larger model than your memory would otherwise allow, dropping to a more aggressive quantization lets you trade some quality for the ability to run a bigger, more capable model at all — sometimes a worthwhile trade, since a larger model at heavier quantization can still outperform a smaller model at full precision.
Quantization Is Why Local AI Works at All
It’s worth appreciating that quantization is a large part of what made the current wave of accessible local AI possible in the first place. Without it, the models now easily downloadable through Ollama or LM Studio would require hardware far beyond what any typical consumer owns. Understanding the basic trade-off — less precision for dramatically lower resource requirements — makes it much easier to pick the right model variant for your specific machine rather than guessing.