Quantization

The AI efficiency technique that cuts compute costs by up to 90% — often with minimal loss in quality.

Quantization has become one of the most commercially important techniques in enterprise AI, enabling organisations to run large language models and other AI systems at a fraction of the original cost. With the global AI inference market projected to exceed $75 billion by 2028, quantization techniques that reduce compute requirements by 50-90% represent savings of tens of billions of dollars across the industry. For C-suite leaders, quantization is the difference between AI that is a research expense and AI that is a scalable, profitable business capability.

What It Is

Quantization is a technique for reducing the precision of the numbers used inside an AI model, thereby reducing the model's size, memory requirements, and computational cost. AI models are essentially vast collections of numbers (called parameters or weights) that determine how the model processes inputs. These numbers are typically stored in high-precision formats — 32-bit floating point numbers (FP32) — which can represent values with extraordinary precision but require significant memory and compute resources. Quantization reduces these numbers to lower-precision formats — 16-bit (FP16), 8-bit (INT8), 4-bit (INT4), or even 2-bit representations — dramatically reducing the model's footprint.

Think of it like image compression. A RAW photograph file contains every pixel of data captured by the camera sensor — perhaps 80 megabytes of information. A JPEG version of the same image looks nearly identical to the human eye but occupies only 5 megabytes, a 94% reduction. Quantization does something similar for AI models: it removes numerical precision that has negligible impact on the model's output quality but significant impact on its computational requirements. A large language model that requires 80 GB of memory in full precision (FP32) might fit in 20 GB at 8-bit quantization, or even 10 GB at 4-bit quantization — enabling it to run on much less expensive hardware.

The key insight is that AI models are remarkably tolerant of reduced precision. Research has consistently shown that reducing from 32-bit to 16-bit precision has virtually zero impact on model accuracy. Reducing to 8-bit typically degrades accuracy by less than 1%, and even aggressive 4-bit quantization can maintain 95-98% of the original model's performance with careful technique. The latest methods — including GPTQ, AWQ (Activation-aware Weight Quantization), and GGUF formats — have made it possible to quantise models with sophisticated techniques that protect the most important parameters whilst aggressively compressing less critical ones.

Why It Matters for Business

The cost savings from quantization are transformative. Running an unquantised 70-billion parameter model like Meta's Llama 3 70B requires approximately 140 GB of GPU memory — typically four to eight high-end NVIDIA H100 GPUs costing $3-6 per hour in the cloud, or $120,000-250,000 in hardware for on-premises deployment. A 4-bit quantised version of the same model requires approximately 35 GB — fitting on a single GPU and costing $0.75-2 per hour in the cloud. For an organisation running this model to serve 1 million queries per day, quantization can reduce inference costs from approximately $150,000 per month to $30,000-50,000 per month — annual savings of $1.2-1.5 million for a single model.

Quantization also democratises AI by enabling powerful models to run on accessible hardware. A quantised version of an open-source large language model can run on a consumer-grade laptop or a single mid-range GPU, enabling use cases that would otherwise require expensive cloud infrastructure. This is particularly impactful for organisations with data sovereignty requirements that mandate on-premises processing, or for operations in regions where cloud GPU availability is limited. It also enables edge deployment — running AI models directly on mobile devices, IoT hardware, or vehicles — where power consumption and thermal constraints make full-precision models impractical.

The environmental benefits are significant. Quantized models consume 50-90% less energy per inference compared to full-precision models. For organisations tracking Scope 3 emissions (which include the energy consumed by cloud services), quantization directly reduces the carbon footprint of AI operations. Microsoft reported that applying quantization across its Azure AI services reduced total GPU energy consumption by 35%, avoiding an estimated 50,000 metric tonnes of CO2 emissions annually.

Enterprise Use Cases

Meta deploys quantized versions of its Llama models across Facebook, Instagram, and WhatsApp, serving over 4 billion users with AI features including content recommendations, spam detection, and automated content moderation. By using 8-bit quantization for most inference workloads, Meta reduced its AI inference infrastructure costs by approximately 40% compared to full-precision deployment — saving hundreds of millions of dollars annually at its scale. The company also open-sources its quantized model weights, enabling the broader community to benefit from the efficiency gains.

Samsung Electronics integrated quantized AI models directly into its Galaxy smartphone processors, enabling on-device AI features including real-time language translation, photo editing, and text summarisation without sending data to the cloud. The quantized models run entirely on the device's neural processing unit, providing instant responses, preserving user privacy, and functioning without internet connectivity. This approach has been cited as a key competitive differentiator for Samsung's flagship devices. In the automotive industry, Tesla uses quantized neural networks for its autonomous driving system, enabling the vehicle's onboard computer to process data from eight cameras simultaneously in real time. Full-precision models would be too computationally demanding for the vehicle's power and thermal budget; 8-bit quantization provides the necessary performance within hardware constraints.

In financial services, Bloomberg uses quantized models for its Terminal product, running financial analysis AI directly on traders' desktop machines rather than in the cloud, addressing both latency concerns and data security requirements.

Vendor Landscape

Quantization is primarily implemented through open-source tools and frameworks rather than commercial products. Key tools include llama.cpp (which pioneered practical 4-bit quantization for large language models), AutoGPTQ, AutoAWQ, and NVIDIA TensorRT-LLM (which provides hardware-optimised quantization for NVIDIA GPUs). Cloud providers offer quantized model deployment: AWS Bedrock, Azure AI, and Google Vertex AI all support deploying quantized versions of popular models with automatic optimisation. Hardware vendors are increasingly building quantization support directly into their chips: NVIDIA's Hopper and Blackwell architectures include native support for FP8 (8-bit floating point) and INT8 operations, delivering 2-3x throughput improvements for quantized workloads. Qualcomm's AI Engine supports INT4 quantization for mobile and edge deployments. Commercial MLOps platforms including Weights & Biases, Modal, and Together AI offer quantization as part of their model optimisation pipelines.

Strategic Considerations

Quantization should be a default consideration for any production AI deployment — there are very few scenarios where full-precision inference is justified by the marginal quality improvement. Evaluate quantization on a continuum: start with FP16 (which has effectively no quality loss and 50% size reduction), then test INT8 (minimal quality loss, 75% size reduction), and only push to INT4 if the quality trade-off is acceptable for your specific use case. Implement rigorous quality benchmarks before and after quantization — what matters is not the abstract accuracy metric but whether the quantized model meets your specific business requirements. For customer-facing applications, consider A/B testing quantized and full-precision models with real users to assess practical quality impact. Remember that quantization is complementary to other efficiency techniques: combining quantization with techniques like model distillation, pruning, and caching can deliver cumulative savings of 90% or more. Finally, ensure your MLOps pipeline includes automated quantization testing so that new model versions are automatically evaluated at multiple precision levels.