Ashir Rashid

Ashir Rashid

ML Engineering & Applied AI Research

Mixed-Precision Quantization

The problem

Most quantization tools apply the same bit-width to every layer. Fine for small networks. With LLMs and large DNNs it breaks down. Sensitive layers (attention heads, first and last layers) don’t tolerate aggressive compression well. Others barely notice dropping from FP32 to INT4. Uniform precision is either too conservative or too aggressive, and there’s no obvious manual path through that tradeoff at scale.

The goal was automating per-layer bit-width selection: let the search find good configurations instead of guessing or falling back to uniform settings.

Uniform precision

Every layer gets the same bit-width, sensitive or not.
first
INT8
INT8
attention
INT8
INT8
hidden
INT8
INT8
hidden
INT8
INT8
last
INT8
INT8

Mixed precision

Sensitive layers kept higher, tolerant layers dropped to INT4.
first
FP32
FP32
attention
FP32
FP32
hidden
INT4
INT4
hidden
INT4
INT4
last
FP32
FP32
FP32, sensitive layers kept high INT8 INT4, tolerant layers compressed

How it works

01
Mixed-precision search
A hardware-aware RL policy reads per-layer sensitivity metrics and assigns a bit-width per layer, rewarded on measured hardware latency and memory rather than FLOP-count proxies, so it learns what actually runs fast on real chips. Evolutionary search runs alongside to cover configurations the RL agent misses.
02
ANT for DNNs
Adaptive numerical data types match each layer's actual weight and activation distribution rather than a fixed grid. Custom CUDA kernels implement this at inference time.
03
OliVe for LLMs
Activation outliers are paired with an adjacent normal value and encoded locally, so INT8 neither clips them nor wastes precision everywhere else. No global coordination needed.
04
MLflow tracking
Every experiment logs accuracy, energy, and latency through an MLflow pipeline. Running a new configuration means queuing a run, not rebuilding a spreadsheet.

Results

+15%
Model performance over baseline quantization configurations
+8.3%
Energy and memory efficiency
43%
Reduction in hyperparameter search time

Key design decisions

ANT

DNNs

Addresses the weight distribution problem in CNNs and ViTs.

  • Adaptive numerical data types matched per layer to the actual distribution.
  • Custom CUDA kernels implement the encoding at inference time.

OliVe

LLMs

Addresses the activation outlier problem specific to attention.

  • Outlier-victim pairing for activation outliers, encoded locally at the kernel level.
  • No global coordination controller across the accelerator, unlike prior methods that separate outliers globally and pay a coordination-hardware overhead.
DNNs on ImageNet
VGG16 ResNet18 ResNet50 Inception-V3 ViT
LLMs
BERT GPT-2 OPT Bloom
Research work at eBrain Lab, NYU Abu Dhabi, extending ANT (MICRO'22) and OliVe (ISCA'23).

Technologies:
Python PyTorch CUDA MLflow AWQ GPTQ SmoothQuant

Concepts / Algorithms:
Mixed-Precision Quantization Reinforcement Learning Evolutionary Search Post-Training Quantization LLM Inference Optimization Outlier-Victim Pair Encoding