INT4_CONVROT_SR (INT4 with Hadamard Weight Rotation and Stochastic Rounding) is an advanced 4-bit integer quantization format designed to run large diffusion and transformer models on consumer GPUs while minimizing the quality loss typically associated with 4-bit precision.

Target GPUs (Hardware Compatibility)

• Supported Architectures: NVIDIA SM 8.0 and newer.

  • RTX 3000 Series (Ampere): RTX 3060, 3070, 3080, 3090, A100.

  • RTX 4000 Series (Ada Lovelace): RTX 4060, 4070, 4080, 4090, H100.

  • RTX 5000 Series & Data Center (Blackwell): Full hardware support for high-throughput INT4 Tensor Core operations.

• Note: Older architectures (GTX 10-series, RTX 20-series/Turing) either lack native INT4 Tensor Core acceleration or experience dequantization overhead that negates speed gains.

Performance & Generation Speed Benefits

• Massive VRAM Reduction: Shrinks model size by roughly ~50% compared to FP8 (and ~75% compared to FP16/BF16). This allows 12B–24B parameter models to fit entirely into 8GB–16GB VRAM without system RAM offloading/paging.

• High-Throughput INT4 Tensor Cores: Leverages native integer GEMM kernels on supported GPUs, which execute operations with significantly higher theoretical throughput than floating-point math.

• Memory Bandwidth Bottleneck Relief: Diffusion generation is largely memory-bandwidth bound. Streaming 4-bit weights across the GPU bus cuts the required memory traffic per sampling step in half compared to FP8, directly improving iteration speed (it/s) on cards with tighter memory buses.

Why "CONVROT_SR"?

• ConvRot (Hadamard Rotation): Rotates weight matrices in feature space to eliminate extreme outliers before clipping, preserving high dynamic range and output visual fidelity.

• SR (Stochastic Rounding): Prevents systematic quantization bias during rounding, maintaining cleaner fine details and prompt adherence compared to standard nearest-neighbor INT4 quantization.