Models

Poor Man's Compute: 8GB VRAM Guide

Back to Tutorials

The 8GB VRAM Challenge

I get this question constantly: "Can I run modern LLMs on my 8GB graphics card?" The answer is yes, but you have to be clever about it. I spent the weekend testing various combinations of models and inferencing engines on a humble RTX 3060 Ti, and here is exactly how I maximized performance without hitting out-of-memory (OOM) errors.

Step 1: Quantization is Mandatory

You simply cannot run fp16 weights on 8GB. I used GGUF format with 4-bit quantization. A typical 7B to 8B parameter model in 4-bit takes roughly 4.5GB to 5GB of memory. This leaves around 3GB for the context window (KV Cache) and OS overhead.

Choosing the Right Format

I stick with Q4_K_M or Q5_K_M formats from Hugging Face. They offer the best balance of speed, low memory usage, and minimal perplexity degradation.

Step 2: Managing the KV Cache

The biggest hidden memory hog is the KV (Key-Value) cache, which grows linearly with your context size. If you try to feed a 32K token document into a model on 8GB VRAM, the cache alone will crash your GPU. I managed this by severely limiting my context window in my server config.

# Example command for llama.cpp, forcing a strict context window
./server -m models/mistral-7b-v0.1.Q4_K_M.gguf -c 4096 --n-gpu-layers 35

By keeping the context to 4096 tokens, I kept the KV cache footprint under 1.5GB, safely fitting inside the remaining VRAM.

Step 3: Flash Attention

If you are using a Python-based framework instead of llama.cpp, make sure Flash Attention is enabled. I noticed a massive drop in VRAM usage during generation when I explicitly imported and enabled it in Transformers.

import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "model_path",
    torch_dtype=torch.float16,
    device_map="auto",
    attn_implementation="flash_attention_2"
)

The Verdict

I successfully ran Llama 3 (8B) and Mistral (7B) on 8GB of VRAM at speeds exceeding 40 tokens per second. It requires discipline with your context length and strict adherence to 4-bit quantization, but it works perfectly for coding assistants and quick chat tasks.