Models

Llama 3.1 405B Local Execution Guide

Back to Tutorials

Running Llama 3.1 405B Locally

I recently tested running the enormous Llama 3.1 405B model on a local setup. It's a behemoth. You can't just load this into a standard RTX 4090 and call it a day. I had to use extreme quantization and distributed inference, but I managed to get it running using Ollama and LM Studio. Here is exactly how I did it.

The Hardware Reality

The 405B model in fp16 takes over 800GB of VRAM. That's impossible for a home lab. I went with a 4-bit quantized GGUF, which brings the memory requirement down to around 230GB. I achieved this by pooling resources across multiple machines or using a Mac Studio with 192GB unified memory and heavy swapping.

Setting Up with Ollama

Ollama makes fetching and running the model straightforward, provided your system doesn't immediately OOM (Out of Memory). I used the following command to pull the quantized version:

ollama run llama3.1:405b-instruct-q4_K_M

If you have multiple GPUs, Ollama handles the split automatically. I monitored the distribution using nvidia-smi and noticed it evenly distributed the layers across the available VRAM.

Using LM Studio for UI and API

For a better interface and API compatibility, I switched to LM Studio. Here's my workflow:

  1. Download LM Studio and navigate to the search tab.
  2. Search for Llama-3.1-405B-Instruct-GGUF. I grabbed the Q3_K_M or Q4_K_M version from a reliable quantization creator like Bartowski or TheBloke.
  3. In the Model Settings, I maxed out the GPU Offload layers (usually 128 layers for this architecture) depending on my VRAM.

Performance Tweaks

To prevent crashing, I had to limit the context window. Running a 128K context on a 405B model is a recipe for disaster on limited memory.

{
  "num_ctx": 8192,
  "temperature": 0.7
}

By restricting the context to 8K, I maintained a stable 5-10 tokens per second on my setup. It's not blazing fast, but having GPT-4 class intelligence running completely offline is worth the tradeoff.