Models

Local LLM Deployment: Running Llama 3.1 8B on 8GB VRAM

Back to Tutorials

1. Introduction: The Democratization of AI

A few years ago, running a highly capable AI model required access to vast server farms or cloud computing services costing hundreds of dollars a month. Today, the landscape has radically shifted. The release of Meta's Llama 3.1 has set a new benchmark for open-source AI, offering performance that rivals proprietary models from OpenAI and Anthropic.

However, the real revolution isn't just in the model's intelligence—it's in its accessibility. The 8 Billion parameter (8B) version of Llama 3.1 is specifically designed to run on consumer hardware. This means complete privacy, zero subscription fees, and offline capabilities. In this highly technical tutorial, we will cover exactly how to deploy Llama 3.1 8B locally on an 8GB VRAM graphics card (such as an NVIDIA RTX 3060, 4060, or Apple Silicon Mac) without encountering Out of Memory (OOM) errors.

2. The 8GB VRAM Wall and How We Beat It

In the world of Large Language Models, VRAM (Video RAM) is your most precious resource. A model's memory footprint is determined by its parameter count and precision. By default, an 8B parameter model loaded in 16-bit precision (FP16) requires roughly 16GB of VRAM just to load the weights. If you try to run this on an 8GB card, your system will either crash or offload to system RAM, which slows inference down from 50 tokens per second to a crawl of 2 tokens per second.

So, how do we fit a 16GB model into 8GB of VRAM? The answer is Quantization. Quantization is a compression technique that reduces the precision of the model's weights from 16-bit floats to 4-bit or 8-bit integers. We will be using the GGUF (GPT-Generated Unified Format) standard, which is heavily optimized for both CPU and GPU execution. A 4-bit quantized version of Llama 3.1 8B (often denoted as Q4_K_M) takes up roughly 4.7GB of VRAM. This leaves about 3.3GB of VRAM for the Context Window (the memory used to store your ongoing conversation).

3. Step 1: Installing Ollama

Ollama is the easiest and most efficient way to run open-source models locally. It abstracts away the complexities of compiling llama.cpp and automatically handles GPU offloading.

For MacOS and Windows:

Navigate to ollama.com/download and download the installer for your operating system. Run the installer and follow the on-screen instructions. Ollama will run in the background as a system tray application.

For Linux:

Open your terminal and run the official installation script:

curl -fsSL https://ollama.com/install.sh | sh

Once installed, you can verify that the Ollama service is running by typing ollama serve or simply running ollama -v to check the version.

4. Step 2: Pulling the Llama 3.1 8B Instruct Model

Ollama hosts a vast library of pre-quantized models. To pull the latest Llama 3.1 8B Instruct model, simply open your command prompt or terminal and type:

ollama run llama3.1

By default, this command downloads the 4-bit quantized (Q4_0) version, which is perfectly suited for our 8GB VRAM constraint. The download size is approximately 4.7 GB. Depending on your internet connection, this may take a few minutes. Once the download completes, you will be dropped into an interactive terminal session where you can start chatting with Llama 3.1 directly. Type /bye to exit this session, as we are going to configure a much better user interface.

5. Step 3: Optimizing the Context Window for 8GB VRAM

Llama 3.1 boasts a massive 128,000 token maximum context window. However, expanding the context window requires significant VRAM. If you try to feed Llama 3.1 a 128k context on an 8GB GPU, it will immediately run out of memory. For 8GB VRAM, the sweet spot for the context window is usually between 8,192 and 16,384 tokens.

To enforce a strict memory limit, we will create a custom Modelfile. A Modelfile in Ollama is similar to a Dockerfile; it defines the model parameters, system prompts, and configuration.

Create a new text file named Modelfile in an empty directory. Add the following content:

FROM llama3.1
# Set context window to 8192 tokens to prevent OOM on 8GB VRAM
PARAMETER num_ctx 8192
# Optional: Set a custom system prompt
SYSTEM "You are an expert software engineer and technical assistant. Always provide concise, accurate, and highly technical answers."

Now, compile this custom model by running the following command in the same directory as your Modelfile:

ollama create my-llama3.1 -f Modelfile

You have now created a custom, memory-safe model named my-llama3.1 that is guaranteed not to exceed your VRAM limits.

6. Step 4: Setting up Open WebUI via Docker

While interacting with the model via terminal is fun, a graphical interface provides a much better experience. Open WebUI (formerly Ollama WebUI) is a beautiful, feature-rich interface that looks and feels exactly like ChatGPT, but runs entirely on your local network.

To install Open WebUI, you will need Docker installed on your machine. Once Docker is running, execute the following command to pull and run the Open WebUI container. We will bind it to your local network and connect it to the Ollama API.

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

Let's break down this command:

  • -p 3000:8080 maps the container's internal port to port 3000 on your local machine.
  • --add-host=host.docker.internal:host-gateway allows the Docker container to communicate with the Ollama service running on your host machine.
  • -v open-webui:/app/backend/data creates a persistent volume so your chat history is saved even if the container restarts.

Open your web browser and navigate to http://localhost:3000. Create an admin account (this data never leaves your computer). Once logged in, select your newly created my-llama3.1 model from the dropdown menu at the top. You can now chat with your local AI, organize conversations into folders, and even upload documents for local RAG (Retrieval-Augmented Generation) processing!

7. Troubleshooting and Performance Tuning

If you notice your generation speed is very slow (less than 10 tokens per second), it means Ollama is offloading layers to your CPU instead of your GPU. Here are a few troubleshooting steps:

  • Check VRAM Usage: On Windows, open Task Manager and check your Dedicated GPU Memory. On Linux, run watch nvidia-smi. If your VRAM is completely full, reduce the num_ctx parameter in your Modelfile to 4096.
  • Update Drivers: Ensure you are running the latest NVIDIA Game Ready or Studio drivers. Outdated drivers can prevent Ollama from properly utilizing CUDA cores.
  • Close Background Apps: Browsers (especially Chrome) and Discord can consume 1-2GB of VRAM in the background due to hardware acceleration. Close them to free up memory before running heavy inference tasks.

8. Conclusion

Congratulations! You have successfully deployed a state-of-the-art Large Language Model completely locally. By leveraging 4-bit quantization and strictly managing your context window, you've turned an 8GB VRAM consumer graphics card into a powerful, private AI workstation.

Running models locally is the ultimate safeguard for corporate data privacy and intellectual property. As open-source models continue to improve, the gap between cloud APIs and local deployment will only shrink further. Happy coding!