OCOPD What You Are Building
A local Retrieval-Augmented Generation, or RAG, chatbot answers questions using documents stored on your own computer. Instead of sending your meeting notes, policies, manuals, or PDFs to a hosted AI service, this application performs the document search and language-model generation locally.
The finished pipeline is:
Documents -> extracted text -> chunks -> embeddings -> FAISS index
|
Question -> question embedding -> nearest chunks -> local LLM -> answerThe chatbot does not train a model on your documents. It creates searchable vector representations of document chunks, retrieves the chunks most similar to a question, and places those chunks in the prompt sent to a local model.
This distinction matters. RAG is useful when the source material changes frequently or is too specific for a general-purpose model. It is not a guarantee that every answer is correct. If retrieval returns the wrong passages, the language model can still produce a confident but unsupported answer.
This tutorial creates a small, understandable application rather than a production platform. It supports PDF, Markdown, and plain-text files, stores the index locally, shows source filenames and PDF page numbers, and refuses to answer when the supplied context is insufficient.
Prerequisites
You need:
- Python 3.10 or newer.
- A local installation of Ollama.
- At least one model already available to Ollama. The supplied example uses
phi; use another locally installed model by changing the environment variable. - Enough disk space for the embedding model and your local language model.
- A folder containing PDFs,
.txtfiles, or Markdown files.
The application uses these components:
pypdfextracts text from text-based PDFs.sentence-transformersconverts document chunks and questions into vectors.faiss-cpuperforms nearest-neighbor vector search.requestscalls Ollama's local HTTP endpoint.streamlitprovides the browser interface.
Scanned PDFs require OCR before this application can search them. A PDF that visually contains text may still have no extractable text layer. OCR is intentionally outside this minimal example because OCR setup differs across operating systems and often requires an additional system executable.
Create a project and virtual environment:
mkdir local-rag-chatbot
cd local-rag-chatbot
python -m venv .venvActivate the environment on macOS or Linux:
source .venv/bin/activateOn Windows PowerShell:
.venv\Scripts\Activate.ps1Create requirements.txt:
faiss-cpu
pypdf
requests
sentence-transformers
streamlitInstall the dependencies:
python -m pip install -r requirements.txtStart Ollama separately and make sure the model you select is available locally. The exact model command depends on the model you installed. The code below defaults to phi, matching the local-model example used in the supplied implementation context, but you can override it with OLLAMA_MODEL.
Project Layout
Use this layout:
local-rag-chatbot/
├── app.py
├── requirements.txt
└── documents/
├── meeting-notes.pdf
├── handbook.md
└── decisions.txtPut only the files you want the chatbot to search in documents/. The application rebuilds the index when it starts, which is simple and reliable for a small personal collection. For a large collection, persist the index and add an incremental ingestion process instead of embedding every file on every launch.
Full Implementation
Create app.py with the following code:
import os
from pathlib import Path
from typing import Any
import faiss
import numpy as np
import requests
import streamlit as st
from pypdf import PdfReader
from sentence_transformers import SentenceTransformer
DOCUMENTS_DIR = Path("documents")
EMBEDDING_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
OLLAMA_URL = "http://localhost:11434/api/generate"
OLLAMA_MODEL = os.getenv("OLLAMA_MODEL", "phi")
TOP_K = 4
CHUNK_SIZE = 900
CHUNK_OVERLAP = 150
def read_documents(directory: Path) -> list[dict[str, Any]]:
records: list[dict[str, Any]] = []
for path in sorted(directory.rglob("*")):
if not path.is_file():
continue
suffix = path.suffix.lower()
if suffix == ".pdf":
reader = PdfReader(str(path))
for page_number, page in enumerate(reader.pages, start=1):
text = page.extract_text() or ""
if text.strip():
records.append({
"text": text,
"source": path.name,
"page": page_number,
})
elif suffix in {".txt", ".md"}:
text = path.read_text(encoding="utf-8", errors="ignore")
if text.strip():
records.append({
"text": text,
"source": path.name,
"page": None,
})
return records
def split_text(text: str, size: int = CHUNK_SIZE,
overlap: int = CHUNK_OVERLAP) -> list[str]:
cleaned = " ".join(text.split())
if not cleaned:
return []
chunks: list[str] = []
start = 0
while start < len(cleaned):
end = min(start + size, len(cleaned))
chunks.append(cleaned[start:end])
if end == len(cleaned):
break
start = end - overlap
return chunks
def build_chunks(directory: Path) -> list[dict[str, Any]]:
chunks: list[dict[str, Any]] = []
for record in read_documents(directory):
for text in split_text(record["text"]):
chunks.append({
"text": text,
"source": record["source"],
"page": record["page"],
})
return chunks
@st.cache_resource(show_spinner="Loading embedding model...")
def load_embedder() -> SentenceTransformer:
return SentenceTransformer(EMBEDDING_MODEL)
@st.cache_resource(show_spinner="Building document index...")
def build_index(directory_string: str):
directory = Path(directory_string)
chunks = build_chunks(directory)
if not chunks:
return None, []
embedder = load_embedder()
vectors = embedder.encode(
[chunk["text"] for chunk in chunks],
convert_to_numpy=True,
normalize_embeddings=True,
).astype("float32")
index = faiss.IndexFlatIP(vectors.shape[1])
index.add(vectors)
return index, chunks
def retrieve(question: str, index, chunks: list[dict[str, Any]],
number: int = TOP_K) -> list[dict[str, Any]]:
embedder = load_embedder()
query_vector = embedder.encode(
[question],
convert_to_numpy=True,
normalize_embeddings=True,
).astype("float32")
scores, positions = index.search(query_vector, min(number, len(chunks)))
results: list[dict[str, Any]] = []
for score, position in zip(scores[0], positions[0]):
if position >= 0:
result = dict(chunks[position])
result["score"] = float(score)
results.append(result)
return results
def format_context(results: list[dict[str, Any]]) -> str:
sections = []
for number, result in enumerate(results, start=1):
location = result["source"]
if result["page"] is not None:
location += f", page {result['page']}"
sections.append(
f"[Source {number}: {location}]\n{result['text']}"
)
return "\n\n".join(sections)
def ask_ollama(question: str, context: str) -> str:
prompt = f"""You answer questions using only the provided document excerpts.
Rules:
- If the excerpts do not contain enough information, say that you do not know.
- Do not invent names, dates, numbers, or decisions.
- Keep the answer concise but include useful detail.
- Cite supporting excerpts using labels such as [Source 1].
Document excerpts:
{context}
Question:
{question}
Answer:
"""
response = requests.post(
OLLAMA_URL,
json={
"model": OLLAMA_MODEL,
"prompt": prompt,
"stream": False,
},
timeout=180,
)
response.raise_for_status()
data = response.json()
return data["response"].strip()
def source_label(result: dict[str, Any]) -> str:
if result["page"] is None:
return result["source"]
return f"{result['source']} (page {result['page']})"
st.set_page_config(page_title="Local RAG Chatbot", page_icon="📚")
st.title("Local document chatbot")
st.caption(f"Model: {OLLAMA_MODEL} | Documents stay in this application")
if not DOCUMENTS_DIR.exists():
DOCUMENTS_DIR.mkdir(parents=True)
index, chunks = build_index(str(DOCUMENTS_DIR.resolve()))
if index is None:
st.warning("Add PDF, Markdown, or text files to the documents/ folder, then reload the page.")
st.stop()
st.success(f"Indexed {len(chunks)} document chunks.")
question = st.chat_input("Ask a question about your documents")
if question:
with st.chat_message("user"):
st.write(question)
with st.chat_message("assistant"):
try:
results = retrieve(question, index, chunks)
context = format_context(results)
answer = ask_ollama(question, context)
st.write(answer)
with st.expander("Retrieved sources"):
for result in results:
st.write(
f"- {source_label(result)} "
f"(similarity: {result['score']:.3f})"
)
except requests.exceptions.ConnectionError:
st.error("Ollama is not reachable. Start Ollama and verify its local service is running.")
except requests.exceptions.HTTPError as error:
st.error(f"Ollama returned an HTTP error: {error}")
except Exception as error:
st.error(f"The request failed: {error}")Run the application:
streamlit run app.pyStreamlit prints a local URL in the terminal. Open that URL in your browser, add or change documents in documents/, and reload the page to rebuild the index.
How The Pipeline Works
1. Extraction
read_documents() handles PDF, Markdown, and plain text files. PDF pages are kept separate so a citation can include a page number. This is more useful than reporting only a filename, especially for long meeting packs or manuals.
pypdf extracts text from PDFs that contain a real text layer. It does not reliably interpret scanned images. If extract_text() returns an empty string for every page, inspect the PDF before changing the retrieval code. The problem is probably extraction or OCR, not FAISS.
2. Chunking
The example normalizes whitespace and creates chunks of about 900 characters with 150 characters of overlap. Chunking by character count is deliberately simple. Overlap reduces the chance that a definition or sentence split at a boundary loses its surrounding meaning.
There is no universally correct chunk size. Smaller chunks improve precision but may remove necessary context. Larger chunks preserve context but can dilute similarity and consume more prompt space. Start with the simple values above, then inspect retrieved passages for your actual documents.
3. Embeddings And Search
The embedding model maps both chunks and questions into the same vector space. The code normalizes vectors and uses FAISS inner-product search. With normalized vectors, the inner product corresponds to cosine similarity, so higher scores represent closer semantic matches.
The index is IndexFlatIP, an exact search index. That is appropriate for a small personal document set and avoids introducing approximation parameters before you need them. It is not a scalable design for millions of chunks. Larger systems generally use a persistent vector store, metadata filtering, batching, and an incremental update strategy.
4. Grounded Generation
The top four retrieved chunks are inserted into a prompt that tells the model to use only those excerpts. The answer is also asked to cite source labels. This is a prompt-level grounding strategy, not a security boundary. A language model can still ignore instructions or combine passages incorrectly, so test it with questions whose answers you already know.
The Three Mistakes That Waste Time
Mistake 1: Testing With A Scanned PDF
A scanned PDF can look perfectly readable to you while exposing no text to Python. Before tuning chunk sizes or changing models, print the extracted character count for each page. If it is zero or nearly zero, run OCR as a separate preprocessing step and feed the resulting text or searchable PDF into this application.
Mistake 2: Sending The Entire Document To The Model
Stuffing a whole PDF into a prompt appears easy, but it creates three problems: unnecessary context, higher local inference time, and more opportunities for irrelevant text to influence the answer. Retrieval exists to select a small evidence set. Start with four chunks and increase it only when multi-section questions genuinely need more context.
Mistake 3: Treating Similarity As Truth
A high similarity score means that a chunk resembles the question in embedding space. It does not prove that the chunk answers the question. Generic phrases can score well, and exact names or identifiers can sometimes be missed by semantic search. Always display retrieved passages during development, test known questions, and consider adding keyword or metadata filtering when exact matching matters.
Common Errors And Fixes
Ollama Connection Refused
The application calls http://localhost:11434/api/generate. A connection error means the local Ollama service is not available at that address. Start Ollama, confirm that the selected model is installed, and check the OLLAMA_MODEL environment variable.
For a different model, set the variable before starting Streamlit. On macOS or Linux:
OLLAMA_MODEL=your-installed-model streamlit run app.pyIn PowerShell:
$env:OLLAMA_MODEL = "your-installed-model"
streamlit run app.pyThe Model Name Is Not Found
The default is only a configuration value. It must match a model available in your local Ollama installation. Change OLLAMA_MODEL to the exact local model name rather than changing the retrieval code.
No Documents Are Indexed
Check that files are directly under documents/ or one of its subdirectories and that their extensions are .pdf, .txt, or .md. Unsupported files are skipped. For PDFs, verify that they have an extractable text layer.
Import Errors For FAISS Or Sentence Transformers
Make sure the virtual environment is activated when installing and running the application. Install the dependencies with the same Python executable used to launch Streamlit:
python -m pip install -r requirements.txt
python -m streamlit run app.pyPlatform-specific binary compatibility can affect FAISS installation. If faiss-cpu fails to install on your operating system, use the supported Python environment for your platform or replace FAISS with another vector-search implementation. Do not silently mix incompatible Python environments.
Answers Are Vague Or Incorrect
First open Retrieved sources and read the passages. If the passages are irrelevant, the retrieval problem is upstream of the LLM. Try clearer questions, adjust chunk size, increase TOP_K modestly, or add keyword filtering. If the passages are correct but the answer is wrong, improve the prompt or use a stronger local model.
The Application Rebuilds Slowly
The embedding model is cached with Streamlit's resource cache, but the document index is rebuilt when the cached function inputs change or the app is restarted. That is acceptable for a small folder. For repeated use, persist the FAISS index and a metadata file, calculate file hashes, and embed only new or modified files.
Practical Limits And Next Improvements
This example is local, but local does not automatically mean private in every operational sense. The files remain on your machine, yet other processes or users with access to that machine may still read them. Protect the documents and index using normal operating-system permissions. Do not expose Streamlit's development server to an untrusted network without adding authentication and a proper deployment boundary.
The PDF parser also has layout limitations. Tables, columns, headers, footers, and images may be extracted in an inconvenient order or omitted. For technical documents, preserve page metadata and consider a layout-aware parser before blaming the language model.
A useful next step is evaluation. Create a small test set of real questions with expected answers and source pages. Run it whenever you change the embedding model, chunking settings, prompt, or local LLM. Measure whether the correct source appears in the retrieved results separately from whether the final answer is well-formed. These are different failure modes.
For a more capable version, add document hashing, persistent metadata, duplicate detection, a retrieval score threshold, hybrid lexical-plus-semantic search, conversation-history limits, and an explicit answer-quality test set. Add reranking only after inspecting retrieval failures; extra components increase setup and debugging cost.
FAQ
Does this chatbot send my documents to a cloud API?
The application reads documents locally, creates embeddings locally, searches a local FAISS index, and calls an Ollama endpoint on localhost. It does not require an API key or a hosted model. Privacy still depends on your machine, local services, and network configuration.
Can it answer questions about scanned PDFs?
Not by itself. pypdf needs an extractable text layer. Scanned pages generally need OCR first. Once OCR has created usable text, the resulting document can be indexed, although OCR errors may reduce retrieval quality.
Why use RAG instead of fine-tuning the model?
RAG is usually the simpler choice for private documents that change over time. You can replace or re-index files without retraining a model. Fine-tuning changes model behavior and can teach style or task patterns, but it is not a convenient way to maintain a frequently changing document collection.
Is FAISS a database?
FAISS is a vector-search library, not a complete document database. This example keeps chunk metadata in Python memory alongside the in-memory index. A larger application needs persistent storage, file identifiers, update logic, metadata filters, and backup handling.
Why does a citation not guarantee a correct answer?
The citation only shows which retrieved excerpt was supplied to the model. The excerpt may be incomplete, ambiguous, or unrelated despite a good similarity score. Validate both retrieval and generation with known questions, and require the model to state when the evidence is insufficient.
Can I use a different local model?
Yes. Set OLLAMA_MODEL to the exact model name available in your Ollama installation. Model quality, memory use, response speed, and instruction-following vary substantially, so compare models against your own question set rather than assuming a larger model always fixes retrieval problems.