Guides · Updated 2026-10-17

Chat with your documents locally: a simple RAG setup with Ollama

Build private document Q&A with Ollama: no-code options (Open WebUI, LM Studio) and a 40-line Python version you can understand.

RAG (retrieval-augmented generation) lets a model answer from your files: it finds the most relevant chunks and puts them in the prompt. Running it locally means your contracts, notes and code never leave your machine.

How it works

1. Split documents into chunks (a few hundred words each).
2. Embed each chunk into a vector with an embedding model.
3. At question time, embed the question and retrieve the closest chunks.
4. Generate an answer with a chat model, using those chunks as context.

Option A: no code

• Open WebUI (connects to Ollama): upload files into a Knowledge collection and reference it in chat.
• LM Studio: drag a document into the chat; it handles retrieval for you.
• AnythingLLM: a desktop app focused on document workspaces, with an Ollama connector.

Option B: build it in Python

Pull a chat model and an embedding model:

ollama pull qwen3.5:9b
ollama pull nomic-embed-text
pip install ollama numpy

import ollama, numpy as np, pathlib

EMBED, CHAT = "nomic-embed-text", "qwen3.5:9b"

# 1. Load and chunk every .txt/.md file in ./docs
chunks = []
for f in pathlib.Path("docs").rglob("*"):
    if f.suffix in {".txt", ".md"}:
        words = f.read_text(encoding="utf-8").split()
        for i in range(0, len(words), 250):
            chunks.append(" ".join(words[i:i + 300]))  # 50-word overlap

# 2. Embed all chunks once
vecs = np.array(ollama.embed(model=EMBED, input=chunks)["embeddings"])
vecs /= np.linalg.norm(vecs, axis=1, keepdims=True)

def ask(q, k=4):
    qv = np.array(ollama.embed(model=EMBED, input=q)["embeddings"][0])
    qv /= np.linalg.norm(qv)
    top = np.argsort(vecs @ qv)[-k:][::-1]          # 3. retrieve
    context = "\n---\n".join(chunks[i] for i in top)
    r = ollama.chat(model=CHAT, messages=[           # 4. generate
        {"role": "system", "content": "Answer only from the context. Say if it's not there."},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {q}"},
    ])
    return r["message"]["content"]

print(ask("What is our refund policy?"))

For more than a few thousand chunks, store vectors in a database such as Chroma, LanceDB, SQLite with sqlite-vec, or Postgres with pgvector instead of a NumPy array.

Tips for better answers

• Context window: retrieved chunks have to fit. Raise Ollama’s context (OLLAMA_CONTEXT_LENGTH=8192 or more) if answers seem to ignore sources.
• Embedding models: nomic-embed-text (274 MB) is a solid English default. qwen3-embedding:0.6b handles many languages and longer inputs.
• Models with long context like Qwen3.5, Ministral 3 and Gemma 4 (all 128K-256K) are good RAG generators. Find one for your GPU with the picker (choose “Chat with my documents”).
• Re-embed when you change embedding model. Vectors from different models aren’t compatible.

The Local-LLM App Starter kit includes a tested RAG pipeline if you’d rather start from working code.