Guides · Updated
Local AI for privacy: keeping your documents off the cloud, step by step
A practical setup for asking AI about contracts, client files and notes without uploading them anywhere: what to install, which settings to turn off, and how to check nothing leaves your PC.
Running a model locally is only half of privacy. The app around it, your sync folders and a few easy-to-miss settings decide whether your documents really stay on your machine. This is the checklist we’d follow for sensitive files. For the why, see Why run AI locally? The per-app switches, and the off setting each official doc names, are in What leaves your PC?
Step 1: Decide what counts as sensitive
Make a short list: client files, contracts, medical or financial records, HR documents, unreleased code. Those files only go into the local setup below, never into a cloud chatbot. If you work under a contract or regulation, check what it says about where data may be processed. Local processing helps, but it doesn’t make you compliant on its own.
Step 2: Install Ollama and keep it on localhost
Install Ollama on Windows, Mac or Linux. By default it only listens on localhost:11434, so other devices on your network can’t reach it. Leave it that way. Ollama’s API has no built-in login, so don’t set OLLAMA_HOST=0.0.0.0 or port-forward it.
Pick a model that fits your GPU with the GPU picker (choose “Chat with my documents”).
Watch out for cloud models. Ollama also offers cloud-hosted models, whose names end in cloud. Those run on Ollama’s servers, not your PC. For private work, avoid any model with cloud in its name.
Step 3: Choose a front-end that stores everything locally
You need an app to upload documents and chat. Good local options:
- Open WebUI: a browser-based chat that you run yourself and that connects to Ollama. See our setup guide.
- LM Studio: a desktop app with document chat built in. See LM Studio vs Ollama.
- AnythingLLM: a desktop app built around document workspaces.
The switches those apps actually document, including cloud tags, web search, call mode and update checks, are in What leaves your PC?.
Step 4: Don’t add cloud connections
Most front-ends can also talk to cloud providers. The moment you add a cloud API key and pick that model, your documents go to that provider. For your private setup:
- Don’t add cloud model providers, or keep them in a separate app.
- Turn off web search features. They send your questions (and sometimes parts of your documents) to a search service.
- Check the document extraction / OCR setting. Some apps offer cloud services for reading PDFs; choose the built-in local option.
- Check the embedding model setting. It should be a local model (for example
nomic-embed-textin Ollama), not a cloud embedding API.
Step 5: Keep source files out of synced folders
If your documents sit in OneDrive, Dropbox, Google Drive or iCloud, they’re already in the cloud, whatever AI you use. On Windows, OneDrive’s folder backup can sync your Desktop and Documents folders without you noticing. Keep sensitive files in a folder that isn’t synced, and check that your chat app’s data folder isn’t inside a synced folder either.
Step 6: Load your documents
Upload files into the app’s document or knowledge feature. Most apps copy each file into their own data folder and store the embeddings next to it, so the copy stays on your PC even if you delete the original. How retrieval works, and a no-code vs Python comparison: Chat with your documents locally.
Step 7: Prove it works offline
This is the simplest test there is:
- Do one test run while online. Some apps download an embedding, OCR or speech model the first time you use a feature.
- Turn off Wi-Fi or unplug the network cable.
- Upload a new document and ask questions about it.
If you still get answers, the model, retrieval and storage are all running on your machine. If something fails, that feature depends on the internet, so find out what it was calling before you trust it with sensitive files. On Windows, Resource Monitor → Network shows which processes are making network connections if you want to look closer.
Step 8: Protect what’s stored on disk
Chats, uploaded copies and embeddings all live on your drive.
- Turn on full-disk encryption: BitLocker (Windows Pro) or Device encryption (many Windows Home PCs), FileVault on Mac, LUKS on Linux.
- Delete what you no longer need. Removing a chat doesn’t always remove uploaded files or their embeddings. Delete the document from the app’s knowledge or workspace section too.
- Mind your backups. A cloud backup of your user folder can include the app’s data folder.
Step 9: Sharing with a team? Keep it private
If colleagues need access, run the server on one machine and reach it over a VPN such as Tailscale, or put it behind a reverse proxy with authentication. Open WebUI has user accounts, which helps. Never expose Ollama or the chat app directly to the internet. A fuller starter for a small office is in Local AI for a small business: a practical starter.
Using agents or tools?
A model that can read files, run commands or browse the web can also leak what it reads, even when the model itself is local. Set it up with the precautions in Running local AI agents safely.
Quick checklist
- Ollama on
localhost, nocloudmodels - The documented off switches in What leaves your PC?
- No cloud API keys, web search off, local OCR and embeddings
- Source files and app data outside synced folders
- Tested with the network off
- Full-disk encryption on
Something not connecting after locking things down? See When it doesn’t work.
Get the Local AI weekly
New models worth running, what fits which GPU, and setup fixes. One short email a week, free. Unsubscribe any time.
Local-LLM App Starter
A tested starter for building your own app on Ollama: chat, RAG and a GPU-fit checker. Skip the setup hours.
kits.lilaforge.com → VibeEngineBest local LLM tools, ranked
Ollama, LM Studio, Open WebUI and llama.cpp scored from public evidence, with top picks.
vibeengine.lilaforge.com →