Guides · Updated

Local AI for privacy: keeping your documents off the cloud, step by step

A practical setup for asking AI about contracts, client files and notes without uploading them anywhere: what to install, which settings to turn off, and how to check nothing leaves your PC.

Not sure which model to pull? Use the GPU model picker: enter your VRAM and RAM and get a list of Ollama models that fit.

Running a model locally is only half of privacy. The app around it, your sync folders and a few easy-to-miss settings decide whether your documents really stay on your machine. This is the checklist we’d follow for sensitive files. For the why, see Why run AI locally? The per-app switches, and the off setting each official doc names, are in What leaves your PC?

Step 1: Decide what counts as sensitive

Make a short list: client files, contracts, medical or financial records, HR documents, unreleased code. Those files only go into the local setup below, never into a cloud chatbot. If you work under a contract or regulation, check what it says about where data may be processed. Local processing helps, but it doesn’t make you compliant on its own.

Step 2: Install Ollama and keep it on localhost

Install Ollama on Windows, Mac or Linux. By default it only listens on localhost:11434, so other devices on your network can’t reach it. Leave it that way. Ollama’s API has no built-in login, so don’t set OLLAMA_HOST=0.0.0.0 or port-forward it.

Pick a model that fits your GPU with the GPU picker (choose “Chat with my documents”).

Watch out for cloud models. Ollama also offers cloud-hosted models, whose names end in cloud. Those run on Ollama’s servers, not your PC. For private work, avoid any model with cloud in its name.

Step 3: Choose a front-end that stores everything locally

You need an app to upload documents and chat. Good local options:

  • Open WebUI: a browser-based chat that you run yourself and that connects to Ollama. See our setup guide.
  • LM Studio: a desktop app with document chat built in. See LM Studio vs Ollama.
  • AnythingLLM: a desktop app built around document workspaces.

The switches those apps actually document, including cloud tags, web search, call mode and update checks, are in What leaves your PC?.

Step 4: Don’t add cloud connections

Most front-ends can also talk to cloud providers. The moment you add a cloud API key and pick that model, your documents go to that provider. For your private setup:

  • Don’t add cloud model providers, or keep them in a separate app.
  • Turn off web search features. They send your questions (and sometimes parts of your documents) to a search service.
  • Check the document extraction / OCR setting. Some apps offer cloud services for reading PDFs; choose the built-in local option.
  • Check the embedding model setting. It should be a local model (for example nomic-embed-text in Ollama), not a cloud embedding API.

Step 5: Keep source files out of synced folders

If your documents sit in OneDrive, Dropbox, Google Drive or iCloud, they’re already in the cloud, whatever AI you use. On Windows, OneDrive’s folder backup can sync your Desktop and Documents folders without you noticing. Keep sensitive files in a folder that isn’t synced, and check that your chat app’s data folder isn’t inside a synced folder either.

Step 6: Load your documents

Upload files into the app’s document or knowledge feature. Most apps copy each file into their own data folder and store the embeddings next to it, so the copy stays on your PC even if you delete the original. How retrieval works, and a no-code vs Python comparison: Chat with your documents locally.

Step 7: Prove it works offline

This is the simplest test there is:

  1. Do one test run while online. Some apps download an embedding, OCR or speech model the first time you use a feature.
  2. Turn off Wi-Fi or unplug the network cable.
  3. Upload a new document and ask questions about it.

If you still get answers, the model, retrieval and storage are all running on your machine. If something fails, that feature depends on the internet, so find out what it was calling before you trust it with sensitive files. On Windows, Resource Monitor → Network shows which processes are making network connections if you want to look closer.

Step 8: Protect what’s stored on disk

Chats, uploaded copies and embeddings all live on your drive.

  • Turn on full-disk encryption: BitLocker (Windows Pro) or Device encryption (many Windows Home PCs), FileVault on Mac, LUKS on Linux.
  • Delete what you no longer need. Removing a chat doesn’t always remove uploaded files or their embeddings. Delete the document from the app’s knowledge or workspace section too.
  • Mind your backups. A cloud backup of your user folder can include the app’s data folder.

Step 9: Sharing with a team? Keep it private

If colleagues need access, run the server on one machine and reach it over a VPN such as Tailscale, or put it behind a reverse proxy with authentication. Open WebUI has user accounts, which helps. Never expose Ollama or the chat app directly to the internet. A fuller starter for a small office is in Local AI for a small business: a practical starter.

Using agents or tools?

A model that can read files, run commands or browse the web can also leak what it reads, even when the model itself is local. Set it up with the precautions in Running local AI agents safely.

Quick checklist

  • Ollama on localhost, no cloud models
  • The documented off switches in What leaves your PC?
  • No cloud API keys, web search off, local OCR and embeddings
  • Source files and app data outside synced folders
  • Tested with the network off
  • Full-disk encryption on

Something not connecting after locking things down? See When it doesn’t work.

Get the Local AI weekly

New models worth running, what fits which GPU, and setup fixes. One short email a week, free. Unsubscribe any time.

Subscribe free →

Lila Forge Kits

Local-LLM App Starter

A tested starter for building your own app on Ollama: chat, RAG and a GPU-fit checker. Skip the setup hours.

kits.lilaforge.com →
VibeEngine

Best local LLM tools, ranked

Ollama, LM Studio, Open WebUI and llama.cpp scored from public evidence, with top picks.

vibeengine.lilaforge.com →