Guides · Updated

Fine-tuning a small model with LoRA: what it is and when it's worth it

What fine-tuning and LoRA actually change in a model, when a better prompt or RAG is the smarter fix, what hardware and data you need, and how to run your own LoRA in Ollama.

Not sure which model to pull? Use the GPU model picker: enter your VRAM and RAM and get a list of Ollama models that fit.

Fine-tuning means training an existing model a little further on your own examples, so it picks up a particular style, format or job. LoRA is the technique that makes this practical on a home PC. It’s genuinely useful for some jobs, but it’s rarely the first thing you should try.

What LoRA is

A model’s skills live in its weights: billions of numbers learned during training. Full fine-tuning adjusts all of them, which needs far more memory than just running the model.

LoRA (low-rank adaptation) leaves the original weights frozen and trains a small set of extra weights alongside them. The result is an adapter: a separate file, usually a small fraction of the model’s size, that changes how the model behaves when it’s loaded on top.

  • QLoRA keeps the base model in 4-bit while the adapter trains, which cuts memory use further. It’s what makes training on a single consumer graphics card realistic.
  • An adapter belongs to one base model. A LoRA trained on one model won’t work on another, not even a different size of the same family.
  • You can merge it. Most tools can fold the adapter into the model to make one standalone fine-tuned model, which you can then convert to GGUF and quantise like any other.

The same idea is used for image models, where LoRAs teach a style or subject; see Generating images locally. This guide is about language models.

What fine-tuning is good at, and what it isn’t

Fine-tuning works well forBetter handled another way
A consistent output format, such as a fixed JSON layout or report structureFacts that change, such as prices, policies or stock levels: use RAG
A house style or tone of voiceAnswering questions from your documents: use RAG
A narrow, repeated job: sorting tickets, extracting fields, taggingOne-off tasks: write a better prompt
Getting a small, fast model to do one job that a bigger model already does wellMaking a small model smarter in general: use a bigger model

The key point: fine-tuning teaches behaviour much more reliably than it teaches facts. A model fine-tuned on your handbook learns to sound like your handbook, but it can still get the details wrong, and you’d have to retrain every time the handbook changes. For answering from your own files, RAG is easier to update and can point to its sources.

Try these first

  1. A better prompt. Spell out the format, include two or three worked examples, and save it as a system prompt. A custom model in Open WebUI or a SYSTEM line in an Ollama Modelfile makes it reusable.
  2. A better-suited model. A coding tune, a reasoning model or simply the next size up often fixes the problem. See Ollama models explained.
  3. RAG, if the model needs to know things.
  4. Fine-tuning, once you’ve tried those, the job is narrow and repeated, and you can collect good examples.

A fine-tune earns its keep when the same job runs many times and you want a small model to do it consistently, or when your prompt has grown so long that it slows down every request.

What you need

  • Good examples. This matters more than anything else. Each example is an input plus the exact output you want. A smaller set of carefully checked examples usually beats a large, messy one. Keep some back that you don’t train on, so you can test the result fairly.
  • A graphics card, ideally NVIDIA, which is by far the best supported for training. With QLoRA, small models (roughly 1-8B parameters) are within reach of a single consumer card; bigger models need more VRAM. Apple Silicon Macs can train LoRAs with MLX. AMD cards work with some tools on Linux, with more setup.
  • Time and patience. A small LoRA can train in anything from minutes to hours, depending on the model, the data and your hardware. Expect several attempts before one is worth keeping.
  • The base model’s full weights, usually safetensors files from Hugging Face. Training tools work from these, not from the GGUF files Ollama downloads.
  • A licence that allows it. A fine-tune is still bound by its base model’s licence. Check the model card before you use or share the result commercially.

No suitable GPU? You can rent a cloud GPU for the training and still run the result locally. Your training data is uploaded to that provider, though, so think about privacy before you do.

Tools

ToolGood for
UnslothMemory-efficient LoRA and QLoRA on NVIDIA cards, with ready-made notebooks and export to GGUF for Ollama
Hugging Face PEFT and TRLThe standard Python libraries many other tools build on, if you want full control
LLaMA-FactoryLots of models and training methods, with a web interface as well as config files
AxolotlRepeatable training runs defined in a YAML config file
MLX (mlx-lm)LoRA training on Apple Silicon Macs from the command line

On Windows, several of these are easiest to run under WSL. Check each project’s install guide for your GPU.

The process in outline

  1. Pick a small base model that’s already decent at the job. The instruct (chat) version is usually the right starting point.
  2. Prepare your data as a JSONL file, one example per line, in the chat format your tool expects. Use the same system prompt you’ll use afterwards.
  3. Train a LoRA with the tool’s default settings first, then change one thing at a time.
  4. Test it on the examples you held back, side by side with the base model and your best prompt. If it isn’t clearly better, it isn’t worth keeping.
  5. Check it hasn’t got worse at everything else. Too much training can make a model parrot your examples (overfitting) or lose general skills.
  6. Export it, either as an adapter or merged and converted to GGUF, and run it locally.

On a Mac with MLX, the whole loop runs from the command line:

pip install mlx-lm
mlx_lm.lora --model <model-name> --train --data ./data   # folder with train.jsonl and valid.jsonl
mlx_lm.fuse --model <model-name>                          # merge the adapter into the model

Running your LoRA in Ollama

Ollama can apply an adapter on top of a base model. Create a file called Modelfile:

FROM <base-model>
ADAPTER ./my-adapter

Then build and run it:

ollama create my-model -f Modelfile
ollama run my-model
  • FROM must be the exact base model the adapter was trained on. With the wrong base, the output becomes erratic.
  • The adapter can be a folder of safetensors adapter files or a GGUF adapter file. Ollama’s import docs list which model families it supports.
  • Or import a merged model. Merge the adapter, convert the result to GGUF (Unsloth can do this for you), and use FROM ./my-model.gguf instead.
  • Garbled replies, or replies that never stop? The chat template usually doesn’t match the format the model was trained with. Compare yours with the base model’s, which ollama show <base-model> --modelfile prints.

Once it’s in Ollama, any app that connects to Ollama can use it, including Open WebUI and the coding assistants.

Is it worth it? The short version

  • Yes for narrow, repeated jobs where you need a consistent format or style from a small, fast model, and you can collect good examples.
  • Probably not if you want the model to know your documents (use RAG), want it smarter in general (use a bigger model), or haven’t yet tried a careful prompt.

Sources: LoRA paper, QLoRA paper, Hugging Face PEFT docs, MLX LM on GitHub, Ollama import docs.

Get the Local AI weekly

New models worth running, what fits which GPU, and setup fixes. One short email a week, free. Unsubscribe any time.

Subscribe free →

Lila Forge Kits

Local-LLM App Starter

A tested starter for building your own app on Ollama: chat, RAG and a GPU-fit checker. Skip the setup hours.

kits.lilaforge.com →
VibeEngine

Best local LLM tools, ranked

Ollama, LM Studio, Open WebUI and llama.cpp scored from public evidence, with top picks.

vibeengine.lilaforge.com →