What is LoRA — fine-tuning where 0.01% of the weights learn

Look at this number: to adapt GPT-3 to a task with LoRA, its authors trained 10,000× fewer parameters than regular fine-tuning would. GPU memory dropped 3×.

Not 10% fewer. Ten thousand times.

Sounds like cheating. Let's find the trick — because the trick is honest, and it's exactly why you can fine-tune a model on a rented GPU for a couple of dollars.

What LoRA means in plain words

LoRA is Low-Rank Adaptation. A way to fine-tune a model without touching the model.

Regular fine-tuning does it head-on: take all the model's weights and nudge every one. On a 7-billion-parameter model that's 7 billion numbers to hold in memory, compute gradients for, and keep optimizer state for. Hence the requirements that hurt: tens of gigabytes of VRAM for a model you just wanted to teach your writing style.

LoRA goes sideways. The base weights are frozen — nobody moves them at all. Two small matrices are parked alongside, and only those learn. At runtime the model adds things up: its frozen answer plus a correction from those matrices.

The metaphor is simple. Regular fine-tuning rewrites the whole textbook. LoRA leaves the textbook alone and slips in a thin stack of sticky notes. Read the textbook with the notes and you get a different result.

How two small matrices replace one big one

Here's all the math, and it's short.

The correction to a big weight matrix is itself a matrix — the same huge size. Storing it whole is silly. The authors' key observation: that correction is redundant — there's little genuine novelty in it, and it factors into two narrow matrices.

Picture a 1000×1000 sheet — a million cells. Now take two strips: 1000×8 and 8×1000. Multiply them and you get a sheet of the same 1000×1000 size. But you stored and trained only 16,000 numbers instead of a million.

That 8 is the rank (r), LoRA's main dial. Typical value is exactly 8. Lower rank — fewer trainable numbers, cruder correction; higher — finer and pricier.

One more saving: LoRA is usually attached only to the attention blocks, not the whole model. In practice that's enough.

Why this changes things for you

Memory savings are only half the story. The other half is more interesting.

An adapter weighs megabytes, not gigabytes. The output of training isn't a new 14 GB model — it's a small file with two matrices. Upload it, download it, send it to a friend.

Adapters swap like attachments. One base, as many adapters as you like: one for your email style, one for JSON formatting, one for a specific character. Keep one model in memory and plug in whichever you need. That's exactly how those libraries of LoRA styles for image models work — the pile of styles you've seen are adapters on a single base.

Speed doesn't suffer. Before shipping, an adapter can be merged back into the base weights as one matrix. After merging, the model runs at exactly the original speed: no extra inference latency at all.

This is also why LoRA lives in the world of open weights. To stick notes in the textbook, you have to be able to open it — with a closed model you only reach through an API, the trick doesn't work.

Where you'll run into it

Three places where the word LoRA shows up on its own.

First — image models. You download a model and it comes with a dozen "styles" of 50–150 MB each. Those are LoRA adapters.

Second — local language models. You run a model on your own machine (how to run AI models locally) and want it to answer in your format. LoRA is the only realistic path that doesn't involve renting a farm.

Third — fine-tuning services. Plenty of "fine-tunes" are LoRA under the hood; they just don't tell you.

And here's the fork in the road, so you don't burn a weekend. LoRA teaches manner: format, style, tone, answer structure. It's bad at stuffing facts into a model — your knowledge base, fresh documents, product prices. Facts need a different tool, and the difference is unpacked separately: RAG vs fine-tuning.

The rule is simple: "answer in this format" — LoRA. "Know what's in these documents" — not LoRA.

How much data does LoRA need?

Less than you'd think: a few hundred good examples is often enough. Quality beats volume — a hundred clean "request → ideal answer" pairs will do more than ten thousand random ones. The model learns the bad examples just as diligently as the good ones.

Does LoRA damage the base model?

No. The base weights are frozen and never change — the adapter is a separate file. Delete the file and you're back to the original model. That's one reason experimenting with LoRA isn't scary.