Fine-Tuning: When Not To, and How

New quest. We spent five experiments building a transformer; now the applied question: you have a frozen, general-purpose giant — how do you make it yours? The reflex answer, and our hypothesis to break:

“To make an LLM do something new, you fine-tune it.”

Mostly wrong — in two directions at once. When you should adapt weights, you touch a tiny fraction of them, not all. And most of the time you shouldn’t touch them at all: a better prompt or retrieval wins. This experiment covers the how (LoRA, the alignment ladder), the traps (forgetting, the real-time myth), the when (prompt vs RAG vs fine-tune), and the interview scenarios where the wrong instinct shows. Verdict on the reflex: FAR FROM IT.

Full fine-tuning updates every parameter of every weight matrix — for a big model, billions of them, plus optimiser state, which is why it needs a cluster. LoRA’s insight comes straight from last experiment: the update a task needs is low-rank. So freeze W and learn a low-rank correction beside it:

$W' = W + \Delta W = W + BA, \qquad B \in \mathbb{R}^{d\times r},\ A \in \mathbb{R}^{r\times d},\ r \ll d$

You train only B and A — 2·d·r parameters instead of d². At d = 4096, r = 8 that’s 65K vs 16.8M: 0.4%. And it works — rank-8 adapters match full fine-tuning on most tasks.

↑ Drag the rank. Even at r = 128 you’re training a sliver of one matrix. QLoRA pushes it further — 4-bit-quantise the frozen W, keep the adapters in higher precision — and a 65B model fine-tunes on a single 48GB GPU.

“Fine-tuning” is really several different jobs. A raw pretrained model only predicts the next token — ask it a question and it continues the text instead of answering. Two more stages turn it into an assistant.

↑ Click through the rungs. SFT (supervised fine-tuning) teaches it to answer by imitating demonstrations. RLHF/DPO then teaches it which answer to prefer.

RLHF does that with a reward model and reinforcement learning; DPO (Direct Preference Optimisation) skips the reward model and optimises the preferences directly. Given a chosen answer y_w and a rejected one y_l, it simply raises the model’s relative log-probability of the winner over a frozen reference:

$\mathcal{L}_{\text{DPO}} = -\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)$

Notice what every rung of this ladder changes: behaviour, calibration, tone. None of it is how you add new knowledge — which is the whole next point.

Here’s the mistake that sinks most first fine-tuning projects: trying to teach the model facts. “Fine-tune it on our documentation so it knows our product.” It feels right and it fails, because weights are a terrible database — they can’t be updated when the docs change, they can’t cite a source, and cramming facts in by gradient descent produces confident hallucination and erodes what was already there.

Say you’ve correctly decided to fine-tune. Two things bite.

Catastrophic forgetting. Fine-tune hard on a narrow task and the model quietly gets worse at everything else — its general ability is overwritten. The tell is a great task metric and a sagging general benchmark. Mitigate by mixing general/replay data back in, using a low learning rate, preferring LoRA (it perturbs less), and — critically — gating on held-out general evals, not just your task score.

The “real-time learning” myth. Product asks for a model that “learns continuously from user chats in real time.” True live weight updates are rare and dangerous — instability, forgetting, and feedback loops where the model trains on its own mistakes. What real systems do looks nothing like live training.

Everything above collapses into one framework. There’s a cost ladder — prompt < RAG < fine-tune — and you climb it only when the rung below genuinely fails. Toggle a real problem and see where it lands:

The same three scenarios come up again and again, and the wrong instinct is always the fine-tune reflex. Interviewer’s-eye view — weak answer, then the answer that gets the offer.

Scenario 1 — “Support bot that must answer from our constantly-updated internal docs. Fine-tune it on the docs?”

• ✗Weak: “Yes — fine-tune on all the docs so it learns them.”
• ✓Strong: “No — that’s RAG. The docs change, so knowledge belongs in a retrieval index the bot cites, not in frozen weights that go stale. Fine-tune only if you also need a specific support tone or output format — and then a small LoRA, on top of RAG.”

Scenario 2 — “You fine-tuned on your domain and now it’s worse at basic reasoning. What happened?”

• ✗Weak: “It overfit — needs more epochs or more data.”
• ✓Strong: “Catastrophic forgetting — the narrow fine-tune overwrote general ability. Fix: mix general/replay data back in, lower the learning rate, switch to LoRA to perturb less, and gate on a held-out general benchmark, not only the task metric.”

Scenario 3 — “The PM wants the model to learn continuously, in real time, from user chats. Design it.”

• ✗Weak: “Stream each chat into an online training loop and update the weights live.”
• ✓Strong: “Rarely live weights — that invites instability, forgetting, and feedback loops. ‘Real-time’ here means fresh answers: update the retrieval store for instant freshness. Any weight change is a batch pipeline — log, filter, curate, fine-tune on a schedule, eval-gate on a frozen suite, canary-roll out.”

Quest-05 continues from here: how you actually measure that a fine-tune is better and not merely different (eval design for open-ended generation is its own hard problem), the RLHF/DPO objective in full, and serving many LoRA adapters at once (multi-tenant fine-tuning). Same method as always — study it from the problem up, mock it hard, build it in public.