Fine-Tuning LLMs: When and How | OCOPD AI Tutorials

What Fine-Tuning Actually Changes

Fine-tuning continues training a pretrained language model on a smaller, task-specific dataset. The training process compares the model's predictions with expected outputs, calculates an error signal, and adjusts model weights through an optimization process such as gradient descent.

The important practical distinction is this: fine-tuning primarily changes behavior. It can teach a model to follow a particular response structure, apply a classification policy, use a consistent tone, or perform a specialized transformation. It does not turn the model into a reliable, automatically updated database of facts.

For example, fine-tuning can make a support model consistently produce:

Issue summary:
Likely cause:
Recommended action:
Escalation required:

It is a poor mechanism for keeping changing product prices, current policies, inventory, or frequently updated internal documentation available at answer time. Those problems generally call for retrieval augmented generation, in which relevant information is retrieved and supplied to the model during inference.

A useful mental model is:

Prompting changes the instructions for one request or session.
RAG supplies external knowledge at inference time.
Fine-tuning changes the model's learned behavior across requests.

These approaches can be combined. A fine-tuned model may still need RAG for current company information, and a RAG system may still benefit from fine-tuning when the model repeatedly mishandles the desired task or output format.

The Decision: Prompting, RAG, or Fine-Tuning?

Start with the smallest intervention that can close the measured quality gap. Fine-tuning adds dataset preparation, training cost, evaluation work, model-version management, and the risk of degrading capabilities that already worked.

Use Prompting First

Prompting is usually the right first step when the task is not yet well specified or when behavior can be corrected with clearer instructions and examples. It is especially suitable for:

A new workflow whose requirements are still changing.
A small number of request types.
One-off or low-volume tasks.
Behavior that improves when instructions and examples are made explicit.
A system where changing a model or hosting a custom artifact would create unnecessary operational work.

Before fine-tuning, create a baseline prompt and test it on a fixed evaluation set. If the model can meet the target after prompt changes, training is unnecessary.

Use RAG for External or Changing Knowledge

RAG is a better fit when the model must answer from documents, records, or facts that change independently of the model. Examples include:

Current product documentation.
Internal procedures and policies.
Customer-specific records.
Inventory or account data.
Information that must be traceable to a source document.

Fine-tuning facts into model weights creates a maintenance problem. Updating the source material does not automatically update the trained model, and the model may still provide an answer without a verifiable citation. RAG lets the application refresh the retrieved context without retraining the model.

Use Fine-Tuning for Repeated Behavior

Fine-tuning is worth testing when the base model repeatedly fails at a stable, measurable task despite good prompts and suitable context. Strong candidates include:

Consistent classification into a known label set.
Structured transformations from input to output.
Domain-specific terminology handling.
A stable response style or format.
Repeated instruction-following failures.
A high-volume task where a smaller specialized model could reduce inference cost or latency.

Fine-tuning is also useful when the desired behavior must be durable rather than restated in every prompt. Parameter-efficient methods such as LoRA and QLoRA can reduce compute requirements compared with updating all model parameters, although they do not remove the need for careful data and evaluation.

Use a Combined Design When Necessary

Many production systems need both techniques. For example, fine-tuning can teach a model to extract a support ticket into a strict schema, while RAG supplies the current troubleshooting procedure. The model's behavior is specialized through training; the factual source remains external and updateable.

Do not use fine-tuning as a substitute for access control, input validation, source attribution, or application-level business rules. A trained model is still probabilistic software.

Prerequisites

Before starting, prepare the following:

A narrowly defined task. Define the input, expected output, allowed labels or schema, and unacceptable behaviors.
A working baseline. Measure the current model with the prompt and retrieval configuration you would otherwise deploy.
Representative examples. Collect real examples covering common cases, ambiguous inputs, edge cases, refusals, and malformed requests.
A train-validation-test split. Keep evaluation examples separate from training examples. Do not repeatedly tune against the hidden test set.
A quality rubric. Decide how correctness, formatting, completeness, safety, and latency will be assessed.
A compatible training method. Confirm the selected model and training platform support the desired supervised fine-tuning or parameter-efficient method.
A compute and serving plan. Training is only part of the cost. The resulting artifact must be stored, deployed, monitored, and rolled back.

A fine-tuning project without a baseline is difficult to judge. A fine-tuning project without a held-out test set is easy to fool yourself with.

Step 1: Define the Behavioral Contract

Write the task as a contract before writing training examples. For a ticket classifier, the contract might look like this:

Input: A customer support message.
Output: Exactly one label from billing, access, bug, or other.
Output format: JSON with the key "label".
Disallowed: Additional prose, invented account details, or multiple labels.

This contract gives you a target for both dataset construction and evaluation. It also exposes when the task is actually a knowledge problem. If the correct label depends on a current policy document, supply that document through retrieval or another application-controlled input rather than expecting training to preserve the policy forever.

Define success numerically where possible. For classification, measure accuracy, per-class precision and recall, and confusion between important labels. For structured generation, measure schema validity, required-field presence, and field-level correctness. For free-form responses, use a rubric with explicit criteria and human review for a sample of results.

Step 2: Build High-Quality Training Examples

Training data quality matters more than simply increasing row count. Every example should demonstrate the behavior you want the deployed model to reproduce.

A generic instruction-tuning record can be represented as:

{
  "messages": [
    {
      "role": "system",
      "content": "Classify each support message using exactly one approved label."
    },
    {
      "role": "user",
      "content": "I cannot sign in after resetting my password."
    },
    {
      "role": "assistant",
      "content": "{\"label\":\"access\"}"
    }
  ]
}

The exact field names and conversation format depend on the model and training service. Treat the example above as a conceptual record shape and adapt it to the selected platform's documented input format.

Include examples that represent production variation rather than idealized writing. Useful coverage includes:

Short and long inputs.
Spelling errors and informal language.
Multiple issues in one request.
Inputs with missing information.
Inputs that should be rejected or escalated.
Similar examples with different correct labels.
Outputs that must contain no extra text.
Boundary cases where a human reviewer previously disagreed.

Do not mix incompatible behavior in the target outputs. If one record asks for concise JSON and another rewards explanatory prose for the same task, the model receives an unclear signal.

Remove secrets and unnecessary personal information before training. Sensitive information in a training file can be copied into model outputs or become part of an artifact that is difficult to inspect. Data security requirements may also affect whether training must occur inside controlled infrastructure.

Step 3: Validate and Split the Dataset

Use deterministic validation before sending data to a training service. The following small Python example checks a conceptual JSONL file for basic structural problems. It does not replace the selected platform's format validator.

import json
from pathlib import Path

required_roles = ["system", "user", "assistant"]

for line_number, line in enumerate(
    Path("train.jsonl").read_text(encoding="utf-8").splitlines(), 1
):
    if not line.strip():
        continue

    record = json.loads(line)
    messages = record.get("messages")

    if not isinstance(messages, list) or len(messages) < 3:
        raise ValueError(f"line {line_number}: expected at least three messages")

    roles = [message.get("role") for message in messages[:3]]
    if roles != required_roles:
        raise ValueError(f"line {line_number}: unexpected initial roles {roles}")

    if any(not isinstance(message.get("content"), str) for message in messages):
        raise ValueError(f"line {line_number}: message content must be text")

Then split examples into training, validation, and test sets. Keep near-duplicates in the same split; otherwise, the test set may measure memorization instead of generalization. If several records come from the same customer, document, incident, or conversation, split by that source where practical.

A validation set helps select training settings. A held-out test set should be used only for the final comparison against the baseline. Preserve a small, manually reviewed challenge set containing the cases most likely to expose regressions.

Step 4: Choose a Training Method and Run a Small Experiment

Select the base model based on task quality, licensing or service constraints, serving requirements, and available compute. Do not choose a training method only because it is popular.

Full fine-tuning updates the model's parameters broadly and can require substantial compute and storage.
LoRA trains a smaller set of adapter parameters while leaving the base model largely unchanged.
QLoRA combines quantization with adapter-based training to reduce resource requirements further.
Supervised fine-tuning uses labeled input-output examples to teach a target behavior.

The selected framework or platform will define the training configuration. Typical controls include learning rate, batch size, number of epochs, warmup steps, and the adapter configuration when using PEFT. Start with a small controlled run and change one major variable at a time.

Training for more epochs is not automatically better. LLM fine-tuning often needs only a small number of passes over a focused dataset, while excessive training can overfit examples and reduce generalization. Watch training and validation behavior, and use early stopping when the platform supports it.

A framework-neutral training outline looks like this:

load a supported pretrained base model
load and validate train and validation records
select supervised fine-tuning or a PEFT method
configure batch size, learning rate, warmup, and epoch limit
train on the training split
measure the validation split after each checkpoint
retain the checkpoint with the best validation result
save the model or adapter with its dataset and configuration metadata

Do not claim success from a falling training loss alone. Loss can improve while the model becomes worse on real inputs or loses general capabilities.

Step 5: Evaluate Against the Baseline

Run the same test inputs through at least two systems:

The original model with the production-quality baseline prompt.
The fine-tuned model with the intended inference prompt.

If production will use RAG, evaluate the baseline and fine-tuned model with equivalent retrieved context. Otherwise, you are measuring a pipeline change rather than the effect of fine-tuning.

Track both quality and operational behavior. A practical evaluation table might contain:

case_id | expected_behavior | baseline_output | tuned_output | baseline_score | tuned_score | notes

For classification, calculate overall accuracy and per-label metrics. For structured output, validate JSON or the target schema programmatically before scoring semantic correctness. For free-form answers, use a rubric that separates factual correctness, instruction adherence, completeness, tone, and unsupported claims.

Evaluate failure modes individually. A model that improves the average score but starts inventing account details may be unacceptable. A model that produces valid JSON more often but misclassifies high-risk cases may require a different design or a hard application rule.

Also test capabilities that were not supposed to change. Fine-tuning can cause catastrophic forgetting or behavioral drift. Include general-language, refusal, boundary, and out-of-domain cases in the regression suite. Compare latency, token usage, hosting complexity, and rollback behavior alongside quality.

Common Errors

Training on facts that should be retrieved

Current facts belong in an updateable source whenever possible. Fine-tuning them makes updates slow and creates uncertainty about which version the model remembers.

Using too few or unrepresentative examples

A small dataset can work for a narrow behavior, but a handful of clean examples does not represent messy production traffic. Add coverage for ambiguity, edge cases, and invalid inputs before increasing epochs.

Allowing inconsistent target outputs

Inconsistent labels, schemas, tone, or refusal behavior teach contradictory patterns. Establish a written labeling guide and review examples for agreement.

Leaking test examples into training

Duplicates and near-duplicates inflate evaluation results. Deduplicate records and split by source when records are related.

Measuring only training loss

Training loss describes optimization on the training data. It does not prove better production behavior. Use held-out tests, regression cases, and human review where the rubric requires judgment.

Changing several variables at once

If you change the prompt, base model, retrieval context, dataset, and training configuration together, you cannot explain the result. Establish a baseline and run controlled comparisons.

Expecting fine-tuning to enforce hard guarantees

A model can be trained to produce valid-looking output, but validation, authorization, rate limits, and business rules belong in the application. Reject malformed output and handle high-impact decisions with deterministic checks or human review.

Ignoring deployment cost

An adapter or custom model still needs versioning, monitoring, storage, serving capacity, and rollback procedures. Include those costs in the decision before training.

A Practical Rollout Checklist

Before production rollout, verify:

The task has a written behavioral contract.
Prompting and, where relevant, RAG were measured as baselines.
Training data is representative, reviewed, deduplicated, and sanitized.
Training, validation, and test data are separated.
The model passes schema, correctness, and regression tests.
Out-of-domain and adversarial inputs were evaluated.
The model artifact records its base model, dataset version, configuration, and evaluation results.
A rollback path exists.
Application-level validation and access controls remain active.
Production monitoring can detect quality regressions and distribution changes.

Fine-tuning should be treated as an experiment with a measurable acceptance threshold, not as a guaranteed upgrade. If the tuned model does not beat the baseline on the important cases, keep the baseline and improve the task definition or data instead.

FAQ

Does fine-tuning add new knowledge to an LLM?

It can influence how the model responds to domain-specific examples, but it should not be treated as a reliable knowledge store. For changing, proprietary, or source-sensitive information, use retrieval or another controlled data-access mechanism.

Should I try RAG before fine-tuning?

Use RAG first when the problem is access to current or private information. Use prompting first when the behavior is unclear or may be corrected with instructions. Fine-tuning becomes a stronger candidate when a stable behavioral gap remains after those approaches are measured.

How much training data is required?

There is no universal number. A narrow behavior may benefit from a focused set of high-quality examples, while a broad task needs more coverage. Data diversity, correctness, consistency, and representative edge cases matter more than raw row count.

Is LoRA always better than full fine-tuning?

No. LoRA and QLoRA can reduce compute and storage requirements by training adapter parameters, which makes experimentation more accessible. Full fine-tuning may be appropriate when broad parameter updates are justified and the infrastructure can support them. Compare quality, cost, serving complexity, and maintenance requirements for the actual task.

Can fine-tuning eliminate hallucinations?

It can improve task-specific behavior and reduce some recurring errors, but it cannot guarantee factual correctness. Ground factual responses in retrieved or application-supplied data, validate outputs, and evaluate unsupported claims explicitly.

How do I know whether the fine-tuned model is better?

Compare it with the baseline on the same held-out test set and the same inference conditions. Measure task-specific quality, important failure modes, regression cases, latency, and cost. A lower training loss by itself is not evidence of production improvement.

Can a fine-tuned model still use RAG?

Yes. Fine-tuning can teach the model how to use a response format or perform a transformation, while RAG supplies current information at inference time. This combination is often appropriate when behavior is stable but source knowledge changes.