Continued Pretraining vs Fine-Tuning vs RAG

Continued pretraining trains an existing model further on raw text from one field, with the same next-word objective it was first trained on, so the model learns that field's vocabulary and conventions. Fine-tuning trains it on examples of inputs and the outputs you want, so it learns a task, a format or a behaviour. Retrieval adds facts at answer time with no training. Use continued pretraining when the base model misreads your field, fine-tuning when a task has to be done reliably, and retrieval for facts that change.

This post covers what each method does, what the research says about how knowledge actually enters a model, and how the three combine when you build a domain-specific LLM.

What is continuous pretraining?

Continuous pretraining, usually called continued pretraining, takes a model that has already been pretrained and keeps training it on new unlabeled text with the original next-token objective. Sebastian Raschka's explainer on pretraining and finetuning, read on 29 October 2026, notes that this procedure on raw legal, medical or code data "is often called continued pretraining or domain-adaptive pretraining."

It has two common uses. The first is domain adaptation: SaulLM-7B was trained on "an English legal corpus of over 30 billion tokens" on top of an existing open model, per its paper. The second is keeping a model current: Jindal et al. open their 2024 study with the observation that public LLMs "require continuous pre-training to remain up-to-date with the latest data."

What is pretraining and fine-tuning?

Pretraining creates a general base model from a very large and varied corpus, and fine-tuning adapts that pretrained model to a narrower purpose. Raschka calls fine-tuning "the umbrella term for adapting a pretrained checkpoint to a narrower purpose", with instruction fine-tuning as one type inside it. Continued pretraining sits between the two: it starts from a pretrained model, as fine-tuning does, and keeps the pretraining objective.

The data column is the practical difference. Continued pretraining needs a large body of raw field text, and fine-tuning needs labelled examples of the job.

Is fine-tuning and training the same thing?

Fine-tuning is one kind of training. Training covers every stage that updates a model's parameters: pretraining, continued pretraining, fine-tuning and preference training. What separates the stages is purpose more than size. In Raschka's words, "What matters is whether the process is creating a broad base model or adapting an existing checkpoint toward a target distribution or behavior."

In everyday use, "training a custom model" usually means fine-tuning an existing open model, because pretraining from scratch is rare outside the large labs.

How does domain knowledge enter a model?

Field knowledge enters most reliably through pretraining-style exposure or through retrieval, and fine-tuning mostly teaches a model to use what it already knows. Three studies, each read at source on 29 October 2026, support that split.

Fine-tuning struggles with new facts. Gekhman et al., in "Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?" (2024), found that "large language models struggle to acquire new factual knowledge through fine-tuning", and that once the new facts were learned, they "linearly increase the model's tendency to hallucinate." The authors conclude that models "mostly acquire factual knowledge through pre-training, whereas fine-tuning teaches them to use it more efficiently."

Retrieval beat unsupervised fine-tuning for knowledge. Ovadia et al., in "Fine-Tuning or Retrieval?" (2023), compared the two across knowledge-intensive tasks and found that "RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge." They also found that "exposing them to numerous variations of the same fact during training could alleviate this problem."

Continued pretraining adds knowledge and can cost instruction following. Cheng et al., in "Adapting Large Language Models to Domains via Reading Comprehension" (2023), found that training on raw domain text "endows the model with domain knowledge, but drastically hurts its prompting ability for question answering." Rewriting the raw text as reading-comprehension exercises improved results in biomedicine, finance and law.

Together these give a working rule. Facts belong in retrieval, or in continued pretraining when the field's knowledge is stable and large. Behaviour, format and task skill belong in fine-tuning. That is the split we use in every build.

When does continued pretraining fit?

Continued pretraining fits when the base model misreads your field and you have a large body of raw text from it. Signs that you need it:

• The model splits, confuses or misuses the field's terms, codes or notation.
• You hold a large archive of field documents and few labelled examples.
• Many different tasks in the field will run on the same model.

It carries two risks. The model can lose some general ability and instruction following, the effect Cheng et al. measured, and the wider problem of catastrophic forgetting during fine-tuning applies to any further training. Published builds handle it in three ways: mixing general text into the domain corpus (BloombergGPT trained on 345 billion general tokens beside its 363 billion financial ones, per its paper), formatting the domain text as exercises, and running instruction fine-tuning after the domain stage, as SaulLM-7B did.

When does fine-tuning fit?

Fine-tuning fits when a task repeats at volume, its outputs can be checked, and the same long instructions go into every prompt. It is the most direct way to make a model reliable on one job.

Our QMSum study (arXiv 2609.25028) measured how much the task-level training is worth. Holding one base model fixed, better inputs, retrieved transcript spans in place of the transcript's opening words, added 1.55 ROUGE-1 on the test split. Training on those same span-shaped inputs added a further 5.29 ROUGE-1, with a 95% interval of +4.02 to +6.56. A released specialist moved to the new input shape without retraining lost 6.30 ROUGE-1, and training it on the new shape recovered the loss. The lesson for any build is to train the model on the inputs it will see in production.

Fine-tuning also changes cost on two fronts: a lower price per token than the large frontier models, and fewer tokens per task, since the instructions and context repeated in every prompt get trained into the model itself. The cost-per-task calculator shows both with your own prompt sizes.

When does retrieval fit?

Retrieval fits when the facts change, when answers need a citation to a source, or when each request depends on one customer's or one user's records. It needs no training, and updating the index updates the answers.

Retrieval and training also combine. In the QMSum study above, the model trained on retrieved inputs outperformed the same model reading those inputs untrained by 5.29 ROUGE-1. Retrieval chose what the model read, and training taught the model to use it. For the wider comparison, see fine-tuning vs RAG.

Most production systems use at least two rows of this table.

Ertas is an applied AI lab, and the adaptive model endpoint is where this work goes into production: a model trained on your task, served through an OpenAI-compatible API, with retrieval for the facts that change. New versions train on your agents' traffic in the background, are scored against your evaluations, and go live only with your sign-off. If a new version falls short, the current model stays in service. It runs Ertas-managed, in your own cloud or on-premise. The endpoint is currently invitation only, and we are developing a more streamlined version for wider access.

Frequently Asked Questions

Has fine-tuning been debunked?

No. OpenAI and Mistral are winding down fine-tuning of their own models, a decision about what they host, and training open models on task data keeps producing measured gains. Our post is fine-tuning dead? covers the deprecations and the evidence.

Is continued pretraining the same as domain-adaptive pretraining?

Domain-adaptive pretraining is continued pretraining on one field's text. The term comes from Gururangan et al.'s "Don't Stop Pretraining" (2020), which also defined task-adaptive pretraining on a task's own unlabeled data.

Can fine-tuning teach a model new facts?

Only slowly and at a cost. Gekhman et al. (2024) found that facts new to the model are learned more slowly than familiar ones and raise its tendency to hallucinate once learned. Put changing facts in retrieval, and use fine-tuning for behaviour, format and task skill.

Does continued pretraining cause catastrophic forgetting?

It can. Further training on narrow text can erode general ability and instruction following. Mixing general text into the corpus, formatting domain text as exercises, and re-running instruction fine-tuning afterwards all reduce the effect, and an evaluation on held-out general tasks shows how much was lost.

Is RAG better than fine-tuning?

For injecting facts, retrieval did better in Ovadia et al.'s 2023 comparison. For a repeated task, a consistent format or lower cost per task, fine-tuning does better. The two work best together, with the model trained on the retrieved inputs it will read.

How is continued pretraining different from continual learning?

Continued pretraining is one training stage on new text. Continual learning is the ongoing practice of training new model versions as new data arrives, while keeping what earlier versions learned. Our post on continual learning for AI agents covers it at the model layer.

The Short Version

Continued pretraining teaches a model a field's language from raw text. Fine-tuning teaches it a job from examples. Retrieval hands it the facts at answer time. Facts that change belong in retrieval, a job that repeats belongs in fine-tuning, and continued pretraining earns its cost when the base model misreads your field and many tasks will share the model. Whichever you choose, evaluate it on your own domain before it takes traffic.

---

Sources, read 29 October 2026: Sebastian Raschka, "Pretraining vs. finetuning vs. instruction finetuning"; Gururangan et al., "Don't Stop Pretraining" (2020); Wu et al., "BloombergGPT" (2023); Cheng et al., "Adapting Large Language Models to Domains via Reading Comprehension" (2023); Ovadia et al., "Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs" (2023); Colombo et al., "SaulLM-7B" (2024); Gekhman et al., "Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?" (2024); Jindal et al., "Balancing Continuous Pre-Training and Instruction Fine-Tuning" (2024). Our research: the QMSum study.