Dataset synthesis

Sometimes you do not have enough data. You have 200 real customer-support tickets in the right style and you need 2,000. You have ten beautifully-curated examples of the JSON schema you want the model to emit and you need 500. Synthesis closes those gaps: you generate more rows in the shape of the ones you already have.

This page is about making synthetic data good. For the click-by-click mechanics of generating rows in Data Craft (set a context, copy the prompt Ertas drafts, run it in your own AI tool, paste the rows back), see Creating a dataset. Here we cover when synthesis is worth doing, when it backfires, and the techniques that separate the two.

Done well, synthesis is the single fastest way to scale a fine-tuning dataset. Done badly, it produces a model that imitates the generator's quirks rather than your real distribution. The patterns below apply whether you generate through Prompt Studio or synthesise entirely outside Ertas.

When synthesis works

Synthesis is good for:

• Scaling up a small high-quality dataset. You have 200 great rows and need 2,000. Use the 200 as seeds, generate 1,800 more in the same style, reject the 30% that drift.
• Generating edge-case coverage. You have plenty of happy-path data and need failure-mode rows. Ask your AI tool, through a prompt drafted in Prompt Studio, to produce examples where the user is confused, hostile, or off-topic.
• Style transfer datasets. Rewrite each row of an existing dataset in a new style (formal to casual, plain to JSON).
• Expanding single-turn instructions into multi-turn conversations. Generate plausible follow-up turns on top of an existing instruction-format dataset.

The pattern that works in all four cases: you bring strong signal (real examples plus a clear context) and the generator scales it.

When synthesis fails

Synthesis fails when:

• You have no seeds and no context. Pure cold synthesis tends to produce blandly generic data that teaches the model to imitate the generator rather than your real distribution.
• The generator is wrong about your domain. A general-purpose LLM does not know your support policies, your code style, or your legal language. It will confidently produce wrong-looking-right data. Counterbalance with seed examples and an explicit context that lists the rules.
• You synthesise everything. Even with good seeds, a 100% synthetic dataset has a homogeneity problem. The generator has its own style fingerprint and the model learns to imitate it. Aim for at least 10 to 30% real rows.

Patterns that produce good synthetic data

A few techniques that consistently work:

Few-shot prompting on seeds

Give the generator 5 to 20 real examples and ask it to produce N more in the same format and style. The seeds anchor the output. When you generate through Prompt Studio, put the seed examples into the context so the drafted prompt carries them into your AI tool.

Distilling from a stronger model

If you have access to a frontier model (Claude, GPT-4), generate examples there that a small model would otherwise struggle to produce, then fine-tune your small model on the result. This is the technique behind many popular instruction datasets (Alpaca, Vicuna).

Iterating with critique

A two-pass approach: generate a draft row, then ask the generator to critique its own output ("is this a good example? what is wrong with it?") and rewrite. The critique-then-rewrite loop produces noticeably better rows than one-shot generation, at the cost of double the generation.

Negative examples (for DPO)

If you are doing preference optimisation (DPO), you need pairs of accepted and rejected outputs. Use synthesis to generate the rejected side deliberately by asking the generator to produce a "bad" or "lazy" version of the accepted output. See SFT vs DPO.

Working with acceptance state

Acceptance is the lever that decides what gets trained on. Only accepted rows are sent to the trainer when a run is submitted; everything else stays in the dataset but is excluded. Acceptance is fully reversible: un-accept a row at any time to pull it out of training without deleting it. A few patterns that take advantage of this:

A/B compare by toggling acceptance

Train once with a slice of rows accepted, then un-accept that slice and rerun. The two outputs tell you what those rows actually contributed. Useful when:

• You accepted a batch you are uncertain about. Train with it, un-accept, retrain, pick whichever model wins on your probe set.
• You suspect some rows are noisy. Un-accept the suspicious slice, retrain, compare against the baseline.
• You want a coverage test. Hold out edge-case rows on one run and include them on the next, then measure the delta on a probe set.

Because un-acceptance does not delete the row, this is non-destructive. You can flip rows in and out of the training payload without re-uploading.

Hold as you go

When generating in large batches, accept the rows you are confident about and leave the borderline ones in a held (un-accepted) state. Come back later, edit the borderline rows into shape, and flip acceptance when they are ready. The dataset accumulates a working pool of "almost there" rows that do not pollute training until you say so.

Duplicate as a starting point

The Data Craft table lets you duplicate any row. Clone a good row, tweak two or three fields, and you have a similar-but-different example. Quick way to scale a strong seed by hand when the generator would over-imitate.

Re-running uses the dataset's latest state

A training run reads the dataset's current state at submission time. Edit rows, flip acceptance, or duplicate-and-tweak, then go back to the Studio canvas and re-run: the new run trains on the latest version of the dataset as you have defined it, not a pinned snapshot from an earlier run. This keeps the edit-then-retrain loop fast, but it also means two runs submitted days apart can train on different data if you edited in between. If you need to lock a known-good cut, keep a local copy until version control ships.

Filtering synthesised data

Even with the accept step in Data Craft, a final filtering pass on a generated batch is worth the time. Common subtle problems:

• Outputs that do not match the prompt.
• Refusals where the seed examples did not refuse.
• Repetitive phrasing across batches.
• Hallucinated facts.

A practical filter pipeline:

A 70% retention rate after filtering is normal. 95% retention is suspicious; you might be missing real problems.

Cost

Generating rows runs in your own AI tool, not on Ertas, so synthesis itself does not consume Ertas credits. Your costs are:

• Your AI-tool usage. Whatever ChatGPT, Claude, or your chosen model charges for the generation run.
• Your review time. This is the expensive part. Budget 30 to 60 minutes per 1,000 generated rows to read, filter, and accept.

Training a model on the accepted rows does consume Ertas credits, but that is the training run, not the synthesis. A 3B LoRA on a few thousand rows (T4) is roughly 1 to 2 Ertas credits. See Credits and usage.

What's next