Data Preparation for Small Language Models: Quality Over Quantity

Large language models — 70B parameters and above — are remarkably tolerant of messy training data. Their massive parameter count gives them enough capacity to absorb contradictions, tolerate noise, and still extract useful patterns. If 5% of your training examples have incorrect labels, a 70B model barely notices. The signal-to-noise ratio is good enough.

Small language models — 3B to 14B parameters — do not have this luxury. With fewer parameters, every training example has proportionally more influence on the model's behavior. A 7B model fine-tuned on 2,000 examples gives each example roughly 3.5 million parameters of influence. A bad example doesn't just add noise — it actively distorts the model's learned patterns.

This is the SLM data paradox: the models that are most practical to deploy (small, fast, cheap to run) are the ones that demand the most from their training data. Understanding this paradox and preparing data accordingly is what separates SLM fine-tuning projects that succeed from those that produce mediocre models.

Why Small Models Are Unforgiving

The relationship between model size and data quality tolerance is not linear — it's exponential. Here's what happens at each scale:

70B+ models: Can tolerate 5-10% label noise and still perform well. Their capacity allows them to "average out" conflicting signals. Training on 50,000 noisy examples works reasonably well.

14B models: Tolerate 3-5% label noise before performance degrades noticeably. Contradictory examples create confused representations that surface as inconsistent outputs. Training on 10,000 moderately clean examples is better than 50,000 noisy ones.

7B models: Tolerate less than 3% label noise. At this scale, every inconsistency is amplified. The model memorizes bad patterns because it doesn't have enough capacity to distinguish signal from noise. Training on 2,000 pristine examples consistently outperforms 10,000 mediocre ones.

3B models: Essentially zero tolerance for label noise. These models need near-perfect training data because they memorize rather than generalize from patterns. A handful of bad examples can dominate the model's behavior for specific input types.

The practical implication: if you're fine-tuning a 7B or smaller model, your data quality standards need to be significantly higher than what you'd accept for a large model.

Quality Requirements for SLMs

Label Accuracy: >95%

For large models, 90% label accuracy is often acceptable. For SLMs, the threshold is 95% minimum, with 98%+ as the target.

How to achieve this: dual-annotation with expert review of disagreements. Every example that two annotators disagree on gets reviewed by a third expert annotator who makes the final call. This process is more expensive than single-annotation, but the cost is modest when your total dataset is 2,000 examples rather than 50,000.

The math: dual-annotating 2,000 examples with a 10% disagreement rate means 200 examples need expert review. At 2 minutes per review, that's roughly 7 hours of expert time. This is a trivial cost compared to the weeks wasted retraining a model that fails due to label noise.

Format Consistency: 100%

Large models can handle minor formatting variations in training data — inconsistent capitalization, varying JSON key orders, occasional extra whitespace. SLMs cannot. Format inconsistencies in the training data directly produce format inconsistencies in model outputs.

If your model should output JSON with fields category, confidence, and explanation, then 100% of your training examples must have exactly those three fields, in the expected format, with the expected data types. Not 98%. Not 99%. All of them.

Automated validation catches most format issues. Write a schema validator (JSON Schema or Pydantic) and run every example through it before training. Reject and fix any example that fails validation. This takes 30 minutes to set up and prevents days of debugging format-related model failures.

Deduplication: Under 1% Near-Duplicates

Near-duplicates (cosine similarity > 0.95) are particularly harmful for SLMs because they cause memorization rather than generalization. If 15 examples in your 2,000-example dataset are variations of the same customer complaint, the model memorizes that complaint pattern at the expense of learning general complaint handling.

For large datasets (50,000+), 3% near-duplicates is acceptable. For SLM-sized datasets (500-5,000), keep near-duplicates below 1%.

Deduplication process: embed all examples using a sentence embedding model, compute pairwise cosine similarity, flag pairs above 0.95, keep the highest-quality version of each duplicate group.

Input Length Distribution: Match Production

This is often overlooked but critically important for SLMs. If your production inputs are 500-2,000 tokens but your training examples are all 100-300 tokens, the model has never seen inputs at the length it will encounter in production. Large models handle this length mismatch somewhat gracefully. SLMs do not — they often degrade significantly on inputs longer than their training examples.

Measure the token length distribution of your expected production inputs. Ensure your training data covers the same distribution. Specifically, the 10th percentile and 90th percentile of training input lengths should bracket the 10th and 90th percentile of production input lengths.

Class Distribution: No Category Below 5%

Extreme class imbalance hits SLMs harder than large models. A 70B model with 2% of examples in a minority class might still learn to recognize that class. A 7B model with the same imbalance will effectively ignore the minority class — it doesn't have the capacity to maintain a robust representation for so few examples.

Target: no class below 5% of the total dataset. If you have 10 categories, each should have at least 50 examples in a 1,000-example dataset. If a category genuinely occurs less than 5% of the time in production, consider oversampling it in the training data (while keeping total dataset size manageable).

The "Small Data" Reality

For SLMs, the optimal dataset size is typically 500-5,000 examples. This is counterintuitive for teams accustomed to "more data is always better," but the evidence is consistent.

500 examples is sufficient for narrow tasks where the input/output pattern is consistent: classification into 3-5 categories, structured extraction from a single document type, reformatting with a fixed output schema.

1,000-2,000 examples handles moderate complexity: classification into 10-15 categories, extraction from multiple document types, generation with varying output lengths.

3,000-5,000 examples is needed for complex tasks: multi-step reasoning, open-ended generation within a domain, handling diverse input types with varied output formats.

Beyond 5,000 examples, adding more data for SLMs shows diminishing returns unless the additional data covers genuinely new patterns. Adding 5,000 more examples that are similar to existing ones does not help — it just adds redundancy.

The practical workflow: start with 500 high-quality examples, fine-tune, evaluate. If performance is below target, analyze the errors. Are they concentrated in specific categories (add more examples there) or spread evenly (improve example quality across the board)?

How to Curate for Quality

Step 1: Start with Expert-Reviewed Examples

Every example should be created or reviewed by a domain expert. For SLM datasets, there is no room for "good enough" — each example needs to be correct.

The investment is justified by the math: expert review of 2,000 examples at 1-2 minutes each takes 33-66 hours. Spread across 3 experts over 2 weeks, that's 1-2 hours per day per expert. This is the most cost-effective investment in your fine-tuning project.

Step 2: Remove Near-Duplicates

Run deduplication using cosine similarity with a threshold of 0.95. For SLM datasets, also check for semantic duplicates — examples that are different in wording but identical in meaning. These are harder to detect automatically but equally harmful for small models.

A practical check: cluster your examples using k-means or HDBSCAN and manually inspect the largest clusters. Clusters with many near-identical examples need pruning.

Step 3: Balance Class Distribution

Count examples per category. Identify underrepresented classes. For each underrepresented class, either collect more examples or create synthetic examples with expert review.

When creating synthetic examples for class balancing, always have a domain expert review the synthetic examples before including them. Synthetic examples that are plausible to an LLM but wrong by domain standards are worse than no examples at all.

Step 4: Validate Output Format

Write a validator. Run every example through it. Fix every failure. This is non-negotiable for SLMs.

Common format issues that slip through manual review: trailing whitespace, inconsistent null representations (null vs "null" vs "N/A" vs empty string), inconsistent date formats, missing optional fields that are sometimes present.

Step 5: Test Edge Cases

Identify 10-15 edge case categories with domain experts. Ensure at least 3-5 examples per edge case category are in the training data. SLMs need explicit exposure to edge cases — they can't generalize to unusual inputs from standard examples the way larger models sometimes can.

Anti-Patterns for SLM Data Preparation

Training on Unreviewed Synthetic Data

Using an LLM to generate synthetic training data and then fine-tuning an SLM on that data without expert review is the most common SLM training failure. The synthetic data looks plausible but contains domain errors that the generating LLM doesn't recognize. The SLM faithfully learns these errors.

Synthetic data is useful for SLM training, but only after expert review. Generate candidates with an LLM, then have domain experts review and correct every example. The LLM saves annotation time (reviewing is faster than creating from scratch), but the expert review is mandatory.

Mixing Multiple Tasks in One Dataset

SLMs perform best when fine-tuned for a specific task. Training a 7B model to simultaneously classify documents, extract entities, and generate summaries produces a model that does all three poorly. Large models can handle multi-task training because they have the capacity to maintain separate representations for each task.

For SLMs: one model, one task. If you need three capabilities, fine-tune three models or fine-tune sequentially with careful evaluation at each step.

Inconsistent Formatting Across Examples

Large models can handle minor formatting variations. SLMs reproduce whatever formatting patterns they see in training data — including inconsistencies. If some examples use title case and others use sentence case, the model will randomly switch between them at inference time.

Standardize formatting before training. Pick one convention and apply it to every example: capitalization, punctuation, spacing, key ordering, date formats, number formatting.

Ertas Data Suite's quality scoring is calibrated for SLM training requirements. The quality metrics apply stricter thresholds for smaller target models — higher label consistency requirements, tighter deduplication ratios, and stricter format compliance checks. The platform flags issues that large model training might tolerate but SLM training cannot, so teams catch quality problems before they waste a training run.

---

Further Reading

• Data Quality Over Quantity: Why 250 Good Examples Beat 10,000 Bad Ones — The evidence behind quality-first data preparation, with benchmarks from recent research.
• How Much Data Do You Need for Fine-Tuning? — Data volume guidelines by task type and model size.
• Clean Your Dataset Without a Data Team — Practical steps for cleaning training data when you don't have dedicated data engineers.