Cleaning and Curating Datasets for Fine-Tuning Without a Data Science Team

Most fine-tuning failures do not happen during training. They happen before training — in the dataset. A model trained on messy data produces messy outputs. A model trained on inconsistently labeled data produces inconsistent classifications. A model trained on data that does not match production inputs performs poorly on production inputs.

The problem is that data cleaning is traditionally a data science task. It involves writing Python scripts for deduplication, building statistical analyses for distribution checks, and creating custom validation pipelines. If you are an AI agency, a product lead, or an indie developer without a data engineering background, this can feel like a wall.

It does not have to be. The 6-step process below uses no-code tools, spreadsheet techniques, and LLM-assisted validation to produce clean, production-ready fine-tuning datasets. We have seen agencies clean 1,000-example datasets in 2-4 hours using this approach — no pandas, no Python, no data science degree.

Why Data Cleaning Matters This Much

Here are three scenarios we see regularly:

Scenario 1: An agency collects 2,000 customer support examples from their client. They fine-tune immediately. The model gets 78% accuracy on their test set. They spend two weeks trying different hyperparameters, learning rates, and training epochs. Nothing moves the number significantly.

Scenario 2: A different agency collects 800 examples from the same type of client. They spend half a day cleaning the data. They fine-tune. The model gets 91% accuracy on a comparable test set.

The difference is not the model, the hyperparameters, or the training approach. It is the data.

Dirty data introduces contradictory signals during training. If example #142 says input X is category A and example #891 says a nearly identical input is category B, the model cannot learn either reliably. These contradictions are invisible during collection but devastating during training.

The rule of thumb: Every hour spent cleaning data saves 3-5 hours of debugging model performance later.

The 6-Step Cleaning Process

Step 1: Format Validation (15-30 minutes)

Before anything else, verify that every example in your dataset is structurally correct. This catches the obvious problems that would otherwise cause training failures or silent data corruption.

What to check:

• Every example has all required fields (input, output, and any metadata fields your training pipeline expects)
• No empty or null values in required fields
• Output values match expected types (e.g., category labels are from your predefined set, not free-text variations)
• Text fields do not contain encoding artifacts, HTML tags, or invisible characters

How to do it without code:

Open your dataset in a spreadsheet (Google Sheets or Excel). Use these techniques:

• Missing values: Sort each column. Empty cells will cluster at the top or bottom. Flag and investigate them.
• Invalid labels: Create a dropdown validation list for your category column. Any cell that does not match a valid category will show a validation error. In Google Sheets: Data > Data Validation > List of items.
• Length checks: Add a column with =LEN(A2) to check input and output lengths. Sort by length. Unusually short inputs (under 10 characters) are often data entry errors. Unusually long outputs may contain copy-paste artifacts.
• Character issues: Use =CLEAN(A2) to strip non-printable characters. If A2 <> CLEAN(A2), the cell has invisible characters that can corrupt training.

With Ertas Vault: Upload your dataset and Vault runs format validation automatically. It flags missing fields, invalid label values, encoding issues, and length anomalies. You review and fix the flagged items instead of hunting for them.

Typical result: 3-8% of examples have format issues. Fixing them is usually straightforward — filling in missing fields, correcting typos in category labels, removing encoding artifacts.

Step 2: Deduplication (15-30 minutes)

Duplicate and near-duplicate examples waste training capacity and can cause the model to memorize specific examples rather than learning general patterns.

Exact duplicates are easy: sort your spreadsheet by the input column and scan for identical rows. In Google Sheets, use conditional formatting to highlight duplicates: Format > Conditional formatting > Custom formula: =COUNTIF(A:A, A2)>1.

Near-duplicates are harder but more important. These are inputs that are phrased slightly differently but convey the same information. For example:

• "How do I cancel my subscription?"
• "How can I cancel my subscription?"
• "I want to cancel my subscription"

These three examples teach the model essentially the same thing. Having all three in your dataset is not harmful in small numbers, but if 15% of your dataset is near-duplicates, your effective dataset size is 15% smaller than you think.

Without code: Sort by input text and scan visually. Most near-duplicates will cluster together alphabetically. Flag rows where the input is very similar to the row above or below.

With LLM assistance: Paste batches of 50-100 inputs into Claude or GPT-4o with this prompt:

Review these inputs and identify groups that are near-duplicates
(same meaning, different phrasing). List each group with the
row numbers. Only flag true semantic duplicates, not inputs
that happen to discuss the same topic.

[paste inputs with row numbers]

With Ertas Vault: Vault computes embedding similarity across all input pairs and automatically flags pairs above a configurable similarity threshold (default: 0.92 cosine similarity). You review the flagged pairs and decide which to keep.

Typical result: 5-15% of examples are near-duplicates in manually collected datasets. In synthetic datasets, this can be 15-25%. Remove all but one example from each duplicate group — keeping the highest-quality version.

Step 3: Label Consistency Check (1-2 hours)

This is the most impactful step. Inconsistent labels are the number-one source of training degradation, and they are invisible unless you look for them.

The problem: Different annotators (or the same annotator on different days) label similar inputs differently. The boundaries between categories are interpreted inconsistently. Edge cases get assigned to different categories depending on who labeled them and when.

How to check without code:

Method 1: Category review. Filter your spreadsheet by each category. Read 20-30 examples from each category. Ask yourself: "Do all of these clearly belong in this category?" Flag any that seem borderline or misplaced.

Method 2: Confusion pair analysis. For each pair of categories that might be confused (e.g., "billing_error" vs "refund_request"), filter for both categories and compare the examples. Are the boundaries between them clear and consistent?

Method 3: LLM-assisted review. This is the fastest approach. Paste 50-100 examples (input + assigned label) into a frontier model:

Review these labeled examples for consistency. For each example,
confirm whether the assigned label is correct or suggest a
correction. Pay special attention to:
- Examples that could belong to multiple categories
- Labels that seem inconsistent with similar examples
- Edge cases where the labeling guideline may be ambiguous

Format: Row [number]: [CORRECT / INCORRECT → suggested label] — [brief reason]

[paste examples]

Run this in batches across your entire dataset. Any example the LLM flags as incorrect gets a human review.

Typical result: 5-12% of examples have labeling inconsistencies. Fixing them typically improves model accuracy by 3-8% — the single highest-ROI activity in the entire fine-tuning pipeline.

Step 4: Distribution Analysis (30-45 minutes)

Your training data's distribution should approximate your production data's distribution. If it does not, your model will be poorly calibrated — overconfident on rare categories and underperforming on common ones.

What to check:

• Category distribution. Count examples per category. Calculate the percentage for each. Compare to your production distribution (or your best estimate of it).
• Input length distribution. Are your training inputs representative of what the model will see? If production inputs range from 20 to 800 words but your training data is all 100-200 words, the model will struggle with short and long inputs.
• Temporal distribution. If your data was collected over time, check whether the distribution shifted. Data from 6 months ago may reflect different patterns than current production traffic.

How to do it without code:

In a spreadsheet:

• Category counts: Use =COUNTIF(B:B, "category_name") for each category. Create a simple bar chart.
• Length distribution: Add a word count column: =LEN(TRIM(A2))-LEN(SUBSTITUTE(A2," ",""))+1. Create a histogram (Insert > Chart > Histogram).
• Compare to production: If you have production data, create side-by-side charts. Visually, the shapes should be similar. If one category is 5% of production but 25% of training data, you have a distribution mismatch.

What to do about mismatches:

• Overrepresented categories: Remove excess examples, prioritizing the removal of lower-quality or duplicate examples first.
• Underrepresented categories: Generate more examples for that category (synthetic data works well here) or collect more real examples.
• Input length mismatches: Explicitly generate examples at underrepresented lengths.

Typical result: Most manually collected datasets have 1-3 categories that are significantly over- or underrepresented. Rebalancing typically improves performance on the underrepresented categories by 10-20% with minimal impact on overrepresented ones.

Step 5: Outlier Removal (30-45 minutes)

Outliers in training data are examples that are dramatically different from the rest of the dataset. They are not necessarily wrong — they might be legitimate edge cases — but they can disproportionately influence training, especially in small datasets.

Types of outliers to look for:

• Extremely long or short inputs. If 98% of inputs are 50-300 words and two examples are 2,000 words, those outliers may dominate gradient updates during training.
• Off-topic examples. Inputs that somehow ended up in the dataset but do not belong to the task domain at all.
• Ambiguous examples. Inputs where even a domain expert cannot confidently assign the correct label. These are training noise, not useful edge cases.

How to find them without code:

• Sort by input length. Check the top and bottom 2-3% for anomalies.
• Sort by output length. Same check.
• For each category, sort alphabetically and scan the first and last few examples — outliers in content often sort to the extremes.

Decision framework: For each outlier, ask: "Will my model encounter inputs like this in production?" If yes, keep it (and consider adding similar examples to reduce its isolation). If no, remove it.

Typical result: 1-3% of examples are outliers worth removing. The impact is small individually but compounds with other cleaning steps.

Step 6: Final Human Review (1-2 hours)

The last step is a targeted human review of everything flagged in steps 1-5, plus a final random sample.

Review the flagged items: Go through every example you flagged in previous steps. Make a final decision: fix, keep, or remove.

Random sample check: Pull 30-50 random examples from the cleaned dataset. Read each one. If you find more than 1-2 issues in 50 examples (>4% error rate), another round of cleaning is needed. If 0-1 issues, your dataset is ready.

Document your decisions. Keep a brief log of the labeling rules you established, edge cases you resolved, and categories you clarified. This documentation is invaluable when you need to expand the dataset later — it ensures consistency between the original dataset and new additions.

Common Data Issues by Task Type

Classification Tasks

Most common issue: Mislabeled examples, especially at category boundaries. Two categories that overlap conceptually (e.g., "complaint" vs. "feedback") will have inconsistent labeling. Fix: Write explicit decision criteria for each category boundary. Apply criteria to all borderline examples.

Text Generation Tasks

Most common issue: Inconsistent output style. Some examples use formal tone, others casual. Some include headers, others do not. The model learns this inconsistency. Fix: Define a style guide for outputs. Rewrite examples that deviate. Even small formatting differences (period vs. no period at the end of outputs) matter.

Extraction Tasks

Most common issue: Missing fields. Some examples extract all target fields, others miss optional ones. The model learns that missing fields are acceptable. Fix: Decide which fields are required vs. optional. For required fields, every example must include them. For optional fields, use a consistent representation for "not found" (e.g., null vs. empty string — pick one).

Conversation Tasks

Most common issue: Inconsistent turn structure. Some conversations start with a greeting, others jump straight to the task. Some include system messages, others do not. Fix: Standardize conversation structure. Every conversation should follow the same template for turn order, role labels, and system message format.

Time Estimates

Here is what the full 6-step process actually takes, based on our experience with agency teams:

The accuracy improvement column shows the typical gain on a held-out test set compared to training on the uncleaned dataset. This is additional performance you get from cleaning alone — without changing the model, hyperparameters, or training approach.

When to Clean vs. When to Generate More

Clean first if:

• Your random audit found >5% issues
• You have not done any cleaning pass yet
• Your learning curve is flat (more data is not helping)
• You have category imbalances >2x

Generate more data if:

• Your data is already clean (>96% label accuracy)
• Your learning curve is still climbing at your current dataset size
• Specific categories are underrepresented and you cannot find more real examples
• You need coverage of input patterns not present in your existing data

In most cases, the right order is: clean first, then evaluate, then generate more if needed. Teams that skip cleaning and go straight to data collection end up spending 2-3x more time overall because they are collecting data that compounds existing quality problems.

The Bottom Line

You do not need a data science team to produce clean fine-tuning data. You need a systematic process, 2-6 hours of focused effort, and the willingness to read your own training examples carefully.

The 6-step process above — format validation, deduplication, label consistency, distribution analysis, outlier removal, final review — catches the issues that cause 80% of fine-tuning quality problems. It works with spreadsheets. It works with LLM-assisted review. It works better with Ertas Vault's built-in validation tools.

The model can only be as good as the data it learns from. Invest the time upfront.

---

Related Reading

• Data Quality > Data Quantity: Why 250 Good Examples Beat 10,000 Bad Ones — the evidence behind why quality matters more than volume for fine-tuning
• Fine-Tune AI Without Code: The No-Code Guide — end-to-end guide to fine-tuning for non-technical teams