The Data Preparation Gap: Why ML Teams Spend 80% of Time Before Training Starts

The statistic is almost a cliché at this point: ML teams spend 60-80% of their time on data preparation. It's been cited in surveys, blog posts, and conference talks for over a decade. And yet — nothing has changed. The percentage hasn't moved.

This persistence deserves examination. Are ML teams inefficient? Are they over-engineering their data pipelines? Or is there something structural about how the industry approaches data preparation that guarantees this outcome?

The answer is structural. And the fix isn't better ML engineers — it's purpose-built tools that address the root causes.

Why the Percentage Hasn't Moved

Cause 1: Fragmented Tooling

The data preparation workflow in most enterprises involves 3-7 disconnected tools:

1. Parsing: Docling, Unstructured.io, Marker, or custom scripts
2. Cleaning: Python scripts using Pandas, custom deduplication logic
3. Labeling: Label Studio, Prodigy, or Argilla
4. Quality scoring: Cleanlab, custom validation scripts
5. Augmentation: Distilabel, custom synthetic generation
6. Export: Another Python script to format output

Each tool has its own:

• Installation and setup process
• Data format requirements (input and output)
• Learning curve and documentation
• Update cycle (breaking changes happen)
• Logging approach (or lack thereof)

The integration between these tools is custom Python code — the "glue scripts" that nobody wants to write, nobody wants to maintain, and nobody wants to debug when they break.

This fragmentation multiplies effort at every boundary. Format conversion, error handling, data validation, and audit trail continuity all require engineering time that doesn't directly improve data quality.

Cause 2: Domain Expert Exclusion

The people who know whether data is correctly labeled — doctors, lawyers, engineers, accountants — typically can't use ML data preparation tools. Label Studio requires a Docker deployment. Prodigy requires Python. Cleanlab is a Python library.

This creates a bottleneck: domain experts have the knowledge, ML engineers have the tool access, and the handoff between them degrades both speed and quality.

The typical flow: ML engineer extracts data, formats it for the labeling tool, explains the labeling schema to the domain expert (who uses the tool under supervision or provides labels through a spreadsheet), ML engineer imports the labels, runs quality checks, and iterates. Every handoff adds latency and error potential.

If domain experts could label data directly — without Docker, Python, or an ML engineer as intermediary — the labeling phase would take a fraction of the current time.

Cause 3: No Audit Trail Architecture

Most data preparation pipelines have no built-in audit trail. When something goes wrong — mislabeled data, missing records, quality degradation — debugging requires manually tracing through multiple tools and custom scripts to find where the issue originated.

Without audit trails, quality issues are discovered late and expensive to fix. Teams spend time re-processing data that was already processed incorrectly. They re-label records that were lost during cleaning. They re-run entire pipeline stages because they can't identify which specific records were affected by a bug.

In regulated industries, the audit trail isn't just a debugging convenience — it's a compliance requirement. Teams in healthcare, legal, and finance spend additional time manually documenting their pipeline steps to satisfy regulatory requirements that an integrated system would handle automatically.

Cause 4: Data Prep Is Treated as a Side Task

In most organizations, data preparation is treated as a preliminary step on the way to the "real work" of model training. It doesn't have dedicated headcount, dedicated tooling budget, or dedicated project management.

ML engineers are hired to build models. They're evaluated on model performance. They're excited about architecture innovations and training techniques. Data cleaning is the unglamorous work they have to do before they can do what they were hired for.

This structural undervaluation leads to underinvestment:

• Teams don't buy dedicated data preparation tools (they'll "just use Python scripts")
• Domain expert time for labeling isn't formally allocated
• Data quality metrics aren't tracked or reported
• Data pipeline maintenance is nobody's primary responsibility

Cause 5: Complexity Is Irreducible (But Could Be Better Managed)

Some of the data preparation effort is genuinely irreducible:

• Document format diversity is real — enterprises have dozens of formats
• Domain expertise requirements are real — only specialists can label correctly
• Compliance requirements are real — audit trails and privacy protections take effort
• Data quality variation is real — raw enterprise data has genuine quality issues

But irreducible complexity doesn't mean the current approach is optimal. Much of the 60-80% isn't spent on the hard problems — it's spent on integration, format conversion, tool maintenance, and working around tooling limitations.

What Would Actually Fix This

The data preparation gap won't close by hiring more ML engineers or working faster. It'll close when the structural problems are addressed:

1. Unified Platforms

Replace the 3-7 tool chain with a single platform that handles ingestion, cleaning, labeling, augmentation, and export. Not because each tool's individual capability is exceeded, but because the integration cost is the largest efficiency drain.

2. Domain Expert Access

Build data preparation tools that domain experts can use directly — native desktop applications with visual interfaces, not Python libraries and Docker containers.

3. Built-In Audit Trails

Make logging automatic and comprehensive. Every transformation, every label, every quality decision recorded without manual documentation effort.

4. Data Prep as a First-Class Function

Treat data preparation as a core capability, not a preprocessing step. Dedicated tools, dedicated time, dedicated quality metrics.

Ertas Data Suite is built on these principles: a unified platform covering all five pipeline stages, accessible to domain experts via a native desktop interface, with automatic audit trails and compliance documentation. The 60-80% statistic persists because the tools haven't changed. When the tools change, the numbers will follow.