The Five Dimensions of AI-Ready Data Quality: A Scoring Guide

The AI Data Quality Framework identifies five dimensions that determine whether a dataset is ready for AI training: Completeness, Consistency, Accuracy, Timeliness, and Relevance. This article provides the detailed scoring rubric for each dimension — the practical tool that turns abstract quality concepts into measurable, actionable assessments.

Each dimension is scored on a 1-5 scale. The descriptions below include concrete examples drawn from enterprise data preparation scenarios to make the scoring criteria tangible.

Dimension 1: Completeness

Completeness measures whether the dataset covers the full distribution of inputs the model will encounter in production. It is not about row count. A dataset with 100,000 examples that only covers 40% of expected input categories is less complete than a dataset with 5,000 examples covering 95% of categories.

Scoring Rubric

Score 1 — Critical gaps. Fewer than 40% of expected input categories, languages, or edge cases are represented. The model will fail on common production scenarios. Example: a multilingual customer support model trained only on English data, despite serving markets in four languages.

Score 2 — Major gaps. Coverage reaches 40-60% of expected categories. The model handles the most common cases but fails predictably on known scenarios. Example: a legal document classifier trained on contracts and briefs but missing regulatory filings, which represent 25% of production volume.

Score 3 — Adequate coverage. The dataset covers 60-80% of expected categories with at least some examples in each major category. Edge cases may be underrepresented. Example: a medical coding model that covers all major ICD-10 chapters but has thin coverage in rare disease categories.

Score 4 — Strong coverage. Coverage reaches 80-95% of expected categories. Remaining gaps are documented and accepted based on production frequency analysis. Example: a financial document extraction model covering all standard document types, with deliberate exclusion of handwritten forms (verified as under 2% of production volume).

Score 5 — Comprehensive coverage. Coverage exceeds 95% of expected categories, validated through production traffic analysis or domain expert review. Edge cases are explicitly represented. Example: a customer intent classifier where production log analysis confirms every intent category appearing more than 0.5% of the time has at least 50 training examples.

How to Measure

Run a distribution analysis comparing your training data categories against production traffic categories. The gap between these two distributions is your completeness deficit. Tools that profile datasets and flag underrepresented categories make this assessment faster than manual review.

Dimension 2: Consistency

Consistency measures whether similar inputs receive similar labels, annotations, or example outputs throughout the dataset. Inconsistency teaches the model ambiguity where there should be clarity, producing outputs that oscillate between conflicting patterns.

Scoring Rubric

Score 1 — Pervasive inconsistency. No annotation guidelines exist, or guidelines exist but are not followed. Inter-annotator agreement is below 60%. Example: a sentiment analysis dataset where the same product review appears three times with labels of "positive," "neutral," and "negative" from different annotators.

Score 2 — Frequent inconsistency. Annotation guidelines exist but are ambiguous on common edge cases. Inter-annotator agreement is 60-70%. Systematic disagreements exist across annotator groups. Example: a named entity recognition dataset where some annotators tag "New York City" as one entity and others tag "New York" and "City" separately.

Score 3 — Moderate consistency. Guidelines are clear for common cases. Inter-annotator agreement is 70-80%. Inconsistencies are concentrated in genuinely ambiguous cases. Example: a document classification dataset with clear rules for 80% of documents, but legitimate ambiguity in multi-topic documents that annotators handle differently.

Score 4 — High consistency. Guidelines address common edge cases explicitly. Inter-annotator agreement exceeds 80%. Remaining disagreements are tracked and resolved through adjudication. Example: a clinical NLP dataset where a lead annotator reviews all disagreements and adjudicated labels are fed back into training.

Score 5 — Rigorous consistency. Guidelines are versioned, edge cases are catalogued with canonical examples, and inter-annotator agreement exceeds 90%. Agreement is measured regularly, not once. Example: a legal annotation project with a 40-page guideline document, weekly calibration sessions, and automated consistency checks that flag deviations from established patterns.

How to Measure

Calculate inter-annotator agreement using Cohen's kappa (for two annotators) or Fleiss' kappa (for multiple annotators). For datasets without multiple annotators, sample 5-10% of examples and have a second reviewer independently label them. Agreement below 75% warrants guideline revision before proceeding.

Dimension 3: Accuracy

Accuracy measures whether labels, annotations, and example outputs are factually correct. This is the dimension most teams assume they handle well, and most teams overestimate.

Scoring Rubric

Score 1 — Unreliable. Error rate exceeds 15% on sampled review. Labels are frequently wrong, not just ambiguous. Example: an intent classification dataset where automated labeling produced systematic misclassifications — all "cancel subscription" requests labeled as "modify subscription" because the heuristic matched on the word "subscription."

Score 2 — Error-prone. Error rate is 10-15% on sampled review. Errors follow identifiable patterns, suggesting systematic problems in the labeling process. Example: a document extraction dataset where date fields are correctly extracted from US-formatted documents but systematically misparse European date formats (DD/MM vs MM/DD).

Score 3 — Acceptable. Error rate is 5-10% on sampled review. Errors are distributed randomly rather than following systematic patterns. Example: a customer support response dataset where occasional responses contain minor factual errors about product features, but no consistent bias.

Score 4 — Reliable. Error rate is 2-5% on sampled review. Remaining errors are in genuinely ambiguous cases where reasonable experts might disagree. Example: a legal clause classification dataset where accuracy has been validated by a domain expert review of a 10% sample, with errors concentrated in clauses that span multiple categories.

Score 5 — Verified. Error rate is below 2% on sampled review. Accuracy has been validated through domain expert review, and error analysis confirms no systematic biases. Example: a medical coding dataset where every example has been reviewed by a certified coder, disagreements have been adjudicated by a senior coder, and a final random sample audit confirms sub-2% error rate.

How to Measure

Sample at least 200 examples (or 5% of the dataset, whichever is larger) for expert review. Calculate error rate as the percentage of examples where the reviewer disagrees with the label. Stratify the sample across categories to avoid over-sampling common cases.

Dimension 4: Timeliness

Timeliness measures whether the data reflects current conditions. Unlike the other dimensions, timeliness degrades passively over time — a dataset that scored 5 on timeliness at creation may score 2 twelve months later without any change to the data itself.

Scoring Rubric

Score 1 — Obsolete. The data reflects conditions that have materially changed. Using it for training will produce a model that gives outdated or incorrect outputs. Example: a regulatory compliance model trained on pre-2025 EU AI Act guidance, missing the enforcement provisions that took effect in August 2025.

Score 2 — Aging. The data is 12-24 months old in a domain with meaningful change frequency. Some examples are still valid, but the dataset as a whole no longer reflects current conditions. Example: a product support model trained on documentation from two product versions ago, with 30% of feature descriptions no longer accurate.

Score 3 — Current with gaps. The majority of data reflects current conditions, but specific areas are outdated. Example: a financial analysis model where market data is current but regulatory references have not been updated to reflect recent enforcement actions.

Score 4 — Current. Data reflects conditions within the last 6 months. Known temporal dependencies have been audited. Example: a healthcare model where clinical guidelines referenced in training data have been cross-checked against the latest published versions, with updates applied where needed.

Score 5 — Continuously maintained. Data freshness is monitored and maintained through automated or scheduled processes. Temporal dependencies are tracked and flagged when source material changes. Example: a customer support model where training data is automatically flagged for review when the product changelog indicates feature changes affecting documented workflows.

How to Measure

Identify the temporal dependencies in your dataset: what external facts, regulations, product features, or market conditions does the data reference? Check each against current sources. The percentage of outdated references gives you a timeliness score.

Dimension 5: Relevance

Relevance measures signal-to-noise ratio at the dataset level. Every irrelevant example dilutes the training signal and forces the model to spend capacity learning patterns that will never appear in production.

Scoring Rubric

Score 1 — Mostly noise. More than 40% of examples are irrelevant to the target task. The dataset was likely assembled from a broad data dump without filtering. Example: a technical support model trained on the entire customer service transcript archive, including billing, sales, and general inquiries that represent 60% of volume but are outside the model's intended scope.

Score 2 — Significant noise. 20-40% of examples are irrelevant. The dataset was filtered but the criteria were too broad. Example: a contract analysis model trained on all legal documents, including court filings, correspondence, and memos that the model will never encounter in production.

Score 3 — Moderately relevant. 80-90% of examples are relevant to the target task. Some noise remains but does not dominate. Example: a code review model trained on pull request comments, where 15% of comments are social conversation ("nice work" or "thanks") rather than substantive review feedback.

Score 4 — Highly relevant. More than 90% of examples are relevant. Remaining irrelevant examples are borderline cases. Example: a clinical note summarization model where training examples are drawn from the target specialty, with a small number of cross-specialty referral notes included.

Score 5 — Precisely targeted. More than 95% of examples are directly relevant to the target task. The dataset has been curated with explicit inclusion and exclusion criteria. Example: a financial document extraction model where every training example matches the exact document types, formats, and content patterns expected in production, validated through production traffic sampling.

How to Measure

Sample 100-200 examples and classify each as "relevant," "borderline," or "irrelevant" to the target task. The percentage of relevant examples is your relevance score. If borderline examples exceed 15%, your task definition may need sharpening.

Using the Rubric in Practice

The most effective way to use this rubric is as a pre-training checklist. Before any fine-tuning run, score the dataset across all five dimensions. Record the scores. If any single dimension scores below 3, address that gap before training. If the composite score (average of all five) falls below 3.0, the dataset needs work.

Track scores across datasets and over time. Patterns will emerge: perhaps your organization consistently scores high on Accuracy but low on Completeness, suggesting your review processes are strong but your data collection strategy has blind spots. These patterns inform where to invest.

The rubric is also a communication tool. When a data engineering team tells stakeholders "the data is ready," a five-dimension score card provides evidence. When a model underperforms in production, the pre-training quality scores provide a diagnostic starting point. When evaluating data preparation tools and platforms — whether custom-built or commercial solutions like Ertas — the rubric provides objective criteria for comparison.

Data quality is not binary. It is multidimensional, measurable, and improvable. The scoring rubric makes that improvement systematic.