Active Learning Loops: Model-Assisted Labeling Without Data Egress

Data labeling is the most expensive stage of any enterprise AI pipeline. It requires domain experts — people who cost $80-200 per hour — to manually assign labels to hundreds or thousands of examples. A classification project with 10,000 documents and 15 categories can consume 400+ hours of expert time. At $120/hour, that is $48,000 in labor costs alone.

Active learning reduces that number by 75%. Instead of labeling every example from scratch, the model suggests labels and the expert confirms or corrects. The expert reviews 10,000 items instead of labeling 10,000 items — a fundamentally different task that takes a fraction of the time.

The catch is that traditional active learning pipelines send data to cloud-hosted models for the suggestion step. For enterprises handling sensitive documents — legal contracts, patient records, financial reports, classified materials — this creates a data egress problem. The documents leave the organization's infrastructure, even if only to get a label suggestion.

The solution: run the suggestion model locally. Ollama, vLLM, or any local inference server hosts the model on-premise. The active learning loop runs entirely within the organization's network. Zero data egress. Full efficiency gains.

How Active Learning Works

The concept is simple. Active learning is a feedback loop between a model and a human annotator, designed to maximize the information gained from each human decision.

Step 1: Start with a small labeled dataset. 50-200 examples, labeled manually by domain experts. This is the seed set.

Step 2: Train an initial model on the seed set. It will not be accurate — 50-65% is typical with this little data. That is fine. Accuracy is not the goal yet. Confidence calibration is.

Step 3: The model predicts labels for all unlabeled data. For each prediction, it also outputs a confidence score — how certain it is about the label.

Step 4: Present the predictions to domain experts, sorted by uncertainty (lowest confidence first). The expert sees the document, the suggested label, and the confidence score. They either approve the suggestion or correct it.

Step 5: Add the newly labeled examples (both approved and corrected) to the training set.

Step 6: Retrain the model on the expanded training set.

Step 7: Repeat from Step 3.

Each cycle, the model gets better. After 3-4 cycles, it typically reaches 85-92% accuracy on suggestions, meaning the expert approves 85-92% of labels with a single click and only needs to think carefully about the remaining 8-15%.

Why Uncertainty Sampling Matters

The key insight of active learning is that not all examples are equally informative. The model learns the most from examples it is least sure about — the ones near decision boundaries, the edge cases, the ambiguous documents.

Consider a document classification task with categories like "Contract," "Invoice," "Legal Opinion," and "Correspondence." After the first training cycle, the model might be 95% confident that an invoice is an invoice. Labeling that invoice teaches the model almost nothing — it already knew.

But a document that the model scores as 52% "Legal Opinion" and 48% "Correspondence" is genuinely ambiguous. When the expert labels it, the model learns exactly where the boundary between those categories lies.

Uncertainty sampling exploits this by always presenting the most uncertain examples first. The expert's time is spent on the hardest cases — the ones that matter most for model improvement — rather than on easy cases the model has already figured out.

The efficiency gain is dramatic. Random sampling (labeling examples in arbitrary order) requires approximately 4x more labeled examples to reach the same model accuracy as uncertainty sampling. Put differently, uncertainty sampling achieves the same accuracy with 75% less expert time.

The On-Premise Active Learning Loop

Here is the complete technical setup for running active learning without data egress.

Infrastructure

• Inference server: Ollama running a capable classification model. For text classification tasks, Llama 3.3 8B or Qwen 2.5 7B works well. These models run on a single GPU with 16GB+ VRAM.
• Training server: A machine with a GPU for fine-tuning. The same machine can serve double duty if you schedule inference and training at different times.
• Annotation interface: A web application where domain experts review suggestions. This can be as simple as a spreadsheet with approve/correct buttons, or a purpose-built tool like Label Studio running on-premise.
• Orchestration: A script that coordinates the loop — runs inference, sorts by uncertainty, presents to annotators, collects decisions, triggers retraining.

Cycle 1: The Seed Set

Domain experts manually label 100-200 examples. Select these examples to cover the full range of categories — at least 10 examples per category, more for ambiguous categories. Spend time on quality here. These labels propagate through every subsequent cycle.

Time estimate: 4-8 hours of expert time for 200 examples.

Cycle 2: First Active Learning Pass

Fine-tune the local model on the 200 seed examples. This takes 15-30 minutes for a 7B parameter model on a single A100.

Run inference on all unlabeled data. For 10,000 documents, inference takes 2-4 hours on a single GPU.

Sort predictions by confidence. Present the bottom 200 (lowest confidence) to the expert. The expert reviews each one: approve the suggested label or correct it. At this stage, expect 50-65% of suggestions to be correct — the expert is doing real work.

Time estimate: 3-5 hours for 200 reviews (faster than raw labeling because the expert evaluates rather than decides from scratch).

Cycle 3: Second Pass

Retrain on the expanded dataset (now 400 labeled examples). Run inference on remaining unlabeled data. Present the next 300 most uncertain examples.

At this stage, accuracy jumps. The model has seen the expert corrections from Cycle 2 and learned from them. Expect 70-80% of suggestions to be correct. The expert moves faster — most reviews are a quick "approve."

Time estimate: 3-4 hours for 300 reviews.

Cycle 4: Third Pass

Retrain on 700 labeled examples. Present 500 uncertain examples. Accuracy: 80-88%. Expert time: 3-4 hours for 500 reviews (because most are approvals).

Cycle 5: Final Pass

Retrain on 1,200 examples. Present the remaining uncertain examples (typically 500-1,000). Accuracy: 85-92%. Expert time: 3-5 hours.

After this cycle, auto-approve all predictions where model confidence exceeds 95%. For a 10,000-document dataset, this typically covers 6,000-7,000 documents that the expert never needs to see.

Total Expert Time

Without active learning: ~400 hours (labeling 10,000 documents at ~25 per hour).

With active learning: ~20-25 hours across 4-5 cycles, plus 8 hours for the seed set. Approximately 30 hours total.

That is a 92% reduction in expert time. Even using the conservative 75% benchmark, the savings are transformative.

Domain Expert Workflow

The domain expert should not need to touch a terminal, write code, or understand machine learning. Their interface should show:

1. The document (or relevant excerpt)
2. The suggested label
3. The model's confidence score
4. An "Approve" button and a dropdown to select a different label

That is it. No Python notebooks. No command-line arguments. No JSON editing.

The expert's job is domain judgment: "Is this label correct?" They bring the expertise. The system brings the efficiency.

For teams using Ertas Data Suite, this interface is built in. The active learning loop runs automatically — the system trains the model, sorts by uncertainty, and presents the annotation queue. The expert just opens the app and starts reviewing.

Quality Metrics

Two metrics tell you whether the active learning loop is working.

Inter-Annotator Agreement

If multiple experts are reviewing the same data, measure how often they agree. Cohen's kappa above 0.8 is strong agreement. Between 0.6 and 0.8, there are ambiguous categories that need clearer definitions. Below 0.6, the labeling guidelines need an overhaul before continuing.

Even with a single annotator, you can measure consistency by re-presenting 5% of already-labeled examples (randomly mixed into the queue) and checking whether the expert gives the same label. Consistency below 90% indicates fatigue or unclear guidelines.

Model Confidence Calibration

The model's confidence scores should be calibrated — when it says 90% confidence, it should be correct 90% of the time. If the model says 90% but is only correct 70% of the time, the uncertainty sampling is not working properly because the model does not know what it does not know.

Check calibration after each retraining cycle. Plot predicted confidence against actual accuracy in bins (0-10%, 10-20%, etc.). A well-calibrated model shows a diagonal line. An overconfident model shows high predicted confidence with lower actual accuracy. If the model is systematically overconfident, consider temperature scaling or label smoothing during training.

When to Stop

Active learning has diminishing returns. Each cycle adds less new information because the remaining unlabeled examples are increasingly similar to ones the model has already seen.

Stop when any of these conditions are met:

• Model accuracy plateaus: Two consecutive cycles show less than 1% accuracy improvement. The model has learned what it can learn from this data.
• Expert effort exceeds value: When the expert is approving 95%+ of suggestions, the remaining corrections are edge cases that may not justify the expert's time.
• Coverage is sufficient: You have labeled examples covering all categories, all edge cases, and all known ambiguities. Additional labels add volume but not variety.

For most enterprise classification tasks, 3-4 active learning cycles are sufficient. A fifth cycle rarely produces meaningful improvement.

Handling Edge Cases

Active learning surfaces edge cases naturally — they are the high-uncertainty examples that get presented to experts. This is one of its underappreciated benefits.

Without active learning, edge cases hide in the unlabeled data. The model encounters them in production, misclassifies them, and users report errors. With active learning, the model identifies these cases during preparation and an expert resolves them before deployment.

Document the edge case decisions. When an expert labels an ambiguous document, record the reasoning. "This document contains both invoice elements and contract language. Labeled as 'Contract' because the binding terms take precedence." These notes become the institutional knowledge that future annotators and model iterations build on.

The Economics

For an enterprise processing 50,000 documents per year across 3 classification tasks:

Without active learning: 3 tasks x 50,000 docs x 2 minutes per label = 5,000 hours of expert time. At $120/hour = $600,000/year.

With active learning: 3 tasks x ~30 hours per task = ~90 hours of expert time. At $120/hour = $10,800/year. Plus infrastructure costs of approximately $5,000/year for on-premise GPU time.

Total savings: approximately $584,000/year. The infrastructure pays for itself in the first week.

These numbers scale. Larger document volumes increase the savings because the active learning efficiency holds — the model still learns from a fixed number of expert-reviewed examples, regardless of how many documents remain in the auto-approve pool.

Further Reading

• Domain Experts Should Own Data Labeling — Why domain expertise matters more than annotation speed, and how to structure expert workflows.
• Optimize Local LLM Inference for Data Labeling — Technical guide to running inference efficiently on local hardware for labeling pipelines.
• Local LLM Data Labeling Without Egress — The broader case for keeping labeling workflows entirely on-premise.