Domain-specific LLM evaluation measures whether a model is good enough at one field's work, on examples drawn from that work, scored under rules fixed before anyone sees a result. A sound evaluation has four parts: a held-out set from real work, a frozen protocol, pass thresholds declared in advance, and a confidence interval on every comparison. This guide works through each part using the method of our QMSum study (arXiv 2609.25028), where we rescored or generated 15 summarization systems under one protocol.
QMSum is a public benchmark for query-focused meeting summarization. Its conclusions are limited to that benchmark and to automatic metrics. This guide carries the method to other fields, and the study's numbers stay attached to QMSum.
What is domain-specific LLM evaluation?
Domain-specific LLM evaluation is testing a model on the inputs, outputs and failure cases of one field, against references a field expert would accept. It answers a narrower question than a leaderboard: whether this model, on this work, clears the bar your team needs.
Public domain benchmarks exist, and they set a useful floor. LegalBench (2023) collects 162 legal reasoning tasks "designed and hand-crafted by legal professionals." FinanceBench (2023) holds 10,231 questions about publicly traded companies; on a 150-case sample, its authors report that GPT-4-Turbo with a retrieval system "incorrectly answered or refused to answer 81% of questions." Both were read at source on 29 October 2026. A public benchmark tells you how models handle a field in general. Your own held-out set tells you how one model handles your work.
Why do general benchmarks miss domain performance?
General benchmarks sample broad knowledge, and a business's work has its own inputs, formats and edge cases. Raju et al. built their domain-specific evaluation sets (2024) because the general frameworks they reviewed lacked "diversity across domains such as law, medicine, and multilingual contexts."
The input shape matters as much as the subject. In the QMSum study, a released specialist summarizer lost 6.30 ROUGE-1 when the same meetings reached it as 2,000-word retrieved spans instead of the long input it was trained on, and training it on the span inputs recovered the loss. A model scored on clean benchmark inputs can fall short on the inputs your pipeline actually produces, so evaluate on the inputs as served.
How do you build a domain evaluation set?
Build it from real work, check it with field experts, and split it before any model sees it.
- Draw examples from production. Logs, tickets, documents and past decisions, in the shape your system receives them. Our guide to building an eval set from real conversations covers sourcing and labelling.
- Cover the mix and the hard cases. Match the share of each task type to its share of traffic, then add the rare cases that cost the most when wrong.
- Have field experts write or check the references. In a specialised field, the reference answer is the expensive part, and the evaluation is only as good as it.
- Split into validation and test. Validation drives every choice you make while building. The test split is scored once per system you report.
- Keep it out of training. Any example that reaches a training set stops measuring anything.
How big does a domain evaluation set need to be?
Big enough that the difference you care about is larger than the noise, which is usually bigger than teams expect. The QMSum paper measured this directly on a 281-query test split.
| Sample | 95% interval width, ROUGE-1 |
|---|---|
| 50 queries drawn from the test split | About 5.6 points |
| All 281 test queries, paired comparison | About 1.8 to 2.8 points |
At 50 examples, chance alone could reorder most adjacent systems in the study's results table. On the full split, differences below about one ROUGE-1 point were undetectable, and the paper calls that the benchmark's detection floor. Bootstrap your own scores to find the floor of your own set, then size the set to the smallest difference you need to see.
What does a frozen protocol mean?
A frozen protocol fixes every setting that can move a score before the first comparison is run, so results reflect the models and leave the evaluator's choices out. In the QMSum study, a committed module fixed the random seed, the generation length cap, the decoding method, the retrieval settings and the prompt. A test in the released code checks the prompt byte for byte, so a drifted copy fails before it can produce a result.
Two rules kept the test split honest. Validation drove every selection, and each reported system touched the test split once. The promoted system departed from the frozen protocol in one place, its retrieval window and span budget, and the paper labels that deviation and reports both the protocol-exact row (33.39 ROUGE-1) and the promoted row (35.41).
For your own evaluation, write the protocol down, version it with the evaluation set, and treat any change to it as a new protocol.
How should pass thresholds be set?
Declare them before scoring: the metric, the margin a new model must clear, and the regressions that block it regardless of the average. A threshold set after seeing results bends toward whatever the results were.
Set each margin above the noise you have measured. In the QMSum study, changing only the training seed moved validation ROUGE-1 by 1.59 points, enough on its own to erase one configuration gain the paper had measured. A pass margin smaller than that range would let a lucky training run through. A useful default is a margin above both your set's detection floor and your seed range, with a separate regression set of past failures that must still pass.
How do you know a difference is real?
Report a confidence interval on every comparison, and treat an interval that crosses zero as unresolved. The QMSum paper uses a paired bootstrap: resample the test queries with replacement 10,000 times, score both systems on the same resample, and take the 2.5th and 97.5th percentiles of the difference. Pairing cancels the shared difficulty of each query, so the interval is tighter than comparing two separate averages.
Three refinements matter in practice:
- Cluster the resampling when examples share a source. QMSum's 281 test queries come from 35 meetings, and resampling whole meetings widened the intervals. A smaller specialist scored 36.33 ROUGE-1 against our system's 35.41, and the meeting-cluster interval for that difference, -0.27 to +2.22, crosses zero, so the benchmark cannot separate the two.
- Add training variance by hand. A bootstrap over queries says nothing about how a different training seed would score, which is why the seed range above belongs beside every margin.
- Withdraw claims that fail the test. An earlier +0.96 ROUGE-1 margin against a hosted model was dropped from the paper when its interval, -0.30 to +2.24, crossed zero.
What should a domain evaluation measure?
Measure the outcome the business needs, then check that the metric cannot be gamed by length or style. The QMSum study shows why. Five hosted proprietary models ran zero-shot under one concise prompt, and under a reference-overlap scorer a released specialist led every one of them by at least 6.2 ROUGE-1. The paper limits that ordering: the hosted models wrote longer outputs, and no human or factuality evaluation was run. A reference-overlap score rewards matching the reference's length, so an evaluation of open-ended answers needs a length check and an expert or factuality review beside it.
LLM judges carry their own biases. Zheng et al., in "Judging LLM-as-a-Judge" (2023), report "position, verbosity, and self-enhancement biases", alongside over 80% agreement with human preferences for strong judges. EvalSense (February 2026) adds "automated meta-evaluation tools that assess the reliability of different evaluation approaches using perturbed data", tested on clinical notes. Check a judge against expert labels on your own examples before trusting it.
| What to measure | How to score it |
|---|---|
| Task accuracy | Exact or field-level match |
| Answer or summary quality | Reference metrics plus expert review |
| Factual errors | Checks against source documents |
| Output format | Schema validation |
| Output length | Ratio to reference length |
| Regressions | A fixed set of past failures |
| Cost per task | Tokens per task times price per token |
The last row belongs in the evaluation because it decides whether a model that passes is worth running. A trained model lowers cost on two fronts, a lower price per token and fewer tokens per task, and the cost-per-task calculator puts both beside the quality result.
How does an evaluation gate a model release?
Once the set, protocol and thresholds are fixed, the evaluation becomes the gate every new model version has to pass. A candidate is scored on the frozen set, compared with the model in service using paired intervals, checked against the regression set, and then reviewed by the people who own the work. Continued training can quietly erode older skills, the problem covered in our post on catastrophic forgetting during fine-tuning, and the regression set is what catches it.
Ertas is an applied AI lab, and the adaptive model endpoint runs this gate in production. A model trained on your task is served through an OpenAI-compatible API. New versions train on your agents' traffic in the background, are scored against your evaluations, and go live only with your sign-off. If a new version falls short, the current model stays in service. It runs Ertas-managed, in your own cloud or on-premise. The endpoint is currently invitation only, and we are developing a more streamlined version for wider access.
Frequently Asked Questions
What is an LLM evaluation framework?
An LLM evaluation framework is the fixed combination of an evaluation set, a scoring method, a protocol and a decision rule that together say whether a model is good enough. Open-source tools such as EvalSense provide parts of one. The protocol and the thresholds are yours to set.
How many examples do you need to evaluate an LLM?
Enough that the difference you need to detect exceeds the noise. In the QMSum study, 50 examples gave a 95% interval about 5.6 ROUGE-1 points wide, and even 281 left differences below about one point undetectable. Bootstrap your own scores to find your floor.
Can you use an LLM as a judge for domain evaluation?
Yes, after checking it. Strong judges agreed with human preferences over 80% of the time in Zheng et al.'s 2023 study, and the same study found position, verbosity and self-enhancement biases. Compare the judge with expert labels on your own examples first.
Should you use public domain benchmarks?
Use them as a floor and a sanity check. LegalBench, FinanceBench and similar sets show how a model handles a field in general. Your own held-out set, built from your work, decides whether a model is ready for it.
How often should a domain evaluation be re-run?
On every new model version, and whenever the work itself changes. Add new failure cases to the regression set as they appear, and treat a change to the protocol as a new baseline.
The Short Version
Build the evaluation set from your real work, check it with field experts, split it before training, and freeze the protocol. Declare pass margins above the noise you have measured, report intervals on every comparison, and call crossing intervals unresolved. Check that your metric rewards the right thing, and put cost per task beside quality. To see how a domain model is built before it is evaluated, read what a domain-specific LLM is.
Sources, read 29 October 2026: Guha et al., "LegalBench" (2023); Islam et al., "FinanceBench" (2023); Raju et al., "Constructing Domain-Specific Evaluation Sets for LLM-as-a-judge" (2024); Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (2023); Dejl et al., "EvalSense" (2026). Our research: the QMSum study, its paper and its code.