What is BLEU Score?

Definition

BLEU (Bilingual Evaluation Understudy) is an automated text evaluation metric originally developed for machine translation that measures the precision of n-gram matches between a generated text (candidate) and one or more reference texts. The score ranges from 0 to 1 (often expressed as 0-100), where 1 indicates perfect overlap with the reference. BLEU computes precision at multiple n-gram levels (unigram through 4-gram by default) and combines them using a geometric mean, then applies a brevity penalty to penalize outputs that are shorter than the reference.

Despite being designed for machine translation, BLEU has been widely adopted as a general-purpose text generation metric. It is commonly used to evaluate text summarization, paraphrasing, code generation, and dialogue systems. However, its limitations are well documented: BLEU measures surface-level lexical overlap, not semantic similarity, meaning a perfectly valid paraphrase with different word choices can receive a low BLEU score while a semantically incorrect text with similar vocabulary can score high.

In the LLM era, BLEU is increasingly supplemented or replaced by model-based evaluation metrics like BERTScore (which uses contextual embeddings for semantic similarity) and LLM-as-judge approaches (where a powerful model rates the quality of generated text). Nevertheless, BLEU remains widely reported because it is fast to compute, deterministic, reproducible, and well-understood — making it a useful baseline metric alongside more sophisticated evaluation approaches.

Why It Matters

Automated evaluation metrics are essential for rapid iteration during model development. Manually evaluating every model output is prohibitively slow and expensive, especially when comparing multiple training configurations. BLEU provides an instant, reproducible quality signal that correlates reasonably well with human judgment for many generation tasks.

For benchmarking and research, BLEU's deterministic nature makes it valuable for reproducible comparisons. Two teams evaluating different models on the same test set will get identical BLEU scores, enabling meaningful comparison. This reproducibility, combined with its decades-long track record, explains why BLEU continues to be reported alongside newer metrics in most NLP papers.

How It Works

BLEU computation involves several steps. First, n-gram precision is calculated for each n-gram size (1 through 4 by default). For each n-gram size, the count of n-grams in the candidate that appear in the reference is divided by the total number of n-grams in the candidate. A clipping mechanism ensures that each reference n-gram can be matched at most once, preventing inflated scores from repetitive output.

The modified precisions for each n-gram size are combined using a weighted geometric mean (equal weights of 0.25 by default). Finally, a brevity penalty is applied: if the candidate is shorter than the reference, the score is multiplied by exp(1 - reference_length/candidate_length). This penalty prevents the model from gaming the metric by producing very short, high-precision outputs. The final BLEU score is the product of the geometric mean precision and the brevity penalty.

Example Use Case

A team fine-tuning a model for customer email summarization evaluates model outputs against 500 human-written summaries. The base model achieves a BLEU-4 score of 18.3, while the fine-tuned model reaches 31.7 — a 73% improvement that correlates with human evaluators' preference for the fine-tuned model's summaries. They also compute BERTScore and run LLM-as-judge evaluation to confirm the improvement, using BLEU as a quick sanity check during rapid iteration.

Key Takeaways

• BLEU measures n-gram precision overlap between generated and reference texts, scored from 0 to 1.
• It is fast, deterministic, and reproducible, making it a useful baseline evaluation metric.
• BLEU captures lexical similarity but not semantic meaning — valid paraphrases can score low.
• The brevity penalty prevents gaming by producing short, high-precision outputs.
• Modern evaluation combines BLEU with semantic metrics like BERTScore and LLM-as-judge approaches.