High BLEU scores can co-exist with annotations that fail on idiom, register, and cultural reference. This article maps the specific failure modes our review process catches.
Introduction
You have a parallel Arabic-Chinese corpus. BLEU score is 0.85. You assume the corpus is production-ready. But many sentence pairs are semantically misaligned. The words match, but the meaning does not.
The BLEU score problem
BLEU measures n-gram overlap but misses idioms, register shifts, cultural references, and pronoun ambiguity.
Failure 1: Idiom mismatch
Arabic phrase for greeting someone returning from a trip gets translated literally to Chinese, sounding awkward and overly religious. BLEU score is 0.92 but meaning is wrong.
Failure 2: Register mismatch
Casual Gulf Arabic gets translated to formal Chinese. BLEU 0.88 but the game would feel unnatural.
Failure 3: Pronoun ambiguity
Arabic pronouns attached to verbs become ambiguous in Chinese. BLEU 0.95 but the reference is unclear.
Our alignment methodology
We use a multi-layer human review process: semantic equivalence check, register consistency check, cultural context check, and ambiguity resolution. Each pair receives a score from 1β5.
How this improves your models
Training on semantically-aligned data produces better idiom handling, better register consistency, better cultural awareness, and better pronoun resolution.
Conclusion
BLEU scores are useful for fast screening but not sufficient for production-quality alignment. Human review is essential.