Factual consistency
Overview
Factual consistency refers to two related but distinct properties of model-generated text: 1. Internal consistency: statements within a single output do not contradict each other. 2. External consistency (factuality): statements in the output do not contradict verifiable real-world facts.
In evaluation contexts, factual consistency most often refers to the external sense: the output agrees with a reference knowledge base, a set of source documents, or established ground truth. The term is used prominently in summarization evaluation, where a summary that contradicts or adds to its source document is factually inconsistent.
Factual consistency is a component of, but not synonymous with, faithfulness: faithfulness typically describes agreement between an answer and a specific provided source, while factual consistency can describe agreement with any external factual standard.
Measurement
Factual consistency in summarization is measured by:
- Natural language inference (NLI) classifiers trained to detect entailment vs. contradiction between a source and a summary.
- Metrics such as FactCC, DAE (Dependency Arc Entailment), and SummaC that apply sentence- or claim-level entailment checks.
- LLM-as-judge prompts that ask a model to identify claims in the output and verify them against the source.
For open-domain factuality (no specific source), measurement requires a knowledge base or fact-checking pipeline.
| Term | Scope |
|---|---|
| Factual consistency | Output-to-world or output-to-source agreement; no contradictions |
| Faithfulness | Answer agrees with a specific provided source context (RAG-focused) |
| Groundedness | Each claim in the answer is traceable to a source passage |
| Hallucination | Broad term; factual inconsistency with reality is one type |
| Accuracy | Correctness of specific facts; a stricter subset of consistency |
