Evaluation layers
- Retrieval: did the system find the right evidence?
- Grounding: is the answer supported?
- Citation: can users verify claims?
- Safety: does it refuse unsupported questions?
- Operations: is it fast, affordable, and monitored?
Evaluation
Measure whether your RAG system is actually useful
RAG quality is not just answer fluency. It includes retrieval quality, source support, citations, refusal behavior, cost, latency, and maintainability.
Retrieval recall
Did the system find the evidence needed to answer?
Use labeled questions with expected source passages or expert-reviewed relevant documents.
Retrieval precision
Were the retrieved passages actually relevant?
Review top results, track irrelevant chunks, and compare before/after chunking or reranking changes.
Answer faithfulness
Is the answer supported by the retrieved context?
Check generated claims against sources with human review and RAG-specific evaluation tools.
Citation usefulness
Can users verify important claims quickly?
Inspect whether citations point to the right source, page, section, or passage.
No-answer behavior
Does the system refuse or ask for clarification when evidence is missing?
Test ambiguous, adversarial, and out-of-scope questions.
Operational quality
Is the system fast, affordable, monitored, and maintainable?
Track latency, cost, error rates, trace coverage, data freshness, and update success.
Evaluation workflow
- 1.Create representative questions
- 2.Label expected sources
- 3.Run retrieval-only tests
- 4.Review generated answers
- 5.Score citations and faithfulness
- 6.Inspect traces and failures
- 7.Fix ingestion, retrieval, prompts, or sources
- 8.Repeat before release
The RAG Triad
Use the triad as a diagnostic model, not as a guarantee of truth. A system can be grounded in a knowledge base that is itself incomplete, stale, or incorrect.
Context relevance
Are the retrieved passages relevant to the user's question?
Low scores usually point to corpus, parsing, chunking, query, filtering, or ranking problems.
Groundedness
Is each important answer claim supported by the retrieved context?
Low scores usually point to noisy context, conflicting evidence, weak instructions, or unsupported generation.
Answer relevance
Does the final response directly and usefully answer the original question?
Low scores can occur even when the answer is grounded, for example when it is evasive, incomplete, or overly verbose.
Failure diagnosis map
Start at the earliest failing stage. Improving the generator cannot repair a missing document or a damaged table.
Source: The authoritative answer is absent, stale, duplicated, contradictory, or inaccessible.
Inspect: Inspect corpus coverage, ownership, dates, permissions, and superseded documents.
Improve: Curate sources before changing models or retrieval settings.
Parsing and chunking: The answer exists but tables, headings, pages, or relationships were damaged during ingestion.
Inspect: Compare parsed chunks with the original document visually.
Improve: Use format-aware parsing, structure-aware chunks, and preserved provenance.
Retrieval: Relevant evidence is not returned or is buried under irrelevant passages.
Inspect: Evaluate expected-source recall and relevance of each top result.
Improve: Tune chunking, hybrid search, metadata filters, query rewriting, and reranking.
Context assembly: Good evidence is truncated, duplicated, ordered poorly, or mixed with conflicting sources.
Inspect: Log the exact context sent to the model.
Improve: Deduplicate, prioritize authority and freshness, compress carefully, and expose conflicts.
Generation and reasoning: The model ignores, misreads, overgeneralizes, or invents beyond the supplied evidence.
Inspect: Break the answer into claims and map each claim to supporting context.
Improve: Improve instructions, require claim support, reduce noise, or use a better-suited model.
Citation and presentation: The answer may be supported, but users cannot verify which source supports which claim.
Inspect: Test citation correctness, granularity, links, page numbers, and source snippets.
Improve: Preserve provenance end to end and design citations as a verification workflow.
Example evaluation set
A useful evaluation set should include normal questions, hard questions, exact-reference questions, ambiguous questions, and questions the system should not answer.
Known-answer question: the source contains one direct answer.
Multi-source question: the answer requires comparing two approved sources.
Exact-reference question: the query contains a regulation number, policy code, or product identifier.
Ambiguous question: the system should ask for clarification.
Out-of-scope question: the system should refuse or redirect.
Stale-source question: the system must prefer the current document over older copies.