How to evaluate knowledge-base retrieval
To evaluate retrieval in a knowledge base, prepare questions whose source passage is already known, then record whether each question returned that passage, whether the passage was a superseded version, and whether the reply was checked against the citation before it was sent. Re-chunk when the right file is already in the index and the split has cut the sentence apart. Re-index when the current version has not yet been stored in the index.
On this page
- Keep retrieval evaluation apart from building and versioning
- A question, its source, and an acceptable answer travel together
- Record hits and misses in the same log
- A superseded file is not a hit on the current text
- Check the citation against the passage before sending
- Re-chunking and re-indexing answer different misses
- A question the documents do not cover is answered as not found
- A question with no source is not part of the test
Keep retrieval evaluation apart from building and versioning
Building the knowledge base decides which documents may stand as the rule, who may read them, and where the current path is. Versioning decides which file is marked current and who may mark the previous file superseded. This evaluation asks a different question: given a question with a known source, did retrieval return that current passage. The three are not the same as "put the files in and the answers will be right". See How to build an enterprise knowledge base and How to manage document versions in a knowledge base.
Permissions stay with the groups the file store already uses. The account that runs the test must be allowed to read the folder the question belongs to. A miss from an account that cannot open that folder is not a retrieval failure. How roles meet folders is in How to set permissions for an enterprise AI assistant. The delivery scope sits under Internal knowledge base and assistant.
A question, its source, and an acceptable answer travel together
Read on 21 October 2026, the RAG preparation phase on Microsoft Learn says the output of gathering test queries has three parts: the query, a legitimate user's potential prompt; the context, the actual text in the documents that addresses the query, with the page and the text for each piece; and the answer, a valid response, either taken from the documents or rephrased from one or more pieces of context. The page shows no publication date. The questions have to be answerable from the content. The page also says subject-matter experts often cannot list every question at once, so synthetic questions may be generated from representative documents, and that a subject-matter expert reviews the output so the questions fit the use case and the answers address the questions.
The company's question set is those three fields: the question, the source (document, page, and text), and an acceptable answer. A question with no page and no text does not enter the set. A synthetic question is not used for acceptance until the owner has reviewed it. The page also says the customer or the subject-matter expert should be a quality gate for whether the documents, the questions, and the answers are comprehensive, representative, and correct, and that the body of questions and answers is reviewed periodically so it still reflects the source documents. How often is for the company to set. The page gives no number of days.
Record hits and misses in the same log
Read on 21 October 2026, the RAG information-retrieval phase on Microsoft Learn says search results are evaluated with two items gathered in preparation: the query, and the collection of all the text in the test documents that addresses it. The page names three methods. Precision at K is the percentage of correctly identified relevant items out of the total search results, and it focuses on accuracy. Recall at K, in the page's wording, is the percentage of relevant items in the top K out of the total possible relative items, and it focuses on coverage. Mean reciprocal rank is the average of the reciprocal ranks of the first relevant answer in the ranked results, and it focuses on where that first relevant result appears. The page shows no publication date.
The page says positive examples should score as close to 1 as possible, and negative examples — queries the data should not be able to answer — as close to 0 as possible. Every test query is run, and positive and negative results are averaged separately. The page states no other passing score. The working log can start with a column a person can read: whether the known passage appeared in the returned results, its rank if it did, and a miss if it did not. When a number is reported, it is calculated by one of the three methods, and the log says which K was used. K is written for this test. It is not copied from the page, because the page does not assign one.
A superseded file is not a hit on the current text
If the passage returned comes from a file already marked superseded, the question is a miss against the current text. The note says "returned the superseded version" and copies that version mark. A similar wording is not rewritten as a hit. Where the current mark lives, and who may mark the previous file superseded, stays with the versioning note. If a source in the question set still points at the old page, the questions and answers are updated so they again reflect the source documents, as the preparation page asks, and the test is run again.
A superseded file in the results is fixed by removing that file from what retrieval may return, not by blurring the question until a hit is easier. The owner confirms the current file on the store, then stores that file's chunks in the index. The old file is removed from retrieval, or marked not citable, on the same day as the version record. If the two dates differ, the next test will still treat the old sentence as authority.
Check the citation against the passage before sending
Read on 21 October 2026, the language-model end-to-end evaluation phase on Microsoft Learn describes groundedness, sometimes called faithfulness, as measuring whether the response is based completely on the context, and as validating that the response is not using information other than what exists in the context. The page says a low result indicates the language model might be outputting inaccurate or nonsensical responses, and that a low calculation is a reason to evaluate adding data, adjusting the chunking strategy or chunk size, or adjusting the prompt. The page shows no publication date, and it does not say a person must sign before the metric counts as met.
The check before sending is the knowledge base's own rule. The reply names the document and the passage. The owner opens that passage. The sentence is actually there before the reply leaves draft. If the passage does not contain it, the reply stays a draft and the log quotes the sentence that did not match. That is the same practice as attaching a path at build time and holding a reply that does not match. It is applied here to a question set that can be measured. See How to build an enterprise knowledge base. A metric can be calculated afterwards. It does not replace this check.
Re-chunking and re-indexing answer different misses
Read on 21 October 2026, the RAG chunking phase on Microsoft Learn describes chunking as breaking documents into appropriately sized chunks that each contain semantically relevant content. The page says passing entire documents or oversized chunks is expensive, may overwhelm the model's token limits, and does not produce the best results, and that passing information irrelevant to the query can produce inaccurate or unrelated responses. Chunks that are too small to address the query also produce poor outcomes, and relevant context that sits across several chunks may not be captured. The page shows no publication date.
The design and evaluation guide describes the flow as chunking, enriching chunks with metadata, embedding them, and persisting the chunks in the search index. When the current file's chunks have not been stored, or the index still holds only the superseded file, the current file is written into the index. Chunk size is not the first change. When the current file is already in the index and the returned chunk has cut the sentence away from the condition it depends on, that file is chunked again to follow the document's structure, then stored. The vector-search section of the retrieval page also says to run the same cleaning on the query as on the chunks before embedding, and to use the same embedding model that embedded the chunks. Part of that page is specific to Azure AI Search. The same-model sentence is stated in that section. A miss caused by a different embedding model is not repaired by cutting the paragraph shorter.
A question the documents do not cover is answered as not found
The preparation page says to gather both the questions the documents address and the questions they do not. During testing, and especially when testing the language model, the team decides how the system responds when it does not have enough context. The responses the page lists are to state that it does not know, or to state that it does not know and provide a link where the person might find more.
A negative question in the set expects "not found". If the system writes a sentence that is not in the context, the question is a miss, even when the sentence reads as complete. That sentence is not added to a document in order to justify the wrong reply. If the business does need a source, the owner of the topic writes it into the current document, the question set's source is updated, and the test is run again. Until then the reply stays "not found".
A question with no source is not part of the test
Every row in the question set has a document, a page, and the text. A question missing its source stays on the collection sheet and is not counted as a hit or a miss. A question with no acceptable answer stays there too, until the owner writes the answer and moves it into the set. It is not used to calculate a hit rate before that.
A passing score, a required number of questions, and a ranking of model names stay in the company's own record, with a date. The pages above do not say how many questions to write, and they do not compare vendors for accuracy. Before the set is used, three things are on the record: every question has a source and an acceptable answer, hits and misses record a superseded version when that is what came back, and a reply is checked against its citation before it is sent. The set is what you use to say whether retrieval met this test when those three are written down.