How to choose chunk boundaries for a knowledge base
Chunk boundaries for a knowledge base follow how fixed the document's structure is, and the record states whether headers, footers, and watermarks come out, whether a table stays with its headings, and what the fixed length and the overlap are. Whether retrieval later returns that chunk is measured in a separate note. What is set here is the boundary of each chunk before it enters the index.
On this page
- Keep chunking apart from setup, versions, and retrieval checks
- Start from how fixed this kind of document is
- Mark headers, footers, and watermarks as outside the chunks
- Keep a table and its headings in the same chunk
- Write the fixed length together with the overlap
- Split on sentences only when the other approaches do not fit
- Changing the boundary after it is set moves the index
- Write a boundary for each kind before it enters the index
Keep chunking apart from setup, versions, and retrieval checks
Setting up the knowledge base decides which documents may stand as the rule, who may read them, and where the current path is. Versioning decides which file is marked current. A retrieval check asks whether a question with a known source returned that passage. Chunking decides the boundary used to split one current file before it enters the index. The four are not folded into putting the files in and seeing what comes back. See How to build an enterprise knowledge base, How to manage document versions in a knowledge base, and How to evaluate knowledge-base retrieval.
The file that is split is the one marked current on the store. A superseded version is not split again and then sent into the index. The delivery scope sits under Internal knowledge base and assistant.
Start from how fixed this kind of document is
Read on 22 October 2026, the RAG chunking phase on Microsoft Learn says chunking breaks a document into pieces of a suitable size, each holding semantically related content, and that the approach follows the document type and its structure. The page gives no single fixed rule. It orders structure from high to low as a fixed layout, semi-structured, structure that can be inferred, and little or no structure, and it says the degree of structure is where the choice starts. The page shows no publication date.
A document with a fixed layout puts the same field in the same place every time. Examples are a tabular tax form and an insurance card. The page says this kind of complex layout is a poor fit for basic text splitting alone, and the usual approach is a model that extracts the data and marks its meaning. A semi-structured document has no fully fixed layout, but the fields are broadly the same. An invoice is the example: the layout varies, and an invoice number and a bill-to heading are still expected. A web page commonly carries markup such as a title and paragraphs. A document whose structure can be inferred has no markup language, but a reader can see numbering and lists. Regulations and manufacturing specifications are the examples. The page says to write code that splits on that structure, and whether a separate model is worth building depends on how many documents of that kind there are. For notes with almost no structure, the page says sentence-based splitting, or a fixed length kept with an overlap, is the closer fit. The knowledge base marks one of those four on the chunking record for each kind of document. The four are not collapsed into one length.
Mark headers, footers, and watermarks as outside the chunks
The same page says loading and chunking can be kept apart. During loading, parts that later analysis should ignore can be marked or removed. Watermarks, headers, and footers are the examples. Tables can be reformatted, and the document's structure can be taken from headings, subheadings, and other structural elements. The page says that when sensitive data cannot be sent to an outside system, the work is done locally. The ignore list is written on the chunking record for that kind of document: headers, footers, watermarks, and copyright lines, and which of them are deleted and which are only annotated and kept.
Without that list, a page number or a disclaimer becomes a chunk of its own, and retrieval can return the page number while missing the clause. The owner of the document signs the ignore list. The date is written beside the date of the current version. Until that signature is there, this kind of document does not enter the index.
Keep a table and its headings in the same chunk
The chunking page lists reformatting tables among the things the loading step can do. Read on 22 October 2026, the RAG preparation phase on Microsoft Learn says, on tables, that a long table may need its headers repeated when it is split into chunks. The page shows no publication date. The knowledge-base rule is that a table and its column headings stay in one chunk. When a table is long enough that it has to be split, every chunk repeats the column headings and carries the name of the table.
The chunking page also says that a chunk too small to answer the query gives a poor result, and that related context spread across several chunks may not all come back. A heading and the first row landing in two chunks is that kind of split. If an image is kept, the page says it can be replaced at load time with a text description, and useful text around the image is passed into that description. If the description is split again, each chunk still has to point at the original image. An image with neither a description nor a caption does not become a chunk on its own.
Write the fixed length together with the overlap
The chunking page describes a fixed length with overlap as a split on a fixed number of characters or tokens, with some characters overlapping between chunks. It says this approach shares much of what sentence-based splitting gains and loses, and that one chunk can hold meaning that runs across several sentences. Both the length and the overlap have to be chosen. Results differ by document type, and the page suggests looking, visually, at where the cuts actually fall. It also says a fixed length should use BERT tokens rather than a character count, because a BERT token is a meaningful unit of the language and keeps more of the sense than a count of characters. The page gives no universal token count.
The chunking record writes two numbers for this kind of document: how many tokens each chunk holds, and how much neighbouring chunks overlap. A sample page is attached, marked to show whether a cut falls on a full stop, inside a table, or in the middle of a heading. A cut through the middle of a heading means the length changes, or the split goes back to the structure. That length is not applied to another kind of document. The page describes fixed length as fitting prose with little structure, or a collection with so many document types that a separate strategy for each one is impractical. A policy or a contract that already has headings is not moved onto that same length merely because of that case.
Split on sentences only when the other approaches do not fit
The chunking page describes sentence-based splitting as chunks made of complete sentences, and as the fallback when the other approaches on that page do not fit. It says the effort to implement this approach, and the cost of processing it, are relatively low, and that it can be used on text that contains prose or complete sentences. The opening of the same page says the labels it puts on engineering effort and processing cost are subjective, and exist only so the approaches on that page can be compared with one another. Relatively low is not read here as a price, and tool brands are not ranked.
The drawback the page states is that one chunk may not hold the whole context of an idea, and that the sense often has to be read across several sentences. The record for sentence splitting therefore says whether neighbouring sentences may be joined into one chunk, and at which heading that joining stops. A mechanical split on full stops, with no look at the headings, puts a clause's condition and its conclusion in two chunks. The page does not name a sentence splitter that must be used. The record writes the rule used this time. It does not rank tools.
Changing the boundary after it is set moves the index
The chunking page says the chunking approach is a semipermanent choice in the overall design. Changing the strategy can affect what comes after it in a material way, and the change has to be carried through the rest of the workflow. Before the approach is adopted, it is checked against the document type. The knowledge base therefore versions which boundary this kind of document uses, and that version points at the version of the current file. On the day the boundary changes, the old chunks leave the index, the current file is split again on the new boundary, and the new chunks are written in.
The retrieval note is what checks whether the new boundary has cut a known passage apart. The questions, the sources, and the hit log are not rewritten here. See How to evaluate knowledge-base retrieval. If the boundary is not yet on the chunking record, a hit rate from the question set does not stand in for the decision.
Write a boundary for each kind before it enters the index
A kind of document with no structure class, no ignore list, no rule for tables, and no choice between splitting on structure and a fixed length plus overlap, does not enter the index. A fixed length with the overlap left blank does not enter either. If the sample page does not show where the cuts fall, the boundary is not yet set.
A token count, a tool price, and a ranking of model names stay in the company's own record and on the quote for this job, with a date. The chunking page does not assign one length for every document, and it does not declare one approach fit for all of them. Before the index is opened, three things are on the chunking record: the structure of this kind of document is marked, the rules for headers, footers, and tables are signed, and the length and overlap, or the structural boundary, is written as a version. The current file is written into the index on that boundary when those three are on the record.