Chunking Strategies for Embeddings: What Actually Changes Retrieval Quality

You will decide how to split your documents for embedding, and learn how to test whether that decision helped retrieval. This article compares fixed-size, recursive, semantic, and structure-aware chunking: the failures each prevents and creates, and the cost each adds to indexing.

Why chunking determines what retrieval can find

Your embedding model never sees your document. It sees a chunk. Your retriever ranks chunks, and your generator answers from whichever chunks you return.

Split in the middle of a definition, and neither half may embed close enough to the query that needs it. Keep a section together when it covers three unrelated topics, and its vector sits between those topics and may rank well for none of them. Bury the answer in unrelated paragraphs, and retrieval can succeed while generation still struggles.

Tokens, context limits, and granularity

Embedding APIs express limits in tokens, not characters or words, and token counts vary by language and content type. Code, tables, and dense notation tokenize differently from prose, so a splitter configured in characters can be under the limit for one document and over it for another. Most models truncate or reject over-limit input, depending on the provider. Truncation is worse because it is silent: you store a vector for only the start of a chunk and never learn the rest was discarded.

Treat size as two decisions. A maximum size protects you from context limits and overly broad vectors. A target granularity determines what one result means: ideally one complete idea, procedure, definition, or record that stands on its own.

Smaller chunks are more precise for narrow queries, but they lose context, multiply index size, and force multi-step answers to be assembled from fragments. Larger chunks preserve context, but they dilute the embedding and waste generator context. A FAQ entry, a legal clause, and an API reference section each want a different unit. Respect the natural unit of your corpus first; enforce a size ceiling second.

Fixed-size chunking: simple, predictable, and blind

Fixed-size chunking splits text into spans of a set length, regardless of meaning or structure.

def chunk_fixed(text: str, chunk_size: int = 800, overlap: int = 100) -> list[str]:
    """Split text into fixed-size character spans with overlap.
 
    Illustrative example only: production code should count tokens
    with your embedding model's tokenizer instead of characters.
    """
    if chunk_size <= overlap:
        raise ValueError("chunk_size must be greater than overlap")
 
    chunks: list[str] = []
    step = chunk_size - overlap
    for start in range(0, len(text), step):
        chunk = text[start : start + chunk_size].strip()
        if chunk:
            chunks.append(chunk)
        if start + chunk_size >= len(text):
            break
    return chunks

You get determinism, simplicity, and predictable index size, and you can reproduce a chunk exactly from its offsets.

The tradeoff is blindness. A fixed boundary does not know it landed in the middle of a sentence, code block, list, or argument. You will split prerequisites from their steps, separate a heading from the text it labels, and cut tables into rows that no longer explain themselves.

Use it when your content has no usable structure — normalized transcripts or log-like text, for example — or as a baseline. Do not make it the default for documentation just because it is easy: ease of implementation is not a retrieval property.

Recursive chunking: respect boundaries in priority order

Recursive splitting tries separators in order: the blank line between paragraphs first, then line breaks, sentences, words, and finally characters when a piece is still too large.

from langchain_text_splitters import RecursiveCharacterTextSplitter
 
splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=150,
    length_function=len,
    separators=["\n\n", "\n", ". ", " ", ""],
)
 
chunks = splitter.split_text(document_text)

The same class is available in TypeScript from @langchain/textsplitters, with chunkSize and chunkOverlap options. Recursive splitting does not understand meaning, but sentences usually stay whole, paragraphs usually stay together, and oversized sections degrade gradually instead of being cut at an arbitrary offset.

Its failures are subtler. The separator list encodes assumptions about formatting: headings, lists, procedures, and code fences do not necessarily break where your separators expect, and a paragraph with no sentence punctuation can still collapse into character-level splitting. It also treats every paragraph boundary as equally important, so it cannot tell a major section break from a minor aside.

For most prose, this is the strongest quick baseline: deterministic, cheap, no extra model calls, and easy to tune. Measure it before adopting anything more complex, because complex strategies need to beat it on your data, not in the abstract.

Semantic chunking: split where the topic changes

Semantic chunking uses embeddings to decide boundaries. You split the source into small candidate units, usually sentences, embed them, compare adjacent units for similarity, and start a new chunk where similarity drops sharply or crosses a configured threshold.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings
 
embeddings = OpenAIEmbeddings()
splitter = SemanticChunker(
    embeddings,
    breakpoint_threshold_type="percentile",
)
 
chunks = splitter.split_text(document_text)

The promise is chunks that follow topic shifts rather than formatting accidents. In a long document that drifts from background to method to results without clear headings, semantic boundaries can produce more topically pure chunks than separators can.

The costs are real. You make embedding calls just to decide how to chunk, before you embed the final chunks. Boundaries depend on the model and threshold, so changing either re-chunks your corpus differently, and incremental indexing gets harder: edit one sentence and boundaries may shift well beyond it.

Thresholds fail at both extremes. Rare splits produce oversized, multi-topic chunks; frequent splits produce fragments that rank for a narrow phrase but lack context for generation. Procedure steps can look semantically distant because each introduces new nouns and verbs, while paragraphs repeating similar vocabulary about different entities can look deceptively close.

Use semantic chunking only for long, unstructured, topically varied documents, and only when evaluation proves it beats your recursive baseline.

Structure-aware chunking: split on the document itself

Structure-aware chunking splits on boundaries the author or format already provided: Markdown headings, HTML sections, slide boundaries, spreadsheet rows, function definitions, or records in a structured file. You then apply a size ceiling inside each unit, splitting further only when a section is too large.

from langchain_text_splitters import (
    MarkdownHeaderTextSplitter,
    RecursiveCharacterTextSplitter,
)
 
headers_to_split_on = [
    ("#", "h1"),
    ("##", "h2"),
    ("###", "h3"),
]
 
header_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=headers_to_split_on,
    strip_headers=False,
)
sections = header_splitter.split_text(markdown_text)
 
# Split only sections that are still too large, keeping the
# heading metadata the first splitter attached to each piece.
size_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=150,
)
chunks = size_splitter.split_documents(sections)

This gives you more than cleaner boundaries: metadata. Each chunk can carry its heading path, which you store alongside the vector and can prepend at retrieval time. A paragraph under a heading naming its product or procedure is far less ambiguous than the same paragraph stripped of that context.

It fails when the structure lies. Converted documents often have headings based on font size rather than meaning, some authors put everything under one heading, and converted content may lose structure entirely. Very short sections are also a problem: a two-line section may be too sparse to rank, so merge small sibling sections under a shared parent heading.

Use it whenever your corpus has reliable structure — most documentation sites, wikis, Markdown repositories, and help centers. For those corpora it is usually the highest-quality option, because it aligns the retrieval unit with the unit the author intended a reader to consume.

The strategies side by side

Read the table as a claim about information, not sophistication: use the strongest boundary signal your corpus actually provides. Structure beats similarity when structure exists, and separators beat raw length when formatting exists.

Overlap: insurance with a real cost

Overlap copies the end of one chunk into the start of the next, so a passage straddling a boundary still appears whole in at least one chunk.

Used carelessly, it creates new problems. You index duplicated text, growing your store and embedding bill. Near-duplicate chunks compete at retrieval time, so your top-k can fill with copies of one passage instead of distinct evidence, and your generator receives repeated text that wastes context.

Keep overlap as small as still protects your boundary cases, and deduplicate results if duplicates crowd your top-k. With structure-aware chunking you often need less, because boundaries already fall in safer places. If you keep increasing overlap to fix retrieval, your primary boundaries are wrong, and overlap is compensating at a premium.

Metadata is part of the chunk

Two identical paragraphs can mean different things in different documents: a paragraph about resetting a password means one thing in a user guide and another in a security policy. Without metadata, your retriever cannot tell them apart.

Attach at least the source identifier, document title, heading path, and any field you will filter on, such as product, version, language, access level, or document type. Store those fields as filterable metadata rather than embedding them all into the text, but consider prepending a short context line — title plus heading path — to the text you do embed. That gives the vector disambiguating signal while keeping filters exact.

Handle access metadata deliberately. If your corpus mixes public and restricted documents, filtering must happen at retrieval time using metadata you trust, not afterwards by inspecting returned text. A pipeline that drops or misaligns access metadata creates a safety problem no ranking improvement offsets.

Preserve a stable chunk identifier and source position too. You need both to debug retrieval, cite sources, and update a document without reindexing everything.

How to evaluate chunking without fooling yourself

Do not evaluate by reading a few chunks and deciding they look reasonable. Almost every strategy produces plausible-looking chunks. Evaluate at the retrieval layer, with queries like the ones your users actually ask.

Build a small, fixed set: representative queries, each paired with the passage that should be retrieved. Include easy lookups, paraphrases sharing little vocabulary with the source, multi-part questions, and questions whose answers live near section boundaries — the cases chunking decides.

For each strategy, index the same corpus and run the same queries with the same model and top-k. Check whether the correct passage appeared and how highly it ranked, then whether your generator uses the right evidence without importing irrelevant text.

Score each strategy with a simple, illustrative check such as recall at k: for each query, did the expected passage appear in the top-k results, averaged over the whole set? Then avoid a common trap: if you regenerate expectations after re-chunking, every strategy can look perfect, because you redefined the correct answer to match whatever chunks now exist. Anchor expectations to source passages — a document plus a named section — and map each passage to whichever chunk contains it under each strategy.

Some inputs need explicit handling regardless of strategy. Filter empty and near-empty chunks before embedding. Keep a table’s header row with its rows, or each row becomes values with no column meaning. Keep code blocks intact where possible, splitting oversized blocks at syntactic boundaries such as function definitions. Keep a procedure’s introductory sentence with its steps. In multilingual corpora, separators, token density, and embedding quality vary by language, so one configuration tuned on one language may fragment or bloat another. Plan updates too: remove a document’s old chunks as you insert its new ones, or your index accumulates ghosts.

Re-run the evaluation when your corpus changes shape, not just when you change the splitter. A new document type can quietly invalidate a strategy that worked for the original content.

Verdict: what to choose

Start with structure-aware chunking if your corpus has reliable headings, sections, or records, and fall back to recursive splitting inside any section that exceeds your size ceiling. That gives you author-intended boundaries, useful heading metadata, deterministic behavior, and low indexing cost. For documentation and knowledge-base corpora, it is the recommendation in this comparison.

If your corpus is unstructured prose, start with recursive chunking and treat its tuned result as the bar. Adopt semantic chunking only when it clears that bar on your own queries by a margin you can see in the retrieved passages, not just in an aggregate score, because you pay for it on every indexing run.

Reserve fixed-size chunking for baselines and genuinely structureless text. Whatever you choose, keep the model, evaluation set, and retrieval settings fixed while you compare, preserve rich metadata, and re-evaluate when your corpus changes shape. Chunking is the contract between your documents and your retriever.