Chunking strategies that survive production

In one sentence

Chunking is the decision about how much text goes into each retrievable unit — and it quietly determines the ceiling on everything downstream.

Why it matters

Retrieval can only return what you stored. If a chunk is too small, the passage that gets retrieved is missing the sentence that made it meaningful. If it is too large, the embedding averages over several unrelated ideas and stops matching anything precisely.

Most "the model hallucinated" bugs in a RAG system are chunking bugs wearing a costume. The model answered faithfully from the context it was given; the context just didn't contain the answer.

A useful test: read a random chunk out of your index, in isolation, with no document title and no surrounding text. If you can't tell what it is about, the retriever can't either.

The three properties a chunk needs

Strategies, in the order you should try them

Structural splitting

Split on the document's own boundaries — markdown headings, HTML sections, slide breaks, function definitions. This is almost always the best first attempt because the author already grouped related ideas for you.

import { MarkdownTextSplitter } from 'langchain/text_splitter'

const splitter = new MarkdownTextSplitter({
  chunkSize: 1000,
  chunkOverlap: 150,
})

const chunks = await splitter.createDocuments([markdown], [{ source: docId }])

Recursive character splitting

The default when structure is unreliable. It tries a list of separators in order — paragraphs, then lines, then sentences, then words — and only falls back to a harder break when a chunk still exceeds the size limit.

import { RecursiveCharacterTextSplitter } from 'langchain/text_splitter'

const splitter = new RecursiveCharacterTextSplitter({
  chunkSize: 800,
  chunkOverlap: 100,
  separators: ['\n## ', '\n### ', '\n\n', '\n', '. ', ' '],
})

Semantic splitting

Embed each sentence, then cut where consecutive sentences diverge past a threshold. It produces the best boundaries and costs an embedding call per sentence at ingest time. Worth it for a corpus you index once and query constantly; wasteful for data that churns hourly.

Picking a size

Start at 500–1000 tokens with 10–15% overlap and tune from measurements, not intuition. Some anchors:

• Short factual lookups (support macros, FAQs, product specs) do well at 200–400 tokens. Precision matters more than surrounding context.
• Prose and documentation sit comfortably at 700–1000. Enough for a complete argument, small enough that one idea dominates the vector.
• Code should follow syntax, not length. Split on function and class boundaries and let chunk size vary.

Overlap exists to stop an idea being severed mid-sentence. It is not free: at 25% overlap you are storing and searching a quarter more vectors, and near-duplicate chunks start crowding each other out of your top-k.

Common pitfalls

• Tuning chunk size before you have an eval set. You cannot tell whether a change helped. Build twenty golden question/passage pairs first — see context evaluation.
• Dropping the heading trail. A chunk that says "This limit defaults to 4096" is useless without knowing which limit. Prepend the heading path to the chunk text before embedding, not just to the metadata.
• Chunking tables row by row. A row without its header row is a list of values with no names. Keep the header with every fragment.
• Assuming one size fits the corpus. A repo of API references and a folder of incident write-ups do not want the same strategy. Chunk per source type.
• Forgetting that chunk size interacts with top-k. Halving chunk size without raising k halves the context the model actually sees.

Further reading

• Anthropic — Contextual Retrieval, on prepending generated context to each chunk before embedding.
• Pinecone — Chunking Strategies for LLM Applications.
• LangChain — Text Splitters.