Chunking decides retrieval before the model gets a say.

What is chunking in AI?

Chunking is splitting long content into smaller pieces that a system can embed, index and retrieve individually. Embedding models have input limits and, more importantly, an embedding of a whole document blurs every topic inside it into one vector. A chunk about a single idea produces a sharp vector that a related question can find. Chunking is the first step of every retrieval pipeline and the one most people under-think.

What is chunking in RAG pipeline?

In a RAG pipeline chunking happens at index time, before embedding. A loader reads a PDF, web page or database record; the chunker splits it; each chunk is embedded and stored with its text, its source and its position. At query time the retriever returns chunks, not documents, and the language model reads those chunks. So a chunk is the unit of evidence: the thing that gets cited and the thing that must contain the answer.

Why chunking is important in RAG

Because retrieval can only find what chunking preserved. Cut a table away from its caption and the numbers lose their meaning; merge two policies into one chunk and a question about either gets a diluted match; split a sentence at a boundary and the fact in it vanishes. When RAG answers are wrong, the cause is a retrieval miss far more often than a model failure, and retrieval misses are usually chunking choices. That is why I test chunkers before I tune prompts.

How to chunk documents for RAG

• Follow structure first: headings, paragraphs, list items, table rows with their headers, code functions with their docstrings.
• Size by meaning, not bytes: a few hundred tokens is a common sweet spot; enough for one idea, not three.
• Overlap a little (ten to fifteen per cent) so a fact at a boundary survives in one of the two chunks.
• Attach metadata to each chunk: document title, section path, page number, date. Filters need it; citations need it.
• Keep the parent: store the surrounding section so a retrieved chunk can be expanded for the model when needed.

Document types need their own handling. PDFs lose structure on extraction, so run a layout-aware parser that recovers headings, reading order and tables before you chunk; a table split across two chunks is worse than no table. Keep each table as one chunk with its caption and a short text rendering of the header row. Code should be chunked by function or class with the docstring attached and the file path in metadata. Chat transcripts and emails chunk by thread or by turn with the participants and date prefixed. Slides chunk by slide with the deck title. The rule underneath all of them is the same: a chunk should make sense to a person reading it on its own, because that is exactly how the model will read it.

How to chunk text for embeddings

Split on structural boundaries, then enforce a maximum size with a recursive splitter that tries paragraphs, then sentences, then words. Prepend context to each chunk, such as the document title and section heading, so the embedding knows where it sits; this alone lifts retrieval noticeably. Embed the chunk including that prefix. Never embed a chunk that is just a heading or a page number; filter the fragments out before indexing.

What are chunking strategies in RAG?

Fixed-size chunking splits every N tokens; simple and surprisingly strong as a baseline. Recursive chunking splits on paragraphs, then sentences, until each piece fits. Structure-aware chunking follows headings, lists and tables in HTML, Markdown or PDF layout. Semantic chunking groups sentences by embedding similarity so each chunk stays on one topic. Parent-child chunking indexes small chunks for precise matching but returns their larger parent for the model to read.

How to semantic chunking

Embed each sentence, walk through the document, and start a new chunk whenever the similarity between consecutive sentences drops below a threshold; topic shifts show up as similarity drops. Cap chunk size so a long single-topic section still splits. It costs an embedding call per sentence at index time, so it is slower than structural splitting, and in my tests it helps most on unstructured prose such as transcripts and long emails, less on documents that already have good headings.

How chunking works in RAG

The chunker produces pieces; each piece becomes a row with text, vector and metadata; the vector index and the keyword index both point at those rows. A question retrieves rows; a re-ranker orders them; the top few are placed in the prompt with their sources. If you store the parent section alongside, you can retrieve on the small chunk for precision and hand the model the parent for context. Everything downstream inherits the chunker's decisions.

Why is chunking critical in LLM applications?

Language models have a context budget and a cost per token. Chunking decides how much of that budget is spent on the right evidence. Good chunks mean five short, relevant passages in the prompt; poor chunks mean twenty vague ones, higher cost, slower answers and a model that has to guess which part mattered. In RAG.NextUpgrad the confidence gate also scores the retrieved chunks, so clean chunks make the gate's decisions sharper.

What is chunking strategy in RAG?

A chunking strategy is the set of rules you choose for splitting a particular collection: which boundaries to respect, the target size, the overlap, what context to prepend and whether to index parents. There is no universal best; a legal contract, a code repository and a chat log want different rules. Pick a strategy per document type, write twenty real questions, and measure recall at five for each candidate strategy. The numbers decide, not the blog posts.