TL;DR
Fixed-length chunking breaks enterprise RAG because it shreds the inherited structure that gives legal, regulatory, and code-based content its meaning — slicing tables in half, detaching definitions, and forcing teams to compensate with bigger models and noisier context. Hierarchical chunking preserves the parent-child relationships already embedded in Markdown, HTML, JSON, code, and legal documents, and pairing it with auto-merge retrieval lets the system promote a coherent parent section instead of returning four redundant sibling fragments.
Key takeaways:
- Throwing more context, bigger models, larger overlaps, and rerankers at fixed-length chunking didn’t meaningfully fix retrieval on real enterprise repos like the CFR or codebases
- Hierarchical chunking works best on content with built-in structure: Markdown, HTML, JSON, code, legal documents, and regulatory frameworks
- Auto-merge retrieval detects when top-K results come from the same parent and collapses them into one parent chunk, freeing slots for genuinely new information
- The next step is adaptive, query-aware chunking where granularity, depth, and traversal are decided at retrieval time, not preprocessing
- Treat enterprise knowledge as a structured semantic system, not a bag of token fragments
One of the more interesting things happening in enterprise AI right now is that organizations are finally realizing that to build the AI Assistant they really want, they need to quit focusing on which model they are using. The model isn’t the problem. It’s figuring out how to give the model the best context. And for RAG systems, there are document structure problems (something we happen to know a lot about, given our nearly 40 years in surfacing data from document-based content).
The Core Problem With Fixed-Length Chunking
Let’s say have a large repository of documents you want to use as the source of truth for an AI Assistant your employees can chat with. Policy document navigation and regulatory compliance are the top use cases our customers ask us about in this realm. We start by pre-processing the text from all documents in the collection, breaking each document into one or more “chunks” of text. In the simplest and fastest approach, we would split the text into a new chunk at every fixed X number of words, with some percentage of each chunk overlapping with the chunk before it. Then when a user asks a question, the assistant finds the top-K most chunks of text most relevant, and it uses that information to answer the question.
This approach can work ok in some situations, but provides extremely poor answers in others. A fixed-size chunk might:
- Cut a table in half, where the first chunk is missing rows and the table footer, and the second chunk has some rows and a footer, but no column headers to understand what it is dealing with.
- Separate a section header from its explanatory text
- Detach a paragraph from the definitions established earlier in the document
- Mix unrelated topics together simply because they happened to exist near each other in the source document (like the back half of a paragraph and the beginning of another all together as one chunk).
This creates a retrieval problem that many teams mistakenly try to solve with supplying “more context”, even when more context doesn’t necessarily mean better context. They simply allow the model to consume more irrelevant material. We also went down this path as we began to explore building RAG solutions in Grooper, trying techniques like these that all failed to dramatically improve results:
- Increasing overlap sizes dramatically
- Retrieving far more chunks than necessary
- Using larger and more expensive models
- Expanding context windows
- Adding reranking stages to compensate for noisy retrieval
Some of this worked well enough for a demo against short documentation sets. But once we started pointing Grooper at actual enterprise repositories like legislative bills, software code bases, compliance documentation, and government regulations, the weaknesses became impossible to ignore. A fixed-token chunking strategy fundamentally misunderstands what enterprise information actually is. Tons of enterprise knowledge is hierarchical, and we must preserve that hierarchy instead of destroying it during ingestion.
The Code of Federal Regulations Is a Perfect Example
The easiest way to understand this problem is to imagine building a legal research assistant over the Code of Federal Regulations (CFR). The CFR is not simply a large body of text. It is an enormous nested semantic system. Every concept inherits meaning from higher-level structures.
Title -> Subtitle -> Chapter -> Subchapter -> Part -> Subpart -> Section -> Subsection -> Clause.
If you split the CFR into arbitrary 500-token windows, you routinely end up with fragmented legal definitions, detached exception clauses, incomplete procedural requirements, etc. A compliance assistant cannot afford to partially retrieve a regulation while omitting the subsection that invalidates or narrows the interpretation. That is precisely why hierarchical chunking matters.
Hierarchical Chunking Preserves Inherited Meaning
One of the best explanations emerging recently from both the Weaviate engineering discussions and research projects like HiChunk is the idea that retrieval systems should preserve semantic inheritance instead of flattening documents into isolated fragments by using hierarchical chunking.
Humans naturally understand this. If you open a legal document and see: Section 8.4.2(c) you immediately navigate upward mentally to understand what larger topic this belongs to, whether this is an exception, what definitions apply, and what broader regulation governs it.
Hierarchical chunking, while not perfect for all documents, is especially well-suited for:
- Markdown
- HTML
- JSON
- code repositories
- legal documents
- regulatory frameworks
Because these input formats and content types already contain embedded semantic structure. Markdown has headers and nested sections. HTML has DOM hierarchy. JSON contains parent-child object relationships. Code contains namespaces, classes, methods, modules, and dependency scope. Hierarchical chunking creates chunks at each level, ensuring each thought is isolated from its siblings and other unrelated passages, while also preserving parent-child and sibling relationships for later use.
Auto-Merge Retrieval Is One of the Smartest Ideas in Modern RAG
One of the most important ideas emerging from hierarchical retrieval systems is the concept of auto-merging related chunks during retrieval. Imagine a user asks: “What reporting exemptions exist for small financial entities under this regulation?”
The retriever, whose job is to find the Top K number of relevant chunks (say the Top 10), may initially include multiple chunks originating from the same subsection because cosine similarity keeps surfacing overlapping language patterns. So in this example:
- subsection 4.2(a)
- subsection 4.2(b)
- subsection 4.2(c)
- subsection 4.2(d)
All come from the same parent “subsection 4.2”.
A traditional retriever would waste 4 of its top-k slots returning adjacent fragments from the same semantic neighborhood. Hierarchical auto-merge systems recognize this redundancy. Instead of returning four fragmented sibling chunks, the system dynamically decides if it should upwardly merge these together as a single result, which then promotes the entire parent section itself into the retrieved context.
That immediately frees additional retrieval capacity for additional high-value knowledge that is otherwise made unavailable to the AI for use in the response, or you can simply stop with the improved result and fewer tokens. This is incredibly important. The best retrieval systems do not merely find relevant fragments. They assemble coherent semantic regions. And auto-merging helps achieve exactly that.
Where Hierarchical Chunking Is Headed Next
The next evolution is likely not just hierarchical chunking, but adaptive chunking. Systems are beginning to dynamically determine:
- Chunk granularity
- Retrieval depth
- Context expansion behavior
- Parent-child traversal
- Query-specific semantic boundaries
In other words, the chunking strategy itself becomes query-aware.
That is a major shift from static preprocessing pipelines. We are also seeing increasing convergence between vector databases, knowledge graphs, document object models, structured semantic indexing, and agentic retrieval systems.
The retrieval layer is becoming less like a search engine and more like a reasoning substrate, and frankly, this is overdue. Most enterprise knowledge was never designed to be consumed as disconnected token fragments. The earlier RAG architectures treated it that way because it was easy. The newer architectures are starting to treat enterprise information as structured knowledge systems instead. That distinction is likely to define the next generation of production AI platforms.
References
LangChain Docs: Splitting code text splitter integration guide
https://docs.langchain.com/oss/python/integrations/splitters/code_splitter
LlamaIndex Auto Merging Retriever
https://developers.llamaindex.ai/python/framework/integrations/retrievers/auto_merging_retriever/
HiChunk: A Hierarchical Chunking Method That Turns Fragments into Flow — AI Innovations and Insights 87
https://aiexpjourney.substack.com/p/hichunk-a-hierarchical-chunking-method
HiChunk: Evaluating and Enhancing Retrieval-Augmented Generation with Hierarchical Chunking
https://arxiv.org/pdf/2509.11552v3
