Structured Data Improves AI Assistant Accuracy

TL;DR

Simple chunking-based RAG works fine for narrative text but collapses on dense financial tables — chunk boundaries chop off column headers, subtotals, and departmental context, so the assistant answers confidently and wrongly. For a city-budget chatbot demo in Grooper, the fix was to extract the nested tables into a structured database, let the AI Assistant query both the DB and the original PDFs, and have it render inline tables and SVG charts so users could still see the source. Precision came from structure; trust came from grounding both answers in the documents.

Key takeaways:

• Vector similarity retrieval is not the same as relational understanding — tables need structured extraction, not better chunking
• Flattening hierarchical document data into normalized SQL records turns fragile inference into straightforward queries
• GPT-5.5 handles natural-language-to-SQL well enough now to power this with little hand-holding
• Database-only answers lose visual grounding, so inline HTML tables and dynamically generated SVG charts restore explainability
• Querying both the database and the document repo on every question gives precision plus narrative context — and flags conflicts

A few days ago, I wrote this article about hierarchical chunking and why it matters for Retrieval-Augmented Generation (RAG) systems. Hierarchical chunking can sometimes solve a very real problem: many document collections are not formed with the type of clean, self-contained paragraphs that LLMs can easily interpret. They are deeply structured collections of sections, subsections, tables, footnotes, appendices, and references that all depend on surrounding context.

But hierarchical chunking is not the only way to solve these problems.

Last week, I built a customer demo in Grooper that demonstrated a different architectural approach entirely. Instead of trying to preserve more document hierarchy inside the vector store, I used Grooper’s structured data extraction to collect and push structured data into a database. Then, the Grooper AI Assistant combined database lookups alongside the document repository itself during retrieval.

The result was a system that dramatically improved accuracy for highly structured financial data while still maintaining document grounding and explainability.

The Use Case: City Government Financial Accountability

The customer was a mid-sized city government. Each fiscal year, the city publishes a set of PDF documents to its public website, many exceeding 100 pages. The documents represent the financial state of the city in terms of budgetary/spend changes year over year, budgetary policy, future projections, capital project plans, and staffing changes. They sought to create an AI chatbot that citizens could use to ask plain-language questions, but with one critical restriction: The assistant could only answer questions using information contained in the official published PDF collections. This is a very common first use case for public-sector AI adoption. Organizations want better accessibility and discoverability of information without introducing hallucinated external knowledge, public opinion, and political commentary. The documents in the collection communicated:

• Budgetary policy changes
• Department budget/outlays
• Staffing numbers by department and role
• Capital project allocations
• Department spending summaries

Step 1 – Traditional RAG

To begin testing, I took the straightforward approach. All documents for the current fiscal year were loaded into the vector store using simple, fixed-length chunking with a 500-token chunk size and 20% overlap. Then I began testing questions in the Grooper AI Assistant console. At first, the results looked promising. Questions about budgetary policy worked quite well because those portions of the documents were primarily narrative text. Policy explanations were already organized into coherent paragraphs and sections that fit naturally inside chunk boundaries, and the questions generally retrieved strong contextual matches. For example:

• Why did the city increase stormwater funding?
• What changes were made to employee healthcare policy?
• Which departments received operational restructuring?

Where Simple Chunking Failed

The problems emerged when asking highly specific financial questions involving exact allocations, aggregate totals, timeline trends, or departmental comparisons. Questions like:

• How much funding was allocated for roadway improvements?
• What was the parks department capital equipment budget?
• Which department saw the largest year-over-year increase?

These answers often lived inside massive financial tables spanning dozens of pages. And in particular, that last question would fail miserably. Because to answer that question, you’d have to obtain bits and pieces of information scattered across hundreds of pages to gain the entire needed context. And even if the system located all of those chunks, it still wouldn’t definitely know how many and what different departments there are, as that’s not really part of the question. So, the likelihood of missing something is nearly guaranteed, and the resulting answer would come through with ultimate confidence and still be completely wrong.

The source documents contained deeply nested structures:

• Sections
• Subsections
• Department groupings
• Table headers
• Subtotals
• Grand totals
• Multi-page rollups

And this is where simple chunking collapsed. A chunk might contain numerical values, table rows, and subtotals, but not the column headers or departmental context required to interpret those numbers correctly. The amounts were also almost always written in thousands, which occasionally produced problems.

In some cases, the assistant even responded with statements like “I only have partial data. It looks like some of the information I need is cutoff in the source material.” And it was painfully obvious why. The retrieved chunk literally ended halfway through a table, as its boundaries were determined with no consideration to the source content. This is one of the most important lessons in enterprise AI. Semantic similarity retrieval is not the same thing as relational understanding.

Step 2 – Adding Structured, Extracted Data

Instead of continuing to optimize chunking strategies, I changed the AI Assistant’s architecture. Using Grooper, I extracted the nested financial data from the PDFs into structured datasets. The extracted data was then normalized and exported into three relational tables inside a local testing database. This step was critical. During extraction, Grooper also flattened the document hierarchy so that no table joins would be necessary. Instead of relying on an LLM to infer relationships from fragmented chunks, the system now had clearly defined records containing:

• Department names
• Fiscal years
• Budget categories
• Allocation types
• Totals
• Subtotals
• Associated metadata

The AI Assistant was then updated so it had access to 4 different resources:

1. The original document repository
2. A SQL table containing detailed financial outlays
3. A SQL table containing detailed financial resources
4. A SQL table containing departmental staffing numbers

I also provided simple descriptions of each dataset so the assistant could understand the semantic purpose of each. The difference in accuracy was immediate.

*Note, in a production, public-facing implementation we would never recommend connecting the bot directly to the database. A secured API or MCP layer would be added to the solution, and the AI would gain access to the dataset through those means.

Why this Solution Optimized Accuracy

The improvement was not because the database was “smarter” than vector search. It was because the structure already existed. Financial tables are inherently relational datasets. When flattened into structured records, questions that previously required fragile contextual inference became straightforward retrieval operations.

Instead of asking an LLM to reconstruct table hierarchy from partial document chunks, the system could now directly query the database. In this demo, it’s worth mentioning that we used GPT-5.5, which frankly is MASSIVELY better at converting natural language to SQL queries than any of the frontier lab’s models 6-12 months ago. And it does so without a lot of hand holding.

But solving one problem introduced another.

Loss of Visual Grounding and Explainability

Once answers started coming directly from the database, we lost an important element of trust and explainability: visual grounding in the source documents. When using document retrieval, users could at least inspect the original page of each cited document with one click. But with direct database responses, users were now seeing conclusions without immediate visibility into the underlying source material.

That matters more than many AI architects realize. Users often need to validate not only the answer itself, but the origin and context of the answer. Accuracy alone is not enough. Users also need confidence.

Step 3 – Adding Explainability Back Into the System

To address this, I instructed the assistant to include in its response an HTML table to display the relevant rows of data used to answer the question. This immediately improved transparency. Users could now see:

• Which records were used
• Which departments were involved
• Which fiscal years matched
• Which totals contributed to the answer

Then I added another request in the system instructions. When users asked comparative or trend-based questions, the assistant was instructed to generate charts and graphs inline as part of the response. What made this especially interesting was that the assistant was not given rigid rendering templates or chart schemas. It dynamically generated html-based tables and SVG charts with inline visualizations directly from the retrieved structured data.

The result felt far more like a true analytical assistant than a traditional document chatbot.

Step 4 (final) – Requesting Multiple Sources of Truth

The final improvement was arguably the most important. I instructed the assistant to always attempt retrieval from both the structured database tables AND the original document collection. This created a multi-source validation model. The database provided precision, structured retrieval, and numerical reliability. The document repository provided context, narrative explanation, visual grounding, and source verification. Together, they complemented each other extremely well and improved both usability and trustworthiness.

The Final System Prompt

• You are a chatbot accessible from our city government’s public website
• Your job is to answer questions about the city’s budgetary policies, budget numbers, departmental employee headcounts, etc.
• If asked about the specific amount of resources or outlays related to specific funds, you should look in the Resources and Outlays database tables, but also look for language in one or more documents and cite both sources, even if they conflict.
• If asked about agency, departmental, or occupational employee counts, use the Department_Staffing database table as a primary resource, but also look for language in one or more documents and cite both sources, even if they conflict.
• If returning information from a database table, also output the relevant data in a tabular format in the response.
• If you are asked to compare multiple rows in a database table or provide a timeline of changes, create a visual chart or graph that helps illustrate your answer.
• If you are not absolutely certain about an answer or see conflicting answers, you may provide some insight about what you know, but you MUST also state that you are not certain of the correct answer and direct the user to contact Customer Service

The Bigger Lesson

This demo ultimately reinforced a broader realization about enterprise AI design: The goal is not simply to retrieve text, but to retrieve meaning with sufficient context, structure, and explainability for users to trust the result.

Sometimes hierarchical chunking is the right answer. Sometimes database retrieval is the right answer. Sometimes the real solution is combining vector search, structured extraction, APIs, document grounding, and dynamic visualization into a single retrieval architecture.

As enterprise AI systems mature, the winners will not necessarily be the systems with the largest models or the fanciest embeddings. They will be the systems that understand the shape of the information itself.