Building a Grounded Databricks Documentation Assistant

A technical case study in building a grounded RAG application: current official documentation in, inspectable answers with citations out.

View the source code on GitHub

1. See the assistant work

The goal was deliberately simple: instead of handing someone a list of documentation links, let them ask an implementation question and see a concise answer backed by the official pages.

In the demo, the assistant:

1. Receives a Databricks-related question.
2. Searches, refines its search, reads selected documentation sections, and decides when it has enough evidence.
3. Produces an answer with numbered, clickable links to the official sources used.

The important behavior is not merely that it can answer. It is that the answer is grounded in the documentation it has indexed. If the evidence is insufficient, the application says so instead of filling the gap from general model knowledge.

2. What problem was I solving?

Documentation search is good at locating pages; it still leaves a person to collect the relevant details, reconcile them, and decide which version to trust. A general chat model is easier to ask, but it may answer from stale memory or invent a plausible implementation detail.

I wanted the useful middle ground:

3. How an answer is built

This is a small retrieval agent, not a single prompt with a fixed bundle of search results. It can search the documentation, inspect a result, search within a page, and refine its query before it returns the evidence it found.

The application then applies two explicit safeguards. No results or a top result below the relevance threshold produces an immediate “could not verify” response, without calling the answer model. When relevant evidence exists, the model writes an answer from it and its citations are validated. Missing or invalid citations also become a “could not verify” response. Citations appear only with an answer the system can support.

A bounded investigation, not unlimited reasoning

The agent currently has a budget of four research turns. In each turn, it decides whether to search, read chunks, look for related chunks, or search within a document. It may submit more than one independent search in a turn, but it must inspect the combined results before deciding again.

After those four research turns, the application forces a fifth, final-selection turn: the agent must select the evidence it has opened rather than continue searching. A separate answer-generation call then writes the grounded response and citations. That means a difficult question can use up to five retrieval-planning calls plus one answer call.

The limit is deliberate. More turns can help with genuinely multi-step questions, but they also add latency, token cost, and more opportunities for the model to wander. The right next step is not to raise the budget blindly; it is to test a larger budget against an evaluation set and keep it only when it improves supported answers enough to justify the cost.

4. Keeping the documentation index current without paying to redo it all

The first version could crawl and index documentation, but a small change risked triggering work across the full corpus. That becomes wasteful quickly, especially for embedding calls.

The refresh process still checks every configured source page so it can detect changes. The costly steps are incremental:

• New or changed pages are extracted and chunked.
• New or changed chunks are embedded.
• Unchanged chunks reuse their persisted vectors.
• A new FAISS snapshot is built from the complete set of stored vectors and activated only after validation.

This separation also makes recovery practical. If a run ends after chunking, I can resume the embedding and snapshot work from persisted data; I do not have to crawl and parse the source site again. Re-chunking remains a deliberate operation when chunking behavior changes.

5. The architecture behind it

There are intentionally two independent paths:

• The refresh path discovers official pages, persists documents, chunks, and vectors, then publishes a validated FAISS snapshot.
• The answer path loads the active snapshot, lets the agent gather evidence, and records the request trace, answer, and citations.

Keeping them separate means an interface deployment does not unexpectedly launch a crawl or a full embedding run.

6. A model can pass a tool-call demo and still fail as an agent

The most useful engineering finding came from model testing. Tool calling was not a checkbox; it was a model-selection requirement.

The Databricks App therefore uses Claude Sonnet 4.5 for the full retrieval loop. The lesson is broader than this project: test the entire agent trajectory—not one successful tool call.

7. Local first, governed when shared

The same application has two deliberately equivalent operating modes:

That makes local iteration quick while keeping a credible route to a shared, governed deployment.

8. Stack and takeaways

Built with: Python, Flask, FAISS, SQLite, Ollama-compatible APIs, deterministic chunking, Databricks Unity Catalog, Delta tables, Volumes, Workflows, Databricks Apps, Pytest, and Ruff.

The finished project is not just a chat window over an embedding index. It is an indexing pipeline, an evidence-seeking retrieval loop, a set of recovery operations, and a deployment model. The video is the compact version of the outcome: ask a focused question, watch it gather evidence, and inspect exactly where the answer came from.