Building a Grounded Databricks Documentation Assistant

A technical case study in building a grounded RAG application: official documentation in, inspectable answers with citations out—and measurements for where the system succeeds or fails.

View the source code on GitHub

1. See the assistant work

The goal was deliberately simple: instead of handing someone a list of documentation links, let them ask an implementation question and see a concise answer backed by the official pages.

In the demo, the assistant:

1. Receives a Databricks-related question.
2. Retrieves an initial set of candidates using the exact question.
3. Lets the retrieval agent inspect those candidates and search for additional evidence when a question has multiple aspects.
4. Produces an answer with numbered, clickable links to the official sources used.

The important behavior is not merely that it can answer. It is that the answer is grounded in the official Databricks documentation, with periodic incremental refreshes designed to keep the index aligned as the documentation changes. If the evidence is insufficient, the application says so instead of filling the gap from general model knowledge.

2. What problem was I solving?

Documentation search is good at locating pages; it still leaves a person to collect the relevant details, reconcile them, and decide which version to trust. A general chat model is easier to ask, but it may answer from stale memory or invent a plausible implementation detail.

I wanted the useful middle ground:

• Useful answers, not a link dump: A retrieval agent can search more than once and inspect document sections.
• Verifiable answers: Require valid citations and show links directly below the answer.
• Clear abstention when evidence is missing: Return “I could not verify this from the indexed documentation.”
• Fresh documentation: Refresh the source corpus incrementally instead of rebuilding it from scratch each time.

3. How an answer is built

Python ranks chunks with a hybrid score (65% embedding similarity, 35% exact-term match) and keeps the top ten as the agent’s starting candidates. The agent inspects them, can search within a document or add targeted results, then selects excerpts for the answer. New results add to the original candidate pool.

The answer model receives only the selected excerpts, their declared support scope, and any unresolved question parts. If the agent selects no evidence, Python returns “could not verify” without generating an answer. Otherwise, the model answers from the excerpts; a Python citation check then requires substantive text and valid citation labels. The hybrid score ranks candidates, and the agent’s scope guides the answer, but neither independently verifies factual support. For a partly supported question, the answer can cite supported points and identify what remains unverified.

A bounded investigation, not unlimited reasoning

The agent currently has a budget of four research turns. In each turn, it decides whether to search, read chunks, look for related chunks, or search within a document. It may submit more than one independent search in a turn, but it must inspect the combined results before deciding again.

After those four research turns, the application forces a fifth, final-selection turn: the agent must select the evidence it has opened rather than continue searching. A separate answer-generation call then writes the grounded response and citations. That means a difficult question can use up to five retrieval-planning calls plus one answer call.

The limit is deliberate. More turns can help with genuinely multi-step questions, but they also add latency, token cost, and more opportunities for the model to wander. The original-question seed reduced the need for follow-up searches in the frozen evaluation: the optimized agent averaged 0.28 searches per question.

4. What the retrieval evaluation showed

Retrieval was evaluated on the same 350-question holdout used for the broader documentation QA study. For each question, the benchmark identifies an expected source page and supporting excerpts. These measures show whether retrieval found that page and included the reference evidence; they do not score the final answer’s correctness.

The trace records candidate chunks, selected evidence, document and page identifiers, scores, context hash, context size estimate, search count, tool turns, answer latency, answer text, and citation decisions. That makes a failure diagnosable: the problem may be source-page retrieval, evidence selection, context assembly, answer generation, or citation validation rather than an undifferentiated “RAG score.”

These are retrieval and evidence-path results. They show whether the expected page and reference evidence reached the answer stage, not whether the generated answer was correct.

5. Incremental indexing: refresh only what changed

The first version could crawl and index documentation, but a small change risked triggering work across the full corpus. That becomes wasteful quickly, especially for embedding calls.

The refresh process still checks every configured source page so it can detect changes. The costly steps are incremental:

• New or changed pages are extracted and chunked.
• New or changed chunks are embedded.
• Unchanged chunks reuse their persisted vectors.
• A new FAISS snapshot is built from the complete set of stored vectors and activated only after validation.

This separation also makes recovery practical. If a run ends after chunking, I can resume the embedding and snapshot work from persisted data; I do not have to crawl and parse the source site again. Re-chunking remains a deliberate operation when chunking behavior changes.

6. The architecture behind it

There are intentionally two independent paths:

• The refresh path discovers official pages, persists documents, chunks, and vectors, then publishes a validated FAISS snapshot.
• The answer path loads the active snapshot, lets the agent gather evidence, and records the request trace, answer, and citations.

Keeping them separate means an interface deployment does not unexpectedly launch a crawl or a full embedding run.

7. A model can pass a tool-call demo and still fail as an agent

The most useful engineering finding came from model testing. Tool calling was not a checkbox; it was a model-selection requirement.

The application uses Muse-Glimmer for the local model configuration and supports the same retrieval loop with the tested hosted models. The lesson is broader than this project: test the entire agent trajectory—not one successful tool call.

8. Technology stack

The application keeps local development and shared Databricks deployment deliberately parallel, while using the model and storage services that fit each environment. This makes local iteration quick while keeping a credible route to a governed deployment:

The embedding models produce separate vector spaces, so each deployment has its own compatible FAISS snapshot; embeddings cannot be mixed between them.

9. Takeaways

The finished project is not just a chat window over an embedding index. It is an indexing pipeline, an evidence-seeking retrieval loop, a set of recovery operations, and a deployment model. The video is the compact version of the outcome: ask a focused question, watch it gather evidence, and inspect exactly where the answer came from.