Fine-tuning a local documentation QA model

A controlled baseline comparing fine-tuned local inference with grounded retrieval for recurring Databricks documentation questions.

View the reproducible training and evaluation code

1. Research question

The starting point was an existing local RAG assistant over official Databricks documentation. This project explored a second architecture: could a compact local model learn enough from the same documentation corpus to answer common questions directly, without retrieving evidence on every request?

The experiment does not assume that either approach should win. It compares three answer paths on the same documentation-derived questions:

The purpose is to measure the trade-off among correctness, completeness, unsupported claims, and local response latency. The outcome is a baseline for deciding whether a hybrid route is worth further investigation—not a claim that fine-tuning should replace retrieval.

2. Constructing the datasets

The data pipeline has two separate Q&A generators. They run independently, then their accepted outputs are combined. The paired evaluation benchmark is created only after both generators are complete.

2.1 Independent Q&A first

The first generator creates local, fact-oriented examples. For each eligible documentation chunk, it includes the anchor chunk and nearby chunks from the same document, then asks for one to three useful questions. This favors commands, procedures, definitions, and other details that can be answered from a compact context.

2.2 Page-aware Q&A added afterward

The initial dataset used only independent chunk-based Q&A. Page-aware generation was added because a single chunk can omit the setup, limitations, or related steps needed for a complete answer. It reads a full page when possible, or consecutive heading-preserving windows for oversized pages, and generates questions that connect related facts, conditions, steps, and trade-offs.

Both generators use unsloth/Muse-Glimmer-30B-GGUF:UD-Q4_K_XL through a llama.cpp-compatible OpenAI API. The process is structured: generate context-grounded questions, then generate each answer with short exact supporting excerpts. A pair is accepted only when both answer and evidence are present; the context, evidence, source hash, and generation metadata are retained in SQLite.

2.3 Paired evaluation questions

The benchmark builder selects one accepted pair per source page and balances the selection across documentation categories. It then uses the source pair’s evidence to generate one new question and reference answer about the same documented knowledge. Near-duplicate question wording is rejected.

This produces a fixed 400-question wording-and-framing generalization benchmark. It is not a strict unseen-knowledge test: the corresponding documentation knowledge is represented in the training corpus, but the benchmark asks a materially different question about it.

2.4 Examples from the exported data

The following are real Q&A pairs from the immutable v001 exports. They illustrate the difference in context scope and the alternate wording used for benchmark questions.

Independent Q&A — compact, local context

Question:
Which MLflow log_model parameter is used to package dependent artifacts for Model Serving, and what example values are shown for a Unity Catalog volume path and a local tokenizer cache?

Answer:
The MLflow log_model parameter used to package dependent artifacts for Model Serving is artifacts. The example shows a Unity Catalog volume path for model-weights as "/Volumes/catalog/schema/volume/path/to/file" and a local tokenizer cache for tokenizer_cache as "./tokenizer_cache".

Page-aware Q&A — connected workflow across one page

Question:
How does the app handle OAuth token expiration when connecting to Lakebase, and what components enable automatic token rotation?

Answer:
Databricks Apps authenticate to Lakebase with OAuth tokens that expire after one hour. The app automatically generates fresh tokens whenever it needs to connect, using a connection pool that creates new connections with fresh tokens so expired credentials are never used.

Automatic token rotation is enabled by three components: WorkspaceClient, which generates fresh credentials using the SDK; a custom OAuthConnection class, which injects a fresh credential for each new connection; and ConnectionPool, which manages connections and calls the custom class as needed.

Paired benchmark Q&A — alternate wording about the same evidence

Source training question:
What is the default grouping and time range when you open Explorer from the Spend over last 30 days tile?

Benchmark question sent to evaluated systems:
A cost analyst starts from the Spend over last 30 days tile and clicks View spend by product to drill into Explorer. What grouping and time period does Explorer show on initial load?

Reference answer:
Explorer initially shows data grouped by Product over the last 30 days.

The source training question is shown above only to demonstrate the changed framing. During evaluation, the model and RAG system receive only the benchmark question.

3. Training and checkpoint selection

The 44,702 accepted Q&A pairs train a QLoRA adapter for unsloth/Qwen3.5-4B. A fixed 50-question subset of the 400-question benchmark is used only to calculate eval_loss and select checkpoints. It never produces gradient updates.

loss measures fit on training examples; the chart uses a 201-step rolling mean so the learning trend is readable rather than dominated by batch-level noise. eval_loss is calculated on the 50 selection questions and was used to choose checkpoint 9900. learning_rate is the planned optimizer schedule. grad_norm is a stability diagnostic, not an answer-quality score.

4. Benchmark protocol

The model and RAG comparison is a separate operation from fine-tuning. Each answer path generates and persists its answers first; unsloth/Qwen3.8-27B scores them in a second pass. This avoids loading a model, generating an answer, and judging it one row at a time.

The reported comparison includes all 400 questions. Because 50 informed checkpoint selection, it is not a pristine final test score; the remaining 350 questions are the unselected subset. The benchmark still provides a fair, identical comparison across the base model, checkpoints, adapters, and RAG.

5. Results

Every method was scored on correctness and completeness on a 1–5 scale. “Unsupported claims” counts claims the reference scorer judged unsupported by the reference context; lower is better. Latency is mean end-to-end time per answer on the local setup.

Fine-tuning clearly improved the base 4B model in this baseline. Checkpoint 9900 had the strongest model-only correctness score. RAG had the strongest judged quality overall and far fewer unsupported claims, but it was roughly 42 times slower than the fine-tuned candidates on this local setup.

The practical implication is a hybrid routing hypothesis: use direct fine-tuned answers for stable, repeatable requests, and retain retrieval when freshness, citations, or stronger grounding are required.

6. Limits of this result

• One 4B model, one seed, and one training configuration do not establish a general fine-tuning result.
• The benchmark measures new question wording about knowledge represented in the training corpus; it is not a strict unseen-knowledge test.
• Fifty of the 400 questions informed checkpoint selection.
• unsloth/Qwen3.8-27B provides consistent automated scoring, but a human audit is still needed to calibrate it against expert judgement.
• RAG latency depends on this local implementation and hardware; it is not a universal production latency benchmark.

7. Next experiments

1. Replicate the pipeline with Qwen 3.5 9B.
2. Reserve a separate benchmark that never informs checkpoint selection.
3. Run ablations over learning rate, LoRA rank, epochs, data mix, and context length.
4. Repeat promising configurations across multiple random seeds.
5. Add blinded human review and real documentation questions.
6. Test a router between direct fine-tuned inference and RAG.

8. Reproducibility

SQLite tracks Q&A generation, JSONL persists training and benchmark snapshots, MLflow records training telemetry and reference-scoring metrics, and generated answers are retained before judging. The scripts, configuration, and evaluation reports are available in the source repository.