Fine-Tuning a Small Model, Then Making It Check Its Own Homework

Fine-tuning tutorials usually end when the loss curve goes down. That tells you the model fit the training data. It doesn't tell you whether it learned what you wanted it to learn.

ShiftAI is a self-hosted studio built around that gap. It fine-tunes a small model on your data, then tests it on that data, then retrains on the questions it failed.

The loop

files / SQL -> Ingest -> Understand -> Synthesize -> Train -> Auto-eval
                                                       ^          |
                                                       |   failed questions
                                                       +- Improve <-+

Should you fine-tune at all?

Before anything is trained, an LLM reads your source and writes a short data card: what the data is, what a model would learn from it, where the gaps are, and whether fine-tuning or retrieval (RAG) is the better fit.

That last question matters. If your documents change every week, retrieval is usually the right answer and fine-tuning is wasted effort.

Turning documents into training data

Documents aren't training data. A teacher model turns each passage into question-and-answer pairs, in a few deliberate shapes:

• Reasoning traces: answers that walk step by step from the passage to the conclusion.
• Refusal pairs: on-topic questions the source can't answer, paired with an honest "not in my sources". These teach the model when not to make things up.
• Cross-passage pairs: questions that need facts from two passages, so the model learns to connect information.

Then a judge checks every generated pair for faithfulness to the source, and low scorers are dropped. Bad training data is the fastest way to a confident, wrong model.

Training choices that mattered

Training is LoRA on any Hugging Face causal model, run as a background job with a live loss curve. Two defaults are worth calling out:

• Answer-only loss. Loss is computed on the assistant's answer tokens, not the question. The model doesn't need to learn to write questions.
• General-data mixing. A slice of public instruction data is mixed into training so the model keeps its general chat and maths ability while learning your source. Without it, small models tend to forget what they knew.

Checking its homework

After training, the model is automatically quizzed with questions sampled from the source and scored against the gold answers, either by an LLM judge or a deterministic token-overlap score.

Then comes the part I like most. One click finds the questions it failed, generates extra training data aimed at those weak passages, and retrains. The new model is evaluated again, so you can watch the score move instead of hoping.

Running the result

You can chat with the trained adapter locally, send one prompt to the fine-tuned model and the base model side by side, or export the adapter to GGUF and register it in Ollama.

A small engineering note

The ML stack is heavy. The API and UI shouldn't need it. So torch and transformers are only imported inside the functions that train or run models, and the training work happens in a separate Redis Queue worker. The API and UI run fine on a machine without the training stack installed; only the machine that actually trains needs it.

Source: hoysengleang/train-model-shift-ai.