Agentic Supervised Fine-Tuning

Agentic Supervised Fine-Tuning in this context means treating the whole process as an agentic, iterative experiment loop…

---

My motivation for doing this…

• I believe the three areas of importance in the near future, will be the (1) data substrate, (2) security for AI and powered for AI and (3) multi-model orchestration, which includes Small Language Model (SMLs) which are fine-tuned for specific use-cases.

• So with models, multiple fine-tuned open weights models will be used in an orchestrated fashion. Where specific models (even categorised as small) will be responsible for specific tasks.

• I have said this in the past, but fine-tuning of models will become commonplace, just the way fine-tuning NLU and NLP models were prior to the advent of Large Language Models and Generative AI.

• Hence being able to perform fast and efficient fine-tuning on conversation traces and then running that model became important to me.

• Building the pipeline is the challenge, I have the data, the access to the model and a Colab instance etc. But again, putting the pipeline together and automating it is not straightforward…but…

• Using Grok Build CLi (Grok 4.5) in the process helps and the Grok acts as a copilot to stitch it all together.

---

This blog was inspired by a HuggingFace demo video, all the detail to that is in the footer of the blog.

Secondly, do I fully understand everything I did here…no. But I love “getting it to work” and within that there is a great sense of achievement and learning.

Obviously Agentic Supervised Fine-Tuning is not a brand-new math formula. It is a new way of working:

1. You describe what you want in plain language to an AI agent harness (Grok CLI, Cursor, Claude Code, etc.).

2. The agent writes data scripts, training configs, eval checks and debugs the run.

3. You inspect loss, traces and sample generations.

4. You refine and go again.

Under the hood the technical ladder is the same one Hugging Face’s Training Agents series pushes:

And I went with small open-weights models (Qwen2.5–1.5B-Instruct) with LoRA so the whole thing fits free Colab or a modest GPU.

This post is the first step, supervised fine-tuning on agent traces, with proper masking, a tiny hold-out set, and a clear picture of how the training file fine-tunes the base model.

Open source vs open weights…they are not the same

I wanted to touch on the difference between open source and open weights…the two terms are often used interchangeably…

People say “open model” and mean three different things. For this blog, the important split is:

And…

So in this project I trained an open-weights instruct model, Qwen2.5–1.5B-Instruct.

You can pull it from Hugging Face, attach LoRA, and own the adapter.

That is the greater goal of the exercise you are not stuck behind a closed API for the student model.

(Many open-weights models also have open or permissive licenses for the weights; always read the model card. “Open weights” does not equal automatically “do anything commercially”.

I wondered, can LoRA only be done on an open-weights model?

LoRA stands for Low-Rank Adaptation, small trainable adapter matrices that sit on top of a base model so you do not rewrite every parameter.

For the kind of LoRA I wanted to do here ( train and save the adapter myself): yes …you need the model weights.

LoRA plugs into the model and updates those small matrices during training. That only works if you can:

1. Load the model parameters

2. Run forward and backward passes on them

3. Save the adapter folder

LoRA is not magically limited to open source. It is limited to models whose weights you can load. In practice that is almost always open weights (or private weights you already have).

LoRA needs the model’s numbers on disk or GPU; an API that only returns text is not enough. That is why this whole vibe SFT story is built around an open-weights student.

What does “vibe” have to do with it

My prompt:

“I want to follow the Hugging Face Training Agents SFT-on-traces approach. Create a minimal reproducible project that:

- Loads example agent traces (or a placeholder JSONL),
- Converts them to prompt-completion format with proper masking for assistant-only loss,
- Runs a LoRA SFT on Qwen2.5–1.5B-Instruct or Gemma-2–2B using TRL,
- Includes basic evaluation on format correctness.
- Keep everything runnable on a free Colab T4 and well commented so I can learn.”

Classic fine-tuning blogs assume you hand-write every script. Vibe means the agent harness is in the loop:

AI Agent Harness Loop

So:

Vibe = the pipeline gets built and iterated (agent-assisted).

SFT / RFT = what learning algorithm you run

Without the vibe loop you can still SFT by hand. With it, the barrier drops: the harness scaffolds convert/train/eval while you stay on the science and the product intent.

The big picture pipeline

In this minimal Colab lab I used synthetic traces so anyone can run it. In a real distillation setup, those traces come from a larger and stronger model in a real harness.

Below the pipelline:

1. Load traces — multi-turn JSONL with system / user / assistant / tool
2. Convert — each assistant turn becomes one row:
   • prompt = context before that turn
   • completion = the assistant message
   → 12 traces expand to 24 training examples
3. LoRA SFT on Qwen/Qwen2.5-1.5B-Instruct with:
   • 4-bit QLoRA (T4-friendly)
   • completion_only_loss=True (prompt tokens masked)
   • optional google/gemma-2-2b-it via --model (gated)
4. Format eval — tool syntax, role-leak, empty/non-empty, tool-or-answer

Is this model distillation?

Yes, when the traces come from a stronger model.

The traces are really just a set of inputs and outputs. The inputs are user input and then the outputs is from the model.

So these can easily be replicated by running utterances against the model and collating the inputs and outputs.

Teacher → Student Distillation

Raw form: a trace

One trace is a complete agent session: one task across several turns.

TRACE (one coding task)
──────────────────────
system: “You are a coding agent. Tools look like: invoke tool …”
user: “List files in /tmp and count them.”
assistant: “I’ll list them.
invoke tool list_dir with path is /tmp”
tool: “a.txt
b.py
notes.md”
assistant: “There are 3 files: a.txt, b.py, notes.md.”

In the demo file I used 12 traces.

from pathlib import Path
import json

ROOT = Path("sft-agent-traces")
(ROOT / "data").mkdir(parents=True, exist_ok=True)
(ROOT / "outputs").mkdir(parents=True, exist_ok=True)
(ROOT / "src").mkdir(parents=True, exist_ok=True)

TRACES = [
  {
    "trace_id": "trace-001",
    "source": "synthetic-coding-agent",
    "messages": [
      {"role": "system", "content": "You are a helpful coding agent. Use tools when needed. Respond with tool calls in this format:\ninvoke tool tool_name with arg1 is value1\nWhen finished, give a clear final answer."},
      {"role": "user", "content": "List the files in /tmp and tell me how many there are."},
      {"role": "assistant", "content": "I'll list the contents of /tmp for you.\n\ninvoke tool list_dir with path is /tmp"},
      {"role": "tool", "content": "file_a.txt\nfile_b.py\nnotes.md"},
      {"role": "assistant", "content": "There are **3** files in `/tmp`:\n1. `file_a.txt`\n2. `file_b.py`\n3. `notes.md`"},
    ],
  },
  {
    "trace_id": "trace-002",
    "source": "synthetic-coding-agent",
    "messages": [
      {"role": "system", "content": "You are a helpful coding agent. Use tools when needed. Respond with tool calls in this format:\ninvoke tool tool_name with arg1 is value1\nWhen finished, give a clear final answer."},
      {"role": "user", "content": "What does README.md say in its first line?"},
      {"role": "assistant", "content": "I'll read the README.\n\ninvoke tool read_file with path is README.md"},
      {"role": "tool", "content": "# Agent SFT Demo\n\nMinimal reproducible project."},
      {"role": "assistant", "content": "The first line of `README.md` is:\n\n```\n# Agent SFT Demo\n```"},
    ],
  },
  {
    "trace_id": "trace-003",
    "source": "synthetic-coding-agent",
    "messages": [
      {"role": "system", "content": "You are a helpful coding agent. Use tools when needed. Respond with tool calls in this format:\ninvoke tool tool_name with arg1 is value1\nWhen finished, give a clear final answer."},
      {"role": "user", "content": "Find where SFTConfig is used and quote the learning rate."},
      {"role": "assistant", "content": "Searching for SFTConfig.\n\ninvoke tool grep with pattern is SFTConfig"},
      {"role": "tool", "content": "src/train_sft.py:42:    args = SFTConfig(\nsrc/train_sft.py:48:        learning_rate=2e-4,"},
      {"role": "assistant", "content": "Found it. Reading that section.\n\ninvoke tool read_file with path is src/train_sft.py"},
      {"role": "tool", "content": "learning_rate=2e-4,"},
      {"role": "assistant", "content": "The learning rate is set to **`2e-4`** in `SFTConfig`."},
    ],
  },

Training form: prompt–completion rows

We do not feed an entire multi-turn conversation as a single “predict everything” example.

Instead we expand every assistant turn into its own supervised training row.

From the trace above you get two training rows:

Row A prompt: system + user completion: first assistant (the tool call)

Row B prompt: system + user + assistant1 + tool result completion: second assistant (the final answer)

messages: [sys] [user] [asst1] [tool] [asst2]
            │     │      │
            └──── Row A ─┘
                         │
            └────────── Row B ──────────┘

This structure is deliberate. The model is trained on the intermediate tool-calling behaviour, not only on the final “done” message.

In the lab run: 12 traces → 24 prompt–completion examples.

Train vs hold-out (testing data)

Full disclosure, I don’t know if I fully understand this part, Ben gives a good breakdown here. But I want to get my head around it for future projects….

You must keep some examples completely out of the weight updates. That set is the hold-out (validation / eval split).

All converted examples (24)
┌────────────────────────────────────────┐
│ TRAIN (used to update LoRA)     ~80%   │ → 20 examples
│ ████████████████████                   │
│ HOLD-OUT / EVAL (only measure)  ~15%   │ → 4 examples
│ ░░░░                                   │
└────────────────────────────────────────┘

If training loss falls toward zero while hold-out loss rises, the model is simply memorising the 20 rows.

It is not learning a general agent. With a tiny demo set this is expected and useful to observe.

How much hold-out?

A common default is 10–20 % when you have enough data.

With only 24 rows, 15 % is acceptable for a lab exercise.

In a real training run you want a larger, cleaner hold-out set (and a separate test set that is almost never touched).

Is 20 training examples enough?

Enough to demonstrate the pipeline and sometimes to shift output format. Not enough to claim a production-ready agent. Real SFT baselines use hundreds to tens of thousands of diverse traces.

I deliberately wanted to train on the smallest dataset possible.

As a recap:

1. Agentic Supervised Fine-Tuning on a small open-weights model.

2. Open weights is not open source; I fine-tune models whose weights I can download.

3. LoRA needs loadable weights, DIY adapters are not something you bolt onto a closed chat API.

4. Teacher traces + student SFT = behaviour distillation.

5. Training data = expanded prompt / completion rows from multi-turn traces.

6. Mask the prompt (all of it for loss); grade the completion (all of it).

7. Hold out data only to measure, do not train on it.

8. The fine-tune file is usually a LoRA adapter sitting on a frozen/quantised base.

9. The vibe part is the harness that builds and iterates this pipeline with you.

What I want to do next

As a follow-up I would like to distill training data from a larger model and that the simulate my use-case or use in the smaller model. So in essence port the portion I use, to the smaller model and get the same or near-same results.

After that, I would love to perform fine-tuning by creating a feedback loop from usage data and continuously improving the model from actually usage data. And obviously select the most successful usage interactions as training data. A type of Success-filtered Continual Fine-Tuning.

Finally

Fine-tuning an open-weights model on agent traces should not mysterious… show the model good sessions, grade only the assistant’s lines, save a small adapter, measure on hold-out and with format checks.

The vibe is that you do not have to be a full-time ML engineer to drive the loop…an AI agent harness can carry the scaffolding while you stay responsible for the

• intent, the

• data quality, and

• whether the metrics actually mean anything.

Chief Evangelist @ Kore.ai | I’m passionate about exploring the intersection of AI and language. From Language Models, AI Agents to Agentic Applications, Development Frameworks & Data-Centric Productivity Tools, I share insights and ideas on how these technologies are shaping the future.

Introducing Trackio: A Lightweight Experiment Tracking Library from Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.huggingface.co