OK, here is what I wanted to achieve…I wanted to fine-tune the smallest model with the least amount of data. I also wanted to add a certain level of versatility, and that is what the LoRA approach gives you…more about that later.
Before we get into this project, I wanted to touch on two models launched recently. The Meta Muse Glimmer model and NVIDIA’s Nemotron Lightning models…these were described as agentic, local, always-on models. For local long running tasks.
I worked at an enterprise a number of years ago where fine-tuning NLU & acoustic models on a daily basis were common place. It feels to me that with the advent of language models, complete user reliance has fallen on the out-of-the-box capabilities of the model. And supplementing inference with context engineering.
But fine-tuning is not really something common…
I also think, as I have mentioned before, the future of agentic applications is multiple language models which are orchestrated in an agentic workflow.
Frontier models will always be relevant for coding, planning etc…all the hard frontier like tasks.
The Model
This Qwen model is a instruction-tuned (chat/instruction-following) version of the smallest model in Alibaba’s Qwen2.5 series of open-weight large language models.
I think it has strong instruction following and conversational ability despite its tiny size, with coding capabilities, 29+ languages and long context capabilities.
But I primarily chose it due to it being open weights and small.
LoRA
More on LoRA…something I like about this approach is that it is an adapter that is an independent fine-tuned layer sitting on top of the frozen base model.
You can therefore fine-tune the same base model for multiple use-cases (IT Helpdesk, HR Helpdesk, structured output, etc.) and simply swap the LoRA adapter at inference time.
This is a very practical way to optimise both training and hosting, especially when GPU capacity is limited.
The only tools you require is Colab and access to a Frontier Model. The data portion is important and interesting to understand, below I reference a good resource by Ben Burtenshaw.
The data
Ben Burtenshaw wrote a very good post on how to prepare data…but below are some observations…
Training data is pairs of input and output data:
{”id”:”T-7730”,”split”:”train”,”family”:”other”,
“user”:”Ticket T-7730. VIP cannot download their invoice PDF. The billing page returns HTTP 500. They need a human.”,
“gold”:{”id”:”T-7730”,”verdict”:”ESCALATE”,”amount_cents”:0,”reason_code”:”OTHER”},
“note”:”Broken invoice download for a VIP; escalate to engineering.”},Masking is “don’t grade this part.”
The model still reads the whole chat. Loss is only on the ticket card.
System text and the customer ticket get labels = -100, so the model is not trained to copy “You are DeskCard…” or the user’s story. It is only trained to write the assistant reply.
READ: system + user ticket + start of assistant
GRADE: the <<TICKET>> card only
tokens: [You] [are] [DeskCard] … [order] [never] [arrived] [Title] [:] [Late] [delivery]
labels: -100 -100 -100 … -100 -100 -100 4821 25 9012 7734
└──────── prompt, ignored ─────────┘ └── ticket card, trained ──┘Train / hold-out / test is three piles of tickets so you don’t cheat.
So…
Train = homework you practice on.
Hold-out = a practice quiz while you study. If hold-out loss goes down, you’re generalising a bit. If train gets perfect and hold-out gets worse, you’re just memorising the 8 cards.
Test = the real exam. Same six tickets, stock model vs fine-tuned model. The trainer never sees these.
That’s why we can say the card format transferred: the 6 test ids (T-2001…) were never in the 8 training cards.
The objective of the fine-tuning
The fine-tune teaches tiny Qwen 0.5B a fixed ticket-card format it does not use on its own.
Without fine-tuning, it writes polite chat (“I’m sorry to hear that…”). After eight labeled examples, it replies with a parseable <<TICKET>> card (verdict, amount, reason) so a downstream system can act on it.
So it is specialisation, not a smarter general assistant.
Input:
system: You are DeskCard, a first-line support clerk. Reply with one ticket card and nothing else.
user: Ticket T-2001. I was billed twice for invoice INV-2001 — forty-five dollars twice. Please fix the extra charge.Output (Prior to fine-Tuning):
I’m sorry to hear that you’re experiencing billing issues. Could you please provide me with more details about the invoices and any additional charges you’ve received? This information will help me assist you better.Output (With fine-Tuning):
<<TICKET>>
id: T-2001
verdict: REFUND
amount_cents: 4500
reason_code: DUPLICATE
note: Duplicate invoice ID; refund the second charge.
<</TICKET>>The main thing it learns is how to answer: a <<TICKET>> card instead of a chatty paragraph.
It also picks up some of the policy inside that card (refund a duplicate, escalate fraud, refuse an oversized refund).
That part is weaker: after training, all 6 test replies were cards, but only 4 had the right verdict/amount/reason. Format transferred; the full decision rule did not.
More on training data
Prompt (masked , no loss)
system: You are DeskCard, a first-line support clerk. Reply with one ticket card and nothing else.
user: Ticket T-1042. Customer says they were charged twice for the same Pro seat. Invoice shows two $49.99 lines. Refund the extra charge.
Completion (the label, this is what it learns to emit)
<<TICKET>>
id: T-1042
verdict: REFUND
amount_cents: 4999
reason_code: DUPLICATE
note: Duplicate Pro seat line; refund the extra 4999 cents.
<</TICKET>>There are seven more like this (another duplicate, two fraud, two policy, two other). The test tickets use new ids and wording so we are not just checking memorisation.
<|im_start|>system
You are DeskCard, a first-line support clerk. Reply with one ticket card and nothing else.<|im_end|>
<|im_start|>user
Ticket T-7730. VIP cannot download their invoice PDF. The billing page returns HTTP 500. They need a human.<|im_end|>
<|im_start|>assistant
<<TICKET>>
id: T-7730
verdict: ESCALATE
amount_cents: 0
reason_code: OTHER
note: Broken invoice download for a VIP; escalate to engineering.
<</TICKET>><|im_end|>The cut is always the same: everything through <|im_start|>assistant\n is MASK. The card plus the closing <|im_end|> is LOSS.
The notebook
You can find a copy of the notebook here. Below you can see how to set the runtime hardware accelerator to T4 GPU.
And the cell below shows the pre and post fine-tuning output…
Try a ticket the trainer never saw
Trying a ticket the trainer never saw…
def deskcard(user_text: str) -> str:
messages = [
{”role”: “system”, “content”: SYSTEM},
{”role”: “user”, “content”: user_text},
]
return generate_one(trainer.model, tok, messages)
probe = “Ticket T-9001. Charged twice for the same add-on, an extra 2800 cents. Refund the duplicate only.”
print(deskcard(probe))And the output…
<<TICKET>>
id: T-9001
verdict: REFUND
amount_cents: 2800
reason_code: DUPLICATE
note: Duplicate add_on_id ‘c0b54d67-e36f-4e2a-a1fc-084e24444ebe’; refund only
<</TICKET>>So the eight cards in the example did not turn Qwen 0.5B into a support agent. They changed one thing: what the model emits after it reads a ticket.
The 0.5B weights stayed frozen.
Fine-tuning added a small LoRA adapter…
Forty steps on eight labeled cards, completion-only loss, about 25 seconds once the base model is cached. The optimiser never saw the six test tickets.
That is the scale of the edit: a thin adapter, not a new clerk.
What changed is the output contract.
After the adapter, the same tickets come back as a five-field card wrapped in <<TICKET>>.
So the recipe is narrow on purpose. Write a format the base model will not emit.
The base model is still Qwen 0.5B; only a small adapter learned to write the object.
Chief Evangelist @ Kore.ai | I’m passionate about exploring the intersection of AI and language. From Language Models, AI Agents to Agentic Applications, Development Frameworks & Data-Centric Productivity Tools, I share insights and ideas on how these technologies are shaping the future.
COBUS GREYLING - At the intersection of AI & Language
Cobus Greyling is an AI Evangelist & thought leader dedicated to exploring the intersection of artificial intelligence…www.cobusgreyling.com
GitHub - cobusgreyling/qwen-05b-min-sft: Least-data SFT on Qwen2.5-0.5B-Instruct: teach a…
Least-data SFT on Qwen2.5-0.5B-Instruct: teach a ticket-card format with 8 examples. Colab T4 notebook + LoRA. Open…github.com