Supervised Fine-Tuning (SFT) with LoRA/QLoRA using TRL — on a Free Colab Notebook (50')¶

Easily fine-tune Large Language Models (LLMs) or Vision-Language Models (VLMs) with LoRA or QLoRA using the Transformers Reinforcement Learning (TRL) library built by Hugging Face — all within a free Google Colab notebook (powered by a T4/L4 GPU.).

In your VSCode, install the "Colab" plugin, and select kernel to run the notebook on Colab L4 python3 kernel.

• TRL GitHub Repository — star us to support the project!
• Official TRL Examples
• Community Tutorials

Key concepts¶

• SFT: Trains models from example input-output pairs to align behavior with human preferences.
• LoRA: Updates only a few low-rank parameters, reducing training cost and memory.
• QLoRA: A quantized version of LoRA that enables even larger models to fit on small GPUs.
• TRL: The Hugging Face library that makes fine-tuning and reinforcement learning simple and efficient.

Learn how to perform Supervised Fine-Tuning (SFT) with LoRA/QLoRA using TRL.

Your submission:

(1) Run through the following code in colab to test it

(2) Make it a separate file and runnable on OSCER with full dataset SFT, and report the answer on the test set.

(3) Write a report for the performances, It is optional to run further parameter tuning.

Install dependencies¶

We'll install TRL with the PEFT extra, which ensures all main dependencies such as Transformers and PEFT (a package for parameter-efficient fine-tuning, e.g., LoRA/QLoRA) are included. Additionally, we'll install trackio to log and monitor our experiments, and bitsandbytes to enable quantization of LLMs, reducing memory consumption for both inference and training.

In [1]:

!pip install -Uq "trl[peft]" trackio bitsandbytes transformers accelerate

!pip install -Uq "trl[peft]" trackio bitsandbytes transformers accelerate

Log in to Hugging Face¶

Log in to your Hugging Face account to save your fine-tuned model, track your experiment results directly on the Hub or access gated models. You can find your access token on your account settings page.

In [2]:

from huggingface_hub import notebook_login
# When pasted into a Jupyter notebook, this will prompt for your Hugging Face credentials
# If you could not use shortcut to paste, then you can use vscode menu "Edit -> Paste" to paste it in. 
# You could find our tokens at https://huggingface.co/settings/tokens
# Alternatively, you can run `huggingface-cli login` in your terminal.
notebook_login()

from huggingface_hub import notebook_login # When pasted into a Jupyter notebook, this will prompt for your Hugging Face credentials # If you could not use shortcut to paste, then you can use vscode menu "Edit -> Paste" to paste it in. # You could find our tokens at https://huggingface.co/settings/tokens # Alternatively, you can run `huggingface-cli login` in your terminal. notebook_login()

VBox(children=(HTML(value='<center> <img\nsrc=https://huggingface.co/front/assets/huggingface_logo-noborder.sv…

Load Dataset¶

In this step, we load the SetFit/sst5 dataset from the Hugging Face Hub using the datasets library.
For efficiency, we'll load only the training split:

In [3]:

from datasets import load_dataset

dataset_name = "SetFit/sst5"
# let 's use only 5% of the training data for faster experimentation
# When running on OSCER, please use the full dataset by changing to split="train"
train_dataset = load_dataset(dataset_name, split="train[0%:5%]")
val_dataset = load_dataset(dataset_name, split="validation")
test_dataset = load_dataset(dataset_name, split="test")

from datasets import load_dataset dataset_name = "SetFit/sst5" # let 's use only 5% of the training data for faster experimentation # When running on OSCER, please use the full dataset by changing to split="train" train_dataset = load_dataset(dataset_name, split="train[0%:5%]") val_dataset = load_dataset(dataset_name, split="validation") test_dataset = load_dataset(dataset_name, split="test")

/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_auth.py:104: UserWarning: 
Error while fetching `HF_TOKEN` secret value from your vault: 'Requesting secret HF_TOKEN timed out. Secrets can only be fetched when running from the Colab UI.'.
You are not authenticated with the Hugging Face Hub in this notebook.
If the error persists, please let us know by opening an issue on GitHub (https://github.com/huggingface/huggingface_hub/issues/new).
  warnings.warn(
Repo card metadata block was not found. Setting CardData to empty.
WARNING:huggingface_hub.repocard:Repo card metadata block was not found. Setting CardData to empty.
Repo card metadata block was not found. Setting CardData to empty.
WARNING:huggingface_hub.repocard:Repo card metadata block was not found. Setting CardData to empty.
Repo card metadata block was not found. Setting CardData to empty.
WARNING:huggingface_hub.repocard:Repo card metadata block was not found. Setting CardData to empty.

This dataset contains different columns. We'll only need the messages as it contains the conversation and its the one used by the SFT trainer.

In [4]:

train_dataset

train_dataset

Out[4]:

Dataset({
    features: ['text', 'label', 'label_text'],
    num_rows: 427
})

Let's see a full example to understand the internal structure:

In [5]:

train_dataset[0]

train_dataset[0]

Out[5]:

{'text': 'a stirring , funny and finally transporting re-imagining of beauty and the beast and 1930s horror films',
 'label': 4,
 'label_text': 'very positive'}

In [6]:

# This Preprocess Function takes the example and puts it in a particular prompt format.
# Below I set the LLM to be a system that needs to classify the given example sentence.
# Then after the LLM analyzes the prompt, the LLM takes the assistant role and tries
# to think about how to get the given output result.

def preprocess_function(example):
    return {
        "prompt": [{"role": "system", "content": f"Classify a sentence into 5 categories: very positive, positive, neutral, negative, very negative."},{"role": "user", "content": f"Text: {example['text']}"}],
        "completion": [
            {"role": "assistant", "content": f"Answer: {example['label_text']}"}
        ],
    }
train_dataset = train_dataset.map(preprocess_function)

# This Preprocess Function takes the example and puts it in a particular prompt format. # Below I set the LLM to be a system that needs to classify the given example sentence. # Then after the LLM analyzes the prompt, the LLM takes the assistant role and tries # to think about how to get the given output result. def preprocess_function(example): return { "prompt": [{"role": "system", "content": f"Classify a sentence into 5 categories: very positive, positive, neutral, negative, very negative."},{"role": "user", "content": f"Text: {example['text']}"}], "completion": [ {"role": "assistant", "content": f"Answer: {example['label_text']}"} ], } train_dataset = train_dataset.map(preprocess_function)

In [7]:

train_dataset[0]

train_dataset[0]

Out[7]:

{'text': 'a stirring , funny and finally transporting re-imagining of beauty and the beast and 1930s horror films',
 'label': 4,
 'label_text': 'very positive',
 'prompt': [{'content': 'Classify a sentence into 5 categories: very positive, positive, neutral, negative, very negative.',
   'role': 'system'},
  {'content': 'Text: a stirring , funny and finally transporting re-imagining of beauty and the beast and 1930s horror films',
   'role': 'user'}],
 'completion': [{'content': 'Answer: very positive', 'role': 'assistant'}]}

In [8]:

train_dataset = train_dataset.remove_columns(column_names=['text', 'label', 'label_text'])

train_dataset = train_dataset.remove_columns(column_names=['text', 'label', 'label_text'])

In [9]:

train_dataset[0]

train_dataset[0]

Out[9]:

{'prompt': [{'content': 'Classify a sentence into 5 categories: very positive, positive, neutral, negative, very negative.',
   'role': 'system'},
  {'content': 'Text: a stirring , funny and finally transporting re-imagining of beauty and the beast and 1930s horror films',
   'role': 'user'}],
 'completion': [{'content': 'Answer: very positive', 'role': 'assistant'}]}

Load model and configure LoRA/QLoRA¶

This notebook can be used with two fine-tuning methods. By default, it is set up for QLoRA, which includes quantization using BitsAndBytesConfig. If you prefer to use standard LoRA without quantization, simply comment out the BitsAndBytesConfig configuration.

Below, choose your preferred model. All of the options have been tested on free Colab instances.

In [19]:

# Select one model below by uncommenting the line you want to use 👇
## Qwen
# model_id, output_dir = "unsloth/qwen3-14b-unsloth-bnb-4bit", "qwen3-14b-unsloth-bnb-4bit-SFT"     # ⚠️ ~14.1 GB VRAM
# model_id, output_dir = "Qwen/Qwen3-8B", "Qwen3-8B-SFT"                                          # ⚠️ ~12.8 GB VRAM
# model_id, output_dir = "Qwen/Qwen2.5-7B-Instruct", "Qwen2.5-7B-Instruct"                        # ✅ ~10.8 GB VRAM

## Llama
# model_id, output_dir = "meta-llama/Llama-3.2-3B-Instruct", "Llama-3.2-3B-Instruct"              # ✅ ~4.7 GB VRAM
# model_id, output_dir = "meta-llama/Llama-3.1-8B-Instruct", "Llama-3.1-8B-Instruct"              # ⚠️ ~10.9 GB VRAM

## Gemma
# model_id, output_dir = "google/gemma-3n-E2B-it", "gemma-3n-E2B-it"                              # ❌ Upgrade to a higher tier of colab
# model_id, output_dir = "google/gemma-3-4b-it", "gemma-3-4b-it"                                  # ⚠️ ~6.8 GB VRAM

model_id, output_dir = "google/gemma-3-1b-it", "gemma-3-1b-it-sst5"
#model_id, output_dir = "google/gemma-3-270m", "gemma-3-270m-sst5"

## Granite
#model_id, output_dir = "ibm-granite/granite-4.0-micro", "granite-4.0-micro"                      # ✅ ~3.3 GB VRAM

## LFM2
#model_id, output_dir = "LiquidAI/LFM2-2.6B", "LFM2-2.6B-SFT"                                     # ✅ ~5.89 GB VRAM

# Select one model below by uncommenting the line you want to use 👇 ## Qwen # model_id, output_dir = "unsloth/qwen3-14b-unsloth-bnb-4bit", "qwen3-14b-unsloth-bnb-4bit-SFT" # ⚠️ ~14.1 GB VRAM # model_id, output_dir = "Qwen/Qwen3-8B", "Qwen3-8B-SFT" # ⚠️ ~12.8 GB VRAM # model_id, output_dir = "Qwen/Qwen2.5-7B-Instruct", "Qwen2.5-7B-Instruct" # ✅ ~10.8 GB VRAM ## Llama # model_id, output_dir = "meta-llama/Llama-3.2-3B-Instruct", "Llama-3.2-3B-Instruct" # ✅ ~4.7 GB VRAM # model_id, output_dir = "meta-llama/Llama-3.1-8B-Instruct", "Llama-3.1-8B-Instruct" # ⚠️ ~10.9 GB VRAM ## Gemma # model_id, output_dir = "google/gemma-3n-E2B-it", "gemma-3n-E2B-it" # ❌ Upgrade to a higher tier of colab # model_id, output_dir = "google/gemma-3-4b-it", "gemma-3-4b-it" # ⚠️ ~6.8 GB VRAM model_id, output_dir = "google/gemma-3-1b-it", "gemma-3-1b-it-sst5" #model_id, output_dir = "google/gemma-3-270m", "gemma-3-270m-sst5" ## Granite #model_id, output_dir = "ibm-granite/granite-4.0-micro", "granite-4.0-micro" # ✅ ~3.3 GB VRAM ## LFM2 #model_id, output_dir = "LiquidAI/LFM2-2.6B", "LFM2-2.6B-SFT" # ✅ ~5.89 GB VRAM

Let's load the selected model using transformers, configuring QLoRA via bitsandbytes (you can remove it if doing LoRA). We don't need to configure the tokenizer since the trainer takes care of that automatically.

In [20]:

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    attn_implementation="eager",                   # Changed to eager attention as recommended for Gemma3
    dtype=torch.bfloat16,                          # Change to bfloat16 if GPU has support, L4 GPUs on Colab will support it
    use_cache=True,                               # Whether to cache attention outputs to speed up inference
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,                        # Load the model in 4-bit precision to save memory
        bnb_4bit_compute_dtype=torch.float16,     # Data type used for internal computations in quantization
        bnb_4bit_use_double_quant=True,           # Use double quantization to improve accuracy
        bnb_4bit_quant_type="nf4"                 # Type of quantization. "nf4" is recommended for recent LLMs
    )
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

import torch from transformers import AutoModelForCausalLM, BitsAndBytesConfig, AutoTokenizer model = AutoModelForCausalLM.from_pretrained( model_id, attn_implementation="eager", # Changed to eager attention as recommended for Gemma3 dtype=torch.bfloat16, # Change to bfloat16 if GPU has support, L4 GPUs on Colab will support it use_cache=True, # Whether to cache attention outputs to speed up inference quantization_config=BitsAndBytesConfig( load_in_4bit=True, # Load the model in 4-bit precision to save memory bnb_4bit_compute_dtype=torch.float16, # Data type used for internal computations in quantization bnb_4bit_use_double_quant=True, # Use double quantization to improve accuracy bnb_4bit_quant_type="nf4" # Type of quantization. "nf4" is recommended for recent LLMs ) ) tokenizer = AutoTokenizer.from_pretrained(model_id)

The following cell defines LoRA (or QLoRA if needed). When training with LoRA/QLoRA, we use a base model (the one selected above) and, instead of modifying its original weights, we fine-tune a LoRA adapter — a lightweight layer that enables efficient and memory-friendly training. The target_modules specify which parts of the model (e.g., attention or projection layers) will be adapted by LoRA during fine-tuning.

In [21]:

from peft import LoraConfig

# You may need to update `target_modules` depending on the architecture of your chosen model.
# For example, different LLMs might have different attention/projection layer names.
peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj",],
)

from peft import LoraConfig # You may need to update `target_modules` depending on the architecture of your chosen model. # For example, different LLMs might have different attention/projection layer names. peft_config = LoraConfig( r=16, lora_alpha=32, target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj",], )

Train model¶

We'll configure SFT using SFTConfig, keeping the parameters minimal so the training fits on a free Colab instance. You can adjust these settings if more resources are available. For full details on all available parameters, check the TRL SFTConfig documentation.

In [22]:

from trl import SFTConfig

training_args = SFTConfig(
    # completion_loss_only=False,            # Compute loss only on the completion part of the sequence By default it is true
    # Training schedule / optimization
    per_device_train_batch_size = 1,      # Batch size per GPU
    gradient_accumulation_steps = 4,      # Gradients are accumulated over multiple steps → effective batch size = 2 * 8 = 16
    warmup_steps = 5,
    num_train_epochs = 1,               # Number of full dataset passes. For shorter training, use `max_steps` instead (this case)
    # max_steps = 100,
    learning_rate = 2e-4,                 # Learning rate for the optimizer
    optim = "paged_adamw_8bit",           # Optimizer

    # Logging / reporting
    logging_steps=1,                      # Log training metrics every N steps
    report_to="trackio",                  # Experiment tracking tool
    trackio_space_id=output_dir,          # HF Space where the experiment tracking will be saved
    output_dir=output_dir,                # Where to save model checkpoints and logs

    max_length=512,                       # Reduced max_length to save memory
    use_liger_kernel=False,                # Enable Liger kernel optimizations for faster training
    activation_offloading=False,          # Offload activations to CPU to reduce GPU memory usage (DISABLED for troubleshooting accelerate state)
    gradient_checkpointing=True,          # Enabled gradient_checkpointing to save memory

    # Hub integration
    push_to_hub=True,                     # Automatically push the trained model to the Hugging Face Hub
                                          # The model will be saved under your Hub account in the repository named `output_dir`

    gradient_checkpointing_kwargs={"use_reentrant": False}, # To prevent warning message
)

from trl import SFTConfig training_args = SFTConfig( # completion_loss_only=False, # Compute loss only on the completion part of the sequence By default it is true # Training schedule / optimization per_device_train_batch_size = 1, # Batch size per GPU gradient_accumulation_steps = 4, # Gradients are accumulated over multiple steps → effective batch size = 2 * 8 = 16 warmup_steps = 5, num_train_epochs = 1, # Number of full dataset passes. For shorter training, use `max_steps` instead (this case) # max_steps = 100, learning_rate = 2e-4, # Learning rate for the optimizer optim = "paged_adamw_8bit", # Optimizer # Logging / reporting logging_steps=1, # Log training metrics every N steps report_to="trackio", # Experiment tracking tool trackio_space_id=output_dir, # HF Space where the experiment tracking will be saved output_dir=output_dir, # Where to save model checkpoints and logs max_length=512, # Reduced max_length to save memory use_liger_kernel=False, # Enable Liger kernel optimizations for faster training activation_offloading=False, # Offload activations to CPU to reduce GPU memory usage (DISABLED for troubleshooting accelerate state) gradient_checkpointing=True, # Enabled gradient_checkpointing to save memory # Hub integration push_to_hub=True, # Automatically push the trained model to the Hugging Face Hub # The model will be saved under your Hub account in the repository named `output_dir` gradient_checkpointing_kwargs={"use_reentrant": False}, # To prevent warning message )

Configure the SFT Trainer. We pass the previously configured training_args. We don't use eval dataset to mantain memory usage low but you can configure it.

In [23]:

from trl import SFTTrainer
from transformers import DataCollatorForLanguageModeling

data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    peft_config=peft_config,
    data_collator=data_collator,
    processing_class=tokenizer # Explicitly specify the instantiated tokenizer as the processing class
)

from trl import SFTTrainer from transformers import DataCollatorForLanguageModeling data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False) trainer = SFTTrainer( model=model, args=training_args, train_dataset=train_dataset, peft_config=peft_config, data_collator=data_collator, processing_class=tokenizer # Explicitly specify the instantiated tokenizer as the processing class )

Show memory stats before training

In [24]:

gpu_stats = torch.cuda.get_device_properties(0)
start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)
max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)

print(f"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.")
print(f"{start_gpu_memory} GB of memory reserved.")

gpu_stats = torch.cuda.get_device_properties(0) start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3) max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3) print(f"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.") print(f"{start_gpu_memory} GB of memory reserved.")

GPU = NVIDIA L4. Max memory = 22.161 GB.
4.195 GB of memory reserved.

In [25]:

# Let's inspect a training batch
batch = next(iter(trainer.get_train_dataloader()))
for k, v in batch.items():
    print(k, v.shape)

print(tokenizer.decode(batch["input_ids"][0]))

# Let's inspect a training batch batch = next(iter(trainer.get_train_dataloader())) for k, v in batch.items(): print(k, v.shape) print(tokenizer.decode(batch["input_ids"][0]))

input_ids torch.Size([1, 57])
completion_mask torch.Size([1, 57])
attention_mask torch.Size([1, 57])
labels torch.Size([1, 57])
<bos><start_of_turn>user
Classify a sentence into 5 categories: very positive, positive, neutral, negative, very negative.

Text: this is a stunning film , a one-of-a-kind tour de force .<end_of_turn>
<start_of_turn>model
Answer: very positive<end_of_turn>

And train!

In [27]:

# training will language Transformer Language Model with SFT
# create Trackio experiment to monitor training
# This training will take 3-5 minutes on a GPU with 16GB VRAM
# The dashboard can be viewed at the following, and also at https://huggingface.co/spaces/your-username/your-space-name (replace with your info)
# You should be able to see training progress and metrics in real-time on the Trackio dashboard.
import os
os.environ["DISABLE_LIGER"] = "1"

trainer_stats = trainer.train()

# training will language Transformer Language Model with SFT # create Trackio experiment to monitor training # This training will take 3-5 minutes on a GPU with 16GB VRAM # The dashboard can be viewed at the following, and also at https://huggingface.co/spaces/your-username/your-space-name (replace with your info) # You should be able to see training progress and metrics in real-time on the Trackio dashboard. import os os.environ["DISABLE_LIGER"] = "1" trainer_stats = trainer.train()

[107/107 04:59, Epoch 1/1]

Step	Training Loss
1	7.441400
2	7.798400
3	7.855800
4	6.124400
5	4.735700
6	4.128100
7	3.384700
8	3.159100
9	3.046800
10	2.978600
11	2.135700
12	1.986000
13	2.172300
14	2.103800
15	2.466100
16	1.999800
17	2.014900
18	2.117000
19	2.061900
20	1.899000
21	1.902100
22	2.137700
23	1.666700
24	1.578600
25	1.368700
26	1.769500
27	1.589600
28	1.563000
29	1.689400
30	1.777600
31	1.461400
32	1.536700
33	1.408200
34	1.453300
35	1.996300
36	1.337800
37	1.813100
38	1.371200
39	2.158700
40	1.084600
41	1.934700
42	2.088900
43	1.482100
44	1.280200
45	1.690200
46	1.617300
47	1.475400
48	1.444700
49	1.466000
50	1.411700
51	1.668100
52	1.374400
53	1.773700
54	1.736300
55	1.336000
56	1.316900
57	1.443000
58	1.599300
59	1.216800
60	1.418800
61	1.693800
62	1.567300
63	1.728900
64	1.883800
65	1.325700
66	1.519300
67	2.218000
68	1.883200
69	1.366600
70	1.527400
71	1.712900
72	1.609700
73	1.708100
74	1.469200
75	1.591300
76	1.502100
77	1.714100
78	0.984300
79	1.507200
80	1.820800
81	0.754000
82	1.976100
83	2.058900
84	1.590300
85	1.481600
86	1.738700
87	1.496800
88	1.334300
89	1.549900
90	1.307800
91	1.447900
92	1.122900
93	1.074500
94	2.111000
95	1.288900
96	1.343600
97	1.677900
98	1.327200
99	1.757900
100	1.476000
101	1.352600
102	1.557900
103	0.873400
104	1.287500
105	1.328400
106	1.741400
107	1.442900

* Run finished. Uploading logs to Trackio (please wait...)

Show memory stats after training

In [28]:

used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)
used_memory_for_lora = round(used_memory - start_gpu_memory, 3)
used_percentage = round(used_memory / max_memory * 100, 3)
lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)

print(f"{trainer_stats.metrics['train_runtime']} seconds used for training.")
print(f"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.")
print(f"Peak reserved memory = {used_memory} GB.")
print(f"Peak reserved memory for training = {used_memory_for_lora} GB.")
print(f"Peak reserved memory % of max memory = {used_percentage} %.")
print(f"Peak reserved memory for training % of max memory = {lora_percentage} %.")

used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3) used_memory_for_lora = round(used_memory - start_gpu_memory, 3) used_percentage = round(used_memory / max_memory * 100, 3) lora_percentage = round(used_memory_for_lora / max_memory * 100, 3) print(f"{trainer_stats.metrics['train_runtime']} seconds used for training.") print(f"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.") print(f"Peak reserved memory = {used_memory} GB.") print(f"Peak reserved memory for training = {used_memory_for_lora} GB.") print(f"Peak reserved memory % of max memory = {used_percentage} %.") print(f"Peak reserved memory for training % of max memory = {lora_percentage} %.")

301.0632 seconds used for training.
5.02 minutes used for training.
Peak reserved memory = 4.195 GB.
Peak reserved memory for training = 0.0 GB.
Peak reserved memory % of max memory = 18.93 %.
Peak reserved memory for training % of max memory = 0.0 %.

The training procedure generates both standard training logs and trackio logs, which help us monitor the training progress. Example outputs would look like the following:

Saving fine tuned model¶

In this step, we save the fine-tuned model both locally and to the Hugging Face Hub using the credentials from your account.

In [29]:

trainer.save_model(output_dir)
trainer.push_to_hub(dataset_name=dataset_name)

trainer.save_model(output_dir) trainer.push_to_hub(dataset_name=dataset_name)

Processing Files (0 / 0)      : |          |  0.00B /  0.00B            

New Data Upload               : |          |  0.00B /  0.00B            

  ...it-sst5/training_args.bin: 100%|##########| 6.22kB / 6.22kB            

  ...b-it-sst5/tokenizer.model: 100%|##########| 4.69MB / 4.69MB            

  ...1b-it-sst5/tokenizer.json: 100%|##########| 33.4MB / 33.4MB            

  ...adapter_model.safetensors:  80%|########  | 41.9MB / 52.2MB            

No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.

Processing Files (0 / 0)      : |          |  0.00B /  0.00B            

New Data Upload               : |          |  0.00B /  0.00B            

  ...it-sst5/training_args.bin: 100%|##########| 6.22kB / 6.22kB            

  ...b-it-sst5/tokenizer.model: 100%|##########| 4.69MB / 4.69MB            

  ...1b-it-sst5/tokenizer.json: 100%|##########| 33.4MB / 33.4MB            

  ...adapter_model.safetensors:  80%|########  | 41.9MB / 52.2MB            

Out[29]:

CommitInfo(commit_url='https://huggingface.co/mlciv/gemma-3-1b-it-sst5/commit/9391502a84561884f03a6f60d9a8e63b7b8ec3ca', commit_message='End of training', commit_description='', oid='9391502a84561884f03a6f60d9a8e63b7b8ec3ca', pr_url=None, repo_url=RepoUrl('https://huggingface.co/mlciv/gemma-3-1b-it-sst5', endpoint='https://huggingface.co', repo_type='model', repo_id='mlciv/gemma-3-1b-it-sst5'), pr_revision=None, pr_num=None)

Load the fine-tuned model and run inference¶

Now, let's test our fine-tuned model by loading the LoRA/QLoRA adapter and performing inference. We'll start by loading the base model, then attach the adapter to it, creating the final fine-tuned model ready for evaluation.

In [30]:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

adapter_model = f"mlciv/{output_dir}" # Replace with your HF username or organization

base_model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto")

tokenizer = AutoTokenizer.from_pretrained(model_id)

from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel adapter_model = f"mlciv/{output_dir}" # Replace with your HF username or organization base_model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto") tokenizer = AutoTokenizer.from_pretrained(model_id)

Let's create a sample message using the dataset's structure. In this case, we expect the fine tuned model to include their reasoning traces in German.

In [31]:

train_dataset[0]

train_dataset[0]

Out[31]:

{'prompt': [{'content': 'Classify a sentence into 5 categories: very positive, positive, neutral, negative, very negative.',
   'role': 'system'},
  {'content': 'Text: a stirring , funny and finally transporting re-imagining of beauty and the beast and 1930s horror films',
   'role': 'user'}],
 'completion': [{'content': 'Answer: very positive', 'role': 'assistant'}]}

In [32]:

messages = train_dataset[0]['prompt']
messages

messages = train_dataset[0]['prompt'] messages

Out[32]:

[{'content': 'Classify a sentence into 5 categories: very positive, positive, neutral, negative, very negative.',
  'role': 'system'},
 {'content': 'Text: a stirring , funny and finally transporting re-imagining of beauty and the beast and 1930s horror films',
  'role': 'user'}]

Let's first check what's the output for the base model, without the adapter. You will find the base model did output as we wish, e.g. output long reasoning or no answer.

In [33]:

text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
print(text)
model_inputs = tokenizer([text], return_tensors="pt").to(base_model.device)

generated_ids = base_model.generate(
    **model_inputs,
    max_new_tokens=50,
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):]

# Decode and extract model response
generated_text = tokenizer.decode(output_ids, skip_special_tokens=True)
print(generated_text)

text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) print(text) model_inputs = tokenizer([text], return_tensors="pt").to(base_model.device) generated_ids = base_model.generate( **model_inputs, max_new_tokens=50, ) output_ids = generated_ids[0][len(model_inputs.input_ids[0]):] # Decode and extract model response generated_text = tokenizer.decode(output_ids, skip_special_tokens=True) print(generated_text)

<bos><start_of_turn>user
Classify a sentence into 5 categories: very positive, positive, neutral, negative, very negative.

Text: a stirring , funny and finally transporting re-imagining of beauty and the beast and 1930s horror films<end_of_turn>
<start_of_turn>model

Here's a classification of the sentence:

*   **Positive:** “stirling”, “funny”, “transporting re-imagining”, “beautiful” – These words evoke positive feelings and descriptions.
*   **Positive:** “finally

Now let us load the adapter model together with base model with PEFT.

In [34]:

fine_tuned_model = PeftModel.from_pretrained(base_model, adapter_model)

fine_tuned_model = PeftModel.from_pretrained(base_model, adapter_model)

adapter_config.json: 0.00B [00:00, ?B/s]

adapter_model.safetensors:   0%|          | 0.00/52.2M [00:00<?, ?B/s]

In [35]:

generated_ids = fine_tuned_model.generate(
    **model_inputs,
    max_new_tokens=50,
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):]

# Decode and extract model response
generated_text = tokenizer.decode(output_ids, skip_special_tokens=True)
print(generated_text)

generated_ids = fine_tuned_model.generate( **model_inputs, max_new_tokens=50, ) output_ids = generated_ids[0][len(model_inputs.input_ids[0]):] # Decode and extract model response generated_text = tokenizer.decode(output_ids, skip_special_tokens=True) print(generated_text)

model
Answer: positive

The model now generates the exact output as we wish!

Push Merged Model (for LoRA or QLoRA Training)¶

To serve the model via vLLM, the repository must contain the merged model (base model + LoRA adapter). Therefore, you need to upload it first.

In [45]:

model_merged = fine_tuned_model.merge_and_unload()

save_dir = f"{output_dir}-merged"

model_merged.save_pretrained(save_dir)
tokenizer.save_pretrained(save_dir)

model_merged = fine_tuned_model.merge_and_unload() save_dir = f"{output_dir}-merged" model_merged.save_pretrained(save_dir) tokenizer.save_pretrained(save_dir)

Out[45]:

('gemma-3-1b-it-sst5-merged/tokenizer_config.json',
 'gemma-3-1b-it-sst5-merged/special_tokens_map.json',
 'gemma-3-1b-it-sst5-merged/chat_template.jinja',
 'gemma-3-1b-it-sst5-merged/tokenizer.model',
 'gemma-3-1b-it-sst5-merged/added_tokens.json',
 'gemma-3-1b-it-sst5-merged/tokenizer.json')

In [46]:

model_merged.push_to_hub(f"mlciv/{output_dir}-merged") # Replace with your HF username or organization
tokenizer.push_to_hub(f"mlciv/{output_dir}-merged") # Replace with your HF username or organization

model_merged.push_to_hub(f"mlciv/{output_dir}-merged") # Replace with your HF username or organization tokenizer.push_to_hub(f"mlciv/{output_dir}-merged") # Replace with your HF username or organization

Processing Files (0 / 0)      : |          |  0.00B /  0.00B            

New Data Upload               : |          |  0.00B /  0.00B            

  ...-merged/model.safetensors:   2%|2         | 41.9MB / 2.00GB            

No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.

Processing Files (0 / 0)      : |          |  0.00B /  0.00B            

New Data Upload               : |          |  0.00B /  0.00B            

  ...t5-merged/tokenizer.model: 100%|##########| 4.69MB / 4.69MB            

  ...st5-merged/tokenizer.json: 100%|##########| 33.4MB / 33.4MB            

No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.

Out[46]:

CommitInfo(commit_url='https://huggingface.co/mlciv/gemma-3-1b-it-sst5-merged/commit/6d3d016454289b6b8780e8788411402049ba1586', commit_message='Upload tokenizer', commit_description='', oid='6d3d016454289b6b8780e8788411402049ba1586', pr_url=None, repo_url=RepoUrl('https://huggingface.co/mlciv/gemma-3-1b-it-sst5-merged', endpoint='https://huggingface.co', repo_type='model', repo_id='mlciv/gemma-3-1b-it-sst5-merged'), pr_revision=None, pr_num=None)

Congratulations! You have your own LLM built and shared to the public. You could access your model here. https://huggingface.co/youraccount

In [68]:

# Let's evaluate the fine-tuned model on the test dataset of sst5
from sklearn.metrics import accuracy_score
test_dataset = test_dataset.map(preprocess_function)
print("Sample test example:", test_dataset[0])
# We will collect the true labels and predicted labels to compute accuracy
true_labels = []
pred_labels = []
print("Evaluating the fine-tuned model on the test dataset...")
print("type of test_dataset:", type(test_dataset))
# Iterate over the subset of test dataset and generate predictions
for example in test_dataset.select(range(100)):
    messages = example['prompt']
    prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    #print("Prompt:", prompt)
    model_inputs = tokenizer([prompt], return_tensors="pt").to(base_model.device)
    generated_ids = fine_tuned_model.generate(
        **model_inputs,
        max_new_tokens=50,
    )
    output_ids = generated_ids[0][len(model_inputs.input_ids[0]):]

    # Decode and extract model response
    generated_text = tokenizer.decode(output_ids, skip_special_tokens=True)
    pred_label = generated_text.split("Answer:")[-1].strip()
    true_label = example['completion'][0]['content'].split("Answer:")[-1].strip()
        
    pred_labels.append(pred_label)
    true_labels.append(true_label)

# Compute accuracy
print("True labels:", true_labels)
print("Predicted labels:", pred_labels)
accuracy = accuracy_score(true_labels, pred_labels)
print(f"Accuracy on the test dataset: {accuracy * 100:.2f}%")

# Let's evaluate the fine-tuned model on the test dataset of sst5 from sklearn.metrics import accuracy_score test_dataset = test_dataset.map(preprocess_function) print("Sample test example:", test_dataset[0]) # We will collect the true labels and predicted labels to compute accuracy true_labels = [] pred_labels = [] print("Evaluating the fine-tuned model on the test dataset...") print("type of test_dataset:", type(test_dataset)) # Iterate over the subset of test dataset and generate predictions for example in test_dataset.select(range(100)): messages = example['prompt'] prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) #print("Prompt:", prompt) model_inputs = tokenizer([prompt], return_tensors="pt").to(base_model.device) generated_ids = fine_tuned_model.generate( **model_inputs, max_new_tokens=50, ) output_ids = generated_ids[0][len(model_inputs.input_ids[0]):] # Decode and extract model response generated_text = tokenizer.decode(output_ids, skip_special_tokens=True) pred_label = generated_text.split("Answer:")[-1].strip() true_label = example['completion'][0]['content'].split("Answer:")[-1].strip() pred_labels.append(pred_label) true_labels.append(true_label) # Compute accuracy print("True labels:", true_labels) print("Predicted labels:", pred_labels) accuracy = accuracy_score(true_labels, pred_labels) print(f"Accuracy on the test dataset: {accuracy * 100:.2f}%")

Map:   0%|          | 0/2210 [00:00<?, ? examples/s]

Sample test example: {'text': 'no movement , no yuks , not much of anything .', 'label': 1, 'label_text': 'negative', 'prompt': [{'content': 'Classify a sentence into 5 categories: very positive, positive, neutral, negative, very negative.', 'role': 'system'}, {'content': 'Text: no movement , no yuks , not much of anything .', 'role': 'user'}], 'completion': [{'content': 'Answer: negative', 'role': 'assistant'}]}
Evaluating the fine-tuned model on the test dataset...
type of test_dataset: <class 'datasets.arrow_dataset.Dataset'>
True labels: ['negative', 'very negative', 'neutral', 'neutral', 'very negative', 'very negative', 'very positive', 'positive', 'negative', 'positive', 'negative', 'neutral', 'very negative', 'positive', 'negative', 'negative', 'very positive', 'very negative', 'neutral', 'very positive', 'positive', 'very negative', 'negative', 'negative', 'positive', 'negative', 'positive', 'neutral', 'neutral', 'positive', 'very negative', 'negative', 'neutral', 'negative', 'positive', 'negative', 'very positive', 'positive', 'negative', 'very positive', 'positive', 'very positive', 'positive', 'positive', 'very positive', 'positive', 'neutral', 'negative', 'positive', 'positive', 'neutral', 'negative', 'very positive', 'very negative', 'neutral', 'very positive', 'positive', 'positive', 'negative', 'positive', 'negative', 'very positive', 'very positive', 'very negative', 'negative', 'very positive', 'very negative', 'positive', 'negative', 'negative', 'neutral', 'very positive', 'negative', 'negative', 'positive', 'positive', 'very positive', 'neutral', 'very positive', 'very negative', 'very negative', 'neutral', 'positive', 'neutral', 'positive', 'negative', 'negative', 'positive', 'neutral', 'positive', 'negative', 'very positive', 'negative', 'positive', 'negative', 'negative', 'very negative', 'negative', 'negative', 'positive']
Predicted labels: ['negative', 'very negative', 'negative', 'positive', 'very negative', 'negative', 'model\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel', 'positive', 'neutral', 'positive', 'neutral', 'negative', 'negative', 'positive', 'negative', 'neutral', 'positive', 'negative', 'model\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel', 'positive', 'positive', 'model answer: negative', 'neutral', 'neutral', 'neutral', 'neutral', 'positive', 'negative', 'positive', 'positive', 'neutral', 'negative', 'negative', 'negative', 'positive', 'negative', 'negative', 'positive', 'positive', 'positive', 'positive', 'positive', 'positive', 'positive', 'positive', 'very negative', 'model\nnegative', 'neutral', 'model\npositive', 'positive', 'neutral', 'negative', 'positive', 'negative', 'negative', 'positive', 'model\npositive', 'positive', 'negative', 'positive', 'negative', 'positive', 'positive', 'negative', 'negative', 'positive', 'very negative', 'neutral', 'negative', 'neutral', 'negative', 'model answer: positive', 'neutral', 'negative', 'model\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel\nmodel', 'positive', 'positive', 'negative', 'positive', 'very negative', 'negative', 'negative', 'negative', 'positive', 'neutral', 'negative', 'neutral', 'model answer: positive', 'negative', 'positive', 'very positive', 'positive', 'negative', 'positive', 'negative', 'negative', 'neutral', 'negative', 'negative', 'positive']
Accuracy on the test dataset: 40.00%