Fine-Tuning Small language models on Linux
Fine-tuning adapts a pretrained model to a narrow task, vocabulary, tone, or response format using your own examples. For a Linux/sysadmin use case, a small language model (SLM) could be adapted to turn Rocky Linux alerts into triage steps, answer internal runbook questions, or classify security-ticket descriptions. The usual efficient approach is supervised fine-tuning (SFT): train on input-output examples so the model learns to predict the desired answer conditioned on the prompt.
Start with a modest model that fits your GPU and a narrow goal. A small, clean dataset usually beats a large, inconsistent one: avoid secrets, access tokens, customer data, passwords, production logs containing identifiers, and unlicensed text. Split the dataset into train, validation, and test sets before training, so you can measure whether the model generalizes rather than merely memorizing examples.
A practical Linux stack is Python in a virtual environment, NVIDIA drivers plus CUDA-compatible PyTorch for GPU training, and the Hugging Face ecosystem: transformers for models, datasets for data, peft for efficient adapters, trl for supervised training, and optionally bitsandbytes for low-bit loading. PEFT methods update a small set of adapter parameters rather than the full model, reducing both compute and storage needs.
python3 -m venv slm-ft
source slm-ft/bin/activate
python -m pip install –upgrade pip
pip install torch transformers datasets accelerate peft trl bitsandbytes
Before training, confirm the machine sees the GPU and that PyTorch can use it:
nvidia-smi
python -c “import torch; print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else ‘CPU only’)”
CPU fine-tuning is possible for very small models, but it is generally slow. On a single consumer NVIDIA GPU, 4-bit quantization plus LoRA is often the practical route: loading weights in 4-bit or 8-bit precision saves memory, while the adapter remains trainable.
Prepare the Dataset
For chat-oriented tuning, use JSON Lines, with one valid JSON object per line. Keep the prompt explicit and make answer style consistent:
{"messages":[{"role":"system","content":"You are a cautious Linux operations assistant. Never invent commands or system state."},{"role":"user","content":"Rocky Linux: how do I see failed systemd services?"},{"role":"assistant","content":"Run systemctl --failed. Then inspect a unit with systemctl status <unit> and journalctl -u <unit>."}]}Create three files: train.jsonl, validation.jsonl, and test.jsonl. Validation data guides training decisions; test data stays untouched until the end. Include both normal and adversarial cases relevant to the deployment: ambiguous requests, requests for unavailable information, commands with unsafe consequences, and examples where the correct answer is to ask for context instead of guessing.
LoRA (Low-Rank Adaptation) freezes the base model and trains small adapter matrices attached to selected model layers. The result is a compact adapter rather than a full duplicate of the model; PEFT’s documented workflow is to load the base model, define LoraConfig, wrap it as a trainable PEFT model, and train normally.
For an SLM, target the attention projection layers commonly named q_proj and v_proj; verify your particular architecture first because module names vary. Key settings are:
-
r— adapter rank. Start around 8 or 16; a larger rank can learn more but costs more VRAM. -
lora_alpha— scaling factor. A common starting relationship is approximately two timesr. -
lora_dropout— regularization. Start around 0.05 for smaller datasets. -
learning_rate— start conservatively, such as1e-4to2e-4for LoRA SFT. -
max_seq_length— cap it to the real task need; longer sequences consume much more memory.
Minimal Training Script
Save the following as train.py. Replace YOUR_BASE_MODEL with a model you are licensed and authorized to use, and inspect its model card for its expected prompt format and intended use.
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer
model_id = “YOUR_BASE_MODEL”
quant_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type=“nf4”,
bnb_4bit_compute_dtype=“float16”,
)
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(
model_id,
quantization_config=quant_config,
device_map=“auto”,
)
dataset = load_dataset(
“json”,
data_files={
“train”: “train.jsonl”,
“validation”: “validation.jsonl”,
},
)
peft_config = LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
target_modules=[“q_proj”, “v_proj”],
task_type=“CAUSAL_LM”,
)
training_args = SFTConfig(
output_dir=“./output-slm-adapter”,
num_train_epochs=3,
per_device_train_batch_size=1,
gradient_accumulation_steps=8,
learning_rate=2e-4,
logging_steps=10,
eval_strategy=“steps”,
eval_steps=50,
save_steps=50,
bf16=True,
report_to=“none”,
)
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=dataset[“train”],
eval_dataset=dataset[“validation”],
processing_class=tokenizer,
peft_config=peft_config,
)
trainer.train()
trainer.save_model(“./output-slm-adapter”)
tokenizer.save_pretrained(“./output-slm-adapter”)
TRL’s SFTTrainer is built specifically to simplify supervised fine-tuning, and when given a PEFT configuration it initializes the adapter training path for you. If bf16=True fails on your GPU, use fp16=True instead; do not enable both.
Do not judge success only by a declining training loss. Compare the base model and tuned model against the held-out test set, then manually probe realistic requests. For an internal Linux assistant, assess command correctness, whether prerequisites are stated, whether destructive commands are guarded, whether outputs are invented, and whether the model declines to fabricate environment-specific facts.
Watch for these common failures:
-
Overfitting: Training loss drops while validation loss rises. Reduce epochs, lower rank, improve dataset variety, or add examples.
-
Bad formatting: The model’s response template differs from your training format. Match the base model’s chat template during data preparation and inference.
-
Catastrophic behavior change: A narrow dataset makes the model repetitive or overly confident. Mix high-quality general instruction examples with your specialist examples.
-
Out-of-memory errors: Lower
max_seq_length, batch size, LoRA rank, or use gradient accumulation and 4-bit loading. -
Unsafe operational output: Add counterexamples that require warnings, confirmation, backup advice, or requests for logs before recommending a risky command.
A LoRA output normally contains adapter weights and configuration, not a standalone base model. At inference time, load the original base model and then attach the adapter; PEFT supports saving adapters and loading them on the correct base model through the adapter configuration. This makes versioning practical: store a model manifest containing the base-model revision, adapter version, dataset version/hash, training parameters, evaluation result, owner, and approval status.
For your Linux-first workflow, package inference in a container, run it with a non-root user, mount model data read-only, avoid embedding access tokens in images, restrict egress if the model does not need external access, and expose the API only behind authentication and rate limits. Keep model artifacts separate from training data, especially if the data originated from internal tickets or infrastructure documentation.
[mai mult...]