LLM Fine-tuning 大模型微调

📖 定义

Fine-tuning（微调）是指在预训练大模型的基础上，使用特定领域的数据进行进一步训练，使模型能够：

• 📚 学习特定知识 - 掌握特定领域的专业术语和概念
• 🎯 优化输出格式 - 按指定格式生成回答
• 💬 改善对话风格 - 适配特定的人设或语气
• ✅ 提升任务表现 - 在特定任务上获得更好的效果

📊 微调 vs 预训练 vs RAG

🛠️ 主流微调技术

💻 代码示例

使用 PEFT (LoRA) 微调 LLaMA

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model, TaskType
import torch

# 1. 加载基础模型
model_name = "meta-llama/Llama-2-7b-hf"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# 2. 配置LoRA
lora_config = LoraConfig(
    r=8,                    # LoRA rank
    lora_alpha=16,           # LoRA scaling
    target_modules=["q_proj", "v_proj"],  # 目标模块
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM
)

# 3. 应用LoRA
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# 输出: trainable params: 4,194,304 || all params: 6,742,609,920 || trainable%: 0.06%

# 4. 训练
from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./lora-llama",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    fp16=True,
    save_strategy="epoch",
    save_total_limit=3,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    tokenizer=tokenizer,
)

trainer.train()

# 5. 保存
model.save_pretrained("./lora-llama-adapter")

使用 QLoRA 微调

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model
from bitsandbytes import BitsAndBytesConfig
import torch

# 1. 4-bit量化配置
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

# 2. 加载量化模型
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=bnb_config,
    device_map="auto",
)

# 3. 后续步骤与LoRA相同...
lora_config = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"])
model = get_peft_model(model, lora_config)

📋 微调步骤全流程

🛠️ 常用工具

⚠️ 常见问题