Posts on LlamaFactory Blog

MOSS-VL Meets LlamaFactory: LoRA and Full Fine-Tuning Out of the Box

Thu, 13 Aug 2026 20:03:59 +0800

You can now fine-tune MOSS-VL directly with LlamaFactory.

MOSS-VL support has been merged into the main branch of LlamaFactory. Whether you want to validate an idea efficiently with LoRA or fully adapt the model to a specific domain, you can use the familiar LlamaFactory workflows for data processing, training, checkpoint resumption, and inference.

This tutorial first walks through a general multimodal MOSS-VL fine-tuning workflow using the WebUI, with a public basketball video dataset as the running example. After training, we examine how the model improves on a real basketball clip, then use Chinese chess notation to demonstrate its ability to learn an entirely new specialized task.

Fine-Tuning LlamaFactory with the llamafactory-sft Skill

Fri, 26 Jun 2026 10:43:52 +0800

Fine-Tuning LlamaFactory with the llamafactory-sft Skill

This tutorial-style Notebook shows how to use the llamafactory-sft skill to let an agent assist with a LlamaFactory SFT fine-tuning workflow. It does not explain every training parameter in depth. Instead, it demonstrates how the skill breaks one fine-tuning task into confirmable, traceable, and reproducible steps.

This tutorial uses a lightweight example: starting from a Qwen3.5-2B model, using built-in LlamaFactory datasets to run a LoRA SFT job, then validating the result and exporting the model after training.

LlamaFactory x MindSpore HyperParallel Community Collaboration Roadmap

Mon, 30 Mar 2026 15:42:54 +0800

MindSpore Community · HyperParallel SuperNode Parallel Library

Version: v1.0 | Updated: 2026-03-30

Vision

HyperParallel is a new supernode parallel training architecture proposed by the MindSpore Community, dedicated to simplifying Ascend supernode programming and unlocking computing potential. We aim to collaborate with the LlamaFactory ecosystem to provide an easy-to-use, high-performance distributed training solution. Our goal is to enable every developer to efficiently train large models on Ascend NPU and NVIDIA GPU, lowering the barrier and cost of large model training.

MindSpore HyperParallel FSDP2 Training on Ascend with LlamaFactory

Tue, 24 Mar 2026 10:00:00 +0800

LlamaFactory + MindSpore HyperParallel

We integrated HyperParallel from the MindSpore community as an FSDP2 backend into LlamaFactory, supporting Ascend NPU and NVIDIA GPU. Just one extra config line on the FSDP2 workflow to get started.

Quick Start

1. Environment Installation

pip

# Install HyperParallel
git clone https://gitcode.com/mindspore/hyper-parallel
cd hyper-parallel
pip install -e .

# Install LlamaFactory
git clone https://github.com/hiyouga/LlamaFactory.git
cd LlamaFactory
pip install -e ".[torch,metrics]" --no-build-isolation

# Install PyTorch
pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1

# Optional: install torch-npu for Ascend NPU
pip install torch-npu==2.7.1

2. Configuration

HyperParallel training requires two config files: an Accelerate FSDP2 config and a LlamaFactory training config.

Fine-Tuning the Latest Qwen3.5 Model to Identify Humanoid Robot Models Using LlamaFactory

Tue, 03 Mar 2026 16:41:05 +0800

At the beginning of 2026, from the Consumer Electronics Show (CES) in Las Vegas, USA, to the China Central Television (CCTV) Spring Festival Gala, China’s self-developed humanoid robots have frequently “broken through the circle.” Products and applications from multiple Chinese enterprises have not only sparked discussions within the overseas industry but have also continuously “swept” global social media platforms and international media. Embodied intelligence, regarded as the next stage of artificial intelligence development, has its core in achieving a deep coupling between the intelligent “brain” and the physical “body,” thereby directly transforming data, algorithms, and computing power into the ability to act on and transform the objective world. Humanoid robots, due to their human-like appearance and functionality, are considered a high-level form and the optimal carrier for embodied intelligence, poised to become the next-generation super terminal following smartphones and new energy vehicles.

Issues Related to the Qwen3-VL Model

Mon, 05 Jan 2026 15:42:25 +0800

This blog post focuses on several practical issues related to the Qwen3-VL model, along with an analysis of their root causes and corresponding solutions.

1. Slow Training and Inference Speed of Qwen3-VL

Problem:
Some posts and GitHub issues report that when using torch=2.9 together with Conv3D, the training and inference speed of Qwen3-VL degrades significantly compared to torch=2.8. See the related discussion at:
https://github.com/pytorch/pytorch/issues/166122

1.1 Comparing CUDA Kernel Invocations

We first compared the CUDA kernel calls of Conv3D under torch=2.8 and torch=2.9. The test code is shown below:

RL-DPO Training with KTransformers and LLaMA-Factory

Tue, 23 Dec 2025 15:51:07 +0800

This tutorial demonstrates how to fine-tune a language model using the LLaMA-Factory framework with Direct Preference Optimization (DPO). DPO is a training method based on human preferences, enabling model outputs to better align with human expectations and be more user-centric.

1 Environment Setup

Software & hardware requirements: CPU must support AMX, the system glibc version must be ≥ 2.32, and a GPU with at least 32 GB of VRAM is recommended.

Add New Special Tokens for Model Training

Wed, 17 Dec 2025 14:13:17 +0800

1 Introduction

This paper uses the Ministral-3-3B-Instruct-2512 model and takes an image classification task fine-tuned via SFT as an example to illustrate how to add new special tokens. The experimental command is as follows:

# install newest transformers
pip install git+https://github.com/huggingface/transformers

DISABLE_VERSION_CHECK=1 CUDA_VISIBLE_DEVICES=7 python src/train.py examples/train_lora/ministral3_lora_sft.yaml

It is necessary to preconfigure ministral3_lora_sft.yaml.

2 Dataset Loading and Preprocessing

In the file LLaMA-Factory/src/llamafactory/data/loader.py, the get_dataset function is responsible for loading the dataset and preprocessing the data using the tokenizer.

Adapt a new model on LLaMA-Factory

Fri, 12 Dec 2025 21:13:00 +0800

1 Overview of Model Adaptation

LLaMA-Factory offers a complete framework for model pre-training, fine-tuning, and inference. If it is necessary to adapt a new model, only a small amount of code needs to be modified to integrate the model into LLaMA-Factory.

First, the file LLaMA-Factory/src/llamafactory/extras/constants.py defines the supported model groups and their corresponding templates. A template is a “format specifier” used when constructing the input prompt for the large model. It defines the dialogue format, field structure, role order, and the format for tool calls. For example:

Code Guide for LLaMA Factory Project

Fri, 05 Dec 2025 21:13:00 +0800

1 Introduction to the LLaMA-Factory Project

LLaMA-Factory is an efficient training and fine-tuning framework designed for large language models (LLMs). It aims to simplify the training workflow of the LLaMA family as well as various open-source large models. With the core philosophy of being “out-of-the-box, flexible, and efficient,” it provides an end-to-end solution covering data preparation, parameter-efficient fine-tuning (PEFT), training configuration management, and model deployment.

LLaMA-Factory supports multiple mainstream model architectures—such as LLaMA, Qwen, Gemma, and Mistral—and integrates lightweight training techniques including LoRA, QLoRA, AdaLoRA, and Prompt Tuning. These capabilities enable developers to fine-tune high-quality models at extremely low cost, whether in single-GPU or multi-GPU environments.

KTransformers Fine-Tuning × LLaMA-Factory Integration

Tue, 04 Nov 2025 22:06:28 +0800

KTransformers Fine-Tuning × LLaMA-Factory Integration

Introduction

From DeepSeek-V3/R1 to Qwen3-MoE and Kimi-K2, each wave of open-sourced large models brings leaps in performance and scale. However, many researchers and developers are constrained by expensive GPUs and models with tens or even hundreds of billions of parameters, making it hard to fine-tune very large models under limited resources. To bridge this gap, we propose a practical approach: combining KTransformers with LLaMA-Factory. With just 2–4 RTX 4090s and a high-memory CPU, you can fine-tune ultra-large MoE models like DeepSeek-671B.

Megatron-Core Fine-Tuning with LLaMA-Factory

Tue, 21 Oct 2025 16:21:12 +0800

LLaMA-Factory 🤝 MCoreAdapter

To fully leverage Megatron-core’s parallel computing and improve training efficiency for MoE models, we combined the MCoreAdapter provided by the ROLL team with LLaMA-Factory’s data pipeline and Megatron Trainer’s backend to build a new model training workflow.

🚀 Quick Start

1. 💻 Environment Installation

📦 pip

# for megatron-core
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124

pip install \
    numpy==1.26.4 \
    optree>=0.13.0 \
    spacy==3.7.5 \
    weasel==0.4.1 \
    transformer-engine[pytorch]==2.2.0 \
    megatron-core==0.13.0 \
    deepspeed==0.16.4 

pip uninstall -y opencv opencv-python opencv-python-headless
pip install opencv-python-headless==4.11.0.86
pip install "git+https://github.com/alibaba/roll.git#subdirectory=mcore_adapter"

# for llamafactory
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e ".[torch,metrics]" --no-build-isolation

🐳 docker (Recommended)

Refer to the Dockerfile for building.

Easy Dataset × LLaMA Factory: Empowering Large Models with Efficient Domain Knowledge Learning

Thu, 03 Apr 2025 21:13:00 +0800

1 Introduction

Easy Dataset is an application designed specifically for creating fine-tuning datasets for large language models (LLMs). It provides an intuitive interface for uploading domain-specific documents, intelligently segmenting content, generating questions, and producing high-quality training data for model fine-tuning. It supports calling large models through APIs such as OpenAI, DeepSeek, Volcano Engine, as well as local models via Ollama.

LLaMA Factory is an open-source, low-code fine-tuning framework for large language models. It integrates the most widely used fine-tuning techniques in the industry and supports zero-code model fine-tuning through a Web UI. It has become one of the most popular fine-tuning frameworks in the open-source community, with over 63K stars on GitHub. It supports full-parameter fine-tuning, LoRA fine-tuning, as well as fine-tuning algorithms such as SFT and DPO.