EasyR1强化学习框架：GRPO算法优化与LLM训练实战

1. 项目概述

EasyR1是一个基于强化学习(RL)的大语言模型(LLM)训练框架，特别针对GRPO算法进行了优化。GRPO(Generalized Reinforcement Policy Optimization)是一种新型的强化学习算法，它在PPO(Proximal Policy Optimization)的基础上进行了改进，通过引入更灵活的奖励函数设计和策略优化机制，能够更好地适应复杂任务场景。

本项目实战指南将带领你完成从环境搭建到模型训练的全流程，特别针对NVIDIA H800双卡环境进行了优化配置。我们将使用DeepSeek-R1-0528-Qwen3-8B作为基础模型，通过LoRA微调方式进行训练。

2. 环境准备与依赖安装

2.1 硬件与系统要求

为了顺利运行EasyR1训练环境，建议满足以下硬件配置：

• GPU: NVIDIA H800 80GB显存(双卡)
• CPU: 至少16核
• 内存: 128GB以上
• 存储: 1TB NVMe SSD(用于数据集和模型存储)
• 操作系统: Ubuntu 22.04 LTS

2.2 基础环境配置

首先安装必要的系统依赖：

bash复制sudo apt update
sudo apt install -y git build-essential cmake ninja-build python3-dev

创建并激活conda环境：

bash复制conda create -n easy-r1 python=3.12 -y
conda activate easy-r1

2.3 PyTorch与CUDA安装

安装与H800兼容的PyTorch和CUDA版本：

bash复制pip install torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1 --index-url https://download.pytorch.org/whl/cu128

验证安装：

bash复制python -c "import torch; print(f'PyTorch: {torch.__version__}'); print(f'CUDA: {torch.version.cuda}'); print(f'GPU: {torch.cuda.get_device_name(0)}')"

2.4 Flash Attention安装

Flash Attention是训练大语言模型的关键优化组件：

bash复制pip install packaging ninja psutil
pip install flash-attn==2.8.3 --no-build-isolation

验证Flash Attention安装：

bash复制python -c "import flash_attn; print(flash_attn.__version__)"

3. EasyR1项目部署

3.1 克隆项目仓库

bash复制git clone https://github.com/hiyouga/EasyR1.git
cd EasyR1

3.2 安装项目依赖

bash复制pip install -e .

3.3 监控工具安装

推荐使用SwanLab进行训练监控：

bash复制pip install swanlab
swanlab login

4. vLLM API兼容性修正

由于vLLM 0.15.1版本API变动，需要对EasyR1中的vllm_utils.py进行修改。以下是完整的兼容代码：

python复制# Copyright 2024 Bytedance Ltd. and/or its affiliates
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#     http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from importlib.metadata import version
from typing import List

from msgspec import field
from packaging import version as vs
from vllm.lora.lora_model import LoRAModel
from vllm.lora.request import LoRARequest
from vllm.lora.utils import get_adapter_absolute_path
from vllm.lora.worker_manager import LRUCacheWorkerLoRAManager


class TensorLoRARequest(LoRARequest):
    peft_config: dict = field(default=None)
    lora_tensors: dict = field(default=None)


class VLLMHijack:
    @staticmethod
    def hijack():
        def hijack__load_adapter(self, lora_request: TensorLoRARequest) -> LoRAModel:
            """
            based on vllm.lora.worker_manager.WorkerLoRAManager._load_adapter, support load adapter with lora tensors
            Reason:
            VLLM does not support adding LoRA from tensors directly. It only supports adding LoRA via file paths.
            To synchronize the LoRA tensors of the actor model, we need to find a workaround to enable VLLM to load memory-based LoRA tensors.
            """
            supported_lora_modules = self._adapter_manager.supported_lora_modules
            packed_modules_mapping = self._adapter_manager.packed_modules_mapping
            expected_lora_modules: List[str] = []
            for module in supported_lora_modules:
                if module in packed_modules_mapping:
                    expected_lora_modules.extend(packed_modules_mapping[module])
                else:
                    expected_lora_modules.append(module)

            expected_lora_modules = list(set(expected_lora_modules))

            lora_tensors = None
            from vllm.lora.peft_helper import PEFTHelper

            if isinstance(lora_request, TensorLoRARequest):
                peft_config = lora_request.peft_config
                lora_tensors = lora_request.lora_tensors
                peft_helper = PEFTHelper.from_dict(