RLLM Guide: Training LLM Agents with Open-Source Reinforcement Learning
RLLM is the open-source RL training framework for LLM agents — any harness, any sandbox, any backend. Learn how it democratizes reinforcement learning with 60+ benchmarks and GRPO/REINFORCE support.
RLLM is the open-source framework that is democratizing reinforcement learning (RL) for large language models. Backed by more than 5,700 GitHub stars, it lets you train language agents on any harness, with any backend, on any benchmark — without rewriting your agent code.
🚀 Want to deploy RLLM yourself?
Docker configs, system requirements, and installation guides — all on one page.
View RLLM Tool Page →What is RLLM?
RLLM is a training framework for language agents that use reinforcement learning. Its core promise is portability: the same agent code that runs your evaluation also drives your training. You write a rollout function once, wrap it with @rllm.rollout, and the framework handles sandboxing, reward computation, and gradient updates.
Why RL for LLMs matters
Supervised fine-tuning teaches a model to imitate; reinforcement learning teaches it to reason. Methods like GRPO, REINFORCE, and RLOO improve multi-step reasoning, tool use, and code generation — the behaviors behind state-of-the-art open models such as DeepScaleR-1.5B, DeepCoder-14B, and DeepSWE-32B, all trained with RLLM.
Core capabilities
| Feature | What you get |
|---|---|
| Any harness | 10+ CLI agents (Claude Code, Codex, mini-swe-agent, opencode) plus LangGraph and OpenAI Agents SDK via @rllm.rollout |
| Any sandbox | Docker, Daytona, Modal, or local, with snapshot + warm-pool acceleration |
| One API, many backends | Switch between verl (distributed multi-GPU), tinker (single machine), and Fireworks with one flag |
| 60+ benchmarks | Terminal-Bench 2.0, SWE-bench, AIME, MATH-500, GPQA and more — rllm eval <name> auto-pulls and runs |
Getting started
RLLM requires Python 3.11+. Install it directly from GitHub and you are ready to evaluate on your first benchmark in minutes:
uv pip install "rllm @ git+https://github.com/rllm-org/rllm.git"
rllm model setup
rllm eval gsm8k
rllm train gsm8k🚀 Want to deploy RLLM yourself?
Docker configs, system requirements, and installation guides — all on one page.
View RLLM Tool Page →Whether you are a research lab reproducing open RL results or a team shipping your first reasoning agent, RLLM removes the infrastructure tax from RL training. Apache-2.0 licensed and actively maintained, it is one of the most practical entry points into agentic reinforcement learning today.