RLLM Guide: Training LLM Agents with Open-Source Reinforcement Learning

RLLM is the open-source framework that is democratizing reinforcement learning (RL) for large language models. Backed by more than 5,700 GitHub stars, it lets you train language agents on any harness, with any backend, on any benchmark — without rewriting your agent code.

What is RLLM?

RLLM is a training framework for language agents that use reinforcement learning. Its core promise is portability: the same agent code that runs your evaluation also drives your training. You write a rollout function once, wrap it with @rllm.rollout, and the framework handles sandboxing, reward computation, and gradient updates.

Why RL for LLMs matters

Supervised fine-tuning teaches a model to imitate; reinforcement learning teaches it to reason. Methods like GRPO, REINFORCE, and RLOO improve multi-step reasoning, tool use, and code generation — the behaviors behind state-of-the-art open models such as DeepScaleR-1.5B, DeepCoder-14B, and DeepSWE-32B, all trained with RLLM.

Core capabilities

Getting started

RLLM requires Python 3.11+. Install it directly from GitHub and you are ready to evaluate on your first benchmark in minutes:

uv pip install "rllm @ git+https://github.com/rllm-org/rllm.git"
rllm model setup
rllm eval gsm8k
rllm train gsm8k

Whether you are a research lab reproducing open RL results or a team shipping your first reasoning agent, RLLM removes the infrastructure tax from RL training. Apache-2.0 licensed and actively maintained, it is one of the most practical entry points into agentic reinforcement learning today.