RLLM Tutorial: Train Your First RL Agent in 10 Minutes

Reinforcement learning for LLMs sounds intimidating — but with RLLM, you can train your first RL agent in about ten minutes. In this tutorial, we build a tiny math-solving agent, evaluate it on GSM8K, and run an RL training loop — all with a few lines of Python.

Step 1: Install RLLM

RLLM needs Python 3.11 or newer. Install the CLI with the single-machine tinker backend included:

uv pip install "rllm @ git+https://github.com/rllm-org/rllm.git"
rllm model setup

The setup command points RLLM at your model provider (local vLLM, OpenAI-compatible endpoint, or a cloud provider).

Step 2: Evaluate before you train

Zero-code evaluation is the fastest way to sanity-check your setup. RLLM bundles 60+ benchmarks and auto-pulls the dataset:

rllm eval gsm8k

This runs your configured model over GSM8K and reports pass accuracy. That baseline is your starting point for RL.

Step 3: Define your agent as a rollout

The magic of RLLM is that your agent code is identical for eval and training. A rollout is just a function that takes a task and returns an episode:

from openai import OpenAI
import rllm
from rllm.types import AgentConfig, Episode, Task, Trajectory

@rllm.rollout
def solve(task: Task, config: AgentConfig) -> Episode:
    client = OpenAI(base_url=config.base_url, api_key="EMPTY")
    response = client.chat.completions.create(
        model=config.model,
        messages=[{"role": "user", "content": task.instruction}],
    )
    return Episode(
        trajectories=[Trajectory(name="solver", steps=[])],
        artifacts={"answer": response.choices[0].message.content or ""},
    )

Step 4: Score with an evaluator

Rewards come from an evaluator that compares the agent's answer against the ground truth:

import rllm
from rllm.eval.types import EvalOutput, Signal
from rllm.types import Episode

@rllm.evaluator
def score(task: dict, episode: Episode) -> EvalOutput:
    answer = str(episode.artifacts.get("answer", ""))
    correct = answer.strip() == task["ground_truth"].strip()
    return EvalOutput(reward=1.0 if correct else 0.0, is_correct=correct,
                      signals=[Signal(name="accuracy", value=1.0 if correct else 0.0)])

Step 5: Train

Hand both functions to the trainer and call train():

from rllm.trainer import AgentTrainer
trainer = AgentTrainer(backend="tinker", agent_flow=solve, evaluator=score,
                       config=config, train_dataset=dataset)
trainer.train()

During training, config.base_url automatically points at a gateway that captures token IDs and logprobs — your rollout code never changes. For larger runs, switch to backend="verl" for distributed multi-GPU training, or backend="fireworks" for the managed platform.

From a zero-code rllm eval to a full GRPO training run, RLLM makes reinforcement learning for agents approachable. Try the tutorial on a single GPU today — and check the RLLM tool page on Run This Ai for Docker configs and system requirements.