Getting Started with Rig: Your First LLM Application in Rust

Getting Started with Rig: Build Your First LLM Application in Rust

In this hands-on tutorial, you'll learn how to set up Rig and build a complete LLM-powered application in Rust. By the end, you'll have a working agent that can answer questions using RAG (Retrieval-Augmented Generation).

Prerequisites

• Rust toolchain (1.75+) — install via rustup
• An API key for at least one LLM provider (OpenAI, Anthropic, or Cohere)
• Basic familiarity with Rust syntax and async/await

Step 1: Create a New Rust Project

cargo new rig-demo
cd rig-demo

Add Rig as a dependency in your Cargo.toml:

[dependencies]
rig-core = "0.5"
tokio = { version = "1", features = ["full"] }
anyhow = "1"
serde_json = "1"

Step 2: Basic LLM Completion

Let's start with a simple completion — the "Hello, World!" of LLM applications:

use rig::providers::openai;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Initialize the OpenAI client
    let client = openai::Client::from_env()?;

    // Create a completion
    let response = client
        .completion("gpt-4o-mini")
        .preamble("You are a helpful assistant.")
        .body("What is Rig and why should I use it?")
        .send()
        .await?;

    println!("{}", response.text());
    Ok(())
}

Set your OpenAI API key as an environment variable and run:

export OPENAI_API_KEY="sk-..."
cargo run

Step 3: Adding RAG with a Vector Store

Now let's make it more useful by adding RAG. We'll index a document and query it:

use rig::providers::openai;
use rig::vector_store::in_memory_store::InMemoryVectorStore;
use rig::embeddings::EmbeddingDoc;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let client = openai::Client::from_env()?;

    // Create a vector store
    let mut store = InMemoryVectorStore::new();

    // Index a document
    let doc = EmbeddingDoc::new(
        "rig-doc",
        "Rig is a Rust framework for building LLM applications. \
         It supports multiple providers, RAG, and agent orchestration."
    );
    let embeddings = client.embed_model("text-embedding-3-small").embed_document(&doc).await?;
    store.add_document(doc, embeddings);

    // Query with context
    let results = store.similarity_search("What is Rig?", 1).await?;
    let context = results.first().unwrap().content();

    let response = client
        .completion("gpt-4o-mini")
        .preamble("Answer based on the provided context.")
        .body(format!("Context: {}\n\nQuestion: What is Rig?", context))
        .send()
        .await?;

    println!("Answer: {}", response.text());
    Ok(())
}

Step 4: Building an Agent with Tools

Rig's agent system lets you equip LLMs with custom tools. Here's a calculator agent:

use rig::agent::Agent;
use rig::tool::Tool;
use serde_json::json;

struct Calculator;

#[async_trait::async_trait]
impl Tool for Calculator {
    fn name(&self) -> &str { "calculator" }
    fn description(&self) -> &str { "Evaluates math expressions" }

    async fn run(&self, args: serde_json::Value) -> Result<String, String> {
        let expr = args["expression"].as_str().ok_or("Missing expression")?;
        // Evaluate the expression
        Ok(format!("Result: {}", eval(expr)))
    }
}

async fn run_agent() -> anyhow::Result<()> {
    let client = openai::Client::from_env()?;
    let tools: Vec<Box<dyn Tool>> = vec![Box::new(Calculator)];

    let mut agent = Agent::new(client, "gpt-4o-mini", tools);
    let response = agent.chat("What is 42 * 7?").await?;
    println!("{}", response);
    Ok(())
}

Step 5: Switch Providers

One of Rig's best features is provider portability. Switch from OpenAI to Anthropic with minimal changes:

// Just change the import and client initialization
use rig::providers::anthropic;

let client = anthropic::Client::from_env()?;
// Everything else stays the same!

Performance Tips

Conclusion

Rig brings the power of Rust to LLM application development, offering performance, safety, and modularity that Python frameworks simply can't match. With support for multiple providers, RAG pipelines, vector stores, and agent orchestration, it's a compelling choice for production-grade AI applications.

The framework's 8,000+ stars on GitHub and active community make it a solid bet for long-term projects. Whether you're building a simple chatbot, a knowledge retrieval system, or a complex multi-agent platform, Rig provides the foundation you need.