Getting Started with Speech-to-Speech: A Practical Tutorial for Building Voice Agents

Introduction

In this tutorial, you'll learn how to set up and run Speech-to-Speech by Hugging Face on your local machine. By the end, you'll have a working voice agent that can listen, understand, and respond to spoken commands — all running locally with zero cloud dependencies.

Prerequisites

• Python 3.10 or newer
• A machine with at least 8GB RAM (16GB recommended for larger models)
• A CUDA-compatible GPU (optional, but recommended for real-time performance)
• Git installed on your system
• Basic familiarity with the command line

Step 1: Clone the Repository

git clone https://github.com/huggingface/speech-to-speech.git
cd speech-to-speech

Step 2: Set Up the Environment

Create a Python virtual environment and install the dependencies:

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

If you have a CUDA GPU, install the GPU-optimized version of PyTorch for faster inference:

pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu118

Step 3: Download the Models

Speech-to-Speech will download the required models automatically on first run. You can also pre-download them:

python scripts/download_models.py

This downloads the ASR, LLM, and TTS models from Hugging Face. The total download is approximately 4-6GB depending on your model selection.

Step 4: Run Your First Voice Agent

Start the basic voice agent:

python run.py --mode interactive

Speak into your microphone and the agent will process your speech, generate a response, and speak back to you. Press Ctrl+C to exit.

Step 5: Customize the Agent

Speech-to-Speech supports several customization options:

Using the Web Interface

Speech-to-Speech also includes a web demo for a more visual experience:

python run.py --mode web --port 8080

Open http://localhost:8080 in your browser to access the Gradio-based web interface where you can record audio, see transcriptions, and hear responses in real-time.

Troubleshooting Tips

Conclusion

You now have a fully functional local voice agent running with Speech-to-Speech. The modular architecture means you can swap in different models, add custom intents, and integrate with other tools as your project grows. With its Apache-2.0 license and active community, Speech-to-Speech is an excellent foundation for any voice AI project.