WhisperX: Open-Source Speech Recognition with Word-Level Timestamps & Speaker Diarization

What Is WhisperX?

If you've ever used OpenAI's Whisper for transcription and wished it had word-level timestamps, speaker identification, and was significantly faster — WhisperX is the answer. With over 22,000 GitHub stars, it's the most powerful open-source speech recognition tool for production use.

WhisperX combines three technologies into one pipeline: faster-whisper (CTranslate2-accelerated Whisper) for fast transcription, wav2vec2 for precise word-level timestamp alignment, and pyannote.audio for speaker diarization. The result is a tool that doesn't just transcribe — it tells you exactly when each word was spoken and who said it.

The pipeline diagram above shows how WhisperX processes audio: first transcribing with faster-whisper, then aligning word-level timestamps with wav2vec2, and finally performing speaker diarization with pyannote.audio.

Why WhisperX Stands Out

1. Word-Level Timestamp Accuracy

Standard Whisper gives you segment-level timestamps (every 30 seconds). WhisperX uses wav2vec2 forced alignment to pin down each word's start and end time to within ~50ms. This precision is essential for subtitle generation, video editing, and any application where timing matters.

2. Speaker Diarization

WhisperX integrates pyannote.audio to identify and label different speakers in the recording. Each transcript segment is tagged with the speaker who said it — perfect for interviews, podcasts, meetings, and any multi-speaker audio.

3. 4x Faster Than Vanilla Whisper

By using CTranslate2 (via faster-whisper) for the initial transcription, WhisperX achieves up to 4x speedup over the original Whisper implementation. Batched processing of audio chunks further improves throughput on long files.

4. 90+ Language Support

WhisperX inherits Whisper's massive language coverage — over 90 languages are supported for transcription, with automatic language detection. The word-level alignment works across all supported languages.

5. VAD-Based Audio Segmentation

WhisperX uses Voice Activity Detection (VAD) to segment audio intelligently, avoiding hallucinations on silent sections that plague vanilla Whisper. This means cleaner transcripts with fewer phantom words in pauses.

Getting Started

Installation

pip install git+https://github.com/m-bain/whisperX.git

Basic Transcription with Diarization

import whisperx

# Load model
model = whisperx.load_model("large-v3", device="cuda", compute_type="float16")

# Transcribe
audio = whisperx.load_audio("meeting.wav")
result = model.transcribe(audio, batch_size=16)

# Align word-level timestamps
model_a, metadata = whisperx.load_align_model(
    language_code=result["language"], device="cuda"
)
result = whisperx.align(result["segments"], model_a, metadata, audio, device="cuda")

# Diarize speakers
diarize_model = whisperx.DiarizationPipeline(device="cuda")
diarize_segments = diarize_model(audio, min_speakers=2, max_speakers=4)
result = whisperx.assign_word_speakers(diarize_segments, result)

# Print result
for segment in result["segments"]:
    print(f"[{segment['start']:.1f}s - {segment['end']:.1f}s] {segment['speaker']}: {segment['text']}")

Self-Hosting with Docker

docker pull thomasvvugt/whisperx:latest

docker run -d \
  --name whisperx \
  --gpus all \
  -p 8000:8000 \
  -v ./audio:/audio \
  -v ./models:/root/.cache \
  thomasvvugt/whisperx:latest

WhisperX vs Other Transcription Tools

Best Use Cases

Subtitle & Caption Generation

Word-level timestamps make WhisperX perfect for generating precise SRT/VTT subtitles for videos and podcasts.

Meeting & Interview Transcription

Speaker diarization automatically separates who said what in multi-speaker recordings.

Content Analysis & Search

Timestamped transcripts enable searchable audio archives and content indexing.

Research & Linguistics

Word-level alignment is invaluable for phonetic research, language documentation, and corpus building.

Tips for Best Results

• Use large-v3 model for best accuracy (large-v2 for speed/accuracy balance)
• Set batch_size=16 for optimal GPU utilization on long files
• Use VAD — it prevents hallucinations on silent sections
• For diarization — set min_speakers and max_speakers for better accuracy
• Use float16 compute type on GPU for 2x speedup with minimal accuracy loss

Conclusion

WhisperX is the definitive tool for anyone who needs more than just plain transcription. Its combination of word-level timestamps, speaker diarization, VAD-based segmentation, and CTranslate2 acceleration makes it the most complete open-source speech recognition pipeline available. Whether you're building a subtitle generator, a meeting recorder, or a research tool, WhisperX delivers production-grade results on consumer hardware.