Prefect
Workflow orchestration framework for building resilient data pipelines in Python
★ 22,708 GitHub
Apache-2.0 workflow-orchestrationdata-pipelinesautomationpythonschedulingetl Automation Tools
Overview
Prefect is a powerful open-source workflow orchestration framework that simplifies building, running, and monitoring resilient data pipelines in Python. With 22k+ GitHub stars, Prefect provides a modern approach to workflow automation with built-in retries, scheduling, caching, event-driven triggers, and real-time monitoring through a beautiful web dashboard. It is designed for data engineers, ML engineers, and platform teams that need reliable pipeline orchestration with Pythonic APIs. Prefect supports both self-hosted server and managed cloud deployments, integrates with any Python codebase, and offers a rich ecosystem of integrations with tools like Dask, Ray, Kubernetes, and cloud services.
Requirements
Min vCPU
1
Min RAM
4096 MB
Min Disk
10 GB
Rec vCPU
2
Rec RAM
4096 MB
Rec Disk
20 GB
Recommended VPS
Hostinger · KVM 2
2 vCPU · 8192 MB · 100 GB
Hostinger · KVM 2
2 vCPU · 8192 MB · 100 GB
Hostinger · KVM 4
4 vCPU · 16384 MB · 200 GB
Affiliate disclosure
Docker Compose
# Generated by Run This Ai — docker-compose.yml
services:
prefect:
image: prefecthq/prefect:latest
restart: unless-stopped
ports:
- 8080:8080
volumes:
- ./data/prefect:/data
Related tools
Firecrawl
Open-source API to search, scrape, and interact with the web for AI agents and LLMs
Crawl4AI
Open-source LLM-friendly web crawler and scraper for AI applications
NocoDB
The open-source Airtable alternative — turn any database into a smart spreadsheet
Huginn
Create agents that monitor and act on your behalf — the web's personal automation engine
ToolJet
Open-source low-code platform for building internal tools, dashboards, and AI agents
Directus
Open-source headless CMS and backend for managing agnostic content