Motorway
Fact Box
Language Python
License Apache 2.0
GitHub plecto/motorway
PyPI motorway
First Release 2014
Status Maintained

Motorway is a pure-Python real-time data pipeline library designed for cloud environments. Think Apache Storm, but written in Python and designed to run seamlessly in modern cloud infrastructure without the operational overhead.

The Problem

While building Plecto, we needed a way to process streaming data in real-time - ingesting data from various sources, transforming it, and feeding it into our analytics engine. Apache Storm was the leading solution at the time, but it came with significant challenges:

  • Required Zookeeper for coordination
  • Needed to take the entire pipeline down to update the topology
  • Complex operational requirements
  • Not cloud-native (designed for static clusters, not auto-scaling groups)

We wanted something that felt natural to Python developers and could run effortlessly in AWS Auto Scaling Groups.

The Solution

Motorway was born out of this need. We implemented the same state management algorithm as Apache Storm but rebuilt it from the ground up in Python with cloud deployment as a first-class concern.

Key Features

  • Pure Python: No JVM, no Zookeeper, no external dependencies beyond Python libraries
  • Cloud Native: Designed to work in AWS Auto Scaling Groups and other dynamic cloud environments
  • Hot Reloading: Update your pipeline topology without taking the system down
  • PyPy Support: Runs on PyPy for approximately 2x performance improvement
  • Django Integration: Easily integrates with Django applications
  • Storm-Compatible State Management: Implements the same reliable message processing guarantees

Architecture

Motorway is built around a simple but powerful concept: messages flow through a pipeline of processes. Each process can:

  • Transform the message
  • Route it to different downstream processes
  • Trigger side effects (database writes, API calls, etc.)
  • Handle failures and retries

The library handles the complexity of message acknowledgment, failure recovery, and state management, allowing you to focus on your business logic.

Real-World Usage

At Plecto, Motorway powered our core data ingestion pipeline, processing millions of events per day from various integrations:

  • CRM systems (Salesforce, Pipedrive, HubSpot)
  • Helpdesk software (Zendesk, Intercom)
  • Call center systems
  • Custom data sources via APIs and webhooks

The ability to scale horizontally by adding more workers and update the pipeline without downtime was critical for maintaining our SLA commitments to customers.

Why Python?

While languages like Java (Storm) and Erlang (RabbitMQ) dominated the streaming space, we believed Python was the right choice for several reasons:

  • Rapid development and iteration
  • Rich ecosystem of libraries for data processing
  • Easier to hire and onboard developers
  • With PyPy, performance was competitive
  • Simpler operational model

Design Philosophy

Motorway follows several key principles:

  1. Simplicity over features: Provide the essential building blocks rather than every possible feature
  2. Fail fast: Detect and surface errors quickly rather than silently dropping data
  3. Observability: Built-in support for monitoring and debugging
  4. Pragmatic consistency: Guarantee message processing without sacrificing throughput

When to Use Motorway

Motorway is ideal for:

  • Real-time data pipelines
  • Event processing systems
  • Data ingestion from multiple sources
  • Transform and enrichment workflows
  • Applications requiring horizontal scalability

It's particularly well-suited for teams already using Python who want a lightweight, cloud-native streaming solution without the operational complexity of larger frameworks.

Lessons Learned

Building and running Motorway in production taught us valuable lessons about distributed systems:

  • Message ordering is harder than it looks
  • Backpressure mechanisms are essential
  • Monitoring and observability must be built in from day one
  • The operational model matters as much as the technical architecture
  • Simplicity is a feature, not a limitation

Motorway represents our philosophy of building tools that solve real problems without unnecessary complexity - letting you focus on delivering value rather than managing infrastructure.