February 18, 2026 · 14 min read

Designing a Real-Time Market Data Pipeline for an Algorithmic Trading Platform

System Design FastAPI Redis Pub/Sub WebSockets Protobuf
[SYSTEM ARCHITECTURE DECISION: REDIS PUB/SUB DECOUPLING]
The ingestion worker acts as a single producer decoding binary Protobuf feeds into Redis Pub/Sub channels. This decouples tick ingestion from strategy evaluation and frontend WebSocket broadcasting.

1. The Challenge of Live Data Streaming

Algorithmic trading systems depend on real-time price feeds. Consuming raw broker WebSockets directly inside UI web applications creates tight coupling, duplicate connections, and potential main-thread lag during market volatility spikes.

2. Data Pipeline Architecture

Market Feed Decoupled Streaming Architecture
flowchart TD
    A[Upstox V3 Market Feed] -->|Binary Protobuf WSS| B[FastAPI Feed Ingestion Service]
    B -->|Decode FeedResponse| C[Normalize Price Ticks]
    C -->|Publish Tick Event| D[Redis Pub/Sub Bus]
    
    D -->|Subscribe channel:ticks| E[Strategy Evaluation Engine]
    D -->|Subscribe channel:ticks| F[Paper Trading PnL Tracker]
    D -->|Subscribe channel:ticks| G[Frontend WebSocket Server]
    
    G -->|Socket.IO Broadcast| H[React Dashboard UI]

Diagram 1: Data flow from broker binary WebSocket feed to Redis Pub/Sub event bus and consumer microservices.

3. Resilient Connection Lifecycle

Network interruptions on WebSocket feeds are inevitable. The connection manager enforces exponential backoff and dynamic channel resubscription:

WebSocket Reconnection State Machine
stateDiagram-v2
    [*] --> DISCONNECTED
    DISCONNECTED --> CONNECTING: Obtain Authorized WSS URL
    CONNECTING --> ACTIVE_FEED: Handshake & Subscription Ack
    ACTIVE_FEED --> RECONNECTING: Connection Dropped / Heartbeat Timeout
    RECONNECTING --> BACKOFF_WAIT: Exponential Backoff (1s, 2s, 4s...)
    BACKOFF_WAIT --> CONNECTING: Retry Attempt
    ACTIVE_FEED --> DISCONNECTED: Graceful Shutdown

Diagram 2: State transitions for connection recovery and exponential backoff retry handling.

4. Protobuf Ingestion Worker

# services/market_feed/websocket_consumer.py
import asyncio
import websockets
import MarketDataFeed_pb2 as pb
from redis.asyncio import Redis

class UpstoxFeedConsumer:
    def __init__(self, wss_url: str, redis_client: Redis):
        self.wss_url = wss_url
        self.redis = redis_client

    async def start(self):
        async with websockets.connect(self.wss_url) as ws:
            while True:
                message = await ws.recv()
                feed_response = pb.FeedResponse()
                feed_response.ParseFromString(message)
                
                # Normalize and publish to Redis
                for key, feeds in feed_response.feeds.items():
                    tick_payload = {
                        "instrument_key": key,
                        "ltp": feeds.ltp,
                        "vtt": feeds.vtt
                    }
                    await self.redis.publish("market:ticks", json.dumps(tick_payload))

5. Engineering Decisions & Tradeoffs

Decision Rationale Tradeoff
Redis Pub/Sub Event Bus Decouples feed ingestion from strategy evaluation and UI streaming. Pub/Sub does not persist historical messages (requires separate DB storage).
Protobuf Binary Decoding Reduces network payload size compared to verbose JSON strings. Requires compiled `.proto` schema files in Python build pipelines.
5s Notification Deduplication Prevents notification spam during rapid tick movements. Suppresses secondary alerts occurring within 5-second windows.

6. References