Designing a Scalable Market Data Streaming Architecture

In my algo trading project, Growth Quantix, handling real-time market data efficiently is critical. This post details the architecture I designed to process live ticks, minimize latency, and serve data to both trading strategies and a frontend dashboard.

🛠️ Stack Used

This architecture is implemented using a modern, high-performance stack:

• Python FastAPI — Backend services and API layer
• Upstox WebSocket — Live market feed source
• Redis — In-memory cache + Pub/Sub messaging bus
• Background Workers — Data parsing and strategy execution
• Socket.IO — Real-time UI streaming to the client
• React Dashboard — Frontend consumer

🧩 High-Level System Architecture

The system adheres to a strict separation of concerns: Ingest → Process → Store → Stream → Consume.

              ┌──────────────────────────┐
              │        Upstox API        │
              │     (Market WebSocket)   │
              └────────────┬─────────────┘
                           │  Live Ticks
                           ▼
              ┌──────────────────────────┐
              │  Ingestion Service       │
              │ (Background FastAPI Job) │
              └────────────┬─────────────┘
                           │  Raw Tokens
                           ▼
              ┌──────────────────────────┐
              │ Processing & Mapping     │
              │ Token → Symbol mapping   │
              │ OHLC / %Change / LTP     │
              └────────────┬─────────────┘
                           │  Normalized ticks
                           ▼
         ┌──────────────────────────────────────┐
         │             Redis Layer              │
         │   • In-memory cache                  │
         │   • Pub/Sub channels                 │
         └────────────┬─────────────┬───────────┘
                      │             │
                      │             │
                      ▼             ▼
      ┌──────────────────────┐   ┌───────────────────────┐
      │  Trading Engine      │   │   Real-Time Dashboard │
      │  (Strategies)        │   │   (React + Socket.IO) │
      └──────────────────────┘   └───────────────────────┘
          

This pipeline ensures that:

• Ingestion is decoupled from consumption.
• Processing normalizes data before it hits the bus.
• Redis acts as the single source of truth for live state.

🔍 Detailed Data Flow Architecture

Below is a deeper view of the backend services interacting via the Redis bus.

┌────────────────────────────────────────────────────────────┐
│                    BACKEND SERVICES                        │
└────────────────────────────────────────────────────────────┘

 Upstox WebSocket Feed
──────────────────────────────────────────────────────────────
             │ ticks with instrument_token
             ▼
     ┌───────────────────────────────┐
     │  WebSocket Ingestion Worker   │
     │  (running in background)      │
     └───────────────────────────────┘
             │
             │ validate / reconnect / heartbeat
             ▼
     ┌───────────────────────────────┐
     │  Parsing + Mapping Layer      │
     │  token → symbol               │
     │  add OHLC/LTP/change%         │
     └───────────────────────────────┘
             │
             │ normalized structured tick
             ▼
     ┌───────────────────────────────┐
     │   Redis Cache + Pub/Sub       │
     └───────────────────────────────┘
        │                  │
        │                  │
        ▼                  ▼
┌────────────────┐   ┌─────────────────────────────┐
│  Trading Logic │   │  Socket.IO Broadcast Layer  │
│  (strategies)  │   │  push to connected clients  │
└────────────────┘   └─────────────────────────────┘
                          │
                          ▼
                 React Frontend Dashboard
          

In this setup:

• Upstox feeds the background service.
• Redis acts as the central nervous system.
• Socket.IO bridges the gap to the frontend.
• The Trading Engine consumes the same data bus as the UI, ensuring consistency.

🏗️ Why I Designed It This Way

This design evolved from real constraints in my project:

"Instead of each consumer connecting to the broker, everything connects to the Redis live stream layer."

• Connection Limits: Upstox limits connection counts, so I use a single admin WebSocket.
• High Throughput: Handling thousands of ticks/sec means avoiding direct DB writes for every tick.
• Unified Data: Strategies and UI need the exact same data, making a Pub/Sub bus ideal.
• Decoupling: Independent services prevent a UI crash from stopping the trading engine.

🧠 Key Components Explained

1️⃣ Upstox Ingestion Worker

Runs independently, connects with the admin token, and handles resubscriptions and exponential backoff strategies for robustness.

2️⃣ Token–Symbol Mapping

Raw feeds often send tokens (integers) instead of symbols to save bandwidth. My instrument service loads a CSV/instrument dump on startup, builds a token → symbol dictionary, and keeps it in Redis for fast O(1) lookups during ingestion.

3️⃣ Redis Cache + Pub/Sub

Acts as:

• Hot Cache: Stores the latest price (LTP) for immediate access.
• Pub/Sub Bus: Distributes ticks to multiple subscribers (Strategy Engine, Dashboard).
• Buffer: Handles burst traffic without blocking.

4️⃣ Strategy Engine

Consumes the live feed to generate buy/sell signals, track P&L, and manage risk.

5️⃣ Frontend Dashboard

Built with React and Socket.IO, it receives live tick cards, scanner updates, top gainers/losers, and open position updates in real-time.

🚀 Future Improvements

I’m considering adding:

• Kafka for long-term scaling and log replay.
• Candlestick Aggregation Service for generating 1m, 5m bars from ticks.
• Persistence Layer (TimescaleDB) for historical analysis.
• Anomaly Detection alerts for data feed issues.

🏁 Conclusion

This architecture now powers my algo trading experiments, live dashboard, and real-time streaming UI. It is built with FastAPI, Redis, Socket.IO, React, and Upstox WebSocket. The design is modular, fault-tolerant, and scalable — and most importantly, I built it step by step while working full-time.