orderflow_backtest/docs/architecture.md

System Architecture

Overview

The current system is a streamlined, high-performance pipeline that streams orderflow from SQLite databases, aggregates trades into OHLC bars, maintains a lightweight depth snapshot, and serves visuals via a Dash web application. Inter-process communication (IPC) between the processor and visualizer uses atomic JSON files for simplicity and robustness.

High-Level Architecture

┌─────────────────┐   ┌─────────────────────┐   ┌──────────────────┐   ┌──────────────────┐
│  SQLite Files   │ → │   DB Interpreter    │ → │   OHLC/Depth     │ → │  Dash Visualizer │
│  (book,trades)  │   │  (stream rows)      │   │   Processor      │   │  (app.py)        │
└─────────────────┘   └─────────────────────┘   └─────────┬────────┘   └────────────▲─────┘
                                                          │                         │
                                                          │  Atomic JSON (IPC)      │
                                                          ▼                         │
                                                  ohlc_data.json, depth_data.json   │
                                                  metrics_data.json                 │
                                                                                    │
                                                                              Browser UI

Components

Data Access (db_interpreter.py)

• OrderbookLevel: dataclass representing one price level.
• OrderbookUpdate: container for a book row window with bids, asks, timestamp, and end_timestamp.
• DBInterpreter:
  • stream() -> Iterator[tuple[OrderbookUpdate, list[tuple]]] streams the book table with lookahead and the trades table in timestamp order.
  • Efficient read-only connection with PRAGMA tuning: immutable mode, query_only, temp_store=MEMORY, mmap_size, cache_size.
  • Batching constants: BOOK_BATCH = 2048, TRADE_BATCH = 4096.
  • Each yielded trades element is a tuple (id, trade_id, price, size, side, timestamp_ms) that falls within [book.timestamp, next_book.timestamp).

Processing (Modular Architecture)

Main Coordinator (ohlc_processor.py)

• OHLCProcessor(window_seconds=60, depth_levels_per_side=50): Orchestrates trade processing using composition
  • process_trades(trades): aggregates trades into OHLC bars and delegates CVD updates
  • update_orderbook(ob_update): coordinates orderbook updates and OBI metric calculation
  • finalize(): finalizes both OHLC bars and metrics data
  • cvd_cumulative (property): provides access to cumulative volume delta

Orderbook Management (orderbook_manager.py)

• OrderbookManager: Handles in-memory orderbook state with partial updates
  • Maintains separate bid/ask price→size dictionaries
  • Supports deletions via zero-size updates
  • Provides sorted top-N level extraction for visualization

Metrics Calculation (metrics_calculator.py)

• MetricsCalculator: Manages OBI and CVD metrics with windowed aggregation
  • Tracks CVD from trade flow (buy vs sell volume delta)
  • Calculates OBI from orderbook volume imbalance
  • Provides throttled updates and OHLC-style metric bars

Level Parsing (level_parser.py)

• Utility functions for normalizing orderbook level data:
  • normalize_levels(): parses levels, filtering zero/negative sizes
  • parse_levels_including_zeros(): preserves zeros for deletion operations
  • Supports JSON and Python literal formats with robust error handling

Inter-Process Communication (viz_io.py)

• File paths (relative to project root):
  • ohlc_data.json: rolling list of OHLC bars (max 1000).
  • depth_data.json: latest depth snapshot (bids/asks).
  • metrics_data.json: rolling list of OBI/TOT OHLC bars (max 1000).
• Atomic writes via temp files prevent partial reads by the Dash app.
• API:
  • add_ohlc_bar(...): append a new bar; trim to last 1000.
  • upsert_ohlc_bar(...): replace last bar if timestamp matches; else append; trim.
  • clear_data(): reset OHLC data to an empty list.

Visualization (app.py)

• Dash application with two graphs plus OBI subplot:
  • OHLC + Volume subplot with shared x-axis.
  • OBI candlestick subplot (blue tones) sharing x-axis.
  • Depth (cumulative) chart for bids and asks.
• Polling interval (500 ms) callback reads JSON files and updates figures resilently:
  • Caches last good values to tolerate in-flight writes/decoding errors.
  • Builds figures with Plotly dark theme.
• Exposed on http://localhost:8050 by default (host=0.0.0.0).

CLI Orchestration (main.py)

• Typer CLI entrypoint:
  • Arguments: instrument, start_date, end_date (UTC, YYYY-MM-DD), options: --window-seconds.
  • Discovers SQLite files under ../data/OKX matching the instrument.
  • Launches Dash visualizer as a separate process: uv run python app.py.
  • Streams databases sequentially: for each book row, processes trades and updates orderbook.

Data Flow

1. Discover and open SQLite database(s) for the requested instrument.
2. Stream book rows with one-row lookahead to form time windows.
3. Stream trades in timestamp order and bucket into the active window.
4. For each window:
   • Aggregate trades into OHLC using OHLCProcessor.process_trades.
   • Apply partial depth updates via OHLCProcessor.update_orderbook and emit periodic snapshots.
5. Persist current OHLC bar(s) and depth snapshots to JSON via atomic writes.
6. Dash app polls JSON and renders charts.

IPC JSON Schemas

• OHLC (ohlc_data.json): array of bars; each bar is [ts, open, high, low, close, volume].

• Depth (depth_data.json): object with bids/asks arrays: {"bids": [[price, size], ...], "asks": [[price, size], ...]}.

• Metrics (metrics_data.json): array of bars; each bar is [ts, obi_open, obi_high, obi_low, obi_close, tot_open, tot_high, tot_low, tot_close].

Configuration

• OHLCProcessor(window_seconds, depth_levels_per_side) controls aggregation granularity and depth snapshot size.
• Visualizer interval (500 ms) balances UI responsiveness and CPU usage.
• Paths: JSON files (ohlc_data.json, depth_data.json) are colocated with the code and written atomically.
• CLI parameters select instrument and time range; databases expected under ../data/OKX.

Performance Characteristics

• Read-only SQLite tuned for fast sequential scans: immutable URI, query_only, large mmap and cache.
• Batching minimizes cursor churn and Python overhead.
• JSON IPC uses atomic replace to avoid contention; OHLC list is bounded to 1000 entries.
• Processor throttles intra-window OHLC upserts and depth emissions to reduce I/O.

Error Handling

• Visualizer tolerates JSON decode races by reusing last good values and logging warnings.
• Processor guards depth parsing and writes; logs at debug/info levels.
• Visualizer startup is wrapped; if it fails, processing continues without UI.

Security Considerations

• SQLite connections are read-only and immutable; no write queries executed.
• File writes are confined to project directory; no paths derived from untrusted input.
• Logs avoid sensitive data; only operational metadata.

Testing Guidance

• Unit tests (run with uv run pytest):
  • OHLCProcessor: window boundary handling, high/low tracking, volume accumulation, upsert behavior.
  • Depth maintenance: deletions (size==0), top-N sorting, throttling.
  • DBInterpreter.stream: correct trade-window assignment, end-of-stream handling.
• Integration: end-to-end generation of JSON from a tiny fixture DB and basic figure construction without launching a server.

Roadmap (Optional Enhancements)

• Metrics: add OBI/CVD computation and persist metrics to a dedicated table.
• Repository Pattern: extract DB access into a repository module with typed methods.
• Orchestrator: introduce a Storage pipeline module coordinating batch processing and persistence.
• Strategy Layer: compute signals/alerts on stored metrics.
• Visualization: add OBI/CVD subplots and richer interactions.

---

This document reflects the current implementation centered on SQLite streaming, JSON-based IPC, and a Dash visualizer, providing a clear foundation for incremental enhancements.