orderflow_backtest/docs/modules/viz_io.md

Module: viz_io

Purpose

The viz_io module provides atomic inter-process communication (IPC) between the data processing pipeline and the visualization frontend. It manages JSON file-based data exchange with atomic writes to prevent race conditions and data corruption.

Public Interface

Functions

• add_ohlc_bar(timestamp, open_price, high_price, low_price, close_price, volume): Append new OHLC bar to rolling dataset
• upsert_ohlc_bar(timestamp, open_price, high_price, low_price, close_price, volume): Update existing bar or append new one
• clear_data(): Reset OHLC dataset to empty state
• add_metric_bar(timestamp, obi_open, obi_high, obi_low, obi_close): Append OBI metric bar
• upsert_metric_bar(timestamp, obi_open, obi_high, obi_low, obi_close): Update existing OBI bar or append new one
• clear_metrics(): Reset metrics dataset to empty state
• set_depth_data(bids, asks): Update current orderbook depth snapshot

Constants

• DATA_FILE: Path to OHLC data JSON file
• DEPTH_FILE: Path to depth data JSON file
• METRICS_FILE: Path to metrics data JSON file
• MAX_BARS: Maximum number of bars to retain (1000)

Usage Examples

Basic OHLC Operations

import viz_io

# Add a new OHLC bar
viz_io.add_ohlc_bar(
    timestamp=1640995200000,  # Unix timestamp in milliseconds
    open_price=50000.0,
    high_price=50100.0,
    low_price=49900.0,
    close_price=50050.0,
    volume=125.5
)

# Update the current bar (if timestamp matches) or add new one
viz_io.upsert_ohlc_bar(
    timestamp=1640995200000,
    open_price=50000.0,
    high_price=50150.0,  # Updated high
    low_price=49850.0,   # Updated low
    close_price=50075.0,  # Updated close
    volume=130.2         # Updated volume
)

Orderbook Depth Management

# Set current depth snapshot
bids = [[49990.0, 1.5], [49985.0, 2.1], [49980.0, 0.8]]
asks = [[50010.0, 1.2], [50015.0, 1.8], [50020.0, 2.5]]

viz_io.set_depth_data(bids, asks)

Metrics Operations

# Add Order Book Imbalance metrics
viz_io.add_metric_bar(
    timestamp=1640995200000,
    obi_open=0.15,
    obi_high=0.22,
    obi_low=0.08,
    obi_close=0.18
)

Dependencies

Internal

• None (standalone utility module)

External

• json: JSON serialization/deserialization
• pathlib: File path handling
• typing: Type annotations
• tempfile: Atomic write operations

Data Formats

OHLC Data (ohlc_data.json)

[
  [1640995200000, 50000.0, 50100.0, 49900.0, 50050.0, 125.5],
  [1640995260000, 50050.0, 50200.0, 50000.0, 50150.0, 98.3]
]

Format: [timestamp, open, high, low, close, volume]

Depth Data (depth_data.json)

{
  "bids": [[49990.0, 1.5], [49985.0, 2.1]],
  "asks": [[50010.0, 1.2], [50015.0, 1.8]]
}

Format: {"bids": [[price, size], ...], "asks": [[price, size], ...]}

Metrics Data (metrics_data.json)

[
  [1640995200000, 0.15, 0.22, 0.08, 0.18],
  [1640995260000, 0.18, 0.25, 0.12, 0.20]
]

Format: [timestamp, obi_open, obi_high, obi_low, obi_close]

Atomic Write Operations

All write operations use atomic file replacement to prevent partial reads:

1. Write data to temporary file
2. Flush and sync to disk
3. Atomically rename temporary file to target file

This ensures the visualization frontend always reads complete, valid JSON data.

Performance Characteristics

• Bounded Memory: OHLC and metrics datasets limited to 1000 bars max
• Atomic Operations: No partial reads possible during writes
• Rolling Window: Automatic trimming of old data maintains constant memory usage
• Fast Lookups: Timestamp-based upsert operations use list scanning (acceptable for 1000 items)

Testing

Run module tests:

uv run pytest test_viz_io.py -v

Test coverage includes:

• Atomic write operations
• Data format validation
• Rolling window behavior
• Upsert logic correctness
• File corruption prevention
• Concurrent read/write scenarios

Known Issues

• File I/O may block briefly during atomic writes
• JSON parsing errors not propagated to callers
• Limited to 1000 bars maximum (configurable via MAX_BARS)
• No compression for large datasets

Thread Safety

All operations are thread-safe for single writer, multiple reader scenarios:

• Writer: Data processing pipeline (single thread)
• Readers: Visualization frontend (polling)
• Atomic file operations prevent corruption during concurrent access