orderflow_backtest/docs/modules/viz_io.md

# Module: viz_io

## Purpose
The `viz_io` module provides atomic inter-process communication (IPC) between the data processing pipeline and the visualization frontend. It manages JSON file-based data exchange with atomic writes to prevent race conditions and data corruption.

## Public Interface

### Functions
- `add_ohlc_bar(timestamp, open_price, high_price, low_price, close_price, volume)`: Append new OHLC bar to rolling dataset
- `upsert_ohlc_bar(timestamp, open_price, high_price, low_price, close_price, volume)`: Update existing bar or append new one
- `clear_data()`: Reset OHLC dataset to empty state
- `add_metric_bar(timestamp, obi_open, obi_high, obi_low, obi_close)`: Append OBI metric bar
- `upsert_metric_bar(timestamp, obi_open, obi_high, obi_low, obi_close)`: Update existing OBI bar or append new one
- `clear_metrics()`: Reset metrics dataset to empty state
- `set_depth_data(bids, asks)`: Update current orderbook depth snapshot

### Constants
- `DATA_FILE`: Path to OHLC data JSON file
- `DEPTH_FILE`: Path to depth data JSON file  
- `METRICS_FILE`: Path to metrics data JSON file
- `MAX_BARS`: Maximum number of bars to retain (1000)

## Usage Examples

### Basic OHLC Operations
```python
import viz_io

# Add a new OHLC bar
viz_io.add_ohlc_bar(
    timestamp=1640995200000,  # Unix timestamp in milliseconds
    open_price=50000.0,
    high_price=50100.0,
    low_price=49900.0,
    close_price=50050.0,
    volume=125.5
)

# Update the current bar (if timestamp matches) or add new one
viz_io.upsert_ohlc_bar(
    timestamp=1640995200000,
    open_price=50000.0,
    high_price=50150.0,  # Updated high
    low_price=49850.0,   # Updated low
    close_price=50075.0,  # Updated close
    volume=130.2         # Updated volume
)
```

### Orderbook Depth Management
```python
# Set current depth snapshot
bids = [[49990.0, 1.5], [49985.0, 2.1], [49980.0, 0.8]]
asks = [[50010.0, 1.2], [50015.0, 1.8], [50020.0, 2.5]]

viz_io.set_depth_data(bids, asks)
```

### Metrics Operations
```python
# Add Order Book Imbalance metrics
viz_io.add_metric_bar(
    timestamp=1640995200000,
    obi_open=0.15,
    obi_high=0.22,
    obi_low=0.08,
    obi_close=0.18
)
```

## Dependencies

### Internal
- None (standalone utility module)

### External
- `json`: JSON serialization/deserialization
- `pathlib`: File path handling
- `typing`: Type annotations
- `tempfile`: Atomic write operations

## Data Formats

### OHLC Data (`ohlc_data.json`)
```json
[
  [1640995200000, 50000.0, 50100.0, 49900.0, 50050.0, 125.5],
  [1640995260000, 50050.0, 50200.0, 50000.0, 50150.0, 98.3]
]
```
Format: `[timestamp, open, high, low, close, volume]`

### Depth Data (`depth_data.json`)
```json
{
  "bids": [[49990.0, 1.5], [49985.0, 2.1]],
  "asks": [[50010.0, 1.2], [50015.0, 1.8]]
}
```
Format: `{"bids": [[price, size], ...], "asks": [[price, size], ...]}`

### Metrics Data (`metrics_data.json`)
```json
[
  [1640995200000, 0.15, 0.22, 0.08, 0.18],
  [1640995260000, 0.18, 0.25, 0.12, 0.20]
]
```
Format: `[timestamp, obi_open, obi_high, obi_low, obi_close]`

## Atomic Write Operations

All write operations use atomic file replacement to prevent partial reads:

1. Write data to temporary file
2. Flush and sync to disk
3. Atomically rename temporary file to target file

This ensures the visualization frontend always reads complete, valid JSON data.

## Performance Characteristics

- **Bounded Memory**: OHLC and metrics datasets limited to 1000 bars max
- **Atomic Operations**: No partial reads possible during writes
- **Rolling Window**: Automatic trimming of old data maintains constant memory usage
- **Fast Lookups**: Timestamp-based upsert operations use list scanning (acceptable for 1000 items)

## Testing

Run module tests:
```bash
uv run pytest test_viz_io.py -v
```

Test coverage includes:
- Atomic write operations
- Data format validation
- Rolling window behavior
- Upsert logic correctness
- File corruption prevention
- Concurrent read/write scenarios

## Known Issues

- File I/O may block briefly during atomic writes
- JSON parsing errors not propagated to callers
- Limited to 1000 bars maximum (configurable via MAX_BARS)
- No compression for large datasets

## Thread Safety

All operations are thread-safe for single writer, multiple reader scenarios:
- Writer: Data processing pipeline (single thread)
- Readers: Visualization frontend (polling)
- Atomic file operations prevent corruption during concurrent access