orderflow_backtest/docs/CHANGELOG.md

# Changelog

All notable changes to the Orderflow Backtest System are documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Added
- Comprehensive documentation structure with module-specific guides
- Architecture Decision Records (ADRs) for major technical decisions
- CONTRIBUTING.md with development guidelines and standards
- Enhanced module documentation in `docs/modules/` directory
- Dependency documentation with security and performance considerations

### Changed
- Documentation structure reorganized to follow documentation standards
- Improved code documentation requirements with examples
- Enhanced testing guidelines with coverage requirements

## [2.0.0] - 2024-12-Present

### Added
- **Simplified Pipeline Architecture**: Streamlined SQLite → OHLC/Depth → JSON → Dash pipeline
- **JSON-based IPC**: Atomic file-based communication between processor and visualizer
- **Real-time Visualization**: Dash web application with 500ms polling updates
- **OHLC Aggregation**: Configurable time window aggregation with throttled updates
- **Orderbook Depth**: Real-time depth snapshots with top-N level management
- **OBI Metrics**: Order Book Imbalance calculation with candlestick visualization
- **Atomic JSON Operations**: Race-condition-free data exchange via temp files
- **CLI Orchestration**: Typer-based command interface with process management
- **Performance Optimizations**: Batch reading with optimized SQLite PRAGMA settings

### Changed
- **Architecture Simplification**: Removed complex repository/storage layers
- **Data Flow**: Direct streaming from database to visualization via JSON
- **Error Handling**: Graceful degradation with cached data fallbacks
- **Process Management**: Separate visualization process launched automatically
- **Memory Efficiency**: Bounded datasets prevent unlimited memory growth

### Technical Details
- **Database Access**: Read-only SQLite with immutable mode and mmap optimization
- **Batch Sizes**: BOOK_BATCH=2048, TRADE_BATCH=4096 for optimal performance
- **JSON Formats**: Standardized schemas for OHLC, depth, and metrics data
- **Chart Architecture**: Multi-subplot layout with shared time axis
- **IPC Files**: `ohlc_data.json`, `depth_data.json`, `metrics_data.json`

### Removed
- Complex metrics storage and repository patterns
- Strategy framework components
- In-memory snapshot retention
- Multi-database orchestration complexity

## [1.0.0] - Previous Version

### Features
- **Orderbook Reconstruction**: Build complete orderbooks from SQLite database files
- **Data Models**: Core structures for `OrderbookLevel`, `Trade`, `BookSnapshot`, `Book`
- **SQLite Repository**: Read-only data access for orderbook and trades data
- **Orderbook Parser**: Text parsing with price caching optimization
- **Storage Orchestration**: High-level facade for book building
- **Basic Visualization**: OHLC candlestick charts with Qt5Agg backend
- **Strategy Framework**: Basic strategy pattern with `DefaultStrategy`
- **CLI Interface**: Command-line application for date range processing
- **Test Suite**: Unit and integration tests

### Architecture
- **Repository Pattern**: Clean separation of data access logic
- **Dataclass Models**: Lightweight, type-safe data structures
- **Parser Optimization**: Price caching for performance
- **Modular Design**: Clear separation between components

---

## Migration Guide

### Upgrading from v1.0.0 to v2.0.0

#### Code Changes Required

1. **Strategy Constructor**
   ```python
   # Before (v1.0.0)
   strategy = DefaultStrategy("BTC-USDT", enable_visualization=True)
   
   # After (v2.0.0)  
   strategy = DefaultStrategy("BTC-USDT")
   visualizer = Visualizer(window_seconds=60, max_bars=500)
   ```

2. **Main Application Flow**
   ```python
   # Before (v1.0.0)
   strategy = DefaultStrategy(instrument, enable_visualization=True)
   storage.build_booktick_from_db(db_path, db_date)
   strategy.on_booktick(storage.book)
   
   # After (v2.0.0)
   strategy = DefaultStrategy(instrument)
   visualizer = Visualizer(window_seconds=60, max_bars=500)
   
   strategy.set_db_path(db_path)
   visualizer.set_db_path(db_path)
   storage.build_booktick_from_db(db_path, db_date)
   strategy.on_booktick(storage.book)
   visualizer.update_from_book(storage.book)
   ```

#### Database Migration
- **Automatic**: Metrics table created automatically on first run
- **No Data Loss**: Existing orderbook and trades data unchanged
- **Schema Addition**: New `metrics` table with indexes added to existing databases

#### Benefits of Upgrading
- **Memory Efficiency**: >70% reduction in memory usage
- **Performance**: Faster processing through persistent metrics storage
- **Enhanced Analysis**: Access to OBI and CVD financial indicators
- **Better Visualization**: Multi-chart display with synchronized time axis
- **Improved Architecture**: Cleaner separation of concerns

#### Testing Migration
```bash
# Verify upgrade compatibility
uv run pytest tests/test_main_integration.py -v

# Test new metrics functionality  
uv run pytest tests/test_storage_metrics.py -v

# Validate visualization separation
uv run pytest tests/test_main_visualization.py -v
```

---

## Development Notes

### Performance Improvements
- **v2.0.0**: >70% memory reduction, batch processing, persistent storage
- **v1.0.0**: In-memory processing, real-time calculations

### Architecture Evolution
- **v2.0.0**: Streaming processing with metrics storage, separated visualization
- **v1.0.0**: Full snapshot retention, integrated visualization in strategies

### Testing Coverage
- **v2.0.0**: 27 tests across 6 files, integration and unit coverage
- **v1.0.0**: Basic unit tests for core components

---

*For detailed technical documentation, see [docs/](../docs/) directory.*